Analytics and Big Data

Have been meaning to write this post for a long time. It’s been almost 4 years since I started working on Hadoop and other Big Data related technologies. I have enjoyed and loved every bit of it so far. Being hands on gives you an edge on how you architect, design and strategize with your data analytics and insights initiatives.

Let me start off by saying “Big Data is a beginning but not a silver bullet for all your problems”. Big Data has always been around in different formats and different places. What changed over the last few years is that people started to realize that Data insights will help them reshape their business. From a technology standpoint we have lot of tools to work with when we talk about data. Historically Data warehouse was the only way to store and retrieve large sets of data. The challenges that came with traditional BI were technology, talent and moving beyond reports.

In the initial days of Big Data world, i.e 2008 or 2009 this was mainly accessible only to developers in the form of Map Reduce or Pig. Over time non programmers have started access to this through R or SQL on top of HIVE. However, if you really need to harness the power of Hadoop you might want to consider MapReduce.

In mid 2011 I had the opportunity to present about the integration of webanalytics and Hadoop at the Techweek Chicago. A lot was learned through this experience. First and foremost its not an easy task to change the analytics culture and especially with new technologies that haven’t been proved in the market.

Continuing on this we presented to a group of key executives at the Gartner peer forum. Again this was interesting since we were way ahead of the game when we looked at others in the industry. This forum also proved that the direction that we took in 2009 and 2010 was very good. Most of this credit goes to our CTO who had a great vision and made that possible through the opportunities within the organization.

In 2012 I met with the #measure community through the ACCELERATE conference that was started by Eric Peterson of Web Analytics Demystified. This was a fun conference to be at. The format was to present 10 tips in 20mins. me along with my product lead Jami presented our key learnings and provided great advice to a large group of people who are really good at analytics but not Big Data/Hadoop usage.

SlideShare
Edit descriptionwww.slideshare.net

I was invited to speak at the Big Data Innovation conference in Bangalore in 2013. This was an interesting presentation since I was able to show the entire life cycle of our infrastructure development and provided some key guidelines to successful implementation and usage. The crowd in Bangalore was very different. What we saw in 2009 in US is what I am seeing in Bangalore in 2013. A lot of curiosity and interest in the Big Data world. Everyone wants to get their hands on this. There is a long way they need to go in terms of using it for the right reason and right place. Talent is amazing and it will just grow over time.

My advice to tech folks is that be active in the community and contribute when possible.