With all the talk about big data analysis, it is easy to forget that the vast majority of organizations are still not leveraging more basic transactional and structured data analysis well enough to be thinking beyond that.  As with anything, it is important that you get the basic and “big stuff” in terms of impact right before trying to go to the next level.  Big data as the term is being used today refers primarily to the more unstructured and interaction types of data that are being generated in such massive volumes today via social networking, web traffic, mobile apps, etc.  Even with the most advanced big data analysis techniques, you still eventually end up with more traditional structured analysis to fully understand the issue once a pattern is detected; the big data techniques are designed to get through the huge volumes and unstructured nature of the data to identify patterns which become a starting point for further analysis.  Everyone wants to leverage all interactions in their analysis, but you need to master the basics before getting caught up in the media frenzy surrounding “big data”.

After more than twenty years building data repositories to support advanced analytics, modeling, and scoring, I am still amazed at how few organizations understand what information is required to realize the real benefits of their data stores.  Most organizations are missing the pot of gold at the end of the rainbow because they have not finished the job; they have expended 80%-90% of the total effort but are missing the piece that could increase their benefits by an order of magnitude or more.

The majority of organizations of reasonable size likely have a data repository of some form that contains basic transactional and customer data and possibly even information on customer contacts or external data.  That is where most organizations leave it and expect to get the large benefits of being able to access and analyze their data.  The problem is that the data is unusable for most types of analysis, and only the most sophisticated users who have the knowledge and ability to access and form the data appropriately can get much benefit from it at all.

So what is the missing information?  It is the dimensional summaries of your data at the customer level - whether that is a business, household, or individual consumer.  This is similar to the data that may exist to some degree in OLAP or reporting summaries you may already have, but it is far more exhaustive and granular.  For example, you may need summaries by customer of their internal activity by timeframe, product group, marketing channel, etc. and combinations of these dimensions.  For the metrics, you will likely be summarizing counts, dollars, minimums, maximums, first, last, etc. for the dimensional combinations.  The important distinction from traditional summaries is the breadth, completeness, and granularity.

As you can see, the number of variables that you create in this summary layer can be quite large – hundreds to even thousands.  A three dimensional OLAP cube with ten values per dimension and a single metric has one thousand total dimensional/metric combinations; a customer summary will likely have several orders of magnitude greater combinations, and the data is at a far more granular level – the individual customer.  This is what is needed to unlock the true potential of your data.

If you want to perform customer level analysis and prediction of any kind, you need data summarized at the customer level.  You want the most complete picture possible of the individual customer to determine what influences their behavior. This is the data needed to support advanced analytics, and without it, you will generate only a tiny fraction of your potential benefits.

About Dave Fickes
Dave Fickes is Clario’s Vice President of Integration and has over 25 years’ experience designing, implementing, and using large scale marketing data platforms that support advanced analytics.  Having spent his career straddling both the analytic and IT worlds, he uniquely understands what is needed from both sides to realize the true value hidden within an organization's data.