Big Data that actually works

I had the pleasure of seeing a presentation by The Nielsen Company (NLSN) as part of TIBCO’s sponsorship of the 2012 Gartner Symposium ITxpo in Orlando, Florida. Ken Rabolt, Chief Data Architect to the CTO of Nielsen presented on Nielsen’s move from independent IT systems spanning 110 countries to a consolidated market research platform that supports a single view of all customers and their research across all geographies. That’s a huge sentence and a huge undertaking.

If you’ve never experienced that kind of effort, it is an amazing feat that can kill the business if it isn’t done carefully and deliberately. But Nielsen knew that we live in a very globalized world that still has plenty of local character. Nielsen’s information systems need to reflect both of those facts.

The Nielsen Company

When I think of Nielsen, I automatically think of Nielsen Ratings, the well-known measure of radio and television popularity and demographics from my childhood. Today’s Nielsen Company is a much broader, much more data analytics-focused market research company. While they still manage research into what consumers watch (but now across television, online and mobile screens), they have a thriving business in analyzing what consumers buy. More on that in a moment.

In the ‘what-they-watch’ business, rather than collecting data just from individual samples (‘Nielsen Families’), they are now collecting the raw data directly from, for example, set top boxes used to deliver digital entertainment. As this type of data becomes more available, Nielsen is handling increasingly large amounts of data coming very, very quickly.

On the consumer markets side, Nielsen measures retail transactional data and consumer behavior across the consumer packaged goods industry. This reflects an enormous amount of data as well, and even more diverse data, and is only rising in volume and velocity as more retailers move toward real-time point of sale systems.

Globalized research

The globalization of many consumer packaged goods companies means that Nielsen has an enormous role in ‘harmonizing’ data that is increasingly more varied, ge0-specific and granular. Raw data can reflect misleading trends if it isn’t interpreted against seasonality, culture, weather and a host of other factors. It is part science and part dark art.

Most globalized companies, Nielsen’s customers, operate ‘war rooms’ where critical decision are made based on Nielsen’s harmonized input. Nestlé, P&G, Unilever many other companies need to ‘follow’ products across not just geographies, but also their many branding and advertising channels like television, in-store and social. Data integration is the key to make everything work and Nielsen’s systems are at the forefront of the integration that makes Big Data ‘work’ and not just interesting discoveries.

Not surprisingly, Nielsen’s systems are stretched and stressed by the changes taking place in mobile and social technology. They are constantly driving toward higher and higher levels of data integration, analytics, and reporting. The more data they take from their research, the more opportunities arise to integrate that information into more granular segmentation of consumers by lifestyle, geography and other factors.

Answers on Demand

Nielsen operates at the center of a virtuous data cycle that puts them in the lead amoung the many companies racing to be Big Data powerhouses. They provide Answers on Demand, the platform name and an appropriate description of what they do. It is a data integration dream come true that few companies can boast.

Their focus on data integration allows Nielsen’s systems to be front-ended (the input data, whether local, regional or international) and back-ended (where customer take the output, the Nielsen product) in standardized ways that allow for any source and any destination, but still make sense internally for systems operation and maintenance. Inside Nielsen, processes, workflows and analytics are streamlined and efficient, even while local algorithms are applied for each customer and their geographies, segments, etc.

Nielsen’s platform is the cutting edge of Big Data, a term they don’t even use, and goes well beyond the hype of Hadoop and simple batch processing of large data sets. Nielsen sits at the epicenter of what drives the economy from the customer and product perspectives. They’re an excellent example of where technology is headed and case study in the benefits of putting enormous focus on data integration ahead of the shiny objects that are hyped but only part of the story.

Below is the Nielsen logical architectural.

Google+

Christopher Taylor