Big data should lead to big dreams

Organizations are collecting more and more data every single minute of every day. It’s no secret that Facebook, for example, processes 2.5 billion pieces of content and over 500 terabytes of data each day, pulling in 2.7 billion Likes and 300 million photos per diem. Facebook also scans a whopping 105 terabytes of data each half hour.

Of course Big Data is nothing new. It was relatively enormous all the way back to the 1960′s when NASA took a shot at the moon. It shouldn’t have been a surprise when Google gave everyone the proverbial heart attack when they revealed they churn almost the entire Internet every couple of days to meet our search needs.

Despite the hype, Big Data didn’t happen all at once…data just got bigger.

So we know that companies manage, manipulate and extrapolate information from ever larger amounts of data. But what we don’t know is whether they are asking anything bigger from it all. They should be.

Limitless information, limited imagination

A customer-focused business with Big Data in its grasp has an unparalleled source of knowledge from an increasing number of sources now; mobile data, social data, transactional data, locational data, financial data, family data, medical data, carbon footprint and consumption data. We even have data about data in the form of log data, as Tesla showed us in rebutting the NY Times.

What’s more, a similar increase of that information is being collected in real-time with lots of integration challenges. (But often stored very traditionally and processed in batch. What use is that for operational decisions?)

But with all this information at hand I’m seeing a worrying trend of organizations still asking the same questions of it and receiving the same answers as before, just with a little bit more data support behind it. When you combine social + mobile + medical + financial + family you get quite a bit more than just demographic segmentation. Too often, that’s the apparent limit to where current thinking goes in customer service and marketing.

Limitless information, limited processing

There is another angle I want to work on here, and something I’ve mentioned aplenty elsewhere. This amount of data requires a modicum of processing power. Whilst you can move it all to the Cloud and leave it to another provider to churn the numbers for you there’s nothing to stop you from applying a little distributed magic yourself and using the idle processors sitting on every desk in the organization. In fact, what if AT&T managed to work out how to use idle clock time in every one of its smartphone to process its own data from its customers ?

SETI@Home famously did this by allowing 3 million users to assign their PCs and PlayStations to solve computational data from radio telescopes. It’s not such a far fetched notion for a business such as mobile provider or a bank to do exactly that via an app.

Limitless information, unlimited possibilities

Big Data actually demands of us big questions. It demands us to think bigger than what we’re currently doing. TIBCO CTO Matt Quinn put it well when he said we should be asking ourselves,

What’s the No. 1 question that you’ve wanted to have answered but you were always told it was impossible? Start from there. Don’t start with what data you have, start with the important question.

The term Big Data may just be part of the industrial hype cycle since it’s all about the Data, but the Big part should be a reminder to dream big.