Big Data: the shift from ocean to stream will hit you like a tsunami

So the fabled Mary Meeker powerpoint tome hit the internet at the D11 conference this week and if you have the patience to wade through the stats the messages are pretty clear: it’s a wearable, shareable future for mankind.

But there’s a shift in how people are willing to share now.

Share it. Delete it.

500m photos are shared on average every day. Of those, 150m are shared via Snapchat, a very nifty little semi-social network app that deletes the photo from the server after it’s been viewed by the recipient. Users can dial in just how long that photo is available for after viewing but once it’s gone it’s gone for good (at least in theory).

And this is a definite trend which will continue for the next few years. As Gen X makes way for Gen Y, the latter doesn’t want their digital footprint to linger for months and years for organizations to benefit from (or leave a trail that can be used negatively) and services like Snapchat offer a way out of the ocean mentality that Gen X has built its empire on. Gen Y is a stream culture as Toby Beresford, a social strategist and founder of Leaderboarded succinctly put it.

Big Data or Disappearing Data?

So where does this leave the Big Data promise? The premise of Big Data analytics is built on having use of massive amounts of data that is constantly being generated and available for weeks, months, years. Big Data was built for the ocean culture of Gen X. But if the mechanism behind Snapchat take off and proliferate across other networks, organizations may find themselves in a difficult position; they’ll have no ocean from which to trawl data, it’ll have slowed to a trickle. Which means they have to react faster and in real-time to data that could potentially appear and disappear within seconds. And with privacy being an ongoing concern for users, having access to that data and being able to store it outside of those parameters could very well breach regulations.

For personal data that we’ve assumed will be around when needed, this could mean a rethinking of event processing and process management.

This is obviously a small use case example and there are many others where the ocean mentality fits the model perfectly (for now, until volumes grow and velocity increases). But with people already crying that Big Data has hit the infamous Trough of Disillusionment the industry needs to re-examine what it means to harness data, analyze it and act on it.

Because in some cases now, you no longer have months to play data scientist, you have seconds.