We’re a funny lot. For years we’ve been talking about data and process as though they were two different things. Guess what…they’re not. Process involves the activities of getting work done and data both describes process and is also the beginning and end points of work. We consume data in processes and we produce data in processes. Data kicks off a process and data ends a process. They are inseparable.
Wait a minute…does that mean digging a hole in the ground is somehow data? Does it consume data and produce data? Does data kick it off and end its process? Absolutely. The timing, location, size, depth, rationale and value of having that hole may not have been digitally captured in the past, but every factor that goes into producing and using that hole is absolutely in the realm of data. It always was.
Where big data factors in
What’s happening, and this is where big data enters the story, is that we are now discovering the data that was uncaptured about hole digging and a myriad of other things. The explosion of data that we’re seeing isn’t from ‘new’ data, but from data that was previously unseen…uncaptured, untapped, unused. Invisible but absolutely in existence.
The way to dig that hole may have been locked in the mind of that worker or given in a hole digging class as unstructured data (from his supervisor as oral instruction). Even the hard work of digging can also be considered data. As we move to automate and machines become a bigger part of our lives, that unstructured data quickly becomes very important. It becomes the instruction manual for how machines operate, optimize and help us dig better, smarter, more efficient holes.
Where process is about to change
And this is why we’re on the verge of something enormous. We call it big data but in reality we’re at the front edge of the digitizing of our world. What tipped the scales in data’s favor were a few advances over the past two decades:
- Fast drop in data storage and retrieval cost
- Quick rise in computational power and techniques
- Easy ways to share information, like the Internet
- Rapid globalization and the need to compete against cheap labor with something smarter
Along the way, Cloud, Mobile and Social happened just to make it interesting, but that’s essentially the landscape we face today. Smart companies are embracing what’s coming next and becoming data and process powerhouses. Watch the progress of Nielsen and others and you’ll see exactly where the global economy is headed.
Thanks, Peter Schooff, for asking the question that inspired this.
All Things D
ARS Technica
Engadget
GigaOM
Mashable
TechCrunch
The Verge
Venture Beat
Wired
Chris Brogan
Brian Solis
Chris Dixon
Clay Shirky Blog
HBR Blog
IT Redux
Jeremiah Owyang
Radar O'Reilly
Seth Godin Blog
SocialMedia Today
Solve for Interesting
The TIBCO Blog
Lifehacker
Chris.. fascinating…
I agree they are inseparable. You cannot process without data and if you fail to generate data during processing, others will be hobbled trying to do their processing.
My focus these days is on big data.
I have a customer who collected 124,000,000 transactions and did not know what to do with the data. They were adding new 3 Tb drives every few weeks. We trimmed it back after careful analysis and decision-making re what data they would be needing and what data they would not be needing. No real worries about getting it right, we know we can always go back an reprocess.
Recently, I was involved in a municipal election and built a multi-dimensional Kbase. (state level legislation, municipal by-laws, staff, long term development plan, all streets with all residents at houses on these streets, all businesses, all contracts awarded/pending, all community sites. Basically, it’s an inventory of everything in and around and about the municipality and there are about 50,000 documents with data coming from multiple, separate relational data bases, newspapers etc.
Challenge - how to consolidate data from multiple, separate record management systems to that we can do free-form/connect-the-dots searches across the entire space.
Solution: Kbase with indexing
I know it will work because I just recently bought for only $99 an indexing solution that lets me find anything across the 30 or so e-mail accounts I have comprising, I could not believe it, a total of 335,000 stored e-mails. I never knew how many e-mails I had, just that I could never find anything I needed when I needed it.
So, yes, let’s do data mining/analytics/process generation but big data goes way beyond this - you need a dashboard of dashboards to view all of your big data and this means having a Kbase that starts with a free-form canvas or sheet, allows you to build multiple hierarchical trees with any mix of structured/unstructured data (text, images, video/audio) and Rule #1 is you want/need to see ALL of this at one screen.
We have had great success with Russian-doll and “rolodex” approaches to organizing data in a graphic environment.
KBases with indexing do things SQL cannot - Look for Wendy’s when your name is MacDonalds and you want to place an outlet where Wendy’s is not, and you will see not only where they are not, but where they are. SQL only tells you what matches your queries.