More data scientists or less complex big data applications?

In Venture Beat yesterday, Meghan Kelly wrote up an article saying that data scientists, “…may soon become one of the most sought-after people in your industry.” with an accompanying infographic (see below). As evidence, Kelly said that data scientist job postings increased 15,000 percent between 2011 and 2012 alone.

I think she has it wrong for three reasons.

Numbers are skewed by Big Data hype and early growth

The first reason Kelly has it wrong has to do with the snapshot in time she’s choosing…2011 to 2012. When the hype around big data kicked into high gear a couple of years ago and we invented the word “data scientist”, we were breaking new ground. The job’s high level of attention is a factor of its newness more than its explosive growth. 1500% growth can be as simple as going from 1 to 1500. It isn’t quite that simple, but you get the point…

Many more companies are getting into the act

Data scientists have always walked among us, they just didn’t have that title and they worked in just a handful of markets. Companies like Nielsen have been crunching very large data sets to find consumer patterns for decades, but they didn’t call it big data. They still don’t.

There was no market for software to solve the problem very few companies had. That changed when advances in technology allow more companies to handle far more data far more easily, causing everyone to get into the act. But just like computerization of the back office in the 70′s, the skill sets to get us through this transition won’t necessarily survive the transition for the third reason…

Applications and automation will simplify much of the complexity

We’re in a phase where front-end applications are scrambling to catch up with the relatively new found capability to distribute storage and processing across many machines (Hadoop clusters, for example). Hadoop is not a polished application with clean interfaces. For now, it takes coder/analytics people to run it well, thus the need for data scientists…for now.

However, a new breed of applications are launching now that allow business users to create their own big data applications in the cloud with a much lower level of IT and data scientist support. The future looks like data scientists working at software companies that sell to many customers, much like finance, marketing and supply chain applications products that mask complexity and sell broadly.

Suggested takeaway

If you’re very early in the game, there’s still time to get where you need to be, but perhaps your focus should be on the myriad of ways to collect, move, refresh, use and govern data in general. The data discovery tools need to sit atop this foundation and if you don’t have it, a team of data scientists may not be your next move.

If you’re a company convinced that you need to hire a score of data scientists, be careful. There’s a pretty high likelihood that by the time to recruit, hire and put those data scientists to work, someone will be selling tools that reduce the complexity and cost of what your team does. There will obviously be exceptions to this rule, but it’s worth tempering some of that recruiting enthusiasm with some skepticism for the hype.

The following infographic was published in Venture Beat along with Kelly’s article:

Google+

Christopher Taylor