Dinosaurs made extinct by Big Data

Over the weekend we watched Moneyball with Brad Pitt. If you haven’t watched it, I highly recommend it. This movie is the ideal platform to explain Big Data and why it matters. It can a be a struggle to see through the hype of any new concept and understand what’s real. As is usually the case, this well-told story conveys the reality very well. It was striking (no pun intended) the number of ways that this movie could be used as a case study for Big Data and analytics.

Intuition vs. data

There’s a great scene in the movie where Brad Pitt’s character is meeting with his scouting team. Most of them are dinosaurs. The conversation goes something like this:

Scout 1: I like Perez

Scout 2: He’s got an ugly girlfriend

Scout 3: An ugly girlfriend means no confidence

As the scouts continue to discuss outlandish variables, it becomes apparent that these highly experienced guys were clueless about what really mattered.

By sheer luck, Brad Pitt soon meets Peter Brand, a young Ivy League economist with a new approach:

Peter Brand: There is an epidemic failure within the game to understand what is really happening and this leads people who run major league baseball teams to misjudge their players and mismanage their teams… Your goal shouldn’t be to buy players. Your goal should be to buy wins and in order to buy wins, you need to buy your runs. Baseball thinking is medieval, they are asking all the wrong questions.

This same type of dynamic is playing out today across a broad range industries. A recent example from healthcare within the state of Michigan shows, “Cardiac patients are nearly three times more likely to undergo heart angioplasty if they live in St. Joseph than if they were in Grand Rapids.” As the counter to this, more and more hospitals, like Appleton, Wisconsin’s Thedacare, are applying a data driven approach that has their physicians supporting a curtailing of ‘creative’ medicine that doesn’t have sufficient evidence. Traditional wine recommendations formulated by a few experts are being replaced with statistically meaningful predictive models based on Big Data. This approach will re-imagine industry after industry.

Pushback from incumbents

Let’s face it, when ways of doing things change, there are winners and losers. The previous generation of baseball scouts, with whole careers invested in the old practices, were clearly on the way out, but not quietly. There was a lot of pushback to the new approach that ended with very senior, very respected people leaving to ‘pursue other opportunities’. There was no other way.

Rapid victory for analytics

The new Big Data approach turns out to be a true game changer, conferring a strong competitive advantage. The Oakland A’s, went on to accomplish a record-setting 20-game winning streak. Other teams took note and the better-funded Red Sox owner tried to hire Billy Bean away.

John Henry: For forty one million, you built a playoff team. You lost Damon, Giambi, Isringhausen, Pena and you won more games without them than you did with them. You won the exact same number of games that the Yankee’s won, but the Yankee’s spent one point four million per win and you paid two hundred and sixty thousand. I know you’ve taken it in the teeth out there, but the first guy through the wall. It always gets bloody, always. It’s the threat and not just the way of doing business, but in their minds it’s threatening the game. But really what it’s threatening is their livelihoods, it’s threatening their jobs, it’s threatening the way that they do things. And every time that happens, whether it’s the government or a way of doing business or whatever it is, the people are holding the reins, have their hands on the switch. They will bet you’re crazy. I mean, anybody who’s not building a team right and rebuilding it using your model, they’re dinosaurs. They’ll be sittin’ on their ass on the sofa in October, watch the Boston Red Sox win the world series.

This is ultimately going to be the story of Big Data: A big data set well analyzed beats a hunch every time.

Who will be sittin’ on their behinds on the sofa in October?