Closing the Big Data Loop

It has been two weeks since TUCON 2012, the TIBCO user conference held annually in Las Vegas, Nevada. TIBCO is well-known as an integration company but used the event to demonstrate their broad platform approach to the biggest challenges of today, like digital customer experience, loyalty, and the topic of this article, Big Data.

The TIBCO CTO, Matt Quinn spoke about the patterns hiding within the rapidly increasing amounts of data flowing across the enterprise. Quinn made an important distinction between data at rest, information sitting in databases or flat files waiting to be queried, and data in motion, which includes streaming data and data stored in-memory (also known as cache).

Quinn made the point that it takes analytics like those offered within TIBCO’s Spotfire product to be able to see what would be invisible to people trying to keep up with the increasing deluge of information. To Quinn, the smart enterprise finds patterns in historical and machine data (log files that up until recently weren’t mined for patterns) that provide insight that can be applied to data that’s coming at today’s ‘full speed’.

Event streams

Event streams, information arriving constantly that describes current conditions, can be acted upon only when it can be understood in the context of what’s happened in the past that has meaning. This idea of interpreting the present based on the past he called, “Closing the loop between data in motion and data at rest.” His real-time approach to Big Data is much broader and more inclusive than the hype around Hadoop and its batch processing limitations.

What makes this approach interesting (and actionable) is the ability to ‘feed’ an event processing system that listens in real-time for known opportunities and risks and allows an organization to act far more quickly than through traditional means. Acting can be manual or automated, but either way, can be significantly more valuable when it happens quickly, before fraud is committed or an opportunity passes.

Big Data pearls

TIBCO has been busy over the past several years developing or acquiring products that stitch together a complete Big Data story, which Quinn described as connect, understand, anticipate and act. His focus on four categories helps the enterprise to understand the bigger picture critical to having a sustainable approach to the enormity of Big Data and analytics. With so many companies wandering in the Big Data wilderness and so much misinformation in the marketplace, this is an important understanding to gain. There is so much focus on Hadoop and other partial solutions that people lose sight of the need to have the core integration, analytics, business events, collaborati0n and business process/connectivity that make Big Data valuable.

Google+

Christopher Taylor