My Navy career involved flying over many parts of the world. In those long hours without much else to do I occupied my mind looking for the patterns of houses, roads, farms and nature.
Still today I watch out the aircraft window and try to make sense of the patterns that I see from miles up in the air.
What’s most interesting about patterns is that some emerge only at a distance, like the watershed of a mountain range. Others only appear upon close inspection, even magnification, like the unique patterns of crystals that make up snow flakes.
Patterns are everywhere. In business, recognizing patterns is how we sell more efficiently, spot issues with products, and manage our employees better. What we’ve done to date, however, pales by comparison with what is emerging in the science of patterns.
Mathematics
The acceleration of the creation, storage and processing of data creates an ideal opportunity to apply mathematics to the discovery of patterns. This is just in time, too. Before the world was so interconnected, an enterprise had control over the data that mattered most to the business. The technologies that grew up in this environment didn’t need to take into account data outside the organization’s control. Enter the Internet and application proliferation.
We no longer have that luxury. Not only is there more data, but there are more sources that at a glance may appear to be unique but on closer inspection are really the same thing represented slightly differently. The science of pattern recognition is key to creating trusted records from the many, disparate sources of information available inside and outside the firewall.
Great examples
When data can be mathematically analyzed for patterns, key information emerges, like:
- Patterns of fraud in the submission of workers compensation claims
- Comparative/competitive pricing of products that are profoundly similar but sold under slightly different ID’s
- Security watch lists that distill various spellings of names to produce a single record that can be easily monitored
- Master patient indexes that allow for a single patient record despite data coming from multiple doctors, laboratories, pharmacies, and other treatment providers
- Event detection when the ‘ground rules’ aren’t already known, like emerging threats from epidemics or credit card fraud
The beauty of patterns is detection of information without knowing exactly what to look for. We’ve always built systems in ‘fragile’ ways that relied on very exact information expectations. Not having that concern is very, very liberating.
Without pattern capabilities, our best efforts are to find and fix these problems after the fact, when the patient has already been prescribed duplicate medicate, the fraudulent claim has been paid, or the dangerous individual is past security. Detecting patterns is the way to insulate ourselves from the ‘duplicity’ of a fast-moving, ever-changing world.
Patterns that we see or find enable us to solve problems without starting from scratch. Systems are now doing what only people could do before, and only when it wasn’t overly complicated. We can get systems to understand ‘close enough’ and take action.
Patterns are a fascinating way to make systems more powerful without making them more fragile.
All Things D
ARS Technica
Engadget
GigaOM
Mashable
TechCrunch
The Verge
Venture Beat
Wired
Chris Brogan
Brian Solis
Chris Dixon
Clay Shirky Blog
HBR Blog
IT Redux
Jeremiah Owyang
Radar O'Reilly
Seth Godin Blog
SocialMedia Today
Solve for Interesting
The TIBCO Blog
Lifehacker
Chris, I had many of the same experiences growing up in a refinery town. During the summer, my mother got me a job at the refinery (that’s the reason I stayed in school, but that’s a story for a different post), and the work that I did was the lowest of the low, laborer job. It was very boring, and I also ended up looking at patterns, but in a different way.
That was where I started recognizing processes, activities, and identifying areas for efficient (although I didn’t know I was doing that at the time). I would sit at the end of a broom or shovel, see how work was being done around me, and recognize ways that it could improve. I saw patterns, but they weren’t just out of the blue. They were tied to a structure or taxonomy.
I had to understand the role of the planner, supervisor, safety rep, pipe fitter, welder, and of course the laborer to recognize these patterns.
Fast-forward to today and I work at APQC. We are very well known for our Process Classification Framework (PCF). It is that inventory that we (and many other organizations) use to think about work and identify patterns.
One of the cool ways we use it to help shape our research agenda is through our Knowledge Base. Our Knowledge Base is the content repository that our members use to access our case studies, white papers, reports, benchmarks, and metrics. We assign each content item a category rating from our PCF. So, very easily, we can see what our members are consuming and where we might want to focus our research.
So, the pattern is there, but without the PCF to give it structure, the pattern makes very little sense. We’ve found the same is true for organizations that use the PCF to govern their actual work. It provides them (and their partners/suppliers) a common understanding of which organization, group, or system is accountable for which process or activity. Something I know you and I have discussed as being very important to the healthcare industry as they move from a fee-for-service model to an accountable care model in the future.
Interesting article and it has so many applications.
I work as an analyst for a large company and I have developed some tools in Excel to identify patterns within large volumes of our operational/transaction data (looking for transactions that are outliers and looking for subtle pattern changes for specific types of transactions). I do this also for traditional accounting sources of data as well (data dump of AP, AR, Cash Postings to look for patterns). It is amazing what you can find buried within the data when applying the right tools against millions of transactions. Fascinating stuff and leads to some valuable insights.