Using Benchmarking Vendors (Part 2 of 2)

In part 1, of this benchmarking series, I covered a few of the instances where you might want to engage a benchmarking vendor to assist you with your projects. In this installment, I outline my recommendations around what to do when you find a vendor (or short list of vendors) that you might want to engage to help you with your project.

When you find a vendor

If you choose to work with a vendor there are a few things you should make sure you investigate in some depth about them.

• Comparability - If you are conducting a predominantly metric benchmarking project, ask about comparability. If they tell you they have data that is truly “apples to apples”, they are lying. Dig deeper. There is no other organization just like you, so there isn’t an exact data match. All data will have some variation in it. Besides, it isn’t about finding an exact comparison, it’s about understanding the data as I discussed in this post.
• Methodology or approach – They should have a fairly standard and robust way they execute benchmarking projects that has been tested, refined, and proven. Granted, it may need to be adjusted based on needs, but if they are inventing the approach each time they conduct a project, Consistency and validity will suffer.
• Validation approach – This is a subset of methodology, but one that deserves some dedicated attention. If you are collecting any type of data or information, there should be validation applied to that information. We promote looking at validation in two ways.
  • Logical – A set of logical standards you should place on any data or information set. This is usually the first level of validation and includes steps like confirming the person providing the information actually make sense (e.g., you are not getting procurement data from someone in marketing). This is especially important as a first pass if you are doing an online survey. Many people will fill out a survey just to get a report.
  • Detailed – When you are conducting a highly metric benchmarking project, this may take the form of statistical validation where you look for outliers in the data. If it is more qualitative, it may include triangulating information provided with other public sources that might be available to add a layer of confirmation. It could also include follow-up questions to a specific organization to ask for examples of where they’ve actually demonstrated something they reported they do in the public domain.
• Peer groups – Talk through how they will define relevant peer groups for your specific situation. If they say they are going to focus just on your industry peers, move on our push back. Here is more information that topic.
• Assistance beyond just learning – You should discuss how they plan to help you develop action plans based on your benchmarking findings. If they don’t have any and leave it up to you to figure out what to do with the findings, just know what you are getting into and that you can handle the change phase of the project. If they do the actual implementation of the findings, make sure you keep an eye on the benchmarking phase. A truly disinterested benchmarking vendor doesn’t care what the findings are as long as they are the best they can find for you. A benchmarking vendor that has to implement the findings may not want to accept the challenge of getting you to the “best practice” standard uncovered during the benchmarking phase.

Just like any area, caveat emptor. Benchmarking is no different. I try to live by a rule I learned long ago. I like to work with vendors that don’t make my tummy hurt. I hope these steps help you find the benchmarking vendor that is right for you.