What you’re measuring is different from how you measure it

This subject comes up almost any time you’re talking about something quantifiable. You want to improve a thing, so you figure out how to measure the thing. Invariably, the two concepts merge. But they’re not the same. The measurement you get is an approximation of what you actually want to measure.

To put it another way: how you measure something is different than why you measure it. This is essentially a restatement of Goodhart’s Law. For example, you don’t really want to reduce the number of bugs, you want to improve the quality of the software. “Quality” is harder to quantify — or even define — but it’s what you actually care about.

The bug example is particularly interesting. You might use bug reports as your measure of quality because that’s readily available. But the number of bug reports isn’t necessarily a measure of quality. In fact, an increase in bug reports might correlate to higher quality because your project is good enough to attract a broad user base. On the other hand, the number of bug reports isn’t necessarily coupled to the number of bugs in the software. After all, bug reports are just the bugs that someone found and took the time to report. There are probably more bugs waiting to be discovered.

So you can set a goal of reducing the number of bug reports filed post-release. But what behavior does that encourage? Will your community make using the bug tracker more hostile so that people stay away from it? Will they just stop requesting people file a bug when they see a complaint on social media or a blog post? Again, this is Goodhart’s Law in action. And even if the behavior doesn’t change, does it really measure what you want it to?

As a program manager, you need to avoid falling into these sorts of traps. Start with understanding the “why?” behind what you’re trying to measure. What is it that you actually hope to accomplish? Then figure out the best measurements you can come up with, knowing that they’ll be imperfect. It’s better to have at least two, that way you’re not gaming one metric. And you need to constantly remind yourself and the rest of the community that how you’re choosing to make a measurement is not the same as what you’re trying to measure.

This post’s featured photo by Alexander Schimmeck on Unsplash.