What you’re measuring is different from how you measure it

A wall of many electrical meters

This subject comes up almost any time you’re talking about something quantifiable. You want to improve a thing, so you figure out how to measure the thing. Invariably, the two concepts merge. But they’re not the same. The measurement you get is an approximation of what you actually want to measure.

To put it another way: how you measure something is different than why you measure it. This is essentially a restatement of Goodhart’s Law. For example, you don’t really want to reduce the number of bugs, you want to improve the quality of the software. “Quality” is harder to quantify — or even define — but it’s what you actually care about.

The bug example is particularly interesting. You might use bug reports as your measure of quality because that’s readily available. But the number of bug reports isn’t necessarily a measure of quality. In fact, an increase in bug reports might correlate to higher quality because your project is good enough to attract a broad user base. On the other hand, the number of bug reports isn’t necessarily coupled to the number of bugs in the software. After all, bug reports are just the bugs that someone found and took the time to report. There are probably more bugs waiting to be discovered.

So you can set a goal of reducing the number of bug reports filed post-release. But what behavior does that encourage? Will your community make using the bug tracker more hostile so that people stay away from it? Will they just stop requesting people file a bug when they see a complaint on social media or a blog post? Again, this is Goodhart’s Law in action. And even if the behavior doesn’t change, does it really measure what you want it to?

As a program manager, you need to avoid falling into these sorts of traps. Start with understanding the “why?” behind what you’re trying to measure. What is it that you actually hope to accomplish? Then figure out the best measurements you can come up with, knowing that they’ll be imperfect. It’s better to have at least two, that way you’re not gaming one metric. And you need to constantly remind yourself and the rest of the community that how you’re choosing to make a measurement is not the same as what you’re trying to measure.

This post’s featured photo by Alexander Schimmeck on Unsplash.

Ben formerly led open source messaging at Docker and was the Fedora Program Manager. He is the author of Program Management for Open Source Projects. Ben is an Open Organization Ambassador and frequent conference speaker. His personal website is Funnel Fiasco.


Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.