Focus release criteria on user-visible behavior

In chapter 11 of Program Management for Open Source Projects, I defined release criteria as “the behaviors the software must (or mustn’t) have to be released.” I chose “behavior” deliberately, but perhaps I should have said “user-visible behavior.” Release criteria aren’t for making perfect software, they’re for making acceptable software. If undesired behavior isn’t visible to the user, it’s a bug, but it’s probably not release-blocking.

I started thinking about this after I read Chris Siebenmann’s “Systemd auto-restarts of units can hide problems from you“. In this post, Chris describes how he — somewhat by accident — discovered that a monitoring service was crashing on a host. Because systemd automatically restarted the service, the failure wasn’t visible to Chris. Is this something to fix? Definitely! Would I block a release based on this? No. The software is functioning well enough that the issue went unnoticed for long periods of time.

But let’s enter an alternate universe where the behavior is more noticeable. If your pre-release testing catches it, you’ll open a bug report. If it violates your release criteria, then you’ll call it a blocker. If you get it fixed, great — you close the report and ship the software. But what if you only get it worked around enough that it’s no longer a blocker? Then you can close the original bug report (which makes blocker tracking simpler) and open a new one about the hidden behavior.

Of course, if you discover it early enough, the blocker process doesn’t have to come into play at all. You fix it well before anyone’s thinking about that. Otherwise, focusing on the user-visible behavior allows you distinguish between what you should fix and what you must fix.

This post’s featured photo by Agence Olloweb on Unsplash.