Should you require a criterion when nominating a release-blocking bug?
In chapter 11 of Program Management for Open Source Projects, I explain why you need to have release criteria. They are your community’s definition of “ready” for a release — what the software must and must not do. In projects of any significant size, you’re not going to be able to fix every bug — not even all of the bugs you know about. So establishing release criteria means you have a standard to measure the release against.
Some bugs obviously violate a criterion. Most are ambiguous and require some discussion and interpretation. This requires a process for nominating and evaluating potential release blockers. So the question is: as part of the process, should you require a nomination to be accompanied by the criterion allegedly violated?
You might be wondering why you wouldn’t require it. It seems obvious that you’d want to, right?
While you certainly want to decide which criterion applies before deciding a bug blocks the release, that’s a question for the final determination. You want the process of nominating a bug to be low friction. The people participating in the blocker decision are probably core contributors who are familiar with your project’s processes. Fedora Linux 37, for example, had roughly 120 release criteria across the basic, beta, and final lists.
The people discovering — and thus nominating — blocker candidates may not be intimately familiar with the project. They could be enthusiastic users giving your beta release a spin. Or maybe they’re downstream developers who just noticed that their automatic tests started failing.
Making the nomination process easier means you’ll end up rejecting more bugs. But the point of the process isn’t to get a high acceptance rate; the point is to catch bugs that violate your release criteria.