The do’s and don’ts of measuring contributions “outside of working hours”
LFX Insights is a handy platform from the Linux Foundation that provides a variety of data on open source projects. Among the statistics it reports is contributions outside of working hours. Some users reported errors with how this information is reported, which got me thinking about the value of this measure. The short version: there’s very little value.
Why measure outside-of-working-hours contributions?
LFX Insights includes this measure as a signal of a project’s sustainability. Projects that rely heavily on people making after hours contributions, the thinking goes, will have a harder time attracting and retaining contributors.
As a software consumer, you don’t want your upstreams to suddenly disappear because that will present supply chain risks. It could mean vulnerabilities go unpatched. It could also mean that new features aren’t added. Either way, this puts the onus on the project’s users to carry the load.
As a project leader, you may be less concerned about whether or not a company downstream has to devote extra engineering time, but you probably do want your contributors to stick around anyway. Onboarding, mentoring, and growing contributors takes a lot of time and effort. You want to make sure people can stick around.
Why this measure fails
Despite the good intentions of measuring contributions outside working hours, the reality fails to deliver. There are some straightfoward reasons for this. Not everyone’s working hours are the same. Not everyone’s working hours are consistent. Some people use a different time zone on their computer. Not everyone’s working days are the same. Holidays vary widely across countries and religions. People (hopefully) take time off.
Then there’s the cultural mismatch. Linux Foundation projects are, to a first approximation, by companies for companies. The Linux Foundation is a 501(c)(6), not a charity, so it makes sense that it would view the world through a business lens. I don’t fault them for that. LF project contributors are more likely to make contributions during the working day than contributors to a “hobbyist” project.
But that workday tendency doesn’t necessarily mean people will stick around projects longer if the project is tied to their job. As the last few years have shown, tech sector layoffs can come for anyone at any time. If someone is only working on an open source project because it’s part of their job, then when the job changes, they’ll probably stop. People who work on an open source project for non-job reasons will likely stick around through job changes.
Thus one could argue that a project with a high degree of outside-working-hours contributions is more sustainable.
What to measure instead
If measuring contributions outside of working hours isn’t helpful, what is? Focus on what you’re worried about. Worried that everyone will disappear? Measure the activity over time. Worried that when a new vulnerability is discovered the lone maintainer will be backpacking through the Alps? Measure the spread of the contributions. Worried that the project doesn’t have enough people to follow secure coding practices? Measure the security posture.
Of course, the best answer is to stop trying to measure sustainability and contribute to making the project more sustainable instead.
This post’s featured photo by Joshua Olsen on Unsplash.
