Growth. It’s the standard by which we measure health and success.
With open source projects however, growth is not always synonymous to success. In some cases, there can be a significant growth in community, but a decrease in overall contributions. Or the software may be expanding in scope, and yet fails to entice new users and contributors. So exactly what is the measure of open source community health and subsequently, success?
You may have figured this out already: it really depends on the community. Because of the diversity of open source projects and the people involved in them, gauges of success can vary greatly. This doesn’t mean however that measuring community health is unattainable. It is simply a matter of determining exactly what success means for your specific project.
Goal Setting: the Pivotal First Step
So naturally the first thing you should do in order to effectively measure community health is to set your goals. You can’t measure for the sake of measuring, and you can’t measure everything otherwise you would just be collecting a lot of useless information. Setting a goal not only gives you a fixed target to aspire to as your community grows and evolves, it also guides you in choosing only the essential metrics.
In the beginning the goal is typically to create software that effectively solves a problem for its users. As the community grows however, other, more specific goals surface and make clearer which metrics should be focused on. A project geared towards improving community participation and contribution will need to focus on metrics that find out what encourages community members to participate. If it is business performance they are more concerned with, the metrics should be focused on identifying which factors contribute the most to business revenue. Once you have your goal, identifying the necessary metrics becomes a whole lot simpler.
Important Notes on Metrics
Apart from being aligned with your project goals, metrics must also be accurate. Now this can be a bit tricky, as various factors including and most especially human interpretation can cause metrics to be inaccurate. It’s important for human error to be minimized in measurements however a small percentage of inaccuracy is acceptable and in most cases, normal. Over time, even a slightly inaccurate result that remains consistent can become an accurate indicator of a trend.
Something else that you need to watch out for are the relationships between metrics. A common mistake is believing that a relationship between two metrics immediately indicates causation, which is not always true. When you choose a metric, you need to carefully consider the relationships it has with other metrics, whether direct or indirect. While many of them may be connected, proving that one directly affects or causes another is a whole other matter that may require a significant amount of experimentation, observation and analysis. So you must take care not to jump to conclusions and if you suspect a correlation, take time to observe and analyze first.
Metrics as Dynamic
Perhaps now you may have a better idea of what you need to measure to properly gauge your community’s health. But will those metrics still hold the same value five or ten years from now? The answer is quite possibly no. Just as communities and their goals and priorities change, so must metrics. This is why it’s important to check every few years which metrics are still relevant to your community, which ones you can forgo, and new ones you may need to include. Remember that some metrics may, at first glance not appear to have a direct relationship with your goals, but when combined and interpreted with others can give you a clearer picture of your project’s current status. When choosing your metrics therefore it helps to keep an open mind and to look at the bigger picture: not just considering the individual value of each metric but seeing how as a whole they can help your project to improve.
[Category: Open Source]