10 metrics you should track in your open source community

How is my open source project currently doing? How many people are contributing to it? What happens if they leave? To get a sense of how well your community is doing, you need to understand their activities. With the right metrics, you can measure this and turn it into useful information. However, this continues to be a challenge for community managers since only a few tools are available to provide these measurements. That is why we gathered the ten most important ones for you, described how they are defined, why they are essential, and explained how you measure them.

Table of Content

1. Social Listening
2. Contribution
3. Time to close
4. Organizational Diversity
5. Contributor Location
6. Time to first response
7. Burstiness
8. Project popularity
9. Bus factor
10. Engagement score
    
    

1. Social Listening

How is "Social Listening" defined? 

Social Listening is a combination of practices across multiple channels. So basically, tracking every mention of your project on different social media platforms. This can be in the form of a post, comment, feedback, like, etc. With this information, you get to know your community more personally. This is useful to foresee trends and discover opportunities. 

Why is this metric important? 

The metric represents the values of a community: its reputation, desires, dislikes, and general community sentiment. By measuring it, you create a measurable business value. Additionally, it gives you insight into community issues or voices which otherwise might have been overheard. Finally, proactively using this metric gives members the feeling that their thoughts and opinions matter.

How can you measure "Social Listening"?

Social Listening is data retrieved from community interactions about your project on social media: 

Step 1: Before collecting data, define a list of keywords, tags, and categories that you want to focus on in your analysis.

Step 2: Use the defined list of keywords and categories to monitor the relevant social media platforms for mentions of your: 

• Projects
• Competitors
• Product 
• Brand 
  

Step 3: Review your collected data and put what you learned into action. What are your strength and weaknesses according to your community? What can you improve to make your community healthier? 

2. Contribution

How is "Contribution" defined? 

There are multiple options to contribute to an open-source project. It is not all about coding. Managing the community, fixing bugs, supporting users, or evangelizing the projects are ways to improve an open-source project. 

Why is this metric important? 

Having a variety of people working on different contribution types indicates a healthy and mature open-source project. Moreover, it shows a well-rounded project with all kinds of people contributing beyond coding. 

How can you measure "Contribution"? 

Measuring contribution can be challenging since it is difficult to compare different contribution types. Depending on the project type, these can vary. Therefore, we have divided the various contribution types into separate lists for you: 

Coding: 

• Writing Code
• Reviewing Code 
• Bug Triaging 
• Quality Assurance and testing

Administration:

• Troubleshooting and Support
• Documentation
• Financial Management

User:

• Community Building and Management
• Organize workshops, meetups 
• User Support and Answering Questions

Marketing & PR: 

• Social Media Management
• Interviews with Technical Press
• Event Organization
• Starting a newsletter 
  

Now, after we clarified how you could contribute to an open-source project, we will explain how you can collect the data to measure the different contribution types:  

• Interviews or Surveys: Create a survey or interview other community members to determine how much others have contributed to the project. 
• Capture all Contributions: Develop a system that allows you to capture all contribution types, not only code-related ones. 
• Automate Classification: Train an AI bot to detect and classify contributions.
• Collect Trace Data: Measure contributions through collaboration tool log data. 
  

3. Time to close

How is "Time to close" defined? 

Time to close is defined as the time between the creation and the closing of an operation. It can be an issue, change request, support ticket, etc. In order to trace this metric, the operation has to be in a closed state. 

Why is this metric important? 

Time to close can be used to determine the responsiveness of your community. It also helps to identify characteristics that impact the operation to tackle future responses more efficiently. In addition, you can discover the areas that still need some refinement and find best practices to improve them. It also helps to identify changes within the community activity, for example, reduction in the diversity of contributions or an indication for a potential maintainer burnout. Finally, draw a parallel to how the time and closure of an issue is related to merge success or failure. 

How can you measure "Time to close"? 

In this case, specific numbers are not that valuable. It is more about the general trend. Depending on the issue or objective, the time to close can generally differ. So, for example, the time to close a bug report may vary from the time to close a new feature request. The data collection approach should be different depending on the project objective. 

Furthermore, you can't find a reason for a longer time to close just by the numbers. It is better to talk with your team and see where potential problems are. Find out which aspect they are currently focusing on so you know why numbers in certain areas are higher than others. Then, together you can come up with a solution on how to fix it. If you want concrete data on time to close, two main measurements provide you with it: 

• Defined time to recover/repair (MTTR)
• Defined time between failures (MTBF)
  

4. Organizational Diversity 

How is "Organizational diversity" defined? 

Organizational diversity is the relative amount of different organizations involved in a project compared to each other. Basically, how much every company contributes to a project compared to each other. When you think about your community's resilience, diversity is an essential factor—the more diverse, the better. 

Why is this metric important? 

If the main contributor with 85% leaves an OSS, the consequences are more severe than a company leaving that only contributes 2%. Therefore, it is essential to identify who contributes most to your open-source project to minimize the risk of this occurring. 

How can you measure "Organizational diversity"? 

First, you need to gather a list of the organizations contributing to your project and associated with it. Secondly, you need to calculate the percentages of contributions from each organization within a pre-defined period. Finally, we prepared some questions with whom you can measure Organizational diversity (in %):

• How many merges/reviews are done by each organization? 
• How many issues are filled by each organization? 
• How much is each organization generally contributing to the project? 
• How many lines of code does each organization contribute? 
• How many organizations are currently contributing to my project?
• How many organizations joined my project in the last month? 
  

5. Contributor Location

How is "Contributor location" defined? 

It is the geographical location from which contributors work or where they live. 

Why is this metric important? 

The location of the contributors helps to understand work practices due to time differences in different time zones. In addition, understanding where the most contributors are located increases engagement in these areas and improves effectiveness. 

How can you measure "Contributor Location"? 

First of all, you need to categorize the term location. Are you referring to a continent, country, state, time zone or city etc.? How concrete do you want to be? The second aspect is time. How long do you want to count contributions: 24hours, a week, a month? The third factor is the type of contributor. What do you count as a contribution? As previously in point 2 explained, there are multiple ways to contribute to a project. You can use different strategies to collect information about the location of your contributors: 

• Profile: You can look up the geographical location of your contributors through their profiles on GitHub.  
• IP address geolocation: Tracking your contributor's location via their IP address helps you figure out the area where the most contributions are made.
• Timestamp: You can infer the geographical location from the timestamp in contributions. 
• Survey: Create a survey and ask your contributors where they are currently based. 
  

The easiest approach will be to look at the contributor's profile location. If that is not available, check their geolocation. Keep in mind that the information on their profile might be outdated or non-sensical. Their IP address can also provide false data since some might use a VPN or other masking tools to disclose their current location. 

6. Time to first response

How is "Time to first response" defined? 

Time to first response measures the time between creating an issue and the first response.

Why is this metric important? 

The first response to an activity can be the most important since it shows your community's activity and engagement level. A long response time can be an indicator for an inactive community. The shorter the response time, the faster issues can be solved. 

How can you measure "Time to first response"? 

You have to define three main aspects. First of all, the role of the responder. Which time of response are you keeping track of? Only the maintainer responses? The second aspect is the automated response. Do you only count the responses from real people, or are bots and other computerized replies also included? The third part is the type of activity. What do you count as an activity? Issues, emails, chats, change requests, PR? So it would be calculated as the following: 

Time to first response of an activity = time first response was posted to the activity - time the activity was created.

7. Burstiness 

How is "Burstiness" defined? 

It is defined as the fluctuation in activity and frequency. It is a metric to understand when and why your community is unusually active. Furthermore, it helps you find a pattern in community activity to predict these spikes in the future. 

Why is this metric important? 

There are multiple reasons why the activity within a community can be higher at certain times and lower at others. With this metric, you can find out the root causes of a burst in activity, provide awareness when a project fluctuates in an activity, and help measure the impact of external activity. You can even predict future bursts that follow a specific pattern and help the community and maintainers to prepare for it. 

What could be the causes of this fluctuation?

• Release cycles
• Hackathons 
• Events
• Critical bugs 
  

How can you measure "Burstiness"?

The following measure can indicate a change of activity: 

• Stars or Forks
• Issues or bug reports
• Feature Requests
• Social Media Activity

8. Project popularity 

How is "project popularity" defined? 

Project popularity is defined as how well known and liked your project is. 

Why is this metric important? 

Project popularity can be used to understand the economic value of your project since this is typically really hard to measure due to a lack of measurable data. For contributors, this metric might also be of interest. With the need to earn a living wage and maximize future employment opportunities, it might be interesting to know how much potential this project has and if it's worth investing in. 

How can you measure "Project popularity"? 

Popularity can be measured by the activity visibility around a project. The popularity metrics change over time. Unfortunately, it is not easy to calculate. Some key figures you can use include stars, forks, likes, social media mentions, or people attending events.

9. Bus factor

How is "Bus factor" defined? 

So Bus factor defines the risk for a community when the most active members leave. The bus factor is the smallest number of people that make 50% of the contributions.  

Why is this metric important? 

It is a compelling metric since it answers the question: "How many contributors can we lose before the project stalls?". In addition, you find out how the work is distributed across the contributors and identify key people who are doing the majority of the work. 

How can you measure "Bus factor"? 

Calculation: 

1. Find out how many contributors are a part of your project.  
2. Find out how much each contributor individually contributes to your project. 
3. Arrange the numbers of contributions in descending order. 
4. Add them up and divide it by 2, then you have the number of total contributions.
5. Add up the number of contributions until you get 50%.
6. Count the number of contributors it took to reach that 50% threshold. 
   

Example:

1. You have 7 contributors 
2. 30, 620,879, 590, 200, 22, 98 
3. Arrange numbers: 879, 620, 590, 200, 98, 30, 22 
4. (879 + 620 + 590 + 200 + 98 + 30 + 22) /2 = 1219,5
5. 879+620= 1499
6. Contributors =2 → Bus factor = 2
   

In our example, two people make more than 50% of the contributions to the project. If those two were to leave, your project would be at risk. Remember that the bus factor may differ over time since the number of contributions also changes. It is just a current representation of contribution. 

10. Engagement score 

How is "Engagement score" defined ? 

The engagement score defines the value of each member for the community. The community consists of users, watchers, maintainers and contributors, with the last group being the most involved one in a project. They invest their time in it, provide feedback and contribute code. 

Why is this metric important? 

OSS is fundamentally driven by the input from their communities. Measuring users and contributors provide the most insight into the scale of the community. To detect those insight cues like GitHub stars or forks aren't always the best indicator for continual engagement since they tend to spike with big press releases or blog articles. 

How can you measure "Engagement score"?

Here are a few trends you can include in your measuring techniques: 

• Variance by Maturity: Most projects peek and then flatten off when they reach full maturity. A projects age and maturity should be included when evaluating the engagement. 
• Variance by Type: It is essential to segment the projects since you can't compare front-end frameworks (with much larger groups of developers) with databases.  
• Skew: Keep in mind that contributions are seriously skewed towards the largest projects, which receive easily 1000 contributors per month. Meanwhile, reaching around 100 contributors per month is already a significant number for most projects. So, you have to identify what you perceive as a meaningful threshold.

‍

Final words

If you want to manage your community, you don't have to use all of these metrics. Instead, it is better to focus on the two most important ones. Understanding the metrics will help you build a strong, healthy open source community and make your project thrive. 

We at Crowd.dev understand the value of these metrics and how difficult it is for community managers to manage their community without them. Apply for our private beta to demystify your open source community.