By holly.ross.drupal on

The Drupal community is diverse. We're from different places. We speak different languages, code in different languages, and use Drupal in different ways. But the one thing we all have in common is that we care about the fate of Drupal. We invest our time, intellect, and emotions into this project. And we want that investment to make the project successful.

We have a lot to be proud of. In the last few years, Drupal use and community contribution has grown tremendously. Some metrics—like comments and commits created each month—are available at drupal.org/metrics. We share project usage and track BuiltWith statistics. We've also started reflecting company contributions in the marketplace and on the organizations list.

The Association's mission is to unite a global open source community to build and promote Drupal. One of its 3-to-5 year goals is to ensure the sustainability of our project and community. To meet that goal, we need to get much better as a community at answering the question: Is Drupal healthy? But that's not an easy question to answer.

In the last board meeting, board members and staff talked about what kinds of metrics show health. In this post, I share some of that thinking, and ask for your ideas. But before we get into recommendations, let’s start at the beginning.

What's project health?

Association staff and board members use “project health” as shorthand for project success. "Health" is robust. Think about your own health. You might think first about physical aspects (Do I have a fever?). But you also might think about emotional aspects, like depression or stress. You may even think about social aspects (Do I have the right people around me to support me and keep me focused on the right habits?). Health isn't binary. Health is multi-dimensional.

So, too, is the health of an open source project. It's easy to take the temperature of a project with metrics like usage or number of committers. But to understand the complexity of our project's health, we need broader measurements. In our work on staff, we've defined four dimensions of product health1 we think we need to explore:

  • Product: whether the business of the Drupal product (the software) is sound. Examples of areas to explore: marketshare; Drupal businesses' revenue.
  • People: whether we have the right kinds of people contributing the right kinds of things. Examples of areas to explore: number of contributors; kinds of contributions; contributors' skills.
  • Process: how the way we do things contributes to the project. Examples of areas to explore: ability to meet published release dates; succession planning.
  • Systems: whether we use the right tools. Examples of areas to explore: testing times for DrupalCI; amount of documentation edits; responses to posts in forums; issue resolutions; commits and integrated repositories.

Acknowledging our limitations

There's the ideal, and then there's reality. We know we won't be able to track everything we want to. So, we'll have to make choices. We'll have to choose metrics that give us directional, even if not precise, accuracy. So, what are some of our limitations?

The future of the project

Knowing what categories of metrics we need to track is necessary, but not enough. Setting metrics also requires knowing where we want the project to go. Taking your temperature is only helpful if you know what you want it to be (and what it means if it's not).

This is a challenge Association staff discussed with the board. The Association doesn't set the software's future. It does, though, need to know where the project is going. To support the project's journey, it needs to know its direction. For example, maybe headless Drupal is the future. If so, we might measure “People” success by how many JavaScript developers contribute. But it’s not 100% clear what the future holds for Drupal. We have work to do before we can identify the best metrics.

Resources

Tracking project health is an Association priority, but it’s not its only mandate. It has to consider the time and expense invested in DrupalCons and Drupal.org too, for example. Unfortunately, budget limitations mean not hiring analysts or consultants to help. For the most part, they mean working with the (wonderful) people already on staff.

So, for example, measuring contribution only by code or comment attribution isn't enough. People contribute in so many ways. (There's even an issue open on this topic already.) Will measuring contribution expand to include other things? Yes. But will it also likely still not give some contributions the attention they deserve? Unfortunately, yes. Hard choices will mean we'll all have to accept some less than ideal outcomes.

Balancing competing frames and other fun factors

Once we know what outcome we want and have found things we can actually measure, we'll still need to do more. Any metric we choose has to also align with the project's mission and our community’s values. We also can't be too dependent on internal metrics. We'll have to measure our success with external indicators too. (See? There's a lot to think about here!)

Examples, please

There’s good news and bad news here. The good news is that we are definitely not alone. Many software projects—including open source projects—have worked on these same issues. There are resources all around us. We've created lists of some of them already.

Bitergia dashboards

Open source projects on GitHub

Project-run dashboards

So what should we track?

There's a lot to consider when we think about the health of the Drupal project. We've just begun this conversation and want to make sure you're part of it from the beginning. So, now we turn things over to you.

Which metrics are indicative of Drupal project health? Share your ideas in the comments section. We'll include your feedback in a document we'll share with you.

1 We borrowed these concepts from Product Lifecycle Management.

Flickr photo: Thomas Haynie

Comments

Shyamala’s picture

Shyamala commented

Just focusing on the people part, would like to ensure we bucket measurement in 4 main areas:

1. Visitation - this group of user would indicate usage, consumption of information
2. Participartion Or Contributions - this brings in actual contributors
3. Advocacy - we need track this more - this is key for growth of a community, bringing in new members
4. Diversity - in skills, in roles, in gender, in culture, in geography

Strategise in these 4 areas is a must for growth of a community.

Greg Boggs’s picture

Greg Boggs commented

How many speakers at DrupalCon are first time drupalcon speakers? Any stats around code of conduct is a good place to understand Health... What percentage of people who identify as female contribute code to Drupal? What percent of people who attend a sprint post at least one patch? How many people post documentation comments or patches? How many contributors identify as white? How many non-code contributions are recognized on Twitter #drupal? How many people who's primary language is not English contribute to Drupal? How many people attend global Drupal events? How many people at OScon know Drupal makes websites? How many people stop at the Drupal booth at oscon or other cons? How many of those sign up for email updates? How many Drupal sites are hacked or defaced because of lack of updates ( maybe keep this internal?) How many sites successfully update the day of critical security announcements? How many brand new contributors add full modes to Drupal.org per month? How are are jr. developers doing in the job market?

droplet’s picture

droplet commented

It would be interesting to know how many commits tagging with "Needs Reroll" and how many credits given to "people doing manually testing / reviews"

catch’s picture

catch commented

One thing that has concerned me for a long time about Drupal metrics is that we've tended to focus only on quantity (number of contributors, number of Drupal 8 release blockers, number of open issues), rather than quality or velocity (how many issues fixed vs. opened, ratio of contributions per contributor). This then has a knock-on effect since those tend to get used for decision making. While BMI isn't perfect either, it sometimes feels a bit like measuring health purely on someone's weight in kilograms, not taking into account height/age etc.

For example, let's say this week there are five critical issues against Drupal core, and last week there were five issues. So no change in the headline number.

These could either be the same five issues, none of which have been fixed yet. Or we could have fixed five issues and discovered five new ones.

Assuming the code base is exactly same apart from the five fixes being committed, this is the difference between 5 unfixed + 5 undiscovered issues, vs. 5 fixed and 5 discovered issues, which is massive in terms of real-terms progress, but on drupalreleasedate.com they'll show up as exactly the same.

https://blog.jquery.com/2010/11/23/team-spotlight-the-jquery-bug-triage-... it's only a blog post from 2010, but it has some of that information that is currently missing from project issue tracking. @xjm tracked some of this for release blockers during the 8.x beta, but this was only possible by running manual queries against a staging database.

Slightly different, but also something which impacts (or rather doesn't but should) decision making:

https://www.drupal.org/node/1627676 (displaying project component usage) - right now we have no information about how much particular core modules or sub-modules of contrib projects are used. 

https://www.drupal.org/node/1248922 - filtering overall project usage by major core version.

Both of these would allow people to make much better decisions about where to prioritise efforts, things that might need splitting out to their own projects, porting etc.

On contributions, I'd really like to see numbers on longetivity of contributors.

For example what are the ratios of people with 1 vs. 20 vs. 50 vs. 100 commits/recorded contributions? You can sort of get that information for core by scrolling down http://www.drupalcores.com/ and seeing what the ranking is for various numbers of commits.

Also how many people continue contributing to the project 3/6/12/24 months after their first recorded contribution?

catch’s picture

catch commented

Examples of areas to explore: ability to meet published release dates; succession planning.

This is an interesting one. For example there are several reasons why deadlines get missed: 1. The deadline seemed reasonable in the first place, but unexpected bugs came up that caused it to be missed. 2. The deadline seemed reasonable in the first place, but unexpected feature additions came up that caused it to be missed. 3. The deadline was over-optimistic in the first place, but was set anyway in the hope of galvanising efforts, which either didn't work or backfired. Also when a deadline gets hit it can be for multiple reasons: 1. The deadline was reasonable and everything went to plan. 2. People worked lots of unpaid overtime to hit it. 3. People worked lots of paid overtime to hit it. 4. Extra people jumped in to help 5. Lots of features got dropped 6. Lots of bugs got swept under the carpet/gaffer taped up to get it out the door. So I'm not sure whether we hit a core release deadline or not is a good metric, or whether any of the above things can be measured quantitatively. I'd rather see us do a bit more in the way of retrospectives after major milestones.

holly.ross.drupal’s picture

holly.ross.drupal commented

Thanks for all the great comments so far. You guys are giving us a great deal to think about and consider!

budda’s picture

budda commented

There’s good news and bad news here.

So what was the bad news ?

holly.ross.drupal’s picture

holly.ross.drupal commented

Good point. Let's go with no bad news. 

levelos’s picture

levelos commented

I haven't heard anything about the variety of projects using Drupal. E.g., if a growing majority of sites that use Drupal are large corporate sites, I would consider that a failure. Drupal's strength has long it's been ability to strike a balance between compelling "out of the box" features and a solid application development framework. Being used for a wide array of use cases, everything from a grassroots nonprofit, to a punchy startup, and, yes, the corporate flagship site would be a sing of continued success.

LIQUID VISUAL’s picture

LIQUID VISUAL commented

Drupal comunity has provided excellent CMS for low budget non profits, with Drupal 8 being the first version that does not work on shared servers.

Is Drupal 7 the last version that will work on shared servers?  Or are we expecting shared servers to develop their resources to meet Drupal 8's requirements?

holly.ross.drupal’s picture

holly.ross.drupal commented

Great question. There is nothing inherent to Drupal 8 that keeps it from working in a shared environment. As adoption increases, and with it demand for shared hosting, we expect that hosting companies will build support for this. The Association is in direct contact with many to help communicate the needs.

TheodorosPloumis’s picture

TheodorosPloumis commented

Why saying that? I have Drupal 8.x sites working shared hosting. Which software of the Drupal requirements is the problem?

Kenndillard’s picture

Kenndillard commented

Hi Holly,

Here are my thoughts. I would urge us to understand the demographics of the people that engage with Drupal. There will be good information for you in knowing that. Actually, that is the key to knowing the actual health of Drupal, if they are engaged and if so then why? You may find that the audience for Drupal is not what was thought or that there are strong numbers in an area the group was not aware of. A CMS minus an engaged audience is just a bunch of modules with no home so why Drupal?  Any product has to know who the people are in order to set the process. Here are a few things I would track:

  1. Any demographic info is a win so ask that.
  2. Who is using it?
  3. Gender
  4. Location
  5. What are they doing with it?
  6. Why Drupal?
  7. What other CMS's have they tried?
  8. What's next for them project wise with Drupal?
  9. Do they have any other project being used in other platform and if so, why have they not been moved to Drupal?

Other things to know:

  1.  Asking was this information helpful to you when people leave forum post or contribute information.
  2. Score the information that people are contributing. (the community)
  3. Who are your movers and shakers in the community and are they themselves engaged?

All in all, it will 100% boil down to understanding who you are talking to, who you would like to be talking to and who is your message actually reaching. In short, people.

Kenn

daviddickerson’s picture

daviddickerson commented

For any project we should invest our time, intellect, and emotions. Then only we will get the best result. There are lots of people struggling with their project. This is due to lack of care. They don't spend their time with dedicated mind. So they won't get the best results. There are Women’s wholesale clothing who offers the best and fashion dresses. So many women like our product. They  know we are dedicated workers. 

Castonguay118’s picture

Castonguay118 commented

Great next process can be carried out in few steps listed below. mobdro for pc great deals of wonderful functions for customers Fine.

Oleary77’s picture

Oleary77 commented

good suitable devices in your house easily. Have   Kodi App  Users should provide their very own web nice.

dangph’s picture

dangph commented

Thank for share

Read more ty le bong da and livescore

mahi2723’s picture

mahi2723 commented

Being a gamer, I lost saved games for lot of times, this is happened due to improper saving of the game and unavailability of proper backup restore setups. This is yesterday’s scenario, now the gaming world is blessed with awesome app for restoring/backup of any app or game information by just a click.

Download the latest titanium backup pro APK on your mobile and enjoy the restore and backup option on your smartphone for free of cost.