OpenBMC Project metrics

[Andrew Jeffery]

On Tue, 10 Dec 2019, at 04:54, krtaylor wrote:
> On 12/8/19 6:06 PM, Andrew Jeffery wrote:
> > On Sat, 7 Dec 2019, at 00:03, krtaylor wrote:
> >> Measuring a project always improves it. That, and I have been
> >> asked to start gathering metrics from several of our contributing
> >> companies.
> > 
> > Where did this discussion occur? Can you provide a link?
> 
> The conversations have happen many times over the last 2 years.
> 
> At TSC and community meetings (not recorded in the meeting minutes that 
> I could find, but it was discussed)
> 
> At release planning meetings (see minutes):
> https://github.com/openbmc/openbmc/wiki/Release-Planning

Ah, okay, but it just briefly talks about metrics directly and not about the
questions we're trying to answer with the metrics?

> 
> I would rather not disclose email (without consent) that I have received 
> privately from several companies supporting this work.

That's okay, but I'm still trying to understand motivations here.

> 
> No one has ever had any objection to gathering this information (until 
> now). Remember, anyone can go see this information any time they want.

I'm not objecting to gathering it at all, I'm objecting to the lack of context in
presentation of the data. Why are we gathering it? What questions are we
answering? "Who commits the most" is the first order question, but why are
we asking it? Are we trying to establish the breadth of contributing companies?
Are we trying to identify first time contributions from companies and work with
them to drive further participation?

> > 
> > I'm concerned that we're trying to create a stick to beat contributing companies with
> > rather than working to find ways increase contributions for mutual benefit. Competition
> > works as a motivator when community members feel safe to take it on but I'm not sure
> > the community is mature enough for that to be true. Adding the context for your
> > statistics might help remove my concerns.
> 
> Honestly I'm surprised at this reaction to a *potential* situation I 
> have never witnessed

This is the problem with the lack of context. We can both have different
ideas about the motivations because they aren't written down anywhere.
It was just a concern of mine, I'm not claiming that it _is_ the motivation,
and I certainly hope that it's not. My concern could be eliminated if we
wrote down why we're gathering the metrics.

> but, I am willing to add any wording that you feel 
> is necessary to create a safe development environment.

I don't think it's about adding words here to create a safe environment, just
I don't expect everyone engaging with the project has 20+ years of experience
in open source software development. Open source work can be intimidating
with scrutiny that can be applied through code review and other interactions,
and not everyone is comfortable with that out of the gate. Certainly we've
done a lot of work internal to IBM to help people become comfortable with
working in open source. What I'd hate to see is people being discouraged from
interacting in the upstream community through metrics that don't have clear
goals.

OpenBMC is flipping the switch on what was a very propriety ecosystem, and
we will have contributors that don't have strong backgrounds in open source.
If the metrics have been discussed in meetings that's fine, but I'd hope the
context would make its way out as well and be attached to the data that we've
collected.

> 
> I value your feedback. When do you feel we as a community are mature 
> enough to start monitoring reviews, commits, and other project data? 

As above I don't think that we can pick a point on a timeline, and I don't
think that's even the point. I think that there should be engagement through
the general project channels (mailing list, IRC, not targeted meetings with
limited audiences) to determine what we're trying to measure, gather the
data, and then _present the data in the context of the question we're trying
to answer_. My consistent complaint is the lack of context.

> Should we hide this "early" data until some future time when it 
> represents everyone equally? 

No, I think you have a misunderstanding of my point here. We just need
to make the question that we're answering is provided with the view of
the data. Context is important.

> Personally, I don't feel like we will ever 
> get to that place. There will always be people that contribute more in 
> one particular area than others and they just can't be upset that they 
> may have done less. Open Source requires thick skin.

It shouldn't necessarily, and we need to work at making sure we're
approachable as a community. Providing context with metrics will give
people the information they need to know how the metrics are being
used by the project.

> 
> Eventually, I'd like to break this data down by project and individual, 
> not just company.

But why? I keep asking this. It's not because there's not a valid answer
and I'm trying to trap you, I just want to understand what the motivations
are.

Anyway, Github provides some of this information for us already. For
example:

https://github.com/openbmc/openbmc/graphs/contributors

Cheers,

Andrew