OpenBMC Project metrics

Tue Jan 7 09:36:03 AEDT 2020

Getting caught up on emails and ARs.

I agree with the others that the context of metrics are not clear nor is
the goal in presenting this data as is.

When I look at the chart, it's not clear to me what it represents either. I
see from Kurts comment in this email that this graph represents the 'merge
metrics', but that is nowhere on the chart. There's also a disclaimer on
the wiki page that these metrics may not be a true representation of
project contribution. So, what's the true value of the metrics then?

What does the y-axis represent in units?
What repositories out of the 140 possible are represented in the graph?

I see charts like this as having one main purpose which is to communicate
who are making contributions, and how do they rank amongst the others. With
the target audience for the metrics being managers. Which bothers me,
because it's too subjective and ambiguous.

We do need to have a clear statement on the goal in providing a chart on
metrics. Here are my thoughts for what I would like to see in the metrics.
Which are focused on the what and not the who.

The primary goal is to represent a holistic view the openbmc project by
providing insight to the progress and health of the openbmc project.

Where progress is represented by the code that attains sufficient quality
to be merged upstream. I would include any upstream projects that were made
on behalf of the openbmc project (i.e., linux, u-boot, etc…).

Health as measured by amount pull requests/code commits being made, reviews
that are in progress, the amount of code added versus deleted, abandoned
code, and aging of commits/reviews.

1 - To show source code contributions that make it into the upstream
repository (i.e., merged)
2 - To show incoming code contributions via pull requests or commits (i.e.,
being reviewed)
3 - To show lines of code additions, deletions, or abandoned
4 - To show contributions via review comments (i.e., number of review
comments)
5 - Snapshots in time weekly, bi-weekly, or monthly (x-axis)

Thanks,
James.

On Tue, Dec 10, 2019 at 3:14 PM Andrew Jeffery <andrew at aj.id.au> wrote:

> On Tue, 10 Dec 2019, at 04:54, krtaylor wrote:
> > On 12/8/19 6:06 PM, Andrew Jeffery wrote:
> > > On Sat, 7 Dec 2019, at 00:03, krtaylor wrote:
> > >> Measuring a project always improves it. That, and I have been
> > >> asked to start gathering metrics from several of our contributing
> > >> companies.
> > >
> > > Where did this discussion occur? Can you provide a link?
> >
> > The conversations have happen many times over the last 2 years.
> >
> > At TSC and community meetings (not recorded in the meeting minutes that
> > I could find, but it was discussed)
> >
> > At release planning meetings (see minutes):
> > https://github.com/openbmc/openbmc/wiki/Release-Planning
>
> Ah, okay, but it just briefly talks about metrics directly and not about
> the
> questions we're trying to answer with the metrics?
>
> >
> > I would rather not disclose email (without consent) that I have received
> > privately from several companies supporting this work.
>
> That's okay, but I'm still trying to understand motivations here.
>
> >
> > No one has ever had any objection to gathering this information (until
> > now). Remember, anyone can go see this information any time they want.
>
> I'm not objecting to gathering it at all, I'm objecting to the lack of
> context in
> presentation of the data. Why are we gathering it? What questions are we
> answering? "Who commits the most" is the first order question, but why are
> we asking it? Are we trying to establish the breadth of contributing
> companies?
> Are we trying to identify first time contributions from companies and work
> with
> them to drive further participation?
>
> > >
> > > I'm concerned that we're trying to create a stick to beat contributing
> companies with
> > > rather than working to find ways increase contributions for mutual
> benefit. Competition
> > > works as a motivator when community members feel safe to take it on
> but I'm not sure
> > > the community is mature enough for that to be true. Adding the context
> for your
> > > statistics might help remove my concerns.
> >
> > Honestly I'm surprised at this reaction to a *potential* situation I
> > have never witnessed
>
> This is the problem with the lack of context. We can both have different
> ideas about the motivations because they aren't written down anywhere.
> It was just a concern of mine, I'm not claiming that it _is_ the
> motivation,
> and I certainly hope that it's not. My concern could be eliminated if we
> wrote down why we're gathering the metrics.
>
> > but, I am willing to add any wording that you feel
> > is necessary to create a safe development environment.
>
> I don't think it's about adding words here to create a safe environment,
> just
> I don't expect everyone engaging with the project has 20+ years of
> experience
> in open source software development. Open source work can be intimidating
> with scrutiny that can be applied through code review and other
> interactions,
> and not everyone is comfortable with that out of the gate. Certainly we've
> done a lot of work internal to IBM to help people become comfortable with
> working in open source. What I'd hate to see is people being discouraged
> from
> interacting in the upstream community through metrics that don't have clear
> goals.
>
> OpenBMC is flipping the switch on what was a very propriety ecosystem, and
> we will have contributors that don't have strong backgrounds in open
> source.
> If the metrics have been discussed in meetings that's fine, but I'd hope
> the
> context would make its way out as well and be attached to the data that
> we've
> collected.
>
> >
> > I value your feedback. When do you feel we as a community are mature
> > enough to start monitoring reviews, commits, and other project data?
>
> As above I don't think that we can pick a point on a timeline, and I don't
> think that's even the point. I think that there should be engagement
> through
> the general project channels (mailing list, IRC, not targeted meetings with
> limited audiences) to determine what we're trying to measure, gather the
> data, and then _present the data in the context of the question we're
> trying
> to answer_. My consistent complaint is the lack of context.
>
> > Should we hide this "early" data until some future time when it
> > represents everyone equally?
>
> No, I think you have a misunderstanding of my point here. We just need
> to make the question that we're answering is provided with the view of
> the data. Context is important.
>
> > Personally, I don't feel like we will ever
> > get to that place. There will always be people that contribute more in
> > one particular area than others and they just can't be upset that they
> > may have done less. Open Source requires thick skin.
>
> It shouldn't necessarily, and we need to work at making sure we're
> approachable as a community. Providing context with metrics will give
> people the information they need to know how the metrics are being
> used by the project.
>
> >
> > Eventually, I'd like to break this data down by project and individual,
> > not just company.
>
> But why? I keep asking this. It's not because there's not a valid answer
> and I'm trying to trap you, I just want to understand what the motivations
> are.
>
> Anyway, Github provides some of this information for us already. For
> example:
>
> https://github.com/openbmc/openbmc/graphs/contributors
>
> Cheers,
>
> Andrew
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ozlabs.org/pipermail/openbmc/attachments/20200106/d7cc93f2/attachment.htm>