OpenBMC Project metrics

[Patrick Williams]

On Fri, Dec 06, 2019 at 07:33:26AM -0600, krtaylor wrote:
> On 12/4/19 4:33 PM, Andrew Jeffery wrote:
> > On Thu, 5 Dec 2019, at 05:14, Kurt Taylor wrote:
> > > 
> > > NOTE: these metrics should be used *very carefully*. They do not
> > > represent the total contributions to the project. We value
> > > contributions many that do not show up in these charts, including
> > > reviews, mail list involvement, IRC involvement, etc.
> > 
> > Given all the caveats and the lopsided view the graphs display, what
> > are we trying to achieve by graphing the metric of commits per company?
> 
> "What gets measured, gets managed" I am a firm believer of this simple
> quote. Measuring a project always improves it. That, and I have been asked
> to start gathering metrics from several of our contributing companies. They
> appreciate it.

I recognize some other projects publish statistics like this and it is
all publicly available information but I personally have slight
apprehension about this data.  This project is no where as mature as the
Linux kernel and the data is highly skewed towards one company.  I have
some concern that this data could be used for political purposes, both
externally in community interaction and internally to member companies
w.r.t. their decisions on future involvement.

The data is public (from Gerrit), no doubt about it, but I think it is
reasonable to question if it is a net-positive or net-negative for the
community to gather the data and put it on Github, to put it on Github
and advertise it, or to put it on Github and the advertisement coming from
the Community Manager.  (ie. there is a spectrum of possible ways to
deal with this data with different pros/cons)

> "Measuring a project always improves it."

Maybe a first step here is answering what is the desired change by
publishing this data?  And who's desire is it?  That isn't obvious to me.

> > It's also not clear to me what the inputs to these graphs are, for instance
> > whether changes to Linux, u-boot, qemu or other major projects that we
> > consume and contribute to are included or whether it's just repositories
> > under the openbmc org on github. If we're excluding upstream projects,
> > why?
> 
> It is only for contributions under openbmc. Other projects have been
> excluded simply because they have their own project metrics. For example:

The commit-count-from-Gerrit approach is slightly disappointing to me for
two reasons:

    1. Commit count does little to assess the impact of the
       contribution.  Ex. a one-line recipe update to add a dependency
       counts the same as a feature.

    2. There are significant contributions on the kernel side done by
       and pretty much exclusively for this project.  The effort
       involved with getting kernel patches upstream is at least an
       order of magnitude higher than userspace changes (see also
       "impact").

> > Where are the scripts to reproduce the graphs? Can you contribute them
> > to openbmc-tools?
> 
> Eventually yes, if my employer will let me do more upstream. :) But, the
> data is publicly available, you can get it yourself from gerrit. Simply go
> to our gerrit dashboard and search something like: " status:merged AND
> after:<date> AND before:<date> AND NOT topic:autobump AND owner:<gerrit id>
> "

One aspect that isn't immediately obvious, since it isn't available via
source code, is how you've done the company assignment.  I suspect the
ones for your employer are correct but for other companies there might be
mistakes or oversights when people are using personal email addresses.

I think this concern also ties into the ask a month ago with the
"computer readable CLA database."  If we had a CLA database and this
tool used it, we would have one place to audit for correctness.

-- 
Patrick Williams
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.ozlabs.org/pipermail/openbmc/attachments/20191206/a20fc054/attachment.sig>