OpenBMC Project metrics

[Andrew Jeffery]

On Sat, 7 Dec 2019, at 00:03, krtaylor wrote:
> On 12/4/19 4:33 PM, Andrew Jeffery wrote:
> > 
> > 
> > On Thu, 5 Dec 2019, at 05:14, Kurt Taylor wrote:
> >> All, I just posted the project merge metrics for September and
> >> October. I will be updating the company/developer lists for November
> >> and posting those shortly. Enjoy.
> >>
> >> https://github.com/openbmc/openbmc/wiki/Project-Metrics
> >>
> >> NOTE: these metrics should be used *very carefully*. They do not
> >> represent the total contributions to the project. We value
> >> contributions many that do not show up in these charts, including
> >> reviews, mail list involvement, IRC involvement, etc.
> > 
> > Given all the caveats and the lopsided view the graphs display, what
> > are we trying to achieve by graphing the metric of commits per company?
> 
> "What gets measured, gets managed" I am a firm believer of this simple 
> quote.

Okay, but my concern is the lack of context. I think we're putting the cart
before the horse in that we as a project need to decide what we want
to manage, determine the appropriate metrics and then perform the
measurements rather than cherry-pick things to measure and present
the data without context. We need to know what questions we're trying to
answer and there is no context available for your data as far as I'm aware,
certainly not at the current revision of the wiki page:

https://github.com/openbmc/openbmc/wiki/Project-Metrics/f15759a7ff5c61fa6713b5602dd0f40820b84d0e

> Measuring a project always improves it. That, and I have been 
> asked to start gathering metrics from several of our contributing 
> companies.

Where did this discussion occur? Can you provide a link?

> They appreciate it.
> 
> > It's also not clear to me what the inputs to these graphs are, for instance
> > whether changes to Linux, u-boot, qemu or other major projects that we
> > consume and contribute to are included or whether it's just repositories
> > under the openbmc org on github. If we're excluding upstream projects,
> > why?
> 
> It is only for contributions under openbmc. Other projects have been 
> excluded simply because they have their own project metrics. For example:
> 
> Linux: 
> https://www.linuxfoundation.org/blog/2016/08/the-top-10-developers-and-companies-contributing-to-the-linux-kernel-in-2015-2016/
> 
> uboot: 
> https://osfc.io/uploads/talk/paper/31/2019_State_U-Boot_development_report.pdf

Sure, but delegating to the upstream projects' statistics buries the
contributions that some are making that are driven by working on OpenBMC.
Often contribution to OpenBMC is by way of improving the kernel. Disregarding
this contributes to the lopsided view that your graphs present.

I'm concerned that we're trying to create a stick to beat contributing companies with
rather than working to find ways increase contributions for mutual benefit. Competition
works as a motivator when community members feel safe to take it on but I'm not sure
the community is mature enough for that to be true. Adding the context for your
statistics might help remove my concerns.

> 
> *Really nice* interactive openstack stats gui: https://www.stackalytics.com/
> 
> > Where are the scripts to reproduce the graphs? Can you contribute them
> > to openbmc-tools?
> 
> Eventually yes, if my employer will let me do more upstream. :) But, the 
> data is publicly available, you can get it yourself from gerrit. Simply 
> go to our gerrit dashboard and search something like: " status:merged 
> AND after:<date> AND before:<date> AND NOT topic:autobump AND 
> owner:<gerrit id> "
> 
> I appreciate your feedback, I will make the specifics of what is 
> measured and how it is done more clear on the project metrics wiki page.

Thanks, but please make sure to address the critical issue: What is the
question whose answer the data from these metrics supports?

Andrew