OpenBMC Project metrics

[krtaylor]

On 12/8/19 6:06 PM, Andrew Jeffery wrote:
> 
> 
> On Sat, 7 Dec 2019, at 00:03, krtaylor wrote:
>> On 12/4/19 4:33 PM, Andrew Jeffery wrote:
>>>
>>>
>>> On Thu, 5 Dec 2019, at 05:14, Kurt Taylor wrote:
>>>> All, I just posted the project merge metrics for September and
>>>> October. I will be updating the company/developer lists for November
>>>> and posting those shortly. Enjoy.
>>>>
>>>> https://github.com/openbmc/openbmc/wiki/Project-Metrics
>>>>
>>>> NOTE: these metrics should be used *very carefully*. They do not
>>>> represent the total contributions to the project. We value
>>>> contributions many that do not show up in these charts, including
>>>> reviews, mail list involvement, IRC involvement, etc.
>>>
>>> Given all the caveats and the lopsided view the graphs display, what
>>> are we trying to achieve by graphing the metric of commits per company?
>>
>> "What gets measured, gets managed" I am a firm believer of this simple
>> quote.
> 
> Okay, but my concern is the lack of context. I think we're putting the cart
> before the horse in that we as a project need to decide what we want
> to manage, determine the appropriate metrics and then perform the
> measurements rather than cherry-pick things to measure and present

Thanks for your opinions! As I have said, this is the first in several 
data points that will be gathered and presented, hence the caveats. I 
had to start somewhere.

> the data without context. We need to know what questions we're trying to
> answer and there is no context available for your data as far as I'm aware,
> certainly not at the current revision of the wiki page:
> 
> https://github.com/openbmc/openbmc/wiki/Project-Metrics/f15759a7ff5c61fa6713b5602dd0f40820b84d0e
> 
>> Measuring a project always improves it. That, and I have been
>> asked to start gathering metrics from several of our contributing
>> companies.
> 
> Where did this discussion occur? Can you provide a link?

The conversations have happen many times over the last 2 years.

At TSC and community meetings (not recorded in the meeting minutes that 
I could find, but it was discussed)

At release planning meetings (see minutes):
https://github.com/openbmc/openbmc/wiki/Release-Planning

I would rather not disclose email (without consent) that I have received 
privately from several companies supporting this work.

No one has ever had any objection to gathering this information (until 
now). Remember, anyone can go see this information any time they want.

> 
>> They appreciate it.
>>
>>> It's also not clear to me what the inputs to these graphs are, for instance
>>> whether changes to Linux, u-boot, qemu or other major projects that we
>>> consume and contribute to are included or whether it's just repositories
>>> under the openbmc org on github. If we're excluding upstream projects,
>>> why?
>>
>> It is only for contributions under openbmc. Other projects have been
>> excluded simply because they have their own project metrics. For example:
>>
>> Linux:
>> https://www.linuxfoundation.org/blog/2016/08/the-top-10-developers-and-companies-contributing-to-the-linux-kernel-in-2015-2016/
>>
>> uboot:
>> https://osfc.io/uploads/talk/paper/31/2019_State_U-Boot_development_report.pdf
> 
> Sure, but delegating to the upstream projects' statistics buries the
> contributions that some are making that are driven by working on OpenBMC.
> Often contribution to OpenBMC is by way of improving the kernel. Disregarding
> this contributes to the lopsided view that your graphs present.

Exactly my point. See carefully wording.
> 
> I'm concerned that we're trying to create a stick to beat contributing companies with
> rather than working to find ways increase contributions for mutual benefit. Competition
> works as a motivator when community members feel safe to take it on but I'm not sure
> the community is mature enough for that to be true. Adding the context for your
> statistics might help remove my concerns.

Honestly I'm surprised at this reaction to a *potential* situation I 
have never witnessed, but, I am willing to add any wording that you feel 
is necessary to create a safe development environment. I have 
participated in open source projects for 20+ years and personally was 
very motivated by contribution metrics, it made it really fun to see if 
I could do better in the community next month!

I value your feedback. When do you feel we as a community are mature 
enough to start monitoring reviews, commits, and other project data? 
Should we hide this "early" data until some future time when it 
represents everyone equally? Personally, I don't feel like we will ever 
get to that place. There will always be people that contribute more in 
one particular area than others and they just can't be upset that they 
may have done less. Open Source requires thick skin. Again, I've never 
seen project metrics reduce productivity/contributions.

Eventually, I'd like to break this data down by project and individual, 
not just company. And, also show data on other extremely valuable 
project contributions such as reviews, IRC meeting participation, 
testing/results, bug reports, maillist involvement and more. Its all 
valuable!

Thanks again!
Kurt Taylor (krtaylor)

>>
>> *Really nice* interactive openstack stats gui: https://www.stackalytics.com/
>>
>>> Where are the scripts to reproduce the graphs? Can you contribute them
>>> to openbmc-tools?
>>
>> Eventually yes, if my employer will let me do more upstream. :) But, the
>> data is publicly available, you can get it yourself from gerrit. Simply
>> go to our gerrit dashboard and search something like: " status:merged
>> AND after:<date> AND before:<date> AND NOT topic:autobump AND
>> owner:<gerrit id> "
>>
>> I appreciate your feedback, I will make the specifics of what is
>> measured and how it is done more clear on the project metrics wiki page.
> 
> Thanks, but please make sure to address the critical issue: What is the
> question whose answer the data from these metrics supports > Andrew
>