Upstreaming downstream Google BMC repositories

[Ed Tanous]

No concerns from me.

-Ed

On Thu, Jan 14, 2021 at 3:18 PM Brandon Kim <brandonkim at google.com> wrote:
>
> Hi everyone,
>
> Wanted to ping this thread to see if there were more concerns on creating an openbmc/google-misc repository.
>
> Thanks!
> Brandon
>
> On Mon, Jan 11, 2021 at 1:51 PM Ed Tanous <ed at tanous.net> wrote:
>>
>> On Mon, Jan 11, 2021 at 1:14 PM Patrick Williams <patrick at stwcx.xyz> wrote:
>> >
>> > Unfortunately we don't have a great policy on any of this.  Hopefully it
>> > is something we can come up with a better definition on in the near
>> > future.
>> >
>> > On Thu, Jan 07, 2021 at 01:20:00PM -0800, Ed Tanous wrote:
>> > > On Thu, Jan 7, 2021 at 10:26 AM Paul Menzel <pmenzel at molgen.mpg.de> wrote:
>> > > > Am 07.01.21 um 18:33 schrieb Benjamin Fair:
>> > > > > On Thu, 7 Jan 2021 at 00:09, Paul Menzel <pmenzel at molgen.mpg.de> wrote:
>> > > > >> Am 07.01.21 um 02:49 schrieb Brandon Kim:
>> > > > >>
>> > > > >>> We're exploring ways of upstreaming some of the downstream repositories
>> > > > >>> from Google to openbmc/* .
>> > > > >>>
>> > > > >>> Half, if not most of the downstream repositories are C++ daemons that are
>> > > > >>> specific to Google so we didn't want to create a bunch of new
>> > > > >>> openbmc/<repo> that no one would use.
>> >
>> > Just out of curiousity, if they're not going to be useful to anyone
>> > except Google, what is the utility of getting them into an openbmc repo?
>> > (There are reasons but I don't want to put words in your mouths)
>>
>> A slight clarification: the theory is they're not useful unless you're
>> building a machine that intends to live within the Google
>> infrastructure (similar to zaius, or q71l).  The meta layer _is_
>> useful outside of google, with companies that design and build the
>> aforementioned platforms.  Having the specific tweaks made available
>> in the public means that the companies we work with can build 1:1 the
>> image that we're operating with, report bugs against it more publicly,
>> and we can share more code in the open, without resorting to
>> public/private forks of OpenBMC for our own purposes which have their
>> own problems as has been proven in the past.
>>
>> The other hope is that if we're wrong, and something within that repo
>> is useful outside of google (seems likely it might happen for
>> something), it's available with a public license and whoever finds it
>> useful can simply move it to a common repo where others can use it,
>> with minimal fuss, or asking us to upstream something we've built in
>> the past.
>>
>> >
>> > > > >>> An idea that Ed gave me was having something like openbmc/google-misc
>> > > > >>> repository for all these repositories and if there are any that seem useful
>> > > > >>> to others, we can break it out into a different, separate repository in
>> > > > >>> openbmc/* layer.
>> > > > >>>
>> > > > >>> Please let me know if this seems like a good idea and I'm open to other
>> > > > >>> suggestions!
>> >
>> > I really dislike dumping-ground repositories.  If we're going to allow
>> > company-specific "misc" repositories, I would really like a policy that
>> > they may *only* be referenced from that company's meta-layer.
>>
>> That sounds completely reasonable, and in-line with our intent.
>>
>> >  If anyone
>> > has any use in that code it really should be broken out into its own
>> > repository with a proper maintenance structure.
>>
>>  +1.
>>
>> >
>> > > > >> Thank you very much for putting in the effort to make these repositories
>> > > > >> public.
>> > > > >>
>> > > > >> Using the openbmc/google-misc approach, how would the git history
>> > > > >> (commit log) be handled?
>> > > > >>
>> > > > >> Personally, I would prefer having small repositories as git makes that
>> > > > >> very easy to handle. Also it might save you time, as you do not have to
>> > > > >> think about what to do with the git history, and do not have to merge it.
>> > > > >
>> > > > > We would most likely squash the history together, in case there's
>> > > > > something confidential or private in the earlier commits.
>> > > >
>> > > > Understood. If that could be avoided, and only the confidential stuff
>> > > > removed, that would be great, as the git history gives a lot of insight
>> > > > into design decisions.
>> > >
>> > > It should be noted, there isn't really much git history to speak of
>> > > for the things we're looking at pushing.
>> >
>> > How was any code of significance developed without a git history?  It is
>> > unfortunate we're going to lose all of this because of how often I tend
>> > to dig through 'git-blame' to understand the "why" on code.
>>
>> A lot of the commits aren't really "openbmc worthy" in that they're
>> not signed off, have no tested statement, might have > 72 character
>> lines ect.  They definitely do have a git history, and if the
>> preference is that we push it with the messy history, we can certainly
>> do it.  I don't have a strong opinion here other than not wanting to
>> rewrite and retest every commit we've had to these things.
>>
>> >
>> > > Some examples of things that would go in this repository.
>> > > 1. The google public keys/certs.  I would hope that non-google systems
>> > > are using their own root certificates.
>> > > 2. Disabling of ssh login flows.  This is done in a very specific way
>> > > that requires interfacing with out of network components and protocols
>> > > that are specific to our systems.  I'd be surprised if anyone found
>> > > this useful.
>> > > 3. In-band telemetry code implementing interfaces for interfacing to
>> > > google infrastructure.  These haven't been built yet, but will likely
>> > > be a translation from the public facing APIs (Dbus/redfish/ipmi) to
>> > > interface them to google infrastructure.  it's unlikely anyone else
>> > > would use this.
>> >
>> > These make me more curious on the value of opening them.
>> >
>> > > > > Many small repos would be easy to handle for us, but OpenBMC may not
>> > > > > want to have lots of small Google-specific repos in their org as this
>> > > > > may make it more cumbersome for others to find the relevant repos that
>> > > > > they're interested in.
>> > > >
>> > > > Understood. On the other, with small repositories, they can only use the
>> > > > parts they need.
>> >
>> > I'm more comfortable with others utilizing this code if it is in a small
>> > repo like "google-ssh-cert".  As others find it valuable we can rename
>> > the repo.
>>
>> Right now the process to create new repositories takes a very long
>> time, and requires interaction with core maintainers for
>> CI/gerrit/github/user groups setup.  That model doesn't scale beyond
>> what we have today if things like certs for every company needs its
>> own repository.  I don't see a way to make it scale to the "new repo
>> for everything" model, unless you had some ideas there.
>>
>> In other tracks I've had good luck simply extracting the history from
>> a subfolder, and pushing that to a new repo when things needed to be
>> split out.
>>
>> >
>> > > See above, if there are pieces that people want to use on non-google
>> > > systems, they don't belong in meta-google.  With that said, your
>> > > statement is incorrect, recipes are not required to be 1:1 with
>> > > repositories.  Multiple recipes can point at subfolders of the same
>> > > repository, allowing you to "use the parts they need" by simply adding
>> > > recipes.  With that said, this is not the intent, and I would much
>> > > rather move code to a more common layer (meta-phosphor for example)
>> > > rather than have non google systems including meta-google in their
>> > > bblayers.conf.
>> > >
>> > > >
>> > > > > There's also overhead for the project maintainers to create the
>> > > > > relevant groups and permissions for each new repo.
>> > > > Please note, that Without knowing the contents of the repositories and
>> > > > the size, this is all just my opinion. If the OpenBMC “admins“ can
>> > > > easily create several repositories, I’d prefer that route.
>> > >
>> > > Today creating new repos is non-trivial, and IMO we already have too
>> > > many of them, which is causing a lot of thrash.  I'd really like to
>> > > see us start consolidating some of these (see my patches to
>> > > consolidate all the meta-layers into openbmc/openbmc with the owners
>> > > plugin)
>> >
>> > What do you mean by "thrash"?  Ideally it should be cheap to create a
>> > repository.
>>
>> Ideally, yes, but we're not there today, and I don't see a path to get us there.
>>
>> >  If there is significant overhead to create a repository
>> > with the current infrastructure we should document those challenges and
>> > look to improve them.
>>
>> The biggest challenge is that a large percentage of the work needs to
>> be done by relatively few people (those with core permissions), and
>> they have their own things to get done, and  rightfully aren't able to
>> prioritize creating new repos/permissions/other stuff.  This topic
>> alone is probably worth an email thread, as it's worth trying to
>> tackle;  I'll try to get that written up.
>>
>> >
>> > I don't have any issue with consolidation of the meta-layers because
>> > those are effectively all built together anyhow.  Right now I'm not in
>> > favor of consolidation of code repositories and we've even talked about
>> > splitting out some pieces (EM and fru-device come to mind to me).
>>
>> We talked about splitting those up?  I'd be a little worried about
>> that, as they're pretty intertwined in their dbus interfaces.
>>
>> We're currently having the problem where nearly any new feature in
>> dbus-sensors needs a commit to entity-manager to add a new schema, and
>> that's non-obvious, and requires a commit done in lockstep to get code
>> through.  I had actually considered the other day doing the opposite,
>> and proposing merging entity-manager and dbus-sensors, but that's a
>> different discussion.
>>
>> >  Can
>> > you quantify what the advantage of a big[ger] multi-function repositories
>> > are?
>>
>> The biggest advantage is time-to-create new repositories and the
>> significant reduction in autobump-like recipe changes that need to be
>> made to keep everything in lockstep.
>>
>> >
>> > --
>> > Patrick Williams