Upstreaming downstream Google BMC repositories

[Brandon Kim]

Hi everyone,

Wanted to ping this thread to see if there were more concerns on creating
an openbmc/google-misc repository.

Thanks!
Brandon

On Mon, Jan 11, 2021 at 1:51 PM Ed Tanous <ed at tanous.net> wrote:

> On Mon, Jan 11, 2021 at 1:14 PM Patrick Williams <patrick at stwcx.xyz>
> wrote:
> >
> > Unfortunately we don't have a great policy on any of this.  Hopefully it
> > is something we can come up with a better definition on in the near
> > future.
> >
> > On Thu, Jan 07, 2021 at 01:20:00PM -0800, Ed Tanous wrote:
> > > On Thu, Jan 7, 2021 at 10:26 AM Paul Menzel <pmenzel at molgen.mpg.de>
> wrote:
> > > > Am 07.01.21 um 18:33 schrieb Benjamin Fair:
> > > > > On Thu, 7 Jan 2021 at 00:09, Paul Menzel <pmenzel at molgen.mpg.de>
> wrote:
> > > > >> Am 07.01.21 um 02:49 schrieb Brandon Kim:
> > > > >>
> > > > >>> We're exploring ways of upstreaming some of the downstream
> repositories
> > > > >>> from Google to openbmc/* .
> > > > >>>
> > > > >>> Half, if not most of the downstream repositories are C++ daemons
> that are
> > > > >>> specific to Google so we didn't want to create a bunch of new
> > > > >>> openbmc/<repo> that no one would use.
> >
> > Just out of curiousity, if they're not going to be useful to anyone
> > except Google, what is the utility of getting them into an openbmc repo?
> > (There are reasons but I don't want to put words in your mouths)
>
> A slight clarification: the theory is they're not useful unless you're
> building a machine that intends to live within the Google
> infrastructure (similar to zaius, or q71l).  The meta layer _is_
> useful outside of google, with companies that design and build the
> aforementioned platforms.  Having the specific tweaks made available
> in the public means that the companies we work with can build 1:1 the
> image that we're operating with, report bugs against it more publicly,
> and we can share more code in the open, without resorting to
> public/private forks of OpenBMC for our own purposes which have their
> own problems as has been proven in the past.
>
> The other hope is that if we're wrong, and something within that repo
> is useful outside of google (seems likely it might happen for
> something), it's available with a public license and whoever finds it
> useful can simply move it to a common repo where others can use it,
> with minimal fuss, or asking us to upstream something we've built in
> the past.
>
> >
> > > > >>> An idea that Ed gave me was having something like
> openbmc/google-misc
> > > > >>> repository for all these repositories and if there are any that
> seem useful
> > > > >>> to others, we can break it out into a different, separate
> repository in
> > > > >>> openbmc/* layer.
> > > > >>>
> > > > >>> Please let me know if this seems like a good idea and I'm open
> to other
> > > > >>> suggestions!
> >
> > I really dislike dumping-ground repositories.  If we're going to allow
> > company-specific "misc" repositories, I would really like a policy that
> > they may *only* be referenced from that company's meta-layer.
>
> That sounds completely reasonable, and in-line with our intent.
>
> >  If anyone
> > has any use in that code it really should be broken out into its own
> > repository with a proper maintenance structure.
>
>  +1.
>
> >
> > > > >> Thank you very much for putting in the effort to make these
> repositories
> > > > >> public.
> > > > >>
> > > > >> Using the openbmc/google-misc approach, how would the git history
> > > > >> (commit log) be handled?
> > > > >>
> > > > >> Personally, I would prefer having small repositories as git makes
> that
> > > > >> very easy to handle. Also it might save you time, as you do not
> have to
> > > > >> think about what to do with the git history, and do not have to
> merge it.
> > > > >
> > > > > We would most likely squash the history together, in case there's
> > > > > something confidential or private in the earlier commits.
> > > >
> > > > Understood. If that could be avoided, and only the confidential stuff
> > > > removed, that would be great, as the git history gives a lot of
> insight
> > > > into design decisions.
> > >
> > > It should be noted, there isn't really much git history to speak of
> > > for the things we're looking at pushing.
> >
> > How was any code of significance developed without a git history?  It is
> > unfortunate we're going to lose all of this because of how often I tend
> > to dig through 'git-blame' to understand the "why" on code.
>
> A lot of the commits aren't really "openbmc worthy" in that they're
> not signed off, have no tested statement, might have > 72 character
> lines ect.  They definitely do have a git history, and if the
> preference is that we push it with the messy history, we can certainly
> do it.  I don't have a strong opinion here other than not wanting to
> rewrite and retest every commit we've had to these things.
>
> >
> > > Some examples of things that would go in this repository.
> > > 1. The google public keys/certs.  I would hope that non-google systems
> > > are using their own root certificates.
> > > 2. Disabling of ssh login flows.  This is done in a very specific way
> > > that requires interfacing with out of network components and protocols
> > > that are specific to our systems.  I'd be surprised if anyone found
> > > this useful.
> > > 3. In-band telemetry code implementing interfaces for interfacing to
> > > google infrastructure.  These haven't been built yet, but will likely
> > > be a translation from the public facing APIs (Dbus/redfish/ipmi) to
> > > interface them to google infrastructure.  it's unlikely anyone else
> > > would use this.
> >
> > These make me more curious on the value of opening them.
> >
> > > > > Many small repos would be easy to handle for us, but OpenBMC may
> not
> > > > > want to have lots of small Google-specific repos in their org as
> this
> > > > > may make it more cumbersome for others to find the relevant repos
> that
> > > > > they're interested in.
> > > >
> > > > Understood. On the other, with small repositories, they can only use
> the
> > > > parts they need.
> >
> > I'm more comfortable with others utilizing this code if it is in a small
> > repo like "google-ssh-cert".  As others find it valuable we can rename
> > the repo.
>
> Right now the process to create new repositories takes a very long
> time, and requires interaction with core maintainers for
> CI/gerrit/github/user groups setup.  That model doesn't scale beyond
> what we have today if things like certs for every company needs its
> own repository.  I don't see a way to make it scale to the "new repo
> for everything" model, unless you had some ideas there.
>
> In other tracks I've had good luck simply extracting the history from
> a subfolder, and pushing that to a new repo when things needed to be
> split out.
>
> >
> > > See above, if there are pieces that people want to use on non-google
> > > systems, they don't belong in meta-google.  With that said, your
> > > statement is incorrect, recipes are not required to be 1:1 with
> > > repositories.  Multiple recipes can point at subfolders of the same
> > > repository, allowing you to "use the parts they need" by simply adding
> > > recipes.  With that said, this is not the intent, and I would much
> > > rather move code to a more common layer (meta-phosphor for example)
> > > rather than have non google systems including meta-google in their
> > > bblayers.conf.
> > >
> > > >
> > > > > There's also overhead for the project maintainers to create the
> > > > > relevant groups and permissions for each new repo.
> > > > Please note, that Without knowing the contents of the repositories
> and
> > > > the size, this is all just my opinion. If the OpenBMC “admins“ can
> > > > easily create several repositories, I’d prefer that route.
> > >
> > > Today creating new repos is non-trivial, and IMO we already have too
> > > many of them, which is causing a lot of thrash.  I'd really like to
> > > see us start consolidating some of these (see my patches to
> > > consolidate all the meta-layers into openbmc/openbmc with the owners
> > > plugin)
> >
> > What do you mean by "thrash"?  Ideally it should be cheap to create a
> > repository.
>
> Ideally, yes, but we're not there today, and I don't see a path to get us
> there.
>
> >  If there is significant overhead to create a repository
> > with the current infrastructure we should document those challenges and
> > look to improve them.
>
> The biggest challenge is that a large percentage of the work needs to
> be done by relatively few people (those with core permissions), and
> they have their own things to get done, and  rightfully aren't able to
> prioritize creating new repos/permissions/other stuff.  This topic
> alone is probably worth an email thread, as it's worth trying to
> tackle;  I'll try to get that written up.
>
> >
> > I don't have any issue with consolidation of the meta-layers because
> > those are effectively all built together anyhow.  Right now I'm not in
> > favor of consolidation of code repositories and we've even talked about
> > splitting out some pieces (EM and fru-device come to mind to me).
>
> We talked about splitting those up?  I'd be a little worried about
> that, as they're pretty intertwined in their dbus interfaces.
>
> We're currently having the problem where nearly any new feature in
> dbus-sensors needs a commit to entity-manager to add a new schema, and
> that's non-obvious, and requires a commit done in lockstep to get code
> through.  I had actually considered the other day doing the opposite,
> and proposing merging entity-manager and dbus-sensors, but that's a
> different discussion.
>
> >  Can
> > you quantify what the advantage of a big[ger] multi-function repositories
> > are?
>
> The biggest advantage is time-to-create new repositories and the
> significant reduction in autobump-like recipe changes that need to be
> made to keep everything in lockstep.
>
> >
> > --
> > Patrick Williams
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ozlabs.org/pipermail/openbmc/attachments/20210114/8816843d/attachment-0001.htm>