OpenBMC on RCS platforms

[Ed Tanous]

On Fri, Apr 23, 2021 at 12:00 PM Timothy Pearson
<tpearson at raptorengineering.com> wrote:
>
>
>
> ----- Original Message -----
> > From: "Ed Tanous" <ed at tanous.net>
> > To: "Timothy Pearson" <tpearson at raptorengineering.com>
> > Cc: "openbmc" <openbmc at lists.ozlabs.org>
> > Sent: Friday, April 23, 2021 12:23:23 PM
> > Subject: Re: OpenBMC on RCS platforms
>
> > On Fri, Apr 23, 2021 at 7:36 AM Timothy Pearson
> > <tpearson at raptorengineering.com> wrote:
> >>
> >
> > First off, this is great feedback, and despite some of my comments
> > below, I do really appreciate you putting it out there.
> >
> >> All,
> >>
> >> I'm reaching out after some internal discussion on how we can better integrate
> >> our platforms with the OpenBMC project.  As many of you may know, we have been
> >> using OpenBMC in our lineup of OpenPOWER-based server and desktop products,
> >> with a number of custom patches on top to better serve our target markets.
> >>
> >> While we have had fairly good success with OpenBMC in the server / datacenter
> >> space, reception has been lukewarm at best in the desktop space.  This is not
> >> too surprising, given OpenBMC's historical focus on datacenter applications,
> >> but it is also becoming an expensive technical and PR pain point for us as the
> >> years go by.  To make matters worse, we're still shielding our desktop /
> >> workstation customer base to some degree from certain design decisions that
> >> persist in upstream OpenBMC, and we'd like to open discussion on all of these
> >> topics to see if a resolution can be found with minimal wasted effort from all
> >> sides.
> >>
> >> Roughly speaking, we see issues in OpenBMC in 5 main areas:
> >>
> >>
> >> == Fan control ==
> >>
> >> Out of all of the various pain points we've dealt with over the years, this has
> >> proven the most costly and is responsible on its own for the lack of RCS
> >> platforms upstream in OpenBMC.
> >>
> >> To be perfectly frank, OpenBMC's current fan control subsystem is a technical
> >> embarrassment, and not up to the high quality seen elsewhere in the project.
> >
> > Which fan control subsystem are you referring to?  Phosphor-fans or
> > phosphor-pid-control?
> >
> >>  Worse, this multi-daemon DBUS-interconnected Rube Goldberg contraption has
> >>  somehow managed to persist over the past 4+ years, likely because it reached a
> >>  complexity level where it is both tightly integrated with the rest of the
> >>  OpenBMC system and extremely difficult to understand, therefore it is equally
> >>  difficult to replace.  Furthering the lack of progress is the fact that it is
> >>  mostly "working" for datacenter applications, so there may be a "don't touch
> >>  what isn't broken" mentality in play.
> >
> > I'm not really sure I agree with that.  If someone came with a design
> > for "We should replace dbus with X", had good technical foundations
> > for why X was better, and was putting forward the monumental effort to
> > do the work, I know that I personally wouldn't be opposed.  For the
> > record, I agree with you about the complexity here, but most of the
> > ideas I've heard to make it better were "Throw everything out and
> > start over", which, if that's what you want to do, by all means do,
> > but I don't think the community is willing to redo all of the untold
> > hours of engineering effort spent over the years the project has
> > existed.
> >
> > FWIW, u-bmc was a project that took the existing kernel, threw out all
> > the userspace and started over.  From my view outside the project,
> > they seem to have failed to gain traction, and only support a couple
> > of platforms.
> >
> >>  From a technical perspective, it is indirected to a sufficient level as to be
> >>  nearly incomprehensible to most people, with the source spread across multiple
> >>  different projects and repositories, yet somehow it remains rigid / fragile
> >>  enough to not support basic features like runtime (or even post-compile) fan
> >>  configuration for a given server.
> >
> > With respect, this statement is incorrect.  On an entity-manager
> > enabled system + phosphor-pid-control, all of the fan control
> > parameters are fully modifiable at runtime either from within the
> > system (through dbus) or through Redfish out of band through the
> > OEMManager API.  If you haven't ported your systems to entity-manager
> > yet, there's quite a bit of people doing it at the moment and are
> > discussing this stuff on discord basically every day that I'm sure
> > would be able to give you some direction on where to start getting
> > your systems moved over.
>
> <snip>
>
> Interesting.  I assume entity-manager is pretty new still?

It's a couple years old at this point (I think first commit was in
2018?).  It has certainly gotten more momentum over time though.

>  A year ago there was zero solution to the problem of runtime configuration, and when I checked several weeks ago the bug report on it [1] had no meaningful progress.

Bug reports aren't generally the best way to get answers in my
experience, especially if it's not a "bug" but an enhancement you want
to make to the overall architecture. The mailing list or discord tends
to get better responses (as you've seen here in this thread).

For what it's worth, Redfish configurable PID loops were checked in
back in October of 2018, so about 2 and a half years old now.
https://github.com/openbmc/bmcweb/commit/af996fe4d12668d1a096e36e791c49690e54c9bb

>  Looks like that's finally changing.
>
> Is the entity manager fairly stable API-wise at this point?

While we do our best to not make backward incompatible configuration
changes (I can't think of any we've done yet) we don't guarantee it,
and certainly can't make any stability guarantees about code we can't
see.  The best way to keep your systems stable is to get them
upstreamed, so when we need to make "might break things" type changes,
we'll have a good idea if anyone is actually using the features in
question, and which systems we should ask maintainers to test changes
against.

More details under "Intent" heading item #3 here:
https://github.com/openbmc/entity-manager/blob/21608383661285e63e97c0457f55817f6e1d6b92/CONFIG_FORMAT.md

>  That might be enough of a game changer for me to go back and get approval for what will effectively be our fourth port of the Talos II systems to OpenBMC.

Glad to see you're interested.

>
> [1] https://github.com/openbmc/openbmc/issues/3595