Adding support for custom SEL records

[Ed Tanous]

On Wed, Oct 19, 2022 at 11:05 AM Bills, Jason M
<jason.m.bills at linux.intel.com> wrote:
>
>
>
> On 10/19/2022 11:10 AM, Brad Bishop wrote:
> > Thanks Jason
> >
> > On Wed, Oct 19, 2022 at 09:50:47AM -0600, Bills, Jason M wrote:
> >
> >> Intel had a requirement to support storing at least 4000 log entries.
> >
> > Ok.  So is it fair to assume anyone using the DBus backend does not have
> > this requirement?
>
> That is my assumption, yes.
> >
> >> At the time, we were able to get about 400 entries on D-Bus before
> >> D-Bus performance became unusable.
> >
> > To anyone using the DBus backend - have you observed similar performance
> > issues?
> >
> > Jason is there a testcase or scenario I can execute to highlighht the
> > issues you refer to concretely?  Maybe something like "create 4000 sels,
> > run ipmitool and see how long it takes?"
>
> To clarify, my understanding is the D-Bus performance issues were not
> isolated to just IPMI.  All of D-Bus for every BMC service was impacted.
>
> If I remember correctly, Ed Tanous is who did the initial evaluation, so
> he may have more detail.  But I think it was similar to what you
> suggest: Create 4000 logs on D-Bus and check the performance.  This
> could be done with ipmitool.


>From what I recall, the requirements were:
- Ability to store 4000 logs in a rotating buffer (original desire was
10,000, but 4000 was picked as a middle-ground that could be
implemented).
- Ability to log 100+ entries per second, including when buffers get
overwritten.
- (abstract) Log storage should be aware of hardware limitations (SPI
flash cell write endurance) and allow writing N logs per minute for
the lifetime of the machine without hardware failure.  (I forget the
value of N).
- "ipmitool sensor list" should return the results from a full sel log
in less than 1 second (renegotiated from 200ms, the faster the
better).
- The logging implementation should be able to support a well-formed,
version controlled, Redfish MessageRegistry to the DMTF
specifications.
- The logging implementation should be able to implement a
well-formed, stable, and ACID compliant IPMI SEL command
implementation.

I don't believe the current DBus implementation can meet the previous
requirements, but if it's capable of that these days, it would be
excellent to consolidate.

> >
> >> I'd also be curious about the reverse question.  Is there any benefit
> >> to storing logs on D-Bus that makes it a better solution?
> >
> > Yes, this is exactly the question I've been trying to ask.  The answer
> > seems only to be that the code is in meta-intel/intel-ipmi-oem - but
> > that is easily fixed by moving the code to
> > meta-phosphor/phosphor-host-ipmid.
> >
> >> At the risk of complicating things more (https://xkcd.com/927/), D-Bus
> >> was the primary solution when Intel joined.  We created the rsyslog
> >> approach because of the limitation imposed by D-Bus.  But I know there
> >> are still those who don't like the rsyslog approach.  Is there a way
> >> we can now get together and define a new logging solution that is
> >> fully upstream and avoids the drawbacks of both existing solutions?
> >
> > I hope so, because doing that would make things a lot easier for our
> > users adopting OpenBMC.
>
> My main requirements are to store many logs (at least 4000 was the
> original number, but I can try to get an updated number if needed) and
> have them persist across BMC reboots.
>
> We currently accomplish this using rsyslog to extract logs from the
> journal and store them in a persistent text file.
>
> How is best to approach starting a new design discussion?  Should we
> continue discussing in this thread?  Start a design doc review?
> Something else?
> >
> > Thanks,
> > brad