Adding support for custom SEL records

[Bills, Jason M]

On 10/24/2022 2:19 PM, Ed Tanous wrote:
> On Mon, Oct 24, 2022 at 12:03 PM Brad Bishop
> <bradleyb at fuzziesquirrel.com> wrote:
>>
>> This is helpful, thanks Ed.
>>
>> On Mon, Oct 24, 2022 at 10:59:51AM -0700, Ed Tanous wrote:
>> >From what I recall, the requirements were:
>>> - Ability to store 4000 logs in a rotating buffer (original desire was
>>> 10,000, but 4000 was picked as a middle-ground that could be
>>> implemented).
>>
>> A DBus object based implementation could meet this requirement, right?
>>
>>> - Ability to log 100+ entries per second, including when buffers get
>>> overwritten.
>>
>> I guess I would not be shocked if DBus objects + serialization might not
>> be able to sustain this rate of incoming logs.  Maybe it depends on the
>> filesystem or how the data is serialized in the filesystem.  The DBus
>> approach creates many files.  Obviously the syslog approach is only
>> using a couple of files.
>>
>> Do you think this kind of requirement is typical?  Quoting Patrick from
>> another thread here:
> 
> There are cases where every sensor will cross a threshold at the same
> moment (generally because a threshold was changed).  Is this a
> "typical" use case, probably not, but if it's trivial to DOS the BMC
> with events with a couple commands, it's not great.
> 
>>
>>> don't think anyone is intending to create 10k events in the span of
>>> a minute
> 
> Intending to, no.  Created in an error case, or in the case of a
> failing piece of hardware, it can happen.
> 
>>
>> 100/s is only 6k in a minute but that is getting pretty close...
>>
>>> - (abstract) Log storage should be aware of hardware limitations (SPI
>>> flash cell write endurance) and allow writing N logs per minute for
>>> the lifetime of the machine without hardware failure.  (I forget the
>>> value of N).
>>
>> Do you think the rsyslog implementation does better at this?  Why?
> 
> @Jason?  I know there were a lot of worries about write expansion (N
> byte sel log turning into 100XN bytes of JSON) wearing out flash
> faster, and having some interactions with jffs2.
> 

Yes.  We ran into this issue both with trying to use individual files 
per SEL entry and with persisting the journal.  At the time, we only had 
4MB of SPI flash space for persistent storage, and both of those methods 
quickly filled that up.

We now have 8MB, but only a small part of that can be dedicated to 
persisting logs, so we still have a requirement that persisting many 
logs should not consume too much persistent storage space.

>>
>>> - "ipmitool sensor list" should return the results from a full sel log
>>> in less than 1 second (renegotiated from 200ms, the faster the
>>> better).
>>
>> Ok, again I would not be shocked if DBus objects weren't able to deliver
>> on this.
>>
>>> - The logging implementation should be able to support a well-formed,
>>> version controlled, Redfish MessageRegistry to the DMTF
>>> specifications.
>>
>> Do you think a DBus object based implementation could meet this
>> requirement?
> 
> Today, as defined, without duplicating the logging strings, no, but
> DBus is just an IPC;  I'm fairly certain we could define a DBus based
> logging mechanism that met these requirements, but the key is that the
> Redfish instance (bmcweb in this case) needs a-priori knowledge of
> every possible log that the system can publish, and version them with
> DMTF-appropriate semver (existing messages changes rev the patch
> version, new messages revs the subminor version).  The existing
> rsyslog implementation places the log strings into bmcweb, so the only
> thing being transferred over the IPC is the MessageId and the
> specific-instance arguments, which in theory, increases the
> performance quite a bit, and avoids duplicating the strings in two
> places.
> 
>>
>>> - The logging implementation should be able to implement a
>>> well-formed, stable, and ACID compliant IPMI SEL command
>>> implementation.
>>
>> Do you think a DBus object based implementation could meet this
>> requirement?
> 
> Same answer, with today's code, no.  DBus is just an IPC, we could
> potentially define one.
> 
>>
>>>
>>> I don't believe the current DBus implementation can meet the previous
>>> requirements,
>>
>> The motivation of my questions above is to understand which requirements
>> cannot be met by something based on DBus objects.
> 
> Cool.
> 
> 
>>
>> Thanks,
>> brad