Link phosphor-hostlogger and bmcweb

[Ed Tanous]

On Fri, May 21, 2021 at 8:11 AM Nan Zhou <nanzhou at google.com> wrote:
>
> On Thu, May 20, 2021 at 04:29:09PM -0700, Nan Zhou wrote:
>>
>> > Hi all,
>> >
>> > In the previous thread (
>> > https://lists.ozlabs.org/pipermail/openbmc/2021-March/025234.html), we
>> > (engineers from Google and Quanta) discussed our attempt to share host
>> > serial logs via Redfish, which includes polling logs via LogService and
>> > streaming log bytes via EventService (e.g. SSE). We went with the event log
>> > architecture
>> > <https://github.com/openbmc/docs/blob/master/architecture/redfish-logging-in-bmcweb.md>
>> > and did the implementation.
>> >
>> > We still want to reuse the phosphor-hostlogger and do some modification. We
>> > believe it is better to try to reuse existing codes if possible and improve
>> > them rather than creating new things that have similar functionalities (in
>> > our case, it is a daemon that could collect logs and persist them).
>> I agree, reusing code is a right choice, but only when it is really possible.
>> For now it looks like you want to remove most of the Hostlogger features to
>> transform it from buffer-like to stream-like service.
>
> Hostlogger will still consume bytes from obmc-console-server and persist logs in files. If the log itself has EOL, it will be splitted in the file as well.
> I thought that is all things we want in the Hostlogger.
>
>> > We want to do the following modification in phosphor-hostlogger (from the
>> > input and output point of view, just very few things will be changed)
>> >
>> > 1. One limitation of phosphor-hostlogger is that it will lose data in the
>> > memory (the ring buffer maintained by hostlogger) when BMC gets force
>> > restarted;
>> When BMC goes to reboot it stops all services, at that moment hostlogger gets
>> a signal and flushes all gathered logs to a file.
>
> Yes, I understand hostlogger registers a sigterm handler. But what if BMC lost its power before sigterm gets sent? That's what I mean by "force restart".
>
> > we propose to remove the ring buffer and write to the log file
> > as soon as some characters are received.
>
>> I am not sure it is a good idea.
>> The host can generate a lot of logs, but we have very limited free space. In
>> addition, such heavy traffic will lead to a quick breakdown of the flash (most
>> available products are guaranteed to withstand around 100,000 P/E cycles only).
>
> I would like to get input from Ed for this point. +Ed Tanous.

Gave my input on the previous email before I read this one.  Lets
follow this up on that thread.

>
>> > This implicitly removes the needs
>> > of the ring buffer, and the persistence triggering (host reboot, sigterm,
>> > etc) in hostlogger. We believe this keeps the same functionality but saves
>> > hundreds of lines of codes in phosphor-hostlogger.
>> You are suggesting to delete the buffer, DBus watcher, log rotate. How are you
>> going to keep the same functionality if you remove everything related to it?
>
> Log rotation is kept. Ring buffer and DBus watcher are great in the original design but become unnecessary if we persist logs immediately. Do we miss
> anything that the previous hostlogger provides (transfer bytes from the unix socket to EOL-separated compressed logs)? I believe in this proposal we have
> less data loss, simpler codes, and make it more performant for Redfish integration.
>
> On Thu, May 20, 2021 at 11:10 PM Artem Senichev <artemsen at gmail.com> wrote:
>>
>> On Thu, May 20, 2021 at 04:29:09PM -0700, Nan Zhou wrote:
>> > Hi all,
>> >
>> > In the previous thread (
>> > https://lists.ozlabs.org/pipermail/openbmc/2021-March/025234.html), we
>> > (engineers from Google and Quanta) discussed our attempt to share host
>> > serial logs via Redfish, which includes polling logs via LogService and
>> > streaming log bytes via EventService (e.g. SSE). We went with the event log
>> > architecture
>> > <https://github.com/openbmc/docs/blob/master/architecture/redfish-logging-in-bmcweb.md>
>> > and did the implementation.
>> >
>> > We still want to reuse the phosphor-hostlogger and do some modification. We
>> > believe it is better to try to reuse existing codes if possible and improve
>> > them rather than creating new things that have similar functionalities (in
>> > our case, it is a daemon that could collect logs and persist them).
>>
>> I agree, reusing code is a right choice, but only when it is really possible.
>> For now it looks like you want to remove most of the Hostlogger features to
>> transform it from buffer-like to stream-like service.
>>
>> > We want to do the following modification in phosphor-hostlogger (from the
>> > input and output point of view, just very few things will be changed)
>> >
>> > 1. One limitation of phosphor-hostlogger is that it will lose data in the
>> > memory (the ring buffer maintained by hostlogger) when BMC gets force
>> > restarted;
>>
>> When BMC goes to reboot it stops all services, at that moment hostlogger gets
>> a signal and flushes all gathered logs to a file.
>>
>> > we propose to remove the ring buffer and write to the log file
>> > as soon as some characters are received.
>>
>> I am not sure it is a good idea.
>> The host can generate a lot of logs, but we have very limited free space. In
>> addition, such heavy traffic will lead to a quick breakdown of the flash (most
>> available products are guaranteed to withstand around 100,000 P/E cycles only).
>>
>> > This implicitly removes the needs
>> > of the ring buffer, and the persistence triggering (host reboot, sigterm,
>> > etc) in hostlogger. We believe this keeps the same functionality but saves
>> > hundreds of lines of codes in phosphor-hostlogger.
>>
>> You are suggesting to delete the buffer, DBus watcher, log rotate. How are you
>> going to keep the same functionality if you remove everything related to it?
>>
>> > 2. We propose not to compress the latest log file. This saves us the
>> > overhead of doing decompression when BMCWeb just needs to retrieve the most
>> > recent logs. There are still going to be log rotations in the file level.
>> > Files other than the latest log file are still going to be compressed. We
>> > can modify existing codes to achieve this or use the linux logrotate
>> > directly.
>> >
>> > Furthermore, we will add host serial logs into BMCWeb, both LogService and
>> > EventService. In LogService, we will teach BMCWeb how to read the latest
>> > log file that is not compressed and the other compressed old logs, and how
>> > to assemble LogEntries out of raw serial logs. We will discuss EventService
>> > in future threads but the very initial idea is to setup inotify on log
>> > files and SSE to stream out new bytes to clients (like what the existing
>> > event logging is doing).
>> >
>> > As we said above, for phosphor-hostlogger, the input is still the
>> > obmc-server unix socket, and the output are still persisted log files. But
>> > the functionality will get improved (less data loss), code gets simplified
>> > (no ring buffer and persistence triggering), and it will become easier and
>> > more performant to get BMCWeb hooked up for log streaming via Redfish.
>> >
>> > Please let us know what you think. We appreciate any feedback. Thank you
>> > very much!
>> >
>> > Sincerely,
>> > Nan
>>
>> --
>> Regards,
>> Artem Senichev
>> Software Engineer, YADRO.