Link phosphor-hostlogger and bmcweb

[Nan Zhou]

On Thu, May 20, 2021 at 04:29:09PM -0700, Nan Zhou wrote:

> > Hi all,
> >
> > In the previous thread (
> > https://lists.ozlabs.org/pipermail/openbmc/2021-March/025234.html), we
> > (engineers from Google and Quanta) discussed our attempt to share host
> > serial logs via Redfish, which includes polling logs via LogService and
> > streaming log bytes via EventService (e.g. SSE). We went with the event
> log
> > architecture
> > <
> https://github.com/openbmc/docs/blob/master/architecture/redfish-logging-in-bmcweb.md
> >
> > and did the implementation.
> >
> > We still want to reuse the phosphor-hostlogger and do some modification.
> We
> > believe it is better to try to reuse existing codes if possible and
> improve
> > them rather than creating new things that have similar functionalities
> (in
> > our case, it is a daemon that could collect logs and persist them).
> I agree, reusing code is a right choice, but only when it is really
> possible.
> For now it looks like you want to remove most of the Hostlogger features to
> transform it from buffer-like to stream-like service.

Hostlogger will still consume bytes from obmc-console-server and persist
logs in files. If the log itself has EOL, it will be splitted in the file
as well.
I thought that is all things we want in the Hostlogger.

> We want to do the following modification in phosphor-hostlogger (from the
> > input and output point of view, just very few things will be changed)
> >
> > 1. One limitation of phosphor-hostlogger is that it will lose data in the
> > memory (the ring buffer maintained by hostlogger) when BMC gets force
> > restarted;
> When BMC goes to reboot it stops all services, at that moment hostlogger
> gets
> a signal and flushes all gathered logs to a file.

Yes, I understand hostlogger registers a sigterm handler. But what if BMC
lost its power before sigterm gets sent? That's what I mean by "force
restart".

> we propose to remove the ring buffer and write to the log file
> as soon as some characters are received.

I am not sure it is a good idea.
> The host can generate a lot of logs, but we have very limited free space.
> In
> addition, such heavy traffic will lead to a quick breakdown of the flash
> (most
> available products are guaranteed to withstand around 100,000 P/E cycles
> only).

I would like to get input from Ed for this point. +Ed Tanous
<edtanous at google.com>.

> This implicitly removes the needs
> > of the ring buffer, and the persistence triggering (host reboot, sigterm,
> > etc) in hostlogger. We believe this keeps the same functionality but
> saves
> > hundreds of lines of codes in phosphor-hostlogger.
> You are suggesting to delete the buffer, DBus watcher, log rotate. How are
> you
> going to keep the same functionality if you remove everything related to
> it?

Log rotation is kept. Ring buffer and DBus watcher are great in the
original design but become unnecessary if we persist logs immediately. Do
we miss
anything that the previous hostlogger provides (transfer bytes from the
unix socket to EOL-separated compressed logs)? I believe in this proposal
we have
less data loss, simpler codes, and make it more performant for Redfish
integration.

On Thu, May 20, 2021 at 11:10 PM Artem Senichev <artemsen at gmail.com> wrote:

> On Thu, May 20, 2021 at 04:29:09PM -0700, Nan Zhou wrote:
> > Hi all,
> >
> > In the previous thread (
> > https://lists.ozlabs.org/pipermail/openbmc/2021-March/025234.html), we
> > (engineers from Google and Quanta) discussed our attempt to share host
> > serial logs via Redfish, which includes polling logs via LogService and
> > streaming log bytes via EventService (e.g. SSE). We went with the event
> log
> > architecture
> > <
> https://github.com/openbmc/docs/blob/master/architecture/redfish-logging-in-bmcweb.md
> >
> > and did the implementation.
> >
> > We still want to reuse the phosphor-hostlogger and do some modification.
> We
> > believe it is better to try to reuse existing codes if possible and
> improve
> > them rather than creating new things that have similar functionalities
> (in
> > our case, it is a daemon that could collect logs and persist them).
>
> I agree, reusing code is a right choice, but only when it is really
> possible.
> For now it looks like you want to remove most of the Hostlogger features to
> transform it from buffer-like to stream-like service.
>
> > We want to do the following modification in phosphor-hostlogger (from the
> > input and output point of view, just very few things will be changed)
> >
> > 1. One limitation of phosphor-hostlogger is that it will lose data in the
> > memory (the ring buffer maintained by hostlogger) when BMC gets force
> > restarted;
>
> When BMC goes to reboot it stops all services, at that moment hostlogger
> gets
> a signal and flushes all gathered logs to a file.
>
> > we propose to remove the ring buffer and write to the log file
> > as soon as some characters are received.
>
> I am not sure it is a good idea.
> The host can generate a lot of logs, but we have very limited free space.
> In
> addition, such heavy traffic will lead to a quick breakdown of the flash
> (most
> available products are guaranteed to withstand around 100,000 P/E cycles
> only).
>
> > This implicitly removes the needs
> > of the ring buffer, and the persistence triggering (host reboot, sigterm,
> > etc) in hostlogger. We believe this keeps the same functionality but
> saves
> > hundreds of lines of codes in phosphor-hostlogger.
>
> You are suggesting to delete the buffer, DBus watcher, log rotate. How are
> you
> going to keep the same functionality if you remove everything related to
> it?
>
> > 2. We propose not to compress the latest log file. This saves us the
> > overhead of doing decompression when BMCWeb just needs to retrieve the
> most
> > recent logs. There are still going to be log rotations in the file level.
> > Files other than the latest log file are still going to be compressed. We
> > can modify existing codes to achieve this or use the linux logrotate
> > directly.
> >
> > Furthermore, we will add host serial logs into BMCWeb, both LogService
> and
> > EventService. In LogService, we will teach BMCWeb how to read the latest
> > log file that is not compressed and the other compressed old logs, and
> how
> > to assemble LogEntries out of raw serial logs. We will discuss
> EventService
> > in future threads but the very initial idea is to setup inotify on log
> > files and SSE to stream out new bytes to clients (like what the existing
> > event logging is doing).
> >
> > As we said above, for phosphor-hostlogger, the input is still the
> > obmc-server unix socket, and the output are still persisted log files.
> But
> > the functionality will get improved (less data loss), code gets
> simplified
> > (no ring buffer and persistence triggering), and it will become easier
> and
> > more performant to get BMCWeb hooked up for log streaming via Redfish.
> >
> > Please let us know what you think. We appreciate any feedback. Thank you
> > very much!
> >
> > Sincerely,
> > Nan
>
> --
> Regards,
> Artem Senichev
> Software Engineer, YADRO.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ozlabs.org/pipermail/openbmc/attachments/20210521/ae4a3f43/attachment.htm>