Link phosphor-hostlogger and bmcweb

[Artem Senichev]

On Thu, May 20, 2021 at 04:29:09PM -0700, Nan Zhou wrote:
> Hi all,
> 
> In the previous thread (
> https://lists.ozlabs.org/pipermail/openbmc/2021-March/025234.html), we
> (engineers from Google and Quanta) discussed our attempt to share host
> serial logs via Redfish, which includes polling logs via LogService and
> streaming log bytes via EventService (e.g. SSE). We went with the event log
> architecture
> <https://github.com/openbmc/docs/blob/master/architecture/redfish-logging-in-bmcweb.md>
> and did the implementation.
> 
> We still want to reuse the phosphor-hostlogger and do some modification. We
> believe it is better to try to reuse existing codes if possible and improve
> them rather than creating new things that have similar functionalities (in
> our case, it is a daemon that could collect logs and persist them).

I agree, reusing code is a right choice, but only when it is really possible.
For now it looks like you want to remove most of the Hostlogger features to
transform it from buffer-like to stream-like service.

> We want to do the following modification in phosphor-hostlogger (from the
> input and output point of view, just very few things will be changed)
> 
> 1. One limitation of phosphor-hostlogger is that it will lose data in the
> memory (the ring buffer maintained by hostlogger) when BMC gets force
> restarted;

When BMC goes to reboot it stops all services, at that moment hostlogger gets
a signal and flushes all gathered logs to a file.

> we propose to remove the ring buffer and write to the log file
> as soon as some characters are received.

I am not sure it is a good idea.
The host can generate a lot of logs, but we have very limited free space. In
addition, such heavy traffic will lead to a quick breakdown of the flash (most
available products are guaranteed to withstand around 100,000 P/E cycles only).

> This implicitly removes the needs
> of the ring buffer, and the persistence triggering (host reboot, sigterm,
> etc) in hostlogger. We believe this keeps the same functionality but saves
> hundreds of lines of codes in phosphor-hostlogger.

You are suggesting to delete the buffer, DBus watcher, log rotate. How are you
going to keep the same functionality if you remove everything related to it?

> 2. We propose not to compress the latest log file. This saves us the
> overhead of doing decompression when BMCWeb just needs to retrieve the most
> recent logs. There are still going to be log rotations in the file level.
> Files other than the latest log file are still going to be compressed. We
> can modify existing codes to achieve this or use the linux logrotate
> directly.
> 
> Furthermore, we will add host serial logs into BMCWeb, both LogService and
> EventService. In LogService, we will teach BMCWeb how to read the latest
> log file that is not compressed and the other compressed old logs, and how
> to assemble LogEntries out of raw serial logs. We will discuss EventService
> in future threads but the very initial idea is to setup inotify on log
> files and SSE to stream out new bytes to clients (like what the existing
> event logging is doing).
> 
> As we said above, for phosphor-hostlogger, the input is still the
> obmc-server unix socket, and the output are still persisted log files. But
> the functionality will get improved (less data loss), code gets simplified
> (no ring buffer and persistence triggering), and it will become easier and
> more performant to get BMCWeb hooked up for log streaming via Redfish.
> 
> Please let us know what you think. We appreciate any feedback. Thank you
> very much!
> 
> Sincerely,
> Nan

-- 
Regards,
Artem Senichev
Software Engineer, YADRO.