Link phosphor-hostlogger and bmcweb

[Nan Zhou]

Hi all,

In the previous thread (
https://lists.ozlabs.org/pipermail/openbmc/2021-March/025234.html), we
(engineers from Google and Quanta) discussed our attempt to share host
serial logs via Redfish, which includes polling logs via LogService and
streaming log bytes via EventService (e.g. SSE). We went with the event log
architecture
<https://github.com/openbmc/docs/blob/master/architecture/redfish-logging-in-bmcweb.md>
and did the implementation.

However, thanks to the maintainers of host-logger, we found the limitation
that we might flood the systemd-journal. Given that serial logs are special
and new data comes more often than almost anything else on the BMC,
flooding systemd-journal will be a fatal problem.

So we are thinking about other directions. The current idea is that we
don't involve DBus at all and BMCWeb reads serial logs from log files
directly.

We still want to reuse the phosphor-hostlogger and do some modification. We
believe it is better to try to reuse existing codes if possible and improve
them rather than creating new things that have similar functionalities (in
our case, it is a daemon that could collect logs and persist them).

We want to do the following modification in phosphor-hostlogger (from the
input and output point of view, just very few things will be changed)

1. One limitation of phosphor-hostlogger is that it will lose data in the
memory (the ring buffer maintained by hostlogger) when BMC gets force
restarted; we propose to remove the ring buffer and write to the log file
as soon as some characters are received. This implicitly removes the needs
of the ring buffer, and the persistence triggering (host reboot, sigterm,
etc) in hostlogger. We believe this keeps the same functionality but saves
hundreds of lines of codes in phosphor-hostlogger.

2. We propose not to compress the latest log file. This saves us the
overhead of doing decompression when BMCWeb just needs to retrieve the most
recent logs. There are still going to be log rotations in the file level.
Files other than the latest log file are still going to be compressed. We
can modify existing codes to achieve this or use the linux logrotate
directly.

Furthermore, we will add host serial logs into BMCWeb, both LogService and
EventService. In LogService, we will teach BMCWeb how to read the latest
log file that is not compressed and the other compressed old logs, and how
to assemble LogEntries out of raw serial logs. We will discuss EventService
in future threads but the very initial idea is to setup inotify on log
files and SSE to stream out new bytes to clients (like what the existing
event logging is doing).

As we said above, for phosphor-hostlogger, the input is still the
obmc-server unix socket, and the output are still persisted log files. But
the functionality will get improved (less data loss), code gets simplified
(no ring buffer and persistence triggering), and it will become easier and
more performant to get BMCWeb hooked up for log streaming via Redfish.

Please let us know what you think. We appreciate any feedback. Thank you
very much!

Sincerely,
Nan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ozlabs.org/pipermail/openbmc/attachments/20210520/ae3bfa29/attachment.htm>