Host Serial Console Logs via Redfish

[Spencer Ku (古世瑜)]

Hi All,
We finally choose method 4 (phosphor-hostlogger + journal + rsyslog + bmcweb) to implement the feature.
I already push the code to phosphor-hostlogger and bmcweb, please take a look for those PRs.
Please let us know if there's any issue, thank you!

Sincerely,
Spencer Ku

> -----Original Message-----
> From: openbmc
> <openbmc-bounces+spencer.ku=quantatw.com at lists.ozlabs.org> On Behalf Of
> Nan Zhou
> Sent: Tuesday, March 9, 2021 5:45 AM
> To: openbmc at lists.ozlabs.org
> Cc: Spencer Ku <spencer.ku at quanta.corp-partner.google.com>; Litzung Chen
> <litzung.chen at quanta.corp-partner.google.com>; Ofer Yehielli
> <ofery at google.com>; Ed Tanous <edtanous at google.com>; Richard Hanley
> <rhanley at google.com>; Justin Chen <juschen at google.com>; Zhenfei Tai
> <ztai at google.com>
> Subject: Host Serial Console Logs via Redfish
> 
> Hi All,
> 
> We are designing and implementing a new logging service in Redfish to
> expose host serial console logs. The goal is that clients can talk to bmc via
> Redfish and get a real-time console (just like a read-only serial console). It will
> improve the debuggability of BMCs.
> 
> We divide the work into two phases. Phase 1 is to use the pull model. That is,
> clients do periodical pull against the Redfish server. In Phase 2, we will
> consider the post model via Redfish Events and subscriptions.
> 
> Implementation for Phase 1 is in
> https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/39093
> <https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgerrit.o
> penbmc-project.xyz%2Fc%2Fopenbmc%2Fbmcweb%2F%2B%2F39093&data=04
> %7C01%7CSpencer.Ku%40quantatw.com%7C99bf794310a94636369908d8e27b
> 8590%7C179b032707fc4973ac738de7313561b2%7C1%7C0%7C637508367471
> 992563%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2lu
> MzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=y4yus%2FhTe7EhF0a
> aWlHpERyTU%2Ba4qw8uMv07H8rh%2F2E%3D&reserved=0> . It is based on
> obmc-console, phosphor-hostlogger, and bmcweb. The basic idea is that
> phosphor-hostlogger collects host serial console logs via obmc-console and
> generates tarballs in a rotation manner. These tarballs are then consumed and
> exposed by a new node in bmcweb log service.
> 
> We found there are some improvements as listed below,
> 
> *	Logs are not exposed to Redfish until they reach BUF_MAXSIZE or
> BUF_MAXTIME (defined in https://github.com/openbmc/phosphor-hostlogger
> <https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.c
> om%2Fopenbmc%2Fphosphor-hostlogger&data=04%7C01%7CSpencer.Ku%40q
> uantatw.com%7C99bf794310a94636369908d8e27b8590%7C179b032707fc497
> 3ac738de7313561b2%7C1%7C0%7C637508367471992563%7CUnknown%7CT
> WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJX
> VCI6Mn0%3D%7C1000&sdata=08iesmkbFcqwCq8Vwhj9LxO8LR1xd9Su%2Bopl
> 1VxKG7Y%3D&reserved=0> ), but we want to achieve a stream-like console.
> We could set BUF_MAXSIZE to 1 or BUF_MAXTIME to a very short interval, but
> it will amplify the overhead of compression and decompression.
> *	Persistence isn’t optional. phosphor-hostlogger doesn’t expose any IPC
> interface. bmcweb can only talk to phosphor-hostlogger via zip files, which
> makes persistence of logs a necessary condition.
> 
> We propose the following methods to improve it.
> 
> 
> *	Method 1: D-Bus Signal; phosphor-hostlogger implements an interface
> which contains a signal. The payload of the signal should contain timestamps
> and log messages.  BmcWeb registers as a listener and once it receives a
> signal, it populates a new LogEntry. BmcWeb should implement its own
> configurable ring buffer to store log entries received from D-Bus.
> *	Method 2: File Watcher; add file watchers in BmcWeb to monitor the log
> files produced by phosphor-hostlogger. This method is similar to method 1. But
> persistence is still a necessary condition.
> *	Method 3: obmc-console + bmcweb: install the console collection and
> ring buffer parts of phosphor-hostlogger as a library. Use the library directly in
> BmcWeb to collect console logs.
> *	Method 4: phosphor-hostlogger + journal + rsyslog + bmcweb: this
> architecture is very similar to what the current OpenBMC uses for
> redfish-event
> <https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.c
> om%2Fopenbmc%2Fdocs%2Fblob%2Fmaster%2Farchitecture%2Fredfish-loggin
> g-in-bmcweb.md&data=04%7C01%7CSpencer.Ku%40quantatw.com%7C99bf79
> 4310a94636369908d8e27b8590%7C179b032707fc4973ac738de7313561b2%7
> C1%7C0%7C637508367472002558%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiM
> C4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&
> sdata=7sVLiODfWs%2FVmfA5%2FJwS6G6t5L%2Fo7Jfy%2B8zq3gboJ3s%3D&res
> erved=0> . Add a new schema for log entries. Publish journal logs in
> phosphor-hostlogger. Add file watchers in BmcWeb to monitor the log files
> produced by rsyslog. rsyslog should have log rotation enabled. Persistence is
> still a necessary condition.
> 
> Before we move forward, we would like to see what your preference is. We are
> willing to see other suggestions and alternatives as well. Thanks!
> 
> Sincerely,
> Nan