Link phosphor-hostlogger and bmcweb

[Nan Zhou]

>
> >
> >> > Or did me missing something?
> >> >
> >> > Also, we already talked about it: there's a problem that if BMC loses
> the
> >> > power before it sends out a signal to hostlogger, data in memory
> won't be
> >> > persisted.
> >> Yes, I agree that this is a problem. But there are ways to fix it
> without
> >> breaking the current functionality of Hostlogger.
> >> We can use rsyslog with external log server, or increase the buffer size
> >> in obmc-console-server, or use systemd-cat with logrotate.
> >> We can even add a new mode to Hostlogger that will not use the buffer,
> but
> >> as I said earlier, there are not many common parts.
> >
> > I guess you are arguing we need a new daemon rather than modify
> Hostlogger, right? +Ed Tanous here to see what his opinion is.
> If Nans use case is in fact totally different, and can't be handled in
> the same application I think that's ok, but I'd like to see the new
> application put in the hostlogger repo so it can share the routines
> that are common (things like finding and opening the unix socket,
> managing the reads, and the zlib integration) and to ensure that users
> find it when searching for code that solves this problem, as on the
> outset they're pretty similar, just with seemingly different rules.
> If we don't put it there, it seems like we'd have to duplicate a lot
> of code.

Thanks a lot for the previous discussion. Following up on this, is it
acceptable if we add a new application (it will be a new binary + a new
systemd service config) in the hostlogger repo to support the stream
application? We can add options to control whether to turn on this new
stream application. In this way, existing applications stay unchanged. As
Ed said, we could use codes that deal with unix sockets and reads. It
should also make sense that console codes are put together in the
same place.

Please let us know what you think. Your confirmation will be very valuable
and will unblock our development.

On Fri, Jun 11, 2021 at 11:31 AM Ed Tanous <edtanous at google.com> wrote:

> On Wed, May 26, 2021 at 12:21 PM Nan Zhou <nanzhou at google.com> wrote:
> >>
> >> > If there are too many logs in a boot cycle, won't the current
> hostlogger
> >> > lose some earlier logs (boot logs) as well?
> >> That's the point!
> >> Hostlogger does not lose these logs. It writes the boot messages, then
> skips
> >> the middle and writes the last 3000 lines when the host shuts down.
> >> We have two log files per host session: start and end.
> >> It is too expensive to store all host output, so mid-session messages
> are
> >> skipped.
> >> It can be easily implemented with a buffer, but I am not sure we can
> achieve
> >> this with logrotate.
> >
> > Thanks for the information. I am not aware of this functionality in the
> current hostlogger. Are you saying it will be implemented in the future or
> I miss it in the current codes.
> > One of our options for log rotations is writing our own codes, I am sure
> we can implement the logic you mentioned above without too much effort.
> > The linux logrotate also has "prerotate scripts", we can carefully name
> the compressed log file and keep the oldest several ones (which have host
> boot logs) out of rotation.
>
> For what it's worth, I don't know if we have to use logrotate
> as-written if it's not a good fit here, but that style of writing to
> disk makes for easily tail-able logs, and has pretty well defined
> behavior for log rotation.  If we could keep the behavior (or the
> behavior + rotating on boot events), even if we didn't use logrotate
> itself, I think it would be a benefit to this.
>
> >
> >> > Or did me missing something?
> >> >
> >> > Also, we already talked about it: there's a problem that if BMC loses
> the
> >> > power before it sends out a signal to hostlogger, data in memory
> won't be
> >> > persisted.
> >> Yes, I agree that this is a problem. But there are ways to fix it
> without
> >> breaking the current functionality of Hostlogger.
> >> We can use rsyslog with external log server, or increase the buffer size
> >> in obmc-console-server, or use systemd-cat with logrotate.
> >> We can even add a new mode to Hostlogger that will not use the buffer,
> but
> >> as I said earlier, there are not many common parts.
> >
> > I guess you are arguing we need a new daemon rather than modify
> Hostlogger, right? +Ed Tanous here to see what his opinion is.
>
> If Nans use case is in fact totally different, and can't be handled in
> the same application I think that's ok, but I'd like to see the new
> application put in the hostlogger repo so it can share the routines
> that are common (things like finding and opening the unix socket,
> managing the reads, and the zlib integration) and to ensure that users
> find it when searching for code that solves this problem, as on the
> outset they're pretty similar, just with seemingly different rules.
> If we don't put it there, it seems like we'd have to duplicate a lot
> of code.
>
> >
> > On Wed, May 26, 2021 at 11:22 AM Artem Senichev <artemsen at gmail.com>
> wrote:
> >>
> >> On Wed, May 26, 2021 at 09:20:38AM -0700, Nan Zhou wrote:
> >> > >
> >> > > > We plan to implement something similar to rotate count
> >> > > > <https://linux.die.net/man/8/logrotate> in linux logrotate. It is
> >> > > basically
> >> > > > like a ring buffer in the file system. We keep N log files. The
> latest
> >> > > log
> >> > > > file is in plain text and the writer keeps appending data to it.
> The rest
> >> > > > N-1 files are compressed.
> >> > > In this case, you will keep full logs without gaps:
> >> > > ```
> >> > > Host start <- log is empty, start logging
> >> > > |
> >> > > [...] <- write file, compress and rotate file
> >> > > |
> >> > > Host reboot or shut down
> >> > > ```
> >> > > If there are too many logs, logrotate removes the oldest one and we
> lose
> >> > > the
> >> > > boot log (form host start).
> >> > > This is the default Hostlogger mode:
> >> > > ```
> >> > > Host start <- log is empty, start logging
> >> > > |
> >> > > [line 3000] <- flush 3000 lines to the persistent file
> >> > > |
> >> > > [...] <- these logs are skipped (the last 3000 lines are in memory)
> >> > > |
> >> > > Host reboot or shut down <- flush last 3000 lines to the file
> >> >
> >> >
> >> > Thanks for your explanation, but I didn't get it. Are you arguing
> that one
> >> > can keep more logs in memory rather than on disk?
> >>
> >> Of course not! =)
> >>
> >> > If there are too many logs in a boot cycle, won't the current
> hostlogger
> >> > lose some earlier logs (boot logs) as well?
> >>
> >> That's the point!
> >> Hostlogger does not lose these logs. It writes the boot messages, then
> skips
> >> the middle and writes the last 3000 lines when the host shuts down.
> >> We have two log files per host session: start and end.
> >> It is too expensive to store all host output, so mid-session messages
> are
> >> skipped.
> >> It can be easily implemented with a buffer, but I am not sure we can
> achieve
> >> this with logrotate.
> >>
> >> > Or did me missing something?
> >> >
> >> > Also, we already talked about it: there's a problem that if BMC loses
> the
> >> > power before it sends out a signal to hostlogger, data in memory
> won't be
> >> > persisted.
> >>
> >> Yes, I agree that this is a problem. But there are ways to fix it
> without
> >> breaking the current functionality of Hostlogger.
> >> We can use rsyslog with external log server, or increase the buffer size
> >> in obmc-console-server, or use systemd-cat with logrotate.
> >> We can even add a new mode to Hostlogger that will not use the buffer,
> but
> >> as I said earlier, there are not many common parts.
> >>
> >> --
> >> Regards,
> >> Artem Senichev
> >> Software Engineer, YADRO.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ozlabs.org/pipermail/openbmc/attachments/20210611/c62cac92/attachment.htm>