Link phosphor-hostlogger and bmcweb

[Ed Tanous]

On Wed, May 26, 2021 at 12:21 PM Nan Zhou <nanzhou at google.com> wrote:
>>
>> > If there are too many logs in a boot cycle, won't the current hostlogger
>> > lose some earlier logs (boot logs) as well?
>> That's the point!
>> Hostlogger does not lose these logs. It writes the boot messages, then skips
>> the middle and writes the last 3000 lines when the host shuts down.
>> We have two log files per host session: start and end.
>> It is too expensive to store all host output, so mid-session messages are
>> skipped.
>> It can be easily implemented with a buffer, but I am not sure we can achieve
>> this with logrotate.
>
> Thanks for the information. I am not aware of this functionality in the current hostlogger. Are you saying it will be implemented in the future or I miss it in the current codes.
> One of our options for log rotations is writing our own codes, I am sure we can implement the logic you mentioned above without too much effort.
> The linux logrotate also has "prerotate scripts", we can carefully name the compressed log file and keep the oldest several ones (which have host boot logs) out of rotation.

For what it's worth, I don't know if we have to use logrotate
as-written if it's not a good fit here, but that style of writing to
disk makes for easily tail-able logs, and has pretty well defined
behavior for log rotation.  If we could keep the behavior (or the
behavior + rotating on boot events), even if we didn't use logrotate
itself, I think it would be a benefit to this.

>
>> > Or did me missing something?
>> >
>> > Also, we already talked about it: there's a problem that if BMC loses the
>> > power before it sends out a signal to hostlogger, data in memory won't be
>> > persisted.
>> Yes, I agree that this is a problem. But there are ways to fix it without
>> breaking the current functionality of Hostlogger.
>> We can use rsyslog with external log server, or increase the buffer size
>> in obmc-console-server, or use systemd-cat with logrotate.
>> We can even add a new mode to Hostlogger that will not use the buffer, but
>> as I said earlier, there are not many common parts.
>
> I guess you are arguing we need a new daemon rather than modify Hostlogger, right? +Ed Tanous here to see what his opinion is.

If Nans use case is in fact totally different, and can't be handled in
the same application I think that's ok, but I'd like to see the new
application put in the hostlogger repo so it can share the routines
that are common (things like finding and opening the unix socket,
managing the reads, and the zlib integration) and to ensure that users
find it when searching for code that solves this problem, as on the
outset they're pretty similar, just with seemingly different rules.
If we don't put it there, it seems like we'd have to duplicate a lot
of code.

>
> On Wed, May 26, 2021 at 11:22 AM Artem Senichev <artemsen at gmail.com> wrote:
>>
>> On Wed, May 26, 2021 at 09:20:38AM -0700, Nan Zhou wrote:
>> > >
>> > > > We plan to implement something similar to rotate count
>> > > > <https://linux.die.net/man/8/logrotate> in linux logrotate. It is
>> > > basically
>> > > > like a ring buffer in the file system. We keep N log files. The latest
>> > > log
>> > > > file is in plain text and the writer keeps appending data to it. The rest
>> > > > N-1 files are compressed.
>> > > In this case, you will keep full logs without gaps:
>> > > ```
>> > > Host start <- log is empty, start logging
>> > > |
>> > > [...] <- write file, compress and rotate file
>> > > |
>> > > Host reboot or shut down
>> > > ```
>> > > If there are too many logs, logrotate removes the oldest one and we lose
>> > > the
>> > > boot log (form host start).
>> > > This is the default Hostlogger mode:
>> > > ```
>> > > Host start <- log is empty, start logging
>> > > |
>> > > [line 3000] <- flush 3000 lines to the persistent file
>> > > |
>> > > [...] <- these logs are skipped (the last 3000 lines are in memory)
>> > > |
>> > > Host reboot or shut down <- flush last 3000 lines to the file
>> >
>> >
>> > Thanks for your explanation, but I didn't get it. Are you arguing that one
>> > can keep more logs in memory rather than on disk?
>>
>> Of course not! =)
>>
>> > If there are too many logs in a boot cycle, won't the current hostlogger
>> > lose some earlier logs (boot logs) as well?
>>
>> That's the point!
>> Hostlogger does not lose these logs. It writes the boot messages, then skips
>> the middle and writes the last 3000 lines when the host shuts down.
>> We have two log files per host session: start and end.
>> It is too expensive to store all host output, so mid-session messages are
>> skipped.
>> It can be easily implemented with a buffer, but I am not sure we can achieve
>> this with logrotate.
>>
>> > Or did me missing something?
>> >
>> > Also, we already talked about it: there's a problem that if BMC loses the
>> > power before it sends out a signal to hostlogger, data in memory won't be
>> > persisted.
>>
>> Yes, I agree that this is a problem. But there are ways to fix it without
>> breaking the current functionality of Hostlogger.
>> We can use rsyslog with external log server, or increase the buffer size
>> in obmc-console-server, or use systemd-cat with logrotate.
>> We can even add a new mode to Hostlogger that will not use the buffer, but
>> as I said earlier, there are not many common parts.
>>
>> --
>> Regards,
>> Artem Senichev
>> Software Engineer, YADRO.