Link phosphor-hostlogger and bmcweb

[Artem Senichev]

On Wed, May 26, 2021 at 08:17:11AM -0700, Nan Zhou wrote:
> >
> > > > > 3. zlib_file.xpp, zlib_exception.xpp:
> > > > > will be removed or slightly changed; we can potentially use the linux
> > > > > logrotate which has built-in compression and file rotation (in this
> > case
> > > > > these compression utilities will be removed).
> > > > > The latest log file isn't compressed any more. History log files are
> > > > > still compressed.
> > > > Just curious, how are you going to remove the oldest messages from the
> > > > latest file in runtime? You are not going to rewrite the entire file on
> > > > every input character, are you?
> > >
> > > The following is my current idea: we will rename the latest file to
> > > something else and notify the writer (hostlogger) to close its old file
> > > descriptor and open a new one (should be doable via linux logrotate and
> > > inotify or some signal handlers, as logrotate is able to send some
> > signals
> > > to hostlogger if a rotation is performed). The writer keeps appending
> > logs
> > > most of the time using the same fd unless the latest file is rotated.
> > This
> > > should be better than truncating the file where the reader (BMCWeb) won't
> > > have race conditions (it might read old snapshots but it is not a big
> > deal
> > > in our case).
> > Currently we can keep the last N lines of the host's output, the oldest
> > messages are removed. It is easy to implement with a buffer in memory.
> > But how are you going to get rid of the old lines if you write data
> > directly
> > to the log file?
> > Rotation will not help you with that (we actually don't need to store such
> > old
> > logs).
> 
> We plan to implement something similar to rotate count
> <https://linux.die.net/man/8/logrotate> in linux logrotate. It is basically
> like a ring buffer in the file system. We keep N log files. The latest log
> file is in plain text and the writer keeps appending data to it. The rest
> N-1 files are compressed.

In this case, you will keep full logs without gaps:
```
Host start <- log is empty, start logging
|
[...] <- write file, compress and rotate file
|
Host reboot or shut down
```

If there are too many logs, logrotate removes the oldest one and we lose the
boot log (form host start).

This is the default Hostlogger mode:
```
Host start <- log is empty, start logging
|
[line 3000] <- flush 3000 lines to the persistent file
|
[...] <- these logs are skipped (the last 3000 lines are in memory)
|
Host reboot or shut down <- flush last 3000 lines to the file
```

-- 
Regards,
Artem Senichev
Software Engineer, YADRO.