/var/log as persistent

[Lei Yu]

On Wed, May 29, 2024 at 8:29 PM Patrick Williams <patrick at stwcx.xyz> wrote:
>
> On Wed, May 29, 2024 at 11:49:15AM +0800, Lei Yu wrote:
> > On Wed, May 29, 2024 at 1:41 AM Patrick Williams <patrick at stwcx.xyz> wrote:
> > >
> > > Hello,
> > >
> > > It was pointed out that I did not do a good job of broadcasting a change
> > > I made back in March, so I am sending this out for awareness now.
> > >
> > > https://gerrit.openbmc.org/c/openbmc/openbmc/+/69959
> > >
> > > A default setting from bitbake (to VOLATILE_LOG_DIR) was causing some
> > > platforms to not be able to persist `/var/log` and instead it was
> > > mounted as a temporary directory.  This meant that even if you
> > > explicitly configured journald to use `/var/log` (instead of the
> > > `/run/log` it uses by default) you would not get persistent journalling.
> > > It also meant that applications like `obmc-console` log files were not
> > > persistent and would be lost in a BMC reboot.
> > >
> >
> > 1. VOLATILE_LOG_DIR is defaulted to `yes` in poky/meta/conf/bitbake.conf
> > 2. With static layout, the above config makes the `/var/log` a
> > volatile dir linked to `/var/volatile/log`, where `/var/volatile` is a
> > tmpfs.
> >
> > Be noted that it's the default for OpenBMC machines with static flash layouts.
> > So the journal log and obmc-console were volatile **by default**.
>
> I was still surprised that the journal was being persisted for you with
> this change.  I'm looking now in more detail at the journald.conf because
> there are only a few platforms that explicitly set `Storage=persistent`.
>
> The primary reason I made this change was because setting the journal
> config had **no effect**, which is, I think very unintuitive.

Agreed that it's "unintuitive" for `Storage=persistent` had **no effect**.
But as I mentioned before, it's the "default" behavior in OpenBMC
builds with static layout.

>
> It seems that the default journal config is `Storage=auto` which means
> that it will be volatile only if `/var/log/journal` exists.  This might
> be why you are now seeing it persist.
>
> > Users should not expect the above logs to be persistent, and if they
> > do, they could config `VOLATILE_LOG_DIR` to `no`, which is done in
> > `mtjade` and `mtmitchell` layer.
>
> Conversely we could have machines that want it to be non-persistent to
> set VOLATILE_LOG_DIR, right?  It isn't obvious why one default is
> "better" than another.
>
> My impression is that VOLATILE_LOG_DIR is default partially because
> syslog is also the default.  As I said, with VOLATILE_LOG_DIR,
> `Storage=persistent` has no affect.
>
> If the only discussion here is really about the systemd-journal, we can
> add a PACKAGECONFIG that chooses between `Storage=volatile` and
> `Storage=persistent`.  It seems we already have 3 different meta-layers
> with a custom journald-config to trigger this, so we might as well
> consolidate those.
>
> > The change `https://gerrit.openbmc.org/c/openbmc/openbmc/+/69959`
> > makes `VOLATILE_LOG_DIR` to `yes` by default in `meta-phosphor` layer,
> > which effectively affects all OpenBMC builds.
> >
> > > I had asked a few machine owners and most of them either had it set to
> > > explicitly unset `VOLATILE_LOG_DIR` in their meta-layer or through some
> > > downstream changes had overwritten it.  So, I made this the default.
> > >
> > > I thought this only affected:
> > >    - machines that explicitly set `Storage=persistent` in the journald
> > >      config.
> > >    - everyone's obmc-console logs.
> > >
> > > Based on the report from a downstream user, it seems like there might be
> > > more effects?  I'm not sure at this point, but advertising it wider.
> >
> > As above information, OpenBMC users were expecting "volatile" logs
> > before, and we should keep it default.
> > So I would suggest we revert the change to keep the consistency about
> > the default volatile log dir.
>
> Were they "expecting"?  This is where I did ask a few of the machine
> owners who have upstream systems in production.  All of them told me
> they overwrite this already.

What do you mean by "overwrite" here? In upstream systems, we only see
`VOLATILE_LOG_DIR = "no"` in `meta-ampere` layer. (Check out to
`828c130f59^` and `git grep VOLATILE_LOG_DIR` shows the result.)
All other layers were using the default `VOLATILE_LOG_DIR = "yes"`

>
> I highly suspect that any commercial system is going to want
> persistence.  I can understand what you have referred to with rsyslog
> for cloud systems.  I think this is one of those cases where whatever
> the default is, it won't satisfy everyone.  (We would be considered a
> "cloud" style system but don't want to rely exclusively on rsyslog
> because you then don't get visibility to debug network issues.)
>
> > And for the reasons why I prefer the volatile log directory:
> > * In most OpenBMC machine builds we see 32/64/128 SPI flashes are
> > used, so the `rwfs` is limited and the frequent "writes" to the SPI
> > flash costs the lifespan.
>
> Are you seeing lifetime issues with SPI-NOR?  Aren't they rated for at
> least 1 million erase cycles?  Even with the static layout, where all
> writes are going to the same location, that should give you 32 years of
> use at 1 erase per second (assuming only 4MiB of rwfs space).

In case you have many logs in obmc-console, there could be much more
erases per second.

>
> > * To collect the logs, we could either use rsyslog or the host-logger
> > (which were "already there") to send the logs to remote servers. So
> > there is no strong requirement to make the logs persistent.
>
> Are you really wanting obmc-console to *not* persist?  Do you think most
> people want this?  This is a bit surprising to me because obmc-console
> already has configuration limiting the size of the log files and it has
> a rotate functionality; the amount of space (and erase cycles) used by it
> is probably low.

I can not speak for others, but what is true is that before the commit
`828c130f59` all OpenBMC upstream builds (except meta-ampere) were
using `VOLATILE_LOG_DIR = "yes"`.