[External] /var/log as persistent

[Patrick Williams]

On Wed, May 29, 2024 at 11:49:15AM +0800, Lei Yu wrote:
> On Wed, May 29, 2024 at 1:41 AM Patrick Williams <patrick at stwcx.xyz> wrote:
> >
> > Hello,
> >
> > It was pointed out that I did not do a good job of broadcasting a change
> > I made back in March, so I am sending this out for awareness now.
> >
> > https://gerrit.openbmc.org/c/openbmc/openbmc/+/69959
> >
> > A default setting from bitbake (to VOLATILE_LOG_DIR) was causing some
> > platforms to not be able to persist `/var/log` and instead it was
> > mounted as a temporary directory.  This meant that even if you
> > explicitly configured journald to use `/var/log` (instead of the
> > `/run/log` it uses by default) you would not get persistent journalling.
> > It also meant that applications like `obmc-console` log files were not
> > persistent and would be lost in a BMC reboot.
> >
> 
> 1. VOLATILE_LOG_DIR is defaulted to `yes` in poky/meta/conf/bitbake.conf
> 2. With static layout, the above config makes the `/var/log` a
> volatile dir linked to `/var/volatile/log`, where `/var/volatile` is a
> tmpfs.
> 
> Be noted that it's the default for OpenBMC machines with static flash layouts.
> So the journal log and obmc-console were volatile **by default**.

I was still surprised that the journal was being persisted for you with
this change.  I'm looking now in more detail at the journald.conf because
there are only a few platforms that explicitly set `Storage=persistent`.

The primary reason I made this change was because setting the journal
config had **no effect**, which is, I think very unintuitive.

It seems that the default journal config is `Storage=auto` which means
that it will be volatile only if `/var/log/journal` exists.  This might
be why you are now seeing it persist.

> Users should not expect the above logs to be persistent, and if they
> do, they could config `VOLATILE_LOG_DIR` to `no`, which is done in
> `mtjade` and `mtmitchell` layer.

Conversely we could have machines that want it to be non-persistent to
set VOLATILE_LOG_DIR, right?  It isn't obvious why one default is
"better" than another.

My impression is that VOLATILE_LOG_DIR is default partially because
syslog is also the default.  As I said, with VOLATILE_LOG_DIR,
`Storage=persistent` has no affect.

If the only discussion here is really about the systemd-journal, we can
add a PACKAGECONFIG that chooses between `Storage=volatile` and
`Storage=persistent`.  It seems we already have 3 different meta-layers
with a custom journald-config to trigger this, so we might as well
consolidate those.

> The change `https://gerrit.openbmc.org/c/openbmc/openbmc/+/69959`
> makes `VOLATILE_LOG_DIR` to `yes` by default in `meta-phosphor` layer,
> which effectively affects all OpenBMC builds.
> 
> > I had asked a few machine owners and most of them either had it set to
> > explicitly unset `VOLATILE_LOG_DIR` in their meta-layer or through some
> > downstream changes had overwritten it.  So, I made this the default.
> >
> > I thought this only affected:
> >    - machines that explicitly set `Storage=persistent` in the journald
> >      config.
> >    - everyone's obmc-console logs.
> >
> > Based on the report from a downstream user, it seems like there might be
> > more effects?  I'm not sure at this point, but advertising it wider.
> 
> As above information, OpenBMC users were expecting "volatile" logs
> before, and we should keep it default.
> So I would suggest we revert the change to keep the consistency about
> the default volatile log dir.

Were they "expecting"?  This is where I did ask a few of the machine
owners who have upstream systems in production.  All of them told me
they overwrite this already.

I highly suspect that any commercial system is going to want
persistence.  I can understand what you have referred to with rsyslog
for cloud systems.  I think this is one of those cases where whatever
the default is, it won't satisfy everyone.  (We would be considered a
"cloud" style system but don't want to rely exclusively on rsyslog
because you then don't get visibility to debug network issues.)

> And for the reasons why I prefer the volatile log directory:
> * In most OpenBMC machine builds we see 32/64/128 SPI flashes are
> used, so the `rwfs` is limited and the frequent "writes" to the SPI
> flash costs the lifespan.

Are you seeing lifetime issues with SPI-NOR?  Aren't they rated for at
least 1 million erase cycles?  Even with the static layout, where all
writes are going to the same location, that should give you 32 years of
use at 1 erase per second (assuming only 4MiB of rwfs space).

> * To collect the logs, we could either use rsyslog or the host-logger
> (which were "already there") to send the logs to remote servers. So
> there is no strong requirement to make the logs persistent.

Are you really wanting obmc-console to *not* persist?  Do you think most
people want this?  This is a bit surprising to me because obmc-console
already has configuration limiting the size of the log files and it has
a rotate functionality; the amount of space (and erase cycles) used by it
is probably low.

-- 
Patrick Williams
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.ozlabs.org/pipermail/openbmc/attachments/20240529/8d75627a/attachment.sig>