[External] /var/log as persistent

[Lei Yu]

On Wed, May 29, 2024 at 1:41 AM Patrick Williams <patrick at stwcx.xyz> wrote:
>
> Hello,
>
> It was pointed out that I did not do a good job of broadcasting a change
> I made back in March, so I am sending this out for awareness now.
>
> https://gerrit.openbmc.org/c/openbmc/openbmc/+/69959
>
> A default setting from bitbake (to VOLATILE_LOG_DIR) was causing some
> platforms to not be able to persist `/var/log` and instead it was
> mounted as a temporary directory.  This meant that even if you
> explicitly configured journald to use `/var/log` (instead of the
> `/run/log` it uses by default) you would not get persistent journalling.
> It also meant that applications like `obmc-console` log files were not
> persistent and would be lost in a BMC reboot.
>

1. VOLATILE_LOG_DIR is defaulted to `yes` in poky/meta/conf/bitbake.conf
2. With static layout, the above config makes the `/var/log` a
volatile dir linked to `/var/volatile/log`, where `/var/volatile` is a
tmpfs.

Be noted that it's the default for OpenBMC machines with static flash layouts.
So the journal log and obmc-console were volatile **by default**.
Users should not expect the above logs to be persistent, and if they
do, they could config `VOLATILE_LOG_DIR` to `no`, which is done in
`mtjade` and `mtmitchell` layer.

The change `https://gerrit.openbmc.org/c/openbmc/openbmc/+/69959`
makes `VOLATILE_LOG_DIR` to `yes` by default in `meta-phosphor` layer,
which effectively affects all OpenBMC builds.

> I had asked a few machine owners and most of them either had it set to
> explicitly unset `VOLATILE_LOG_DIR` in their meta-layer or through some
> downstream changes had overwritten it.  So, I made this the default.
>
> I thought this only affected:
>    - machines that explicitly set `Storage=persistent` in the journald
>      config.
>    - everyone's obmc-console logs.
>
> Based on the report from a downstream user, it seems like there might be
> more effects?  I'm not sure at this point, but advertising it wider.

As above information, OpenBMC users were expecting "volatile" logs
before, and we should keep it default.
So I would suggest we revert the change to keep the consistency about
the default volatile log dir.

And for the reasons why I prefer the volatile log directory:
* In most OpenBMC machine builds we see 32/64/128 SPI flashes are
used, so the `rwfs` is limited and the frequent "writes" to the SPI
flash costs the lifespan.
* To collect the logs, we could either use rsyslog or the host-logger
(which were "already there") to send the logs to remote servers. So
there is no strong requirement to make the logs persistent.

-- 
BRs,
Lei YU