[OpenBmc][JournalD] Persisting Journal Logs

[Rohit Pai]

Persistent Journal does not work with jffs2 file system. 
More info: https://github.com/systemd/systemd/issues/25712

-----Original Message-----
From: Alisa Gataullina <argataullina at gmail.com> 
Sent: Saturday, December 3, 2022 12:08 AM
To: Rohit Pai <ropai at nvidia.com>
Cc: openbmc at lists.ozlabs.org
Subject: Re: [OpenBmc][JournalD] Persisting Journal Logs

External email: Use caution opening links or attachments


On Thu, 2022-12-01 at 14:04 +0000, Rohit Pai wrote:
> I am running open bmc with Linux 5.15 and systemd 251 (251.2+).
> I have configured persistent logging for journal.
> /etc/systemd/journald.conf
> [Journal]
> Storage=persistent
>
> Created folder /var/log/journal. This is mounted on mtd flash 
> partition.
> The rootfs has overlay with RO filesystem and RW filesystem coming 
> from mtd partition.
>
> ls -alt /var/log/journal/
> drwxr-sr-x    2 root     systemd-         0 Jan  1 00:03
> 2b4305f670484d1fa6b9c4deee336b91
>
> Jouranld creates a folder under /var/log/journal but I dont see 
> anything getting stored here ever.
> I don't see journal being persistent across reboots.
> Journal logs are kept only in /run/log/journal, and this is tmpfs in 
> the system and gets erased on each reboot.
> I have tried journalctl --flush to see if anything gets pushed to 
> inside /var/log/journal but nothing is stored apart from the folder 
> name.
> journalctl --rotate also has no impact.
> I seem to be doing everything as per the journalctl documentation but 
> still it’s not working.
>
> Anyone else has seen this issue, any help?
>
> Thanks
> Rohit PAI

Hello

To change journal log directory:

systemd-tmpfiles --create --prefix /var/log/journal systemctl restart systemd-journald

It needs to check journal log rotate settings. But this isn't recommended way for embedded system- too many rewriting cycles. The best way is to use remote syslog

To mount /var/log on persistent storage with yocto you could add VOLATILE_LOG_DIR = "no"  into conf file of machine.
After that all RedFsh, IPMI and all others logs defined with rsyslog.conf will be there on persistent storage. Also yocto created directory /var/log/journal but journal use default log path /run/log/journal. And you need to setup it.

Best Regards,
Alisa