Preventing a service from spamming the journal

[Andrew Geissler]

With the enablement of persistent journal (i.e. writing the journal to
a persistent location in the bmc filesystem) with witherspoon, we had
to configure a maximum size of 2.5MB.  This size can provide 2000+
entries but if you have a “bad” service it can quickly wrap out other
critical data.  The primary goal is always to fix those “bad” services
but sometimes the solution is not easy, or you don’t discover the bad
service until after you’ve lost your data.

Reading through
https://www.freedesktop.org/software/systemd/man/journald.conf.html
there are couple of solutions within journald, but none are ideal.

RateLimitIntervalSec=, RateLimitBurst=
These allow us to configure, how many entries a service can make
within a defined time limit.  I did a quick test on our system where I
set these to 60 sec and 10 entries max.  The journal was nice and
easily readable due to these 2 messages :)

Dec 19 15:35:48 witherspoon systemd-journald[2426]: Suppressed 71
messages from /init.scope
Dec 19 15:36:14 witherspoon systemd-journald[2426]: Suppressed 356
messages from /system.slice/phosphor-ipmi-host.service

The problem with this is it’s a very broad solution, there are def
times where we want certain services to be able to log more then the
limit we’d like to impose on other services.

Another possibility is to change MaxLevelStore= to something like
“notice”.  This means all info and debug logs are dropped.  Once
again, we have a lot of services that log things useful for debug in
the debug or info levels of journald.

So, thoughts or ideas are appreciated.  I’m thinking at a minimum, we
could tweak the defaults for RateLimit.  Currently they’re at 1000
messages every 30 seconds.  500 messages every 10 minutes seems more
reasonable to me in the openbmc env.

Andrew