Critical BMC process failure recovery

[Neil Bradley]

Hey Andrew!

At least initially, the requirements don't really seem like requirements - they seem like what someone's idea of what they think a solution would be.  For example, why reset 3 times? Why not 10? Or 2? Seems completely arbitrary. If the BMC resets twice in a row, there's no reason to think it would be OK the 3rd time. It's kinda like how people have been known do 4-5 firmware updates to "fix" a problem and it "still doesn't work". 😉

If the ultimate goal is availability, then there's more nuance to the discussion to be had. Let's assume the goal is "highest availability possible".

With that in mind, defining what "failure" is gets to be a bit more convoluted. Back when we did the CMM code for the Intel modular server, we had a several-pronged approach:

1) Run procmon - Look for any service that is supposed to be running (but isn't) and restart it and/or its process dependency tree.
2) Create a monitor (either a standalone program or a script) that periodically connects to the various services available - IPMI, web, KVM, etc.... - think of it like a functional "ping". A bit more involved, as this master control program (Tron reference 😉 ) would have to speak sentiently to each service to gauge how alive it is. There have been plenty of situations where a BMC is otherwise healthy but one service wasn't working, and it's overkill to have a 30-45 second outage while the BMC restarts.
3) Kernel panics were set to automatically reboot the BMC, with a double whammy of the hardware watchdog being enabled in case the CPU didn't reset.

There's more to it than this, as sometimes you'd have to quiesce procmon to not restart services that, through normal operation, would cease functioning for a brief period of time, so tuning would be required.

-->Neil

-----Original Message-----
From: openbmc <openbmc-bounces+neil_bradley=phoenix.com at lists.ozlabs.org> On Behalf Of Andrew Geissler
Sent: Monday, October 19, 2020 12:53 PM
To: OpenBMC Maillist <openbmc at lists.ozlabs.org>
Subject: Critical BMC process failure recovery

Greetings,

I've started initial investigation into two IBM requirements:

- Reboot the BMC if a "critical" process fails and can not recover
- Limit the amount of times the BMC reboots for recovery
  - Limit should be configurable, i.e. 3 resets within 5 minutes
  - If limit reached, display error to panel (if one available) and halt
    the BMC.

The goal here is to have the BMC try and get itself back into a working state via a reboot of itself.

This same reboot logic and limits would also apply to kernel panics and/or BMC hardware watchdog expirations.

Some thoughts that have been thrown around internally:

- Spend more time ensuring code doesn't fail vs. handling them failing
- Put all BMC code into a single application so it's all or nothing (vs. 
  trying to pick and choose specific applications and dealing with all of
  the intricacies of restarting individual ones)
- Rebooting the BMC and getting the proper ordering of service starts is
  sometimes easier then testing every individual service restart for recovery
  paths

"Critical" processes would be things like mapper or dbus-broker. There's definitely a grey area though with other services so we'd need some guidelines around defining them and allow the meta layers to have a way to deem whichever they want critical.

So anyway, just throwing this out there to see if anyone has any input or is looking for something similar.

High level, I'd probably start looking into utilizing systemd as much as possible. "FailureAction=reboot-force" in the critical services and something that monitors for these types of reboots and enforces the reboot limits.

Andrew