Critical BMC process failure recovery

[Patrick Williams]

Hi Andrew,

I like the proposal to reuse what systemd already provides.  It does
look like Lei pointed to some existing bbclass that could be enhanced
for this purpose so that any recipe can simply 'inherit ...' and maybe
set a variable to indicate that it is providing "critical services".

On Mon, Oct 19, 2020 at 02:53:11PM -0500, Andrew Geissler wrote:
> Greetings,
> 
> I've started initial investigation into two IBM requirements:
> 
> - Reboot the BMC if a "critical" process fails and can not recover
> - Limit the amount of times the BMC reboots for recovery
>   - Limit should be configurable, i.e. 3 resets within 5 minutes

I like that it has a time bound on it here.  If the reset didn't have a
time bound that would be a problem to me because it means that a slow
memory leak could eventually get the BMCs into this state.

Do you need to do anything in relationship with the WDT and failover
settings there?  I'm thinking you'll need to do something to ensure that
you don't swap flash banks between these resets.  Do you need to do N
resets on one flash bank and then M on the other?

It seems that the most likely cause of N resets in a short time is some
sort of flash corruption, BMC chip error, or a bug aggravated some RWFS
setting.  None of these are particularly recovered by the reset but at
least you know your in a bad situation at that point.

>   - If limit reached, display error to panel (if one available) and halt
>     the BMC.

And then what?  What is the remediation for this condition?  Are there
any services, such as SSH, that will continue to run in this state?  I
hope the only answer for remediation is physical access / power cycle.

-- 
Patrick Williams
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.ozlabs.org/pipermail/openbmc/attachments/20201020/85233cae/attachment.sig>