Critical BMC process failure recovery

[Bills, Jason M]

On 10/19/2020 7:58 PM, Lei Yu wrote:
> Hi Andrew,
> 
> In Intel-BMC/openbmc, there are watchdog configs for every service
> that in case it fails, it will reset the BMC using the watchdog. See
> the below related configs and scripts.
> 
> https://github.com/Intel-BMC/openbmc/blob/intel/meta-openbmc-mods/meta-common/classes/systemd-watchdog.bbclass
> https://github.com/Intel-BMC/openbmc/blob/intel/meta-openbmc-mods/meta-common/recipes-phosphor/watchdog/system-watchdog/watchdog-reset.sh
> 
> It probably meets most of the requirements.
As an FYI - this approach has been very aggressive, so it is resetting 
more often than we would like and doesn't seem to be recovering some of 
the cases that it should.

We are considering disabling this watchdog reset in our platforms for 
now, and will likely need to refine the approach before we would enable 
it again.

> 
> 
> On Tue, Oct 20, 2020 at 3:54 AM Andrew Geissler <geissonator at gmail.com> wrote:
>>
>> Greetings,
>>
>> I've started initial investigation into two IBM requirements:
>>
>> - Reboot the BMC if a "critical" process fails and can not recover
>> - Limit the amount of times the BMC reboots for recovery
>>    - Limit should be configurable, i.e. 3 resets within 5 minutes
>>    - If limit reached, display error to panel (if one available) and halt
>>      the BMC.
>>
>> The goal here is to have the BMC try and get itself back into a working state
>> via a reboot of itself.
>>
>> This same reboot logic and limits would also apply to kernel panics and/or
>> BMC hardware watchdog expirations.
>>
>> Some thoughts that have been thrown around internally:
>>
>> - Spend more time ensuring code doesn't fail vs. handling them failing
>> - Put all BMC code into a single application so it's all or nothing (vs.
>>    trying to pick and choose specific applications and dealing with all of
>>    the intricacies of restarting individual ones)
>> - Rebooting the BMC and getting the proper ordering of service starts is
>>    sometimes easier then testing every individual service restart for recovery
>>    paths
>>
>> "Critical" processes would be things like mapper or dbus-broker. There's
>> definitely a grey area though with other services so we'd need some
>> guidelines around defining them and allow the meta layers to have a way
>> to deem whichever they want critical.
>>
>> So anyway, just throwing this out there to see if anyone has any input
>> or is looking for something similar.
>>
>> High level, I'd probably start looking into utilizing systemd as much as
>> possible. "FailureAction=reboot-force" in the critical services and something
>> that monitors for these types of reboots and enforces the reboot limits.
>>
>> Andrew
> 
> 
>