Creating event logs for blackout power losses

[Bills, Jason M]

On 6/28/2019 8:26 AM, Matt Spinler wrote:
> Hi,
> We've been asked by field support to create an OpenBMC event log when we 
> detect that
> we suffered a power blackout, by which I mean the system is up and 
> running, and then the
> BMC suddenly reboots and when it comes back up main power is off. The 
> issue for it is
> https://github.com/ibm-openbmc/dev/issues/677.
> 
> I have a proposal in there for how to detect it in the chassis state 
> manager code, which
> basically just consists of persisting the chassis power state and then 
> checking it against
> the actual value on startup, and an error proposed at
> https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-dbus-interfaces/+/22791. 
> 
> Does anyone have any other ideas or thoughts?  Is it OK if we make this 
> the default behavior, or
> would it be preferred we enable it with a compile flag?

Sorry for the late replay as I have been out of the office.

In my testing on this type of failure, I found that there is a race 
condition between the Chassis and BMC losing power after the blackout. 
If the Chassis loses power while the BMC is still active, it can 
sometimes detect it, change the Chassis State to Off, and persist it 
before losing power.  When this happens, the next boot will see only the 
Off state persisted and not detect the blackout.

To work around this, we have added a delay when persisting the Chassis 
State.  In a normal Off, it will be persisted after the delay; in a 
blackout off, the delay will prevent the BMC from persisting the Off 
state allowing the blackout to be detected.

> 
> Thanks,
> Matt