Creating event logs for blackout power losses

[Bills, Jason M]

On 7/24/2019 1:10 PM, Matt Spinler wrote:
> 
> 
> On 7/24/2019 2:53 PM, Bills, Jason M wrote:
>> On 6/28/2019 8:26 AM, Matt Spinler wrote:
>>> Hi,
>>> We've been asked by field support to create an OpenBMC event log when 
>>> we detect that
>>> we suffered a power blackout, by which I mean the system is up and 
>>> running, and then the
>>> BMC suddenly reboots and when it comes back up main power is off. The 
>>> issue for it is
>>> https://github.com/ibm-openbmc/dev/issues/677.
>>>
>>> I have a proposal in there for how to detect it in the chassis state 
>>> manager code, which
>>> basically just consists of persisting the chassis power state and 
>>> then checking it against
>>> the actual value on startup, and an error proposed at
>>> https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-dbus-interfaces/+/22791. 
>>>
>>> Does anyone have any other ideas or thoughts?  Is it OK if we make 
>>> this the default behavior, or
>>> would it be preferred we enable it with a compile flag?
>>
>> Sorry for the late replay as I have been out of the office.
>>
>> In my testing on this type of failure, I found that there is a race 
>> condition between the Chassis and BMC losing power after the blackout. 
>> If the Chassis loses power while the BMC is still active, it can 
>> sometimes detect it, change the Chassis State to Off, and persist it 
>> before losing power.  When this happens, the next boot will see only 
>> the Off state persisted and not detect the blackout.
>>
>> To work around this, we have added a delay when persisting the Chassis 
>> State.  In a normal Off, it will be persisted after the delay; in a 
>> blackout off, the delay will prevent the BMC from persisting the Off 
>> state allowing the blackout to be detected.
> 
> Hi, thanks for the response.
> Did you do this in a downstream phosphor-state-manager repository,
> or somewhere else?  If in phosphor-state-manager, would you consider 
> sharing it?
> 

It's in a downstream x86-power-control repository that I haven't had a 
chance to push to the upstream x86-power-control, yet.

>>
>>>
>>> Thanks,
>>> Matt
>>
>