Enabling OpenBMC Debug

[Andrew Geissler]

> On Feb 12, 2020, at 2:28 PM, Justin Thaler <thalerj at linux.vnet.ibm.com> wrote:
> 
> This is a really good set of information! I must say I think the obmcutil command is a good idea, for development purposes.
> 
> On 2/11/20 11:58 AM, Joseph Reynolds wrote:
>> On 2/11/20 9:51 AM, Andrew Geissler wrote:
>>> I find myself running a lot of different commands to enable a variety of
>>> debug data on OpenBMC when debugging different issues (usually BIOS
>>> communication issues). I also end up giving these commands to a lot of people
>>> when recreating issues for me. Stuff like this:
>>> 
>>> # enable debug logs in journal
>>> sed -i 's/info/debug/' /lib/systemd/journald.conf.d/journald-maxlevel-policy.conf
>>> systemctl restart systemd-journald.service
>>> 
>>> # Enable BIOS communication service debug
>>> sed -i 's/mboxd/mboxd -vv/' /lib/systemd/system/mboxd.service
>>> sed -i 's/btbridged/btbridged --vv/' /lib/systemd/system/org.openbmc.HostIpmi.service
>>> sed -i 's/ipmid/ipmid -d 0xff/'  /lib/systemd/system/phosphor-ipmi-host.service
>>> sed -i 's/0/1/' /etc/default/pldmd
>>> systemctl daemon-reload
>>> systemctl restart mboxd.service
>>> systemctl restart org.openbmc.HostIpmi.service
>>> systemctl restart phosphor-ipmi-host.service
>>> systemctl restart pldmd.service
>>> 
>>> I was contemplating wrapping the above stuff in a obmcutil command so instead
>>> of telling people to do all of the above (and dealing with situations when
>>> those commands change), I could say something like run "obmcutil debugon".
>>> 
> What about a REST API command tied to this? It could help when dealing with external to BMC problems.

No more generic REST API’s :) But in my response to Joseph I did ponder whether
we could get something added to Redfish for this. Enabling extra debug on a system
seems like a fairly normal use case.

> 
>>> Any thoughts out there? Would finer granularity of the debug be useful?
>>> Are there other forms of debug people would like enabled?
> Sometimes we have to recreate issues that occur on a system, a rather unfortunate situation in and of itself. I'll have to counter with a question of my own. Do you think this would be good to enable during a recreate, and we don't risk flooding the useful info?

Yes, ideally we don’t need the recreate but when we do, having the ability to
enable extra debug without a lot of work could be quite useful.

> 
> 
>> Thank you for the information.  I keep learning all the time.
>> These settings factor into service scenarios and also deployment readiness.  For example:
>> - I want to turn off debug.  For example, I want a debug-off command.
>> - I would want a way to validate (or at least show) these settings when testing firmware image release candidates.  For example, another obmcutil command "debug-show" which uses "grep -H" instead of "sed".
>> - I would want some documentation added the BMC administrator's guide to make them aware of OpenBMC debug support.  Draft: The obmcutil command can control debug settings for a variety of the components running on the BMC.  For more information, see https://github.com/openbmc/phosphor-state-manager/blob/master/obmcutil
>> The obmcutil tool is a shell script, so folks can take just the pieces they need.  That is, don't be more granular at this time.
>> In summary, it seems like a good idea.  It seems like we should ask our service architects to weight in.  I'll go ask mine....
>> - Joseph
>>> 
>>> Andrew
> Thanks,
> Justin