Enabling OpenBMC Debug

[Andrew Geissler]

> On Feb 11, 2020, at 11:58 AM, Joseph Reynolds <jrey at linux.ibm.com> wrote:
> 
> On 2/11/20 9:51 AM, Andrew Geissler wrote:
>> I find myself running a lot of different commands to enable a variety of
>> debug data on OpenBMC when debugging different issues (usually BIOS
>> communication issues). I also end up giving these commands to a lot of people
>> when recreating issues for me. Stuff like this:
>> 
>> # enable debug logs in journal
>> sed -i 's/info/debug/' /lib/systemd/journald.conf.d/journald-maxlevel-policy.conf
>> systemctl restart systemd-journald.service
>> 
>> # Enable BIOS communication service debug
>> sed -i 's/mboxd/mboxd -vv/' /lib/systemd/system/mboxd.service
>> sed -i 's/btbridged/btbridged --vv/' /lib/systemd/system/org.openbmc.HostIpmi.service
>> sed -i 's/ipmid/ipmid -d 0xff/'  /lib/systemd/system/phosphor-ipmi-host.service
>> sed -i 's/0/1/' /etc/default/pldmd
>> systemctl daemon-reload
>> systemctl restart mboxd.service
>> systemctl restart org.openbmc.HostIpmi.service
>> systemctl restart phosphor-ipmi-host.service
>> systemctl restart pldmd.service
>> 
>> I was contemplating wrapping the above stuff in a obmcutil command so instead
>> of telling people to do all of the above (and dealing with situations when
>> those commands change), I could say something like run "obmcutil debugon".
>> 
>> Any thoughts out there? Would finer granularity of the debug be useful?
>> Are there other forms of debug people would like enabled?
> 
> Thank you for the information.  I keep learning all the time.
> 
> These settings factor into service scenarios and also deployment readiness.  For example:
> - I want to turn off debug.  For example, I want a debug-off command.
> - I would want a way to validate (or at least show) these settings when testing firmware image release candidates.  For example, another obmcutil command "debug-show" which uses "grep -H" instead of "sed".
> - I would want some documentation added the BMC administrator's guide to make them aware of OpenBMC debug support.  Draft: The obmcutil command can control debug settings for a variety of the components running on the BMC.  For more information, see https://github.com/openbmc/phosphor-state-manager/blob/master/obmcutil

This is interesting, I was not really thinking of these being used by our
external service type teams. I was mostly thinking of this as a development
only tool. Used early in bringup of new systems when lots of new bugs
and issues are being discovered.  Within our company, the rule is that
the default software you ship, must generate and collect enough
information to debug any failure. Asking a customer to turn on extra
debug is frowned upon (but at times still needed). I do wonder if we
could interest the DMTF in some sort of “DebugLevel” property on
the Manager interface? It does make me think though that if were
to formalize this behind an actual external interface, it would then
make sense to formalize across OpenBMC how we enable different
levels of debug. For example, a consistent —vebose/—debug option
that is passed to applications on startup. Or a more consistent use
of systemd-journald levels across applications.

> 
> The obmcutil tool is a shell script, so folks can take just the pieces they need.  That is, don't be more granular at this time.
> 
> In summary, it seems like a good idea.  It seems like we should ask our service architects to weight in.  I'll go ask mine....
> 
> - Joseph
> 
>> 
>> Andrew