BMC Performance Profiler

[Pasha Ghabussi]

Hello all,

We would really appreciate it if you can take a few minutes to read the
following proposal and let us know your thoughts and suggestions.

We are developing a tool to investigate performance problems by looking at
DBus traffic dumps. Current DBus inspection and visualization tools do not
represent the DBus events similar to a typical performance profiler.
Additionally, these tools do not address typical BMC workloads such as IPMI
and ASIO. Hence, identifying potential performance problems requires
inspecting the raw BMC DBus traffic, which can become a long and complex
process. We want to add a graphical interface to webui-vue to visualize the
DBus traffic to address the abovementioned problem.

There have been DBus and IPMI performance-related discussions in the
OpenBMC community, both of which can be helped by this work: IPMI-related
issues started to appear as early as in 2017. One issue (#2630)
<https://github.com/openbmc/openbmc/issues/2630> describes a problem
related to large numbers of sensors. Its follow-up (#3098)
<https://github.com/openbmc/openbmc/issues/3098> mentions “hostboot crashes
due to poor IPMI performance”. Another issue (#2519)
<https://github.com/openbmc/openbmc/issues/2519> describes a commonly-seen
problem of IPMI taking very long to respond (> 5s).
There are also discussions on RedFish performance
<https://lists.ozlabs.org/pipermail/openbmc/2018-February/010735.html> on
the mailing list; A patch
<https://lists.ozlabs.org/pipermail/openbmc/2016-June/003380.html>
optimized DBus performance by introducing a cache for name translation.

All the performance investigations listed above involve DBus and may be
helped by this work.

We are planning to use the BMCweb file hosting functionality to access the
DBus event dumps and visualize the events in the web UI. The available
profiling tools such as dbus-pcap
<https://github.com/openbmc/openbmc-tools/tree/master/amboar/obmc-scripts/dbus-pcap>,
Wireshark <https://www.wireshark.org/>, Bustle
<https://gitlab.freedesktop.org/bustle/bustle>, Snyh
<https://github.com/snyh/dbus-profiler>, or DFeet
<https://wiki.gnome.org/action/show/Apps/DFeet?action=show&redirect=DFeet>
do not provide the exact functionality we are looking for. Our goal is to
develop functionalities similar to other widely used profilers such as
GPUView or VTune Profiler.

One alternative solution considered was to stream DBus requests over
websocket, but the existing websocket endpoints available on BMC webserver
do not provide the exact information we need.

Requirements and Scalability:

   -

   Should provide the adequate functionalities to filter, visualize the
   events timeline, and group the DBus traffic based on multiple criteria such
   as type, source, destination, path, interface, demon signatures, and more.
   -

   Should support capture of DBus messages using as little resources as
   possible.
   -

   Should be able to show many (~thousands of) entries on screen
   simultaneously
   -

   Integration with webui-vue


Thank you
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ozlabs.org/pipermail/openbmc/attachments/20201005/2413f976/attachment-0001.htm>