ObjectMapper - quantity limitations?

[Andrew Jeffery]

On Wed, 10 Feb 2021, at 02:06, Ambrozewicz, Adrian wrote:
> W dniu 1/27/2021 o 18:14, Ed Tanous pisze:
> > On Wed, Jan 27, 2021 at 9:04 AM Ambrozewicz, Adrian
> > <adrian.ambrozewicz at linux.intel.com> wrote:
> >>
> >> Hello,
> >>
> >> I'm doing some performance measurements of OpenBMC telemetry subsystem.
> >> I'm using my custom app, which spawns valid D-Bus Sensors, I configure
> >> TelemetryService to monitor them and EventService to push MetricReports
> >> to external server.
> > 
> > Which sensor code are you using?
> > 
> My homebrew app, which I run like so fake-sensors -c 
> "/xyz/openbmc_project/inventory/system/board/AC_Baseboard" --loglevel 1 
> --sensors {sensors_cnt} --interval {interval_ms} :)
> It's just for tests.
> 
> >>
> >> I observe certain limitation on my system. Each sensor is mapped as two
> >> objects in ObjectMapper hierarchy. It seems that I am able to correctly
> >> create up to 1500 sensors. When I go above this limit part of the
> >> sensors are not represented in ObjectMapper tree.
> > 
> > When I wrote it originally, there were no arbitrary limits on how many
> > objects the mapper could cache or return, but considering how big your
> > responses will be, I'm guessing you're hitting the dbus per-message
> > limit.  You don't mention if you're seeing any errors in the
> > journalctl log from either the broker or object manager.  That might
> > give you more clues.
> > 
> 
> I've come back to the topic and I observe two kind of behaviors:
> - mapperx crash (pretty rare)
> - error "Error getting associations from ...".
> 
> The former still requires debugging, but for the latter I have simple 
> explanation - only one application spawns thousands of sensors and is 
> bombarding ObjectMapper with InterfaceAdded signals, while receiving 
> 'Get' on Associations interface (and possibly other calls). I've 
> introduced a workaround - up to 3 retries of do_associations() on 
> ETIMEDOUT, which allowed mapperx to correctly consume 2000 sensors.
> 
> However, performance could be better. On AST2600 listing of 2000 sensors 
> by ObjectMapper takes 130s, 60s for 1000. Furthermore - it's just local 
> workaround for bigger problem - D-Bus scalability in embedded system.
> 
> It looks like sooner or later we need to figure out more lightweight IPC 
> for telemetry.

I posted a bit about this on another performance-related thread[1]: I've put together a process-shared-memory shared library that could make these exchanges quite light-weight with less of the usual awkwardness of shared memory. It's still young and needs further work, but might be interesting all the same:

https://github.com/amboar/shmapper#libshmap

[1] https://lore.kernel.org/openbmc/09f8bed7-e19e-4ce9-850f-ae870ae89937@www.fastmail.com/

Andrew