ObjectMapper - quantity limitations?

[Ambrozewicz, Adrian]

W dniu 1/27/2021 o 18:14, Ed Tanous pisze:
> On Wed, Jan 27, 2021 at 9:04 AM Ambrozewicz, Adrian
> <adrian.ambrozewicz at linux.intel.com> wrote:
>>
>> Hello,
>>
>> I'm doing some performance measurements of OpenBMC telemetry subsystem.
>> I'm using my custom app, which spawns valid D-Bus Sensors, I configure
>> TelemetryService to monitor them and EventService to push MetricReports
>> to external server.
> 
> Which sensor code are you using?
> 
My homebrew app, which I run like so fake-sensors -c 
"/xyz/openbmc_project/inventory/system/board/AC_Baseboard" --loglevel 1 
--sensors {sensors_cnt} --interval {interval_ms} :)
It's just for tests.

>>
>> I observe certain limitation on my system. Each sensor is mapped as two
>> objects in ObjectMapper hierarchy. It seems that I am able to correctly
>> create up to 1500 sensors. When I go above this limit part of the
>> sensors are not represented in ObjectMapper tree.
> 
> When I wrote it originally, there were no arbitrary limits on how many
> objects the mapper could cache or return, but considering how big your
> responses will be, I'm guessing you're hitting the dbus per-message
> limit.  You don't mention if you're seeing any errors in the
> journalctl log from either the broker or object manager.  That might
> give you more clues.
> 

I've come back to the topic and I observe two kind of behaviors:
- mapperx crash (pretty rare)
- error "Error getting associations from ...".

The former still requires debugging, but for the latter I have simple 
explanation - only one application spawns thousands of sensors and is 
bombarding ObjectMapper with InterfaceAdded signals, while receiving 
'Get' on Associations interface (and possibly other calls). I've 
introduced a workaround - up to 3 retries of do_associations() on 
ETIMEDOUT, which allowed mapperx to correctly consume 2000 sensors.

However, performance could be better. On AST2600 listing of 2000 sensors 
by ObjectMapper takes 130s, 60s for 1000. Furthermore - it's just local 
workaround for bigger problem - D-Bus scalability in embedded system.

It looks like sooner or later we need to figure out more lightweight IPC 
for telemetry.

Regards,
Adrian