Prioritizing URIs with tight performance requirement in openBmc with bmcweb

[Rohit Pai]

Hello Ed, 

We have been analyzing time spent in route handler code to get some time measurements with different approaches. 
At least we know from our tests that majority of the time is spent in route handlers and not in JSON serialization which happens outside. 

To recap our test, we have two http clients doing thermal metric and stats/counters metric URI polling simultaneously from the BMC. 
Thermal metric URI has around 100 sensors and has tight latency perf requirement of 500ms. 
Stats/counter metric URI has around 2500 properties to fetch from the backend which uses the GetManagedObjects API. 
Time analysis was done on the latency measurement of stats/counter URI as this impacts the latency of thermal metric URI with the current bmcweb single threaded nature. 

Method 1 - Object Manger call to the backend service, bmcweb handler code processes the response and prepares the required JSON objects. 
a. Backend dbus call turnaround time                                              - 584 ms 
b. Logic in bmcweb route handler code to prepare response      - 365 ms 
c. Total URI latency                                                                               - 1019 ms

Method 2 - Backend populates all the needed properties in a single aggregate property. 
a. Backend dbus call turnaround time                                              - 161 ms 
b. Logic in bmcweb route handler code to prepare response      - 71   ms 
c. Total URI latency                                                                               - 291 ms

Method 3 - Bmcweb reads all the properties from a file fd. Here goal is to eliminate latency and load coming by using dbus as an IPC for large payloads. 
a. fd read call in bmcweb                                                                     - 64 ms 
b. JSON objection population from the read file contents             - 96 ms 
c. Total URI latency                                                                                - 254 ms 
The file contents were in JSON format. If we can replace this with efficient data structure which can be used with fd passing, then I think we can further optimize point b. 
Optimization around CPU bound logic in handler code would certainly help the latency of the other requests pending in the queue. 

I will try the multi-threaded solution put by you in the coming days and share the results. 

Thanks 
Rohit PAI 


-----Original Message-----
From: Ed Tanous <edtanous at google.com> 
Sent: Friday, June 2, 2023 12:04 AM
To: Rohit Pai <ropai at nvidia.com>
Cc: openbmc at lists.ozlabs.org
Subject: Re: Prioritizing URIs with tight performance requirement in openBmc with bmcweb

External email: Use caution opening links or attachments


On Wed, May 24, 2023 at 10:56 AM Ed Tanous <edtanous at google.com> wrote:
>
> On Wed, May 24, 2023 at 9:26 AM Ed Tanous <edtanous at google.com> wrote:
> >
> > There's likely a few other minor things that would need fixed, but 
> > the above is the general gist.
>
> I spent an hour or so and put together a simple POC of multithreading 
> bmcweb.  It's not threadsafe for all the global structures (although I 
> did one as an example), and has a lot of safety issues we'll need to 
> work through, but will give us a hint about whether multi-threading 
> bmcweb will solve your performance problem:
> https://gerrit.openbmc.org/c/openbmc/bmcweb/+/63710
>
> PTAL

Rohit, Were you able to try this?