Performance: BMCWEB prioritizing URI response in multiple redfish clients environment

[Ed Tanous]

On Wed, Nov 16, 2022 at 9:49 AM sharad yadav <sharad.openbmc at gmail.com> wrote:
>
> Thanks for your response. Responses inline.
>
> >  Can you elaborate on which URLs your clients are accessing?  Is this
> >  on an upstream SHA1 of bmcweb?  Do said clients implement HTTP
> >  keep-alive?  Keep-alive is important, because otherwise the bmc is
> >  negotiating TLS for every request, which can be quite slow, especially
> >  when done in parallel.
>
> We are at the latest bmcweb upstream code.
> We have 4 type of clients collecting set of URIs:
> All thermals: /redfish/v1/Chassis/<str>/Thermal/
> All other sensors: /redfish/v1/Chassis/<str>/Sensors/<str>/

When you say latest, do you really mean latest (rebased within the
last couple weeks).  There were some pretty significant performance
increases made in these APIs.  There's even yet another still pending
that you might try:
https://gerrit.openbmc.org/c/openbmc/bmcweb/+/57653

> All inventory: /redfish/v1/Chassis/<str>/ and /redfish/v1/Systems/<str>/Processors/<str>/
> Log services: /redfish/v1/Systems/<str>/LogServices/EventLog/ and /redfish/v1/Systems/<str>/LogServices/EventLog/Entries/
> /redfish/v1/Systems/<str>/LogServices/EventLog/Entries/<str>/

If you remove the two EventLog APIs, do your problems go away?  These
endpoints are known to use blocking filesystem calls (which is now a
documented anti-pattern, and we have alternatives like io-uring) but
could be the source of your blocking.  That would be a lot more doable
to fix to use non-blocking calls.

>
> All clients implement keep-alive so clients avoid the TLS handshake penalty.
>
> >  I'm not sure how a priority system would work in this case.  Can you
> >  elaborate on your idea?  There is currently no queue outside of the
> >  TCP kernel, and async tasks in io_context.  Which queue were you
> >  planning on assigning priorities to?
>
> We want to prioritize our thermal sensor URI.
> Is there a way to prioritize HTTP requests over D-Bus response messages in bmcweb?

I can't piece apart what you just asked for.  To create a response for
a sensor object, the flow goes:
Http -> bmcweb -> dbus -> sensor daemon

if you "prioritize" http socket actions, then your end state will be
no faster (possibly slower) because the dbus calls required to get
that data are de-prioritized.  Can you please expand on what you're
thinking, ideally pointing at the specific queues (in code) that you'd
like to make a priority queue?

>
> Thanks,
> Sharad
>
> On Wed, 16 Nov 2022 at 21:59, Ed Tanous <edtanous at google.com> wrote:
>>
>> On Wed, Nov 16, 2022 at 5:28 AM sharad yadav <sharad.openbmc at gmail.com> wrote:
>> >
>> > Hi All,
>> >
>> > We are facing a delay(till 6seconds) in bmcweb URIs responses when there are multiple clients(4 in our case) performing GET requests in parallel on different URIs.
>>
>> Can you elaborate on which URLs your clients are accessing?  Is this
>> on an upstream SHA1 of bmcweb?  Do said clients implement HTTP
>> keep-alive?  Keep-alive is important, because otherwise the bmc is
>> negotiating TLS for every request, which can be quite slow, especially
>> when done in parallel.
>>
>> Can you put together an example script that reproduces the issue?
>>
>> > We have a very important telemetry URI, used for thermal monitoring. On standalone, telemetry URI response is within 200ms.
>> > But with multiple clients, telemetry URI response reaches till 6seconds.
>> >
>> > Can we get help on how we can assign priority to our telemetry URI over other URIs in bmcweb?
>>
>> I'm not sure how a priority system would work in this case.  Can you
>> elaborate on your idea?  There is currently no queue outside of the
>> TCP kernel, and async tasks in io_context.  Which queue were you
>> planning on assigning priorities to?
>>
>> >
>> > Thanks,
>> > Sharad
>> >
>> >
>> >
>> >
>> >
>> >