IPMI SOL performance

[Tom Joseph]

On Tuesday 20 March 2018 06:05 AM, Jeremy Kerr wrote:
> Hi Tom,
>
>> I wanted to discuss about an issue related to IPMI SOL.
> Cool!
>
> Given that:
>
>> The same issue is not hit by the ssh console on port 2200 and does not
>> miss the timeout of the petitboot menu option.
> - and that we don't see the same behaviour with other SoL
> implementations (ie, different BMC firmware on similar hardware), it
> sounds to me like this might be due to our IPMI implementation?

> Have you confirmed that we're making the best use of the SoL payloads?
> (ie, we're not using smaller TX buffers for any reason).

The accepted character count field in the SOL payload is what the 
client(ipmitool) uses to acknowledge to BMC.
Since this is a 1 byte field, so the maximum SOL character data is 
restricted to 255.
>
>> IPMI uses UDP protocol and the maximum size of the  SOL payload per
>> packet according to the specification is 255 bytes. Also the ack for
>> each of the IPMI packet requires a command lookup
>> and execution. Compared with IPMI SOL, ssh supports 64kb payloads and
>> ack does not need an application lookup, so the round-trip timing is
>> faster. IPMI specification for SOL is written with UART speed as
>> reference, but the VUART pushes the data at a much faster rate.
> The console multiplexer will only push data as fast as *all* consumers
> can consume it. As a bit of background: obmc-console-server will read
> from the VUART into a global 16kB ringbuffer, and write to each of the
> consumers (ie, the sockets that ssh and IPMI SoL daemons connect to).
>
> If those writes block (for any one of those consumers), the ringbuffer
> tail is not advanced. Only once the slowest consumer has unblocked and
> consumed data, we advance the tail.
Is it guaranteed that the Petitboot menu option will be shown only after 
the obmc-console-server,
reads it from the VUART? When does the petitboot menu timeout begin?
>
> Once the ringbuffer is full though, we stop reading from the VUART until
> there is more space available (ie, the slowest consumer has caught up).
>
> This means that we have 16kB of space for bursty data, after which the
> daemon will slow down reads from the VUART device.
>
> So it sounds like our IPMI SoL implementation is slower than other BMC
> implementations, as we don't see the delay on other platforms. Can we
> address that?
This is good piece of information. IPMI server on BMC listens on 
"\0obmc-console" socket and reads from the socket
  on a EPOLLIN. The read console data is updated in a buffer (double 
ended queue) and sent to the ipmitool
based on the flow rate. Since the console data is read into the IPMI 
buffer( and sending it to the client at the speed of client)
is the reason that obmc-console-server does not slow down.

One possible option is to put a limit on the buffer size and read from 
the socket only when the buffer is within that limits. The trade-off
would be ssh based console would output at the speed of IPMI.

> As others have mentioned, there's currently an issue with the VUART
> hardware or driver dropping serial data on high load, but that's
> somewhat separate from IPMI performance. I'm currently working on that
> now :)
>
> Regards,
>
>
> Jeremy
>