Multiple BMCs in a system: IPMB? Redfish? MCTP?

[Andrew Jeffery]

On Fri, 8 May 2020, at 05:14, Nancy Yuen wrote:
> 
> 
> On Mon, May 4, 2020 at 3:31 AM Andrew Jeffery <andrew at aj.id.au> wrote:
> > 
> > 
> >  On Thu, 30 Apr 2020, at 17:08, Deepak Kodihalli wrote:
> >  > On 30/04/20 4:21 am, Nancy Yuen wrote:
> >  > > I've talked with some people a while back (long while back) about 
> >  > > multiple BMCs in a system. Either for redundancy or managing separate 
> >  > > parts of a system. I'm wondering what other people are thinking in 
> >  > > this area if at all.
> >  > > 
> >  > > We are considering similar designs and I'm looking into options for 
> >  > > BMC-BMC communications. Some BMCs may not be externally 
> >  > > accessible. Here are some options that we've looked at:
> >  > > 
> >  > > 1. i2c/IPMB
> >  > > 2. usbnet/Redfish
> >  > > 3. i2c/MCTP/PLDM or something else?
> >  > > 4. internal network via switch chip/Redfish or MCTP
> >  > > 
> >  > > I'd like to reduce our use of IPMI so I want to avoid (1).
> >  > > 
> >  > > ----------
> >  > > Nancy
> >  > 
> >  > Hi Nancy,
> >  > 
> >  > I think it depends on whether the BMCs need to talk to each other for 
> >  > platform management, or if they manage their own hosts and we need one 
> >  > of the BMCs to broadcast out of band requests and aggregate responses.
> >  > 
> >  > For the former I think PLDM over MCTP could be a good fit. This is more 
> >  > of an "inband" use-case in my opinion so I'm not sure if Redfish is well 
> >  > suited. For the latter, a Redfish based aggregation is a good option.
> > 
> >  I think this looks like the right approach. Certainly some thought would be
> >  required in designing the MCTP networks given the (intentionally) limited
> >  number of endpoint IDs.
> > 
> >  Andrew
> 
> Hi Andrew, Deepak,
> 
> Sorry for the delayed response!
> 
> For the purposes of this discussion I have BMCs managing their own 
> host: BMC3<--->BMC2<--->BMC1/controller<--> external. I wrote BMC1 but 
> there maybe some other controller there that isn't strictly a BMC. BMC2 
> would be "passing through" information between BMC1 and BMC3. But I 
> don't necessarily want BMC2 to do any form of aggregation from 
> information from BMC3. For BMC1/Controller and BMC 2 we are thinking 
> Redfish. 

So to restate, you're looking to bridge data from BMC1/controller to BMC3
via BMC2?

MCTP caters to this but it probably depends on what transport medium
you were thinking of using. Redfish for BMC1-to-BMC2 suggests networking,
which starts to go outside the use-cases for which MCTP is intended.

Others might be able to comment on whether Redfish is a good fit for a data
bridge.

Andrew