[Skiboot] [PATCH 1/2] npu2: Rework phb-index assignments for virtual PHBs

[Oliver O'Halloran]

On Fri, Aug 2, 2019 at 11:37 PM Frederic Barrat <fbarrat at linux.ibm.com> wrote:
>
> Le 02/08/2019 à 10:10, Oliver O'Halloran a écrit :
> > On Thu, 2019-08-01 at 10:54 +0200, Frederic Barrat wrote:
>
> Yes, what we do for opencapi deviates from nvlink (1 PHB per link as
> opposed to 1 per NPU). That's unfortunate and confusing but 1) we had to
> because of hardware considerations related to BAR assignments and 2) the
> opencapi layout is the similar to real PCIs as you describe above, so we
> didn't feel too bad about it.

Fair enough. The only situation I can see where having seperate PHBs
might be an issue is if we wanted to implement fence recovery. For
real PHBs we can unfence them individually even when they share a PEC,
but I dunno about the NPU. Hopefully we'll never need to do that
either.

> I agree we should shoot for fixed phb-index and opal-phbid, as it makes
> things simpler. It's also a bit troubling to see those virtual PHBs with
> low opal IDs, when they are related to devices plugged to the 2nd chip.
>
> For Axone, it seems doable: 6 PHBs + 3 NPUs per chip (I'm not aware of
> PCI changes, if any, for Axone. Is it still 3 PECs and 6 PHBs per
> chip?).

Yeah, looks like there's been no changes.

> In the recently merged npu3 patches, we have a 1-to-1 mapping
> between NPU and PHB for nvlink, and the upcoming opencapi patches will
> keep it that way.
>
> For P9, it becomes ugly really fast if we want to preserve compatibility
> with current phb/domain IDs seen by userland (and we should). At the
> same time, while desirable, I don't think we really *have to* have fixed
> phbid either. On a given platform, the dynamic IDs end up always being
> the same anyway.
>
> So if I do the right thing for Axone, but keep things as is (with this
> patch) on P9, would that fly?

That's fine. I figured we'd have to keep the existing behaviour for
already released systems. We could make the new numbering the default
for Axone and onwards and for unreleased P9 systems.

Anyway I was thinking we should use: <chip_id> * 0x10 as the base PHB
index for each chip so we can support 16 PHBs per chip. Would that be
enough?