[Skiboot] [PATCH 1/2] npu2: Rework phb-index assignments for virtual PHBs

[Frederic Barrat]

Le 05/08/2019 à 13:13, Oliver O'Halloran a écrit :
> On Fri, Aug 2, 2019 at 11:37 PM Frederic Barrat <fbarrat at linux.ibm.com> wrote:
>>
>> Le 02/08/2019 à 10:10, Oliver O'Halloran a écrit :
>>> On Thu, 2019-08-01 at 10:54 +0200, Frederic Barrat wrote:
>>
>> Yes, what we do for opencapi deviates from nvlink (1 PHB per link as
>> opposed to 1 per NPU). That's unfortunate and confusing but 1) we had to
>> because of hardware considerations related to BAR assignments and 2) the
>> opencapi layout is the similar to real PCIs as you describe above, so we
>> didn't feel too bad about it.
> 
> Fair enough. The only situation I can see where having seperate PHBs
> might be an issue is if we wanted to implement fence recovery. For
> real PHBs we can unfence them individually even when they share a PEC,
> but I dunno about the NPU. Hopefully we'll never need to do that
> either.


For the NPU, we can fence/unfence on a per-link (i.e brick) basis.


>> I agree we should shoot for fixed phb-index and opal-phbid, as it makes
>> things simpler. It's also a bit troubling to see those virtual PHBs with
>> low opal IDs, when they are related to devices plugged to the 2nd chip.
>>
>> For Axone, it seems doable: 6 PHBs + 3 NPUs per chip (I'm not aware of
>> PCI changes, if any, for Axone. Is it still 3 PECs and 6 PHBs per
>> chip?).
> 
> Yeah, looks like there's been no changes.
> 
>> In the recently merged npu3 patches, we have a 1-to-1 mapping
>> between NPU and PHB for nvlink, and the upcoming opencapi patches will
>> keep it that way.
>>
>> For P9, it becomes ugly really fast if we want to preserve compatibility
>> with current phb/domain IDs seen by userland (and we should). At the
>> same time, while desirable, I don't think we really *have to* have fixed
>> phbid either. On a given platform, the dynamic IDs end up always being
>> the same anyway.
>>
>> So if I do the right thing for Axone, but keep things as is (with this
>> patch) on P9, would that fly?
> 
> That's fine. I figured we'd have to keep the existing behaviour for
> already released systems. We could make the new numbering the default
> for Axone and onwards and for unreleased P9 systems.


Well, I'm not too keen on changing it on P9/npu2 just for unreleased 
systems. I'd favor keeping it simple and all P9 systems the same. If it 
has to work for the already GA'd systems, it will work for the newer 
ones too.


> Anyway I was thinking we should use: <chip_id> * 0x10 as the base PHB
> index for each chip so we can support 16 PHBs per chip. Would that be
> enough?


16 PHBs would cover axone easily. What comes next is still a bit fuzzy, 
at least to me, but I guess we can always revisit then.
Is the chip ID numbering the same on axone? Does it mean the 2nd chip 
would see PHBs starting at 0x80?

   Fred