PowerNV PCIe hotplug support?

[Oliver O'Halloran]

On Thu, Dec 28, 2023 at 3:16 PM Timothy Pearson
<tpearson at raptorengineering.com> wrote:
>
> I've been evaluating some new options for our POWER9-based hardware in the NVMe space, and would like some clarification on the current status of PCIe hotplug for the PowerNV platforms.
>
> From what I understand, the pnv_php driver provides the basic hotplug functionality on PowerNV.  What I'm not clear on is to what extent this is intended to flow downstream to attached PCIe switches.

I did a bunch of work on NVMe hotplug back in the day and it worked
fine then. Most of that work was done with Gen3 PLX switches though
which are considerably dumber than the newer ones though.

> I have a test setup here that consists of a PMC 8533 switch and several downstream NVMe drives, with the switch attached directly to the PHB4 root port.  After loading the pnv_php module, I can disconnect the downstream NVMe devices by either using echo 0 on /sys/bus/pcu/slots/Snnnnnnn/power, or by doing a physical surprise unplug, however nothing I try can induce a newly plugged device to train and be detected on the bus.  Even trying a echo 0 and subsequent echo 1 to /sys/bus/pcu/slots/Snnnnnnn/power only results in the device going offline, there seems to be no way to bring the device back online short of a reboot.
>
> Hotplug of other devices connected directly to the PHB4 appears to work properly (I can online and offline via the power node); the issue seems to be restricted to downstream devices connected to the (theoretically hotplug capable) PMC 8533 switch.

I'd suspect either the PCIe interrupt is not being delivered for some
reason (EEH might be isolating the PCIe switch port?) or the removal
is triggering downstream port containment on the switch. Check the
port capability status with lspci. IIRC pnv_php doesn't know anything
about DPC so you might need to have skiboot disable that by default to
keep the kernel happy.

> Is this the intended behavior for downstream (non-IBM) PCIe ports?  Raptor can provide resources to assist in a fix if needed, but I would like to understand if this is a bug or an unimplemented feature first, and if the latter what the main issues are likely to be in implementation.

It *should* work and the WARN()/BUG() spew you're seeing are bugs that
just need fixing. That said, hotplug on PNV is a headache for a bunch
of reasons most of which are due to EEH. Something I was working
towards before I left IBM was refactoring how EEH worked internally so
we could eliminate the need for pnv_php and go back to using the
standard pcieport driver. Unfortunately, that's a big job and I can't
even remember how much of that work actually made it upstream.