PowerNV PCIe hotplug support?

[Timothy Pearson]

I've been evaluating some new options for our POWER9-based hardware in the NVMe space, and would like some clarification on the current status of PCIe hotplug for the PowerNV platforms.

>From what I understand, the pnv_php driver provides the basic hotplug functionality on PowerNV.  What I'm not clear on is to what extent this is intended to flow downstream to attached PCIe switches.

I have a test setup here that consists of a PMC 8533 switch and several downstream NVMe drives, with the switch attached directly to the PHB4 root port.  After loading the pnv_php module, I can disconnect the downstream NVMe devices by either using echo 0 on /sys/bus/pcu/slots/Snnnnnnn/power, or by doing a physical surprise unplug, however nothing I try can induce a newly plugged device to train and be detected on the bus.  Even trying a echo 0 and subsequent echo 1 to /sys/bus/pcu/slots/Snnnnnnn/power only results in the device going offline, there seems to be no way to bring the device back online short of a reboot.

Hotplug of other devices connected directly to the PHB4 appears to work properly (I can online and offline via the power node); the issue seems to be restricted to downstream devices connected to the (theoretically hotplug capable) PMC 8533 switch.

Is this the intended behavior for downstream (non-IBM) PCIe ports?  Raptor can provide resources to assist in a fix if needed, but I would like to understand if this is a bug or an unimplemented feature first, and if the latter what the main issues are likely to be in implementation.

Thank you!