[bug report] powerpc: per device MSI irq domain

[Nilay Shroff]

Hi Nam,

I have been using an NVMe disk on my PowerPC system that supports up to
129 MSI-X interrupt vectors. Everything worked fine until Linux kernel
v6.18, after which the NVMe driver stopped detecting the disk because
the driver probe now fails.

After further investigation, I found that the probe failure in v6.18
occurs during PCI/MSI-X vector allocation. A git bisect identified
commit daaa574aba6f (“powerpc/pseries/msi: Switch to msi_create_parent_
irq_domain()”) as the first bad commit.

Additional debugging showed that the driver probe fails when calling
msi_create_device_irq_domain(). My working hypothesis is that, although
the PCIe NVMe device advertises support for 129 MSI-X vectors, the pSeries
firmware can supply only 128 MSI vectors to the device. This mismatch 
appears to cause MSI-X domain creation to fail, which ultimately results
in the NVMe driver failing to probe the device.

Device & MSI-X capability:
==========================

# lspci 
0524:28:00.0 Non-Volatile memory controller: KIOXIA Corporation NVMe SSD Controller CM7 2.5" (rev 01)

# lspci -vvv -s 0524:28:00.0 | grep -A2 MSI-X
	Capabilities: [b0] MSI-X: Enable+ Count=129 Masked-
		Vector table: BAR=0 offset=00005200
		PBA: BAR=0 offset=0000d600

Relevant device tree excerpt (DTS):

pci at 800000020000585 {
    ...
    ibm,pe-total-#msi = <0x80>;            /* 128 available under this PHB */
    ...
    pci1014,6d1 at 0 {
        ...
        ibm,msi-x-ranges = <0x1c 0x01>;
        ibm,req#msi-x        = <0x81>;     /* device supports 0x81 == 129 */
        ...
    }
}

As shown above, The device supports 0x81 (129) MSI-X vectors (ibm,req#msi-x),
but the PHB reports ibm,pe-total-#msi = 0x80 (128), indicating the platform/firmware
provides only 128 MSI vectors for devices under that PHB.

Debugfs IRQ domain (on a kernel just before the bad commit):
===========================================================

# cat /sys/kernel/debug/irq/domains/:pci at 800000020000524-3
name:   :pci at 800000020000524-3
 size:   0
 mapped: 65
 flags:  0x00000013
    IRQ_DOMAIN_FLAG_HIERARCHY
    IRQ_DOMAIN_NAME_ALLOCATED
    IRQ_DOMAIN_FLAG_MSI
 parent: pSeries-MSI-1316
    name:   pSeries-MSI-1316
     size:   128
     mapped: 65
     flags:  0x00000003
        IRQ_DOMAIN_FLAG_HIERARCHY
        IRQ_DOMAIN_NAME_ALLOCATED
     parent: :interrupt-controller at 400209f0000
        ...

This shows the parent domain (pSeries-MSI-1316) has size: 128.