[PATCH] powerpc/iommu: Do not do platform domain attach atctions after probe

Jason Gunthorpe jgg at nvidia.com
Fri Oct 6 00:35:11 AEDT 2023

POWER throws a splat at boot, it looks like the DMA ops were probably
changed while a driver was attached. Something is still weird about how
power sequences its bootup. Previously this was hidden since the core
iommu code did nothing during probe, now it calls

Make spapr_tce_platform_iommu_attach_dev() do nothing on the probe time
call like it did before.

  WARNING: CPU: 0 PID: 8 at arch/powerpc/kernel/iommu.c:407 __iommu_free+0x1e4/0x1f0
  Modules linked in: sd_mod t10_pi crc64_rocksoft crc64 sg ibmvfc mlx5_core(+) scsi_transport_fc ibmveth mlxfw psample dm_multipath dm_mirror dm_region_hash dm_log dm_mod fuse
  CPU: 0 PID: 8 Comm: kworker/0:0 Not tainted 6.6.0-rc3-next-20230929-auto #1
  Hardware name: IBM,9080-HEX POWER10 (raw) 0x800200 0xf000006 of:IBM,FW1030.30 (NH1030_062) hv:phyp pSeries
  Workqueue: events work_for_cpu_fn
  NIP:  c00000000005f6d4 LR: c00000000005f6d0 CTR: 00000000005ca81c
  REGS: c000000003a27890 TRAP: 0700   Not tainted (6.6.0-rc3-next-20230929-auto)
  MSR:  800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>  CR: 48000824  XER: 00000008
  CFAR: c00000000020f738 IRQMASK: 0
  GPR00: c00000000005f6d0 c000000003a27b30 c000000001481800 000000000000017
  GPR04: 00000000ffff7fff c000000003a27950 c000000003a27948 0000000000000027
  GPR08: c000000c18c07c10 0000000000000001 0000000000000027 c000000002ac8a08
  GPR12: 0000000000000000 c000000002ff0000 c00000000019cc88 c000000003042300
  GPR16: 0000000000000000 0000000000000000 0000000000000000 c000000003071ab0
  GPR20: c00000000349f80d c000000003215440 c000000003215480 61c8864680b583eb
  GPR24: 0000000000000000 000000007fffffff 0800000020000000 0000000000000010
  GPR28: 0000000000020000 0000800000020000 c00000000c5dc800 c00000000c5dc880
  NIP [c00000000005f6d4] __iommu_free+0x1e4/0x1f0
  LR [c00000000005f6d0] __iommu_free+0x1e0/0x1f0
  Call Trace:
  [c000000003a27b30] [c00000000005f6d0] __iommu_free+0x1e0/0x1f0 (unreliable)
  [c000000003a27bc0] [c00000000005f848] iommu_free+0x28/0x70
  [c000000003a27bf0] [c000000000061518] iommu_free_coherent+0x68/0xa0
  [c000000003a27c20] [c00000000005e8d4] dma_iommu_free_coherent+0x24/0x40
  [c000000003a27c40] [c00000000024698c] dma_free_attrs+0x10c/0x140
  [c000000003a27c90] [c008000000dcb8d4] mlx5_cmd_cleanup+0x5c/0x90 [mlx5_core]
  [c000000003a27cc0] [c008000000dc45a0] mlx5_mdev_uninit+0xc8/0x100 [mlx5_core]
  [c000000003a27d00] [c008000000dc4ac4] probe_one+0x3ec/0x530 [mlx5_core]
  [c000000003a27d90] [c0000000008c5edc] local_pci_probe+0x6c/0x110
  [c000000003a27e10] [c000000000189c98] work_for_cpu_fn+0x38/0x60
  [c000000003a27e40] [c00000000018d1d0] process_scheduled_works+0x230/0x4f0
  [c000000003a27f10] [c00000000018ff14] worker_thread+0x1e4/0x500
  [c000000003a27f90] [c00000000019cdb8] kthread+0x138/0x140
  [c000000003a27fe0] [c00000000000df98] start_kernel_thread+0x14/0x18
  Code: 481b004d 60000000 e89e0028 3c62ffe0 3863dd20 481b0039 60000000 e89e0038 3c62ffe0 3863dd38 481b0025 60000000 <0fe00000> 4bffff20 60000000 3c4c0142
  ---[ end trace 0000000000000000 ]---
  iommu_free: invalid entry
      entry     = 0x8000000203d0
      dma_addr  = 0x8000000203d0000
      Table     = 0xc00000000c5dc800
      bus#      = 0x1
      size      = 0x20000
      startOff  = 0x800000000000
      index     = 0x70200016

Fixes: 2ad56efa80db ("powerpc/iommu: Setup a default domain and remove set_platform_dma_ops")
Reported-by: Tasmiya Nalatwad <tasmiya at linux.vnet.ibm.com>
Link: https://lore.kernel.org/r/d06cee81-c47f-9d62-dfc6-4c77b60058db@linux.vnet.ibm.com
Tested-by: Tasmiya Nalatwad <tasmiya at linux.vnet.ibm.com>
Signed-off-by: Jason Gunthorpe <jgg at nvidia.com>
 arch/powerpc/kernel/iommu.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

Joerg, this would need to go to your tree, thanks

diff --git a/arch/powerpc/kernel/iommu.c b/arch/powerpc/kernel/iommu.c
index d6ad3fde85a212..bf19932147517b 100644
--- a/arch/powerpc/kernel/iommu.c
+++ b/arch/powerpc/kernel/iommu.c
@@ -1280,13 +1280,19 @@ struct iommu_table_group_ops spapr_tce_table_group_ops = {
  * A simple iommu_ops to allow less cruft in generic VFIO code.
-static int spapr_tce_platform_iommu_attach_dev(struct iommu_domain *dom,
-					       struct device *dev)
+static int
+spapr_tce_platform_iommu_attach_dev(struct iommu_domain *platform_domain,
+				    struct device *dev)
+	struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
 	struct iommu_group *grp = iommu_group_get(dev);
 	struct iommu_table_group *table_group;
 	int ret = -EINVAL;
+	/* At first attach the ownership is already set */
+	if (!domain)
+		return 0;
 	if (!grp)
 		return -ENODEV;

base-commit: 20700abdf2393c8c0bd3a100e3111d0fcd3d5bf3

More information about the Linuxppc-dev mailing list