[OpenPower-Firmware] PCI Initialization with OPAL

[Russell Currey]

On Sat, 2016-07-23 at 13:19 -0700, Nathan Whitehorn wrote:
> Is there an overview of what steps are required to get a basically 
> functional PCI bus after kexec() runs? I'm trying to bring up FreeBSD on 
> the Tyan reference system and am hitting a wall.

Hello, that sounds like fun.

> 
> As a first pass, I am hoping to bypass all the IOMMU and PE features and 
> ignore MSIs, mapping everything to PE 1. This is what I am doing now, 
> which works in QEMU at least:
> 1. OPAL_RESET_PCI_IODA_TABLE(phb)
> 2. OPAL_PCI_SET_PE(phb, 1, 0, OpalPciBusAny, 
> OPAL_IGNORE_RID_DEVICE_NUMBER, OPAL_IGNORE_RID_FUNCTION_NUMBER, OPAL_MAP_PE)
> 3. OPAL_PCI_MAP_PE_MMIO_WINDOW(phb, 1, OPAL_M32_WINDOW_TYPE, 0, i) for i 
> < ibm,opal-num-pes
> 4. Enable M64 window based on property:
>                  opal_call(OPAL_PCI_SET_PHB_MEM_WINDOW, phb,
>                      OPAL_M64_WINDOW_TYPE, 0 /* index */,
>                      ((uint64_t)m64window[2] << 32) | m64window[3], 0,
>                      ((uint64_t)m64window[4] << 32) | m64window[5]);
>                  opal_call(OPAL_PCI_MAP_PE_MMIO_WINDOW, phb,
>                      OPAL_PCI_DEFAULT_PE, OPAL_M64_WINDOW_TYPE,
>                      0 /* index */, 0);
>                  opal_call(OPAL_PCI_PHB_MMIO_ENABLE, phb,
>                      OPAL_M64_WINDOW_TYPE, 0, OPAL_ENABLE_M64_NON_SPLIT);
> 5. OPAL_PCI_MAP_PE_DMA_WINDOW_REAL(phb, 1, 2, 0, max_memory_address)
> 6. Release any EEH freeze
> 
> This results in continual MMIO freezes every few ms and no LSI 
> interrupts are ever emitted (I think because the bus freezes before the 
> driver can manage to ask for them). I suspect these freezes are due to 
> failing DMAs or MSIs left over from Linux. If I try to reset the PHB:
> 1. OPAL_PCI_RESET(phb, OPAL_RESET_PHB_COMPLETE, ASSERT_RESET);
> 2. wait on OPAL_POLL_PCI to return <= 0
> 3. OPAL_PCI_RESET(phb, OPAL_RESET_PHB_COMPLETE, DEASSERT_RESET);
> 4. wait on OPAL_POLL_PCI to return <= 0
> 
> Then the system stops responding slightly after step 2 (which fails with 
> OPAL_HARDWARE), the IPMI controller turns off and I lose the ability to 
> either see the serial console or power-cycle the system over IMPI and 
> half to reset the machine with the power switch.

Do you have a public git tree you're working out of?  I don't know why this is
happening but I'd love to take a look.

> 
> Any ideas on what I am missing would be appreciated -- I'm happy to 
> write a basic PCI bringup guide for the docs once this works as well.
> 
> Thanks,
> Nathan
> _______________________________________________
> OpenPower-Firmware mailing list
> OpenPower-Firmware at lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/openpower-firmware