[OpenPower-Firmware] PCI Initialization with OPAL

[Nathan Whitehorn]

Is there an overview of what steps are required to get a basically 
functional PCI bus after kexec() runs? I'm trying to bring up FreeBSD on 
the Tyan reference system and am hitting a wall.

As a first pass, I am hoping to bypass all the IOMMU and PE features and 
ignore MSIs, mapping everything to PE 1. This is what I am doing now, 
which works in QEMU at least:
1. OPAL_RESET_PCI_IODA_TABLE(phb)
2. OPAL_PCI_SET_PE(phb, 1, 0, OpalPciBusAny, 
OPAL_IGNORE_RID_DEVICE_NUMBER, OPAL_IGNORE_RID_FUNCTION_NUMBER, OPAL_MAP_PE)
3. OPAL_PCI_MAP_PE_MMIO_WINDOW(phb, 1, OPAL_M32_WINDOW_TYPE, 0, i) for i 
< ibm,opal-num-pes
4. Enable M64 window based on property:
                 opal_call(OPAL_PCI_SET_PHB_MEM_WINDOW, phb,
                     OPAL_M64_WINDOW_TYPE, 0 /* index */,
                     ((uint64_t)m64window[2] << 32) | m64window[3], 0,
                     ((uint64_t)m64window[4] << 32) | m64window[5]);
                 opal_call(OPAL_PCI_MAP_PE_MMIO_WINDOW, phb,
                     OPAL_PCI_DEFAULT_PE, OPAL_M64_WINDOW_TYPE,
                     0 /* index */, 0);
                 opal_call(OPAL_PCI_PHB_MMIO_ENABLE, phb,
                     OPAL_M64_WINDOW_TYPE, 0, OPAL_ENABLE_M64_NON_SPLIT);
5. OPAL_PCI_MAP_PE_DMA_WINDOW_REAL(phb, 1, 2, 0, max_memory_address)
6. Release any EEH freeze

This results in continual MMIO freezes every few ms and no LSI 
interrupts are ever emitted (I think because the bus freezes before the 
driver can manage to ask for them). I suspect these freezes are due to 
failing DMAs or MSIs left over from Linux. If I try to reset the PHB:
1. OPAL_PCI_RESET(phb, OPAL_RESET_PHB_COMPLETE, ASSERT_RESET);
2. wait on OPAL_POLL_PCI to return <= 0
3. OPAL_PCI_RESET(phb, OPAL_RESET_PHB_COMPLETE, DEASSERT_RESET);
4. wait on OPAL_POLL_PCI to return <= 0

Then the system stops responding slightly after step 2 (which fails with 
OPAL_HARDWARE), the IPMI controller turns off and I lose the ability to 
either see the serial console or power-cycle the system over IMPI and 
half to reset the machine with the power switch.

Any ideas on what I am missing would be appreciated -- I'm happy to 
write a basic PCI bringup guide for the docs once this works as well.

Thanks,
Nathan