[OpenPower-Firmware] PCI Initialization with OPAL

[Nathan Whitehorn]

On 07/24/16 23:16, Russell Currey wrote:
> On Sat, 2016-07-23 at 13:19 -0700, Nathan Whitehorn wrote:
>> Is there an overview of what steps are required to get a basically
>> functional PCI bus after kexec() runs? I'm trying to bring up FreeBSD on
>> the Tyan reference system and am hitting a wall.
> Hello, that sounds like fun.
>
>> As a first pass, I am hoping to bypass all the IOMMU and PE features and
>> ignore MSIs, mapping everything to PE 1. This is what I am doing now,
>> which works in QEMU at least:
>> 1. OPAL_RESET_PCI_IODA_TABLE(phb)
>> 2. OPAL_PCI_SET_PE(phb, 1, 0, OpalPciBusAny,
>> OPAL_IGNORE_RID_DEVICE_NUMBER, OPAL_IGNORE_RID_FUNCTION_NUMBER, OPAL_MAP_PE)
>> 3. OPAL_PCI_MAP_PE_MMIO_WINDOW(phb, 1, OPAL_M32_WINDOW_TYPE, 0, i) for i
>> < ibm,opal-num-pes
>> 4. Enable M64 window based on property:
>>                   opal_call(OPAL_PCI_SET_PHB_MEM_WINDOW, phb,
>>                       OPAL_M64_WINDOW_TYPE, 0 /* index */,
>>                       ((uint64_t)m64window[2] << 32) | m64window[3], 0,
>>                       ((uint64_t)m64window[4] << 32) | m64window[5]);
>>                   opal_call(OPAL_PCI_MAP_PE_MMIO_WINDOW, phb,
>>                       OPAL_PCI_DEFAULT_PE, OPAL_M64_WINDOW_TYPE,
>>                       0 /* index */, 0);
>>                   opal_call(OPAL_PCI_PHB_MMIO_ENABLE, phb,
>>                       OPAL_M64_WINDOW_TYPE, 0, OPAL_ENABLE_M64_NON_SPLIT);
>> 5. OPAL_PCI_MAP_PE_DMA_WINDOW_REAL(phb, 1, 2, 0, max_memory_address)
>> 6. Release any EEH freeze
>>
>> This results in continual MMIO freezes every few ms and no LSI
>> interrupts are ever emitted (I think because the bus freezes before the
>> driver can manage to ask for them). I suspect these freezes are due to
>> failing DMAs or MSIs left over from Linux. If I try to reset the PHB:
>> 1. OPAL_PCI_RESET(phb, OPAL_RESET_PHB_COMPLETE, ASSERT_RESET);
>> 2. wait on OPAL_POLL_PCI to return <= 0
>> 3. OPAL_PCI_RESET(phb, OPAL_RESET_PHB_COMPLETE, DEASSERT_RESET);
>> 4. wait on OPAL_POLL_PCI to return <= 0
>>
>> Then the system stops responding slightly after step 2 (which fails with
>> OPAL_HARDWARE), the IPMI controller turns off and I lose the ability to
>> either see the serial console or power-cycle the system over IMPI and
>> half to reset the machine with the power switch.
> Do you have a public git tree you're working out of?  I don't know why this is
> happening but I'd love to take a look.

You can see the code here:

https://svnweb.freebsd.org/base/projects/powernv/powerpc/powernv/opal_pci.c

It's the #ifdef NOTYET section that breaks everything really horribly.
-Nathan

>
>> Any ideas on what I am missing would be appreciated -- I'm happy to
>> write a basic PCI bringup guide for the docs once this works as well.
>>
>> Thanks,
>> Nathan
>> _______________________________________________
>> OpenPower-Firmware mailing list
>> OpenPower-Firmware at lists.ozlabs.org
>> https://lists.ozlabs.org/listinfo/openpower-firmware