[PATCH V5 1/6] powerpc/powernv: don't enable SRIOV when VF BAR has non 64bit-prefetchable BAR

[Wei Yang]

On Fri, Oct 09, 2015 at 09:02:16AM +0000, David Laight wrote:
>From: Benjamin Herrenschmidt
>> Sent: 09 October 2015 09:15
>> On Fri, 2015-10-09 at 10:46 +0800, Wei Yang wrote:
>> > On PHB_IODA2, we enable SRIOV devices by mapping IOV BAR with M64 BARs. If
>> > a SRIOV device's IOV BAR is not 64bit-prefetchable, this is not assigned
>> > from 64bit prefetchable window, which means M64 BAR can't work on it.
>> 
>> Won't this cause a lot of devices to become unsupported for us ? Or do
>> all devices we care about have their BARs marked prefetchable ?
>> 
>> > The reason is PCI bridges support only 2 windows and the kernel code
>> > programs bridges in the way that one window is 32bit-nonprefetchable and
>> > the other one is 64bit-prefetchable. So if devices' IOV BAR is 64bit and
>> > non-prefetchable, it will be mapped into 32bit space and therefore M64
>> > cannot be used for it.
>> >
>> > This patch makes this explicit.
>> 
>> So PCIe allows for non-prefetchable BARs to be put under prefetchable
>> bridge windows as long as the mapping done by the CPU doesn't prefetch,
>> I believe. Well it's a natural conclusion of the weird note "Additional
>> Guidance on the Prefetchable Bit in Memory Space BARs" page 596 of PCIe
>> spec v3.0... it also says that devices should be pretty much free to
>> set their prefetchable bit even if they have side effects so.
>> 
>> So maybe we should have that option, rather than just not using the
>> devices, allow them to be allocate via the prefetchable window...
>
>I'm trying to understand some of this for other reasons...
>(Mostly because NetBSD doesn't allow PCIe addresses above 4G.)

I guess here means the pci root bridge's windows are all below 4G?

>
>AFAICT there are two options '32bit' and '64bit prefetchable'.
>

For a pci bridge, the bridge window could be:
1. 32bit non-prefetchable and 32bit prefetchable
2. 32bit non-prefetchable and 64bit prefetchable

>I presume the latter allows addresses above 4G (although I've
>not looked closely enough to see how that gets written into the BAR).
>

Allow, and also could below 4G.

>That seems problematic on architectures like x86 where the bios
>initialises the BARs but doesn't (usually) know whether the OS
>will be 32 or 64bit.
>

I didn't get the point.

Usually when bios handle the MMIO assignment, kernel will mostly leave it
alone, as I know.

>Whether the cpu maps the addresses prefetchable ought to depend
>on the way the driver maps the area - since it is the driver
>that knows whether prefetching is sensible.
>
>	David
>
>

-- 
Richard Yang
Help you, Help me