[PATCH V5 1/6] powerpc/powernv: don't enable SRIOV when VF BAR has non 64bit-prefetchable BAR

[David Laight]

From: Benjamin Herrenschmidt
> Sent: 09 October 2015 09:15
> On Fri, 2015-10-09 at 10:46 +0800, Wei Yang wrote:
> > On PHB_IODA2, we enable SRIOV devices by mapping IOV BAR with M64 BARs. If
> > a SRIOV device's IOV BAR is not 64bit-prefetchable, this is not assigned
> > from 64bit prefetchable window, which means M64 BAR can't work on it.
> 
> Won't this cause a lot of devices to become unsupported for us ? Or do
> all devices we care about have their BARs marked prefetchable ?
> 
> > The reason is PCI bridges support only 2 windows and the kernel code
> > programs bridges in the way that one window is 32bit-nonprefetchable and
> > the other one is 64bit-prefetchable. So if devices' IOV BAR is 64bit and
> > non-prefetchable, it will be mapped into 32bit space and therefore M64
> > cannot be used for it.
> >
> > This patch makes this explicit.
> 
> So PCIe allows for non-prefetchable BARs to be put under prefetchable
> bridge windows as long as the mapping done by the CPU doesn't prefetch,
> I believe. Well it's a natural conclusion of the weird note "Additional
> Guidance on the Prefetchable Bit in Memory Space BARs" page 596 of PCIe
> spec v3.0... it also says that devices should be pretty much free to
> set their prefetchable bit even if they have side effects so.
> 
> So maybe we should have that option, rather than just not using the
> devices, allow them to be allocate via the prefetchable window...

I'm trying to understand some of this for other reasons...
(Mostly because NetBSD doesn't allow PCIe addresses above 4G.)

AFAICT there are two options '32bit' and '64bit prefetchable'.

I presume the latter allows addresses above 4G (although I've
not looked closely enough to see how that gets written into the BAR).

That seems problematic on architectures like x86 where the bios
initialises the BARs but doesn't (usually) know whether the OS
will be 32 or 64bit.

Whether the cpu maps the addresses prefetchable ought to depend
on the way the driver maps the area - since it is the driver
that knows whether prefetching is sensible.

	David