[PATCH 2/2] powerpc/powernv: Double VF BAR size for compound PE

[Gavin Shan]

On Fri, Jul 17, 2015 at 10:14:43AM +1000, Gavin Shan wrote:
>When VF BAR size is equal to 128MB or bigger than that, we extend
>the corresponding PF's IOV BAR to cover number of total VFs supported
>by the PF. Otherwise, we extend the PF's IOV BAR to cover 256 VFs.
>For the former case, we have to create compound PE, which includes
>4 VFs. Those 4 VFs included in the compound PE can't be passed through
>to different guests, which isn't good.
>
>The gate (128MB) was choosen based on the assumption that each PHB
>supports 64GB M64 space and one PF's IOV BAR can be extended to be
>as huge as 1/4 of that, which is 16GB. However, the IOV BAR can be
>extended to half of PHB's M64 window when the PF seats behind the
>root port. In that case, the gate can be enlarged to be 256MB to
>avoid compound PE as we can.
>
>Signed-off-by: Gavin Shan <gwshan at linux.vnet.ibm.com>
>---
> arch/powerpc/platforms/powernv/pci-ioda.c | 21 ++++++++++++++++-----
> 1 file changed, 16 insertions(+), 5 deletions(-)
>
>diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
>index 6ec62b9..5b2e88f 100644
>--- a/arch/powerpc/platforms/powernv/pci-ioda.c
>+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
>@@ -2721,6 +2721,7 @@ static void pnv_pci_ioda_fixup_iov_resources(struct pci_dev *pdev)
> 	struct resource *res;
> 	int i;
> 	resource_size_t size;
>+	resource_size_t limit;
> 	struct pci_dn *pdn;
> 	int mul, total_vfs;
>
>@@ -2730,6 +2731,18 @@ static void pnv_pci_ioda_fixup_iov_resources(struct pci_dev *pdev)
> 	hose = pci_bus_to_host(pdev->bus);
> 	phb = hose->private_data;
>
>+	/*
>+	 * When the PF seats behind root port, the IOV BAR can
>+	 * consume half of the PHB's M64 window. Otherwise,
>+	 * 1/4 of the PHB's M64 window can be consumed to the
>+	 * maximal degree.
>+	 */
>+	if (!pci_is_root_bus(pdev->bus) &&
>+	    pci_is_root_bus(pdev->bus->self->bus))
>+		limit = 128;
>+	else
>+		limit = 256;
>+

I sent it too fast. The limit should be reversed: 256 when PF seats behind the
root port. Otherwise, it should be 128. I will send follow-up v2 after waiting
for couple of days in case there are some comments for this revision.

> 	pdn = pci_get_pdn(pdev);
> 	pdn->vfs_expanded = 0;
>
>@@ -2748,11 +2761,9 @@ static void pnv_pci_ioda_fixup_iov_resources(struct pci_dev *pdev)
> 		}
>
> 		size = pci_iov_resource_size(pdev, i + PCI_IOV_RESOURCES);
>-
>-		/* bigger than 64M */
>-		if (size > (1 << 26)) {
>-			dev_info(&pdev->dev, "PowerNV: VF BAR%d: %pR IOV size is bigger than 64M, roundup power2\n",
>-				 i, res);
>+		if (size >= (limit * 0x100000)) {
>+			dev_info(&pdev->dev, "PowerNV: VF BAR%d: %pR IOV size is bigger than %lldMB, roundup power2\n",
>+				 i, res, limit);
> 			pdn->m64_per_iov = M64_PER_IOV;
> 			mul = roundup_pow_of_two(total_vfs);
> 			break;

Thanks,
Gavin