[RFC PATCH] powerpc: fsl_pci: fix inbound ATMU entries for systems with >4G RAM

[Scott Wood]

On 08/24/2016 02:48 PM, Tillmann Heidsieck wrote:
> For systems with >4G of RAM, the current implementation adds a second
> inbound PCIe window starting at 128G this leaves all memory from 4G to
> 128G inaccessible to inbound PCIe transactions.

The second inbound window is at 256G, and it maps all of RAM.  Note that
for accesses in this window, the PCI address is different from the host
address.

> The according errors can be observed by using the EDAC driver for MPC85XX.
> 
> This patch changes this behaviour by adding the second window starting
> at 4G. The current implementation still leaves memory beyond 68G
> unmapped as this would require yet another ATMU entry.

Windows must be size-aligned, as per the description of PEXIWBARn[WBA].
 This window is illegal.

By trying to identity-map everything you also break the assumption that
we don't need swiotlb to avoid the PEXCSRBAR region on PCI devices
capable of generating 64-bit addresses.

> Tested on a T4240 with 12G of RAM and an AMD E6760 PCIe card working with
> the in-tree radeon driver.

I have no problem using an e1000e card on a t4240 with 24 GiB RAM.

Looking at the radeon driver I see that there's a possibility of a dma
mask that is exactly 40 bits.  I think there's an off-by-one error in
fsl_pci_dma_set_mask().  If you change "dma_mask >=
DMA_BIT_MASK(MAX_PHYS_ADDR_BITS)" to "dma_mask >
DMA_BIT_MASK(MAX_PHYS_ADDR_BITS)", does that make radeon work without
this patch?


> Signed-off-by: Tillmann Heidsieck <theidsieck at leenox.de>
> ---
>  arch/powerpc/sysdev/fsl_pci.c | 40 ++++++++++++++++++++--------------------
>  1 file changed, 20 insertions(+), 20 deletions(-)
> 
> diff --git a/arch/powerpc/sysdev/fsl_pci.c b/arch/powerpc/sysdev/fsl_pci.c
> index 0ef9df49f0f2..260983037904 100644
> --- a/arch/powerpc/sysdev/fsl_pci.c
> +++ b/arch/powerpc/sysdev/fsl_pci.c
> @@ -349,17 +349,13 @@ static void setup_pci_atmu(struct pci_controller *hose)
>  	}
>  
>  	sz = min(mem, paddr_lo);
> -	mem_log = ilog2(sz);
> +	mem_log = order_base_2(sz);
>  
>  	/* PCIe can overmap inbound & outbound since RX & TX are separated */
>  	if (early_find_capability(hose, 0, 0, PCI_CAP_ID_EXP)) {
> -		/* Size window to exact size if power-of-two or one size up */
> -		if ((1ull << mem_log) != mem) {
> -			mem_log++;
> -			if ((1ull << mem_log) > mem)
> -				pr_info("%s: Setting PCI inbound window "
> -					"greater than memory size\n", name);
> -		}
> +		if ((1ull << mem_log) > mem)
> +			pr_info("%s: Setting PCI inbound window greater than memory size\n",
> +				name);
>  
>  		piwar |= ((mem_log - 1) & PIWAR_SZ_MASK);
>  

This change is unrelated.

BTW, for some reason your patch is not showing up in Patchwork.

-Scott