FSL 64-bit DMA window question

[Scott Wood]

On 06/07/2013 05:39:53 PM, Benjamin Herrenschmidt wrote:
> On Fri, 2013-06-07 at 17:34 -0500, Scott Wood wrote:
> > > You also need to account for other on-chip MMIOs no ? Or do you  
> never
> > > intend to let PCI devices hit them ?
> >
> > I don't think we normally need that (other than MSIs, which have a
> > special window under 4G)...  The question is whether it's better to  
> let
> > odd use cases work without having to manually move the DMA window,  
> at
> > the cost of decreased out-of-the-box performance with common PCIe  
> cards.
> 
> Sorry, I didn't parse what the tradeoff is here...

I just meant that a PCIe device targeting something other than RAM,  
CCSR (it's not just MSIs that are mapped through the special window),  
or another PCIe is a rather unusual case -- not something that you'd  
see by just plugging in an ordinary PCIe card.  The tradeoff is that if  
we accommodate this strange use case, boards like the radeon would need  
to use swiotlb (once we fix the > versus >= bug) unless the user tweaks  
the DMA window.

It may be best to stick with the default that makes everything work,  
even if a broken PCIe card ends up being a bit slower out of the box,  
even if the PCIe-to-MMIO use case is weird and would require custom  
setup anyway.  OTOH, did we ever care about 32-bit DMA being able to  
access MMIO?

> Putting the inbound window above memory seems like a reasonable  
> option then
> (at least as a default that the board can override).
> 
> I sincerely hope those radeons are the only remaining things with  
> that sort of
> limitation but you never know ... it's all x86 fault of course for  
> never having
> used high DMA windows :-)

Hmm... Why are we using this special window at all?  Can't we just have  
one large identity mapping starting at zero, with the MSI window  
presumably taking precedence?  The manual is a bit vague on whether  
inbound windows can overlap with priority... in EP mode it says it  
does, and the RC section is silent.  Do we currently ever overlap  
PCICSRBAR with a real window?

-Scott