FSL 64-bit DMA window question

[Scott Wood]

On 06/07/2013 05:01:28 PM, Scott Wood wrote:
> On 06/07/2013 07:09:20 AM, Benjamin Herrenschmidt wrote:
>> On Fri, 2013-06-07 at 09:44 +0000, Zang Roy-R61911 wrote:
>> >
>> > > -----Original Message-----
>> > > From: Benjamin Herrenschmidt [mailto:benh at kernel.crashing.org]
>> > >
>> > > On Fri, 2013-06-07 at 07:58 +0000, Zang Roy-R61911 wrote:
>> > > >
>> > > > > -----Original Message-----
>> > > > > From: Benjamin Herrenschmidt  
>> [mailto:benh at kernel.crashing.org]
>> > > > >
>> > > > > Hi Folks !
>> > > > >
>> > > > > Is there any specific reason why you chose 1T (40 bit) as the
>> > > location of
>> > > > > the 64-bit DMA window ?
>> > > > >
>> > > > > It happens that most current radeon adapters cannot DMA  
>> there, they
>> > > have
>> > > > > a 40-bit DMA limit. I seem to be getting things to work fine  
>> using a
>> > > 39-
>> > > > > bit window, but I suppose that might collide with something  
>> else ?
>> > > > T4240 has 40bit physical address ability.
>> > > > "
>> > > > This chip's 40-bit, physical address map consists of local  
>> space and
>> > > external address
>> > > > space. For the local address map, 32 local access windows  
>> (LAWs) define
>> > > mapping
>> > > > within the local 40-bit (1 TB) address space. Inbound and  
>> outbound
>> > > translation windows
>> > > > can map the chip into a larger system address space such as  
>> the RapidIO
>> > > or PCIe 64-bit
>> > > > address environment. This functionality is included in the  
>> address
>> > > translation and
>> > > > mapping units (ATMUs).
>> > > >
>> > > > "
>> > > > That should be the reason to set the DMA window to 40-bit.
>> > > I see. However if the top half of that space isn't used by  
>> default with
>> > > whatever is our current setup, it makes sense to move down the  
>> 64-bit
>> > > DMA window to allow those adapters to function don't you think ?
>> > Good to me.
>> > 40 bit DMA will prevent your radeon video card from working. Right?
>> > Your P5020 DS system only support 36 bit physical address.
>> 
>> We should probably put the "64-bit DMA" address in the device-tree,
>> that way if somebody wish to do differently they can.
> 
> I thought the device tree was for describing the hardware, rather  
> than configuration? :-)
> A kernel command line option might be more appropriate, unless you  
> just mean describing the difference between e6500 (which supports 40  
> bit addresses) and previous chips (which support 36 bits), rather  
> than an ability to move it earlier even on e6500.
> 
> That said, the current code looks broken -- it checks whether a card  
> can do 40-bit DMA, and if it can, it sets the DMA offset to (1ULL <<  
> 40), thus requiring 41-bit DMA.  It should be > instead of >= in  
> fsl_pci_dma_set_mask.
> 
> Maybe we could by default use the size of actual RAM, rather than the  
> physical address space.  Then only odd scenarios such as DMA to  
> non-kernel-owned RAM would need manual adjustment (MSIs would still  
> go through the special window below 4G).
> 
> -Scott