User allocated memory for DMA

[Olof Johansson]

Amit K Tiwari wrote:

>I am writing a driver for a PCIX board to run on Y-HPC (64-bit Yellow
>Dog Linux pre-installed on Apple XServe). The application is to
>process the data acquired from the board and send the results out. The
>data needs to be acquired by DMA into a user allocated buffer. The
>amount of data is huge and I need about 1.5 GB per DMA operation.
>  
>
The current IOMMU isn't quite written with the expectation that the user 
will keep a 1.5GB mapping of DMA memory active for an extensive period 
of time. It should work, but let me know if you see any strangeness.

>Here is what I am doing:
>1. malloc and mlock the memory in user mode.
>2. get the user pages (get_user_pages) in the driver. 
>3. Find out how much physically contiguous memory did the application
>get (Pass through all the pages and see if the physical addresses got
>from page_to_phys are contiguous).
>4. Prepare a scatter-gather list of contiguous regions. 
>5. pci_map_sg the sg list to get the DMA addresses for each of the entries.
>
>My problem is that at step 3 I do not get any contiguous region.
>Surprisingly, some of the physical addresses are such that the low
>order page (say page 151) has high order physical address (say,
>1ef234000) while the next high order page (say page 152) has the next
>lower physical address(say 1ef233000) and this trend continues for
>some 10 pages after which the addresses totally far apart.
>  
>
This doesn't really surprise me. You have no guarantees where in 
physical memory that a process will get its pages allocated, they can be 
anywhere. The nice thing is that with the IOMMU, you can get a mostly 
contigous address range as seen by the PCI adapter. The translation 
between the two is transparent to the DMA operation.

>As a result of this, my scatter gather list has as many entries 
>as there are pages allocated. 
>
>Am I doing something bad here? 
>I know there is limit on memory that can be allocated through
>pci_alloc_*. Is there any other way in which I can allocate memory for
>DMA?
>  
>
Have you considered making your application use large pages? If so, 
you'd at least get 16MB contigous at a time.


-Olof