User allocated memory for DMA

[Jean-Christophe Dubois]

On Mon, 2004-12-06 at 18:29 +0530, Amit K Tiwari wrote:
> I am writing a driver for a PCIX board to run on Y-HPC (64-bit Yellow
> Dog Linux pre-installed on Apple XServe). The application is to
> process the data acquired from the board and send the results out. The
> data needs to be acquired by DMA into a user allocated buffer. The
> amount of data is huge and I need about 1.5 GB per DMA operation.
> 
> Here is what I am doing:
> 1. malloc and mlock the memory in user mode.

Just curious, but is all the G5 memory DMAable? And is your PCIX card
DMA engine 64 bits capable? If yes maybe you should consider an approach
similar to the bigphysarea patch. This way you could be sure to get one
single contiguous block the size you want. As a drawback it may mean
that a lot of memory will not be available to Linux but that may be a
price you are ready to pay for your application as it will consume this
much memory anyway. If required it is trivial to implement some kind of
filesystem on top of it to manage this memory region from user space (we
have done something like this here even if we didn't need such big chunk
of memory).

It is a design rather different from the scatter/gather list but it
might be more efficient for you overall.

> 2. get the user pages (get_user_pages) in the driver. 
> 3. Find out how much physically contiguous memory did the application
> get (Pass through all the pages and see if the physical addresses got
> from page_to_phys are contiguous).
> 4. Prepare a scatter-gather list of contiguous regions. 
> 5. pci_map_sg the sg list to get the DMA addresses for each of the entries.
> 
> My problem is that at step 3 I do not get any contiguous region.
> Surprisingly, some of the physical addresses are such that the low
> order page (say page 151) has high order physical address (say,
> 1ef234000) while the next high order page (say page 152) has the next
> lower physical address(say 1ef233000) and this trend continues for
> some 10 pages after which the addresses totally far apart.
> 
> As a result of this, my scatter gather list has as many entries 
> as there are pages allocated. 
> 
> Am I doing something bad here? 
> I know there is limit on memory that can be allocated through
> pci_alloc_*. Is there any other way in which I can allocate memory for
> DMA?
> 
> Thanks, 
> Amit K T
> _______________________________________________
> Linuxppc64-dev mailing list
> Linuxppc64-dev at ozlabs.org
> https://ozlabs.org/cgi-bin/mailman/listinfo/linuxppc64-dev