User allocated memory for DMA

[Amit K Tiwari]

Thanks, it helped a lot.
Here is what works - 
1. Increase the shared memory limit (sysctl kernel.shmmax)
2. Configure the large pages (echo 256 > /proc/sys/vm/nr_hugepages)
3. Use shmget to get a handle to huge pages - Set the  SHM_HUGETLB
bits in flags.



On Mon, 06 Dec 2004 08:21:59 -0600, Olof Johansson <olof at austin.ibm.com> wrote:
> Amit K Tiwari wrote:
> 
> >I am writing a driver for a PCIX board to run on Y-HPC (64-bit Yellow
> >Dog Linux pre-installed on Apple XServe). The application is to
> >process the data acquired from the board and send the results out. The
> >data needs to be acquired by DMA into a user allocated buffer. The
> >amount of data is huge and I need about 1.5 GB per DMA operation.
> >
> >
> The current IOMMU isn't quite written with the expectation that the user
> will keep a 1.5GB mapping of DMA memory active for an extensive period
> of time. It should work, but let me know if you see any strangeness.
> 
> 
> 
> >Here is what I am doing:
> >1. malloc and mlock the memory in user mode.
> >2. get the user pages (get_user_pages) in the driver.
> >3. Find out how much physically contiguous memory did the application
> >get (Pass through all the pages and see if the physical addresses got
> >from page_to_phys are contiguous).
> >4. Prepare a scatter-gather list of contiguous regions.
> >5. pci_map_sg the sg list to get the DMA addresses for each of the entries.
> >
> >My problem is that at step 3 I do not get any contiguous region.
> >Surprisingly, some of the physical addresses are such that the low
> >order page (say page 151) has high order physical address (say,
> >1ef234000) while the next high order page (say page 152) has the next
> >lower physical address(say 1ef233000) and this trend continues for
> >some 10 pages after which the addresses totally far apart.
> >
> >
> This doesn't really surprise me. You have no guarantees where in
> physical memory that a process will get its pages allocated, they can be
> anywhere. The nice thing is that with the IOMMU, you can get a mostly
> contigous address range as seen by the PCI adapter. The translation
> between the two is transparent to the DMA operation.
> 
> >As a result of this, my scatter gather list has as many entries
> >as there are pages allocated.
> >
> >Am I doing something bad here?
> >I know there is limit on memory that can be allocated through
> >pci_alloc_*. Is there any other way in which I can allocate memory for
> >DMA?
> >
> >
> Have you considered making your application use large pages? If so,
> you'd at least get 16MB contigous at a time.
> 
> 
> -Olof
>