DMA to User-Space
john.p.price at l-3com.com
john.p.price at l-3com.com
Thu Nov 5 06:02:26 EST 2009
Jonathan, what kind of memory is fpga_drv.strmdata? memory from the
kernel stack, something you allocated? You'll notice you were trying to
access location 0.
To use dma streaming memory you must call dma_map_single() to get the
physical address of fpga_drv.strmdata. That memory must be contiguous.
Then you may call remap_pfn_range with the page offset of the physical
address returned from the dma_map_single() call. Then when the cpu
accesses that region you must sync the region for with dma direction for
cpu or when the device needs to access it, for the device.
LDD3 seems a bit dated on this topic it says to use nopage instead of
remap_pfn_range(), in my experiments remap_pfn_range seemed ok. I would
ask others on the list for their experience on the topic.
Ultimately I went with mapping user space provided address and
implementing dma with scatter-gather.
-----Original Message-----
From: linuxppc-dev-bounces+john.p.price=l-3com.com at lists.ozlabs.org
[mailto:linuxppc-dev-bounces+john.p.price=l-3com.com at lists.ozlabs.org]
On Behalf Of Jonathan Haws
Sent: Wednesday, November 04, 2009 12:40 PM
To: Bo.Liu at windriver.com
Cc: linuxppc-dev at lists.ozlabs.org
Subject: RE: DMA to User-Space
> Jonathan Haws wrote:
> > All,
> >
> > I have what may be an unconventional question:
> >
> > Our application consists of data being captured by an FPGA,
> processed, and transferred to SDRAM. I simply give the FPGA an
> address of where I want it stored in SDRAM and it simply DMAs the
> data over and interrupts me when finished. I then take that data
> and store it to disk.
> >
> > I have code in user space that handles all of the writing to disk
> nicely and fast enough for my application (I am capturing data at
> about 35-40 Mbytes/sec).
> >
> > My question is this: is it possible to give a user-space pointer
> to the FPGA to DMA to? It seems like I would have problems with
> alignment, address manipulation, and a whole slew of other issues.
> >
> > What would be the best way to accomplish something like that? I
> want to handle all the disk access in user-space, but I do not want
> to have to copy 40 MB/s from kernel space to user-space either.
> >
> You can maintain a DMA buffer in kernel, then mmap to user space.
> And
> maybe you need some handshake between FPGA and the apps to balance
> input
> datas with datas to disk.
> > I can maintain an allocated, DMA-safe buffer in kernel space if
> needed. Can I simply get a user-space pointer to that buffer? What
> calls are needed to translate addresses?
> >
> Use remap_pfn_range() in your kernel DMA buffer manipulation driver
> .mmap() handler to export DMA buffer address to user space.
>
Can you provide an example for how to do that? I have an mmap routine
to map BARs that the FPGA maintains and I can access those, however when
I try to map the DMA buffer and access what is in it, the system
crashes. Here is the mmap function I have right now:
/* fpga_mmap()
*
* Description:
* The purpose of this function is to serve as a 'file_operation'
* which maps different PCI resources into the calling processes
* memory space.
*
* NOTE: The file offset are in page size; i.e.:
* offset 0 in process's mmap syscall -> BAR0
* offset 4096 in process's mmap syscall -> BAR1
* offset 8192 in process's mmap syscall -> BAR2
* offset 12288 -> Streaming DMA buffer
*
* Arguments:
* struct file *filp -- struct representing an open
file
* struct vm_area_struct *vma -- struct representing memory
'segment'
*
* Returns:
* int -- indication of success or failure
*
*/
int fpga_mmap(struct file *filp, struct vm_area_struct *vma)
{
struct pci_dev *dev;
unsigned long addressToMap;
uint8_t mapType = 0; /* 0 = IO, 1 = memory */
/* Get the PCI device */
dev = (struct pci_dev*)(filp->private_data);
/* Map in the appropriate BAR based on page offset */
if (vma->vm_pgoff == FPGA_CONFIG_SPACE)
{
/* Map BAR1 (the CONFIG area) */
printk(KERN_ALERT "FPGA: Mapping BAR1 (CONFIG BAR).\n");
addressToMap = pci_resource_start(dev,
FPGA_CONFIG_SPACE);
printk(KERN_ALERT "FPGA: PCI BAR1 (CONFIG BAR) Size ->
%#08x.\n",
pci_resource_len(dev, FPGA_CONFIG_SPACE));
mapType = 0;
}
else if(vma->vm_pgoff == FPGA_TEST_SPACE)
{
/* Map BAR2 (the TEST area) */
printk(KERN_ALERT "FPGA: Mapping BAR2 (TEST BAR).\n");
addressToMap = (pci_resource_start(dev, FPGA_TEST_SPACE)
+
pci_resource_len(dev, FPGA_TEST_SPACE)) -
FPGA_TEST_LENGTH;
printk(KERN_ALERT "FPGA: PCI BAR2 (TEST BAR) Size ->
%#08x.\n",
pci_resource_len(dev, FPGA_TEST_SPACE));
mapType = 0;
}
else if(vma->vm_pgoff == 3)
{
addressToMap = (unsigned long)&fpga_drv.strmData[0];
mapType = 1;
}
else
{
printk(KERN_ALERT " FPGA: Invalid BAR mapping
specified.\n");
return ERROR;
}
/* Execute the mapping */
vma->vm_flags |= VM_IO;
vma->vm_flags |= VM_RESERVED;
vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
printk(KERN_ALERT "FPGA: vmSize -> 0x%x.\n",
(unsigned int)(vma->vm_end - vma->vm_start));
if( mapType == 0 )
{
if(io_remap_pfn_range(vma, vma->vm_start, addressToMap
>> PAGE_SHIFT,
vma->vm_end - vma->vm_start,
vma->vm_page_prot) != 0)
{
printk(KERN_ALERT "FPGA: Failed to map BAR PCI
space to user space.\n");
return ERROR;
}
}
else
{
printk(KERN_NOTICE "FPGA: Mapping stream ptr (%#08x) to
user space\n",(uint32_t)&fpga_drv.strmData[0]);
printk(KERN_NOTICE "FPGA: Setting strmData[0][0] to
0x37\n");
fpga_drv.strmData[0][0] = 0x37;
if(remap_pfn_range(vma, vma->vm_start, addressToMap >>
PAGE_SHIFT,
vma->vm_end - vma->vm_start,
vma->vm_page_prot) != 0)
{
printk(KERN_ALERT "FPGA: Failed to map BAR PCI
space to user space.\n");
return ERROR;
}
}
return OK;
}
And here is the failure:
~ # ./main
FPGA: vmSize -> 0x4000.
FPGA: Mapping stream ptr (0xd10613bc) to user space
FPGA: Setting strmData[0][0] to 0x37
Unable to handle kernel paging request for data at address 0x00000000
Faulting instruction address: 0xd105f5e0
Oops: Kernel access of bad area, sig: 11 [#1]
PREEMPT Kilauea
Modules linked in: fpgaDriver
NIP: d105f5e0 LR: d105f5d8 CTR: c003fff4
REGS: cf86dde0 TRAP: 0300 Not tainted (2.6.30.3-wolverine-dirty)
MSR: 00029030 <EE,ME,CE,IR,DR> CR: 24008022 XER: 0000005f
DEAR: 00000000, ESR: 00800000
TASK = cf873950[281] 'main' THREAD: cf86c000
GPR00: 00000037 cf86de90 cf873950 00000028 17d78400 c0302028 c0302068
00000000
GPR08: 00000000 00000000 00000003 c0300000 24008024 100e89c8 100d6590
100917d8
GPR16: 100d0000 100d0000 c0320000 cfaca07c 00000001 cf8c0230 00000003
000000fb
GPR24: 00000000 cf8a6840 cfaca07c 00000000 d10613bc d106128c 00000004
cfaca07c
NIP [d105f5e0] fpga_mmap+0x9c/0x214 [fpgaDriver]
LR [d105f5d8] fpga_mmap+0x94/0x214 [fpgaDriver]
Call Trace:
[cf86de90] [d105f5d8] fpga_mmap+0x94/0x214 [fpgaDriver] (unreliable)
[cf86deb0] [c00c1a4c] mmap_region+0x29c/0x408
[cf86df10] [c000377c] sys_mmap+0x78/0x100
[cf86df40] [c00103fc] ret_from_syscall+0x0/0x3c
Instruction dump:
386302a8 48000735 3b9d0130 3c60d106 7f84e378 386302c4 48000721 3c60d106
386302f8 48000715 813d0130 38000037 <98090000> 7fe3fb78 809f0004
80df0008
---[ end trace 4393e8cf23caef19 ]---
Where am I going wrong?
Using mmap() would work well for me.
Also, what about this:
1. I open /dev/mem and get a file descriptor
2. I use mmap to reserve some physical addresses for my buffers in user
space.
3. I give that address to the FPGA for DMA use.
4. When I get the FPGA interrupt, I invalidate the data cache and write
the data to disk
Does that sound like it would work? Would the address I receive from
mmap() and pass to the FPGA be the actual physical address, or would I
need to send the physical address to the FPGA and use the mmap() address
to access and write to disk?
Thanks for the help!
Jonathan
_______________________________________________
Linuxppc-dev mailing list
Linuxppc-dev at lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev
More information about the Linuxppc-dev
mailing list