DMA to User-Space

Thu Nov 5 04:40:22 EST 2009

> Jonathan Haws wrote:
> > All,
> >
> > I have what may be an unconventional question:
> >
> > Our application consists of data being captured by an FPGA,
> processed, and transferred to SDRAM.  I simply give the FPGA an
> address of where I want it stored in SDRAM and it simply DMAs the
> data over and interrupts me when finished.  I then take that data
> and store it to disk.
> >
> > I have code in user space that handles all of the writing to disk
> nicely and fast enough for my application (I am capturing data at
> about 35-40 Mbytes/sec).
> >
> > My question is this:  is it possible to give a user-space pointer
> to the FPGA to DMA to?  It seems like I would have problems with
> alignment, address manipulation, and a whole slew of other issues.
> >
> > What would be the best way to accomplish something like that?  I
> want to handle all the disk access in user-space, but I do not want
> to have to copy 40 MB/s from kernel space to user-space either.
> >
> You can maintain a DMA buffer in kernel, then mmap to user space.
> And
> maybe you need some handshake between FPGA and the apps to balance
> input
> datas  with datas to disk.
> > I can maintain an allocated, DMA-safe buffer in kernel space if
> needed.  Can I simply get a user-space pointer to that buffer?  What
> calls are needed to translate addresses?
> >
> Use remap_pfn_range()  in your kernel DMA buffer manipulation driver
> .mmap() handler to export DMA  buffer address to user space.
> 

Can you provide an example for how to do that?  I have an mmap routine to map BARs that the FPGA maintains and I can access those, however when I try to map the DMA buffer and access what is in it, the system crashes.  Here is the mmap function I have right now:

/* fpga_mmap()
 *
 *  Description:
 *       The purpose of this function is to serve as a 'file_operation'
 *       which maps different PCI resources into the calling processes
 *       memory space.
 *
 *       NOTE:  The file offset are in page size; i.e.:
 *       offset 0 in process's mmap syscall -> BAR0
 *       offset 4096 in process's mmap syscall -> BAR1
 *       offset 8192 in process's mmap syscall -> BAR2
 *	   offset 12288 -> Streaming DMA buffer
 *
 *  Arguments:
 *       struct file             *filp -- struct representing an open file
 *       struct vm_area_struct   *vma  -- struct representing memory 'segment'
 *
 *  Returns:
 *       int -- indication of success or failure
 *
 */
int fpga_mmap(struct file *filp, struct vm_area_struct *vma)
{
	struct pci_dev *dev;
	unsigned long addressToMap;
	uint8_t mapType = 0; /* 0 = IO, 1 = memory */

	/* Get the PCI device */
	dev = (struct pci_dev*)(filp->private_data);

	/* Map in the appropriate BAR based on page offset */
	if (vma->vm_pgoff == FPGA_CONFIG_SPACE)
	{
		/* Map BAR1 (the CONFIG area) */
		printk(KERN_ALERT "FPGA: Mapping BAR1 (CONFIG BAR).\n");
		addressToMap = pci_resource_start(dev, FPGA_CONFIG_SPACE);
		printk(KERN_ALERT "FPGA: PCI BAR1 (CONFIG BAR) Size -> %#08x.\n",
			pci_resource_len(dev, FPGA_CONFIG_SPACE));
		mapType = 0;
	}
	else if(vma->vm_pgoff == FPGA_TEST_SPACE)
	{
		/* Map BAR2 (the TEST area) */
		printk(KERN_ALERT "FPGA: Mapping BAR2 (TEST BAR).\n");
		addressToMap = (pci_resource_start(dev, FPGA_TEST_SPACE) +
			pci_resource_len(dev, FPGA_TEST_SPACE)) - FPGA_TEST_LENGTH;
		printk(KERN_ALERT "FPGA: PCI BAR2 (TEST BAR) Size -> %#08x.\n",
			pci_resource_len(dev, FPGA_TEST_SPACE));
		mapType = 0;
	}
	else if(vma->vm_pgoff == 3)
	{

		addressToMap = (unsigned long)&fpga_drv.strmData[0];
		mapType = 1;
	}
	else
	{
		printk(KERN_ALERT " FPGA: Invalid BAR mapping specified.\n");
		return ERROR;
	}

	/* Execute the mapping */
	vma->vm_flags |= VM_IO;
	vma->vm_flags |= VM_RESERVED;
	vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
	printk(KERN_ALERT "FPGA: vmSize -> 0x%x.\n",
			(unsigned int)(vma->vm_end - vma->vm_start));

	if( mapType == 0 )
	{
		if(io_remap_pfn_range(vma, vma->vm_start, addressToMap >> PAGE_SHIFT,
				vma->vm_end - vma->vm_start, vma->vm_page_prot) != 0)
		{
			printk(KERN_ALERT "FPGA: Failed to map BAR PCI space to user space.\n");
			return ERROR;
		}
	}
	else
	{
		printk(KERN_NOTICE "FPGA: Mapping stream ptr (%#08x) to user space\n",(uint32_t)&fpga_drv.strmData[0]);
		printk(KERN_NOTICE "FPGA: Setting strmData[0][0] to 0x37\n");
		fpga_drv.strmData[0][0] = 0x37;
		if(remap_pfn_range(vma, vma->vm_start, addressToMap >> PAGE_SHIFT,
				vma->vm_end - vma->vm_start, vma->vm_page_prot) != 0)
		{
			printk(KERN_ALERT "FPGA: Failed to map BAR PCI space to user space.\n");
			return ERROR;
		}
	}

	return OK;
}

And here is the failure:

~ # ./main
FPGA: vmSize -> 0x4000.
FPGA: Mapping stream ptr (0xd10613bc) to user space
FPGA: Setting strmData[0][0] to 0x37
Unable to handle kernel paging request for data at address 0x00000000
Faulting instruction address: 0xd105f5e0
Oops: Kernel access of bad area, sig: 11 [#1]
PREEMPT Kilauea
Modules linked in: fpgaDriver
NIP: d105f5e0 LR: d105f5d8 CTR: c003fff4
REGS: cf86dde0 TRAP: 0300   Not tainted  (2.6.30.3-wolverine-dirty)
MSR: 00029030 <EE,ME,CE,IR,DR>  CR: 24008022  XER: 0000005f
DEAR: 00000000, ESR: 00800000
TASK = cf873950[281] 'main' THREAD: cf86c000
GPR00: 00000037 cf86de90 cf873950 00000028 17d78400 c0302028 c0302068 00000000
GPR08: 00000000 00000000 00000003 c0300000 24008024 100e89c8 100d6590 100917d8
GPR16: 100d0000 100d0000 c0320000 cfaca07c 00000001 cf8c0230 00000003 000000fb
GPR24: 00000000 cf8a6840 cfaca07c 00000000 d10613bc d106128c 00000004 cfaca07c
NIP [d105f5e0] fpga_mmap+0x9c/0x214 [fpgaDriver]
LR [d105f5d8] fpga_mmap+0x94/0x214 [fpgaDriver]
Call Trace:
[cf86de90] [d105f5d8] fpga_mmap+0x94/0x214 [fpgaDriver] (unreliable)
[cf86deb0] [c00c1a4c] mmap_region+0x29c/0x408
[cf86df10] [c000377c] sys_mmap+0x78/0x100
[cf86df40] [c00103fc] ret_from_syscall+0x0/0x3c
Instruction dump:
386302a8 48000735 3b9d0130 3c60d106 7f84e378 386302c4 48000721 3c60d106
386302f8 48000715 813d0130 38000037 <98090000> 7fe3fb78 809f0004 80df0008
---[ end trace 4393e8cf23caef19 ]---

Where am I going wrong?

Using mmap() would work well for me.

Also, what about this:

1. I open /dev/mem and get a file descriptor
2. I use mmap to reserve some physical addresses for my buffers in user space.
3. I give that address to the FPGA for DMA use.
4. When I get the FPGA interrupt, I invalidate the data cache and write the data to disk

Does that sound like it would work?  Would the address I receive from mmap() and pass to the FPGA be the actual physical address, or would I need to send the physical address to the FPGA and use the mmap() address to access and write to disk?

Thanks for the help!

Jonathan