Memory mapping PCI memory region to user space

[Wyse, Chris]

Hi,

Thanks for all the help.

My ideal solution would have been to use sysfs, but I didn't want to
upgrade the kernel.  Therefore, I used Dave's program as a guide for
setting the protect bits appropriately, and was able to get the driver
working.  The driver required the pfn_pte macro change specified in my
original post.  It appears to be a kernel bug on PPC processors with a
greater than 32 bit address space.


Chris Wyse
Member of Technical Staff
Embedded Technologies
860-749-1556 office
860-978-0849 cell
413-778-9101 fax
http://www.windriver.com
  


-----Original Message-----
From: David Hawkins [mailto:dwh at ovro.caltech.edu] 
Sent: Thursday, March 23, 2006 12:47 PM
To: Kumar Gala
Cc: Wyse, Chris; Linuxppc-Embedded ((E-Mail))
Subject: Re: Memory mapping PCI memory region to user space

Hi Kumar,

>> When I was testing the Yosemite board as the host, I found that I 
>> could set the endian flag on the mmapped page, which then made the 
>> PCI device registers read as 32-bit quantities read back with the 
>> same layout under both x86 and PPC hosts.
> 
> Hmm, I guess I would handle this like how the reset of the kernel 
> handle is with the io routines handling the swapping.  Not sure if 
> there is any advantage to using the endian flag.  I guess if you have 
> something you are treating as just memory there would be.

I haven't used the feature, I just tested it to see what it did.

The application case I thought of was this; the PCI boards I built (that
I am revising, and replacing the DSP with a PPC) have an 8MB PCI region
that I can mmap from the host. I have a test suite that runs from the
host that manipulates registers on the boards to download FPGAs etc.
When the boards are used in a real system, the onboard DSP is generally
used, and the host just talks to the DSP.

However, for the test suite, if I have a header with definitions
like:

#define CONTROL_FPGA_ENABLE   (1 << 0)
#define CONTROL_FPGA_DONE_BIT (1 << 1)

that correspond to bits in a 32-bit PCI mmapped register. Then code in
the user-space test suite that did something like

   pci_addr[CONTROL_OFFSET] |= CONTROL_FPGA_ENABLE;

would instead need to be re-written, eg.,

   write_le32(&pci_addr[CONTROL_OFFSET], CONTROL_FPGA_ENABLE);

to be portable.

I definitely agree that this is how kernel-level code should be written,
but user-space code ... well, if I want to reuse code already written,
setting the page endian flag and reusing the code would seem like the
way to go. (This isn't what I need to do, since my host will still be an
x86, the PPC will be a target device, but I still need to think about
the endian issues).

Now of course that I have seen the consequences of my coding, I'll be
more careful to deal with endianness more appropriately.

Its a tricky trade-off though. I could define control ioctl's that hide
all the endianness issues ... but then the driver just gets bigger. I
think the appropriate solution for the user-space test code would be to
use CPU-to-little-endian routines, and wrap the lot in a re-usable
library that the test suite links against.

> There isn't a sysfs flag for the endianness page attribute since  
> thats a PPC book-e specific feature.  We could possible expand things

> to support it but, I've been trying to actively avoid using the 'E'
bit.

Ok, I haven't received the 8349E board that I am waiting on, so I hadn't
spotted that the PAGE_ENDIAN flag was Book E specific.

Thanks for your insight.

Cheers
Dave