Problem with DMAing into hugetlb pages on ppc64

[Sexton, Matt]

Hi all,

I originally posed this question on lkml and am reposting a slightly
modified version here.  

On a ppc64 platform running 2.6.13-1, I'm having trouble with an IO
device DMAing into a memory buffer allocated by mmap'ing a hugetlbfs
file and whose physical addresses were determined by calling
get_user_pages().  The DMAed data often show up in the wrong 16MB page.

Specifically, I have written a driver for a PCI device that writes data
into a user-allocated memory buffer.  The user passes the virtual
address of the buffer to the driver, which calls get_user_pages() to
lock the pages and to get the page information in order to be able to
build a scatter list of contiguous physical blocks to pass to the DMA
engine on the device. The device then writes a known pattern to the
buffer, which the user space program can verify.

This process works fine on ia32 and ppc64 using malloc'ed memory.  This
process also works fine on ia32 when obtaining the memory by mmap'ing a
file on a hugetlbfs filesystem.  The huge pages [2MB in the case of
ia32] are used to reduce the number of entries in the scatter list.

The process doesn't work so well on ppc64 with hugetlbfs (and 16MB
pages).  Often, the data is written to the wrong 16MB pages, from the
perspective of the user space program.  The data is correct within a
16MB page - it's just written to the wrong page.  Perhaps the wrong
physical address is returned by get_user_pages()?  Or perhaps there is
some inconsistency between get_user_pages() and mmap()?  [I did receive
one response with test results that indicated this was not the case.]

Any suggestions on what I could be doing wrong in this specific case?
Any known problems with the kernel in this case?

Please CC me on any replies.

Thanks,
Matt