scatter/gather DMA and cache coherency

[Eugene Surovegin]

On Thu, Feb 16, 2006 at 05:51:20PM +1030, Phil Nitschke wrote:
> Hi,
> 
> I've been using a PCI device driver developed by a third party company.
> It uses a scatter/gather DMA I/O to transfer data from the PCI device
> into user memory.  When using a buffer size of about 1 MB, the driver
> achieves a transfer bandwidth of about 60 MB/s, on a 66 MHz, 32-bit
> bus.
> 
> The problem is, that sometimes the data is corrupt (usually on the first
> transfer).  We've concluded that the problem is related to cache
> coherency.  The Artesyn 2.6.10 reference kernel (branched from the
> kernel at penguinppc.org) must be built with CONFIG_NOT_COHERENT_CACHE=y,
> as Artesyn have never successfully verified operation with hardware
> coherency enabled.
> My understanding is that their Marvel system controller (MV64460)
> supports cache snooping, but their Linux kernel support hasn't caught up
> yet.
> 
> So if I understand my situation correctly, the device driver must use
> software-enforced coherency to avoid data corruption.  Is this correct?
> 
> What currently happens is this:
> 
> The buffers are allocated with get_user_pages(...)
> 
> After each DMA transfer is complete, the driver invalidates the cache
> using  __dma_sync_page(...)

No, buffers must be invalidated _before_ DMA transfer, not after. 
Also, don't use internal PPC functions like __dma_sync_page. Please, 
read Documentation/DMA-API.txt for official API.

-- 
Eugene