How to handle cache when I allocate phys memory?

[Benjamin Herrenschmidt]

On Wed, 2011-10-12 at 16:08 -0500, Ayman El-Khashab wrote:
> I'm using the 460sx (440 core) so no snooping here.  What
> I've done is reserved the top of memory for my driver.  My
> driver can read/write the memory and I can mmap it just
> fine.  The problem is I want to enable caching on the mmap
> for performance but I don't know / can't figure out how to
> tell the kernel to sync the cache after it gets dma data
> from the device or after i put data into it from user space.
> I know how to do it from regular devices, but not when I've
> allocated the physical memory myself.  I suppose what I am
> looking for is something akin to dma_sync_single cpu/device.
> 
> In my device driver, I am allocating the memory like this, 
> in this case the buffer is about 512MB.
> 
>  vma->vm_flags |= VM_LOCKED | VM_RESERVED;
> 
>  /* map the physical area into one buffer */
>  rc = remap_pfn_range(vma, vma->vm_start, 
>                          (PHYS_MEM_ADDR)>>PAGE_SHIFT, 
>                          len, vma->vm_page_prot);
> 
> Is this going to give me the best performance, or is there
> something more I can do?
> 
> Failing that, what is the best way to do this (i need a very
> large contiguous buffer).  it runs in batch mode, so it
> DMAs, stops, cpu reads, cpu writes, repeat ...

Did you try looking at what the dma_* functions do under the hood and
call it directly (or reproducing it) ?

Basically it boils down to using dcbf instructions to flush dirty data
or dcbi to invalidate cache lines.

Cheers,
Ben.