AW: PowerPC PCI DMA issues (prefetch/coherency?)

[Mikhail Zolotaryov]

Hi,

Why manage cache lines  manually, if appropriate code is a part of 
__dma_sync / dma_sync_single_for_device of DMA API ? (implies 
CONFIG_NOT_COHERENT_CACHE enabled, as default for Sequoia Board)

Prodyut Hazarika wrote:
> Hi Adam,
>
>   
>> Yes, I am using the 440EPx (same as the sequoia board). 
>> Our ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
>>     
> (using
>   
>> the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on timing)
>> end up being partially corrupted when we try to parse the data in the
>> virtual page. We have confirmed the data is good before the PCI-IDE
>> bridge. We are creating two 8K pages and map them to physical DMA
>>     
> memory
>   
>> using single-entry scatter/gather structs. When a DMA block is
>> corrupted, we see a random portion of it (always a multiple of 16byte
>> cache lines) is overwritten with old data from the last time the
>>     
> buffer
>   
>> was used. 
>>     
>
> This looks like a cache coherency problem.
> Can you ensure that the TLB entries corresponding to the DMA region has
> the CacheInhibit bit set.
> You will need a BDI connected to your system.
>
> Also, you will need to invalidate and flush the lines appropriately,
> since in 440 cores,
> L1Cache coherency is managed entirely by software.
> Please look at drivers/net/ibm_newemac/mal.c and core.c for example on
> how to do it.
>
> Thanks
> Prodyut
>
> On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
>   
>> Hi Adam,
>>
>>     
>>> Are you sure there is L2 cache on the 440?
>>>       
>> It depends on the SoC you are using. SoC like 460EX (Canyonlands
>>     
> board)
>   
>> have L2Cache.
>> It seems you are using a Sequoia board, which has a 440EPx SoC. 440EPx
>> has a 440 cpu core, but no L2Cache.
>> Could you please tell me which SoC you are using?
>> You can also refer to the appropriate dts file to see if there is L2C.
>> For example, in canyonlands.dts (460EX based board), we have the L2C
>> entry.
>>         L2C0: l2c {
>>               ...
>>         }
>>
>>     
>>> I am seeing this problem with our custom IDE driver which is based on
>>>       
>
>   
>>> pretty old code. Our driver uses pci_alloc_consistent() to allocate
>>>       
> the
>   
>>> physical DMA memory and alloc_pages() to allocate a virtual page. It 
>>> then uses pci_map_sg() to map to a scatter/gather buffer. Perhaps I 
>>> should convert these to the DMA API calls as you suggest.
>>>       
>> Could you give more details on the consistency problem? It is a good
>> idea to change to the new DMA APIs, but pci_alloc_consistent() should
>> work too
>>
>> Thanks
>> Prodyut	
>>
>> On Thu, 2009-09-03 at 19:57 +1000, Benjamin Herrenschmidt wrote:
>>     
>>> On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
>>>       
>>>> Hi Adam,
>>>>
>>>> If you have a look in include/asm-ppc/pgtable.h for the following
>>>>         
>> section:
>>     
>>>> #ifdef CONFIG_44x
>>>> #define _PAGE_BASE    (_PAGE_PRESENT | _PAGE_ACCESSED |
>>>>         
>> _PAGE_GUARDED)
>>     
>>>> #else
>>>> #define _PAGE_BASE    (_PAGE_PRESENT | _PAGE_ACCESSED)
>>>> #endif
>>>>
>>>> Try adding _PAGE_COHERENT to the appropriate line above and see if
>>>>         
>> that 
>>     
>>>> fixes your issue - this causes the 'M' bit to be set on the page
>>>>         
>> which 
>>     
>>>> sure enforce cache coherency. If it doesn't, you'll need to check
>>>>         
>> the 
>>     
>>>> 'M' bit isn't being masked out in head_44x.S (it was originally
>>>>         
>> masked 
>>     
>>>> out on arch/powerpc, but was fixed in later kernels when the cache
>>>>         
>
>   
>>>> coherency issues with non-SMP systems were resolved).
>>>>         
>>> I have some doubts about the usefulness of doing that for 4xx.
>>>       
> AFAIK,
>   
>>> the 440 core just ignores M.
>>>
>>> The problem lies probably elsewhere. Maybe the L2 cache coherency
>>>       
>> isn't
>>     
>>> enabled or not working ?
>>>
>>> The L1 cache on 440 is simply not coherent, so drivers have to make
>>>       
>> sure
>>     
>>> they use the appropriate DMA APIs which will do cache flushing when
>>> needed.
>>>
>>> Adam, what driver is causing you that sort of problems ?
>>>
>>> Cheers,
>>> Ben.
>>>
>>>
>>>