83xx: Marking or Allocating Pages as Cache-Inhibited

Sat Mar 7 03:30:19 EST 2009

Thank you for your help!  That bit resolved all of the RDMA/WDMA coherency
issues on the CSB side...except:

We expose a 1MB region of memory from CSB via a BAR (BAR1, if it matters) to
the Host.  This region is also not behaving correctly with respect to
coherency on SOME hosts; again, disabling our cache makes it work correctly
on all hosts.  We have set PEX_DEVICE_CONTROL in PCI-E Config Space (0x54)
to 0x2010 (sorry about the endianness below).  We thought that CLEARING the
no-snoop bit here would indicate that snooping was required for this
region...is this a similar issue?

- Ben

On Fri, Mar 6, 2009 at 10:12 AM, Ben Menchaca <ben.menchaca at gmail.com>wrote:

> Testing now...it looks like it (almost) works, though!  Why does setting
> no-snoop cause snooping to work?  More on the effect on setting that bit in
> a few minutes...need more testing.
>
> ACR is 0x00030300.
>
> - Ben
>
>
> On Fri, Mar 6, 2009 at 12:30 AM, Liu Dave-R63238 <DaveLiu at freescale.com>wrote:
>
>>  Did you enable the descriptor bit 3 to have a try?
>>
>>  ------------------------------
>> *From:* Ben Menchaca [mailto:ben.menchaca at gmail.com]
>> *Sent:* Friday, March 06, 2009 2:10 PM
>>
>> *To:* Liu Dave-R63238
>> *Cc:* linuxppc-dev at ozlabs.org
>> *Subject:* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited
>>
>> I can look at ACR morning...although I can say with a fair amount of
>> certainty that I have not changed it from the POR value.
>>
>> I will try enabling No Snoop for CSB in the descriptor (bit 3,
>> yes?)...this seems a bit counterintuitive to me.
>>
>> What is the hope regarding these two?  Some combination I am not seeing?
>>
>>
>> On Thu, Mar 5, 2009 at 11:40 PM, Liu Dave-R63238 <DaveLiu at freescale.com>wrote:
>>
>>>  what is the value of ACR register?
>>>
>>>  ------------------------------
>>> *From:* Ben Menchaca [mailto:ben.menchaca at gmail.com]
>>> *Sent:* Friday, March 06, 2009 1:38 PM
>>> *To:* Liu Dave-R63238
>>> *Cc:* linuxppc-dev at ozlabs.org
>>> *Subject:* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited
>>>
>>>   1.  BAT2 in linux is set to WIMG=0010, and covers all 64M
>>> 2.  PEX_DEVICE_CONTROL in PCI-E Config Space (0x54): 0x1020
>>> 3.  PEX_xDMA_CTRL is set to 0x00000401 at the initiation of the DMA.
>>> 4.  OWAR0 is set to 0xFFFFF005, so NSNP is 0.
>>> 5.  The DMA descriptor (randomly chosen when I hit a trigger...just
>>> ignore the size...) contains 0002AFF3 at offset 0, so nosnoops are cleared.
>>>
>>>
>>> Core is 400MHz, and CSB is 133MHz.
>>>
>>> - Ben
>>>
>>> On Thu, Mar 5, 2009 at 11:27 PM, Liu Dave-R63238 <DaveLiu at freescale.com>wrote:
>>>
>>>> and what settings  is DMA description bit 3?
>>>>
>>>> > -----Original Message-----
>>>> > From: linuxppc-dev-bounces+daveliu=freescale.com at ozlabs.org
>>>> > [mailto:linuxppc-dev-bounces+daveliu <linuxppc-dev-bounces%2Bdaveliu>
>>>> =freescale.com at ozlabs.org]
>>>>   >  On Behalf Of Liu Dave-R63238
>>>> > Sent: Friday, March 06, 2009 1:22 PM
>>>> > To: Ben Menchaca; linuxppc-dev at ozlabs.org
>>>> > Subject: RE: 83xx: Marking or Allocating Pages as Cache-Inhibited
>>>> >
>>>> > Did you enable the snoop bit at PEX_WDMA_CTRL[SNOOP] and
>>>> > PEX_RDMA_CTRL[SNOOP]?
>>>> >
>>>> > What is the freq settings? CORE/CSB bus.
>>>> >
>>>> > Thanks, Dave
>>>> >
>>>> > ________________________________
>>>> >
>>>> >       From: linuxppc-dev-bounces+daveliu=freescale.com at ozlabs.org
>>>> > [mailto:linuxppc-dev-bounces+daveliu <linuxppc-dev-bounces%2Bdaveliu>
>>>> =freescale.com at ozlabs.org]
>>>> >  On Behalf Of Ben Menchaca
>>>> >       Sent: Friday, March 06, 2009 12:33 PM
>>>> >       To: linuxppc-dev at ozlabs.org
>>>> >       Subject: 83xx: Marking or Allocating Pages as Cache-Inhibited
>>>> >
>>>> >
>>>> >       I am working on a Freescale 8314e design, and the
>>>> > embedded device is configured as a PCI-e endpoint running a
>>>> > 2.6.27-5 kernel.  For context, we have written a kernel
>>>> > module which, among other things, uses the RDMA/WDMA engine
>>>> > in the PCI-e IP block.  On the host side, these DMAs are
>>>> > coherent.  However, on the embedded side, things are quite a
>>>> > bit less rosy; we must manually flush/invalidate cache lines
>>>> > for WDMA/RDMAs to occur successfully.  After speaking with
>>>> > (several) FAEs at Freescale, we believe there is a
>>>> > configuration issue that is the cause, but we have yet to
>>>> > have anyone successfully point to it.
>>>> >
>>>> >       Disabling the data cache altogether resolves the issue
>>>> > entirely, but of course, also completely tanks performance.
>>>> > As a temporary workaround, I would like to simply mark the
>>>> > pages (obtained currently via dma_alloc_coherent) involved as
>>>> > cache-inhibited.  I have attempted to do this via some
>>>> > snippets remaining in fec.c (va_to_pte, uncache_pte to set
>>>> > _PAGE_NO_CACHE, flush_tlb_page, then unmap_pte), but this is
>>>> > almost certainly braindead; va_to_pte is not a part of the
>>>> > 83xx source, as far as I can tell; 8xx only.
>>>> >
>>>> >       A quick pointer in the correct direction for marking
>>>> > pages as cache-inhibited on a 2.6.27-5 kernel would be
>>>> > appreciated, or if my approach to a workaround is flawed, a
>>>> > pointer to the correct way would be great.
>>>> >
>>>> >       Ben Menchaca
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > Linuxppc-dev mailing list
>>>> > Linuxppc-dev at ozlabs.org
>>>> > https://ozlabs.org/mailman/listinfo/linuxppc-dev
>>>> >
>>>> >
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ozlabs.org/pipermail/linuxppc-dev/attachments/20090306/fab2bfaf/attachment.htm>