83xx: Marking or Allocating Pages as Cache-Inhibited

Mon Mar 9 14:27:39 EST 2009

The only problem host found so far is a newer Asus 970FX motherboard,
regardless of OS.

We are seeing that some time after LTSSM finishes, but long before OS load,
our PEX_DEVICE_CONTROL register is changed.  On the "working" motherboards,
NO_SNOOP is enabled; if I read the spec correctly, this means that TLPs with
no-snoop are permitted.  On the non-working motherboards, NO_SNOOP is
disabled; this is supposed to mean that TLPs with no-snoop are not permitted
(again, if I understand correctly).  Is it possible that there has been
another misinterpretation of this bit?  Something regarding the generation
of snoops on CSB when it is cleared?

This bit could be a complete red herring; it was one of the few differences
in the config space, however, so it was my best guess.

- Ben

On Sun, Mar 8, 2009 at 9:56 PM, Liu Dave-R63238 <DaveLiu at freescale.com>wrote:

>  Hi Ben,
>
> The second issue. you told me "some hosts" has problem,
> and some hosts worked well.
>
> what is the problem-hosts?
>
> The issue seems like the hosts did set the NO SNOOP attribute
> bit at TLP.
>
> The PEX_DEVICE_CONTROL is standard PCI configuration space
> register, it controls the behavior of the initiator's transaction.
> For 8315, it is outbound, not inbound transaction.
>
> Thanks, Dave
>
>  ------------------------------
> *From:* Ben Menchaca [mailto:ben.menchaca at gmail.com]
> *Sent:* Saturday, March 07, 2009 12:30 AM
> *To:* Liu Dave-R63238
> *Cc:* linuxppc-dev at ozlabs.org
> *Subject:* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited
>
> Thank you for your help!  That bit resolved all of the RDMA/WDMA coherency
> issues on the CSB side...except:
>
> We expose a 1MB region of memory from CSB via a BAR (BAR1, if it matters)
> to the Host.  This region is also not behaving correctly with respect to
> coherency on SOME hosts; again, disabling our cache makes it work correctly
> on all hosts.  We have set PEX_DEVICE_CONTROL in PCI-E Config Space (0x54)
> to 0x2010 (sorry about the endianness below).  We thought that CLEARING
> the no-snoop bit here would indicate that snooping was required for this
> region...is this a similar issue?
>
> - Ben
>
> On Fri, Mar 6, 2009 at 10:12 AM, Ben Menchaca <ben.menchaca at gmail.com>wrote:
>
>> Testing now...it looks like it (almost) works, though!  Why does setting
>> no-snoop cause snooping to work?  More on the effect on setting that bit in
>> a few minutes...need more testing.
>>
>> ACR is 0x00030300.
>>
>> - Ben
>>
>>
>> On Fri, Mar 6, 2009 at 12:30 AM, Liu Dave-R63238 <DaveLiu at freescale.com>wrote:
>>
>>>  Did you enable the descriptor bit 3 to have a try?
>>>
>>>  ------------------------------
>>>  *From:* Ben Menchaca [mailto:ben.menchaca at gmail.com]
>>> *Sent:* Friday, March 06, 2009 2:10 PM
>>>
>>> *To:* Liu Dave-R63238
>>> *Cc:* linuxppc-dev at ozlabs.org
>>> *Subject:* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited
>>>
>>>   I can look at ACR morning...although I can say with a fair amount of
>>> certainty that I have not changed it from the POR value.
>>>
>>> I will try enabling No Snoop for CSB in the descriptor (bit 3,
>>> yes?)...this seems a bit counterintuitive to me.
>>>
>>> What is the hope regarding these two?  Some combination I am not seeing?
>>>
>>>
>>> On Thu, Mar 5, 2009 at 11:40 PM, Liu Dave-R63238 <DaveLiu at freescale.com>wrote:
>>>
>>>>  what is the value of ACR register?
>>>>
>>>>  ------------------------------
>>>> *From:* Ben Menchaca [mailto:ben.menchaca at gmail.com]
>>>> *Sent:* Friday, March 06, 2009 1:38 PM
>>>> *To:* Liu Dave-R63238
>>>> *Cc:* linuxppc-dev at ozlabs.org
>>>> *Subject:* Re: 83xx: Marking or Allocating Pages as Cache-Inhibited
>>>>
>>>>   1.  BAT2 in linux is set to WIMG=0010, and covers all 64M
>>>> 2.  PEX_DEVICE_CONTROL in PCI-E Config Space (0x54): 0x1020
>>>> 3.  PEX_xDMA_CTRL is set to 0x00000401 at the initiation of the DMA.
>>>> 4.  OWAR0 is set to 0xFFFFF005, so NSNP is 0.
>>>> 5.  The DMA descriptor (randomly chosen when I hit a trigger...just
>>>> ignore the size...) contains 0002AFF3 at offset 0, so nosnoops are cleared.
>>>>
>>>>
>>>> Core is 400MHz, and CSB is 133MHz.
>>>>
>>>> - Ben
>>>>
>>>> On Thu, Mar 5, 2009 at 11:27 PM, Liu Dave-R63238 <DaveLiu at freescale.com
>>>> > wrote:
>>>>
>>>>> and what settings  is DMA description bit 3?
>>>>>
>>>>> > -----Original Message-----
>>>>> > From: linuxppc-dev-bounces+daveliu=freescale.com at ozlabs.org
>>>>> > [mailto:linuxppc-dev-bounces+daveliu<linuxppc-dev-bounces%2Bdaveliu>
>>>>> =freescale.com at ozlabs.org]
>>>>>  >  On Behalf Of Liu Dave-R63238
>>>>> > Sent: Friday, March 06, 2009 1:22 PM
>>>>> > To: Ben Menchaca; linuxppc-dev at ozlabs.org
>>>>> > Subject: RE: 83xx: Marking or Allocating Pages as Cache-Inhibited
>>>>> >
>>>>> > Did you enable the snoop bit at PEX_WDMA_CTRL[SNOOP] and
>>>>> > PEX_RDMA_CTRL[SNOOP]?
>>>>> >
>>>>> > What is the freq settings? CORE/CSB bus.
>>>>> >
>>>>> > Thanks, Dave
>>>>> >
>>>>> > ________________________________
>>>>> >
>>>>> >       From: linuxppc-dev-bounces+daveliu=freescale.com at ozlabs.org
>>>>> > [mailto:linuxppc-dev-bounces+daveliu<linuxppc-dev-bounces%2Bdaveliu>
>>>>> =freescale.com at ozlabs.org]
>>>>> >  On Behalf Of Ben Menchaca
>>>>> >       Sent: Friday, March 06, 2009 12:33 PM
>>>>> >       To: linuxppc-dev at ozlabs.org
>>>>> >       Subject: 83xx: Marking or Allocating Pages as Cache-Inhibited
>>>>> >
>>>>> >
>>>>> >       I am working on a Freescale 8314e design, and the
>>>>> > embedded device is configured as a PCI-e endpoint running a
>>>>> > 2.6.27-5 kernel.  For context, we have written a kernel
>>>>> > module which, among other things, uses the RDMA/WDMA engine
>>>>> > in the PCI-e IP block.  On the host side, these DMAs are
>>>>> > coherent.  However, on the embedded side, things are quite a
>>>>> > bit less rosy; we must manually flush/invalidate cache lines
>>>>> > for WDMA/RDMAs to occur successfully.  After speaking with
>>>>> > (several) FAEs at Freescale, we believe there is a
>>>>> > configuration issue that is the cause, but we have yet to
>>>>> > have anyone successfully point to it.
>>>>> >
>>>>> >       Disabling the data cache altogether resolves the issue
>>>>> > entirely, but of course, also completely tanks performance.
>>>>> > As a temporary workaround, I would like to simply mark the
>>>>> > pages (obtained currently via dma_alloc_coherent) involved as
>>>>> > cache-inhibited.  I have attempted to do this via some
>>>>> > snippets remaining in fec.c (va_to_pte, uncache_pte to set
>>>>> > _PAGE_NO_CACHE, flush_tlb_page, then unmap_pte), but this is
>>>>> > almost certainly braindead; va_to_pte is not a part of the
>>>>> > 83xx source, as far as I can tell; 8xx only.
>>>>> >
>>>>> >       A quick pointer in the correct direction for marking
>>>>> > pages as cache-inhibited on a 2.6.27-5 kernel would be
>>>>> > appreciated, or if my approach to a workaround is flawed, a
>>>>> > pointer to the correct way would be great.
>>>>> >
>>>>> >       Ben Menchaca
>>>>> >
>>>>> >
>>>>> > _______________________________________________
>>>>> > Linuxppc-dev mailing list
>>>>> > Linuxppc-dev at ozlabs.org
>>>>> > https://ozlabs.org/mailman/listinfo/linuxppc-dev
>>>>> >
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ozlabs.org/pipermail/linuxppc-dev/attachments/20090308/8c188d55/attachment.htm>