The only problem host found so far is a newer Asus 970FX motherboard, regardless of OS.<br><br>We are seeing that some time after LTSSM finishes, but long before OS load, our PEX_DEVICE_CONTROL register is changed. On the "working" motherboards, NO_SNOOP is enabled; if I read the spec correctly, this means that TLPs with no-snoop are permitted. On the non-working motherboards, NO_SNOOP is disabled; this is supposed to mean that TLPs with no-snoop are not permitted (again, if I understand correctly). Is it possible that there has been another misinterpretation of this bit? Something regarding the generation of snoops on CSB when it is cleared?<br>
<br>This bit could be a complete red herring; it was one of the few differences in the config space, however, so it was my best guess.<br><br>- Ben<br><br><div class="gmail_quote">On Sun, Mar 8, 2009 at 9:56 PM, Liu Dave-R63238 <span dir="ltr"><<a href="mailto:DaveLiu@freescale.com">DaveLiu@freescale.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div><div class="im">
<div dir="ltr" align="left">
<div dir="ltr" align="left"><span><font color="#0000ff" size="2" face="Arial">Hi Ben,</font></span></div>
<div dir="ltr" align="left"><span><font color="#0000ff" size="2" face="Arial"></font></span> </div>
<div dir="ltr" align="left"><span><font color="#0000ff" size="2" face="Arial">The second issue. you told me "some hosts" has
problem,</font></span></div>
<div dir="ltr" align="left"><span><font color="#0000ff" size="2" face="Arial">and some hosts worked well.</font></span></div>
<div dir="ltr" align="left"><span><font color="#0000ff" size="2" face="Arial"></font></span> </div>
<div dir="ltr" align="left"><span><font color="#0000ff" size="2" face="Arial">what is the problem-hosts?</font></span></div>
<div dir="ltr" align="left"><span><font color="#0000ff" size="2" face="Arial"></font></span> </div>
<div dir="ltr" align="left"><span><font color="#0000ff" size="2" face="Arial">The issue seems like the hosts did set the NO SNOOP
attribute</font></span></div>
<div dir="ltr" align="left"><span><font color="#0000ff" size="2" face="Arial">bit at TLP.</font></span></div>
<div dir="ltr" align="left"><span><font color="#0000ff" size="2" face="Arial"></font></span> </div>
<div dir="ltr" align="left"><span><font color="#0000ff" size="2" face="Arial">The PEX_DEVICE_CONTROL is standard PCI configuration
space</font></span></div>
<div dir="ltr" align="left"><span><font color="#0000ff" size="2" face="Arial">register, it controls the behavior of the initiator's
transaction.</font></span></div>
<div dir="ltr" align="left"><span><font color="#0000ff" size="2" face="Arial">For 8315, it is outbound, not inbound
transaction.</font></span></div>
<div dir="ltr" align="left"><span><font color="#0000ff" size="2" face="Arial"></font></span> </div>
<div dir="ltr" align="left"><span><font color="#0000ff" size="2" face="Arial">Thanks, Dave</font></span></div></div><br>
</div><blockquote style="border-left: 2px solid rgb(0, 0, 255); padding-left: 5px; margin-left: 5px; margin-right: 0px;">
<div dir="ltr" align="left" lang="en-us">
<hr>
<font size="2" face="Tahoma"><div class="im"><b>From:</b> Ben Menchaca
[mailto:<a href="mailto:ben.menchaca@gmail.com" target="_blank">ben.menchaca@gmail.com</a>] <br></div><div class="im"><b>Sent:</b> Saturday, March 07, 2009
12:30 AM<br></div><div><div></div><div class="h5"><b>To:</b> Liu Dave-R63238<br><b>Cc:</b>
<a href="mailto:linuxppc-dev@ozlabs.org" target="_blank">linuxppc-dev@ozlabs.org</a><br><b>Subject:</b> Re: 83xx: Marking or Allocating
Pages as Cache-Inhibited<br></div></div></font><br></div><div><div></div><div class="h5">
<div></div>Thank you for your help! That bit resolved all of the
RDMA/WDMA coherency issues on the CSB side...except:<br><br>We expose a 1MB
region of memory from CSB via a BAR (BAR1, if it matters) to the Host.
This region is also not behaving correctly with respect to coherency on SOME
hosts; again, disabling our cache makes it work correctly on all hosts.
We have set <font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;">PEX_DEVICE_CONTROL in PCI-E
Config Space (0x54) to </font></font></font></font></font><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;">0x2010 (sorry about the
endianness below). We thought that CLEARING the no-snoop bit here would
indicate that snooping was required for this region...is this a similar
issue?<br><br>- Ben<br></font></font></font></font></font><br>
<div class="gmail_quote">On Fri, Mar 6, 2009 at 10:12 AM, Ben Menchaca <span dir="ltr"><<a href="mailto:ben.menchaca@gmail.com" target="_blank">ben.menchaca@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Testing
now...it looks like it (almost) works, though! Why does setting
no-snoop cause snooping to work? More on the effect on setting that
bit in a few minutes...need more testing.<br> <br>ACR is
0x00030300.<br><font color="#888888"><br>- Ben</font>
<div>
<div></div>
<div><br><br>
<div class="gmail_quote">On Fri, Mar 6, 2009 at 12:30 AM, Liu Dave-R63238
<span dir="ltr"><<a href="mailto:DaveLiu@freescale.com" target="_blank">DaveLiu@freescale.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>
<div dir="ltr" align="left"><span><font color="#0000ff" size="2" face="Arial">Did
you enable the descriptor bit 3 to have a try?</font></span></div><br>
<blockquote style="border-left: 2px solid rgb(0, 0, 255); padding-left: 5px; margin-left: 5px; margin-right: 0px;">
<div dir="ltr" align="left" lang="en-us">
<hr>
<font size="2" face="Tahoma">
<div><b>From:</b> Ben Menchaca [mailto:<a href="mailto:ben.menchaca@gmail.com" target="_blank">ben.menchaca@gmail.com</a>] <br></div><b>Sent:</b> Friday,
March 06, 2009 2:10 PM
<div>
<div></div>
<div><br><b>To:</b> Liu Dave-R63238<br><b>Cc:</b> <a href="mailto:linuxppc-dev@ozlabs.org" target="_blank">linuxppc-dev@ozlabs.org</a><br><b>Subject:</b> Re: 83xx:
Marking or Allocating Pages as
Cache-Inhibited<br></div></div></font><br></div>
<div>
<div></div>
<div>
<div></div>I can look at ACR morning...although I can say with a fair
amount of certainty that I have not changed it from the POR
value.<br><br>I will try enabling No Snoop for CSB in the descriptor
(bit 3, yes?)...this seems a bit counterintuitive to me.<br><br>What is
the hope regarding these two? Some combination I am not
seeing?<br><br><br>
<div class="gmail_quote">On Thu, Mar 5, 2009 at 11:40 PM, Liu Dave-R63238
<span dir="ltr"><<a href="mailto:DaveLiu@freescale.com" target="_blank">DaveLiu@freescale.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>
<div dir="ltr" align="left"><span><font color="#0000ff" size="2" face="Arial">what is the value of ACR register?</font></span></div><br>
<blockquote style="border-left: 2px solid rgb(0, 0, 255); padding-left: 5px; margin-left: 5px; margin-right: 0px;">
<div dir="ltr" align="left" lang="en-us">
<hr>
<font size="2" face="Tahoma"><b>From:</b> Ben Menchaca [mailto:<a href="mailto:ben.menchaca@gmail.com" target="_blank">ben.menchaca@gmail.com</a>] <br><b>Sent:</b> Friday,
March 06, 2009 1:38 PM<br><b>To:</b> Liu Dave-R63238<br><b>Cc:</b>
<a href="mailto:linuxppc-dev@ozlabs.org" target="_blank">linuxppc-dev@ozlabs.org</a><br><b>Subject:</b> Re:
83xx: Marking or Allocating Pages as
Cache-Inhibited<br></font><br></div>
<div>
<div></div>
<div>
<div></div><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;">1. BAT2 in linux
is set to WIMG=0010, and covers all 64M<br>2.
PEX_DEVICE_CONTROL in PCI-E Config Space (0x54): 0x1020<br>3.
PEX_xDMA_CTRL is set to <span></span>0x00000401 at the initiation of
the DMA.<br>4. OWAR0 is set to 0xFFFFF005, so NSNP is
0.<br>5. The DMA descriptor (randomly chosen when I hit a
trigger...just ignore the size...) contains 0002AFF3 at offset 0, so
nosnoops are cleared.</font></font></font></font></font><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"> <br>
<br>Core is
400MHz, and CSB is 133MHz.</font></font></font></font><br><font color="#888888"><br>- Ben<br></font><br>
<div class="gmail_quote">On Thu, Mar 5, 2009 at 11:27 PM, Liu
Dave-R63238 <span dir="ltr"><<a href="mailto:DaveLiu@freescale.com" target="_blank">DaveLiu@freescale.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">and
what settings is DMA description bit 3?<br>
<div><br>> -----Original Message-----<br>> From:
linuxppc-dev-bounces+daveliu=<a href="http://freescale.com" target="_blank">freescale.com</a>@<a href="http://ozlabs.org" target="_blank">ozlabs.org</a><br>> [mailto:<a href="mailto:linuxppc-dev-bounces%2Bdaveliu" target="_blank">linuxppc-dev-bounces+daveliu</a>=<a href="http://freescale.com" target="_blank">freescale.com</a>@<a href="http://ozlabs.org" target="_blank">ozlabs.org</a>]<br>
</div>
<div>
<div></div>
<div>> On Behalf Of Liu Dave-R63238<br>> Sent: Friday,
March 06, 2009 1:22 PM<br>> To: Ben Menchaca; <a href="mailto:linuxppc-dev@ozlabs.org" target="_blank">linuxppc-dev@ozlabs.org</a><br>> Subject: RE:
83xx: Marking or Allocating Pages as
Cache-Inhibited<br>><br>> Did you enable the snoop bit at
PEX_WDMA_CTRL[SNOOP] and<br>>
PEX_RDMA_CTRL[SNOOP]?<br>><br>> What is the freq settings?
CORE/CSB bus.<br>><br>> Thanks, Dave<br>><br>>
________________________________<br>><br>>
From: linuxppc-dev-bounces+daveliu=<a href="http://freescale.com" target="_blank">freescale.com</a>@<a href="http://ozlabs.org" target="_blank">ozlabs.org</a><br>>
[mailto:<a href="mailto:linuxppc-dev-bounces%2Bdaveliu" target="_blank">linuxppc-dev-bounces+daveliu</a>=<a href="http://freescale.com" target="_blank">freescale.com</a>@<a href="http://ozlabs.org" target="_blank">ozlabs.org</a>]<br>
>
On Behalf Of Ben Menchaca<br>> Sent:
Friday, March 06, 2009 12:33 PM<br>> To:
<a href="mailto:linuxppc-dev@ozlabs.org" target="_blank">linuxppc-dev@ozlabs.org</a><br>>
Subject: 83xx: Marking or Allocating Pages as
Cache-Inhibited<br>><br>><br>> I am
working on a Freescale 8314e design, and the<br>> embedded
device is configured as a PCI-e endpoint running a<br>>
2.6.27-5 kernel. For context, we have written a
kernel<br>> module which, among other things, uses the
RDMA/WDMA engine<br>> in the PCI-e IP block. On the host
side, these DMAs are<br>> coherent. However, on the
embedded side, things are quite a<br>> bit less rosy; we must
manually flush/invalidate cache lines<br>> for WDMA/RDMAs to
occur successfully. After speaking with<br>> (several)
FAEs at Freescale, we believe there is a<br>> configuration
issue that is the cause, but we have yet to<br>> have anyone
successfully point to it.<br>><br>>
Disabling the data cache altogether resolves the issue<br>>
entirely, but of course, also completely tanks
performance.<br>> As a temporary workaround, I would like to
simply mark the<br>> pages (obtained currently via
dma_alloc_coherent) involved as<br>> cache-inhibited. I
have attempted to do this via some<br>> snippets remaining in
fec.c (va_to_pte, uncache_pte to set<br>> _PAGE_NO_CACHE,
flush_tlb_page, then unmap_pte), but this is<br>> almost
certainly braindead; va_to_pte is not a part of the<br>> 83xx
source, as far as I can tell; 8xx only.<br>><br>>
A quick pointer in the correct direction for
marking<br>> pages as cache-inhibited on a 2.6.27-5 kernel
would be<br>> appreciated, or if my approach to a workaround is
flawed, a<br>> pointer to the correct way would be
great.<br>><br>> Ben
Menchaca<br>><br>><br></div></div>>
_______________________________________________<br>>
Linuxppc-dev mailing list<br>> <a href="mailto:Linuxppc-dev@ozlabs.org" target="_blank">Linuxppc-dev@ozlabs.org</a><br>> <a href="https://ozlabs.org/mailman/listinfo/linuxppc-dev" target="_blank">https://ozlabs.org/mailman/listinfo/linuxppc-dev</a><br>
><br>><br></blockquote></div><br></div></div></blockquote></div></blockquote></div><br></div></div></blockquote></div></blockquote></div><br></div></div></blockquote></div><br></div></div></blockquote></div>
</blockquote></div><br>