Thank you for your help! That bit resolved all of the RDMA/WDMA coherency issues on the CSB side...except:<br><br>We expose a 1MB region of memory from CSB via a BAR (BAR1, if it matters) to the Host. This region is also not behaving correctly with respect to coherency on SOME hosts; again, disabling our cache makes it work correctly on all hosts. We have set <font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;">PEX_DEVICE_CONTROL in PCI-E
Config Space (0x54) to </font></font></font></font></font><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;">0x2010 (sorry about the endianness below). We thought that CLEARING the no-snoop bit here would indicate that snooping was required for this region...is this a similar issue?<br>
<br>- Ben<br></font></font></font></font></font><br><div class="gmail_quote">On Fri, Mar 6, 2009 at 10:12 AM, Ben Menchaca <span dir="ltr"><<a href="mailto:ben.menchaca@gmail.com">ben.menchaca@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Testing now...it looks like it (almost) works, though! Why does setting no-snoop cause snooping to work? More on the effect on setting that bit in a few minutes...need more testing.<br>
<br>ACR is 0x00030300.<br><font color="#888888"><br>- Ben</font><div><div></div><div class="h5"><br>
<br><div class="gmail_quote">On Fri, Mar 6, 2009 at 12:30 AM, Liu Dave-R63238 <span dir="ltr"><<a href="mailto:DaveLiu@freescale.com" target="_blank">DaveLiu@freescale.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>
<div dir="ltr" align="left"><span><font size="2" color="#0000ff" face="Arial">Did you enable the descriptor bit 3 to have a
try?</font></span></div><br>
<blockquote style="border-left: 2px solid rgb(0, 0, 255); padding-left: 5px; margin-left: 5px; margin-right: 0px;">
<div dir="ltr" align="left" lang="en-us">
<hr>
<font size="2" face="Tahoma"><div><b>From:</b> Ben Menchaca
[mailto:<a href="mailto:ben.menchaca@gmail.com" target="_blank">ben.menchaca@gmail.com</a>] <br></div><b>Sent:</b> Friday, March 06, 2009 2:10
PM<div><div></div><div><br><b>To:</b> Liu Dave-R63238<br><b>Cc:</b>
<a href="mailto:linuxppc-dev@ozlabs.org" target="_blank">linuxppc-dev@ozlabs.org</a><br><b>Subject:</b> Re: 83xx: Marking or Allocating
Pages as Cache-Inhibited<br></div></div></font><br></div><div><div></div><div>
<div></div>I can look at ACR morning...although I can say with a fair amount
of certainty that I have not changed it from the POR value.<br><br>I will try
enabling No Snoop for CSB in the descriptor (bit 3, yes?)...this seems a bit
counterintuitive to me.<br><br>What is the hope regarding these two?
Some combination I am not seeing?<br><br><br>
<div class="gmail_quote">On Thu, Mar 5, 2009 at 11:40 PM, Liu Dave-R63238 <span dir="ltr"><<a href="mailto:DaveLiu@freescale.com" target="_blank">DaveLiu@freescale.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>
<div dir="ltr" align="left"><span><font size="2" color="#0000ff" face="Arial">what is
the value of ACR register?</font></span></div><br>
<blockquote style="border-left: 2px solid rgb(0, 0, 255); padding-left: 5px; margin-left: 5px; margin-right: 0px;">
<div dir="ltr" align="left" lang="en-us">
<hr>
<font size="2" face="Tahoma"><b>From:</b> Ben Menchaca [mailto:<a href="mailto:ben.menchaca@gmail.com" target="_blank">ben.menchaca@gmail.com</a>] <br><b>Sent:</b> Friday, March
06, 2009 1:38 PM<br><b>To:</b> Liu Dave-R63238<br><b>Cc:</b> <a href="mailto:linuxppc-dev@ozlabs.org" target="_blank">linuxppc-dev@ozlabs.org</a><br><b>Subject:</b> Re: 83xx:
Marking or Allocating Pages as Cache-Inhibited<br></font><br></div>
<div>
<div></div>
<div>
<div></div><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;">1. BAT2 in linux is set
to WIMG=0010, and covers all 64M<br>2. PEX_DEVICE_CONTROL in PCI-E
Config Space (0x54): 0x1020<br>3. PEX_xDMA_CTRL is set to
<span></span>0x00000401 at the initiation of the DMA.<br>4. OWAR0 is
set to 0xFFFFF005, so NSNP is 0.<br>5. The DMA descriptor (randomly
chosen when I hit a trigger...just ignore the size...) contains 0002AFF3
at offset 0, so nosnoops are
cleared.</font></font></font></font></font><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"><font style="font-size: 10pt; font-family: Tahoma;"> <br>
<br>Core is
400MHz, and CSB is 133MHz.</font></font></font></font><br><font color="#888888"><br>- Ben<br></font><br>
<div class="gmail_quote">On Thu, Mar 5, 2009 at 11:27 PM, Liu Dave-R63238
<span dir="ltr"><<a href="mailto:DaveLiu@freescale.com" target="_blank">DaveLiu@freescale.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">and
what settings is DMA description bit 3?<br>
<div><br>> -----Original Message-----<br>> From:
linuxppc-dev-bounces+daveliu=<a href="http://freescale.com" target="_blank">freescale.com</a>@<a href="http://ozlabs.org" target="_blank">ozlabs.org</a><br>> [mailto:<a href="mailto:linuxppc-dev-bounces%2Bdaveliu" target="_blank">linuxppc-dev-bounces+daveliu</a>=<a href="http://freescale.com" target="_blank">freescale.com</a>@<a href="http://ozlabs.org" target="_blank">ozlabs.org</a>]<br>
</div>
<div>
<div></div>
<div>> On Behalf Of Liu Dave-R63238<br>> Sent: Friday, March
06, 2009 1:22 PM<br>> To: Ben Menchaca; <a href="mailto:linuxppc-dev@ozlabs.org" target="_blank">linuxppc-dev@ozlabs.org</a><br>> Subject: RE: 83xx:
Marking or Allocating Pages as Cache-Inhibited<br>><br>> Did you
enable the snoop bit at PEX_WDMA_CTRL[SNOOP] and<br>>
PEX_RDMA_CTRL[SNOOP]?<br>><br>> What is the freq settings?
CORE/CSB bus.<br>><br>> Thanks, Dave<br>><br>>
________________________________<br>><br>>
From: linuxppc-dev-bounces+daveliu=<a href="http://freescale.com" target="_blank">freescale.com</a>@<a href="http://ozlabs.org" target="_blank">ozlabs.org</a><br>> [mailto:<a href="mailto:linuxppc-dev-bounces%2Bdaveliu" target="_blank">linuxppc-dev-bounces+daveliu</a>=<a href="http://freescale.com" target="_blank">freescale.com</a>@<a href="http://ozlabs.org" target="_blank">ozlabs.org</a>]<br>
> On
Behalf Of Ben Menchaca<br>> Sent: Friday, March
06, 2009 12:33 PM<br>> To: <a href="mailto:linuxppc-dev@ozlabs.org" target="_blank">linuxppc-dev@ozlabs.org</a><br>>
Subject: 83xx: Marking or Allocating Pages as
Cache-Inhibited<br>><br>><br>> I am
working on a Freescale 8314e design, and the<br>> embedded device is
configured as a PCI-e endpoint running a<br>> 2.6.27-5 kernel.
For context, we have written a kernel<br>> module which, among
other things, uses the RDMA/WDMA engine<br>> in the PCI-e IP block.
On the host side, these DMAs are<br>> coherent. However,
on the embedded side, things are quite a<br>> bit less rosy; we must
manually flush/invalidate cache lines<br>> for WDMA/RDMAs to occur
successfully. After speaking with<br>> (several) FAEs at
Freescale, we believe there is a<br>> configuration issue that is the
cause, but we have yet to<br>> have anyone successfully point to
it.<br>><br>> Disabling the data cache
altogether resolves the issue<br>> entirely, but of course, also
completely tanks performance.<br>> As a temporary workaround, I would
like to simply mark the<br>> pages (obtained currently via
dma_alloc_coherent) involved as<br>> cache-inhibited. I have
attempted to do this via some<br>> snippets remaining in fec.c
(va_to_pte, uncache_pte to set<br>> _PAGE_NO_CACHE, flush_tlb_page,
then unmap_pte), but this is<br>> almost certainly braindead;
va_to_pte is not a part of the<br>> 83xx source, as far as I can
tell; 8xx only.<br>><br>> A quick pointer in
the correct direction for marking<br>> pages as cache-inhibited on a
2.6.27-5 kernel would be<br>> appreciated, or if my approach to a
workaround is flawed, a<br>> pointer to the correct way would be
great.<br>><br>> Ben
Menchaca<br>><br>><br></div></div>>
_______________________________________________<br>> Linuxppc-dev
mailing list<br>> <a href="mailto:Linuxppc-dev@ozlabs.org" target="_blank">Linuxppc-dev@ozlabs.org</a><br>> <a href="https://ozlabs.org/mailman/listinfo/linuxppc-dev" target="_blank">https://ozlabs.org/mailman/listinfo/linuxppc-dev</a><br>
><br>><br></blockquote></div><br></div></div></blockquote></div></blockquote></div><br></div></div></blockquote></div>
</blockquote></div><br>
</div></div></blockquote></div><br>