<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.2900.3492" name=GENERATOR></HEAD>
<BODY>
<DIV dir=ltr align=left>
<DIV dir=ltr align=left><SPAN class=456321902-09032009><FONT face=Arial
color=#0000ff size=2>Hi Ben,</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=456321902-09032009><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=456321902-09032009><FONT face=Arial
color=#0000ff size=2>The second issue. you told me "some hosts" has
problem,</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=456321902-09032009><FONT face=Arial
color=#0000ff size=2>and some hosts worked well.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=456321902-09032009><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=456321902-09032009><FONT face=Arial
color=#0000ff size=2>what is the problem-hosts?</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=456321902-09032009><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=456321902-09032009><FONT face=Arial
color=#0000ff size=2>The issue seems like the hosts did set the NO SNOOP
attribute</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=456321902-09032009><FONT face=Arial
color=#0000ff size=2>bit at TLP.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=456321902-09032009><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=456321902-09032009><FONT face=Arial
color=#0000ff size=2>The PEX_DEVICE_CONTROL is standard PCI configuration
space</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=456321902-09032009><FONT face=Arial
color=#0000ff size=2>register, it controls the behavior of the initiator's
transaction.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=456321902-09032009><FONT face=Arial
color=#0000ff size=2>For 8315, it is outbound, not inbound
transaction.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=456321902-09032009><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=456321902-09032009><FONT face=Arial
color=#0000ff size=2>Thanks, Dave</FONT></SPAN></DIV></DIV><BR>
<BLOCKQUOTE
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> Ben Menchaca
[mailto:ben.menchaca@gmail.com] <BR><B>Sent:</B> Saturday, March 07, 2009
12:30 AM<BR><B>To:</B> Liu Dave-R63238<BR><B>Cc:</B>
linuxppc-dev@ozlabs.org<BR><B>Subject:</B> Re: 83xx: Marking or Allocating
Pages as Cache-Inhibited<BR></FONT><BR></DIV>
<DIV></DIV>Thank you for your help! That bit resolved all of the
RDMA/WDMA coherency issues on the CSB side...except:<BR><BR>We expose a 1MB
region of memory from CSB via a BAR (BAR1, if it matters) to the Host.
This region is also not behaving correctly with respect to coherency on SOME
hosts; again, disabling our cache makes it work correctly on all hosts.
We have set <FONT style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"><FONT
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"><FONT
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"><FONT
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"><FONT
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma">PEX_DEVICE_CONTROL in PCI-E
Config Space (0x54) to </FONT></FONT></FONT></FONT></FONT><FONT
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"><FONT
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"><FONT
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"><FONT
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"><FONT
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma">0x2010 (sorry about the
endianness below). We thought that CLEARING the no-snoop bit here would
indicate that snooping was required for this region...is this a similar
issue?<BR><BR>- Ben<BR></FONT></FONT></FONT></FONT></FONT><BR>
<DIV class=gmail_quote>On Fri, Mar 6, 2009 at 10:12 AM, Ben Menchaca <SPAN
dir=ltr><<A
href="mailto:ben.menchaca@gmail.com">ben.menchaca@gmail.com</A>></SPAN>
wrote:<BR>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">Testing
now...it looks like it (almost) works, though! Why does setting
no-snoop cause snooping to work? More on the effect on setting that
bit in a few minutes...need more testing.<BR> <BR>ACR is
0x00030300.<BR><FONT color=#888888><BR>- Ben</FONT>
<DIV>
<DIV></DIV>
<DIV class=h5><BR><BR>
<DIV class=gmail_quote>On Fri, Mar 6, 2009 at 12:30 AM, Liu Dave-R63238
<SPAN dir=ltr><<A href="mailto:DaveLiu@freescale.com"
target=_blank>DaveLiu@freescale.com</A>></SPAN> wrote:<BR>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">
<DIV>
<DIV dir=ltr align=left><SPAN><FONT face=Arial color=#0000ff size=2>Did
you enable the descriptor bit 3 to have a try?</FONT></SPAN></DIV><BR>
<BLOCKQUOTE
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: rgb(0,0,255) 2px solid; MARGIN-RIGHT: 0px">
<DIV lang=en-us dir=ltr align=left>
<HR>
<FONT face=Tahoma size=2>
<DIV><B>From:</B> Ben Menchaca [mailto:<A
href="mailto:ben.menchaca@gmail.com"
target=_blank>ben.menchaca@gmail.com</A>] <BR></DIV><B>Sent:</B> Friday,
March 06, 2009 2:10 PM
<DIV>
<DIV></DIV>
<DIV><BR><B>To:</B> Liu Dave-R63238<BR><B>Cc:</B> <A
href="mailto:linuxppc-dev@ozlabs.org"
target=_blank>linuxppc-dev@ozlabs.org</A><BR><B>Subject:</B> Re: 83xx:
Marking or Allocating Pages as
Cache-Inhibited<BR></DIV></DIV></FONT><BR></DIV>
<DIV>
<DIV></DIV>
<DIV>
<DIV></DIV>I can look at ACR morning...although I can say with a fair
amount of certainty that I have not changed it from the POR
value.<BR><BR>I will try enabling No Snoop for CSB in the descriptor
(bit 3, yes?)...this seems a bit counterintuitive to me.<BR><BR>What is
the hope regarding these two? Some combination I am not
seeing?<BR><BR><BR>
<DIV class=gmail_quote>On Thu, Mar 5, 2009 at 11:40 PM, Liu Dave-R63238
<SPAN dir=ltr><<A href="mailto:DaveLiu@freescale.com"
target=_blank>DaveLiu@freescale.com</A>></SPAN> wrote:<BR>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">
<DIV>
<DIV dir=ltr align=left><SPAN><FONT face=Arial color=#0000ff
size=2>what is the value of ACR register?</FONT></SPAN></DIV><BR>
<BLOCKQUOTE
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: rgb(0,0,255) 2px solid; MARGIN-RIGHT: 0px">
<DIV lang=en-us dir=ltr align=left>
<HR>
<FONT face=Tahoma size=2><B>From:</B> Ben Menchaca [mailto:<A
href="mailto:ben.menchaca@gmail.com"
target=_blank>ben.menchaca@gmail.com</A>] <BR><B>Sent:</B> Friday,
March 06, 2009 1:38 PM<BR><B>To:</B> Liu Dave-R63238<BR><B>Cc:</B>
<A href="mailto:linuxppc-dev@ozlabs.org"
target=_blank>linuxppc-dev@ozlabs.org</A><BR><B>Subject:</B> Re:
83xx: Marking or Allocating Pages as
Cache-Inhibited<BR></FONT><BR></DIV>
<DIV>
<DIV></DIV>
<DIV>
<DIV></DIV><FONT style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"><FONT
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"><FONT
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"><FONT
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"><FONT
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma">1. BAT2 in linux
is set to WIMG=0010, and covers all 64M<BR>2.
PEX_DEVICE_CONTROL in PCI-E Config Space (0x54): 0x1020<BR>3.
PEX_xDMA_CTRL is set to <SPAN></SPAN>0x00000401 at the initiation of
the DMA.<BR>4. OWAR0 is set to 0xFFFFF005, so NSNP is
0.<BR>5. The DMA descriptor (randomly chosen when I hit a
trigger...just ignore the size...) contains 0002AFF3 at offset 0, so
nosnoops are cleared.</FONT></FONT></FONT></FONT></FONT><FONT
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"><FONT
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"><FONT
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"><FONT
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"> <BR><BR>Core is
400MHz, and CSB is 133MHz.</FONT></FONT></FONT></FONT><BR><FONT
color=#888888><BR>- Ben<BR></FONT><BR>
<DIV class=gmail_quote>On Thu, Mar 5, 2009 at 11:27 PM, Liu
Dave-R63238 <SPAN dir=ltr><<A href="mailto:DaveLiu@freescale.com"
target=_blank>DaveLiu@freescale.com</A>></SPAN> wrote:<BR>
<BLOCKQUOTE class=gmail_quote
style="PADDING-LEFT: 1ex; MARGIN: 0pt 0pt 0pt 0.8ex; BORDER-LEFT: rgb(204,204,204) 1px solid">and
what settings is DMA description bit 3?<BR>
<DIV><BR>> -----Original Message-----<BR>> From:
linuxppc-dev-bounces+daveliu=<A href="http://freescale.com"
target=_blank>freescale.com</A>@<A href="http://ozlabs.org"
target=_blank>ozlabs.org</A><BR>> [mailto:<A
href="mailto:linuxppc-dev-bounces%2Bdaveliu"
target=_blank>linuxppc-dev-bounces+daveliu</A>=<A
href="http://freescale.com" target=_blank>freescale.com</A>@<A
href="http://ozlabs.org" target=_blank>ozlabs.org</A>]<BR></DIV>
<DIV>
<DIV></DIV>
<DIV>> On Behalf Of Liu Dave-R63238<BR>> Sent: Friday,
March 06, 2009 1:22 PM<BR>> To: Ben Menchaca; <A
href="mailto:linuxppc-dev@ozlabs.org"
target=_blank>linuxppc-dev@ozlabs.org</A><BR>> Subject: RE:
83xx: Marking or Allocating Pages as
Cache-Inhibited<BR>><BR>> Did you enable the snoop bit at
PEX_WDMA_CTRL[SNOOP] and<BR>>
PEX_RDMA_CTRL[SNOOP]?<BR>><BR>> What is the freq settings?
CORE/CSB bus.<BR>><BR>> Thanks, Dave<BR>><BR>>
________________________________<BR>><BR>>
From: linuxppc-dev-bounces+daveliu=<A
href="http://freescale.com" target=_blank>freescale.com</A>@<A
href="http://ozlabs.org" target=_blank>ozlabs.org</A><BR>>
[mailto:<A href="mailto:linuxppc-dev-bounces%2Bdaveliu"
target=_blank>linuxppc-dev-bounces+daveliu</A>=<A
href="http://freescale.com" target=_blank>freescale.com</A>@<A
href="http://ozlabs.org" target=_blank>ozlabs.org</A>]<BR>>
On Behalf Of Ben Menchaca<BR>> Sent:
Friday, March 06, 2009 12:33 PM<BR>> To:
<A href="mailto:linuxppc-dev@ozlabs.org"
target=_blank>linuxppc-dev@ozlabs.org</A><BR>>
Subject: 83xx: Marking or Allocating Pages as
Cache-Inhibited<BR>><BR>><BR>> I am
working on a Freescale 8314e design, and the<BR>> embedded
device is configured as a PCI-e endpoint running a<BR>>
2.6.27-5 kernel. For context, we have written a
kernel<BR>> module which, among other things, uses the
RDMA/WDMA engine<BR>> in the PCI-e IP block. On the host
side, these DMAs are<BR>> coherent. However, on the
embedded side, things are quite a<BR>> bit less rosy; we must
manually flush/invalidate cache lines<BR>> for WDMA/RDMAs to
occur successfully. After speaking with<BR>> (several)
FAEs at Freescale, we believe there is a<BR>> configuration
issue that is the cause, but we have yet to<BR>> have anyone
successfully point to it.<BR>><BR>>
Disabling the data cache altogether resolves the issue<BR>>
entirely, but of course, also completely tanks
performance.<BR>> As a temporary workaround, I would like to
simply mark the<BR>> pages (obtained currently via
dma_alloc_coherent) involved as<BR>> cache-inhibited. I
have attempted to do this via some<BR>> snippets remaining in
fec.c (va_to_pte, uncache_pte to set<BR>> _PAGE_NO_CACHE,
flush_tlb_page, then unmap_pte), but this is<BR>> almost
certainly braindead; va_to_pte is not a part of the<BR>> 83xx
source, as far as I can tell; 8xx only.<BR>><BR>>
A quick pointer in the correct direction for
marking<BR>> pages as cache-inhibited on a 2.6.27-5 kernel
would be<BR>> appreciated, or if my approach to a workaround is
flawed, a<BR>> pointer to the correct way would be
great.<BR>><BR>> Ben
Menchaca<BR>><BR>><BR></DIV></DIV>>
_______________________________________________<BR>>
Linuxppc-dev mailing list<BR>> <A
href="mailto:Linuxppc-dev@ozlabs.org"
target=_blank>Linuxppc-dev@ozlabs.org</A><BR>> <A
href="https://ozlabs.org/mailman/listinfo/linuxppc-dev"
target=_blank>https://ozlabs.org/mailman/listinfo/linuxppc-dev</A><BR>><BR>><BR></BLOCKQUOTE></DIV><BR></DIV></DIV></BLOCKQUOTE></DIV></BLOCKQUOTE></DIV><BR></DIV></DIV></BLOCKQUOTE></DIV></BLOCKQUOTE></DIV><BR></DIV></DIV></BLOCKQUOTE></DIV><BR></BLOCKQUOTE></BODY></HTML>