83xx: Marking or Allocating Pages as Cache-Inhibited

Mon Mar 9 13:56:07 EST 2009

Hi Ben,

The second issue. you told me "some hosts" has problem,
and some hosts worked well.

what is the problem-hosts?

The issue seems like the hosts did set the NO SNOOP attribute
bit at TLP.

The PEX_DEVICE_CONTROL is standard PCI configuration space
register, it controls the behavior of the initiator's transaction.
For 8315, it is outbound, not inbound transaction.

Thanks, Dave

________________________________

	From: Ben Menchaca [mailto:ben.menchaca at gmail.com] 
	Sent: Saturday, March 07, 2009 12:30 AM
	To: Liu Dave-R63238
	Cc: linuxppc-dev at ozlabs.org
	Subject: Re: 83xx: Marking or Allocating Pages as
Cache-Inhibited

	Thank you for your help!  That bit resolved all of the RDMA/WDMA
coherency issues on the CSB side...except:

	We expose a 1MB region of memory from CSB via a BAR (BAR1, if it
matters) to the Host.  This region is also not behaving correctly with
respect to coherency on SOME hosts; again, disabling our cache makes it
work correctly on all hosts.  We have set PEX_DEVICE_CONTROL in PCI-E
Config Space (0x54) to 0x2010 (sorry about the endianness below).  We
thought that CLEARING the no-snoop bit here would indicate that snooping
was required for this region...is this a similar issue?

	- Ben

	On Fri, Mar 6, 2009 at 10:12 AM, Ben Menchaca
<ben.menchaca at gmail.com> wrote:

		Testing now...it looks like it (almost) works, though!
Why does setting no-snoop cause snooping to work?  More on the effect on
setting that bit in a few minutes...need more testing.

		ACR is 0x00030300.

		- Ben 

		On Fri, Mar 6, 2009 at 12:30 AM, Liu Dave-R63238
<DaveLiu at freescale.com> wrote:

			Did you enable the descriptor bit 3 to have a
try?

________________________________

				From: Ben Menchaca
[mailto:ben.menchaca at gmail.com] 

				Sent: Friday, March 06, 2009 2:10 PM 

				To: Liu Dave-R63238
				Cc: linuxppc-dev at ozlabs.org
				Subject: Re: 83xx: Marking or Allocating
Pages as Cache-Inhibited

				I can look at ACR morning...although I
can say with a fair amount of certainty that I have not changed it from
the POR value.

				I will try enabling No Snoop for CSB in
the descriptor (bit 3, yes?)...this seems a bit counterintuitive to me.

				What is the hope regarding these two?
Some combination I am not seeing?

				On Thu, Mar 5, 2009 at 11:40 PM, Liu
Dave-R63238 <DaveLiu at freescale.com> wrote:

				what is the value of ACR register?

________________________________

				From: Ben Menchaca
[mailto:ben.menchaca at gmail.com] 
				Sent: Friday, March 06, 2009 1:38 PM
				To: Liu Dave-R63238
				Cc: linuxppc-dev at ozlabs.org
				Subject: Re: 83xx: Marking or Allocating
Pages as Cache-Inhibited

				1.  BAT2 in linux is set to WIMG=0010,
and covers all 64M
				2.  PEX_DEVICE_CONTROL in PCI-E Config
Space (0x54): 0x1020
				3.  PEX_xDMA_CTRL is set to 0x00000401
at the initiation of the DMA.
				4.  OWAR0 is set to 0xFFFFF005, so NSNP
is 0.
				5.  The DMA descriptor (randomly chosen
when I hit a trigger...just ignore the size...) contains 0002AFF3 at
offset 0, so nosnoops are cleared.  

				Core is 400MHz, and CSB is 133MHz.

				- Ben

				On Thu, Mar 5, 2009 at 11:27 PM, Liu
Dave-R63238 <DaveLiu at freescale.com> wrote:

				and what settings  is DMA description
bit 3?

				> -----Original Message-----
				> From:
linuxppc-dev-bounces+daveliu=freescale.com at ozlabs.org
				> [mailto:linuxppc-dev-bounces+daveliu
<mailto:linuxppc-dev-bounces%2Bdaveliu> =freescale.com at ozlabs.org]

				>  On Behalf Of Liu Dave-R63238
				> Sent: Friday, March 06, 2009 1:22 PM
				> To: Ben Menchaca;
linuxppc-dev at ozlabs.org
				> Subject: RE: 83xx: Marking or
Allocating Pages as Cache-Inhibited
				>
				> Did you enable the snoop bit at
PEX_WDMA_CTRL[SNOOP] and
				> PEX_RDMA_CTRL[SNOOP]?
				>
				> What is the freq settings? CORE/CSB
bus.
				>
				> Thanks, Dave
				>
				> ________________________________
				>
				>       From:
linuxppc-dev-bounces+daveliu=freescale.com at ozlabs.org
				> [mailto:linuxppc-dev-bounces+daveliu
<mailto:linuxppc-dev-bounces%2Bdaveliu> =freescale.com at ozlabs.org]
				>  On Behalf Of Ben Menchaca
				>       Sent: Friday, March 06, 2009
12:33 PM
				>       To: linuxppc-dev at ozlabs.org
				>       Subject: 83xx: Marking or
Allocating Pages as Cache-Inhibited
				>
				>
				>       I am working on a Freescale
8314e design, and the
				> embedded device is configured as a
PCI-e endpoint running a
				> 2.6.27-5 kernel.  For context, we have
written a kernel
				> module which, among other things, uses
the RDMA/WDMA engine
				> in the PCI-e IP block.  On the host
side, these DMAs are
				> coherent.  However, on the embedded
side, things are quite a
				> bit less rosy; we must manually
flush/invalidate cache lines
				> for WDMA/RDMAs to occur successfully.
After speaking with
				> (several) FAEs at Freescale, we
believe there is a
				> configuration issue that is the cause,
but we have yet to
				> have anyone successfully point to it.
				>
				>       Disabling the data cache
altogether resolves the issue
				> entirely, but of course, also
completely tanks performance.
				> As a temporary workaround, I would
like to simply mark the
				> pages (obtained currently via
dma_alloc_coherent) involved as
				> cache-inhibited.  I have attempted to
do this via some
				> snippets remaining in fec.c
(va_to_pte, uncache_pte to set
				> _PAGE_NO_CACHE, flush_tlb_page, then
unmap_pte), but this is
				> almost certainly braindead; va_to_pte
is not a part of the
				> 83xx source, as far as I can tell; 8xx
only.
				>
				>       A quick pointer in the correct
direction for marking
				> pages as cache-inhibited on a 2.6.27-5
kernel would be
				> appreciated, or if my approach to a
workaround is flawed, a
				> pointer to the correct way would be
great.
				>
				>       Ben Menchaca
				>
				>

				>
_______________________________________________
				> Linuxppc-dev mailing list
				> Linuxppc-dev at ozlabs.org
				>
https://ozlabs.org/mailman/listinfo/linuxppc-dev
				>
				>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ozlabs.org/pipermail/linuxppc-dev/attachments/20090309/cc7542c4/attachment.htm>