[Cbe-oss-dev] [patch 9/9] powerpc/cell: Add DMA_ATTR_STRONG_ORDERING dma attribute and use in IOMMU code

[Arnd Bergmann]

On Tuesday 15 July 2008, Roland Dreier wrote:
> Sorry for the late comments, I missed this when it went by before.
> 
>  > +DMA_ATTR_STRONG_ORDERING
>  > +----------------------
>  > +
>  > +DMA_ATTR_STRONG_ORDERING specifies that previous reads and writes are
>  > +performed in the order in which they're received by the IOMMU; thus
>  > +reads and writes may not pass each other.
> 
> I don't understand what this is trying to say.  What is "previous"
> referring to? What does "received by the IOMMU" mean -- do you mean 
> issued onto the bus by the CPU?

This is all about inbound transfers, i.e. DMAs coming from the I/O bridge
into the CPU, both DMA read and DMA write.

The relevant paragraph in the specification is 
"If the SO bits in the I/O page table entry = ‘11’ and the IOIF S-bit is ‘1’,
this READ or WRITE cannot be placed on the EIB until all previous READs and
WRITEs from this CVCID and IOID have gotten an ACK or NULL type snoop response."

Normally, this is only true for accesses going to the same cache line,
accesses from one device to different cache lines that are issued in order
also send their response in-order (unless you get an I/O exception, which means
you're toast), but can arrive at the I/O location out of order.

My interpretation is that strong ordering basically turns our whole I/O
subsystem into single-issue non-posted accesses (all devices are currently
configured to use the same CVCID and IOID), so we really should not
do that.

> When you say "reads and writes may not 
> pass each other," do you mean just that reads may not pass writes and
> writes may not pass reads, or do you mean that reads also can't pass
> reads and writes can't pass writes?
> 
> Since I don't know exactly what this attribute does, I can't be sure,
> but it seems that making weak ordering the default is dangerous in that
> it breaks drivers that expect usual memory ordering semantics.  Would it
> be safer/better to make strong ordering the default and then add a
> "WEAK_ORDERING" attribute that drivers can use as an optimization?

With all our existing hardware, the I/O bridge overrides the setting
to select weak ordering. The purpose of this patch is to change the
default so that the bridge does not force weak ordering any more and
some drivers are free to set strong ordering without impacting performance
on the other drivers.

Strong ordering is only active when both the bridge and the IOMMU enable
it, but for correctly written drivers, this only results in a slowdown.

	Arnd <><