MMIO and gcc re-ordering issue

[Jeremy Higdon]

On Mon, Jun 02, 2008 at 11:56:39AM +0200, Jes Sorensen wrote:
> Jeremy Higdon wrote:
> >We don't actually have that problem on the Altix.  All writes issued
> >by CPU X will be ordered with respect to each other.  But writes by
> >CPU X and CPU Y will not be, unless an mmiowb() is done by the
> >original CPU before the second CPU writes.  I.e.
> >
> >	CPU X	writel
> >	CPU X	writel
> >	CPU X	mmiowb
> >
> >	CPU Y	writel
> >	...
> >
> >Note that this implies some sort of locking.  Also note that if in
> >the above, CPU Y did the mmiowb, that would not work.
> 
> Hmmm,
> 
> Then it's less bad than I thought - my apologies for the confusion.
> 
> Would we be able to use Ben's trick of setting a per cpu flag in
> writel() then and checking that in spin unlock issuing the mmiowb()
> there if needed?


Yes, that should work fine.  You will get more mmiowb's than you
need, since some drivers, such as Fusion, don't need them.  On the
Origins (older SGI MIPS-based Numa), the 'sync' instruction had
the same effect as mmiowb() with respect to mmio write ordering,
and it was issued unconditionally in the spin unlock.  It was
cheaper than mmiowb, however.

If it matters, we could invent and use writel_relaxed() to get
performance back in drivers we care about....

jeremy