[PATCH] Document Linux's memory barriers [try #2]

[Jesse Barnes]

On Wednesday, March 8, 2006 4:35 pm, Paul Mackerras wrote:
> David Howells writes:
> > On NUMA PowerPC, should mmiowb() be a SYNC or an EIEIO instruction
> > then? Those do inter-component synchronisation.
>
> We actually have quite heavy synchronization in read*/write* on PPC,
> and mmiowb can safely be a no-op.  It would be nice to be able to have
> lighter-weight synchronization, but I'm sure we would see lots of
> subtle driver bugs cropping up if we did.  write* do a full memory
> barrier (sync) after the store, and read* explicitly wait for the data
> to come back before.
>
> If you ask me, the need for mmiowb on some platforms merely shows that
> those platforms' implementations of spinlocks and read*/write* are
> buggy...

Or maybe they just wanted to keep them fast.  I don't know why you 
compromised so much in your implementation of read/write and 
lock/unlock, but given how expensive synchronization is, I'd think it 
would be better in the long run to make the barrier types explicit (or 
at least a subset of them) to maximize performance.  The rules for using 
the barriers really aren't that bad... for mmiowb() you basically want 
to do it before an unlock in any critical section where you've done PIO 
writes.  

Of course, that doesn't mean there isn't confusion about existing 
barriers.  There was a long thread a few years ago (Jes worked it all 
out, iirc) regarding some subtle memory ordering bugs in the tty layer 
that ended up being due to ia64's very weak spin_unlock ordering 
guarantees (one way memory barrier only), but I think that's mainly an 
artifact of how ill defined the semantics of the various arch specific 
routines are in some cases.

That's why I suggested in an earlier thread that you enumerate all the 
memory ordering combinations on ppc and see if we can't define them all.  
Then David can roll the implicit ones up into his document, or we can 
add the appropriate new operations to the kernel.  Really getting 
barriers right shouldn't be much harder than getting DMA mapping right, 
from a driver writers POV (though people often get that wrong I guess).

Jesse