[PATCH] Document Linux's memory barriers [try #4]

[David Howells]

Linus Torvalds <torvalds at osdl.org> wrote:

> That's not that different from doing
> 
> 	ptr = read a
> 	data = read [ptr]
> 
>   and speculating the result of the first read.

But that would lead to the situation I suggested (q == &b and d == a), not the
one Paul suggested (q == &b and d == old b) because we'd speculate on the old
value of the pointer, and so see it before it's updated, and thus still
pointing to a.

> The cache is fully coherent, but the coherency isn't _ordered_.
> 
> Remember: the smp_wmb() only orders on the _writer_ side. Not on the 
> reader side. The writer may send out the stuff in a particular order, but 
> the reader might see them in a different order because _it_ might queue 
> the bus events internally for its caches (in particular, it could end up 
> delaying updating a particular way in the cache because it's busy).

Ummm... So whilst smp_wmb() commits writes to the mercy of the cache coherency
system in a particular order, the updates can be passed over from one cache to
another and committed to the reader's cache in any order, and can even be
delayed:

	CPU 1		CPU 2		COMMENT
	===============	===============	=======================================
					a == 0, b == 1 and p == &a, q == &a
	b = 2;
	smp_wmb();			Make sure b is changed before p
	<post b=2>
			<queue b=2>
	p = &b;		q = p;
	<post p=&b>
			<queue p=&b>
			d = *q;
			<commit p=&b>
			<post q=p>
			<read *q>	Reads from b before b updated in cache
			<post d=*q>
			<commit b=2>

I presume the Alpha MB instruction forces cache queue completion in addition
to a partial ordering on memory accesses:

	CPU 1		CPU 2		COMMENT
	===============	===============	=======================================
					a == 0, b == 1 and p == &a, q == &a
	b = 2;
	smp_wmb();			Make sure b is changed before p
	<post b=2>
			<queue b=2>
	p = &b;		q = p;
	<post p=&b>
			<queue p=&b>
			smp_read_barrier_depends();
			<commit b=2>
			<commit p=&b>
			d = *q;
			<post q=p>
			<read *q>	Reads new value of b
			<post d=*q>


David