[PATCH] Document Linux's memory barriers [try #2]

[Alan Cox]

On Wed, Mar 08, 2006 at 11:26:41AM -0800, Linus Torvalds wrote:
> Actually, since the different NUMA things may have different paths to the 
> PCI thing, I don't think even the mmiowb() will really help. It has 
> nothing to serialize _with_.

It serializes to the bridge. On the Altix for example this is done by reading
a local status register with the pending write count in it and waiting until
the chip reports the write has propogated across the fabric. At that point it
has hit the bridge and the usual PCI posting applies, but the PCI ordering
rule will also apply so the write won't be passed by another write issued
after the spinlock is then dropped.

> The IO might be posted somewhere on a PCI bridge, and and depending on the 
> posting rules, the mmiowb() just isn't relevant for IO coming through 
> another path.

Yes. mmiowb only serializes to the bridge. Thats how it is defined in the
documentation. Thats enough to sort out things like the example with locks,
but where a read from the device would be overkill.

> general, or even very commonly. The undeniable fact is that "big NUMA" 
> machines need to validate the drivers they use separately. The fact that 
> it works on a normal PC - and that it's been tested to death there - does 
> not guarantee much anything.

mmiowb comes about from the Altix folks strangley enough.

> The good news, of course, is that you don't use that kind of "big NUMA" 
> system the same way you'd use a regular desktop SMP. You don't plug in 
> random devices into it and just expect them to work. I'd hope ;)

Various core drivers like tg3 use mmiowb()

Alan