Memory barriers and spin_unlock safety

[Linus Torvalds]

On Sat, 4 Mar 2006, Paul Mackerras wrote:
> 
> > If so, a simple write barrier should be sufficient. That's exactly what 
> > the x86 write barriers do too, ie stores to magic IO space are _not_ 
> > ordered wrt a normal [smp_]wmb() (or, as per how this thread started, a 
> > spin_unlock()) at all.
> 
> By magic IO space, do you mean just any old memory-mapped device
> register in a PCI device, or do you mean something else?

Any old memory-mapped device that has been marked as write-combining in 
the MTRR's or page tables.

So the rules from the PC side (and like it or not, they end up being 
what all the drivers are tested with) are:

 - regular stores are ordered by write barriers
 - PIO stores are always synchronous
 - MMIO stores are ordered by IO semantics
	- PCI ordering must be honored:
	  * write combining is only allowed on PCI memory resources
	    that are marked prefetchable. If your host bridge does write 
	    combining in general, it's not a "host bridge", it's a "host 
	    disaster".
	  * for others, writes can always be posted, but they cannot
	    be re-ordered wrt either reads or writes to that device
	    (ie a read will always be fully synchronizing)
	- io_wmb must be honored

In addition, it will help a hell of a lot if you follow the PC notion of 
"per-region extra rules", ie you'd default to the non-prefetchable 
behaviour even for areas that are prefetchable from a PCI standpoint, but 
allow some way to relax the ordering rules in various ways.

PC's use MTRR's or page table hints for this, but it's actually perfectly 
possible to do it by virtual address (ie decide on "ioremap()" time by 
looking at some bits that you've saved away to remap it to a certain 
virtual address range, and then use the virtual address as a hint for 
readl/writel whether you need to serialize or not).

On x86, we already use the "virtual address" trick to distinguish between 
PIO and MMIO for the newer ioread/iowrite interface (the older 
inb/outb/readb/writeb interfaces obviously don't need that, since the IO 
space is statically encoded in the function call itself).

The reason I mention the MTRR emulation is again just purely compatibility 
with drivers that get 99.9% of all the testing on a PC platform.

		Linus