[PATCH] Document Linux's memory barriers [try #4]

[Paul Mackerras]

David Howells writes:

> Paul Mackerras <paulus at samba.org> wrote:
> > There shouldn't be any problem here, because readw/writew _must_
> > ensure that the device accesses are serialized.
> 
> No. That depends on the properties of the memory window readw/writew write
> through, the properties of the CPU wrt memory accesses, and what explicit
> barriers at interpolated inside readw/writew themselves.

The properties of the memory window are certainly relevant.  For a
non-prefetchable PCI MMIO region, the readw/writew must ensure that
the accesses are serialized w.r.t. each other, although not
necessarily serialized with accesses to normal memory.  That is a
requirement that the driver writer can rely on, and the implementor of
readw/writew must ensure is met, taking into account the properties of
the CPU (presumably by putting explicit barriers inside readw/write).

For prefetchable regions, or if the cookie used with readw/writew has
been obtained by something other than the normal ioremap{,_nocache},
then it's more of an open question.

> > Of course, on an SMP system it would be quite possible for the
> > interrupt to be taken on another CPU, and in that case disabling
> > interrupts (I assume that by "DISABLE IRQ" you mean
> > local_irq_disable() or some such)
> 
> Yes. There are quite a few different ways to disable interrupts.

I think it wasn't clear to me which of the following you meant:

(a) Telling this CPU not to take any interrupts (e.g. local_irq_disable())
(b) Telling the interrupt controller not to allow interrupts from that
    device (e.g. disable_irq(irq_num))
(c) Telling the device not to generate interrupts in some
    device-specific fashion

They all have different characteristics w.r.t. timing and
synchronization, so I think it's important to be clear which one you
mean.  For example, if it's (c), then after doing a writel (or
whatever) to the device, you then need at least to do a readl to make
sure the write has got to the device, and even then there might be an
interrupt signal still wending its way through the interrupt
controller etc., which might arrive after the readl has finished.

I think you meant (a), but DISABLE_IRQ actually sounds more like (b).

Paul.