[Cbe-oss-dev] [RFC 4/9] AXON - Ethernet over PCI-E driver

[Benjamin Herrenschmidt]

> I thought that was about what my example tries to do, but you seem
> to ignore the problem that you can't have real shared memory here,
> only DMA transfers.

You can have real shared memory (as long as you move the PIM around),
though that's a non issue. DMA transfers and shared memory are
equivalent. Let's say shared host memory to simplify the guest doing
everything with DMA.

> In the data structure I laid out (you'd have one per direction),
> there are distinct variables that are written only by the emitter
> or by the receiver and only read by the other side. Of course,
> these should stay in separate cache lines in each coherency
> domain.

Yup.

> I guess one point you made that could simplify the scheme is that
> the message area should not be separated per direction but depending
> on who is writing into it. If you only do DMA reads and write into
> local buffers, that should further simplify the model.

Yup.

> > That works fine for lock-less and almost barrier-less NAPI poll(). In
> > addition, you can add a mecanism to trigger interrupts (based on
> > threshold, or a "I want an IRQ" bit somewhere or whatever), in which
> > case the ISR needs to perform an MMIO read on the host end to flush
> > store buffers, and then schedules a NAPI poll.
> 
> right, except that we don't have an MMIO read here at all, it's always
> a DMA transfer between the two memory domains.

Not exactly. The Host can do an MMIO read from some random place on Axon
to guarantee PCIe write buffers are flushed when getting an irq. It
doesn't have to in that model, but that means possible "delay" of a
packet rx, though that's a completely non-issue if we use MSIs.

Ben.