[Cbe-oss-dev] elaboration of Axon driver?

[Arnd Bergmann]

On Thursday 11 December 2008, Marcus G. Daniels wrote:

> I've been using the Axon driver on Roadrunner IBM triblades and I'm 
> wondering if any thought has been given to supporting DAPL?   In 
> particular, I'm interested in marshaling function calls with complex 
> types as arguments, perhaps with RDMA/RPC.  Since most of the numerics 
> work on Roadrunner is done on the SPUs of the Cell processors, and is 
> very specialized for performance, it's appealing to me keep the Opteron 
> side of the code relatively abstract.  An Axon implementation of MPI 
> would be another option, although the use case I have in mind is 
> factoring monolithic codes to run on multiple tiers of the system, with 
> MPI on the outside (across the cluster).   Also, simple message passing 
> can be done reasonably well with DaCS anyway.  I'd like to be able to 
> break up codes such that some pieces could be RPCed to the PPU of the 
> Cell. 
> 
> Wondering if anyone has thought about these issues.

My current plan is to retire the current driver on the long run, and
move to the standard ib-verbs interface with a new device driver
implementation that also supports other systems than just the triblade.
You can see my presentation slides about my ideas at 
http://userweb.kernel.org/%7Earnd/papers/plumbers08/plumbers-slides.pdf .

IFAICT, DAPL is basically dead for new development, but there is already
experimental DaCS code on top of ib-verbs. The ib-verbs interface in turn
is supported all current infiniband and ethernet/iWarp implementations,
and it can be used in all current MPI libraries.

When we have a device driver library that supports ibverbs on top of
generic shared-memory interconnects (PCI, PCIe, IOIF, RapidIO, ...),
we can use the dma-engine interface in Linux in order to speed up
transfers on the Axon implementation. The same driver layer will also
provide a virtio abstraction, so that we can use virtio high-level
drivers (e.g. virtio-net, virtio-block, 9p file system, ...) directly
on the interconnect, potentially across larger switched fabrics than
just the point-to-point connections in Triblade.

	Arnd <><