Freescale RapidIO controller shortcomings, memory access and DMA support

[Alex Dubov]

Greetings.

I'm currently involved in a project which aims to use a RapidIO fabric as,
essentially, a cluster interconnect. That is, our system contains a
multitude of processing elements, rather than a host + some dumb
peripherals.

Unfortunately, it appears that Freescale's rapidio controller is not well
suited for this kind of use and the situation doesn't seem to get any
better in new chips. Just for the sake of general rant, I shall mention
the missing functionality:

1. Support for sending/receiving raw RIO packets - could be used to emulate
all the necessary features in software, if it was available.

2. Support for indirect descriptors in inbound message queue, akin to the
one in outbound. It will allow not to waste tons of space in the incoming
message buffer, and, even more important, will allow for Source ID based
message dispatch. Frankly, I don't understand what RIO core designers were
thinking - piling of protocol layers on top of RIO messages is not a good
idea (and piling the whole Ethernet + friends protocol stack over there is
even worse).

3. Proper IO TLB would be nice, but even without it there are accepted ways
to manage DMA from peripherals without reverting to overly complex hw
schemes. After all, it won't be a big deal for a controller to support
out-of-memory interrupt event, which would allow managing of DMA transfers
in concurrent fashion, leading to a true zero-copy IO.

4. Related to the previous point, there's no control, nor indication about
which message sizes are actually used for outbound window originated
transactions. Sending 16 bytes of actual phy data for each 4 processor
bytes can be a little bit wasteful (I hope, that there's some buffering
going on, but absence of control is depressing).

--- end of the rant ---

I understand, that Zhang Wei and Li Yang did some work in the direction
of memory mapped RIO access. Therefore, I wonder: what is the final fate
of their submission? Is there some sort of summary, what functionality
had been added and how was it envisaged to be used from user space?

I'm currently working on a fork of rapidio subsystem, which I intend to
bring to a feature completeness as far as possible (given the shortcoming
above) and use it as a cluster interconnect. I don't want to step on other
people's rakes and I would like a good cluster functionality support
merged eventually into the mainline kernel, so any feedback is very
much appreciated.



      __________________________________________________________________________________
See what's on at the movies in your area. Find out now: http://au.movies.yahoo.com/session-times/