[Cbe-oss-dev] Spider DMA wrongness

[Benjamin Herrenschmidt]

Hi folks !

I've got the iommu code up and running now. However, I noticed some
serious bogosity with spider, I suspect most of it is already well known
though but heh... let's make sure it sticks :-) (Linas, I know you had a
patch for some of this, what is the status ?)

The problem is related to the way the descriptors are managed mostly.
But here we go in no special order. I can try to spend some time fixing
these, though probably not before next week or so.

 - First of all, not our fault, but alloc_etherdev doesn't (yet) have an
alloc_etherdev_node() version thus we end up with a data structure on
the wrong node, pretty bloody annoying. I'm not even talking about skb's
on the wrong node here but purely the netdevice and priv datas

 - Since we allocate our descriptor ring as part of the netdev privates,
they -also- end up in the wrong node. I'd much prefer we use
pci_allocate_consistent() for the rings in fact. Currently, our
implementation for it doesn't do node local allocations but I'm just
fixing it right now :-) Having the descriptors on the wrong node is
probably more of a performances killer than having the packets on the
wrong node (bandwidth accross nodes is ok, latency sucks)

 - The descriptor ring mixes up descriptors themselves and driver
specific data. This is pretty bad. That means bad cache behaviour and
the descriptors as seen by the hardware aren't nicely next to each other
in memory, thus defeating any possible attempt at prefetching the chip
might be doing (most network chips do aggressive prefetching of
descriptors, I don't know for sure about spider). We should really
really really (I insist :-) split the actual descriptors manipulated by
the driver and the device (DMA) from the driver data structures. The
later should be in a separate array, possibly next to the driver priv,
while the actual HW descriptors used for DMA could be allocated
separately with pci_alloc_consistent() and nicely cache aligned, not
sharing any cache line with anything else.

Cheers,
Ben.