[Cbe-oss-dev] Spider DMA wrongness

[Christoph Hellwig]

On Tue, Nov 07, 2006 at 04:43:07PM +1100, Benjamin Herrenschmidt wrote:
> Hi folks !
> 
> I've got the iommu code up and running now. However, I noticed some
> serious bogosity with spider, I suspect most of it is already well known
> though but heh... let's make sure it sticks :-) (Linas, I know you had a
> patch for some of this, what is the status ?)
> 
> The problem is related to the way the descriptors are managed mostly.
> But here we go in no special order. I can try to spend some time fixing
> these, though probably not before next week or so.
> 
>  - First of all, not our fault, but alloc_etherdev doesn't (yet) have an
> alloc_etherdev_node() version thus we end up with a data structure on
> the wrong node, pretty bloody annoying. I'm not even talking about skb's
> on the wrong node here but purely the netdevice and priv datas

I had done this, but davem didn't really like it.  I can bounce the
patch to you and if you have some hard numbers we can try to push it
again.

>  - Since we allocate our descriptor ring as part of the netdev privates,
> they -also- end up in the wrong node. I'd much prefer we use
> pci_allocate_consistent() for the rings in fact. Currently, our
> implementation for it doesn't do node local allocations but I'm just
> fixing it right now :-) Having the descriptors on the wrong node is
> probably more of a performances killer than having the packets on the
> wrong node (bandwidth accross nodes is ok, latency sucks)

And for corretness it should use dma_alloc_coherent.  In fact that one
should be node-local these days, I think I submitted a patch for it.

>  - The descriptor ring mixes up descriptors themselves and driver
> specific data. This is pretty bad. That means bad cache behaviour and
> the descriptors as seen by the hardware aren't nicely next to each other
> in memory, thus defeating any possible attempt at prefetching the chip
> might be doing (most network chips do aggressive prefetching of
> descriptors, I don't know for sure about spider). We should really
> really really (I insist :-) split the actual descriptors manipulated by
> the driver and the device (DMA) from the driver data structures. The
> later should be in a separate array, possibly next to the driver priv,
> while the actual HW descriptors used for DMA could be allocated
> separately with pci_alloc_consistent() and nicely cache aligned, not
> sharing any cache line with anything else.

Yes, agreed.