[Cbe-oss-dev] Spider DMA wrongness

[Benjamin Herrenschmidt]

> >  - First of all, not our fault, but alloc_etherdev doesn't (yet) have an
> > alloc_etherdev_node() version thus we end up with a data structure on
> > the wrong node, pretty bloody annoying. I'm not even talking about skb's
> > on the wrong node here but purely the netdevice and priv datas
> 
> I had done this, but davem didn't really like it.  I can bounce the
> patch to you and if you have some hard numbers we can try to push it
> again.

Not really since PCI will migrate the thread doing the probe to the
device's node before calling probe() it should still get allocated on
the proper node unless I've missed something.

> >  - Since we allocate our descriptor ring as part of the netdev privates,
> > they -also- end up in the wrong node. I'd much prefer we use
> > pci_allocate_consistent() for the rings in fact. Currently, our
> > implementation for it doesn't do node local allocations but I'm just
> > fixing it right now :-) Having the descriptors on the wrong node is
> > probably more of a performances killer than having the packets on the
> > wrong node (bandwidth accross nodes is ok, latency sucks)
> 
> And for corretness it should use dma_alloc_coherent.  In fact that one
> should be node-local these days, I think I submitted a patch for it.

It was not when you have no iommu enabled (when using the PCI direct DMA
ops), I fixed that in one of the patches I posted today. Since I now use
the direct ops on cell, it makes sense. Yes, we should use
dma_alloc_coherent (or pci_alloc_consistent which is just a wrapper for
pci_dev on the former).

> >  - The descriptor ring mixes up descriptors themselves and driver
> > specific data. This is pretty bad. That means bad cache behaviour and
> > the descriptors as seen by the hardware aren't nicely next to each other
> > in memory, thus defeating any possible attempt at prefetching the chip
> > might be doing (most network chips do aggressive prefetching of
> > descriptors, I don't know for sure about spider). We should really
> > really really (I insist :-) split the actual descriptors manipulated by
> > the driver and the device (DMA) from the driver data structures. The
> > later should be in a separate array, possibly next to the driver priv,
> > while the actual HW descriptors used for DMA could be allocated
> > separately with pci_alloc_consistent() and nicely cache aligned, not
> > sharing any cache line with anything else.
> 
> Yes, agreed.

Ben.