Worst case performance of up()

[Adrian Cox]

First the background: I've been investigating poor performance of a
Firewire capture application, running on a dual-7450 board with a 2.6.17
kernel. The kernel is based on a slightly earlier version of the
mpc7448hpc2 board port, using arch/powerpc, which I've not yet updated
to reflect the changes made when the board support entered the
mainstream kernel. 

The application runs smoothly on a single processor. On the dual
processor machine, the application sometimes suffers a drop in
frame-rate, simultaneous with high CPU usage by the Firewire kernel
thread.

Further investigation reveals that the kernel thread spends most of the
time in one line: up(&fi->complete_sem) in __queue_complete_req() in
drivers/iee1394/raw1394.c.  It seems that whenever the userspace thread
calling raw1394_read() is scheduled on the opposite CPU to the kernel
thread, the kernel thread takes much longer to execute up() - typically
10000 times longer.

Does anybody have any ideas what could make up() take so long in this
circumstance? I'd expect cache transfers to make the operation about 100
times slower, but this looks like repeated cache ping-pong between the
two CPUs.

-- 
Adrian Cox <adrian at humboldt.co.uk>