Busy waits take much longer in driver code vs. application code

[Matias Garcia]

Not sure if this is arch-dependant. If not, I'll register on the kernel
mailing list.

I'm working with some legacy driver code (an IOCTL) that busy waits
after requesting an operation from an FPGA. It reads a register in a
loop until the FPGA has finished the operation. The operation is
supposed to take about 1ms, but the driver code is returning after 11ms
or more. If I instead run the same busy loop inside the application that
calls the IOCTL, it takes <1ms. Instrumenting the driver loop shows that
the loop is only executed a couple of times about 5ms apart (though
printk may skew this slightly). The application loop is run MUCH more
often.

Platform is P2020 with kernel 2.6.37 running two applications. The one
in question is run with FIFO scheduler at priority 10, the other one is
run with FIFO scheduler at priority 50. All other processes are
regular/default priority.

The only difference in interface between calling the IOCTL and
reading/writing registers through the device file is that read/writes
use copy to/from user while the IOCTL calls io[read|write]32be. No data
needs to go to user-space for this operation.

A couple of questions:

1. Why is the application code running WAY faster than the IOCTL call? I
thought driver code was executed in process context at the same priority
as the calling process.

2. Can you suggest a better way to implement the busy wait in the driver
code if I can get the priorities right? Should I even leave it in there?
I'm doing a lot of reading, but could really use an expert opinion.

Any light shed will be toasted heartily.

Thanks,
Matias Garcia