Performance differences with IRQ_ALL_CPUS

[Kumar Gala]

On Apr 21, 2009, at 10:56 AM, Subodh Nijsure wrote:

> Hi,
>
>
> I have board with MPC8572E (dual core PPC). I have one board running  
> kernel
> with IRQ_ALL_CPUS set to Y and another with that option turned off.   
> Kernel
> version #2.6.26
>
>
> With IRQ_ALL_CPUS turned off
> ( Here interrupts all go to CPU0 )
>
> hdparm -tT /dev/hda
> /dev/hda:
> Timing cached reads:   3048 MB in  2.00 seconds = 1523.79 MB/sec
> Timing buffered disk reads:   10 MB in  3.07 seconds =   3.25 MB/sec
>
> cat /proc/interrupts
>           CPU0       CPU1
> 18:    1394100          0   OpenPIC   Edge      ide0
>
> With IRQ_ALL_CPUS turned on I see
> (Here interrupts go to CPU0 and CPU1 )
>
> hdparm -tT /dev/hda
>
> /dev/hda:
> Timing cached reads:   1076 MB in  2.00 seconds = 538.01 MB/sec
> Timing buffered disk reads:   10 MB in  3.08 seconds =   3.25 MB/sec
>
> cat /proc/interrupts
>           CPU0       CPU1
> 18:      44951      54765   OpenPIC   Edge      ide0
>
> I "expected" that with IRQ_ALL_CPUS -- interrupt sharing I would be  
> able to
> get higher througput but I see things otherway around.
>
> Would someone care to enlighten me as to why when I set IRQ_ALL_CPUS  
> disk
> I/O performance goes down so much?  Under what circumstances should  
> one then
> turn on IRQ_ALL_CPUS option on PPC platform?
>
> Regards,
> /Subodh

This is probably because you are getting ill defined behavior on  
8572.  The 8572 (and any FSL MPIC) doesn't actually support having the  
PIC distribute interrupts to the different CPUs.  Its technically  
"illegal" to set more than one destination processor.

If you want to try and distribute interrupts than I suggest looking at  
irqbalance.

I'll look at a patch to deal with IRQ_ALL_CPUS to not setup the insane  
HW scenario.

- k