Performance differences with IRQ_ALL_CPUS

[Kumar Gala]

On Apr 21, 2009, at 5:01 PM, Subodh Nijsure wrote:

>
>
>> From: Kumar Gala
>> Sent: Tuesday, April 21, 2009 1:02 PM
>
>>
>>
>> On Apr 21, 2009, at 10:56 AM, Subodh Nijsure wrote:
>>
>>> Hi,
>>>
>>>
>>> I have board with MPC8572E (dual core PPC). I have one
>> board running
>>> kernel
>>> with IRQ_ALL_CPUS set to Y and another with that option
>> turned off.
>>> Kernel
>>> version #2.6.26
>>>
>>>
>>> With IRQ_ALL_CPUS turned off
>>> ( Here interrupts all go to CPU0 )
>>>
>>> hdparm -tT /dev/hda
>>> /dev/hda:
>>> Timing cached reads:   3048 MB in  2.00 seconds = 1523.79 MB/sec
>>> Timing buffered disk reads:   10 MB in  3.07 seconds =   3.25 MB/sec
>>>
>>> cat /proc/interrupts
>>>          CPU0       CPU1
>>> 18:    1394100          0   OpenPIC   Edge      ide0
>>>
>>> With IRQ_ALL_CPUS turned on I see
>>> (Here interrupts go to CPU0 and CPU1 )
>>>
>>> hdparm -tT /dev/hda
>>>
>>> /dev/hda:
>>> Timing cached reads:   1076 MB in  2.00 seconds = 538.01 MB/sec
>>> Timing buffered disk reads:   10 MB in  3.08 seconds =   3.25 MB/sec
>>>
>>> cat /proc/interrupts
>>>          CPU0       CPU1
>>> 18:      44951      54765   OpenPIC   Edge      ide0
>>>
>>> I "expected" that with IRQ_ALL_CPUS -- interrupt sharing I would be
>>> able to get higher througput but I see things otherway around.
>>>
>>> Would someone care to enlighten me as to why when I set
>> IRQ_ALL_CPUS
>>> disk I/O performance goes down so much?  Under what circumstances
>>> should one then turn on IRQ_ALL_CPUS option on PPC platform?
>>>
>>> Regards,
>>> /Subodh
>>
>> This is probably because you are getting ill defined behavior
>> on 8572.  The 8572 (and any FSL MPIC) doesn't actually
>> support having the PIC distribute interrupts to the different
>> CPUs.  Its technically "illegal" to set more than one
>> destination processor.
>>
>> If you want to try and distribute interrupts than I suggest
>> looking at irqbalance.
>>
>
> Kumar, thanks for the info.
> Okay I will research irqbalance (from http://irqbalance.org/?) and  
> see how
> that goes.
>
> If I want to lock say ethernet interrupts to (core0) CPU0 and Ide  
> interrupts
> to (core1) CPU1 is there a way to fix that using device-tree  
> specification?

No. We've actively avoided trying to convene such affinity in the  
device tree.

However you can do:

cat 1 > /proc/irq/ENET_IRQ_NUM/affinity
cat 2 > /proc/irq/IDE_IRQ/affinity

at boot time.

This will get you affinity binding to a core.

The affinity file is bitmask related to core # (with lsb being core0  
and msb being core31)

- k