Feedback requested on switching the exception wrapper used for the PMU interrupt on ppc64

[Olof Johansson]

On May 13, 2008, at 5:05 PM, Corey Ashford wrote:

> Hello,
>
> One of the things I've been working on is porting perfmon2 to ppc64.  
> We've made a fair amount of progress on it, and support is available  
> in libpfm and the perfmon2 kernel patch.
>
> One of the things we had to work around was the "lazy interrupt  
> disabling" mechanism in ppc64 Linux.  The problem was that the PMU  
> exception handler (0xf00) uses the STD_EXCEPTION_PSERIES wrapper,  
> which does not support lazy interrupt disabling.
>
> This is desirable for Oprofile's use of the PMU since its handler is  
> fairly simple and being able to profile interrupt protected code is  
> desirable.  However, it causes problems for perfmon2, since the  
> operations it performs on the thread of its PMU interrupt handler  
> can cause a deadlock condition (it can end up calling spin_lock, for  
> example).
>
> Initially, to work around this, we created special spin_lock_irqsave  
> and spin_unlock_irqrestore macros for perfmon2 which we could  
> override for POWER to define them as functions which do hard  
> disables and restores.
>
> However, not all of the places that we need to disable interrupts  
> were occurring from within the perfmon2 code.  Specifically, getting  
> PMU interrupts in the middle of a schedule() call (where interrupts  
> were expected to be disabled) was causing kernel hangs.
>
> To fix this, I've gone back and removed the special spin_lock macros  
> we defined in perfmon2 and have ifdef'd arch/powerpc/kernel/ 
> head_64.S file as follows:
>
> /*** pSeries interrupt support ***/
>
>        /* moved from 0xf00 */
> + #ifdef CONFIG_PERFMON
> +         MASKABLE_EXCEPTION_PSERIES(., performance_monitor)
> + #else
>        STD_EXCEPTION_PSERIES(., performance_monitor)
> + #endif
>
> /*
> * An interrupt came in while soft-disabled; clear EE in SRR1,
> * clear paca->hard_enabled and return.
>
> The downside of this change is that if someone is using Oprofile and  
> they have enabled perfmon in their kernel, they will not get profile  
> samples that occur in interrupt-protected regions of the kernel.  
> However, they still can by configuring perfmon out of their kernel.
>
> What do you think of this idea?  Is this something that you would  
> object to when perfmon2 does go upstream to LKML?  I think we'd want  
> to add some documentation somewhere that describes this side-effect  
> of enabling perfmon in the ppc64 Linux kernel.  Do you have  
> suggestions on where that should go?  I'm thinking perhaps in both  
> the basic_profiling.txt and perfmon2.txt files in the Documentation  
> subdirectory.

Since you didn't post the perfmon2 code, I'll ask instead of go look:

Do you have a single entry point from performance_monitor into  
perfmon2? If so, it's nicer to check and see if interrupts are soft  
disabled right upon entry (before taking any locks, etc), and if they  
are just return without doing more work.

PMU interrupts generally won't re-arm themselves so you'll obviously  
have to deal with that as well but I'm sure you're already aware of  
that...


-Olof