[PATCH][RFC] preempt_count corruption across H_CEDE call with CONFIG_PREEMPT on pseries

[Ankita Garg]

Hi Darren,

On Thu, Jul 22, 2010 at 11:24:13AM -0700, Darren Hart wrote:
> 
> With some instrumentation we were able to determine that the
> preempt_count() appears to change across the extended_cede_processor()
> call.  Specifically across the plpar_hcall_norets(H_CEDE) call. On
> PREEMPT_RT we call this with preempt_count=1 and return with
> preempt_count=0xffffffff. On mainline with CONFIG_PREEMPT=y, the value
> is different (0x65) but is still incorrect.

I was trying to reproduce this issue on a 2.6.33.7-rt29 kernel. I could
easily reproduce this on the RT kernel and not the non-RT kernel.
However, I hit it every single time I did a cpu online operation. I also
noticed that the issue persists even when I disable H_CEDE by passing
the "cede_offline=0" kernel commandline parameter. Could you pl confirm
if you observe the same in your setup ? 

However, the issue still remains. Will spend few cycles looking into
this issue.

> 
> Also of interest is that this path
> cpu_idle()->cpu_die()->pseries_mach_cpu_die() to start_secondary()
> enters with a preempt_count=1 if it wasn't corrupted across the hcall.
> The early boot path from _start however appears to call
> start_secondary() with a preempt_count of 0.
> 
> The following patch is most certainly not correct, but it does eliminate
> the situation on mainline 100% of the time (there is still a 25%
> reproduction rate on PREEMPT_RT). Can someone comment on:
> 
> 1) How can the preempt_count() get mangled across the H_CEDE hcall?
> 2) Should we call preempt_enable() in cpu_idle() prior to cpu_die() ?
> 
> Hacked-up-by: Darren Hart <dvhltc at us.ibm.com>
> 
> Index: linux-2.6.33.6/arch/powerpc/platforms/pseries/hotplug-cpu.c
> ===================================================================
> --- linux-2.6.33.6.orig/arch/powerpc/platforms/pseries/hotplug-cpu.c
> +++ linux-2.6.33.6/arch/powerpc/platforms/pseries/hotplug-cpu.c
> @@ -138,6 +138,7 @@ static void pseries_mach_cpu_die(void)
>  			 * Kernel stack will be reset and start_secondary()
>  			 * will be called to continue the online operation.
>  			 */
> +			preempt_count() = 0;
>  			start_secondary_resume();
>  		}
>  	}
> 
> 

-- 
Regards,
Ankita Garg (ankita at in.ibm.com)
Linux Technology Center
IBM India Systems & Technology Labs,
Bangalore, India