[PATCH v02] powerpc/pseries: Check for ceded CPU's during LPAR migration

[Michael Ellerman]

Tyrel Datwyler <tyreld at linux.vnet.ibm.com> writes:
> On 01/31/2019 02:21 PM, Tyrel Datwyler wrote:
>> On 01/31/2019 01:53 PM, Michael Bringmann wrote:
>>> On 1/30/19 11:38 PM, Michael Ellerman wrote:
>>>> Michael Bringmann <mwb at linux.vnet.ibm.com> writes:
>>>>> This patch is to check for cede'ed CPUs during LPM.  Some extreme
>>>>> tests encountered a problem ehere Linux has put some threads to
>>>>> sleep (possibly to save energy or something), LPM was attempted,
>>>>> and the Linux kernel didn't awaken the sleeping threads, but issued
>>>>> the H_JOIN for the active threads.  Since the sleeping threads
>>>>> are not awake, they can not issue the expected H_JOIN, and the
>>>>> partition would never suspend.  This patch wakes the sleeping
>>>>> threads back up.
>>>>
>>>> I'm don't think this is the right solution.
>>>>
>>>> Just after your for loop we do an on_each_cpu() call, which sends an IPI
>>>> to every CPU, and that should wake all CPUs up from CEDE.
>>>>
>>>> If that's not happening then there is a bug somewhere, and we need to
>>>> work out where.
>>>
>>> Let me explain the scenario of the LPM case that Pete Heyrman found, and
>>> that Nathan F. was working upon, previously.
>>>
>>> In the scenario, the partition has 5 dedicated processors each with 8 threads
>>> running.
>> 
>> Do we CEDE processors when running dedicated? I thought H_CEDE was part of the
>> Shared Processor LPAR option.
>
> Looks like the cpuidle-pseries driver uses CEDE with dedicated processors as
> long as firmware supports SPLPAR option.
>
>> 
>>>
>>> From the PHYP data we can see that on VP 0, threads 3, 4, 5, 6 and 7 issued
>>> a H_CEDE requesting to save energy by putting the requesting thread into
>>> sleep mode.  In this state, the thread will only be awakened by H_PROD from
>>> another running thread or from an external user action (power off, reboot
>>> and such).  Timers and external interrupts are disabled in this mode.
>> 
>> Not according to PAPR. A CEDE'd processor should awaken if signaled by external
>> interrupt such as decrementer or IPI as well.
>
> This statement should still apply though. From PAPR:
>
> 14.11.3.3 H_CEDE
> The architectural intent of this hcall() is to have the virtual processor, which
> has no useful work to do, enter a wait state ceding its processor capacity to
> other virtual processors until some useful work appears, signaled either through
> an interrupt or a prod hcall(). To help the caller reduce race conditions, this
> call may be made with interrupts disabled but the semantics of the hcall()
> enable the virtual processor’s interrupts so that it may always receive wake up
> interrupt signals.

Thanks for digging that out of PAPR.

H_CEDE must respond to IPIs, we have no logic to H_PROD CPUs that are
idle in order to wake them up.

There must be something else going on here.

cheers