Problems with THP in v4.5-rc4 on POWER

[Aneesh Kumar K.V]

Paul Mackerras <paulus at ozlabs.org> writes:

> On Sat, Feb 20, 2016 at 12:39:42PM +1100, Paul Mackerras wrote:
>> It seems there's something wrong with our transparent hugepage
>> implementation on POWER server processors as of v4.5-rc4.  I have seen
>> the email thread on "[BUG] random kernel crashes after THP rework on
>> s390 (maybe also on PowerPC and ARM)", but this doesn't seem exactly
>> the same as that (though it may of course be related).
>> 
>> I have been testing v4.5-rc4 with Aneesh's patch "powerpc/mm/hash:
>> Clear the invalid slot information correctly" on top, on a KVM guest
>> with 160 vcpus (threads=8) and 32GB of memory backed by 16MB large
>> pages, running on a POWER8 machine running a 4.4.1 host kernel (20
>> cores * 8 threads, 128GB of RAM).  The guest kernel is compiled with
>> THP enabled and set to "always" (i.e. not "madvise").
>> 
>> On this setup, when doing something like a large kernel compile, I see
>> random segfaults happening (in gcc, cc1, sh, etc.).  I also see bursts
>> of messages like this on the host console:
>> 
>> [50957.570859] Harmless Hypervisor Maintenance interrupt [Recovered]
>> [50957.570864]  Error detail: Processor Recovery done
>> [50957.570869]  HMER: 2040000000000000
>
> When I use a merge of v4.5-rc4 with the fixes branch from the powerpc
> tree, I don't see these messages any more, presumably due to
> "powerpc/mm: Fix Multi hit ERAT cause by recent THP update".  With my
> patch, I still see that it is finding HPTEs to invalidate, but without
> my patch, even though it is presumably leaving HPTEs around, I don't
> see any errors (such as random segfaults) occurring.
>

Yes that patch should fix that. With thp, we use the deposited page
table to store hash pte slot information. On splitting, we were doing a
withdraw of deposited table, without doing the required hash pte flush.
This fix is to call pmdp_huge_split_prepare, which does the required
flush and also clear the _PAGE_USER. Clearning _PAGE_USER ensure that we
don't be inserting any other hash pte entries w.r.t to this THP.

-aneesh