lpar issue for ZONE_DEVICE p2pmem in 4.14-rc

[Balbir Singh]

On Wed, Oct 18, 2017 at 7:38 AM, Stephen  Bates <sbates at raithlin.com> wrote:
> Hi All
>
> I am hoping someone can help shed some light on an issue I am seeing with my attempt to add p2pmem [1] to the ppc64el kernel. p2pmem is a (currently) out-of-tree patchset that allows one to add device memory with struct page backing into the Linux kernel. It leverages MEMORY_HOTPLUG and ZONE_DEVICE which were added to ppc64 in 4.14 so I thought it would be fun to try it out.
>
> We constructed a patchset based off 4.14-rc1 [1] and build a kernel based off the pseries defconfig and ran this on upstream qemu-system-ppc64. The exact command to run QEMU was:
>
> qemu-system-ppc64 \
>     -machine pseries \
>     -cpu power8 \
>     -smp 1 -m 2048 \
>     -kernel ~/kernel/linux-ppc64el/vmlinux \
>     -append "nvme.use_cmb=24 console=hvc root=/dev/sda rootwait rw" \
>     -serial mon:stdio -drive file=image-ppc64el.img,if=scsi,format=raw,index=0 \
>     -nographic \
>     -drive file=../images/nvme.qcow2,if=none,id=nvme1,snapshot=on \
>     -device nvme,drive=nvme1,serial=nvme1 \
>     -drive file=../images/nvme2.qcow2,if=none,id=nvme2,snapshot=on \
>     -device nvme,drive=nvme2,serial=nvme2,cmb_size_mb=64
>
> This resulted in the following extract from dmesg when registering the p2pmem associated with one of the NVMe SSDs.
>
> [    3.508497] nvme 0000:00:03.0: enabling device (0100 -> 0102)
> [    3.510743] nvme 0000:00:03.0: Using 64-bit direct DMA at offset 800000000000000
> [    3.535706] p2pmem p2pmem0: registered
> [    3.537780] lpar: Attempting to resize HPT to shift 21
> [    3.539251] Unable to resize hash page table to target order 21: -1

I am guessing that the hotplug of ZONE_DEVICE memory was done
incorrectly as it lead to HPT resizing (the system thinking this is
normal memory). Ideally one would expect that the driver would online
ZONE_DEVICE memory and not go through the HOTPLUG path. Are you using
devm_memremap_pages() path to add these pages?

Balbir Singh.