[PATCH] powerpc/mm: Fix RECLAIM_DISTANCE

[Gavin Shan]

On Mon, Jan 30, 2017 at 12:02:40PM +1100, Anton Blanchard wrote:
>> Anton suggested that NUMA distances in powerpc mattered and hurted
>> performance without this setting. We need to validate to see if this
>> is still true. A simple way to start would be benchmarking
>
>The original issue was that we never reclaimed local clean pagecache.
>
>I just tried all settings for /proc/sys/vm/zone_reclaim_mode and none
>of them caused me to reclaim local clean pagecache! We are very broken.
>
>I would think we have test cases for this, but here is a dumb one.
>First something to consume memory:
>
># cat alloc.c
>
>#include <stdlib.h>
>#include <unistd.h>
>#include <string.h>
>#include <assert.h>
>
>int main(int argc, char *argv[])
>{
>	void *p;
>
>	unsigned long size;
>
>	size = strtoul(argv[1], NULL, 0);
>
>	p = malloc(size);
>	assert(p);
>	memset(p, 0, size);
>	printf("%p\n", p);
>
>	sleep(3600);
>
>	return 0;
>}
>
>Now create a file to consume pagecache. My nodes have 32GB each, so
>I create 16GB, enough to consume half of the node:
>
>dd if=/dev/zero of=/tmp/file bs=1G count=16
>
>Clear out our pagecache:
>
>sync
>echo 3 > /proc/sys/vm/drop_caches
>
>Bring it in on node 0:
>
>taskset -c 0 cat /tmp/file > /dev/null
>
>Consume 24GB of memory on node 0:
>
>taskset -c 0 ./alloc 25769803776
>
>In all zone reclaim modes, the pagecache never gets reclaimed:
>
># grep FilePages /sys/devices/system/node/node0/meminfo
>
>Node 0 FilePages:      16757376 kB
>
>And our alloc process shows lots of off node memory used:
>
>3ff9a4630000 default anon=393217 dirty=393217 N0=112474 N1=220490 N16=60253 kernelpagesize_kB=64
>
>Clearly nothing is working. Gavin, if your patch fixes this we should
>get it into stable too.
>

Anton, thanks for the detailed test case. I tried what you suggested
on the box that has only one node. The memory capacity is 16GB. So
the parameters I used are different from what you had. First of all,
I observed same behaviour that the pagecache can't be reclaimed when
allocating memory for heap. With the patch applied, the pagecache
can be dropped for page reclaim and more details are showed as below
Everything looks good. I'll put your testcase, its result and stable tag
to next revision.

   root at palm8:/home/gavin# grep FilePages /sys/devices/system/node/node0/meminfo
   Node 0 FilePages:        142400 kB

   root at palm8:/home/gavin# sync
   root at palm8:/home/gavin# echo 3 > /proc/sys/vm/drop_caches

   root at palm8:/home/gavin# grep FilePages /sys/devices/system/node/node0/meminfo
   Node 0 FilePages:         62848 kB

   root at palm8:/home/gavin# du -sh file
   8.1G    file
   root at palm8:/home/gavin# cat file > /dev/null
   root at palm8:/home/gavin# grep FilePages /sys/devices/system/node/node0/meminfo
   Node 0 FilePages:       8448000 kB

   root at palm8:/home/gavin# ./alloc 17179869184
   root at palm8:/home/gavin# grep FilePages /sys/devices/system/node/node0/meminfo
   Node 0 FilePages:        387584 kB

Thanks,
Gavin