[PATCH] powerpc: Set a smaller value for RECLAIM_DISTANCE to enable zone reclaim

[Anton Blanchard]

Hi Mel,

> You're pretty much on the button here. Only one thread at a time enters
> zone_reclaim. The others back off and try the next zone in the zonelist
> instead. I'm not sure what the original intention was but most likely it
> was to prevent too many parallel reclaimers in the same zone potentially
> dumping out way more data than necessary.
> 
> > I'm not sure if there is an easy way to fix this without penalising other
> > workloads though.
> > 
> 
> You could experiment with waiting on the bit if the GFP flags allowi it? The
> expectation would be that the reclaim operation does not take long. Wait
> on the bit, if you are making the forward progress, recheck the
> watermarks before continueing.

Thanks to you and Christoph for some suggestions to try. Attached is a
chart showing the results of the following tests:


baseline.txt
The current ppc64 default of zone_reclaim_mode = 0. As expected we see
no change in remote node memory usage even after 10 iterations.

zone_reclaim_mode.txt
Now we set zone_reclaim_mode = 1. On each iteration we continue to improve,
but even after 10 runs of stream we have > 10% remote node memory usage.

reclaim_4096_pages.txt
Instead of reclaiming 32 pages at a time, we try for a much larger batch
of 4096. The slope is much steeper but it still takes around 6 iterations
to get almost all local node memory.

wait_on_busy_flag.txt
Here we busy wait if the ZONE_RECLAIM_LOCKED flag is set. As you suggest
we would need to check the GFP flags etc, but so far it looks the most
promising. We only get a few percent of remote node memory on the first
iteration and get all local node by the second.


Perhaps a combination of larger batch size and waiting on the busy
flag is the way to go?

Anton
-------------- next part --------------
A non-text attachment was scrubbed...
Name: stream_test:_percentage_off_node_memory.png
Type: image/png
Size: 34767 bytes
Desc: not available
URL: <http://lists.ozlabs.org/pipermail/linuxppc-dev/attachments/20100223/8e2f4db6/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: reclaim_4096_pages.patch
Type: text/x-diff
Size: 376 bytes
Desc: not available
URL: <http://lists.ozlabs.org/pipermail/linuxppc-dev/attachments/20100223/8e2f4db6/attachment-0002.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: wait_on_ZONE_RECLAIM_LOCKED.patch
Type: text/x-diff
Size: 482 bytes
Desc: not available
URL: <http://lists.ozlabs.org/pipermail/linuxppc-dev/attachments/20100223/8e2f4db6/attachment-0003.patch>