NUMA memory block size

Olof Johansson olof at austin.ibm.com
Sat Apr 3 13:20:48 EST 2004


Hi,

Some of our new hardware has an LMB size of 16MB, which results in the
memory device nodes covering only 16MB each. Right now the numa code will
round up and down each device node to start at MEMORY_INCREMENT
boundaries, which is 256MB (old LMB size).

If the break between domains still happens on 256MB boundaries, things
will go OK since the overlapped entries will just be skipped, but if the
break is not on an even 256MB boundary bad things happen:

...
memory region 72000000 to 73000000 maps to domain 0
memory region 73000000 to 74000000 maps to domain 1
...

This results in one 256MB region being added to both domains, since the
0x74000000 will be rounded down (and 0x70000000 is rounded up)

Below patch fixes this by setting a 16MB MEMORY_INCREMENT instead. It sure
helps on systems with 16MB LMB, but I don't have enough detail knowledge
of the numa aspects to tell if it might break something else.

Anyone else care to share wisdom? Anton? Dave?

If there are no significant problems caused by this I'd like to feed this
upstream soon.


Thanks,

Olof

Olof Johansson                                        Office: 4F005/905
Linux on Power Development                            IBM Systems Group
Email: olof at austin.ibm.com                          Phone: 512-838-9858
All opinions are my own and not those of IBM


===== include/asm-ppc64/mmzone.h 1.18 vs edited =====
--- 1.18/include/asm-ppc64/mmzone.h	Fri Mar 12 21:18:15 2004
+++ edited/include/asm-ppc64/mmzone.h	Fri Apr  2 20:38:13 2004
@@ -25,7 +25,7 @@

 #define MAX_MEMORY (1UL << 41)
 /* 256MB regions */
-#define MEMORY_INCREMENT_SHIFT 28
+#define MEMORY_INCREMENT_SHIFT 24
 #define MEMORY_INCREMENT (1UL << MEMORY_INCREMENT_SHIFT)

 /* NUMA debugging, will not work on a DLPAR machine */


** Sent via the linuxppc64-dev mail list. See http://lists.linuxppc.org/





More information about the Linuxppc64-dev mailing list