ppc64 NUMA code needs an IQ injection

[Sonny Rao]

On Fri, Nov 12, 2004 at 01:00:38PM +1100, Anton Blanchard wrote:
> 
> Hi,
> 
> I was shown a box containing 128GB RAM of which only 32GB was visible
> inside Linux. The dmesg shows:
> 
> memory region 0 to 10000000 maps to domain 0
> memory region 10000000 to 20000000 maps to domain 0
> memory region 20000000 to 30000000 maps to domain 1
> memory region 30000000 to 40000000 maps to domain 2
> Hole in node, disabling region start 40000000 length 10000000
> ...
> Hole in node, disabling region start 1800000000 length 10000000
> memory region 1810000000 to 1820000000 maps to domain 3
> 
> The device tree highlights the strange behaviour. The NUMA domain is the
> last number:
> 
> ./memory at 0/ibm,associativity
>                  00000003 00000000 00000000 00000000
> ./memory at 10000000/ibm,associativity
>                  00000003 00000000 00000000 00000000
> ./memory at 20000000/ibm,associativity
>                  00000003 00000000 00000000 00000001
> ./memory at 30000000/ibm,associativity
>                  00000003 00000000 00000000 00000002
> ./memory at 40000000/ibm,associativity
>                  00000003 00000000 00000000 00000000
> ./memory at 50000000/ibm,associativity
>                  00000003 00000000 00000000 00000000
> 
> Firmware thinks there are 2 256MB regions on node 1 and node 2. This
> turned out to be a firmware bug, however its a bit rude to disable 3/4
> of the machines memory because of it.
> 
> Our numa code expects each node to be contiguous and to appear in order,
> and gets upset with the above layout.

I've seen this on a Power5 machine as well.  It was an 8-way 570 with
128GB of ram running a partition with all 8 CPUS and RAM, then I shut
that partition down and activated a partition with only 4 CPUS and
64GB of RAM, but Linux only gave me 32GB.  I also saw similar messages
with "Hole in node, disabling region."

I believe it was running the latest Firmware as well (0434E ?), has it
been fixed in a newer version?

Sonny