[Linuxppc-users] memmap kernel option on ubuntu 16.04 power8

[Michael Ellerman]

Hi Brian,

Brian Horton <brianh at linux.vnet.ibm.com> writes:
> Greetings all.
>
> I'm trying to run some tests with less memory than what's installed. the 
> 'mem=128G' option works, but it ends up putting all of the memory on the 
> first numa node:

It doesn't put memory anywhere, it just clamps the total memory, and it
happens that the first (lowest address) 128G of RAM is all on node 0.

>     available: 2 nodes (0,8)
>     node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
>     22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
>       41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62
>     63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79
>     node 0 size: 130695 MB
>     node 0 free: 10765 MB
>     node 8 cpus: 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97
>     98 99 100 101 102 103 104 105 106 107 108 109 110 111 112
>     113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129
>     130 131 132 133 134 135 136 137 138 139 140 141 142 143 1
>     44 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159
>     node 8 size: 0 MB
>     node 8 free: 0 MB
>
>
> I want 64G on each of the 2 numa nodes. Found this kernel option:
>
>          memmap=nn[KMG]$ss[KMG]
>                          [KNL,ACPI] Mark specific memory as reserved.
>                          Region of memory to be reserved is from ss to ss+nn.
>                          Example: Exclude memory from 0x18690000-0x1869ffff
>                                   memmap=64K$0x18690000
>                                   or
>                                   memmap=0x10000$0x18690000
>
> and it works on x86:
>
>     available: 2 nodes (0-1)
>     node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 24 25 26 27 28 29 30 31 32 33
>     34 35
>     node 0 size: 64315 MB
>     node 0 free: 364 MB
>     node 1 cpus: 12 13 14 15 16 17 18 19 20 21 22 23 36 37 38 39 40 41
>     42 43 44 45 46 47
>     node 1 size: 64507 MB
>     node 1 free: 134 MB
>
>
> but i can't get it to work on Ubuntu 16.04 Power8.

It's not supported on Power.

Further up in the docs it says:

	memmap=exactmap	[KNL,X86] Enable setting of an exact
			E820 memory map, as specified by the user.

Where the X86 means x86 only.

> someone said that it worked on RHEL, so maybe it's an Ubuntu-specific bug..

OK that would be interesting, but I suspect it's not true :)

> thoughts? ideas?

AFAIK there's no way to achieve what you're after, without changing
kernel code or creating a custom device tree.

The mem= option is very primitive, it just clamps the total as I said
above, it's not NUMA aware.

What hardware/platform are you trying to do this on? Are you familiar
with kexec?

cheers