[Skiboot] Memory allocations / free HEAP space

[Stewart Smith]

Hi all,

I've been looking at HEAP usage of skiboot booting in various
environments.

We currently reserve 12MB for heap (or 11MB once there's my gcov
modifications). On Mambo, we only use about 600k -
but this is due to there being only one cpu and pretty much no
devices. On a dual socket P8, we use a lot more - on an FSP system we
currently only have 2.3MB free when we boot the kernel. This is possibly
getting a bit close.

We can take one (or both) of these actions:
1) reduce memory usage
2) add extra heap.

I added a small patch to dump out the allocations and free space,
getting a decent view as to both memory usage and free space
fragmentation.

The big allocations are:
[15649665335,5]     0x00100010 hw/fsp/fsp-console.c:543.

We preallocate memory for each possible console. We possibly don't need
to allocate all of these on startup, perhaps only when console is opened?

[15649677058,5]     0x0000c010 hw/fsp/fsp-mem-err.c:386.

This should probably be converted to use core/pool.c rather than custom
pool.

[15649684021,5]     0x00040010 hw/fsp/fsp-elog-read.c:537.
[15649691303,5]     0x00001010 hw/fsp/fsp-elog-read.c:515.

Instinctively I think this should be core/pool.c rather than custom one,
but I haven't looked into details.

We also probably don't need to statically allocate the error log buffer
to read from FSP?


[15649698639,5]     0x00010010 hw/fsp/fsp-elog-write.c:398.

We do probably want to keep the panic buffer allocated at boot time,
although in the code path that uses it, we probably also want to avoid
allocations (which it doesn't look like we really succeed at).

[15649706019,5]     0x00040010 hw/fsp/fsp-elog-write.c:405.
[15649713362,5]     0x00010010 hw/fsp/fsp-elog-write.c:412.

probably also should be core/pool.c

[15649720676,5]     0x000e1010 core/pool.c:66.

This is actually from somewhere else, not sure where though.

[15649741434,5]     0x00100010 hw/fsp/fsp.c:1083.

This is fsp_inbound_buf which I'm not convinced needs to be always
allocated and I'm not convinced we really need (it looks like only
fsp-leds use it)

[15649753411,5]     0x0007fb90 core/hostservices.c:422.

This is hservice_lid_load, so it's HBRT.

[15650025347,5]     0x00100010 hw/fsp/fsp-sensor.c:724.

sensor_buffer. Without looking closely, cannot work out if there's an
"easy" way to not use this much memory for duration of
runtime... perhaps this is something we just have to eat?


[15651004495,5]     0x0007fb90 core/hostservices.c:422.

Huh... didn't expect to see this again... I wonder if we are leaking
memory across load requests or if this is a different LID?

[15651055312,5]     0x00100010 core/nvram.c:228.

This is because we allocate a copy of NVRAM in OPAL and do all reads
from our cache but write-through to NVRAM.

What we *should* have done of course is have an API that was async, but
the Linux code now exists that does this:

        rc = opal_read_nvram(__pa(buf), count, off);
        if (rc != OPAL_SUCCESS)
                return -EIO;

and this is the API, that it doesn't block.

We *could* still adapt, but we'd have to deprecate things over a long
period of time, so it looks like we're stuck (unless NVRAM suddenly
jumps in size and we *have* to fix this).

[15652018374,5]     0x00027a30 hw/phb3.c:4188.
[15652359587,5]     0x00020010 hw/phb3.c:2217.
[15652434447,5]     0x00027a30 hw/phb3.c:4188.
[15652574357,5]     0x00027a30 hw/phb3.c:4188.
[15652714381,5]     0x00027a30 hw/phb3.c:4188.
[15652847063,5]     0x00027a30 hw/phb3.c:4188.
[15652979102,5]     0x00027a30 hw/phb3.c:4188.
[15653119254,5]     0x00027a30 hw/phb3.c:4188.
[15653259644,5]     0x00027a30 hw/phb3.c:4188.

These look like the cost of doing PCI.