[TECH TOPIC] Reaching consensus on CONFIG_HIGHMEM phaseout

[Arnd Bergmann]

High memory is one of the least popular features of the Linux kernel.
Added in 1999 for linux-2.3.16 to support large x86 machines, there
are very few systems that still need it. I talked about about this
recently at the Embedded Linux Conference on 32-bit systems [1][2][3]
and there were a few older discussions before[4][5][6].

While removing a feature that is actively used is clearly a regression
and not normally done, I expect removing highmem is going to happen
at some point anyway when there are few enough users, but the question
is when that time will be.

I'm still collecting information about which of the remaining highmem
users plan to keep updating their kernels and for what reason. Some
users obviously are alarmed about potentially losing this ability,
so I hope to get a broad consensus on a specific timeline for how long
we plan to support highmem in the page cache and to give every user
sufficient time to migrate to a well-tested alternative setup if that
is possible, or stay on a highmem-enabled LTS kernel for as long
as necessary.

These are the assumptions I'm making, based on both what I have
presented in my talk and feedback I have received so far, let me
know if something is missing here:

- Highmem in new kernels is almost exclusively an embedded Linux
  topic. While there were a few late 32-bit desktop and laptop
  systems that had more than 2GB of RAM, these were fairly short
  lived and have long been unsupported by both the originally
  shipping operating systems and most Linux distros. Notable
  Examples include Pentium 4 "Northwood" desktops (sold 2003-2004),
  Core Duo laptops (2006-2007), and Arm Chromebooks (rk3288,
  Tegra k1, Exynos 5800, sold 2014-2017). Some PowerPC G4 Macs
  and Atom Netbooks could be upgraded to 2GB.
  There are a small number of users, but they really love these
  devices and want to keep them alive, especially since they
  mark the peak of the respective 32-bit product lines.

- Within the embedded market, highmem is mostly used on ARMv7
  based SoCs, but a few others also need it and still get kernel
  updates:
  PowerPC 85xx, 86xx and QoriQ P1/P2/P3/P4 (produced 2003-2021)
  were used in some long-lived embedded systems with 2GB of RAM
  or more. Mediatek MT7621 (MIPS32r3, introduced 2014 but still
  sold) needs highmem to reach the upper 64MB of the 512MB physical
  memory. Ingenic JZ4780 (MIPS32, released 2012) was used in the
  short-lived MIPS Creator CI20 with 1GB of RAM (256MB lowmem)
  and probably othes. Sparc32/LEON seems to be limited to 192MB of
  lowmem as a kernel design choice. Vortex86DX3 supports 2GB
  DDR3 memory.
  The kernel also supports highmem on ARMv4, ARMv5, ARMv6,
  PPC4xx, PPC82xx, PPC83xx, ARC, CSKY, Microblaze and Xtensa,
  as well as additional MIPS SoCs,  but so far I could not find
  any indication of any such machine with more than 1GB that
  keeps getting kernel updates.

- The vast majority of new embedded ARMv7 machines have 1GB of
  RAM or less, which on many SoCs is a physical limitation
  a narrow DDR3 memory interface, as well as a cost tradeoff.
  The 1GB case is interesting because that usually means having
  only 768MB of lowmem plus 256MB of highmem, as well as 3GB
  of virtual addressing. I expect that almost all applications
  on these work just as well with CONFIG_VMSPLIT_3G_OPT, changing
  the limit to 1GB of lowmem and 2816MB of user address space.
  The same thing should work on x86 and powerpc (CONFIG_LOWMEM_SIZE)
  but not on mips and sparc where the limit is not configurable.

- A few Arm SoCs have sparse physical address ranges for
  their RAM, e.g. range per memory controllers like the Renesas
  R-Car Gen 2 or Broadcom BCM4708. These currently require highmem
  even on configurations with less than 1GB RAM, until we change
  the way that sparsemem is handled to support rearranging the
  linear map to fill all of lowmem. This still needs more work.

- ARMv7 machines with 2GB remain in production, in particular
  with the popular i.MX6Dual/Quad chip that has a 64-bit wide DDR3
  interface and guaranteed manufacturer support until 2035.
  This is the configuration I expect to see struggle the most.
  Setting CONFIG_VMSPLIT_2G or CONFIG_VMSPLIT_2G_OPT should work
  for most of the users but can break if an application runs out
  of virtual address space, so this does require extensive testing,
  and possibly user space changes. An example of possibly affected
  userspace is Firefox, which needs more address space than other
  applications but can perhaps be replaced with another embedded
  browser.
  
- ARMv7 machines with 4GB and more exist and keep getting
  kernel upgrades, but to my knowledge are not in production any
  more. These are mainly 2010-2015 era chips based on rare
  out-of-order cores like A15, A17 or PJ4 that were designed for
  low-end servers, chromebooks and network equipment but replaced
  with 64-bit chips shortly after. We had planned to bring a
  CONFIG_VMSPLIT_4G_4G option to ARMv7VE to keep supporting the full
  memory at a performance penalty, but currently have no plan to
  finish this (volunteers welcome).
  There is still some hope to keep them working with a combination
  of CONFIG_VMSPLIT_2G and a modified ZRAM that can use high
  pages without CONFIG_HIGHMEM, but whether this works depends
  a lot on the application. I expect most of these products to
  stop getting kernel updates in the next few years due to aging
  hardware and increasing cost for updating out-of-tree patches
  on platforms that are not fully upstream. I do not remember any
  such devices with official support beyond 2030.

My proposal based on the above assumptions is to gradually phase
out highmem over the next 2 years for mainline kernels, obviously
both the individual items and the timeline are completely up for
debate:

1. mark CONFIG_HIGHMEM as 'depends on EXPERT', updating the
   Kconfig description to point to the kernel summit discussion
   any any decisions made here.

2. Change the ARMv7 Kconfig defaults to CONFIG_VMSPLIT_3G_OPT
   and HIGHMEM=n, to make it more likely to find possible
   regressions with that, without changing much for users.
   Possibly do the same for x86 and powerpc.

3. Start removing __GFP_HIGHMEM from in-kernel allocations
   other than the page cache, in particular from individual
   device drivers and filesystem metadata where there is
   already little benefit.

4. Remove HIGHMEM as an option from platforms that are thought
   to no longer need it (arc, armv5, ppc4xx, ppc82xx, ppc83xx,
   csky, microblaze, xtensa), mainly to validate the
   assertion that these use only lowmem.

5. Split highmem on zram out into a separate Kconfig option
   that can be enabled without CONFIG_HIGHMEM or CONFIG_EXPERT

6. Finish the "densemem" replacement for sparsemem, ideally
   allowing both very sparse lowmem areas and a boot-time
   vmsplit selection instead of the compile-time one.

7. Finally, remove the highmem pagecache option, leaving only
   zram and custom device drivers as a way to access high
   pages.

That last step should wait for an LTS kernel, ideally a version
that the CIP project's SLTS kernel is planning to keep supporting
for 10 years. The newest SLTS kernel was 6.12 last year, and the
phb-crytal-ball suggests that the next one in December 2026 may
be linux-7.4, the one after that in December 2028 (linux-7.15?)
seems too far out to plan for but would be another option.

Unless there is an easy consensus on this on the mailing list,
I would like to lead a discussion session at the kernel summit
in order to get closer to a decision.

     Arnd 

[1] https://osseu2025.sched.com/event/25VmZ/32-bit-linux-support-now-and-in-the-future-arnd-bergmann-linaro
[2] https://www.youtube.com/watch?v=QiOMiyGCoTw
[3] https://lwn.net/Articles/1035727/
[4] https://lore.kernel.org/all/0047f565-ada4-491a-b157-f2d8dfde0ac0@app.fastmail.com/
[5] https://lwn.net/Articles/813201/
[6] https://lpc.events/event/16/contributions/1183/attachments/1062/2028/highmem-api-2022-09-12.pdf