[6.4.0-rc7-next-20230620] Boot failure on IBM Power LPAR

Tue Jun 20 22:11:57 AEST 2023

6.4.0-rc7-next-20230620 fails to boot on IBM Power LPAR with following

[ 5.548368] BUG: Unable to handle kernel data access at 0x95bdcf954bc34e73
[ 5.548380] Faulting instruction address: 0xc000000000548090
[ 5.548384] Oops: Kernel access of bad area, sig: 11 [#1]
[ 5.548387] LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=2048 NUMA pSeries
[ 5.548391] Modules linked in: nf_tables(E) nfnetlink(E) sunrpc(E) binfmt_misc(E) pseries_rng(E) aes_gcm_p10_crypto(E) drm(E) drm_panel_orientation_quirks(E) xfs(E) libcrc32c(E) sd_mod(E) sr_mod(E) t10_pi(E) crc64_rocksoft_generic(E) cdrom(E) crc64_rocksoft(E) crc64(E) sg(E) ibmvscsi(E) scsi_transport_srp(E) ibmveth(E) vmx_crypto(E) fuse(E)
[ 5.548413] CPU: 1 PID: 789 Comm: systemd-udevd Tainted: G E 6.4.0-rc7-next-20230620 #1
[ 5.548417] Hardware name: IBM,9080-HEX POWER10 (raw) 0x800200 0xf000006 of:IBM,FW1030.20 (NH1030_058) hv:phyp pSeries
[ 5.548421] NIP: c000000000548090 LR: c000000000547fbc CTR: c0000000004206f0
[ 5.548424] REGS: c0000000afb536f0 TRAP: 0380 Tainted: G E (6.4.0-rc7-next-20230620)
[ 5.548427] MSR: 800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 88028202 XER: 20040000
[ 5.548436] CFAR: c000000000547fc4 IRQMASK: 0 
[ 5.548436] GPR00: c000000000547fbc c0000000afb53990 c0000000014b1600 0000000000000000 
[ 5.548436] GPR04: 0000000000000cc0 00000000000034d8 0000000000000e6f ed5e02cab43c21e0 
[ 5.548436] GPR08: 0000000000000e6e 0000000000000058 0000001356ea0000 0000000000002000 
[ 5.548436] GPR12: c0000000004206f0 c0000013fffff300 0000000000000000 0000000000000000 
[ 5.548436] GPR16: 0000000000000000 0000000000000000 0000000000000000 c000000092f43708 
[ 5.548436] GPR20: c000000092f436b0 0000000000000000 fffffffffff7dfff c0000000afa80000 
[ 5.548436] GPR24: c000000002b87aa0 00000000000000b8 c000000000159914 0000000000000cc0 
[ 5.548436] GPR28: 95bdcf954bc34e1b c00000000a1fafc0 0000000000000000 c000000003019800 
[ 5.548473] NIP [c000000000548090] kmem_cache_alloc+0x1a0/0x420
[ 5.548480] LR [c000000000547fbc] kmem_cache_alloc+0xcc/0x420
[ 5.548485] Call Trace:
[ 5.548487] [c0000000afb53990] [c000000000547fbc] kmem_cache_alloc+0xcc/0x420 (unreliable)
[ 5.548493] [c0000000afb53a00] [c000000000159914] vm_area_dup+0x44/0xf0
[ 5.548499] [c0000000afb53a40] [c00000000015a638] dup_mmap+0x298/0x8b0
[ 5.548504] [c0000000afb53bb0] [c00000000015acd0] dup_mm.constprop.0+0x80/0x180
[ 5.548509] [c0000000afb53bf0] [c00000000015bdc0] copy_process+0xc00/0x1510
[ 5.548514] [c0000000afb53cb0] [c00000000015c848] kernel_clone+0xb8/0x5a0
[ 5.548519] [c0000000afb53d30] [c00000000015ceb8] __do_sys_clone+0x88/0xd0
[ 5.548524] [c0000000afb53e10] [c000000000033bcc] system_call_exception+0x13c/0x340
[ 5.548529] [c0000000afb53e50] [c00000000000d05c] system_call_vectored_common+0x15c/0x2ec
[ 5.548534] --- interrupt: 3000 at 0x7fff87f0c178
[ 5.548538] NIP: 00007fff87f0c178 LR: 0000000000000000 CTR: 0000000000000000
[ 5.548540] REGS: c0000000afb53e80 TRAP: 3000 Tainted: G E (6.4.0-rc7-next-20230620)
[ 5.548544] MSR: 800000000000f033 <SF,EE,PR,FP,ME,IR,DR,RI,LE> CR: 44004204 XER: 00000000
[ 5.548552] IRQMASK: 0 
[ 5.548552] GPR00: 0000000000000078 00007ffffde8cb80 00007fff88637500 0000000001200011 
[ 5.548552] GPR04: 0000000000000000 0000000000000000 0000000000000000 00007fff888bd490 
[ 5.548552] GPR08: 0000000000000001 0000000000000000 0000000000000000 0000000000000000 
[ 5.548552] GPR12: 0000000000000000 00007fff888c4c00 0000000000000002 00007ffffde95698 
[ 5.548552] GPR16: 00007ffffde95690 00007ffffde95688 00007ffffde956a0 0000000000000028 
[ 5.548552] GPR20: 0000000132bca308 0000000000000001 0000000000000001 0000000000000315 
[ 5.548552] GPR24: 0000000000000003 0000000000000040 0000000000000000 0000000000000003 
[ 5.548552] GPR28: 0000000000000000 0000000000000000 00007ffffde8cf24 0000000000000045 
[ 5.548586] NIP [00007fff87f0c178] 0x7fff87f0c178
[ 5.548589] LR [0000000000000000] 0x0
[ 5.548591] --- interrupt: 3000
[ 5.548593] Code: e93f0000 7ce95214 e9070008 7f89502a e9270010 2e3c0000 41920258 2c290000 41820250 813f0028 e8ff00b8 38c80001 <7fdc482a> 7d3c4a14 79250022 552ac03e 
[ 5.548605] ---[ end trace 0000000000000000 ]---
[ 5.550849] pstore: backend (nvram) writing error (-1)
[ 5.550852] 
Starting Network Manager...
[ 5.566384] BUG: Bad rss-counter state mm:00000000dc60f1c1 type:MM_ANONPAGES val:36
[ 5.568784] BUG: Bad rss-counter state mm:000000008eb9341b type:MM_ANONPAGES val:36
[ 5.689774] BUG: Bad rss-counter state mm:00000000edbda345 type:MM_ANONPAGES val:36
[ 5.692187] BUG: Bad rss-counter state mm:000000003f7ec21f type:MM_ANONPAGES val:36
[ 5.705947] BUG: Bad rss-counter state mm:00000000cdbb7cfd type:MM_ANONPAGES val:36
[ 6.550855] Kernel panic - not syncing: Fatal exception
[ 6.568226] Rebooting in 10 seconds..

The problem was introduced in 6.4.0-rc7-next-20230619. I tried git bisect, but unsure of the
result reported by it. Bisect points to following patch

# git bisect bad
70c94cc2eefd4f98d222834cbe7512804977c2d4 is the first bad commit
commit 70c94cc2eefd4f98d222834cbe7512804977c2d4
Merge: 48f5ee5c48c3 3fe08f7d5e80
Author: Stephen Rothwell <sfr at canb.auug.org.au>
Date:   Tue Jun 20 09:43:25 2023 +1000

    Merge branch 'mm-everything' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
         # Conflicts:
         #       mm/mmap.c

git bisect start
# status: waiting for both good and bad commits
# bad: [9dbf40840551df336c95ce2a3adbdd25ed53c0ef] Add linux-next specific files for 20230620
git bisect bad 9dbf40840551df336c95ce2a3adbdd25ed53c0ef
# status: waiting for good commit(s), bad commit known
# good: [45a3e24f65e90a047bef86f927ebdc4c710edaa1] Linux 6.4-rc7
git bisect good 45a3e24f65e90a047bef86f927ebdc4c710edaa1
# bad: [175cde0dcc05c0905adeb55dff5ac49da96552b3] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git
git bisect bad 175cde0dcc05c0905adeb55dff5ac49da96552b3
# bad: [d16e40b24a7d258d166fbfe46f0f565a21204df7] Merge branch 'xtensa-for-next' of git://github.com/jcmvbkbc/linux-xtensa.git
git bisect bad d16e40b24a7d258d166fbfe46f0f565a21204df7
# bad: [2be5f21481bf5606654520c19bd016090522f5d4] Merge branch 'for-next' of https://git.kernel.org/pub/scm/linux/kernel/git/krzk/linux-mem-ctrl.git
git bisect bad 2be5f21481bf5606654520c19bd016090522f5d4
# bad: [1dfd9944d721bef26f49d00220ce86efeb77711d] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git
git bisect bad 1dfd9944d721bef26f49d00220ce86efeb77711d
# good: [34fd86722257374f73bb6da13a60cc19b0344e99] mm: zswap: remove shrink from zpool interface
git bisect good 34fd86722257374f73bb6da13a60cc19b0344e99
# good: [48f5ee5c48c342bd82fa04eefc8a41048a6165fc] Merge branch 'mm-nonmm-stable' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
git bisect good 48f5ee5c48c342bd82fa04eefc8a41048a6165fc
# good: [dfd058ab9bef3f6590fb349ae1a2dfa7fc3ee50e] mm/gup: do not return 0 from pin_user_pages_fast() for bad args
git bisect good dfd058ab9bef3f6590fb349ae1a2dfa7fc3ee50e
# good: [ec336aa83162fe0f3d554baed2d4e2589b69ec6e] scripts/mksysmap: Fix badly escaped '$'
git bisect good ec336aa83162fe0f3d554baed2d4e2589b69ec6e
# good: [b08e8297596bb6f80351dc50fc1b8c2250d3a318] modpost: show offset from symbol for section mismatch warnings
git bisect good b08e8297596bb6f80351dc50fc1b8c2250d3a318
# good: [14b17c0b28bbd853c43d1a815019091497b5b436] watchdog/hardlockup: sort hardlockup detector related config values a logical way
git bisect good 14b17c0b28bbd853c43d1a815019091497b5b436
# good: [1e5db612cc70f3137aa48978b267afff17eb222d] watchdog/hardlockup: define HARDLOCKUP_DETECTOR_ARCH
git bisect good 1e5db612cc70f3137aa48978b267afff17eb222d
# good: [3fe08f7d5e80b3f822673b70fcc6be8dbee58f76] Merge branch 'mm-nonmm-unstable' into mm-everything
git bisect good 3fe08f7d5e80b3f822673b70fcc6be8dbee58f76
# good: [9ac40f75debfcb20c93de71b434ae73add1f692d] linux/export.h: rename 'sec' argument to 'license'
git bisect good 9ac40f75debfcb20c93de71b434ae73add1f692d
# bad: [70c94cc2eefd4f98d222834cbe7512804977c2d4] Merge branch 'mm-everything' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
git bisect bad 70c94cc2eefd4f98d222834cbe7512804977c2d4
# first bad commit: [70c94cc2eefd4f98d222834cbe7512804977c2d4] Merge branch 'mm-everything' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

- Sachin