[powerpc] Kernel oops while running xfstests w/ext4 (5.18-rc6-next-20220510)

Thu May 12 00:35:27 AEST 2022


> On 11-May-2022, at 2:56 PM, Sachin Sant <sachinp at linux.ibm.com> wrote:
> 
> While running xfstests (specifically ext4/032) w/ext4 on a POWER9 LPAR running
> linux-next version 5.18.0-rc6-next-20220510 following crash is seen:
> 
> [  472.486440] EXT4-fs (loop0): resized filesystem to 41943040
> [  472.760888] BUG: Kernel NULL pointer dereference at 0x0000002c
> [  472.760891] Faulting instruction address: 0xc0000000007729f4
> [  472.760894] Oops: Kernel access of bad area, sig: 11 [#1]
> [  472.760913] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
> [  472.760921] Modules linked in: loop(E) dm_mod(E) nft_fib_inet(E) nft_fib_ipv4(E) nft_fib_ipv6(E) nft_fib(E) nft_reject_inet(E) nf_reject_ipv4(E) nf_reject_ipv6(E) nft_reject(E) nft_ct(E) nft_chain_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) ip_set(E) bonding(E) rfkill(E) tls(E) nf_tables(E) libcrc32c(E) nfnetlink(E) sunrpc(E) pseries_rng(E) vmx_crypto(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sd_mod(E) sg(E) lpfc(E) nvmet_fc(E) nvmet(E) ibmvscsi(E) scsi_transport_srp(E) ibmveth(E) nvme_fc(E) nvme(E) nvme_fabrics(E) nvme_core(E) t10_pi(E) scsi_transport_fc(E) crc64_rocksoft(E) crc64(E) tg3(E) ipmi_devintf(E) ipmi_msghandler(E) fuse(E)
> [  472.761006] CPU: 8 PID: 5139 Comm: kworker/u193:0 Tainted: G            E     5.18.0-rc6-next-20220510 #2
> [  472.761013] Workqueue: loop0 loop_rootcg_workfn [loop]
> [  472.761027] NIP:  c0000000007729f4 LR: c00000000077331c CTR: c0000000009e9ac0
> [  472.761032] REGS: c00000002d95b3a0 TRAP: 0380   Tainted: G            E      (5.18.0-rc6-next-20220510)
> [  472.761038] MSR:  800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>  CR: 24008822  XER: 00000000
> [  472.761057] CFAR: c000000000772b80 IRQMASK: 0 
> [  472.761057] GPR00: c00000000077331c c00000002d95b640 c000000002a7cf00 c00000002d95b8e0 
> [  472.761057] GPR04: c00000006fd58200 0000000000000001 0000000000000010 0000000000000040 
> [  472.761057] GPR08: 0000000000000020 0000000000000000 0000000000010000 c0080000089570f8 
> [  472.761057] GPR12: 0000000000008000 c00000001ec46300 0000000000000000 c000000054e32200 
> [  472.761057] GPR16: 5deadbeef0000100 0000000000000000 0000000000000000 0000000000000000 
> [  472.761057] GPR20: 000000007fffffff c009fffffc817a00 c00000002d95b748 c00000002d95b8e0 
> [  472.761057] GPR24: 0000000000000001 0000000000000000 c0000000842b1c00 0000000000000000 
> [  472.761057] GPR28: 0000000000000000 0000000000000000 c00000006fd58200 c00000002d95b8e0 
> [  472.761126] NIP [c0000000007729f4] blk_add_rq_to_plug+0x74/0x1d0
> [  472.761135] LR [c00000000077331c] blk_mq_try_issue_list_directly+0x18c/0x1d0
> [  472.761141] Call Trace:
> [  472.761144] [c00000002d95b640] [c0000000842b1c00] 0xc0000000842b1c00 (unreliable)
> [  472.761153] [c00000002d95b680] [c000000000773244] blk_mq_try_issue_list_directly+0xb4/0x1d0
> [  472.761160] [c00000002d95b6d0] [c00000000077b38c] blk_mq_sched_insert_requests+0x13c/0x240
> [  472.761168] [c00000002d95b720] [c000000000772658] blk_mq_flush_plug_list+0x118/0x440
> [  472.761175] [c00000002d95b7c0] [c00000000075ecbc] __blk_flush_plug+0x17c/0x200
> [  472.761183] [c00000002d95b840] [c00000000075efe0] blk_finish_plug+0x50/0x70
> [  472.761190] [c00000002d95b870] [c00000000061a2a4] __iomap_dio_rw+0x444/0x960
> [  472.761200] [c00000002d95ba60] [c00000000061a7e0] iomap_dio_rw+0x20/0x90
> [  472.761208] [c00000002d95ba80] [c008000008c56424] ext4_file_read_iter+0x17c/0x2d0 [ext4]
> [  472.761237] [c00000002d95bac0] [c008000009822aa8] lo_rw_aio.isra.36+0x260/0x320 [loop]
> [  472.761245] [c00000002d95bb40] [c008000009824030] loop_process_work+0x448/0xb70 [loop]
> [  472.761253] [c00000002d95bc90] [c000000000183744] process_one_work+0x2b4/0x5b0
> [  472.761262] [c00000002d95bd30] [c000000000183ab8] worker_thread+0x78/0x600
> [  472.761269] [c00000002d95bdc0] [c0000000001901d4] kthread+0x124/0x130
> [  472.761276] [c00000002d95be10] [c00000000000ce04] ret_from_kernel_thread+0x5c/0x64
> [  472.761284] Instruction dump:
> [  472.761288] 893f0014 38e00040 39000020 2fa90000 7d283f9e 7e8a4840 409400b4 e93e0000 
> [  472.761300] e9290068 71290008 40820024 3d400001 <813d002c> 614affff 7e895040 41950090 
> [  472.761314] ---[ end trace 0000000000000000 ]---
> [  472.769088] 
> [  473.769091] Kernel panic - not syncing: Fatal exception
> 
> 5.18.0-rc6-next-20220509 build did not exhibit this problem.
> Will try git bisect and report back with results.
> 

Unfortunately git bisect doesn’t seem to help.
first bad commit: [3aedd17333a514c6f2542ed305d940e7a970a6f2] 
          Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc.git

# git bisect log
git bisect start
# bad: [3bf222d317a20170ee17f082626c1e0f83537e13] Add linux-next specific files for 20220510
git bisect bad 3bf222d317a20170ee17f082626c1e0f83537e13
# good: [c5eb0a61238dd6faf37f58c9ce61c9980aaffd7a] Linux 5.18-rc6
git bisect good c5eb0a61238dd6faf37f58c9ce61c9980aaffd7a
# good: [5b0c7020c4bf2cfed914323a7d58777c47df6bf8] Merge branch 'drm-next' of git://git.freedesktop.org/git/drm/drm.git
git bisect good 5b0c7020c4bf2cfed914323a7d58777c47df6bf8
# bad: [d11e84253a18f5afa5bb6e7da8509ba249435c0c] Merge branch 'next' of git://git.kernel.org/pub/scm/virt/kvm/kvm.git
git bisect bad d11e84253a18f5afa5bb6e7da8509ba249435c0c
# good: [24f8d248db54c383b01e373355d8655810832192] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input.git
git bisect good 24f8d248db54c383b01e373355d8655810832192
# bad: [52be5c3c2431cb82789bd9cdb5169f7a2538a8af] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/robh/linux.git
git bisect bad 52be5c3c2431cb82789bd9cdb5169f7a2538a8af
# good: [020a5624b05bb35b24beb1e405ecf38a37c3c238] Merge branch 'pcmcia-next' of git://git.kernel.org/pub/scm/linux/kernel/git/brodo/linux.git
git bisect good 020a5624b05bb35b24beb1e405ecf38a37c3c238
# bad: [528beacf1e5a082c15738652115e8eed516e925d] Merge branch 'master' of https://scm.osdn.net/gitroot/tomoyo/tomoyo-test1.git
git bisect bad 528beacf1e5a082c15738652115e8eed516e925d
# bad: [e8f0eb6a4d1b9775d1ba3ed5b95bbb9cdbe663a0] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-power-supply.git
git bisect bad e8f0eb6a4d1b9775d1ba3ed5b95bbb9cdbe663a0
# good: [7a0587496a6233b9ffa8441573d38f8844751066] dt-bindings: mmc: sdhci-am654: Add flag to force setting of TESTCD bit
git bisect good 7a0587496a6233b9ffa8441573d38f8844751066
# good: [17a9f73d45ea74b2beb009c29ad38569990c3453] dt-bindings: mmc: sdhci-msm: Add compatible string for sm8150
git bisect good 17a9f73d45ea74b2beb009c29ad38569990c3453
# good: [3474b838f4204c21d108183d9268611d961a428f] dt-bindings: Drop undocumented i.MX iomuxc-gpr bindings in examples
git bisect good 3474b838f4204c21d108183d9268611d961a428f
# good: [0c9ee5ba7555016afd1efc9598c3f83de5d83470] mmc: sdhci-brcmstb: Fix compiler warning
git bisect good 0c9ee5ba7555016afd1efc9598c3f83de5d83470
# good: [d96a89407e5f682d1cb22569d91784506c784863] power: supply: bq24190_charger: using pm_runtime_resume_and_get instead of pm_runtime_get_sync
git bisect good d96a89407e5f682d1cb22569d91784506c784863
# bad: [6d4c27440f59a40aa8df6ad5ed3ff8ae50048870] Merge branch 'for-mfd-next' of git://git.kernel.org/pub/scm/linux/kernel/git/lee/mfd.git
git bisect bad 6d4c27440f59a40aa8df6ad5ed3ff8ae50048870
# bad: [3aedd17333a514c6f2542ed305d940e7a970a6f2] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc.git
git bisect bad 3aedd17333a514c6f2542ed305d940e7a970a6f2
# first bad commit: [3aedd17333a514c6f2542ed305d940e7a970a6f2] Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc.git

-Sachin