[Bisected] [commit dd6c3c544126] [linux-next] [6.8.0-rc2] Task hungs for infinite time call traces
Tasmiya Nalatwad
tasmiya at linux.vnet.ibm.com
Mon Feb 5 20:29:21 AEDT 2024
[linux-next] [6.8.0-rc2-next-20240130] [FC/XFS] Task hungs for infinite
time while running bonnie test XFS filesystem
Bisected the issue. Git bisect points to the below commit
commit dd6c3c5441263723305a9c52c5ccc899a4653000
workqueue: Move pwq_dec_nr_in_flight() to the end of work
item handling
--- Traces ---
[ 981.280811] Call Trace:
[ 981.280813] [c0000001d10b7560] [c000000006e9b934] 0xc000000006e9b934 (unreliable)
[ 981.280820] [c0000001d10b7710] [c00000000001fbac] __switch_to+0x13c/0x220
[ 981.280827] [c0000001d10b7770] [c000000001002840] __schedule+0x268/0x7c4
[ 981.280832] [c0000001d10b7840] [c000000001002ddc] schedule+0x40/0x108
[ 981.280836] [c0000001d10b78b0] [c00000000100b748] schedule_timeout+0x19c/0x1c0
[ 981.280841] [c0000001d10b7980] [c000000001004044] __wait_for_common+0x148/0x340
[ 981.280845] [c0000001d10b7a10] [c00000000018fa98] __flush_workqueue+0x15c/0x530
[ 981.280852] [c0000001d10b7ab0] [c008000008f89e9c] xfs_inodegc_flush+0x54/0x15c [xfs]
[ 981.280913] [c0000001d10b7b00] [c008000008f9f47c] xfs_unmountfs+0x30/0x1e4 [xfs]
[ 981.280969] [c0000001d10b7b80] [c008000008fa825c] xfs_fs_put_super+0x5c/0x110 [xfs]
[ 981.281023] [c0000001d10b7bf0] [c0000000005c8774] generic_shutdown_super+0xc0/0x16c
[ 981.281029] [c0000001d10b7c60] [c0000000005c8a50] kill_block_super+0x30/0x68
[ 981.281034] [c0000001d10b7c90] [c008000008fa5c54] xfs_kill_sb+0x28/0x4c [xfs]
[ 981.281088] [c0000001d10b7cc0] [c0000000005ca9d4] deactivate_locked_super+0x70/0x144
[ 981.281093] [c0000001d10b7cf0] [c000000000605728] cleanup_mnt+0x10c/0x1d8
[ 981.281098] [c0000001d10b7d40] [c00000000019b5e4] task_work_run+0xe0/0x16c
[ 981.281102] [c0000001d10b7d90] [c000000000022974] do_notify_resume+0x134/0x13c
[ 981.281107] [c0000001d10b7dc0] [c000000000032378] interrupt_exit_user_prepare_main+0x1ac/0x264
[ 981.281112] [c0000001d10b7e20] [c000000000032580] syscall_exit_prepare+0x150/0x178
[ 981.281116] [c0000001d10b7e50] [c00000000000d068] system_call_vectored_common+0x168/0x2ec
[ 981.281122] --- interrupt: 3000 at 0x7fffaed4c11c
[ 981.281125] NIP: 00007fffaed4c11c LR: 0000000000000000 CTR: 0000000000000000
[ 981.281128] REGS: c0000001d10b7e80 TRAP: 3000 Not tainted (6.8.0-rc2-next-20240130-auto)
[ 981.281131] MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 48002402 XER: 00000000
[ 981.281139] IRQMASK: 0
[ 981.281139] GPR00: 0000000000000034 00007fffec649770 00007fffaef07f00 0000000000000000
[ 981.281139] GPR04: 0000000000000000 ffffffffff000000 0000000000000000 0000000000000001
[ 981.281139] GPR08: 000000014cd61390 0000000000000000 0000000000000000 0000000000000000
[ 981.281139] GPR12: 0000000000000000 00007fffaefbc140 000000000ee6b280 00007fffec649a30
[ 981.281139] GPR16: 00007fffec649bd8 0000000118b66478 0000000000000000 0000000000000000
[ 981.281139] GPR20: ffffffffffffffff ffffffffffffffff 0000000000000000 0000000000000000
[ 981.281139] GPR24: 00007fffec64b0b0 0000000118b663d8 0000000118b66a58 0000000000000000
[ 981.281139] GPR28: 000000014cd61250 0000000000000000 000000014cd61370 000000014cd61140
[ 981.281175] NIP [00007fffaed4c11c] 0x7fffaed4c11c
[ 981.281177] LR [0000000000000000] 0x0
[ 981.281179] --- interrupt: 3000
[ 1104.160797] INFO: taskumount:32506 blocked for more than 245 seconds.
[ 1104.160811] Not tainted 6.8.0-rc2-next-20240130-auto #1
[ 1104.160814] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1104.160818]task:umount state:D stack:0 pid:32506 tgid:32506 ppid:32329 flags:0x00042080
[ 1104.160826] Call Trace:
[ 1104.160829] [c0000001d10b7560] [c000000006e9b934] 0xc000000006e9b934 (unreliable)
[ 1104.160836] [c0000001d10b7710] [c00000000001fbac] __switch_to+0x13c/0x220
[ 1104.160844] [c0000001d10b7770] [c000000001002840] __schedule+0x268/0x7c4
[ 1104.160851] [c0000001d10b7840] [c000000001002ddc] schedule+0x40/0x108
[ 1104.160856] [c0000001d10b78b0] [c00000000100b748] schedule_timeout+0x19c/0x1c0
[ 1104.160862] [c0000001d10b7980] [c000000001004044] __wait_for_common+0x148/0x340
[ 1104.160867] [c0000001d10b7a10] [c00000000018fa98] __flush_workqueue+0x15c/0x530
[ 1104.160873] [c0000001d10b7ab0] [c008000008f89e9c] xfs_inodegc_flush+0x54/0x15c [xfs]
[ 1104.160938] [c0000001d10b7b00] [c008000008f9f47c] xfs_unmountfs+0x30/0x1e4 [xfs]
[ 1104.160993] [c0000001d10b7b80] [c008000008fa825c] xfs_fs_put_super+0x5c/0x110 [xfs]
[ 1104.161047] [c0000001d10b7bf0] [c0000000005c8774] generic_shutdown_super+0xc0/0x16c
[ 1104.161053] [c0000001d10b7c60] [c0000000005c8a50] kill_block_super+0x30/0x68
[ 1104.161059] [c0000001d10b7c90] [c008000008fa5c54] xfs_kill_sb+0x28/0x4c [xfs]
[ 1104.161112] [c0000001d10b7cc0] [c0000000005ca9d4] deactivate_locked_super+0x70/0x144
[ 1104.161118] [c0000001d10b7cf0] [c000000000605728] cleanup_mnt+0x10c/0x1d8
[ 1104.161123] [c0000001d10b7d40] [c00000000019b5e4] task_work_run+0xe0/0x16c
[ 1104.161126] [c0000001d10b7d90] [c000000000022974] do_notify_resume+0x134/0x13c
[ 1104.161131] [c0000001d10b7dc0] [c000000000032378] interrupt_exit_user_prepare_main+0x1ac/0x264
[ 1104.161136] [c0000001d10b7e20] [c000000000032580] syscall_exit_prepare+0x150/0x178
[ 1104.161141] [c0000001d10b7e50] [c00000000000d068] system_call_vectored_common+0x168/0x2ec
[ 1104.161146] --- interrupt: 3000 at 0x7fffaed4c11c
[ 1104.161149] NIP: 00007fffaed4c11c LR: 0000000000000000 CTR: 0000000000000000
[ 1104.161152] REGS: c0000001d10b7e80 TRAP: 3000 Not tainted (6.8.0-rc2-next-20240130-auto)
[ 1104.161155] MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 48002402 XER: 00000000
[ 1104.161163] IRQMASK: 0
[ 1104.161163] GPR00: 0000000000000034 00007fffec649770 00007fffaef07f00 0000000000000000
[ 1104.161163] GPR04: 0000000000000000 ffffffffff000000 0000000000000000 0000000000000001
[ 1104.161163] GPR08: 000000014cd61390 0000000000000000 0000000000000000 0000000000000000
[ 1104.161163] GPR12: 0000000000000000 00007fffaefbc140 000000000ee6b280 00007fffec649a30
[ 1104.161163] GPR16: 00007fffec649bd8 0000000118b66478 0000000000000000 0000000000000000
[ 1104.161163] GPR20: ffffffffffffffff ffffffffffffffff 0000000000000000 0000000000000000
[ 1104.161163] GPR24: 00007fffec64b0b0 0000000118b663d8 0000000118b66a58 0000000000000000
[ 1104.161163] GPR28: 000000014cd61250 0000000000000000 000000014cd61370 000000014cd61140
[ 1104.161199] NIP [00007fffaed4c11c] 0x7fffaed4c11c
[ 1104.161202] LR [0000000000000000] 0x0
[ 1104.161204] --- interrupt: 3000
[ 1227.040796] INFO: taskumount:32506 blocked for more than 368 seconds.
[ 1227.040810] Not tainted 6.8.0-rc2-next-20240130-auto #1
[ 1227.040814] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1227.040817]task:umount state:D stack:0 pid:32506 tgid:32506 ppid:32329 flags:0x00042080
[ 1227.040823] Call Trace:
[ 1227.040826] [c0000001d10b7560] [c000000006e9b934] 0xc000000006e9b934 (unreliable)
[ 1227.040832] [c0000001d10b7710] [c00000000001fbac] __switch_to+0x13c/0x220
[ 1227.040839] [c0000001d10b7770] [c000000001002840] __schedule+0x268/0x7c4
[ 1227.040844] [c0000001d10b7840] [c000000001002ddc] schedule+0x40/0x108
[ 1227.040848] [c0000001d10b78b0] [c00000000100b748] schedule_timeout+0x19c/0x1c0
[ 1227.040853] [c0000001d10b7980] [c000000001004044] __wait_for_common+0x148/0x340
[ 1227.040857] [c0000001d10b7a10] [c00000000018fa98] __flush_workqueue+0x15c/0x530
[ 1227.040863] [c0000001d10b7ab0] [c008000008f89e9c] xfs_inodegc_flush+0x54/0x15c [xfs]
[ 1227.040925] [c0000001d10b7b00] [c008000008f9f47c] xfs_unmountfs+0x30/0x1e4 [xfs]
[ 1227.040980] [c0000001d10b7b80] [c008000008fa825c] xfs_fs_put_super+0x5c/0x110 [xfs]
[ 1227.041035] [c0000001d10b7bf0] [c0000000005c8774] generic_shutdown_super+0xc0/0x16c
[ 1227.041041] [c0000001d10b7c60] [c0000000005c8a50] kill_block_super+0x30/0x68
[ 1227.041045] [c0000001d10b7c90] [c008000008fa5c54] xfs_kill_sb+0x28/0x4c [xfs]
[ 1227.041099] [c0000001d10b7cc0] [c0000000005ca9d4] deactivate_locked_super+0x70/0x144
[ 1227.041105] [c0000001d10b7cf0] [c000000000605728] cleanup_mnt+0x10c/0x1d8
[ 1227.041109] [c0000001d10b7d40] [c00000000019b5e4] task_work_run+0xe0/0x16c
[ 1227.041113] [c0000001d10b7d90] [c000000000022974] do_notify_resume+0x134/0x13c
[ 1227.041118] [c0000001d10b7dc0] [c000000000032378] interrupt_exit_user_prepare_main+0x1ac/0x264
[ 1227.041123] [c0000001d10b7e20] [c000000000032580] syscall_exit_prepare+0x150/0x178
[ 1227.041128] [c0000001d10b7e50] [c00000000000d068] system_call_vectored_common+0x168/0x2ec
[ 1227.041133] --- interrupt: 3000 at 0x7fffaed4c11c
[ 1227.041136] NIP: 00007fffaed4c11c LR: 0000000000000000 CTR: 0000000000000000
[ 1227.041139] REGS: c0000001d10b7e80 TRAP: 3000 Not tainted (6.8.0-rc2-next-20240130-auto)
[ 1227.041142] MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 48002402 XER: 00000000
[ 1227.041150] IRQMASK: 0
[ 1227.041150] GPR00: 0000000000000034 00007fffec649770 00007fffaef07f00 0000000000000000
[ 1227.041150] GPR04: 0000000000000000 ffffffffff000000 0000000000000000 0000000000000001
[ 1227.041150] GPR08: 000000014cd61390 0000000000000000 0000000000000000 0000000000000000
[ 1227.041150] GPR12: 0000000000000000 00007fffaefbc140 000000000ee6b280 00007fffec649a30
[ 1227.041150] GPR16: 00007fffec649bd8 0000000118b66478 0000000000000000 0000000000000000
[ 1227.041150] GPR20: ffffffffffffffff ffffffffffffffff 0000000000000000 0000000000000000
[ 1227.041150] GPR24: 00007fffec64b0b0 0000000118b663d8 0000000118b66a58 0000000000000000
[ 1227.041150] GPR28: 000000014cd61250 0000000000000000 000000014cd61370 000000014cd61140
[ 1227.041186] NIP [00007fffaed4c11c] 0x7fffaed4c11c
[ 1227.041188] LR [0000000000000000] 0x0
[ 1227.041190] --- interrupt: 3000
--
Regards,
Tasmiya Nalatwad
IBM Linux Technology Center
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ozlabs.org/pipermail/linuxppc-dev/attachments/20240205/bdd010fa/attachment-0001.htm>
More information about the Linuxppc-dev
mailing list