sysrq-b fails miserably to reboot PowerNV box

Sun Oct 9 09:06:12 AEDT 2016

On Sat, 2016-10-08 at 22:16 +1100, Anton Blanchard wrote:
> Hi,
> 
> Unfortunately sysrq-b seems to tie us up in knots, instead of
> rebooting
> the box. This is mainline from today.

Argh yeah reboot from interrupt context, what a PITA...

ppc_md.machine_shutdown is called which is supposed to be a kexec
thing, it clears the PCI IODA tables and does OPAL shutdown which
frees the IRQs among other things.

Then the actual reboot is called which does things that are duplicates
of what machine_shutdown does ...

Somebody needs to dig in there ... I can try next week if I have
time but somebody else feel free to beat me to it.

Ben.

> Anton
> --
> 
> Trying to free IRQ 17 from IRQ context!
> ------------[ cut here ]------------
> WARNING: CPU: 32 PID: 0 at kernel/irq/manage.c:1460
> __free_irq+0x298/0x380
> Modules linked in: kvm_hv kvm binfmt_misc vmx_crypto crc32c_vpmsum
> leds_powernv powernv_rng led_class powernv_op_panel rng_core ib_iser
> rdma_cm iw_cm ib_cm ib_core libiscsi scsi_transport_iscsi autofs4
> CPU: 32 PID: 0 Comm: swapper/32 Tainted: G      D         4.8.0-
> 11288-gb66484c #785
> task: c0000017f91fce00 task.stack: c0000017f927c000
> NIP: c00000000012b4e8 LR: c00000000012b4e4 CTR: c000000000668950
> REGS: c000001fffe8b5c0 TRAP: 0700   Tainted:
> G      D          (4.8.0-11288-gb66484c)
> MSR: 9000000000029033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 28004224  XER:
> 20000000
> CFAR: c00000000000c2e0 SOFTE: 1 
> GPR00: c00000000012b4e4 c000001fffe8b840 c000000000dbc100
> 0000000000000027 
> GPR04: c0000007fb80ada0 c0000007fb81f598 0000000000000000
> c000000000dee9a8 
> GPR08: 0000000000000007 0000000000000000 00000007fab60000
> 9000000000001003 
> GPR12: 0000000000004400 c00000000fe08000 c0000017f927ff90
> 0000000000000000 
> GPR16: 0000000000000000 0000000000000000 0000000000000000
> 0000000000000000 
> GPR20: 0000000000000001 c000000000d35d70 c000001fffe8bb00
> 0000000000000000 
> GPR24: 0000000000000000 c0000007e3330400 c000000000d31de0
> 000000000000000f 
> GPR28: 0000000000000011 c000000ff7ed1400 0000000000000000
> 0000000000000011 
> NIP [c00000000012b4e8] __free_irq+0x298/0x380
> LR [c00000000012b4e4] __free_irq+0x294/0x380
> Call Trace:
> [c000001fffe8b840] [c00000000012b4e4] __free_irq+0x294/0x380
> (unreliable)
> [c000001fffe8b8d0] [c00000000012b6ec] free_irq+0x7c/0xb0
> [c000001fffe8b900] [c000000000072748] opal_event_shutdown+0x88/0xd0
> [c000001fffe8b940] [c00000000006db8c] opal_shutdown+0x1c/0x90
> [c000001fffe8b970] [c000000000069d00] pnv_shutdown+0x20/0x40
> [c000001fffe8b990] [c000000000026a18] machine_restart+0x38/0x90
> [c000001fffe8ba00] [c0000000000e7538] emergency_restart+0x28/0x40
> [c000001fffe8ba20] [c000000000568c44] sysrq_handle_reboot+0x24/0x40
> [c000001fffe8ba40] [c000000000569490] __handle_sysrq+0xe0/0x250
> [c000001fffe8bae0] [c00000000057f1a0] hvc_poll+0x230/0x380
> [c000001fffe8bb80] [c0000000005803cc] hvc_handle_interrupt+0x1c/0x50
> [c000001fffe8bba0] [c00000000012a2b0]
> __handle_irq_event_percpu+0x90/0x2f0
> [c000001fffe8bc60] [c00000000012a540]
> handle_irq_event_percpu+0x30/0x90
> [c000001fffe8bca0] [c00000000012a5fc] handle_irq_event+0x5c/0xc0
> [c000001fffe8bcd0] [c00000000012f5b4] handle_level_irq+0xd4/0x1e0
> [c000001fffe8bd00] [c000000000128e44] generic_handle_irq+0x44/0x70
> [c000001fffe8bd20] [c000000000072600] opal_handle_events+0xd0/0x120
> [c000001fffe8bd70] [c0000000000726a8] opal_interrupt+0x38/0x50
> [c000001fffe8bda0] [c00000000012a2b0]
> __handle_irq_event_percpu+0x90/0x2f0
> [c000001fffe8be60] [c00000000012a540]
> handle_irq_event_percpu+0x30/0x90
> [c000001fffe8bea0] [c00000000012a5fc] handle_irq_event+0x5c/0xc0
> [c000001fffe8bed0] [c00000000012f9f0] handle_fasteoi_irq+0xe0/0x290
> [c000001fffe8bf00] [c000000000128e44] generic_handle_irq+0x44/0x70
> [c000001fffe8bf20] [c00000000001641c] __do_irq+0x8c/0x210
> [c000001fffe8bf90] [c000000000027ed4] call_do_irq+0x14/0x24
> [c0000017f927fac0] [c000000000016634] do_IRQ+0x94/0x140
> [c0000017f927fb10] [c000000000008e8c]
> hardware_interrupt_common+0x10c/0x180
> --- interrupt: 501 at arch_local_irq_restore+0x5c/0x90
>     LR = arch_local_irq_restore+0x40/0x90
> [c0000017f927fe00] [c000000000155e44]
> tick_broadcast_oneshot_control+0x54/0x70 (unreliable)
> [c0000017f927fe20] [c0000000007b6da4] cpuidle_enter_state+0x1b4/0x440
> [c0000017f927fe80] [c000000000114da4] call_cpuidle+0x44/0x90
> [c0000017f927fea0] [c000000000115138] cpu_startup_entry+0x1f8/0x320
> [c0000017f927ff20] [c000000000043540] start_secondary+0x320/0x340
> [c0000017f927ff90] [c00000000000b56c]
> start_secondary_prolog+0x10/0x14
> Instruction dump:
> 7d2903a6 7fc3f378 f8410018 7d2c4b78 4e800421 e8410018 4bfffe68
> 3c62ffd6 
> 7f84e378 38639270 48811f35 60000000 <0fe00000> 4bfffdb4 7d2903a6
> 7fc3f378 
> ---[ end trace dab28cb0b01424b6 ]---
> EEH: Frozen PHB#3-PE#1 detected
> EEH: PE location: U78C9.001.WZS003V-P1-C10     , PHB location: N/A
> CPU: 25 PID: 0 Comm: swapper/25 Tainted: G      D W       4.8.0-
> 11288-gb66484c #785
> Call Trace:
> [c0000007f6483690] [c00000000093e6fc] dump_stack+0xb0/0xf0
> (unreliable)
> [c0000007f64836d0] [c000000000039c30]
> eeh_dev_check_failure+0x3f0/0x530
> [c0000007f6483780] [c000000000039df4] eeh_check_failure+0x84/0xd0
> [c0000007f64837c0] [c000000000677fb0] tg3_read32+0x80/0x90
> [c0000007f64837f0] [c000000000698acc] tg3_timer+0x5ec/0x1170
> [c0000007f6483850] [c000000000142334] call_timer_fn+0x54/0x1d0
> [c0000007f64838e0] [c000000000142600] expire_timers+0x120/0x1d0
> [c0000007f6483950] [c0000000001428a8] run_timer_softirq+0x1f8/0x260
> [c0000007f64839e0] [c0000000000c0478] __do_softirq+0x158/0x3e0
> [c0000007f6483ac0] [c0000000000c0958] irq_exit+0xe8/0x120
> [c0000007f6483ae0] [c0000000000400f0] doorbell_exception+0xa0/0xd0
> [c0000007f6483b10] [c00000000000a20c] h_doorbell_common+0x10c/0x180
> --- interrupt: e81 at arch_local_irq_restore+0x5c/0x90
>     LR = arch_local_irq_restore+0x40/0x90
> [c0000007f6483e00] [c000000000155e44]
> tick_broadcast_oneshot_control+0x54/0x70 (unreliable)
> [c0000007f6483e20] [c0000000007b6da4] cpuidle_enter_state+0x1b4/0x440
> [c0000007f6483e80] [c000000000114da4] call_cpuidle+0x44/0x90
> [c0000007f6483ea0] [c000000000115138] cpu_startup_entry+0x1f8/0x320
> [c0000007f6483f20] [c000000000043540] start_secondary+0x320/0x340
> [c0000007f6483f90] [c00000000000b56c]
> start_secondary_prolog+0x10/0x14
> EEH: Detected PCI bus error on PHB#3-PE#1
> EEH: This PCI device has failed 1 times in the last hour
> EEH: Notify device drivers to shutdown
> tg3 0003:09:00.0 eth0: PCI I/O error detected
> tg3 0003:09:00.1 eth1: PCI I/O error detected
> tg3 0003:09:00.2 eth2: PCI I/O error detected
> tg3 0003:09:00.3 eth3: PCI I/O error detected
> EEH: Collect temporary log
> PHB3 PHB#3 Diag-data (Version: 1)
> brdgCtl:     00000002
> RootSts:     0002002f 00400000 b0830008 00100147 00002800
> RootErrSts:  00000000 00008000 00000000
> PhbSts:      0000001c00000000 0000001c00000000
> Lem:         4000000050000000 42498e327f502eae 0000000000000000
> OutErr:      2000000000000000 2000000000000000 0204006000003b50
> 1000080000000000
> InAErr:      0000008800000000 0000000800000000 001d090000000800
> 0800000ff3b47380
> PE[  0] A/B: 8000002100000000 8000000000000000
> PE[  1] A/B: 8000000000000000 8000000000000000
> PE[  2] A/B: 8000000000000000 8000000000000000
> PE[  3] A/B: 8000000000000000 8000000000000000
> PE[  4] A/B: 8000000000000000 8000000000000000
> PE[  5] A/B: 8000000000000000 8000000000000000
> PE[  6] A/B: 8000000000000000 8000000000000000
> PE[  7] A/B: 8000000000000000 8000000000000000
> PE[  8] A/B: 8000000000000000 8000000000000000
> PE[  9] A/B: 8000000000000000 8000000000000000
> PE[ 10] A/B: 8000000000000000 8000000000000000
> PE[ 11] A/B: 8000000000000000 8000000000000000
> PE[ 12] A/B: 8000000000000000 8000000000000000
> PE[ 13] A/B: 8000000000000000 8000000000000000
> PE[ 14] A/B: 8000000000000000 8000000000000000
> PE[ 15] A/B: 8000000000000000 8000000000000000
> PE[ 16] A/B: 8000000000000000 8000000000000000
> PE[ 17] A/B: 8000000000000000 8000000000000000
> PE[ 18] A/B: 8000000000000000 8000000000000000
> PE[ 19] A/B: 8000000000000000 8000000000000000
> PE[ 20] A/B: 8000000000000000 8000000000000000
> PE[ 21] A/B: 8000000000000000 8000000000000000
> PE[ 22] A/B: 8000000000000000 8000000000000000
> PE[ 23] A/B: 8000000000000000 8000000000000000
> PE[ 24] A/B: 8000000000000000 8000000000000000
> PE[ 25] A/B: 8000000000000000 8000000000000000
> PE[ 26] A/B: 8000000000000000 8000000000000000
> PE[ 27] A/B: 8000000000000000 8000000000000000
> PE[ 28] A/B: 8000000000000000 8000000000000000
> PE[ 29] A/B: 8000000000000000 8000000000000000
> PE[ 30] A/B: 8000000000000000 8000000000000000
> PE[ 31] A/B: 8000000000000000 8000000000000000
> PE[ 32] A/B: 8000000000000000 8000000000000000
> PE[ 33] A/B: 8000000000000000 8000000000000000
> PE[ 34] A/B: 8000000000000000 8000000000000000
> PE[ 35] A/B: 8000000000000000 8000000000000000
> PE[ 36] A/B: 8000000000000000 8000000000000000
> PE[ 37] A/B: 8000000000000000 8000000000000000
> PE[ 38] A/B: 8000000000000000 8000000000000000
> PE[ 39] A/B: 8000000000000000 8000000000000000
> PE[ 40] A/B: 8000000000000000 8000000000000000
> PE[ 41] A/B: 8000000000000000 8000000000000000
> PE[ 42] A/B: 8000000000000000 8000000000000000
> PE[ 43] A/B: 8000000000000000 8000000000000000
> PE[ 44] A/B: 8000000000000000 8000000000000000
> PE[ 45] A/B: 8000000000000000 8000000000000000
> PE[ 46] A/B: 8000000000000000 8000000000000000
> PE[ 47] A/B: 8000000000000000 8000000000000000
> PE[ 48] A/B: 8000000000000000 8000000000000000
> PE[ 49] A/B: 8000000000000000 8000000000000000
> PE[ 50] A/B: 8000000000000000 8000000000000000
> PE[ 51] A/B: 8000000000000000 8000000000000000
> PE[ 52] A/B: 8000000000000000 8000000000000000
> PE[ 53] A/B: 8000000000000000 8000000000000000
> PE[ 54] A/B: 8000000000000000 8000000000000000
> PE[ 55] A/B: 8000000000000000 8000000000000000
> PE[ 56] A/B: 8000000000000000 8000000000000000
> PE[ 57] A/B: 8000000000000000 8000000000000000
> PE[ 58] A/B: 8000000000000000 8000000000000000
> PE[ 59] A/B: 8000000000000000 8000000000000000
> PE[ 60] A/B: 8000000000000000 8000000000000000
> PE[ 61] A/B: 8000000000000000 8000000000000000
> PE[ 62] A/B: 8000000000000000 8000000000000000
> PE[ 63] A/B: 8000000000000000 8000000000000000
> PE[ 64] A/B: 8000000000000000 8000000000000000
> PE[ 65] A/B: 8000000000000000 8000000000000000
> PE[ 66] A/B: 8000000000000000 8000000000000000
> PE[ 67] A/B: 8000000000000000 8000000000000000
> PE[ 68] A/B: 8000000000000000 8000000000000000
> PE[ 69] A/B: 8000000000000000 8000000000000000
> PE[ 70] A/B: 8000000000000000 8000000000000000
> PE[ 71] A/B: 8000000000000000 8000000000000000
> PE[ 72] A/B: 8000000000000000 8000000000000000
> PE[ 73] A/B: 8000000000000000 8000000000000000
> PE[ 74] A/B: 8000000000000000 8000000000000000
> PE[ 75] A/B: 8000000000000000 8000000000000000
> PE[ 76] A/B: 8000000000000000 8000000000000000
> PE[ 77] A/B: 8000000000000000 8000000000000000
> PE[ 78] A/B: 8000000000000000 8000000000000000
> PE[ 79] A/B: 8000000000000000 8000000000000000
> PE[ 80] A/B: 8000000000000000 8000000000000000
> PE[ 81] A/B: 8000000000000000 8000000000000000
> PE[ 82] A/B: 8000000000000000 8000000000000000
> PE[ 83] A/B: 8000000000000000 8000000000000000
> PE[ 84] A/B: 8000000000000000 8000000000000000
> PE[ 85] A/B: 8000000000000000 8000000000000000
> PE[ 86] A/B: 8000000000000000 8000000000000000
> PE[ 87] A/B: 8000000000000000 8000000000000000
> PE[ 88] A/B: 8000000000000000 8000000000000000
> PE[ 89] A/B: 8000000000000000 8000000000000000
> PE[ 90] A/B: 8000000000000000 8000000000000000
> PE[ 91] A/B: 8000000000000000 8000000000000000
> PE[ 92] A/B: 8000000000000000 8000000000000000
> PE[ 93] A/B: 8000000000000000 8000000000000000
> PE[ 94] A/B: 8000000000000000 8000000000000000
> PE[ 95] A/B: 8000000000000000 8000000000000000
> PE[ 96] A/B: 8000000000000000 8000000000000000
> PE[ 97] A/B: 8000000000000000 8000000000000000
> PE[ 98] A/B: 8000000000000000 8000000000000000
> PE[ 99] A/B: 8000000000000000 8000000000000000
> PE[100] A/B: 8000000000000000 8000000000000000
> PE[101] A/B: 8000000000000000 8000000000000000
> PE[102] A/B: 8000000000000000 8000000000000000
> PE[103] A/B: 8000000000000000 8000000000000000
> PE[104] A/B: 8000000000000000 8000000000000000
> PE[105] A/B: 8000000000000000 8000000000000000
> PE[106] A/B: 8000000000000000 8000000000000000
> PE[107] A/B: 8000000000000000 8000000000000000
> PE[108] A/B: 8000000000000000 8000000000000000
> PE[109] A/B: 8000000000000000 8000000000000000
> PE[110] A/B: 8000000000000000 8000000000000000
> PE[111] A/B: 8000000000000000 8000000000000000
> PE[112] A/B: 8000000000000000 8000000000000000
> PE[113] A/B: 8000000000000000 8000000000000000
> PE[114] A/B: 8000000000000000 8000000000000000
> PE[115] A/B: 8000000000000000 8000000000000000
> PE[116] A/B: 8000000000000000 8000000000000000
> PE[117] A/B: 8000000000000000 8000000000000000
> PE[118] A/B: 8000000000000000 8000000000000000
> PE[119] A/B: 8000000000000000 8000000000000000
> PE[120] A/B: 8000000000000000 8000000000000000
> PE[121] A/B: 8000000000000000 8000000000000000
> PE[122] A/B: 8000000000000000 8000000000000000
> PE[123] A/B: 8000000000000000 8000000000000000
> PE[124] A/B: 8000000000000000 8000000000000000
> PE[125] A/B: 8000000000000000 8000000000000000
> PE[126] A/B: 8000000000000000 8000000000000000
> PE[127] A/B: 8000000000000000 8000000000000000
> PE[128] A/B: 8000000000000000 8000000000000000
> PE[129] A/B: 8000000000000000 8000000000000000
> PE[130] A/B: 8000000000000000 8000000000000000
> PE[131] A/B: 8000000000000000 8000000000000000
> PE[132] A/B: 8000000000000000 8000000000000000
> PE[133] A/B: 8000000000000000 8000000000000000
> PE[134] A/B: 8000000000000000 8000000000000000
> PE[135] A/B: 8000000000000000 8000000000000000
> PE[136] A/B: 8000000000000000 8000000000000000
> PE[137] A/B: 8000000000000000 8000000000000000
> PE[138] A/B: 8000000000000000 8000000000000000
> PE[139] A/B: 8000000000000000 8000000000000000
> PE[140] A/B: 8000000000000000 8000000000000000
> PE[141] A/B: 8000000000000000 8000000000000000
> PE[142] A/B: 8000000000000000 8000000000000000
> PE[143] A/B: 8000000000000000 8000000000000000
> PE[144] A/B: 8000000000000000 8000000000000000
> PE[145] A/B: 8000000000000000 8000000000000000
> PE[146] A/B: 8000000000000000 8000000000000000
> PE[147] A/B: 8000000000000000 8000000000000000
> PE[148] A/B: 8000000000000000 8000000000000000
> PE[149] A/B: 8000000000000000 8000000000000000
> PE[150] A/B: 8000000000000000 8000000000000000
> PE[151] A/B: 8000000000000000 8000000000000000
> PE[152] A/B: 8000000000000000 8000000000000000
> PE[153] A/B: 8000000000000000 8000000000000000
> PE[154] A/B: 8000000000000000 8000000000000000
> PE[155] A/B: 8000000000000000 8000000000000000
> PE[156] A/B: 8000000000000000 8000000000000000
> PE[157] A/B: 8000000000000000 8000000000000000
> PE[158] A/B: 8000000000000000 8000000000000000
> PE[159] A/B: 8000000000000000 8000000000000000
> PE[160] A/B: 8000000000000000 8000000000000000
> PE[161] A/B: 8000000000000000 8000000000000000
> PE[162] A/B: 8000000000000000 8000000000000000
> PE[163] A/B: 8000000000000000 8000000000000000
> PE[164] A/B: 8000000000000000 8000000000000000
> PE[165] A/B: 8000000000000000 8000000000000000
> PE[166] A/B: 8000000000000000 8000000000000000
> PE[167] A/B: 8000000000000000 8000000000000000
> PE[168] A/B: 8000000000000000 8000000000000000
> PE[169] A/B: 8000000000000000 8000000000000000
> PE[170] A/B: 8000000000000000 8000000000000000
> PE[171] A/B: 8000000000000000 8000000000000000
> PE[172] A/B: 8000000000000000 8000000000000000
> PE[173] A/B: 8000000000000000 8000000000000000
> PE[174] A/B: 8000000000000000 8000000000000000
> PE[175] A/B: 8000000000000000 8000000000000000
> PE[176] A/B: 8000000000000000 8000000000000000
> PE[177] A/B: 8000000000000000 8000000000000000
> PE[178] A/B: 8000000000000000 8000000000000000
> PE[179] A/B: 8000000000000000 8000000000000000
> PE[180] A/B: 8000000000000000 8000000000000000
> PE[181] A/B: 8000000000000000 8000000000000000
> PE[182] A/B: 8000000000000000 8000000000000000
> PE[183] A/B: 8000000000000000 8000000000000000
> PE[184] A/B: 8000000000000000 8000000000000000
> PE[185] A/B: 8000000000000000 8000000000000000
> PE[186] A/B: 8000000000000000 8000000000000000
> PE[187] A/B: 8000000000000000 8000000000000000
> PE[188] A/B: 8000000000000000 8000000000000000
> PE[189] A/B: 8000000000000000 8000000000000000
> PE[190] A/B: 8000000000000000 8000000000000000
> PE[191] A/B: 8000000000000000 8000000000000000
> PE[192] A/B: 8000000000000000 8000000000000000
> PE[193] A/B: 8000000000000000 8000000000000000
> PE[194] A/B: 8000000000000000 8000000000000000
> PE[195] A/B: 8000000000000000 8000000000000000
> PE[196] A/B: 8000000000000000 8000000000000000
> PE[197] A/B: 8000000000000000 8000000000000000
> PE[198] A/B: 8000000000000000 8000000000000000
> PE[199] A/B: 8000000000000000 8000000000000000
> PE[200] A/B: 8000000000000000 8000000000000000
> PE[201] A/B: 8000000000000000 8000000000000000
> PE[202] A/B: 8000000000000000 8000000000000000
> PE[203] A/B: 8000000000000000 8000000000000000
> PE[204] A/B: 8000000000000000 8000000000000000
> PE[205] A/B: 8000000000000000 8000000000000000
> PE[206] A/B: 8000000000000000 8000000000000000
> PE[207] A/B: 8000000000000000 8000000000000000
> PE[208] A/B: 8000000000000000 8000000000000000
> PE[209] A/B: 8000000000000000 8000000000000000
> PE[210] A/B: 8000000000000000 8000000000000000
> PE[211] A/B: 8000000000000000 8000000000000000
> PE[212] A/B: 8000000000000000 8000000000000000
> PE[213] A/B: 8000000000000000 8000000000000000
> PE[214] A/B: 8000000000000000 8000000000000000
> PE[215] A/B: 8000000000000000 8000000000000000
> PE[216] A/B: 8000000000000000 8000000000000000
> PE[217] A/B: 8000000000000000 8000000000000000
> PE[218] A/B: 8000000000000000 8000000000000000
> PE[219] A/B: 8000000000000000 8000000000000000
> PE[220] A/B: 8000000000000000 8000000000000000
> PE[221] A/B: 8000000000000000 8000000000000000
> PE[222] A/B: 8000000000000000 8000000000000000
> PE[223] A/B: 8000000000000000 8000000000000000
> PE[224] A/B: 8000000000000000 8000000000000000
> PE[225] A/B: 8000000000000000 8000000000000000
> PE[226] A/B: 8000000000000000 8000000000000000
> PE[227] A/B: 8000000000000000 8000000000000000
> PE[228] A/B: 8000000000000000 8000000000000000
> PE[229] A/B: 8000000000000000 8000000000000000
> PE[230] A/B: 8000000000000000 8000000000000000
> PE[231] A/B: 8000000000000000 8000000000000000
> PE[232] A/B: 8000000000000000 8000000000000000
> PE[233] A/B: 8000000000000000 8000000000000000
> PE[234] A/B: 8000000000000000 8000000000000000
> PE[235] A/B: 8000000000000000 8000000000000000
> PE[236] A/B: 8000000000000000 8000000000000000
> PE[237] A/B: 8000000000000000 8000000000000000
> PE[238] A/B: 8000000000000000 8000000000000000
> PE[239] A/B: 8000000000000000 8000000000000000
> PE[240] A/B: 8000000000000000 8000000000000000
> PE[241] A/B: 8000000000000000 8000000000000000
> PE[242] A/B: 8000000000000000 8000000000000000
> PE[243] A/B: 8000000000000000 8000000000000000
> PE[244] A/B: 8000000000000000 8000000000000000
> PE[245] A/B: 8000000000000000 8000000000000000
> PE[246] A/B: 8000000000000000 8000000000000000
> PE[247] A/B: 8000000000000000 8000000000000000
> PE[248] A/B: 8000000000000000 8000000000000000
> PE[249] A/B: 8000000000000000 8000000000000000
> PE[250] A/B: 8000000000000000 8000000000000000
> PE[251] A/B: 8000000000000000 8000000000000000
> PE[252] A/B: 8000000000000000 8000000000000000
> PE[253] A/B: 8000000000000000 8000000000000000
> PE[254] A/B: 8000000000000000 8000000000000000
> PE[255] A/B: 8000802309000000 8800000ff3b47380
> EEH: Reset without hotplug activity
> EEH: Notify device drivers the completion of reset
> NMI watchdog: BUG: soft lockup - CPU#19 stuck for 22s!
> [kworker/19:2:2695]
> Modules linked in: kvm_hv kvm binfmt_misc vmx_crypto crc32c_vpmsum
> leds_powernv powernv_rng led_class powernv_op_panel rng_core ib_iser
> rdma_cm iw_cm ib_cm ib_core libiscsi scsi_transport_iscsi autofs4
> CPU: 19 PID: 2695 Comm: kworker/19:2 Tainted: G      D W       4.8.0-
> 11288-gb66484c #785
> Workqueue: events wait_rcu_exp_gp
> task: c0000007f4c10300 task.stack: c0000007f4e88000
> NIP: c00000000015ff28 LR: c000000000137d94 CTR: c0000000000681e0
> REGS: c0000007f4e8b890 TRAP: 0901   Tainted: G      D
> W        (4.8.0-11288-gb66484c)
> MSR: 900000010280b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE,TM[E]>  CR:
> 48002422  XER: 20000000
> CFAR: c00000000015ff30 SOFTE: 1 
> GPR00: c000000000137d94 c0000007f4e8bb10 c000000000dbc100
> 0000000000000020 
> GPR04: c0000000001374a0 c000000000ce1280 0000000000000000
> c000000000dee9a8 
> GPR08: c00000000fe04c00 0000000000000001 c0000007fb4e3680
> 0000000000000000 
> GPR12: c0000000000681e0 c00000000fe04c00 
> NIP [c00000000015ff28] smp_call_function_single+0x98/0x180
> LR [c000000000137d94] wait_rcu_exp_gp+0x294/0xaf0
> Call Trace:
> [c0000007f4e8bb10] [c00000000015ff58]
> smp_call_function_single+0xc8/0x180 (unreliable)
> [c0000007f4e8bb50] [c000000000137d94] wait_rcu_exp_gp+0x294/0xaf0
> [c0000007f4e8bc50] [c0000000000dbab0] process_one_work+0x2a0/0x5a0
> [c0000007f4e8bce0] [c0000000000dbe58] worker_thread+0xa8/0x6a0
> [c0000007f4e8bd80] [c0000000000e41d0] kthread+0x120/0x130
> [c0000007f4e8be30] [c00000000000bf68]
> ret_from_kernel_thread+0x5c/0x74
> Instruction dump:
> 409e0078 e9280030 3d42fff0 394a7580 7d4a4a14 812a0018 71290001
> 41820020 
> 60000000 60420000 7c210b78 7c421378 <812a0018> 71290001 4082fff0
> 7c2004ac 
> NMI watchdog: BUG: soft lockup - CPU#32 stuck for 22s! [swapper/32:0]
> Modules linked in: kvm_hv kvm binfmt_misc vmx_crypto crc32c_vpmsum
> leds_powernv powernv_rng led_class powernv_op_panel rng_core ib_iser
> rdma_cm iw_cm ib_cm ib_core libiscsi scsi_transport_iscsi autofs4
> CPU: 32 PID: 0 Comm: swapper/32 Tainted: G      D W    L  4.8.0-
> 11288-gb66484c #785
> task: c0000017f91fce00 task.stack: c0000017f927c000
> NIP: c00000000012a698 LR: c00000000012a840 CTR: 0000000000000000
> REGS: c000001fffe8b520 TRAP: 0901   Tainted: G      D
> W    L   (4.8.0-11288-gb66484c)
> MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 44004882  XER:
> 00000000
> CFAR: c00000000012a6a4 SOFTE: 1 
> GPR00: c00000000012a840 c000001fffe8b7a0 c000000000dbc100
> c000000ff7ed1400 
> GPR04: 0000000000000011 0000000000000000 0000000000000000
> 0000000000000000 
> GPR08: c000000ff7ed1400 0000000000040000 c0000007f90000b0
> c0000007f9000028 
> GPR12: 0000000000000040 c00000000fe08000 
> NIP [c00000000012a698] __synchronize_hardirq+0x38/0xa0
> LR [c00000000012a840] synchronize_irq+0x30/0xd0
> Call Trace:
> [c000001fffe8b7a0] [c000000000339548] remove_proc_subtree+0x168/0x1c0
> (unreliable)
> [c000001fffe8b7e0] [c00000000012a840] synchronize_irq+0x30/0xd0
> [c000001fffe8b840] [c00000000012b37c] __free_irq+0x12c/0x380
> [c000001fffe8b8d0] [c00000000012b6ec] free_irq+0x7c/0xb0
> [c000001fffe8b900] [c000000000072748] opal_event_shutdown+0x88/0xd0
> [c000001fffe8b940] [c00000000006db8c] opal_shutdown+0x1c/0x90
> [c000001fffe8b970] [c000000000069d00] pnv_shutdown+0x20/0x40
> [c000001fffe8b990] [c000000000026a18] machine_restart+0x38/0x90
> [c000001fffe8ba00] [c0000000000e7538] emergency_restart+0x28/0x40
> [c000001fffe8ba20] [c000000000568c44] sysrq_handle_reboot+0x24/0x40
> [c000001fffe8ba40] [c000000000569490] __handle_sysrq+0xe0/0x250
> [c000001fffe8bae0] [c00000000057f1a0] hvc_poll+0x230/0x380
> [c000001fffe8bb80] [c0000000005803cc] hvc_handle_interrupt+0x1c/0x50
> [c000001fffe8bba0] [c00000000012a2b0]
> __handle_irq_event_percpu+0x90/0x2f0
> [c000001fffe8bc60] [c00000000012a540]
> handle_irq_event_percpu+0x30/0x90
> [c000001fffe8bca0] [c00000000012a5fc] handle_irq_event+0x5c/0xc0
> [c000001fffe8bcd0] [c00000000012f5b4] handle_level_irq+0xd4/0x1e0
> [c000001fffe8bd00] [c000000000128e44] generic_handle_irq+0x44/0x70
> [c000001fffe8bd20] [c000000000072600] opal_handle_events+0xd0/0x120
> [c000001fffe8bd70] [c0000000000726a8] opal_interrupt+0x38/0x50
> [c000001fffe8bda0] [c00000000012a2b0]
> __handle_irq_event_percpu+0x90/0x2f0
> [c000001fffe8be60] [c00000000012a540]
> handle_irq_event_percpu+0x30/0x90
> [c000001fffe8bea0] [c00000000012a5fc] handle_irq_event+0x5c/0xc0
> [c000001fffe8bed0] [c00000000012f9f0] handle_fasteoi_irq+0xe0/0x290
> [c000001fffe8bf00] [c000000000128e44] generic_handle_irq+0x44/0x70
> [c000001fffe8bf20] [c00000000001641c] __do_irq+0x8c/0x210
> [c000001fffe8bf90] [c000000000027ed4] call_do_irq+0x14/0x24
> [c0000017f927fac0] [c000000000016634] do_IRQ+0x94/0x140
> [c0000017f927fb10] [c000000000008e8c]
> hardware_interrupt_common+0x10c/0x180
> --- interrupt: 501 at arch_local_irq_restore+0x5c/0x90
>     LR = arch_local_irq_restore+0x40/0x90
> [c0000017f927fe00] [c000000000155e44]
> tick_broadcast_oneshot_control+0x54/0x70 (unreliable)
> [c0000017f927fe20] [c0000000007b6da4] cpuidle_enter_state+0x1b4/0x440
> [c0000017f927fe80] [c000000000114da4] call_cpuidle+0x44/0x90
> [c0000017f927fea0] [c000000000115138] cpu_startup_entry+0x1f8/0x320
> [c0000017f927ff20] [c000000000043540] start_secondary+0x320/0x340
> [c0000017f927ff90] [c00000000000b56c]
> start_secondary_prolog+0x10/0x14
> Instruction dump:
> 7c0802a6 fba1ffe8 fbc1fff0 3ba3018c fbe1fff8 7c7e1b78 f8010010
> f821ffc1 
> 48000010 60420000 7c210b78 7c421378 <e93e0128> 81290000 75290004
> 4082ffec 