<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hello Oliver,<div class=""><br class=""><div><blockquote type="cite" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">That's very odd. rcu_reclaim_tiny() is probably being folded into</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">rcu_process_callbacks() by the compiler and the crash occurs when</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">branching to the callback function from the rcu_head</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">(</span><a href="https://elixir.bootlin.com/linux/v6.18-rc5/source/kernel/rcu/tiny.c#L95" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">https://elixir.bootlin.com/linux/v6.18-rc5/source/kernel/rcu/tiny.c#L95</a><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">).</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">That said, the "callback" address it branched to (0xc00c000000000000)</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">is actually the base of the vmemmap (i.e. the struct page array) so I</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">doubt that's actually the callback address stored in the rcu_head. You</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">can use xmon to dump the registers and examine memory to confirm this.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">It's hard to say why this is happening, but it's pretty likely to</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">either be the compiler optimizing away code you'd prefer to keep or a</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">bug in the core itself.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">I'd compare the disasm of rcu_process_callbacks() with and without</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">your wait loop added and see how the emitted code changes. If adding</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">the loop changes nothing then it might be a logic bug in microwatt</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">itself or some other timing induced problem.</span><br class=""></blockquote><br class=""></div><div>I just replaced the wait-loop with the following (and it also prevents the</div><div>kernel panic):</div><div><br class=""></div><div>> <span class="Apple-tab-span" style="white-space: pre;"> </span>volatile uint32_t dumb = 0xFEDC0000;</div><div>><span class="Apple-tab-span" style="white-space:pre"> </span>dumb++;</div><br class=""></div><div class="">- Here is the disasm of the `<span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">rcu_process_callbacks</span>` with the injected code:</div><div class=""><br class=""></div><div class=""><div class="">c00000000008c5e8 <rcu_process_callbacks>:</div><div class="">c00000000008c5e8: 32 00 4c 3c addis r2,r12,50</div><div class="">c00000000008c5ec: 18 ca 42 38 addi r2,r2,-13800</div><div class="">c00000000008c5f0: a6 02 08 7c mflr r0</div><div class="">c00000000008c5f4: a1 c3 23 48 bl c0000000002c8994 <_savegpr0_27></div><div class="">c00000000008c5f8: a1 ff 21 f8 stdu r1,-96(r1)</div><div class="">c00000000008c5fc: fa 00 6d 88 lbz r3,250(r13)</div><div class="">c00000000008c600: 01 00 69 60 ori r9,r3,1</div><div class="">c00000000008c604: fa 00 2d 99 stb r9,250(r13)</div><div class="">c00000000008c608: 2b 01 42 3d addis r10,r2,299</div><div class="">c00000000008c60c: 2b 01 22 3d addis r9,r2,299</div><div class="">c00000000008c610: d0 3c 4a e9 ld r10,15568(r10)</div><div class="">c00000000008c614: c8 3c 29 39 addi r9,r9,15560</div><div class="">c00000000008c618: 00 48 2a 7c cmpd r10,r9</div><div class="">c00000000008c61c: 14 00 82 40 bne c00000000008c630 <rcu_process_callbacks+0x48></div><div class="">c00000000008c620: 6d d6 f8 4b bl c000000000019c8c <arch_local_irq_restore+0x8></div><div class="">c00000000008c624: 00 00 00 60 nop</div><div class="">c00000000008c628: 60 00 21 38 addi r1,r1,96</div><div class="">c00000000008c62c: b8 c3 23 48 b c0000000002c89e4 <_restgpr0_27></div><div class="">c00000000008c630: 00 00 0a e9 ld r8,0(r10)</div><div class="">c00000000008c634: 2b 01 e2 3c addis r7,r2,299</div><div class="">c00000000008c638: c8 3c e7 eb ld r31,15560(r7)</div><div class="">c00000000008c63c: c8 3c 07 f9 std r8,15560(r7)</div><div class="">c00000000008c640: 00 00 00 39 li r8,0</div><div class="">c00000000008c644: 00 00 0a f9 std r8,0(r10)</div><div class="">c00000000008c648: 2b 01 42 3d addis r10,r2,299</div><div class="">c00000000008c64c: d8 3c 0a e9 ld r8,15576(r10)</div><div class="">c00000000008c650: 2b 01 42 3d addis r10,r2,299</div><div class="">c00000000008c654: d0 3c 4a e9 ld r10,15568(r10)</div><div class="">c00000000008c658: 00 50 28 7c cmpd r8,r10</div><div class="">c00000000008c65c: 08 00 82 40 bne c00000000008c664 <rcu_process_callbacks+0x7c></div><div class="">c00000000008c660: 10 00 29 f9 std r9,16(r9)</div><div class="">c00000000008c664: 08 00 29 f9 std r9,8(r9)</div><div class="">c00000000008c668: dc fe 80 3f lis r28,-292 <<<<====== "THIS IS INJECTED CODE"</div><div class="">c00000000008c66c: 21 d6 f8 4b bl c000000000019c8c <arch_local_irq_restore+0x8></div><div class="">c00000000008c670: 00 00 00 60 nop</div><div class="">c00000000008c674: 00 00 60 3b li r27,0</div><div class="">c00000000008c678: 00 00 3f 2c cmpdi r31,0</div><div class="">c00000000008c67c: ac ff 82 41 beq c00000000008c628 <rcu_process_callbacks+0x40></div><div class="">c00000000008c680: 00 00 df eb ld r30,0(r31)</div><div class="">c00000000008c684: 00 00 3e 2c cmpdi r30,0</div><div class="">c00000000008c688: 08 00 82 41 beq c00000000008c690 <rcu_process_callbacks+0xa8></div><div class="">c00000000008c68c: 2c f2 00 7c dcbtct 0,r30</div><div class="">c00000000008c690: 2c 00 81 93 stw r28,44(r1)</div><div class="">c00000000008c694: 08 00 bf eb ld r29,8(r31)</div><div class="">c00000000008c698: 00 00 3d 2c cmpdi r29,0</div><div class="">c00000000008c69c: 2c 00 21 81 lwz r9,44(r1)</div><div class="">c00000000008c6a0: 01 00 29 39 addi r9,r9,1</div><div class="">c00000000008c6a4: 2c 00 21 91 stw r9,44(r1)</div><div class="">c00000000008c6a8: 10 00 82 40 bne c00000000008c6b8 <rcu_process_callbacks+0xd0></div><div class="">c00000000008c6ac: 78 fb e3 7f mr r3,r31</div><div class="">c00000000008c6b0: 15 f8 03 48 bl c0000000000cbec4 <kmem_dump_obj+0x8></div><div class="">c00000000008c6b4: 00 00 00 60 nop</div><div class="">c00000000008c6b8: 08 00 7f fb std r27,8(r31)</div><div class="">c00000000008c6bc: 78 fb e3 7f mr r3,r31</div><div class="">c00000000008c6c0: a6 03 a9 7f mtctr r29</div><div class="">c00000000008c6c4: 78 eb ac 7f mr r12,r29</div><div class="">c00000000008c6c8: 18 00 41 f8 std r2,24(r1)</div><div class="">c00000000008c6cc: 78 f3 df 7f mr r31,r30</div><div class="">c00000000008c6d0: 21 04 80 4e bctrl</div><div class="">c00000000008c6d4: 18 00 41 e8 ld r2,24(r1)</div><div class="">c00000000008c6d8: a0 ff ff 4b b c00000000008c678 <rcu_process_callbacks+0x90></div></div><div class=""><br class=""></div><div class="">- And here is the disasm without the injected code:</div><div class=""><br class=""></div><div class=""><div class="">c00000000008c5e8 <rcu_process_callbacks>:</div><div class="">c00000000008c5e8: 32 00 4c 3c addis r2,r12,50</div><div class="">c00000000008c5ec: 18 ca 42 38 addi r2,r2,-13800</div><div class="">c00000000008c5f0: a6 02 08 7c mflr r0</div><div class="">c00000000008c5f4: 95 c3 23 48 bl c0000000002c8988 <_savegpr0_28></div><div class="">c00000000008c5f8: c1 ff 21 f8 stdu r1,-64(r1)</div><div class="">c00000000008c5fc: fa 00 6d 88 lbz r3,250(r13)</div><div class="">c00000000008c600: 01 00 69 60 ori r9,r3,1</div><div class="">c00000000008c604: fa 00 2d 99 stb r9,250(r13)</div><div class="">c00000000008c608: 2b 01 42 3d addis r10,r2,299</div><div class="">c00000000008c60c: 2b 01 22 3d addis r9,r2,299</div><div class="">c00000000008c610: d0 3c 4a e9 ld r10,15568(r10)</div><div class="">c00000000008c614: c8 3c 29 39 addi r9,r9,15560</div><div class="">c00000000008c618: 00 48 2a 7c cmpd r10,r9</div><div class="">c00000000008c61c: 14 00 82 40 bne c00000000008c630 <rcu_process_callbacks+0x48></div><div class="">c00000000008c620: 6d d6 f8 4b bl c000000000019c8c <arch_local_irq_restore+0x8></div><div class="">c00000000008c624: 00 00 00 60 nop</div><div class="">c00000000008c628: 40 00 21 38 addi r1,r1,64</div><div class="">c00000000008c62c: ac c3 23 48 b c0000000002c89d8 <_restgpr0_28></div><div class="">c00000000008c630: 00 00 0a e9 ld r8,0(r10)</div><div class="">c00000000008c634: 2b 01 e2 3c addis r7,r2,299</div><div class="">c00000000008c638: c8 3c e7 eb ld r31,15560(r7)</div><div class="">c00000000008c63c: c8 3c 07 f9 std r8,15560(r7)</div><div class="">c00000000008c640: 00 00 00 39 li r8,0</div><div class="">c00000000008c644: 00 00 0a f9 std r8,0(r10)</div><div class="">c00000000008c648: 2b 01 42 3d addis r10,r2,299</div><div class="">c00000000008c64c: d8 3c 0a e9 ld r8,15576(r10)</div><div class="">c00000000008c650: 2b 01 42 3d addis r10,r2,299</div><div class="">c00000000008c654: d0 3c 4a e9 ld r10,15568(r10)</div><div class="">c00000000008c658: 00 50 28 7c cmpd r8,r10</div><div class="">c00000000008c65c: 08 00 82 40 bne c00000000008c664 <rcu_process_callbacks+0x7c></div><div class="">c00000000008c660: 10 00 29 f9 std r9,16(r9)</div><div class="">c00000000008c664: 08 00 29 f9 std r9,8(r9)</div><div class="">c00000000008c668: 00 00 80 3b li r28,0</div><div class="">c00000000008c66c: 21 d6 f8 4b bl c000000000019c8c <arch_local_irq_restore+0x8></div><div class="">c00000000008c670: 00 00 00 60 nop</div><div class="">c00000000008c674: 00 00 3f 2c cmpdi r31,0</div><div class="">c00000000008c678: b0 ff 82 41 beq c00000000008c628 <rcu_process_callbacks+0x40></div><div class="">c00000000008c67c: 00 00 df eb ld r30,0(r31)</div><div class="">c00000000008c680: 00 00 3e 2c cmpdi r30,0</div><div class="">c00000000008c684: 08 00 82 41 beq c00000000008c68c <rcu_process_callbacks+0xa4></div><div class="">c00000000008c688: 2c f2 00 7c dcbtct 0,r30</div><div class="">c00000000008c68c: 08 00 bf eb ld r29,8(r31)</div><div class="">c00000000008c690: 00 00 3d 2c cmpdi r29,0</div><div class="">c00000000008c694: 10 00 82 40 bne c00000000008c6a4 <rcu_process_callbacks+0xbc></div><div class="">c00000000008c698: 78 fb e3 7f mr r3,r31</div><div class="">c00000000008c69c: 19 f8 03 48 bl c0000000000cbeb4 <kmem_dump_obj+0x8></div><div class="">c00000000008c6a0: 00 00 00 60 nop</div><div class="">c00000000008c6a4: 08 00 9f fb std r28,8(r31)</div><div class="">c00000000008c6a8: 78 fb e3 7f mr r3,r31</div><div class="">c00000000008c6ac: a6 03 a9 7f mtctr r29</div><div class="">c00000000008c6b0: 78 eb ac 7f mr r12,r29</div><div class="">c00000000008c6b4: 18 00 41 f8 std r2,24(r1)</div><div class="">c00000000008c6b8: 78 f3 df 7f mr r31,r30</div><div class="">c00000000008c6bc: 21 04 80 4e bctrl</div><div class="">c00000000008c6c0: 18 00 41 e8 ld r2,24(r1)</div><div class="">c00000000008c6c4: b0 ff ff 4b b c00000000008c674 <rcu_process_callbacks+0x8c></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Sounds like some sorta compiler optimizations, right?</div></div><div class=""><br class=""></div><div class="">Bests,</div><div class="">Manili</div></body></html>