Wedge400 (AST2520) OpenBMC stuck at reboot

[Tao Ren]

Hi Konstantin,

Thanks for the sharing. The watchdog control logic in the script is
similar to aspeed_wdt_restart(), but the good part is: system is still
reachable if watchdog cannot reset the system successfully.

Hi Ryan,

Have you ever seen the problem in your environment? Looks like it is
affecting multiple ASPEED platforms. Any suggestions?

BTW, I'm running Linux 5.15 in Wedge400 AST2520A2 OpenBMC.


Cheers,

Tao

On Thu, Sep 22, 2022 at 10:59:12AM +0300, Konstantin Klubnichkin wrote:
> <div>- все</div><div> </div><div>Hello!</div><div> </div><div>I've faced this issue.</div><div>Finally my solution was to modify shutdown script:</div><div> </div><div>======================================================</div><div><div># tcsattr(tty, TIOCDRAIN, mode) to drain tty messages to console   </div><div>test -t 1 && stty cooked 0<&1                                      </div><div>                                                                   </div><div>echo "Syncing..."                                                  </div><div>sync || :                                          </div><div>sync || :                          </div><div>sync || :                           </div><div>                                       </div><div>echo "Stopping WDTs..."                                                                                      </div><div>rev=$(ast_getrev || :)              </div><div>if [ "$rev" = "G5" ]; then           </div><div>        devmem 0x1e78500c 32 0 || :              </div><div>        devmem 0x1e78502c 32 0 || :  </div><div>        devmem 0x1e78504c 32 0 || :            </div><div>fi                                             </div><div>if [ "$rev" = "G6" ]; then                                         </div><div>        devmem 0x1e78500c 32 0 || :                                </div><div>        devmem 0x1e78504c 32 0 || :                                </div><div>        devmem 0x1e78508c 32 0 || :                                </div><div>        devmem 0x1e7850cc 32 0 || :                                </div><div>        devmem 0x1e78510c 32 0 || :                                </div><div>        devmem 0x1e78514c 32 0 || :                                </div><div>        devmem 0x1e78518c 32 0 || :                                </div><div>        devmem 0x1e7851cc 32 0 || :                                </div><div>fi                                                                 </div><div>                                                                   </div><div>sled_hb_hb || :                                                    </div><div>                                                                   </div><div>echo "Setting up WDT1 for ARM reboot"                              </div><div># Set timeout to 5 seconds                                         </div><div>devmem 0x1e785004 32 0x4c4b40 || :                                 </div><div># Load counter reload value to counter register                    </div><div>devmem 0x1e785008 32 0x4755 || :                                   </div><div># Enable WDT1, reset ARM core only, use first flash (AST2500 only),</div><div># disable interrupt,  use 1MHz clock (AST2500 only)</div><div>devmem 0x1e78500c 32 0x53 || :                          </div><div>                                            </div><div>echo -n "WDT1CR " || :                      </div><div>devmem 0x1e78500c || :                      </div><div>                                            </div><div>echo "Last heart beats following..."                            </div><div>                                                                </div><div>while true; do                                                  </div><div>        echo "KNOCK knock..."                                   </div><div>        sleep 1                                                 </div><div>done                                                                                                                                                                               </div><div> </div><div>echo "WARNING!!!! ZOMBIE ATTACK!!!" </div><div>                                                              </div><div># Execute the command systemd told us to ...                    </div><div>if test -d /oldroot  && test "$1"                               </div><div>then                                                            </div><div>        if test "$1" = kexec                                    </div><div>        then                                                    </div><div>                $1 -f -e                                        </div><div>        else                                                    </div><div>                $1 -f                                           </div><div>        fi                                                      </div><div>fi    </div></div><div><div>======================================================</div></div><div> </div><div>22.09.2022, 01:09, "Tao Ren" <rentao.bupt at gmail.com>:</div><blockquote><p>Hi there,<br /><br />Recently I noticed a few Wedge400 (AST2520A2) units stuck after "reboot"<br />command. It's hard to reproduce (affecting ~1 out of 1,000 units), but<br />once it happens, I have to power cycle the chassis to recover OpenBMC.<br /><br />I checked aspeed_wdt.c and manually played with watchdog registers, but<br />everything looks normal to me. Did anyone hit the similar error before?<br />Any suggestions which area I should look into?<br /><br />Below are the last few lines of logs before OpenBMC hangs:<br /><br />bmc-oob login:<br />INIT: Sending processes configured via /etc/inittab the TERM signal<br />Stopping OpenBSD Secure Shell server: sshdstopped /usr/sbin/sshd (pid 7397 1189)<br />Stopping ntpd: done<br />stopping rsyslogd ... done<br />Stopping random number generator daemon.<br />Deconfiguring network interfaces... done.<br />Sending all processes the TERM signal...<br />rackmond[1747]: Got request exit[ 528.383133] watchdog: watchdog0: watchdog did not stop!<br />Sending all processes the KILL signal...<br />Unmounting remote filesystems...<br />Deactivating swap...<br />Unmounting local filesystems...<br />Rebooting... [ 529.725009] reboot: Restarting system<br /><br /><br />Cheers,<br /><br />Tao</p></blockquote><div> </div><div> </div><div>-- </div><div>Best regards,</div><div>Konstantin Klubnichkin,</div><div>lead firmware engineer,</div><div>server hardware R&D group,</div><div>Yandex Moscow office.</div><div>tel: +7-903-510-33-33</div><div> </div>