boot failure when read-write fs is full

[Rohit Pai]

Hello All,

Currently I am investigating the bootup failures which I see on our 0penBmc based boards when the rw-fs is full.
The rw-fs can become full because of many reasons. One example being too frequent bmc dump creation which are stored in rw-fs.

I have allocated 16MB for the read-write fs and part of the init sequence there is overlay file system which is being mounted as the root-fs which combines the ro-fs and the rw-fs.


mount<https://grok.openbmc.org/s?defs=mount&project=openbmc> -t overlay<https://grok.openbmc.org/s?defs=overlay&project=openbmc> -o lowerdir<https://grok.openbmc.org/s?defs=lowerdir&project=openbmc>=$rodir<https://grok.openbmc.org/s?defs=%24rodir&project=openbmc>,upperdir<https://grok.openbmc.org/s?defs=upperdir&project=openbmc>=$upper<https://grok.openbmc.org/s?defs=%24upper&project=openbmc>,workdir<https://grok.openbmc.org/s?defs=workdir&project=openbmc>=$work<https://grok.openbmc.org/s?defs=%24work&project=openbmc> cow<https://grok.openbmc.org/s?defs=cow&project=openbmc> /root<https://grok.openbmc.org/s?defs=root&project=openbmc>

Above 'mount overlay' command fails with the below error when the upperdir which the rw-fs is full.


chroot: can't execute '/bin/sh': No such file or directory

Unable to confirm /sbin/init is an executable non-empty file

in merged file system mounted at /root.

Change Root test failed!

Fatal error, triggering kernel panic!

Basically, when the 'overlayfs' fails there is no rootfs mounted on the /. So, I am thinking the subsequent init sequence fails.

I am very much interested in knowing if anyone has any thoughts on solving this issue or has encountered and already found some existing solutions.
One solution which is in my mind is to capture the failure of mount overlay command and do a self-clean-up procedure on the rw-fs with a white-list policy.
Thanks for any kind input.

Regards,
Rohit PAI



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ozlabs.org/pipermail/openbmc/attachments/20220411/43ea9b05/attachment-0001.htm>