Firmware update error: Cannot UBI create volume

[rgrs]

Hi All,

I have a strange behavior where only one image is visible in webUI.
I get UBI "no space left" errors when I try to update/flash new image.
I dont have steps to reproduce and dont know how my BMC ended up in this state.

How do I recover from this error? /proc/mtd shows all partitions/volumes properly.
Is there any way to recover alternate partition without erasing SPI from uboot?

Logs:

Dec 17 09:26:21 obmc phosphor-version-software-manager[1531]: Untaring
Dec 17 09:26:21 obmc phosphor-mapper[1300]: Found invalid association on path /xyz/openbmc_project/software/c35db0b8
Dec 17 09:26:21 obmc phosphor-mapper[1300]: Found invalid association on path /xyz/openbmc_project/software/c35db0b8
Dec 17 09:26:21 obmc phosphor-mapper[1300]: Found invalid association on path /xyz/openbmc_project/software/c35db0b8
Dec 17 09:26:22 obmc phosphor-image-updater[1464]: BMC image activating - BMC reboots are disabled.
Dec 17 09:26:22 obmc systemd[1]: Starting Enable a guard that blocks BMC reboot...
Dec 17 09:26:23 obmc systemd[1]: reboot-guard-enable.service: Succeeded.
Dec 17 09:26:23 obmc systemd[1]: Started Enable a guard that blocks BMC reboot.
Dec 17 09:26:24 obmc systemd[1]: Created slice system-obmc\x2dflash\x2dbmc\x2dubiro.slice.
Dec 17 09:26:24 obmc systemd[1]: Starting Store read-only images c35db0b8 to BMC storage...
Dec 17 09:26:24 obmc systemd[1]: Starting Create BMC read-write ubi volume...
Dec 17 09:26:25 obmc systemd[1]: obmc-flash-bmc-ubirw.service: Succeeded.
Dec 17 09:26:25 obmc systemd[1]: Started Create BMC read-write ubi volume.
Dec 17 09:26:27 obmc kernel: ubi4 error: ubi_create_volume: not enough PEBs, only 315 available
Dec 17 09:26:27 obmc kernel: ubi4 error: ubi_create_volume: cannot create volume 0, error -28
Dec 17 09:26:27 obmc obmc-flash-bmc[2205]: ubimkvol: error!: cannot UBI create volume
Dec 17 09:26:27 obmc obmc-flash-bmc[2205]:           error 28 (No space left on device)
Dec 17 09:26:27 obmc systemd[1]: obmc-flash-bmc-ubiro at c35db0b8.service: Main process exited, code=exited, status=255/EXCN
Dec 17 09:26:27 obmc systemd[1]: obmc-flash-bmc-ubiro at c35db0b8.service: Failed with result 'exit-code'.
Dec 17 09:26:27 obmc systemd[1]: Failed to start Store read-only images c35db0b8 to BMC storage.
Dec 17 09:26:27 obmc phosphor-image-updater[1464]: BMC activation has ended - BMC reboots are re-enabled.
Dec 17 09:26:27 obmc systemd[1]: obmc-flash-bmc-ubiro at c35db0b8.service: Triggering OnFailure= dependencies.
Dec 17 09:26:27 obmc systemd[1]: Starting Removes the guard that blocks BMC reboot...
Dec 17 09:26:27 obmc systemd[1]: Created slice system-obmc\x2dflash\x2dbmc\x2dubiro\x2dremove.slice.
Dec 17 09:26:27 obmc systemd[1]: Starting Deletes read-only and kernel ubi volume c35db0b8...
Dec 17 09:26:28 obmc systemd[1]: reboot-guard-disable.service: Succeeded.
Dec 17 09:26:28 obmc systemd[1]: Started Removes the guard that blocks BMC reboot.
Dec 17 09:26:28 obmc systemd[1]: obmc-flash-bmc-ubiro-remove at c35db0b8.service: Succeeded.
Dec 17 09:26:28 obmc systemd[1]: Started Deletes read-only and kernel ubi volume c35db0b8.
root at obmc:~#
root at obmc:~#

root at obmc:~# ubinfo
UBI version:                    1
Count of UBI devices:           2
UBI control device major/minor: 10:59
Present UBI devices:            ubi0, ubi4

root at obmc:~# ubinfo -d 0 -n 0
Volume ID:   0 (on ubi0)
Type:        static
Alignment:   1
Size:        293 LEBs (19164544 bytes, 18.2 MiB)
Data bytes:  19128320 bytes (18.2 MiB)
State:       OK
Name:        rofs-956f13ca
Character device major/minor: 246:1
root at obmc:~# ubinfo -d 0 -n 1
Volume ID:   1 (on ubi0)
Type:        static
Alignment:   1
Size:        43 LEBs (2812544 bytes, 2.6 MiB)
Data bytes:  2767008 bytes (2.6 MiB)
State:       OK
Name:        kernel-956f13ca
Character device major/minor: 246:2
root at obmc:~# ubinfo -d 0 -n 2
Volume ID:   2 (on ubi0)
Type:        dynamic
Alignment:   1
Size:        97 LEBs (6344576 bytes, 6.0 MiB)
State:       OK
Name:        rwfs
Character device major/minor: 246:3
root at obmc:~# ubinfo -d 0 -n 3
ubinfo: error!: cannot get information about UBI volume 3 on ubi0
        error 2 (No such file or directory)

root at obmc:~# ubinfo -d 4 -n 0
ubinfo: error!: cannot get information about UBI volume 0 on ubi4
        error 2 (No such file or directory)
root at obmc:~# ubinfo -d 4 -n 1
Volume ID:   1 (on ubi4)
Type:        static
Alignment:   1
Size:        43 LEBs (2812544 bytes, 2.6 MiB)
Data bytes:  2767008 bytes (2.6 MiB)
State:       OK
Name:        kernel-7e7e1530
Character device major/minor: 245:2
root at obmc:~# ubinfo -d 4 -n 2
Volume ID:   2 (on ubi4)
Type:        dynamic
Alignment:   1
Size:        97 LEBs (6344576 bytes, 6.0 MiB)
State:       OK
Name:        rwfs
Character device major/minor: 245:3
root at obmc:~# ubinfo -d 4 -n 3
Volume ID:   3 (on ubi4)
Type:        static
Alignment:   1
Size:        43 LEBs (2812544 bytes, 2.6 MiB)
Data bytes:  2767008 bytes (2.6 MiB)
State:       OK
Name:        kernel-956f13ca
Character device major/minor: 245:4
root at obmc:~#
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ozlabs.org/pipermail/openbmc/attachments/20191217/735e7670/attachment-0001.htm>