OpenBMC replacing AMI AST2500 BMC fw on Gigabyte MC12-LE0 - questions

[Andrew Jeffery]

On Tue, 2024-05-28 at 20:37 +0200, Johannes Truschnigg wrote:
> Hi again! :)
> 
> On Tue, May 28, 2024 at 09:56:38AM +0930, Andrew Jeffery wrote:
> 
> > > So I guess I have at least six questions now:
> > > 
> > > 1.) What happened when the kernel called it quits, u-boot reloaded and decided
> > > to format some of its flash?
> > 
> > Not sure, but you'd do well to boot a kernel that doesn't try to mount
> > partitions from the flash.
> 
> Understood. Is that always the case for OpenBMC kernel images with default
> config?

Depends on what you mean by default. Which platform did you build for?

> 
> The subsequent attempts using a more up-to-date release of gigaflash worked,
> and while fooling around with culvert and gigaflash (to check if gigaflash
> always produced the same result and to find out if the switches that Gigabyte's
> provided wrapper shellscript used to updaye fw actually made a difference), I
> noticed that at some point, culvert suddenly could (correctly, bit-by-bit
> identically!) dump the ROM, too!

Right, culvert is likely missing some trick with initialising the flash
controller. I'm trying to understand what that might be :)

> 
> For posterity, this is what it looked like when gigaflash successfully dumped
> the image for me:
> 
> ```
> # Tools/gigaflash_x64 -dump test.img -cs 0 -2500
> gigaflash v2.0.10
> Failed to connect BMC, try to dump image!
> --- Dump image from BMC...
> Find ASPEED Device 1a03:2000 on 4:0.0 
> MMIO Virtual Address: a4e84000 
> Relocate IO Base: f000 
> Found ASPEED Device 1a03:2500 rev. 41 
> Static Memory Controller Information: 
> CS0 Flash Type is SPI 
> CS1 Flash Type is SPI 
> CS2 Flash Type is SPI 
> CS3 Flash Type is NOR 
> CS4 Flash Type is NOR 
> Boot CS is 0 
> Option Information: 
> CS: 0 
> Flash Type: SPI 
> [Warning] Don't AC OFF or Reboot System During BMC Firmware Update!! 
> [SOCFLASH] Flash ID : 20ba20 
> Find Flash Chip #1: Numonyx N25Q512 
> Backup Flash Chip O.K.                 
> --- Dump image finished
> --- Wait 90 secs for BMC ready...
> ```
> 
> 
> I tried to arrive at a minimal reproducer on what I had to do after a power
> cut to get `culvert read firmware` to work, and in the end, it seems to be
> that one needs to run
> 
> `gigaflash -dump some_filename -2500`
> 
> ... once, and *only then* culvert can read the data as well.

Okay, handy to know. That jives with my comment above.

> 
> I did also try to get mmiotrace, which you mentioned in the GH issue (I have
> never worked with it before, but I am familiar with ftrace, so I am not 100%
> but reasonably certain that I did not hold it wrong), to work, but I could not
> make it emit any tracing data while either culvert or gigaflash were dumping
> ROM. Only when (un)loading the `ast` driver, lots of tracing data could be
> collected.
> 
> I do have strace capture of `gigaflash` running for the first time after a
> reboot, but all the juicy bits seem to hide behind mmap() anyway, so I will
> only provide it upon request.

Well, consider this a request :D What it's mapping is of interest to
me, along with what it's opening more broadly, and any calls to
ioperm() or iopl().

> 
> Then it dawned on me that `culvert probe` before and after the gigaflash
> "unlock" might hold the key, and when diffing two runs' results (one before,
> when culvert could not dump anything, and one after, when it very well could)
> yielded this result:
> 
> ```
> $ diff -u1 culvert_probe_initial culvert_probe_after_dump 
> --- culvert_probe_initial
> +++ culvert_probe_after_dump
> @@ -121,4 +121,4 @@
>  [*] ahb_readl: 0x1e6e2070: 0xf120f286
> -[*] ahb_readl: 0x1e789100: 0x00000000
> -ilpc:  Permissive
> +[*] ahb_readl: 0x1e789100: 0x00000040
> +ilpc:  Restricted
>  [*] ahb_readl: 0x1e6e2070: 0xf120f286
> ```

So something (it's unclear to me whether the BMC has rebooted in your
test here) has set the bit to disable writes via the iLPC2AHB bridge.

The reason I ask for the strace above is the mmiotrace output in the
github issue only shows a couple of accesses to the P2A. I'm wondering
whether gigaflash is exploiting iLPC2AHB and LPC FW/MEM cycles instead
(or perhaps I need to improve my understanding of the limits of
mmiotrace).

> 
> The result is both very encouraging and also disappointing, because as you
> intially theorized, the BMC boots fine with OpenBMC flashed onto it - but none
> of its host system manmagement capabilities actually work. I do have the
> vanilla OpenBMC web application available now, with DHCP, DNS, NTP et al.
> working fine for the BMC, can log in via SSH, but all the peripherals the BMC
> ought to be able to manage and hook into do not work at present.

Yes, this is expected. Configuring the BMC applications to match the
integration into the board is the biggest challenge.

> 
> 
> The next step that I would want to take is that I find a way to revert to the
> BMC stock fw with having only OpenBMC running, since the host apparently cannot
> boot any more (same situation as with the stock BMC fw when u-boot had
> initialized, but no BMC system was allowed to boot up, afaict - the power
> button/contacts just do nothing) in this state. After that, I would like to
> establish a sane (and hopefully easy) way to convert the board's BMC firmware
> from OpenBMC to stock, and vice versa.

The least-effort hack would be to place the stock firmware at
/run/initramfs/image-bmc and reboot OpenBMC - this will write the stock
firmware image back to the flash for you.

> 
> Once I have established a surefire and straightfoward way to do what I have
> done in such meandering and clumsy attempts, I would like to learn more about
> how the "M" is actually put into this whole "BMC" thing, and see how far I can
> take that. The stock fw has some interesting description files regarding i2c
> configs that might come in handy, but I am just not educated enough (yet, I
> hope) to make real sense of it :)

Yeah, the I2C configs will certainly help.

Breaking into the stock firmware on the hardware and tracing things
like GPIO accesses would go a long way. With the tools at hand it
shouldn't be too difficult :)

> 
> Can you perhaps offer me advice on how to flash arbitrary new SPI flash
> contents from either OpenBMC's u-boot or an OpenBMC root shell, or what I would
> need to look at in detail to learn how to do that?

See the comment above regarding /run/initramfs/image-bmc. However you
can also boot to an initrd and use flashcp. The main thing is you need
to make sure no other accesses to the mtd device are taking place
(hence suggesting the initrd environment).

Andrew