[OpenPower-Firmware] Questions about core bring-up

[Krystian Hebel]

Hello,

during our struggle with making a port of coreboot for Talos II we hit a
dead end in istep 16.1. Even when using a 1:1 copy of HOMER produced by
Hostboot (which has different HRMOR, but we haven't got to a point where
it makes a difference) we cannot get through deadman loop. It seems that
the core is woken up and executes its self restore code, but then it hits
another 'stop' instruction [1] from which it doesn't return. At that point
PPM_SSHOTR has a value of 0x8fbfb00000000000 for cores, 0x0bbbf00000000000
for cache chiplets. I can manually wake threads by doing 'pdbg stop'
followed by SRESET through DIRECT_CONTROLS and, given that this was done
fast enough, code properly sends a message to stop the timer to SBE;
otherwise checkstop is generated (I've masked it for easier debugging).

First question: what wakes up the core after the first 'stop' (SBE, I
presume?) and what should wake it after self restore?

I managed to get SGPE logs (default verbosity) as described in comment in
script I prepared [2] and got this:

     Initializing External Interrupt Routing Registers
     Setup: Clear Type 0,2,3,5,6,7 and ipi_lo_3 interrupts
     Setup: Prepare CME[0x8000] to be Booted
     Setup: Booting CME in Hostboot Mode
     Setup: Remove Hostboot Istep4 Special Wakeup
     Setup: Unmask Type 0,2,3,5,6 and ipi_lo_3 interrupts, keep type7 
masked as unused
     Setup: SGPE STOP READY
     Core[1] sent PIG Type[3] with Payload [0xF]
     Core Request Entry Confirmed
     Unblock Entry Thread
     DB2 MessageID 2(Rclk Entry) sent to core 1
     Setup: Entry done. Enable Type0/2/3/6 Interrupt
     Core[1] sent PIG Type[0] with Payload [0x10B]
     Core Request Entry Allowed as Resonant Clock Disable is Completed
     Unblock Entry Thread
     SE.8A: Quad[0] EX_L2[2] Stopping L2 Clocks
     SE.11A: Quad[0] EX_PG[3] Shutting Cache Down
     Setup: Entry done. Enable Type0/2/3/6 Interrupt

It stops there. It starts the same as with Hostboot, except Hostboot keeps
going forward after this. As cache chiplets are at STOP 11 at this point,
I couldn't read logs from CMEs.

Is there a way of obtaining CME's output before it gets turned off? I tried
enabling CHTM_TRACE_CME, CME_TRACE_ARRAY and setting CHTM config in SGPE
header and I got _something_, but without information about its format it
isn't really useful. Is it something that could be publicly documented?

Are there any additional conditions that must be met for 16.1 to work? By
additional I mean anything more than 15.1-15.4 and functional RAM. We have
skipped OCC initialization from 6.11 for example, is it required for proper
core startup?

The last question is a cosmetic one: is there a way of tracking the time
during these steps, before TOD is set up? I know I can save and restore
TB register, but the amount of time required for jumps between STOP states
would be lost.

[1] 
https://github.com/open-power/hcode/blob/master/import/chips/p9/procedures/utils/stopreg/p9_core_restore_routines.S#L429
[2] 
https://github.com/3mdeb/openpower-coreboot-docs/blob/core_startup/devnotes/scripts/dump_occ_sram.sh

-- 
Krystian Hebel
Firmware Engineer
https://3mdeb.com | @3mdeb_com