<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" ><font size="2" face="Default Monospace,Courier New,Courier,monospace" >> First question: what wakes up the core after the first 'stop' (SBE, I<br>> presume?) and what should wake it after self restore?</font></div>
<div dir="ltr" >The core wakes up due to an interrupt, not any kind of manual control. (Going a bit from memory here) The SBE unblocks interrupts after it sees all of the cores go into STOP11 state.</div>
<div class="mail-signature-container" dir="ltr" > </div>
<div class="mail-signature-container" dir="ltr" >Which side of the deadman loop are you failing on? There are actually 2 ways it can fail. We can either not get the cores into STOP11 or we cannot wakeup after that. Both lead to the same checkstop firing.</div>
<div class="mail-signature-container" dir="ltr" > </div>
<div class="mail-signature-container" dir="ltr" >> <font size="2" face="Default Monospace,Courier New,Courier,monospace" >We have skipped OCC initialization from 6.11 for example, is it required for proper<br>> core startup?</font></div>
<div class="mail-signature-container" dir="ltr" >No, OCC is not required, only the SPGE/PGPE engines that we start in step15/16. </div>
<div class="mail-signature-container" dir="ltr" > </div>
<div class="mail-signature-container" dir="ltr" >> <font size="2" face="Default Monospace,Courier New,Courier,monospace" >is there a way of tracking the time<br>> during these steps, before TOD is set up? I know I can save and restore<br>> TB register, but the amount of time required for jumps between STOP states<br>> would be lost.</font></div>
<div class="mail-signature-container" dir="ltr" >There is not. Pre-TOD, there is no timer that I'm aware of except TB. I suppose you could use the SBE's ticker to count if you needed to. The STOP time should be very tiny when it works correctly so in reality there isn't too much time getting lost.</div>
<div class="mail-signature-container" dir="ltr" > </div>
<div class="mail-signature-container" dir="ltr" >You've already dug into the guts of the CME/SGPE code more than I ever have so I'll forward your questions on to the chip team to see if they have anything to add.</div>
<div class="mail-signature-container" dir="ltr" ><br>--<br>Dan Crowell<br>Senior Software Engineer - Power Systems Enablement Firmware<br>IBM Rochester: t/l 553-2987 dcrowell@us.ibm.com</div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" data-history-expanded="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: "Krystian Hebel" <krystian.hebel@3mdeb.com><br>Sent by: "OpenPower-Firmware" <openpower-firmware-bounces+dcrowell=us.ibm.com@lists.ozlabs.org><br>To: openpower-firmware@lists.ozlabs.org<br>Cc: firmware@3mdeb.com<br>Subject: [EXTERNAL] [OpenPower-Firmware] Questions about core bring-up<br>Date: Tue, Aug 24, 2021 10:27 AM<br>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >Hello,<br><br>during our struggle with making a port of coreboot for Talos II we hit a<br>dead end in istep 16.1. Even when using a 1:1 copy of HOMER produced by<br>Hostboot (which has different HRMOR, but we haven't got to a point where<br>it makes a difference) we cannot get through deadman loop. It seems that<br>the core is woken up and executes its self restore code, but then it hits<br>another 'stop' instruction [1] from which it doesn't return. At that point<br>PPM_SSHOTR has a value of 0x8fbfb00000000000 for cores, 0x0bbbf00000000000<br>for cache chiplets. I can manually wake threads by doing 'pdbg stop'<br>followed by SRESET through DIRECT_CONTROLS and, given that this was done<br>fast enough, code properly sends a message to stop the timer to SBE;<br>otherwise checkstop is generated (I've masked it for easier debugging).<br><br>First question: what wakes up the core after the first 'stop' (SBE, I<br>presume?) and what should wake it after self restore?<br><br>I managed to get SGPE logs (default verbosity) as described in comment in<br>script I prepared [2] and got this:<br><br> Initializing External Interrupt Routing Registers<br> Setup: Clear Type 0,2,3,5,6,7 and ipi_lo_3 interrupts<br> Setup: Prepare CME[0x8000] to be Booted<br> Setup: Booting CME in Hostboot Mode<br> Setup: Remove Hostboot Istep4 Special Wakeup<br> Setup: Unmask Type 0,2,3,5,6 and ipi_lo_3 interrupts, keep type7<br>masked as unused<br> Setup: SGPE STOP READY<br> Core[1] sent PIG Type[3] with Payload [0xF]<br> Core Request Entry Confirmed<br> Unblock Entry Thread<br> DB2 MessageID 2(Rclk Entry) sent to core 1<br> Setup: Entry done. Enable Type0/2/3/6 Interrupt<br> Core[1] sent PIG Type[0] with Payload [0x10B]<br> Core Request Entry Allowed as Resonant Clock Disable is Completed<br> Unblock Entry Thread<br> SE.8A: Quad[0] EX_L2[2] Stopping L2 Clocks<br> SE.11A: Quad[0] EX_PG[3] Shutting Cache Down<br> Setup: Entry done. Enable Type0/2/3/6 Interrupt<br><br>It stops there. It starts the same as with Hostboot, except Hostboot keeps<br>going forward after this. As cache chiplets are at STOP 11 at this point,<br>I couldn't read logs from CMEs.<br><br>Is there a way of obtaining CME's output before it gets turned off? I tried<br>enabling CHTM_TRACE_CME, CME_TRACE_ARRAY and setting CHTM config in SGPE<br>header and I got _something_, but without information about its format it<br>isn't really useful. Is it something that could be publicly documented?<br><br>Are there any additional conditions that must be met for 16.1 to work? By<br>additional I mean anything more than 15.1-15.4 and functional RAM. We have<br>skipped OCC initialization from 6.11 for example, is it required for proper<br>core startup?<br><br>The last question is a cosmetic one: is there a way of tracking the time<br>during these steps, before TOD is set up? I know I can save and restore<br>TB register, but the amount of time required for jumps between STOP states<br>would be lost.<br><br>[1]<br><a href="https://github.com/open-power/hcode/blob/master/import/chips/p9/procedures/utils/stopreg/p9_core_restore_routines.S#L429" target="_blank">https://github.com/open-power/hcode/blob/master/import/chips/p9/procedures/utils/stopreg/p9_core_restore_routines.S#L429</a> <br>[2]<br><a href="https://github.com/3mdeb/openpower-coreboot-docs/blob/core_startup/devnotes/scripts/dump_occ_sram.sh" target="_blank">https://github.com/3mdeb/openpower-coreboot-docs/blob/core_startup/devnotes/scripts/dump_occ_sram.sh</a> <br><br>--<br>Krystian Hebel<br>Firmware Engineer<br><a href="https://3mdeb.com" target="_blank">https://3mdeb.com</a> | @3mdeb_com<br><br>_______________________________________________<br>OpenPower-Firmware mailing list<br>OpenPower-Firmware@lists.ozlabs.org<br><a href="https://lists.ozlabs.org/listinfo/openpower-firmware" target="_blank">https://lists.ozlabs.org/listinfo/openpower-firmware</a> </font></div></blockquote>
<div dir="ltr" > </div></div><BR>
<BR>