[Skiboot] [PATCH] core/fast-reboot.c: Add sreset opal call

[Alistair Popple]

On Fri, 23 Dec 2016 11:32:02 AM Stewart Smith wrote:
> Alistair Popple <alistair at popple.id.au> writes:
> > Sending a NMI to other CPUs regardless of their current state requires
> > a way to reset them. POWER hardware has a method of directly injecting
> > resets via direct thread control, however this only works if the
> > thread is not active (eg. in a sleep or nap state).
> >
> > Resetting an active thread can be performed either via forcing the
> > threads to an inactive state (as fast reboot does) or by ramming an
> > instruction sequence that simulates an sreset. This patch implements
> > the latter as forcing a thread to the inactive state is not ideal for
> > debug purposes as the threads loose state.
> 
> do we have an idea about what's going to be involved with it on P9 ?

As Ben mentioned not yet. I have been hacking around on pdbg though and one of
the aims is to make it easier to integrate code from that into skiboot. That
way the instruction ramming code could be the same between pdbg/skiboot
although practically I think that's a little way off yet.

<snip>

> > +
> > +	/* Disable HV mode on thread */
> > +	ram_mode &= ~(PPC_BIT(2) >> thread_id*2);
> > +	xscom_write(chip_id, XSCOM_ADDR_P8_EX(core_id,
> > P8_EX_RAM_MODE_REG), ram_mode);
> 
> I'm guessing the only real *sensible* way to deal with any of these
> xscoms failing is to bail out and return a "good luck with that" error
> code back to the OS so it can then... just cry really.

I did kind of ignore all the return codes because I didn't really know what to
do if a scom access failed. There are plenty of other places that also don't
check check scom access return codes so perhaps we need a version of
scom_read/write that either panics or prints a warning in case of xscom
failure?

In the case of instruction ramming the only real thing we could do to recover
is print tears and die as the box will be totally hosed or about to checkstop.

I could add an assert(xscom_write(...)) I guess? Or would you rather return
to the OS? That said the OS itself is probably somewhat sick which is why this
function was called in the first place, so it might be better to just do the
simple thing and die.

> 
> 
> > +/*
> > + * Apply an sreset to the given threads in a core. When ramming
> > + * instructions the whole core must be quiesced so we can't apply an
> > + * sreset to active threads on the same core as we're running
> > + */
> > +static int64_t sreset_core(struct cpu_thread *cpu, unsigned int thread_mask)
> > +{
> > +	uint32_t thread_id;
> > +	uint32_t sreset_mask = 0, ram_mask = 0;
> > +	uint64_t ras_status;
> > +	int timeout;
> > +	int64_t rc = 0;
> > +	unsigned int max_thread_id = pir_to_thread_id(-1U);
> > +	unsigned int max_thread_mask = (1 << (max_thread_id + 1)) - 1;
> > +
> > +	assert(cpu == cpu->primary);
> > +	thread_mask &= max_thread_mask;
> > +	if (this_cpu() == cpu) {
> > +		prlog(PR_WARNING, "SRESET: Unable to reset threads on self\n");
> > +		return OPAL_PARAMETER;
> > +	}
> > +
> > +	prlog(PR_INFO, "SRESET: Start reset for cpu 0x%x thread_mask 0x%x\n",
> > +	      cpu->pir, thread_mask);
> > +
> > +	/* cpu is the primary thread */
> > +	if (set_special_wakeup(cpu) != OPAL_SUCCESS)
> > +		return OPAL_BUSY;
> > +
> > +	/* Stop threads selected for sreset */
> > +	for (thread_id = 0; thread_id <= max_thread_id; thread_id++) {
> > +		if (!((1 << thread_id) & thread_mask))
> > +			continue;
> > +
> > +		set_direct_ctl(cpu, thread_id, P8_DIRECT_CTL_STOP);
> > +		ras_status = get_ras_status(cpu, thread_id);
> > +		if (!(ras_status & RAS_STATUS_THREAD_ACTIVE))
> > +			sreset_mask |= 1 << thread_id;
> > +	}
> > +
> > +	if (thread_mask == sreset_mask)
> > +		/* All threads selected for sreset can be sreset directly */
> > +		prlog(PR_INFO, "SRESET: All threads inactive, doing direct sreset\n");
> > +		for (thread_id = 0; thread_id <= max_thread_id; thread_id++)
> > +			if ((1 << thread_id) & thread_mask) {
> > +				set_direct_ctl(cpu, thread_id, P8_DIRECT_CTL_SRESET);
> > +				rc = OPAL_SUCCESS;
> > +				goto out;
> > +			}
> > +
> > +	/* Need to emulate sreset so stop all other threads */
> > +	for (thread_id = 0; thread_id <= max_thread_id; thread_id++)
> > +		if (!((1 << thread_id) & thread_mask))
> > +			set_direct_ctl(cpu, thread_id, P8_DIRECT_CTL_STOP);
> > +
> > +	/* Work out which threads to sreset and which need sreset emulation */
> > +	for (thread_id = 0; thread_id <= max_thread_id; thread_id++) {
> > +		ras_status = get_ras_status(cpu, thread_id);
> > +		if (!(ras_status & RAS_STATUS_THREAD_ACTIVE))
> > +			sreset_mask |= 1 << thread_id;
> > +		else {
> > +			for (timeout = 0; timeout < RAS_STATUS_TIMEOUT; timeout++) {
> > +				ras_status = get_ras_status(cpu, thread_id);
> > +				if ((ras_status & RAS_STATUS_SRQ_EMPTY)
> > +				    && (ras_status & RAS_STATUS_LSU_QUIESCED)
> > +				    && (ras_status & RAS_STATUS_TS_QUIESCE)) {
> > +					ram_mask |= 1 << thread_id;
> > +					break;
> > +				}
> > +			}
> > +		}
> > +	}
> > +
> > +	/*
> > +	 * To emulate sreset we need to make sure all threads on a core are either:
> > +	 *  a) Quiesced
> > +	 *  b) Not active (recorded in sreset_mask)
> > +	 * So skip ramming threads if we're not in the right state.
> > +	 */
> > +	if ((ram_mask & thread_mask) && (ram_mask | sreset_mask) != max_thread_mask) {
> > +		prlog(PR_ERR, "SRESET: Unable to quiesce all threads for ramming (sreset_mask 0x%08x / ram_mask 0x%08x)\n",
> > +		      sreset_mask, ram_mask);
> > +		ram_mask = 0;
> > +		rc = OPAL_PARTIAL;
> > +	}
> > +
> > +	/* We need to ram threads before doing the direct sresets as
> > +	 * that makes the threads active */
> > +	for (thread_id = 0; thread_id <= max_thread_id; thread_id++) {
> > +		if (!((1 << thread_id) & thread_mask))
> > +			continue;
> > +
> > +		if ((1 << thread_id) & ram_mask)
> > +			emulate_sreset(cpu, thread_id);
> > +	}
> > +
> > +	for (thread_id = 0; thread_id <= max_thread_id; thread_id++) {
> > +		if (!((1 << thread_id) & thread_mask))
> > +			continue;
> > +
> > +		if ((1 << thread_id) & sreset_mask) {
> > +			prlog(PR_ERR, "SRESET: cpu 0x%x thread 0x%x not active, applying SRESET directly\n",
> > +			      cpu->pir, thread_id);
> > +			set_direct_ctl(cpu, thread_id, P8_DIRECT_CTL_SRESET);
> > +		}
> > +	}
> > +
> > +out:
> > +	/* Start all threads */
> > +	for (thread_id = 0; thread_id <= max_thread_id; thread_id++)
> > +		set_direct_ctl(cpu, thread_id, P8_DIRECT_CTL_START);
> > +
> > +	clr_special_wakeup(cpu);
> > +
> > +	return rc;
> > +}
> > +
> > +#define SYS_RESET_ALLBUTSELF -2
> 
> should be in opal-api.h ?

It sure should. Will fix.

> > +int64_t signal_system_reset(int cpu_nr)
> > +{
> > +	int64_t rc = 0;
> > +	struct cpu_thread *cpu;
> > +	uint32_t thread_id;
> > +
> > +	if (proc_gen != proc_gen_p8)
> > +		return OPAL_UNSUPPORTED;
> > +
> > +	/* Reset a single CPU */
> > +	if (cpu_nr >= 0) {
> > +		cpu = find_cpu_by_server(cpu_nr);
> > +		if (!cpu)
> > +			return OPAL_PARAMETER;
> > +
> > +		thread_id = pir_to_thread_id(cpu->pir);
> > +		cpu = cpu->primary;
> > +		return sreset_core(cpu, 1 << thread_id);
> > +	}
> > +
> > +	/* Otherwise reset all CPUs */
> > +	for_each_cpu(cpu) {
> > +		if (cpu->primary == this_cpu()->primary)
> > +			continue;
> > +
> > +		if (cpu->primary != cpu)
> > +			continue;
> > +
> > +		/* sreset all threads on a core */
> > +		rc |= sreset_core(cpu, -1);
> > +	}
> > +
> > +	return rc;
> > +}
> > diff --git a/doc/opal-api/opal-signal-system-reset-128.txt b/doc/opal-api/opal-signal-system-reset-128.txt
> > new file mode 100644
> > index 0000000..bb1b869
> > --- /dev/null
> > +++ b/doc/opal-api/opal-signal-system-reset-128.txt
> > @@ -0,0 +1,30 @@
> > +OPAL_SIGNAL_SYSTEM_RESET
> > +-------------------
> > +
> > +#define OPAL_SIGNAL_SYSTEM_RESET			128
> > +
> > +int64_t signal_system_reset(int cpu_nr)
> > +
> > +Arguments:
> > +
> > +  int cpu_nr
> > +    Either the cpu server number of the target cpu to reset or
> > +    SYS_RESET_ALLBUTSELF (-2) to indicate all but the current cpu
> > +    should be reset.
> 
> I'm thinking we should just copy what's going on in papr and also accept
> -1 = target all online threads including the caller
> 
> even if we just return OPAL_UNSUPPORTED or OPAL_PARTIAL or something for
> the -1 case.

Good idea. We could in theory add support to do all online threads in future
so being able to detect that with OPAL_UNSUPPORTED is probably good. For the
moment though Nick and I concluded it would be easier to do the equivalent in
Linux with SYS_RESET_ALLBUTSELF and another call to reset SELF from another
processor.

> > +This OPAL call causes the specified cpu(s) to be reset to the system
> > +reset exception handler (0x100). Sleeping cpus will be woken with
> > +SRR1[42:45] = 0x0100 indicating an interrupt caused by SCOM when in
> > +power saving mode. Active cpus will also indicate interrupt caused by
> > +SCOM but will have SRR1[46:47] = 0 as the interrupt did not occur
> > +during a power saving mode.
> > +
> > +Resetting active threads on the same core as this call is run is
>  > +currently not supported.
> > +
> > +Return Values:
> > +OPAL_SUCCESS: the power down was updated successful
> > +OPAL_PARAMETER: a parameter was incorrect
> > +OPAL_HARDWARE: hardware indicated failure during instruction ramming
> > +OPAL_PARTIAL: not all requested cpus could be reset at this time
> > +OPAL_UNSUPPORTED: this processor generation is not supported
> 
>                      or requested operation is not supported?
>                      or should that be OPAL_PARAMETER?
>                      (for -1)

OPAL_UNSUPPORTED for -1 makes sense to me. Will update it.

> 
> > index e1a8a4d..e4761ee 100644
> > --- a/platforms/astbmc/common.c
> > +++ b/platforms/astbmc/common.c
> > @@ -356,4 +356,6 @@ void astbmc_early_init(void)
> >  	uart_init();
> >
> >  	prd_init();
> > +
> > +	opal_register(OPAL_SIGNAL_SYSTEM_RESET, signal_system_reset, 1);
> >  }
> 
> Anything that makes this specific to astbmc systems?

I don't think so, other than I've only tested it on astbmc systems.