[Skiboot] [RFC 1/1] OPAL : Support spr save-restore using new opal call

[Nicholas Piggin]

On Tue, 24 Jul 2018 11:19:07 +0530
Vaidyanathan Srinivasan <svaidy at linux.vnet.ibm.com> wrote:

> * Nicholas Piggin <npiggin at gmail.com> [2018-07-24 15:36:48]:
> 
> > On Tue, 24 Jul 2018 10:06:12 +1000
> > Benjamin Herrenschmidt <benh at kernel.crashing.org> wrote:
> >   
> > > On Mon, 2018-07-23 at 23:41 +1000, Nicholas Piggin wrote:  
> > > > Right, so we should make OPAL do the whole thing (except catch the
> > > > SRESET wakeup which Linux has to do and has well architected SRR1
> > > > bits).
> > > >     
> > > > > 
> > > > > We could have a separate OPAL entry point for the wakeup case but
> > > > > putting the entirety of the enter/exit into OPAL is complicated.
> > > > > 
> > > > > We have to arbitrate between multiple threads entering different
> > > > > states. We don't know until we exit how deep the core actually went and
> > > > > what actually needs to be restored. It's a function of the SRR1 bits on
> > > > > return *and* the state that was requested.
> > > > > 
> > > > > There's also the need to synchronize/rendez-vous threads on wakeup in
> > > > > some cases such as TB resync, and dealing with big vs. small core.
> > > > > 
> > > > > Doing it all in OPAL would require *all* stop states to go to OPAL, I
> > > > > don't think that's a great idea for performance.    
> > > > 
> > > > Oh you want Linux native and OPAL drivers to be used simultaneously?
> > > > Is that the right thing to optimise for?    
> > > 
> > > We want Linux native for states that are known to work (hopefully
> > > STOP0/1) and OPAL for things that might need additional resources saved
> > > or restored and/or workarounds.
> > > 
> > > At least that's how I thought of it so far... you prefer having an "all
> > > or nothing" approach ?  
> > 
> > As an option, yes. And once you have that option I don't think there is
> > much point requiring kernel to know anything about deep states or even
> > the STOP instruction or PSSCR register.
> > 
> > I would not object to an OPAL API that requires all deep states to be
> > entered via OPAL but allows shallow states (GPR loss only) to be used
> > natively by the kernel concurrently. The shallow states are pretty well
> > architected and the deep states not nearly so performance critical.  
> 
> I agree with your design point.  However based on our experience
> across different platforms, we did not want to draw a line for shallow
> vs deep for this API.

I don't see a problem with doing that. If the state came to require some
workaround other than simple GPR restore, then it can be marked in
device tree as being a deep state (which it is by definition, isn't it?)

The biggest performance win and most critical is SMT mode switch, so if
we only had stop0 available for native kernel I don't think that would
be a problem.

When we start turning off clocks and throwing away caches and SPRs, I
don't think the OPAL call will be a huge problem.

>  The framework can be used as fallback for any
> state if required.  But in practical situations, we expect to hook
> onto this one from the deepest state like STOP11 where we still have
> few quirks to deal with.  The STOP11 case is a clear win with OPAL
> path because it is so slow and kernel's context management will not
> speed it up.

The problem with that is that it's all still expecting the kernel to
do the right synchronisation and know about scope of resource loss.
What if we had some issue that required synchronisation between a pair
of cores? Or if we had to use a different entry sequence to execute
the 'stop' instruction (AFAIK that is not actually architected, but
assuming we don't do something crazy like change it, there could be
an errata)? Then we lose the whole API.

IMO we need a *true* back compatible API (at least one that relies on
no more than SRESET/MCE + SRR1 powersave wakeup indication). And then
we can think of reasonable additions to make performance a bit better.
I think allowing native shallow stop concurrently is reasonable. Is
allowing native concurrent deep stops? I'm not so sure. I'd like to see
some numbers though.

Thanks,
Nick