[Cbe-oss-dev] PATCH [0/3] spusched enhancements

[Luke Browning]

Hi Arnd and Christoph,

This patchset should significantly improve the performance of the system
when the SPUs are over committed. I have tested these changes on the
simulator with good results. I can now run several instances of the
matrix multiplication testcase at the same time in a few minutes.  

I haven't tested 'isolated' spu contexts yet.  

Do you have a testcase for 'isolated' contexts?

Here are the main performance problems that I have addressed. 

1) when an spe context is preempted, notification is provided to the
controlling thread.  This is unnecessary as involuntary context switches
should be transparent.  PPE load is artificially inflated.

2) the controlling thread manages the transition from SPU_STATE_SAVED to
SPU_STATE_RUNNABLE.  This is inefficient as the controlling context
needs to be woken in most cases, which introduces latencies into the
scheduler.  The spus may not be fully utilized.  In fact, this is what
we observed in the matrix_Mul benchmark. 

3) the controlling thread has three potentially blocking critical
sections. 1) spu_activate.  2) spe events and errors. 3) spu_reactivate.
Only 2) has to be performed by the controlling thread.  1) and 3) can be
performed by the controlling thread or another thread.  It is more
efficient to do this under the same context, so that you can combine the
unbind and bind in a single step.  Not only does this minimize system
overhead but it makes more rapid use of the SPUs.  

Here is a brief description of the major design point.   

I completed the abstraction of the context structure so that state save
and restore can be performed asynchronously.  The controlling thread has
less work to do now.  It simply initializes the context, passes it to
the scheduler, and waits for spe events and errors.  It is not woken to
activate the context, queue it on the runqueue, or bind the context
anymore.  These tasks are done asynchronously in the scheduler.
Involuntary context switches are transparent to the controlling context
now.  Basically, unbinds and binds now happen in the same critical
section under one thread.  

Note that 'isolated' contexts are still handled in the old way. That is,
contexts created with SPE_CREATE_NOSCHED are managed synchronously by
the controlling thread which tightly controls its execution. This is
needed to complete the initialization of 'isolated' contexts.  

I think these changes will make it a lot easier to implement gang
scheduling as we need the ability to preempt N contexts and bind N
contexts in a single critical section.  

Basically, this enhancement should eliminate about half of the PPE
context switches when SPEs are over committed.  We should see some good
results ...

Cheers, Luke