[Cbe-oss-dev] [PATCH 1/5] MARS: yield in workload queue wait busy loop

[Arnd Bergmann]

On Wednesday 16 July 2008, Kazunori Asayama wrote:
> Arnd Bergmann wrote:

> > Mailbox stalls should of course be avoided, but I suppose there is a way
> > to combine synchronization through atomic memory updates with mailbox
> > accesses to achieve this.
> 
> One possible solution is a combination of mailbox and futex as 
> following. It is just an idea at this point.
> 
>    - Use one futex per synchronization object. Maybe the futex can be
>      the first 32bit of the sync object and it also can be used for
>      atomic operations for SPE-to-SPE and PPE-to-SPE communication.
>    - A PPE thread waits for synchronization object by using the futex.
>    - An SPE notifies the PPE a change of the synchronization object's
>      status via its mailboxes. The EA of the futex is passed as the
>      notification.
>    - Use one PPE thread per SPE context to process the notifications.
>      This thread has a loop which receives notifications (== futex's EA)
>      via the mailboxes then wakes waiting threads up by using the futex.
 
I would hope that you can do something simpler, involving either futex
or mailbox but not both, and no extra threads.
A futex has the advantage that you can use it as a blocking call in either
direction (PPU to SPU and SPU to PPU), while you typically want to avoid
having the SPU blocked on a mailbox.
Then again, a futex_wait on the SPU running MARS prevents you from running
other tasks in the same SPU context, MARS should probably do that only
when it runs out of things to work on.

As an example, I think you can do a ring buffer between PPU and SPU like
follows:

* Have a data buffer of up to 65536 objects (bytes) that is only written
to by one side.
* Have one 32 bit value with two offsets into that buffer, one for reader
and writer respectively.
* Use atomic updates on the offset when data is read or written in the
buffer.
* only when writing to a full buffer or reading from an empty buffer, you
need to wait. The PPU can wait by calling futex_wait or reading from an empty
mailbox, or by writing to a full mailbox. The SPU can wait by calling
futex_wait, scheduling another MARS task or doing some other stop-and-signal
call.
* When writing to an empty buffer or reading from a full buffer, notify
the other side.

	Arnd <><