[Cbe-oss-dev] LWG_RFC00002-0 : SPE Runtime mailbox functions

[Arnd Bergmann]

On Friday 31 March 2006 19:08, Daniel Brokenshire wrote:
> LWG_RFC00002-0
> 
> SPE Runtime mailbox functions
> 
> ------------------------------------------------------------------------
> 
> Severity: 2             Status: submitted       Date attained: 03/31/2006
>                         Resolution: <r>         Date resolved: <mm/dd/yy>
>                                                 Date made: <mm/dd/yy>
> 
> Summary: Correct inconsistencies and limitation is the MFC mailbox functions
> provided by the SPE Runtime Managment library
> 
> Documents effected: 
>          SPE Runtime Management Library         Version: 1.1
> 
> 
> Sections and pages effected:
>          SPE Runtime Management Library         page 26
>          
>          Change the C Specification to read:
> 
>          "#include <libspe.h>
>           int spe_read_out_mbox(speid_t speid, size_t entries, unsigned int *data);"

As you said, that prototype is incompatible with the current one, causing
silent run-time misbehaviour when running an application with the wrong
version of the shared library, as well as breaking source-level
compatibility.

I wonder if we can do one of these things to limit the impact of
this change:

1. Give the prototype a new name, while keeping the previous version in
   place. E.g. the new name could be spe_read_out_mbox_multi (ok, we'd
   better come up with something better than that).
2. Defer the change of the prototype until version 2.0 of the standard.
   We should still update the specification to reflect the real behaviour,
   but at least it will remain compatible within the 1.x series.
3. Use symbol-versioning to keep binary compatibility, while breaking only
   compilation (this can be combined with 2.).

>          Change the Description to read:
> 
>          "The spe_read_out_mbox functions returns a requested number of 
>           outbound mailbox entries, as specified by the entries parameter, 
>           corresponding to the SPE thread specified by the speid parameter,
>           to the array specified by data. 
> 
>           This read has blocking symantics such that control is not returned 
>           to the caller until all the requested entries are read. To avoid 
>           blocking, spe_stat_out_mbox can be called to determine the 
>           number of available mailbox entries prior to reading the outbound
>           mailbox."

AFAIK, spe_read_out_mbox can not block, there is no interrupt associated with
the functionality. Only the interrupt-mailbox and the spe_write_in_mbox ()
are able to provide blocking behaviour due to the design of the hardware.

>          Add the following additional Parameters:
> 
>          "entries    Specifies the number of 32-bit mailbox entries to be read
>                      from the outbound mailbox
> 
>           data       Pointer to the array in which the mailbox entries are to
>                      be placed. The array must be sufficiently large to hold
>                      the requested number of mailbox entries."
> 
>          Change the Return Values description to read:
> 
>          "On success, spe_read_out_mbox returns 0 and the data array is filled
>           with the specified number of mailbox entries. On failure, -1 is 
>           returned."

Should it also set errno when it hits an error?

>          SPE Runtime Management Library         page 27
> 
>          Remove the spe_stat_out_intr_mbox function.
> 
>          SPE Runtime Management Library         page 28
> 
>          Change the C Specification to read:
> 
>          "#include <libspe.h>
>           int spe_write_in_mbox(speid_t speid, size_t entries, unsigned int *data);"

same comment on the prototype as above.

>          Change the Description to read:
> 
>          "The spe_write_in_mbox functions places the requested number of 
>           32-bit mailbox entries into the inbound mailbox for the SPE thread 
>           specified by the speid parameter. The entries parameter specifies 
>           the requested number data array elements to be written to the 
>           inbound mailbox.
> 
>           This write has blocking symantics such that control is not returned 
>           to the caller until all the requested data is written to the mailbox.
>           To avoid blocking, spe_stat_in_mbox can be called to determine the 
>           number of available mailbox entries prior to writing to the inbound
>           mailbox."

The kernel already implements both blocking and non-blocking mode for mailbox
writes, it can be selected when opening the underlying file (O_NONBLOCK).
Maybe we should provide a way of surfacing that in the library instead of
recommending the use of spe_stat_in_mbox, which causes additional overhead.
 
>          Add the following additional Parameters:
> 
>          "entries    Specifies the number of 32-bit mailbox entries to be 
>                      written to the inbound mailbox
> 
>           data       Pointer to the array of data to be written into the 
>                      inbound mailbox."
> 
> 
> Submitter:      Dan Brokenshire         Date accepted: 03/31/2006
> Requester:      Dan Brokenshire
> Assigned to:    Dan Brokenshire
> Impact on software: libspe, many testcases and code samples
> Keywords:       MFC, mailboxes
> 
> ------------------------------------------------------------------------
> 
> The current SPE Runtime Management Library mailbox functions have several
> limitations and inconsistencies. This include:
>  * spe_read_out_mbox is specified to return -1 if no mailbox data is available.
>    With such an interface, programmers have no way to destinguish between no
>    available data and data equal to -1 without "stat"ing the mailbox before
>    reading. 

ouch...

>  * spe_write_in_mbox is specified to overwrite the last mailbox entry if it is
>    full and that spe_stat_in_mbox must be called to ensure space is available 
>    before writing to the queue. The current implementation actually blocks
>    waiting for available space.

It could also be set to non-blocking mode, as mentioned above. However, then it
would perform no action at all instead of overwriting the last element and we'd
still have to change the specification.

>  * Mailboxes are defined to have capacity. However, this interface only allows
>    one to read and write a single word at a time. This becomes inefficient for
>    applications that which to send multiple words at a time through the 
>    mailbox.

It would probably be a good idea to also change the kernel interface then
to allow read/write of multiple elements in a single system call. This
extension would be compatible in the, except that the kernel would no
longer be required to return an error condition if asked to transfer more
than four bytes.

>  * The spe_stat_out_intr_mbox routine appears to be senseless without the
>    ability to read the outbound interrupting mailbox. Since the interrupting 
>    mailbox data is read by the operating system and buffered in the event 
>    queue, the need for "stat"ing the outbound interrupting mailbox does not
>    exist.

Actually, the kernel does not treat the interrupt mailbox different from the
normal out mailbox, except that it has both blocking and nonblocking mode.
All the event buffering for it happens inside of the library.

Since we are already about to make a major change to the mailbox interface,
can we at the same time add a new spe_read_out_intr_mbox? I believe that
for many user scenarios that would be a lot easier to use than the
currently available event mechanism.

> This RFC proposes the following changes to address all of these issues. 
> 1) Modify the spe_read_out_mbox function to:
>    - Add an additional size_t parameter that specifies the number of mailbox 
>      entries to be read. 
>    - Add an additional unsigned int * parameter to for returning mailbox data.
>    - Change the return value to an int that returns success status only.
>    - Change the documentation to indicate that the function blocks until all
>      the requested data is read or an error is detected.
> 2) Modify the spe_write_in_mbox function to:
>    - Add an additional size_t parameter that specifies the number 32-bit data
>      words to be written to the SPE's inbound mailbox.
>    - Change the data type of the "data" parameter to be a pointer to an 
>      unsigned integer.
>    - Change the description to state that control is not returned to the 
>      caller until the "data" is successfully placed in the SPE threads mailbox
>      and 0 is returned. If the specified thread is invalid or terminates before
>      the data is written to the mailbox, then -1 is returned.
> 3) Remove the spe_stat_out_intr_mbox function.
> 
> =============================== end LWG_RFC ================================

Thanks,

	Arnd <><