distributed tests of a patch

Damien Lespiau damien.lespiau at intel.com
Thu Sep 24 22:57:52 AEST 2015

On Thu, Sep 24, 2015 at 02:34:10PM +0200, Thomas Monjalon wrote:
> Hi Damien,
> 2015-09-24 12:01, Damien Lespiau:
> > On Thu, Jul 23, 2015 at 01:26:22AM +0200, Thomas Monjalon wrote:
> > > An advantage of sending test reports to a mailing list, is to be able
> > > to receive them without the need of patchwork, and eventually do some
> > > custom processing. It is also possible to offload the storage of the
> > > test reports in the mailing list archives. Last advantage, the mailing list
> > > administration allows to whitelist the test contributors.
> [...]
> > > Let's consider a project having 3 mailing lists:
> > > - devel@ to receive the patches and comments.
> > > - test@ is a mailing list without archive to send patches to test.
> > > - reports@ is a mailing list with archives to store the test reports.
> > > Each patch from devel@ is sent to test@ with prefix [patchwork id].
> > > Private tests may be requested by sending patches to test at .
> [...]
> > > Each test platform apply the patch (and previous patches of the series) on its
> > > local git tree.
> > 
> > This relies on parsing more mails. Parsing mailing-list is done because
> > patches are sent as mail but I wouldn't use the same mechanism for what
> > should be more of an API or at least more structured data.
> Why a test report would be more structured than a patch?
> A patch has a title, an history, an explanation, some tags, some file
> references, etc

Parsing a patch is really awful. There are plenty of ambiguous details.
Parsing a mailing-list is not the pinnacle of RPC mechanisms.

That also means test infrastructures have to be set up to receive emails
and forward them to a parsing script and be able to send emails back.

To me, that's a no-go. Let's not use emails as an RPC mechanism?

> > I'd send the test-results directly as replies to original patches, with
> > consolidated results for a defined list of tests. That way, the report
> > mail can be useful in autonomy.
> > 
> > This also works for tests that would only run for full series (Vs
> > running for each individual patch of a series), replying to the
> > cover-letter (or first patch if no cover-letter).
> No we need to test patches individually to easily know where are the
> problems.

That's great. But that's not what I need. I need both.

> [...]
> > > When refreshing one of these views, it is possible to see the test progress,
> > > i.e. how many tests are complete.
> > > In this scheme, there is no definitive global test status.
> > > The idea is to wait to have a common number of success and no error (removing
> > > possible false positives) to consider a patch as good. This judgement is done
> > > by the maintainers.
> > 
> > I'd really like some idea of test completion. I think it's quite key to
> > even consider doing more work with the patch, make sure all tests have
> > been run.
> The goal is to be really open and distributed. It means new tests can spawn and
> some of them may be removed or disabled without central management. It allows
> to leverage a community for testing without management issues.

That's something debatable. If you are part of a community and have a
separate test system, asking the admin (if you're not admin yourself) to
add an entry for your test (giving a name, maybe contact email for that
test, etc..) is not the end of the world?

It is true that that not caring about testing completion simplifies
things though.


More information about the Patchwork mailing list