distributed tests of a patch

[Damien Lespiau]

On Thu, Sep 24, 2015 at 02:34:10PM +0200, Thomas Monjalon wrote:
> Hi Damien,
> 
> 2015-09-24 12:01, Damien Lespiau:
> > On Thu, Jul 23, 2015 at 01:26:22AM +0200, Thomas Monjalon wrote:
> > > An advantage of sending test reports to a mailing list, is to be able
> > > to receive them without the need of patchwork, and eventually do some
> > > custom processing. It is also possible to offload the storage of the
> > > test reports in the mailing list archives. Last advantage, the mailing list
> > > administration allows to whitelist the test contributors.
> [...]
> > > Let's consider a project having 3 mailing lists:
> > > - devel@ to receive the patches and comments.
> > > - test@ is a mailing list without archive to send patches to test.
> > > - reports@ is a mailing list with archives to store the test reports.
> > > Each patch from devel@ is sent to test@ with prefix [patchwork id].
> > > Private tests may be requested by sending patches to test at .
> [...]
> > > Each test platform apply the patch (and previous patches of the series) on its
> > > local git tree.
> > 
> > This relies on parsing more mails. Parsing mailing-list is done because
> > patches are sent as mail but I wouldn't use the same mechanism for what
> > should be more of an API or at least more structured data.
> 
> Why a test report would be more structured than a patch?
> A patch has a title, an history, an explanation, some tags, some file
> references, etc

Parsing a patch is really awful. There are plenty of ambiguous details.
Parsing a mailing-list is not the pinnacle of RPC mechanisms.

That also means test infrastructures have to be set up to receive emails
and forward them to a parsing script and be able to send emails back.

To me, that's a no-go. Let's not use emails as an RPC mechanism?

> > I'd send the test-results directly as replies to original patches, with
> > consolidated results for a defined list of tests. That way, the report
> > mail can be useful in autonomy.
> > 
> > This also works for tests that would only run for full series (Vs
> > running for each individual patch of a series), replying to the
> > cover-letter (or first patch if no cover-letter).
> 
> No we need to test patches individually to easily know where are the
> problems.

That's great. But that's not what I need. I need both.

> [...]
> > > When refreshing one of these views, it is possible to see the test progress,
> > > i.e. how many tests are complete.
> > > In this scheme, there is no definitive global test status.
> > > The idea is to wait to have a common number of success and no error (removing
> > > possible false positives) to consider a patch as good. This judgement is done
> > > by the maintainers.
> > 
> > I'd really like some idea of test completion. I think it's quite key to
> > even consider doing more work with the patch, make sure all tests have
> > been run.
> 
> The goal is to be really open and distributed. It means new tests can spawn and
> some of them may be removed or disabled without central management. It allows
> to leverage a community for testing without management issues.

That's something debatable. If you are part of a community and have a
separate test system, asking the admin (if you're not admin yourself) to
add an entry for your test (giving a name, maybe contact email for that
test, etc..) is not the end of the world?

It is true that that not caring about testing completion simplifies
things though.

-- 
Damien