distributed tests of a patch

[Thomas Monjalon]

Hi Damien,

2015-09-24 12:01, Damien Lespiau:
> On Thu, Jul 23, 2015 at 01:26:22AM +0200, Thomas Monjalon wrote:
> > An advantage of sending test reports to a mailing list, is to be able
> > to receive them without the need of patchwork, and eventually do some
> > custom processing. It is also possible to offload the storage of the
> > test reports in the mailing list archives. Last advantage, the mailing list
> > administration allows to whitelist the test contributors.
[...]
> > Let's consider a project having 3 mailing lists:
> > - devel@ to receive the patches and comments.
> > - test@ is a mailing list without archive to send patches to test.
> > - reports@ is a mailing list with archives to store the test reports.
> > Each patch from devel@ is sent to test@ with prefix [patchwork id].
> > Private tests may be requested by sending patches to test at .
[...]
> > Each test platform apply the patch (and previous patches of the series) on its
> > local git tree.
> 
> This relies on parsing more mails. Parsing mailing-list is done because
> patches are sent as mail but I wouldn't use the same mechanism for what
> should be more of an API or at least more structured data.

Why a test report would be more structured than a patch?
A patch has a title, an history, an explanation, some tags, some file
references, etc

> You're also making the understanding what a series is part of the
> client, I did it as part of patchwork itself. There are quite complex
> things about series: eg. knowing that a v2 patch is actually part of a
> bigger series, but just that patch was re-sent as reply of the v1
> because the remaining patches are all r-b'ed. These complex things are
> done in patchwork, exposing high level concepts to test
> infrastructures/scripts.
> 
> What I have in store: Patchwork knows what is a series and presents a
> Series object with all the info needed. a RSS feed with extra elements
> in a patchwork namespace to get a list of events and API entry points
> ("new-series-revision", REST URL). From there it's easy to retrieve the
> list of the patches to apply. The logic of knowing to re-test of full
> series with a single patch as v2 (and other interesting mailing-lits
> usages) is abstract by patchwork.

Nice.
As I said earlier,
> > Each patch from devel@ is sent to test@ with prefix [patchwork id].
So we should be able to benefit of patchwork series awareness.

> > Each test platform must reply 1 or more reports with exactly the same subject
> > to the original sender or reports@ if List-Id was devel at .
[...]
> > When the report is received in reports@, the patchwork id entry is updated to
> > be associated to a new test entry including the test label, status and the URL
> > to the report archives.
> 
> So. You are now decoupling the patch from the test results email. I'd
> argue that a report mail that is not a reply to the patch is not as
> useful as a direct reply, because the context is lost and one needs to
> match the patch mail with the result manually. Sure patchwork does that,
> but it means the emails are roughly useless.

Actually, the report is a reply to the forwarded patch. Maybe we could use the
original message id when forwarding to test instances (adding patchwork id).
The recipient of the test report may be the patch author in case of private
testing (without using patchwork), or it may be a mailing list: the devel@ one
or a dedicated reports@ one to avoid spamming.

> I'd send the test-results directly as replies to original patches, with
> consolidated results for a defined list of tests. That way, the report
> mail can be useful in autonomy.
> 
> This also works for tests that would only run for full series (Vs
> running for each individual patch of a series), replying to the
> cover-letter (or first patch if no cover-letter).

No we need to test patches individually to easily know where are the problems.

[...]
> > When refreshing one of these views, it is possible to see the test progress,
> > i.e. how many tests are complete.
> > In this scheme, there is no definitive global test status.
> > The idea is to wait to have a common number of success and no error (removing
> > possible false positives) to consider a patch as good. This judgement is done
> > by the maintainers.
> 
> I'd really like some idea of test completion. I think it's quite key to
> even consider doing more work with the patch, make sure all tests have
> been run.

The goal is to be really open and distributed. It means new tests can spawn and
some of them may be removed or disabled without central management. It allows
to leverage a community for testing without management issues.

Thanks for the discussion