distributed tests of a patch

[Thomas Monjalon]

2015-09-24 13:57, Damien Lespiau:
> On Thu, Sep 24, 2015 at 02:34:10PM +0200, Thomas Monjalon wrote:
> > 2015-09-24 12:01, Damien Lespiau:
> > > On Thu, Jul 23, 2015 at 01:26:22AM +0200, Thomas Monjalon wrote:
> > > > When refreshing one of these views, it is possible to see the test progress,
> > > > i.e. how many tests are complete.
> > > > In this scheme, there is no definitive global test status.
> > > > The idea is to wait to have a common number of success and no error (removing
> > > > possible false positives) to consider a patch as good. This judgement is done
> > > > by the maintainers.
> > > 
> > > I'd really like some idea of test completion. I think it's quite key to
> > > even consider doing more work with the patch, make sure all tests have
> > > been run.
> > 
> > The goal is to be really open and distributed. It means new tests can spawn and
> > some of them may be removed or disabled without central management. It allows
> > to leverage a community for testing without management issues.
> 
> That's something debatable. If you are part of a community and have a
> separate test system, asking the admin (if you're not admin yourself) to
> add an entry for your test (giving a name, maybe contact email for that
> test, etc..) is not the end of the world?

I think an admin must give the right to an entity to send some test reports,
e.g. whitelisting the email. Then this entity manage its own set of tests.
Sometimes they can add or update a test, sometimes they can make some errors
and have the urgent need to disable a test sending some false positives.
It is important to give this entity the agility of managing which tests are
run in a reactive way. If they are not reactive enough, they can be removed
from the whitelist.
But asking to a central admin for each test enabling/disabling is clearly a
bottleneck leading to a failure of the distributed concept IMHO.

> It is true that that not caring about testing completion simplifies
> things though.