[PULL] github.com/stephenfin/development

[Finucane, Stephen]

> On Fri, Oct 16, 2015 at 04:11:50PM +0100, Finucane, Stephen wrote:
> > You appear to be insisting there should be a more static configuration
> > of "runnable tests". However, I don't think this would scale or allow
> > the fluidity of Thomas' proposal.
> 
> I'm insisting it's *possible* to have a list of pre-defined tests, not
> that it would be the only way to report test results. I can see merits
> in both policies. I actually don't care how it's implemented as long as
> it's reasonable and integrated into patchwork.
>
> > In the normalized case, allowing a new CI system to start reporting
> > test statuses would require (a) allowing them access to the mailing
> > list
> 
> Nop. The test results are sent to patchwork, which then replies on
> behalf of the tester. Several configurable policies could be possible,
> depending on what project want to do:
>
>   - send a consolidated email if we have a list of pre-defined tests
>   - send per-test mails otherwise?
>   - send consolidated mails on full series (remember the discussion
>     about testing full series because it can impractical to do it on
>     every patch, not just a corner case, we need it for the project I
>     work on)
>   - ...
> 
> No need for the test system to be on the mailing list, no mail setup on
> the client side is all win to me.
> 
> > and (b) configuring patchwork to enter this new tests. This, to me, is
> > a barrier.
> 
> This doesn't have to be the case. Configurable behaviours are totally
> fine.
> 
> > I hadn't planned on letting patchwork send any more emails, if I'm
> > being totally honest. If you think an email would be useful, could we
> > develop a script that checks for a given list of 'contexts' on a patch
> > and asserts that those contexts all exist and have a 'passed' state.
> 
> Given that patchwork is there to augment mails & mailing-lists, it
> seemed natural to me to have it replying to patches with test results.
> 
> An external script could mostly work. However I'd like it to be
> integrated so we can do things like replying to the cover letter of the
> series, which needs full access to the patchwork.

OK, I can see the benefits of this scenario in some cases: let's add the functionality that you need without having to compromise on what other groups need. If so, I propose adding the following changes to the existing model. These can and should all be done in follow up series to keep each series as small and modular as possible.


# 1) Add additional project configuration options:

We should add the following options to the 'Project' model

 * Add an optional list property to specify the tests contexts that have to run in order for a patch to be marked as "testing complete". If set, an email will be sent on test completion. If unset, no emails will be sent and Thomas' process can be used
 * Add a Boolean property to state whether to restrict results to these contexts only or whether to allow any and all contexts (but basically ignore the ones not in the list)
 * Add a Boolean property to restrict publishing of 'checks' to maintainers or to all patchwork users. This can be expanded on later (see below)
 * [OPTIONAL] Add an option to turn 'Check' support on/off. Some people won't want the extra column in the summary field cluttering their view. This should be set to 'enabled' (i.e. use 'Checks') by default, IMO.

# 2) Add additional 'reporter' or 'bot'-type user:

Currently basic patchwork user rights are needed to submit a check. We could enforce this to allow maintainer-only rights (as stated above) but it would be better to add an additional 'reporter' role that would be allowed to publish results but not change the 'state' of a patch or do anything else. A bit like a daemon/service user in Linux, I guess.


What do you think? Is this a good compromise that solves your needs? If so, I don't think there is a great reason to normalize the Check model for one field so let's keep it as is. Let's get this existing work merged and I'll submit the work for the project configuration options by early next week. These two series above will require modifications to the 'Project' and 'User' models respectively and won't take me long (I've already started on it). I should have it ready for review by the middle of next week.

Cheers,
Stephen