[PATCH 00/10] Add series support

[Finucane, Stephen]

On 16 Jun 16:42, Russell Currey wrote:
> On Thu, 2016-06-16 at 09:21 +1000, Andrew Donnellan wrote:
> > On 15/06/16 19:07, Finucane, Stephen wrote:
> 
> <snip>
> 
> > > The best option we might have, if per-series reporting is really
> > > necessary, is to allow Check uploading against a Series endpoint. This
> > > would actually cause N Checks to be created - one for each Patch in the
> > > series - meaning each Patch could still be individually queried. It
> > > would be a bit of a lie (we didn't actually test the patch by itself,
> > > therefore it might be broken) and I wouldn't promote this workflow
> > > myself (bisectability FTW), but it could be a good way of dealing with
> > > extremely long-running or resource-intensive test suites, where
> > > per-patch validation would be too expensive.
> > 
> > I suppose this would be better than nothing, though apart from being 
> > mildly misleading it would also generate a lot of spam. If I'm 
> > submitting 5 test results against the entire series and 1-2 test results 
> > against each individual patch, and we have all of those results 
> > appearing on every patch, things could get more than a little bit 
> > confusing. Personally, I'd be inclined to submit series-wide test 
> > results as checks on the very last patch in the series instead of using 
> > this.

What kind of spam? There would be a list of checks in the patches
table (the 1-2 test results you talk about) along with the details of
the checks on each page (useful, IMO). We haven't configured anything
by way of email, but we can definitely take this into account when
implementing this.

> I don't think adding checks on the very last patch is particularly intuitive.
> Having a Series endpoint that creates checks on every patch in the series would
> be fine so long as there was some obvious way of distinguishing what was run on
> the series as a whole and what was run on the individual patch, whether it's
> something arbitrary like "[series]" in the check name or if there was some flag
> that triggered a visual differentiation.

I don't think it's that intuitive either, tbh. We have relatively
free-form fields in the 'context' and 'description' fields however: you
could do something as simple as reporting with a different context when
you ran a patch against a series rather than against the individual
patch? You could also provide a not in the description saying as much.

> The reason I brought this up is that Andrew and I wrote a tool to automatically
> test series as they're indexed in Patchwork that was built on the Series support
> in the fdo fork, I'll play around with this series and see if anything major is
> missing that would prevent us from doing the same with this implementation.

Sounds great. There will be a few differences in how they work, but I'm
happy to describe this when we get around to it. FWIM, I'm working on a
Jenkins powered solution, and I've also started working on a Buildbot
plugin.

Stephen