Checks API and pwclient

[Finucane, Stephen]

On 17 Feb 10:40, Andy Doan wrote:
> I was looking at trying to make use of the "checks" API via XMLRPC.
> The first problem I noticed is that the client passes invalid
> parameters to the server, so the command doesn't work. I started to
> look at fixing this, but have gotten bogged down in how this was
> intended to work.
> 
> I'm wondering if there are plans or proposed patches around this
> feature, and if not is there interest in making the feature more
> usable?

Hey Andy,

So there's two questions here IMO: how is the API supposed to work, and
how usable is it in its current form? I've given long answers to both below.

> The things I'd like from the API are:
> 
>  * list checks for a given patch or maybe list of patches.
>  * add a check to a patch
> 
> If this sounds sensible, I'd be willing to create a patch-set.

So there is work around series support and a REST API implementation,
as discussed below. These might impact anything you do in the long run
(though I am totally open to input here). As far as near-term work on
the XML-RPC API and/or pwclient tools go, however, I'd be delighted to
see any contributions you could make and will certainly work with you
to get them in. I'm not working on these right now and don't think
anyone else is either. Unless someone else replies with different input
then please feel free to proceed :)

Cheers,
Stephen

---

Q. How is the Check API _supposed_ to work?

A. The idea of checks is that they store the 'status' of test along
   with metadata (a brief 'description' and a 'url' to stored logs).
   Tests from different CI systems (e.g. different vendors) or
   different types of tests (e.g. unit tests, integration tests, coding
   style tests) are indicated by a unique 'context'. If a "check" is
   submitted for a patch that already has a "check" with the same
   'context', then this is assumed to be an update and the existing
   check is no longer available to the user. This is illustrated in
   some dodgy pseudocode below:

     >> create new check with context 'intel/checkstyle' and status
         'success'
     >> list checks for patch x
     [Check(context='intel/checkstyle', status='success', ...)]
     >> create new check with context 'intel/checkstyle' and status
         'fail'
     >> list checks for patch x
     [Check(context='intel/checkstyle', status'fail', ...)]
     >> create new check with context 'intel/perf' and status
         'success'
     >> list checks for patch x
     [Check(context='intel/checkstyle', status='fail', ...),
      Check(context='intel/perf', status='success', ...)]

   So that's what's _supposed_ to happen.

Q. How usable is the Check API in it's current form?

A. So in it's current form, using the API requires a _lot_ of work
   from the perspective of the tester. There are reasons for this are
   two-fold:

     1. The XML-RPC API
     2. The lack of series support

   The former is an issue because XML-RPC is a really awkward standard
   to develop against today: it's not that widely used anymore and, as
   a result of this, tools like Jenkins don't tend to provide plugins
   to interact with such APIs. As a result, implementing a testing
   infrastructure requires a lot of scripts to both pull changes from
   patchwork and post results. This is _way_ too much work IMO. Maybe
   we could develop our own plugins for the likes of Jenkins (and I may
   do this at some point), but what we need long-term is to use a more
   popular "protocol". The clear choice is REST, so I'm working on a
   REST API.

   The latter, on the other hand, is an issue because if we're going to
   test patch four of a five patch series then we have to know about
   the relationship with patch one. Once again this is currently
   "doable" using a lot of custom scripting on the client-end, but
   patchwork should be doing this for us. We badly need patchwork to
   recognise series relationships.

   There are a few folks who have done a lot of work on both features,
   but we have nothing that can be upstreamed yet. There are two things
   we need to be careful of. We need to ensure that any packages we use
   are available on enterprise distros like Debian, Ubuntu LTS etc. In
   addition, any changes have to be stable and absolutely cannot result
   in dropped patches.

   Both of these features will form part of the next release, v2.0.0. 
   I'm working on both features diligently, but REST APIs are
   difficult to scope out and secure, while series support is a really
   tough problem in general (people aren't robots and do silly things).
   If we're really lucky someone will come in with a fully functional
   solution to one or both of the above and the problem will be solved.
   Until that happens though, this takes priority over further
   refinement of the actual Check feature.