[snowpatch] [RFC] patchwork: get more than one page's worth of series

[Andrew Donnellan]

At present, when getting the list of series from Patchwork, we request a
single "page" of results (up to 20 series). This more or less works for
daemon mode up to a certain point - it may cause problems for an extremely
busy Patchwork instance, but for most setups it'll work fine.

However, when we're not running in daemon mode, and we specify --count=200,
we'd like to actually try and get 200 series, rather than testing the first
20 over and over again...

Instead of grabbing a single page, follow the "next" link that Patchwork
returns and request as many pages as possible. Change the query string to
get the maximum of 100 series per page, rather than the default 20.

This approach is rather naive, and won't work very well on Patchwork setups
with a large existing database.

Closes: #17 ("Enable the ability to go to page 2 (and beyond) of Patchwork
results")
Signed-off-by: Andrew Donnellan <andrew.donnellan at au1.ibm.com>

---

Depends-on: http://patchwork.ozlabs.org/patch/613373/

We need to work out a smarter method than just getting every single series
that Patchwork has ever seen across all projects - I don't think it's as
simple as trying to grab however many series are specified by --count...
Discussion going on at https://github.com/ruscur/snowpatch/issues/17
---
 src/main.rs      |  2 +-
 src/patchwork.rs | 12 +++++++++---
 2 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/src/main.rs b/src/main.rs
index d0177af..776a538 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -265,7 +265,7 @@ fn main() {
 
     // Poll patchwork for new series. For each series, get patches, apply and test.
     'daemon: loop {
-        let series_list = patchwork.get_series_query().unwrap().results.unwrap();
+        let series_list = patchwork.get_series_query().unwrap();
         for series in series_list {
             // If it's already been tested, we can skip it
             if series.test_state.is_some() {
diff --git a/src/patchwork.rs b/src/patchwork.rs
index c8b9179..279933f 100644
--- a/src/patchwork.rs
+++ b/src/patchwork.rs
@@ -37,7 +37,7 @@ use utils;
 
 // TODO: more constants.  constants for format strings of URLs and such.
 pub static PATCHWORK_API: &'static str = "/api/1.0";
-pub static PATCHWORK_QUERY: &'static str = "?ordering=-last_updated&related=expand";
+pub static PATCHWORK_QUERY: &'static str = "?ordering=-last_updated&related=expand&perpage=100";
 
 // /api/1.0/projects/*/series/
 
@@ -176,10 +176,16 @@ impl PatchworkServer {
             .header(Connection::close()).send()
     }
 
-    pub fn get_series_query(&self) -> Result<SeriesList, DecoderError> {
+    pub fn get_series_query(&self) -> Result<Vec<Series>, DecoderError> {
         let url = format!("{}{}/series/{}", &self.url,
                           PATCHWORK_API, PATCHWORK_QUERY);
-        json::decode(&self.get(&url).unwrap())
+        let mut page: SeriesList = try!(json::decode(&self.get(&url).unwrap()));
+        let mut series = page.results.unwrap();
+        while page.next.is_some() {
+            page = try!(json::decode(&self.get(&url).unwrap()));
+            series.extend(page.results.unwrap());
+        }
+        Ok(series)
     }
 
     pub fn get_patch(&self, series: &Series) -> PathBuf {
-- 
Andrew Donnellan              OzLabs, ADL Canberra
andrew.donnellan at au1.ibm.com  IBM Australia Limited