[RFC PATCH] REST: Add new setting for maximum API page size

[Daniel Axtens]

>>>> FWIW we now have this applied on patchwork.ozlabs.org and it appears to
>>>> be working. Would like some more input as to what an appropriate default
>>>> limit is.
>>> 
>>> My completely fact-free feeling/opinion is that if it takes more than
>>> ~1sec to run on Ozlabs, it's probably too high. Apart from that, I'm not
>>> fussed.
>>
>> OzLabs.org is not a fast machine. 1 second round trip time == 100 per 
>> page. 500 per page == ~2.3 seconds round trip.
>>
>> A single 500 item load is still under half the time of doing 5 * 100 
>> item loads...

I wonder if we could let authenticated users do slower queries. Maybe
something for later. Let's split the difference at 250 then, I'd be
happy to merge that.

> I bet MySQL would execute the query quicker :)

In general (and I have tested this) mysql 5.7 executes patchwork queries
slower than postgres 9.6 - this is on my laptop on an SSD with a couple
100ks of patches across 2 or 3 projects.

> Anyway, that sounds really quite slow and we should look at why on earth
> it's that slow - is there some kind of horrific hidden join or poor
> indexing or query plan or something?

Yes.

So the 1.0 -> 2.0 transition was not good for performance because it
turns out databases do not map well to object-oriented structures.* I
didn't think to check for performance at the time, so we're
progressively improving that by denormalising stuff. (In fact that was a
major driver of the 2.1 release!) There is still further to go.

Regards,
Daniel

* Yes, I probably should have realised this at the time. something
  something unpaid volunteers something something patch review something
  something.

>
> -- 
> Stewart Smith
> OPAL Architect, IBM.