[PATCH 01/11] Improve patch listing performance (~3x)

[Stewart Smith]

Stephen Finucane <stephen at that.guru> writes:
> On Fri, 2018-08-10 at 18:00 +1000, Stewart Smith wrote:
>> There's two main bits that are really expensive when composing the list
>> of patches for a project: the query getting the list, and the query
>> finding the series for each patch.
>> 
>> If we look at the query getting the list, it gets a lot of unnesseccary
>> fields such as 'headers' and 'content', even though we tell Django not
>> to. It turns out that Django seems to ignore the Submission relationship
>> and I have no idea how to force it to ignore that thing (defer doesn't
>> work) but if we go only, then it works okay.
>> 
>> From my import of ~8000 messages for a few projects, my laptop query
>> time (MySQL, as setup by whatever the docker-compose things do) goes
>> from:
>> 
>> http://localhost:8000/project/linux-kernel/list/
>> FROM:
>> 342ms SQL queries cold cache, 268ms warm cache
>> TO:
>> 118ms SQL queries cold cache, 88ms warm cache
>> 
>> Which is... non trivial to say the least.
>> 
>> The big jump is the patches.only change, and the removal of ordering
>> on the patchseries takes a further 10ms off. For some strange reason, it
>> seems rather hard to tell Django that you don't care what order the
>> results come back in for that query (if we do, then the db server has to
>> do a sort rather than just return each row)
>> 
>> Signed-off-by: Stewart Smith <stewart at linux.ibm.com>
>
> Tested this and it looks good to me. There's a migration missing (it's
> instead squashed into the first "add index" patch) but I can fix that
> at merge time.

Ahh, sorry about that.

> I was concerned that the combination of '.only' followed immediately by
> '.prefetch_related' (for checks and series) would be an issue and
> indeed it seems that, at this patch set at least, we do generate a
> query per check. However, this is no different to before and it
> resolved in a later check. As such:
>
> Reviewed-by: Stephen Finucane <stephen at that.guru>
>
> I'm not going to push this (or any of the series) yet in case Daniel
> wants to weigh in but will do so next week if there are no complaints.

Thanks for testing it all, increases my confidence with the patches as
well :)

-- 
Stewart Smith
OPAL Architect, IBM.