[RFC PATCH] 3x improvement in patch listing

[Stewart Smith]

Daniel Axtens <dja at axtens.net> writes:
> Daniel Axtens <dja at axtens.net> writes:
>
>> Konstantin Ryabitsev <konstantin at linuxfoundation.org> writes:
>>
>>> On Wed, Aug 08, 2018 at 05:40:06PM +1000, Stewart Smith wrote:
>>>>There's two main bits that are really expensive when composing the list
>>>>of patches for a project: the query getting the list, and the query
>>>>finding the series for each patch.
>>>
>>> Stewart:
>>>
>>> Thanks for working on this! Do you think this would help with pagination 
>>> as well? I find that in semi-abandoned projects like 
>>> https://patchwork.kernel.org/project/qemu-devel/list/ it takes a few 
>>> seconds to load the list view due to 800+ pages of unprocessed patches.  
>>> I am currently considering an automated script that would auto-archive 
>>> patches older than 6 months, but if simply improving pagination times 
>>> would fix the issue, then I wouldn't need to bother.
>>
>> It should do. From a brief look at his patch, it fixes a hot path in
>> __init__.py that goes through the paginator (and which I was *sure* I
>> had looked at, but apparently I had not!) I'll do a quick sanity check
>
> Ah, I have figured out what I had thought I had fixed!
>
> So you're hitting the issue where the db has to fetch the 'diff',
> 'content' and 'headers' out of Submission just to list the patches,
> which is clearly utterly bonkers. This is *despite* the defer() we
> attempt to do - jk reported a Django bug for it: [1].

Yeah, that'd explain it. I was kind of going "wtf" a lot at that, and
then doing something kind of crazy to try and get things down to
something even *remotely* sane.

I was almost tempted to rewrite the page just as raw SQL rather than
using Django at all, at least then it won't randomly regress in the
future. I may be a cranky database guy with a healthy disdain for the
SQL that comes out of ORMs though :)

> I suppose it's worth taking the patch - the Django fix certainly isn't
> going to hit the 1.11 stable tree and we're not supporting 2.0 just
> yet. I do still want to check it properly first, so ...
>
>> on it and hopefully apply it to master and stable/2.1 within the next
>> few days. (but no promises!)
>
> ... this lack-of-a-firm-timeline still applies.

Any random testing / whatever I can do locally to try and help?

This is literally my first time poking at anything Django.

-- 
Stewart Smith
OPAL Architect, IBM.