[RFC PATCH] 3x improvement in patch listing

[Daniel Axtens]

Daniel Axtens <dja at axtens.net> writes:

> Konstantin Ryabitsev <konstantin at linuxfoundation.org> writes:
>
>> On Wed, Aug 08, 2018 at 05:40:06PM +1000, Stewart Smith wrote:
>>>There's two main bits that are really expensive when composing the list
>>>of patches for a project: the query getting the list, and the query
>>>finding the series for each patch.
>>
>> Stewart:
>>
>> Thanks for working on this! Do you think this would help with pagination 
>> as well? I find that in semi-abandoned projects like 
>> https://patchwork.kernel.org/project/qemu-devel/list/ it takes a few 
>> seconds to load the list view due to 800+ pages of unprocessed patches.  
>> I am currently considering an automated script that would auto-archive 
>> patches older than 6 months, but if simply improving pagination times 
>> would fix the issue, then I wouldn't need to bother.
>
> It should do. From a brief look at his patch, it fixes a hot path in
> __init__.py that goes through the paginator (and which I was *sure* I
> had looked at, but apparently I had not!) I'll do a quick sanity check

Ah, I have figured out what I had thought I had fixed!

So you're hitting the issue where the db has to fetch the 'diff',
'content' and 'headers' out of Submission just to list the patches,
which is clearly utterly bonkers. This is *despite* the defer() we
attempt to do - jk reported a Django bug for it: [1].

It looks like Debian picked up the patch for their release, but it
didn't actually make it into Django 1.11.6 ([1], comment 8)

I attempted a monkey-patch inside Patchwork [2] - the patch was NAKed,
but OzLabs is/was carrying a local fix.

I suppose it's worth taking the patch - the Django fix certainly isn't
going to hit the 1.11 stable tree and we're not supporting 2.0 just
yet. I do still want to check it properly first, so ...

> on it and hopefully apply it to master and stable/2.1 within the next
> few days. (but no promises!)

... this lack-of-a-firm-timeline still applies.

Regards,
Daniel

[1] https://code.djangoproject.com/ticket/28549#ticket
[2] https://patchwork.ozlabs.org/patch/809294/