[PATCH v3 3/5] models: Convert Series-Patch relationship to 1:N

[Stephen Finucane]

On Sat, 2018-10-27 at 11:26 +1100, Daniel Axtens wrote:
> Daniel Axtens <dja at axtens.net> writes:
> 
> > Hi Stephen,
> > 
> > > > I'm cloning the lkml public-inbox git repos and attempting to import
> > > > them into patchwork (which requires a new helper script). That'll be
> > > > over a million messages. Then I will time the migration and watch the
> > > > memory usage. That's the one thing I want to check; apart from that I
> > > > have no qualms with the series.
> > > > 
> > > > If you want to do this yourself and report back I'm very happy to take
> > > > your word for the results, otherwise it should be no more than a day or
> > > > two given that I've now finished with the URL schema mangling.
> > > 
> > > If you're able to do it, I'm happy to wait. There's also the Bootstrap
> > > 4 thing but that's more subjective and not as crucial to have eyes on
> > > (it's supposed to look almost the same for now).
> > 
> > I have imported 2 of the 7 repos - each import takes 5-6 hours :(
> > In total I now have 340k patches and I think roughly the same number of
> > comments. On my laptop, the db queries to display the list page takes a
> > terrifying 12 seconds, so I have no idea how kernel.org's lkml pw is so
> > fast. (It has almost a million patches!)
> > 
> > I then tried to apply the patches but they no longer apply to master. So
> > I pulled out a branch where I had applied some before, applied the
> > remainder, and rebased that. 
> > 
> > I then ran 'time python3 manage.py migrate'.
> > 
> > The migration took 6 and a half minutes on my laptop and nothing was
> > oomkilled. So even for people with significantly more patches, the
> > migration should be doable.
> > 
> > Tests pass.
> > 
> > Tested-by: Daniel Axtens <dja at axtens.net>
> > (you'll have to add that manually as the patch isn't showing up in pw)
> 
> It seems I forgot to check this (d'oh!), but it now takes 112 db queries
> for me to load the patch list page. They're almost all of the form

Damn, I missed that. Sorry. Going forward, it seems we can test this
using assertNumQueries [1] and I'm going to work on introducing this
for the REST API test cases at least. In any case, I've the issue at
hand resolved now but forgot to CC you. Find it at [2].

Stephen

[1] https://docs.djangoproject.com/en/2.1/topics/testing/tools/
[2] http://patchwork.ozlabs.org/patch/990331/

> SELECT
> `patchwork_patch`.`submission_ptr_id`, `patchwork_patch`.`series_id`
> FROM `patchwork_patch` WHERE `patchwork_patch`.`submission_ptr_id` = NNNN
> 
> Stephen, could you check this out please?
> 
> Regards,
> Daniel
> 
> 
> > 
> > Regards,
> > Daniel
> > > 
> > > I'll stick the URL patch on my review queue for sometime this week.
> > > 
> > > Cheers,
> > > Stephen