[PATCH 0/6] A sketch of flattening the models with live migration

[Daniel Axtens]

Daniel Axtens <dja at axtens.net> writes:

> We want to flatten CoverLetter and Patch into Submission for performance
> and simplicity reasons.
>
> We also want to avoid some of the debacles of previous migrations where
> sites were offline for hours to perform the migration, or needed to
> drop the migration out of Python entirely and in to raw SQL.
>
> So this is a sketch or outline of an approach were we migrate diff
> and pull_url from Patch to Submission using live-ish migrations (not
> fully live, but with no data migrations in the schema changes) and
> then drop the CoverLetter model.

I now have a full implementation which I'm still cleaning up and
rebasing in the lead up to an RFC:
https://github.com/daxtens/patchwork/tree/flatten

It does away with the Patch and CoverLetter models - everything is now a
Submission and we have submission.is_patch()/Submission.patch_objects
and submission.is_cover()/Submission.cover_objects instead.

Daniel

>
> The way this works is pretty straight forward and is documented in
> more detail in the patches - basically patches 1-3 are all minor
> fixups, then you bring down your instance, apply patch 4, do the
> schema migration, bring up your instance, perform the data migration
> in the background, then bring down your instance, apply patches 5 and
> 6, do those migrations, and then you're done. Total downtime should be
> very short (hopefully no more than a few minutes), depending on how
> long it takes to add columns to your database.
>
> Lots to do before this is a full approach - I still have to migrate
> all the rest of the fields, and then clean up everything that depends
> on a Patch model existing. But I feel like this at least demonstrates
> that we can do the really hard work of the migration - moving the data
> around - in a way that is friendly to sysadmins of large instances.
>
> Daniel Axtens (6):
>   pyenv: also install requirements for python2
>   tests: make test suite pass with XML-RPC disabled
>   parsearchive,mail: use repr() to get a human readable exception
>   Begin live migration of diff and pull_url to Submission
>   Migration part 2: drop old fields
>   Burninate CoverLetter
>
>  patchwork/admin.py                            |   3 -
>  patchwork/api/cover.py                        |  16 +--
>  patchwork/api/embedded.py                     |   2 +-
>  patchwork/api/filters.py                      |  13 ++-
>  patchwork/api/patch.py                        |  10 +-
>  .../commands/livemigrate-v22-v23.py           |  35 ++++++
>  patchwork/management/commands/parsearchive.py |   6 +-
>  patchwork/management/commands/parsemail.py    |   2 +-
>  .../0037_prepare_old_diff_pull_url.py         |  35 ++++++
>  .../0038_submission_new_diff_pull_url.py      |  25 ++++
>  patchwork/migrations/0039_sanity_check.py     |  27 +++++
>  .../migrations/0040_drop_old_diff_pull_url.py |  23 ++++
>  patchwork/migrations/0041_drop_coverletter.py |  29 +++++
>  patchwork/models.py                           | 110 +++++++++++++++---
>  patchwork/parser.py                           |   9 +-
>  patchwork/signals.py                          |   4 +-
>  patchwork/templates/patchwork/submission.html |  10 +-
>  patchwork/tests/api/test_cover.py             |   5 +-
>  patchwork/tests/test_about.py                 |  24 +++-
>  patchwork/tests/test_series.py                |  14 +--
>  patchwork/tests/test_xmlrpc.py                |   3 +
>  patchwork/tests/utils.py                      |   6 +-
>  patchwork/views/cover.py                      |  25 ++--
>  patchwork/views/patch.py                      |   1 +
>  patchwork/views/xmlrpc.py                     |   5 +-
>  tools/docker/Dockerfile                       |  11 +-
>  26 files changed, 378 insertions(+), 75 deletions(-)
>  create mode 100644 patchwork/management/commands/livemigrate-v22-v23.py
>  create mode 100644 patchwork/migrations/0037_prepare_old_diff_pull_url.py
>  create mode 100644 patchwork/migrations/0038_submission_new_diff_pull_url.py
>  create mode 100644 patchwork/migrations/0039_sanity_check.py
>  create mode 100644 patchwork/migrations/0040_drop_old_diff_pull_url.py
>  create mode 100644 patchwork/migrations/0041_drop_coverletter.py
>
> -- 
> 2.20.1