[PATCH 0/6] A sketch of flattening the models with live migration

[Daniel Axtens]

We want to flatten CoverLetter and Patch into Submission for performance
and simplicity reasons.

We also want to avoid some of the debacles of previous migrations where
sites were offline for hours to perform the migration, or needed to
drop the migration out of Python entirely and in to raw SQL.

So this is a sketch or outline of an approach were we migrate diff
and pull_url from Patch to Submission using live-ish migrations (not
fully live, but with no data migrations in the schema changes) and
then drop the CoverLetter model.

The way this works is pretty straight forward and is documented in
more detail in the patches - basically patches 1-3 are all minor
fixups, then you bring down your instance, apply patch 4, do the
schema migration, bring up your instance, perform the data migration
in the background, then bring down your instance, apply patches 5 and
6, do those migrations, and then you're done. Total downtime should be
very short (hopefully no more than a few minutes), depending on how
long it takes to add columns to your database.

Lots to do before this is a full approach - I still have to migrate
all the rest of the fields, and then clean up everything that depends
on a Patch model existing. But I feel like this at least demonstrates
that we can do the really hard work of the migration - moving the data
around - in a way that is friendly to sysadmins of large instances.

Daniel Axtens (6):
  pyenv: also install requirements for python2
  tests: make test suite pass with XML-RPC disabled
  parsearchive,mail: use repr() to get a human readable exception
  Begin live migration of diff and pull_url to Submission
  Migration part 2: drop old fields
  Burninate CoverLetter

 patchwork/admin.py                            |   3 -
 patchwork/api/cover.py                        |  16 +--
 patchwork/api/embedded.py                     |   2 +-
 patchwork/api/filters.py                      |  13 ++-
 patchwork/api/patch.py                        |  10 +-
 .../commands/livemigrate-v22-v23.py           |  35 ++++++
 patchwork/management/commands/parsearchive.py |   6 +-
 patchwork/management/commands/parsemail.py    |   2 +-
 .../0037_prepare_old_diff_pull_url.py         |  35 ++++++
 .../0038_submission_new_diff_pull_url.py      |  25 ++++
 patchwork/migrations/0039_sanity_check.py     |  27 +++++
 .../migrations/0040_drop_old_diff_pull_url.py |  23 ++++
 patchwork/migrations/0041_drop_coverletter.py |  29 +++++
 patchwork/models.py                           | 110 +++++++++++++++---
 patchwork/parser.py                           |   9 +-
 patchwork/signals.py                          |   4 +-
 patchwork/templates/patchwork/submission.html |  10 +-
 patchwork/tests/api/test_cover.py             |   5 +-
 patchwork/tests/test_about.py                 |  24 +++-
 patchwork/tests/test_series.py                |  14 +--
 patchwork/tests/test_xmlrpc.py                |   3 +
 patchwork/tests/utils.py                      |   6 +-
 patchwork/views/cover.py                      |  25 ++--
 patchwork/views/patch.py                      |   1 +
 patchwork/views/xmlrpc.py                     |   5 +-
 tools/docker/Dockerfile                       |  11 +-
 26 files changed, 378 insertions(+), 75 deletions(-)
 create mode 100644 patchwork/management/commands/livemigrate-v22-v23.py
 create mode 100644 patchwork/migrations/0037_prepare_old_diff_pull_url.py
 create mode 100644 patchwork/migrations/0038_submission_new_diff_pull_url.py
 create mode 100644 patchwork/migrations/0039_sanity_check.py
 create mode 100644 patchwork/migrations/0040_drop_old_diff_pull_url.py
 create mode 100644 patchwork/migrations/0041_drop_coverletter.py

-- 
2.20.1