[PATCH v2 0/9] Tools and fixes for parallel parsing

[Daniel Axtens]

Thomas Petazzoni reported that Patchwork would occasionally lose
Buildroot email. Andrew - having talked to jk and sfr - suggested that
this may be race-condition related.

I investigated and found some bugs. I first had to develop some tools.
Along the way I found other unrelated bugs too.

Patches 1-4 are tooling - ways to do parallel parsing of messages and
get and compare the output. (Patch 1 fixes an issue I found when
running the tool from patch 2)

Patch 5 is an unrelated fix that came up along the way and
demonstrates that humans remain the best fuzzers, and that Python's
email module is still adorably* quirky.

Patch 6 is a bug that came up very quickly in testing but is unlikely
to be the actual bug Buildroot is hitting, as it can only occur the
first time an email address is seen.

Patch 7 is a related tidy-up/optimisation.

Patch 8 fixes up a MySQL-only bug, but also adds some robustness.

I think patch 9 closes the most likely issue for Buildroot patches.

Pending review, patches 5, 6, 8 and 9 should go to stable.

V2: Address review comments from Andrew, see patches.
    Added the new python script to the automatic pep8 testing.
    Minor improvement to parallel parse script to allow different
     pythons to be used.

Regards,
Daniel

Cc: Andrew Donnellan <andrew.donnellan at au1.ibm.com>
Cc: Thomas Petazzoni <thomas.petazzoni at bootlin.com>

Daniel Axtens (9):
  tools/docker: assume terminal supports utf-8
  debugging: add command to dump patches and series
  tools/scripts: split a mbox N ways
  tools/scripts: parallel_parsearchive - load archives in parallel
  parser: Handle even more exotically broken headers
  parser: close a TOCTTOU bug on Person creation
  parser: avoid an unnecessary UPDATE of Person
  parser: use Patch.objects.create instead of save()
  parser: don't fail on multiple SeriesReferences

 patchwork/management/commands/debug_dump.py | 46 ++++++++++++++
 patchwork/parser.py                         | 96 +++++++++++++++++++----------
 patchwork/tests/fuzztests/x-face.mbox       | 58 +++++++++++++++++
 patchwork/tests/test_parser.py              | 44 ++++++-------
 tools/docker/Dockerfile                     |  1 +
 tools/scripts/parallel_parsearchive.sh      | 61 ++++++++++++++++++
 tools/scripts/split_mail.py                 | 80 ++++++++++++++++++++++++
 tox.ini                                     |  2 +-
 8 files changed, 330 insertions(+), 58 deletions(-)
 create mode 100644 patchwork/management/commands/debug_dump.py
 create mode 100644 patchwork/tests/fuzztests/x-face.mbox
 create mode 100755 tools/scripts/parallel_parsearchive.sh
 create mode 100755 tools/scripts/split_mail.py

-- 
2.14.1