[PATCH v2 0/9] Tools and fixes for parallel parsing

[Daniel Axtens]

Patches 5-9 applied to master and stable/2.0.

Regards,
Daniel

Daniel Axtens <dja at axtens.net> writes:

> Thomas Petazzoni reported that Patchwork would occasionally lose
> Buildroot email. Andrew - having talked to jk and sfr - suggested that
> this may be race-condition related.
>
> I investigated and found some bugs. I first had to develop some tools.
> Along the way I found other unrelated bugs too.
>
> Patches 1-4 are tooling - ways to do parallel parsing of messages and
> get and compare the output. (Patch 1 fixes an issue I found when
> running the tool from patch 2)
>
> Patch 5 is an unrelated fix that came up along the way and
> demonstrates that humans remain the best fuzzers, and that Python's
> email module is still adorably* quirky.
>
> Patch 6 is a bug that came up very quickly in testing but is unlikely
> to be the actual bug Buildroot is hitting, as it can only occur the
> first time an email address is seen.
>
> Patch 7 is a related tidy-up/optimisation.
>
> Patch 8 fixes up a MySQL-only bug, but also adds some robustness.
>
> I think patch 9 closes the most likely issue for Buildroot patches.
>
> Pending review, patches 5, 6, 8 and 9 should go to stable.
>
> V2: Address review comments from Andrew, see patches.
>     Added the new python script to the automatic pep8 testing.
>     Minor improvement to parallel parse script to allow different
>      pythons to be used.
>
> Regards,
> Daniel
>
> Cc: Andrew Donnellan <andrew.donnellan at au1.ibm.com>
> Cc: Thomas Petazzoni <thomas.petazzoni at bootlin.com>
>
> Daniel Axtens (9):
>   tools/docker: assume terminal supports utf-8
>   debugging: add command to dump patches and series
>   tools/scripts: split a mbox N ways
>   tools/scripts: parallel_parsearchive - load archives in parallel
>   parser: Handle even more exotically broken headers
>   parser: close a TOCTTOU bug on Person creation
>   parser: avoid an unnecessary UPDATE of Person
>   parser: use Patch.objects.create instead of save()
>   parser: don't fail on multiple SeriesReferences
>
>  patchwork/management/commands/debug_dump.py | 46 ++++++++++++++
>  patchwork/parser.py                         | 96 +++++++++++++++++++----------
>  patchwork/tests/fuzztests/x-face.mbox       | 58 +++++++++++++++++
>  patchwork/tests/test_parser.py              | 44 ++++++-------
>  tools/docker/Dockerfile                     |  1 +
>  tools/scripts/parallel_parsearchive.sh      | 61 ++++++++++++++++++
>  tools/scripts/split_mail.py                 | 80 ++++++++++++++++++++++++
>  tox.ini                                     |  2 +-
>  8 files changed, 330 insertions(+), 58 deletions(-)
>  create mode 100644 patchwork/management/commands/debug_dump.py
>  create mode 100644 patchwork/tests/fuzztests/x-face.mbox
>  create mode 100755 tools/scripts/parallel_parsearchive.sh
>  create mode 100755 tools/scripts/split_mail.py
>
> -- 
> 2.14.1