[PATCH v2 0/9] Tools and fixes for parallel parsing
dja at axtens.net
Wed Mar 7 01:53:55 AEDT 2018
Patches 5-9 applied to master and stable/2.0.
Daniel Axtens <dja at axtens.net> writes:
> Thomas Petazzoni reported that Patchwork would occasionally lose
> Buildroot email. Andrew - having talked to jk and sfr - suggested that
> this may be race-condition related.
> I investigated and found some bugs. I first had to develop some tools.
> Along the way I found other unrelated bugs too.
> Patches 1-4 are tooling - ways to do parallel parsing of messages and
> get and compare the output. (Patch 1 fixes an issue I found when
> running the tool from patch 2)
> Patch 5 is an unrelated fix that came up along the way and
> demonstrates that humans remain the best fuzzers, and that Python's
> email module is still adorably* quirky.
> Patch 6 is a bug that came up very quickly in testing but is unlikely
> to be the actual bug Buildroot is hitting, as it can only occur the
> first time an email address is seen.
> Patch 7 is a related tidy-up/optimisation.
> Patch 8 fixes up a MySQL-only bug, but also adds some robustness.
> I think patch 9 closes the most likely issue for Buildroot patches.
> Pending review, patches 5, 6, 8 and 9 should go to stable.
> V2: Address review comments from Andrew, see patches.
> Added the new python script to the automatic pep8 testing.
> Minor improvement to parallel parse script to allow different
> pythons to be used.
> Cc: Andrew Donnellan <andrew.donnellan at au1.ibm.com>
> Cc: Thomas Petazzoni <thomas.petazzoni at bootlin.com>
> Daniel Axtens (9):
> tools/docker: assume terminal supports utf-8
> debugging: add command to dump patches and series
> tools/scripts: split a mbox N ways
> tools/scripts: parallel_parsearchive - load archives in parallel
> parser: Handle even more exotically broken headers
> parser: close a TOCTTOU bug on Person creation
> parser: avoid an unnecessary UPDATE of Person
> parser: use Patch.objects.create instead of save()
> parser: don't fail on multiple SeriesReferences
> patchwork/management/commands/debug_dump.py | 46 ++++++++++++++
> patchwork/parser.py | 96 +++++++++++++++++++----------
> patchwork/tests/fuzztests/x-face.mbox | 58 +++++++++++++++++
> patchwork/tests/test_parser.py | 44 ++++++-------
> tools/docker/Dockerfile | 1 +
> tools/scripts/parallel_parsearchive.sh | 61 ++++++++++++++++++
> tools/scripts/split_mail.py | 80 ++++++++++++++++++++++++
> tox.ini | 2 +-
> 8 files changed, 330 insertions(+), 58 deletions(-)
> create mode 100644 patchwork/management/commands/debug_dump.py
> create mode 100644 patchwork/tests/fuzztests/x-face.mbox
> create mode 100755 tools/scripts/parallel_parsearchive.sh
> create mode 100755 tools/scripts/split_mail.py
More information about the Patchwork