[PATCH v2 0/9] Tools and fixes for parallel parsing
dja at axtens.net
Sun Feb 25 01:50:11 AEDT 2018
Thomas Petazzoni reported that Patchwork would occasionally lose
Buildroot email. Andrew - having talked to jk and sfr - suggested that
this may be race-condition related.
I investigated and found some bugs. I first had to develop some tools.
Along the way I found other unrelated bugs too.
Patches 1-4 are tooling - ways to do parallel parsing of messages and
get and compare the output. (Patch 1 fixes an issue I found when
running the tool from patch 2)
Patch 5 is an unrelated fix that came up along the way and
demonstrates that humans remain the best fuzzers, and that Python's
email module is still adorably* quirky.
Patch 6 is a bug that came up very quickly in testing but is unlikely
to be the actual bug Buildroot is hitting, as it can only occur the
first time an email address is seen.
Patch 7 is a related tidy-up/optimisation.
Patch 8 fixes up a MySQL-only bug, but also adds some robustness.
I think patch 9 closes the most likely issue for Buildroot patches.
Pending review, patches 5, 6, 8 and 9 should go to stable.
V2: Address review comments from Andrew, see patches.
Added the new python script to the automatic pep8 testing.
Minor improvement to parallel parse script to allow different
pythons to be used.
Cc: Andrew Donnellan <andrew.donnellan at au1.ibm.com>
Cc: Thomas Petazzoni <thomas.petazzoni at bootlin.com>
Daniel Axtens (9):
tools/docker: assume terminal supports utf-8
debugging: add command to dump patches and series
tools/scripts: split a mbox N ways
tools/scripts: parallel_parsearchive - load archives in parallel
parser: Handle even more exotically broken headers
parser: close a TOCTTOU bug on Person creation
parser: avoid an unnecessary UPDATE of Person
parser: use Patch.objects.create instead of save()
parser: don't fail on multiple SeriesReferences
patchwork/management/commands/debug_dump.py | 46 ++++++++++++++
patchwork/parser.py | 96 +++++++++++++++++++----------
patchwork/tests/fuzztests/x-face.mbox | 58 +++++++++++++++++
patchwork/tests/test_parser.py | 44 ++++++-------
tools/docker/Dockerfile | 1 +
tools/scripts/parallel_parsearchive.sh | 61 ++++++++++++++++++
tools/scripts/split_mail.py | 80 ++++++++++++++++++++++++
tox.ini | 2 +-
8 files changed, 330 insertions(+), 58 deletions(-)
create mode 100644 patchwork/management/commands/debug_dump.py
create mode 100644 patchwork/tests/fuzztests/x-face.mbox
create mode 100755 tools/scripts/parallel_parsearchive.sh
create mode 100755 tools/scripts/split_mail.py
More information about the Patchwork