[PATCH 0/9] Tools and fixes for parallel parsing
dja at axtens.net
Thu Feb 22 01:17:07 AEDT 2018
Thomas Petazzoni reported that Patchwork would occasionally lose
Buildroot email. Andrew - having talked to jk and sfr - suggested that
this may be race-condition related.
I investigated and found some bugs. I first had to develop some tools.
Along the way I found other unrelated bugs too.
Patches 1-4 are tooling - ways to do parallel parsing of messages and
get and compare the output. (Patch 1 fixes an issue I found when
running the tool from patch 2)
Patch 5 is an unrelated fix that came up along the way and
demonstrates that humans remain the best fuzzers, and that Python's
email module is still adorably* quirky.
Patch 6 is a bug that came up very quickly in testing but is unlikely
to be the actual bug Buildroot is hitting, as it can only occur the
first time an email address is seen.
Patch 7 is a related tidy-up/optimisation.
Patch 8 fixes up a MySQL-only bug, but also adds some robustness.
I think patch 9 closes the most likely issue for Buildroot patches.
Pending review, patches 5, 6, 8 and 9 should go to stable.
Cc: Andrew Donnellan <andrew.donnellan at au1.ibm.com>
Cc: Thomas Petazzoni <thomas.petazzoni at free-electrons.com>
Daniel Axtens (9):
tools/docker: assume terminal supports utf-8
debugging: add command to dump patches and series
tools/scripts: split a mbox N ways
tools/scripts: parallel_parsearchive - load archives in parallel
parser: Handle even more exotically broken headers
parser: close a TOCTTOU bug on Person creation
parser: avoid an unnecessary UPDATE of Person
parser: use Patch.objects.create instead of save()
parser: don't fail on multiple SeriesReferences
patchwork/management/commands/debug_dump.py | 46 +++++++++++++++
patchwork/parser.py | 92 +++++++++++++++++++----------
patchwork/tests/fuzztests/x-face.mbox | 58 ++++++++++++++++++
patchwork/tests/test_parser.py | 44 +++++++-------
tools/docker/Dockerfile | 1 +
tools/scripts/parallel_parsearchive.sh | 55 +++++++++++++++++
tools/scripts/split_mail.py | 76 ++++++++++++++++++++++++
7 files changed, 317 insertions(+), 55 deletions(-)
create mode 100644 patchwork/management/commands/debug_dump.py
create mode 100644 patchwork/tests/fuzztests/x-face.mbox
create mode 100755 tools/scripts/parallel_parsearchive.sh
create mode 100755 tools/scripts/split_mail.py
More information about the Patchwork