[PATCH v2 0/7] Improvements to message parsing

[Daniel Axtens]

As described in my previous writeup, this cleaned up series includes:
 - some fixes to make testing easier by passing around command-line options
 - fixes for headers with invalid characters
 - A modified version of Stephen's parsemail managment command migration
 - more tests

I suspect the mailing list might drop patch 6 as it contains mbox
files, so I'll also send a pull request with the details on how to
pull it from git.

---
v2:
 - export fix, thanks Andy
 - broaden fixes to all headers, thanks Stephen
 - handle cases where the declared encoding doesn't work
 
Thanks to all for the feedback on the series.
---


Daniel Axtens (7):
  parsemail.sh: make testing easier
  parsemail-batch.sh: pass command-line options to parsemail
  Test for header preservation
  parser: parse headers containing invalid characters or codings
  parser: subject_check needs to call clean_header
  parser tests: test parsing of interesting mail
  parsemail: Convert to a management command

 patchwork/bin/parsemail-batch.sh                   |   8 +-
 patchwork/bin/parsemail.py                         | 114 ------------------
 patchwork/bin/parsemail.sh                         |  17 ++-
 patchwork/management/commands/parsemail.py         |  84 +++++++++++++
 patchwork/parser.py                                | 108 +++++++++++++----
 patchwork/settings/base.py                         |  60 +++++++++-
 patchwork/tests/__init__.py                        |  23 ++++
 patchwork/tests/mail/0012-invalid-header-char.mbox |  72 +++++++++++
 patchwork/tests/mail/0013-with-utf8-body.mbox      |  46 +++++++
 .../mail/0014-with-unencoded-utf8-headers.mbox     |  46 +++++++
 .../tests/mail/0015-with-invalid-utf8-headers.mbox |  46 +++++++
 patchwork/tests/test_management.py                 | 112 +++++++++++++++++
 patchwork/tests/test_parser.py                     | 133 ++++++++++++++++++++-
 patchwork/tests/utils.py                           |   2 +-
 14 files changed, 719 insertions(+), 152 deletions(-)
 delete mode 100755 patchwork/bin/parsemail.py
 create mode 100644 patchwork/management/commands/parsemail.py
 create mode 100644 patchwork/tests/mail/0012-invalid-header-char.mbox
 create mode 100644 patchwork/tests/mail/0013-with-utf8-body.mbox
 create mode 100644 patchwork/tests/mail/0014-with-unencoded-utf8-headers.mbox
 create mode 100644 patchwork/tests/mail/0015-with-invalid-utf8-headers.mbox
 create mode 100644 patchwork/tests/test_management.py

-- 
2.7.4