[PATCH] parser: Fix parsing of pull request emails with CRLF line endings on Python 2

Andrew Donnellan andrew.donnellan at au1.ibm.com
Mon Jan 8 18:49:30 AEDT 2018


On 08/01/18 18:46, Andrew Donnellan wrote:
> When using Python 2, an incoming email that uses CRLF line endings won't
> have the line endings converted by the email parser. This means that the
> regex to detect pull request emails will fail to match.
> 
> Clean up the line endings when we extract the email body to fix this. (On
> Python 3, the email parser policy fixes this for us at the initial email
> parsing stage.)
> 
> Add a test pull request mbox with CRLF line endings to ensure we don't
> regress.

Hmm, sudden realisation that git handles line endings for us, so perhaps 
we don't have an easy way of including a test here.


> 
> Closes: #148 ("Parsing pull request emails broken on Python 2")
> Signed-off-by: Andrew Donnellan <andrew.donnellan at au1.ibm.com>
> ---
>   patchwork/parser.py                                |   4 +
>   .../mail/0017-git-pull-request-crlf-newlines.mbox  | 348 +++++++++++++++++++++
>   patchwork/tests/test_parser.py                     |   3 +
>   3 files changed, 355 insertions(+)
>   create mode 100644 patchwork/tests/mail/0017-git-pull-request-crlf-newlines.mbox
> 
> diff --git a/patchwork/parser.py b/patchwork/parser.py
> index 1568bc4..7c677db 100644
> --- a/patchwork/parser.py
> +++ b/patchwork/parser.py
> @@ -666,9 +666,13 @@ def clean_content(content):
>       """Remove cruft from the email message.
>   
>       Catch signature (-- ) and list footer (_____) cruft.
> +
> +    Change to Unix line endings (the Python 3 email module does this for us,
> +    but not Python 2).
>       """
>       sig_re = re.compile(r'^(-- |_+)\n.*', re.S | re.M)
>       content = sig_re.sub('', content)
> +    content = content.replace('\r\n', '\n')
>   
>       return content.strip()
>   
> diff --git a/patchwork/tests/mail/0017-git-pull-request-crlf-newlines.mbox b/patchwork/tests/mail/0017-git-pull-request-crlf-newlines.mbox
> new file mode 100644
> index 0000000..bad78ae
> --- /dev/null
> +++ b/patchwork/tests/mail/0017-git-pull-request-crlf-newlines.mbox
> @@ -0,0 +1,348 @@
> +From benh at kernel.crashing.org Fri Oct 22 11:51:02 2010
> +Return-Path: <linuxppc-dev-bounces+jk=ozlabs.org at lists.ozlabs.org>
> +X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on bilbo.ozlabs.org
> +X-Spam-Level:
> +X-Spam-Status: No, score=0.0 required=3.0 tests=none autolearn=disabled
> +	version=3.3.1
> +X-Original-To: jk at ozlabs.org
> +Delivered-To: jk at ozlabs.org
> +Received: from bilbo.ozlabs.org (localhost [127.0.0.1])
> +	by ozlabs.org (Postfix) with ESMTP id ED4B3100937
> +	for <jk at ozlabs.org>; Fri, 22 Oct 2010 14:51:54 +1100 (EST)
> +Received: by ozlabs.org (Postfix)
> +	id BF799B70CB; Fri, 22 Oct 2010 14:51:50 +1100 (EST)
> +Delivered-To: linuxppc-dev at ozlabs.org
> +Received: from gate.crashing.org (gate.crashing.org [63.228.1.57])
> +	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
> +	(Client did not present a certificate)
> +	by ozlabs.org (Postfix) with ESMTPS id 94629B7043
> +	for <linuxppc-dev at ozlabs.org>; Fri, 22 Oct 2010 14:51:49 +1100 (EST)
> +Received: from [IPv6:::1] (localhost.localdomain [127.0.0.1])
> +	by gate.crashing.org (8.14.1/8.13.8) with ESMTP id o9M3p3SP018234;
> +	Thu, 21 Oct 2010 22:51:04 -0500
> +Subject: [git pull] Please pull powerpc.git next branch
> +From: Benjamin Herrenschmidt <benh at kernel.crashing.org>
> +To: Linus Torvalds <torvalds at linux-foundation.org>
> +Date: Fri, 22 Oct 2010 14:51:02 +1100
> +Message-ID: <1287719462.2198.37.camel at pasglop>
> +Mime-Version: 1.0
> +X-Mailer: Evolution 2.30.3
> +Cc: linuxppc-dev list <linuxppc-dev at ozlabs.org>,
> + Andrew Morton <akpm at linux-foundation.org>,
> + Linux Kernel list <linux-kernel at vger.kernel.org>
> +X-BeenThere: linuxppc-dev at lists.ozlabs.org
> +X-Mailman-Version: 2.1.13
> +Precedence: list
> +List-Id: Linux on PowerPC Developers Mail List <cbe-oss-dev.ozlabs.org>
> +List-Unsubscribe: <https://lists.ozlabs.org/options/linuxppc-dev>,
> +	<mailto:linuxppc-dev-request at lists.ozlabs.org?subject=unsubscribe>
> +List-Archive: <http://lists.ozlabs.org/pipermail/linuxppc-dev>
> +List-Post: <mailto:linuxppc-dev at lists.ozlabs.org>
> +List-Help: <mailto:linuxppc-dev-request at lists.ozlabs.org?subject=help>
> +List-Subscribe: <https://lists.ozlabs.org/listinfo/linuxppc-dev>,
> +	<mailto:linuxppc-dev-request at lists.ozlabs.org?subject=subscribe>
> +Content-Type: text/plain;
> +  charset="us-ascii"
> +Content-Transfer-Encoding: 7bit
> +Sender: linuxppc-dev-bounces+jk=ozlabs.org at lists.ozlabs.org
> +Errors-To: linuxppc-dev-bounces+jk=ozlabs.org at lists.ozlabs.org
> +X-UID: 11446
> +X-Length: 16781
> +Status: R
> +X-Status: N
> +X-KMail-EncryptionState:
> +X-KMail-SignatureState:
> +X-KMail-MDN-Sent:
> +
> +Hi Linus !
> +
> +Here's powerpc's batch for this merge window. Mostly bits and pieces,
> +such as Anton doing some performance tuning left and right, and the
> +usual churn. One hilight is the support for the new Freescale e5500 core
> +(64-bit BookE). Another one is that we now wire up the whole lot of
> +socket calls as direct syscalls in addition to the old style indirect
> +method.
> +
> +Cheers,
> +Ben.
> +
> +The following changes since commit e10117d36ef758da0690c95ecffc09d5dd7da479:
> +  Linus Torvalds (1):
> +        Merge branch 'upstream-linus' of git://git.kernel.org/.../jgarzik/libata-dev
> +
> +are available in the git repository at:
> +
> +  git://git.kernel.org/pub/scm/linux/kernel/git/benh/powerpc.git next
> +
> +Andreas Schwab (1):
> +      powerpc: Remove fpscr use from [kvm_]cvt_{fd,df}
> +
> +Anton Blanchard (5):
> +      powerpc: Optimise 64bit csum_partial
> +      powerpc: Optimise 64bit csum_partial_copy_generic and add csum_and_copy_from_user
> +      powerpc: Add 64bit csum_and_copy_to_user
> +      powerpc: Feature nop out reservation clear when stcx checks address
> +      powerpc: Check end of stack canary at oops time
> +
> +Arnd Bergmann (1):
> +      powerpc/spufs: Use llseek in all file operations
> +
> +Benjamin Herrenschmidt (4):
> +      powerpc/dma: Add optional platform override of dma_set_mask()
> +      powerpc/dart_iommu: Support for 64-bit iommu bypass window on PCIe
> +      Merge remote branch 'kumar/merge' into next
> +      Merge remote branch 'jwb/next' into next
> +
> +Denis Kirjanov (1):
> +      powerpc: Use is_32bit_task() helper to test 32-bit binary
> +
> +Harninder Rai (1):
> +      powerpc/85xx: add cache-sram support
> +
> +Ian Munsie (1):
> +      powerpc: Wire up direct socket system calls
> +
> +Ilya Yanok (1):
> +      powerpc/mpc83xx: Support for MPC8308 P1M board
> +
> +Joe Perches (2):
> +      powerpc: Use static const char arrays
> +      powerpc: Remove pr_<level> uses of KERN_<level>
> +
> +Josh Boyer (1):
> +      powerpc/44x: Update ppc44x_defconfig
> +
> +Julia Lawall (7):
> +      powerpc/via-pmu-led.c: Add of_node_put to avoid memory leak
> +      powerpc/maple: Add of_node_put to avoid memory leak
> +      powerpc/powermac/pfunc_core.c: Add of_node_put to avoid memory leak
> +      powerpc/cell: Add of_node_put to avoid memory leak
> +      powerpc/chrp/nvram.c: Add of_node_put to avoid memory leak
> +      powerpc/irq.c: Add of_node_put to avoid memory leak
> +      i2c/i2c-pasemi.c: Fix unsigned return type
> +
> +Kumar Gala (11):
> +      powerpc/ppc64e: Fix link problem when building ppc64e_defconfig
> +      powerpc/fsl-pci: Fix MSI support on 83xx platforms
> +      powerpc/mpc8xxx_gpio: Add support for 'qoriq-gpio' controllers
> +      powerpc/fsl-booke: Add PCI device ids for P2040/P3041/P5010/P5020 QoirQ chips
> +      powerpc/fsl-booke: Add p3041 DS board support
> +      powerpc: Fix compile error with paca code on ppc64e
> +      powerpc/fsl-booke: Add support for FSL 64-bit e5500 core
> +      powerpc/fsl-booke: Add support for FSL Arch v1.0 MMU in setup_page_sizes
> +      powerpc/fsl-booke64: Use TLB CAMs to cover linear mapping on FSL 64-bit chips
> +      powerpc/fsl-booke: Add p5020 DS board support
> +      powerpc/fsl-booke: Add e55xx (64-bit) smp defconfig
> +
> +Matthew McClintock (7):
> +      powerpc/mm: Assume first cpu is boot_cpuid not 0
> +      powerpc/kexec: make masking/disabling interrupts generic
> +      powerpc/85xx: Remove call to mpic_teardown_this_cpu in kexec
> +      powerpc/85xx: Minor fixups for kexec on 85xx
> +      powerpc/85xx: flush dcache before resetting cores
> +      powerpc/fsl_soc: Search all global-utilities nodes for rstccr
> +      powerpc/fsl_booke: Add support to boot from core other than 0
> +
> +Michael Neuling (1):
> +      powerpc: Move arch_sd_sibling_asym_packing() to smp.c
> +
> +Nathan Fontenot (3):
> +      powerpc/pseries: Export device tree updating routines
> +      powerpc/pseries: Export rtas_ibm_suspend_me()
> +      powerpc/pseries: Partition migration in the kernel
> +
> +Nishanth Aravamudan (8):
> +      powerpc/pci: Fix return type of BUID_{HI,LO} macros
> +      powerpc/dma: Fix dma_iommu_dma_supported compare
> +      powerpc/dma: Fix check for direct DMA support
> +      powerpc/vio: Use put_device() on device_register failure
> +      powerpc/viobus: Free TCE table on device release
> +      powerpc/pseries: Use kmemdup
> +      powerpc/pci: Cleanup device dma setup code
> +      powerpc/pseries/xics: Use cpu_possible_mask rather than cpu_all_mask
> +
> +Paul Gortmaker (1):
> +      powerpc: Fix invalid page flags in create TLB CAM path for PTE_64BIT
> +
> +Paul Mackerras (5):
> +      powerpc: Abstract indexing of lppaca structs
> +      powerpc: Dynamically allocate most lppaca structs
> +      powerpc: Account time using timebase rather than PURR
> +      powerpc/pseries: Re-enable dispatch trace log userspace interface
> +      powerpc/perf: Fix sampling enable for PPC970
> +
> +Scott Wood (1):
> +      oprofile/fsl emb: Don't set MSR[PMM] until after clearing the interrupt.
> +
> +Sean MacLennan (2):
> +      powerpc: Fix incorrect .stabs entry for copy_32.S
> +      powerpc: mtmsrd not defined
> +
> +Shaohui Xie (1):
> +      fsl_rio: Add comments for sRIO registers.
> +
> +Stephen Rothwell (1):
> +      powerpc: define a compat_sys_recv cond_syscall
> +
> +Timur Tabi (5):
> +      powerpc: export ppc_proc_freq and ppc_tb_freq as GPL symbols
> +      powerpc/watchdog: Allow the Book-E driver to be compiled as a module
> +      powerpc/p1022: Add probing for individual DMA channels
> +      powerpc/85xx: add ngPIXIS FPGA device tree node to the P1022DS board
> +      powerpc/watchdog: Make default timeout for Book-E watchdog a Kconfig option
> +
> +Tirumala Marri (1):
> +      powerpc/44x: Add support for the AMCC APM821xx SoC
> +
> +matt mooney (1):
> +      powerpc/Makefiles: Change to new flag variables
> +
> + arch/powerpc/boot/addnote.c                    |    4 +-
> + arch/powerpc/boot/dts/bluestone.dts            |  254 +++++++++++++
> + arch/powerpc/boot/dts/mpc8308_p1m.dts          |  332 ++++++++++++++++
> + arch/powerpc/boot/dts/p1022ds.dts              |   11 +
> + arch/powerpc/configs/44x/bluestone_defconfig   |   68 ++++
> + arch/powerpc/configs/e55xx_smp_defconfig       |   84 ++++
> + arch/powerpc/configs/ppc44x_defconfig          |    9 +-
> + arch/powerpc/configs/ppc64e_defconfig          |    4 +-
> + arch/powerpc/include/asm/checksum.h            |   10 +
> + arch/powerpc/include/asm/compat.h              |    4 +-
> + arch/powerpc/include/asm/cputable.h            |   14 +-
> + arch/powerpc/include/asm/dma-mapping.h         |   14 +-
> + arch/powerpc/include/asm/elf.h                 |    2 +-
> + arch/powerpc/include/asm/exception-64s.h       |    3 +-
> + arch/powerpc/include/asm/fsl_85xx_cache_sram.h |   48 +++
> + arch/powerpc/include/asm/kexec.h               |    1 +
> + arch/powerpc/include/asm/kvm_fpu.h             |    4 +-
> + arch/powerpc/include/asm/lppaca.h              |   29 ++
> + arch/powerpc/include/asm/machdep.h             |    3 +
> + arch/powerpc/include/asm/mmu-book3e.h          |   15 +
> + arch/powerpc/include/asm/paca.h                |   10 +-
> + arch/powerpc/include/asm/page_64.h             |    4 +-
> + arch/powerpc/include/asm/ppc-pci.h             |    4 +-
> + arch/powerpc/include/asm/ppc_asm.h             |   50 ++-
> + arch/powerpc/include/asm/processor.h           |    4 +-
> + arch/powerpc/include/asm/pte-common.h          |    7 +
> + arch/powerpc/include/asm/rtas.h                |    1 +
> + arch/powerpc/include/asm/systbl.h              |   19 +
> + arch/powerpc/include/asm/system.h              |    4 +-
> + arch/powerpc/include/asm/time.h                |    5 -
> + arch/powerpc/include/asm/unistd.h              |   21 +-
> + arch/powerpc/kernel/Makefile                   |    4 +-
> + arch/powerpc/kernel/align.c                    |    4 +-
> + arch/powerpc/kernel/asm-offsets.c              |   12 +-
> + arch/powerpc/kernel/cpu_setup_44x.S            |    1 +
> + arch/powerpc/kernel/cpu_setup_fsl_booke.S      |   15 +
> + arch/powerpc/kernel/cputable.c                 |   43 ++-
> + arch/powerpc/kernel/crash.c                    |   13 +-
> + arch/powerpc/kernel/dma-iommu.c                |   21 +-
> + arch/powerpc/kernel/dma.c                      |   20 +-
> + arch/powerpc/kernel/entry_64.S                 |   40 ++
> + arch/powerpc/kernel/fpu.S                      |   10 -
> + arch/powerpc/kernel/head_fsl_booke.S           |   10 +-
> + arch/powerpc/kernel/irq.c                      |    6 +-
> + arch/powerpc/kernel/lparcfg.c                  |   14 +-
> + arch/powerpc/kernel/machine_kexec.c            |   24 ++
> + arch/powerpc/kernel/machine_kexec_32.c         |    4 +
> + arch/powerpc/kernel/paca.c                     |   70 ++++-
> + arch/powerpc/kernel/pci-common.c               |    4 +-
> + arch/powerpc/kernel/ppc970-pmu.c               |    2 +
> + arch/powerpc/kernel/process.c                  |   12 -
> + arch/powerpc/kernel/ptrace.c                   |    2 +-
> + arch/powerpc/kernel/rtas.c                     |    4 +-
> + arch/powerpc/kernel/setup_32.c                 |    2 +-
> + arch/powerpc/kernel/smp.c                      |   14 +-
> + arch/powerpc/kernel/time.c                     |  275 +++++++-------
> + arch/powerpc/kernel/traps.c                    |    5 +
> + arch/powerpc/kernel/vdso.c                     |    6 +-
> + arch/powerpc/kernel/vdso32/Makefile            |    6 +-
> + arch/powerpc/kernel/vdso64/Makefile            |    6 +-
> + arch/powerpc/kernel/vio.c                      |   10 +-
> + arch/powerpc/kvm/Makefile                      |    2 +-
> + arch/powerpc/kvm/book3s_paired_singles.c       |   44 +--
> + arch/powerpc/kvm/emulate.c                     |    4 +-
> + arch/powerpc/kvm/fpu.S                         |    8 -
> + arch/powerpc/lib/Makefile                      |    7 +-
> + arch/powerpc/lib/checksum_64.S                 |  482 +++++++++++++++++-------
> + arch/powerpc/lib/checksum_wrappers_64.c        |  102 +++++
> + arch/powerpc/lib/copy_32.S                     |    2 +-
> + arch/powerpc/lib/ldstfp.S                      |   36 +-
> + arch/powerpc/lib/locks.c                       |    4 +-
> + arch/powerpc/lib/sstep.c                       |    8 +
> + arch/powerpc/math-emu/Makefile                 |    2 +-
> + arch/powerpc/mm/Makefile                       |    6 +-
> + arch/powerpc/mm/fault.c                        |    6 +
> + arch/powerpc/mm/fsl_booke_mmu.c                |   15 +-
> + arch/powerpc/mm/mmu_context_nohash.c           |    6 +-
> + arch/powerpc/mm/mmu_decl.h                     |    5 +-
> + arch/powerpc/mm/tlb_nohash.c                   |   56 +++-
> + arch/powerpc/mm/tlb_nohash_low.S               |    2 +-
> + arch/powerpc/oprofile/Makefile                 |    4 +-
> + arch/powerpc/oprofile/backtrace.c              |    2 +-
> + arch/powerpc/oprofile/op_model_fsl_emb.c       |   15 +-
> + arch/powerpc/platforms/44x/Kconfig             |   16 +
> + arch/powerpc/platforms/44x/ppc44x_simple.c     |    1 +
> + arch/powerpc/platforms/83xx/Kconfig            |    4 +-
> + arch/powerpc/platforms/83xx/mpc830x_rdb.c      |    3 +-
> + arch/powerpc/platforms/85xx/Kconfig            |   28 ++-
> + arch/powerpc/platforms/85xx/Makefile           |    2 +
> + arch/powerpc/platforms/85xx/p1022_ds.c         |    2 +
> + arch/powerpc/platforms/85xx/p3041_ds.c         |   64 ++++
> + arch/powerpc/platforms/85xx/p5020_ds.c         |   69 ++++
> + arch/powerpc/platforms/85xx/smp.c              |   83 ++++-
> + arch/powerpc/platforms/Kconfig.cputype         |    8 +-
> + arch/powerpc/platforms/cell/ras.c              |    4 +-
> + arch/powerpc/platforms/cell/spider-pic.c       |    4 +-
> + arch/powerpc/platforms/cell/spufs/file.c       |   18 +
> + arch/powerpc/platforms/chrp/nvram.c            |    4 +-
> + arch/powerpc/platforms/iseries/Makefile        |    2 +-
> + arch/powerpc/platforms/iseries/dt.c            |    4 +-
> + arch/powerpc/platforms/iseries/smp.c           |    2 +-
> + arch/powerpc/platforms/maple/setup.c           |    1 +
> + arch/powerpc/platforms/powermac/pfunc_core.c   |    9 +-
> + arch/powerpc/platforms/pseries/Makefile        |   13 +-
> + arch/powerpc/platforms/pseries/dlpar.c         |    7 +-
> + arch/powerpc/platforms/pseries/dtl.c           |  224 +++++++++---
> + arch/powerpc/platforms/pseries/lpar.c          |   25 ++-
> + arch/powerpc/platforms/pseries/mobility.c      |  362 ++++++++++++++++++
> + arch/powerpc/platforms/pseries/pseries.h       |    9 +
> + arch/powerpc/platforms/pseries/setup.c         |   52 +++
> + arch/powerpc/platforms/pseries/xics.c          |    2 +-
> + arch/powerpc/sysdev/Makefile                   |    5 +-
> + arch/powerpc/sysdev/dart_iommu.c               |   74 ++++-
> + arch/powerpc/sysdev/fsl_85xx_cache_ctlr.h      |  101 +++++
> + arch/powerpc/sysdev/fsl_85xx_cache_sram.c      |  159 ++++++++
> + arch/powerpc/sysdev/fsl_85xx_l2ctlr.c          |  231 +++++++++++
> + arch/powerpc/sysdev/fsl_msi.c                  |    9 +-
> + arch/powerpc/sysdev/fsl_pci.c                  |   60 +++-
> + arch/powerpc/sysdev/fsl_pci.h                  |    1 +
> + arch/powerpc/sysdev/fsl_rio.c                  |   65 ++--
> + arch/powerpc/sysdev/fsl_soc.c                  |   20 +-
> + arch/powerpc/sysdev/mpc8xxx_gpio.c             |    3 +
> + arch/powerpc/sysdev/pmi.c                      |    2 +-
> + arch/powerpc/xmon/Makefile                     |    4 +-
> + drivers/i2c/busses/i2c-pasemi.c                |    2 +-
> + drivers/macintosh/via-pmu-led.c                |    4 +-
> + drivers/watchdog/Kconfig                       |   22 +-
> + drivers/watchdog/booke_wdt.c                   |   47 ++-
> + include/linux/pci_ids.h                        |    8 +
> + kernel/sys_ni.c                                |    1 +
> + 130 files changed, 3676 insertions(+), 683 deletions(-)
> + create mode 100644 arch/powerpc/boot/dts/bluestone.dts
> + create mode 100644 arch/powerpc/boot/dts/mpc8308_p1m.dts
> + create mode 100644 arch/powerpc/configs/44x/bluestone_defconfig
> + create mode 100644 arch/powerpc/configs/e55xx_smp_defconfig
> + create mode 100644 arch/powerpc/include/asm/fsl_85xx_cache_sram.h
> + create mode 100644 arch/powerpc/lib/checksum_wrappers_64.c
> + create mode 100644 arch/powerpc/platforms/85xx/p3041_ds.c
> + create mode 100644 arch/powerpc/platforms/85xx/p5020_ds.c
> + create mode 100644 arch/powerpc/platforms/pseries/mobility.c
> + create mode 100644 arch/powerpc/sysdev/fsl_85xx_cache_ctlr.h
> + create mode 100644 arch/powerpc/sysdev/fsl_85xx_cache_sram.c
> + create mode 100644 arch/powerpc/sysdev/fsl_85xx_l2ctlr.c
> +
> +
> +_______________________________________________
> +Linuxppc-dev mailing list
> +Linuxppc-dev at lists.ozlabs.org
> +https://lists.ozlabs.org/listinfo/linuxppc-dev
> diff --git a/patchwork/tests/test_parser.py b/patchwork/tests/test_parser.py
> index c21040e..9dca54a 100644
> --- a/patchwork/tests/test_parser.py
> +++ b/patchwork/tests/test_parser.py
> @@ -545,6 +545,9 @@ class PatchParseTest(PatchTest):
>       def test_git_pull_request(self):
>           self._test_pull_request_parse('0001-git-pull-request.mbox')
>   
> +    def test_git_pull_request_crlf_newlines(self):
> +        self._test_pull_request_parse('0017-git-pull-request-crlf-newlines.mbox')
> +
>       def test_git_pull_wrapped_request(self):
>           self._test_pull_request_parse('0002-git-pull-request-wrapped.mbox')
>   
> 

-- 
Andrew Donnellan              OzLabs, ADL Canberra
andrew.donnellan at au1.ibm.com  IBM Australia Limited



More information about the Patchwork mailing list