[PATCH] parser: Fix parsing of pull request emails with CRLF line endings on Python 2
Andrew Donnellan
andrew.donnellan at au1.ibm.com
Mon Jan 8 18:49:30 AEDT 2018
On 08/01/18 18:46, Andrew Donnellan wrote:
> When using Python 2, an incoming email that uses CRLF line endings won't
> have the line endings converted by the email parser. This means that the
> regex to detect pull request emails will fail to match.
>
> Clean up the line endings when we extract the email body to fix this. (On
> Python 3, the email parser policy fixes this for us at the initial email
> parsing stage.)
>
> Add a test pull request mbox with CRLF line endings to ensure we don't
> regress.
Hmm, sudden realisation that git handles line endings for us, so perhaps
we don't have an easy way of including a test here.
>
> Closes: #148 ("Parsing pull request emails broken on Python 2")
> Signed-off-by: Andrew Donnellan <andrew.donnellan at au1.ibm.com>
> ---
> patchwork/parser.py | 4 +
> .../mail/0017-git-pull-request-crlf-newlines.mbox | 348 +++++++++++++++++++++
> patchwork/tests/test_parser.py | 3 +
> 3 files changed, 355 insertions(+)
> create mode 100644 patchwork/tests/mail/0017-git-pull-request-crlf-newlines.mbox
>
> diff --git a/patchwork/parser.py b/patchwork/parser.py
> index 1568bc4..7c677db 100644
> --- a/patchwork/parser.py
> +++ b/patchwork/parser.py
> @@ -666,9 +666,13 @@ def clean_content(content):
> """Remove cruft from the email message.
>
> Catch signature (-- ) and list footer (_____) cruft.
> +
> + Change to Unix line endings (the Python 3 email module does this for us,
> + but not Python 2).
> """
> sig_re = re.compile(r'^(-- |_+)\n.*', re.S | re.M)
> content = sig_re.sub('', content)
> + content = content.replace('\r\n', '\n')
>
> return content.strip()
>
> diff --git a/patchwork/tests/mail/0017-git-pull-request-crlf-newlines.mbox b/patchwork/tests/mail/0017-git-pull-request-crlf-newlines.mbox
> new file mode 100644
> index 0000000..bad78ae
> --- /dev/null
> +++ b/patchwork/tests/mail/0017-git-pull-request-crlf-newlines.mbox
> @@ -0,0 +1,348 @@
> +From benh at kernel.crashing.org Fri Oct 22 11:51:02 2010
> +Return-Path: <linuxppc-dev-bounces+jk=ozlabs.org at lists.ozlabs.org>
> +X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on bilbo.ozlabs.org
> +X-Spam-Level:
> +X-Spam-Status: No, score=0.0 required=3.0 tests=none autolearn=disabled
> + version=3.3.1
> +X-Original-To: jk at ozlabs.org
> +Delivered-To: jk at ozlabs.org
> +Received: from bilbo.ozlabs.org (localhost [127.0.0.1])
> + by ozlabs.org (Postfix) with ESMTP id ED4B3100937
> + for <jk at ozlabs.org>; Fri, 22 Oct 2010 14:51:54 +1100 (EST)
> +Received: by ozlabs.org (Postfix)
> + id BF799B70CB; Fri, 22 Oct 2010 14:51:50 +1100 (EST)
> +Delivered-To: linuxppc-dev at ozlabs.org
> +Received: from gate.crashing.org (gate.crashing.org [63.228.1.57])
> + (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
> + (Client did not present a certificate)
> + by ozlabs.org (Postfix) with ESMTPS id 94629B7043
> + for <linuxppc-dev at ozlabs.org>; Fri, 22 Oct 2010 14:51:49 +1100 (EST)
> +Received: from [IPv6:::1] (localhost.localdomain [127.0.0.1])
> + by gate.crashing.org (8.14.1/8.13.8) with ESMTP id o9M3p3SP018234;
> + Thu, 21 Oct 2010 22:51:04 -0500
> +Subject: [git pull] Please pull powerpc.git next branch
> +From: Benjamin Herrenschmidt <benh at kernel.crashing.org>
> +To: Linus Torvalds <torvalds at linux-foundation.org>
> +Date: Fri, 22 Oct 2010 14:51:02 +1100
> +Message-ID: <1287719462.2198.37.camel at pasglop>
> +Mime-Version: 1.0
> +X-Mailer: Evolution 2.30.3
> +Cc: linuxppc-dev list <linuxppc-dev at ozlabs.org>,
> + Andrew Morton <akpm at linux-foundation.org>,
> + Linux Kernel list <linux-kernel at vger.kernel.org>
> +X-BeenThere: linuxppc-dev at lists.ozlabs.org
> +X-Mailman-Version: 2.1.13
> +Precedence: list
> +List-Id: Linux on PowerPC Developers Mail List <cbe-oss-dev.ozlabs.org>
> +List-Unsubscribe: <https://lists.ozlabs.org/options/linuxppc-dev>,
> + <mailto:linuxppc-dev-request at lists.ozlabs.org?subject=unsubscribe>
> +List-Archive: <http://lists.ozlabs.org/pipermail/linuxppc-dev>
> +List-Post: <mailto:linuxppc-dev at lists.ozlabs.org>
> +List-Help: <mailto:linuxppc-dev-request at lists.ozlabs.org?subject=help>
> +List-Subscribe: <https://lists.ozlabs.org/listinfo/linuxppc-dev>,
> + <mailto:linuxppc-dev-request at lists.ozlabs.org?subject=subscribe>
> +Content-Type: text/plain;
> + charset="us-ascii"
> +Content-Transfer-Encoding: 7bit
> +Sender: linuxppc-dev-bounces+jk=ozlabs.org at lists.ozlabs.org
> +Errors-To: linuxppc-dev-bounces+jk=ozlabs.org at lists.ozlabs.org
> +X-UID: 11446
> +X-Length: 16781
> +Status: R
> +X-Status: N
> +X-KMail-EncryptionState:
> +X-KMail-SignatureState:
> +X-KMail-MDN-Sent:
> +
> +Hi Linus !
> +
> +Here's powerpc's batch for this merge window. Mostly bits and pieces,
> +such as Anton doing some performance tuning left and right, and the
> +usual churn. One hilight is the support for the new Freescale e5500 core
> +(64-bit BookE). Another one is that we now wire up the whole lot of
> +socket calls as direct syscalls in addition to the old style indirect
> +method.
> +
> +Cheers,
> +Ben.
> +
> +The following changes since commit e10117d36ef758da0690c95ecffc09d5dd7da479:
> + Linus Torvalds (1):
> + Merge branch 'upstream-linus' of git://git.kernel.org/.../jgarzik/libata-dev
> +
> +are available in the git repository at:
> +
> + git://git.kernel.org/pub/scm/linux/kernel/git/benh/powerpc.git next
> +
> +Andreas Schwab (1):
> + powerpc: Remove fpscr use from [kvm_]cvt_{fd,df}
> +
> +Anton Blanchard (5):
> + powerpc: Optimise 64bit csum_partial
> + powerpc: Optimise 64bit csum_partial_copy_generic and add csum_and_copy_from_user
> + powerpc: Add 64bit csum_and_copy_to_user
> + powerpc: Feature nop out reservation clear when stcx checks address
> + powerpc: Check end of stack canary at oops time
> +
> +Arnd Bergmann (1):
> + powerpc/spufs: Use llseek in all file operations
> +
> +Benjamin Herrenschmidt (4):
> + powerpc/dma: Add optional platform override of dma_set_mask()
> + powerpc/dart_iommu: Support for 64-bit iommu bypass window on PCIe
> + Merge remote branch 'kumar/merge' into next
> + Merge remote branch 'jwb/next' into next
> +
> +Denis Kirjanov (1):
> + powerpc: Use is_32bit_task() helper to test 32-bit binary
> +
> +Harninder Rai (1):
> + powerpc/85xx: add cache-sram support
> +
> +Ian Munsie (1):
> + powerpc: Wire up direct socket system calls
> +
> +Ilya Yanok (1):
> + powerpc/mpc83xx: Support for MPC8308 P1M board
> +
> +Joe Perches (2):
> + powerpc: Use static const char arrays
> + powerpc: Remove pr_<level> uses of KERN_<level>
> +
> +Josh Boyer (1):
> + powerpc/44x: Update ppc44x_defconfig
> +
> +Julia Lawall (7):
> + powerpc/via-pmu-led.c: Add of_node_put to avoid memory leak
> + powerpc/maple: Add of_node_put to avoid memory leak
> + powerpc/powermac/pfunc_core.c: Add of_node_put to avoid memory leak
> + powerpc/cell: Add of_node_put to avoid memory leak
> + powerpc/chrp/nvram.c: Add of_node_put to avoid memory leak
> + powerpc/irq.c: Add of_node_put to avoid memory leak
> + i2c/i2c-pasemi.c: Fix unsigned return type
> +
> +Kumar Gala (11):
> + powerpc/ppc64e: Fix link problem when building ppc64e_defconfig
> + powerpc/fsl-pci: Fix MSI support on 83xx platforms
> + powerpc/mpc8xxx_gpio: Add support for 'qoriq-gpio' controllers
> + powerpc/fsl-booke: Add PCI device ids for P2040/P3041/P5010/P5020 QoirQ chips
> + powerpc/fsl-booke: Add p3041 DS board support
> + powerpc: Fix compile error with paca code on ppc64e
> + powerpc/fsl-booke: Add support for FSL 64-bit e5500 core
> + powerpc/fsl-booke: Add support for FSL Arch v1.0 MMU in setup_page_sizes
> + powerpc/fsl-booke64: Use TLB CAMs to cover linear mapping on FSL 64-bit chips
> + powerpc/fsl-booke: Add p5020 DS board support
> + powerpc/fsl-booke: Add e55xx (64-bit) smp defconfig
> +
> +Matthew McClintock (7):
> + powerpc/mm: Assume first cpu is boot_cpuid not 0
> + powerpc/kexec: make masking/disabling interrupts generic
> + powerpc/85xx: Remove call to mpic_teardown_this_cpu in kexec
> + powerpc/85xx: Minor fixups for kexec on 85xx
> + powerpc/85xx: flush dcache before resetting cores
> + powerpc/fsl_soc: Search all global-utilities nodes for rstccr
> + powerpc/fsl_booke: Add support to boot from core other than 0
> +
> +Michael Neuling (1):
> + powerpc: Move arch_sd_sibling_asym_packing() to smp.c
> +
> +Nathan Fontenot (3):
> + powerpc/pseries: Export device tree updating routines
> + powerpc/pseries: Export rtas_ibm_suspend_me()
> + powerpc/pseries: Partition migration in the kernel
> +
> +Nishanth Aravamudan (8):
> + powerpc/pci: Fix return type of BUID_{HI,LO} macros
> + powerpc/dma: Fix dma_iommu_dma_supported compare
> + powerpc/dma: Fix check for direct DMA support
> + powerpc/vio: Use put_device() on device_register failure
> + powerpc/viobus: Free TCE table on device release
> + powerpc/pseries: Use kmemdup
> + powerpc/pci: Cleanup device dma setup code
> + powerpc/pseries/xics: Use cpu_possible_mask rather than cpu_all_mask
> +
> +Paul Gortmaker (1):
> + powerpc: Fix invalid page flags in create TLB CAM path for PTE_64BIT
> +
> +Paul Mackerras (5):
> + powerpc: Abstract indexing of lppaca structs
> + powerpc: Dynamically allocate most lppaca structs
> + powerpc: Account time using timebase rather than PURR
> + powerpc/pseries: Re-enable dispatch trace log userspace interface
> + powerpc/perf: Fix sampling enable for PPC970
> +
> +Scott Wood (1):
> + oprofile/fsl emb: Don't set MSR[PMM] until after clearing the interrupt.
> +
> +Sean MacLennan (2):
> + powerpc: Fix incorrect .stabs entry for copy_32.S
> + powerpc: mtmsrd not defined
> +
> +Shaohui Xie (1):
> + fsl_rio: Add comments for sRIO registers.
> +
> +Stephen Rothwell (1):
> + powerpc: define a compat_sys_recv cond_syscall
> +
> +Timur Tabi (5):
> + powerpc: export ppc_proc_freq and ppc_tb_freq as GPL symbols
> + powerpc/watchdog: Allow the Book-E driver to be compiled as a module
> + powerpc/p1022: Add probing for individual DMA channels
> + powerpc/85xx: add ngPIXIS FPGA device tree node to the P1022DS board
> + powerpc/watchdog: Make default timeout for Book-E watchdog a Kconfig option
> +
> +Tirumala Marri (1):
> + powerpc/44x: Add support for the AMCC APM821xx SoC
> +
> +matt mooney (1):
> + powerpc/Makefiles: Change to new flag variables
> +
> + arch/powerpc/boot/addnote.c | 4 +-
> + arch/powerpc/boot/dts/bluestone.dts | 254 +++++++++++++
> + arch/powerpc/boot/dts/mpc8308_p1m.dts | 332 ++++++++++++++++
> + arch/powerpc/boot/dts/p1022ds.dts | 11 +
> + arch/powerpc/configs/44x/bluestone_defconfig | 68 ++++
> + arch/powerpc/configs/e55xx_smp_defconfig | 84 ++++
> + arch/powerpc/configs/ppc44x_defconfig | 9 +-
> + arch/powerpc/configs/ppc64e_defconfig | 4 +-
> + arch/powerpc/include/asm/checksum.h | 10 +
> + arch/powerpc/include/asm/compat.h | 4 +-
> + arch/powerpc/include/asm/cputable.h | 14 +-
> + arch/powerpc/include/asm/dma-mapping.h | 14 +-
> + arch/powerpc/include/asm/elf.h | 2 +-
> + arch/powerpc/include/asm/exception-64s.h | 3 +-
> + arch/powerpc/include/asm/fsl_85xx_cache_sram.h | 48 +++
> + arch/powerpc/include/asm/kexec.h | 1 +
> + arch/powerpc/include/asm/kvm_fpu.h | 4 +-
> + arch/powerpc/include/asm/lppaca.h | 29 ++
> + arch/powerpc/include/asm/machdep.h | 3 +
> + arch/powerpc/include/asm/mmu-book3e.h | 15 +
> + arch/powerpc/include/asm/paca.h | 10 +-
> + arch/powerpc/include/asm/page_64.h | 4 +-
> + arch/powerpc/include/asm/ppc-pci.h | 4 +-
> + arch/powerpc/include/asm/ppc_asm.h | 50 ++-
> + arch/powerpc/include/asm/processor.h | 4 +-
> + arch/powerpc/include/asm/pte-common.h | 7 +
> + arch/powerpc/include/asm/rtas.h | 1 +
> + arch/powerpc/include/asm/systbl.h | 19 +
> + arch/powerpc/include/asm/system.h | 4 +-
> + arch/powerpc/include/asm/time.h | 5 -
> + arch/powerpc/include/asm/unistd.h | 21 +-
> + arch/powerpc/kernel/Makefile | 4 +-
> + arch/powerpc/kernel/align.c | 4 +-
> + arch/powerpc/kernel/asm-offsets.c | 12 +-
> + arch/powerpc/kernel/cpu_setup_44x.S | 1 +
> + arch/powerpc/kernel/cpu_setup_fsl_booke.S | 15 +
> + arch/powerpc/kernel/cputable.c | 43 ++-
> + arch/powerpc/kernel/crash.c | 13 +-
> + arch/powerpc/kernel/dma-iommu.c | 21 +-
> + arch/powerpc/kernel/dma.c | 20 +-
> + arch/powerpc/kernel/entry_64.S | 40 ++
> + arch/powerpc/kernel/fpu.S | 10 -
> + arch/powerpc/kernel/head_fsl_booke.S | 10 +-
> + arch/powerpc/kernel/irq.c | 6 +-
> + arch/powerpc/kernel/lparcfg.c | 14 +-
> + arch/powerpc/kernel/machine_kexec.c | 24 ++
> + arch/powerpc/kernel/machine_kexec_32.c | 4 +
> + arch/powerpc/kernel/paca.c | 70 ++++-
> + arch/powerpc/kernel/pci-common.c | 4 +-
> + arch/powerpc/kernel/ppc970-pmu.c | 2 +
> + arch/powerpc/kernel/process.c | 12 -
> + arch/powerpc/kernel/ptrace.c | 2 +-
> + arch/powerpc/kernel/rtas.c | 4 +-
> + arch/powerpc/kernel/setup_32.c | 2 +-
> + arch/powerpc/kernel/smp.c | 14 +-
> + arch/powerpc/kernel/time.c | 275 +++++++-------
> + arch/powerpc/kernel/traps.c | 5 +
> + arch/powerpc/kernel/vdso.c | 6 +-
> + arch/powerpc/kernel/vdso32/Makefile | 6 +-
> + arch/powerpc/kernel/vdso64/Makefile | 6 +-
> + arch/powerpc/kernel/vio.c | 10 +-
> + arch/powerpc/kvm/Makefile | 2 +-
> + arch/powerpc/kvm/book3s_paired_singles.c | 44 +--
> + arch/powerpc/kvm/emulate.c | 4 +-
> + arch/powerpc/kvm/fpu.S | 8 -
> + arch/powerpc/lib/Makefile | 7 +-
> + arch/powerpc/lib/checksum_64.S | 482 +++++++++++++++++-------
> + arch/powerpc/lib/checksum_wrappers_64.c | 102 +++++
> + arch/powerpc/lib/copy_32.S | 2 +-
> + arch/powerpc/lib/ldstfp.S | 36 +-
> + arch/powerpc/lib/locks.c | 4 +-
> + arch/powerpc/lib/sstep.c | 8 +
> + arch/powerpc/math-emu/Makefile | 2 +-
> + arch/powerpc/mm/Makefile | 6 +-
> + arch/powerpc/mm/fault.c | 6 +
> + arch/powerpc/mm/fsl_booke_mmu.c | 15 +-
> + arch/powerpc/mm/mmu_context_nohash.c | 6 +-
> + arch/powerpc/mm/mmu_decl.h | 5 +-
> + arch/powerpc/mm/tlb_nohash.c | 56 +++-
> + arch/powerpc/mm/tlb_nohash_low.S | 2 +-
> + arch/powerpc/oprofile/Makefile | 4 +-
> + arch/powerpc/oprofile/backtrace.c | 2 +-
> + arch/powerpc/oprofile/op_model_fsl_emb.c | 15 +-
> + arch/powerpc/platforms/44x/Kconfig | 16 +
> + arch/powerpc/platforms/44x/ppc44x_simple.c | 1 +
> + arch/powerpc/platforms/83xx/Kconfig | 4 +-
> + arch/powerpc/platforms/83xx/mpc830x_rdb.c | 3 +-
> + arch/powerpc/platforms/85xx/Kconfig | 28 ++-
> + arch/powerpc/platforms/85xx/Makefile | 2 +
> + arch/powerpc/platforms/85xx/p1022_ds.c | 2 +
> + arch/powerpc/platforms/85xx/p3041_ds.c | 64 ++++
> + arch/powerpc/platforms/85xx/p5020_ds.c | 69 ++++
> + arch/powerpc/platforms/85xx/smp.c | 83 ++++-
> + arch/powerpc/platforms/Kconfig.cputype | 8 +-
> + arch/powerpc/platforms/cell/ras.c | 4 +-
> + arch/powerpc/platforms/cell/spider-pic.c | 4 +-
> + arch/powerpc/platforms/cell/spufs/file.c | 18 +
> + arch/powerpc/platforms/chrp/nvram.c | 4 +-
> + arch/powerpc/platforms/iseries/Makefile | 2 +-
> + arch/powerpc/platforms/iseries/dt.c | 4 +-
> + arch/powerpc/platforms/iseries/smp.c | 2 +-
> + arch/powerpc/platforms/maple/setup.c | 1 +
> + arch/powerpc/platforms/powermac/pfunc_core.c | 9 +-
> + arch/powerpc/platforms/pseries/Makefile | 13 +-
> + arch/powerpc/platforms/pseries/dlpar.c | 7 +-
> + arch/powerpc/platforms/pseries/dtl.c | 224 +++++++++---
> + arch/powerpc/platforms/pseries/lpar.c | 25 ++-
> + arch/powerpc/platforms/pseries/mobility.c | 362 ++++++++++++++++++
> + arch/powerpc/platforms/pseries/pseries.h | 9 +
> + arch/powerpc/platforms/pseries/setup.c | 52 +++
> + arch/powerpc/platforms/pseries/xics.c | 2 +-
> + arch/powerpc/sysdev/Makefile | 5 +-
> + arch/powerpc/sysdev/dart_iommu.c | 74 ++++-
> + arch/powerpc/sysdev/fsl_85xx_cache_ctlr.h | 101 +++++
> + arch/powerpc/sysdev/fsl_85xx_cache_sram.c | 159 ++++++++
> + arch/powerpc/sysdev/fsl_85xx_l2ctlr.c | 231 +++++++++++
> + arch/powerpc/sysdev/fsl_msi.c | 9 +-
> + arch/powerpc/sysdev/fsl_pci.c | 60 +++-
> + arch/powerpc/sysdev/fsl_pci.h | 1 +
> + arch/powerpc/sysdev/fsl_rio.c | 65 ++--
> + arch/powerpc/sysdev/fsl_soc.c | 20 +-
> + arch/powerpc/sysdev/mpc8xxx_gpio.c | 3 +
> + arch/powerpc/sysdev/pmi.c | 2 +-
> + arch/powerpc/xmon/Makefile | 4 +-
> + drivers/i2c/busses/i2c-pasemi.c | 2 +-
> + drivers/macintosh/via-pmu-led.c | 4 +-
> + drivers/watchdog/Kconfig | 22 +-
> + drivers/watchdog/booke_wdt.c | 47 ++-
> + include/linux/pci_ids.h | 8 +
> + kernel/sys_ni.c | 1 +
> + 130 files changed, 3676 insertions(+), 683 deletions(-)
> + create mode 100644 arch/powerpc/boot/dts/bluestone.dts
> + create mode 100644 arch/powerpc/boot/dts/mpc8308_p1m.dts
> + create mode 100644 arch/powerpc/configs/44x/bluestone_defconfig
> + create mode 100644 arch/powerpc/configs/e55xx_smp_defconfig
> + create mode 100644 arch/powerpc/include/asm/fsl_85xx_cache_sram.h
> + create mode 100644 arch/powerpc/lib/checksum_wrappers_64.c
> + create mode 100644 arch/powerpc/platforms/85xx/p3041_ds.c
> + create mode 100644 arch/powerpc/platforms/85xx/p5020_ds.c
> + create mode 100644 arch/powerpc/platforms/pseries/mobility.c
> + create mode 100644 arch/powerpc/sysdev/fsl_85xx_cache_ctlr.h
> + create mode 100644 arch/powerpc/sysdev/fsl_85xx_cache_sram.c
> + create mode 100644 arch/powerpc/sysdev/fsl_85xx_l2ctlr.c
> +
> +
> +_______________________________________________
> +Linuxppc-dev mailing list
> +Linuxppc-dev at lists.ozlabs.org
> +https://lists.ozlabs.org/listinfo/linuxppc-dev
> diff --git a/patchwork/tests/test_parser.py b/patchwork/tests/test_parser.py
> index c21040e..9dca54a 100644
> --- a/patchwork/tests/test_parser.py
> +++ b/patchwork/tests/test_parser.py
> @@ -545,6 +545,9 @@ class PatchParseTest(PatchTest):
> def test_git_pull_request(self):
> self._test_pull_request_parse('0001-git-pull-request.mbox')
>
> + def test_git_pull_request_crlf_newlines(self):
> + self._test_pull_request_parse('0017-git-pull-request-crlf-newlines.mbox')
> +
> def test_git_pull_wrapped_request(self):
> self._test_pull_request_parse('0002-git-pull-request-wrapped.mbox')
>
>
--
Andrew Donnellan OzLabs, ADL Canberra
andrew.donnellan at au1.ibm.com IBM Australia Limited
More information about the Patchwork
mailing list