parsemail.sh hanging with large mails
andy.doan at linaro.org
Fri Sep 22 00:01:57 AEST 2017
On 09/21/2017 01:43 AM, David Demelier wrote:
> Hello all,
> It looks like parsemail.sh takes very long time to parse a large mail.
> I have a 20MB mail file and running parsemail.sh on it takes forever
> and starts eating all the memory and CPU time.
> See after 3 minutes:
> 547 www 1 103 0 351M 239M CPU1 1 8:31 100.68%
> What can I do to debug more?
This isn't super helpful, but I've found a similar thing on my instance
and the issue seems to be when calling "parse_patch". It basically winds
up taking a really big buffer, splitting into an array of lines, and
building up another big buffer. My solution has always been too much of
a hack to propose here, but we work around it with something like:
# Don't try and process >5Mb patches, they flood the server
if len(patch) > 5000000:
raise MemoryError('patch too large to process: %d' % len(patch))
patch = parse_patch(patch)
And we just don't bother tracking large patches.
More information about the Patchwork