parsemail.sh hanging with large mails

[Andy Doan]

On 09/21/2017 01:43 AM, David Demelier wrote:
> Hello all,
> 
> It looks like parsemail.sh takes very long time to parse a large mail.
> I have a 20MB mail file and running parsemail.sh on it takes forever
> and starts eating all the memory and CPU time.
> 
> See after 3 minutes:
> 
>    547 www           1 103    0   351M   239M CPU1    1   8:31 100.68%
> python2.7
> 
> 
> What can I do to debug more?
> 

This isn't super helpful, but I've found a similar thing on my instance 
and the issue seems to be when calling "parse_patch". It basically winds 
up taking a really big buffer, splitting into an array of lines, and 
building up another big buffer. My solution has always been too much of 
a hack to propose here, but we work around it with something like:

     # Don't try and process >5Mb patches, they flood the server
     if len(patch) > 5000000:
         raise MemoryError('patch too large to process: %d' % len(patch))
     patch = parse_patch(patch)[0]

And we just don't bother tracking large patches.