Structured feeds

[Daniel Borkmann]

On 11/9/19 8:41 AM, Johannes Berg wrote:
> On Sat, 2019-11-09 at 01:18 +1100, Daniel Axtens wrote:
>>>
>>   - code that efficiently reads a public-inbox git repository/folder of
>>     git repositories and feeds it into the existing parser. I have very
>>     inefficient code that converts public-inbox to an mbox and then
>>     parses that, but I'm sure you can do better with a git library.
> 
> Somebody (Daniel Borkmann?) posted a (very fast) public-inbox git to
> maildir converter, with procmail support. I assume that would actually
> satisfy this step already, since you can just substitute the patchwork
> parser for procmail.
> 
>>   - careful thought about how to do this incrementally. It's obvious how
>>     to do email incrementally, but I think you need to keep an extra bit
>>     of state around to incrementally parse the git archive. I think.
> 
> Not sure he had an incremental mode figured out there, but that can't
> really be all *that* hard, just store the last-successfully-parsed git
> sha1?

Yep, that is what it is doing, so that we only need to walk the repo(s)
upon a new git fetch to the point where we stopped last time.

Thanks,
Daniel