[c-lightning] Splitting gossipd

[Rusty Russell]

Hi all,

        Now we're post 0.6, I want to address some technical debt.
Much of that means enhancing unit tests and writing some canned protocol
tests (which will let us do fuzz testing), but we need to make sure our
architecture is correct otherwise we'll just end up writing tests twice.

Background:

The current gossipd maintains the graph topology, but *also* connects
and chats to all peers which are not part of an active channel.  These
two roles are quite separate, though it's slightly more efficient than
having a separate daemon because we don't need to pass messages back and
forth between them.

Unfortunately, the transition between gossipd and the other daemons has
been a source of bugs: it'd be nice to isolate this.  Thus I've begun
splitting the "connect and handshake" role of gossipd into a new
"connectd".  That's turned out pretty clean.

connectd itself has two parts: firstly it is responsible for answering
incoming connections, and trying to connect out to peers we want to talk
to, then handing them to the master.  Secondly, it takes peers from the
master which don't have a channel, and sits there forwarding gossip.

This "peer in gossip mode" handling is completely separate from the
"establish connection" part of connectd; worse, if the master decides it
*does* want to talk to a channel, it has to sync any partially read and
written gossip messages.

If, instead, we have a per-peer daemon which handles "just gossiping",
this is not a problem, as each daemon can be synchronous (thus
simpler).  It consumes a bit more resources per peer, but it now means
that *all peers* post-init are isolated into their own daemons.

I've yet to complete my implementation, but I thougth I'd share the
design so the patch doesn't come as a complete shock.  

Cheers,
Rusty.