[c-lightning] Plugins/passthrough for next version?

[Christian Decker]

I think we have quite a few different options to foster extensibility on
top of a focused and lean lightning implementation, each with its own
pros and cons:

 - Driver: a program that controls the lightning daemon, tails its
   output, or has another way to stream events from the daemon, and
   interacts with the JSON-RPC we already have. That's basically how we
   drive the tests, both for c-lightning [1] as well as the
   integration tests across all implementations [2]. It's a bit cludgy
   since we tail the logs which were not intended for machine
   consumption, but it works right now and is how I've been building a
   number of applications. An event stream, e.g., streaming JSON events,
   would make this a lot easier.
 - Sidecar: this is just a small variation of the above, in that it
   doesn't control the daemon, and has to rely on polling it. It's
   higher overhead, slower and would also benefit from an event stream.
 - Callbacks: c-lightning hooks into certain events and calls out to a
   registered external program if something of interest has
   happened. This is basically the model `-blocknotify` option of
   bitcoind. It's simple and allows the external tool to be written in
   any conceivable language, but it forces the external tool to manage
   state across multiple runs, and may come with some
   overhead. Ultimately it'd result in people just using it as an event
   stream that pipes into a long running program anyway.
 - Embedded plugins: this is probably the most ambitious in that we
   build plugins that are executed directly in the context of
   c-lightning, either by compiling them in, dynamically loading them,
   or embedding an interpreter. The first two options would again limit
   us to languages that work well in C, while the last would give us
   free choice of languages, and why not, we could even implement a
   small wasm interpreter, e.g., wac [3]. It'd be the least overhead
   option, but also the hardest to pull off.

We already have number of tools that use the Driver or the Sidecar
model, and we have at least one PR that would give us a Callback
extension point (PR #1715). As for the last option, it'd still limit us
to Rust, C and C++, until the wasm community has outlined how to
interface with a garbage collector. The embedded plugins would also be
completely isolated, which is good for autopilots and similar
self-contained things, but not so good for wallets and things that
actually need to talk to the outside world.

Out of these options we probably should prioritize one, and try to make
that as appealing as possible. As simple as it sounds, just providing a
streaming API to get notified of internal events would get us quite far
already, as long as we don't need to change the automated behavior like
#1715 tries to do.

Cheers,
Christian

[1] https://github.com/ElementsProject/lightning/tree/master/tests
[2] https://github.com/cdecker/lightning-integration
[3] https://github.com/kanaka/wac