[c-lightning] Plugins/passthrough for next version?

[Rusty Russell]

Christian Decker <decker.christian at gmail.com> writes:
> I think we have quite a few different options to foster extensibility on
> top of a focused and lean lightning implementation, each with its own
> pros and cons:
>
>  - Driver: a program that controls the lightning daemon, tails its
>    output, or has another way to stream events from the daemon, and
>    interacts with the JSON-RPC we already have. That's basically how we
>    drive the tests, both for c-lightning [1] as well as the
>    integration tests across all implementations [2]. It's a bit cludgy
>    since we tail the logs which were not intended for machine
>    consumption, but it works right now and is how I've been building a
>    number of applications. An event stream, e.g., streaming JSON events,
>    would make this a lot easier.

Yes, we probably need to add some pub-sub API to the JSON-RPC interface
eventually.  It's fairly easy to do; the hard part is selecting and
defining the events.

>  - Sidecar: this is just a small variation of the above, in that it
>    doesn't control the daemon, and has to rely on polling it. It's
>    higher overhead, slower and would also benefit from an event stream.
>  - Callbacks: c-lightning hooks into certain events and calls out to a
>    registered external program if something of interest has
>    happened. This is basically the model `-blocknotify` option of
>    bitcoind. It's simple and allows the external tool to be written in
>    any conceivable language, but it forces the external tool to manage
>    state across multiple runs, and may come with some
>    overhead. Ultimately it'd result in people just using it as an event
>    stream that pipes into a long running program anyway.

This is what you do when you don't want to create an API :)  But we
already have an API, so I dislike this option.  As you say, you end up
building a layer on top of it to make it usable...

My rough thinking was that we would activate extensions with the
callback model on startup (presumably handing it any options which
matched its prefix), and it would tell us what commands it provided,
then we'd relay those JSON RPC commands to it (and return the replies).

>  - Embedded plugins: this is probably the most ambitious in that we
>    build plugins that are executed directly in the context of
>    c-lightning, either by compiling them in, dynamically loading them,
>    or embedding an interpreter. The first two options would again limit
>    us to languages that work well in C, while the last would give us
>    free choice of languages, and why not, we could even implement a
>    small wasm interpreter, e.g., wac [3]. It'd be the least overhead
>    option, but also the hardest to pull off.

It's dangerous in C, and makes bug reporting complex; to diag your crash
I might need to reproduce your plugins, etc.

Plus: since we use multi-process for isolation, not using that for
plugins seems weird.

Cheers,
Rusty.