[c-lightning] Plug-in dependency management

[Rusty Russell]

ZmnSCPxj <ZmnSCPxj at protonmail.com> writes:
> Good morning Corne,
>
>> A recent PR[1] made me wonder: is lightningd asymptotically evolving
>> towards an architecture where all functionality is in a set of mutually
>> dependent plug-ins, with lightningd as a sort of an RPC microkernel? I'm
>> not really against this (it provides nice features like this PR), but
>> will this be a workable end result?
>
> This is my understanding as well, though I must admit I have not explicitly discussed this with my C-Lightning Core colleagues.
> At the most extreme, even things like `channeld` et al may very well be plugins communicating to each other via JSON-RPC commands.
> The move to having `bitcoin-cli` calls be mediated by a plugin is, to my mind, already a start.
>
>> In the end, we're probably going to need some kind of dependency
>> checking between plug-ins. I guess that, with Rusty in the team, we
>> could take Linux kernel modules as an example - if I'm not mistaken,
>> Linux does some dependency checking when I do insmod, and it even
>> resolves them with modprobe. I think it's all about providing and
>> requiring interfaces - multiple implementations could provide the same
>> interface.
>>
>> Maybe we're not at the point yet where the problem for this solution
>> really exists, but it is something to keep in mind for the future.
>
> Given that plugins can be started and queried of their desired interface prior to being actually connected to the "main" RPC microkernel/bus architecture, it would be possible to just start every plugin at startup, learn their provided interfaces (including conflicting interfaces), then figure out which ones the user wants to enable via some command-line option.
>
> We "only" need to add some kind of `"requires_command"` field to the plugin manifest.

That's possible, but I think in general we'd start them all and they
could activate themselves lazily?

I agree with ZmnSCPxj's predictions, and in particular see a few
near-term problems:

1. plugins can't name-clash, which makes it awkward to offer two plugins
   which want to do similar things.  This can be fixed when we need with a
   <plugin>.name convention.

2. only one plugin can register on each hook at the moment.

3. we often want alternatives at runtime; for example a backend that can
   query bitcoind via RPC or another which uses the p2p network, or
   neutrino or even a block explorer.  

If we use namespaces to fix the first one, we can use priorities to
solve the second and third ones I think.  We might want to eventually
tweak priorities at runtime, but that's longer term.

Note also that darosior's rpc_command hook has also added scarily
powerful capabilities on what plugins can do; you can really mess with
someone's head using this.

Cheers,
Rusty.