[c-lightning] Plug-in dependency management

[ZmnSCPxj]

Good morning Corne,

> A recent PR[1] made me wonder: is lightningd asymptotically evolving
> towards an architecture where all functionality is in a set of mutually
> dependent plug-ins, with lightningd as a sort of an RPC microkernel? I'm
> not really against this (it provides nice features like this PR), but
> will this be a workable end result?

This is my understanding as well, though I must admit I have not explicitly discussed this with my C-Lightning Core colleagues.
At the most extreme, even things like `channeld` et al may very well be plugins communicating to each other via JSON-RPC commands.
The move to having `bitcoin-cli` calls be mediated by a plugin is, to my mind, already a start.

> In the end, we're probably going to need some kind of dependency
> checking between plug-ins. I guess that, with Rusty in the team, we
> could take Linux kernel modules as an example - if I'm not mistaken,
> Linux does some dependency checking when I do insmod, and it even
> resolves them with modprobe. I think it's all about providing and
> requiring interfaces - multiple implementations could provide the same
> interface.
>
> Maybe we're not at the point yet where the problem for this solution
> really exists, but it is something to keep in mind for the future.

Given that plugins can be started and queried of their desired interface prior to being actually connected to the "main" RPC microkernel/bus architecture, it would be possible to just start every plugin at startup, learn their provided interfaces (including conflicting interfaces), then figure out which ones the user wants to enable via some command-line option.

We "only" need to add some kind of `"requires_command"` field to the plugin manifest.

Then the user can just load specific plugins in the lightning configuration.
If some plugin indicates it requires a command, then the loader searches for a plugin that provides that command and adds it to the list of plugins it needs to `init` later.
(It could even be arranged to `init` in order of most-depended-upon to least-depended-upon).
If multiple plugins provide the same command, perhaps a lightning configuration option could be used by the user to hint which one he or she prefers, and fail startup in case of such conflict.
(This would allow for a general method to select a backend, for example, once we are able to move the bitcoin backend to plugins)

Regards,
ZmnSCPxj