From corne at bitonic.nl Mon Nov 18 20:43:26 2019 From: corne at bitonic.nl (=?UTF-8?Q?Corn=c3=a9_Plooy?=) Date: Mon, 18 Nov 2019 10:43:26 +0100 Subject: [c-lightning] Plug-in dependency management Message-ID: Hi, A recent PR[1] made me wonder: is lightningd asymptotically evolving towards an architecture where all functionality is in a set of mutually dependent plug-ins, with lightningd as a sort of an RPC microkernel? I'm not really against this (it provides nice features like this PR), but will this be a workable end result? In the end, we're probably going to need some kind of dependency checking between plug-ins. I guess that, with Rusty in the team, we could take Linux kernel modules as an example - if I'm not mistaken, Linux does some dependency checking when I do insmod, and it even resolves them with modprobe. I think it's all about providing and requiring *interfaces* - multiple implementations could provide the same interface. Maybe we're not at the point yet where the problem for this solution really exists, but it is something to keep in mind for the future. CJP [1] https://github.com/ElementsProject/lightning/pull/3260 From ZmnSCPxj at protonmail.com Mon Nov 18 21:19:58 2019 From: ZmnSCPxj at protonmail.com (ZmnSCPxj) Date: Mon, 18 Nov 2019 10:19:58 +0000 Subject: [c-lightning] Plug-in dependency management In-Reply-To: References: Message-ID: Good morning Corne, > A recent PR[1] made me wonder: is lightningd asymptotically evolving > towards an architecture where all functionality is in a set of mutually > dependent plug-ins, with lightningd as a sort of an RPC microkernel? I'm > not really against this (it provides nice features like this PR), but > will this be a workable end result? This is my understanding as well, though I must admit I have not explicitly discussed this with my C-Lightning Core colleagues. At the most extreme, even things like `channeld` et al may very well be plugins communicating to each other via JSON-RPC commands. The move to having `bitcoin-cli` calls be mediated by a plugin is, to my mind, already a start. > In the end, we're probably going to need some kind of dependency > checking between plug-ins. I guess that, with Rusty in the team, we > could take Linux kernel modules as an example - if I'm not mistaken, > Linux does some dependency checking when I do insmod, and it even > resolves them with modprobe. I think it's all about providing and > requiring interfaces - multiple implementations could provide the same > interface. > > Maybe we're not at the point yet where the problem for this solution > really exists, but it is something to keep in mind for the future. Given that plugins can be started and queried of their desired interface prior to being actually connected to the "main" RPC microkernel/bus architecture, it would be possible to just start every plugin at startup, learn their provided interfaces (including conflicting interfaces), then figure out which ones the user wants to enable via some command-line option. We "only" need to add some kind of `"requires_command"` field to the plugin manifest. Then the user can just load specific plugins in the lightning configuration. If some plugin indicates it requires a command, then the loader searches for a plugin that provides that command and adds it to the list of plugins it needs to `init` later. (It could even be arranged to `init` in order of most-depended-upon to least-depended-upon). If multiple plugins provide the same command, perhaps a lightning configuration option could be used by the user to hint which one he or she prefers, and fail startup in case of such conflict. (This would allow for a general method to select a backend, for example, once we are able to move the bitcoin backend to plugins) Regards, ZmnSCPxj From rusty at rustcorp.com.au Thu Nov 21 16:52:35 2019 From: rusty at rustcorp.com.au (Rusty Russell) Date: Thu, 21 Nov 2019 16:22:35 +1030 Subject: [c-lightning] Plug-in dependency management In-Reply-To: References: Message-ID: <877e3tn5cs.fsf@rustcorp.com.au> ZmnSCPxj writes: > Good morning Corne, > >> A recent PR[1] made me wonder: is lightningd asymptotically evolving >> towards an architecture where all functionality is in a set of mutually >> dependent plug-ins, with lightningd as a sort of an RPC microkernel? I'm >> not really against this (it provides nice features like this PR), but >> will this be a workable end result? > > This is my understanding as well, though I must admit I have not explicitly discussed this with my C-Lightning Core colleagues. > At the most extreme, even things like `channeld` et al may very well be plugins communicating to each other via JSON-RPC commands. > The move to having `bitcoin-cli` calls be mediated by a plugin is, to my mind, already a start. > >> In the end, we're probably going to need some kind of dependency >> checking between plug-ins. I guess that, with Rusty in the team, we >> could take Linux kernel modules as an example - if I'm not mistaken, >> Linux does some dependency checking when I do insmod, and it even >> resolves them with modprobe. I think it's all about providing and >> requiring interfaces - multiple implementations could provide the same >> interface. >> >> Maybe we're not at the point yet where the problem for this solution >> really exists, but it is something to keep in mind for the future. > > Given that plugins can be started and queried of their desired interface prior to being actually connected to the "main" RPC microkernel/bus architecture, it would be possible to just start every plugin at startup, learn their provided interfaces (including conflicting interfaces), then figure out which ones the user wants to enable via some command-line option. > > We "only" need to add some kind of `"requires_command"` field to the plugin manifest. That's possible, but I think in general we'd start them all and they could activate themselves lazily? I agree with ZmnSCPxj's predictions, and in particular see a few near-term problems: 1. plugins can't name-clash, which makes it awkward to offer two plugins which want to do similar things. This can be fixed when we need with a .name convention. 2. only one plugin can register on each hook at the moment. 3. we often want alternatives at runtime; for example a backend that can query bitcoind via RPC or another which uses the p2p network, or neutrino or even a block explorer. If we use namespaces to fix the first one, we can use priorities to solve the second and third ones I think. We might want to eventually tweak priorities at runtime, but that's longer term. Note also that darosior's rpc_command hook has also added scarily powerful capabilities on what plugins can do; you can really mess with someone's head using this. Cheers, Rusty.