[c-lightning] Towards an autopilot

[ZmnSCPxj]

Good morning Igor and other c-lightningers,

Please see: https://github.com/ZmnSCPxj/lightning/tree/pilotd

This is very unfinished yet: lightningd does not even start lightning_pilotd at all other than to check that it exists and has the same version as itself, and lightning_pilotd does nothing other than spam the logs with its messages every 10 seconds.  Hopefully I can fill this in more this week.

> However, this can also be achieved by simply extending the set of JSON-RPC commands to include the ones we need for autopilot. One could run autopilot as a separate process connecting to the same socket we use now or as a part of the whatever app is driving c-lightning.
>
> This would be the simplest approach but without an "official" autopilot it's missing standardisation and the dogfooding aspect.

Precisely.  Note that we do not "need" anything for autopilot other than channel usage statistics (which were merged just recently although I may add some more statistics) in order to judge when to close.

But a framework for querying lightningd state, and for plugging in own algorithms, would be a reasonable investment towards encouraging other programmers to add more nuanced, smarter, more necessary algorithms as needed for their own purposes.

> Standardisation we need for algorithms such as your cyclic hubs - such strategies have network effects and the more nodes apply the same algorithm the better it achieves its stated goal. So having an official or default c-lightning autopilot could possibly be beneficial to the nascent network.
>
> On the technical side from your description it wasn't clear to me how a user could run a custom autopilotd, do we kill the official one and replace with ours? Or do we change the algorithm in the wrangler? The wrangler whole?

My intent, is, the lightningd will have a --pilot-algo setting (which can be set also by `lightning-cli pilot algo "yoursetting"` command).  And pilotd will query this setting and give to wrangler.  If wrangler notice, this setting was changed, it will clear its algorithm objects and recreate them according to the algo setting.

The algo string is something of the form "algoname(arg, arg); algoname(arg); algoname()".

Algorithm plugins would register their algorithm names and their constructors.  When the wrangler processes the algo string, it finds the algorithm name, and calls the constructor with the arguments given in the algo string.  The constructor should return an algorithm object with two entry points, the "funding channel needed" entrypoint and the "periodic polling" entrypoint, either of which may be NULL (a pointless algorithm, would have all entrypoints NULL).  Below is a sketch:

/* lpa = lightningd pilot algorithm */
struct lpa_algo {
    void *algo_contents;
    void (*funding)(struct lpa_run*, void *algo_contents);
    void (*periodic)(struct lpa_run*, void *algo_contents);
    /* Needed: destroys this object */
    void (*destroy)(void *algo_contents);
};

The wrangler exposes the funding and periodic entrypoints.  If the driver calls the wrangler funding function, the wrangler goes through each algorithm in order and calls its funding entrypoint if present.  Similar for periodic-polling.  Each wrangler can manipulate an array of proposed peers to open channels to, and array of proposed channels to close (for ease-of-programming we add a small utility function that just appends a peer/channel to the open/close array): each algorithm can append their proposals to the array, filter the array, or rearrange the array as it sees fit.

The funding entrypoint is only called if the driver finds we have onchain funds and the array of proposed peers to open channels to is empty.  The periodic-polling entrypoint is periodically called every few cycles (the driver checks for funds on each cycle, and calls periodic-polling every few such cycles).

Algorithm plugins, once I get around to implementing them, could be stored in a `pilotplugins` directory inside lightningdir; on startup of `lightning_pilotd`, every .so in it is dlopen()ed and we look for a `lightningd_pilot_algo_register` function that performs the registration of algorithm name and constructor.  For builtin algorithms we just call a similar such function for each algorithm we implement as built-in to the pilotd.

Regards,
ZmnSCPxj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ozlabs.org/pipermail/c-lightning/attachments/20180326/f5014356/attachment.html>