[c-lightning] Towards an autopilot

Sun Mar 25 04:27:34 AEDT 2018

Morning ZmnSCPxj!

And here I was thinking you're gonna propose to hardcode cyclic superhubs.
:)

You have really put some though into this and now I see it makes perfect
sense to support many strategies or roles; consumer, bridger, evil
centraliser etc... I'd like c-lightning to be the swiss knife
implementation of lightning, partly cause I run one instance on my phone.

However, this can also be achieved by simply extending the set of JSON-RPC
commands to include the ones we need for autopilot. One could run autopilot
as a separate process connecting to the same socket we use now or as a part
of the whatever app is driving c-lightning.

This would be the simplest approach but without an "official" autopilot
it's missing standardisation and the dogfooding aspect.

Standardisation we need for algorithms such as your cyclic hubs - such
strategies have network effects and the more nodes apply the same algorithm
the better it achieves its stated goal. So having an official or default
c-lightning autopilot could possibly be beneficial to the nascent network.

On the technical side from your description it wasn't clear to me how a
user could run a custom autopilotd, do we kill the official one and replace
with ours? Or do we change the algorithm in the wrangler? The wrangler
whole?

Having said all that, I believe the best way forward is for you to grace us
with a pull request where we could see the proposed code in action. Write a
small cyclic superhubs implementation and we go from there. :)

Cheers,
Igor

On 23 March 2018 at 02:00, <c-lightning-request at lists.ozlabs.org> wrote:

> Send c-lightning mailing list submissions to
>         c-lightning at lists.ozlabs.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://lists.ozlabs.org/listinfo/c-lightning
> or, via email, send a message with subject or body 'help' to
>         c-lightning-request at lists.ozlabs.org
>
> You can reach the person managing the list at
>         c-lightning-owner at lists.ozlabs.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of c-lightning digest..."
>
>
> Today's Topics:
>
>    1. Towards an autopilot (ZmnSCPxj at protonmail.com)
>    2. Re: Towards an autopilot (Ján Sáreník)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 22 Mar 2018 00:14:02 -0400
> From: ZmnSCPxj at protonmail.com
> To: C Lightning List <c-lightning at lists.ozlabs.org>
> Subject: [c-lightning] Towards an autopilot
> Message-ID:
>         <EtW4eQJ6o4no8cpfQV4xmit4vqq9cnOoVhOiveKaSEFk4A8pAQAT73ZR8vGZ
> TsV5XIZT9U-vskh5ZqvHyMux0ReIjoIhicsCtUhDx3lrmp0=@protonmail.com>
>
> Content-Type: text/plain; charset="utf-8"
>
> Good morning c-lightningers,
>
> Ideally, a nontechnical human user will simply cause funds to be sent to
> their c-lightning node, and then some time after be able to send funds on
> Lightning.
>
> A step towards that ideal, is an autopilot (similar to that of lnd) which
> manages onchain funds and opens channels to the network as it detects funds.
>
> So I propose an autopilotd, of the below architecture
>
> --
>
> At lightningd, we add support by a new autopilot_control.[ch] module.
> This provides a `struct autopilotd`.  Creating this object by
> `new_autopilotd` will launch the `lightning_autopilotd` subd.  Destroying
> this tal-allocated object will shut down forcibly the subd.
>
> Internally, the `new_autopilotd` will need to be provided a socket.  This
> socket is access to the JSON-RPC (i.e. it is backed by a `struct
> json_connection` that is the other end of the socket).
>
> In addition, the `new_autopilotd` will create a new socket, which will be
> the status socket.
>
> The `lightning_autopilotd` program will then receive two sockets on
> startup: the JSON-RPC control socket and the status socket.
>
> The autopilot primarily interacts with lightningd via the JSON-RPC
> socket.  The status socket simply contains autopilotd->lightningd status
> messages, which are put into a billboard (similar to the channel-level
> daemon billboards).  The lightningd has no need to send commands to the
> autopilot: enabling autopilot simply means starting `lightning_autopilotd`
> (i.e. creating a `struct autopilotd`), disabling autopilot simply means
> killing `lightning_autopilotd` and closing its attached sockets (i.e.
> destroying the `struct autopilotd`).  Configuration data for the autopilot
> can be provided in the `listconfigs` command, which the autopilotd can
> query via the JSON-RPC socket.
>
> Why does autopilot interact primarily via the JSON-RPC interface?  Others
> may have better ideas on how to manage channels, perhaps with better
> integration with higher-level user-facing applications.  Such applications
> are likely to use JSON-RPC.  By using JSON-RPC ourselves for our built-in
> autopilot, we can find out if there are missing features/commands/info in
> JSON-RPC that prevent a good autopilot from being implemented via JSON-RPC,
> i.e. we are forced to eat our own dog food.
>
> So, this is the architecture as seen from the lightningd side: we hand
> over a JSON-RPC command interface, and a status interface that we store in
> the logs.  We provide RPC commands to turn on/off the autopilot and change
> its configuration, and query its billboard.
>
> --
>
> At the autopilotd side, we split up the autopilot into two parts: a driver
> and an algorithm wrangler.
>
> The driver is obsessed with one thing: converting onchain funds into
> onLightning funds.  Periodically (every 3 seconds?) it performs
> `listfunds`.  If that shows any onchain funds, driver gets a candidate node
> from its queue of candidate nodes, tries to connect and fundchannel to it,
> until it has run out of onchain funds.  If that queue of candidate nodes is
> empty, it then requests for a new queue of candidate nodes from the
> algorithm wrangler.
>
> The algorithm wrangler calls out to a list of algorithms.  Those
> algorithms are given an (initially empty) sequence of candidate nodes, and
> can manipulate that sequence of nodes (add entries primarily, remove
> entries, rearrange..).  The algorithms the wrangler uses can be configured
> at lightningd (the wrangler uses `listconfigs` to get the configuration),
> as a comma-separated list of algorithm names.  When the algorithm wrangler
> finishes calling all the listed algorithms, it returns the resulting
> sequence of candidate nodes to the driver, which then proceeds to its work
> of putting onchain funds onLightning.
>
> Example algorithms could be:
>
> * `random` - get all known nodes from `listchannels` command, randomly
> select one.
> * `customer` - get payments from `listpayments`, check which payee node we
> pay to often, and select the most-often-paid node that we do not have a
> direct channel to already.
> * `superhub` - see "Towards a gridlike Lightning Network" topic on
> lightning-dev: https://lists.linuxfoundation.org/pipermail/lightning-dev/
> 2018-March/001108.html
> * `bridger` - get all known nodes from `listchannels` command, look for
> nodes with least number of public channels (if that node becomes a popular
> payee we have a chance of being on one of the few viable routes to it).
> Select one from the lowest bin as candidate.
> * `EVILCENTRALIZERDONOTUSE` - get all known nodes from `listchannels`,
> look for nodes with most number of public channels (more likely that node
> will be able to route our own payments to our destination with all the
> channels it has).  Select one from the highest bin as candidate.
> * `blockstream` - add store.blockstream.com as candidate hahahahahaha.
>
> Now a good number of the above algorithms call `listchannels` and other
> queries from the lightningd.  It would be wasteful to re-query lightningd
> each time, if the user configures multiple algorithms that each call
> `listchannels`.  One thing we can do is to cache such queries.  Start the
> cache empty.  When an algorithm asks for a query, check if it is already
> done and try to get it from the cache, else go actually perform the query
> and store it.  Then if the user lists multiple algorithms that call the
> same query on lightningd, only the first algorithm needs to wait for the
> reply, and succeeding algorithms simply reuse the same data.  The cache is
> reset just before the algorithm wrangler completes and returns the list of
> nodes.
>
> For example, someone who wants to become a hub node may want the
> `superhub` and `bridger` algorithms to run, and as a fallback also the
> `random` algorithm to sometimes connect to random nodes in case they become
> useful someday.  Someone who just wants to pay for groceries and coffee
> might want `customer` and `EVILCENTRALIZERDONOTUSE`, and once Blockstream
> starts selling coffee maybe `blockstream`.
>
> The resulting list of nodes after all algorithms have run may have
> duplicates, i.e. if multiple algorithms think those are good candidates.
> The wrangler can also prioritize nodes that have duplicates over nodes that
> do not have duplicates.
>
> Separating bits of logic into the "algorithm" system suggests we can
> provide a plug-in system eventually, by which external projects can provide
> new algorithms that change or tweak how autopilot works.
>
> --
>
> In addition, we want to consider the question of closing channels.
>
> Obviously we will want good statistics, e.g. how many payments we have
> succeeded on that channel, how much those payments are, etc.
>
> A similar system for algorithms could be done too, although I am unsure
> how the driver decides when to close channels.
>
> Perhaps the wrangler can unify things somewhat: each algorithm has
> multiple possible entry points, one for opening, one for closing, and those
> are optional to the algorithm (an algorithm can have entrypoint for opening
> channel, entrypoint for closing channel, or both).  Then the driver calls
> the wrangler on whether it wants to open channels or close them.  This will
> let us configure autopilot with a single list.
>
> Regards,
> ZmnSCPxj
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://lists.ozlabs.org/pipermail/c-lightning/
> attachments/20180322/23ba84c7/attachment-0001.html>
>
> ------------------------------
>
> Message: 2
> Date: Thu, 22 Mar 2018 08:17:31 +0000
> From: Ján Sáreník <jajomojo at gmail.com>
> To: c-lightning at lists.ozlabs.org
> Subject: Re: [c-lightning] Towards an autopilot
> Message-ID:
>         <CAE5c8tHOb7dJ5zM3KOb9NUqn44WhKkjvuGnEJdU2D1MgOSu9EA at mail.
> gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Good morning ZmnSCPxj!
>
> On Thu, Mar 22, 2018, 05:14 <ZmnSCPxj at protonmail.com> wrote:
>
> > --
> >
> > In addition, we want to consider the question of closing channels.
> >
> > Obviously we will want good statistics, e.g. how many payments we have
> > succeeded on that channel, how much those payments are, etc.
> >
> > A similar system for algorithms could be done too, although I am unsure
> > how the driver decides when to close channels.
> >
>
> What about a simple bittorrent-like ratio comparing the fee that was paid
> for opening the channel and the amount which flew through the channel?
>
> I like the idea of autopilot. Thank you for putting it into words!
>
> Best regards, Ján
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://lists.ozlabs.org/pipermail/c-lightning/
> attachments/20180322/9efa29e6/attachment-0001.html>
>
> ------------------------------
>
> Subject: Digest Footer
>
> c-lightning mailing list
> c-lightning at lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/c-lightning
>
>
> ------------------------------
>
> End of c-lightning Digest, Vol 2, Issue 3
> *****************************************
>

-- 
*Igor Cota*
Codex Apertus Ltd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ozlabs.org/pipermail/c-lightning/attachments/20180324/b70d5e5d/attachment-0001.html>