<div dir="ltr"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Morning <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">ZmnSCPxj!</span></span><div><br></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"></span>And here I was thinking you're gonna propose to hardcode cyclic superhubs. :)</div><div><br></div><div>You have really put some though into this and now I see it makes perfect sense to support many strategies or roles; consumer, bridger, evil centraliser etc... I'd like c-lightning to be the swiss knife implementation of lightning, partly cause I run one instance on my phone.</div><div><br></div><div>However, this can also be achieved by simply extending the set of JSON-RPC commands to include the ones we need for autopilot. One could run autopilot as a separate process connecting to the same socket we use now or as a part of the whatever app is driving c-lightning.<br></div><div><br></div><div>This would be the simplest approach but without an "official" autopilot it's missing standardisation and the dogfooding aspect. </div><div><br></div><div>Standardisation we need for algorithms such as your cyclic hubs - such strategies have network effects and the more nodes apply the same algorithm the better it achieves its stated goal. So having an official or default c-lightning autopilot could possibly be beneficial to the nascent network.</div><div><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">On the technical side from your description it wasn't clear to me how a user could run a custom autopilotd, do we kill the official one and replace with ours? Or do we change the algorithm in the wrangler? The wrangler whole?</div><div><br></div><div>Having said all that, I believe the best way forward is for you to grace us with a pull request where we could see the proposed code in action. Write a small cyclic superhubs implementation and we go from there. :)</div><div><br></div><div>Cheers,<br>Igor</div><div><br></div><div><div class="gmail_extra"><div class="gmail_quote">On 23 March 2018 at 02:00, <span dir="ltr"><<a href="mailto:c-lightning-request@lists.ozlabs.org" target="_blank">c-lightning-request@lists.ozlabs.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Send c-lightning mailing list submissions to<br>
<a href="mailto:c-lightning@lists.ozlabs.org">c-lightning@lists.ozlabs.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://lists.ozlabs.org/listinfo/c-lightning" rel="noreferrer" target="_blank">https://lists.ozlabs.org/<wbr>listinfo/c-lightning</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:c-lightning-request@lists.ozlabs.org">c-lightning-request@lists.<wbr>ozlabs.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:c-lightning-owner@lists.ozlabs.org">c-lightning-owner@lists.<wbr>ozlabs.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of c-lightning digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Towards an autopilot (<a href="mailto:ZmnSCPxj@protonmail.com">ZmnSCPxj@protonmail.com</a>)<br>
2. Re: Towards an autopilot (Ján Sáreník)<br>
<br>
<br>
------------------------------<wbr>------------------------------<wbr>----------<br>
<br>
Message: 1<br>
Date: Thu, 22 Mar 2018 00:14:02 -0400<br>
From: <a href="mailto:ZmnSCPxj@protonmail.com">ZmnSCPxj@protonmail.com</a><br>
To: C Lightning List <<a href="mailto:c-lightning@lists.ozlabs.org">c-lightning@lists.ozlabs.org</a>><br>
Subject: [c-lightning] Towards an autopilot<br>
Message-ID:<br>
<<wbr>EtW4eQJ6o4no8cpfQV4xmit4vqq9cn<wbr>OoVhOiveKaSEFk4A8pAQAT73ZR8vGZ<wbr>TsV5XIZT9U-<wbr>vskh5ZqvHyMux0ReIjoIhicsCtUhDx<wbr>3lrmp0=@<a href="http://protonmail.com" rel="noreferrer" target="_blank">protonmail.com</a>><br>
<br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Good morning c-lightningers,<br>
<br>
Ideally, a nontechnical human user will simply cause funds to be sent to their c-lightning node, and then some time after be able to send funds on Lightning.<br>
<br>
A step towards that ideal, is an autopilot (similar to that of lnd) which manages onchain funds and opens channels to the network as it detects funds.<br>
<br>
So I propose an autopilotd, of the below architecture<br>
<br>
--<br>
<br>
At lightningd, we add support by a new autopilot_control.[ch] module. This provides a `struct autopilotd`. Creating this object by `new_autopilotd` will launch the `lightning_autopilotd` subd. Destroying this tal-allocated object will shut down forcibly the subd.<br>
<br>
Internally, the `new_autopilotd` will need to be provided a socket. This socket is access to the JSON-RPC (i.e. it is backed by a `struct json_connection` that is the other end of the socket).<br>
<br>
In addition, the `new_autopilotd` will create a new socket, which will be the status socket.<br>
<br>
The `lightning_autopilotd` program will then receive two sockets on startup: the JSON-RPC control socket and the status socket.<br>
<br>
The autopilot primarily interacts with lightningd via the JSON-RPC socket. The status socket simply contains autopilotd->lightningd status messages, which are put into a billboard (similar to the channel-level daemon billboards). The lightningd has no need to send commands to the autopilot: enabling autopilot simply means starting `lightning_autopilotd` (i.e. creating a `struct autopilotd`), disabling autopilot simply means killing `lightning_autopilotd` and closing its attached sockets (i.e. destroying the `struct autopilotd`). Configuration data for the autopilot can be provided in the `listconfigs` command, which the autopilotd can query via the JSON-RPC socket.<br>
<br>
Why does autopilot interact primarily via the JSON-RPC interface? Others may have better ideas on how to manage channels, perhaps with better integration with higher-level user-facing applications. Such applications are likely to use JSON-RPC. By using JSON-RPC ourselves for our built-in autopilot, we can find out if there are missing features/commands/info in JSON-RPC that prevent a good autopilot from being implemented via JSON-RPC, i.e. we are forced to eat our own dog food.<br>
<br>
So, this is the architecture as seen from the lightningd side: we hand over a JSON-RPC command interface, and a status interface that we store in the logs. We provide RPC commands to turn on/off the autopilot and change its configuration, and query its billboard.<br>
<br>
--<br>
<br>
At the autopilotd side, we split up the autopilot into two parts: a driver and an algorithm wrangler.<br>
<br>
The driver is obsessed with one thing: converting onchain funds into onLightning funds. Periodically (every 3 seconds?) it performs `listfunds`. If that shows any onchain funds, driver gets a candidate node from its queue of candidate nodes, tries to connect and fundchannel to it, until it has run out of onchain funds. If that queue of candidate nodes is empty, it then requests for a new queue of candidate nodes from the algorithm wrangler.<br>
<br>
The algorithm wrangler calls out to a list of algorithms. Those algorithms are given an (initially empty) sequence of candidate nodes, and can manipulate that sequence of nodes (add entries primarily, remove entries, rearrange..). The algorithms the wrangler uses can be configured at lightningd (the wrangler uses `listconfigs` to get the configuration), as a comma-separated list of algorithm names. When the algorithm wrangler finishes calling all the listed algorithms, it returns the resulting sequence of candidate nodes to the driver, which then proceeds to its work of putting onchain funds onLightning.<br>
<br>
Example algorithms could be:<br>
<br>
* `random` - get all known nodes from `listchannels` command, randomly select one.<br>
* `customer` - get payments from `listpayments`, check which payee node we pay to often, and select the most-often-paid node that we do not have a direct channel to already.<br>
* `superhub` - see "Towards a gridlike Lightning Network" topic on lightning-dev: <a href="https://lists.linuxfoundation.org/pipermail/lightning-dev/2018-March/001108.html" rel="noreferrer" target="_blank">https://lists.linuxfoundation.<wbr>org/pipermail/lightning-dev/<wbr>2018-March/001108.html</a><br>
* `bridger` - get all known nodes from `listchannels` command, look for nodes with least number of public channels (if that node becomes a popular payee we have a chance of being on one of the few viable routes to it). Select one from the lowest bin as candidate.<br>
* `EVILCENTRALIZERDONOTUSE` - get all known nodes from `listchannels`, look for nodes with most number of public channels (more likely that node will be able to route our own payments to our destination with all the channels it has). Select one from the highest bin as candidate.<br>
* `blockstream` - add <a href="http://store.blockstream.com" rel="noreferrer" target="_blank">store.blockstream.com</a> as candidate hahahahahaha.<br>
<br>
Now a good number of the above algorithms call `listchannels` and other queries from the lightningd. It would be wasteful to re-query lightningd each time, if the user configures multiple algorithms that each call `listchannels`. One thing we can do is to cache such queries. Start the cache empty. When an algorithm asks for a query, check if it is already done and try to get it from the cache, else go actually perform the query and store it. Then if the user lists multiple algorithms that call the same query on lightningd, only the first algorithm needs to wait for the reply, and succeeding algorithms simply reuse the same data. The cache is reset just before the algorithm wrangler completes and returns the list of nodes.<br>
<br>
For example, someone who wants to become a hub node may want the `superhub` and `bridger` algorithms to run, and as a fallback also the `random` algorithm to sometimes connect to random nodes in case they become useful someday. Someone who just wants to pay for groceries and coffee might want `customer` and `EVILCENTRALIZERDONOTUSE`, and once Blockstream starts selling coffee maybe `blockstream`.<br>
<br>
The resulting list of nodes after all algorithms have run may have duplicates, i.e. if multiple algorithms think those are good candidates. The wrangler can also prioritize nodes that have duplicates over nodes that do not have duplicates.<br>
<br>
Separating bits of logic into the "algorithm" system suggests we can provide a plug-in system eventually, by which external projects can provide new algorithms that change or tweak how autopilot works.<br>
<br>
--<br>
<br>
In addition, we want to consider the question of closing channels.<br>
<br>
Obviously we will want good statistics, e.g. how many payments we have succeeded on that channel, how much those payments are, etc.<br>
<br>
A similar system for algorithms could be done too, although I am unsure how the driver decides when to close channels.<br>
<br>
Perhaps the wrangler can unify things somewhat: each algorithm has multiple possible entry points, one for opening, one for closing, and those are optional to the algorithm (an algorithm can have entrypoint for opening channel, entrypoint for closing channel, or both). Then the driver calls the wrangler on whether it wants to open channels or close them. This will let us configure autopilot with a single list.<br>
<br>
Regards,<br>
ZmnSCPxj<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.ozlabs.org/pipermail/c-lightning/attachments/20180322/23ba84c7/attachment-0001.html" rel="noreferrer" target="_blank">http://lists.ozlabs.org/<wbr>pipermail/c-lightning/<wbr>attachments/20180322/23ba84c7/<wbr>attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Thu, 22 Mar 2018 08:17:31 +0000<br>
From: Ján Sáreník <<a href="mailto:jajomojo@gmail.com">jajomojo@gmail.com</a>><br>
To: <a href="mailto:c-lightning@lists.ozlabs.org">c-lightning@lists.ozlabs.org</a><br>
Subject: Re: [c-lightning] Towards an autopilot<br>
Message-ID:<br>
<<a href="mailto:CAE5c8tHOb7dJ5zM3KOb9NUqn44WhKkjvuGnEJdU2D1MgOSu9EA@mail.gmail.com">CAE5c8tHOb7dJ5zM3KOb9NUqn44Wh<wbr>KkjvuGnEJdU2D1MgOSu9EA@mail.<wbr>gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Good morning ZmnSCPxj!<br>
<br>
On Thu, Mar 22, 2018, 05:14 <<a href="mailto:ZmnSCPxj@protonmail.com">ZmnSCPxj@protonmail.com</a>> wrote:<br>
<br>
> --<br>
><br>
> In addition, we want to consider the question of closing channels.<br>
><br>
> Obviously we will want good statistics, e.g. how many payments we have<br>
> succeeded on that channel, how much those payments are, etc.<br>
><br>
> A similar system for algorithms could be done too, although I am unsure<br>
> how the driver decides when to close channels.<br>
><br>
<br>
What about a simple bittorrent-like ratio comparing the fee that was paid<br>
for opening the channel and the amount which flew through the channel?<br>
<br>
I like the idea of autopilot. Thank you for putting it into words!<br>
<br>
Best regards, Ján<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.ozlabs.org/pipermail/c-lightning/attachments/20180322/9efa29e6/attachment-0001.html" rel="noreferrer" target="_blank">http://lists.ozlabs.org/<wbr>pipermail/c-lightning/<wbr>attachments/20180322/9efa29e6/<wbr>attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
c-lightning mailing list<br>
<a href="mailto:c-lightning@lists.ozlabs.org">c-lightning@lists.ozlabs.org</a><br>
<a href="https://lists.ozlabs.org/listinfo/c-lightning" rel="noreferrer" target="_blank">https://lists.ozlabs.org/<wbr>listinfo/c-lightning</a><br>
<br>
<br>
------------------------------<br>
<br>
End of c-lightning Digest, Vol 2, Issue 3<br>
******************************<wbr>***********<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><b>Igor Cota</b><div>Codex Apertus Ltd<br></div></div></div></div></div>
</div></div></div>