[c-lightning] Towards an autopilot

[ZmnSCPxj]

Good morning Rusty,


> > Too large - can already judge from routemap from `listchannels`. We already record channel capacity, we just need to report this in `listchannels` too. We can judge a node as too large by summing up the channel capacities of all its channels listed in `listchannels` command.
> 
> Well, it can simply use a heuristic that biasses away from supernodes.
> 
> But that needs to be spelled out. I believe biasses are better than
> 
> absolute rules, for network health.

We seem to use different terminology.  To me, biasing is itself an absolute rule.  For instance I might randomly disqualify nodes from a list of candidates, with a weighted coin that is biased towards disqualifying more often a node that has larger total channel capacity.

I would call a strict filter a rule that says "remove all nodes with total channel capacity >= N".

I agree that biasing is better indeed than a strict filter.  One could also use some kind of desirability score that is penalized for nodes with larger total channel capacity, and I would also consider this technique to be a form of biasing.

> 
> > Too new - we could store on DB the earliest time we saw a `node_announcement`. However when a `node_announcment` is pruned we should take care to also prune it in the database. We could add a new `listnodes` command that reports nodes, address hints for that node if any, and the earliest we heard about the node.
> 
> Not useful for new nodes. But I suggest we use the channel depth as a
> 
> proxy for age; bias against nodes which have no channels older than 2
> 
> weeks, say. The bias away from giant nodes should be stronger than
> 
> this, however.
> 
> > Too unreliable - when a payment fails, we know the erring node and we know the node after the erring node. The node after the erring node is the one that is unreliable (the erring node is the one that reported the error, so we know it is reliably alive and responsive). Keep a counter for each node that is just the number of times it was pointed at by the erring node as failing. We get some noise due to channel balance issues, but this is better than nothing. We could also increase the unreliability counter (or have a separate counter) if we connect to the node and fail.
> 
> We don't make enough payments to be informed, (especially at startup).
> 
> We can test for reachability, of course. When we have a channel we can
> 
> make a test payment through the node, too.

Yes, it would be possible to make test payments to probe the network without creating new connections.  We could generate a random hash and attempt to pay a tiny amount (say twice our own dust setting) to a random node who is unlikely to know the preimage.

> > > This bias against new nodes
> > > 
> > >     should weaken for successive channels,
> > >     
> > 
> > Under my proposed architecture, you can simply apply another algorithm
> > 
> > to be executed after the other algorithms. That algorithm would check
> > 
> > `listpeers` to know how many channels it has. If it has none, it
> > 
> > filters out too new/too large/too unreliable from the array of
> > 
> > proposed fundees. If it has a small number (one or two, or some
> > 
> > tweakable setting) it can move them to the end of the array (so that
> > 
> > they get funded last if ever), or flip a weighted coin (weighed for
> > 
> > 100% removed if channels is 0, reducing greatly as the number of
> > 
> > channels increases) to remove them from the array of proposed fundees.
> 
> Pluggable algorithms are great for researchers, but absolutely wrong for
> 
> normal users. IOW, every option you add halves your userbase, and
> 
> they'll leave it on the default anyway!
> 
> And no user will ever be as informed about the nuances of their choices
> 
> as you are. That means it's our responsibility to Make It Just Work,
> 
> and every option we add is an abrogation of that responsibility.
> 

It is also our responsibility to find out what would Make It Just Work too, which requires us to do research for ourselves.

Further some users may have specialized use-cases or needs (for instance, an employer whose employees accept Bitcoin over LN might want to keep channels to those employees, and say a few days before payday if the capacity of those channels is low on the employeer side could close those channels in preparation for opening them again with new funds just before payday)

> > >     The payment should use a heuristic like:
> > >     
> > >     a. Can I send like normal?
> > >     
> > >     b. Can I rebalance channels to send?
> > >     
> > >     c. Do I need to set up a new channel?
> > >     
> > 
> > Seems more appropriate for `lightningd/payalgo`.
> 
> Perhaps, but the other option is to have the pilot do it, and only
> 
> provide low-level controls?

One idea I have floating in my head is a "c-lightning bus" that wraps the RPC: it connects to the "real" lightningd RPC and presents its own compatible RPC.  It extends its offered RPC with the ability to monitor specific commands: e.g. I could register interest in issued `pay` commands, and then poll the bus for `pay` commands issued by the UI, in order to better inform myself how failures occur and so on.

Regards,
ZmnSCPxj