[c-lightning] Shared-memory Routemap
ZmnSCPxj at protonmail.com
Tue Sep 24 19:02:16 AEST 2019
Sent with ProtonMail Secure Email.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Tuesday, September 24, 2019 12:12 PM, Rusty Russell <rusty at rustcorp.com.au> wrote:
> ZmnSCPxj ZmnSCPxj at protonmail.com writes:
> > Hi all,
> > Rusty expressed the opinion, to move routing techniques to plugins.
> > And he made also https://github.com/ElementsProject/lightning/pull/3075 .
> > Simply put, the consideration of this pull request is to reduce representation of nodes and channels, so contains fields only for purpose of routemaps.
> > And each plugin separately reads the `gossip_store` file to create an in-memory routemap.
> > My concern, is that this routemap is (to my mind, needlessly) duplicated across each plugin instance.
> > Consider that we may eventually have a `getroutequick` and `permuteroute` and `recalcroute` implementat1ions (as well as legacy `getroute` implementation), possibly in separate plugins for easier code management.
> Well, each node needs to generate its own index (mapping scids to
> offset, and nodeids to offsets). But that makes sense, since they may
> have different needs, and the index is only 10MB currently.
> I checked with the million-channels-project: it's still only 72MB for
> the indices. And it takes about 2.3 seconds to load (vs 200msec for the
> current gossip store) on my laptop.
72Mb seems large to replicate a few times to me, at least on single-board-computer-level devices.
Though of course it will take a while to reach a million channels anyway, and might not be achieved on the public network anyway.
> > Synchronization
> > ================
> > As the memory is shared, we should take care the synchronization across multiple threads-of-control.
> > Thus, we should care point, to use a proper synchronization structure.
> You've reinvented one of my favorite projects: libantithread :)
I checked, and it seems some difference.
libantithread needs to control the startup of each plugin that accesses the shared memory.
And I think a child antithread can only read from the single pool it was created in?
Are shared objects supposed to be passed via `at_tell` and `at_tell_parent`?
Can we use plain `void*` for pointers to shared objects as well?
It seems not easy to adapt to using libantithread given our current plugin system.
> But it's overkill for the moment, and complex and fragile, as you
> describe. Using a database for the indexes might be a halfway model
> (either sqlite3 or tdb), but it's simply not worth it.
> I've done some bugfixes, and here are the current numbers for my laptop
> and my rPi:
> The result is that the current gossip_store:
> - load time (-O3 -flto laptop): 90msec
> - load time (-O3 rPi): outlier 3110msec, mean 500msec
> - Total memory: 7.0MB:
> - 3.8MB for the array of channels
> - 2.0MB for the channel htable to map scid -> channel.
> - 460k for the array of nodes
> - 510k for the node htable to map nodeid -> node.
> - 245k for the array of channels inside each node
> For the million-channels-project (~780MB of gossip):
> - load time (-O3 -flto laptop): 1153msec
> - load time (-O3 rPi): outlier 50,000msec, mean 35,000 msec
> - Total memory: 67.4MB:
> - 39.7MB for the array of channels
> - 16.0MB for the channel htable to map scid -> channel.
> - 2.4MB for the array of nodes
> - 2.0MB for the node htable to map nodeid -> node.
> - 7.3MB for the array of channels inside each node
Thank you for these numbers.
I still feel it worthwhile to consider sharing the memory for the routemap.
In principle, nothing requires `permuteroute` to live in the same plugin as `recalcroute`, for instance, but the fact of replication of the index seems to push them into the same plugin instead (admittedly, `recalcroute` can simply use `listchannels` instead without maintaining its own routing index).
More information about the c-lightning