[c-lightning] Shared-memory Routemap

[ZmnSCPxj]

With a properly defined interface, it would be possible to implement with both non-shared and shared memory techniques.

For instance, the interface I defined would be doable with the Rusty technique of having each plugin read the `gossip_store` and individually create data structures, just as well as for a shared-memory implementation.

Constructing a view would refresh the process-local data structure via `check_gossip_store` (instead of requesting access to shared memory from `routemapd`) and releasing the view would do nothing (instead of releasing the shared memory access back to `routemapd` to let `routemapd` continue its background updating).

We also need to figure out how to signal deletions in some common way between both techniques.
For example, constructing the `struct routemap` may require registration of a deletion callback.
For shared-memory technique the callback would be called on reception of a `routemap_object_deletion` notification.
For process-local technique the callback would be called during the construction of a view.


Due to the need to register a notification, however, there are some complications with how to construct the  `struct routemap` when using shared-memory technique.
It may be doable by having the routemap system subsume the plugin initialization and registration process, i.e. the routemap system exposes new APIs that replace those in `libplugin` and allow the shared-routemap technique to add the notification handler.


(the whole point of proper API design is to allow implementations to be replaced with little to no impact on the callers of the API.)
(it would be possible to create a proper API for the process-local version that we can later upgrade to using shared-memory (because reducing swap usage is always a good thing regardless, the only question is whether to prioritize or not) without changing plugin code, and with sufficiently good API design skill this can be done without even recompiling the plugin (though may need to relink to a different version of the routemap library).)


Regards,
ZmnSCPxj

> Hi C Lightning List!
>
> Sorry for top-posting without quotes. I just want to let
> you know I read it all and understand ZmnSCPxj's concern
> about memory.
>
> On the other hand I also understand the consumption
> is pretty minimal and hardly comparable to anything
> else which is not written in C.
>
> Let me remember the "old" times (I may be quite young here).
> Yes, I remember computers with 4, 8, 16, 32 megabytes of memory.
> On a Pentium with 32MB of RAM I ran my first Linux (loaded via
> oadlin from FAT, using UMSDOS to store UNIX file attributes).
>
> Now the least RAM (512MB) in household is on an ARM box
> which I got for free at some conference. Much more memory
> on refurbished Androids which serve me as Linux servers...
> If possible, let us do some experiments, e.g. would it be
> possible to have the same featureful version with and without
> the memory-sharing change? Is it possible to separate this
> change so that both versions can live for some time?
>
> I am happy to help with testing. All machines here
> (arm, arm64, i686, x86_64 all both on glibc and musl libc)
> are willing to help.
> Have a good weekend!
> Kind regards, Jasan