Alternative to the filesystem overlay

[William Kennington]

Sounds like this is just a data schema update type of issue, where you
just need a service to run at first update boot and upgrade the files
as expected by the change.

We could probably do this with systemd services that are
ConditionNeedsUpdate= and
https://www.freedesktop.org/software/systemd/man/systemd-update-done.service.html

I'd prefer to see us move to a mostly immutable filesystem with
symlinks that map into a rwfs for specific files / directories we
expect to mutate at runtime. I don't think we should do any smart
types of writeback caching at the filesystem level as that will just
lead to confusion about persistence that is hard to debug. Not to
mention tools like rsync don't guarantee atomic snapshotting during
the copy for file consistency. Our applications should be designed in
such a way that they are always consistent on disk. Ex. writing text
files should be done to a temporary, then moved over the old one. That
way the update is atomic and we don't have partially written files.

On Fri, Aug 16, 2019 at 2:10 AM Alexander A. Filippov
<a.filippov at yadro.com> wrote:
>
> On Thu, Aug 15, 2019 at 12:38:16PM -0500, Adriana Kobylak wrote:
> > In OpenBMC we've been using an overlay to make the read-only filesystem (or
> > part of it) writable. It hasn't come without its issues, like missing
> > directories (openbmc/openbmc#3564) and various flavors of corruption
> > (openbmc/openbmc#3578 and issues when adding files to /etc in the build
> > causing files that had been previously modified on the bmc to become
> > corrupted).
>
> There is one more issue for the same type - openbmc/openbmc#3468.
> And it can't be solved by the way you propose.
>
> >
> > An alternative would be to use a bind mount and manage the file merging by
> > an OpenBMC app. Doing some experimentation, seems this solves the issues
> > seen above. The way that it'd work would be:
> > - The init script or initramfs would do an rsync from /etc to the bind mount
> > destination directory in the read-write filesystem, skipping the contents of
> > a list which would contain the files that have been modified in the BMC and
> > should be preserved.
> > - A new OpenBMC app would monitor /etc for changes via inotify for example,
> > and if a file is modified, it'd add it to the "list".
> >
> > Considerations:
> > - The /etc dir mainly being a configuration destination does not get
> > modified often so the app that is watching the directory would not be
> > triggered often.
> > - The bind mount would duplicate the contents of etc although it's not much.
> > If space is a concern, some exploration could be done to have the bind mount
> > be a tmpfs and have the monitor app copy the modified files to the
> > read-write filesystem instead of adding them to a list.
> >
> > Thoughts?
> >
>
> Most of these issues can be solved by the simple shell scripts. We could modify
> the initramfs/init scenario to exec these scripts at the moment when ROFS and
> RWFS have mounted but are not combined to OVERLAY yet.
>
> Of course, this method doesn't prevent an occurrence of any new issues of this
> type but brings an easy way to solve them.
>
> --
> Regards,
> Alexander