Alternative to the filesystem overlay

[Adriana Kobylak]

In OpenBMC we've been using an overlay to make the read-only filesystem 
(or part of it) writable. It hasn't come without its issues, like 
missing directories (openbmc/openbmc#3564) and various flavors of 
corruption (openbmc/openbmc#3578 and issues when adding files to /etc in 
the build causing files that had been previously modified on the bmc to 
become corrupted).

An alternative would be to use a bind mount and manage the file merging 
by an OpenBMC app. Doing some experimentation, seems this solves the 
issues seen above. The way that it'd work would be:
- The init script or initramfs would do an rsync from /etc to the bind 
mount destination directory in the read-write filesystem, skipping the 
contents of a list which would contain the files that have been modified 
in the BMC and should be preserved.
- A new OpenBMC app would monitor /etc for changes via inotify for 
example, and if a file is modified, it'd add it to the "list".

Considerations:
- The /etc dir mainly being a configuration destination does not get 
modified often so the app that is watching the directory would not be 
triggered often.
- The bind mount would duplicate the contents of etc although it's not 
much. If space is a concern, some exploration could be done to have the 
bind mount be a tmpfs and have the monitor app copy the modified files 
to the read-write filesystem instead of adding them to a list.

Thoughts?