Merging multiple erofs file systems on the same block device

[Daan De Meyer]

> On 2023/5/2 18:03, Daan De Meyer wrote:
> >> On 2023/5/1 22:09, Daan De Meyer wrote:
> >>> Hi,
> >>>
> >>> I've been looking into erofs as an initramfs replacement by using
> >>> root=/dev/ram0 to tell the kernel to load the initramfs as a ramdisk.
> >>
> >> Sorry, I'm on vacation now.
> >>
> >> May I ask what's your detailed use cases?  Sure, you could use
> >> /dev/ram0 as a replacement, but currently it still takes double
> >> memory compared with initramfs since ramdisk doesn't support FSDAX
> >> for now (by enabling FSDAX, it won't take double memory at all.)
> >
> > I'm experimenting with larger initramfses and running into memory
> > bottlenecks since the entire compressed cpio has to be decompressed
> > into memory. I was hoping to use erofs as a replacement that could stay
> > compressed, where only the files that are actually accessed are
> > decompressed at runtime.
>
> Sorry for late reply.
>
> Okay, that makes sense, although FSDAX cannot be used as this way since
> decompressed data is needed for mmapped accesses.

Can you clarify why using a ramdisk would take double the memory if FSDAX
is not used?

> >
> >> Actually I think ramdisk FSDAX is useful and I might sync up this on
> >> the following LSF/MM/BPF 2023.
> >>
> >>> However, by using a ramdisk instead of the usual compressed cpio, I
> >>> would lose the feature where the kernel merges multiple individual
> >>> cpios together into a single tmpfs filesystem. Looking at the
> >>> documentation for erofs, I noticed that erofs already seems to support
> >>> merging multiple erofs filesystems on separate block devices using the
> >>> device= cmdline option. Would it be possible to extend this so that
> >> Here `device=` is actually used to refer to seperate blobs with the
> >> merged metadata.  For example, you could have
> >>
> >>     device=/dev/ram1 original tar1
> >>     device=/dev/ram2 original tar2
> >>     /dev/ram0        merged metadata for tar1 + tar2.
> >>
> >> which means, if you'd like to merge multiple EROFS filesystems, you
> >> might need another step to build a merged metadata in advance in order
> >> to merges multiple individual tarballs together, which could be built
> >> when applying images or booting (by using a special bootloader with
> >> such functionality.)
> >
> > Ahh, I misunderstood the device= option then.
> >
> >> EROFS doesn't support stacking multiple fses runtimely since it seems
> >> a duplicated feature of overlayfs (you could consider using overlayfs
> >> honestly.)
> >
> > I would love to use overlayfs, but there's no way to specify to the kernel that
> > the initrd should be set up as an overlayfs of a set of ram disks. It would be
> > interesting if I could put multiple filesystems in the initrd and the
> > kernel would
> > notice and automatically set up an overlayfs of them.
>
> I didn't use overlayfs as this way so I'm not sure as well.  Yet as a wild
> guess, you could specify a ramdisk with a customized init to stack
> overlayfs like this in the userspace?  Not sure though...

This approach generally works, but if you want to enforce all mounted
filesystems to be dm-verity protected, you now need to add verity data
for all these filesystems to the initramfs. That's why we're looking
for a solution where the kernel sets up the filesystem mount of the
ramdisk which is special cased and doesn't require verity data to be
present. Anyway, the overlayfs issue is not something for erofs to
solve. If at all desirable, it should probably be filesystem
independent where the kernel just looks for multiple filesystems in
the initrd buffer and sets up an overlay mount using each found
filesystem.

Cheers,

Daan De Meyer


On Fri, 5 May 2023 at 07:05, Gao Xiang <hsiangkao at linux.alibaba.com> wrote:
>
>
>
> On 2023/5/2 18:03, Daan De Meyer wrote:
> >> On 2023/5/1 22:09, Daan De Meyer wrote:
> >>> Hi,
> >>>
> >>> I've been looking into erofs as an initramfs replacement by using
> >>> root=/dev/ram0 to tell the kernel to load the initramfs as a ramdisk.
> >>
> >> Sorry, I'm on vacation now.
> >>
> >> May I ask what's your detailed use cases?  Sure, you could use
> >> /dev/ram0 as a replacement, but currently it still takes double
> >> memory compared with initramfs since ramdisk doesn't support FSDAX
> >> for now (by enabling FSDAX, it won't take double memory at all.)
> >
> > I'm experimenting with larger initramfses and running into memory
> > bottlenecks since the entire compressed cpio has to be decompressed
> > into memory. I was hoping to use erofs as a replacement that could stay
> > compressed, where only the files that are actually accessed are
> > decompressed at runtime.
>
> Sorry for late reply.
>
> Okay, that makes sense, although FSDAX cannot be used as this way since
> decompressed data is needed for mmapped accesses.
>
> >
> >> Actually I think ramdisk FSDAX is useful and I might sync up this on
> >> the following LSF/MM/BPF 2023.
> >>
> >>> However, by using a ramdisk instead of the usual compressed cpio, I
> >>> would lose the feature where the kernel merges multiple individual
> >>> cpios together into a single tmpfs filesystem. Looking at the
> >>> documentation for erofs, I noticed that erofs already seems to support
> >>> merging multiple erofs filesystems on separate block devices using the
> >>> device= cmdline option. Would it be possible to extend this so that
> >> Here `device=` is actually used to refer to seperate blobs with the
> >> merged metadata.  For example, you could have
> >>
> >>     device=/dev/ram1 original tar1
> >>     device=/dev/ram2 original tar2
> >>     /dev/ram0        merged metadata for tar1 + tar2.
> >>
> >> which means, if you'd like to merge multiple EROFS filesystems, you
> >> might need another step to build a merged metadata in advance in order
> >> to merges multiple individual tarballs together, which could be built
> >> when applying images or booting (by using a special bootloader with
> >> such functionality.)
> >
> > Ahh, I misunderstood the device= option then.
> >
> >> EROFS doesn't support stacking multiple fses runtimely since it seems
> >> a duplicated feature of overlayfs (you could consider using overlayfs
> >> honestly.)
> >
> > I would love to use overlayfs, but there's no way to specify to the kernel that
> > the initrd should be set up as an overlayfs of a set of ram disks. It would be
> > interesting if I could put multiple filesystems in the initrd and the
> > kernel would
> > notice and automatically set up an overlayfs of them.
>
> I didn't use overlayfs as this way so I'm not sure as well.  Yet as a wild
> guess, you could specify a ramdisk with a customized init to stack
> overlayfs like this in the userspace?  Not sure though...
>
> Thanks,
> Gao Xiang