Merging multiple erofs file systems on the same block device

[Gao Xiang]

On 2023/5/5 16:19, Daan De Meyer wrote:
>> On 2023/5/2 18:03, Daan De Meyer wrote:
>>>> On 2023/5/1 22:09, Daan De Meyer wrote:
>>>>> Hi,
>>>>>
>>>>> I've been looking into erofs as an initramfs replacement by using
>>>>> root=/dev/ram0 to tell the kernel to load the initramfs as a ramdisk.
>>>>
>>>> Sorry, I'm on vacation now.
>>>>
>>>> May I ask what's your detailed use cases?  Sure, you could use
>>>> /dev/ram0 as a replacement, but currently it still takes double
>>>> memory compared with initramfs since ramdisk doesn't support FSDAX
>>>> for now (by enabling FSDAX, it won't take double memory at all.)
>>>
>>> I'm experimenting with larger initramfses and running into memory
>>> bottlenecks since the entire compressed cpio has to be decompressed
>>> into memory. I was hoping to use erofs as a replacement that could stay
>>> compressed, where only the files that are actually accessed are
>>> decompressed at runtime.
>>
>> Sorry for late reply.
>>
>> Okay, that makes sense, although FSDAX cannot be used as this way since
>> decompressed data is needed for mmapped accesses.
> 
> Can you clarify why using a ramdisk would take double the memory if FSDAX
> is not used?

You could take a look at
https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt

and section "ramfs and ramdisk" describes the details, but FSDAX could avoid
this problem.

> 
>>>
>>>> Actually I think ramdisk FSDAX is useful and I might sync up this on
>>>> the following LSF/MM/BPF 2023.
>>>>
>>>>> However, by using a ramdisk instead of the usual compressed cpio, I
>>>>> would lose the feature where the kernel merges multiple individual
>>>>> cpios together into a single tmpfs filesystem. Looking at the
>>>>> documentation for erofs, I noticed that erofs already seems to support
>>>>> merging multiple erofs filesystems on separate block devices using the
>>>>> device= cmdline option. Would it be possible to extend this so that
>>>> Here `device=` is actually used to refer to seperate blobs with the
>>>> merged metadata.  For example, you could have
>>>>
>>>>      device=/dev/ram1 original tar1
>>>>      device=/dev/ram2 original tar2
>>>>      /dev/ram0        merged metadata for tar1 + tar2.
>>>>
>>>> which means, if you'd like to merge multiple EROFS filesystems, you
>>>> might need another step to build a merged metadata in advance in order
>>>> to merges multiple individual tarballs together, which could be built
>>>> when applying images or booting (by using a special bootloader with
>>>> such functionality.)
>>>
>>> Ahh, I misunderstood the device= option then.
>>>
>>>> EROFS doesn't support stacking multiple fses runtimely since it seems
>>>> a duplicated feature of overlayfs (you could consider using overlayfs
>>>> honestly.)
>>>
>>> I would love to use overlayfs, but there's no way to specify to the kernel that
>>> the initrd should be set up as an overlayfs of a set of ram disks. It would be
>>> interesting if I could put multiple filesystems in the initrd and the
>>> kernel would
>>> notice and automatically set up an overlayfs of them.
>>
>> I didn't use overlayfs as this way so I'm not sure as well.  Yet as a wild
>> guess, you could specify a ramdisk with a customized init to stack
>> overlayfs like this in the userspace?  Not sure though...
> 
> This approach generally works, but if you want to enforce all mounted
> filesystems to be dm-verity protected, you now need to add verity data
> for all these filesystems to the initramfs. That's why we're looking
> for a solution where the kernel sets up the filesystem mount of the
> ramdisk which is special cased and doesn't require verity data to be
> present. Anyway, the overlayfs issue is not something for erofs to
> solve. If at all desirable, it should probably be filesystem
> independent where the kernel just looks for multiple filesystems in
> the initrd buffer and sets up an overlay mount using each found
> filesystem.

I'm not sure if you could still specify one root filesystem with
dm-verity and it mounts filesystems of rest layers and then uses
overlayfs properly with a customized init.

Yes, anyway, you could enhance kernel init code as well.

Thanks,
Gao Xiang

> 
> Cheers,
> 
> Daan De Meyer