Merging multiple erofs file systems on the same block device

[Gao Xiang]

On 2023/5/2 18:03, Daan De Meyer wrote:
>> On 2023/5/1 22:09, Daan De Meyer wrote:
>>> Hi,
>>>
>>> I've been looking into erofs as an initramfs replacement by using
>>> root=/dev/ram0 to tell the kernel to load the initramfs as a ramdisk.
>>
>> Sorry, I'm on vacation now.
>>
>> May I ask what's your detailed use cases?  Sure, you could use
>> /dev/ram0 as a replacement, but currently it still takes double
>> memory compared with initramfs since ramdisk doesn't support FSDAX
>> for now (by enabling FSDAX, it won't take double memory at all.)
> 
> I'm experimenting with larger initramfses and running into memory
> bottlenecks since the entire compressed cpio has to be decompressed
> into memory. I was hoping to use erofs as a replacement that could stay
> compressed, where only the files that are actually accessed are
> decompressed at runtime.

Sorry for late reply.

Okay, that makes sense, although FSDAX cannot be used as this way since
decompressed data is needed for mmapped accesses.

> 
>> Actually I think ramdisk FSDAX is useful and I might sync up this on
>> the following LSF/MM/BPF 2023.
>>
>>> However, by using a ramdisk instead of the usual compressed cpio, I
>>> would lose the feature where the kernel merges multiple individual
>>> cpios together into a single tmpfs filesystem. Looking at the
>>> documentation for erofs, I noticed that erofs already seems to support
>>> merging multiple erofs filesystems on separate block devices using the
>>> device= cmdline option. Would it be possible to extend this so that
>> Here `device=` is actually used to refer to seperate blobs with the
>> merged metadata.  For example, you could have
>>
>>     device=/dev/ram1 original tar1
>>     device=/dev/ram2 original tar2
>>     /dev/ram0        merged metadata for tar1 + tar2.
>>
>> which means, if you'd like to merge multiple EROFS filesystems, you
>> might need another step to build a merged metadata in advance in order
>> to merges multiple individual tarballs together, which could be built
>> when applying images or booting (by using a special bootloader with
>> such functionality.)
> 
> Ahh, I misunderstood the device= option then.
> 
>> EROFS doesn't support stacking multiple fses runtimely since it seems
>> a duplicated feature of overlayfs (you could consider using overlayfs
>> honestly.)
> 
> I would love to use overlayfs, but there's no way to specify to the kernel that
> the initrd should be set up as an overlayfs of a set of ram disks. It would be
> interesting if I could put multiple filesystems in the initrd and the
> kernel would
> notice and automatically set up an overlayfs of them.

I didn't use overlayfs as this way so I'm not sure as well.  Yet as a wild
guess, you could specify a ramdisk with a customized init to stack
overlayfs like this in the userspace?  Not sure though...

Thanks,
Gao Xiang