Worse performance than SquashFS for small filesystems

[Gao Xiang]

Hi Paul,

On Tue, May 31, 2022 at 10:58:01AM +0200, Paul Spooren wrote:
> Hi,
> 
> Thank you for the quick response!
> 
> > On 31. May 2022, at 02:00, Gao Xiang <xiang at kernel.org> wrote:
> > 
> > On Mon, May 30, 2022 at 08:41:34PM +0200, Paul Spooren wrote:
> >> Hi,
> >> 
> >> I’m an OpenWrt developer and we work on an operation system for routers and overall network facing embedded devices. Since storage is often limited on our targeted devices we use SquashFS to compress everything, total image sizes are less than 10 MegaBytes.
> >> 
> >> Reading about erofs I’m very keen to adopt it for our case, however giving it a first try the storage performance seems to be significantly _worse_ than the default SquashFS implementation. Since you ran benchmarks in the past, could you give me advise if I’m missing anything?
> > 
> > What the meaning of `performance'? I don't think the size of images
> > mean runtime performance anyway.
> 
> Performance as in looking at how well it performs on minimizing the space requirement.

Originally, EROFS had a goal to have a better runtime performance with
minimal memory overhead (with some inplace techniques).

Strictly speaking, if you'd like to minimize the storage space, there
are many FUSE fses which can do this, but such fses doesn't consider
more about memory footprints or runtime I/O amplification and that's
what EROFS addresses exactly.

But yeah, we tend to perform a generic in-kernel fses, I think we can
do in this way (saving more space without impact runtime performance)
with our best effort.

> 
> > 
> >> 
> >> For the test I used erofs-utils as of a134ce7c39a5427343029e97a62665157bef9bc3 (2022-05-17) and compressed the x86/64 root filesystem of a standard OpenWrt image[1]. I’m seeing a difference of one MegaByte which is about 30% worse in this context.
> >> 
> >> My test:
> >> 
> >> $ ./staging_dir/host/bin/mkfs.erofs -zlzma erofs.root ./build_dir/target-x86_64_musl/root-x86
> > 
> > Have you try with
> > 
> > -zlzma -C 65536 -Eztailpacking
> > 
> > by default, EROFS uses 4k compression rather than Squashfs 128k.
> 
> That indeed improves the total size, I’m now down to 3.6MB (vs 3.3MB for squashfs).

Good to hear that.

> 
> > 
> >> mkfs.erofs 1.4
> >> <W> erofs: EXPERIMENTAL MicroLZMA feature in use. Use at your own risk!
> >> <W> erofs: Note that it may take more time since the compressor is still single-threaded for now.
> >> Build completed.
> >> 
> >> $ mksquashfs -comp xz ./build_dir/target-x86_64_musl/root-x86 squashfs.root
> >> 
> >> $ ls -lh *.root
> >> -rw-r--r-- 1 ubuntu ubuntu 4.3M May 30 20:27 erofs.root
> >> -rw-r--r-- 1 ubuntu ubuntu 3.3M May 30 20:28 squashfs.root
> > 
> > Even EROFS now supports big pcluster and ztailpacking features,
> > Squashfs supports a feature called fragments which compresses
> > several tails of files in a fragment. It's obviously beneficial
> > to the final size of images but it can read unrelated data of
> > other files even such files are very small.
> > 
> > You can try a big file with EROFS and Squashfs, and you can
> > see the difference.
> > 
> > Btw, MicroLZMA is still an experimental feature for now, due to 
> > three reasons:
> > - XZ utils hasn't release a stable version for now, which I think
> >   Lasse will release a stable version this year;
> > 
> > - EROFS has a finalized on-disk MircoLZMA support since Linux 5.16,
> >   so users can mount MicroLZMA since 5.16.  Yet currently EROFS is
> >   not quite optimized for such slow algorithm, since it needs a
> >   completely different strategy from LZ4.  I'm working on that
> >   together with folio work;
> > 
> > - We need a similiar `fragment' feature to catch squashfs under a
> >   lot of files.
> 
> Is that feature likely to be implemented for EROFS or is that out of scope?

As far as I know, Yue Hu has some interest working on this. But that is
not a promise anyway, I can seek time working on this as well.

> 
> > 
> >> 
> >> Is erofs just not meant for such small storages?
> > 
> > If you care more about the size of images, I personally prefer to
> > squashfs for now until we handle all the things above.
> 
> Sure, I didn’t plan to migrate things over to EROFS just like that but it looks like a good candidate in the future!

Yeah, I think anyway openwrt can support it as an alternative fs, but
anyway, if you pursue extreme storage saving, the solution may vary
endlessly and could be much complicated (and definitely impact runtime
performance.)

But we can do in this way further if it seems simply like fragments.

> 
> Thank you and everyone involved for working on EROFS!
> 

Thanks,
Gao Xing

> Best,
> Paul Spooren
> 
> > 
> > Thanks,
> > Gao Xiang
> > 
> >> 
> >> Thanks for all further comments!
> >> 
> >> Best,
> >> Paul
> >> 
> >> [1]: https://downloads.openwrt.org/snapshots/targets/x86/64/openwrt-x86-64-rootfs.tar.gz