erofs + zstd : a super combination!

[Gao Xiang]

(cc linux-erofs mailing list for archival use only)

On Tue, Oct 18, 2022 at 04:26:09PM -0500, André Goddard Rosa wrote:
> On Tue, Oct 18, 2022 at 4:15 PM Gao Xiang <hsiangkao at linux.alibaba.com>
> wrote:
> 
> > Hi André,
> >
> > On Tue, Oct 18, 2022 at 03:34:18PM -0500, André Goddard Rosa wrote:
> > > Hi Gao Xiang, good day!
> > >
> > > I noticed that you were trying to add support for zstd to erofs in #1399
> > > [1] , but you mentioned it was a low priority at the time. By looking at
> > > this nice article [2], erofs+zstd are probably going to give an excellent
> > > compression rate with a really fast decompression, so it can compress
> > > better than LZ4 with a decompression performance much better than
> > MicroLZMA
> > > (it's outstanding you worked with Lasse Collin on that!) benefitting a
> > lot
> > > of small systems with low memory.
> > >
> > > As you mentioned, some slight variation resembling a binary search could
> > be
> > > done in order to make use of zstd. Also, Yann Collet, lz4&zstd author,
> > > suggested a similar mechanism on #204 [3].
> > >
> > > Because of the great potential for this, I'd like to ask what are your
> > > plans with regard to it. Please let me know if you would need any help
> > with
> > > it or if you have anything in progress or plans you could share with me.
> >
> > There are several ways to make EROFS work with zstd.  Of course [3] will
> > be a way, but I've seen they have a superblock compression support as
> > well. Or I can just compress to several blocks and leave the last
> > remaining block as is even if it's not full.
> >
> > I will try to implment zstd in my spare time in the next few months, but
> > I cannot promise anything.  Since once the implementation is done, it
> > has to be supported all the time in order to make EROFS compatibility.
> > Therefore, the formal version will be done with care.
> >
> 
> Great, thanks for the quick update Gao Xiang!
> 
> Please let me know if you'd benefit from any help in implementing this
> feature. I can definitely help to test too.
> 
> In many cases, the compression speed is less important than the
> decompression time,
> so even if there's no perfectly optimized zstd backend, it could still
> work. Also, it would be

Yes, even we try to compress to several blocks and leave the last remaining
block non-full.  It's still comparable to btrfs or f2fs since they both
work as this way.  And anyway, the decompression side is unrelated to the
compression side.

> really nice if it could be supported on older devices running kernel 4.19+
> like erofs
> originally did.

I mostly work for the upstream kernel and LTS kernels are bugfixes only
so they don't really accept feature backports and I'm afraid I don't
have full time to look after so many old LTS kernels with new features..

Thanks,
Gao Xiang

> 
> Thanks and best regards,
> Andre