[WIP] [PATCH v0.0-20200229 00/11] ez: LZMA fixed-sized output compression

[Gao Xiang]

Hi Eric,

On Fri, Feb 28, 2020 at 08:58:42PM -0800, Eric Biggers wrote:
> On Sat, Feb 29, 2020 at 12:50:06PM +0800, Gao Xiang via Linux-erofs wrote:
> > From: Gao Xiang <gaoxiang25 at huawei.com>
> > 
> > This is a WIP PREVIEW patchset, just for archiving to open
> > source community only.
> > 
> > For now, it implements LZMA SDK-like GetOptimumFast approach
> > and GetOptimum is still on schedule.
> > 
> > It's still buggy, lack of formal APIs and actively under
> > development for a while...
> > 
> > Usage:
> > $ ./run.sh
> > $ ./a.out output.bin.lzma infile
> > 
> > It will compress the beginning as much as possible into
> > 4k RAW LZMA block.
> 
> Why not just use liblzma?

I discuss with Lasse and Igor in private, they have no recent plan to
develop fixed-sized output compression for now so it could not have
a formal upstream release in the near future.

After I digged into liblzma, it has many filters and LZMA2 wrapper so
it needs some more hack for now. I tend that Lasse could implement it
(Lasse said no recent, but it can be potential upstreamed. In fact,
liblzma is odd fixes status recent years compared with LZMA SDK...
However LZMA SDK coding style is so weird.)

On the other hand, it's just a product when I learned LZMA internals,
just like libdeflate in some extent. Bacause I need to do some coding
so I can learn LZMA internals and get how to do next step by step
because I'm not a LZMA expert at first.

> 
> Also, if you care enough about compression ratio to use LZMA instead of
> Zstandard, why use only a 4 KB blocksize?

They are all on schedule. I need some priority because I only have 24
hours a day and the content of my HUAWEI job is not all about EROFS. :(

If Zstandard developers have more interests on it. I'm happy to integrate
Zstd to EROFS as well. Otherwise, I need to do them myself. As a quick
glance, Zstandard uses multi-pass approach, so it may be hard for me to
do a quick hack. And I don't know all Zstandard internals as well.

Why use only a 4 KB blocksize --- It's just that I need to find a cleverer
and elegant way to do in-place decompression for larger physical cluster.
I need to think it over otherwise it could have potential compatibility
concerns and make EROFS more complex in the future.

In fact, Lasse raised a clever idea month ago. I will sort it out after
LZMA algortithm for EROFS is ready.

LZMA and larger physical cluster will be implemented this year If nothing
unexpected happens. I'm happy more experts and volunteers could have interest
in it but if only me, I only can do my best in my spare time...

Thanks,
Gao Xiang

> 
> - Eric