[PATCH v6 5/5] erofs-utils: mkfs: introduce inner-file multi-threaded compression

[Gao Xiang]

On 2024/3/14 20:37, Yifan Zhao wrote:
> Currently, the creation of EROFS compressed image creation is
> single-threaded, which suffers from performance issues. This patch
> attempts to address it by compressing the large file in parallel.
> 
> Specifically, each input file larger than 16MB is splited into segments,
> and each worker thread compresses a segment as if it were a separate
> file. Finally, the main thread merges all the compressed segments.
> 
> Multi-threaded compression is not compatible with -Ededupe,
> -E(all-)fragments and -Eztailpacking for now.
> 
> Signed-off-by: Yifan Zhao <zhaoyifan at sjtu.edu.cn>
> Co-authored-by: Tong Xin <xin_tong at sjtu.edu.cn>

I did some updates yesterday and I just posted v7.

BTW, I also found an issue that the output cannot be stablized with
"mkfs.erofs -zlz4hc,12 --worker=72" and enwik9 dataset.  I'm still
looking into that since it's an unexpected behavior.

Thanks,
Gao Xiang