[PATCH v6 5/5] erofs-utils: mkfs: introduce inner-file multi-threaded compression

[Gao Xiang]

On 2024/3/15 10:39, Gao Xiang wrote:
> 
> 
> On 2024/3/14 20:37, Yifan Zhao wrote:
>> Currently, the creation of EROFS compressed image creation is
>> single-threaded, which suffers from performance issues. This patch
>> attempts to address it by compressing the large file in parallel.
>>
>> Specifically, each input file larger than 16MB is splited into segments,
>> and each worker thread compresses a segment as if it were a separate
>> file. Finally, the main thread merges all the compressed segments.
>>
>> Multi-threaded compression is not compatible with -Ededupe,
>> -E(all-)fragments and -Eztailpacking for now.
>>
>> Signed-off-by: Yifan Zhao <zhaoyifan at sjtu.edu.cn>
>> Co-authored-by: Tong Xin <xin_tong at sjtu.edu.cn>
> 
> I did some updates yesterday and I just posted v7.
> 
> BTW, I also found an issue that the output cannot be stablized with
> "mkfs.erofs -zlz4hc,12 --worker=72" and enwik9 dataset.  I'm still
> looking into that since it's an unexpected behavior.

I found the issue is really with "-Eztailpacking", and now fixed.

Thanks,
Gao Xiang