[PATCH 0/7] erofs-utils: mkfs: introduce multi-threaded compression

[Yifan Zhao]

Hi all,

Currently, mkfs.erofs is single-threaded and the time-consuming process
of creating EROFS images has emerged as a critical factor. Let's introduce
multi-threaded compression to speed up the process. This patchset is based
on the previous work of Gao Xiang and the multi-threading framework has
been introduced in the first two patches. The following patches introduce
inner-file and inter-file multi-threaded compression.

The inner-file multi-threaded compression is implemented by dividing the
input file into multiple segments and compressing them in parallel.

The inter-file multi-threaded compression is implemented by compressing
multiple files in parallel, and we let the main thread reaps the compressed
files and writes them to the output image to ensure the reproducibility of
the output image.

This patchset is only a basic implementation without support for dedupe,
(all-)fragments and ztailpacking features. We will add support for them
in the future.

The performance of the multi-threaded compression is tested on a Dell r740
server with 2 Intel Xeon Gold 5215M CPUs (10 cores per CPU). We test the
compression time of serveral workloads with lzma algorithm. The results are
as follows:

workload: enwik9 (1GB)

| # of threads | Time (s) | Speedup |
|--------------|----------|---------|
| 1            |  216.08  | 1.00x   |
| 2            |  110.15  | 1.96x   |
| 4            |  57.22   | 3.78x   |
| 8            |  31.4    | 6.88x   |
| 16           |  16.99   | 12.72x  |

workload: Linux-6.7-rc4 source (1.55GB)

| # of threads | Time (s) | Speedup |
|--------------|----------|---------|
| 1            |  288.65  | 1.00x   |
| 2            |  140.66  | 2.05x   |
| 4            |  79.64   | 3.62x   |
| 8            |  54.74   | 5.27x   |
| 16           |  47.08   | 6.13x   |

workload: Arch container with Xfce desktop (4.19GB)

| # of threads | Time (s) | Speedup |
|--------------|----------|---------|
| 1            |  834.26  | 1.00x   |
| 2            |  430.73  | 1.94x   |
| 4            |  267.96  | 3.11x   |
| 8            |  205.67  | 4.06x   |
| 16           |  183.26  | 4.55x   |

Gao Xiang (2):
  erofs-utils: introduce multi-threading framework
  erofs-utils: add a helper to get available processors

Yifan Zhao (5):
  erofs-utils: mkfs: add --worker=# parameter
  erofs-utils: mkfs: optionally print warning in erofs_compressor_init
  erofs-utils: extend multi-threading framework for per-thread fields
  erofs-utils: mkfs: introduce inner-file multi-threaded compression
  erofs-utils: mkfs: introduce inter-file multi-threaded compression

 configure.ac                |  17 +
 include/erofs/compress.h    |  18 +
 include/erofs/config.h      |   5 +
 include/erofs/internal.h    |   6 +
 include/erofs/list.h        |   8 +
 include/erofs/queue.h       |  22 +
 include/erofs/workqueue.h   |  48 +++
 lib/Makefile.am             |   4 +
 lib/compress.c              | 819 +++++++++++++++++++++++++++++-------
 lib/compressor.c            |   7 +-
 lib/compressor.h            |   5 +-
 lib/compressor_deflate.c    |  10 +-
 lib/compressor_libdeflate.c |   6 +-
 lib/compressor_liblzma.c    |   9 +-
 lib/compressor_lz4.c        |   2 +-
 lib/compressor_lz4hc.c      |   2 +-
 lib/config.c                |  16 +
 lib/inode.c                 | 346 ++++++++++++---
 lib/queue.c                 |  64 +++
 lib/workqueue.c             | 222 ++++++++++
 mkfs/main.c                 |  38 ++
 21 files changed, 1442 insertions(+), 232 deletions(-)
 create mode 100644 include/erofs/queue.h
 create mode 100644 include/erofs/workqueue.h
 create mode 100644 lib/queue.c
 create mode 100644 lib/workqueue.c

-- 
2.43.0