[PATCH 0/4] erofs-utils: code-refactoring for erofs compressor
Guo Xuenan
guoxuenan at huawei.com
Thu Jun 15 20:17:23 AEST 2023
simplify and unify erofs compressor code.
Test by following script:
```shell
#!/bin/bash
FSCK=`which fsck.erofs`
MKFS=`which mkfs.erofs`
DPFS=`which dump.erofs`
SRC_DIR=./src
IMGDIR=./image/
mkdir -p ${IMGDIR}
create_erofsimg()
{
local img=$1
echo "MKFS [mkfs.erofs $@]"
$MKFS $@ ${SRC_DIR}
}
#test mkfs.erofs
create_erofsimg ${IMGDIR}/erofs-raw32.img -E force-inode-compact || exit -11
create_erofsimg ${IMGDIR}/erofs-raw64.img -E force-inode-extended || exit -12
create_erofsimg ${IMGDIR}/erofs-lzma-4k.img -zlzma,9 || exit -13
create_erofsimg ${IMGDIR}/erofs-lzma-bigpcluster-8k.img -zlzma,9 -C 8192 || exit -14
create_erofsimg ${IMGDIR}/erofs-lzma-rand-bigpcluster-1M.img -zlzma,9 -C 1048576 --random-pclusterblks || exit -15
create_erofsimg ${IMGDIR}/erofs-lz4-4k.img -zlz4hc,9 || exit -16
create_erofsimg ${IMGDIR}/erofs-lz4-rand-bigpcluster-1M.img -zlz4hc,9 -C 1048576 --random-pclusterblks || exit -17
create_erofsimg ${IMGDIR}/erofs-rand-alg-1M.img -zlz4:lzma,9:lz4hc,9 -C 1048576 --random-pclusterblks --random-algorithms || exit -18
create_erofsimg ${IMGDIR}/erofs-rand-frag-1M.img -zlz4:lzma,9:lz4hc,9 -C 8192 -Efragments,8192 --random-pclusterblks --random-algorithms || exit -19
create_erofsimg ${IMGDIR}/erofs-rand-frag-dedupe-1M.img -zlz4:lzma,9:lz4hc,9 -C 1048576 -Efragments,1048576 -Ededupe --random-algorithms || exit -20
#test {dump,fsck}.erofs
for img in `ls ${IMGDIR}/*.img`
do
echo "FSCK_DUMP==================$img====================="
mkdir -p ${img}_dir && $FSCK $img && $FSCK --extract=${img}_dir $img
$DPFS $img
done
ls -F ${IMGDIR}/
```
Output:
--------------------------------------------
MKFS [mkfs.erofs ./image//erofs-raw32.img -E force-inode-compact]
mkfs.erofs 1.6-g566da452-dirty
Build completed.
MKFS [mkfs.erofs ./image//erofs-raw64.img -E force-inode-extended]
mkfs.erofs 1.6-g566da452-dirty
Build completed.
MKFS [mkfs.erofs ./image//erofs-lzma-4k.img -zlzma,9]
mkfs.erofs 1.6-g566da452-dirty
<W> erofs: z_erofs_compress_init() Line[1105] EXPERIMENTAL MicroLZMA feature in use. Use at your own risk!
<W> erofs: z_erofs_compress_init() Line[1106] Note that it may take more time since the compressor is still single-threaded for now.
Build completed.
MKFS [mkfs.erofs ./image//erofs-lzma-bigpcluster-8k.img -zlzma,9 -C 8192]
mkfs.erofs 1.6-g566da452-dirty
<W> erofs: z_erofs_compress_init() Line[1105] EXPERIMENTAL MicroLZMA feature in use. Use at your own risk!
<W> erofs: z_erofs_compress_init() Line[1106] Note that it may take more time since the compressor is still single-threaded for now.
Build completed.
MKFS [mkfs.erofs ./image//erofs-lzma-rand-bigpcluster-1M.img -zlzma,9 -C 1048576 --random-pclusterblks]
mkfs.erofs 1.6-g566da452-dirty
<W> erofs: z_erofs_compress_init() Line[1105] EXPERIMENTAL MicroLZMA feature in use. Use at your own risk!
<W> erofs: z_erofs_compress_init() Line[1106] Note that it may take more time since the compressor is still single-threaded for now.
Build completed.
MKFS [mkfs.erofs ./image//erofs-lz4-4k.img -zlz4hc,9]
mkfs.erofs 1.6-g566da452-dirty
Build completed.
MKFS [mkfs.erofs ./image//erofs-lz4-rand-bigpcluster-1M.img -zlz4hc,9 -C 1048576 --random-pclusterblks]
mkfs.erofs 1.6-g566da452-dirty
Build completed.
MKFS [mkfs.erofs ./image//erofs-rand-alg-1M.img -zlz4:lzma,9:lz4hc,9 -C 1048576 --random-pclusterblks --random-algorithms]
mkfs.erofs 1.6-g566da452-dirty
<W> erofs: z_erofs_compress_init() Line[1105] EXPERIMENTAL MicroLZMA feature in use. Use at your own risk!
<W> erofs: z_erofs_compress_init() Line[1106] Note that it may take more time since the compressor is still single-threaded for now.
Build completed.
MKFS [mkfs.erofs ./image//erofs-rand-frag-1M.img -zlz4:lzma,9:lz4hc,9 -C 8192 -Efragments,8192 --random-pclusterblks --random-algorithms]
mkfs.erofs 1.6-g566da452-dirty
<W> erofs: main() Line[790] EXPERIMENTAL compressed fragments feature in use. Use at your own risk!
<W> erofs: z_erofs_compress_init() Line[1105] EXPERIMENTAL MicroLZMA feature in use. Use at your own risk!
<W> erofs: z_erofs_compress_init() Line[1106] Note that it may take more time since the compressor is still single-threaded for now.
Build completed.
MKFS [mkfs.erofs ./image//erofs-rand-frag-dedupe-1M.img -zlz4:lzma,9:lz4hc,9 -C 1048576 -Efragments,1048576 -Ededupe --random-algorithms]
mkfs.erofs 1.6-g566da452-dirty
<W> erofs: main() Line[790] EXPERIMENTAL compressed fragments feature in use. Use at your own risk!
<W> erofs: main() Line[793] EXPERIMENTAL data deduplication feature in use. Use at your own risk!
<W> erofs: z_erofs_compress_init() Line[1105] EXPERIMENTAL MicroLZMA feature in use. Use at your own risk!
<W> erofs: z_erofs_compress_init() Line[1106] Note that it may take more time since the compressor is still single-threaded for now.
Build completed.
FSCK_DUMP==================./image//erofs-lz4-4k.img=====================
Filesystem magic number: 0xE0F5E1E2
Filesystem blocks: 6
Filesystem inode metadata start block: 0
Filesystem shared xattr metadata start block: 0
Filesystem root nid: 36
Filesystem lz4_max_distance: 65535
Filesystem sb_extslots: 0
Filesystem inode count: 7
Filesystem created: Thu Jun 15 18:11:51 2023
Filesystem features: sb_csum mtime 0padding
Filesystem UUID: not available
FSCK_DUMP==================./image//erofs-lz4-rand-bigpcluster-1M.img=====================
Filesystem magic number: 0xE0F5E1E2
Filesystem blocks: 6
Filesystem inode metadata start block: 0
Filesystem shared xattr metadata start block: 0
Filesystem root nid: 37
Filesystem compr_algs: lz4, lz4hc
Filesystem sb_extslots: 0
Filesystem inode count: 7
Filesystem created: Thu Jun 15 18:11:51 2023
Filesystem features: sb_csum mtime 0padding compr_cfgs big_pcluster
Filesystem UUID: not available
FSCK_DUMP==================./image//erofs-lzma-4k.img=====================
Filesystem magic number: 0xE0F5E1E2
Filesystem blocks: 6
Filesystem inode metadata start block: 0
Filesystem shared xattr metadata start block: 0
Filesystem root nid: 37
Filesystem compr_algs: lzma
Filesystem sb_extslots: 0
Filesystem inode count: 7
Filesystem created: Thu Jun 15 18:11:50 2023
Filesystem features: sb_csum mtime 0padding compr_cfgs big_pcluster
Filesystem UUID: not available
FSCK_DUMP==================./image//erofs-lzma-bigpcluster-8k.img=====================
Filesystem magic number: 0xE0F5E1E2
Filesystem blocks: 6
Filesystem inode metadata start block: 0
Filesystem shared xattr metadata start block: 0
Filesystem root nid: 37
Filesystem compr_algs: lzma
Filesystem sb_extslots: 0
Filesystem inode count: 7
Filesystem created: Thu Jun 15 18:11:50 2023
Filesystem features: sb_csum mtime 0padding compr_cfgs big_pcluster
Filesystem UUID: not available
FSCK_DUMP==================./image//erofs-lzma-rand-bigpcluster-1M.img=====================
Filesystem magic number: 0xE0F5E1E2
Filesystem blocks: 6
Filesystem inode metadata start block: 0
Filesystem shared xattr metadata start block: 0
Filesystem root nid: 37
Filesystem compr_algs: lzma
Filesystem sb_extslots: 0
Filesystem inode count: 7
Filesystem created: Thu Jun 15 18:11:50 2023
Filesystem features: sb_csum mtime 0padding compr_cfgs big_pcluster
Filesystem UUID: not available
FSCK_DUMP==================./image//erofs-rand-alg-1M.img=====================
Filesystem magic number: 0xE0F5E1E2
Filesystem blocks: 6
Filesystem inode metadata start block: 0
Filesystem shared xattr metadata start block: 0
Filesystem root nid: 37
Filesystem compr_algs: lz4, lz4hc, lzma
Filesystem sb_extslots: 0
Filesystem inode count: 7
Filesystem created: Thu Jun 15 18:11:51 2023
Filesystem features: sb_csum mtime 0padding compr_cfgs big_pcluster
Filesystem UUID: not available
FSCK_DUMP==================./image//erofs-rand-frag-1M.img=====================
Filesystem magic number: 0xE0F5E1E2
Filesystem blocks: 2
Filesystem inode metadata start block: 0
Filesystem shared xattr metadata start block: 0
Filesystem root nid: 37
Filesystem packed nid: 71
Filesystem compr_algs: lz4, lz4hc, lzma
Filesystem sb_extslots: 0
Filesystem inode count: 8
Filesystem created: Thu Jun 15 18:11:51 2023
Filesystem features: sb_csum mtime 0padding compr_cfgs big_pcluster fragments dedupe
Filesystem UUID: not available
FSCK_DUMP==================./image//erofs-rand-frag-dedupe-1M.img=====================
Filesystem magic number: 0xE0F5E1E2
Filesystem blocks: 2
Filesystem inode metadata start block: 0
Filesystem shared xattr metadata start block: 0
Filesystem root nid: 37
Filesystem packed nid: 71
Filesystem compr_algs: lz4, lz4hc, lzma
Filesystem sb_extslots: 0
Filesystem inode count: 8
Filesystem created: Thu Jun 15 18:11:51 2023
Filesystem features: sb_csum mtime 0padding compr_cfgs big_pcluster fragments dedupe
Filesystem UUID: not available
FSCK_DUMP==================./image//erofs-raw32.img=====================
Filesystem magic number: 0xE0F5E1E2
Filesystem blocks: 17
Filesystem inode metadata start block: 0
Filesystem shared xattr metadata start block: 0
Filesystem root nid: 36
Filesystem lz4_max_distance: 65535
Filesystem sb_extslots: 0
Filesystem inode count: 7
Filesystem created: Thu Jun 15 18:11:50 2023
Filesystem features: sb_csum mtime
Filesystem UUID: not available
FSCK_DUMP==================./image//erofs-raw64.img=====================
Filesystem magic number: 0xE0F5E1E2
Filesystem blocks: 17
Filesystem inode metadata start block: 0
Filesystem shared xattr metadata start block: 0
Filesystem root nid: 36
Filesystem lz4_max_distance: 65535
Filesystem sb_extslots: 0
Filesystem inode count: 7
Filesystem created: Thu Jun 15 18:11:50 2023
Filesystem features: sb_csum mtime
Filesystem UUID: not available
erofs-lz4-4k.img erofs-lzma-4k.img_dir/ erofs-rand-alg-1M.img erofs-rand-frag-dedupe-1M.img_dir/
erofs-lz4-4k.img_dir/ erofs-lzma-bigpcluster-8k.img erofs-rand-alg-1M.img_dir/ erofs-raw32.img
erofs-lz4-rand-bigpcluster-1M.img erofs-lzma-bigpcluster-8k.img_dir/ erofs-rand-frag-1M.img erofs-raw32.img_dir/
erofs-lz4-rand-bigpcluster-1M.img_dir/ erofs-lzma-rand-bigpcluster-1M.img erofs-rand-frag-1M.img_dir/ erofs-raw64.img
erofs-lzma-4k.img erofs-lzma-rand-bigpcluster-1M.img_dir/ erofs-rand-frag-dedupe-1M.img erofs-raw64.img_dir/
Guo Xuenan (4):
erofs-utils: lib: refactor erofs compressors init
erofs-utils: lib: unify all identical compressor print function
erofs-utils: simplify erofs compressor init and exit
erofs-utils: dump: add some superblock fields
dump/main.c | 13 +++
fsck/main.c | 17 +---
include/erofs/compress.h | 7 +-
include/erofs/internal.h | 1 +
lib/compress.c | 87 ++++++++------------
lib/compressor.c | 173 ++++++++++++++++++++++++++++-----------
lib/compressor.h | 23 ++++--
lib/compressor_liblzma.c | 4 -
lib/compressor_lz4.c | 2 -
lib/compressor_lz4hc.c | 3 -
lib/super.c | 5 ++
mkfs/main.c | 18 +---
12 files changed, 207 insertions(+), 146 deletions(-)
--
2.31.1
More information about the Linux-erofs
mailing list