[RFC PATCH v5 0/4] erofs-utils: compressed fragments feature

[Yue Hu]

From: Yue Hu <huyue2 at coolpad.com>

In order to achieve greater compression ratio, let's introduce
compressed fragments feature which can merge tail of per-file or the
whole files into one special inode to reach the target.

And we can also set pcluster size to fragments inode for different
compression requirments.

In this patchset, we also improve the uncompressed data layout of
compressed files. Just write it from 'clusterofs' instead of 0 since it
can benefit from in-place I/O. For now, it only goes with fragments.

The main idea above is from Xiang.

Here is some test data of Linux 5.10.87 source code under Ubuntu 18.04:

linux-5.10.87 (erofs, uncompressed)                1.1G

linux-5.10.87 (erofs, lz4hc,12 4k fragments,4k)    301M
linux-5.10.87 (erofs, lz4hc,12 8k fragments,8k)    268M
linux-5.10.87 (erofs, lz4hc,12 16k fragments,16k)  242M
linux-5.10.87 (erofs, lz4hc,12 32k fragments,32k)  225M
linux-5.10.87 (erofs, lz4hc,12 64k fragments,64k)  217M

linux-5.10.87 (erofs, lz4hc,12 4k vanilla)         396M
linux-5.10.87 (erofs, lz4hc,12 8k vanilla)         376M
linux-5.10.87 (erofs, lz4hc,12 16k vanilla)        364M
linux-5.10.87 (erofs, lz4hc,12 32k vanilla)        359M
linux-5.10.87 (erofs, lz4hc,12 64k vanilla)        358M

Usage:
mkfs.erofs -zlz4hc,12 -C65536 -Efragments,65536 foo.erofs.img foo/

Changes since v4:
 - naming on-disk offset as interlaced offset and add a advise to hint the data
   layout (Gao Xiang);
 - improve the packed inode pcluster size checking condition (Gao Xiang);
 - add field is_packed_inode instead of using is_src parameter passed (Gao Xiang);
 - per-inode eof bug_on (Gao Xiang);
 - extract the interlaced data layout part as a seprate patch;
 - change the advise bit of fragment feature.
 - re-implement the function to read interlaced data;
 - update compresstion ratio check;

Changes since v3:
 - fuse minor change and modify fragments naming related suggested by Xiang;
 - refator fragments build code;
 - rebase to latest commit 547bea3cb71a.

Changes since v2:
 - mainly reimplment the decompression logic for fragment inode due to
   kernel side;
 - fix compatibility issue to old image with ztailpacking feature;
 - move code of super.c in patch 3/3 to patch 1/3;
 - minor naming change.

Changes since v1:
 - mainly optimize index space for fragment inode;
 - add merging tail with len <= pclustersize into fragments directly;
 - use a inode instead of nid to avoid multiple load fragments;
 - fix memory leak of building fragments;
 - minor change to diff special fragments with normal inode.
 - rebase to commit cb058526 with patch [1];
 - code cleanup.

Note that inode will be extended version (64 bytes) due to mtime, may
use 'force-inode-compact' option to reduce the size if mtime careless.

[1] https://lore.kernel.org/linux-erofs/20220722053610.23912-1-huyue2@coolpad.com/

Yue Hu (4):
  erofs-utils: fuse: support interlaced uncompressed data for compressed
    files
  erofs-utils: lib: support fragments data decompression
  erofs-utils: lib: support interlaced uncompressed data layout when
    compressing
  erofs-utils: mkfs: introduce compressed fragments support

 include/erofs/compress.h   |  2 +-
 include/erofs/config.h     |  3 +-
 include/erofs/decompress.h |  3 ++
 include/erofs/fragments.h  | 25 +++++++++++
 include/erofs/inode.h      |  1 +
 include/erofs/internal.h   |  8 ++++
 include/erofs_fs.h         | 28 +++++++++---
 lib/Makefile.am            |  4 +-
 lib/compress.c             | 92 ++++++++++++++++++++++++++++----------
 lib/data.c                 | 28 +++++++++++-
 lib/decompress.c           | 16 ++++++-
 lib/fragments.c            | 58 ++++++++++++++++++++++++
 lib/inode.c                | 47 ++++++++++++++++---
 lib/super.c                | 24 +++++++++-
 lib/zmap.c                 | 26 +++++++++++
 mkfs/main.c                | 61 ++++++++++++++++++++++---
 16 files changed, 379 insertions(+), 47 deletions(-)
 create mode 100644 include/erofs/fragments.h
 create mode 100644 lib/fragments.c

-- 
2.17.1