[RFC PATCH v3 0/3] erofs-utils: compressed fragments feature

[Gao Xiang]

Hi Yue,

On Wed, Aug 03, 2022 at 11:51:27AM +0800, Yue Hu wrote:
> In order to achieve greater compression ratio, let's introduce
> compressed fragments feature which can merge tail of per-file or the
> whole files into one special inode to reach the target.
> 
> And we can also set pcluster size to fragments inode for different
> compression requirments.
> 
> In this patchset, we also improve the uncompressed data layout of
> compressed files. Just write it from 'clusterofs' instead of 0 since it
> can benefit from in-place I/O. For now, it only goes with fragments.
> 
> The main idea above is from Xiang.

Thanks for your hard work! I will take a deep try this weekend,

Also I'd like to enable logical cluster size != 4k for big pcluster with
large pclustersize in order to reduce the size of compression indexes.

In such cases, I think compact indexes are unnecessary.  I think it's
already supported on the kernel side, so we just need to implement the
userspace side.

Thanks,
Gao Xiang

> 
> Here is some test data of Linux 5.10.87 source code under Ubuntu 18.04:
> 
> linux-5.10.87 (erofs, uncompressed)                1.1G
> 
> linux-5.10.87 (erofs, lz4hc,12 4k fragments,4k)    301M
> linux-5.10.87 (erofs, lz4hc,12 8k fragments,8k)    268M
> linux-5.10.87 (erofs, lz4hc,12 16k fragments,16k)  242M
> linux-5.10.87 (erofs, lz4hc,12 32k fragments,32k)  225M
> linux-5.10.87 (erofs, lz4hc,12 64k fragments,64k)  217M
> 
> linux-5.10.87 (erofs, lz4hc,12 4k vanilla)         396M
> linux-5.10.87 (erofs, lz4hc,12 8k vanilla)         376M
> linux-5.10.87 (erofs, lz4hc,12 16k vanilla)        364M
> linux-5.10.87 (erofs, lz4hc,12 32k vanilla)        359M
> linux-5.10.87 (erofs, lz4hc,12 64k vanilla)        358M
> 
> Usage:
> mkfs.erofs -zlz4hc,12 -C65536 -Efragments,65536 foo.erofs.img foo/
> 
> Changes since v2:
>  - mainly reimplment the decompression logic for fragment inode due to
>    kernel side;
>  - fix compatibility issue to old image with ztailpacking feature;
>  - move code of super.c in patch 3/3 to patch 1/3;
>  - minor naming change.
> 
> Changes since v1:
>  - mainly optimize index space for fragment inode;
>  - add merging tail with len <= pclustersize into fragments directly;
>  - use a inode instead of nid to avoid multiple load fragments;
>  - fix memory leak of building fragments;
>  - minor change to diff special fragments with normal inode.
>  - rebase to commit cb058526 with patch [1];
>  - code cleanup.
> 
> Note that inode will be extended version (64 bytes) due to mtime, may
> use 'force-inode-compact' option to reduce the size if mtime careless.
> 
> [1] https://lore.kernel.org/linux-erofs/20220722053610.23912-1-huyue2@coolpad.com/
> 
> Yue Hu (3):
>   erofs-utils: lib: add support for fragments data decompression
>   erofs-utils: lib: support on-disk offset for shifted decompression
>   erofs-utils: introduce compressed fragments support
> 
>  include/erofs/compress.h   |   3 +-
>  include/erofs/config.h     |   3 +-
>  include/erofs/decompress.h |   3 ++
>  include/erofs/fragments.h  |  25 +++++++++
>  include/erofs/inode.h      |   2 +
>  include/erofs/internal.h   |   9 ++++
>  include/erofs_fs.h         |  27 +++++++---
>  lib/Makefile.am            |   4 +-
>  lib/compress.c             | 108 +++++++++++++++++++++++++++----------
>  lib/data.c                 |  28 +++++++++-
>  lib/decompress.c           |  10 +++-
>  lib/fragments.c            |  76 ++++++++++++++++++++++++++
>  lib/inode.c                |  43 ++++++++++-----
>  lib/super.c                |  24 ++++++++-
>  lib/zmap.c                 |  26 +++++++++
>  mkfs/main.c                |  64 +++++++++++++++++++---
>  16 files changed, 393 insertions(+), 62 deletions(-)
>  create mode 100644 include/erofs/fragments.h
>  create mode 100644 lib/fragments.c
> 
> -- 
> 2.17.1