[PATCH v3] erofs-utils: introduce segment size to limit the max input stream

[Gao Xiang]

Hi Guifu,

On Thu, Jun 11, 2020 at 01:17:28AM +0800, Li Guifu via Linux-erofs wrote:
> Limit the max input stream size by adding segment compression
> (e.g. 4M segment size), it will benefits:
>  - more friendly to block diff (and more details about this);
>  - it can also be used for parallel compression in the same file.

Subject: erofs-utils: introduce segment compression

Support segment compression which seperates files in several logic
units (segments) and each segment is compressed independently.

Advantages:
 - more friendly for data differencing;
 - it can also be used for parallel compression in the same file later.

> 
> Signed-off-by: Li Guifu <bluce.lee at aliyun.com>
> ---
>  include/erofs/config.h |  1 +
>  lib/compress.c         | 11 +++++++++++
>  lib/config.c           |  1 +
>  3 files changed, 13 insertions(+)
> 
> diff --git a/include/erofs/config.h b/include/erofs/config.h
> index 2f09749..9125c1e 100644
> --- a/include/erofs/config.h
> +++ b/include/erofs/config.h
> @@ -36,6 +36,7 @@ struct erofs_configure {
>  	char *c_src_path;
>  	char *c_compr_alg_master;
>  	int c_compr_level_master;
> +	unsigned int c_compr_seg_size;	/* max segment compress size */

Add a command line argument "-S segment size" for this, and
update the manpage as well.

>  	int c_force_inodeversion;
>  	/* < 0, xattr disabled and INT_MAX, always use inline xattrs */
>  	int c_inline_xattr_tolerance;
> diff --git a/lib/compress.c b/lib/compress.c
> index 6cc68ed..8fdbfb2 100644
> --- a/lib/compress.c
> +++ b/lib/compress.c
> @@ -32,6 +32,8 @@ struct z_erofs_vle_compress_ctx {
>  
>  	erofs_blk_t blkaddr;	/* pointing to the next blkaddr */
>  	u16 clusterofs;
> +	unsigned int comprlimits;
> +	unsigned int comr_seg_size;

Let's update the whole logic as what we discussed on phone just now.

Thanks,
Gao Xiang