[PATCH v2 4/4] erofs-utils: mkfs: support EROFS index-only image generation from S3

[zhaoyifan (H)]

On 2025/8/1 15:52, Gao Xiang wrote:
>
>
> On 2025/8/1 15:30, Yifan Zhao wrote:
>> From: zhaoyifan <zhaoyifan28 at huawei.com>
>>
>> This patch introduces experimental S3 support for mkfs.erofs, 
>> allowing EROFS
>> images to be generated from AWS S3 (and other S3 API compatible 
>> services).
>>
>> Currently the functionality is limited:
>> - only index-only EROFS image generation are supported
>> - only AWS Signature Version 2 is supported
>> - only S3 object name and size are respected during image generation
>>
>> Signed-off-by: Yifan Zhao <zhaoyifan28 at huawei.com>
>> ---
>> change since v1:
>> - get rid of erofs_init_empty_dir() as described in commit cea8581
>>
>>   lib/liberofs_s3.h |   2 +
>>   lib/remotes/s3.c  | 605 +++++++++++++++++++++++++++++++++++++++++++++-
>>   mkfs/main.c       |   5 +-
>>   3 files changed, 609 insertions(+), 3 deletions(-)
>>
>> diff --git a/lib/liberofs_s3.h b/lib/liberofs_s3.h
>> index 16a06c9..a975fbb 100644
>> --- a/lib/liberofs_s3.h
>> +++ b/lib/liberofs_s3.h
>> @@ -33,6 +33,8 @@ struct erofs_s3 {
>>       enum s3erofs_signature_version sig;
>>   };
>>   +int s3erofs_build_trees(struct erofs_inode *root, struct erofs_s3 
>> *s3cfg);
>> +
>>   #ifdef __cplusplus
>>   }
>>   #endif
>> diff --git a/lib/remotes/s3.c b/lib/remotes/s3.c
>> index 358ee91..a062a50 100644
>> --- a/lib/remotes/s3.c
>> +++ b/lib/remotes/s3.c
>> @@ -4,4 +4,607 @@
>>    *             http://www.huawei.com/
>>    * Created by Yifan Zhao <zhaoyifan28 at huawei.com>
>>    */
>> -#include "liberofs_s3.h"
>> \ No newline at end of file
>
> "\ No newline at end of file"
>
> Let's check the editor first, in principle the patches
> shouldn't have this.
>
I forget to add a newline in newly added files. Fix it in v3.
>> ...
>> +
>> +static struct s3erofs_object_info *
>> +s3erofs_get_next_object(struct s3erofs_object_iterator *it)
>> +{
>> +    int ret = 0;
>> +
>> +    if (it->cur < it->total) {
>> +        return &it->objects[it->cur++];
>> +    }
>> +
>> +    if (it->is_truncated) {
>> +        ret = s3erofs_list_objects(it);
>> +        if (ret < 0)
>> +            return ERR_PTR(ret);
>> +
>> +        it->cur = 0;
>> +        return &it->objects[it->cur++];
>> +    }
>> +
>> +    return NULL;
>> +}
>> +
>> +static int s3erofs_global_init(void)
>> +{
>> +    int ret;
>
> Does it support multiple remotes?
>
> Use a mutex to protect this for multiple instances?

For now we only support one remote and there is no race issues here.

Could we add the mutex in the following patches, together with multi

remotes/multithreading support?

>
>> +
>> +    ret = curl_global_init(CURL_GLOBAL_DEFAULT);
>> +    if (ret != CURLE_OK)
>> +        return -EIO;
>> +
>> +    easy_curl = curl_easy_init();
>> +    if (!easy_curl) {
>> +        curl_global_cleanup();
>> +        return -EIO;
>> +    }
>> +
>> +    curl_easy_setopt(easy_curl, CURLOPT_WRITEFUNCTION, 
>> s3erofs_request_write_memory_cb);
>> +    curl_easy_setopt(easy_curl, CURLOPT_FOLLOWLOCATION, 1L);
>> +    curl_easy_setopt(easy_curl, CURLOPT_TIMEOUT, 30L);
>> +
>> +    xmlInitParser();
>> +
>> +    return ret;
>> +}
>> +
>> +static void s3erofs_global_exit(void)
>> +{
>> +    if (!easy_curl)
>> +        return;
>> +
>> +    xmlCleanupParser();
>> +
>> +    curl_easy_cleanup(easy_curl);
>> +    easy_curl = NULL;
>> +
>> +    curl_global_cleanup();
>> +}
>> +
>> +int s3erofs_build_trees(struct erofs_inode *root, struct erofs_s3 
>> *s3cfg)
>> +{
>> +    struct erofs_sb_info *sbi = root->sbi;
>> +    struct s3erofs_object_iterator *iter;
>> +    struct s3erofs_object_info *obj;
>> +    struct erofs_dentry *d;
>> +    struct erofs_inode *inode;
>> +    struct stat st;
>> +    bool dumb;
>> +    int ret = 0;
>> +
>> +    st.st_uid = getuid();
>> +    st.st_gid = getgid();
>> +
>> +    /* XXX */
>> +    st.st_mtime = sbi->epoch;
>
>
>     st.st_mtime = sbi->epoch + sbi->build_time?
>
> build_time can be specified by users too.

Fixed in v3.


Thanks,

Yifan Zhao

>
> Thanks,
> Gao Xiang