[PATCH 19/26] netfs: New writeback implementation

Vadim Fedorenko vadim.fedorenko at linux.dev
Sat Mar 30 12:06:09 AEDT 2024


On 29/03/2024 10:34, Naveen Mamindlapalli wrote:
>> -----Original Message-----
>> From: David Howells <dhowells at redhat.com>
>> Sent: Thursday, March 28, 2024 10:04 PM
>> To: Christian Brauner <christian at brauner.io>; Jeff Layton <jlayton at kernel.org>;
>> Gao Xiang <hsiangkao at linux.alibaba.com>; Dominique Martinet
>> <asmadeus at codewreck.org>
>> Cc: David Howells <dhowells at redhat.com>; Matthew Wilcox
>> <willy at infradead.org>; Steve French <smfrench at gmail.com>; Marc Dionne
>> <marc.dionne at auristor.com>; Paulo Alcantara <pc at manguebit.com>; Shyam
>> Prasad N <sprasad at microsoft.com>; Tom Talpey <tom at talpey.com>; Eric Van
>> Hensbergen <ericvh at kernel.org>; Ilya Dryomov <idryomov at gmail.com>;
>> netfs at lists.linux.dev; linux-cachefs at redhat.com; linux-afs at lists.infradead.org;
>> linux-cifs at vger.kernel.org; linux-nfs at vger.kernel.org; ceph-
>> devel at vger.kernel.org; v9fs at lists.linux.dev; linux-erofs at lists.ozlabs.org; linux-
>> fsdevel at vger.kernel.org; linux-mm at kvack.org; netdev at vger.kernel.org; linux-
>> kernel at vger.kernel.org; Latchesar Ionkov <lucho at ionkov.net>; Christian
>> Schoenebeck <linux_oss at crudebyte.com>
>> Subject: [PATCH 19/26] netfs: New writeback implementation
>>
>> The current netfslib writeback implementation creates writeback requests of
>> contiguous folio data and then separately tiles subrequests over the space
>> twice, once for the server and once for the cache.  This creates a few
>> issues:
>>
>>   (1) Every time there's a discontiguity or a change between writing to only
>>       one destination or writing to both, it must create a new request.
>>       This makes it harder to do vectored writes.
>>
>>   (2) The folios don't have the writeback mark removed until the end of the
>>       request - and a request could be hundreds of megabytes.
>>
>>   (3) In future, I want to support a larger cache granularity, which will
>>       require aggregation of some folios that contain unmodified data (which
>>       only need to go to the cache) and some which contain modifications
>>       (which need to be uploaded and stored to the cache) - but, currently,
>>       these are treated as discontiguous.
>>
>> There's also a move to get everyone to use writeback_iter() to extract
>> writable folios from the pagecache.  That said, currently writeback_iter()
>> has some issues that make it less than ideal:
>>
>>   (1) there's no way to cancel the iteration, even if you find a "temporary"
>>       error that means the current folio and all subsequent folios are going
>>       to fail;
>>
>>   (2) there's no way to filter the folios being written back - something
>>       that will impact Ceph with it's ordered snap system;
>>
>>   (3) and if you get a folio you can't immediately deal with (say you need
>>       to flush the preceding writes), you are left with a folio hanging in
>>       the locked state for the duration, when really we should unlock it and
>>       relock it later.
>>
>> In this new implementation, I use writeback_iter() to pump folios,
>> progressively creating two parallel, but separate streams and cleaning up
>> the finished folios as the subrequests complete.  Either or both streams
>> can contain gaps, and the subrequests in each stream can be of variable
>> size, don't need to align with each other and don't need to align with the
>> folios.
>>
>> Indeed, subrequests can cross folio boundaries, may cover several folios or
>> a folio may be spanned by multiple folios, e.g.:
>>
>>           +---+---+-----+-----+---+----------+
>> Folios:  |   |   |     |     |   |          |
>>           +---+---+-----+-----+---+----------+
>>
>>             +------+------+     +----+----+
>> Upload:    |      |      |.....|    |    |
>>             +------+------+     +----+----+
>>
>>           +------+------+------+------+------+
>> Cache:   |      |      |      |      |      |
>>           +------+------+------+------+------+
>>
>> The progressive subrequest construction permits the algorithm to be
>> preparing both the next upload to the server and the next write to the
>> cache whilst the previous ones are already in progress.  Throttling can be
>> applied to control the rate of production of subrequests - and, in any
>> case, we probably want to write them to the server in ascending order,
>> particularly if the file will be extended.
>>
>> Content crypto can also be prepared at the same time as the subrequests and
>> run asynchronously, with the prepped requests being stalled until the
>> crypto catches up with them.  This might also be useful for transport
>> crypto, but that happens at a lower layer, so probably would be harder to
>> pull off.
>>
>> The algorithm is split into three parts:
>>
>>   (1) The issuer.  This walks through the data, packaging it up, encrypting
>>       it and creating subrequests.  The part of this that generates
>>       subrequests only deals with file positions and spans and so is usable
>>       for DIO/unbuffered writes as well as buffered writes.
>>
>>   (2) The collector. This asynchronously collects completed subrequests,
>>       unlocks folios, frees crypto buffers and performs any retries.  This
>>       runs in a work queue so that the issuer can return to the caller for
>>       writeback (so that the VM can have its kswapd thread back) or async
>>       writes.
>>
>>   (3) The retryer.  This pauses the issuer, waits for all outstanding
>>       subrequests to complete and then goes through the failed subrequests
>>       to reissue them.  This may involve reprepping them (with cifs, the
>>       credits must be renegotiated, and a subrequest may need splitting),
>>       and doing RMW for content crypto if there's a conflicting change on
>>       the server.
>>
>> [!] Note that some of the functions are prefixed with "new_" to avoid
>> clashes with existing functions.  These will be renamed in a later patch
>> that cuts over to the new algorithm.
>>
>> Signed-off-by: David Howells <dhowells at redhat.com>
>> cc: Jeff Layton <jlayton at kernel.org>
>> cc: Eric Van Hensbergen <ericvh at kernel.org>
>> cc: Latchesar Ionkov <lucho at ionkov.net>
>> cc: Dominique Martinet <asmadeus at codewreck.org>
>> cc: Christian Schoenebeck <linux_oss at crudebyte.com>
>> cc: Marc Dionne <marc.dionne at auristor.com>
>> cc: v9fs at lists.linux.dev
>> cc: linux-afs at lists.infradead.org
>> cc: netfs at lists.linux.dev
>> cc: linux-fsdevel at vger.kernel.org

[..snip..]

>> +/*
>> + * Begin a write operation for writing through the pagecache.
>> + */
>> +struct netfs_io_request *new_netfs_begin_writethrough(struct kiocb *iocb, size_t
>> len)
>> +{
>> +	struct netfs_io_request *wreq = NULL;
>> +	struct netfs_inode *ictx = netfs_inode(file_inode(iocb->ki_filp));
>> +
>> +	mutex_lock(&ictx->wb_lock);
>> +
>> +	wreq = netfs_create_write_req(iocb->ki_filp->f_mapping, iocb->ki_filp,
>> +				      iocb->ki_pos, NETFS_WRITETHROUGH);
>> +	if (IS_ERR(wreq))
>> +		mutex_unlock(&ictx->wb_lock);
>> +
>> +	wreq->io_streams[0].avail = true;
>> +	trace_netfs_write(wreq, netfs_write_trace_writethrough);
> 
> Missing mutex_unlock() before return.
> 

mutex_unlock() happens in new_netfs_end_writethrough()

> Thanks,
> Naveen
> 


X-sender: <netdev+bounces-83486-peter.schumann=secunet.com at vger.kernel.org>
X-Receiver: <peter.schumann at secunet.com> ORCPT=rfc822;peter.schumann at secunet.com NOTIFY=NEVER; X-ExtendedProps=BQAMAAIAAAUAWAAXAEgAAACdOWm+FoEIR7KnWe1lIx8pQ049U2NodW1hbm4gUGV0ZXIsT1U9VXNlcnMsT1U9TWlncmF0aW9uLERDPXNlY3VuZXQsREM9ZGUFAGwAAgAADwA2AAAATWljcm9zb2Z0LkV4Y2hhbmdlLlRyYW5zcG9ydC5NYWlsUmVjaXBpZW50LkRpc3BsYXlOYW1lDwAPAAAAU2NodW1hbm4sIFBldGVyBQA8AAIAAAUAHQAPAAwAAABtYngtZXNzZW4tMDEFAA4AEQAuyVP5XtO9RYbNJlr9VbVbBQALABcAvgAAAEOSGd+Q7QVIkVZ3ffGxE8RDTj1EQjQsQ049RGF0YWJhc2VzLENOPUV4Y2hhbmdlIEFkbWluaXN0cmF0aXZlIEdyb3VwIChGWURJQk9IRjIzU1BETFQpLENOPUFkbWluaXN0cmF0aXZlIEdyb3VwcyxDTj1zZWN1bmV0LENOPU1pY3Jvc29mdCBFeGNoYW5nZSxDTj1TZXJ2aWNlcyxDTj1Db25maWd1cmF0aW9uLERDPXNlY3VuZXQsREM9ZGUFABIADwBgAAAAL289c2VjdW5ldC9vdT1FeGNoYW5nZSBBZG1pbmlzdHJhdGl2ZSBHcm91cCAoRllESUJPSEYyM1NQRExUKS9jbj1SZWNpcGllbnRzL2NuPVBldGVyIFNjaHVtYW5uNWU3BQBHAAIAAAUARgAHAAMAAAAFAEMAAgAABQAWAAIAAAUAagAJAAEAAAAAAAAABQAVABYAAgAAAA8ANQAAAE1pY3Jvc29mdC5FeGNoYW5nZS5UcmFuc3BvcnQuRGlyZWN0b3J5RGF0YS5Jc1Jlc291cmNlAgAABQAUABEAnTlpvhaBCEeyp1ntZSMfKQUAIwACAAEFACIADwAxAAAAQXV0b1Jlc3BvbnNlU3VwcHJlc3M6IDANClRyYW5zbWl0SGlzdG9yeTogRmFsc2UNCg8ALwAAAE1pY3Jvc29mdC5FeGNoYW5nZS5UcmFuc3BvcnQuRXhwYW5zaW9uR3JvdXBUeXBlDwAVAAAATWVtYmVyc0dyb3VwRXhwYW5zaW9uBQAmAAIAAQ==
X-CreatedBy: MSExchange15
X-HeloDomain: a.mx.secunet.com
X-ExtendedProps: BQBjAAoA3kymlidQ3AgFAGEACAABAAAABQA3AAIAAA8APAAAAE1pY3Jvc29mdC5FeGNoYW5nZS5UcmFuc3BvcnQuTWFpbFJlY2lwaWVudC5Pcmdhbml6YXRpb25TY29wZREAAAAAAAAAAAAAAAAAAAAAAAUASQACAAEFAAQAFCABAAAAGgAAAHBldGVyLnNjaHVtYW5uQHNlY3VuZXQuY29tBQAGAAIAAQ8AKgAAAE1pY3Jvc29mdC5FeGNoYW5nZS5UcmFuc3BvcnQuUmVzdWJtaXRDb3VudAcAAwAAAA8ACQAAAENJQXVkaXRlZAIAAQUAAgAHAAEAAAAFAAMABwAAAAAABQAFAAIAAQUAYgAKADEAAADQigAABQBkAA8AAwAAAEh1YgUAKQACAAEPAD8AAABNaWNyb3NvZnQuRXhjaGFuZ2UuVHJhbnNwb3J0LkRpcmVjdG9yeURhdGEuTWFpbERlbGl2ZXJ5UHJpb3JpdHkPAAMAAABMb3c=
X-Source: SMTP:Default MBX-ESSEN-02
X-SourceIPAddress: 62.96.220.36
X-EndOfInjectedXHeaders: 28032
Received: from cas-essen-02.secunet.de (10.53.40.202) by
 mbx-essen-02.secunet.de (10.53.40.198) with Microsoft SMTP Server
 (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id
 15.1.2507.37; Sat, 30 Mar 2024 02:06:35 +0100
Received: from a.mx.secunet.com (62.96.220.36) by cas-essen-02.secunet.de
 (10.53.40.202) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35 via Frontend
 Transport; Sat, 30 Mar 2024 02:06:35 +0100
Received: from localhost (localhost [127.0.0.1])
	by a.mx.secunet.com (Postfix) with ESMTP id 3EE60201E5
	for <peter.schumann at secunet.com>; Sat, 30 Mar 2024 02:06:35 +0100 (CET)
X-Virus-Scanned: by secunet
X-Spam-Flag: NO
X-Spam-Score: -2.751
X-Spam-Level:
X-Spam-Status: No, score=-2.751 tagged_above=-999 required=2.1
	tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1,
	DKIM_VALID_AU=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249,
	MAILING_LIST_MULTI=-1, RCVD_IN_DNSWL_NONE=-0.0001,
	SPF_HELO_NONE=0.001, SPF_PASS=-0.001]
	autolearn=unavailable autolearn_force=no
Authentication-Results: a.mx.secunet.com (amavisd-new);
	dkim=pass (1024-bit key) header.d=linux.dev
Received: from a.mx.secunet.com ([127.0.0.1])
	by localhost (a.mx.secunet.com [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id 6tvewKec8aDo for <peter.schumann at secunet.com>;
	Sat, 30 Mar 2024 02:06:31 +0100 (CET)
Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=147.75.48.161; helo=sy.mirrors.kernel.org; envelope-from=netdev+bounces-83486-peter.schumann=secunet.com at vger.kernel.org; receiver=peter.schumann at secunet.com 
DKIM-Filter: OpenDKIM Filter v2.11.0 a.mx.secunet.com 617F120519
Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org [147.75.48.161])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by a.mx.secunet.com (Postfix) with ESMTPS id 617F120519
	for <peter.schumann at secunet.com>; Sat, 30 Mar 2024 02:06:31 +0100 (CET)
Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by sy.mirrors.kernel.org (Postfix) with ESMTPS id 2491AB2154C
	for <peter.schumann at secunet.com>; Sat, 30 Mar 2024 01:06:27 +0000 (UTC)
Received: from localhost.localdomain (localhost.localdomain [127.0.0.1])
	by smtp.subspace.kernel.org (Postfix) with ESMTP id 7E78810FF;
	Sat, 30 Mar 2024 01:06:22 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="VugHsG2Y"
X-Original-To: netdev at vger.kernel.org
Received: from out-173.mta0.migadu.com (out-173.mta0.migadu.com [91.218.175.173])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id 762E77E8
	for <netdev at vger.kernel.org>; Sat, 30 Mar 2024 01:06:19 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.173
ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116;
	t=1711760782; cv=none; b=dASfLrrkRxmD6WmYvcvyTFgLXAgqW4qcP8FwVw/FT8ajSayU1k2jNzB6oEhlAk4YxWiFWUStYosH2VKaROBs5wKHQh4Rsxe59gs4L4KuJN+VlHKDa1iIm9ShtgGS6jAthHnsiMpAE+me1GueQZILnQSEjyu5ZoBpE9mg1Ojzukk=
ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org;
	s=arc-20240116; t=1711760782; c=relaxed/simple;
	bh=0Tab6a8G72hAdQQTifWJdxmg84+y/tA9VvkWPFI9VQA=;
	h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From:
	 In-Reply-To:Content-Type; b=J9cTiVFLYlA2DR+fBNIsoiV/11LbFxExG+qAmCsON2fksIZjZEAFWqHQx2zJk8Dqn3t/Quqw4LH8Yjb2qqlthM0L82RcciykTG9EQ9SPWlqiRoPPhuerZSz/amNX1IgyImsufFdXk4+oiQpzCA0LsWzVgTdTI9x4oenmDhjahZI=
ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=VugHsG2Y; arc=none smtp.client-ip=91.218.175.173
Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev
Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev
Message-ID: <08dd01e3-c45e-47d9-bcde-55f7d1edc480 at linux.dev>
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1;
	t=1711760777;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:cc:mime-version:mime-version:content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding:
	 in-reply-to:in-reply-to:references:references;
	bh=BFgFwDTUylEnErMgQV0Ufr9/Ufnl/0omKSnqywixpVg=;
	b=VugHsG2Y5vYBP20CvcOMmGaEI/5A/PvLnTsQTtTA0ZebTuac4nORyH8iRZe/CwFs5RLRhJ
	4Ih/prbwbd8/OSA0Wv9Z9Z9JdeLOJUf8/vLW1xeGCG/2qNeI4CXYcIw3EixotT7o6oviEg
	ZM4gfY/Y4bUjm5TsY8pyZBWQLZ0Jv74=
Date: Fri, 29 Mar 2024 18:06:09 -0700
Precedence: bulk
X-Mailing-List: netdev at vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe at vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe at vger.kernel.org>
MIME-Version: 1.0
Subject: Re: [PATCH 19/26] netfs: New writeback implementation
Content-Language: en-US
To: Naveen Mamindlapalli <naveenm at marvell.com>,
 David Howells <dhowells at redhat.com>, Christian Brauner
 <christian at brauner.io>, Jeff Layton <jlayton at kernel.org>,
 Gao Xiang <hsiangkao at linux.alibaba.com>,
 Dominique Martinet <asmadeus at codewreck.org>
Cc: Matthew Wilcox <willy at infradead.org>, Steve French <smfrench at gmail.com>,
 Marc Dionne <marc.dionne at auristor.com>, Paulo Alcantara <pc at manguebit.com>,
 Shyam Prasad N <sprasad at microsoft.com>, Tom Talpey <tom at talpey.com>,
 Eric Van Hensbergen <ericvh at kernel.org>, Ilya Dryomov <idryomov at gmail.com>,
 "netfs at lists.linux.dev" <netfs at lists.linux.dev>,
 "linux-cachefs at redhat.com" <linux-cachefs at redhat.com>,
 "linux-afs at lists.infradead.org" <linux-afs at lists.infradead.org>,
 "linux-cifs at vger.kernel.org" <linux-cifs at vger.kernel.org>,
 "linux-nfs at vger.kernel.org" <linux-nfs at vger.kernel.org>,
 "ceph-devel at vger.kernel.org" <ceph-devel at vger.kernel.org>,
 "v9fs at lists.linux.dev" <v9fs at lists.linux.dev>,
 "linux-erofs at lists.ozlabs.org" <linux-erofs at lists.ozlabs.org>,
 "linux-fsdevel at vger.kernel.org" <linux-fsdevel at vger.kernel.org>,
 "linux-mm at kvack.org" <linux-mm at kvack.org>,
 "netdev at vger.kernel.org" <netdev at vger.kernel.org>,
 "linux-kernel at vger.kernel.org" <linux-kernel at vger.kernel.org>,
 Latchesar Ionkov <lucho at ionkov.net>,
 Christian Schoenebeck <linux_oss at crudebyte.com>
References: <20240328163424.2781320-1-dhowells at redhat.com>
 <20240328163424.2781320-20-dhowells at redhat.com>
 <SJ2PR18MB5635A86C024316BC5E57B79EA23A2 at SJ2PR18MB5635.namprd18.prod.outlook.com>
X-Report-Abuse: Please report any abuse attempt to abuse at migadu.com and include these headers.
From: Vadim Fedorenko <vadim.fedorenko at linux.dev>
In-Reply-To: <SJ2PR18MB5635A86C024316BC5E57B79EA23A2 at SJ2PR18MB5635.namprd18.prod.outlook.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Migadu-Flow: FLOW_OUT
Return-Path: netdev+bounces-83486-peter.schumann=secunet.com at vger.kernel.org
X-MS-Exchange-Organization-OriginalArrivalTime: 30 Mar 2024 01:06:35.2990
 (UTC)
X-MS-Exchange-Organization-Network-Message-Id: 407005da-cf0f-43eb-e721-08dc5055a526
X-MS-Exchange-Organization-OriginalClientIPAddress: 62.96.220.36
X-MS-Exchange-Organization-OriginalServerIPAddress: 10.53.40.202
X-MS-Exchange-Organization-Cross-Premises-Headers-Processed: cas-essen-02.secunet.de
X-MS-Exchange-Organization-OrderedPrecisionLatencyInProgress: LSRV=mbx-essen-02.secunet.de:TOTAL-HUB=25969.715|SMR=0.330(SMRDE=0.005|SMRC=0.324(SMRCL=0.103|X-SMRCR=0.323))|CAT=0.079(CATOS=0.001
 |CATRESL=0.033(CATRESLP2R=0.026)|CATORES=0.043(CATRS=0.043(CATRS-Index
 Routing Agent=0.041
 )))|QDM=5760.243|SMSC=0.016|SMS=2.529(SMSMBXD-INC=2.522)|UNK=0.001|QDM=5821.174|SMSC=0.014
 |SMS=3.274(SMSMBXD-INC=3.267)|QDM=8272.090|SMSC=0.022|SMS=4.740(SMSMBXD-INC=4.719
 )|QDM=6095.229|UNK=0.001|CAT=0.027(CATRESL=0.026(CATRESLP2R=0.012))|QDM=5.054|UNK=0.004
 |CAT=0.022(CATRESL=0.020(CATRESLP2R=0.009))|QDM=5.047|UNK=0.004|CAT=0.005(CATRESL=0.004
 (CATRESLP2R=0.002));2024-03-30T08:19:25.041Z
X-MS-Exchange-Forest-ArrivalHubServer: mbx-essen-02.secunet.de
X-MS-Exchange-Organization-AuthSource: cas-essen-02.secunet.de
X-MS-Exchange-Organization-AuthAs: Anonymous
X-MS-Exchange-Organization-FromEntityHeader: Internet
X-MS-Exchange-Organization-OriginalSize: 15012
X-MS-Exchange-Organization-HygienePolicy: Standard
X-MS-Exchange-Organization-MessageLatency: SRV=cas-essen-02.secunet.de:TOTAL-FE=0.025|SMR=0.022(SMRPI=0.020(SMRPI-FrontendProxyAgent=0.020))|SMS=0.002
X-MS-Exchange-Organization-Recipient-Limit-Verified: True
X-MS-Exchange-Organization-TotalRecipientCount: 1
X-MS-Exchange-Organization-Rules-Execution-History: 0b0cf904-14ac-4724-8bdf-482ee6223cf2%%%fd34672d-751c-45ae-a963-ed177fcabe23%%%d8080257-b0c3-47b4-b0db-23bc0c8ddb3c%%%95e591a2-5d7d-4afa-b1d0-7573d6c0a5d9%%%f7d0f6bc-4dcc-4876-8c5d-b3d6ddbb3d55%%%16355082-c50b-4214-9c7d-d39575f9f79b
X-MS-Exchange-Forest-RulesExecuted: mbx-essen-02
X-MS-Exchange-Organization-RulesExecuted: mbx-essen-02
X-MS-Exchange-Forest-IndexAgent-0: AQ0CZW4AAZIQAAAPAAADH4sIAAAAAAAEAJ1Ze29bx7E/lMSnHnYedp
 LeB/bmj1aKKdpxgqKRVUOp7da+iJ3AdpoCQSAsD5fkqQ7Psuchmr35
 HBfod7kf7v5mZnd5SFmpUUGmlruzs/P8zez6/776NlP3v7p774u79+
 /d/1J9fu/kiy/76oW+NCZTz/UsyUapnus0TdQit6U52e89fKiO6efb
 PJkkmU7Vc1MUemJ4kpf/mNvZiXqsL5ORemoXJk0LdTqayugsN6OpLg
 exnT1k6lcmK0/U62mVFyO97OPUPJ6q+7/rKy/SvS/Vd8+Z9rU9UY+m
 eVKUic7UH3JdZSZXp7GfOhvK1CCxDx+o/zbjsfpGL0ubqdO/pjw4uz
 B5ZtKBzScPHzDPP2mr/oK9E3U6LejvhbZnaZJVbwY6TYZ6qFnWB+qx
 hTmSv1WGRCyTzJS8/1QXMz0yVXEW25FZ5Ca+YO68+Ch+J0M8AMuynJ
 qF+iFJY/tGGC+SNF2eJdk4B389EpnVq9JcGtjYZDDTaTEb8+hsMtNJ
 GpjlsXqc2CwzwmmGicGIJ850RcayuaP9TlepVV+nsc5KnWt1Oo/PZr
 BCZYaJF+7VdKlnzOm7XBd6pF7g4DkPz2ZJnNvCjj3taztTr3U6N0t1
 WtrZWcljt/gkT2L1Z50xr6cmK4YmnyDSTg0WLqdrzlHP0qVWj/Olnd
 lLdZqMZFRXlNnAD+MCDivKYiBuG5nLB4qHx7GOp2Zct7Vf0WHTmoGF
 p9ucgOZygnBayeW3Z29bis18KikAEUx6leDyq+tFNbkNi/bvqR4W9f
 OY67i4hq/QzGZnF5dawu8B2QXU19EyP5m+SvKNLmG2Qufqmc0uyPpp
 FU/tWcLfBuAM94Q8lCzGusnMENEPajri3BZIibwameGyNLV8r4Z/NT
 FS/sfvvn796Kn6/Ku793/7k7jxRL1ADizypDRDKKKS2Tw1MwCELnE2
 bRcYmBoVVzkCv5R9yNNrd6k4N7o0RY0gN8hi2FnZMfOLbVYmk8pWhR
 rbNLFqpEutdDZSyMlMFWaOzChNulRlkoJTUQ1XLC6BQCBTxVzHkm7l
 IolNX9ksNuDnVk1OhMRzbZ4DdKCgUlIESbUamwWzSooCx5wEzZU6/P
 xIPQErkmVmiEdufkNbRknh9EjKpQJ7reIpEtmooSkXhOdkgAQ4V1rI
 kC4dQ/qxmUHIwpuZmAy7a8RDW077KinVrCpKJyS4Z3CVM8Ogxos1me
 kL6IEtU52PyECwqVWX8LtFKoorikFdrftH7Fa2fwHi7De0+ZI1rLkO
 SEb+AxCATQV1UyYwZNcxDWuSOOHUMZtdh++xrdIRrKKmVTaCOBQHam
 YmmgJ1XaovjtSzTI2rssrh0WdqAZQkZYpqPrd5CaapBoLl4kc1yXVW
 YQYe6KvFNAFCE4hvyJTkMN9kkuNEsfZYFXYWdC+BVBySOsmg4syOkn
 ECbTkoD5nrmusQlpnBOsSaWPoMcXXEijNvEcZzFZ4xH1/UmAn3wA4m
 quapBTiOhJN4b/2IYzWsyr7Px3TZr/EDWQFdoW/JUQM2RS1QkXCDel
 JLJKeFhV3Jw6ySKZWheKcYxfeqqIXDOf7mh0c0b96UuY6lJNO6HqbB
 omM0JCzyHH3KKuNg5kIno5rsVzgzuymEZiNKNoqDKMIpwIEHPJOpBD
 Uk3UxVn5+ZRewsSVAU2thI1NIZ7IM+qQgOY7W0lRonHK6flmaGINP5
 8tOaSU2eM3SQCEZnhbjCgaGgF0d7mhJMFRTyfqFgT0ws0rruI6vGqK
 gPNpLxquDjJC0d1jl2Q0MIQTYrIT3n5zFbqpxunEHiUiYQNsNL6hEq
 JSbKKWyAQyyBBKKjyPRcFcsCmj/YSENSytmHQkI7Xek7LAqwSGYzM0
 oEpskTwv6wgPBERDG9oXVaFVMJCzRt2Ot0McVRn7eQtVIzLoWTP5Ew
 lUiTbD3QVWrjC9KhJHj0+D6qvIcXVEqQBSmFGQrClFGoymgbBRIUXM
 MJP5+CXb5KE8BRSQhL6Lte6AifrsuOeTWbO69Jfs5zCwAqiuSS7MWY
 zmi/AC2KXZqatE+JHYof9ALRrGBHxClCj+iruVQ8igk0x8UUBvCxJq
 FZr5XoASBwSdn3JKEAo0JD1cUzl2qMXPJANdHzou8r8RovLBpkstvJ
 mwBXgNJLwC8lP/Mqkr8Dt6WaeFRDVz/JxKfMwbIkdMa1dL6uiGp1Z4
 xA3F9XksRHS+x7iaFFlYFMBorMEIwxdwwFYRqC1BnL5szPxxjRDbml
 QMeOUrVE6U3LZB4gDYAxmAzqYON/7uAW5v/VP2UkX+WWxoxOlPpZrf
 7VP39eMf35XzxkY5tsXNHdWc3dWQn2PZeck5Uw4c+Afn4OM5tSvRP7
 t2hS2/HWP3KTo7JxclWmt/75Fw8JzW0tO2uxRVmBcK9i7hrmJp8lpS
 SZTicWWT+dSdV2GU6ZS0nKGUZkGYqkq+i+iNf60kDBAOIIXEZSc4Pe
 IIUQDjAvE2qXUZSlqugUaThaUlp64bnG5rYsUxJCElSifD5PE8kxyv
 PcSjVklEECY//I6UidUS21uJHr0xk6WzrJCuT3gi2GqzpBq+vQnBJT
 M9tQlXYXMRpGkooLj0NEutTH1L2BSzJ2mIaM47qFbIRpsMuMVun/CN
 JTdY3z5Vwqu3QvQ+OsTw2PWKzQ1KxTx/4WXPTQn1ck2xLX+dxmMG9K
 XaTDH+Y4B8OwS6ovyg3AutYLi128RHyRg88Dn5m/bMySybQM8qJwjK
 tUyhaa2IK62xonqQVcxqdwH27upBi6X7uASVO9hBVR+WtuCD22vwCI
 kSvY0o7Ha21fLXwhV4HgQDXP2Gu5MeyYK1cg2sbtWO71Wej0gkyb22
 oiFqOGuY/t8YWWik3BD+TMWKf1DkUK8KoQ1tzDB4gYcsdIXAs4wYWX
 grbeQa9dDTPXihRifY6muS0Sbrulo55rP7KkfFWE0iU/5JDHz769W2
 XDajw2q8sTxRG9I9HfjaW33atiiyih29dArLUeZn55VaJHdU3qLb00
 LEWoQ2O4qPDxJoKIQsAnCM8NwxJBW1IFdL6q9zlVxrVcI2Jws8N5lS
 FbsIHL4GTOLTCp8mx1/YDIORmoxm51UTysM/nzc2bA90nCzItioecj
 ji9AIdEf8Y2ZjLLJ7cqFkMxJ+ixXsTfXyJ+iJi/yVtNB5D5qxW1VIl
 EZc64JFgZDsf3q5WFizXpIU6e+7pv1njY3fP56omuC5UubXpLcDCPc
 6hE0HnJg0kNXf+PyjEwYkQZ85x/SzsxMbEkN9qjvLtS10kSHcOPE6U
 spdFSPGumu6NiXz39gq8Tr2CmA654ysDZOk5gT0b1h2M1uW9B85Zsf
 /+sn9cIy5NO9jq5r1oF4lcUu43KG0XHyhnKFNP8UbfT5p9ztXdpEUD
 hOdUGgKR3iG3rkghyBi8BBsaoLsAzAfSRRzN064qF0l3S5zFfhoci6
 MrtYgd5Kh1doOM3oGPh4PFy++wt6HJ/807fuQOifYP/J82ugf4e3wE
 B79YX8l1/Hac/qOf/dnxBpX+2N+xfetwP5215ew+IvvggHqrc+NG/w
 uOaFdr+33/txMCiyZD4Y/ETfsOvO3c/kr/pM/cFMBAS5a7Fz9y7AmR
 Je4mo4sHrI8CzuykD6Q5H1PLHnPj8/o0CX2SGddc4nOZaHbtdFYuOh
 +ow++3x1OpceIDWZvITc+R93HIHXdSfByX9Tv1cvvv/mmwfXkWcIBh
 wUl29AWZs7pBrphiTG8cOL5Bxz86Mjx2vFcVaV5s05VaPDXxOn44eL
 IX+9SupEkoPkAVMMQFKvH3T8cHw+0wyR6DTrK/0Vv1/+CbvmVCBfPH
 n9x1fnP7x89vrJ66cvv/3+T0+PamYB7h0+e3X+5OXLQxLy6OjqIaKn
 1N130vT4IdzhbtQ/3vtpoC9RNaA+PGBqR9N7mXExwcZgCfqqNnMuNP
 VY4SPxq56j0lBYrol3BEBEzBpXqwdMu9/boPEtJEJ+FZbordeD8ojz
 hN7psgtuQdz/Szqe+70o2oq2txpRD78YRNs7jWazETUj/LZbUXs/Om
 hGNztRtx29txO1mlGrHXU7jeiDqIvV7aiLyU7Uw5YD/EatrWgHHDpR
 1In2Me5GPdB3G9GHNOhsRc2dqNmKOu1G9H7UYQ697WgHZ4Hm38CH+H
 e2oxY2dqIDHHQb24mmu0VkG6cTcSvaa9JMi8lIQvDvEP8uxpjhE/d2
 GtHHNGhDFyHrRrtQ/Ga07Tf2sNqItvHvVtSRScgGbtuk2o6cCG7bUa
 cX7bUa0XtsIpkUwfyhZJBbfAp/JZN+7KRty9f/YOZNJ3YXM79iPiyt
 o/mEacLMjme7miHDNuun/Ls/1DGJdqHOR1GnzvZ9ctO2bGmRhdur1W
 h3Y6ZLZuk0yVDCEwIf7Eb7TWJL4on8bZINbKOtxjZ9sk9l9SD6CAPo
 3mFXHkQ32mzeDvm03Yt2m86qpKl83YW/eKZJIdGW8bZzwU742mS9wK
 0RbcEprcYHjShqsGphHgEMZTHAXsxwEG7vRDewHadg7Gda/NnkGNhi
 gT/YYdtiHpNbnB2QOZwi8gThoZEcyoYl8baI5qDNEQ6esiRM8Nlt3B
 Y+zJ9ExXaRR6RlO+wzf7KYSCsSykGwFafkTQkbkZAPdcYX+UG5H30i
 cgYtRHhP3GVH3xLJxSD70UdtNm8QDEsiOSlLGQF92yzwXofz0dthj+
 WhwPZkuyGMg6gyaDbeFyPIoWK0louc3XAui7EtEopSImcwNUwhu2AN
 BrH3rmMrhmXV2t3ow5BxMhDD0kzNy8JBnNj0abvr1ZQl+Mjn1A6nTN
 fPN8WGcuhWtB9kEIIOh434osV4ImEjgyazFcMeRDclMuvGF3BDkItP
 OxxsDNcd76+2DIK7d8lfuy2/He4TYwZd6pEm6nPM7Av+i1OaHqyCQe
 rzlIzRzRYHrazit+fHAoNbnGiyPaQhywnJv25F/2i0ov9tNKNEJIKH
 96J9UQsGZqHaknMhuLZ8UgbbS8Jtc6S0vCVkiz9wt8fhucXwIILIXo
 yxhM/d6GDbRWJnx2dwXd1w7hYXkyaLGiJXCDrR+x7+d/xMd4NGNN1m
 AQ44rQNwitnYWp2WF5U3tkVN4eBT56DltRBKsXSTJRRuHtUka/eDVU
 OGhdRk1boBS5hty7sAmXcrYAMbaqee38gwSZQQL11X8FshslrRe/Uk
 lr1XsVbwANVDysgu41PLezbEO1sMUXFDmAh6NT26h/BYj5aDLjtaCI
 JTZJWxQQrmzbo8bZ+40iM0V9lzWxJIhAmgItywKjG24/EyALmHkwNB
 IJDtsT2hQs8jaDe6Hawne1sOg6lo4Fz+KhjzYYA0h9keG/gU0agnZ7
 FJ90IUrcdMb4MPs/oAbcK+g4erfRk1a1caPTfJjq53T90tB5n1LsNN
 As82ekB2/Wb3EY5b7/J4fq1F6pC0m82Xm6z1ayuy9qp1gh1+hVUxix
 hTGgoW4xMCdZafDdjdpl7MIR/QgCGfAgnytKgg4JRbG1s4JG7I5C5n
 685qTBt7Lo9a6FvbFK4fNWmytbM2eVt23WZLNl0LGVZbjUbUpWakW5
 s8lnGL/Os6xL3oYxjBT4LDx5jf4+Z6/Swa96hr7vjmCwQ3muRfZ0+q
 G43oP/3XvehWm33N3LqcYgfhq6vY1A67ON+LPmwBGf4f/nChsmgpAA
 ABCtYZPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTE2
 Ij8+DQo8RW1haWxTZXQ+DQogIDxWZXJzaW9uPjE1LjAuMC4wPC9WZX
 JzaW9uPg0KICA8RW1haWxzPg0KICAgIDxFbWFpbCBTdGFydEluZGV4
 PSIxMDUiIFBvc2l0aW9uPSJPdGhlciI+DQogICAgICA8RW1haWxTdH
 Jpbmc+ZGhvd2VsbHNAcmVkaGF0LmNvbTwvRW1haWxTdHJpbmc+DQog
 ICAgPC9FbWFpbD4NCiAgICA8RW1haWwgU3RhcnRJbmRleD0iMTk3Ii
 BQb3NpdGlvbj0iT3RoZXIiPg0KICAgICAgPEVtYWlsU3RyaW5nPmNo
 cmlzdGlhbkBicmF1bmVyLmlvPC9FbWFpbFN0cmluZz4NCiAgICA8L0
 VtYWlsPg0KICAgIDxFbWFpbCBTdGFydEluZGV4PSIyMzMiIFBvc2l0
 aW9uPSJPdGhlciI+DQogICAgICA8RW1haWxTdHJpbmc+amxheXRvbk
 BrZXJuZWwub3JnPC9FbWFpbFN0cmluZz4NCiAgICA8L0VtYWlsPg0K
 ICAgIDxFbWFpbCBTdGFydEluZGV4PSIyNjkiIFBvc2l0aW9uPSJPdG
 hlciI+DQogICAgICA8RW1haWxTdHJpbmc+aHNpYW5na2FvQGxpbnV4
 LmFsaWJhYmEuY29tPC9FbWFpbFN0cmluZz4NCiAgICA8L0VtYWlsPg
 0KICAgIDxFbWFpbCBTdGFydEluZGV4PSIzMjMiIFBvc2l0aW9uPSJP
 dGhlciI+DQogICAgICA8RW1haWxTdHJpbmc+YXNtYWRldXNAY29kZX
 dyZWNrLm9yZzwvRW1haWxTdHJpbmc+DQogICAgPC9FbWFpbD4NCiAg
 ICA8RW1haWwgU3RhcnRJbmRleD0iNDEyIiBQb3NpdGlvbj0iT3RoZX
 IiPg0KICAgICAgPEVtYWlsU3RyaW5nPndpbGx5QGluZnJhZGVhZC5v
 cmc8L0VtYWlsU3RyaW5nPg0KICAgIDwvRW1haWw+DQogICAgPEVtYW
 lsIFN0YXJ0SW5kZXg9IjQ0OCIgUG9zaXRpb249Ik90aGVyIj4NCiAg
 ICAgIDxFbWFpbFN0cmluZz5zbWZyZW5jaEBnbWFpbC5jb208L0VtYW
 lsU3RyaW5nPg0KICAgIDwvRW1haWw+DQogICAgPEVtYWlsIFN0YXJ0
 SW5kZXg9IjQ4NiIgUG9zaXRpb249Ik90aGVyIj4NCiAgICAgIDxFbW
 FpbFN0cmluZz5tYXJjLmRpb25uZUBhdXJpc3Rvci5jb208L0VtYWls
 U3RyaW5nPg0KICAgIDwvRW1haWw+DQogICAgPEVtYWlsIFN0YXJ0SW
 5kZXg9IjUzMCIgUG9zaXRpb249Ik90aGVyIj4NCiAgICAgIDxFbWFp
 bFN0cmluZz5wY0BtYW5ndWViaXQuY29tPC9FbWFpbFN0cmluZz4NCi
 AgICA8L0VtYWlsPg0KICAgIDxFbWFpbCBTdGFydEluZGV4PSI1Njki
 IFBvc2l0aW9uPSJPdGhlciI+DQogICAgICA8RW1haWxTdHJpbmc+c3
 ByYXNhZEBtaWNyb3NvZnQuY29tPC9FbWFpbFN0cmluZz4NCiAgICA8
 L0VtYWlsPg0KICAgIDxFbWFpbCBTdGFydEluZGV4PSI2MDUiIFBvc2
 l0aW9uPSJPdGhlciI+DQogICAgICA8RW1haWxTdHJpbmc+dG9tQHRh
 bHBleS5jb208L0VtYWlsU3RyaW5nPg0KICAgIDwvRW1haWw+DQogIC
 AgPEVtYWlsIFN0YXJ0SW5kZXg9IjY0NyIgUG9zaXRpb249Ik90aGVy
 Ij4NCiAgICAgIDxFbWFpbFN0cmluZz5lcmljdmhAa2VybmVsLm9yZz
 wvRW1haWxTdHJpbmc+DQogICAgPC9FbWFpbD4NCiAgICA8RW1haWwg
 U3RhcnRJbmRleD0iNjgxIiBQb3NpdGlvbj0iT3RoZXIiPg0KICAgIC
 AgPEVtYWlsU3RyaW5nPmlkcnlvbW92QGdtYWlsLmNvbTwvRW1haWxT
 dHJpbmc+DQogICAgPC9FbWFpbD4NCiAgICA8RW1haWwgU3RhcnRJbm
 RleD0iNzA2IiBQb3NpdGlvbj0iT3RoZXIiPg0KICAgICAgPEVtYWls
 U3RyaW5nPm5ldGZzQGxpc3RzLmxpbnV4LmRldjwvRW1haWxTdHJpbm
 c+DQogICAgPC9FbWFpbD4NCiAgICA8RW1haWwgU3RhcnRJbmRleD0i
 NzI5IiBQb3NpdGlvbj0iT3RoZXIiPg0KICAgICAgPEVtYWlsU3RyaW
 5nPmxpbnV4LWNhY2hlZnNAcmVkaGF0LmNvbTwvRW1haWxTdHJpbmc+
 DQogICAgPC9FbWFpbD4NCiAgICA8RW1haWwgU3RhcnRJbmRleD0iNz
 U1IiBQb3NpdGlvbj0iT3RoZXIiPg0KICAgICAgPEVtYWlsU3RyaW5n
 PmxpbnV4LWFmc0BsaXN0cy5pbmZyYWRlYWQub3JnPC9FbWFpbFN0cm
 luZz4NCiAgICA8L0VtYWlsPg0KICAgIDxFbWFpbCBTdGFydEluZGV4
 PSI3OTAiIFBvc2l0aW9uPSJPdGhlciI+DQogICAgICA8RW1haWxTdH
 Jpbmc+bGludXgtY2lmc0B2Z2VyLmtlcm5lbC5vcmc8L0VtYWlsU3Ry
 aW5nPg0KICAgIDwvRW1haWw+DQogICAgPEVtYWlsIFN0YXJ0SW5kZX
 g9IjgxOCIgUG9zaXRpb249Ik90aGVyIj4NCiAgICAgIDxFbWFpbFN0
 cmluZz5saW51eC1uZnNAdmdlci5rZXJuZWwub3JnPC9FbWFpbFN0cm
 luZz4NCiAgICA8L0VtYWlsPg0KICAgIDxFbWFpbCBTdGFydEluZGV4
 PSI4NTUiIFBvc2l0aW9uPSJPdGhlciI+DQogICAgICA8RW1haWxTdH
 Jpbmc+ZGV2ZWxAdmdlci5rZXJuZWwub3JnPC9FbWFpbFN0cmluZz4N
 CiAgICA8L0VtYWlsPg0KICAgIDxFbWFpbCBTdGFydEluZGV4PSI4Nz
 giIFBvc2l0aW9uPSJPdGhlciI+DQogICAgICA8RW1haWxTdHJpbmc+
 djlmc0BsaXN0cy5saW51eC5kZXY8L0VtYWlsU3RyaW5nPg0KICAgID
 wvRW1haWw+DQogICAgPEVtYWlsIFN0YXJ0SW5kZXg9IjkwMCIgUG9z
 aXRpb249Ik90aGVyIj4NCiAgICAgIDxFbWFpbFN0cmluZz5saW51eC
 1lcm9mc0BsaXN0cy5vemxhYnMub3JnPC9FbWFpbFN0cmluZz4NCiAg
 ICA8L0VtYWlsPg0KICAgIDxFbWFpbCBTdGFydEluZGV4PSI5NDEiIF
 Bvc2l0aW9uPSJPdGhlciI+DQogICAgICA8RW1haWxTdHJpbmc+ZnNk
 ZXZlbEB2Z2VyLmtlcm5lbC5vcmc8L0VtYWlsU3RyaW5nPg0KICAgID
 wvRW1haWw+DQogICAgPEVtYWlsIFN0YXJ0SW5kZXg9Ijk2NiIgUG9z
 aXRpb249Ik90aGVyIj4NCiAgICAgIDxFbWFpbFN0cmluZz5saW51eC
 1tbUBrdmFjay5vcmc8L0VtYWlsU3RyaW5nPg0KICAgIDwvRW1haWw+
 DQogICAgPEVtYWlsIFN0YXJ0SW5kZXg9Ijk4NiIgUG9zaXRpb249Ik
 90aGVyIj4NCiAgICAgIDxFbWFpbFN0cmluZz5uZXRkZXZAdmdlci5r
 ZXJuZWwub3JnPC9FbWFpbFN0cmluZz4NCiAgICA8L0VtYWlsPg0KIC
 AgIDxFbWFpbCBTdGFydEluZGV4PSIxMDIxIiBQb3NpdGlvbj0iT3Ro
 ZXIiPg0KICAgICAgPEVtYWlsU3RyaW5nPmtlcm5lbEB2Z2VyLmtlcm
 5lbC5vcmc8L0VtYWlsU3RyaW5nPg0KICAgIDwvRW1haWw+DQogICAg
 PEVtYWlsIFN0YXJ0SW5kZXg9IjEwNjMiIFBvc2l0aW9uPSJPdGhlci
 I+DQogICAgICA8RW1haWxTdHJpbmc+bHVjaG9AaW9ua292Lm5ldDwv
 RW1haWxTdHJpbmc+DQogICAgPC9FbWFpbD4NCiAgICA8RW1haWwgU3
 RhcnRJbmRleD0iMTEwOSIgUG9zaXRpb249Ik90aGVyIj4NCiAgICAg
 IDxFbWFpbFN0cmluZz5saW51eF9vc3NAY3J1ZGVieXRlLmNvbTwvRW
 1haWxTdHJpbmc+DQogICAgPC9FbWFpbD4NCiAgPC9FbWFpbHM+DQo8
 L0VtYWlsU2V0PgEOzgFSZXRyaWV2ZXJPcGVyYXRvciwxMCwwO1JldH
 JpZXZlck9wZXJhdG9yLDExLDM7UG9zdERvY1BhcnNlck9wZXJhdG9y
 LDEwLDI7UG9zdERvY1BhcnNlck9wZXJhdG9yLDExLDA7UG9zdFdvcm
 RCcmVha2VyRGlhZ25vc3RpY09wZXJhdG9yLDEwLDY7UG9zdFdvcmRC
 cmVha2VyRGlhZ25vc3RpY09wZXJhdG9yLDExLDA7VHJhbnNwb3J0V3
 JpdGVyUHJvZHVjZXIsMjAsOQ==
X-MS-Exchange-Forest-IndexAgent: 1 7753
X-MS-Exchange-Forest-EmailMessageHash: E2F20B10
X-MS-Exchange-Forest-Language: en
X-MS-Exchange-Organization-Processed-By-Journaling: Journal Agent
X-MS-Exchange-Organization-Transport-Properties: DeliveryPriority=Low
X-MS-Exchange-Organization-Prioritization: 2:RC:REDACTED-015ab0b5b79b98c9c07fb4b31a88d5d2 at secunet.com:12/10|SR
X-MS-Exchange-Organization-IncludeInSla: False:RecipientCountThresholdExceeded

On 29/03/2024 10:34, Naveen Mamindlapalli wrote:
>> -----Original Message-----
>> From: David Howells <dhowells at redhat.com>
>> Sent: Thursday, March 28, 2024 10:04 PM
>> To: Christian Brauner <christian at brauner.io>; Jeff Layton <jlayton at kernel.org>;
>> Gao Xiang <hsiangkao at linux.alibaba.com>; Dominique Martinet
>> <asmadeus at codewreck.org>
>> Cc: David Howells <dhowells at redhat.com>; Matthew Wilcox
>> <willy at infradead.org>; Steve French <smfrench at gmail.com>; Marc Dionne
>> <marc.dionne at auristor.com>; Paulo Alcantara <pc at manguebit.com>; Shyam
>> Prasad N <sprasad at microsoft.com>; Tom Talpey <tom at talpey.com>; Eric Van
>> Hensbergen <ericvh at kernel.org>; Ilya Dryomov <idryomov at gmail.com>;
>> netfs at lists.linux.dev; linux-cachefs at redhat.com; linux-afs at lists.infradead.org;
>> linux-cifs at vger.kernel.org; linux-nfs at vger.kernel.org; ceph-
>> devel at vger.kernel.org; v9fs at lists.linux.dev; linux-erofs at lists.ozlabs.org; linux-
>> fsdevel at vger.kernel.org; linux-mm at kvack.org; netdev at vger.kernel.org; linux-
>> kernel at vger.kernel.org; Latchesar Ionkov <lucho at ionkov.net>; Christian
>> Schoenebeck <linux_oss at crudebyte.com>
>> Subject: [PATCH 19/26] netfs: New writeback implementation
>>
>> The current netfslib writeback implementation creates writeback requests of
>> contiguous folio data and then separately tiles subrequests over the space
>> twice, once for the server and once for the cache.  This creates a few
>> issues:
>>
>>   (1) Every time there's a discontiguity or a change between writing to only
>>       one destination or writing to both, it must create a new request.
>>       This makes it harder to do vectored writes.
>>
>>   (2) The folios don't have the writeback mark removed until the end of the
>>       request - and a request could be hundreds of megabytes.
>>
>>   (3) In future, I want to support a larger cache granularity, which will
>>       require aggregation of some folios that contain unmodified data (which
>>       only need to go to the cache) and some which contain modifications
>>       (which need to be uploaded and stored to the cache) - but, currently,
>>       these are treated as discontiguous.
>>
>> There's also a move to get everyone to use writeback_iter() to extract
>> writable folios from the pagecache.  That said, currently writeback_iter()
>> has some issues that make it less than ideal:
>>
>>   (1) there's no way to cancel the iteration, even if you find a "temporary"
>>       error that means the current folio and all subsequent folios are going
>>       to fail;
>>
>>   (2) there's no way to filter the folios being written back - something
>>       that will impact Ceph with it's ordered snap system;
>>
>>   (3) and if you get a folio you can't immediately deal with (say you need
>>       to flush the preceding writes), you are left with a folio hanging in
>>       the locked state for the duration, when really we should unlock it and
>>       relock it later.
>>
>> In this new implementation, I use writeback_iter() to pump folios,
>> progressively creating two parallel, but separate streams and cleaning up
>> the finished folios as the subrequests complete.  Either or both streams
>> can contain gaps, and the subrequests in each stream can be of variable
>> size, don't need to align with each other and don't need to align with the
>> folios.
>>
>> Indeed, subrequests can cross folio boundaries, may cover several folios or
>> a folio may be spanned by multiple folios, e.g.:
>>
>>           +---+---+-----+-----+---+----------+
>> Folios:  |   |   |     |     |   |          |
>>           +---+---+-----+-----+---+----------+
>>
>>             +------+------+     +----+----+
>> Upload:    |      |      |.....|    |    |
>>             +------+------+     +----+----+
>>
>>           +------+------+------+------+------+
>> Cache:   |      |      |      |      |      |
>>           +------+------+------+------+------+
>>
>> The progressive subrequest construction permits the algorithm to be
>> preparing both the next upload to the server and the next write to the
>> cache whilst the previous ones are already in progress.  Throttling can be
>> applied to control the rate of production of subrequests - and, in any
>> case, we probably want to write them to the server in ascending order,
>> particularly if the file will be extended.
>>
>> Content crypto can also be prepared at the same time as the subrequests and
>> run asynchronously, with the prepped requests being stalled until the
>> crypto catches up with them.  This might also be useful for transport
>> crypto, but that happens at a lower layer, so probably would be harder to
>> pull off.
>>
>> The algorithm is split into three parts:
>>
>>   (1) The issuer.  This walks through the data, packaging it up, encrypting
>>       it and creating subrequests.  The part of this that generates
>>       subrequests only deals with file positions and spans and so is usable
>>       for DIO/unbuffered writes as well as buffered writes.
>>
>>   (2) The collector. This asynchronously collects completed subrequests,
>>       unlocks folios, frees crypto buffers and performs any retries.  This
>>       runs in a work queue so that the issuer can return to the caller for
>>       writeback (so that the VM can have its kswapd thread back) or async
>>       writes.
>>
>>   (3) The retryer.  This pauses the issuer, waits for all outstanding
>>       subrequests to complete and then goes through the failed subrequests
>>       to reissue them.  This may involve reprepping them (with cifs, the
>>       credits must be renegotiated, and a subrequest may need splitting),
>>       and doing RMW for content crypto if there's a conflicting change on
>>       the server.
>>
>> [!] Note that some of the functions are prefixed with "new_" to avoid
>> clashes with existing functions.  These will be renamed in a later patch
>> that cuts over to the new algorithm.
>>
>> Signed-off-by: David Howells <dhowells at redhat.com>
>> cc: Jeff Layton <jlayton at kernel.org>
>> cc: Eric Van Hensbergen <ericvh at kernel.org>
>> cc: Latchesar Ionkov <lucho at ionkov.net>
>> cc: Dominique Martinet <asmadeus at codewreck.org>
>> cc: Christian Schoenebeck <linux_oss at crudebyte.com>
>> cc: Marc Dionne <marc.dionne at auristor.com>
>> cc: v9fs at lists.linux.dev
>> cc: linux-afs at lists.infradead.org
>> cc: netfs at lists.linux.dev
>> cc: linux-fsdevel at vger.kernel.org

[..snip..]

>> +/*
>> + * Begin a write operation for writing through the pagecache.
>> + */
>> +struct netfs_io_request *new_netfs_begin_writethrough(struct kiocb *iocb, size_t
>> len)
>> +{
>> +	struct netfs_io_request *wreq = NULL;
>> +	struct netfs_inode *ictx = netfs_inode(file_inode(iocb->ki_filp));
>> +
>> +	mutex_lock(&ictx->wb_lock);
>> +
>> +	wreq = netfs_create_write_req(iocb->ki_filp->f_mapping, iocb->ki_filp,
>> +				      iocb->ki_pos, NETFS_WRITETHROUGH);
>> +	if (IS_ERR(wreq))
>> +		mutex_unlock(&ictx->wb_lock);
>> +
>> +	wreq->io_streams[0].avail = true;
>> +	trace_netfs_write(wreq, netfs_write_trace_writethrough);
> 
> Missing mutex_unlock() before return.
> 

mutex_unlock() happens in new_netfs_end_writethrough()

> Thanks,
> Naveen
> 


X-sender: <netdev+bounces-83486-steffen.klassert=secunet.com at vger.kernel.org>
X-Receiver: <steffen.klassert at secunet.com> ORCPT=rfc822;steffen.klassert at secunet.com
X-CreatedBy: MSExchange15
X-HeloDomain: mbx-dresden-01.secunet.de
X-ExtendedProps: BQBjAAoANE2mlidQ3AgFADcAAgAADwA8AAAATWljcm9zb2Z0LkV4Y2hhbmdlLlRyYW5zcG9ydC5NYWlsUmVjaXBpZW50Lk9yZ2FuaXphdGlvblNjb3BlEQAAAAAAAAAAAAAAAAAAAAAADwA/AAAATWljcm9zb2Z0LkV4Y2hhbmdlLlRyYW5zcG9ydC5EaXJlY3RvcnlEYXRhLk1haWxEZWxpdmVyeVByaW9yaXR5DwADAAAATG93
X-Source: SMTP:Default MBX-ESSEN-02
X-SourceIPAddress: 10.53.40.199
X-EndOfInjectedXHeaders: 15568
Received: from mbx-dresden-01.secunet.de (10.53.40.199) by
 mbx-essen-02.secunet.de (10.53.40.198) with Microsoft SMTP Server
 (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id
 15.1.2507.37; Sat, 30 Mar 2024 02:06:35 +0100
Received: from a.mx.secunet.com (62.96.220.36) by cas-essen-01.secunet.de
 (10.53.40.201) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35 via Frontend
 Transport; Sat, 30 Mar 2024 02:06:34 +0100
Received: from localhost (localhost [127.0.0.1])
	by a.mx.secunet.com (Postfix) with ESMTP id B2D8120847
	for <steffen.klassert at secunet.com>; Sat, 30 Mar 2024 02:06:34 +0100 (CET)
X-Virus-Scanned: by secunet
X-Spam-Flag: NO
X-Spam-Score: -2.751
X-Spam-Level:
X-Spam-Status: No, score=-2.751 tagged_above=-999 required=2.1
	tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1,
	DKIM_VALID_AU=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249,
	MAILING_LIST_MULTI=-1, RCVD_IN_DNSWL_NONE=-0.0001,
	SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: a.mx.secunet.com (amavisd-new);
	dkim=pass (1024-bit key) header.d=linux.dev
Received: from a.mx.secunet.com ([127.0.0.1])
	by localhost (a.mx.secunet.com [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id rTuZ69sK9a_2 for <steffen.klassert at secunet.com>;
	Sat, 30 Mar 2024 02:06:31 +0100 (CET)
Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=147.75.199.223; helo=ny.mirrors.kernel.org; envelope-from=netdev+bounces-83486-steffen.klassert=secunet.com at vger.kernel.org; receiver=steffen.klassert at secunet.com 
DKIM-Filter: OpenDKIM Filter v2.11.0 a.mx.secunet.com 29E82201E5
Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org [147.75.199.223])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by a.mx.secunet.com (Postfix) with ESMTPS id 29E82201E5
	for <steffen.klassert at secunet.com>; Sat, 30 Mar 2024 02:06:31 +0100 (CET)
Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by ny.mirrors.kernel.org (Postfix) with ESMTPS id E8EA61C21430
	for <steffen.klassert at secunet.com>; Sat, 30 Mar 2024 01:06:29 +0000 (UTC)
Received: from localhost.localdomain (localhost.localdomain [127.0.0.1])
	by smtp.subspace.kernel.org (Postfix) with ESMTP id EB85617D2;
	Sat, 30 Mar 2024 01:06:22 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="VugHsG2Y"
X-Original-To: netdev at vger.kernel.org
Received: from out-173.mta0.migadu.com (out-173.mta0.migadu.com [91.218.175.173])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id 762E77E8
	for <netdev at vger.kernel.org>; Sat, 30 Mar 2024 01:06:19 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.173
ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116;
	t=1711760782; cv=none; b=dASfLrrkRxmD6WmYvcvyTFgLXAgqW4qcP8FwVw/FT8ajSayU1k2jNzB6oEhlAk4YxWiFWUStYosH2VKaROBs5wKHQh4Rsxe59gs4L4KuJN+VlHKDa1iIm9ShtgGS6jAthHnsiMpAE+me1GueQZILnQSEjyu5ZoBpE9mg1Ojzukk=
ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org;
	s=arc-20240116; t=1711760782; c=relaxed/simple;
	bh=0Tab6a8G72hAdQQTifWJdxmg84+y/tA9VvkWPFI9VQA=;
	h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From:
	 In-Reply-To:Content-Type; b=J9cTiVFLYlA2DR+fBNIsoiV/11LbFxExG+qAmCsON2fksIZjZEAFWqHQx2zJk8Dqn3t/Quqw4LH8Yjb2qqlthM0L82RcciykTG9EQ9SPWlqiRoPPhuerZSz/amNX1IgyImsufFdXk4+oiQpzCA0LsWzVgTdTI9x4oenmDhjahZI=
ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=VugHsG2Y; arc=none smtp.client-ip=91.218.175.173
Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev
Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev
Message-ID: <08dd01e3-c45e-47d9-bcde-55f7d1edc480 at linux.dev>
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1;
	t=1711760777;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:cc:mime-version:mime-version:content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding:
	 in-reply-to:in-reply-to:references:references;
	bh=BFgFwDTUylEnErMgQV0Ufr9/Ufnl/0omKSnqywixpVg=;
	b=VugHsG2Y5vYBP20CvcOMmGaEI/5A/PvLnTsQTtTA0ZebTuac4nORyH8iRZe/CwFs5RLRhJ
	4Ih/prbwbd8/OSA0Wv9Z9Z9JdeLOJUf8/vLW1xeGCG/2qNeI4CXYcIw3EixotT7o6oviEg
	ZM4gfY/Y4bUjm5TsY8pyZBWQLZ0Jv74=
Date: Fri, 29 Mar 2024 18:06:09 -0700
Precedence: bulk
X-Mailing-List: netdev at vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe at vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe at vger.kernel.org>
MIME-Version: 1.0
Subject: Re: [PATCH 19/26] netfs: New writeback implementation
Content-Language: en-US
To: Naveen Mamindlapalli <naveenm at marvell.com>,
 David Howells <dhowells at redhat.com>, Christian Brauner
 <christian at brauner.io>, Jeff Layton <jlayton at kernel.org>,
 Gao Xiang <hsiangkao at linux.alibaba.com>,
 Dominique Martinet <asmadeus at codewreck.org>
Cc: Matthew Wilcox <willy at infradead.org>, Steve French <smfrench at gmail.com>,
 Marc Dionne <marc.dionne at auristor.com>, Paulo Alcantara <pc at manguebit.com>,
 Shyam Prasad N <sprasad at microsoft.com>, Tom Talpey <tom at talpey.com>,
 Eric Van Hensbergen <ericvh at kernel.org>, Ilya Dryomov <idryomov at gmail.com>,
 "netfs at lists.linux.dev" <netfs at lists.linux.dev>,
 "linux-cachefs at redhat.com" <linux-cachefs at redhat.com>,
 "linux-afs at lists.infradead.org" <linux-afs at lists.infradead.org>,
 "linux-cifs at vger.kernel.org" <linux-cifs at vger.kernel.org>,
 "linux-nfs at vger.kernel.org" <linux-nfs at vger.kernel.org>,
 "ceph-devel at vger.kernel.org" <ceph-devel at vger.kernel.org>,
 "v9fs at lists.linux.dev" <v9fs at lists.linux.dev>,
 "linux-erofs at lists.ozlabs.org" <linux-erofs at lists.ozlabs.org>,
 "linux-fsdevel at vger.kernel.org" <linux-fsdevel at vger.kernel.org>,
 "linux-mm at kvack.org" <linux-mm at kvack.org>,
 "netdev at vger.kernel.org" <netdev at vger.kernel.org>,
 "linux-kernel at vger.kernel.org" <linux-kernel at vger.kernel.org>,
 Latchesar Ionkov <lucho at ionkov.net>,
 Christian Schoenebeck <linux_oss at crudebyte.com>
References: <20240328163424.2781320-1-dhowells at redhat.com>
 <20240328163424.2781320-20-dhowells at redhat.com>
 <SJ2PR18MB5635A86C024316BC5E57B79EA23A2 at SJ2PR18MB5635.namprd18.prod.outlook.com>
X-Report-Abuse: Please report any abuse attempt to abuse at migadu.com and include these headers.
From: Vadim Fedorenko <vadim.fedorenko at linux.dev>
In-Reply-To: <SJ2PR18MB5635A86C024316BC5E57B79EA23A2 at SJ2PR18MB5635.namprd18.prod.outlook.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Migadu-Flow: FLOW_OUT
Return-Path: netdev+bounces-83486-steffen.klassert=secunet.com at vger.kernel.org
X-MS-Exchange-Organization-OriginalArrivalTime: 30 Mar 2024 01:06:34.7524
 (UTC)
X-MS-Exchange-Organization-Network-Message-Id: 87894eac-5cd8-4523-8273-08dc5055a4d3
X-MS-Exchange-Organization-OriginalClientIPAddress: 62.96.220.36
X-MS-Exchange-Organization-OriginalServerIPAddress: 10.53.40.201
X-MS-Exchange-Organization-Cross-Premises-Headers-Processed: cas-essen-01.secunet.de
X-MS-Exchange-Organization-OrderedPrecisionLatencyInProgress: LSRV=cas-essen-01.secunet.de:TOTAL-FE=0.012|SMR=0.012(SMRPI=0.009(SMRPI-FrontendProxyAgent=0.009));2024-03-30T01:06:34.764Z
X-MS-Exchange-Forest-ArrivalHubServer: mbx-essen-02.secunet.de
X-MS-Exchange-Organization-AuthSource: cas-essen-01.secunet.de
X-MS-Exchange-Organization-AuthAs: Anonymous
X-MS-Exchange-Organization-OriginalSize: 15019
X-MS-Exchange-Organization-Transport-Properties: DeliveryPriority=Low
X-MS-Exchange-Organization-Prioritization: 2:ShadowRedundancy
X-MS-Exchange-Organization-IncludeInSla: False:ShadowRedundancy

On 29/03/2024 10:34, Naveen Mamindlapalli wrote:
>> -----Original Message-----
>> From: David Howells <dhowells at redhat.com>
>> Sent: Thursday, March 28, 2024 10:04 PM
>> To: Christian Brauner <christian at brauner.io>; Jeff Layton <jlayton at kernel.org>;
>> Gao Xiang <hsiangkao at linux.alibaba.com>; Dominique Martinet
>> <asmadeus at codewreck.org>
>> Cc: David Howells <dhowells at redhat.com>; Matthew Wilcox
>> <willy at infradead.org>; Steve French <smfrench at gmail.com>; Marc Dionne
>> <marc.dionne at auristor.com>; Paulo Alcantara <pc at manguebit.com>; Shyam
>> Prasad N <sprasad at microsoft.com>; Tom Talpey <tom at talpey.com>; Eric Van
>> Hensbergen <ericvh at kernel.org>; Ilya Dryomov <idryomov at gmail.com>;
>> netfs at lists.linux.dev; linux-cachefs at redhat.com; linux-afs at lists.infradead.org;
>> linux-cifs at vger.kernel.org; linux-nfs at vger.kernel.org; ceph-
>> devel at vger.kernel.org; v9fs at lists.linux.dev; linux-erofs at lists.ozlabs.org; linux-
>> fsdevel at vger.kernel.org; linux-mm at kvack.org; netdev at vger.kernel.org; linux-
>> kernel at vger.kernel.org; Latchesar Ionkov <lucho at ionkov.net>; Christian
>> Schoenebeck <linux_oss at crudebyte.com>
>> Subject: [PATCH 19/26] netfs: New writeback implementation
>>
>> The current netfslib writeback implementation creates writeback requests of
>> contiguous folio data and then separately tiles subrequests over the space
>> twice, once for the server and once for the cache.  This creates a few
>> issues:
>>
>>   (1) Every time there's a discontiguity or a change between writing to only
>>       one destination or writing to both, it must create a new request.
>>       This makes it harder to do vectored writes.
>>
>>   (2) The folios don't have the writeback mark removed until the end of the
>>       request - and a request could be hundreds of megabytes.
>>
>>   (3) In future, I want to support a larger cache granularity, which will
>>       require aggregation of some folios that contain unmodified data (which
>>       only need to go to the cache) and some which contain modifications
>>       (which need to be uploaded and stored to the cache) - but, currently,
>>       these are treated as discontiguous.
>>
>> There's also a move to get everyone to use writeback_iter() to extract
>> writable folios from the pagecache.  That said, currently writeback_iter()
>> has some issues that make it less than ideal:
>>
>>   (1) there's no way to cancel the iteration, even if you find a "temporary"
>>       error that means the current folio and all subsequent folios are going
>>       to fail;
>>
>>   (2) there's no way to filter the folios being written back - something
>>       that will impact Ceph with it's ordered snap system;
>>
>>   (3) and if you get a folio you can't immediately deal with (say you need
>>       to flush the preceding writes), you are left with a folio hanging in
>>       the locked state for the duration, when really we should unlock it and
>>       relock it later.
>>
>> In this new implementation, I use writeback_iter() to pump folios,
>> progressively creating two parallel, but separate streams and cleaning up
>> the finished folios as the subrequests complete.  Either or both streams
>> can contain gaps, and the subrequests in each stream can be of variable
>> size, don't need to align with each other and don't need to align with the
>> folios.
>>
>> Indeed, subrequests can cross folio boundaries, may cover several folios or
>> a folio may be spanned by multiple folios, e.g.:
>>
>>           +---+---+-----+-----+---+----------+
>> Folios:  |   |   |     |     |   |          |
>>           +---+---+-----+-----+---+----------+
>>
>>             +------+------+     +----+----+
>> Upload:    |      |      |.....|    |    |
>>             +------+------+     +----+----+
>>
>>           +------+------+------+------+------+
>> Cache:   |      |      |      |      |      |
>>           +------+------+------+------+------+
>>
>> The progressive subrequest construction permits the algorithm to be
>> preparing both the next upload to the server and the next write to the
>> cache whilst the previous ones are already in progress.  Throttling can be
>> applied to control the rate of production of subrequests - and, in any
>> case, we probably want to write them to the server in ascending order,
>> particularly if the file will be extended.
>>
>> Content crypto can also be prepared at the same time as the subrequests and
>> run asynchronously, with the prepped requests being stalled until the
>> crypto catches up with them.  This might also be useful for transport
>> crypto, but that happens at a lower layer, so probably would be harder to
>> pull off.
>>
>> The algorithm is split into three parts:
>>
>>   (1) The issuer.  This walks through the data, packaging it up, encrypting
>>       it and creating subrequests.  The part of this that generates
>>       subrequests only deals with file positions and spans and so is usable
>>       for DIO/unbuffered writes as well as buffered writes.
>>
>>   (2) The collector. This asynchronously collects completed subrequests,
>>       unlocks folios, frees crypto buffers and performs any retries.  This
>>       runs in a work queue so that the issuer can return to the caller for
>>       writeback (so that the VM can have its kswapd thread back) or async
>>       writes.
>>
>>   (3) The retryer.  This pauses the issuer, waits for all outstanding
>>       subrequests to complete and then goes through the failed subrequests
>>       to reissue them.  This may involve reprepping them (with cifs, the
>>       credits must be renegotiated, and a subrequest may need splitting),
>>       and doing RMW for content crypto if there's a conflicting change on
>>       the server.
>>
>> [!] Note that some of the functions are prefixed with "new_" to avoid
>> clashes with existing functions.  These will be renamed in a later patch
>> that cuts over to the new algorithm.
>>
>> Signed-off-by: David Howells <dhowells at redhat.com>
>> cc: Jeff Layton <jlayton at kernel.org>
>> cc: Eric Van Hensbergen <ericvh at kernel.org>
>> cc: Latchesar Ionkov <lucho at ionkov.net>
>> cc: Dominique Martinet <asmadeus at codewreck.org>
>> cc: Christian Schoenebeck <linux_oss at crudebyte.com>
>> cc: Marc Dionne <marc.dionne at auristor.com>
>> cc: v9fs at lists.linux.dev
>> cc: linux-afs at lists.infradead.org
>> cc: netfs at lists.linux.dev
>> cc: linux-fsdevel at vger.kernel.org

[..snip..]

>> +/*
>> + * Begin a write operation for writing through the pagecache.
>> + */
>> +struct netfs_io_request *new_netfs_begin_writethrough(struct kiocb *iocb, size_t
>> len)
>> +{
>> +	struct netfs_io_request *wreq = NULL;
>> +	struct netfs_inode *ictx = netfs_inode(file_inode(iocb->ki_filp));
>> +
>> +	mutex_lock(&ictx->wb_lock);
>> +
>> +	wreq = netfs_create_write_req(iocb->ki_filp->f_mapping, iocb->ki_filp,
>> +				      iocb->ki_pos, NETFS_WRITETHROUGH);
>> +	if (IS_ERR(wreq))
>> +		mutex_unlock(&ictx->wb_lock);
>> +
>> +	wreq->io_streams[0].avail = true;
>> +	trace_netfs_write(wreq, netfs_write_trace_writethrough);
> 
> Missing mutex_unlock() before return.
> 

mutex_unlock() happens in new_netfs_end_writethrough()

> Thanks,
> Naveen
> 




More information about the Linux-erofs mailing list