<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
在 2021/11/23 10:59, Jianhua1 Hao 郝建华 via Linux-erofs 写道:<br>
<blockquote type="cite" cite="mid:669247783482488c8654c55e67fe1ac3@xiaomi.com">
<style>body { line-height: 1.5; }p { margin-top: 0px; margin-bottom: 0px; }body { font-size: 14px; font-family: "Microsoft YaHei UI"; color: rgb(0, 0, 0); line-height: 1.5; }body { font-size: 14px; font-family: "Microsoft YaHei UI"; color: rgb(0, 0, 0); line-height: 1.5; }</style>
<div><span></span>
<pre style="overflow-wrap: break-word; margin: 0em;"><span style="white-space: pre-wrap; text-decoration-style: initial; text-decoration-color: initial;"><font face="courier, courier new, monospace" color="#a01e1e"><b>We also found that it is easy to cause deadlock in the kswap scene， We observed the following deadlock in the stress test under low
memory scenario，</b></font></span><span style="font-family: courier, "courier new", monospace; white-space: pre-wrap; color: rgb(160, 30, 30);"> <b></b></span><font face="courier, courier new, monospace" color="#a01e1e"><span style="white-space: pre-wrap;"><b>Same as "erofs: fix deadlock when shrink erofs slab".</b></span></font></pre>
<pre style="overflow-wrap: break-word; margin: 0em;"><font face="courier, courier new, monospace" color="#a01e1e"><span style="white-space: pre-wrap;"><b>
</b></span></font></pre>
<pre style="overflow-wrap: break-word; margin: 0em;"><p class="MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; white-space: normal;"><span style="font-size: 9pt;">Thread A: <o:p></o:p></span><span style="font-size: 9pt; background-color: transparent;">Thread B:</span></p><p class="MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; white-space: normal;"><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;">erofs_try_to_release_workgroup(</span><span style="background-color: transparent; font-size: 12px; color: rgb(112, 173, 71);">grp = 0xFFFFFF87ADFEE610</span><span style="background-color: transparent; font-size: 12px;">)</span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-size: 9pt;">erofs_insert_workgroup()</span></p><p class="MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; white-space: normal;"><span style="font-size: 9pt;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;">erofs_workgroup_try_to_freeze(grp, 1)</span><span style="background-color: transparent; font-size: 9pt;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="background-color: transparent; font-size: 9pt;">//xa lock is held here</span></p><p class="MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; white-space: normal;"><span style="font-size: 9pt;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;">//set ref count to EROFS_LOCKED_MAGIC</span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-size: 9pt;">xa_lock(&sbi->managed_pslots);</span></p><p class="MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; white-space: normal;"><span style="font-size: 9pt;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;">atomic_cmpxchg(&grp->refcount, val,</span><span style="background-color: transparent; font-size: 12px;"> </span><span style="background-color: transparent; font-size: 12px; color: red;">EROFS_LOCKED_MAGIC</span><span style="background-color: transparent; font-size: 12px;">)</span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-size: 9pt;">pre = __xa_cmpxchg(&sbi->managed_pslots, grp->index, NULL, grp, GFP_NOFS);</span></p><p class="MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; white-space: normal;"><span style="font-size: 9pt;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;">xa_erase(&sbi->managed_pslots, grp->index)</span><span style="background-color: transparent; font-size: 12px;"> </span><span style="background-color: transparent; font-size: 9pt;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-size: 9pt;">erofs_workgroup_get(pre) //</span><span style="background-color: transparent; font-size: 9pt; color: rgb(112, 173, 71);">pre = grp = 0xFFFFFF87ADFEE610</span><span style="background-color: transparent; font-size: 9pt;"> </span></p><p class="MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; white-space: normal;"><span style="font-size: 9pt;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;">//stuck there to wait for xa lock, already held by thread B</span><span style="background-color: transparent; font-size: 9pt;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="background-color: transparent; font-size: 9pt;">erofs_wait_on_workgroup_freezed(grp);</span></p><p class="MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; white-space: normal;"><span style="font-size: 9pt;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;">xa_lock(xa); </span><span style="background-color: transparent; font-size: 9pt;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="background-color: transparent; font-family: "Microsoft YaHei UI"; font-size: 14px;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="background-color: transparent; font-size: 9pt;">//wait ref count to be unlocked, which should be done by thread A</span></p><p class="MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; white-space: normal;"><span style="font-size: 9pt;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="font-family: "Microsoft YaHei UI"; font-size: 14px; background-color: transparent;"> </span><span style="background-color: transparent; font-size: 9pt;">atomic_cond_read_relaxed(&grp->refcount, VAL != </span><span style="background-color: transparent; font-size: 8pt; color: red;">EROFS_LOCKED_MAGIC</span><span style="background-color: transparent; font-size: 8pt;">);</span></p><p class="MsoNormal" style="margin: 0in; white-space: normal;"><font face="courier, courier new, monospace" color="#a01e1e"><span style="text-decoration-style: initial; text-decoration-color: initial;">
</span></font></p><p class="MsoNormal" style="margin: 0in;"><span style="white-space: normal; text-decoration-style: initial; text-decoration-color: initial;"><font face="courier, courier new, monospace" color="#a01e1e">Follow-up fix:</font></span><span style="color: rgb(160, 30, 30); font-family: courier, "courier new", monospace; white-space: pre-wrap; background-color: transparent;">it need to hold the xa lock before freeze the workgroup</span></p><pre style="font-family: courier, "courier new", monospace; white-space: pre-wrap; overflow-wrap: break-word; margin-top: 0px; margin-bottom: 0px; color: rgb(160, 30, 30);">beacuse we will operate xarry?</pre><p class="MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; white-space: normal;"><span style="font-size: 9pt;"> <o:p></o:p></span></p><p class="MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; white-space: normal;"><span style="font-size: 9pt;"> <o:p></o:p></span></p><p class="MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; white-space: normal;"><span style="font-size: 9pt;">
</span></p></pre>
</div>
</blockquote>
Hi, JianHua,<br>
<br>
The fix is in the patch, please test it kindly if you have
condition.<br>
<a class="moz-txt-link-freetext" href="https://lore.kernel.org/linux-erofs/YZcJpDs3FKpSfzAE@B-P7TQMD6M-0146/T/#t">https://lore.kernel.org/linux-erofs/YZcJpDs3FKpSfzAE@B-P7TQMD6M-0146/T/#t</a><br>
<br>
---<br>
fs/erofs/utils.c | 9 +++++++--<br>
1 file changed, 7 insertions(+), 2 deletions(-)<br>
<br>
diff --git a/fs/erofs/utils.c b/fs/erofs/utils.c<br>
index 84da2c280012..84a59f075dd1 100644<br>
--- a/fs/erofs/utils.c<br>
+++ b/fs/erofs/utils.c<br>
@@ -150,7 +150,7 @@ static bool
erofs_try_to_release_workgroup(struct erofs_sb_info *sbi,<br>
* however in order to avoid some race conditions, add a<br>
* DBG_BUGON to observe this in advance.<br>
*/<br>
- DBG_BUGON(xa_erase(&sbi->managed_pslots, grp->index)
!= grp);<br>
+ DBG_BUGON(__xa_erase(&sbi->managed_pslots,
grp->index) != grp);<br>
<br>
/* last refcount should be connected with its managed pslot.
*/<br>
erofs_workgroup_unfreeze(grp, 0);<br>
@@ -165,15 +165,20 @@ static unsigned long
erofs_shrink_workstation(struct erofs_sb_info *sbi,<br>
unsigned int freed = 0;<br>
unsigned long index;<br>
<br>
+ xa_lock(&sbi->managed_pslots);<br>
xa_for_each(&sbi->managed_pslots, index, grp) {<br>
/* try to shrink each valid workgroup */<br>
if (!erofs_try_to_release_workgroup(sbi, grp))<br>
continue;<br>
+ xa_unlock(&sbi->managed_pslots);<br>
<br>
++freed;<br>
if (!--nr_shrink)<br>
- break;<br>
+ return freed;<br>
+ xa_lock(&sbi->managed_pslots);<br>
}<br>
+ xa_unlock(&sbi->managed_pslots);<br>
+<br>
return freed;<br>
}<br>
<br>
<br>
Thanks,<br>
Jianan<br>
<br>
<blockquote type="cite" cite="mid:669247783482488c8654c55e67fe1ac3@xiaomi.com">
<div>
<pre style="overflow-wrap: break-word; margin: 0em;"><p class="MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; white-space: normal;"><span style="font-size: 9pt;"> <o:p></o:p></span></p><p class="MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; white-space: normal;"><span style="font-size: 9pt;"> </span></p></pre>
</div>
<div><br>
</div>
<hr style="width: 210px; height: 1px;" size="1" color="#b5c4df" align="left">
<div><span>
<div style="MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE:
10pt">
<div>Jianhua1 Hao</div>
</div>
</span></div>
#/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！
This e-mail and its attachments contain confidential information
from XIAOMI, which is intended only for the person or entity whose
address is listed above. Any use of the information contained
herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than
the intended recipient(s) is prohibited. If you receive this
e-mail in error, please notify the sender by phone or email
immediately and delete it!******/#
</blockquote>
<br>
</body>
</html>