<div dir="ltr"><div dir="auto">>
Jacob: Can you repeat bisection please? Why did you skip VMA lock-based<br>
page fault commits in your bisection?</div><div dir="auto"><br></div><div dir="auto">All skips were due to compile errors of the form:<br>make[3]: 'install_headers' is up to date.<br>In file included from ./include/linux/memcontrol.h:20,<br> from ./include/linux/swap.h:9,<br> from ./include/linux/suspend.h:5,<br> from arch/x86/kernel/asm-offsets.c:14:<br>./include/linux/mm.h: In function ‘vma_try_start_write’:<br>./include/linux/mm.h:702:37: error: ‘struct vm_area_struct’ has no member named ‘vm_lock’<br> 702 | if (!down_write_trylock(&vma->vm_lock->lock))<br> | ^~<br>./include/linux/mm.h:706:22: error: ‘struct vm_area_struct’ has no member named ‘vm_lock’<br> 706 | up_write(&vma->vm_lock->lock);<br> | ^~<br>make[1]: *** [scripts/Makefile.build:114: arch/x86/kernel/asm-offsets.s] Error 1<br>make: *** [Makefile:1286: prepare0] Error 2</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jul 2, 2023, 08:27 Bagas Sanjaya <<a href="mailto:bagasdotme@gmail.com" target="_blank">bagasdotme@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
I notice a regression report on Bugzilla [1]. Quoting from it:<br>
<br>
> After upgrading to kernel version 6.4.0 from 6.3.9, I noticed frequent but random crashes in a user space program. After a lot of reduction, I have come up with the following reproducer program:<br>
> <br>
> $ uname -a<br>
> Linux jacob 6.4.1-gentoo #1 SMP PREEMPT_DYNAMIC Sat Jul 1 19:02:42 EDT 2023 x86_64 AMD Ryzen 9 7950X3D 16-Core Processor AuthenticAMD GNU/Linux<br>
> $ cat repro.c<br>
> #define _GNU_SOURCE<br>
> #include <sched.h><br>
> #include <sys/wait.h><br>
> #include <unistd.h><br>
> <br>
> void *threadSafeAlloc(size_t n) {<br>
> static size_t end_index = 0;<br>
> static char buffer[1 << 25];<br>
> size_t start_index = __atomic_load_n(&end_index, __ATOMIC_SEQ_CST);<br>
> while (1) {<br>
> if (start_index + n > sizeof(buffer)) _exit(1);<br>
> if (__atomic_compare_exchange_n(&end_index, &start_index, start_index + n, 1, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST)) return buffer + start_index;<br>
> }<br>
> }<br>
> <br>
> int thread(void *arg) {<br>
> size_t i;<br>
> size_t n = 1 << 7;<br>
> char *items;<br>
> (void)arg;<br>
> while (1) {<br>
> items = threadSafeAlloc(n);<br>
> for (i = 0; i != n; i += 1) items[i] = '@';<br>
> for (i = 0; i != n; i += 1) if (items[i] != '@') _exit(2);<br>
> }<br>
> }<br>
> <br>
> int main(void) {<br>
> static size_t stacks[2][1 << 9];<br>
> size_t i;<br>
> for (i = 0; i != 2; i += 1) clone(&thread, &stacks[i] + 1, CLONE_THREAD | CLONE_VM | CLONE_SIGHAND, NULL);<br>
> while (1) {<br>
> if (fork() == 0) _exit(0);<br>
> (void)wait(NULL);<br>
> }<br>
> }<br>
> $ cc repro.c<br>
> $ ./a.out<br>
> $ echo $?<br>
> 2<br>
> <br>
> After tuning the various parameters for my computer, exit code 2, which indicates that memory corruption was detected, occurs approximately 99% of the time. Exit code 1, which occurs approximately 1% of the time, means it ran out of statically-allocated memory before reproducing the issue, and increasing the memory usage any more only leads to diminishing returns. There is also something like a 0.1% chance that it segfaults due to memory corruption elsewhere than in the statically-allocated buffer.<br>
> <br>
> With this reproducer in hand, I was able to perform the following bisection:<br>
> <br>
> git bisect start<br>
> # status: waiting for both good and bad commits<br>
> # bad: [6995e2de6891c724bfeb2db33d7b87775f913ad1] Linux 6.4<br>
> git bisect bad 6995e2de6891c724bfeb2db33d7b87775f913ad1<br>
> # status: waiting for good commit(s), bad commit known<br>
> # good: [457391b0380335d5e9a5babdec90ac53928b23b4] Linux 6.3<br>
> git bisect good 457391b0380335d5e9a5babdec90ac53928b23b4<br>
> # good: [d42b1c47570eb2ed818dc3fe94b2678124af109d] Merge tag 'devicetree-for-6.4-1' of git://<a href="http://git.kernel.org/pub/scm/linux/kernel/git/robh/linux" rel="noreferrer noreferrer" target="_blank">git.kernel.org/pub/scm/linux/kernel/git/robh/linux</a><br>
> git bisect good d42b1c47570eb2ed818dc3fe94b2678124af109d<br>
> # bad: [58390c8ce1bddb6c623f62e7ed36383e7fa5c02f] Merge tag 'iommu-updates-v6.4' of git://<a href="http://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu" rel="noreferrer noreferrer" target="_blank">git.kernel.org/pub/scm/linux/kernel/git/joro/iommu</a><br>
> git bisect bad 58390c8ce1bddb6c623f62e7ed36383e7fa5c02f<br>
> # good: [888d3c9f7f3ae44101a3fd76528d3dd6f96e9fd0] Merge tag 'sysctl-6.4-rc1' of git://<a href="http://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux" rel="noreferrer noreferrer" target="_blank">git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux</a><br>
> git bisect good 888d3c9f7f3ae44101a3fd76528d3dd6f96e9fd0<br>
> # bad: [86e98ed15b3e34460d1b3095bd119b6fac11841c] Merge tag 'cgroup-for-6.4' of git://<a href="http://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup" rel="noreferrer noreferrer" target="_blank">git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup</a><br>
> git bisect bad 86e98ed15b3e34460d1b3095bd119b6fac11841c<br>
> # bad: [7fa8a8ee9400fe8ec188426e40e481717bc5e924] Merge tag 'mm-stable-2023-04-27-15-30' of git://<a href="http://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm" rel="noreferrer noreferrer" target="_blank">git.kernel.org/pub/scm/linux/kernel/git/akpm/mm</a><br>
> git bisect bad 7fa8a8ee9400fe8ec188426e40e481717bc5e924<br>
> # bad: [0120dd6e4e202e19a0e011e486fb2da40a5ea279] zram: make zram_bio_discard more self-contained<br>
> git bisect bad 0120dd6e4e202e19a0e011e486fb2da40a5ea279<br>
> # good: [fce0b4213edb960859dcc65ea414c8efb11948e1] mm/page_alloc: add helper for checking if check_pages_enabled<br>
> git bisect good fce0b4213edb960859dcc65ea414c8efb11948e1<br>
> # bad: [59f876fb9d68a4d8c20305d7a7a0daf4ee9478a8] mm: avoid passing 0 to __ffs()<br>
> git bisect bad 59f876fb9d68a4d8c20305d7a7a0daf4ee9478a8<br>
> # good: [0050d7f5ee532f92e8ab1efcec6547bfac527973] afs: split afs_pagecache_valid() out of afs_validate()<br>
> git bisect good 0050d7f5ee532f92e8ab1efcec6547bfac527973<br>
> # good: [2ac0af1b66e3b66307f53b1cc446514308ec466d] mm: fall back to mmap_lock if vma->anon_vma is not yet set<br>
> git bisect good 2ac0af1b66e3b66307f53b1cc446514308ec466d<br>
> # skip: [0d2ebf9c3f7822e7ba3e4792ea3b6b19aa2da34a] mm/mmap: free vm_area_struct without call_rcu in exit_mmap<br>
> git bisect skip 0d2ebf9c3f7822e7ba3e4792ea3b6b19aa2da34a<br>
> # skip: [70d4cbc80c88251de0a5b3e8df3275901f1fa99a] powerc/mm: try VMA lock-based page fault handling first<br>
> git bisect skip 70d4cbc80c88251de0a5b3e8df3275901f1fa99a<br>
> # good: [444eeb17437a0ef526c606e9141a415d3b7dfddd] mm: prevent userfaults to be handled under per-vma lock<br>
> git bisect good 444eeb17437a0ef526c606e9141a415d3b7dfddd<br>
> # bad: [e06f47a16573decc57498f2d02f9af3bb3e84cf2] s390/mm: try VMA lock-based page fault handling first<br>
> git bisect bad e06f47a16573decc57498f2d02f9af3bb3e84cf2<br>
> # skip: [0bff0aaea03e2a3ed6bfa302155cca8a432a1829] x86/mm: try VMA lock-based page fault handling first<br>
> git bisect skip 0bff0aaea03e2a3ed6bfa302155cca8a432a1829<br>
> # skip: [cd7f176aea5f5929a09a91c661a26912cc995d1b] arm64/mm: try VMA lock-based page fault handling first<br>
> git bisect skip cd7f176aea5f5929a09a91c661a26912cc995d1b<br>
> # good: [52f238653e452e0fda61e880f263a173d219acd1] mm: introduce per-VMA lock statistics<br>
> git bisect good 52f238653e452e0fda61e880f263a173d219acd1<br>
> # bad: [c7f8f31c00d187a2c71a241c7f2bd6aa102a4e6f] mm: separate vma->lock from vm_area_struct<br>
> git bisect bad c7f8f31c00d187a2c71a241c7f2bd6aa102a4e6f<br>
> # only skipped commits left to test<br>
> # possible first bad commit: [c7f8f31c00d187a2c71a241c7f2bd6aa102a4e6f] mm: separate vma->lock from vm_area_struct<br>
> # possible first bad commit: [0d2ebf9c3f7822e7ba3e4792ea3b6b19aa2da34a] mm/mmap: free vm_area_struct without call_rcu in exit_mmap<br>
> # possible first bad commit: [70d4cbc80c88251de0a5b3e8df3275901f1fa99a] powerc/mm: try VMA lock-based page fault handling first<br>
> # possible first bad commit: [cd7f176aea5f5929a09a91c661a26912cc995d1b] arm64/mm: try VMA lock-based page fault handling first<br>
> # possible first bad commit: [0bff0aaea03e2a3ed6bfa302155cca8a432a1829] x86/mm: try VMA lock-based page fault handling first<br>
> <br>
> I do not usually see any kernel log output while running the program, just occasional logs about user space segfaults.<br>
<br>
See Bugzilla for the full thread.<br>
<br>
Jacob: Can you repeat bisection please? Why did you skip VMA lock-based<br>
page fault commits in your bisection?<br>
<br>
Anyway, I'm adding it to regzbot:<br>
<br>
#regzbot introduced: 0bff0aaea03e2a..c7f8f31c00d187 <a href="https://bugzilla.kernel.org/show_bug.cgi?id=217624#regzbot" rel="noreferrer noreferrer" target="_blank">https://bugzilla.kernel.org/show_bug.cgi?id=217624<br>
#regzbot</a> title: Memory corruption in multithreaded user space program while calling fork (possibly caused by trying VMA lock-based page fault)<br>
<br>
Thanks.<br>
<br>
[1]: <a href="https://bugzilla.kernel.org/show_bug.cgi?id=217624" rel="noreferrer noreferrer" target="_blank">https://bugzilla.kernel.org/show_bug.cgi?id=217624</a><br>
<br>
-- <br>
An old man doll... just what I always wanted! - Clara<br>
</blockquote></div>