{"thread":{"id":"59127","subject":"[CI]: Is t7527 known to be flakey?","startedAt":"2023-01-20T02:52:06Z","lastAt":"2023-01-25T21:13:27Z","messageCount":8,"participants":["Junio C Hamano","Jeff Hostetler","SZEDER Gábor"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"470762","messageId":"xmqqtu0lzzn2.fsf@gitster.g","threadId":"59127","inReplyTo":null,"subject":"[CI]: Is t7527 known to be flakey?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-20T02:52:01Z","receivedAt":"2023-01-20T02:52:06Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"The said test failed its linux-musl job in its first attempt, but\nre-running the failed job passed.\n\n https://github.com/git/git/actions/runs/3963948890/jobs/6792356234\n (seen@e096683 attempt #1 linux-musl)\n\n https://github.com/git/git/actions/runs/3963948890/jobs/6792850313\n (seen@e096683 attempt #2 linux-musl)\n\n"},{"id":"470784","messageId":"d5e2cd60-49f7-91bf-0678-70e3671b1cad@jeffhostetler.com","threadId":"59127","inReplyTo":"xmqqtu0lzzn2.fsf@gitster.g","subject":"Re: [CI]: Is t7527 known to be flakey?","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2023-01-20T15:23:29Z","receivedAt":"2023-01-20T15:23:41Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 1/19/23 9:52 PM, Junio C Hamano wrote:\n> The said test failed its linux-musl job in its first attempt, but\n> re-running the failed job passed.\n> \n> https://github.com/git/git/actions/runs/3963948890/jobs/6792356234\n> (seen@e096683 attempt #1 linux-musl)\n> \n> https://github.com/git/git/actions/runs/3963948890/jobs/6792850313\n> (seen@e096683 attempt #2 linux-musl)\n> \n\nThis is on Linux, so it would be using the linux inotify backend.\nLet me add Eric to the \"To:\" line for visibility. And see if he\nhas experienced this during his development of it.\n\nI've not looked at the inotify code so I can't say if there are\nraces there or not. Tests that move directories feel like good\ncandidates for race conditions -- since the daemon doesn't get a\nrecursive view of the tree with inotify() and must simulate that\nand manage the individual directories, but again I don't want to\nassume that.\n\nJeff\n\n"},{"id":"470788","messageId":"xmqqmt6dxlhv.fsf@gitster.g","threadId":"59127","inReplyTo":"d5e2cd60-49f7-91bf-0678-70e3671b1cad@jeffhostetler.com","subject":"Re: [CI]: Is t7527 known to be flakey?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-20T15:40:28Z","receivedAt":"2023-01-20T15:40:52Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Hostetler writes:\n\n> This is on Linux, so it would be using the linux inotify backend.\n> Let me add Eric to the \"To:\" line for visibility.\n\nThanks for redirection.\n"},{"id":"470837","messageId":"20230121102355.GA2155@szeder.dev","threadId":"59127","inReplyTo":"xmqqtu0lzzn2.fsf@gitster.g","subject":"Re: [CI]: Is t7527 known to be flakey?","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2023-01-21T10:23:55Z","receivedAt":"2023-01-21T10:24:41Z","isPatch":false,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Jan 19, 2023 at 06:52:01PM -0800, Junio C Hamano wrote:\n> The said test failed its linux-musl job in its first attempt, but\n> re-running the failed job passed.\n> \n> https://github.com/git/git/actions/runs/3963948890/jobs/6792356234\n> (seen@e096683 attempt #1 linux-musl)\n> \n> https://github.com/git/git/actions/runs/3963948890/jobs/6792850313\n> (seen@e096683 attempt #2 linux-musl)\n\nt7527 is quite slow, even with the right selection of test cases, but\nthis little tweak makes it much faster:\n\n diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\n index 0e497ba98d..4210ef644c 100755\n --- a/t/t7527-builtin-fsmonitor.sh\n +++ b/t/t7527-builtin-fsmonitor.sh\n @@ -676,7 +676,10 @@ test_expect_success 'cleanup worktrees' '\n # cause incorrect results when the untracked-cache is enabled.\n \n test_lazy_prereq UNTRACKED_CACHE '\n -\tgit update-index --test-untracked-cache\n +\t# This check takes a very long time, but I know it works on\n +\t# my system, so let's fake it.\n +\t#git update-index --test-untracked-cache\n +\ttrue\n '\n \n test_expect_success 'Matrix: setup for untracked-cache,fsmonitor matrix' '\n\nThis with the right selection of test cases makes --stress\npracticable, and the test tends to fail after a handful of\nrepetitions:\n\n ./t7527-builtin-fsmonitor.sh --stress -r 8,23-58\n\nI saw different failures in multiple test cases, e.g.:\n\nUnexpected output in case 55:\n\n expecting success of 7527.55 'Matrix[uc:true][fsm:true] move_directory_contents_deeper': \n \t\tmatrix_clean_up_repo &&\n \t\t$fn &&\n \t\tif test $uc = false && test $fsm = false\n \t\tthen\n \t\t\tgit status --porcelain=v1 >.git/expect.$fn\n \t\telse\n \t\t\tgit status --porcelain=v1 >.git/actual.$fn &&\n \t\t\ttest_cmp .git/expect.$fn .git/actual.$fn\n \t\tfi\n \t\n + matrix_clean_up_repo\n + git reset --hard HEAD\n HEAD is now at 1d1edcb initial\n + git clean -fd\n + move_directory_contents_deeper\n + mkdir T1/_new_\n + mv T1/F1 T1/F2 T1/T2 T1/_new_\n + test true = false\n + git status --porcelain=v1\n error: read error: Connection reset by peer\n error: could not read IPC response\n + test_cmp .git/expect.move_directory_contents_deeper .git/actual.move_directory_contents_deeper\n + test 2 -ne 2\n + eval diff -u \"$@\"\n + diff -u .git/expect.move_directory_contents_deeper .git/actual.move_directory_contents_deeper\n --- .git/expect.move_directory_contents_deeper\t2023-01-21 09:47:12.677410349 +0000\n +++ .git/actual.move_directory_contents_deeper\t2023-01-21 09:47:14.045448573 +0000\n @@ -7,3 +7,4 @@\n D T1/T2/T3/T4/F1\n D T1/T2/T3/T4/F2\n ?? T1/_new_/\n +?? dir1\n error: last command exited with $?=1\n not ok 55 - Matrix[uc:true][fsm:true] move_directory_contents_deeper\n\nSIGPIPE in 'git status' cases 42, 43 and 55:\n\n expecting success of 7527.42 'Matrix[uc:false][fsm:true] move_directory_contents_deeper':\n matrix_clean_up_repo &&\n $fn &&\n if test $uc = false && test $fsm = false\n then\n git status --porcelain=v1 >.git/expect.$fn\n else\n git status --porcelain=v1 >.git/actual.$fn &&\n test_cmp .git/expect.$fn .git/actual.$fn\n fi\n \n + matrix_clean_up_repo\n + git reset --hard HEAD\n HEAD is now at 1d1edcb initial\n + git clean -fd\n + move_directory_contents_deeper\n + mkdir T1/_new_\n + mv T1/F1 T1/F2 T1/T2 T1/_new_\n + test false = false\n + test true = false\n + git status --porcelain=v1\n error: last command exited with $?=141\n not ok 42 - Matrix[uc:false][fsm:true] move_directory_contents_deeper\n\n expecting success of 7527.43 'Matrix[uc:false][fsm:true] move_directory_up': \n \t\tmatrix_clean_up_repo &&\n \t\t$fn &&\n \t\tif test $uc = false && test $fsm = false\n \t\tthen\n \t\t\tgit status --porcelain=v1 >.git/expect.$fn\n \t\telse\n \t\t\tgit status --porcelain=v1 >.git/actual.$fn &&\n \t\t\ttest_cmp .git/expect.$fn .git/actual.$fn\n \t\tfi\n \t\n + matrix_clean_up_repo\n + git reset --hard HEAD\n HEAD is now at 1d1edcb initial\n + git clean -fd\n Removing T1/_new_/\n + move_directory_up\n + mv T1/T2/T3 T1\n + test false = false\n + test true = false\n + git status --porcelain=v1\n error: last command exited with $?=141\n not ok 43 - Matrix[uc:false][fsm:true] move_directory_up\n\n expecting success of 7527.55 'Matrix[uc:true][fsm:true] move_directory_contents_deeper':\n matrix_clean_up_repo &&\n $fn &&\n if test $uc = false && test $fsm = false\n then\n git status --porcelain=v1 >.git/expect.$fn\n else\n git status --porcelain=v1 >.git/actual.$fn &&\n test_cmp .git/expect.$fn .git/actual.$fn\n fi\n \n + matrix_clean_up_repo\n + git reset --hard HEAD\n HEAD is now at 1d1edcb initial\n + git clean -fd\n + move_directory_contents_deeper\n + mkdir T1/_new_\n + mv T1/F1 T1/F2 T1/T2 T1/_new_\n + test true = false\n + git status --porcelain=v1\n error: last command exited with $?=141\n\nI find it interesting that the output of 'git status' is redirected to\na file in all these cases, and yet it gets a SIGPIPE.\n\n\nI also saw the test hang a couple of times, e.g.:\n\n $ ./t7527-builtin-fsmonitor.sh --stress -r 8,23-58\n OK 6.1\n OK 7.1\n OK 1.1\n OK 2.1\n OK 3.1\n OK 5.1\n OK 4.1\n OK 0.1\n OK 6.2\n OK 1.2\n OK 2.2\n OK 7.2\n OK 5.2\n OK 0.2\n OK 4.2\n OK 6.3\n OK 7.3\n OK 2.3\n OK 0.3\n OK 4.3\n OK 6.4\n OK 7.4\n OK 2.4\n OK 0.4\n OK 4.4\n OK 6.5\n OK 7.5\n OK 2.5\n OK 0.5\n OK 4.5\n OK 6.6\n OK 7.6\n OK 2.6\n OK 0.6\n OK 4.6\n OK 6.7\n OK 7.7\n OK 2.7\n OK 0.7\n OK 4.7\n OK 6.8\n OK 7.8\n OK 2.8\n OK 0.8\n OK 4.8\n OK 6.9\n OK 7.9\n OK 2.9\n OK 0.9\n OK 4.9\n OK 6.10\n OK 7.10\n OK 2.10\n OK 0.10\n OK 4.10\n OK 6.11\n OK 7.11\n OK 2.11\n OK 0.11\n OK 4.11\n OK 6.12\n OK 7.12\n OK 2.12\n OK 0.12\n OK 4.12\n OK 6.13\n OK 7.13\n OK 2.13\n OK 0.13\n OK 6.14\n OK 4.13\n OK 7.14\n OK 2.14\n OK 0.14\n OK 6.15\n OK 4.14\n OK 2.15\n OK 7.15\n OK 0.15\n FAIL 7.16\n OK 6.16\n OK 2.16\n OK 4.15\n OK 0.16\n\nAt this point the test script should print the log of the failed job,\nbut it hangs instead, as there are a number of stuck fsmonitor--daemon\nand status processes (notice how the stress test starts with 8 jobs,\nbut the last repetition only has 4):\n\n $ ps aux |grep git\n szeder 1857100 0.0 0.1 72272 4452 pts/2 Sl+ 21:40 0:00 /home/szeder/src/git/git fsmonitor--daemon run --detach --ipc-threads=8\n szeder 1857779 0.0 0.1 6560 4152 pts/2 S+ 21:40 0:00 /home/szeder/src/git/git status --porcelain=v1\n szeder 1860020 0.0 0.1 88664 4312 pts/2 Sl+ 21:40 0:00 /home/szeder/src/git/git fsmonitor--daemon run --detach --ipc-threads=8\n szeder 1860668 0.0 0.1 6560 4040 pts/2 S+ 21:40 0:00 /home/szeder/src/git/git status --porcelain=v1\n szeder 1860749 0.0 0.1 96860 4528 pts/2 Sl+ 21:40 0:00 /home/szeder/src/git/git fsmonitor--daemon run --detach --ipc-threads=8\n szeder 1861281 0.0 0.1 6560 4272 pts/2 S+ 21:40 0:00 /home/szeder/src/git/git status --porcelain=v1\n\n"},{"id":"470932","messageId":"f7449e73-7f50-67ea-2be4-2037f98a69f3@jeffhostetler.com","threadId":"59127","inReplyTo":"20230121102355.GA2155@szeder.dev","subject":"Re: [CI]: Is t7527 known to be flakey?","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2023-01-23T16:56:53Z","receivedAt":"2023-01-23T16:57:03Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 1/21/23 5:23 AM, SZEDER Gábor wrote:\n> On Thu, Jan 19, 2023 at 06:52:01PM -0800, Junio C Hamano wrote:\n>> The said test failed its linux-musl job in its first attempt, but\n>> re-running the failed job passed.\n>>\n>> https://github.com/git/git/actions/runs/3963948890/jobs/6792356234\n>> (seen@e096683 attempt #1 linux-musl)\n>>\n>> https://github.com/git/git/actions/runs/3963948890/jobs/6792850313\n>> (seen@e096683 attempt #2 linux-musl)\n> \n> t7527 is quite slow, even with the right selection of test cases, but\n> this little tweak makes it much faster:\n> \n> diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\n> index 0e497ba98d..4210ef644c 100755\n> --- a/t/t7527-builtin-fsmonitor.sh\n> +++ b/t/t7527-builtin-fsmonitor.sh\n> @@ -676,7 +676,10 @@ test_expect_success 'cleanup worktrees' '\n> # cause incorrect results when the untracked-cache is enabled.\n> \n> test_lazy_prereq UNTRACKED_CACHE '\n> -\tgit update-index --test-untracked-cache\n> +\t# This check takes a very long time, but I know it works on\n> +\t# my system, so let's fake it.\n> +\t#git update-index --test-untracked-cache\n> +\ttrue\n> '\n> \n> test_expect_success 'Matrix: setup for untracked-cache,fsmonitor matrix' '\n> \n> This with the right selection of test cases makes --stress\n> practicable, and the test tends to fail after a handful of\n> repetitions:\n> \n> ./t7527-builtin-fsmonitor.sh --stress -r 8,23-58\n> \n> I saw different failures in multiple test cases, e.g.:\n> \n> Unexpected output in case 55:\n> \n> expecting success of 7527.55 'Matrix[uc:true][fsm:true] move_directory_contents_deeper':\n> \t\tmatrix_clean_up_repo &&\n> \t\t$fn &&\n> \t\tif test $uc = false && test $fsm = false\n> \t\tthen\n> \t\t\tgit status --porcelain=v1 >.git/expect.$fn\n> \t\telse\n> \t\t\tgit status --porcelain=v1 >.git/actual.$fn &&\n> \t\t\ttest_cmp .git/expect.$fn .git/actual.$fn\n> \t\tfi\n> \t\n> + matrix_clean_up_repo\n> + git reset --hard HEAD\n> HEAD is now at 1d1edcb initial\n> + git clean -fd\n> + move_directory_contents_deeper\n> + mkdir T1/_new_\n> + mv T1/F1 T1/F2 T1/T2 T1/_new_\n> + test true = false\n> + git status --porcelain=v1\n> error: read error: Connection reset by peer\n> error: could not read IPC response\n> + test_cmp .git/expect.move_directory_contents_deeper .git/actual.move_directory_contents_deeper\n> + test 2 -ne 2\n> + eval diff -u \"$@\"\n> + diff -u .git/expect.move_directory_contents_deeper .git/actual.move_directory_contents_deeper\n> --- .git/expect.move_directory_contents_deeper\t2023-01-21 09:47:12.677410349 +0000\n> +++ .git/actual.move_directory_contents_deeper\t2023-01-21 09:47:14.045448573 +0000\n> @@ -7,3 +7,4 @@\n> D T1/T2/T3/T4/F1\n> D T1/T2/T3/T4/F2\n> ?? T1/_new_/\n> +?? dir1\n> error: last command exited with $?=1\n> not ok 55 - Matrix[uc:true][fsm:true] move_directory_contents_deeper\n> \n> SIGPIPE in 'git status' cases 42, 43 and 55:\n> \n> expecting success of 7527.42 'Matrix[uc:false][fsm:true] move_directory_contents_deeper':\n> matrix_clean_up_repo &&\n> $fn &&\n> if test $uc = false && test $fsm = false\n> then\n> git status --porcelain=v1 >.git/expect.$fn\n> else\n> git status --porcelain=v1 >.git/actual.$fn &&\n> test_cmp .git/expect.$fn .git/actual.$fn\n> fi\n> \n> + matrix_clean_up_repo\n> + git reset --hard HEAD\n> HEAD is now at 1d1edcb initial\n> + git clean -fd\n> + move_directory_contents_deeper\n> + mkdir T1/_new_\n> + mv T1/F1 T1/F2 T1/T2 T1/_new_\n> + test false = false\n> + test true = false\n> + git status --porcelain=v1\n> error: last command exited with $?=141\n> not ok 42 - Matrix[uc:false][fsm:true] move_directory_contents_deeper\n> \n> expecting success of 7527.43 'Matrix[uc:false][fsm:true] move_directory_up':\n> \t\tmatrix_clean_up_repo &&\n> \t\t$fn &&\n> \t\tif test $uc = false && test $fsm = false\n> \t\tthen\n> \t\t\tgit status --porcelain=v1 >.git/expect.$fn\n> \t\telse\n> \t\t\tgit status --porcelain=v1 >.git/actual.$fn &&\n> \t\t\ttest_cmp .git/expect.$fn .git/actual.$fn\n> \t\tfi\n> \t\n> + matrix_clean_up_repo\n> + git reset --hard HEAD\n> HEAD is now at 1d1edcb initial\n> + git clean -fd\n> Removing T1/_new_/\n> + move_directory_up\n> + mv T1/T2/T3 T1\n> + test false = false\n> + test true = false\n> + git status --porcelain=v1\n> error: last command exited with $?=141\n> not ok 43 - Matrix[uc:false][fsm:true] move_directory_up\n> \n> expecting success of 7527.55 'Matrix[uc:true][fsm:true] move_directory_contents_deeper':\n> matrix_clean_up_repo &&\n> $fn &&\n> if test $uc = false && test $fsm = false\n> then\n> git status --porcelain=v1 >.git/expect.$fn\n> else\n> git status --porcelain=v1 >.git/actual.$fn &&\n> test_cmp .git/expect.$fn .git/actual.$fn\n> fi\n> \n> + matrix_clean_up_repo\n> + git reset --hard HEAD\n> HEAD is now at 1d1edcb initial\n> + git clean -fd\n> + move_directory_contents_deeper\n> + mkdir T1/_new_\n> + mv T1/F1 T1/F2 T1/T2 T1/_new_\n> + test true = false\n> + git status --porcelain=v1\n> error: last command exited with $?=141\n> \n> I find it interesting that the output of 'git status' is redirected to\n> a file in all these cases, and yet it gets a SIGPIPE.\n\nThe `git status` command talks to the running `git fsmonitor--daemon`\non a Unix domain socket (or a Named Pipe on Windows), so a SIGPIPE is\npossible.\n\nWas this on Linux or MacOS ?\n\nJeff\n\n\n\n\n> \n> \n> I also saw the test hang a couple of times, e.g.:\n> \n> $ ./t7527-builtin-fsmonitor.sh --stress -r 8,23-58\n> OK 6.1\n> OK 7.1\n> OK 1.1\n> OK 2.1\n> OK 3.1\n> OK 5.1\n> OK 4.1\n> OK 0.1\n> OK 6.2\n> OK 1.2\n> OK 2.2\n> OK 7.2\n> OK 5.2\n> OK 0.2\n> OK 4.2\n> OK 6.3\n> OK 7.3\n> OK 2.3\n> OK 0.3\n> OK 4.3\n> OK 6.4\n> OK 7.4\n> OK 2.4\n> OK 0.4\n> OK 4.4\n> OK 6.5\n> OK 7.5\n> OK 2.5\n> OK 0.5\n> OK 4.5\n> OK 6.6\n> OK 7.6\n> OK 2.6\n> OK 0.6\n> OK 4.6\n> OK 6.7\n> OK 7.7\n> OK 2.7\n> OK 0.7\n> OK 4.7\n> OK 6.8\n> OK 7.8\n> OK 2.8\n> OK 0.8\n> OK 4.8\n> OK 6.9\n> OK 7.9\n> OK 2.9\n> OK 0.9\n> OK 4.9\n> OK 6.10\n> OK 7.10\n> OK 2.10\n> OK 0.10\n> OK 4.10\n> OK 6.11\n> OK 7.11\n> OK 2.11\n> OK 0.11\n> OK 4.11\n> OK 6.12\n> OK 7.12\n> OK 2.12\n> OK 0.12\n> OK 4.12\n> OK 6.13\n> OK 7.13\n> OK 2.13\n> OK 0.13\n> OK 6.14\n> OK 4.13\n> OK 7.14\n> OK 2.14\n> OK 0.14\n> OK 6.15\n> OK 4.14\n> OK 2.15\n> OK 7.15\n> OK 0.15\n> FAIL 7.16\n> OK 6.16\n> OK 2.16\n> OK 4.15\n> OK 0.16\n> \n> At this point the test script should print the log of the failed job,\n> but it hangs instead, as there are a number of stuck fsmonitor--daemon\n> and status processes (notice how the stress test starts with 8 jobs,\n> but the last repetition only has 4):\n> \n> $ ps aux |grep git\n> szeder 1857100 0.0 0.1 72272 4452 pts/2 Sl+ 21:40 0:00 /home/szeder/src/git/git fsmonitor--daemon run --detach --ipc-threads=8\n> szeder 1857779 0.0 0.1 6560 4152 pts/2 S+ 21:40 0:00 /home/szeder/src/git/git status --porcelain=v1\n> szeder 1860020 0.0 0.1 88664 4312 pts/2 Sl+ 21:40 0:00 /home/szeder/src/git/git fsmonitor--daemon run --detach --ipc-threads=8\n> szeder 1860668 0.0 0.1 6560 4040 pts/2 S+ 21:40 0:00 /home/szeder/src/git/git status --porcelain=v1\n> szeder 1860749 0.0 0.1 96860 4528 pts/2 Sl+ 21:40 0:00 /home/szeder/src/git/git fsmonitor--daemon run --detach --ipc-threads=8\n> szeder 1861281 0.0 0.1 6560 4272 pts/2 S+ 21:40 0:00 /home/szeder/src/git/git status --porcelain=v1\n> \n"},{"id":"470944","messageId":"20230123181216.GB2155@szeder.dev","threadId":"59127","inReplyTo":"f7449e73-7f50-67ea-2be4-2037f98a69f3@jeffhostetler.com","subject":"Re: [CI]: Is t7527 known to be flakey?","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2023-01-23T18:12:16Z","receivedAt":"2023-01-23T18:12:23Z","isPatch":false,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Jan 23, 2023 at 11:56:53AM -0500, Jeff Hostetler wrote:\n> Was this on Linux or MacOS ?\n\nOn an average-ish Linux (an Ubuntu LTS variant). So the issue is not\nspecific to musl.\n\n"},{"id":"471032","messageId":"7fa84280-db1d-8618-571a-ce0ac7a26135@jeffhostetler.com","threadId":"59127","inReplyTo":"20230123181216.GB2155@szeder.dev","subject":"Re: [CI]: Is t7527 known to be flakey?","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2023-01-25T19:02:40Z","receivedAt":"2023-01-25T19:02:46Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 1/23/23 1:12 PM, SZEDER Gábor wrote:\n> On Mon, Jan 23, 2023 at 11:56:53AM -0500, Jeff Hostetler wrote:\n>> Was this on Linux or MacOS ?\n> \n> On an average-ish Linux (an Ubuntu LTS variant). So the issue is not\n> specific to musl.\n> \n\nOK, thanks. I wasn't worried about \"musl\", but rather whether\nyou were running the stress test on a Linux or Mac.\n\nSince they have different backends (inotify vs FSEvent) and all\nthe code that touches the filesystem is different, it would best\nto start on the correct OS when trying to repro it.\n\n\nCan you tell from your stess test whether the fsmonitor-daemon\nis crashing? (It might be subtle since the daemon is auto-started\nif necessary, so it might be crashing and silently getting restarted\nby the next command.)\n\nI ask because a SIGPIPE in the client would make me think that the\nserver suddenly closed the connection unexpectedly, like if it had\nSIGSEGV'd or something.\n\nI won't have time to spin up a Linux VM until next week, so I\nwon't be able to investigate this for a bit.\n\nJeff\n"},{"id":"471033","messageId":"20230125211244.GC2155@szeder.dev","threadId":"59127","inReplyTo":"7fa84280-db1d-8618-571a-ce0ac7a26135@jeffhostetler.com","subject":"Re: [CI]: Is t7527 known to be flakey?","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2023-01-25T21:12:44Z","receivedAt":"2023-01-25T21:13:27Z","isPatch":false,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Jan 25, 2023 at 02:02:40PM -0500, Jeff Hostetler wrote:\n> Can you tell from your stess test whether the fsmonitor-daemon\n> is crashing? (It might be subtle since the daemon is auto-started\n> if necessary, so it might be crashing and silently getting restarted\n> by the next command.)\n> \n> I ask because a SIGPIPE in the client would make me think that the\n> server suddenly closed the connection unexpectedly, like if it had\n> SIGSEGV'd or something.\n\nLast time around I only looked at the failing test case, and didn't\nnotice anything that might have indicated the cause of the SIGPIPE.\nThis time I chanced to look a bit further up in the test log, and:\n\n expecting success of 7527.55 'Matrix[uc:true][fsm:true] move_directory_contents_deeper':\n matrix_clean_up_repo &&\n $fn &&\n if test $uc = false && test $fsm = false\n then\n git status --porcelain=v1 >.git/expect.$fn\n else\n git status --porcelain=v1 >.git/actual.$fn &&\n test_cmp .git/expect.$fn .git/actual.$fn\n fi\n \n + matrix_clean_up_repo\n + git reset --hard HEAD\n HEAD is now at 1d1edcb initial\n + git clean -fd\n + move_directory_contents_deeper\n + mkdir T1/_new_\n + mv T1/F1 T1/F2 T1/T2 T1/_new_\n + test true = false\n + git status --porcelain=v1\n error: read error: Connection reset by peer\n error: could not read IPC response\n + test_cmp .git/expect.move_directory_contents_deeper .git/actual.move_directory_contents_deeper\n + test 2 -ne 2\n + eval diff -u \"$@\"\n + diff -u .git/expect.move_directory_contents_deeper .git/actual.move_directory_contents_deeper\n ok 55 - Matrix[uc:true][fsm:true] move_directory_contents_deeper\n \n expecting success of 7527.56 'Matrix[uc:true][fsm:true] move_directory_up':\n matrix_clean_up_repo &&\n $fn &&\n if test $uc = false && test $fsm = false\n then\n git status --porcelain=v1 >.git/expect.$fn\n else\n git status --porcelain=v1 >.git/actual.$fn &&\n test_cmp .git/expect.$fn .git/actual.$fn\n fi\n \n + matrix_clean_up_repo\n + git reset --hard HEAD\n HEAD is now at 1d1edcb initial\n + git clean -fd\n Removing T1/_new_/\n + move_directory_up\n + mv T1/T2/T3 T1\n + test true = false\n + git status --porcelain=v1\n error: last command exited with $?=141\n not ok 56 - Matrix[uc:true][fsm:true] move_directory_up\n\nNotice that \"error: read error: Connection reset by peer\" in the\nprevious, still successful test case! I ran it a couple of times, and\nsaw the same error message in the still successful '42 -\nMatrix[uc:false][fsm:true] move_directory_contents_deeper' followed by\na SIGPIPE caused failure in the next test case. And now that I knew\nwhat to look for, I noticed this error message in the very first test\nfailure I reported the other day, which didn't fail because of\nSIGPIPE, and in that case the error message was printed in the failed\ntest case.\n\nAnd there were a few cases that failed because of SIGPIPE but there\nwere no error messages at all.\n\nI can't say what caused these errors, but I doubt that anything\nsegfaulted, because segfaults are logged in syslog, and I haven't\nfound any such syslog entries coinciding with stress testing.\n\n\n"}]}