{"thread":{"id":"62220","subject":"git commit hangs when run concurrently on macos","startedAt":"2024-09-30T20:58:43Z","lastAt":"2024-10-01T04:30:10Z","messageCount":2,"participants":["Bennet Huber","Eric Sunshine"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"503771","messageId":"CAJ7FMP0RqfUc0TKxJWn+tGq6tRhhABmbpd+xJMc7SRL6F5=JgA@mail.gmail.com","threadId":"62220","inReplyTo":null,"subject":"git commit hangs when run concurrently on macos","fromName":"Bennet Huber","fromEmail":"bennet.huber@gmail.com","sentAt":"2024-09-30T20:58:16Z","receivedAt":"2024-09-30T20:58:43Z","isPatch":false,"sender":{"key":"bennet.huber@gmail.com","avatar":null},"body":"Thank you for filling out a Git bug report!\nPlease answer the following questions to help us understand your issue.\n\nWhat did you do before the bug happened? (Steps to reproduce your issue)\n\nRan several \"git commit\" commands in parallel on separate repos.\nHere's a simple test that does that.\n\n```bash\ndir-foreach-parallel () {\n    subdirs=(\"${(@f)$(find ./ -maxdepth 1 -mindepth 1 -type d)}\")\n    for d in $subdirs\n    do\n        echo \"$d\"\n        pushd \"$d\" >/dev/null\n        $@ &\n        popd >/dev/null\n    done\n}\n\nmkdir -p /tmp/gitTest\ncd /tmp/gitTest\nfor ((i=0;i<100;i++)) do mkdir git_$i; done\n\nrunTest() {\n    git init\n    USE_FS_MONITOR=true\n    echo -e \"[user]\\n\\tname = Gopher\\n\\temail =\ngopher@example.com\\n[core]\\n\\tfsmonitor = $USE_FS_MONITOR\\n\"\n>.git/config\n    GIT_TRACE=1 git commit --allow-empty -m `pwd` &>gitci.log\n}\n\ndir-foreach-parallel runTest\n```\n\nAfter running this, wait for output to subside, then run `jobs`.\nOn my machine, I usually see 5-10 hung \"git commit\" jobs.\nRunning `ps -efww | grep \"git commit\"` will show the repos they belong\nto, which you can then use to\nexamine the corresponding gitci.log files.  The last line of all the\nfiles I examined for hung \"git\ncommit\" commands is:\n\n> 3:39:04.020533 run-command.c:758       trace: start_command: /opt/homebrew/opt/git/libexec/git-core/git fsmonitor--daemon run --detach --ipc-threads=8\n\nSetting `USE_FS_MONITOR=false` in the code above consistently results\nin no hung \"git commit\" commands, which leads me to suspect the\nproblem has to do with the fsmonitor feature.  It's also a decent,\nthough difficult to discover, workaround.\n\n\nWhat did you expect to happen? (Expected behavior)\nConcurrent git commands in separate repos shouldn't interfere with each other.\n\nWhat happened instead? (Actual behavior)\nSeveral \"git commit\" commands hung and failed to complete due to\nfsmonitor not starting.\n\nWhat's different between what you expected and what actually happened?\nSuccess\n\nAnything else you want to add:\n* I tried to repro this on a linux box and failed. It seems to have\nsomething to do with the macos\nfsmonitor implementation.\n  * `uname -a` of linux box that succeeds: Linux <hostname>\n6.8.0-1016-aws #17-Ubuntu SMP Mon Sep  2 13:48:07 UTC 2024 x86_64\nx86_64 x86_64 GNU/Linux\n* I'm running an ARM Apple M3 Pro chip, so maybe ARM specific?\n* The only mac I have access to is a corporate one, which has a ton of\nsecurity monitoring software running on it in the background.  It's\nconceivable this could be interfering with fsmonitor somehow.  I\nunfortunately can't test this on a clean mac.\n* This use case may seem contrived, but I actually lost a lot of time\nbecause of it.  I was trying\n  to run some unit tests in parallel which involved running \"git\ncommit\" commands.  It took me\na long time to figure out why they were randomly hanging.  In\naddition, I've previously used the `dir-foreach-parallel git\ncommit...` pattern for actual development, when working on software\nwhich requires changes across several dependent git repos.\n\nPlease review the rest of the bug report below.\nYou can delete any lines you don't wish to share.\n\n\n[System Info]\ngit version:\ngit version 2.46.2\ncpu: arm64\nno commit associated with this build\nsizeof-long: 8\nsizeof-size_t: 8\nshell-path: /bin/sh\nfeature: fsmonitor--daemon\nlibcurl: 8.4.0\nzlib: 1.2.12\nuname: Darwin 23.6.0 Darwin Kernel Version 23.6.0: Mon Jul 29 21:14:30\nPDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T6030 arm64\ncompiler info: clang: 15.0.0 (clang-1500.3.9.4)\nlibc info: no libc information available\n$SHELL (typically, interactive shell): /bin/zsh\n\n\n[Enabled Hooks]\nnot run from a git repository - no hooks to show\n"},{"id":"503798","messageId":"CAPig+cSBvU8Ne29SpV7fZeRPmU25EPQxxxmE+agyM303kDoPCA@mail.gmail.com","threadId":"62220","inReplyTo":"CAJ7FMP0RqfUc0TKxJWn+tGq6tRhhABmbpd+xJMc7SRL6F5=JgA@mail.gmail.com","subject":"Re: git commit hangs when run concurrently on macos","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2024-10-01T04:29:58Z","receivedAt":"2024-10-01T04:30:10Z","isPatch":false,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Sep 30, 2024 at 4:58 PM Bennet Huber <bennet.huber@gmail.com> wrote:\n> After running this, wait for output to subside, then run `jobs`.\n> On my machine, I usually see 5-10 hung \"git commit\" jobs.\n>\n> > 3:39:04.020533 run-command.c:758       trace: start_command: /opt/homebrew/opt/git/libexec/git-core/git fsmonitor--daemon run --detach --ipc-threads=8\n>\n> Setting `USE_FS_MONITOR=false` in the code above consistently results\n> in no hung \"git commit\" commands, which leads me to suspect the\n> problem has to do with the fsmonitor feature.  It's also a decent,\n> though difficult to discover, workaround.\n\nI don't use fsmonitor and haven't particularly been following the\nissue, but there are some active threads on the mailing list regarding\nfsmonitor hanging on macOS. Perhaps they are relevant to your case.\n\nhttps://lore.kernel.org/git/20240930194600.GA1805534@coredump.intra.peff.net/T/\n\nhttps://lore.kernel.org/git/20240509162219.GA1707955@coredump.intra.peff.net/T/\n\nhttps://lore.kernel.org/git/pull.1802.git.1727577690390.gitgitgadget@gmail.com/T/\n"}]}