{"thread":{"id":"62632","subject":"failed to read .git/worktrees/test-worktree-xdrphX/commondir: Success","startedAt":"2024-12-12T20:23:56Z","lastAt":"2024-12-14T13:41:37Z","messageCount":3,"participants":["Brendan Jackman","Phillip Wood"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"509037","messageId":"CAJUwWUMQF_tN_hyBMh+F4NtAHVAx1z+g0HYY=PD9-jd5RdcR9g@mail.gmail.com","threadId":"62632","inReplyTo":null,"subject":"failed to read .git/worktrees/test-worktree-xdrphX/commondir: Success","fromName":"Brendan Jackman","fromEmail":"bhenryj0117@gmail.com","sentAt":"2024-12-12T20:23:44Z","receivedAt":"2024-12-12T20:23:56Z","isPatch":false,"sender":{"key":"bhenryj0117@gmail.com","avatar":null},"body":"I am developing a tool that uses Git via its CLI. The tool creates a\nbunch of worktrees on startup and tears them down on shutdown.\n\nWhen running its tests, it does this a lot. Sometimes, the git\nworktree add command fails with exit code 128. The output just looks\nlike:\n\nfailed to read .git/worktrees/test-worktree-xdrphX/commondir: Success\n\nNaturally, whenever I try to strace anything, the bug stops\nreproducing. But, I have tried manually adding logging to my\napplication to see the Git commands it's running.\n\nThe test harness sets up a fresh repo and puts some commits in it,\nthen runs multiple instances of the application (on the same repo).\nThe one that fails seems to only be running these two commands:\n\n[\"-c\", \"color.ui=false\", \"log\", \"-n1\", \"--format=%H %T\", \"HEAD\"]\n[\"-c\", \"color.ui=false\", \"worktree\", \"add\",\n\"/tmp/limmat-child3qE96Q/worktrees/test-worktree-sGVqA8\", \"HEAD\"]\n\nAnd that second command fails with the error above.\n\nAny idea what's going wrong here? Is what I'm doing (i.e. creating\nworktrees in parallel) actually legit? Anything I can do to help\nnarrow down the issue?\n\n.:: Details\n\nI've built git from next on Ubuntu 24.04.1 LTS, here's the git\nbugreport output:\nhttps://gist.github.com/bjackman/3377add9e8d7b747d9f74b52518e9101/edit\n\nBut, I also experienced this issue with the Ubuntu packaged version of Git.\n\nThe application with the hacks to add logging of Git commands is here:\nhttps://github.com/bjackman/limmat/tree/git-bug\n\nHere's an example of the test failing in CI:\nhttps://github.com/bjackman/limmat/actions/runs/12303338582/job/34338222752#step:5:337\n\nIf you're willing to try and reproduce locally, in theory it should be\nenough to put your Git build in $PATH, clone my repo, install Cargo\n(https://doc.rust-lang.org/cargo/getting-started/installation.html),\nand then from the base of my repo run  `cargo test race` in a loop, in\nmy case it usually fails relatively quickly.\n\nMany thanks,\nBrendan\n"},{"id":"509106","messageId":"e56b3047-3af2-4353-a0ac-d3af65fe70eb@gmail.com","threadId":"62632","inReplyTo":"CAJUwWUMQF_tN_hyBMh+F4NtAHVAx1z+g0HYY=PD9-jd5RdcR9g@mail.gmail.com","subject":"Re: failed to read .git/worktrees/test-worktree-xdrphX/commondir: Success","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2024-12-13T14:53:22Z","receivedAt":"2024-12-13T14:53:26Z","isPatch":false,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Brendan\n\nOn 12/12/2024 20:23, Brendan Jackman wrote:\n> I am developing a tool that uses Git via its CLI. The tool creates a\n> bunch of worktrees on startup and tears them down on shutdown.\n> \n> When running its tests, it does this a lot. Sometimes, the git\n> worktree add command fails with exit code 128. The output just looks\n> like:\n> \n> failed to read .git/worktrees/test-worktree-xdrphX/commondir: Success\n> \n> Naturally, whenever I try to strace anything, the bug stops\n> reproducing.\n\nHow annoying, I suspect it is slowing things down enough that the race \ncondition triggering this error does not happen\n\n> [...]\n> Any idea what's going wrong here? Is what I'm doing (i.e. creating\n> worktrees in parallel) actually legit?\n\nUnfortunately with the current implementation of \"git worktree add\" it \nisn't. When git tries to add a new worktree it first checks to see if it \nalready exists. As a worktree may be on removable media it cannot simply \ncheck that the directory does not exist, it also has to check that there \nisn't a locked worktree at that path. To check for a locked worktree it \nhas to list all the worktrees in the repository and if another process \nis creating a new worktree at the same time the process trying the list \nthe worktrees ends up seeing an corrupt wortree because the other \nprocess hasn't finished creating all the required files yet. I think \nthis is fixable in principle by teaching the code that lists the \nworktrees to ignore \".git/worktrees/*.lock\" and creating new worktree \nunder \".git/worktrees/$id.lock\" and then renaming \"$id.lock\" to \"$id\" \nonce all the files are written. There are probably some subtleties \nthough as we would need to ensure the processes that are spawned to \nsetup the worktree use the temporary \".lock\" directory.\n\n> Anything I can do to help narrow down the issue?\n\nIf you're interested in fixing the problem the relevant code lives in \nworktree.c and builtin/worktree.c. The easiest solution is probably to \ntry introduce some locking in your program so you don't run two \ninstances of \"git worktree add\" at the same time.\n\nBest Wishes\n\nPhillip\n\n> .:: Details\n> \n> I've built git from next on Ubuntu 24.04.1 LTS, here's the git\n> bugreport output:\n> https://gist.github.com/bjackman/3377add9e8d7b747d9f74b52518e9101/edit\n> \n> But, I also experienced this issue with the Ubuntu packaged version of Git.\n> \n> The application with the hacks to add logging of Git commands is here:\n> https://github.com/bjackman/limmat/tree/git-bug\n> \n> Here's an example of the test failing in CI:\n> https://github.com/bjackman/limmat/actions/runs/12303338582/job/34338222752#step:5:337\n> \n> If you're willing to try and reproduce locally, in theory it should be\n> enough to put your Git build in $PATH, clone my repo, install Cargo\n> (https://doc.rust-lang.org/cargo/getting-started/installation.html),\n> and then from the base of my repo run  `cargo test race` in a loop, in\n> my case it usually fails relatively quickly.\n> \n> Many thanks,\n> Brendan\n> \n\n"},{"id":"509117","messageId":"CAJUwWUMWq6MA8YFYWk+B0st79gGRyud8PGx5ia5AkuZwVnrdMQ@mail.gmail.com","threadId":"62632","inReplyTo":"e56b3047-3af2-4353-a0ac-d3af65fe70eb@gmail.com","subject":"Re: failed to read .git/worktrees/test-worktree-xdrphX/commondir: Success","fromName":"Brendan Jackman","fromEmail":"bhenryj0117@gmail.com","sentAt":"2024-12-14T13:41:25Z","receivedAt":"2024-12-14T13:41:37Z","isPatch":false,"sender":{"key":"bhenryj0117@gmail.com","avatar":null},"body":"On Fri, Dec 13, 2024 at 3:53 PM Phillip Wood <phillip.wood123@gmail.com> wrote:\n> Unfortunately with the current implementation of \"git worktree add\" it\n> isn't. When git tries to add a new worktree it first checks to see if it\n> already exists. As a worktree may be on removable media it cannot simply\n> check that the directory does not exist, it also has to check that there\n> isn't a locked worktree at that path. To check for a locked worktree it\n> has to list all the worktrees in the repository and if another process\n> is creating a new worktree at the same time the process trying the list\n> the worktrees ends up seeing an corrupt wortree because the other\n> process hasn't finished creating all the required files yet. I think\n> this is fixable in principle by teaching the code that lists the\n> worktrees to ignore \".git/worktrees/*.lock\" and creating new worktree\n> under \".git/worktrees/$id.lock\" and then renaming \"$id.lock\" to \"$id\"\n> once all the files are written. There are probably some subtleties\n> though as we would need to ensure the processes that are spawned to\n> setup the worktree use the temporary \".lock\" directory.\n>\n> > Anything I can do to help narrow down the issue?\n>\n> If you're interested in fixing the problem the relevant code lives in\n> worktree.c and builtin/worktree.c. The easiest solution is probably to\n> try introduce some locking in your program so you don't run two\n> instances of \"git worktree add\" at the same time.\n\nOK, thanks very much for the info :)\n\nSo I think the right approach for me is:\n\n1. Do a silly retry-based workaround in my application to make the\ntests stop flaking (doing proper locking is actually nontrivial\nbecause the git commands get run across separate processes).\n2. If (big \"if\" unfortunately) and when I get the time, contribute a\nfix to Git as you suggested - thanks for the code pointers.\n3. Then I can just have my tests depend on an updated Git. I've never\nactually seen the issue in real usage of the application so I think\nthat's likely to be good enough.\n\nCheers and thanks again for the help,\nBrendan\n"}]}