threads / discuss / 62632

failed to read .git/worktrees/test-worktree-xdrphX/commondir: Success

Subject: failed to read .git/worktrees/test-worktree-xdrphX/commondir: Success

## tl;dr

3 messages between Dec 12, 2024 and Dec 14, 2024.

replies: 2people: 2as markdown or json

Brendan Jackman· Dec 12, 2024, 20:23 UTC · lore

I am developing a tool that uses Git via its CLI. The tool creates a bunch of worktrees on startup and tears them down on shutdown.

When running its tests, it does this a lot. Sometimes, the git worktree add command fails with exit code 128. The output just looks like:

failed to read .git/worktrees/test-worktree-xdrphX/commondir: Success

Naturally, whenever I try to strace anything, the bug stops reproducing. But, I have tried manually adding logging to my application to see the Git commands it's running.

The test harness sets up a fresh repo and puts some commits in it, then runs multiple instances of the application (on the same repo). The one that fails seems to only be running these two commands:

["-c", "color.ui=false", "log", "-n1", "--format=%H %T", "HEAD"] ["-c", "color.ui=false", "worktree", "add", "/tmp/limmat-child3qE96Q/worktrees/test-worktree-sGVqA8", "HEAD"]

And that second command fails with the error above.

Any idea what's going wrong here? Is what I'm doing (i.e. creating worktrees in parallel) actually legit? Anything I can do to help narrow down the issue?

.:: Details

I've built git from next on Ubuntu 24.04.1 LTS, here's the git bugreport output: https://gist.github.com/bjackman/3377add9e8d7b747d9f74b52518e9101/edit

But, I also experienced this issue with the Ubuntu packaged version of Git.

The application with the hacks to add logging of Git commands is here: https://github.com/bjackman/limmat/tree/git-bug

Here's an example of the test failing in CI: https://github.com/bjackman/limmat/actions/runs/12303338582/job/34338222752#step:5:337

If you're willing to try and reproduce locally, in theory it should be enough to put your Git build in $PATH, clone my repo, install Cargo (https://doc.rust-lang.org/cargo/getting-started/installation.html), and then from the base of my repo run `cargo test race` in a loop, in my case it usually fails relatively quickly.

Many thanks, Brendan

Phillip Wood· Dec 13, 2024, 14:53 UTC · re: Brendan Jackman · lore

Re: failed to read .git/worktrees/test-worktree-xdrphX/commondir: Success

Hi Brendan

On 12/12/2024 20:23, Brendan Jackman wrote:

Show 11 quoted lines

> I am developing a tool that uses Git via its CLI. The tool creates a
> bunch of worktrees on startup and tears them down on shutdown.
> 
> When running its tests, it does this a lot. Sometimes, the git
> worktree add command fails with exit code 128. The output just looks
> like:
> 
> failed to read .git/worktrees/test-worktree-xdrphX/commondir: Success
> 
> Naturally, whenever I try to strace anything, the bug stops
> reproducing.

How annoying, I suspect it is slowing things down enough that the race condition triggering this error does not happen

> [...]
> Any idea what's going wrong here? Is what I'm doing (i.e. creating
> worktrees in parallel) actually legit?

Unfortunately with the current implementation of "git worktree add" it isn't. When git tries to add a new worktree it first checks to see if it already exists. As a worktree may be on removable media it cannot simply check that the directory does not exist, it also has to check that there isn't a locked worktree at that path. To check for a locked worktree it has to list all the worktrees in the repository and if another process is creating a new worktree at the same time the process trying the list the worktrees ends up seeing an corrupt wortree because the other process hasn't finished creating all the required files yet. I think this is fixable in principle by teaching the code that lists the worktrees to ignore ".git/worktrees/*.lock" and creating new worktree under ".git/worktrees/$id.lock" and then renaming "$id.lock" to "$id" once all the files are written. There are probably some subtleties though as we would need to ensure the processes that are spawned to setup the worktree use the temporary ".lock" directory.

> Anything I can do to help narrow down the issue?

If you're interested in fixing the problem the relevant code lives in worktree.c and builtin/worktree.c. The easiest solution is probably to try introduce some locking in your program so you don't run two instances of "git worktree add" at the same time.

Best Wishes

Phillip

Show 23 quoted lines

> .:: Details
> 
> I've built git from next on Ubuntu 24.04.1 LTS, here's the git
> bugreport output:
> https://gist.github.com/bjackman/3377add9e8d7b747d9f74b52518e9101/edit
> 
> But, I also experienced this issue with the Ubuntu packaged version of Git.
> 
> The application with the hacks to add logging of Git commands is here:
> https://github.com/bjackman/limmat/tree/git-bug
> 
> Here's an example of the test failing in CI:
> https://github.com/bjackman/limmat/actions/runs/12303338582/job/34338222752#step:5:337
> 
> If you're willing to try and reproduce locally, in theory it should be
> enough to put your Git build in $PATH, clone my repo, install Cargo
> (https://doc.rust-lang.org/cargo/getting-started/installation.html),
> and then from the base of my repo run  `cargo test race` in a loop, in
> my case it usually fails relatively quickly.
> 
> Many thanks,
> Brendan
>

Brendan Jackman· Dec 14, 2024, 13:41 UTC · re: Phillip Wood · lore

Re: failed to read .git/worktrees/test-worktree-xdrphX/commondir: Success

On Fri, Dec 13, 2024 at 3:53 PM Phillip Wood <phillip.wood123@gmail.com> wrote:

Show 22 quoted lines

> Unfortunately with the current implementation of "git worktree add" it
> isn't. When git tries to add a new worktree it first checks to see if it
> already exists. As a worktree may be on removable media it cannot simply
> check that the directory does not exist, it also has to check that there
> isn't a locked worktree at that path. To check for a locked worktree it
> has to list all the worktrees in the repository and if another process
> is creating a new worktree at the same time the process trying the list
> the worktrees ends up seeing an corrupt wortree because the other
> process hasn't finished creating all the required files yet. I think
> this is fixable in principle by teaching the code that lists the
> worktrees to ignore ".git/worktrees/*.lock" and creating new worktree
> under ".git/worktrees/$id.lock" and then renaming "$id.lock" to "$id"
> once all the files are written. There are probably some subtleties
> though as we would need to ensure the processes that are spawned to
> setup the worktree use the temporary ".lock" directory.
>
> > Anything I can do to help narrow down the issue?
>
> If you're interested in fixing the problem the relevant code lives in
> worktree.c and builtin/worktree.c. The easiest solution is probably to
> try introduce some locking in your program so you don't run two
> instances of "git worktree add" at the same time.

OK, thanks very much for the info :)

So I think the right approach for me is:

1. Do a silly retry-based workaround in my application to make the
tests stop flaking (doing proper locking is actually nontrivial
because the git commands get run across separate processes).
2. If (big "if" unfortunately) and when I get the time, contribute a
fix to Git as you suggested - thanks for the code pointers.
3. Then I can just have my tests depend on an updated Git. I've never
actually seen the issue in real usage of the application so I think
that's likely to be good enough.

Cheers and thanks again for the help, Brendan