"lock file exists" when fetching in bare clone of repository

> The fetch fails, returning status 255, with the below error message:
> 
>         remote: Enumerating objects: 18733, done.
>         remote: Counting objects: 100% (562/562), done.
>         remote: Compressing objects: 100% (448/448), done.
>         remote: Total 18733 (delta 399), reused 144 (delta 112), pack-reused 18171
>         Receiving objects: 100% (18733/18733), 25.32 MiB | 7.93 MiB/s, done.
>         Resolving deltas: 100% (14608/14608), done.
>         From <VALID-URL>
> 
>         [.... quite a number of branches and tags .....]
> 
>         error: cannot lock ref 'refs/heads/ptv-2164': Unable to create '/Users/joe.drew/tmp/./refs/heads/ptv-2164.lock': File exists.

> 
>         Another git process seems to be running in this repository, e.g.
>         an editor opened by 'git commit'. Please make sure all processes
>         are terminated then try again. If it still fails, a git process
>         may have crashed in this repository earlier:
>         remove the file manually to continue.
> 
> This file doesn't exist; in fact, no such files exist:
> 
>         $ ls refs/heads
>         $
> 
> However, the branch _does_ exist on the remote.

> On Wed, Aug 20, 2025 at 08:54:37PM +0000, Joe Drew wrote:
> 
> > The fetch fails, returning status 255, with the below error message:
> > 
> >         remote: Enumerating objects: 18733, done.
> >         remote: Counting objects: 100% (562/562), done.
> >         remote: Compressing objects: 100% (448/448), done.
> >         remote: Total 18733 (delta 399), reused 144 (delta 112), pack-reused 18171
> >         Receiving objects: 100% (18733/18733), 25.32 MiB | 7.93 MiB/s, done.
> >         Resolving deltas: 100% (14608/14608), done.
> >         From <VALID-URL>
> > 
> >         [.... quite a number of branches and tags .....]
> > 
> >         error: cannot lock ref 'refs/heads/ptv-2164': Unable to create '/Users/joe.drew/tmp/./refs/heads/ptv-2164.lock': File exists.
> 
> > 
> >         Another git process seems to be running in this repository, e.g.
> >         an editor opened by 'git commit'. Please make sure all processes
> >         are terminated then try again. If it still fails, a git process
> >         may have crashed in this repository earlier:
> >         remove the file manually to continue.
> > 
> > This file doesn't exist; in fact, no such files exist:
> > 
> >         $ ls refs/heads
> >         $
> > 
> > However, the branch _does_ exist on the remote.
> 
> This is a wild guess, but: are there any case collisions with that
> branch name (e.g., PTV-2164 or something) in the upstream repo?
> 
> If so, and assuming you're on a case-insensitive filesystem, then the
> lock files would collide. You wouldn't see anything after the fact
> because git-fetch itself would create the colliding lockfile, and then
> clean it up after hitting the fatal error.

> If that is the case, you can try using the reftables backend in v2.51.0.
> It doesn't use the filesystem for its ref storage or locking. Something
> like:
> 
>   git init --bare --ref-format=reftable

> > This error _does not_ happen in the Apple-supplied version of git:
> > `git version 2.39.5 (Apple Git-154)`, but does in 2.51.0, which I
> > installed with homebrew. (If this is a packaging error, I'll happily
> > report to homebrew.)
> 
> That is definitely weird, and not something I'd expect if it's just a
> case collision. Is it possible for you to build Git from source? If so,
> and the problem happens with your build of 2.51.0 but not v2.39.5, it
> would be very enlightening to see the results of "git bisect". We can
> provide more guidance if you need with that process.

> > > This error _does not_ happen in the Apple-supplied version of git:
> > > `git version 2.39.5 (Apple Git-154)`, but does in 2.51.0, which I
> > > installed with homebrew. (If this is a packaging error, I'll happily
> > > report to homebrew.)
> > 
> > That is definitely weird, and not something I'd expect if it's just a
> > case collision. Is it possible for you to build Git from source? If so,
> > and the problem happens with your build of 2.51.0 but not v2.39.5, it
> > would be very enlightening to see the results of "git bisect". We can
> > provide more guidance if you need with that process.
> 
> Curious indeed. Another relevant part could be the new batched mode that
> we have recently introduced in git-fetch(1). Previously we would've
> updated refs one by one, and potentially that led to us writing the same
> ref multiple times due to case-insensitivity? The result would be quite
> broken in that case, but at least the fetch would've finished. Does make
> me wonder though what subsequent fetches did, and whether we always tend
> to flip-flop the value of such a ref.

> But with batched transactions that's not possible anymore, as we would
> indeed try to lock the same ref twice. Assuming that this really is the
> case, I wonder whether we should detect this case, tell the user that
> it's impossible to store all refs on their system, and then continue
> regardless while ejecting that specific ref from the transaction.

> On Thu, Aug 21, 2025 at 09:15:47AM +0200, Patrick Steinhardt wrote:
> 
> > > > This error _does not_ happen in the Apple-supplied version of git:
> > > > `git version 2.39.5 (Apple Git-154)`, but does in 2.51.0, which I
> > > > installed with homebrew. (If this is a packaging error, I'll happily
> > > > report to homebrew.)
> > > 
> > > That is definitely weird, and not something I'd expect if it's just a
> > > case collision. Is it possible for you to build Git from source? If so,
> > > and the problem happens with your build of 2.51.0 but not v2.39.5, it
> > > would be very enlightening to see the results of "git bisect". We can
> > > provide more guidance if you need with that process.
> > 
> > Curious indeed. Another relevant part could be the new batched mode that
> > we have recently introduced in git-fetch(1). Previously we would've
> > updated refs one by one, and potentially that led to us writing the same
> > ref multiple times due to case-insensitivity? The result would be quite
> > broken in that case, but at least the fetch would've finished. Does make
> > me wonder though what subsequent fetches did, and whether we always tend
> > to flip-flop the value of such a ref.
> 
> Ah, good catch. That would explain the behavior change perfectly. My money is
> on a case collision, then. :)
> 
> > But with batched transactions that's not possible anymore, as we would
> > indeed try to lock the same ref twice. Assuming that this really is the
> > case, I wonder whether we should detect this case, tell the user that
> > it's impossible to store all refs on their system, and then continue
> > regardless while ejecting that specific ref from the transaction.
> 
> Maybe it is too optimistic to think that they will just start using
> reftable in Git 3.0, and we can avoid extra code to detect this?

>     migrate your repository to use the "reftable" backend with the
>     following command:
>
>         git refs migrate --ref-format=reftable
>
>     Please keep in mind that not all implementations of Git support this
>     new format yet. So if you use tools other than Git to access this
>     repository it may not be an option to migrate to reftables.
>
> The last part is what I'm working on right now. libgit2 is fully
> functional now, but what is still lacking is reviews.
>
> Patrick

> Patrick Steinhardt <ps@pks.im> writes:
> 
> >     You're on a case-insensitive filesystem, and the remote you are
> >     trying to fetch from has references that only differ in casing. It
> >     is impossible to store such references with the "files" backend. You
> 
> "backend." -> "backend on your system."
> 
> >     can either accept this as-is, in which case you won't be able to
> >     store all remote references on disk. Or you can alternatively
> 
> I do not see the former as a viable choice, though.  When this
> happens, the clone or fetch fails and the user cannot catch up to
> the upstream development, no?  You have to futz with the fetch
> refspec to cause refs your filesystem cannot store ignored in order
> to make progress on other refs, but that is making the user do more
> than accepting this as-is.

> On Thu, Aug 21, 2025 at 09:05:30AM -0700, Junio C Hamano wrote:
>> Patrick Steinhardt <ps@pks.im> writes:
>>
>> >     You're on a case-insensitive filesystem, and the remote you are
>> >     trying to fetch from has references that only differ in casing. It
>> >     is impossible to store such references with the "files" backend. You
>>
>> "backend." -> "backend on your system."
>>
>> >     can either accept this as-is, in which case you won't be able to
>> >     store all remote references on disk. Or you can alternatively
>>
>> I do not see the former as a viable choice, though.  When this
>> happens, the clone or fetch fails and the user cannot catch up to
>> the upstream development, no?  You have to futz with the fetch
>> refspec to cause refs your filesystem cannot store ignored in order
>> to make progress on other refs, but that is making the user do more
>> than accepting this as-is.
>
> Oh, yes, right now it's too involved. What I'm proposing is to mark the
> transaction as allowed-to-fail, and in that case we'd be able to fetch
> and store refs in this case again. The result would still be broken, but
> it would be broken in a similar way as before. There's one difference
> though: we'd accept the _first_ conflicting ref now, whereas before we
> accepted the _last_ conflicting ref.

> The fix itself isn't too involved:
>
> diff --git a/refs/files-backend.c b/refs/files-backend.c
> index 088b52c740..5c31b02e6b 100644
> --- a/refs/files-backend.c
> +++ b/refs/files-backend.c
> @@ -776,6 +776,8 @@ static enum ref_transaction_error
> lock_raw_ref(struct files_ref_store *refs,
>             goto retry;
>         } else {
>             unable_to_lock_message(ref_file.buf, myerr, err);
> +			if (myerr == EEXIST)
> +				ret = REF_TRANSACTION_ERROR_NAME_CONFLICT;
>             goto error_return;
>         }
>     }

> Karthik Nayak <karthik.188@gmail.com> writes:
>
>> The fix itself isn't too involved:
>>
>> diff --git a/refs/files-backend.c b/refs/files-backend.c
>> index 088b52c740..5c31b02e6b 100644
>> --- a/refs/files-backend.c
>> +++ b/refs/files-backend.c
>> @@ -776,6 +776,8 @@ static enum ref_transaction_error
>> lock_raw_ref(struct files_ref_store *refs,
>>             goto retry;
>>         } else {
>>             unable_to_lock_message(ref_file.buf, myerr, err);
>> +			if (myerr == EEXIST)
>> +				ret = REF_TRANSACTION_ERROR_NAME_CONFLICT;
>>             goto error_return;
>>         }
>>     }
>

> Without this change we'd return REF_TRANSACTION_ERROR_GENERIC; does
> the caller treat NAME_CONFLICT any specially?  Or is the "fix" you
> talk about just about giving a different message and no other
> behaviour changes involved?  I guess a more specific message that is
> 99% of the time more correct is an improvement over an overly
> generic "some error happened" message.  But I thought the original
> issue was that the user cannot make any progress when the ref
> updates are transactional.  Does returning NAME_CONFLICT from here
> tell the machinery that we are allowed to break transactional
> semantics somehow?
>

> In any case, I wonder if refs_verify_refnames_available() should
> notice that we are using files backend on a case challenged
> filesystem, and change the behaviour a bit.  This additional check
> needs to be implemented as a backend call via refs->be->something()
> to tighten the outcome.  It appears to me that the function in the
> current form does not even notice a D/F conflict when there is a
> branch 'd' and the transaction requests to create a new branch 'D/f'
> on a case challenged system if files backend is in use, since the
> function is in the generic layer and behaves case sensitively (which
> is the right thing to do---we are talking about detecting backend
> specific glitches here).
>
> Thanks.
>

> This error type is bubbled up to `files_transaction_prepare()` which
> tries to lock each reference update by calling `lock_ref_for_update()`.
> So if the locking fails, we check if the rejection type can be ignored,
> which is done by calling `ref_transaction_maybe_set_rejected()`.
>
> Only during batched updates would errors be ignore and only for
> non-generic errors. So this change would specifically only apply for
> batched updates. Currently that is used by:
>
>     1. git fetch
>     2. git receive-pack
>     3. git update-ref --batch-updates
>
> And for all three scenarios I think it makes sense to add this in.

> Karthik Nayak <karthik.188@gmail.com> writes:
>
>> This error type is bubbled up to `files_transaction_prepare()` which
>> tries to lock each reference update by calling `lock_ref_for_update()`.
>> So if the locking fails, we check if the rejection type can be ignored,
>> which is done by calling `ref_transaction_maybe_set_rejected()`.
>>
>> Only during batched updates would errors be ignore and only for
>> non-generic errors. So this change would specifically only apply for
>> batched updates. Currently that is used by:
>>
>>     1. git fetch
>>     2. git receive-pack
>>     3. git update-ref --batch-updates
>>
>> And for all three scenarios I think it makes sense to add this in.
>
> Thanks for an extra explanation.
>
> These pieces of information would be necessary in the proposed log
> message when it becomes a real patch.

>> But with batched transactions that's not possible anymore, as we would
>> indeed try to lock the same ref twice. Assuming that this really is the
>> case, I wonder whether we should detect this case, tell the user that
>> it's impossible to store all refs on their system, and then continue
>> regardless while ejecting that specific ref from the transaction.
>
> Maybe it is too optimistic to think that they will just start using
> reftable in Git 3.0, and we can avoid extra code to detect this?

> If that is the case, you can try using the reftables backend in v2.51.0.
> It doesn't use the filesystem for its ref storage or locking. Something
> like:
>
>  git init --bare --ref-format=reftable

>> This error _does not_ happen in the Apple-supplied version of git:
>> `git version 2.39.5 (Apple Git-154)`, but does in 2.51.0, which I
>> installed with homebrew. (If this is a packaging error, I'll happily
>> report to homebrew.)
>
> That is definitely weird, and not something I'd expect if it's just a
> case collision. Is it possible for you to build Git from source? If so,
> and the problem happens with your build of 2.51.0 but not v2.39.5, it
> would be very enlightening to see the results of "git bisect". We can
> provide more guidance if you need with that process.

> If so, and assuming you're on a case-insensitive filesystem, then the
> lock files would collide. You wouldn't see anything after the fact
> because git-fetch itself would create the colliding lockfile, and then
> clean it up after hitting the fatal error.
> 
> If that is the case, you can try using the reftables backend in v2.51.0.
> It doesn't use the filesystem for its ref storage or locking. Something
> like:
> 
>   git init --bare --ref-format=reftable

> I'm sorry but I do not have an email client I could use to post to the
> git mailing list.
> Please could you post my thoughts on this to the list.
>
> https://lore.kernel.org/git/20250820213323.GA1667633@coredump.intra.peff.net/
>
> I use git 2.51.0 built using Macports on a very old version of macOS
>
> There are a couple of repos on github that were showing this problem:
> vectorgraphics/asymptote
> ghc/ghc
>
> Both of them were bare clones.
>
> I applied  --ref-format=reftable to the vectorgraphics/asymptote repo
> and a fetch
> went through to the end successfully
>
> The only snag I can see is that cgit no longer works. It says :
> Failed to open github/vectorgraphics/asymptote: No such file or directory
>
> The version of cgit is about 5 years old

> On Wed, Aug 20, 2025 at 05:33:23PM -0400, Jeff King wrote:
> > I'm sorry but I do not have an email client I could use to post to the
> > git mailing list.
> > Please could you post my thoughts on this to the list.
> >
> > https://lore.kernel.org/git/20250820213323.GA1667633@coredump.intra.peff.net/
> >
> > I use git 2.51.0 built using Macports on a very old version of macOS
> >
> > There are a couple of repos on github that were showing this problem:
> > vectorgraphics/asymptote
> > ghc/ghc
> >
> > Both of them were bare clones.
> >
> > I applied  --ref-format=reftable to the vectorgraphics/asymptote repo
> > and a fetch
> > went through to the end successfully
> >
> > The only snag I can see is that cgit no longer works. It says :
> > Failed to open github/vectorgraphics/asymptote: No such file or directory
> >
> > The version of cgit is about 5 years old
> 
> I think that's all expected. Unlike most systems which call Git commands
> like for-each-ref, etc, to access the ref store, cgit is linked directly
> against Git's C code (something we don't support, but the cgit authors
> accepted as a risk). So an old version of cgit will link against an old
> version of Git, and won't know about reftable at all.
> 
> It _might_ not even be able to detect the directory as a git repo,
> though I thought the reftable format was designed to let even old
> versions of Git see the repo (e.g., the dummy value in HEAD).