{"thread":{"id":"63501","subject":"HEAD.lock and git maintenance","startedAt":"2025-05-22T16:54:10Z","lastAt":"2025-05-28T06:51:15Z","messageCount":4,"participants":["david asraf","Patrick Steinhardt","Emily Shaffer"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"518671","messageId":"CANi7bVAkNc+gY1NoXfJuDRjxjZLTgL8Lfn8_ZmWsvLAoiLPkNg@mail.gmail.com","threadId":"63501","inReplyTo":null,"subject":"HEAD.lock and git maintenance","fromName":"david asraf","fromEmail":"dasraf9@gmail.com","sentAt":"2025-05-22T16:53:58Z","receivedAt":"2025-05-22T16:54:10Z","isPatch":false,"sender":{"key":"dasraf9@gmail.com","avatar":null},"body":"Thank you for filling out a Git bug report!\n\nPlease answer the following questions to help us understand your issue.\n\nWhat did you do before the bug happened? (Steps to reproduce your issue)\n\nWe have a system that runs many git commands on a local repo connected\nto a remote repo on GitHub via HTTPS. Our system creates many commits\nand works with many un-staged files. Every once in a while, we run the\nfollowing sequence of commands:\n\ngit stash --all\n\ngit checkout b1\n\ngit remote -v\n\ngit fetch\n\ngit status --branch --porcelain=v1 -u\n\ngit checkout b2\n\ngit stash pop\n\nWe start this sequence from branch b1 and record the output for internal use.\n\nWhat did you expect to happen? (Expected behavior)\n\nWe expected git checkout b2 to succeed consistently.\n\nWhat happened instead? (Actual behavior)\n\ngit checkout b2 sometimes fails because the HEAD.lock file already exists.\n\nWhat's different between what you expected and what actually happened?\n\nThe git checkout b2 command, which previously succeeded consistently,\nnow occasionally fails due to the presence of a HEAD.lock file. This\nissue started occurring after upgrading Git from version 2.39.5 to\n2.47.2.\n\nAnything else you want to add:\n\nUsing GIT_TRACE_PERFORMANCE, we noticed that a Git maintenance process\n(/usr/libexec/git-core/git maintenance run --auto --no-quiet --detach)\nsometimes starts after the git fetch command, occasionally in detached\nmode. We suspect this operation is causing the issue because we've\nverified that the git maintenance command requires HEAD.lock before it\nstarts running. We are considering setting maintenance.autoDetach to\nfalse. We are unsure if this is a bug or if it is working as intended,\nand would appreciate your comments on this.\n\nThanks, David.\n\nPlease review the rest of the bug report below.\n\nYou can delete any lines you don't wish to share.\n\n[System Info]\n\ngit version: 2.47.2\n"},{"id":"518926","messageId":"aDRq6oIgkSfAepcP@pks.im","threadId":"63501","inReplyTo":"CANi7bVAkNc+gY1NoXfJuDRjxjZLTgL8Lfn8_ZmWsvLAoiLPkNg@mail.gmail.com","subject":"Re: HEAD.lock and git maintenance","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-05-26T13:21:46Z","receivedAt":"2025-05-26T13:21:53Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nOn Thu, May 22, 2025 at 07:53:58PM +0300, david asraf wrote:\n> Thank you for filling out a Git bug report!\n> \n> Please answer the following questions to help us understand your issue.\n> \n> What did you do before the bug happened? (Steps to reproduce your issue)\n> \n> We have a system that runs many git commands on a local repo connected\n> to a remote repo on GitHub via HTTPS. Our system creates many commits\n> and works with many un-staged files. Every once in a while, we run the\n> following sequence of commands:\n> \n> git stash --all\n> \n> git checkout b1\n> \n> git remote -v\n> \n> git fetch\n> \n> git status --branch --porcelain=v1 -u\n> \n> git checkout b2\n> \n> git stash pop\n> \n> We start this sequence from branch b1 and record the output for internal use.\n> \n> What did you expect to happen? (Expected behavior)\n> \n> We expected git checkout b2 to succeed consistently.\n> \n> What happened instead? (Actual behavior)\n> \n> git checkout b2 sometimes fails because the HEAD.lock file already exists.\n> \n> What's different between what you expected and what actually happened?\n> \n> The git checkout b2 command, which previously succeeded consistently,\n> now occasionally fails due to the presence of a HEAD.lock file. This\n> issue started occurring after upgrading Git from version 2.39.5 to\n> 2.47.2.\n> \n> Anything else you want to add:\n> \n> Using GIT_TRACE_PERFORMANCE, we noticed that a Git maintenance process\n> (/usr/libexec/git-core/git maintenance run --auto --no-quiet --detach)\n> sometimes starts after the git fetch command, occasionally in detached\n> mode. We suspect this operation is causing the issue because we've\n> verified that the git maintenance command requires HEAD.lock before it\n> starts running. We are considering setting maintenance.autoDetach to\n> false. We are unsure if this is a bug or if it is working as intended,\n> and would appreciate your comments on this.\n\nthanks for your report! A couple months ago there was a similar\ndiscussion with someone else, but I cannot find that thread anymore,\nunfortunately.\n\nThe root cause here is repository maintenance with `--auto --detach`\nwill detach before spawning git-gc(1). This command may decide to pack\nyour references and thus cause them to be locked. This then triggers a\nrace condition, where the next Git command that wants to modify refs may\nnot be able to lock \"packed-refs\" because we are still busy repacking\nthem.\n\nThe actual timeout to lock the \"packed-refs\" file is configurable via\n\"core.packedRefsTimeout\", so bumping this value may make the problem\nless likely to happen. But it's only papering over the actual issue.\n\nI'll send a patch series soonish that fixes this issue. I think the\nsolution would be to make git-maintenance(1) learn about tasks that\nshould run previous and after daemonizing the process to avoid this race\ncondition. The effect would be that the caller of auto-maintenance will\nnot continue before refs have been packed, which is similar to what\ngit-gc(1) used to do in the past.\n\nPatrick\n"},{"id":"519002","messageId":"CAJoAoZ=OGOWVWQJNSk0YAVA0V_O68Y4ycXdw6d8bJ0=OhnNGeQ@mail.gmail.com","threadId":"63501","inReplyTo":"aDRq6oIgkSfAepcP@pks.im","subject":"Re: HEAD.lock and git maintenance","fromName":"Emily Shaffer","fromEmail":"nasamuffin@google.com","sentAt":"2025-05-27T16:33:50Z","receivedAt":"2025-05-27T16:34:04Z","isPatch":false,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Mon, May 26, 2025 at 6:22 AM Patrick Steinhardt <ps@pks.im> wrote:\n>\n> Hi,\n>\n> On Thu, May 22, 2025 at 07:53:58PM +0300, david asraf wrote:\n> > Thank you for filling out a Git bug report!\n> >\n> > Please answer the following questions to help us understand your issue.\n> >\n> > What did you do before the bug happened? (Steps to reproduce your issue)\n> >\n> > We have a system that runs many git commands on a local repo connected\n> > to a remote repo on GitHub via HTTPS. Our system creates many commits\n> > and works with many un-staged files. Every once in a while, we run the\n> > following sequence of commands:\n> >\n> > git stash --all\n> >\n> > git checkout b1\n> >\n> > git remote -v\n> >\n> > git fetch\n> >\n> > git status --branch --porcelain=v1 -u\n> >\n> > git checkout b2\n> >\n> > git stash pop\n> >\n> > We start this sequence from branch b1 and record the output for internal use.\n> >\n> > What did you expect to happen? (Expected behavior)\n> >\n> > We expected git checkout b2 to succeed consistently.\n> >\n> > What happened instead? (Actual behavior)\n> >\n> > git checkout b2 sometimes fails because the HEAD.lock file already exists.\n> >\n> > What's different between what you expected and what actually happened?\n> >\n> > The git checkout b2 command, which previously succeeded consistently,\n> > now occasionally fails due to the presence of a HEAD.lock file. This\n> > issue started occurring after upgrading Git from version 2.39.5 to\n> > 2.47.2.\n> >\n> > Anything else you want to add:\n> >\n> > Using GIT_TRACE_PERFORMANCE, we noticed that a Git maintenance process\n> > (/usr/libexec/git-core/git maintenance run --auto --no-quiet --detach)\n> > sometimes starts after the git fetch command, occasionally in detached\n> > mode. We suspect this operation is causing the issue because we've\n> > verified that the git maintenance command requires HEAD.lock before it\n> > starts running. We are considering setting maintenance.autoDetach to\n> > false. We are unsure if this is a bug or if it is working as intended,\n> > and would appreciate your comments on this.\n>\n> thanks for your report! A couple months ago there was a similar\n> discussion with someone else, but I cannot find that thread anymore,\n> unfortunately.\n\nGoogle had a big problem with this behavior about a year ago, I'm not\nsure if we got far with a thread about it though. That may be what\nyou're thinking of.\n\n>\n> The root cause here is repository maintenance with `--auto --detach`\n> will detach before spawning git-gc(1). This command may decide to pack\n> your references and thus cause them to be locked. This then triggers a\n> race condition, where the next Git command that wants to modify refs may\n> not be able to lock \"packed-refs\" because we are still busy repacking\n> them.\n>\n> The actual timeout to lock the \"packed-refs\" file is configurable via\n> \"core.packedRefsTimeout\", so bumping this value may make the problem\n> less likely to happen. But it's only papering over the actual issue.\n>\n> I'll send a patch series soonish that fixes this issue. I think the\n> solution would be to make git-maintenance(1) learn about tasks that\n> should run previous and after daemonizing the process to avoid this race\n> condition. The effect would be that the caller of auto-maintenance will\n> not continue before refs have been packed, which is similar to what\n> git-gc(1) used to do in the past.\n\nWe'll look forward to this series with interest, thanks.\n\n - Emily\n\n>\n> Patrick\n>\n"},{"id":"519046","messageId":"aDayWsRRA3ur-Pwi@pks.im","threadId":"63501","inReplyTo":"CAJoAoZ=OGOWVWQJNSk0YAVA0V_O68Y4ycXdw6d8bJ0=OhnNGeQ@mail.gmail.com","subject":"Re: HEAD.lock and git maintenance","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-05-28T06:51:06Z","receivedAt":"2025-05-28T06:51:15Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, May 27, 2025 at 09:33:50AM -0700, Emily Shaffer wrote:\n> On Mon, May 26, 2025 at 6:22 AM Patrick Steinhardt <ps@pks.im> wrote:\n[snip]\n> > > Using GIT_TRACE_PERFORMANCE, we noticed that a Git maintenance process\n> > > (/usr/libexec/git-core/git maintenance run --auto --no-quiet --detach)\n> > > sometimes starts after the git fetch command, occasionally in detached\n> > > mode. We suspect this operation is causing the issue because we've\n> > > verified that the git maintenance command requires HEAD.lock before it\n> > > starts running. We are considering setting maintenance.autoDetach to\n> > > false. We are unsure if this is a bug or if it is working as intended,\n> > > and would appreciate your comments on this.\n> >\n> > thanks for your report! A couple months ago there was a similar\n> > discussion with someone else, but I cannot find that thread anymore,\n> > unfortunately.\n> \n> Google had a big problem with this behavior about a year ago, I'm not\n> sure if we got far with a thread about it though. That may be what\n> you're thinking of.\n\nYeah, I remembered it was Google that had problems, but I thought that\nthis was at max half a year ago. So I didn't care to look back further\nthan that :)\n\n> > The root cause here is repository maintenance with `--auto --detach`\n> > will detach before spawning git-gc(1). This command may decide to pack\n> > your references and thus cause them to be locked. This then triggers a\n> > race condition, where the next Git command that wants to modify refs may\n> > not be able to lock \"packed-refs\" because we are still busy repacking\n> > them.\n> >\n> > The actual timeout to lock the \"packed-refs\" file is configurable via\n> > \"core.packedRefsTimeout\", so bumping this value may make the problem\n> > less likely to happen. But it's only papering over the actual issue.\n> >\n> > I'll send a patch series soonish that fixes this issue. I think the\n> > solution would be to make git-maintenance(1) learn about tasks that\n> > should run previous and after daemonizing the process to avoid this race\n> > condition. The effect would be that the caller of auto-maintenance will\n> > not continue before refs have been packed, which is similar to what\n> > git-gc(1) used to do in the past.\n> \n> We'll look forward to this series with interest, thanks.\n\nFor reference, I've sent the series yesterday via [1]. I'll keep you\nCc'd on that series from now on.\n\nPatrick\n\n[1]: <20250527-b4-pks-maintenance-ref-lock-race-v1-0-e1ceb2dea66e@pks.im>\n"}]}