# Possible race condition with git-rebase + .git/index.lock 14 messages from 2019-03-11 to 2022-10-28. Participants: Sergio Durigan Junior, Eric Sunshine, Duy Nguyen, Elijah Newren, Phillip Wood, Jeff King, Mingun. Thread: https://gitlist.dev/t/50712 ## Sergio Durigan Junior, 2019-03-11 21:38 Subject: Possible race condition with git-rebase + .git/index.lock Message-ID: <87k1h55bx0.fsf@sergiodj.net> URL: https://gitlist.dev/e/87k1h55bx0.fsf%40sergiodj.net ``` Hi there, I've been bit by this bug many times, and always forget to report it here, but this time I finally remembered. The scenario is this: - I have a local repository which tracks the upstream of the project. - I have 120+ local commits. - Every week or so, I have to rebase my local commits on top of upstream's origin/master. The workflow is simple: # git fetch origin && git rebase origin/master This works without problems most of the time (well, usually there are conflicts and all, but that's a burden I have to carry). However, sometimes I notice that git fails with: # git rebase origin/master ... Applying: commitX Applying: commitY Applying: commitZ fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists. The first thing I did was to check whether the index.lock file existed, but it doesn't. I also I thought it might have something to do with "commitZ", but after I ran "git rebase --abort" and tried rebasing again, I noticed that it would stop at another commit. This seems like a race condition, but after inspecting builtin/rebase.c it doesn't seem like the "git am" commands are run in parallel (and it wouldn't make sense if they did, anyway). I tried using GIT_TRACE, but haven't seen any useful information in the logs. I tried using the "next" branch, and found that the problem exists there as well. I wonder if anyone else has seen this problem. Thanks, -- Sergio GPG key ID: 237A 54B1 0287 28BF 00EF 31F4 D0EB 7628 65FC 5E36 Please send encrypted e-mail if possible http://sergiodj.net/ ``` ## Eric Sunshine, 2019-03-11 22:50 Subject: Re: Possible race condition with git-rebase + .git/index.lock Message-ID: URL: https://gitlist.dev/e/CAPig%2BcS_NLgwr6F2OqMpMZ3GBO-Cyru3G3UBxiT4ULJ_V1Bqvg%40mail.gmail.com In-Reply-To: <87k1h55bx0.fsf@sergiodj.net> ``` On Mon, Mar 11, 2019 at 6:17 PM Sergio Durigan Junior wrote: > # git rebase origin/master > ... > Applying: commitX > Applying: commitY > Applying: commitZ > fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists. > > The first thing I did was to check whether the index.lock file existed, > but it doesn't. What platform is this on? If Windows, I'm wondering if something (such as a virus scanner) is holding the lock file open long enough to prevent Git from actually deleting it (and perhaps Git doesn't notice the failed deletion -- until the next time it tries to take the lock). ``` ## Sergio Durigan Junior, 2019-03-11 22:53 Subject: Re: Possible race condition with git-rebase + .git/index.lock Message-ID: <87bm2h58fy.fsf@sergiodj.net> URL: https://gitlist.dev/e/87bm2h58fy.fsf%40sergiodj.net In-Reply-To: ``` On Monday, March 11 2019, Eric Sunshine wrote: > On Mon, Mar 11, 2019 at 6:17 PM Sergio Durigan Junior > wrote: >> # git rebase origin/master >> ... >> Applying: commitX >> Applying: commitY >> Applying: commitZ >> fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists. >> >> The first thing I did was to check whether the index.lock file existed, >> but it doesn't. > > What platform is this on? If Windows, I'm wondering if something (such > as a virus scanner) is holding the lock file open long enough to > prevent Git from actually deleting it (and perhaps Git doesn't notice > the failed deletion -- until the next time it tries to take the lock). This is on GNU/Linux (Fedora 28, to be specific). I haven't had the chance to test this on other distros, unfortunately. And since it seems to be a race condition, it's kind of hard to perform tests that trigger this. Thanks, -- Sergio GPG key ID: 237A 54B1 0287 28BF 00EF 31F4 D0EB 7628 65FC 5E36 Please send encrypted e-mail if possible http://sergiodj.net/ ``` ## Duy Nguyen, 2019-03-12 10:28 Subject: Re: Possible race condition with git-rebase + .git/index.lock Message-ID: URL: https://gitlist.dev/e/CACsJy8ANLkz%3D3cj1dAuHdUCkrgQzos%3D90EEG0n901o3QAp3PUQ%40mail.gmail.com In-Reply-To: <87k1h55bx0.fsf@sergiodj.net> ``` On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior wrote: > This works without problems most of the time (well, usually there are > conflicts and all, but that's a burden I have to carry). However, > sometimes I notice that git fails with: > > # git rebase origin/master > ... > Applying: commitX > Applying: commitY > Applying: commitZ > fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists. > > The first thing I did was to check whether the index.lock file existed, > but it doesn't. Is the output this clean? What I'm looking for is signs of automatic garbage collection kicking in the middle of the rebase. Something like "Auto packing the repository blah blah for optimum performance". -- Duy ``` ## Sergio Durigan Junior, 2019-03-12 16:45 Subject: Re: Possible race condition with git-rebase + .git/index.lock Message-ID: <877ed459eh.fsf@sergiodj.net> URL: https://gitlist.dev/e/877ed459eh.fsf%40sergiodj.net In-Reply-To: ``` On Tuesday, March 12 2019, Duy Nguyen wrote: > On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior > wrote: >> This works without problems most of the time (well, usually there are >> conflicts and all, but that's a burden I have to carry). However, >> sometimes I notice that git fails with: >> >> # git rebase origin/master >> ... >> Applying: commitX >> Applying: commitY >> Applying: commitZ >> fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists. >> >> The first thing I did was to check whether the index.lock file existed, >> but it doesn't. > > Is the output this clean? What I'm looking for is signs of automatic > garbage collection kicking in the middle of the rebase. Something like > "Auto packing the repository blah blah for optimum performance". Yeah, this is the exact output. I also thought about "git gc", but I don't see any output related to it. Is it possible that it's being executed in the background and not printing anything? Thanks, -- Sergio GPG key ID: 237A 54B1 0287 28BF 00EF 31F4 D0EB 7628 65FC 5E36 Please send encrypted e-mail if possible http://sergiodj.net/ ``` ## Elijah Newren, 2019-03-12 19:23 Subject: Re: Possible race condition with git-rebase + .git/index.lock Message-ID: URL: https://gitlist.dev/e/CABPp-BFnxhiXfvZUZndD-_htMEw0bZzrLRFpAF9u5YV3wi6qnA%40mail.gmail.com In-Reply-To: <877ed459eh.fsf@sergiodj.net> ``` On Tue, Mar 12, 2019 at 9:48 AM Sergio Durigan Junior wrote: > On Tuesday, March 12 2019, Duy Nguyen wrote: > > > On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior > > wrote: > >> This works without problems most of the time (well, usually there are > >> conflicts and all, but that's a burden I have to carry). However, > >> sometimes I notice that git fails with: > >> > >> # git rebase origin/master > >> ... > >> Applying: commitX > >> Applying: commitY > >> Applying: commitZ > >> fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists. > >> > >> The first thing I did was to check whether the index.lock file existed, > >> but it doesn't. > > > > Is the output this clean? What I'm looking for is signs of automatic > > garbage collection kicking in the middle of the rebase. Something like > > "Auto packing the repository blah blah for optimum performance". > > Yeah, this is the exact output. I also thought about "git gc", but I > don't see any output related to it. Is it possible that it's being > executed in the background and not printing anything? Any chance you have a cronjob or other task execution mechanism that is running git commands in various directories (even simple commands like 'git status' since it's not read-only contrary to what some expect, or 'git fetch' which might trigger a background gc)? ``` ## Sergio Durigan Junior, 2019-03-12 19:32 Subject: Re: Possible race condition with git-rebase + .git/index.lock Message-ID: <8736nr6g94.fsf@sergiodj.net> URL: https://gitlist.dev/e/8736nr6g94.fsf%40sergiodj.net In-Reply-To: ``` On Tuesday, March 12 2019, Elijah Newren wrote: > On Tue, Mar 12, 2019 at 9:48 AM Sergio Durigan Junior > wrote: >> On Tuesday, March 12 2019, Duy Nguyen wrote: >> >> > On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior >> > wrote: >> >> This works without problems most of the time (well, usually there are >> >> conflicts and all, but that's a burden I have to carry). However, >> >> sometimes I notice that git fails with: >> >> >> >> # git rebase origin/master >> >> ... >> >> Applying: commitX >> >> Applying: commitY >> >> Applying: commitZ >> >> fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists. >> >> >> >> The first thing I did was to check whether the index.lock file existed, >> >> but it doesn't. >> > >> > Is the output this clean? What I'm looking for is signs of automatic >> > garbage collection kicking in the middle of the rebase. Something like >> > "Auto packing the repository blah blah for optimum performance". >> >> Yeah, this is the exact output. I also thought about "git gc", but I >> don't see any output related to it. Is it possible that it's being >> executed in the background and not printing anything? > > Any chance you have a cronjob or other task execution mechanism that > is running git commands in various directories (even simple commands > like 'git status' since it's not read-only contrary to what some > expect, or 'git fetch' which might trigger a background gc)? Nope, nothing like this. AFAIK, nothing is touching that repository at the same time that I am. Besides, even if I wait some minutes before trying again, the bug manifests again. -- Sergio GPG key ID: 237A 54B1 0287 28BF 00EF 31F4 D0EB 7628 65FC 5E36 Please send encrypted e-mail if possible http://sergiodj.net/ ``` ## Elijah Newren, 2019-03-12 19:39 Subject: Re: Possible race condition with git-rebase + .git/index.lock Message-ID: URL: https://gitlist.dev/e/CABPp-BH2k2yz-MrEvN2dX1jLVJydxOBv-bu-H5M_SXO_aLK8hw%40mail.gmail.com In-Reply-To: <8736nr6g94.fsf@sergiodj.net> ``` On Tue, Mar 12, 2019 at 12:32 PM Sergio Durigan Junior wrote: > > On Tuesday, March 12 2019, Elijah Newren wrote: > > > On Tue, Mar 12, 2019 at 9:48 AM Sergio Durigan Junior > > wrote: > >> On Tuesday, March 12 2019, Duy Nguyen wrote: > >> > >> > On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior > >> > wrote: > >> >> This works without problems most of the time (well, usually there are > >> >> conflicts and all, but that's a burden I have to carry). However, > >> >> sometimes I notice that git fails with: > >> >> > >> >> # git rebase origin/master > >> >> ... > >> >> Applying: commitX > >> >> Applying: commitY > >> >> Applying: commitZ > >> >> fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists. > >> >> > >> >> The first thing I did was to check whether the index.lock file existed, > >> >> but it doesn't. > >> > > >> > Is the output this clean? What I'm looking for is signs of automatic > >> > garbage collection kicking in the middle of the rebase. Something like > >> > "Auto packing the repository blah blah for optimum performance". > >> > >> Yeah, this is the exact output. I also thought about "git gc", but I > >> don't see any output related to it. Is it possible that it's being > >> executed in the background and not printing anything? > > > > Any chance you have a cronjob or other task execution mechanism that > > is running git commands in various directories (even simple commands > > like 'git status' since it's not read-only contrary to what some > > expect, or 'git fetch' which might trigger a background gc)? > > Nope, nothing like this. AFAIK, nothing is touching that repository at > the same time that I am. Besides, even if I wait some minutes before > trying again, the bug manifests again. Well, even though you didn't see the output Duy was looking for, if you set gc.auto to 0 and gc.autoPackLimit to 0 (and maybe gc.autoDetach to false just for good measure), does the problem still occur? ``` ## Sergio Durigan Junior, 2019-03-12 19:58 Subject: Re: Possible race condition with git-rebase + .git/index.lock Message-ID: <87y35j50hb.fsf@sergiodj.net> URL: https://gitlist.dev/e/87y35j50hb.fsf%40sergiodj.net In-Reply-To: ``` On Tuesday, March 12 2019, Elijah Newren wrote: > On Tue, Mar 12, 2019 at 12:32 PM Sergio Durigan Junior > wrote: >> >> On Tuesday, March 12 2019, Elijah Newren wrote: >> >> > On Tue, Mar 12, 2019 at 9:48 AM Sergio Durigan Junior >> > wrote: >> >> On Tuesday, March 12 2019, Duy Nguyen wrote: >> >> >> >> > On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior >> >> > wrote: >> >> >> This works without problems most of the time (well, usually there are >> >> >> conflicts and all, but that's a burden I have to carry). However, >> >> >> sometimes I notice that git fails with: >> >> >> >> >> >> # git rebase origin/master >> >> >> ... >> >> >> Applying: commitX >> >> >> Applying: commitY >> >> >> Applying: commitZ >> >> >> fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists. >> >> >> >> >> >> The first thing I did was to check whether the index.lock file existed, >> >> >> but it doesn't. >> >> > >> >> > Is the output this clean? What I'm looking for is signs of automatic >> >> > garbage collection kicking in the middle of the rebase. Something like >> >> > "Auto packing the repository blah blah for optimum performance". >> >> >> >> Yeah, this is the exact output. I also thought about "git gc", but I >> >> don't see any output related to it. Is it possible that it's being >> >> executed in the background and not printing anything? >> > >> > Any chance you have a cronjob or other task execution mechanism that >> > is running git commands in various directories (even simple commands >> > like 'git status' since it's not read-only contrary to what some >> > expect, or 'git fetch' which might trigger a background gc)? >> >> Nope, nothing like this. AFAIK, nothing is touching that repository at >> the same time that I am. Besides, even if I wait some minutes before >> trying again, the bug manifests again. > > Well, even though you didn't see the output Duy was looking for, if > you set gc.auto to 0 and gc.autoPackLimit to 0 (and maybe > gc.autoDetach to false just for good measure), does the problem still > occur? Thanks, I was indeed going to try this. I'll test when I have time, and will let you know. -- Sergio GPG key ID: 237A 54B1 0287 28BF 00EF 31F4 D0EB 7628 65FC 5E36 Please send encrypted e-mail if possible http://sergiodj.net/ ``` ## Phillip Wood, 2019-03-13 20:40 Subject: Re: Possible race condition with git-rebase + .git/index.lock Message-ID: URL: https://gitlist.dev/e/e0e1e159-844d-c9bf-e1cc-ada05384ef76%40gmail.com In-Reply-To: <8736nr6g94.fsf@sergiodj.net> ``` Hi Sergio On 12/03/2019 19:32, Sergio Durigan Junior wrote: > On Tuesday, March 12 2019, Elijah Newren wrote: > >> On Tue, Mar 12, 2019 at 9:48 AM Sergio Durigan Junior >> wrote: >>> On Tuesday, March 12 2019, Duy Nguyen wrote: >>> >>>> On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior >>>> wrote: >>>>> This works without problems most of the time (well, usually there are >>>>> conflicts and all, but that's a burden I have to carry). However, >>>>> sometimes I notice that git fails with: >>>>> >>>>> # git rebase origin/master >>>>> ... >>>>> Applying: commitX >>>>> Applying: commitY >>>>> Applying: commitZ >>>>> fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists. >>>>> >>>>> The first thing I did was to check whether the index.lock file existed, >>>>> but it doesn't. >>>> >>>> Is the output this clean? What I'm looking for is signs of automatic >>>> garbage collection kicking in the middle of the rebase. Something like >>>> "Auto packing the repository blah blah for optimum performance". >>> >>> Yeah, this is the exact output. I also thought about "git gc", but I >>> don't see any output related to it. Is it possible that it's being >>> executed in the background and not printing anything? >> >> Any chance you have a cronjob or other task execution mechanism that >> is running git commands in various directories (even simple commands >> like 'git status' since it's not read-only contrary to what some >> expect, or 'git fetch' which might trigger a background gc)? > > Nope, nothing like this. AFAIK, nothing is touching that repository at > the same time that I am. Besides, even if I wait some minutes before > trying again, the bug manifests again. > Do you have any of the files in the repository open in an editor or ide while you are rebasing? I have seen this message when rebasing and one of the files is open in emacs which runs git status when it sees a file has changed which locks the index while the rebase is running. Best Wishes Phillip ``` ## Sergio Durigan Junior, 2019-03-13 20:48 Subject: Re: Possible race condition with git-rebase + .git/index.lock Message-ID: <87pnqu4i1n.fsf@sergiodj.net> URL: https://gitlist.dev/e/87pnqu4i1n.fsf%40sergiodj.net In-Reply-To: ``` On Wednesday, March 13 2019, Phillip Wood wrote: > Hi Sergio > > On 12/03/2019 19:32, Sergio Durigan Junior wrote: >> On Tuesday, March 12 2019, Elijah Newren wrote: >> >>> On Tue, Mar 12, 2019 at 9:48 AM Sergio Durigan Junior >>> wrote: >>>> On Tuesday, March 12 2019, Duy Nguyen wrote: >>>> >>>>> On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior >>>>> wrote: >>>>>> This works without problems most of the time (well, usually there are >>>>>> conflicts and all, but that's a burden I have to carry). However, >>>>>> sometimes I notice that git fails with: >>>>>> >>>>>> # git rebase origin/master >>>>>> ... >>>>>> Applying: commitX >>>>>> Applying: commitY >>>>>> Applying: commitZ >>>>>> fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists. >>>>>> >>>>>> The first thing I did was to check whether the index.lock file existed, >>>>>> but it doesn't. >>>>> >>>>> Is the output this clean? What I'm looking for is signs of automatic >>>>> garbage collection kicking in the middle of the rebase. Something like >>>>> "Auto packing the repository blah blah for optimum performance". >>>> >>>> Yeah, this is the exact output. I also thought about "git gc", but I >>>> don't see any output related to it. Is it possible that it's being >>>> executed in the background and not printing anything? >>> >>> Any chance you have a cronjob or other task execution mechanism that >>> is running git commands in various directories (even simple commands >>> like 'git status' since it's not read-only contrary to what some >>> expect, or 'git fetch' which might trigger a background gc)? >> >> Nope, nothing like this. AFAIK, nothing is touching that repository at >> the same time that I am. Besides, even if I wait some minutes before >> trying again, the bug manifests again. >> > > Do you have any of the files in the repository open in an editor or > ide while you are rebasing? I have seen this message when rebasing and > one of the files is open in emacs which runs git status when it sees a > file has changed which locks the index while the rebase is running. Hey Phillip, Huh, I do have some of the files opened in Emacs! They're in background, but indeed, now that you mentioned I remember that Emacs keeps track of changes and invokes "git status" sometimes. Next time I see the bug, I'll try closing the files and see if it happens again. Thanks for the insight! -- Sergio GPG key ID: 237A 54B1 0287 28BF 00EF 31F4 D0EB 7628 65FC 5E36 Please send encrypted e-mail if possible http://sergiodj.net/ ``` ## Jeff King, 2019-03-13 20:59 Subject: Re: Possible race condition with git-rebase + .git/index.lock Message-ID: <20190313205926.GB5397@sigill.intra.peff.net> URL: https://gitlist.dev/e/20190313205926.GB5397%40sigill.intra.peff.net In-Reply-To: <87pnqu4i1n.fsf@sergiodj.net> ``` On Wed, Mar 13, 2019 at 04:48:36PM -0400, Sergio Durigan Junior wrote: > Huh, I do have some of the files opened in Emacs! They're in > background, but indeed, now that you mentioned I remember that Emacs > keeps track of changes and invokes "git status" sometimes. Next time I > see the bug, I'll try closing the files and see if it happens again. If Emacs (I guess maybe magit?) is running "git status" behind the scenes, you might benefit from teaching it to use "git --no-optional-locks status" instead. See the section "BACKGROUND REFRESH" in "git help status" for more discussion. -Peff ``` ## Sergio Durigan Junior, 2019-03-13 21:07 Subject: Re: Possible race condition with git-rebase + .git/index.lock Message-ID: <87lg1i4h6g.fsf@sergiodj.net> URL: https://gitlist.dev/e/87lg1i4h6g.fsf%40sergiodj.net In-Reply-To: <20190313205926.GB5397@sigill.intra.peff.net> ``` On Wednesday, March 13 2019, Jeff King wrote: > On Wed, Mar 13, 2019 at 04:48:36PM -0400, Sergio Durigan Junior wrote: > >> Huh, I do have some of the files opened in Emacs! They're in >> background, but indeed, now that you mentioned I remember that Emacs >> keeps track of changes and invokes "git status" sometimes. Next time I >> see the bug, I'll try closing the files and see if it happens again. > > If Emacs (I guess maybe magit?) is running "git status" behind the > scenes, you might benefit from teaching it to use "git > --no-optional-locks status" instead. See the section "BACKGROUND > REFRESH" in "git help status" for more discussion. Thanks Jeff. That's going to be helpful. -- Sergio GPG key ID: 237A 54B1 0287 28BF 00EF 31F4 D0EB 7628 65FC 5E36 Please send encrypted e-mail if possible http://sergiodj.net/ ``` ## Mingun, 2022-10-28 16:04 Subject: Re: Possible race condition with git-rebase + .git/index.lock - probably a gettext issue Message-ID: <20221028160451.2040-1-alexander_sergey@mail.ru> URL: https://gitlist.dev/e/20221028160451.2040-1-alexander_sergey%40mail.ru In-Reply-To: <87lg1i4h6g.fsf@sergiodj.net> ``` --- It seems that this bug is related to localization library, similar to this diesel issue: https://github.com/diesel-rs/diesel/discussions/2947#discussioncomment-2025857. Also, this bug is highly depend on the free space capacity on a drive where working copy is located. I encounter this bug each time when my ssd is nearly full (about 6 GB of free space on 120 GB drive). About 90% of time, when I try to use `git svn rebase` in such conditions, the operation failed: ˋˋˋ $ git svn rebase warning: пропущен уже применённый коммит ef0998895a6 warning: пропущен уже применённый коммит d1dbbd6ef8b подсказка: use --reapply-cherry-picks to include skipped commits подсказка: Disable this message with "git config advice.skippedCherryPicks false" fatal: Не удалось создать «/path/to/my/git/repo/.git/index.lock»: Файл существует. Похоже, что другой процесс git запущен в этом репозитории, например редактор открыт из «git commit». Пожалуйста, убедитесь, что все процессы были завершены и потом попробуйте снова. Если это не поможет, то возможно процесс git был ранее завершен с ошибкой в этом репозитории: удалите файл вручную для продолжения. rebase refs/remotes/origin/trunk: command returned error: 128 Ветка push удалена (была 5c2212d372e). $ git rebase --continue fatal: не удалось прочитать файл журнала «.git/rebase-merge/message»: Нет такого файла или каталога ˋˋˋ At the same time, when after I free space on my drive or set "C" locale, the operation is completed from first attempt. I hope this will help to localize and fix the issue. ```