threads / discuss / 50712

Possible race condition with git-rebase + .git/index.lock

Subject: Possible race condition with git-rebase + .git/index.lock

## tl;dr

14 messages between Mar 11, 2019 and Oct 28, 2022.

replies: 13people: 7as markdown or json

Sergio Durigan Junior· Mar 11, 2019, 21:38 UTC · lore

Hi there,

I've been bit by this bug many times, and always forget to report it here, but this time I finally remembered.

The scenario is this:

- I have a local repository which tracks the upstream of the project.

- I have 120+ local commits.

- Every week or so, I have to rebase my local commits on top of
  upstream's origin/master.

The workflow is simple:

  # git fetch origin && git rebase origin/master

This works without problems most of the time (well, usually there are conflicts and all, but that's a burden I have to carry). However, sometimes I notice that git fails with:

  # git rebase origin/master
  ...
  Applying: commitX
  Applying: commitY
  Applying: commitZ
  fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists.

The first thing I did was to check whether the index.lock file existed, but it doesn't.

I also I thought it might have something to do with "commitZ", but after I ran "git rebase --abort" and tried rebasing again, I noticed that it would stop at another commit.

This seems like a race condition, but after inspecting builtin/rebase.c it doesn't seem like the "git am" commands are run in parallel (and it wouldn't make sense if they did, anyway).

I tried using GIT_TRACE, but haven't seen any useful information in the logs. I tried using the "next" branch, and found that the problem exists there as well. I wonder if anyone else has seen this problem.

Thanks,

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

Eric Sunshine· Mar 11, 2019, 22:50 UTC · re: Sergio Durigan Junior · lore

Re: Possible race condition with git-rebase + .git/index.lock

On Mon, Mar 11, 2019 at 6:17 PM Sergio Durigan Junior <sergiodj@sergiodj.net> wrote:

Show 9 quoted lines

>   # git rebase origin/master
>   ...
>   Applying: commitX
>   Applying: commitY
>   Applying: commitZ
>   fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists.
>
> The first thing I did was to check whether the index.lock file existed,
> but it doesn't.

What platform is this on? If Windows, I'm wondering if something (such as a virus scanner) is holding the lock file open long enough to prevent Git from actually deleting it (and perhaps Git doesn't notice the failed deletion -- until the next time it tries to take the lock).

Sergio Durigan Junior· Mar 11, 2019, 22:53 UTC · re: Eric Sunshine · lore

Re: Possible race condition with git-rebase + .git/index.lock

On Monday, March 11 2019, Eric Sunshine wrote:

Show 16 quoted lines

> On Mon, Mar 11, 2019 at 6:17 PM Sergio Durigan Junior
> <sergiodj@sergiodj.net> wrote:
>>   # git rebase origin/master
>>   ...
>>   Applying: commitX
>>   Applying: commitY
>>   Applying: commitZ
>>   fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists.
>>
>> The first thing I did was to check whether the index.lock file existed,
>> but it doesn't.
>
> What platform is this on? If Windows, I'm wondering if something (such
> as a virus scanner) is holding the lock file open long enough to
> prevent Git from actually deleting it (and perhaps Git doesn't notice
> the failed deletion -- until the next time it tries to take the lock).

This is on GNU/Linux (Fedora 28, to be specific). I haven't had the chance to test this on other distros, unfortunately. And since it seems to be a race condition, it's kind of hard to perform tests that trigger this.

Thanks,

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

Duy Nguyen· Mar 12, 2019, 10:28 UTC · re: Sergio Durigan Junior · lore

Re: Possible race condition with git-rebase + .git/index.lock

On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior <sergiodj@sergiodj.net> wrote:

Show 13 quoted lines

> This works without problems most of the time (well, usually there are
> conflicts and all, but that's a burden I have to carry).  However,
> sometimes I notice that git fails with:
>
>   # git rebase origin/master
>   ...
>   Applying: commitX
>   Applying: commitY
>   Applying: commitZ
>   fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists.
>
> The first thing I did was to check whether the index.lock file existed,
> but it doesn't.

Is the output this clean? What I'm looking for is signs of automatic garbage collection kicking in the middle of the rebase. Something like "Auto packing the repository blah blah for optimum performance".

-- 
Duy

Sergio Durigan Junior· Mar 12, 2019, 16:45 UTC · re: Duy Nguyen · lore

Re: Possible race condition with git-rebase + .git/index.lock

On Tuesday, March 12 2019, Duy Nguyen wrote:

Show 19 quoted lines

> On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior
> <sergiodj@sergiodj.net> wrote:
>> This works without problems most of the time (well, usually there are
>> conflicts and all, but that's a burden I have to carry).  However,
>> sometimes I notice that git fails with:
>>
>>   # git rebase origin/master
>>   ...
>>   Applying: commitX
>>   Applying: commitY
>>   Applying: commitZ
>>   fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists.
>>
>> The first thing I did was to check whether the index.lock file existed,
>> but it doesn't.
>
> Is the output this clean? What I'm looking for is signs of automatic
> garbage collection kicking in the middle of the rebase. Something like
> "Auto packing the repository blah blah for optimum performance".

Yeah, this is the exact output. I also thought about "git gc", but I don't see any output related to it. Is it possible that it's being executed in the background and not printing anything?

Thanks,

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

Elijah Newren· Mar 12, 2019, 19:23 UTC · re: Sergio Durigan Junior · lore

Re: Possible race condition with git-rebase + .git/index.lock

On Tue, Mar 12, 2019 at 9:48 AM Sergio Durigan Junior <sergiodj@sergiodj.net> wrote:

Show 25 quoted lines

> On Tuesday, March 12 2019, Duy Nguyen wrote:
>
> > On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior
> > <sergiodj@sergiodj.net> wrote:
> >> This works without problems most of the time (well, usually there are
> >> conflicts and all, but that's a burden I have to carry).  However,
> >> sometimes I notice that git fails with:
> >>
> >>   # git rebase origin/master
> >>   ...
> >>   Applying: commitX
> >>   Applying: commitY
> >>   Applying: commitZ
> >>   fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists.
> >>
> >> The first thing I did was to check whether the index.lock file existed,
> >> but it doesn't.
> >
> > Is the output this clean? What I'm looking for is signs of automatic
> > garbage collection kicking in the middle of the rebase. Something like
> > "Auto packing the repository blah blah for optimum performance".
>
> Yeah, this is the exact output.  I also thought about "git gc", but I
> don't see any output related to it.  Is it possible that it's being
> executed in the background and not printing anything?

Any chance you have a cronjob or other task execution mechanism that is running git commands in various directories (even simple commands like 'git status' since it's not read-only contrary to what some expect, or 'git fetch' which might trigger a background gc)?

Sergio Durigan Junior· Mar 12, 2019, 19:32 UTC · re: Elijah Newren · lore

Re: Possible race condition with git-rebase + .git/index.lock

On Tuesday, March 12 2019, Elijah Newren wrote:

Show 32 quoted lines

> On Tue, Mar 12, 2019 at 9:48 AM Sergio Durigan Junior
> <sergiodj@sergiodj.net> wrote:
>> On Tuesday, March 12 2019, Duy Nguyen wrote:
>>
>> > On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior
>> > <sergiodj@sergiodj.net> wrote:
>> >> This works without problems most of the time (well, usually there are
>> >> conflicts and all, but that's a burden I have to carry).  However,
>> >> sometimes I notice that git fails with:
>> >>
>> >>   # git rebase origin/master
>> >>   ...
>> >>   Applying: commitX
>> >>   Applying: commitY
>> >>   Applying: commitZ
>> >>   fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists.
>> >>
>> >> The first thing I did was to check whether the index.lock file existed,
>> >> but it doesn't.
>> >
>> > Is the output this clean? What I'm looking for is signs of automatic
>> > garbage collection kicking in the middle of the rebase. Something like
>> > "Auto packing the repository blah blah for optimum performance".
>>
>> Yeah, this is the exact output.  I also thought about "git gc", but I
>> don't see any output related to it.  Is it possible that it's being
>> executed in the background and not printing anything?
>
> Any chance you have a cronjob or other task execution mechanism that
> is running git commands in various directories (even simple commands
> like 'git status' since it's not read-only contrary to what some
> expect, or 'git fetch' which might trigger a background gc)?

Nope, nothing like this. AFAIK, nothing is touching that repository at the same time that I am. Besides, even if I wait some minutes before trying again, the bug manifests again.

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

Elijah Newren· Mar 12, 2019, 19:39 UTC · re: Sergio Durigan Junior · lore

Re: Possible race condition with git-rebase + .git/index.lock

On Tue, Mar 12, 2019 at 12:32 PM Sergio Durigan Junior <sergiodj@sergiodj.net> wrote:

Show 39 quoted lines

>
> On Tuesday, March 12 2019, Elijah Newren wrote:
>
> > On Tue, Mar 12, 2019 at 9:48 AM Sergio Durigan Junior
> > <sergiodj@sergiodj.net> wrote:
> >> On Tuesday, March 12 2019, Duy Nguyen wrote:
> >>
> >> > On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior
> >> > <sergiodj@sergiodj.net> wrote:
> >> >> This works without problems most of the time (well, usually there are
> >> >> conflicts and all, but that's a burden I have to carry).  However,
> >> >> sometimes I notice that git fails with:
> >> >>
> >> >>   # git rebase origin/master
> >> >>   ...
> >> >>   Applying: commitX
> >> >>   Applying: commitY
> >> >>   Applying: commitZ
> >> >>   fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists.
> >> >>
> >> >> The first thing I did was to check whether the index.lock file existed,
> >> >> but it doesn't.
> >> >
> >> > Is the output this clean? What I'm looking for is signs of automatic
> >> > garbage collection kicking in the middle of the rebase. Something like
> >> > "Auto packing the repository blah blah for optimum performance".
> >>
> >> Yeah, this is the exact output.  I also thought about "git gc", but I
> >> don't see any output related to it.  Is it possible that it's being
> >> executed in the background and not printing anything?
> >
> > Any chance you have a cronjob or other task execution mechanism that
> > is running git commands in various directories (even simple commands
> > like 'git status' since it's not read-only contrary to what some
> > expect, or 'git fetch' which might trigger a background gc)?
>
> Nope, nothing like this.  AFAIK, nothing is touching that repository at
> the same time that I am.  Besides, even if I wait some minutes before
> trying again, the bug manifests again.

Well, even though you didn't see the output Duy was looking for, if you set gc.auto to 0 and gc.autoPackLimit to 0 (and maybe gc.autoDetach to false just for good measure), does the problem still occur?

Sergio Durigan Junior· Mar 12, 2019, 19:58 UTC · re: Elijah Newren · lore

Re: Possible race condition with git-rebase + .git/index.lock

On Tuesday, March 12 2019, Elijah Newren wrote:

Show 46 quoted lines

> On Tue, Mar 12, 2019 at 12:32 PM Sergio Durigan Junior
> <sergiodj@sergiodj.net> wrote:
>>
>> On Tuesday, March 12 2019, Elijah Newren wrote:
>>
>> > On Tue, Mar 12, 2019 at 9:48 AM Sergio Durigan Junior
>> > <sergiodj@sergiodj.net> wrote:
>> >> On Tuesday, March 12 2019, Duy Nguyen wrote:
>> >>
>> >> > On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior
>> >> > <sergiodj@sergiodj.net> wrote:
>> >> >> This works without problems most of the time (well, usually there are
>> >> >> conflicts and all, but that's a burden I have to carry).  However,
>> >> >> sometimes I notice that git fails with:
>> >> >>
>> >> >>   # git rebase origin/master
>> >> >>   ...
>> >> >>   Applying: commitX
>> >> >>   Applying: commitY
>> >> >>   Applying: commitZ
>> >> >>   fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists.
>> >> >>
>> >> >> The first thing I did was to check whether the index.lock file existed,
>> >> >> but it doesn't.
>> >> >
>> >> > Is the output this clean? What I'm looking for is signs of automatic
>> >> > garbage collection kicking in the middle of the rebase. Something like
>> >> > "Auto packing the repository blah blah for optimum performance".
>> >>
>> >> Yeah, this is the exact output.  I also thought about "git gc", but I
>> >> don't see any output related to it.  Is it possible that it's being
>> >> executed in the background and not printing anything?
>> >
>> > Any chance you have a cronjob or other task execution mechanism that
>> > is running git commands in various directories (even simple commands
>> > like 'git status' since it's not read-only contrary to what some
>> > expect, or 'git fetch' which might trigger a background gc)?
>>
>> Nope, nothing like this.  AFAIK, nothing is touching that repository at
>> the same time that I am.  Besides, even if I wait some minutes before
>> trying again, the bug manifests again.
>
> Well, even though you didn't see the output Duy was looking for, if
> you set gc.auto to 0 and gc.autoPackLimit to 0 (and maybe
> gc.autoDetach to false just for good measure), does the problem still
> occur?

Thanks, I was indeed going to try this. I'll test when I have time, and will let you know.

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

Phillip Wood· Mar 13, 2019, 20:40 UTC · re: Sergio Durigan Junior · lore

Re: Possible race condition with git-rebase + .git/index.lock

Hi Sergio

On 12/03/2019 19:32, Sergio Durigan Junior wrote:

Show 39 quoted lines

> On Tuesday, March 12 2019, Elijah Newren wrote:
> 
>> On Tue, Mar 12, 2019 at 9:48 AM Sergio Durigan Junior
>> <sergiodj@sergiodj.net> wrote:
>>> On Tuesday, March 12 2019, Duy Nguyen wrote:
>>>
>>>> On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior
>>>> <sergiodj@sergiodj.net> wrote:
>>>>> This works without problems most of the time (well, usually there are
>>>>> conflicts and all, but that's a burden I have to carry).  However,
>>>>> sometimes I notice that git fails with:
>>>>>
>>>>>    # git rebase origin/master
>>>>>    ...
>>>>>    Applying: commitX
>>>>>    Applying: commitY
>>>>>    Applying: commitZ
>>>>>    fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists.
>>>>>
>>>>> The first thing I did was to check whether the index.lock file existed,
>>>>> but it doesn't.
>>>>
>>>> Is the output this clean? What I'm looking for is signs of automatic
>>>> garbage collection kicking in the middle of the rebase. Something like
>>>> "Auto packing the repository blah blah for optimum performance".
>>>
>>> Yeah, this is the exact output.  I also thought about "git gc", but I
>>> don't see any output related to it.  Is it possible that it's being
>>> executed in the background and not printing anything?
>>
>> Any chance you have a cronjob or other task execution mechanism that
>> is running git commands in various directories (even simple commands
>> like 'git status' since it's not read-only contrary to what some
>> expect, or 'git fetch' which might trigger a background gc)?
> 
> Nope, nothing like this.  AFAIK, nothing is touching that repository at
> the same time that I am.  Besides, even if I wait some minutes before
> trying again, the bug manifests again.
>

Do you have any of the files in the repository open in an editor or ide while you are rebasing? I have seen this message when rebasing and one of the files is open in emacs which runs git status when it sees a file has changed which locks the index while the rebase is running.

Best Wishes

Phillip

Sergio Durigan Junior· Mar 13, 2019, 20:48 UTC · re: Phillip Wood · lore

Re: Possible race condition with git-rebase + .git/index.lock

On Wednesday, March 13 2019, Phillip Wood wrote:

Show 47 quoted lines

> Hi Sergio
>
> On 12/03/2019 19:32, Sergio Durigan Junior wrote:
>> On Tuesday, March 12 2019, Elijah Newren wrote:
>>
>>> On Tue, Mar 12, 2019 at 9:48 AM Sergio Durigan Junior
>>> <sergiodj@sergiodj.net> wrote:
>>>> On Tuesday, March 12 2019, Duy Nguyen wrote:
>>>>
>>>>> On Tue, Mar 12, 2019 at 5:18 AM Sergio Durigan Junior
>>>>> <sergiodj@sergiodj.net> wrote:
>>>>>> This works without problems most of the time (well, usually there are
>>>>>> conflicts and all, but that's a burden I have to carry).  However,
>>>>>> sometimes I notice that git fails with:
>>>>>>
>>>>>>    # git rebase origin/master
>>>>>>    ...
>>>>>>    Applying: commitX
>>>>>>    Applying: commitY
>>>>>>    Applying: commitZ
>>>>>>    fatal: Unable to create '/home/xyz/dir1/dir2/.git/index.lock': File exists.
>>>>>>
>>>>>> The first thing I did was to check whether the index.lock file existed,
>>>>>> but it doesn't.
>>>>>
>>>>> Is the output this clean? What I'm looking for is signs of automatic
>>>>> garbage collection kicking in the middle of the rebase. Something like
>>>>> "Auto packing the repository blah blah for optimum performance".
>>>>
>>>> Yeah, this is the exact output.  I also thought about "git gc", but I
>>>> don't see any output related to it.  Is it possible that it's being
>>>> executed in the background and not printing anything?
>>>
>>> Any chance you have a cronjob or other task execution mechanism that
>>> is running git commands in various directories (even simple commands
>>> like 'git status' since it's not read-only contrary to what some
>>> expect, or 'git fetch' which might trigger a background gc)?
>>
>> Nope, nothing like this.  AFAIK, nothing is touching that repository at
>> the same time that I am.  Besides, even if I wait some minutes before
>> trying again, the bug manifests again.
>>
>
> Do you have any of the files in the repository open in an editor or
> ide while you are rebasing? I have seen this message when rebasing and
> one of the files is open in emacs which runs git status when it sees a
> file has changed which locks the index while the rebase is running.

Hey Phillip,

Huh, I do have some of the files opened in Emacs! They're in background, but indeed, now that you mentioned I remember that Emacs keeps track of changes and invokes "git status" sometimes. Next time I see the bug, I'll try closing the files and see if it happens again.

Thanks for the insight!

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

Jeff King· Mar 13, 2019, 20:59 UTC · re: Sergio Durigan Junior · lore

Re: Possible race condition with git-rebase + .git/index.lock

On Wed, Mar 13, 2019 at 04:48:36PM -0400, Sergio Durigan Junior wrote:

> Huh, I do have some of the files opened in Emacs!  They're in
> background, but indeed, now that you mentioned I remember that Emacs
> keeps track of changes and invokes "git status" sometimes.  Next time I
> see the bug, I'll try closing the files and see if it happens again.

If Emacs (I guess maybe magit?) is running "git status" behind the scenes, you might benefit from teaching it to use "git --no-optional-locks status" instead. See the section "BACKGROUND REFRESH" in "git help status" for more discussion.

-Peff

Sergio Durigan Junior· Mar 13, 2019, 21:07 UTC · re: Jeff King · lore

Re: Possible race condition with git-rebase + .git/index.lock

On Wednesday, March 13 2019, Jeff King wrote:

Show 11 quoted lines

> On Wed, Mar 13, 2019 at 04:48:36PM -0400, Sergio Durigan Junior wrote:
>
>> Huh, I do have some of the files opened in Emacs!  They're in
>> background, but indeed, now that you mentioned I remember that Emacs
>> keeps track of changes and invokes "git status" sometimes.  Next time I
>> see the bug, I'll try closing the files and see if it happens again.
>
> If Emacs (I guess maybe magit?) is running "git status" behind the
> scenes, you might benefit from teaching it to use "git
> --no-optional-locks status" instead. See the section "BACKGROUND
> REFRESH" in "git help status" for more discussion.

Thanks Jeff.  That's going to be helpful.

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

Mingun· Oct 28, 2022, 16:04 UTC · re: Sergio Durigan Junior · lore

Re: Possible race condition with git-rebase + .git/index.lock - probably a gettext issue

--- It seems that this bug is related to localization library, similar to this diesel issue: https://github.com/diesel-rs/diesel/discussions/2947#discussioncomment-2025857.

Also, this bug is highly depend on the free space capacity on a drive where working copy is located. I encounter this bug each time when my ssd is nearly full (about 6 GB of free space on 120 GB drive).

About 90% of time, when I try to use `git svn rebase` in such conditions, the operation failed:

``` $ git svn rebase warning: пропущен уже применённый коммит ef0998895a6 warning: пропущен уже применённый коммит d1dbbd6ef8b подсказка: use --reapply-cherry-picks to include skipped commits подсказка: Disable this message with "git config advice.skippedCherryPicks false" fatal: Не удалось создать «/path/to/my/git/repo/.git/index.lock»: Файл существует.

Похоже, что другой процесс git запущен в этом репозитории, например редактор открыт из «git commit». Пожалуйста, убедитесь, что все процессы были завершены и потом попробуйте снова. Если это не поможет, то возможно процесс git был ранее завершен с ошибкой в этом репозитории: удалите файл вручную для продолжения. rebase refs/remotes/origin/trunk: command returned error: 128

Ветка push удалена (была 5c2212d372e). $ git rebase --continue fatal: не удалось прочитать файл журнала «.git/rebase-merge/message»: Нет такого файла или каталога ```

At the same time, when after I free space on my drive or set "C" locale, the operation is completed from first attempt.

I hope this will help to localize and fix the issue.