threads / discuss / 47762

Are concurrent git operations on the same repo safe?

Subject: Are concurrent git operations on the same repo safe?

## tl;dr

5 messages between Feb 5, 2018 and Feb 6, 2018.

replies: 4people: 4as markdown or json

Ian Norton· Feb 5, 2018, 20:16 UTC · lore

Hi all,

I'm generally used to the idea that if a thing is not marked as "thread-safe" then it isn't thread safe, but I thought I'd ask anyway to be sure.

Is it safe for me to do several operations with git concurrently on the same local repo? Specifically I'm trying to speed up "git submodule update" by doing several at the same time. I've noticed some odd side effects afterwards though when trying to commit changes to my super project.

Apologies if this is answered elsewhere, my google-foo is weak today.

Many thanks

Ian

Duy Nguyen· Feb 6, 2018, 10:16 UTC · re: Ian Norton · lore

Re: Are concurrent git operations on the same repo safe?

On Tue, Feb 6, 2018 at 3:16 AM, Ian Norton <inorton@gmail.com> wrote:

Show 8 quoted lines

> Hi all,
>
> I'm generally used to the idea that if a thing is not marked as
> "thread-safe" then it isn't thread safe, but I thought I'd ask anyway
> to be sure.
>
> Is it safe for me to do several operations with git concurrently on
> the same local repo?

Off the top of my head, object database access (e.g. things in .git/objects), refs updates (.git/refs) and .git/index should handle concurrent operations fine (or in the worst case you get "some operation is ongoing, aborted" and you need to try again but not corruption or anything else). I think we generally try to make it safe concurrently.

> Specifically I'm trying to speed up "git
> submodule update" by doing several at the same time.  I've noticed
> some odd side effects afterwards though when trying to commit changes
> to my super project.

submodule is a hot area with lots of development lately I think, perhaps you're seeing some bugs... CCing at least one submodule person...

Show 5 quoted lines

> Apologies if this is answered elsewhere, my google-foo is weak today.
>
> Many thanks
>
> Ian

-- 
Duy

Stefan Beller· Feb 6, 2018, 17:30 UTC · re: Duy Nguyen · lore

Re: Are concurrent git operations on the same repo safe?

On Tue, Feb 6, 2018 at 2:16 AM, Duy Nguyen <pclouds@gmail.com> wrote:

Show 31 quoted lines

> On Tue, Feb 6, 2018 at 3:16 AM, Ian Norton <inorton@gmail.com> wrote:
>> Hi all,
>>
>> I'm generally used to the idea that if a thing is not marked as
>> "thread-safe" then it isn't thread safe, but I thought I'd ask anyway
>> to be sure.
>>
>> Is it safe for me to do several operations with git concurrently on
>> the same local repo?
>
> Off the top of my head, object database access (e.g. things in
> .git/objects), refs updates (.git/refs) and .git/index should handle
> concurrent operations fine (or in the worst case you get "some
> operation is ongoing, aborted" and you need to try again but not
> corruption or anything else). I think we generally try to make it safe
> concurrently.
>
>> Specifically I'm trying to speed up "git
>> submodule update" by doing several at the same time.  I've noticed
>> some odd side effects afterwards though when trying to commit changes
>> to my super project.
>
> submodule is a hot area with lots of development lately I think,
> perhaps you're seeing some bugs... CCing at least one submodule
> person...
>
>> Apologies if this is answered elsewhere, my google-foo is weak today.
>>
>> Many thanks
>>
>> Ian

"git submodule update" has the network part parallelized, but not the local part IIRC. (That is cloning/fetching submodules can be done with "-j <n>", but the local checkout is still serial for UX purposes, i.e. it wants to stop at the first conflict and only have one conflict at a time)

Which odd side effects do you see? I'd be curious to see if that is a bug in the code or documentation.

Thanks, Stefan

Jonathan Nieder· Feb 6, 2018, 20:10 UTC · re: Ian Norton · lore

Re: Are concurrent git operations on the same repo safe?

Ian Norton wrote:

>                      Specifically I'm trying to speed up "git
> submodule update" by doing several at the same time.

Can you say more about this? E.g. how can I reproduce your experience? Is there a script I can run?

Thanks, Jonathan

Ian Norton· Feb 6, 2018, 20:26 UTC · re: Jonathan Nieder · lore

Re: Are concurrent git operations on the same repo safe?

Sure, my office is still devoid of power, i'll have to get back to you to be more precise but I was using a morally equivalent process to https://gitlab.com/inorton/git-multi-sync

cd repo python -m gitmultisync --update-submodules

where I had a superproject containing 5-6 submodules, some of which were quite large (1-2Gb history)

Afterwards, I had trouble doing "git pull -r" in my submodules to pull in newer changes.

On 6 February 2018 at 20:10, Jonathan Nieder <jrnieder@gmail.com> wrote:

Show 10 quoted lines

> Ian Norton wrote:
>
>>                      Specifically I'm trying to speed up "git
>> submodule update" by doing several at the same time.
>
> Can you say more about this?  E.g. how can I reproduce your experience?
> Is there a script I can run?
>
> Thanks,
> Jonathan