# Are concurrent git operations on the same repo safe? 5 messages from 2018-02-05 to 2018-02-06. Participants: Ian Norton, Duy Nguyen, Stefan Beller, Jonathan Nieder. Thread: https://gitlist.dev/t/47762 ## Ian Norton, 2018-02-05 20:16 Subject: Are concurrent git operations on the same repo safe? Message-ID: URL: https://gitlist.dev/e/CAGUnuBHY9nQvGiROm4S_JvBWMrzieHC1FE50fJqxaauzZhH7xw%40mail.gmail.com ``` Hi all, I'm generally used to the idea that if a thing is not marked as "thread-safe" then it isn't thread safe, but I thought I'd ask anyway to be sure. Is it safe for me to do several operations with git concurrently on the same local repo? Specifically I'm trying to speed up "git submodule update" by doing several at the same time. I've noticed some odd side effects afterwards though when trying to commit changes to my super project. Apologies if this is answered elsewhere, my google-foo is weak today. Many thanks Ian ``` ## Duy Nguyen, 2018-02-06 10:16 Subject: Re: Are concurrent git operations on the same repo safe? Message-ID: URL: https://gitlist.dev/e/CACsJy8Bhu6RvedKgOx8uExrJuo0-b66JvnZA9d9v4Uk6mSwPQQ%40mail.gmail.com In-Reply-To: ``` On Tue, Feb 6, 2018 at 3:16 AM, Ian Norton wrote: > Hi all, > > I'm generally used to the idea that if a thing is not marked as > "thread-safe" then it isn't thread safe, but I thought I'd ask anyway > to be sure. > > Is it safe for me to do several operations with git concurrently on > the same local repo? Off the top of my head, object database access (e.g. things in .git/objects), refs updates (.git/refs) and .git/index should handle concurrent operations fine (or in the worst case you get "some operation is ongoing, aborted" and you need to try again but not corruption or anything else). I think we generally try to make it safe concurrently. > Specifically I'm trying to speed up "git > submodule update" by doing several at the same time. I've noticed > some odd side effects afterwards though when trying to commit changes > to my super project. submodule is a hot area with lots of development lately I think, perhaps you're seeing some bugs... CCing at least one submodule person... > Apologies if this is answered elsewhere, my google-foo is weak today. > > Many thanks > > Ian -- Duy ``` ## Stefan Beller, 2018-02-06 17:30 Subject: Re: Are concurrent git operations on the same repo safe? Message-ID: URL: https://gitlist.dev/e/CAGZ79kb%2BSB3q720Q_QbpPTLsY-gfsYHGdSvK2TGOtRXwLMQyOg%40mail.gmail.com In-Reply-To: ``` On Tue, Feb 6, 2018 at 2:16 AM, Duy Nguyen wrote: > On Tue, Feb 6, 2018 at 3:16 AM, Ian Norton wrote: >> Hi all, >> >> I'm generally used to the idea that if a thing is not marked as >> "thread-safe" then it isn't thread safe, but I thought I'd ask anyway >> to be sure. >> >> Is it safe for me to do several operations with git concurrently on >> the same local repo? > > Off the top of my head, object database access (e.g. things in > .git/objects), refs updates (.git/refs) and .git/index should handle > concurrent operations fine (or in the worst case you get "some > operation is ongoing, aborted" and you need to try again but not > corruption or anything else). I think we generally try to make it safe > concurrently. > >> Specifically I'm trying to speed up "git >> submodule update" by doing several at the same time. I've noticed >> some odd side effects afterwards though when trying to commit changes >> to my super project. > > submodule is a hot area with lots of development lately I think, > perhaps you're seeing some bugs... CCing at least one submodule > person... > >> Apologies if this is answered elsewhere, my google-foo is weak today. >> >> Many thanks >> >> Ian "git submodule update" has the network part parallelized, but not the local part IIRC. (That is cloning/fetching submodules can be done with "-j ", but the local checkout is still serial for UX purposes, i.e. it wants to stop at the first conflict and only have one conflict at a time) Which odd side effects do you see? I'd be curious to see if that is a bug in the code or documentation. Thanks, Stefan ``` ## Jonathan Nieder, 2018-02-06 20:10 Subject: Re: Are concurrent git operations on the same repo safe? Message-ID: <20180206201047.GE104086@aiede.svl.corp.google.com> URL: https://gitlist.dev/e/20180206201047.GE104086%40aiede.svl.corp.google.com In-Reply-To: ``` Ian Norton wrote: > Specifically I'm trying to speed up "git > submodule update" by doing several at the same time. Can you say more about this? E.g. how can I reproduce your experience? Is there a script I can run? Thanks, Jonathan ``` ## Ian Norton, 2018-02-06 20:26 Subject: Re: Are concurrent git operations on the same repo safe? Message-ID: URL: https://gitlist.dev/e/CAGUnuBHRH5NV31yDiBPdZouNicdfU9z9iPK1wcBNrEhoo%2BXw8A%40mail.gmail.com In-Reply-To: <20180206201047.GE104086@aiede.svl.corp.google.com> ``` Sure, my office is still devoid of power, i'll have to get back to you to be more precise but I was using a morally equivalent process to https://gitlab.com/inorton/git-multi-sync cd repo python -m gitmultisync --update-submodules where I had a superproject containing 5-6 submodules, some of which were quite large (1-2Gb history) Afterwards, I had trouble doing "git pull -r" in my submodules to pull in newer changes. On 6 February 2018 at 20:10, Jonathan Nieder wrote: > Ian Norton wrote: > >> Specifically I'm trying to speed up "git >> submodule update" by doing several at the same time. > > Can you say more about this? E.g. how can I reproduce your experience? > Is there a script I can run? > > Thanks, > Jonathan ```