{"thread":{"id":"47762","subject":"Are concurrent git operations on the same repo safe?","startedAt":"2018-02-05T20:16:43Z","lastAt":"2018-02-06T20:26:52Z","messageCount":5,"participants":["Ian Norton","Duy Nguyen","Stefan Beller","Jonathan Nieder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"338247","messageId":"CAGUnuBHY9nQvGiROm4S_JvBWMrzieHC1FE50fJqxaauzZhH7xw@mail.gmail.com","threadId":"47762","inReplyTo":null,"subject":"Are concurrent git operations on the same repo safe?","fromName":"Ian Norton","fromEmail":"inorton@gmail.com","sentAt":"2018-02-05T20:16:38Z","receivedAt":"2018-02-05T20:16:43Z","isPatch":false,"sender":{"key":"inorton@gmail.com","avatar":null},"body":"Hi all,\n\nI'm generally used to the idea that if a thing is not marked as\n\"thread-safe\" then it isn't thread safe, but I thought I'd ask anyway\nto be sure.\n\nIs it safe for me to do several operations with git concurrently on\nthe same local repo? Specifically I'm trying to speed up \"git\nsubmodule update\" by doing several at the same time. I've noticed\nsome odd side effects afterwards though when trying to commit changes\nto my super project.\n\nApologies if this is answered elsewhere, my google-foo is weak today.\n\nMany thanks\n\nIan\n"},{"id":"338496","messageId":"CACsJy8Bhu6RvedKgOx8uExrJuo0-b66JvnZA9d9v4Uk6mSwPQQ@mail.gmail.com","threadId":"47762","inReplyTo":"CAGUnuBHY9nQvGiROm4S_JvBWMrzieHC1FE50fJqxaauzZhH7xw@mail.gmail.com","subject":"Re: Are concurrent git operations on the same repo safe?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-02-06T10:16:56Z","receivedAt":"2018-02-06T10:17:40Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Feb 6, 2018 at 3:16 AM, Ian Norton wrote:\n> Hi all,\n>\n> I'm generally used to the idea that if a thing is not marked as\n> \"thread-safe\" then it isn't thread safe, but I thought I'd ask anyway\n> to be sure.\n>\n> Is it safe for me to do several operations with git concurrently on\n> the same local repo?\n\nOff the top of my head, object database access (e.g. things in\n.git/objects), refs updates (.git/refs) and .git/index should handle\nconcurrent operations fine (or in the worst case you get \"some\noperation is ongoing, aborted\" and you need to try again but not\ncorruption or anything else). I think we generally try to make it safe\nconcurrently.\n\n> Specifically I'm trying to speed up \"git\n> submodule update\" by doing several at the same time. I've noticed\n> some odd side effects afterwards though when trying to commit changes\n> to my super project.\n\nsubmodule is a hot area with lots of development lately I think,\nperhaps you're seeing some bugs... CCing at least one submodule\nperson...\n\n> Apologies if this is answered elsewhere, my google-foo is weak today.\n>\n> Many thanks\n>\n> Ian\n\n\n\n-- \nDuy\n"},{"id":"338521","messageId":"CAGZ79kb+SB3q720Q_QbpPTLsY-gfsYHGdSvK2TGOtRXwLMQyOg@mail.gmail.com","threadId":"47762","inReplyTo":"CACsJy8Bhu6RvedKgOx8uExrJuo0-b66JvnZA9d9v4Uk6mSwPQQ@mail.gmail.com","subject":"Re: Are concurrent git operations on the same repo safe?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-02-06T17:30:47Z","receivedAt":"2018-02-06T17:31:10Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Feb 6, 2018 at 2:16 AM, Duy Nguyen wrote:\n> On Tue, Feb 6, 2018 at 3:16 AM, Ian Norton wrote:\n>> Hi all,\n>>\n>> I'm generally used to the idea that if a thing is not marked as\n>> \"thread-safe\" then it isn't thread safe, but I thought I'd ask anyway\n>> to be sure.\n>>\n>> Is it safe for me to do several operations with git concurrently on\n>> the same local repo?\n>\n> Off the top of my head, object database access (e.g. things in\n> .git/objects), refs updates (.git/refs) and .git/index should handle\n> concurrent operations fine (or in the worst case you get \"some\n> operation is ongoing, aborted\" and you need to try again but not\n> corruption or anything else). I think we generally try to make it safe\n> concurrently.\n>\n>> Specifically I'm trying to speed up \"git\n>> submodule update\" by doing several at the same time. I've noticed\n>> some odd side effects afterwards though when trying to commit changes\n>> to my super project.\n>\n> submodule is a hot area with lots of development lately I think,\n> perhaps you're seeing some bugs... CCing at least one submodule\n> person...\n>\n>> Apologies if this is answered elsewhere, my google-foo is weak today.\n>>\n>> Many thanks\n>>\n>> Ian\n\n\"git submodule update\" has the network part parallelized,\nbut not the local part IIRC. (That is cloning/fetching submodules\ncan be done with \"-j \", but the local checkout is still serial for\nUX purposes, i.e. it wants to stop at the first conflict and only\nhave one conflict at a time)\n\nWhich odd side effects do you see?\nI'd be curious to see if that is a bug in the code or documentation.\n\nThanks,\nStefan\n"},{"id":"338533","messageId":"20180206201047.GE104086@aiede.svl.corp.google.com","threadId":"47762","inReplyTo":"CAGUnuBHY9nQvGiROm4S_JvBWMrzieHC1FE50fJqxaauzZhH7xw@mail.gmail.com","subject":"Re: Are concurrent git operations on the same repo safe?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-06T20:10:47Z","receivedAt":"2018-02-06T20:11:05Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ian Norton wrote:\n\n> Specifically I'm trying to speed up \"git\n> submodule update\" by doing several at the same time.\n\nCan you say more about this? E.g. how can I reproduce your experience?\nIs there a script I can run?\n\nThanks,\nJonathan\n"},{"id":"338536","messageId":"CAGUnuBHRH5NV31yDiBPdZouNicdfU9z9iPK1wcBNrEhoo+Xw8A@mail.gmail.com","threadId":"47762","inReplyTo":"20180206201047.GE104086@aiede.svl.corp.google.com","subject":"Re: Are concurrent git operations on the same repo safe?","fromName":"Ian Norton","fromEmail":"inorton@gmail.com","sentAt":"2018-02-06T20:26:32Z","receivedAt":"2018-02-06T20:26:52Z","isPatch":false,"sender":{"key":"inorton@gmail.com","avatar":null},"body":"Sure, my office is still devoid of power, i'll have to get back to you\nto be more precise but I was using a morally equivalent process to\nhttps://gitlab.com/inorton/git-multi-sync\n\ncd repo\npython -m gitmultisync --update-submodules\n\nwhere I had a superproject containing 5-6 submodules, some of which\nwere quite large (1-2Gb history)\n\nAfterwards, I had trouble doing \"git pull -r\" in my submodules to pull\nin newer changes.\n\nOn 6 February 2018 at 20:10, Jonathan Nieder wrote:\n> Ian Norton wrote:\n>\n>> Specifically I'm trying to speed up \"git\n>> submodule update\" by doing several at the same time.\n>\n> Can you say more about this? E.g. how can I reproduce your experience?\n> Is there a script I can run?\n>\n> Thanks,\n> Jonathan\n"}]}