{"thread":{"id":"48895","subject":"Are clone/checkout operations deterministic?","startedAt":"2018-07-17T09:14:34Z","lastAt":"2018-07-17T19:52:19Z","messageCount":5,"participants":["J. Paul Reed","Ævar Arnfjörð Bjarmason","Stefan Beller","Duy Nguyen","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"352753","messageId":"20180717091429.GA31043@sigkill.com","threadId":"48895","inReplyTo":null,"subject":"Are clone/checkout operations deterministic?","fromName":"J. Paul Reed","fromEmail":"preed@sigkill.com","sentAt":"2018-07-17T09:14:29Z","receivedAt":"2018-07-17T09:14:34Z","isPatch":false,"sender":{"key":"preed@sigkill.com","avatar":"https://gravatar.com/avatar/a6d898ae24f856b851376d22623fd35bb37134ad6df0483b84b2b135c4878237?d=mp&s=160"},"body":"\nHey Git Devs,\n\nI have a bit of an odd question: do git clone/checkout operations have a\ndeterministic ordering?\n\nThat is: are files guaranteed to be laid down onto disk in any specific\n(and deterministic) order as a clone and/or checkout operations occurs?\n(And if so, is it alphabetical order? Depth-first? Something else?)\n\nIn case the answer is different (and I'd guess that it might be?), I'm\nmostly interested in the initial clone case... but it would be great to\nknow if, indeed, the answer is different for just-checkouts too.\n\nI did some cursory googling, but nothing hopped out at me as an answer to\nthis question.\n\nThanks!\n\n-preed\n-- \nJ. Paul Reed                                              PGP: 0xDF8708F8\n=========================================================================\nI've never seen an airplane yet that can read the type ratings on your\npilot's license.                                       -- Chuck Boedecker\n\n"},{"id":"352755","messageId":"87bmb6chvm.fsf@evledraar.gmail.com","threadId":"48895","inReplyTo":"20180717091429.GA31043@sigkill.com","subject":"Re: Are clone/checkout operations deterministic?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-07-17T09:48:45Z","receivedAt":"2018-07-17T09:48:51Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jul 17 2018, J. Paul Reed wrote:\n\n> Hey Git Devs,\n>\n> I have a bit of an odd question: do git clone/checkout operations have a\n> deterministic ordering?\n>\n> That is: are files guaranteed to be laid down onto disk in any specific\n> (and deterministic) order as a clone and/or checkout operations occurs?\n> (And if so, is it alphabetical order? Depth-first? Something else?)\n>\n> In case the answer is different (and I'd guess that it might be?), I'm\n> mostly interested in the initial clone case... but it would be great to\n> know if, indeed, the answer is different for just-checkouts too.\n>\n> I did some cursory googling, but nothing hopped out at me as an answer to\n> this question.\n\nIn practice I think clone, checkout, reset etc. always work in the same\norder you see with `git ls-tree -r --name-only HEAD`, but as far as I\nknow this has never been guaranteed or documented, and shouldn't be\nrelied on.\n\nE.g. there's probably cases where writing files in parallel is going to\nbe faster than writing them sequentially. We don't have such a mode just\nbecause nobody's written a patch for it, but having that patch would\nbreak any assumptions of our current order.\n"},{"id":"352809","messageId":"CAGZ79kbJkocY0oe2_efEA3s0dGgR4tnvwpvUTNuJuSSYY=MazQ@mail.gmail.com","threadId":"48895","inReplyTo":"87bmb6chvm.fsf@evledraar.gmail.com","subject":"Re: Are clone/checkout operations deterministic?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-07-17T17:58:30Z","receivedAt":"2018-07-17T17:58:44Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Jul 17, 2018 at 2:48 AM Ævar Arnfjörð Bjarmason\n<avarab@gmail.com> wrote:\n>\n>\n> On Tue, Jul 17 2018, J. Paul Reed wrote:\n>\n> > Hey Git Devs,\n> >\n> > I have a bit of an odd question: do git clone/checkout operations have a\n> > deterministic ordering?\n> >\n> > That is: are files guaranteed to be laid down onto disk in any specific\n> > (and deterministic) order as a clone and/or checkout operations occurs?\n> > (And if so, is it alphabetical order? Depth-first? Something else?)\n> >\n> > In case the answer is different (and I'd guess that it might be?), I'm\n> > mostly interested in the initial clone case... but it would be great to\n> > know if, indeed, the answer is different for just-checkouts too.\n> >\n> > I did some cursory googling, but nothing hopped out at me as an answer to\n> > this question.\n>\n> In practice I think clone, checkout, reset etc. always work in the same\n> order you see with `git ls-tree -r --name-only HEAD`, but as far as I\n> know this has never been guaranteed or documented, and shouldn't be\n> relied on.\n\nThe transmission of packfiles is non-deterministic, as the packfiles\n(which are packed for each clone separately when using core git as\na server) are not packed in a deterministic fashion, but in a threaded\nenvironment which allows different packing orders.\n\nIf you clone from a server that gives you exactly the same pack at\nall times (assuming the remote repo doesn't change refs), then\ncheckout is currently deterministic in unpacking files to the working tree.\n\n>\n> E.g. there's probably cases where writing files in parallel is going to\n> be faster than writing them sequentially. We don't have such a mode just\n> because nobody's written a patch for it, but having that patch would\n> break any assumptions of our current order.\n\n+cc Ben who is looking into that, but hasn't spoken up on the mailing list yet.\n"},{"id":"352810","messageId":"CACsJy8A0KUyxK_2NAMh+da9yithZM5d68rhqEVZe3NcMxinAjA@mail.gmail.com","threadId":"48895","inReplyTo":"87bmb6chvm.fsf@evledraar.gmail.com","subject":"Re: Are clone/checkout operations deterministic?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-07-17T18:06:44Z","receivedAt":"2018-07-17T18:07:13Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jul 17, 2018 at 11:50 AM Ævar Arnfjörð Bjarmason\n<avarab@gmail.com> wrote:\n> In practice I think clone, checkout, reset etc. always work in the same\n> order you see with `git ls-tree -r --name-only HEAD`, but as far as I\n> know this has never been guaranteed or documented, and shouldn't be\n> relied on.\n>\n> E.g. there's probably cases where writing files in parallel is going to\n> be faster than writing them sequentially. We don't have such a mode just\n> because nobody's written a patch for it, but having that patch would\n\nWell I did have some patches [1] but as usually I did not follow\nthrough. Interestingly there's actually concern about timestamp order\n[2] even back then\n\n[1] https://public-inbox.org/git/20160415095139.GA3985@lanh/\n[2] https://public-inbox.org/git/CAP8UFD0WZHriY340eh3K6ygzb0tXnoT+XaY8+c2k+N2x9UBYxA@mail.gmail.com/\n\n> break any assumptions of our current order.\n-- \nDuy\n"},{"id":"352824","messageId":"20180717195215.GA26218@sigill.intra.peff.net","threadId":"48895","inReplyTo":"87bmb6chvm.fsf@evledraar.gmail.com","subject":"Re: Are clone/checkout operations deterministic?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-07-17T19:52:16Z","receivedAt":"2018-07-17T19:52:19Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 17, 2018 at 11:48:45AM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> In practice I think clone, checkout, reset etc. always work in the same\n> order you see with `git ls-tree -r --name-only HEAD`, but as far as I\n> know this has never been guaranteed or documented, and shouldn't be\n> relied on.\n\nI think this paragraph is correct in general (and I agree with the\nsentiment that this is subject to change in future versions).\n\nThere is one concrete case I know that has non-deterministic order in\ncurrent versions: long-lived clean/smudge filters can defer their\nresponse. The LFS filter uses this to tell Git \"no, I'm still\ndownloading the content\", at which point Git will proceed with checking\nout other local files (or even other LFS files that happen to arrive\nsooner).\n\nDepending on what one wants to do with the determinism, it may be OK to\nignore that case. ;)\n\n-Peff\n"}]}