{"thread":{"id":"53277","subject":"Multiple pushurls, different 'objects\\info\\packs'","startedAt":"2020-04-21T18:07:57Z","lastAt":"2020-04-22T17:51:18Z","messageCount":3,"participants":["Mikhail Strelnikov","brian m. carlson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"395840","messageId":"CAAihzVJTN5V3o_aFeUuJH_8+nqZrKHJww3vFQ9K5W0DtbeM14w@mail.gmail.com","threadId":"53277","inReplyTo":null,"subject":"Multiple pushurls, different 'objects\\info\\packs'","fromName":"Mikhail Strelnikov","fromEmail":"mikhail.strelnikov@gmail.com","sentAt":"2020-04-21T18:07:42Z","receivedAt":"2020-04-21T18:07:57Z","isPatch":false,"sender":{"key":"mikhail.strelnikov@gmail.com","avatar":null},"body":"Hi,\n\nI have a repo with two pushurls configured like this:\n\n\nC:\\folder\\1>git init --bare\nInitialized empty Git repository in C:/folder/1/\n\nC:\\folder\\2>git init --bare\nInitialized empty Git repository in C:/folder/2/\n\nC:\\folder\\w>git init\nInitialized empty Git repository in C:/folder/w/.git/\n\nC:\\folder\\w>git add work.txt\n\nC:\\folder\\w>git commit -m \"Initial commit\"\n[master (root-commit) 1b314f3] Initial commit\n1 file changed, 1 insertion(+)\ncreate mode 100644 work.txt\n\nC:\\folder\\w>git remote add origin C:\\folder\\1\n\nC:\\folder\\w>git remote set-url origin --push --add C:\\folder\\1\n\nC:\\folder\\w>git remote set-url origin --push --add C:\\folder\\2\n\nC:\\folder\\w>git push --set-upstream origin master\n\n\nI would expect those two folders (C:\\folder\\1 and C:\\folder\\2) to\ncontain exactly the same bytes. And they did for quite some time. But\nnow there is a difference in 'objects\\info\\packs' (and some of\nobjects\\pack\\pack-*.idx/pack are also different).\n\n(all the commits are the same in both and all my data is also the same\nand 'fast-export --all' yields the same result)\n\nI'd like to know what might have caused this nondeterminism and if\nthere is something to do to prevent that.\n\n(my current version of git is 2.25.0.windows.1, but I guess this\ndivergence happened several versions back)\n\nThanks.\n"},{"id":"395880","messageId":"20200422031455.GG6465@camp.crustytoothpaste.net","threadId":"53277","inReplyTo":"CAAihzVJTN5V3o_aFeUuJH_8+nqZrKHJww3vFQ9K5W0DtbeM14w@mail.gmail.com","subject":"Re: Multiple pushurls, different 'objects\\info\\packs'","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-04-22T03:14:55Z","receivedAt":"2020-04-22T03:15:32Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-04-21 at 18:07:42, Mikhail Strelnikov wrote:\n> Hi,\n> \n> I have a repo with two pushurls configured like this:\n> \n> \n> C:\\folder\\1>git init --bare\n> Initialized empty Git repository in C:/folder/1/\n> \n> C:\\folder\\2>git init --bare\n> Initialized empty Git repository in C:/folder/2/\n> \n> C:\\folder\\w>git init\n> Initialized empty Git repository in C:/folder/w/.git/\n> \n> C:\\folder\\w>git add work.txt\n> \n> C:\\folder\\w>git commit -m \"Initial commit\"\n> [master (root-commit) 1b314f3] Initial commit\n> 1 file changed, 1 insertion(+)\n> create mode 100644 work.txt\n> \n> C:\\folder\\w>git remote add origin C:\\folder\\1\n> \n> C:\\folder\\w>git remote set-url origin --push --add C:\\folder\\1\n> \n> C:\\folder\\w>git remote set-url origin --push --add C:\\folder\\2\n> \n> C:\\folder\\w>git push --set-upstream origin master\n> \n> \n> I would expect those two folders (C:\\folder\\1 and C:\\folder\\2) to\n> contain exactly the same bytes. And they did for quite some time. But\n> now there is a difference in 'objects\\info\\packs' (and some of\n> objects\\pack\\pack-*.idx/pack are also different).\n> \n> (all the commits are the same in both and all my data is also the same\n> and 'fast-export --all' yields the same result)\n> \n> I'd like to know what might have caused this nondeterminism and if\n> there is something to do to prevent that.\n\nYou can get nondeterminism because the push to each repository happens\nindependently and delta compression is multithreaded.  You can therefore\ncompute different packs on push and get different packs in the result.\n\nYou could try to avoid it by disabling threading for pushes, but that\nhas to be done on each client that pushes to them.  In general, this is\nnot worth worrying about as long as the data is intact (that is, it\npasses git fsck) and the refs are identical.  It is also not especially\neasy to avoid, since determinism of pack files is not considered a goal\nof Git.\n\nMaybe if you tell us a little more about your reason for wanting\nbit-for-bit identical replicas we can provide some assistance in helping\nyou achieve your goals.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"395920","messageId":"CAAihzVLiB4Cx3BC66bqXMVgAgacRiNGpviUJHROtiJYwxNYCSg@mail.gmail.com","threadId":"53277","inReplyTo":"20200422031455.GG6465@camp.crustytoothpaste.net","subject":"Re: Multiple pushurls, different 'objects\\info\\packs'","fromName":"Mikhail Strelnikov","fromEmail":"mikhail.strelnikov@gmail.com","sentAt":"2020-04-22T17:45:04Z","receivedAt":"2020-04-22T17:51:18Z","isPatch":false,"sender":{"key":"mikhail.strelnikov@gmail.com","avatar":null},"body":"Thanks Brian,\n\n> delta compression is multithreaded\n\nThat is great news that this is expected behavior and I'm not dealing\nwith some sort of obscure data corruption. :)\nNow see there is `git config --global pack.threads 1` (I'm doing\neverything locally, so this might work for me)\n\n> reason for wanting bit-for-bit identical\n\nMy plan was to have 3 pushurls and shoot one of them if it is not\nbit-for-bit identical to the other two. But you are right, all I need\nis `git fsck`.\n\nOn Wed, Apr 22, 2020 at 6:15 AM brian m. carlson\n<sandals@crustytoothpaste.net> wrote:\n>\n> On 2020-04-21 at 18:07:42, Mikhail Strelnikov wrote:\n> > Hi,\n> >\n> > I have a repo with two pushurls configured like this:\n> >\n> >\n> > C:\\folder\\1>git init --bare\n> > Initialized empty Git repository in C:/folder/1/\n> >\n> > C:\\folder\\2>git init --bare\n> > Initialized empty Git repository in C:/folder/2/\n> >\n> > C:\\folder\\w>git init\n> > Initialized empty Git repository in C:/folder/w/.git/\n> >\n> > C:\\folder\\w>git add work.txt\n> >\n> > C:\\folder\\w>git commit -m \"Initial commit\"\n> > [master (root-commit) 1b314f3] Initial commit\n> > 1 file changed, 1 insertion(+)\n> > create mode 100644 work.txt\n> >\n> > C:\\folder\\w>git remote add origin C:\\folder\\1\n> >\n> > C:\\folder\\w>git remote set-url origin --push --add C:\\folder\\1\n> >\n> > C:\\folder\\w>git remote set-url origin --push --add C:\\folder\\2\n> >\n> > C:\\folder\\w>git push --set-upstream origin master\n> >\n> >\n> > I would expect those two folders (C:\\folder\\1 and C:\\folder\\2) to\n> > contain exactly the same bytes. And they did for quite some time. But\n> > now there is a difference in 'objects\\info\\packs' (and some of\n> > objects\\pack\\pack-*.idx/pack are also different).\n> >\n> > (all the commits are the same in both and all my data is also the same\n> > and 'fast-export --all' yields the same result)\n> >\n> > I'd like to know what might have caused this nondeterminism and if\n> > there is something to do to prevent that.\n>\n> You can get nondeterminism because the push to each repository happens\n> independently and delta compression is multithreaded.  You can therefore\n> compute different packs on push and get different packs in the result.\n>\n> You could try to avoid it by disabling threading for pushes, but that\n> has to be done on each client that pushes to them.  In general, this is\n> not worth worrying about as long as the data is intact (that is, it\n> passes git fsck) and the refs are identical.  It is also not especially\n> easy to avoid, since determinism of pack files is not considered a goal\n> of Git.\n>\n> Maybe if you tell us a little more about your reason for wanting\n> bit-for-bit identical replicas we can provide some assistance in helping\n> you achieve your goals.\n> --\n> brian m. carlson: Houston, Texas, US\n> OpenPGP: https://keybase.io/bk2204\n"}]}