{"thread":{"id":"60112","subject":"Corrupt repo with force push when using a reference repo","startedAt":"2023-08-15T04:16:36Z","lastAt":"2023-08-22T04:05:55Z","messageCount":3,"participants":["Grant Singleton","brian m. carlson","U-AP\\GSingleton"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"480668","messageId":"MN2PR03MB5072F08C68E7F32C9F5AC866A614A@MN2PR03MB5072.namprd03.prod.outlook.com","threadId":"60112","inReplyTo":null,"subject":"Corrupt repo with force push when using a reference repo","fromName":"Grant Singleton","fromEmail":"grant_singleton@trimble.com","sentAt":"2023-08-15T04:05:34Z","receivedAt":"2023-08-15T04:16:36Z","isPatch":false,"sender":{"key":"grant_singleton@trimble.com","avatar":null},"body":"Thank you for filling out a Git bug report!\nPlease answer the following questions to help us understand your issue.\n\nWhat did you do before the bug happened? (Steps to reproduce your issue)\n\nWith 3 clones of a repo, one on branch-a one a mirror,  another using the mirror as a reference repo on branch-b, do the following\n\n# in the repo on branch-a, make a change, commit and push\ncd bad-object-test\necho $(( RANDOM % 1000 + 1 )) > file.txt\ngit ci -am \"change a file\"\ngit push\n\n# do a git fetch in the reference repo\ncd ../bad-object-test.git/\ngit fetch\n\n# do a git pull in the repo on branch-b\ncd ../bad-object-test-with-reference-repo/\ngit pull\n\n# All good so far\n\n# in the repo on branch-a, discard HEAD and force push\ncd ../bad-object-test\ngit reset --hard HEAD~1\ngit push --force\n\n# do a git fetch in the reference repo and then prune loose objects\ncd ../bad-object-test.git/\ngit fetch\ngit prune -v --expire=now\n\n# do a git pull in the repo on branch-b\ncd ../bad-object-test-with-reference-repo\ngit pull\n\n\nWhat did you expect to happen? (Expected behavior)\n\nthe repo on branch-b does a force update of romote tracking branch branch-a\n\n\nWhat happened instead? (Actual behavior)\nxxx> git pull\nfatal: bad orefs/remotes/origin/branch-a\nerror: https://github.com/xxxxx/yyyyyyy.git did not send all necessary objects\n\n\nWhat's different between what you expected and what actually happened?\n\nI expected the force push to succeed, but insted I get a corrupt repo\n\n\nAnything else you want to add:\n\ngit rev-parse refs/remotes/origin/branch-a\nwill return the hash of the commit that was discarded by the git reset command\n\n\nPlease review the rest of the bug report below.\nYou can delete any lines you don't wish to share.\n\n\n[System Info]\ngit version:\ngit version 2.41.0\ncpu: x86_64\nno commit associated with this build\nsizeof-long: 8\nsizeof-size_t: 8\nshell-path: /bin/sh\nuname: Linux 5.15.0-79-generic #86-Ubuntu SMP Mon Jul 10 16:07:21 UTC 2023 x86_64\ncompiler info: gnuc: 11.3\nlibc info: glibc: 2.35\n$SHELL (typically, interactive shell): /bin/bash\n\n\n[Enabled Hooks]"},{"id":"480701","messageId":"ZNwkh47mBJxcSYkm@tapette.crustytoothpaste.net","threadId":"60112","inReplyTo":"MN2PR03MB5072F08C68E7F32C9F5AC866A614A@MN2PR03MB5072.namprd03.prod.outlook.com","subject":"Re: Corrupt repo with force push when using a reference repo","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2023-08-16T01:21:11Z","receivedAt":"2023-08-16T01:22:13Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2023-08-15 at 04:05:34, Grant Singleton wrote:\n> Thank you for filling out a Git bug report!\n> Please answer the following questions to help us understand your issue.\n> \n> What did you do before the bug happened? (Steps to reproduce your issue)\n> \n> With 3 clones of a repo, one on branch-a one a mirror,  another using the mirror as a reference repo on branch-b, do the following\n> \n> # in the repo on branch-a, make a change, commit and push\n> cd bad-object-test\n> echo $(( RANDOM % 1000 + 1 )) > file.txt\n> git ci -am \"change a file\"\n> git push\n> \n> # do a git fetch in the reference repo\n> cd ../bad-object-test.git/\n> git fetch\n> \n> # do a git pull in the repo on branch-b\n> cd ../bad-object-test-with-reference-repo/\n> git pull\n> \n> # All good so far\n> \n> # in the repo on branch-a, discard HEAD and force push\n> cd ../bad-object-test\n> git reset --hard HEAD~1\n> git push --force\n> \n> # do a git fetch in the reference repo and then prune loose objects\n> cd ../bad-object-test.git/\n> git fetch\n> git prune -v --expire=now\n\nWhen you clone using --shared or --reference, you implicitly depend on\nthe objects in the referenced repository existing without being pruned\nfor the integrity of the repository.\n\nThe manual page for git clone says this:\n\n  NOTE: this is a possibly dangerous operation; do not use it unless you\n  understand what it does. If you clone your repository using this\n  option and then delete branches (or use any other Git command that\n  makes any existing commit unreferenced) in the source repository, some\n  objects may become unreferenced (or dangling). These objects may be\n  removed by normal Git operations (such as git commit) which\n  automatically call git maintenance run --auto. (See\n  git‐maintenance(1).) If these objects are removed and were referenced\n  by the cloned repository, then the cloned repository will become\n  corrupt.\n\nIf you want to change the references in any non-fast-forward way or do\nany sort of GC or pruning in the reference repository, you need to run\n`git repack -a` in the non-reference repository to repack all of the\nobjects in that repository and dissociate it from the reference\nrepository.\n\nIn general, unless you are _very_ sure you know what you're doing, you\nusually want to use `--dissociate` with `--reference` such that you\ndetach immediately.\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"480915","messageId":"ZOQ0GxPn-anJlLxQ@GSINGLE-NZ-DE.ap.trimblecorp.net","threadId":"60112","inReplyTo":"ZNwkh47mBJxcSYkm@tapette.crustytoothpaste.net","subject":"Re: Corrupt repo with force push when using a reference repo","fromName":"U-AP\\GSingleton","fromEmail":"grant_singleton@trimble.com","sentAt":"2023-08-22T04:05:47Z","receivedAt":"2023-08-22T04:05:55Z","isPatch":false,"sender":{"key":"grant_singleton@trimble.com","avatar":null},"body":"On Wed, Aug 16, 2023 at 01:21:11AM +0000, brian m. carlson wrote:\n> On 2023-08-15 at 04:05:34, Grant Singleton wrote:\n> > Thank you for filling out a Git bug report!\n> > Please answer the following questions to help us understand your issue.\n> > \n> > What did you do before the bug happened? (Steps to reproduce your issue)\n> > \n> > With 3 clones of a repo, one on branch-a one a mirror,  another using the mirror as a reference repo on branch-b, do the following\n> > \n> > # in the repo on branch-a, make a change, commit and push\n> > cd bad-object-test\n> > echo $(( RANDOM % 1000 + 1 )) > file.txt\n> > git ci -am \"change a file\"\n> > git push\n> > \n> > # do a git fetch in the reference repo\n> > cd ../bad-object-test.git/\n> > git fetch\n> > \n> > # do a git pull in the repo on branch-b\n> > cd ../bad-object-test-with-reference-repo/\n> > git pull\n> > \n> > # All good so far\n> > \n> > # in the repo on branch-a, discard HEAD and force push\n> > cd ../bad-object-test\n> > git reset --hard HEAD~1\n> > git push --force\n> > \n> > # do a git fetch in the reference repo and then prune loose objects\n> > cd ../bad-object-test.git/\n> > git fetch\n> > git prune -v --expire=now\n> \n> When you clone using --shared or --reference, you implicitly depend on\n> the objects in the referenced repository existing without being pruned\n> for the integrity of the repository.\n> \n> The manual page for git clone says this:\n> \n>   NOTE: this is a possibly dangerous operation; do not use it unless you\n>   understand what it does. If you clone your repository using this\n>   option and then delete branches (or use any other Git command that\n>   makes any existing commit unreferenced) in the source repository, some\n>   objects may become unreferenced (or dangling). These objects may be\n>   removed by normal Git operations (such as git commit) which\n>   automatically call git maintenance run --auto. (See\n>   git‐maintenance(1).) If these objects are removed and were referenced\n>   by the cloned repository, then the cloned repository will become\n>   corrupt.\n> \n> If you want to change the references in any non-fast-forward way or do\n> any sort of GC or pruning in the reference repository, you need to run\n> `git repack -a` in the non-reference repository to repack all of the\n> objects in that repository and dissociate it from the reference\n> repository.\n> \n> In general, unless you are _very_ sure you know what you're doing, you\n> usually want to use `--dissociate` with `--reference` such that you\n> detach immediately.\n> -- \n> brian m. carlson (he/him or they/them)\n> Toronto, Ontario, CA\n\nOk, sorry and thanks, I should have read more of the documentation.\n\nFor some context, we're doing this on a bunch of Jenkins agents for our\nCI/CD pipeline on a somewhat bloating monolithic repo. We have 30-40\nclones on each machine, so the reference repos save us a lot of network\ntraffic, time to fetch and disk space.\n\nSo IIUC I can do a `git repack -kad` in the reference repo to pack loose\nobjects and keep unreachable commits that may be referenced by other\nrepos using the reference repo.\n\nGrant Singleton\nChristchurch, New Zealand\n\n"}]}