# Corrupt repo with force push when using a reference repo 3 messages from 2023-08-15 to 2023-08-22. Participants: Grant Singleton, brian m. carlson, U-AP\GSingleton. Thread: https://gitlist.dev/t/60112 ## Grant Singleton, 2023-08-15 04:05 Subject: Corrupt repo with force push when using a reference repo Message-ID: URL: https://gitlist.dev/e/MN2PR03MB5072F08C68E7F32C9F5AC866A614A%40MN2PR03MB5072.namprd03.prod.outlook.com ``` Thank you for filling out a Git bug report! Please answer the following questions to help us understand your issue. What did you do before the bug happened? (Steps to reproduce your issue) With 3 clones of a repo, one on branch-a one a mirror, another using the mirror as a reference repo on branch-b, do the following # in the repo on branch-a, make a change, commit and push cd bad-object-test echo $(( RANDOM % 1000 + 1 )) > file.txt git ci -am "change a file" git push # do a git fetch in the reference repo cd ../bad-object-test.git/ git fetch # do a git pull in the repo on branch-b cd ../bad-object-test-with-reference-repo/ git pull # All good so far # in the repo on branch-a, discard HEAD and force push cd ../bad-object-test git reset --hard HEAD~1 git push --force # do a git fetch in the reference repo and then prune loose objects cd ../bad-object-test.git/ git fetch git prune -v --expire=now # do a git pull in the repo on branch-b cd ../bad-object-test-with-reference-repo git pull What did you expect to happen? (Expected behavior) the repo on branch-b does a force update of romote tracking branch branch-a What happened instead? (Actual behavior) xxx> git pull fatal: bad orefs/remotes/origin/branch-a error: https://github.com/xxxxx/yyyyyyy.git did not send all necessary objects What's different between what you expected and what actually happened? I expected the force push to succeed, but insted I get a corrupt repo Anything else you want to add: git rev-parse refs/remotes/origin/branch-a will return the hash of the commit that was discarded by the git reset command Please review the rest of the bug report below. You can delete any lines you don't wish to share. [System Info] git version: git version 2.41.0 cpu: x86_64 no commit associated with this build sizeof-long: 8 sizeof-size_t: 8 shell-path: /bin/sh uname: Linux 5.15.0-79-generic #86-Ubuntu SMP Mon Jul 10 16:07:21 UTC 2023 x86_64 compiler info: gnuc: 11.3 libc info: glibc: 2.35 $SHELL (typically, interactive shell): /bin/bash [Enabled Hooks] ``` ## brian m. carlson, 2023-08-16 01:21 Subject: Re: Corrupt repo with force push when using a reference repo Message-ID: URL: https://gitlist.dev/e/ZNwkh47mBJxcSYkm%40tapette.crustytoothpaste.net In-Reply-To: ``` On 2023-08-15 at 04:05:34, Grant Singleton wrote: > Thank you for filling out a Git bug report! > Please answer the following questions to help us understand your issue. > > What did you do before the bug happened? (Steps to reproduce your issue) > > With 3 clones of a repo, one on branch-a one a mirror, another using the mirror as a reference repo on branch-b, do the following > > # in the repo on branch-a, make a change, commit and push > cd bad-object-test > echo $(( RANDOM % 1000 + 1 )) > file.txt > git ci -am "change a file" > git push > > # do a git fetch in the reference repo > cd ../bad-object-test.git/ > git fetch > > # do a git pull in the repo on branch-b > cd ../bad-object-test-with-reference-repo/ > git pull > > # All good so far > > # in the repo on branch-a, discard HEAD and force push > cd ../bad-object-test > git reset --hard HEAD~1 > git push --force > > # do a git fetch in the reference repo and then prune loose objects > cd ../bad-object-test.git/ > git fetch > git prune -v --expire=now When you clone using --shared or --reference, you implicitly depend on the objects in the referenced repository existing without being pruned for the integrity of the repository. The manual page for git clone says this: NOTE: this is a possibly dangerous operation; do not use it unless you understand what it does. If you clone your repository using this option and then delete branches (or use any other Git command that makes any existing commit unreferenced) in the source repository, some objects may become unreferenced (or dangling). These objects may be removed by normal Git operations (such as git commit) which automatically call git maintenance run --auto. (See git‐maintenance(1).) If these objects are removed and were referenced by the cloned repository, then the cloned repository will become corrupt. If you want to change the references in any non-fast-forward way or do any sort of GC or pruning in the reference repository, you need to run `git repack -a` in the non-reference repository to repack all of the objects in that repository and dissociate it from the reference repository. In general, unless you are _very_ sure you know what you're doing, you usually want to use `--dissociate` with `--reference` such that you detach immediately. -- brian m. carlson (he/him or they/them) Toronto, Ontario, CA ``` ## U-AP\GSingleton, 2023-08-22 04:05 Subject: Re: Corrupt repo with force push when using a reference repo Message-ID: URL: https://gitlist.dev/e/ZOQ0GxPn-anJlLxQ%40GSINGLE-NZ-DE.ap.trimblecorp.net In-Reply-To: ``` On Wed, Aug 16, 2023 at 01:21:11AM +0000, brian m. carlson wrote: > On 2023-08-15 at 04:05:34, Grant Singleton wrote: > > Thank you for filling out a Git bug report! > > Please answer the following questions to help us understand your issue. > > > > What did you do before the bug happened? (Steps to reproduce your issue) > > > > With 3 clones of a repo, one on branch-a one a mirror, another using the mirror as a reference repo on branch-b, do the following > > > > # in the repo on branch-a, make a change, commit and push > > cd bad-object-test > > echo $(( RANDOM % 1000 + 1 )) > file.txt > > git ci -am "change a file" > > git push > > > > # do a git fetch in the reference repo > > cd ../bad-object-test.git/ > > git fetch > > > > # do a git pull in the repo on branch-b > > cd ../bad-object-test-with-reference-repo/ > > git pull > > > > # All good so far > > > > # in the repo on branch-a, discard HEAD and force push > > cd ../bad-object-test > > git reset --hard HEAD~1 > > git push --force > > > > # do a git fetch in the reference repo and then prune loose objects > > cd ../bad-object-test.git/ > > git fetch > > git prune -v --expire=now > > When you clone using --shared or --reference, you implicitly depend on > the objects in the referenced repository existing without being pruned > for the integrity of the repository. > > The manual page for git clone says this: > > NOTE: this is a possibly dangerous operation; do not use it unless you > understand what it does. If you clone your repository using this > option and then delete branches (or use any other Git command that > makes any existing commit unreferenced) in the source repository, some > objects may become unreferenced (or dangling). These objects may be > removed by normal Git operations (such as git commit) which > automatically call git maintenance run --auto. (See > git‐maintenance(1).) If these objects are removed and were referenced > by the cloned repository, then the cloned repository will become > corrupt. > > If you want to change the references in any non-fast-forward way or do > any sort of GC or pruning in the reference repository, you need to run > `git repack -a` in the non-reference repository to repack all of the > objects in that repository and dissociate it from the reference > repository. > > In general, unless you are _very_ sure you know what you're doing, you > usually want to use `--dissociate` with `--reference` such that you > detach immediately. > -- > brian m. carlson (he/him or they/them) > Toronto, Ontario, CA Ok, sorry and thanks, I should have read more of the documentation. For some context, we're doing this on a bunch of Jenkins agents for our CI/CD pipeline on a somewhat bloating monolithic repo. We have 30-40 clones on each machine, so the reference repos save us a lot of network traffic, time to fetch and disk space. So IIUC I can do a `git repack -kad` in the reference repo to pack loose objects and keep unreachable commits that may be referenced by other repos using the reference repo. Grant Singleton Christchurch, New Zealand ```