{"thread":{"id":"62693","subject":"\"git fetch\" fails for a --reference clone after an outer forced push","startedAt":"2024-12-25T17:23:20Z","lastAt":"2024-12-27T19:09:37Z","messageCount":5,"participants":["Mirochnik, Oleg V","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"509556","messageId":"SJ0PR11MB581445B096273126D18F3724C60C2@SJ0PR11MB5814.namprd11.prod.outlook.com","threadId":"62693","inReplyTo":null,"subject":"\"git fetch\" fails for a --reference clone after an outer forced push","fromName":"Mirochnik, Oleg V","fromEmail":"oleg.v.mirochnik@intel.com","sentAt":"2024-12-25T17:23:13Z","receivedAt":"2024-12-25T17:23:20Z","isPatch":false,"sender":{"key":"oleg.v.mirochnik@intel.com","avatar":null},"body":"Hi all,\n\nTo reduce traffic, we use local mirrors via \"git clone --reference mirror ...\" commands.\nAnd we regularly get problems with update of few weeks old clones and must run something like below to get rid of outdated branches:\n\ngit for-each-ref --format=\"%(refname)\" | while read ref; do\n    git show-ref --quiet --verify $ref 2>/dev/null || git update-ref -d $ref\ndone\n\nThis can be quite long operation as number of refs is significant.\n\nDetails are below.\n\nThanks,\nOleg.\n\n\nThank you for filling out a Git bug report!\nPlease answer the following questions to help us understand your issue.\n\nWhat did you do before the bug happened? (Steps to reproduce your issue)\n\n$ cat ./doit\n#!/bin/sh\nset -xe\nrm -rf tst\nmkdir tst\ncd tst\nmkdir master\ngit -C master init --bare\ngit clone master local\ntouch local/foo\ngit -C local add .\ngit -C local commit -m init-commit\ngit -C local push\necho foo > local/foo\ngit -C local commit -a -m dummy-commit\ngit -C local push origin HEAD:refs/heads/dummy\ngit clone --mirror file://`pwd`/master mirror\ngit clone --reference `pwd`/mirror file://`pwd`/master local1\ngit -C local1 log --oneline origin/dummy\ngit -C local commit --amend -m new-dummy-commit\ngit -C local push -f origin HEAD:dummy\ngit -C mirror fetch\ngit -C mirror gc --prune=now\ngit -C local1 fetch\ngit -C local1 log --oneline origin/dummy\n\nWhat did you expect to happen? (Expected behavior)\n\nNo errors; last few lines like:\n\n+ git -C local1 fetch\nFrom file:///tmp/tst/master\n + 552a293...248c7fa dummy      -> origin/dummy  (forced update)\n+ git -C local1 log --oneline origin/dummy\n248c7fa (origin/dummy) new-dummy-commit\n0636429 (HEAD -> master, origin/master, origin/HEAD) init-commit\n\nWhat happened instead? (Actual behavior)\n\n+ git -C local1 fetch\nfatal: bad object refs/remotes/origin/dummy\nerror: file:///tmp/tst/master did not send all necessary objects\n\nWhat's different between what you expected and what actually happened?\n\nObvious - should work and does not work\n\nAnything else you want to add:\n\nIt works if \"git -C mirror gc --prune=now\" step is skipped or if local path is used for the clone steps instead of the \"file://\"\nThe same issue is with https://github.com/...\nThe same issue if use \"git -C local1 remote update --prune\" instead of the \"git -C local1 fetch\"\n\nPlease review the rest of the bug report below.\nYou can delete any lines you don't wish to share.\n\npodman ub24 container is used to get a newer git version.\nExactly the same behavior is for the git v2.39.1\n\n[System Info]\ngit version:\ngit version 2.43.0\ncpu: x86_64\nno commit associated with this build\nsizeof-long: 8\nsizeof-size_t: 8\nshell-path: /bin/sh\nuname: Linux 5.14.0-70.13.1.el9_0.x86_64 #1 SMP PREEMPT Thu Apr 14 12:42:38 EDT 2022 x86_64\ncompiler info: gnuc: 13.2\nlibc info: glibc: 2.39\n$SHELL (typically, interactive shell): <unset>\n\n\n[Enabled Hooks]\nnot run from a git repository - no hooks to show\n"},{"id":"509565","messageId":"20241226155251.GA69868@coredump.intra.peff.net","threadId":"62693","inReplyTo":"SJ0PR11MB581445B096273126D18F3724C60C2@SJ0PR11MB5814.namprd11.prod.outlook.com","subject":"Re: \"git fetch\" fails for a --reference clone after an outer forced push","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-12-26T15:52:51Z","receivedAt":"2024-12-26T15:52:53Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 25, 2024 at 05:23:13PM +0000, Mirochnik, Oleg V wrote:\n\n> What did you do before the bug happened? (Steps to reproduce your issue)\n> \n> $ cat ./doit\n> #!/bin/sh\n> set -xe\n> rm -rf tst\n> mkdir tst\n> cd tst\n> mkdir master\n> git -C master init --bare\n> git clone master local\n> touch local/foo\n> git -C local add .\n> git -C local commit -m init-commit\n> git -C local push\n> echo foo > local/foo\n> git -C local commit -a -m dummy-commit\n> git -C local push origin HEAD:refs/heads/dummy\n> git clone --mirror file://`pwd`/master mirror\n> git clone --reference `pwd`/mirror file://`pwd`/master local1\n> git -C local1 log --oneline origin/dummy\n> git -C local commit --amend -m new-dummy-commit\n> git -C local push -f origin HEAD:dummy\n> git -C mirror fetch\n> git -C mirror gc --prune=now\n> git -C local1 fetch\n> git -C local1 log --oneline origin/dummy\n>\n> [...]\n>\n> What happened instead? (Actual behavior)\n> \n> + git -C local1 fetch\n> fatal: bad object refs/remotes/origin/dummy\n> error: file:///tmp/tst/master did not send all necessary objects\n\nThis is the expected behavior, and what the warning in \"git help clone\"\nis talking about:\n\n  NOTE: this is a possibly dangerous operation; do not use it unless you\n  understand what it does. If you clone your repository using this\n  option and then delete branches (or use any other Git command that\n  makes any existing commit unreferenced) in the source repository, some\n  objects may become unreferenced (or dangling). These objects may be\n  removed by normal Git operations (such as git commit) which\n  automatically call git maintenance run --auto. (See\n  git-maintenance(1).) If these objects are removed and were referenced\n  by the cloned repository, then the cloned repository will become\n  corrupt.\n\nYour \"mirror\" repository has no idea that other repositories are\ndepending on it. To safely do a \"git gc\" there, it would need to know\nall of the objects that are referenced by the dependent repositories, to\ncount them as reachable.\n\nOne way to do that is something like:\n\n  1. Enable the \"preciousObjects\" flag in the mirror repo, to prevent\n     accidental destruction (e.g., from auto-gc):\n\n       git -C mirror config core.repositoryFormatVersion 1\n       git -C mirror config extensions.preciousObjects true\n\n  2. When you do want to run gc on the mirror repo, collect all of the\n     references from child repos first:\n\n       # collect references from all child repos; the destination\n       # doesn't really matter here, and you could even delete\n       # refs/child/* after the gc if you want\n       for $repo in local*; do\n         git -C mirror fetch --prune ../$repo refs/*:refs/child/$repo/*\n       done\n\n       # now gc, disabling preciousObjects temporarily\n       git -c extensions.preciousObjects=false gc --prune=now\n\nThis is (roughly) what a site like GitHub is doing on the backend with\nrepository forks. But Git doesn't ship any scripts to help with it, and\nI don't offhand know of any public ones. I assume GitLab does something\nsimilar, and their system may be open source.\n\nSome gotchas:\n\n  - this is obviously racy with simultaneous updates to the local repos\n\n  - you'd probably want to fetch HEAD as well, to cover detached HEADs\n\n  - it won't cover blobs/trees referenced by the index of each child\n    repo (but those are probably going to be local to those repos\n    anyway).\n\n  - it won't cover reflogs in the local repos either (but it's not the\n    end of the world if a reflog entry goes stale)\n\nAnother, perhaps simpler approach, is to just never expire objects from\nthe mirror repo (with the obvious downside being that you might carry\nobjects forever that nobody cares about). You can set gc.pruneExpire to\nsomething high, and then look into gc.cruftPacks to store the old\nobjects in a more efficient form.\n\nHope that helps.\n\n-Peff\n"},{"id":"509573","messageId":"SJ0PR11MB58148000A9DCB33CCFD792A0C60D2@SJ0PR11MB5814.namprd11.prod.outlook.com","threadId":"62693","inReplyTo":"20241226155251.GA69868@coredump.intra.peff.net","subject":"RE: \"git fetch\" fails for a --reference clone after an outer forced push","fromName":"Mirochnik, Oleg V","fromEmail":"oleg.v.mirochnik@intel.com","sentAt":"2024-12-26T17:36:34Z","receivedAt":"2024-12-26T17:36:47Z","isPatch":false,"sender":{"key":"oleg.v.mirochnik@intel.com","avatar":null},"body":"Hi Jeff,\n\nThank you for the detailed explanation. It explains perfectly why skipping the git-gc for the mirror in the reproducer fixes the problem.\nBut one thing is still confusing and inconsistent IMO - why using local path for clone still works (even with the git-gc in both master and mirror which removes the obsoleted commit object) while it should not (based on your explanation and the \"git help clone\" snippet you provided)\nCould you please clarify?\n\nThanks,\nOleg.\n\nSteps:\n$ cat doit\n#!/bin/sh\nset -xe\nrm -rf tst\nmkdir tst\ncd tst\nmkdir master\ngit -C master init --bare\ngit clone master local\ntouch local/foo\ngit -C local add .\ngit -C local commit -m init-commit\ngit -C local push\necho foo > local/foo\ngit -C local commit -a -m dummy-commit\nrev=`git -C local log -1 --format=%h`\ngit -C local push origin HEAD:refs/heads/dummy\ngit clone --mirror file://`pwd`/master mirror\ngit clone --reference `pwd`/mirror `pwd`/master local1\ngit -C local1 log --oneline origin/dummy\ngit -C local commit --amend -m new-dummy-commit\ngit -C local push -f origin HEAD:dummy\ngit -C local log --oneline origin/dummy\ngit -C mirror fetch\ngit -C mirror gc --prune=now\ngit -C mirror log -1 --oneline $rev -- || echo FAIL\ngit -C master gc --prune=now\ngit -C master log -1 --oneline $rev -- || echo FAIL\ngit -C local1 fetch\n#git -C local1 remote update --prune\ngit -C local1 log --oneline origin/dummy\n\nTail of the script execution output:\n\n+ git -C mirror fetch\nremote: Enumerating objects: 5, done.\nremote: Counting objects: 100% (5/5), done.\nremote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0\nUnpacking objects: 100% (3/3), 230 bytes | 230.00 KiB/s, done.\nFrom file:///tmp/tst/master\n + 6e72f6a...553da2c dummy      -> dummy  (forced update)\n+ git -C mirror gc --prune=now\nEnumerating objects: 6, done.\nCounting objects: 100% (6/6), done.\nDelta compression using up to 96 threads\nCompressing objects: 100% (2/2), done.\nWriting objects: 100% (6/6), done.\nBuilding bitmaps: 100% (2/2), done.\nTotal 6 (delta 1), reused 4 (delta 0), pack-reused 0\n+ git -C mirror log -1 --oneline 6e72f6a --\nfatal: bad revision '6e72f6a'\n+ echo FAIL\nFAIL\n+ git -C master gc --prune=now\nEnumerating objects: 6, done.\nCounting objects: 100% (6/6), done.\nDelta compression using up to 96 threads\nCompressing objects: 100% (2/2), done.\nWriting objects: 100% (6/6), done.\nBuilding bitmaps: 100% (2/2), done.\nTotal 6 (delta 1), reused 0 (delta 0), pack-reused 0\n+ git -C master log -1 --oneline 6e72f6a --\nfatal: bad revision '6e72f6a'\n+ echo FAIL\nFAIL\n+ git -C local1 fetch\nFrom /tmp/tst/master\n + 6e72f6a...553da2c dummy      -> origin/dummy  (forced update)\n+ git -C local1 log --oneline origin/dummy\n553da2c (origin/dummy) new-dummy-commit\nf5eab7f (HEAD -> master, origin/master, origin/HEAD) init-commit\n\n\n\n-----Original Message-----\nFrom: Jeff King <peff@peff.net> \nSent: Thursday, December 26, 2024 7:53 AM\nTo: Mirochnik, Oleg V <oleg.v.mirochnik@intel.com>\nCc: git@vger.kernel.org\nSubject: Re: \"git fetch\" fails for a --reference clone after an outer forced push\n\nOn Wed, Dec 25, 2024 at 05:23:13PM +0000, Mirochnik, Oleg V wrote:\n\n> What did you do before the bug happened? (Steps to reproduce your \n> issue)\n> \n> $ cat ./doit\n> #!/bin/sh\n> set -xe\n> rm -rf tst\n> mkdir tst\n> cd tst\n> mkdir master\n> git -C master init --bare\n> git clone master local\n> touch local/foo\n> git -C local add .\n> git -C local commit -m init-commit\n> git -C local push\n> echo foo > local/foo\n> git -C local commit -a -m dummy-commit git -C local push origin \n> HEAD:refs/heads/dummy git clone --mirror file://`pwd`/master mirror \n> git clone --reference `pwd`/mirror file://`pwd`/master local1 git -C \n> local1 log --oneline origin/dummy git -C local commit --amend -m \n> new-dummy-commit git -C local push -f origin HEAD:dummy git -C mirror \n> fetch git -C mirror gc --prune=now git -C local1 fetch git -C local1 \n> log --oneline origin/dummy\n>\n> [...]\n>\n> What happened instead? (Actual behavior)\n> \n> + git -C local1 fetch\n> fatal: bad object refs/remotes/origin/dummy\n> error: file:///tmp/tst/master did not send all necessary objects\n\nThis is the expected behavior, and what the warning in \"git help clone\"\nis talking about:\n\n  NOTE: this is a possibly dangerous operation; do not use it unless you\n  understand what it does. If you clone your repository using this\n  option and then delete branches (or use any other Git command that\n  makes any existing commit unreferenced) in the source repository, some\n  objects may become unreferenced (or dangling). These objects may be\n  removed by normal Git operations (such as git commit) which\n  automatically call git maintenance run --auto. (See\n  git-maintenance(1).) If these objects are removed and were referenced\n  by the cloned repository, then the cloned repository will become\n  corrupt.\n\nYour \"mirror\" repository has no idea that other repositories are depending on it. To safely do a \"git gc\" there, it would need to know all of the objects that are referenced by the dependent repositories, to count them as reachable.\n\nOne way to do that is something like:\n\n  1. Enable the \"preciousObjects\" flag in the mirror repo, to prevent\n     accidental destruction (e.g., from auto-gc):\n\n       git -C mirror config core.repositoryFormatVersion 1\n       git -C mirror config extensions.preciousObjects true\n\n  2. When you do want to run gc on the mirror repo, collect all of the\n     references from child repos first:\n\n       # collect references from all child repos; the destination\n       # doesn't really matter here, and you could even delete\n       # refs/child/* after the gc if you want\n       for $repo in local*; do\n         git -C mirror fetch --prune ../$repo refs/*:refs/child/$repo/*\n       done\n\n       # now gc, disabling preciousObjects temporarily\n       git -c extensions.preciousObjects=false gc --prune=now\n\nThis is (roughly) what a site like GitHub is doing on the backend with repository forks. But Git doesn't ship any scripts to help with it, and I don't offhand know of any public ones. I assume GitLab does something similar, and their system may be open source.\n\nSome gotchas:\n\n  - this is obviously racy with simultaneous updates to the local repos\n\n  - you'd probably want to fetch HEAD as well, to cover detached HEADs\n\n  - it won't cover blobs/trees referenced by the index of each child\n    repo (but those are probably going to be local to those repos\n    anyway).\n\n  - it won't cover reflogs in the local repos either (but it's not the\n    end of the world if a reflog entry goes stale)\n\nAnother, perhaps simpler approach, is to just never expire objects from the mirror repo (with the obvious downside being that you might carry objects forever that nobody cares about). You can set gc.pruneExpire to something high, and then look into gc.cruftPacks to store the old objects in a more efficient form.\n\nHope that helps.\n\n-Peff\n"},{"id":"509580","messageId":"20241227040551.GA86251@coredump.intra.peff.net","threadId":"62693","inReplyTo":"SJ0PR11MB58148000A9DCB33CCFD792A0C60D2@SJ0PR11MB5814.namprd11.prod.outlook.com","subject":"Re: \"git fetch\" fails for a --reference clone after an outer forced push","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-12-27T04:05:51Z","receivedAt":"2024-12-27T04:05:55Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Dec 26, 2024 at 05:36:34PM +0000, Mirochnik, Oleg V wrote:\n\n> But one thing is still confusing and inconsistent IMO - why using\n> local path for clone still works (even with the git-gc in both master\n> and mirror which removes the obsoleted commit object) while it should\n> not (based on your explanation and the \"git help clone\" snippet you\n> provided)\n> Could you please clarify?\n\nWhen clone is given a regular path (and not a file:// url), the\n\"--local\" mode optimizations are used by default. And in that mode, all\nof the objects from the source repo are actually hard-linked into the\ncloned repo. We still set up a pointer to the --reference repository,\nbut none of its objects will be used immediately (since we should have\neverything we need by hard-linking the source repo).\n\nSo here:\n\n> git clone --reference `pwd`/mirror `pwd`/master local1\n\n...there will be hard-links from local1 to the packs in master. So even\nif \"mirror\" gets rid of those objects, there will still be copies in\nlocal1.\n\nI think if you were to repack local1, it would drop those duplicate\nobjects (because it passes \"-l\" to \"git-repack\" under the hood). And\nthen you'd still be subject to the same problems if \"mirror\" also gets\nrid of its copies.\n\n-Peff\n"},{"id":"509646","messageId":"SJ0PR11MB581453EE2F26A8AAA4D68158C60E2@SJ0PR11MB5814.namprd11.prod.outlook.com","threadId":"62693","inReplyTo":"20241227040551.GA86251@coredump.intra.peff.net","subject":"RE: \"git fetch\" fails for a --reference clone after an outer forced push","fromName":"Mirochnik, Oleg V","fromEmail":"oleg.v.mirochnik@intel.com","sentAt":"2024-12-27T19:09:29Z","receivedAt":"2024-12-27T19:09:37Z","isPatch":false,"sender":{"key":"oleg.v.mirochnik@intel.com","avatar":null},"body":"Thanks!\n\n-----Original Message-----\nFrom: Jeff King <peff@peff.net> \nSent: Thursday, December 26, 2024 8:06 PM\nTo: Mirochnik, Oleg V <oleg.v.mirochnik@intel.com>\nCc: git@vger.kernel.org\nSubject: Re: \"git fetch\" fails for a --reference clone after an outer forced push\n\nOn Thu, Dec 26, 2024 at 05:36:34PM +0000, Mirochnik, Oleg V wrote:\n\n> But one thing is still confusing and inconsistent IMO - why using \n> local path for clone still works (even with the git-gc in both master \n> and mirror which removes the obsoleted commit object) while it should \n> not (based on your explanation and the \"git help clone\" snippet you\n> provided)\n> Could you please clarify?\n\nWhen clone is given a regular path (and not a file:// url), the \"--local\" mode optimizations are used by default. And in that mode, all of the objects from the source repo are actually hard-linked into the cloned repo. We still set up a pointer to the --reference repository, but none of its objects will be used immediately (since we should have everything we need by hard-linking the source repo).\n\nSo here:\n\n> git clone --reference `pwd`/mirror `pwd`/master local1\n\n...there will be hard-links from local1 to the packs in master. So even if \"mirror\" gets rid of those objects, there will still be copies in local1.\n\nI think if you were to repack local1, it would drop those duplicate objects (because it passes \"-l\" to \"git-repack\" under the hood). And then you'd still be subject to the same problems if \"mirror\" also gets rid of its copies.\n\n-Peff\n"}]}