{"thread":{"id":"63889","subject":"Fetching upstream remote fails if repo was a blobless clone","startedAt":"2025-08-01T09:31:34Z","lastAt":"2025-08-02T19:55:20Z","messageCount":6,"participants":["Justin Su","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"523185","messageId":"CAB=S_8+LMsSpnRWQZwK2Dj63WdcPy1vp+aJ=erDbf_aaPoU3cA@mail.gmail.com","threadId":"63889","inReplyTo":null,"subject":"Fetching upstream remote fails if repo was a blobless clone","fromName":"Justin Su","fromEmail":"injustsu@gmail.com","sentAt":"2025-08-01T09:30:56Z","receivedAt":"2025-08-01T09:31:34Z","isPatch":false,"sender":{"key":"injustsu@gmail.com","avatar":"https://gravatar.com/avatar/038335c7765082d30381cabb16967c569c317af7c4d49addc996dcf9720ada06?d=mp&s=160"},"body":"Thank you for filling out a Git bug report!\nPlease answer the following questions to help us understand your issue.\n\nWhat did you do before the bug happened? (Steps to reproduce your issue)\n\n```\ngit clone --filter=blob:none git@github.com:injust/delta\ncd delta/\ngit remote add upstream git@github.com:dandavison/delta\ngit fetch upstream\n```\n\nWhat did you expect to happen? (Expected behavior) Fetch the upstream repo\n\nWhat happened instead? (Actual behavior)\n\n```\nremote: Enumerating objects: 1578, done.\nremote: Counting objects: 100% (776/776), done.\nremote: Compressing objects: 100% (15/15), done.\nremote: Total 1578 (delta 772), reused 761 (delta 761), pack-reused 802 (from 2)\nReceiving objects: 100% (1578/1578), 2.67 MiB | 5.34 MiB/s, done.\nResolving deltas: 100% (1156/1156), completed with 356 local objects.\nfatal: did not receive expected object 0020d54b979cc8cf59a13406f98bfe515b190559\nfatal: fetch-pack: invalid index-pack output\n```\n\nAnything else you want to add: This only happens if a blobless clone\nis performed\n\n\n[System Info]\ngit version:\ngit version 2.50.1\ncpu: x86_64\nno commit associated with this build\nsizeof-long: 8\nsizeof-size_t: 8\nshell-path: /bin/sh\nfeature: fsmonitor--daemon\nlibcurl: 8.6.0\nzlib: 1.2.12\nSHA-1: SHA1_DC\nSHA-256: SHA256_BLK\nuname: Darwin 24.5.0 Darwin Kernel Version 24.5.0: Tue Apr 22 19:53:26\nPDT 2025; root:xnu-11417.121.6~2/RELEASE_X86_64 x86_64\ncompiler info: clang: 16.0.0 (clang-1600.0.26.6)\nlibc info: no libc information available\n$SHELL (typically, interactive shell): /usr/local/bin/fish\n"},{"id":"523265","messageId":"20250802093226.GF3711639@coredump.intra.peff.net","threadId":"63889","inReplyTo":"CAB=S_8+LMsSpnRWQZwK2Dj63WdcPy1vp+aJ=erDbf_aaPoU3cA@mail.gmail.com","subject":"Re: Fetching upstream remote fails if repo was a blobless clone","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-08-02T09:32:26Z","receivedAt":"2025-08-02T09:32:27Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Aug 01, 2025 at 05:30:56AM -0400, Justin Su wrote:\n\n> git clone --filter=blob:none git@github.com:injust/delta\n> cd delta/\n> git remote add upstream git@github.com:dandavison/delta\n> git fetch upstream\n> [...]\n\n> ```\n> remote: Enumerating objects: 1578, done.\n> remote: Counting objects: 100% (776/776), done.\n> remote: Compressing objects: 100% (15/15), done.\n> remote: Total 1578 (delta 772), reused 761 (delta 761), pack-reused 802 (from 2)\n> Receiving objects: 100% (1578/1578), 2.67 MiB | 5.34 MiB/s, done.\n> Resolving deltas: 100% (1156/1156), completed with 356 local objects.\n> fatal: did not receive expected object 0020d54b979cc8cf59a13406f98bfe515b190559\n> fatal: fetch-pack: invalid index-pack output\n> ```\n\nHmm. I can't reproduce here, but I wonder if it is dependent on the\nserver-side repo state. E.g., if it depends on the fetch from the\nupstream remote receiving a delta against a blob from origin. And that\nmight change if the server repacks, or even if there is another push.\n\nIs it still happening for you?\n\n-Peff\n"},{"id":"523285","messageId":"CAB=S_8JYbFHJ6WQSyzGO5ns8Fe-NPCdyKjWLmRrsZ1UiZJsReg@mail.gmail.com","threadId":"63889","inReplyTo":"20250802093226.GF3711639@coredump.intra.peff.net","subject":"Re: Fetching upstream remote fails if repo was a blobless clone","fromName":"Justin Su","fromEmail":"injustsu@gmail.com","sentAt":"2025-08-02T18:02:57Z","receivedAt":"2025-08-02T18:03:34Z","isPatch":false,"sender":{"key":"injustsu@gmail.com","avatar":"https://gravatar.com/avatar/038335c7765082d30381cabb16967c569c317af7c4d49addc996dcf9720ada06?d=mp&s=160"},"body":"On Sat, Aug 2, 2025 at 5:32 AM Jeff King <peff@peff.net> wrote:\n\n> Is it still happening for you?\n\nYes, I can still reproduce with that sequence of commands.\n"},{"id":"523286","messageId":"CAB=S_8+aDwMNQkawY-Mod35EDm20mi_=xmmwfngU6As799ppqw@mail.gmail.com","threadId":"63889","inReplyTo":"CAB=S_8JYbFHJ6WQSyzGO5ns8Fe-NPCdyKjWLmRrsZ1UiZJsReg@mail.gmail.com","subject":"Re: Fetching upstream remote fails if repo was a blobless clone","fromName":"Justin Su","fromEmail":"injustsu@gmail.com","sentAt":"2025-08-02T18:28:24Z","receivedAt":"2025-08-02T18:29:00Z","isPatch":false,"sender":{"key":"injustsu@gmail.com","avatar":"https://gravatar.com/avatar/038335c7765082d30381cabb16967c569c317af7c4d49addc996dcf9720ada06?d=mp&s=160"},"body":"Turns out this was because I had `transfer.fsckObjects = true` in my\nglobal config.\n\nI think you should be able to repro if you change the last command to\n`git -c fetch.fsckObjects=true fetch upstream`.\n\n\nOn Sat, Aug 2, 2025 at 2:02 PM Justin Su <injustsu@gmail.com> wrote:\n>\n> On Sat, Aug 2, 2025 at 5:32 AM Jeff King <peff@peff.net> wrote:\n>\n> > Is it still happening for you?\n>\n> Yes, I can still reproduce with that sequence of commands.\n"},{"id":"523292","messageId":"20250802193110.GA1774743@coredump.intra.peff.net","threadId":"63889","inReplyTo":"CAB=S_8+aDwMNQkawY-Mod35EDm20mi_=xmmwfngU6As799ppqw@mail.gmail.com","subject":"Re: Fetching upstream remote fails if repo was a blobless clone","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-08-02T19:31:10Z","receivedAt":"2025-08-02T19:31:12Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Aug 02, 2025 at 02:28:24PM -0400, Justin Su wrote:\n\n> Turns out this was because I had `transfer.fsckObjects = true` in my\n> global config.\n> \n> I think you should be able to repro if you change the last command to\n> `git -c fetch.fsckObjects=true fetch upstream`.\n\nThanks, I can reproduce easily now. The object in question isn't\nmentioned directly in the pack at all, as an incoming object or as a\ndelta. It's mentioned by a tree, c5b8c11446. And then when we fsck, we\nhit it via fsck_walk_tree(). And then when we've finished indexing the\npack, we check for any objects that were mentioned but which we don't\nhave. And we don't have 0020d54b979, so we barf.\n\nI assume what's happening is that 0020d54b979 is contained in the origin\nrepo, but we don't fetch (because of the blob:none filter). And then\nwhen we talk to the upstream repo, it assumes we _do_ have it because of\nthe commits that we claimed to have. And that looks like the case. In\nthe partial clone we can do:\n\n  $ git rev-list --objects --all --missing=print-info | grep 0020d54b\n  ?0020d54b979cc8cf59a13406f98bfe515b190559 path=src/features/navigate.rs type=blob\n\nThere it is, mentioned by the origin repo.\n\nSo it is perfectly normal for us to be missing this object, and\nindex-pack is wrong to complain. Curiously, there's this code in\nfetch-pack.c:\n\n                  if (args->from_promisor)\n                          /*\n                           * create_promisor_file() may be called afterwards but\n                           * we still need index-pack to know that this is a\n                           * promisor pack. For example, if transfer.fsckobjects\n                           * is true, index-pack needs to know that .gitmodules\n                           * is a promisor object (so that it won't complain if\n                           * it is missing).\n                           */\n                          strvec_push(&cmd.args, \"--promisor\");\n\nwhich you'd think would kick in here. And I confirmed that the\nindex-pack which barfs is passed that option.\n\nSo I dunno. Clearly there is a bug, but it's not clear to me how this\ncode is actually supposed to work.\n\nDoing this:\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 0a5c8a1ac8..e01cf7238b 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -262,9 +262,14 @@ static unsigned check_object(struct object *obj)\n \t\tunsigned long size;\n \t\tint type = odb_read_object_info(the_repository->objects,\n \t\t\t\t\t\t&obj->oid, &size);\n-\t\tif (type <= 0)\n+\t\tif (type <= 0) {\n+\t\t\tif (is_promisor_object(the_repository, &obj->oid)) {\n+\t\t\t\tobj->flags |= FLAG_CHECKED;\n+\t\t\t\treturn 1;\n+\t\t\t}\n \t\t\tdie(_(\"did not receive expected object %s\"),\n \t\t\t      oid_to_hex(&obj->oid));\n+\t\t}\n \t\tif (type != obj->type)\n \t\t\tdie(_(\"object %s: expected type %s, found %s\"),\n \t\t\t    oid_to_hex(&obj->oid),\n\nmakes the problem go away. But I feel like I'm probably missing\nsomething (and that function is rather expensive to run, though maybe\nnot so bad if the alternative is crashing).\n\n+cc Jonathan Tan as the author of the code comment above for any wisdom.\n\n-Peff\n"},{"id":"523293","messageId":"20250802195518.GA1800293@coredump.intra.peff.net","threadId":"63889","inReplyTo":"20250802193110.GA1774743@coredump.intra.peff.net","subject":"Re: Fetching upstream remote fails if repo was a blobless clone","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-08-02T19:55:18Z","receivedAt":"2025-08-02T19:55:20Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Aug 02, 2025 at 03:31:11PM -0400, Jeff King wrote:\n\n> Doing this:\n> \n> diff --git a/builtin/index-pack.c b/builtin/index-pack.c\n> index 0a5c8a1ac8..e01cf7238b 100644\n> --- a/builtin/index-pack.c\n> +++ b/builtin/index-pack.c\n> @@ -262,9 +262,14 @@ static unsigned check_object(struct object *obj)\n>  \t\tunsigned long size;\n>  \t\tint type = odb_read_object_info(the_repository->objects,\n>  \t\t\t\t\t\t&obj->oid, &size);\n> -\t\tif (type <= 0)\n> +\t\tif (type <= 0) {\n> +\t\t\tif (is_promisor_object(the_repository, &obj->oid)) {\n> +\t\t\t\tobj->flags |= FLAG_CHECKED;\n> +\t\t\t\treturn 1;\n> +\t\t\t}\n>  \t\t\tdie(_(\"did not receive expected object %s\"),\n>  \t\t\t      oid_to_hex(&obj->oid));\n> +\t\t}\n>  \t\tif (type != obj->type)\n>  \t\t\tdie(_(\"object %s: expected type %s, found %s\"),\n>  \t\t\t    oid_to_hex(&obj->oid),\n> \n> makes the problem go away. But I feel like I'm probably missing\n> something (and that function is rather expensive to run, though maybe\n> not so bad if the alternative is crashing).\n> \n> +cc Jonathan Tan as the author of the code comment above for any wisdom.\n\nAnd here is a minimal reproduction that doesn't depend on any other\nrepositories:\n\n-- >8 --\n# Server has two commits, with two blobs for file, old and new.\ngit init server\n(\n\tcd server\n\techo old >file\n\tgit add .\n\tgit commit -m old\n\techo new >file\n\tgit commit -am new\n\n\tgit config uploadpack.allowfilter true\n)\n\n# The fork has built a new tree which mentions the old file.\ngit clone server fork\n(\n\tcd fork\n\tgit reset --hard HEAD^\n\techo content >unrelated\n\tgit add .\n\tgit commit -m unrelated\n)\n\n# After our partial clone, we have the new blob (because we faulted it in to\n# checkout), but not the old one (because it is buried in history).\ngit clone --no-local --filter=blob:none server repo\ncd repo\n\n# This will get the tree at the tip of the fork repo, which mentions old. When\n# we fsck that tree, we'll see that it mentions the old blob, and expect to\n# find it. But we won't due to the partial clone (though we could get it if we\n# wanted from the server repo).\ngit -c transfer.fsckObjects=true fetch ../fork\n-- >8 --\n\nThat fails with stock git now, like this:\n\n  fatal: did not receive expected object 3367afdbbf91e638efe983616377c60477cc6612\n  fatal: index-pack failed\n\nbut succeeds with the patch above.\n\n-Peff\n"}]}