{"thread":{"id":"66675","subject":"[PATCH] pack-objects: handle missing objects when excluding promisor objects","startedAt":"2026-10-27T07:00:28Z","lastAt":"2026-10-28T11:50:51Z","messageCount":5,"participants":["Patrick Steinhardt","Mark C. Chu-Carroll","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"556436","messageId":"20261027-b4-pks-pack-objects-repack-links-with-missing-objects-v1-1-f50f90651274@pks.im","threadId":"66675","inReplyTo":null,"subject":"[PATCH] pack-objects: handle missing objects when excluding promisor objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-27T07:00:28Z","receivedAt":"2026-10-27T07:00:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In a partial clone Git tries to maintain two different sets of objects:\npromisor objects should be those that we have received from the remote,\nand any such object may not be fully connected. Locally created objects\non the other hand should be stored as non-promisor objects. The goal is\nthat promisor objects should never reference non-promisor objects as the\n\"promisor\" property is contagious, where a promisor implicitly means\nthat any of its referenced objects are promisor objects, too.\n\nSo when git-index-pack(1) indexes a packfile from a promisor remote, and\nthat packfile has objects that refer to non-promisor objects, then it\ntries to reinstate that partition by executing `git pack-objects\n--exclude-promisor-objects-best-effort`. That command packs local,\nnon-promisor objects that become reachable from a promisor object into a\nnew promisor pack. This best-effort mode tries to exclude promisor\nobjects by using an `is_not_in_promisor_pack_obj()` callback for the\nrev-walk: for every object that we see, we check whether it's part of a\npromisor pack. And if it is, we exclude that object.\n\nBut in a partial clone we may fail to find some of the objects during\nthe object walk. The callback only stops us from walking into the\nparents of a commit that lives in a promisor pack, but the commit itself\nis still yielded, and `traverse_commit_list()` then queues its root\ntree. And in a repository that was cloned with a \"tree:0\" filter that\ntree does not exist, so `is_not_in_promisor_pack_obj()` fails to read it\nand Git would `BUG()`:\n\n    BUG: builtin/pack-objects.c:5088: should_include_obj should only be called on existing objects\n    error: pack-objects died of signal 6\n    fatal: could not finish pack-objects to repack local links\n    fatal: index-pack failed\n\nThis bug can trigger in situations where we have e.g. a combination of\nshallow and partial clones, and that bug has existed ever since we\nintroduced the logic in c08589efdc5 (index-pack: repack local links into\npromisor packs, 2024-11-01).\n\nBut more recently, in d43b9acb05b (setup: unify setup of shallow file,\n2026-07-07) we have introduced another case where it can occur. With\nthis change, Git now propagates the \"--shallow-file\" option to child\nprocesses via GIT_SHALLOW_FILE. But when unshallowing or deepening a\nrepository we'd pass \"--shallow-file=\" to `git index-pack --promisor`,\nand thus we'd now also propagate that environment variable to `git\npack-objects --exclude-promisor-objects-best-effort`. And as that asks\nit to ignore the shallow boundary we may eventually try to read beyond\nthe shallow boundary and hit the missing tree of a promisor commit.\n\nWhile we could stop propagating the environment variable to the child\nprocess that would not help us to fix the first case. And besides, all\nother global Git options also propagate to child processes in the same\nway, so it feels wrong to behave differently here.\n\nInstead, we can simply observe the fact that a missing object cannot be\na local non-promisor object by definition, and hence there is nothing\nfor us to pack in that case. So instead of calling `BUG()`, we should\nrather just skip such objects. And coincidentally, that's also what\n`repack_local_links()` (which invokes git-pack-objects(1) with\n\"--exclude-promisor-objects-best-effort\") already does.\n\nReported-by: Daniel Zoghalchali <daniel@latchkey.dev>\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\nHi,\n\nthis patch fixes the issue reported in [1]. Thanks!\n\nPatrick\n\n[1]: <CAJwQ=KQynymg4CJOLkyebACYVQa+LhXB2UifniTASkLyArWejA@mail.gmail.com>\n---\n builtin/pack-objects.c   |  3 ++-\n t/t5616-partial-clone.sh | 55 ++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 57 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex ca3a891dfbc..d6caed16d97 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -5085,7 +5085,8 @@ static int is_not_in_promisor_pack_obj(struct object *obj, void *data UNUSED)\n \t};\n \n \tif (odb_read_object_info_extended(the_repository->objects, &obj->oid, &info, 0))\n-\t\tBUG(\"should_include_obj should only be called on existing objects\");\n+\t\t/* Missing; assume it is a promisor object */\n+\t\treturn 0;\n \treturn source_info.source->type != ODB_SOURCE_PACKED || !source_info.u.packed.pack->pack_promisor;\n }\n \ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex 7cc6f58a31f..97a72e161ce 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -747,6 +747,61 @@ test_expect_success 'after fetching descendants of non-promisor commits, gc work\n \tgit -C partial gc --prune=now\n '\n \n+for opt in --unshallow --deepen=1\n+do\n+\ttest_expect_success \"filtered fetch $opt of a moved ref repacks local links\" '\n+\t\ttest_when_finished \"rm -rf source target\" &&\n+\n+\t\tgit init source &&\n+\t\ttest_commit_bulk -C source 4 &&\n+\t\tgit -C source config set uploadpack.allowFilter true &&\n+\t\tgit -C source config set uploadpack.allowAnySHA1InWant true &&\n+\t\tCOMMIT=$(git -C source rev-parse main~1) &&\n+\n+\t\tgit init target &&\n+\t\tgit -C target remote add origin \"file://$(pwd)/source\" &&\n+\t\tgit -C target fetch --no-tags --depth=1 origin \"+$COMMIT:refs/remotes/origin/main\" &&\n+\n+\t\t# The fetched pack contains the new tip, whose parent is the\n+\t\t# local non-promisor shallow tip, so index-pack repacks the\n+\t\t# latter into a promisor pack. The pack-objects process doing\n+\t\t# so inherits the shallow file passed to index-pack and thus\n+\t\t# walks into the promisor commits below the former boundary,\n+\t\t# whose trees do not exist.\n+\t\tgit -C target fetch --no-tags --filter=tree:0 \"$opt\" origin +refs/heads/main:refs/remotes/origin/main &&\n+\t\tgit -C target rev-parse --verify refs/remotes/origin/main >actual &&\n+\t\tgit -C source rev-parse --verify refs/heads/main >expect &&\n+\t\ttest_cmp expect actual &&\n+\t\tgit -C target fsck\n+\t'\n+done\n+\n+test_expect_success 'repacking local links tolerates missing trees of promisor commits' '\n+\ttest_when_finished \"rm -rf source target\" &&\n+\n+\tgit init source &&\n+\ttest_commit_bulk -C source 4 &&\n+\tgit -C source config set uploadpack.allowFilter true &&\n+\tgit -C source config set uploadpack.allowAnySHA1InWant true &&\n+\tCOMMIT=$(git -C source rev-parse main~1) &&\n+\n+\t# Fetch a single commit as a non-promisor object, then unshallow it\n+\t# with a filter that omits trees. The commits below the former\n+\t# shallow boundary are now promisor objects whose trees are missing.\n+\tgit init target &&\n+\tgit -C target remote add origin \"file://$(pwd)/source\" &&\n+\tgit -C target fetch --no-tags --depth=1 origin \"+$COMMIT:refs/remotes/origin/main\" &&\n+\tgit -C target fetch --no-tags --filter=tree:0 --unshallow origin \"+$COMMIT:refs/remotes/origin/main\" &&\n+\ttest_path_is_missing target/.git/shallow &&\n+\n+\t# Fetching a descendant of the local non-promisor commit causes\n+\t# index-pack to repack that commit into a promisor pack. The revision\n+\t# walk must not die when it reaches the missing trees of the promisor\n+\t# commits below it.\n+\tgit -C target fetch --no-tags --filter=tree:0 origin +refs/heads/main:refs/remotes/origin/main &&\n+\tgit -C target rev-parse --verify refs/remotes/origin/main &&\n+\tgit -C target fsck\n+'\n \n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n\n---\nbase-commit: 7e34d8174dcda3435d1404c9a30bef46afe6c13e\nchange-id: 20261026-b4-pks-pack-objects-repack-links-with-missing-objects-ba49defedef1\n\n\n"},{"id":"556526","messageId":"03704f3590183aadc16d5d81bdc64e9a@fastmail.com","threadId":"66675","inReplyTo":"20261027-b4-pks-pack-objects-repack-links-with-missing-objects-v1-1-f50f90651274@pks.im","subject":"Re: [PATCH] pack-objects: handle missing objects kkkjwhen excluding promisor objects","fromName":"Mark C. Chu-Carroll","fromEmail":"markchucarroll@fastmail.com","sentAt":"2026-10-27T16:02:11Z","receivedAt":"2026-10-27T16:02:11Z","isPatch":true,"sender":{"key":"markchucarroll@fastmail.com","avatar":null},"body":"\nOn Tue, 27 Oct 2026 08:00:28 +0100, Patrick Steinhardt <ps@pks.im> wrote:\n> In a partial clone Git tries to maintain two different sets of objects:\n> promisor objects should be those that we have received from the remote,\n> and any such object may not be fully connected. Locally created objects\n> on the other hand should be stored as non-promisor objects. The goal is\n> that promisor objects should never reference non-promisor objects as the\n> \"promisor\" property is contagious, where a promisor implicitly means\n> that any of its referenced objects are promisor objects, too.\n\nWordsmithing nitpick: this reads awkwardly. You start by saying there \nare two different kinds of things, and then introduce one of them,\nfollowed by a comma, followed by something which _isn't_ the second \ntype of thing.\n\nMaybe:\nIn a partial clone, Git tries to maintain two different sets of objects:\npromisor objects (which are those we have recieved from a remote, and may \nnot be fully connected), and non-promisor objects (including any locally\ncreated objects). The goal is that .... \n\n\n\n\n\n"},{"id":"556613","messageId":"auGjjv-zVYPFQLIK@pks.im","threadId":"66675","inReplyTo":"03704f3590183aadc16d5d81bdc64e9a@fastmail.com","subject":"Re: [PATCH] pack-objects: handle missing objects kkkjwhen excluding promisor objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-28T07:26:06Z","receivedAt":"2026-10-28T07:26:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Oct 27, 2026 at 12:02:11PM -0400, Mark C. Chu-Carroll wrote:\n> \n> On Tue, 27 Oct 2026 08:00:28 +0100, Patrick Steinhardt <ps@pks.im> wrote:\n> > In a partial clone Git tries to maintain two different sets of objects:\n> > promisor objects should be those that we have received from the remote,\n> > and any such object may not be fully connected. Locally created objects\n> > on the other hand should be stored as non-promisor objects. The goal is\n> > that promisor objects should never reference non-promisor objects as the\n> > \"promisor\" property is contagious, where a promisor implicitly means\n> > that any of its referenced objects are promisor objects, too.\n> \n> Wordsmithing nitpick: this reads awkwardly. You start by saying there \n> are two different kinds of things, and then introduce one of them,\n> followed by a comma, followed by something which _isn't_ the second \n> type of thing.\n> \n> Maybe:\n> In a partial clone, Git tries to maintain two different sets of objects:\n> promisor objects (which are those we have recieved from a remote, and may \n> not be fully connected), and non-promisor objects (including any locally\n> created objects). The goal is that .... \n\nThat reads better indeed. Queued locally for now, I'll wait a bit longer\nbefore sending this out. Thanks!\n\nPatrick\n\n"},{"id":"556615","messageId":"xmqq7bj2e0o1.fsf@gitster.g","threadId":"66675","inReplyTo":"20261027-b4-pks-pack-objects-repack-links-with-missing-objects-v1-1-f50f90651274@pks.im","subject":"Re: [PATCH] pack-objects: handle missing objects when excluding promisor objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-10-28T07:37:02Z","receivedAt":"2026-10-28T07:37:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> this patch fixes the issue reported in [1]. Thanks!\n>  builtin/pack-objects.c   |  3 ++-\n>  t/t5616-partial-clone.sh | 55 ++++++++++++++++++++++++++++++++++++++++++++++++\n>  2 files changed, 57 insertions(+), 1 deletion(-)\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index ca3a891dfbc..d6caed16d97 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -5085,7 +5085,8 @@ static int is_not_in_promisor_pack_obj(struct object *obj, void *data UNUSED)\n>  \t};\n>  \n>  \tif (odb_read_object_info_extended(the_repository->objects, &obj->oid, &info, 0))\n> -\t\tBUG(\"should_include_obj should only be called on existing objects\");\n> +\t\t/* Missing; assume it is a promisor object */\n> +\t\treturn 0;\n\nIt is not entirely the fault of this patch, but the code relies on\n.fetch_if_missing being set to 0 elsewhere.  While that is not wrong\nper se, shouldn't this call to odb_read_object_info_extended() pass\nthe OBJECT_INFO_SKIP_FETCH_OBJECT bit explicitly for clarity?\n\n>  \treturn source_info.source->type != ODB_SOURCE_PACKED || !source_info.u.packed.pack->pack_promisor;\n>  }\n\nThanks.\n\n"},{"id":"556640","messageId":"auHhm1NAL_dRnp1c@pks.im","threadId":"66675","inReplyTo":"xmqq7bj2e0o1.fsf@gitster.g","subject":"Re: [PATCH] pack-objects: handle missing objects when excluding promisor objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-28T11:50:51Z","receivedAt":"2026-10-28T11:50:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Oct 28, 2026 at 12:37:02AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n> \n> > this patch fixes the issue reported in [1]. Thanks!\n> >  builtin/pack-objects.c   |  3 ++-\n> >  t/t5616-partial-clone.sh | 55 ++++++++++++++++++++++++++++++++++++++++++++++++\n> >  2 files changed, 57 insertions(+), 1 deletion(-)\n> >\n> > diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> > index ca3a891dfbc..d6caed16d97 100644\n> > --- a/builtin/pack-objects.c\n> > +++ b/builtin/pack-objects.c\n> > @@ -5085,7 +5085,8 @@ static int is_not_in_promisor_pack_obj(struct object *obj, void *data UNUSED)\n> >  \t};\n> >  \n> >  \tif (odb_read_object_info_extended(the_repository->objects, &obj->oid, &info, 0))\n> > -\t\tBUG(\"should_include_obj should only be called on existing objects\");\n> > +\t\t/* Missing; assume it is a promisor object */\n> > +\t\treturn 0;\n> \n> It is not entirely the fault of this patch, but the code relies on\n> .fetch_if_missing being set to 0 elsewhere.  While that is not wrong\n> per se, shouldn't this call to odb_read_object_info_extended() pass\n> the OBJECT_INFO_SKIP_FETCH_OBJECT bit explicitly for clarity?\n\nYeah. This is fine in practice because we always unset\n`fetch_if_missing` with \"--exclude-promisor-objects[-best-effort]\", but\nadding the flag explicitly for clarity feels sensible to me.\n\nPatrick\n\n"}]}