{"thread":{"id":"66857","subject":"[BUG] index-pack lazily fetches REF_DELTA bases that are in the pack being indexed","startedAt":"2026-11-16T17:38:38Z","lastAt":"2026-11-16T17:38:38Z","messageCount":1,"participants":["לירון עדן"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"558565","messageId":"CAG7kbaYpxb1-4E9CCG_=DKK4K1RV0b3qsYfUf6Qp57fakW0XPQ@mail.gmail.com","threadId":"66857","inReplyTo":null,"subject":"[BUG] index-pack lazily fetches REF_DELTA bases that are in the pack being indexed","fromName":"לירון עדן","fromEmail":"lironeden@gmail.com","sentAt":"2026-11-16T17:38:38Z","receivedAt":"2026-11-16T17:38:38Z","isPatch":false,"sender":{"key":"lironeden@gmail.com","avatar":null},"body":"Thank you for filling out a Git bug report!\nPlease answer the following questions to help us understand your issue.\n\nWhat did you do before the bug happened? (Steps to reproduce your issue)\n\nIn a blobless partial clone, I fetched new commits from a server that\nsends a thin pack in which the deltas with a base inside the pack are\nREF_DELTA (the client did not negotiate ofs-delta).\n\nThe script below reproduces it with stock git over file://. Setting\nrepack.useDeltaBaseOffset=false on the client stands in for a server\nthat does not support ofs-delta.\n\n----- 8< -----\n#!/bin/sh\n# index-pack lazily fetches REF_DELTA bases that are inside the pack it\n# is indexing (partial clone + thin pack + REF_DELTA for in-pack bases).\nset -e\nD=$(mktemp -d)\ncd \"$D\"\n\ncommit () {\ngit add .\ngit -c user.name=t -c user.email=t@example.com commit -q -m \"$1\"\n}\n\ngit init -q server\n(\ncd server\n# One wide tree, so that later versions of it delta well.\nfor i in $(seq 1 300)\ndo\necho \"base $i\" >file$i\ndone\ncommit base\ngit config uploadpack.allowFilter true\ngit config uploadpack.allowAnySHA1InWant true\n)\n\ngit clone -q --no-checkout --filter=blob:none \"file://$D/server\" client\n\n# Do not ask for ofs-delta, so every delta arrives as REF_DELTA. This is\n# what a client gets from a server that does not support ofs-delta.\ngit -C client config repack.useDeltaBaseOffset false\n\nold=$(git -C server rev-parse HEAD)\n(\ncd server\n# Each new commit changes one file, i.e. one entry of the tree.\nfor n in 1 2 3 4 5 6 7 8\ndo\necho \"new $n\" >>file$n\ncommit \"new $n\"\ndone\n)\nnew=$(git -C server rev-parse HEAD)\n\nGIT_TRACE2_EVENT=\"$D/trace2\" GIT_TRACE_PACKET=\"$D/packet\" \\\ngit -C client fetch -q origin\n\necho \"git version: $(git version)\"\necho \"processes spawned below 'git fetch':\"\nsed -n 's/.*\"event\":\"cmd_name\".*\"hierarchy\":\"\\([^\"]*\\)\".*/\\1/p' \"$D/trace2\" |\nsort | uniq -c\n\ngit -C server rev-list --objects --filter=blob:none \"$old..$new\" |\ncut -c1-40 | sort >\"$D/in-pack\"\nsed -n 's/.*fetch> want \\([0-9a-f]\\{40\\}\\).*/\\1/p' \"$D/packet\" |\ngrep -v \"$new\" | sort -u >\"$D/lazy-wants\"\n\necho \"objects requested by the nested lazy fetch: $(wc -l <\"$D/lazy-wants\")\"\necho \"of those, already in the pack being indexed: $(comm -12\n\"$D/lazy-wants\" \"$D/in-pack\" | wc -l)\"\nrm -rf \"$D\"\n----- >8 -----\n\nWhat did you expect to happen? (Expected behavior)\n\nA single fetch. Every delta base is either in the received pack or\nalready in the local object store, so nothing has to be fetched lazily.\n\nWhat happened instead? (Actual behavior)\n\nindex-pack starts a nested lazy fetch for delta bases that are in the\npack it is indexing. Output of the script with 2.56.0:\n\ngit version: git version 2.56.0\nprocesses spawned below 'git fetch':\n1 fetch\n1 fetch/index-pack\n1 fetch/index-pack/fetch\n1 fetch/index-pack/fetch/index-pack\n1 fetch/index-pack/fetch/maintenance\n1 fetch/index-pack/fetch/upload-pack\n1 fetch/index-pack/fetch/upload-pack/pack-objects\n1 fetch/maintenance\n1 fetch/upload-pack\n1 fetch/upload-pack/pack-objects\nobjects requested by the nested lazy fetch: 6\nof those, already in the pack being indexed: 6\n\nWith the repack.useDeltaBaseOffset line removed (ofs-delta negotiated)\nthere is no nested fetch and both counts are 0.\n\nWhat's different between what you expected and what actually happened?\n\nA redundant fetch is started from inside index-pack. In the script it\ncosts one extra round trip. Against a real server it turned into\nunbounded recursion, described below.\n\nAnything else you want to add:\n\nWhere it comes from, as far as I can tell: fix_unresolved_deltas() in\nbuiltin/index-pack.c, which is reached only when the pack is thin. In a\nrepository with a promisor remote it first prefetches delta bases:\n\nfor (i = 0; i < nr_ref_deltas; i++) {\nstruct ref_delta_entry *d = sorted_by_pos[i];\nif (!odb_read_object_info_extended(the_repository->objects,\n&d->oid, NULL,\nOBJECT_INFO_FOR_PREFETCH))\ncontinue;\noid_array_append(&to_fetch, &d->oid);\n}\npromisor_remote_get_direct(the_repository, to_fetch.oid, to_fetch.nr);\n\nThis loop goes over all REF_DELTA entries, including the ones that were\nalready resolved against a base inside the pack. The loop right after\nit skips those (real_type != OBJ_REF_DELTA), but the prefetch loop does\nnot. The objects of the incoming pack are not in the object store yet,\nso their bases look missing and are requested from the promisor remote.\n\nSkipping the already resolved entries would not cover everything: a\ndelta chained onto another in-pack delta whose own base is outside the\npack stays unresolved at this point, and its base has no known object\nID yet. Resolving with the locally available bases first, and only then\nprefetching the bases of what is still unresolved, would avoid the\nrequest in that case too. I have not written or tested a patch.\n\nHow I ran into it: clients 2.50.1.windows.1 and 2.53.0 (Linux) against\nAzure DevOps Services, which did not advertise ofs-delta in the packet\ntrace. The flow was:\n\ngit clone --no-checkout --filter=blob:none --single-branch \\\n--depth 1 <url> .\ngit checkout <commit that is not in the clone>\n\nThe checkout lazily fetched the commit. The server answered with a thin\npack of about 44,000 objects (commits and trees). Its out-of-pack bases\nwere trees of the shallow commit, which the client had. index-pack then\nlazily requested about 9,700 objects, all of them in that pack. The\nserver answered that request with another thin pack holding the same\nobjects, the nested index-pack did the same again, and so on. Every\nlevel kept its fetch, remote-https, fetch-pack and index-pack processes\nalive and downloaded about 14 MB again. I saw 12 levels in about a\nminute before I interrupted it, and several hundred levels in an\nunattended job that ended only when it hit its memory limit.\n\n4360734e90 (promisor-remote: prevent infinite recursion when lazy\nfetching, 2026-10-02) on master caps the nesting at 5, which bounds\nthis, but the redundant requests are still made. The prefetch block\nabove is the same in v2.56.1, master, next and seen as of 2026-11-15.\n\nFetching the commit explicitly before the checkout avoided it for me:\n\nGIT_NO_LAZY_FETCH=1 git fetch --depth 1 origin <commit>\ngit checkout <commit>\n\nPlease review the rest of the bug report below.\nYou can delete any lines you don't wish to share.\n\n\n[System Info]\ngit version:\ngit version 2.56.0\ncpu: arm64\nno commit associated with this build\nsizeof-long: 8\nsizeof-size_t: 8\nshell-path: /bin/sh\nrust: disabled\nfeature: fsmonitor--daemon\ngettext: enabled\nlibcurl: 8.23.0\nzlib: 1.2.12\nSHA-1: SHA1_DC\nSHA-256: SHA256_BLK\ndefault-ref-format: files\ndefault-hash: sha1\nuname: Darwin 25.6.0 Darwin Kernel Version 25.6.0: Mon Oct 12 00:41:41\nPDT 2026; root:xnu-12377.161.15.701.48~2/RELEASE_ARM64_T6050 arm64\ncompiler info: clang: 21.0.0 (clang-2100.3.34.2)\nlibc info: no libc information available\n$SHELL (typically, interactive shell): /bin/bash\n\n"}]}