# [BUG] Lazy fetch in partial clone fails with "not our ref" since 5946a564 (odb/source: make read_object_info() pluggable) 5 messages from 2026-11-03 to 2026-11-04. Participants: Patrick Ohly, Kristofer Karlsson, Patrick Steinhardt. Thread: https://gitlist.dev/t/66753 ## Patrick Ohly, 2026-11-03 13:24 Subject: [BUG] Lazy fetch in partial clone fails with "not our ref" since 5946a564 (odb/source: make read_object_info() pluggable) Message-ID: URL: https://gitlist.dev/e/yrjh1p92qc7w.fsf%40pohly-mobl1.fritz.box ``` Hello, "git merge" in a blobless partial clone fails with "not our ref" while lazily fetching a missing object. This started to occur today in the Kubernetes CI, completely breaking the merging of PRs. We should have been using git 2.54.0, the affected version, for a few days at least without issues, so it can't be ruled out that this is at least partly caused by a change on the GitHub side. We are about to revert to git 2.52.0 to unblock the Kubernetes project. What follows is an agent analysis of the problem. I'm not rewriting it because it seems succinct and useful as-is: What did you do before the bug happened? (Steps to reproduce) git init k && cd k R=https://github.com/kubernetes/kubernetes.git git fetch --filter=blob:none $R 6575724b63e50cd5dd454e774bc293c6de1e4e12 git checkout 6575724b63e50cd5dd454e774bc293c6de1e4e12 git fetch --filter=blob:none $R 787dd27af44c14c22d3bae46bf62b4a0271efc06 git -c user.name=a -c user.email=a@b merge --no-ff 787dd27af44c14c22d3bae46bf62b4a0271efc06 What did you expect to happen? The merge succeeds, with missing blobs fetched lazily from the promisor remote. What happened instead? fatal: remote error: upload-pack: not our ref 27b64cc863187e62d8b47196a78c045412d32ac9 fatal: could not fetch 3c804b88bcf177a1435ba770fa9054c25e1315c3 from promisor remote Bisect result I bisected v2.52.0 (good) to v2.54.0 (bad). The first bad commit is: 5946a564cddc0bf471f27ae4c3fe205441e3ef65 odb/source: make `read_object_info()` function pluggable The parent, 3bc3177ad7 ( 2.53.0.185.g3bc3177ad7 ), is good. Versions tested ┌───────────────────────────────────────────────────────┬────────┐ │ Version │ Result │ ├───────────────────────────────────────────────────────┼────────┤ │ 2.39.5, 2.43.4, 2.45.2, 2.47.2, 2.49.1, 2.52.0 │ OK │ ├───────────────────────────────────────────────────────┼────────┤ │ 2.54.0, 2.56.0 │ fail │ ├───────────────────────────────────────────────────────┼────────┤ │ master (v2.56.0-341-gebdc087dc, built with NO_RUST=1) │ fail │ └───────────────────────────────────────────────────────┴────────┘ Notes • The remote is github.com over HTTPS. I haven't tried other hosts. • I haven't analysed the cause. My guess is that the new  OBJECT_INFO_SECOND_READ  and reprepare path changes how the promisor lazy fetch is triggered. The client then requests an object by OID that the server refuses with "not our ref". • The failure is deterministic. The first PR head merges fine on 2.52.0 and older. [System Info] git version 2.56.GIT (also 2.54.0 and 2.56.0), Linux x86_64 (Debian stable container) -- Best Regards Patrick Ohly Cloud Software Architect ``` ## Kristofer Karlsson, 2026-11-04 10:11 Subject: Re: [BUG] Lazy fetch in partial clone fails with "not our ref" since 5946a564 (odb/source: make read_object_info() pluggable) Message-ID: URL: https://gitlist.dev/e/CAL71e4MqMdGk5ryhNxvVNN6Z02%2B45B933ZfUuvj4hM92N5a6hA%40mail.gmail.com In-Reply-To: ``` On Tue, 3 Nov 2026 at 14:31, Patrick Ohly wrote: > > Hello, > > "git merge" in a blobless partial clone fails with "not our ref" while > lazily fetching a missing object. This started to occur today in the > Kubernetes CI, completely breaking the merging of PRs. We should have > been using git 2.54.0, the affected version, for a few days at least > without issues, so it can't be ruled out that this is at least partly > caused by a change on the GitHub side. We are about to revert to git > 2.52.0 to unblock the Kubernetes project. > > [...] > > I bisected v2.52.0 (good) to v2.54.0 (bad). The first bad commit is: > > 5946a564cddc0bf471f27ae4c3fe205441e3ef65 > odb/source: make `read_object_info()` function pluggable > > The parent, 3bc3177ad7 ( 2.53.0.185.g3bc3177ad7 ), is good. Thanks for the detailed report. I think I have found the cause: I can reproduce the failure with your steps and make it go away with a small change to the loose object cache. The problem is in the semantics of the loose object cache, which answers quick existence checks. It is a two-layer cache: a bitmap keyed by the first byte of the object ID, recording which "objects/XX/" subdirectories have been listed, and an oidtree with the objects found there. The intended invariant is that once a subdirectory has been listed, the oidtree holds every loose object in it, so that "not in the oidtree" means "does not exist". Writing a loose object does not update the cache, which breaks the invariant if the subdirectory had already been listed. In practice, that means a quick lookup of any object in the same subdirectory earlier in the same process. In your example, the object the server refuses, 27b64cc8, is not in either parent nor in the merge base. It is the content-merged blob that the merge itself just wrote as a loose object. merge-ort's prefetch had already listed "27/", so the checkout of the merge result thinks the blob is missing and asks the server for an object that only exists locally. The server rejects the whole batch, and we then die on the first object in it that really is missing, 3c804b88, which is why the two OIDs in the error differ. The bug is older than 5946a564cd (odb/source: make `read_object_info()` function pluggable), but that commit, which matches the bisect result, is what exposed it. Before it, any non-quick lookup that missed called odb_reprepare(), and that also threw away the whole loose cache. A merge does such lookups between writing the merged blob and checking out the result, so in practice the stale cache was discarded before anything could use it. Since that commit, the retry only reprepares the packfile store, and the stale cache survives until the checkout. I think the fix is to maintain the invariant: after a successful loose object write, if the subdirectory has already been listed, insert the new OID into the oidtree. (Clearing the subdirectory's bit would also work, at the cost of listing it again.) I have a local fix with a unit test for the loose source. An end-to-end test is awkward, since it needs a merged blob whose first byte matches one of the prefetched objects. I could polish this into a proper patch, but since Patrick Steinhardt has been reworking this part of the code, I think his thoughts on the approach would be useful first. Thanks, Kristofer ``` ## Patrick Ohly, 2026-11-04 11:56 Subject: Re: [BUG] Lazy fetch in partial clone fails with "not our ref" since 5946a564 (odb/source: make read_object_info() pluggable) Message-ID: URL: https://gitlist.dev/e/yrjhld78q06y.fsf%40pohly-mobl1.fritz.box In-Reply-To: ``` Kristofer Karlsson writes: > On Tue, 3 Nov 2026 at 14:31, Patrick Ohly wrote: >> >> Hello, >> >> "git merge" in a blobless partial clone fails with "not our ref" while >> lazily fetching a missing object. This started to occur today in the >> Kubernetes CI, completely breaking the merging of PRs. We should have >> been using git 2.54.0, the affected version, for a few days at least >> without issues, so it can't be ruled out that this is at least partly >> caused by a change on the GitHub side. We are about to revert to git >> 2.52.0 to unblock the Kubernetes project. >> >> [...] >> >> I bisected v2.52.0 (good) to v2.54.0 (bad). The first bad commit is: >> >> 5946a564cddc0bf471f27ae4c3fe205441e3ef65 >> odb/source: make `read_object_info()` function pluggable >> >> The parent, 3bc3177ad7 ( 2.53.0.185.g3bc3177ad7 ), is good. > > Thanks for the detailed report. I think I have found the cause: I > can reproduce the failure with your steps and make it go away with a > small change to the loose object cache. Any idea why this suddenly started to occur? We were using 2.54.0 for several days without issues prior to the failure. For example, https://storage.googleapis.com/kubernetes-ci-logs/pr-logs/pull/batch/pull-kubernetes-cmd/2117551425255903232/clone-log.txt merged several different PRs for https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/batch/pull-kubernetes-cmd/2117551425255903232 using us-docker.pkg.dev/k8s-infra-prow/images/clonerefs:v20261028-c06373a3e as environment. It contains 2.54.0. -- Best Regards Patrick Ohly Cloud Software Architect ``` ## Patrick Steinhardt, 2026-11-04 12:19 Subject: Re: [BUG] Lazy fetch in partial clone fails with "not our ref" since 5946a564 (odb/source: make read_object_info() pluggable) Message-ID: URL: https://gitlist.dev/e/ausi3vZtRdB_40Bc%40pks.im In-Reply-To: ``` Hi, On Tue, Nov 03, 2026 at 02:24:51PM +0100, Patrick Ohly wrote: > Hello, > > "git merge" in a blobless partial clone fails with "not our ref" while > lazily fetching a missing object. This started to occur today in the > Kubernetes CI, completely breaking the merging of PRs. We should have > been using git 2.54.0, the affected version, for a few days at least > without issues, so it can't be ruled out that this is at least partly > caused by a change on the GitHub side. We are about to revert to git > 2.52.0 to unblock the Kubernetes project. > > What follows is an agent analysis of the problem. I'm not rewriting it > because it seems succinct and useful as-is: > > What did you do before the bug happened? (Steps to reproduce) > > git init k && cd k > R=https://github.com/kubernetes/kubernetes.git > git fetch --filter=blob:none $R 6575724b63e50cd5dd454e774bc293c6de1e4e12 > git checkout 6575724b63e50cd5dd454e774bc293c6de1e4e12 > git fetch --filter=blob:none $R 787dd27af44c14c22d3bae46bf62b4a0271efc06 > git -c user.name=a -c user.email=a@b merge --no-ff 787dd27af44c14c22d3bae46bf62b4a0271efc06 > > What did you expect to happen? > > The merge succeeds, with missing blobs fetched lazily from the promisor remote. > > What happened instead? > > fatal: remote error: upload-pack: not our ref 27b64cc863187e62d8b47196a78c045412d32ac9 > fatal: could not fetch 3c804b88bcf177a1435ba770fa9054c25e1315c3 from promisor remote thanks for this well-written report, I can reproduce the bug. > Bisect result > > I bisected v2.52.0 (good) to v2.54.0 (bad). The first bad commit is: > > 5946a564cddc0bf471f27ae4c3fe205441e3ef65 > odb/source: make `read_object_info()` function pluggable > > The parent, 3bc3177ad7 ( 2.53.0.185.g3bc3177ad7 ), is good. The root cause of this is actually older than the commit you have bisected to: we never update the loose object cache when writing objects, so in case we have already populated the specific sharding directory that the newly written object belongs to the cache will be stale. And as object existence checks are answered via the cache, we will now pretend as if the object didn't exist at all. The commit changes logic to not flush the loose object cache anymore in case we cannot find an object. But the fact that flushing the loose object cache fixes the bug was pure coincidence, as it required us to do interleaving objects reads and object existence checks. I've sent fixes via [1]. Thanks! Patrick [1]: <20261104-pks-odb-loose-stale-cache-v1-0-a00d5cdb5e7d@pks.im> ``` ## Patrick Steinhardt, 2026-11-04 12:22 Subject: Re: [BUG] Lazy fetch in partial clone fails with "not our ref" since 5946a564 (odb/source: make read_object_info() pluggable) Message-ID: URL: https://gitlist.dev/e/ausjegMf8AM-URB2%40pks.im In-Reply-To: ``` On Wed, Nov 04, 2026 at 12:56:53PM +0100, Patrick Ohly wrote: > Kristofer Karlsson writes: > > > On Tue, 3 Nov 2026 at 14:31, Patrick Ohly wrote: > >> > >> Hello, > >> > >> "git merge" in a blobless partial clone fails with "not our ref" while > >> lazily fetching a missing object. This started to occur today in the > >> Kubernetes CI, completely breaking the merging of PRs. We should have > >> been using git 2.54.0, the affected version, for a few days at least > >> without issues, so it can't be ruled out that this is at least partly > >> caused by a change on the GitHub side. We are about to revert to git > >> 2.52.0 to unblock the Kubernetes project. > >> > >> [...] > >> > >> I bisected v2.52.0 (good) to v2.54.0 (bad). The first bad commit is: > >> > >> 5946a564cddc0bf471f27ae4c3fe205441e3ef65 > >> odb/source: make `read_object_info()` function pluggable > >> > >> The parent, 3bc3177ad7 ( 2.53.0.185.g3bc3177ad7 ), is good. > > > > Thanks for the detailed report. I think I have found the cause: I > > can reproduce the failure with your steps and make it go away with a > > small change to the loose object cache. > > Any idea why this suddenly started to occur? > > We were using 2.54.0 for several days without issues prior to the > failure. For example, > https://storage.googleapis.com/kubernetes-ci-logs/pr-logs/pull/batch/pull-kubernetes-cmd/2117551425255903232/clone-log.txt > merged several different PRs for > https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/batch/pull-kubernetes-cmd/2117551425255903232 > using > us-docker.pkg.dev/k8s-infra-prow/images/clonerefs:v20261028-c06373a3e > as environment. It contains 2.54.0. The issue here depends on the actual objects IDs that you're merging. You need to merge at least one object with the same first two hex characters into a newly written object that same first two characters. If so, you can trigger a stale loose object cache that makes Git think that the object does not exist locally, and it then tries to fetch it from the promisor remote. I've sent a fix for this bug via [1]. Thanks! Patrick [1]: <20261104-pks-odb-loose-stale-cache-v1-0-a00d5cdb5e7d@pks.im> ```