{"thread":{"id":"66753","subject":"[BUG] Lazy fetch in partial clone fails with \"not our ref\" since 5946a564 (odb/source: make read_object_info() pluggable)","startedAt":"2026-11-03T13:24:51Z","lastAt":"2026-11-04T12:22:18Z","messageCount":5,"participants":["Patrick Ohly","Kristofer Karlsson","Patrick Steinhardt"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"557316","messageId":"yrjh1p92qc7w.fsf@pohly-mobl1.fritz.box","threadId":"66753","inReplyTo":null,"subject":"[BUG] Lazy fetch in partial clone fails with \"not our ref\" since 5946a564 (odb/source: make read_object_info() pluggable)","fromName":"Patrick Ohly","fromEmail":"patrick.ohly@intel.com","sentAt":"2026-11-03T13:24:51Z","receivedAt":"2026-11-03T13:24:51Z","isPatch":false,"sender":{"key":"patrick.ohly@intel.com","avatar":null},"body":"Hello,\n\n\"git merge\" in a blobless partial clone fails with \"not our ref\" while\nlazily fetching a missing object. This started to occur today in the\nKubernetes CI, completely breaking the merging of PRs. We should have\nbeen using git 2.54.0, the affected version, for a few days at least\nwithout issues, so it can't be ruled out that this is at least partly\ncaused by a change on the GitHub side. We are about to revert to git\n2.52.0 to unblock the Kubernetes project.\n\nWhat follows is an agent analysis of the problem. I'm not rewriting it\nbecause it seems succinct and useful as-is:\n\nWhat did you do before the bug happened? (Steps to reproduce)\n\ngit init k && cd k\nR=https://github.com/kubernetes/kubernetes.git\ngit fetch --filter=blob:none $R 6575724b63e50cd5dd454e774bc293c6de1e4e12\ngit checkout 6575724b63e50cd5dd454e774bc293c6de1e4e12\ngit fetch --filter=blob:none $R 787dd27af44c14c22d3bae46bf62b4a0271efc06\ngit -c user.name=a -c user.email=a@b merge --no-ff 787dd27af44c14c22d3bae46bf62b4a0271efc06\n\nWhat did you expect to happen?\n\nThe merge succeeds, with missing blobs fetched lazily from the promisor remote.\n\nWhat happened instead?\n\nfatal: remote error: upload-pack: not our ref 27b64cc863187e62d8b47196a78c045412d32ac9\nfatal: could not fetch 3c804b88bcf177a1435ba770fa9054c25e1315c3 from promisor remote\n\nBisect result\n\nI bisected v2.52.0 (good) to v2.54.0 (bad). The first bad commit is:\n\n5946a564cddc0bf471f27ae4c3fe205441e3ef65\nodb/source: make `read_object_info()` function pluggable\n\nThe parent, 3bc3177ad7 ( 2.53.0.185.g3bc3177ad7 ), is good.\n\nVersions tested\n\n┌───────────────────────────────────────────────────────┬────────┐\n│ Version                                               │ Result │\n├───────────────────────────────────────────────────────┼────────┤\n│ 2.39.5, 2.43.4, 2.45.2, 2.47.2, 2.49.1, 2.52.0        │ OK     │\n├───────────────────────────────────────────────────────┼────────┤\n│ 2.54.0, 2.56.0                                        │ fail   │\n├───────────────────────────────────────────────────────┼────────┤\n│ master (v2.56.0-341-gebdc087dc, built with NO_RUST=1) │ fail   │\n└───────────────────────────────────────────────────────┴────────┘\n\nNotes\n\n• The remote is github.com over HTTPS. I haven't tried other hosts.\n• I haven't analysed the cause. My guess is that the new  OBJECT_INFO_SECOND_READ  and reprepare path changes how the promisor lazy fetch is triggered. The client then requests an object by OID that the server refuses with \"not our ref\".\n• The failure is deterministic. The first PR head merges fine on 2.52.0 and older.\n\n[System Info]\n\ngit version 2.56.GIT (also 2.54.0 and 2.56.0), Linux x86_64 (Debian stable container)\n\n-- \nBest Regards\n\nPatrick Ohly\nCloud Software Architect\n"},{"id":"557420","messageId":"CAL71e4MqMdGk5ryhNxvVNN6Z02+45B933ZfUuvj4hM92N5a6hA@mail.gmail.com","threadId":"66753","inReplyTo":"yrjh1p92qc7w.fsf@pohly-mobl1.fritz.box","subject":"Re: [BUG] Lazy fetch in partial clone fails with \"not our ref\" since 5946a564 (odb/source: make read_object_info() pluggable)","fromName":"Kristofer Karlsson","fromEmail":"krka@spotify.com","sentAt":"2026-11-04T10:11:21Z","receivedAt":"2026-11-04T10:11:21Z","isPatch":false,"sender":{"key":"krka@spotify.com","avatar":"https://avatars.githubusercontent.com/u/5349266?v=4"},"body":"On Tue, 3 Nov 2026 at 14:31, Patrick Ohly <patrick.ohly@intel.com> wrote:\n>\n> Hello,\n>\n> \"git merge\" in a blobless partial clone fails with \"not our ref\" while\n> lazily fetching a missing object. This started to occur today in the\n> Kubernetes CI, completely breaking the merging of PRs. We should have\n> been using git 2.54.0, the affected version, for a few days at least\n> without issues, so it can't be ruled out that this is at least partly\n> caused by a change on the GitHub side. We are about to revert to git\n> 2.52.0 to unblock the Kubernetes project.\n>\n> [...]\n>\n> I bisected v2.52.0 (good) to v2.54.0 (bad). The first bad commit is:\n>\n> 5946a564cddc0bf471f27ae4c3fe205441e3ef65\n> odb/source: make `read_object_info()` function pluggable\n>\n> The parent, 3bc3177ad7 ( 2.53.0.185.g3bc3177ad7 ), is good.\n\nThanks for the detailed report.  I think I have found the cause: I\ncan reproduce the failure with your steps and make it go away with a\nsmall change to the loose object cache.\n\nThe problem is in the semantics of the loose object cache, which\nanswers quick existence checks.  It is a two-layer cache: a bitmap\nkeyed by the first byte of the object ID, recording which\n\"objects/XX/\" subdirectories have been listed, and an oidtree with\nthe objects found there.  The intended invariant is that once a\nsubdirectory has been listed, the oidtree holds every loose object\nin it, so that \"not in the oidtree\" means \"does not exist\".\n\nWriting a loose object does not update the cache, which breaks the\ninvariant if the subdirectory had already been listed.  In\npractice, that means a quick lookup of any object in the same\nsubdirectory earlier in the same process.\n\nIn your example, the object the server refuses, 27b64cc8, is not in\neither parent nor in the merge base.  It is the content-merged blob\nthat the merge itself just wrote as a loose object.  merge-ort's\nprefetch had already listed \"27/\", so the checkout of the merge\nresult thinks the blob is missing and asks the server for an object\nthat only exists locally.  The server rejects the whole batch, and\nwe then die on the first object in it that really is missing,\n3c804b88, which is why the two OIDs in the error differ.\n\nThe bug is older than 5946a564cd (odb/source: make\n`read_object_info()` function pluggable), but that commit, which\nmatches the bisect result, is what exposed it.  Before it, any\nnon-quick lookup that missed called odb_reprepare(), and that also\nthrew away the whole loose cache.  A merge does such lookups between\nwriting the merged blob and checking out the result, so in practice\nthe stale cache was discarded before anything could use it.  Since\nthat commit, the retry only reprepares the packfile store, and the\nstale cache survives until the checkout.\n\nI think the fix is to maintain the invariant: after a successful\nloose object write, if the subdirectory has already been listed,\ninsert the new OID into the oidtree.  (Clearing the subdirectory's\nbit would also work, at the cost of listing it again.)\n\nI have a local fix with a unit test for the loose source.  An\nend-to-end test is awkward, since it needs a merged blob whose first\nbyte matches one of the prefetched objects.\n\nI could polish this into a proper patch, but since Patrick\nSteinhardt has been reworking this part of the code, I think his\nthoughts on the approach would be useful first.\n\nThanks,\nKristofer\n\n"},{"id":"557427","messageId":"yrjhld78q06y.fsf@pohly-mobl1.fritz.box","threadId":"66753","inReplyTo":"CAL71e4MqMdGk5ryhNxvVNN6Z02+45B933ZfUuvj4hM92N5a6hA@mail.gmail.com","subject":"Re: [BUG] Lazy fetch in partial clone fails with \"not our ref\" since 5946a564 (odb/source: make read_object_info() pluggable)","fromName":"Patrick Ohly","fromEmail":"patrick.ohly@intel.com","sentAt":"2026-11-04T11:56:53Z","receivedAt":"2026-11-04T11:56:53Z","isPatch":false,"sender":{"key":"patrick.ohly@intel.com","avatar":null},"body":"Kristofer Karlsson <krka@spotify.com> writes:\n\n> On Tue, 3 Nov 2026 at 14:31, Patrick Ohly <patrick.ohly@intel.com> wrote:\n>>\n>> Hello,\n>>\n>> \"git merge\" in a blobless partial clone fails with \"not our ref\" while\n>> lazily fetching a missing object. This started to occur today in the\n>> Kubernetes CI, completely breaking the merging of PRs. We should have\n>> been using git 2.54.0, the affected version, for a few days at least\n>> without issues, so it can't be ruled out that this is at least partly\n>> caused by a change on the GitHub side. We are about to revert to git\n>> 2.52.0 to unblock the Kubernetes project.\n>>\n>> [...]\n>>\n>> I bisected v2.52.0 (good) to v2.54.0 (bad). The first bad commit is:\n>>\n>> 5946a564cddc0bf471f27ae4c3fe205441e3ef65\n>> odb/source: make `read_object_info()` function pluggable\n>>\n>> The parent, 3bc3177ad7 ( 2.53.0.185.g3bc3177ad7 ), is good.\n>\n> Thanks for the detailed report.  I think I have found the cause: I\n> can reproduce the failure with your steps and make it go away with a\n> small change to the loose object cache.\n\nAny idea why this suddenly started to occur?\n\nWe were using 2.54.0 for several days without issues prior to the\nfailure. For example,\nhttps://storage.googleapis.com/kubernetes-ci-logs/pr-logs/pull/batch/pull-kubernetes-cmd/2117551425255903232/clone-log.txt\nmerged several different PRs for\nhttps://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/batch/pull-kubernetes-cmd/2117551425255903232\nusing\nus-docker.pkg.dev/k8s-infra-prow/images/clonerefs:v20261028-c06373a3e\nas environment. It contains 2.54.0.\n\n-- \nBest Regards\n\nPatrick Ohly\nCloud Software Architect\n\n"},{"id":"557431","messageId":"ausi3vZtRdB_40Bc@pks.im","threadId":"66753","inReplyTo":"yrjh1p92qc7w.fsf@pohly-mobl1.fritz.box","subject":"Re: [BUG] Lazy fetch in partial clone fails with \"not our ref\" since 5946a564 (odb/source: make read_object_info() pluggable)","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-04T12:19:42Z","receivedAt":"2026-11-04T12:19:42Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nOn Tue, Nov 03, 2026 at 02:24:51PM +0100, Patrick Ohly wrote:\n> Hello,\n> \n> \"git merge\" in a blobless partial clone fails with \"not our ref\" while\n> lazily fetching a missing object. This started to occur today in the\n> Kubernetes CI, completely breaking the merging of PRs. We should have\n> been using git 2.54.0, the affected version, for a few days at least\n> without issues, so it can't be ruled out that this is at least partly\n> caused by a change on the GitHub side. We are about to revert to git\n> 2.52.0 to unblock the Kubernetes project.\n> \n> What follows is an agent analysis of the problem. I'm not rewriting it\n> because it seems succinct and useful as-is:\n> \n> What did you do before the bug happened? (Steps to reproduce)\n> \n> git init k && cd k\n> R=https://github.com/kubernetes/kubernetes.git\n> git fetch --filter=blob:none $R 6575724b63e50cd5dd454e774bc293c6de1e4e12\n> git checkout 6575724b63e50cd5dd454e774bc293c6de1e4e12\n> git fetch --filter=blob:none $R 787dd27af44c14c22d3bae46bf62b4a0271efc06\n> git -c user.name=a -c user.email=a@b merge --no-ff 787dd27af44c14c22d3bae46bf62b4a0271efc06\n> \n> What did you expect to happen?\n> \n> The merge succeeds, with missing blobs fetched lazily from the promisor remote.\n> \n> What happened instead?\n> \n> fatal: remote error: upload-pack: not our ref 27b64cc863187e62d8b47196a78c045412d32ac9\n> fatal: could not fetch 3c804b88bcf177a1435ba770fa9054c25e1315c3 from promisor remote\n\nthanks for this well-written report, I can reproduce the bug.\n\n> Bisect result\n> \n> I bisected v2.52.0 (good) to v2.54.0 (bad). The first bad commit is:\n> \n> 5946a564cddc0bf471f27ae4c3fe205441e3ef65\n> odb/source: make `read_object_info()` function pluggable\n> \n> The parent, 3bc3177ad7 ( 2.53.0.185.g3bc3177ad7 ), is good.\n\nThe root cause of this is actually older than the commit you have\nbisected to: we never update the loose object cache when writing\nobjects, so in case we have already populated the specific sharding\ndirectory that the newly written object belongs to the cache will be\nstale. And as object existence checks are answered via the cache, we\nwill now pretend as if the object didn't exist at all.\n\nThe commit changes logic to not flush the loose object cache anymore in\ncase we cannot find an object. But the fact that flushing the loose\nobject cache fixes the bug was pure coincidence, as it required us to do\ninterleaving objects reads and object existence checks.\n\nI've sent fixes via [1]. Thanks!\n\nPatrick\n\n[1]: <20261104-pks-odb-loose-stale-cache-v1-0-a00d5cdb5e7d@pks.im>\n\n"},{"id":"557432","messageId":"ausjegMf8AM-URB2@pks.im","threadId":"66753","inReplyTo":"yrjhld78q06y.fsf@pohly-mobl1.fritz.box","subject":"Re: [BUG] Lazy fetch in partial clone fails with \"not our ref\" since 5946a564 (odb/source: make read_object_info() pluggable)","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-04T12:22:18Z","receivedAt":"2026-11-04T12:22:18Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Nov 04, 2026 at 12:56:53PM +0100, Patrick Ohly wrote:\n> Kristofer Karlsson <krka@spotify.com> writes:\n> \n> > On Tue, 3 Nov 2026 at 14:31, Patrick Ohly <patrick.ohly@intel.com> wrote:\n> >>\n> >> Hello,\n> >>\n> >> \"git merge\" in a blobless partial clone fails with \"not our ref\" while\n> >> lazily fetching a missing object. This started to occur today in the\n> >> Kubernetes CI, completely breaking the merging of PRs. We should have\n> >> been using git 2.54.0, the affected version, for a few days at least\n> >> without issues, so it can't be ruled out that this is at least partly\n> >> caused by a change on the GitHub side. We are about to revert to git\n> >> 2.52.0 to unblock the Kubernetes project.\n> >>\n> >> [...]\n> >>\n> >> I bisected v2.52.0 (good) to v2.54.0 (bad). The first bad commit is:\n> >>\n> >> 5946a564cddc0bf471f27ae4c3fe205441e3ef65\n> >> odb/source: make `read_object_info()` function pluggable\n> >>\n> >> The parent, 3bc3177ad7 ( 2.53.0.185.g3bc3177ad7 ), is good.\n> >\n> > Thanks for the detailed report.  I think I have found the cause: I\n> > can reproduce the failure with your steps and make it go away with a\n> > small change to the loose object cache.\n> \n> Any idea why this suddenly started to occur?\n> \n> We were using 2.54.0 for several days without issues prior to the\n> failure. For example,\n> https://storage.googleapis.com/kubernetes-ci-logs/pr-logs/pull/batch/pull-kubernetes-cmd/2117551425255903232/clone-log.txt\n> merged several different PRs for\n> https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/batch/pull-kubernetes-cmd/2117551425255903232\n> using\n> us-docker.pkg.dev/k8s-infra-prow/images/clonerefs:v20261028-c06373a3e\n> as environment. It contains 2.54.0.\n\nThe issue here depends on the actual objects IDs that you're merging.\nYou need to merge at least one object with the same first two hex\ncharacters into a newly written object that same first two characters.\nIf so, you can trigger a stale loose object cache that makes Git think\nthat the object does not exist locally, and it then tries to fetch it\nfrom the promisor remote.\n\nI've sent a fix for this bug via [1]. Thanks!\n\nPatrick\n\n[1]: <20261104-pks-odb-loose-stale-cache-v1-0-a00d5cdb5e7d@pks.im>\n\n"}]}