patch, 2 partsodb/source-loose: fix failing existence checks caused by stale cache

Hi,

this small patch series fixes the bug reported in [1].

In summary, when git-merge(1) writes merged blobs it may be unable to read them back because the loose object cache does not get updated. Consequently, we may think that the blobs do not exist locally, and thus in a promisor remote we may perform a backfill fetch. But as the remote doesn't know those objects we'll fail.

Thanks!

Patrick

[1]: yrjh1p92qc7w.fsf@pohly-mobl1.fritz.box

---
Patrick Steinhardt (2):
      odb/source-loose: extract logic to retrieve subdirectory bitmap
      odb/source-loose: fix existence checks for newly written objects

 odb/source-loose.c       | 54 +++++++++++++++++++++++++++++++++++++++---------
 t/t5616-partial-clone.sh | 43 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 87 insertions(+), 10 deletions(-)

--- base-commit: 750f5de7dfe9a4062f95088212b063f189d50e91 change-id: 20261104-pks-odb-loose-stale-cache-c16107987290

The loose object cache tracks which sharding directories have already been scanned via a bitmap. The logic to retrieve the bitmap and mask for a given object ID is inlined into the function that populates the cache, but we're about to add a second function that will need to perform the same logic.

Extract the logic into a separate function to prepare for this.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-loose.c | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)

The loose object source maintains a cache of object IDs that it uses to answer existence checks fast. This cache is lazily populated: whenever we are asked about a specific object ID, we scan the whole sharding directory it would live in and populate the cache with the objects we find. Subsequent existence checks are then answered from the cache without hitting the filesystem.

But when writing a new loose object we don't update that cache. That's fine in the case where the cache for its sharding directory has not yet been populated, because we would then populate it on a subsequent read and thus surface the newly written object. But when it _is_ populated already, then any subsequent existence check will claim that the object does not exist at all until the cache is flushed.

This issue can cause git-merge(1) to die in a partial clone because we try to fetch an object that we have just written, as reported in [1]:

  1. `prefetch_for_content_merges()` performs quick existence checks for
     all blobs that need a content merge to figure out which ones need
     to be fetched from the promisor remote. These lookups populate the
     loose object cache for the respective sharding directories.

  2. We lazily fetch the missing blobs, which writes a new packfile.

  3. We perform the merge and write the resulting blobs as loose
     objects. The cache is not updated, so for all blobs whose sharding
     directory was populated in (1), the object existence checks will
     fail.

  4. When checking out the merge result, `prefetch_cache_entries()`
     performs existence checks for all index entries that need to be
     written to the worktree. But due to the stale cache, the merged
     blobs are reported as missing, and consequently we try to fetch
     them from the promisor remote. This fetch fails because the remote
     will most likely not have them.

This issue bisects to 5946a564cdd (odb/source: make `read_object_info()` function pluggable, 2026-03-05), because in that commit we stopped flushing the loose object cache when we try to read a missing object. That was done intentionally though: a failing read is only retried when it is a non-QUICK read, whereas the cache only affects QUICK reads. So flushing the cache for a non-QUICK read wouldn't ever cause us to find that object.

So the logic described here only ever worked by chance, as the cache was incoherent after writes even before that commit. It only used to work because of the interleaving sequence of QUICK and non-QUICK reads: when we read the lazily-fetched blobs in (3) via a non-QUICK read, the first read fails because the new packfile has not yet been added to the "packed" source. On the retry we then used to flush the loose object cache, and that in turn means that the subsequent existence check in (4) rescanned the sharding directories and found the merged blobs.

We could in theory fix this by flushing all the loose object caches again, like we used to do before 5946a564cdd. But that wouldn't address the root cause: the cache would still be stale unless somebody does a non-QUICK read for a missing object. Instead, fix the bug by updating the cache whenever we write a new loose object.

[1]: <yrjh1p92qc7w.fsf@pohly-mobl1.fritz.box>

Reported-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-loose.c       | 30 +++++++++++++++++++++++++++---
 t/t5616-partial-clone.sh | 43 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 70 insertions(+), 3 deletions(-)

Patrick Steinhardt <ps@pks.im> writes:

Not about this change but the lack of "if (compat)" stuff in the previous hunk puzzled me. It turns out that the logic to write a single loose object is so big that write_loose_object(), which is the subject of the previous hunk, helps another function, odb_source_loose_write_object(), by being its helper function, and this caller has the corresponding "if (compat)" logic to produce the hash interoperability data.

Looking very good.

On Wed, 4 Nov 2026 at 13:17, Patrick Steinhardt <ps@pks.im> wrote:

The fix looks good to me, and I verified that the test fails without it. My own local fix is logically equivalent, which is a good sign, though perhaps that biased my review.

Thanks, Kristofer

On 11/4/2026 7:57 AM, Kristofer Karlsson wrote:

I also think the fix looks good. (I haven't verified it locally like Kristofer did, but the test is good.)

Thanks, -Stolee

Kristofer Karlsson <krka@spotify.com> writes:

Thanks, both.