`git gc` says "unable to read" but `git fsck` happy

Here's an example session:

    % LANG=C git fsck --strict; LANG=C git gc
    Checking object directories: 100% (256/256), done.
    error in tree 2699d230e3b592ae42506d7b5c969a7ac6a4593c: zeroPaddedFilemode: contains zero-padded file modes
    Checking objects: 100% (462555/462555), done.
    Verifying commits in commit graph: 100% (117904/117904), done.
    Enumerating objects: 462573, done.
    Counting objects: 100% (462573/462573), done.
    Delta compression using up to 8 threads
    Compressing objects: 100% (155363/155363), done.
    fatal: unable to read f5e44b38fc8f7e15e5e6718090d05b09912254fa
    fatal: failed to run repack
    %

How come it can't read `f5e44b38fc8f7e15e5e6718090d05b09912254fa` during "repack" while `git fsck` says everything is fine?

More importantly: how do I diagnose this further and fix it?

Rumors on the net suggest that `git gc --aggressive` may circumvent this problem occasionally, but those don't seem to know what they're talking about, and in my case it didn't make any difference (except that it takes more time :-).

        Stefan

On Wed, Mar 29, 2023 at 06:05:24PM -0400, Stefan Monnier wrote:

Do you use separate worktrees? It sounds like it might be similar to this case:

  https://lore.kernel.org/git/c6246ed5-bffc-7af9-1540-4e2071eff5dc@kdbg.org/

If so, there are patches in the current "master" (but not in a released version yet) that fix fsck to detect this.

If it is the same problem (which would be a blob or maybe cached tree missing in one of the worktree's index files), then probably you'd either:

  1. Accept the loss and blow away that worktree's index file (or
     perhaps even the whole worktree, and just recreate it).

  2. Try to "git add" the file in question to recreate the blob
     (assuming that the file itself is still hanging around).

The original corruption bug itself (gc not taking into account worktree index files) has been fixed for a while, so the theory is that this can be lingering corruption from a repack by an older version of Git. But if you have evidence to the contrary, we'd like to hear that, too. ;)

I don't think --aggressive would help at all. In theory --prune=now might, but I think even that won't help if the problem is that the object is referenced in an index file.

It could also be a totally unrelated bug, perhaps where we are too eager to complain about missing objects in unreachable history we're trying to retain. In which case "git gc --prune=now" _would_ help (but it might be nice to save a copy of the repository before trying, because that would indicate a bug we still need to fix, and your repo is worth investigating).

-Peff

Very much so, indeed!

That's sounds exactly right. I was actually preparing to file a separate bug report because of a similar problem I had identified where a worktree's `index` caused a similar problem (`git fsck` happy but `git gc` fails) except it was found much earlier in `git gc`, causing a "bad object" error almost right away.

Good, thanks.

Hmm... the problem is "that": I have about a hundred worktrees for this repository. But yes, I can just throw away all those `index` files, I guess.

My suspicion is that the origin of the broken state is elsewhere (maybe a power failure?) because the problem appeared "simultaneously" (a few days apart, really) for two different repositories.

Indeed, I had also tried `--prune=now` and it did not help. Thanks,

        Stefan

On Thu, Mar 30, 2023 at 09:01:39AM -0400, Stefan Monnier wrote:

If you try "git fsck" from the tip of master, it should identify the worktree index that is the source of the problem, I think. You might need to pass "--name-objects".

Hmm. I wouldn't expect that to happen specifically with this worktree thing, but of course many bets are off with power failures.

-Peff

On Mär 30 2023, Jeff King wrote:

I had the same problem, and after Junio refreshed my memory by pointing me to this thread, I updated to the brand new git 2.41 and re-ran git fsck. That duely identified problems in two of the worktree indexes (invalid sha1 pointer in resolve-undo). After recreating those indexes there were no more complaints from git gc.