Re: Two issues with mark_reachable_objects

From: Jeff King <peff@peff.net>
Date: Apr 27, 2021, 14:41 UTC
Message-ID: <YIgijn93f639Pp7Z@coredump.intra.peff.net>
In-Reply-To: <CAJ-dYSOVx0egnyxJb6ZjgWvEDR=19QPgc70JQ7cXUjUPZ1XDiQ@mail.gmail.com>

On Tue, Apr 27, 2021 at 11:45:01AM +0100, David Emett wrote:

> I noticed that this only happened when the fetch triggered an automatic GC at
> the end. After a bit of digging I discovered two separate issues:
> 
> 1) It seems that FETCH_HEAD is not considered a root by mark_reachable_objects.

Right, as you discovered, this is known and intentional. I don't have anything to add to the thread you linked already.

Show 6 quoted lines

> 2) If the bitmap_git branch in mark_reachable_objects is taken, the mark_recent
>    argument is ignored. This doesn't _completely_ break "git prune"'s --expire
>    option, as it turns out there is another explicit mtime check in
>    prune_object (prune.c). If that check passes this is not propagated to
>    referenced objects though. So even if a dangling commit is recent, a prune
>    can discard old objects it references.

But this one is a scary bug. As you note, it's not _completely_ breaking "--expire", but it is totally disabling the "reachable from recent" safety added in d3038d22f9 (prune: keep objects reachable from recent objects, 2014-10-15).

The bug here was introduced by me, and comes from a matter of timing. Despite what you'll see in the project history, the "use bitmaps" patch actually predates the "reachable from recent" one. I just didn't clean it up and upstream it until 2019, and failed to notice the bad interaction between the two.

Show 5 quoted lines

> I assume (2) is not intentional, given that "git gc --help" explicitly says
> "Any object with modification time newer than the --prune date is kept, along
> with everything reachable from it." Is it safe to just run the mark_recent
> block after the bitmap_git block? Could add_unseen_recent_objects_to_traversal
> just be called at the start of the bitmap_git block if mark_recent?

So no, definitely not intentional.

I think we'd just want to run the whole mark_recent block after doing the bitmap traversal.

There may be some subtlety with reusing the rev_info struct again. I think we'd want to reset the pending objects list after calling into the bitmap code. It _usually_ does an actual traversal that consumes the list, but not necessarily. I think traverse_bitmap_commit_list() probably ought to be the one to do it, so it behaves more like traverse_commit_list(). (OTOH, I don't think it's _too_ bad if we don't; we'd include those already-seen objects in our traversal, but they should all by definition have the SEEN bit set, so we'd stop there).

I don't think we want to add_unseen_recent_objects_to_traversal() to include it as part of the same traversal, for the same reason the non-bitmap traversal does not combine them: the mark_recent traversal is best-effort. We set revs->ignore_missing_links to be tolerant of already-broken segments of history.

It's possible that we could do the second mark_recent traversal also with bitmaps (but still separately). I can't offhand think of a reason that ignore_missing_links wouldn't behave well there. But since we expect it to be small, I'd be more comfortable just using the regular traversal code.

-Peff