Re: [PATCH] gc: do not warn about too many loose objects

From: Jonathan Nieder <jrnieder@gmail.com>
Date: Jul 16, 2018, 21:31 UTC
Message-ID: <20180716213104.GG11513@aiede.svl.corp.google.com>
In-Reply-To: <CABPp-BEpCF9FE7eJwZWjY+bMsjDQnnDaSrHO+e3DtDDsR-=7Hg@mail.gmail.com>

Elijah Newren wrote:

> I totally agree with your general plan to put unreferenced loose
> objects into a pack.  However, I don't think these objects should be
> part of that pack; they should just be deleted instead.

This might be the wrong thread to discuss it, but did you follow the reference/prune race that Peff mentioned? The simplest cure I'm aware of to it does involve writing those objects to a pack. The idea is to enforce a straightforward contract:

There are two kinds of packs: GC and UNREACHABLE_GARBAGE.

Every object in a GC pack has a minimum lifetime of <ttl> (let's say "1 days") from the time they are read. If you start making use of an object from a GC pack (e.g. by creating a new object referencing it), you have three days to ensure it's referenced.

Each UNREACHABLE_GARBAGE pack has a <ttl> (let's say "3 days") from the time it is created. Objects in an UNREACHABLE_GARBAGE have no minimum ttl from the time they are read. If you want to start making use of an object from an UNREACHABLE_GARBAGE pack (e.g. by creating a new object referencing it), then copy it and everything it references to a GC pack.

To avoid a proliferation of UNREACHABLE_GARBAGE packs, there's a rule for coalescing them, but that's not relevant here.

It is perfectly possible for an object in a GC pack to reference an object in an UNREACHABLE_GARBAGE pack via writes racing with gc, but that's fine --- on the next gc run, the unreachable garbage objects get copied to a GC pack.

We've been using this on a JGit DfsRepository based server for > 2 years now and it's been working well. More details are in the "Loose objects and unreachable objects" section in Documentation/technical/ mentioned before.

Thanks, Jonathan