Re: How de-duplicate similar repositories with alternates

From: Jeff King <peff@peff.net>
Date: Dec 4, 2018, 07:06 UTC
Message-ID: <20181204070602.GB11010@sigill.intra.peff.net>
In-Reply-To: <CAGZ79ka1sjU+rHctRP4SVMP0GQsK2iZghkU46=f96ugqvX5Neg@mail.gmail.com>

On Thu, Nov 29, 2018 at 10:55:49AM -0800, Stefan Beller wrote:

Show 15 quoted lines

> On Thu, Nov 29, 2018 at 7:00 AM Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
> >
> > A co-worker asked me today how space could be saved when you have
> > multiple checkouts of the same repository (at different revs) on the
> > same machine. I said since these won't block-level de-duplicate well[1]
> > one way to do this is with alternates.
> 
> Another way is to use git-worktree, which would solve the gc issues
> mentioned below?
> 
> I view alternates as a historic artefact as the deduping
> of objects client side can be done using worktrees, and on the
> serverside - I think - most of the git hosters use namespaces
> and put a fork network into the same repository and use pack islands.

Nope, we definitely use alternates. The ref namespace support in Git is not nearly complete enough to run a modern hosting site; it only kicks in for upload-pack and receive-pack. Other commands (e.g., rev-list to traverse for a history-view page) have no support at all. So we share object storage, but not ref storage.

In theory the caller could namespace requests (e.g., the user asks for "foo", the web site feeds "refs/forks/$id/refs/heads/foo" to git). But any bugs are a lot more likely to lead to security problems (oops, you accidentally wrote into somebody else's fork!). And ref storage has traditionally been a sore point for scaling, so giving each fork its own repo and refs helps break that up.

By contrast, object storage is pretty easy to share. It scales reasonably well, and the security model is much simpler due to the immutable nature of object names.

-Peff