Re: Add a "Flattened Cache" to `git --clone`?

From: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Date: May 14, 2020, 21:44 UTC
Message-ID: <20200514214404.bcbjskgi52bwedlh@chatter.i7.local>
In-Reply-To: <xmqqr1vmp5wf.fsf@gitster.c.googlers.com>

On Thu, May 14, 2020 at 02:23:44PM -0700, Junio C Hamano wrote:

Show 8 quoted lines

> > I think something like git-caching-proxy would be a neat project, 
> > because it would significantly improve mirroring for CI deployments 
> > without requiring that each individual job implements clone.bundle 
> > prefetching.
> 
> What are we improving with such a proxy, though?
> 
> Not bandwidth to the client, apparently.

Well, if it sits in front of the CI subnet, then it *does* save bandwidth.

Here's an example with the exact situation we have:

- the Gerrit server is on the US West Coast
- the CI builder is on the East Coast
- each CI job does a full transfer of the multi-MB repo across the 
  continent, even when cloning shallow

We solve this by having a local mirror of the repository, but this requires active mirroring to be pre-setup. A caching proxy that could:

- receive a request for a repository
- stream the response back to the client
- cache objects locally
- use local cache to construct future requests, so only missing objects 
  are fetched from the remote repo regardless of the haves on the actual 
  client...

..now, that would be kinda neat, but I'm not sure how sane or fragile that setup would be. :)

Show 5 quoted lines

> I thought that with the
> reachability bitmap on the server side with reusing packed object,
> it was more or less a solved problem that the server end spends way
> too much time enumerating, deltifying and compressing the object
> data?

Indeed, it's not really solving anything for this case.

-K