Re: Add a "Flattened Cache" to `git --clone`?

From: Eric Wong <e@yhbt.net>
Date: May 15, 2020, 21:42 UTC
Message-ID: <20200515214257.GA21855@dcvr>
In-Reply-To: <20200514214404.bcbjskgi52bwedlh@chatter.i7.local>

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:

Show 12 quoted lines

> On Thu, May 14, 2020 at 02:23:44PM -0700, Junio C Hamano wrote:
> > > I think something like git-caching-proxy would be a neat project, 
> > > because it would significantly improve mirroring for CI deployments 
> > > without requiring that each individual job implements clone.bundle 
> > > prefetching.
> > 
> > What are we improving with such a proxy, though?
> > 
> > Not bandwidth to the client, apparently. 
> 
> Well, if it sits in front of the CI subnet, then it *does* save 
> bandwidth.

Agreed.

Show 16 quoted lines

> Here's an example with the exact situation we have:
> 
> - the Gerrit server is on the US West Coast
> - the CI builder is on the East Coast
> - each CI job does a full transfer of the multi-MB repo across the 
>   continent, even when cloning shallow
> 
> We solve this by having a local mirror of the repository, but this 
> requires active mirroring to be pre-setup. A caching proxy that could:
> 
> - receive a request for a repository
> - stream the response back to the client
> - cache objects locally
> - use local cache to construct future requests, so only missing objects 
>   are fetched from the remote repo regardless of the haves on the actual 
>   client...

An off-the-shelf HTTP caching proxy (e.g. polipo, Squid) could do a good enough job with dumb HTTP clones (via GIT_SMART_HTTP=0 env).

With well-packed repos, the dumb HTTP transfer cost shouldn't be too high (and git 2.10+ got way faster on the client side with poorly-packed repos, thanks to the Linux kernel-derived list.h).

The occasional full repack on the source git server will invalidate caches and result in a giant download; but it's better than no caching at all and doing giant cross-country transfers all day long.

That said, I'm not sure if any client-side caching proxies can MITM HTTPS and save bandwidth with HTTPS everywhere, nowadays. I seem to recall polipo being abandoned because of HTTPS. Maybe there's a caching HTTPS MITM proxy out there...