Re: Git Scaling: What factors most affect Git performance for a large repo?

From: David Turner <dturner@twopensource.com>
Date: Feb 23, 2015, 20:23 UTC
Message-ID: <1424722996.27803.29.camel@leckie>
In-Reply-To: <xmqqegpkz4cf.fsf@gitster.dls.corp.google.com>

On Fri, 2015-02-20 at 12:59 -0800, Junio C Hamano wrote:

Show 27 quoted lines

> David Turner <dturner@twopensource.com> writes:
> 
> > On Fri, 2015-02-20 at 06:38 +0700, Duy Nguyen wrote:
> >> >    * 'git push'?
> >> 
> >> This one is not affected by how deep your repo's history is, or how
> >> wide your tree is, so should be quick..
> >> 
> >> Ah the number of refs may affect both git-push and git-pull. I think
> >> Stefan knows better than I in this area.
> >
> > I can tell you that this is a bit of a problem for us at Twitter.  We
> > have over 100k refs, which adds ~20MiB of downstream traffic to every
> > push.
> >
> > I added a hack to improve this locally inside Twitter: The client sends
> > a bloom filter of shas that it believes that the server knows about; the
> > server sends only the sha of master and any refs that are not in the
> > bloom filter.  The client  uses its local version of the servers' refs
> > as if they had just been sent....
> 
> Interesting.
> 
> Care to extend the discussion to improve the protocol exchange,
> which starts at $gmane/263932 [*1*], where I list the known issues
> around the current protocol (and a possible way to correct them in
> footnotes)?

At Twitter, we changed to an entirely different clone strategy for our largest repo: instead of using git clone, we use bittorrent (on a tarball of the repo). For git pull, we maintain a journal of all pushes ever made to the server (data and ref updates); each client keeps track of their location in that journal. So now pull does not require any computation on the server; the client just requests the segment of the journal that they don't have. Then the client replays the journal. This scheme isn't perfect: clients end up with data about even transitory and long-dead branches, and there is presently no way to redact data (although that would be possible to add). And of course shallow and sparse clones are impossible. But it works quite well for Twitter's needs. As I understand it, the hope is to implement redaction and then submit patches upstream.

I say "we", but I personally did not do any of the above work. Because I haven't looked into most of these issues personally, I'm reluctant to say too much on protocol improvements. I would want to better understand the constraints. I do think there is value in having a diversity of possible protocols to handle different use cases. As repositories grow, traditional full-repo clones become less viable. Network transfer and client-side performance both suffer. In a repo the size of (say) WebKit, the traditional model works. In a repo the size of Facebook's monorepo, it starts to break down. So Facebook does entirely shallow clones (using hg, but the problems are similar in git). Commands like log and blame instead call out to a server to gather history data. At Google, whose repo is I think two or three orders of magnitude larger than WebKit, all local copies are both shallow and sparse; there is also support for "sparse commits" -- so that a commit that affects (say) ten thousand files across the entire tree can be kept to a reasonable size.

<end digression>

Twitter's journal scheme explains why I implemented bloom filter pushes -- the number of refs does not significantly affect pull performance, but pushes still go through the normal git machinery, so we wanted an optimization to reduce latency there.