Re: Git Scaling: What factors most affect Git performance for a large repo?

From: Junio C Hamano <gitster@pobox.com>
Date: Feb 20, 2015, 20:59 UTC
Message-ID: <xmqqegpkz4cf.fsf@gitster.dls.corp.google.com>
In-Reply-To: <1424392969.30029.15.camel@leckie>

David Turner <dturner@twopensource.com> writes:

Show 18 quoted lines

> On Fri, 2015-02-20 at 06:38 +0700, Duy Nguyen wrote:
>> >    * 'git push'?
>> 
>> This one is not affected by how deep your repo's history is, or how
>> wide your tree is, so should be quick..
>> 
>> Ah the number of refs may affect both git-push and git-pull. I think
>> Stefan knows better than I in this area.
>
> I can tell you that this is a bit of a problem for us at Twitter.  We
> have over 100k refs, which adds ~20MiB of downstream traffic to every
> push.
>
> I added a hack to improve this locally inside Twitter: The client sends
> a bloom filter of shas that it believes that the server knows about; the
> server sends only the sha of master and any refs that are not in the
> bloom filter.  The client  uses its local version of the servers' refs
> as if they had just been sent....

Interesting.

Care to extend the discussion to improve the protocol exchange, which starts at $gmane/263932 [*1*], where I list the known issues around the current protocol (and a possible way to correct them in footnotes)?

[Footnote]

*1* http://thread.gmane.org/gmane.comp.version-control.git/263898/focus=263932