Re: the 100 mb push

From: Shawn O. Pearce <spearce@spearce.org>
Date: Dec 15, 2009, 19:42 UTC
Message-ID: <20091215194212.GC18319@spearce.org>
In-Reply-To: <20091215192338.GA16654@gnu.kitenet.net>

Joey Hess <joey@kitenet.net> wrote:

> Is it normal for git push to sometimes transfer much more data
> than seems necessary? Here is a case where that happens:

Yes.

Show 8 quoted lines

> Meanwhile, the testsuite branch is a 100+ mb monster, containing a lot
> of big binaries. In it, a small change has been made in the origin
> repo. In the local repo, a *lot* of *big* files have been deleted from
> the same branch, about 20 mb of files were removed all told. But the diff
> for this change should be quite small.
> 
> So, testsuite needs to be merged before it can be pushed, but git push
> doesn't tell me that. Instead, it goes off and does this for 2+ hours:

The problem here is, unlike fetch, push does not do a common ancestor negotiation. The sending side (your push client) just assumes the remote side has *only* what the remote side advertised.

Since the remote side advertised a commit on the testsuite branch that your client doesn't have, your client was forced to assume there was no common ancestor and sent the entire thing.

This usually doesn't show up that badly because the delta tends to be smaller (no huge binary files), tends to be a strict fast forward (so your client contains what the remote advertised), and tags may help to limit the upload size by being at fixed points in the history (so at worst you upload since the last tag).

Junio wrote a patch series for git push over a year ago to make
it do common ancestor negotiation like git fetch does, but it had
a deadlock problem and the patch series got dropped.  Not enough
people were interested to help Junio carry it through to being
ready for inclusion.

-- 
Shawn.