Re: [PATCH] rev-list: add "--full-objects" flag.

From: Linus Torvalds <torvalds@osdl.org>
Date: Jul 8, 2005, 02:39 UTC
Message-ID: <Pine.LNX.4.58.0507071928220.25104@g5.osdl.org>
In-Reply-To: <7vy88ica8e.fsf@assigned-by-dhcp.cox.net>

On Thu, 7 Jul 2005, Junio C Hamano wrote:

Show 8 quoted lines

> 
> However it does not automatically mean that the avenue I have
> been pursuing would not work; the server side preparation needs
> to be a bit more careful than what I sent, which unconditionally
> runs "prune-packed".  It instead should leave the files that
> "--whole-trees" would have packed as plain SHA1 files, so that
> the bulk is obtained by statically generated packs and the rest
> can be handled in the commit-chain walker as before.

I really think the commit-chain walker needs to run locally (ie at the server end, or after fetching all the objects from the server).

I don't know how much you've tried out the git-http-pull and git-ssh-pull things, but their performance was quite horrid for anything half-way bigger, because of the totally synchronized IO.

The "fetch one object, parse it, fetch the next one, parse that.." approach is just horrible.

I ended up preferring the "rsync" thing even though rsync sucked badly on big object stores too, if only because when rsync got working, it at least nicely pipelined the transfers, and would transfer things ten times faster than git-ssh-pull did (maybe I'm exaggerating, but I don't think so, it really felt that way).

And the thing is, if you purely follow one tree (which is likely the common case for a lot of users), then you are actually always likely better off with the "mirror it" model. Which is _not_ a good model for developers (for example, me rsync'ing from Jeff's kernel repository always got me hundreds of useless objects), but it's fine for somebody who actually just wants to track somebody else.

And then you really can use just rsync or wget or ncftpget or anything else that has a "fetch recursively, optimizing existing objects" mode.

Now, re-packing ends up causing some double transmissions, but I bet the cost of those are going to be less than the cost of the "ping-pong for each object" approach. Especially as most of the repacked objects will be deltas if the repacking is done properly.

		Linus