Re: "git-send-pack"

From: H. Peter Anvin <hpa@zytor.com>
Date: Jun 30, 2005, 22:00 UTC
Message-ID: <42C46B86.8070006@zytor.com>
In-Reply-To: <Pine.LNX.4.58.0506301432500.14331@ppc970.osdl.org>

Linus Torvalds wrote:

Show 25 quoted lines

> 
> On Thu, 30 Jun 2005, H. Peter Anvin wrote:
> 
>>It does that, but it only have to do that when the actual file has 
>>changed.  That's acceptable, at least for the repository sizes we're 
>>likely to deal with within the medium term.
> 
> 
> Well, realize that "incremental packs" deltify a lot worse than a "big
> pack", since pack-files don't do deltas to objects outside the pack-file.
> 
> So we'd get _some_ compression, but not as much as possible. The current
> kernel compresses down to a single 63 MB pack-file (that's with the 2.6.11
> tree too, not just the HEAD history), but without deltas it weights in at
> about 177 MB.
> 
> So a "sum of incremental packs" should be somewhere in between those two
> values, even today. For a single kernel archive.
> 
> So repository sizes aren't exactly trivial. I don't know how expensive
> that rsync hash thing is, but one thing you lose is the ability to
> hardlink objects, so if you have a few kernel repositories at some point
> it doesn't fit in the cache any more, and then the rsync will have to read
> that much pack object stuff from disk in addition to doing the hash. Ugh.
>

The bulk of the cost in doing the hashing comes from having to read the file.

Well, if you grow a single pack file with appending, then you can have delta references to earlier objects within the same pack file.

At least at this point, we'd handle a few very large files a lot better than an enormous swarm of smaller ones.

In the end, it might be that the right thing to do for git on kernel.org is to have a single, unified object store which isn't accessible by anything other than git-specific protocols. There would have to be some way of dealing with, for example, conflicting tags that apply to different repositories, though.

	-hpa