Re: [RFC] Design for http-pull on repo with packs

From: Dan Holmsand <holmsand@gmail.com>
Date: Jul 11, 2005, 15:53 UTC
Message-ID: <42D2960E.3050008@gmail.com>
In-Reply-To: <7v4qb2ni73.fsf@assigned-by-dhcp.cox.net>

Junio C Hamano wrote:

Show 7 quoted lines

> One very minor problem I have with Holmsand approach [*1*] is
> that the original Barkalow puller allowed a really dumb http
> server by not requiring directory index at all.  For somebody
> like me with a cheap ISP account [*2*], it was great that I did
> not have to update 256 index.html files for objects/??/
> directories.  Admittedly, it would be just one directory
> object/pack/, but still...

I totally agree that you shouldn't have to do any special kind of prepping to serve a repository thru http. Which was why I thought it was a good thing to use the default directory listing of the web-server, assuming that this feature would be available on most servers... Apparently not yours, though :-(

And Cogito already relies on directory listings (to find tags to download).

But if git-repack-script generates a "pack index file" automagically, then of course everything is fine.

Show 12 quoted lines

> On the other hand, picking an optimum set of packs from
> overlapping set of packs is indeed a very interesting (and hard
> combinatorial) problem to solve.  I am hoping that in practice
> people would not force clients to do it with "interesting" set
> of packs.  I would hope them to have just a full pack and
> incrementals, never having ovelaps, like Linus plans to do on
> his kernel repo.
> 
> On the other hand, for somebody like Jeff Garzik with 50 heads,
> it might make some sense to have a handful different overlapping
> packs, optimized for different sets of people wanting to pull
> some but not all of his heads.

Well, it is an interresting problem... But I don't think that the solution is to create more pack files. In fact, you'd want as few pack files as possible, for maximum overall efficiency.

I did a little experiment. I cloned Linus' current tree, and git repacked everything (that's 63M + 3.3M worth of pack files). Then I got something like 25 or so of Jeff's branches. That's 6.9M of object files, and 1.4M packed. Total size: 70M for the entire .git/objects/pack directory.

Repacking all of that to a single pack file gives, somewhat surprisingly, a pack size of 62M (+ 1.3M index). In other words, the cost of getting all those branches, and all of the new stuff from Linus, turns out to be *negative* (probably due to some strange deltification coincidence).

I think that this shows that (at least in this case), having many branches isn't particularly wasteful (1.4M in this case with one incremental pack).

And that fewer packs beats many packs quite handily.

The big problem, however, comes when Jeff (or anyone else) decides to repack. Then, if you fetch both his repo and Linus', you might end up with several really big pack files, that mostly overlap. That could easily mean storing most objects many times, if you don't do some smart selective un/repacking when fetching.

/dan