Re: Repacking many disconnected blobs

On Wed, 14 Jun 2006, Keith Packard wrote:

Ok. That's actually _easily_ rectifiable, because it turns out that your behaviour is something that re-packing is actually really good at handling.

The thing is, "git repack" (the wrapper function) is all about finding all the heads of a repository, and then tellign the _real_ packing logic which objects to pack.

In other words, it literally boils down to basically

	git-rev-list --all --objects $rev_list |
		git-pack-objects --non-empty $pack_objects .tmp-pack

where "$rev_list" and "$pack_objects" are just extra flags to the two phases that you don't really care about.

But the important point to recognize is that the pack generation itself doesn't care about reachability or anything else AT ALL. The pack is just a jumble of objects, nothing more. Which is exactly what you want.

Absolutely. And it's even easy.

What you should do is to just generate a list of objects every once in a while, and pass that list off to "git-pack-objects", which will create a pack-file for you. Then you just move the generated pack-file (and index file) into the .git/objects/pack directory, and then you can run the normal "git-prune-packed", and you're done.

There's just two small subtle points to look out for:

 - You can list the objects with "most important first" order first, if 
   you can.  That will improve locality later (the packing will try to 
   generate the pack so that the order you gave the objects in will be a 
   rough order of the resul - the first objects will be together at the 
   beginning, the last objects will be at the end)

   This is not a huge deal. If you don't have a good order, give them in 
   any order, and then after you're done (and you do have branches and 
   tag-heads), the final repack (with a regular "git repack") will fix it 
   all up.

   You'll still get all of the size/access advantage of packfiles without 
   this, it just won't have the additional "nice IO patterns within the 
   packfile" behaviour (which mainly matters for the cold-cache case, so 
   you may well not care).

 - append the filename the object is associated with to the object name on 
   the list, if at all possible. This is what git-pack-objects will use as 
   part of the heuristic for finding the deltas, so this is actually a big 
   deal. If you forget (or mess up) the filename, packing will still 
   _work_ - it's just a heuristic, after all, and there are a few others 
   too - but the pack-file will have inferior delta chains.

   (The name doesn't have to be the "real name", it really only needs to 
   be something unique per *,v file, but real name is probably best)

   The corollary to this is that it's better to generate the pack-file 
   from a list of every version of a few files than it is to generate it 
   from a few versions of every file. Ie, if you process things one file 
   at a time, and create every object for that file, that is actually good 
   for packing, since there will be the optimal delta opportunity.

In other words, you should just feed git-pack-file a list of objects in the form "<sha1><space><filename>\n", and git-pack-file will do the rest.

Just as a stupid example, if you were to want to pack just the _tree_ that is the current version of a git archive, you'd do

	git-rev-list --objects HEAD^{tree} |
		git-pack-objects --non-empty .tmp-pack

which you can try on the current git tree just to see (the first line will generate a list of all objects reachable from the current _tree_: no history at all, the second line will create two files under the name of ".tmp-pack-<sha1-of-object-list>.{pack|idx}".

The reason I suggest doing this for the current tree of the git archive is simply that you can look at the git-rev-list output with "less", and see for yourself what it actually does (and there are just a few hundred objects there: a few tree objects, and the blob objects for every file in the current HEAD).

So the git pack-format is actually _optimal_ for your particular case, exactly because the pack-files don't actually care about any high-level semantics: all they contain is a list of objects.

So in phase 1, when you generate all the objects, the simplest thing to do is to literally just remember the last five thousand objects or so as you generate them, and when that array of objects fills up, you just start the "git-pack-objects" thing, and feed it the list of objects, move the pack-file into .git/objects/pack/pack-... and do a "git prune-packed".

Then you just continue.

So this should all fit the parsecvs approach very well indeed.

		Linus