Re: fetching packs and storing them as packs

From: Linus Torvalds <torvalds@osdl.org>
Date: Oct 28, 2006, 04:18 UTC
Message-ID: <Pine.LNX.4.64.0610272109500.3849@g5.osdl.org>
In-Reply-To: <20061028034206.GA14044@spearce.org>

On Fri, 27 Oct 2006, Shawn Pearce wrote:

Show 7 quoted lines

> 
> So a reader-writer lock is preferred over
> a non-locking solution such as I posted in
> http://article.gmane.org/gmane.comp.version-control.git/30288 ?
> 
> Not to mention that such a solution would also fix the -d issue
> Linus points out above.

Be very careful.

There's a good reason why git doesn't use locking, and tends to use the "create file exclusively and move over the old version after having tested that the old version is still relevant" approach.

Two _major_ issues:

 - just about any other locking algorithm simply doesn't work on some 
   filesystems. And then you're just royally screwed.

 - I want to be able to push out, regardless of whether there is somebody 
   (or millions of somebodies) reading the repository at the same time. So 
   locking is not acceptable for "normal operations" at all - at most this 
   would be a "keep a repack from interfering with another repack" kind of 
   thing.

I would MUCH rather we just rename the index/pack file to something that git can _use_, but that "git repack -a -d" won't remove. In other words, rather than locking, it would be much better to just use a naming rule: when we download a new pack, the new pack will be called

	new-pack-<SHA1ofobjectlist>.pack
	new-pack-<SHA1ofobjectlist>.idx

and we just make the rule that "git repack -a -d" will only ever touch packs that are called just "pack-*.{pack|idx}", and never anything else.

It really is that simple. Allow normal git object opens to open the "temporary file" naming version too (so that you can install the refs before the rename, and all the objects will be visible), but don't allow "git repack" to remove packs that are in the process of being installed.

Race removed, and no locking really needed. At most, we might need to be able to match up a "new-pack-*.idx" file with a "pack-*.pack" file when we open pack-files, simply because we can't rename two files atomically, so the pack-file and index file would potentially exist with "different" names for a short window.

That kind of small semantic changes are _way_ better than introducing locking, which will inevitably have much worse error cases (not working, stale locks, inability to push because something is really slow, or any number of other problems).