Re: fetching packs and storing them as packs

From: Junio C Hamano <junkio@cox.net>
Date: Oct 27, 2006, 06:57 UTC
Message-ID: <7viri6i6uu.fsf@assigned-by-dhcp.cox.net>
In-Reply-To: <BAYC1-PASMTP10C050A5FAA4C70AD57679AE040@CEZ.ICE>

Sean <seanlkml@sympatico.ca> writes:

Show 11 quoted lines

> On Thu, 26 Oct 2006 21:42:29 -0400
> Shawn Pearce <spearce@spearce.org> wrote:
>
>> This is an issue for "central" repositories that people push into
>> and which might be getting repacked according to a cronjob.
>> 
>> Unfortunately I don't have a solution.  I tried to come up with
>> one but didn't.  :-)
>
> What about creating a temporary ref before pushing, and then removing
> it only after the HEAD has been updated?

That won't work. If repack is faster than index-pack, repack would fail to find necessary objects, barf, and would not remove the existing or new pack, and then index-pack would eventually succeed and when it does at least your repository is complete even though it may still have redundant objects in packs.

So in that sense, it is not a disaster, so it might be a good enough solution.

I'd almost say "heavy repository-wide operations like 'repack -a -d' and 'prune' should operate under a single repository lock", but historically we've avoided locks and instead tried to do things optimistically and used compare-and-swap to detect conflicts, so maybe that avenue might be worth pursuing.

How about (I'm thinking aloud and I'm sure there will be holes -- I won't think about prune for now)...

* "repack -a -d":

 (1) initially run show-ref (or "ls-remote .") and store the
     result in .git/$ref_pack_lock_file;

 (2) enumerate existing packs;

 (3) do the usual "rev-list --all | pack-objects" thing; this
     may end up including more objects than what are reachable
     from the result of (1) if somebody else updates refs in the
     meantime;

 (4) enumerate existing packs; if there is difference from (2)
     other than what (3) created, that means somebody else added
     a pack in the meantime; stop and do not do the "-d" part;

 (5) run "ls-remote ." again and compare it with what it got in
     (1); if different, somebody else updated a ref in the
     meantime; stop and do not do the "-d" part;

 (6) do the "-d" part as usual by removing packs we saw in (2)
     but do not remove the pack we created in (3);

 (7) remove .git/$ref_pack_lock_file.

* "fetch --thin" and "index-pack --stdin":

 (1) check the .git/$ref_pack_lock_file, and refuse to operate
    if there is such (this is not strictly needed for
    correctness but only to give an early exit);

 (2) create a new pack under a temporary name, and when
     complete, make the pack/index pair .pack and .idx;

 (3) update the refs.