# [PATCH] git-repack: create new packs inside $PACKDIR, not cwd 8 messages from 2006-09-04 to 2006-09-04. Participants: Martin Langhoff, Martin Waitz, Junio C Hamano, Martin Langhoff (CatalystIT). Thread: https://gitlist.dev/t/5469 ## Martin Langhoff, 2006-09-04 05:42 Subject: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd Message-ID: <11573485523752-git-send-email-martin@catalyst.net.nz> URL: https://gitlist.dev/e/11573485523752-git-send-email-martin%40catalyst.net.nz ``` Avoid failing when cwd is !writable by writing the packfiles directly in the $PACKDIR. Without this, git-repack was failing when run from crontab by non-root user accounts. For large repositories, this also makes the mv operation a lot cheaper, and avoids leaving temp packfiles around the fs upon failure. Signed-off-by: Martin Langhoff --- git-repack.sh | 9 +++++---- 1 files changed, 5 insertions(+), 4 deletions(-) diff --git a/git-repack.sh b/git-repack.sh index 584a732..ccc8e43 100755 --- a/git-repack.sh +++ b/git-repack.sh @@ -42,11 +42,13 @@ case ",$all_into_one," in find . -type f \( -name '*.pack' -o -name '*.idx' \) -print` ;; esac + +mkdir -p "$PACKDIR" || exit pack_objects="$pack_objects $local $quiet $no_reuse_delta$extra" name=$( { git-rev-list --objects --all $rev_list || echo "git-rev-list died with exit code $?" } | - git-pack-objects --non-empty $pack_objects .tmp-pack) || + git-pack-objects --non-empty $pack_objects "$PACKDIR/.tmp-pack") || exit 1 if [ -z "$name" ]; then echo Nothing new to pack. @@ -54,7 +56,6 @@ else if test "$quiet" != '-q'; then echo "Pack pack-$name created." fi - mkdir -p "$PACKDIR" || exit for sfx in pack idx do @@ -64,8 +65,8 @@ else "$PACKDIR/old-pack-$name.$sfx" fi done && - mv -f .tmp-pack-$name.pack "$PACKDIR/pack-$name.pack" && - mv -f .tmp-pack-$name.idx "$PACKDIR/pack-$name.idx" && + mv -f "$PACKDIR/.tmp-pack-$name.pack" "$PACKDIR/pack-$name.pack" && + mv -f "$PACKDIR/.tmp-pack-$name.idx" "$PACKDIR/pack-$name.idx" && test -f "$PACKDIR/pack-$name.pack" && test -f "$PACKDIR/pack-$name.idx" || { echo >&2 "Couldn't replace the existing pack with updated one." -- 1.4.2.gdfe7 -- VGER BF report: S 1 ``` ## Martin Waitz, 2006-09-04 09:08 Subject: Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd Message-ID: <20060904090833.GF17042@admingilde.org> URL: https://gitlist.dev/e/20060904090833.GF17042%40admingilde.org In-Reply-To: <11573485523752-git-send-email-martin@catalyst.net.nz> ``` hoi :) On Mon, Sep 04, 2006 at 05:42:32PM +1200, Martin Langhoff wrote: > Avoid failing when cwd is !writable by writing the > packfiles directly in the $PACKDIR. what if other GIT commands are being called while you repack? Wouldn't they try to use the new packs, even while they are not completely written? Perhaps it is better to create a new subdirectory $PACKDIR/.tmp/ and create the new pack files there? -- Martin Waitz ``` ## Junio C Hamano, 2006-09-04 09:36 Subject: Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd Message-ID: <7vveo4nfbg.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vveo4nfbg.fsf%40assigned-by-dhcp.cox.net In-Reply-To: <20060904090833.GF17042@admingilde.org> ``` Martin Waitz writes: > hoi :) > > On Mon, Sep 04, 2006 at 05:42:32PM +1200, Martin Langhoff wrote: >> Avoid failing when cwd is !writable by writing the >> packfiles directly in the $PACKDIR. > > what if other GIT commands are being called while you repack? > Wouldn't they try to use the new packs, even while they are not > completely written? Given that a new pack is created first by writing out .pack and then .idx, and that the using side ignores .pack without corresponding .idx, we are talking about very small window, but you are right. Writing into $cwd was certainly a carelessness; we tend to use $GIT_DIR/ for this kind of thing. -- VGER BF report: U 0.926108 ``` ## Junio C Hamano, 2006-09-04 09:50 Subject: Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd Message-ID: <7vr6ysneor.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vr6ysneor.fsf%40assigned-by-dhcp.cox.net In-Reply-To: <7vveo4nfbg.fsf@assigned-by-dhcp.cox.net> ``` Junio C Hamano writes: > Writing into $cwd was certainly a carelessness; we tend to use > $GIT_DIR/ for this kind of thing. In other words... -- >8 -- From: Martin Langhoff Date: Mon, 4 Sep 2006 17:42:32 +1200 Subject: [PATCH] git-repack: create new packs inside $GIT_DIR, not cwd Avoid failing when cwd is !writable by writing the packfiles in $GIT_DIR, which is more in line with other commands. Without this, git-repack was failing when run from crontab by non-root user accounts. For large repositories, this also makes the mv operation a lot cheaper, and avoids leaving temp packfiles around the fs upon failure. Signed-off-by: Martin Langhoff Signed-off-by: Junio C Hamano --- git-repack.sh | 11 +++++++---- 1 files changed, 7 insertions(+), 4 deletions(-) diff --git a/git-repack.sh b/git-repack.sh index 584a732..b525fc5 100755 --- a/git-repack.sh +++ b/git-repack.sh @@ -24,8 +24,10 @@ do shift done -rm -f .tmp-pack-* PACKDIR="$GIT_OBJECT_DIRECTORY/pack" +PACKTMP="$GIT_DIR/.tmp-$$-pack" +rm -f "$PACKTMP"-* +trap 'rm -f "$PACKTMP"-*' 0 1 2 3 15 # There will be more repacking strategies to come... case ",$all_into_one," in @@ -42,11 +44,12 @@ case ",$all_into_one," in find . -type f \( -name '*.pack' -o -name '*.idx' \) -print` ;; esac + pack_objects="$pack_objects $local $quiet $no_reuse_delta$extra" name=$( { git-rev-list --objects --all $rev_list || echo "git-rev-list died with exit code $?" } | - git-pack-objects --non-empty $pack_objects .tmp-pack) || + git-pack-objects --non-empty $pack_objects "$PACKTMP") || exit 1 if [ -z "$name" ]; then echo Nothing new to pack. @@ -64,8 +67,8 @@ else "$PACKDIR/old-pack-$name.$sfx" fi done && - mv -f .tmp-pack-$name.pack "$PACKDIR/pack-$name.pack" && - mv -f .tmp-pack-$name.idx "$PACKDIR/pack-$name.idx" && + mv -f "$PACKTMP-$name.pack" "$PACKDIR/pack-$name.pack" && + mv -f "$PACKTMP-$name.idx" "$PACKDIR/pack-$name.idx" && test -f "$PACKDIR/pack-$name.pack" && test -f "$PACKDIR/pack-$name.idx" || { echo >&2 "Couldn't replace the existing pack with updated one." -- 1.4.2.g99d7d -- VGER BF report: U 0.870206 ``` ## Martin Langhoff (CatalystIT), 2006-09-04 10:03 Subject: Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd Message-ID: <44FBF9E0.9050800@catalyst.net.nz> URL: https://gitlist.dev/e/44FBF9E0.9050800%40catalyst.net.nz In-Reply-To: <7vr6ysneor.fsf@assigned-by-dhcp.cox.net> ``` Junio C Hamano wrote: > In other words... Can't be offline 2 hs to read a book... ;-) Actually, I had thought the pack reading code would focus on filenames following pack-.pack pattern and corresponding idx files, and that .tmp-* was safe to have there. My bad. BTW, I think there's a small error. ... > --- a/git-repack.sh > +++ b/git-repack.sh > @@ -24,8 +24,10 @@ do > shift > done > > -rm -f .tmp-pack-* > PACKDIR="$GIT_OBJECT_DIRECTORY/pack" > +PACKTMP="$GIT_DIR/.tmp-$$-pack" > +rm -f "$PACKTMP"-* > +trap 'rm -f "$PACKTMP"-*' 0 1 2 3 15 Your packtmp includes $$ which means that rm -f "$PACKTMP" will only clear out old packs only if the pid of the old-and-probably-dead process matches ours... and then a hyphen. so instead I propose... +trap 'rm -f "$GIT_DIR/.tmp-*-pack"' 0 1 2 3 15 cheers, martin -- ----------------------------------------------------------------------- Martin @ Catalyst .Net .NZ Ltd, PO Box 11-053, Manners St, Wellington WEB: http://catalyst.net.nz/ PHYS: Level 2, 150-154 Willis St OFFICE: +64(4)916-7224 MOB: +64(21)364-017 Make things as simple as possible, but no simpler - Einstein ----------------------------------------------------------------------- -- VGER BF report: U 0.900798 ``` ## Junio C Hamano, 2006-09-04 10:13 Subject: Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd Message-ID: <7vlkp0ndmj.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vlkp0ndmj.fsf%40assigned-by-dhcp.cox.net In-Reply-To: <44FBF9E0.9050800@catalyst.net.nz> ``` "Martin Langhoff (CatalystIT)" writes: > BTW, I think there's a small error. > > Your packtmp includes $$ which means that rm -f "$PACKTMP" will only > clear out old packs.. That was deliberate. I hate programs that clean things up behind user's back. The first "rm" is to get rid of what would collide with what we are going to do (i.e. protecting ourselves) and "trap rm" is to make sure we do not leave the cruft we know we are going to create. I'd rather leave other people's cruft around, unless the purpose of the command is to clean things up, and repack is hardly that. -- VGER BF report: H 0.149712 ``` ## Martin Langhoff (CatalystIT), 2006-09-04 10:46 Subject: Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd Message-ID: <44FC041F.6010002@catalyst.net.nz> URL: https://gitlist.dev/e/44FC041F.6010002%40catalyst.net.nz In-Reply-To: <7vlkp0ndmj.fsf@assigned-by-dhcp.cox.net> ``` Junio C Hamano wrote: > "Martin Langhoff (CatalystIT)" writes: > > >>BTW, I think there's a small error. >> >>Your packtmp includes $$ which means that rm -f "$PACKTMP" will only >>clear out old packs.. > > > That was deliberate. I hate programs that clean things up > behind user's back. The first "rm" is to get rid of what would > collide with what we are going to do (i.e. protecting ourselves) > and "trap rm" is to make sure we do not leave the cruft we know > we are going to create. I'd rather leave other people's cruft > around, unless the purpose of the command is to clean things up, > and repack is hardly that. > Ah, ok. I misunderstood the use of trap -- of course, re-reading the man pages, it makes sense. A-ok with me, then, and sorry about the noise. martin -- ----------------------------------------------------------------------- Martin @ Catalyst .Net .NZ Ltd, PO Box 11-053, Manners St, Wellington WEB: http://catalyst.net.nz/ PHYS: Level 2, 150-154 Willis St OFFICE: +64(4)916-7224 MOB: +64(21)364-017 Make things as simple as possible, but no simpler - Einstein ----------------------------------------------------------------------- -- VGER BF report: H 0.0618878 ``` ## Junio C Hamano, 2006-09-04 20:16 Subject: Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd Message-ID: <7vslj7mlp9.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vslj7mlp9.fsf%40assigned-by-dhcp.cox.net In-Reply-To: <44FC041F.6010002@catalyst.net.nz> ``` "Martin Langhoff (CatalystIT)" writes: > Ah, ok. I misunderstood the use of trap -- of course, re-reading the > man pages, it makes sense. However the more I think about it your original idea of using .tmp-pack directly in .git/objects/pack/ *should* have worked (modulo two repack instances using the same name, which is fixed by $$ there). Not insisting on "pack-" prefix I consider is a feature, but I do not see a reason to take a file that begin with a dot. Well, probably too late to deprecate, maybe not. I dunno. ```