threads / patch / 5469

patchgit-repack: create new packs inside $PACKDIR, not cwd

Subject: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd

## tl;dr

8 messages between Sep 4, 2006 and Sep 4, 2006. Diffs are folded; open one to read it.

replies: 7people: 3as markdown or json

Martin Langhoff· Sep 4, 2006, 05:42 UTC · lore

Avoid failing when cwd is !writable by writing the packfiles directly in the $PACKDIR.

Without this, git-repack was failing when run from crontab by non-root user accounts. For large repositories, this also makes the mv operation a lot cheaper, and avoids leaving temp packfiles around the fs upon failure.

Signed-off-by: Martin Langhoff <martin@catalyst.net.nz>
---
 git-repack.sh |    9 +++++----
 1 files changed, 5 insertions(+), 4 deletions(-)

Show changes to git-repack.sh +5 −4

diff --git a/git-repack.sh b/git-repack.sh
index 584a732..ccc8e43 100755
--- a/git-repack.sh
+++ b/git-repack.sh
@@ -42,11 +42,13 @@ case ",$all_into_one," in
 	    find . -type f \( -name '*.pack' -o -name '*.idx' \) -print`
 	;;
 esac
+
+mkdir -p "$PACKDIR" || exit
 pack_objects="$pack_objects $local $quiet $no_reuse_delta$extra"
 name=$( { git-rev-list --objects --all $rev_list ||
 	  echo "git-rev-list died with exit code $?"
 	} |
-	git-pack-objects --non-empty $pack_objects .tmp-pack) ||
+	git-pack-objects --non-empty $pack_objects "$PACKDIR/.tmp-pack") ||
 	exit 1
 if [ -z "$name" ]; then
 	echo Nothing new to pack.
@@ -54,7 +56,6 @@ else
 	if test "$quiet" != '-q'; then
 	    echo "Pack pack-$name created."
 	fi
-	mkdir -p "$PACKDIR" || exit
 
 	for sfx in pack idx
 	do
@@ -64,8 +65,8 @@ else
 				"$PACKDIR/old-pack-$name.$sfx"
 		fi
 	done &&
-	mv -f .tmp-pack-$name.pack "$PACKDIR/pack-$name.pack" &&
-	mv -f .tmp-pack-$name.idx  "$PACKDIR/pack-$name.idx" &&
+	mv -f "$PACKDIR/.tmp-pack-$name.pack" "$PACKDIR/pack-$name.pack" &&
+	mv -f "$PACKDIR/.tmp-pack-$name.idx"  "$PACKDIR/pack-$name.idx" &&
 	test -f "$PACKDIR/pack-$name.pack" &&
 	test -f "$PACKDIR/pack-$name.idx" || {
 		echo >&2 "Couldn't replace the existing pack with updated one."

-- 
1.4.2.gdfe7


-- 
VGER BF report: S 1

Martin Waitz· Sep 4, 2006, 09:08 UTC · re: Martin Langhoff · lore

Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd

hoi :)

On Mon, Sep 04, 2006 at 05:42:32PM +1200, Martin Langhoff wrote:

> Avoid failing when cwd is !writable by writing the
> packfiles directly in the $PACKDIR.

what if other GIT commands are being called while you repack? Wouldn't they try to use the new packs, even while they are not completely written?

Perhaps it is better to create a new subdirectory $PACKDIR/.tmp/ and create the new pack files there?

-- 
Martin Waitz

Junio C Hamano· Sep 4, 2006, 09:36 UTC · re: Martin Waitz · lore

Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd

Martin Waitz <tali@admingilde.org> writes:

Show 9 quoted lines

> hoi :)
>
> On Mon, Sep 04, 2006 at 05:42:32PM +1200, Martin Langhoff wrote:
>> Avoid failing when cwd is !writable by writing the
>> packfiles directly in the $PACKDIR.
>
> what if other GIT commands are being called while you repack?
> Wouldn't they try to use the new packs, even while they are not
> completely written?

Given that a new pack is created first by writing out .pack and then .idx, and that the using side ignores .pack without corresponding .idx, we are talking about very small window, but you are right.

Writing into $cwd was certainly a carelessness; we tend to use $GIT_DIR/ for this kind of thing.

-- 
VGER BF report: U 0.926108

Junio C Hamano· Sep 4, 2006, 09:50 UTC · re: Junio C Hamano · lore

Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd

Junio C Hamano <junkio@cox.net> writes:

> Writing into $cwd was certainly a carelessness; we tend to use
> $GIT_DIR/ for this kind of thing.

In other words...

-- >8 --
From: Martin Langhoff <martin@catalyst.net.nz>
Date: Mon, 4 Sep 2006 17:42:32 +1200
Subject: [PATCH] git-repack: create new packs inside $GIT_DIR, not cwd

Avoid failing when cwd is !writable by writing the packfiles in $GIT_DIR, which is more in line with other commands.

Signed-off-by: Martin Langhoff <martin@catalyst.net.nz>
Signed-off-by: Junio C Hamano <junkio@cox.net>
---
 git-repack.sh |   11 +++++++----
 1 files changed, 7 insertions(+), 4 deletions(-)

Show changes to git-repack.sh +7 −4

diff --git a/git-repack.sh b/git-repack.sh
index 584a732..b525fc5 100755
--- a/git-repack.sh
+++ b/git-repack.sh
@@ -24,8 +24,10 @@ do
 	shift
 done
 
-rm -f .tmp-pack-*
 PACKDIR="$GIT_OBJECT_DIRECTORY/pack"
+PACKTMP="$GIT_DIR/.tmp-$$-pack"
+rm -f "$PACKTMP"-*
+trap 'rm -f "$PACKTMP"-*' 0 1 2 3 15
 
 # There will be more repacking strategies to come...
 case ",$all_into_one," in
@@ -42,11 +44,12 @@ case ",$all_into_one," in
 	    find . -type f \( -name '*.pack' -o -name '*.idx' \) -print`
 	;;
 esac
+
 pack_objects="$pack_objects $local $quiet $no_reuse_delta$extra"
 name=$( { git-rev-list --objects --all $rev_list ||
 	  echo "git-rev-list died with exit code $?"
 	} |
-	git-pack-objects --non-empty $pack_objects .tmp-pack) ||
+	git-pack-objects --non-empty $pack_objects "$PACKTMP") ||
 	exit 1
 if [ -z "$name" ]; then
 	echo Nothing new to pack.
@@ -64,8 +67,8 @@ else
 				"$PACKDIR/old-pack-$name.$sfx"
 		fi
 	done &&
-	mv -f .tmp-pack-$name.pack "$PACKDIR/pack-$name.pack" &&
-	mv -f .tmp-pack-$name.idx  "$PACKDIR/pack-$name.idx" &&
+	mv -f "$PACKTMP-$name.pack" "$PACKDIR/pack-$name.pack" &&
+	mv -f "$PACKTMP-$name.idx"  "$PACKDIR/pack-$name.idx" &&
 	test -f "$PACKDIR/pack-$name.pack" &&
 	test -f "$PACKDIR/pack-$name.idx" || {
 		echo >&2 "Couldn't replace the existing pack with updated one."

-- 
1.4.2.g99d7d



-- 
VGER BF report: U 0.870206

Martin Langhoff (CatalystIT)· Sep 4, 2006, 10:03 UTC · re: Junio C Hamano · lore

Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd

Junio C Hamano wrote:

> In other words...

Can't be offline 2 hs to read a book... ;-) Actually, I had thought the pack reading code would focus on filenames following pack-<id>.pack pattern and corresponding idx files, and that .tmp-* was safe to have there. My bad.

BTW, I think there's a small error.

...

Show 11 quoted lines

> --- a/git-repack.sh
> +++ b/git-repack.sh
> @@ -24,8 +24,10 @@ do
>  	shift
>  done
>  
> -rm -f .tmp-pack-*
>  PACKDIR="$GIT_OBJECT_DIRECTORY/pack"
> +PACKTMP="$GIT_DIR/.tmp-$$-pack"
> +rm -f "$PACKTMP"-*
> +trap 'rm -f "$PACKTMP"-*' 0 1 2 3 15

Your packtmp includes $$ which means that rm -f "$PACKTMP" will only clear out old packs only if the pid of the old-and-probably-dead process matches ours... and then a hyphen.

so instead I propose...

+trap 'rm -f "$GIT_DIR/.tmp-*-pack"' 0 1 2 3 15

cheers,

martin

-- 
-----------------------------------------------------------------------
Martin @ Catalyst .Net .NZ  Ltd, PO Box 11-053, Manners St,  Wellington
WEB: http://catalyst.net.nz/           PHYS: Level 2, 150-154 Willis St
OFFICE: +64(4)916-7224                              MOB: +64(21)364-017
       Make things as simple as possible, but no simpler - Einstein
-----------------------------------------------------------------------

-- 
VGER BF report: U 0.900798

Junio C Hamano· Sep 4, 2006, 10:13 UTC · re: Martin Langhoff (CatalystIT) · lore

Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd

"Martin Langhoff (CatalystIT)" <martin@catalyst.net.nz> writes:

> BTW, I think there's a small error.
>
> Your packtmp includes $$ which means that rm -f "$PACKTMP" will only
> clear out old packs..

That was deliberate. I hate programs that clean things up behind user's back. The first "rm" is to get rid of what would collide with what we are going to do (i.e. protecting ourselves) and "trap rm" is to make sure we do not leave the cruft we know we are going to create. I'd rather leave other people's cruft around, unless the purpose of the command is to clean things up, and repack is hardly that.

-- 
VGER BF report: H 0.149712

Martin Langhoff (CatalystIT)· Sep 4, 2006, 10:46 UTC · re: Junio C Hamano · lore

Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd

Junio C Hamano wrote:

Show 17 quoted lines

> "Martin Langhoff (CatalystIT)" <martin@catalyst.net.nz> writes:
> 
> 
>>BTW, I think there's a small error.
>>
>>Your packtmp includes $$ which means that rm -f "$PACKTMP" will only
>>clear out old packs..
> 
> 
> That was deliberate.  I hate programs that clean things up
> behind user's back.  The first "rm" is to get rid of what would
> collide with what we are going to do (i.e. protecting ourselves)
> and "trap rm" is to make sure we do not leave the cruft we know
> we are going to create.  I'd rather leave other people's cruft
> around, unless the purpose of the command is to clean things up,
> and repack is hardly that.
>

Ah, ok. I misunderstood the use of trap -- of course, re-reading the man pages, it makes sense.

A-ok with me, then, and sorry about the noise.

martin

-- 
-----------------------------------------------------------------------
Martin @ Catalyst .Net .NZ  Ltd, PO Box 11-053, Manners St,  Wellington
WEB: http://catalyst.net.nz/           PHYS: Level 2, 150-154 Willis St
OFFICE: +64(4)916-7224                              MOB: +64(21)364-017
       Make things as simple as possible, but no simpler - Einstein
-----------------------------------------------------------------------

-- 
VGER BF report: H 0.0618878

Junio C Hamano· Sep 4, 2006, 20:16 UTC · re: Martin Langhoff (CatalystIT) · lore

Re: [PATCH] git-repack: create new packs inside $PACKDIR, not cwd

"Martin Langhoff (CatalystIT)" <martin@catalyst.net.nz> writes:

> Ah, ok. I misunderstood the use of trap -- of course, re-reading the
> man pages, it makes sense.

However the more I think about it your original idea of using .tmp-pack directly in .git/objects/pack/ *should* have worked (modulo two repack instances using the same name, which is fixed by $$ there). Not insisting on "pack-" prefix I consider is a feature, but I do not see a reason to take a file that begin with a dot.

Well, probably too late to deprecate, maybe not.  I dunno.