threads / rfc / 4650

[RFC] git-fetch - repack in the background after fetching

Subject: [RFC] git-fetch - repack in the background after fetching

## tl;dr

7 messages between Jun 24, 2006 and Jun 25, 2006.

replies: 6people: 5as markdown or json

Martin Langhoff· Jun 24, 2006, 11:30 UTC · lore

Check whether we have a large set of unpacked objects and repack after the fetch, but don't for the user to wait for us. Conditional on core.autorepack =! no.

Having ' handle concurrent pruning of packed objects' (637cdd9d1d997fca34a1fc668fed1311e30fe95f) from Jeff King it should be safe to repack and prune in the background.

Signed-off-by: Martin Langhoff <martin@catalyst.net.nz>
---

This is a follow up to a similar patch earlier http://www.gelato.unsw.edu.au/archives/git/0605/21401.html -- is there interest in making GIT more friendly to users who don't know or care about packing and repacking their repos?

I loathe to do this conditionally only on the count of unpacked objects. If there's a quick'n'dirty way of asking portably whether the machine is busy or otherwise resource-constrained (ie: on battery) it should use it to avoid running repack at inconvenient times.

---
 git-fetch.sh |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)
diff --git a/git-fetch.sh b/git-fetch.sh
index 48818f8..7211318 100755
--- a/git-fetch.sh
+++ b/git-fetch.sh
@@ -427,3 +427,12 @@ case ",$update_head_ok,$orig_head," in
 	fi
 	;;
 esac
+
+if test "$(git-repo-config --get core.autorepack)" != 'no'
+then
+	if test $(git rev-list --unpacked --all | wc -l) -gt 1000
+	then
+		echo "Repacking in the background"
+		nice git repack -a -d -q &
+	fi
+fi

-- 
1.4.1.rc1.g59c8
Junio C Hamano· Jun 25, 2006, 03:12 UTC · re: Martin Langhoff · lore

Re: [RFC] git-fetch - repack in the background after fetching

Martin Langhoff <martin@catalyst.net.nz> writes:
> This is a follow up to a similar patch earlier
> http://www.gelato.unsw.edu.au/archives/git/0605/21401.html -- is there 
> interest in making GIT more friendly to users who don't know or care
> about packing and repacking their repos?

I would be a bit worried about the niced background repack racing against another instance of itself spawned by the same parent.

> I loathe to do this conditionally only on the count of unpacked
> objects. If there's a quick'n'dirty way of asking portably whether
> the machine is busy or otherwise resource-constrained (ie: on battery)
> it should use it to avoid running repack at inconvenient times.
count-objects might be lighter weight than rev-list --unpacked.

If you mean to make core.autorepack to be boolean, checking for string 'no' is not the right way.

	git repo-config --bool --get core.autorepack

But it does not matter if that variable is a string that is almost always true unless the value is "no".

Ryan Anderson· Jun 25, 2006, 10:10 UTC · re: Junio C Hamano · lore

[PATCH] Repack should try to prevent itself from running twice, concurrently.

Signed-off-by: Ryan Anderson <ryan@michonline.com>
---
 git-repack.sh |   11 +++++++++++
 1 files changed, 11 insertions(+), 0 deletions(-)
diff --git a/git-repack.sh b/git-repack.sh
index eb75c8c..20f9b55 100755
--- a/git-repack.sh
+++ b/git-repack.sh
@@ -24,6 +24,15 @@ do
 	shift
 done
 
+if [ -f $GIT_DIR/repack.lock ]
+then
+	echo "Existing repack job appears to be running."
+	echo "Remove $GIT_DIR/repack.lock if this is not the case."
+	exit 1
+else
+	echo $$ > $GIT_DIR/repack.lock
+fi
+
 rm -f .tmp-pack-*
 PACKDIR="$GIT_OBJECT_DIRECTORY/pack"
 
@@ -83,3 +92,5 @@ case "$no_update_info" in
 t) : ;;
 *) git-update-server-info ;;
 esac
+
+rm $GIT_DIR/repack.lock

-- 
1.4.1.rc1.gacb70
Johannes Schindelin· Jun 25, 2006, 10:17 UTC · re: Ryan Anderson · lore

Re: [PATCH] Repack should try to prevent itself from running twice, concurrently.

Hi,
On Sun, 25 Jun 2006, Ryan Anderson wrote:
Show 8 quoted lines
> +if [ -f $GIT_DIR/repack.lock ]
> +then
> +	echo "Existing repack job appears to be running."
> +	echo "Remove $GIT_DIR/repack.lock if this is not the case."
> +	exit 1
> +else
> +	echo $$ > $GIT_DIR/repack.lock
> +fi

It is not like it is being an atomic operation, but then, we are not going to call repack multiple times a second. I'd say it is sufficient.

Ciao, Dscho

Linus Torvalds· Jun 25, 2006, 03:53 UTC · re: Martin Langhoff · lore

Re: [RFC] git-fetch - repack in the background after fetching

On Sat, 24 Jun 2006, Martin Langhoff wrote:
>
> Check whether we have a large set of unpacked objects and repack
> after the fetch, but don't for the user to wait for us. Conditional
> on core.autorepack =! no.
I don't think this is safe.
It's also done stupidly.

Instead of askign how many unpacked objects we have with the (expensive) git-rev-list, why not just do

	ls "$GIT_DIR/objects/00" | wc -l
which is pretty much guaranteed to be faster and easier.

However, the more worrisome thing about background repacking is that while it should be safe against normal users, if you have two _repacks_ at the same time, they can decide to remove each others packs. Yeah, yeah, that's pretty damn unlikely, but hey, "pretty damn unlikely" is not "impossible".

Also, I think you'd want to repack with "-l", in case the thing is set up with an alternate object directory.

			Linus
Johannes Schindelin· Jun 25, 2006, 09:25 UTC · re: Linus Torvalds · lore

Re: [RFC] git-fetch - repack in the background after fetching

Hi,
On Sat, 24 Jun 2006, Linus Torvalds wrote:
> However, the more worrisome thing about background repacking is that while 
> it should be safe against normal users, if you have two _repacks_ at the 
> same time, they can decide to remove each others packs. Yeah, yeah, that's 
> pretty damn unlikely, but hey, "pretty damn unlikely" is not "impossible".
Why not introduce a lock file for repack?

Ciao, Dscho

Linus Torvalds· Jun 25, 2006, 17:29 UTC · re: Johannes Schindelin · lore

Re: [RFC] git-fetch - repack in the background after fetching

On Sun, 25 Jun 2006, Johannes Schindelin wrote:
Show 9 quoted lines
> 
> On Sat, 24 Jun 2006, Linus Torvalds wrote:
> 
> > However, the more worrisome thing about background repacking is that while 
> > it should be safe against normal users, if you have two _repacks_ at the 
> > same time, they can decide to remove each others packs. Yeah, yeah, that's 
> > pretty damn unlikely, but hey, "pretty damn unlikely" is not "impossible".
> 
> Why not introduce a lock file for repack?

You can do that. The problem is, lock-files are really hard to do right, and portably. Especially from scripts.

But _I_ think the basic issue is that it's wrong to even try to do this background repack.

Git does explicit repacking. That's just how it is. If the worry is that people forget to pack often enough, why not just have the "git pull" script _tell_ the user, something like

	if [lots of unpacked objects]; then
		echo "You've got a boatload of unpacked objects now."
		echo "Maybe you'd like to repack using"
		echo "   git repack -a -d"
		echo "Thank you for not smoking"
	fi >&2
which is educational on so many levels.
		Linus

← back to recent threads