# [RFC] git-fetch - repack in the background after fetching 7 messages from 2006-06-24 to 2006-06-25. Participants: Martin Langhoff, Junio C Hamano, Linus Torvalds, Johannes Schindelin, Ryan Anderson. Thread: https://gitlist.dev/t/4650 ## Martin Langhoff, 2006-06-24 11:30 Subject: [RFC] git-fetch - repack in the background after fetching Message-ID: <11511486003924-git-send-email-martin@catalyst.net.nz> URL: https://gitlist.dev/e/11511486003924-git-send-email-martin%40catalyst.net.nz ``` Check whether we have a large set of unpacked objects and repack after the fetch, but don't for the user to wait for us. Conditional on core.autorepack =! no. Having ' handle concurrent pruning of packed objects' (637cdd9d1d997fca34a1fc668fed1311e30fe95f) from Jeff King it should be safe to repack and prune in the background. Signed-off-by: Martin Langhoff --- This is a follow up to a similar patch earlier http://www.gelato.unsw.edu.au/archives/git/0605/21401.html -- is there interest in making GIT more friendly to users who don't know or care about packing and repacking their repos? I loathe to do this conditionally only on the count of unpacked objects. If there's a quick'n'dirty way of asking portably whether the machine is busy or otherwise resource-constrained (ie: on battery) it should use it to avoid running repack at inconvenient times. --- git-fetch.sh | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-) diff --git a/git-fetch.sh b/git-fetch.sh index 48818f8..7211318 100755 --- a/git-fetch.sh +++ b/git-fetch.sh @@ -427,3 +427,12 @@ case ",$update_head_ok,$orig_head," in fi ;; esac + +if test "$(git-repo-config --get core.autorepack)" != 'no' +then + if test $(git rev-list --unpacked --all | wc -l) -gt 1000 + then + echo "Repacking in the background" + nice git repack -a -d -q & + fi +fi -- 1.4.1.rc1.g59c8 ``` ## Junio C Hamano, 2006-06-25 03:12 Subject: Re: [RFC] git-fetch - repack in the background after fetching Message-ID: <7vy7vmormn.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vy7vmormn.fsf%40assigned-by-dhcp.cox.net In-Reply-To: <11511486003924-git-send-email-martin@catalyst.net.nz> ``` Martin Langhoff writes: > This is a follow up to a similar patch earlier > http://www.gelato.unsw.edu.au/archives/git/0605/21401.html -- is there > interest in making GIT more friendly to users who don't know or care > about packing and repacking their repos? I would be a bit worried about the niced background repack racing against another instance of itself spawned by the same parent. > I loathe to do this conditionally only on the count of unpacked > objects. If there's a quick'n'dirty way of asking portably whether > the machine is busy or otherwise resource-constrained (ie: on battery) > it should use it to avoid running repack at inconvenient times. count-objects might be lighter weight than rev-list --unpacked. If you mean to make core.autorepack to be boolean, checking for string 'no' is not the right way. git repo-config --bool --get core.autorepack But it does not matter if that variable is a string that is almost always true unless the value is "no". ``` ## Linus Torvalds, 2006-06-25 03:53 Subject: Re: [RFC] git-fetch - repack in the background after fetching Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0606242049500.3747%40g5.osdl.org In-Reply-To: <11511486003924-git-send-email-martin@catalyst.net.nz> ``` On Sat, 24 Jun 2006, Martin Langhoff wrote: > > Check whether we have a large set of unpacked objects and repack > after the fetch, but don't for the user to wait for us. Conditional > on core.autorepack =! no. I don't think this is safe. It's also done stupidly. Instead of askign how many unpacked objects we have with the (expensive) git-rev-list, why not just do ls "$GIT_DIR/objects/00" | wc -l which is pretty much guaranteed to be faster and easier. However, the more worrisome thing about background repacking is that while it should be safe against normal users, if you have two _repacks_ at the same time, they can decide to remove each others packs. Yeah, yeah, that's pretty damn unlikely, but hey, "pretty damn unlikely" is not "impossible". Also, I think you'd want to repack with "-l", in case the thing is set up with an alternate object directory. Linus ``` ## Johannes Schindelin, 2006-06-25 09:25 Subject: Re: [RFC] git-fetch - repack in the background after fetching Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.63.0606251122260.29667%40wbgn013.biozentrum.uni-wuerzburg.de In-Reply-To: ``` Hi, On Sat, 24 Jun 2006, Linus Torvalds wrote: > However, the more worrisome thing about background repacking is that while > it should be safe against normal users, if you have two _repacks_ at the > same time, they can decide to remove each others packs. Yeah, yeah, that's > pretty damn unlikely, but hey, "pretty damn unlikely" is not "impossible". Why not introduce a lock file for repack? Ciao, Dscho ``` ## Ryan Anderson, 2006-06-25 10:10 Subject: [PATCH] Repack should try to prevent itself from running twice, concurrently. Message-ID: <11512302144123-git-send-email-ryan@michonline.com> URL: https://gitlist.dev/e/11512302144123-git-send-email-ryan%40michonline.com In-Reply-To: <7vy7vmormn.fsf@assigned-by-dhcp.cox.net> ``` Signed-off-by: Ryan Anderson --- git-repack.sh | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-) diff --git a/git-repack.sh b/git-repack.sh index eb75c8c..20f9b55 100755 --- a/git-repack.sh +++ b/git-repack.sh @@ -24,6 +24,15 @@ do shift done +if [ -f $GIT_DIR/repack.lock ] +then + echo "Existing repack job appears to be running." + echo "Remove $GIT_DIR/repack.lock if this is not the case." + exit 1 +else + echo $$ > $GIT_DIR/repack.lock +fi + rm -f .tmp-pack-* PACKDIR="$GIT_OBJECT_DIRECTORY/pack" @@ -83,3 +92,5 @@ case "$no_update_info" in t) : ;; *) git-update-server-info ;; esac + +rm $GIT_DIR/repack.lock -- 1.4.1.rc1.gacb70 ``` ## Johannes Schindelin, 2006-06-25 10:17 Subject: Re: [PATCH] Repack should try to prevent itself from running twice, concurrently. Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.63.0606251216250.29667%40wbgn013.biozentrum.uni-wuerzburg.de In-Reply-To: <11512302144123-git-send-email-ryan@michonline.com> ``` Hi, On Sun, 25 Jun 2006, Ryan Anderson wrote: > +if [ -f $GIT_DIR/repack.lock ] > +then > + echo "Existing repack job appears to be running." > + echo "Remove $GIT_DIR/repack.lock if this is not the case." > + exit 1 > +else > + echo $$ > $GIT_DIR/repack.lock > +fi It is not like it is being an atomic operation, but then, we are not going to call repack multiple times a second. I'd say it is sufficient. Ciao, Dscho ``` ## Linus Torvalds, 2006-06-25 17:29 Subject: Re: [RFC] git-fetch - repack in the background after fetching Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0606251025100.3747%40g5.osdl.org In-Reply-To: ``` On Sun, 25 Jun 2006, Johannes Schindelin wrote: > > On Sat, 24 Jun 2006, Linus Torvalds wrote: > > > However, the more worrisome thing about background repacking is that while > > it should be safe against normal users, if you have two _repacks_ at the > > same time, they can decide to remove each others packs. Yeah, yeah, that's > > pretty damn unlikely, but hey, "pretty damn unlikely" is not "impossible". > > Why not introduce a lock file for repack? You can do that. The problem is, lock-files are really hard to do right, and portably. Especially from scripts. But _I_ think the basic issue is that it's wrong to even try to do this background repack. Git does explicit repacking. That's just how it is. If the worry is that people forget to pack often enough, why not just have the "git pull" script _tell_ the user, something like if [lots of unpacked objects]; then echo "You've got a boatload of unpacked objects now." echo "Maybe you'd like to repack using" echo " git repack -a -d" echo "Thank you for not smoking" fi >&2 which is educational on so many levels. Linus ```