threads / patch / 11475

patchgit-am: Run git gc only once and not for every patch.

Subject: [PATCH] git-am: Run git gc only once and not for every patch.

## tl;dr

9 messages between Jan 4, 2008 and Jan 5, 2008. Diffs are folded; open one to read it.

replies: 8people: 3as markdown or json

Michael Stefaniuc· Jan 4, 2008, 18:59 UTC · lore

With "too many unreachable loose objects" git gc --auto will always trigger. This clutters the output of git am and thus git rebase.

The work flow of the Wine project doesn't include git merge. git rebase is therefor used to track the origin. This will produce soon too many loose objects for git gc --auto's taste. Pruning the repository would "fix" it. But we tell Wine developers new to git to NOT prune as long as they aren't confident enough with git; just as a safety net in case they have thrown away month of work.

Signed-off-by: Michael Stefaniuc <mstefani@redhat.com>
---
 git-am.sh |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
Show changes to git-am.sh +2 −2
diff --git a/git-am.sh b/git-am.sh
index 65c634f..5f0f241 100755
--- a/git-am.sh
+++ b/git-am.sh
@@ -469,9 +469,9 @@ do
 		"$GIT_DIR"/hooks/post-applypatch
 	fi
 
-	git gc --auto
-
 	go_next
 done
 
+git gc --auto
+
 rm -fr "$dotest"

-- 
1.5.4.rc2
Nicolas Pitre· Jan 4, 2008, 19:38 UTC · re: Michael Stefaniuc · lore

Re: [PATCH] git-am: Run git gc only once and not for every patch.

On Fri, 4 Jan 2008, Michael Stefaniuc wrote:
Show 9 quoted lines
> With "too many unreachable loose objects" git gc --auto will always
> trigger. This clutters the output of git am and thus git rebase.
> 
> The work flow of the Wine project doesn't include git merge. git rebase
> is therefor used to track the origin. This will produce soon too many
> loose objects for git gc --auto's taste. Pruning the repository would
> "fix" it. But we tell Wine developers new to git to NOT prune as long as
> they aren't confident enough with git; just as a safety net in case they
> have thrown away month of work.
The safety is the reflog.  What it refers to doesn't get pruned.
Nicolas
Michael Stefaniuc· Jan 4, 2008, 20:21 UTC · re: Nicolas Pitre · lore

Re: [PATCH] git-am: Run git gc only once and not for every patch.

Nicolas Pitre wrote:
Show 13 quoted lines
> On Fri, 4 Jan 2008, Michael Stefaniuc wrote:
> 
>> With "too many unreachable loose objects" git gc --auto will always
>> trigger. This clutters the output of git am and thus git rebase.
>>
>> The work flow of the Wine project doesn't include git merge. git rebase
>> is therefor used to track the origin. This will produce soon too many
>> loose objects for git gc --auto's taste. Pruning the repository would
>> "fix" it. But we tell Wine developers new to git to NOT prune as long as
>> they aren't confident enough with git; just as a safety net in case they
>> have thrown away month of work.
> 
> The safety is the reflog.  What it refers to doesn't get pruned.

Then git gc --auto should just prune too and not spam, right? But the reflog is only there for branches that still exist; the rest is gone. A git stash clear will also remove the reflog for the stash.

Regardless of the safety of git prune i don't see a reason why git gc --auto needs to be called for every patch in a git-am run and not once at the end.

bye
	michael
Nicolas Pitre· Jan 4, 2008, 20:58 UTC · re: Michael Stefaniuc · lore

Re: [PATCH] git-am: Run git gc only once and not for every patch.

On Fri, 4 Jan 2008, Michael Stefaniuc wrote:
Show 15 quoted lines
> Nicolas Pitre wrote:
> > On Fri, 4 Jan 2008, Michael Stefaniuc wrote:
> > 
> >> With "too many unreachable loose objects" git gc --auto will always
> >> trigger. This clutters the output of git am and thus git rebase.
> >>
> >> The work flow of the Wine project doesn't include git merge. git rebase
> >> is therefor used to track the origin. This will produce soon too many
> >> loose objects for git gc --auto's taste. Pruning the repository would
> >> "fix" it. But we tell Wine developers new to git to NOT prune as long as
> >> they aren't confident enough with git; just as a safety net in case they
> >> have thrown away month of work.
> > 
> > The safety is the reflog.  What it refers to doesn't get pruned.
> Then git gc --auto should just prune too and not spam, right?

Pruning might be dangerous if some other operation is happening concurrently, which is why it is not done by default.

> But the reflog is only there for branches that still exist; the rest is
> gone. A git stash clear will also remove the reflog for the stash.

But the reflog for HEAD remains nevertheless, and it contains everything that happened to be the current branch.

> Regardless of the safety of git prune i don't see a reason why git gc
> --auto needs to be called for every patch in a git-am run and not once
> at the end.
Indeed.
Nicolas
Junio C Hamano· Jan 4, 2008, 20:38 UTC · re: Nicolas Pitre · lore

Re: [PATCH] git-am: Run git gc only once and not for every patch.

Nicolas Pitre <nico@cam.org> writes:
Show 13 quoted lines
> On Fri, 4 Jan 2008, Michael Stefaniuc wrote:
>
>> With "too many unreachable loose objects" git gc --auto will always
>> trigger. This clutters the output of git am and thus git rebase.
>> 
>> The work flow of the Wine project doesn't include git merge. git rebase
>> is therefor used to track the origin. This will produce soon too many
>> loose objects for git gc --auto's taste. Pruning the repository would
>> "fix" it. But we tell Wine developers new to git to NOT prune as long as
>> they aren't confident enough with git; just as a safety net in case they
>> have thrown away month of work.
>
> The safety is the reflog.  What it refers to doesn't get pruned.
What Nico said.

More importantly, ones who are not confident with git would not be able to resurrect unreachables that are left unpruned anyway. The unreachables are by definition not connected to anything, so they cannot do much better than grepping through droppings left by "fsck --lost-found", which they probably even do not know how to do yet.

Michael Stefaniuc· Jan 4, 2008, 21:08 UTC · re: Junio C Hamano · lore

Re: [PATCH] git-am: Run git gc only once and not for every patch.

Junio C Hamano wrote:
Show 16 quoted lines
> Nicolas Pitre <nico@cam.org> writes:
> 
>> On Fri, 4 Jan 2008, Michael Stefaniuc wrote:
>>
>>> With "too many unreachable loose objects" git gc --auto will always
>>> trigger. This clutters the output of git am and thus git rebase.
>>>
>>> The work flow of the Wine project doesn't include git merge. git rebase
>>> is therefor used to track the origin. This will produce soon too many
>>> loose objects for git gc --auto's taste. Pruning the repository would
>>> "fix" it. But we tell Wine developers new to git to NOT prune as long as
>>> they aren't confident enough with git; just as a safety net in case they
>>> have thrown away month of work.
>> The safety is the reflog.  What it refers to doesn't get pruned.
> 
> What Nico said.

Not sure if you read my reply to Nico but the reflog is not there for deleted branches. Nor for a cleared stash. Common operations where one can loose work by mistake.

Show 6 quoted lines
> More importantly, ones who are not confident with git would not
> be able to resurrect unreachables that are left unpruned anyway.
> The unreachables are by definition not connected to anything, so
> they cannot do much better than grepping through droppings left
> by "fsck --lost-found", which they probably even do not know how
> to do yet.

The smarter ones come and ask on irc. We helped a few people with git problems on the Wine channels. Depending on the amount of time someone put into his "lost" work grepping through "fsck --lost-found" might be well worth it.

Nevertheless is there a reason why git gc needs to run after applying each patch in git-am? Why can't it run just once at the end? git prune is optional and there's no reason to penalize a user that doesn't feel safe to run it by cluttering the output of git am/rebase. There is also a time penalty: git gc --auto on a pruned tree runs so fast that it isn't measurable but on my unpruned wine it took 1.5 seconds. Waiting 1.5 seconds per am/rebase is acceptable; wasting 1.5 seconds per patch in the mailbox/rebase isn't that much fun if there are more than a handful of patches to apply.

bye
	michael
-- 
Michael Stefaniuc                           Tel.: +49-711-96437-199
Consulting Communications Engineer          Fax.: +49-711-96437-111
--------------------------------------------------------------------
Reg. Adresse: Red Hat GmbH, Hauptstätter Strasse 58, 70178 Stuttgart
Handelsregister: Amtsgericht Stuttgart HRB 153243
Geschäftsführer: Brendan Lane, Charlie Peters, Michael Cunningham,
                 Werner Knoblich
Junio C Hamano· Jan 4, 2008, 22:00 UTC · re: Michael Stefaniuc · lore

Re: [PATCH] git-am: Run git gc only once and not for every patch.

Michael Stefaniuc <mstefani@redhat.com> writes:
>> What Nico said.
> Not sure if you read my reply to Nico but the reflog is not there for
> deleted branches. Nor for a cleared stash. Common operations where one
> can loose work by mistake.
I am sure you read what Nico pointed out about HEAD reflog.
Show 9 quoted lines
> Nevertheless is there a reason why git gc needs to run after applying
> each patch in git-am? Why can't it run just once at the end? git prune
> is optional and there's no reason to penalize a user that doesn't feel
> safe to run it by cluttering the output of git am/rebase. There is also
> a time penalty: git gc --auto on a pruned tree runs so fast that it
> isn't measurable but on my unpruned wine it took 1.5 seconds. Waiting
> 1.5 seconds per am/rebase is acceptable; wasting 1.5 seconds per patch
> in the mailbox/rebase isn't that much fun if there are more than a
> handful of patches to apply.

"Optional" does not mean "whether you do it or not you will see identical behaviour". Not pruning may slow things down but you may still choose not to prune and suffer the penalty --- that's your choice. But we still produce the correct result (hopefully ;-) --- and that is what "Optional" means.

The message from "gc --auto" may serve as a coalmine canary for you to know when to prune.

I do not think moving "gc --auto" outside the loop hurts in practice because you are not likely to be rebasing a truly huge series every day, but cruft can accumulate during "git am" run and the "gc --auto" inside loop was meant to clean them up. The idea was taken from importers that run repack every once in a while (e.g. cvsimport runs every 1k commits), but "gc --auto" was designed to be much more lightweight than a full repack and that was the reason it was placed in the loop without counting "every N commits".

Junio C Hamano· Jan 5, 2008, 06:55 UTC · re: Junio C Hamano · lore

Re: [PATCH] git-am: Run git gc only once and not for every patch.

Junio C Hamano <gitster@pobox.com> writes:
Show 10 quoted lines
> ...
> I do not think moving "gc --auto" outside the loop hurts in
> practice because you are not likely to be rebasing a truly huge
> series every day, but cruft can accumulate during "git am" run
> and the "gc --auto" inside loop was meant to clean them up.  The
> idea was taken from importers that run repack every once in a
> while (e.g. cvsimport runs every 1k commits), but "gc --auto"
> was designed to be much more lightweight than a full repack and
> that was the reason it was placed in the loop without counting
> "every N commits".

Having said all that, I'll take your patch as-is except that I'd drop the later part of the commit log message that explains Wine's practice.

Michael Stefaniuc· Jan 5, 2008, 16:23 UTC · re: Junio C Hamano · lore

Re: [PATCH] git-am: Run git gc only once and not for every patch.

Junio C Hamano wrote:
Show 14 quoted lines
> Junio C Hamano <gitster@pobox.com> writes:
> 
>> ...
>> I do not think moving "gc --auto" outside the loop hurts in
>> practice because you are not likely to be rebasing a truly huge
>> series every day, but cruft can accumulate during "git am" run
>> and the "gc --auto" inside loop was meant to clean them up.  The
>> idea was taken from importers that run repack every once in a
>> while (e.g. cvsimport runs every 1k commits), but "gc --auto"
>> was designed to be much more lightweight than a full repack and
>> that was the reason it was placed in the loop without counting
>> "every N commits".
> 
> Having said all that, I'll take your patch as-is except that I'd
Thanks.
> drop the later part of the commit log message that explains
> Wine's practice.

Yeah, sorry about that. That should have been outside of the commit message anyway. But forgot about it as the Wine maintainer takes only the subject as commit message.

bye
	michael

← back to recent threads