threads / discuss / 27572

gc getting called on each git command ... what's wrong?

Subject: gc getting called on each git command ... what's wrong?

## tl;dr

10 messages between Jun 8, 2011 and Jun 16, 2011.

replies: 9people: 6as markdown or json

Geoff Russell· Jun 8, 2011, 01:33 UTC · lore
Hi all,
I'm running git version 1.7.0.4 on Ubuntu 10.04 LTS

As of today, almost every time I do a git command, gc is getting invoked. This is a multi-gigabyte repository with over half a million objects, so this takes a while ... and I'm guessing that it shouldn't be happening anyway!

I've run an fsck (which doesn't do a gc!) and the repository looks clean ... no output.

I have packSizeLimit set to 30M ... not sure why I did this, was investigating something I didn't understand. There are 96 pack files.

Any help greatly appreciated, many thanks,

Cheers, Geoff

Peter Harris· Jun 8, 2011, 01:48 UTC · re: Geoff Russell · lore

Re: gc getting called on each git command ... what's wrong?

On Tue, Jun 7, 2011 at 9:33 PM, Geoff Russell wrote:
>
> As of today, almost every time I do a git command, gc is getting
> invoked.
>   There are 96 pack files.

That's why. See gc.autopacklimit in "git help config" -- by default, git will gc if there are more than 50 pack files.

Peter Harris
Drew Northup· Jun 8, 2011, 16:02 UTC · re: Peter Harris · lore

Re: gc getting called on each git command ... what's wrong?

On Tue, 2011-06-07 at 21:48 -0400, Peter Harris wrote:
> On Tue, Jun 7, 2011 at 9:33 PM, Geoff Russell wrote:
> >
> > As of today, almost every time I do a git command, gc is getting
> > invoked.
<re-added>
> >   I have packSizeLimit set to 30M 
</re-added>
> >   There are 96 pack files.
> 
> That's why. See gc.autopacklimit in "git help config" -- by default,
> git will gc if there are more than 50 pack files.

Do we want to consider ignoring (or automatically doubling, or something like that) gc.autopacklimit if that number of packs meet or exceed gc.packSizeLimit? I have no idea what the patch for this might look like, but it seems to make more sense than this situation.

Just a random brain fart...
-- 
-Drew Northup
________________________________________________
"As opposed to vegetable or mineral error?"
-John Pescatore, SANS NewsBites Vol. 12 Num. 59
Brandon Casey· Jun 8, 2011, 16:29 UTC · re: Drew Northup · lore

Re: gc getting called on each git command ... what's wrong?

On 06/08/2011 11:02 AM, Drew Northup wrote:
Show 21 quoted lines
> 
> On Tue, 2011-06-07 at 21:48 -0400, Peter Harris wrote:
>> On Tue, Jun 7, 2011 at 9:33 PM, Geoff Russell wrote:
>>>
>>> As of today, almost every time I do a git command, gc is getting
>>> invoked.
> <re-added>
>>>   I have packSizeLimit set to 30M 
> </re-added>
>>>   There are 96 pack files.
>>
>> That's why. See gc.autopacklimit in "git help config" -- by default,
>> git will gc if there are more than 50 pack files.
> 
> Do we want to consider ignoring (or automatically doubling, or something
> like that) gc.autopacklimit if that number of packs meet or exceed
> gc.packSizeLimit? I have no idea what the patch for this might look
> like, but it seems to make more sense than this situation.
> 
> Just a random brain fart...
> 
Or just ignore the packs that exceed pack.packSizeLimit...
diff --git a/builtin/gc.c b/builtin/gc.c
index ff5f73b..7be14ab 100644
--- a/builtin/gc.c
+++ b/builtin/gc.c
@@ -26,6 +26,7 @@ static int pack_refs = 1;
 static int aggressive_window = 250;
 static int gc_auto_threshold = 6700;
 static int gc_auto_pack_limit = 50;
+static off_t pack_size_limit;
 static const char *prune_expire = "2.weeks.ago";
 
 #define MAX_ADD 10
@@ -64,6 +65,10 @@ static int gc_config(const char *var, const char *value, void *cb)
 		}
 		return git_config_string(&prune_expire, var, value);
 	}
+	if (!strcmp(var, "pack.packsizelimit")) {
+		pack_size_limit = git_config_ulong(var, value);
+		return 0;
+	}
 	return git_default_config(var, value, cb);
 }
 
@@ -135,10 +140,8 @@ static int too_many_packs(void)
 			continue;
 		if (p->pack_keep)
 			continue;
-		/*
-		 * Perhaps check the size of the pack and count only
-		 * very small ones here?
-		 */
+		if (pack_size_limit && p->pack_size >= pack_size_limit)
+			continue;
 		cnt++;
 	}
 	return gc_auto_pack_limit <= cnt;
Junio C Hamano· Jun 8, 2011, 16:50 UTC · re: Drew Northup · lore

Re: gc getting called on each git command ... what's wrong?

Drew Northup <drew.northup@maine.edu> writes:
Show 7 quoted lines
>> That's why. See gc.autopacklimit in "git help config" -- by default,
>> git will gc if there are more than 50 pack files.
>
> Do we want to consider ignoring (or automatically doubling, or something
> like that) gc.autopacklimit if that number of packs meet or exceed
> gc.packSizeLimit? I have no idea what the patch for this might look
> like, but it seems to make more sense than this situation.

This is unrelated to the auto-gc, but it also would be fruitful to question if it is a sane setting to limit packfiles to 30M, when the repository needs 100 of them (total around 3G??). Just like having too many loose object files degrade performance (and that is one of the reasons we pack them in the first place), having many packs will degrade performance unnecessarily and to a worse degree, as "check which pack has this particular object" code has to examine all packs, unlike the loose object case where we let the .git/objects/?? fan-out to give us some hashing and the filesystem to do the heavylifting for us.

Jakub Narebski· Jun 8, 2011, 17:09 UTC · re: Peter Harris · lore

Re: gc getting called on each git command ... what's wrong?

Peter Harris <git@peter.is-a-geek.org> writes:
Show 9 quoted lines
> On Tue, Jun 7, 2011 at 9:33 PM, Geoff Russell wrote:
> >
> > As of today, almost every time I do a git command, gc is getting
> > invoked.
> 
> >   There are 96 pack files.
> 
> That's why. See gc.autopacklimit in "git help config" -- by default,
> git will gc if there are more than 50 pack files.

Actually it looks like it is combination of this and packSizeLimit set to 30M. Git notices that it has too many packfiles, and tries to repack them, but packlimit forces Git to split it into small packfiles... and end up with more packfiles than limit anyway.

Perhaps git should notice that it has nonsensical combination of options...

-- 
Jakub Narebski
Poland
ShadeHawk on #git
Geoff Russell· Jun 15, 2011, 01:28 UTC · re: Jakub Narebski · lore

Re: gc getting called on each git command ... what's wrong?

On Thu, Jun 9, 2011 at 2:39 AM, Jakub Narebski <jnareb@gmail.com> wrote:
Show 12 quoted lines
>
> Peter Harris <git@peter.is-a-geek.org> writes:
>
> > On Tue, Jun 7, 2011 at 9:33 PM, Geoff Russell wrote:
> > >
> > > As of today, almost every time I do a git command, gc is getting
> > > invoked.
> >
> > >   There are 96 pack files.
> >
> > That's why. See gc.autopacklimit in "git help config" -- by default,
> > git will gc if there are more than 50 pack files.

Thanks to everybody. This is exactly what was happening and the problems went away when I set the packSizeLimit higher ... 3000M

Show 8 quoted lines
>
> Actually it looks like it is combination of this and packSizeLimit set
> to 30M.  Git notices that it has too many packfiles, and tries to
> repack them, but packlimit forces Git to split it into small
> packfiles... and end up with more packfiles than limit anyway.
>
> Perhaps git should notice that it has nonsensical combination of
> options...

That would be nice. It should be reasonably easy to work out that the packSizeLimit will guarantee too many pack files after the gc. Disobeying a users wishes shouldn't be undertaken lightly, but sometimes we stuff up :)

Cheers, Geoff.

--
6 Fifth Ave,
St Morris, S.A. 5068
Australia
Ph: 041 8805 184 / 08 8332 5069
http://perfidy.com.au
Jakub Narebski· Jun 15, 2011, 15:35 UTC · lore

Re: gc getting called on each git command ... what's wrong?

On Wed, 15 Jun 2011, Geoff Russell wrote:
Show 19 quoted lines
> On Thu, Jun 9, 2011 at 2:39 AM, Jakub Narebski <jnareb@gmail.com> wrote:
> > Peter Harris <git@peter.is-a-geek.org> writes:
> > > On Tue, Jun 7, 2011 at 9:33 PM, Geoff Russell wrote:
> > > >
> > > > As of today, almost every time I do a git command, gc is getting
> > > > invoked.
> > >
> > > >   There are 96 pack files.
> > >
> > > That's why. See gc.autopacklimit in "git help config" -- by default,
> > > git will gc if there are more than 50 pack files.
> >
> > Actually it looks like it is combination of this and packSizeLimit set
> > to 30M.  Git notices that it has too many packfiles, and tries to
> > repack them, but packlimit forces Git to split it into small
> > packfiles... and end up with more packfiles than limit anyway.
> 
> Thanks to everybody. This is exactly what was happening and the problems
> went away when I set the packSizeLimit higher ... 3000M
 
Why did you set packSizeLimit at all?
 
Show 8 quoted lines
> >
> > Perhaps git should notice that it has nonsensical combination of
> > options...
> 
> That would be nice. It should be reasonably easy to work out that the
> packSizeLimit will guarantee too many pack files after the gc.
> Disobeying a users wishes shouldn't be undertaken lightly, but sometimes
> we stuff up :) 

Well, git can simply notice that each except perhaps on file has size greater or equal to gc.packSizeLimit, and then ignore gc.autopacklimit hint, because repacking would not reduce number of packs, and not lower it below gc.autopacklimit.

If `git gc` is called interactively, we can warn user about this situation...
-- 
Jakub Narebski
Poland
Geoff Russell· Jun 16, 2011, 01:46 UTC · re: Jakub Narebski · lore

Re: gc getting called on each git command ... what's wrong?

2011/6/16 Jakub Narebski <jnareb@gmail.com>
Show 5 quoted lines
>
>
> Why did you set packSizeLimit at all?
>
>

Some time ago (31/8/2010) I had a problem which seemed to be caused by large packs (>4GB), you can find it in the git list with a subject of "Large pack causes git clone failures ... what to do?"

Anyway, I set packSizeLimit and fiddled around for a bit ... eventually the problem went away when I moved the central repository to another machine with less load and more memory. At which point I gave a sigh of relief and forgot to remove the packSizeLimit until recently bitten. But the original problem was probably nothing to do with large packs and hasn't recurred.

Cheers, Geoff.

Jakub Narebski· Jun 16, 2011, 14:14 UTC · re: Geoff Russell · lore

Re: gc getting called on each git command ... what's wrong?

On Thu, 16 Jun 2011, Geoff Russell wrote:
Show 7 quoted lines
> 2011/6/16 Jakub Narebski <jnareb@gmail.com>
> >
> > Why did you set packSizeLimit at all?
> 
> Some time ago (31/8/2010) I had a problem which seemed to be caused by
> large packs (>4GB), you can find it in the git list with a subject of
> "Large pack causes git clone failures ... what to do?"

So why did you set packSizeLimit to such ridiculous low value, instead of 2g (2 GB) or something?

-- 
Jakub Narebski
Poland

← back to recent threads