{"thread":{"id":"27572","subject":"gc getting called on each git command ... what's wrong?","startedAt":"2011-06-08T01:33:41Z","lastAt":"2011-06-16T14:14:13Z","messageCount":10,"participants":["Geoff Russell","Peter Harris","Drew Northup","Brandon Casey","Junio C Hamano","Jakub Narebski"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"169529","messageId":"BANLkTi=oUARfwvNFNj-_FvZdwxQgibqPOg@mail.gmail.com","threadId":"27572","inReplyTo":null,"subject":"gc getting called on each git command ... what's wrong?","fromName":"Geoff Russell","fromEmail":"geoffrey.russell@gmail.com","sentAt":"2011-06-08T01:33:41Z","receivedAt":"2011-06-08T01:33:41Z","isPatch":false,"sender":{"key":"geoffrey.russell@gmail.com","avatar":"https://gravatar.com/avatar/c30f497ccfa6bf06d86f30bd2ba092a2dd124c61c6bc902f7cb5c3f6486947de?d=mp&s=160"},"body":"Hi all,\n\nI'm running git version 1.7.0.4 on Ubuntu 10.04 LTS\n\nAs of today, almost every time I do a git command, gc is getting\ninvoked. This is a multi-gigabyte repository with over\nhalf a million objects, so this takes a while ... and I'm guessing\nthat it shouldn't be happening anyway!\n\nI've run an fsck (which doesn't do a gc!) and the repository looks\nclean ... no output.\n\nI have packSizeLimit set to 30M ... not sure why I did this, was\ninvestigating something I\ndidn't understand.  There are 96 pack files.\n\nAny help greatly appreciated, many thanks,\n\nCheers,\nGeoff\n"},{"id":"169531","messageId":"BANLkTiksHRmp102XDJP5+-CLGj8hZXUR=g@mail.gmail.com","threadId":"27572","inReplyTo":"BANLkTi=oUARfwvNFNj-_FvZdwxQgibqPOg@mail.gmail.com","subject":"Re: gc getting called on each git command ... what's wrong?","fromName":"Peter Harris","fromEmail":"git@peter.is-a-geek.org","sentAt":"2011-06-08T01:48:29Z","receivedAt":"2011-06-08T01:48:29Z","isPatch":false,"sender":{"key":"git@peter.is-a-geek.org","avatar":null},"body":"On Tue, Jun 7, 2011 at 9:33 PM, Geoff Russell wrote:\n>\n> As of today, almost every time I do a git command, gc is getting\n> invoked.\n\n>   There are 96 pack files.\n\nThat's why. See gc.autopacklimit in \"git help config\" -- by default,\ngit will gc if there are more than 50 pack files.\n\nPeter Harris\n"},{"id":"169628","messageId":"1307548930.19438.30.camel@drew-northup.unet.maine.edu","threadId":"27572","inReplyTo":"BANLkTiksHRmp102XDJP5+-CLGj8hZXUR=g@mail.gmail.com","subject":"Re: gc getting called on each git command ... what's wrong?","fromName":"Drew Northup","fromEmail":"drew.northup@maine.edu","sentAt":"2011-06-08T16:02:10Z","receivedAt":"2011-06-08T16:02:10Z","isPatch":false,"sender":{"key":"drew.northup@maine.edu","avatar":"https://avatars.githubusercontent.com/u/18331571?v=4"},"body":"\nOn Tue, 2011-06-07 at 21:48 -0400, Peter Harris wrote:\n> On Tue, Jun 7, 2011 at 9:33 PM, Geoff Russell wrote:\n> >\n> > As of today, almost every time I do a git command, gc is getting\n> > invoked.\n<re-added>\n> >   I have packSizeLimit set to 30M \n</re-added>\n> >   There are 96 pack files.\n> \n> That's why. See gc.autopacklimit in \"git help config\" -- by default,\n> git will gc if there are more than 50 pack files.\n\nDo we want to consider ignoring (or automatically doubling, or something\nlike that) gc.autopacklimit if that number of packs meet or exceed\ngc.packSizeLimit? I have no idea what the patch for this might look\nlike, but it seems to make more sense than this situation.\n\nJust a random brain fart...\n\n-- \n-Drew Northup\n________________________________________________\n\"As opposed to vegetable or mineral error?\"\n-John Pescatore, SANS NewsBites Vol. 12 Num. 59\n"},{"id":"169633","messageId":"mHQrb8DBZuqQY8Hc3UzxeMQ_IVnTRI-lDtGLq4eawjiPycP6aZDRrQ@cipher.nrlssc.navy.mil","threadId":"27572","inReplyTo":"1307548930.19438.30.camel@drew-northup.unet.maine.edu","subject":"Re: gc getting called on each git command ... what's wrong?","fromName":"Brandon Casey","fromEmail":"brandon.casey.ctr@nrlssc.navy.mil","sentAt":"2011-06-08T16:29:48Z","receivedAt":"2011-06-08T16:29:48Z","isPatch":false,"sender":{"key":"brandon.casey.ctr@nrlssc.navy.mil","avatar":null},"body":"On 06/08/2011 11:02 AM, Drew Northup wrote:\n> \n> On Tue, 2011-06-07 at 21:48 -0400, Peter Harris wrote:\n>> On Tue, Jun 7, 2011 at 9:33 PM, Geoff Russell wrote:\n>>>\n>>> As of today, almost every time I do a git command, gc is getting\n>>> invoked.\n> <re-added>\n>>>   I have packSizeLimit set to 30M \n> </re-added>\n>>>   There are 96 pack files.\n>>\n>> That's why. See gc.autopacklimit in \"git help config\" -- by default,\n>> git will gc if there are more than 50 pack files.\n> \n> Do we want to consider ignoring (or automatically doubling, or something\n> like that) gc.autopacklimit if that number of packs meet or exceed\n> gc.packSizeLimit? I have no idea what the patch for this might look\n> like, but it seems to make more sense than this situation.\n> \n> Just a random brain fart...\n> \n\nOr just ignore the packs that exceed pack.packSizeLimit...\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex ff5f73b..7be14ab 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -26,6 +26,7 @@ static int pack_refs = 1;\n static int aggressive_window = 250;\n static int gc_auto_threshold = 6700;\n static int gc_auto_pack_limit = 50;\n+static off_t pack_size_limit;\n static const char *prune_expire = \"2.weeks.ago\";\n \n #define MAX_ADD 10\n@@ -64,6 +65,10 @@ static int gc_config(const char *var, const char *value, void *cb)\n \t\t}\n \t\treturn git_config_string(&prune_expire, var, value);\n \t}\n+\tif (!strcmp(var, \"pack.packsizelimit\")) {\n+\t\tpack_size_limit = git_config_ulong(var, value);\n+\t\treturn 0;\n+\t}\n \treturn git_default_config(var, value, cb);\n }\n \n@@ -135,10 +140,8 @@ static int too_many_packs(void)\n \t\t\tcontinue;\n \t\tif (p->pack_keep)\n \t\t\tcontinue;\n-\t\t/*\n-\t\t * Perhaps check the size of the pack and count only\n-\t\t * very small ones here?\n-\t\t */\n+\t\tif (pack_size_limit && p->pack_size >= pack_size_limit)\n+\t\t\tcontinue;\n \t\tcnt++;\n \t}\n \treturn gc_auto_pack_limit <= cnt;\n"},{"id":"169637","messageId":"7vhb802row.fsf@alter.siamese.dyndns.org","threadId":"27572","inReplyTo":"1307548930.19438.30.camel@drew-northup.unet.maine.edu","subject":"Re: gc getting called on each git command ... what's wrong?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-06-08T16:50:23Z","receivedAt":"2011-06-08T16:50:23Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Drew Northup <drew.northup@maine.edu> writes:\n\n>> That's why. See gc.autopacklimit in \"git help config\" -- by default,\n>> git will gc if there are more than 50 pack files.\n>\n> Do we want to consider ignoring (or automatically doubling, or something\n> like that) gc.autopacklimit if that number of packs meet or exceed\n> gc.packSizeLimit? I have no idea what the patch for this might look\n> like, but it seems to make more sense than this situation.\n\nThis is unrelated to the auto-gc, but it also would be fruitful to\nquestion if it is a sane setting to limit packfiles to 30M, when the\nrepository needs 100 of them (total around 3G??). Just like having too\nmany loose object files degrade performance (and that is one of the\nreasons we pack them in the first place), having many packs will degrade\nperformance unnecessarily and to a worse degree, as \"check which pack has\nthis particular object\" code has to examine all packs, unlike the loose\nobject case where we let the .git/objects/?? fan-out to give us some\nhashing and the filesystem to do the heavylifting for us.\n"},{"id":"169641","messageId":"m3pqmodzee.fsf@localhost.localdomain","threadId":"27572","inReplyTo":"BANLkTiksHRmp102XDJP5+-CLGj8hZXUR=g@mail.gmail.com","subject":"Re: gc getting called on each git command ... what's wrong?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2011-06-08T17:09:12Z","receivedAt":"2011-06-08T17:09:12Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Peter Harris <git@peter.is-a-geek.org> writes:\n\n> On Tue, Jun 7, 2011 at 9:33 PM, Geoff Russell wrote:\n> >\n> > As of today, almost every time I do a git command, gc is getting\n> > invoked.\n> \n> >   There are 96 pack files.\n> \n> That's why. See gc.autopacklimit in \"git help config\" -- by default,\n> git will gc if there are more than 50 pack files.\n\nActually it looks like it is combination of this and packSizeLimit set\nto 30M.  Git notices that it has too many packfiles, and tries to\nrepack them, but packlimit forces Git to split it into small\npackfiles... and end up with more packfiles than limit anyway.\n\nPerhaps git should notice that it has nonsensical combination of\noptions...\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"170038","messageId":"BANLkTikdP6XKac9t8v2kyUEta3CB+Ue0PQ@mail.gmail.com","threadId":"27572","inReplyTo":"m3pqmodzee.fsf@localhost.localdomain","subject":"Re: gc getting called on each git command ... what's wrong?","fromName":"Geoff Russell","fromEmail":"geoffrey.russell@gmail.com","sentAt":"2011-06-15T01:28:54Z","receivedAt":"2011-06-15T01:28:54Z","isPatch":false,"sender":{"key":"geoffrey.russell@gmail.com","avatar":"https://gravatar.com/avatar/c30f497ccfa6bf06d86f30bd2ba092a2dd124c61c6bc902f7cb5c3f6486947de?d=mp&s=160"},"body":"On Thu, Jun 9, 2011 at 2:39 AM, Jakub Narebski <jnareb@gmail.com> wrote:\n>\n> Peter Harris <git@peter.is-a-geek.org> writes:\n>\n> > On Tue, Jun 7, 2011 at 9:33 PM, Geoff Russell wrote:\n> > >\n> > > As of today, almost every time I do a git command, gc is getting\n> > > invoked.\n> >\n> > >   There are 96 pack files.\n> >\n> > That's why. See gc.autopacklimit in \"git help config\" -- by default,\n> > git will gc if there are more than 50 pack files.\n\nThanks to everybody. This is exactly what was happening and the problems went\naway when I set the packSizeLimit higher ... 3000M\n\n>\n> Actually it looks like it is combination of this and packSizeLimit set\n> to 30M.  Git notices that it has too many packfiles, and tries to\n> repack them, but packlimit forces Git to split it into small\n> packfiles... and end up with more packfiles than limit anyway.\n>\n> Perhaps git should notice that it has nonsensical combination of\n> options...\n\nThat would be nice. It should be reasonably easy to work out that the\npackSizeLimit\nwill guarantee too many pack files after the gc. Disobeying a users\nwishes shouldn't\nbe undertaken lightly, but sometimes we stuff up :)\n\nCheers,\nGeoff.\n\n--\n6 Fifth Ave,\nSt Morris, S.A. 5068\nAustralia\nPh: 041 8805 184 / 08 8332 5069\nhttp://perfidy.com.au\n"},{"id":"170071","messageId":"201106151735.07137.jnareb@gmail.com","threadId":"27572","inReplyTo":"BANLkTi=w10KQ3MSd5YuYR+S=eMgywNTY-A@mail.gmail.com","subject":"Re: gc getting called on each git command ... what's wrong?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2011-06-15T15:35:06Z","receivedAt":"2011-06-15T15:35:06Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Wed, 15 Jun 2011, Geoff Russell wrote:\n> On Thu, Jun 9, 2011 at 2:39 AM, Jakub Narebski <jnareb@gmail.com> wrote:\n> > Peter Harris <git@peter.is-a-geek.org> writes:\n> > > On Tue, Jun 7, 2011 at 9:33 PM, Geoff Russell wrote:\n> > > >\n> > > > As of today, almost every time I do a git command, gc is getting\n> > > > invoked.\n> > >\n> > > >   There are 96 pack files.\n> > >\n> > > That's why. See gc.autopacklimit in \"git help config\" -- by default,\n> > > git will gc if there are more than 50 pack files.\n> >\n> > Actually it looks like it is combination of this and packSizeLimit set\n> > to 30M.  Git notices that it has too many packfiles, and tries to\n> > repack them, but packlimit forces Git to split it into small\n> > packfiles... and end up with more packfiles than limit anyway.\n> \n> Thanks to everybody. This is exactly what was happening and the problems\n> went away when I set the packSizeLimit higher ... 3000M\n \nWhy did you set packSizeLimit at all?\n\n \n> >\n> > Perhaps git should notice that it has nonsensical combination of\n> > options...\n> \n> That would be nice. It should be reasonably easy to work out that the\n> packSizeLimit will guarantee too many pack files after the gc.\n> Disobeying a users wishes shouldn't be undertaken lightly, but sometimes\n> we stuff up :) \n\nWell, git can simply notice that each except perhaps on file has\nsize greater or equal to gc.packSizeLimit, and then ignore gc.autopacklimit\nhint, because repacking would not reduce number of packs, and not lower\nit below gc.autopacklimit.\n\nIf `git gc` is called interactively, we can warn user about this situation...\n\n-- \nJakub Narebski\nPoland\n"},{"id":"170111","messageId":"BANLkTimYsvHqZsxYA34o7s94c9V8vzkHbw@mail.gmail.com","threadId":"27572","inReplyTo":"201106151735.07137.jnareb@gmail.com","subject":"Re: gc getting called on each git command ... what's wrong?","fromName":"Geoff Russell","fromEmail":"geoffrey.russell@gmail.com","sentAt":"2011-06-16T01:46:21Z","receivedAt":"2011-06-16T01:46:21Z","isPatch":false,"sender":{"key":"geoffrey.russell@gmail.com","avatar":"https://gravatar.com/avatar/c30f497ccfa6bf06d86f30bd2ba092a2dd124c61c6bc902f7cb5c3f6486947de?d=mp&s=160"},"body":"2011/6/16 Jakub Narebski <jnareb@gmail.com>\n>\n>\n> Why did you set packSizeLimit at all?\n>\n>\n\nSome time ago (31/8/2010) I had a problem which seemed to be caused by\nlarge packs (>4GB), you\ncan find it in the git list with a subject of \"Large pack causes git\nclone failures ... what to do?\"\n\nAnyway, I set packSizeLimit and fiddled around for a bit ...\neventually the problem went\naway when I moved the central repository to another machine with less load and\nmore memory. At which point I gave a sigh of relief and forgot to\nremove the packSizeLimit\nuntil recently bitten. But the original problem was probably nothing\nto do with large\npacks and hasn't recurred.\n\nCheers,\nGeoff.\n"},{"id":"170133","messageId":"201106161614.14656.jnareb@gmail.com","threadId":"27572","inReplyTo":"BANLkTimYsvHqZsxYA34o7s94c9V8vzkHbw@mail.gmail.com","subject":"Re: gc getting called on each git command ... what's wrong?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2011-06-16T14:14:13Z","receivedAt":"2011-06-16T14:14:13Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Thu, 16 Jun 2011, Geoff Russell wrote:\n> 2011/6/16 Jakub Narebski <jnareb@gmail.com>\n> >\n> > Why did you set packSizeLimit at all?\n> \n> Some time ago (31/8/2010) I had a problem which seemed to be caused by\n> large packs (>4GB), you can find it in the git list with a subject of\n> \"Large pack causes git clone failures ... what to do?\"\n\nSo why did you set packSizeLimit to such ridiculous low value, instead\nof 2g (2 GB) or something?\n\n-- \nJakub Narebski\nPoland\n"}]}