{"thread":{"id":"16735","subject":"\"git gc\" doesn't seem to remove loose objects any more","startedAt":"2008-12-15T12:52:34Z","lastAt":"2008-12-15T19:38:37Z","messageCount":12,"participants":["Bruce Stephens","Mikael Magnusson","Björn Steinbrink","Theodore Tso","Mark Brown","Johan Herland","Jakub Narebski","Brandon Casey","Nicolas Pitre"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"97954","messageId":"808wqhzjl9.fsf@tiny.isode.net","threadId":"16735","inReplyTo":null,"subject":"\"git gc\" doesn't seem to remove loose objects any more","fromName":"Bruce Stephens","fromEmail":"bruce.stephens@isode.com","sentAt":"2008-12-15T12:52:34Z","receivedAt":"2008-12-15T12:52:34Z","isPatch":false,"sender":{"key":"bruce.stephens@isode.com","avatar":null},"body":"I couldn't see a test for this, but perhaps I'm just missing it?\n\n brs% git count-objects\n 161 objects, 1552 kilobytes\n brs% git gc\n Counting objects: 80621, done.\n Compressing objects: 100% (22372/22372), done.\n Writing objects: 100% (80621/80621), done.\n Total 80621 (delta 57160), reused 80305 (delta 56884)\n brs% git count-objects\n 207 objects, 2048 kilobytes\n\n\nAnd I see lots of directories under .git/objects which confirms\nthings.\n\nI don't think I've changed any relevant configuration.\n\nThis is with 8befc50c49e8a271fd3cd7fb34258fe88d1dfcad (also whatever\nversion I used before, erm, probably\nde0db422782ddaf7754ac5b03fdc6dc5de1a9ae4), and possibly earlier\nversions---I've just started noticing now that the number of loose\nobjects has started causing git gui to complain.\n\n(Hmm, I note that git gui reports a larger number of loose objects\nthan git count-objects. Ah, OK, it really is just an approximation,\nso no surprise.)\n"},{"id":"97956","messageId":"237967ef0812150538n671c22b8gaf7a7b5dcaf68433@mail.gmail.com","threadId":"16735","inReplyTo":"808wqhzjl9.fsf@tiny.isode.net","subject":"Re: \"git gc\" doesn't seem to remove loose objects any more","fromName":"Mikael Magnusson","fromEmail":"mikachu@gmail.com","sentAt":"2008-12-15T13:38:56Z","receivedAt":"2008-12-15T13:38:56Z","isPatch":false,"sender":{"key":"mikachu@gmail.com","avatar":null},"body":"2008/12/15 Bruce Stephens :\n> I couldn't see a test for this, but perhaps I'm just missing it?\n>\n> brs% git count-objects\n> 161 objects, 1552 kilobytes\n> brs% git gc\n> Counting objects: 80621, done.\n> Compressing objects: 100% (22372/22372), done.\n> Writing objects: 100% (80621/80621), done.\n> Total 80621 (delta 57160), reused 80305 (delta 56884)\n> brs% git count-objects\n> 207 objects, 2048 kilobytes\n>\n>\n> And I see lots of directories under .git/objects which confirms\n> things.\n>\n> I don't think I've changed any relevant configuration.\n>\n> This is with 8befc50c49e8a271fd3cd7fb34258fe88d1dfcad (also whatever\n> version I used before, erm, probably\n> de0db422782ddaf7754ac5b03fdc6dc5de1a9ae4), and possibly earlier\n> versions---I've just started noticing now that the number of loose\n> objects has started causing git gui to complain.\n>\n> (Hmm, I note that git gui reports a larger number of loose objects\n> than git count-objects. Ah, OK, it really is just an approximation,\n> so no surprise.)\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n\nIIRC git gc only removes loose objects older than two weeks, if you\nreally want to remove them now, run git prune. But make sure no other\ngit process can be active when you run it, or it could possibly step\non something.\n\n-- \nMikael Magnusson\n"},{"id":"97960","messageId":"80oczdy1iv.fsf@tiny.isode.net","threadId":"16735","inReplyTo":"237967ef0812150538n671c22b8gaf7a7b5dcaf68433@mail.gmail.com","subject":"Re: \"git gc\" doesn't seem to remove loose objects any more","fromName":"Bruce Stephens","fromEmail":"bruce.stephens@isode.com","sentAt":"2008-12-15T14:08:08Z","receivedAt":"2008-12-15T14:08:08Z","isPatch":false,"sender":{"key":"bruce.stephens@isode.com","avatar":null},"body":"\"Mikael Magnusson\" writes:\n\n[...]\n\n> IIRC git gc only removes loose objects older than two weeks, if you\n> really want to remove them now, run git prune. But make sure no other\n> git process can be active when you run it, or it could possibly step\n> on something.\n\nOK, that makes sense. Obviously I misunderstood this. That doesn't\nexplain why the number of objects might increase after \"git gc\", but\nperhaps that's for a good reason too.\n\nSurely \"git gui\"'s warning is unhelpful, then: it warns I have more\nthan 2000 loose objects (in another checkout), offers to compress my\ndatabase, and I end up with an unchanged repository (which it still\ncomplains about)? Is this warning just redundant now that we've got\n\"git gc --auto\"?\n"},{"id":"97961","messageId":"20081215140834.GA3684@atjola.homenet","threadId":"16735","inReplyTo":"237967ef0812150538n671c22b8gaf7a7b5dcaf68433@mail.gmail.com","subject":"Re: \"git gc\" doesn't seem to remove loose objects any more","fromName":"Björn Steinbrink","fromEmail":"b.steinbrink@gmx.de","sentAt":"2008-12-15T14:08:34Z","receivedAt":"2008-12-15T14:08:34Z","isPatch":false,"sender":{"key":"b.steinbrink@gmx.de","avatar":"https://avatars.githubusercontent.com/u/230962?v=4"},"body":"On 2008.12.15 14:38:56 +0100, Mikael Magnusson wrote:\n> 2008/12/15 Bruce Stephens :\n> > I couldn't see a test for this, but perhaps I'm just missing it?\n> >\n> > brs% git count-objects\n> > 161 objects, 1552 kilobytes\n> > brs% git gc\n> > Counting objects: 80621, done.\n> > Compressing objects: 100% (22372/22372), done.\n> > Writing objects: 100% (80621/80621), done.\n> > Total 80621 (delta 57160), reused 80305 (delta 56884)\n> > brs% git count-objects\n> > 207 objects, 2048 kilobytes\n> >\n> >\n> > And I see lots of directories under .git/objects which confirms\n> > things.\n> >\n> > I don't think I've changed any relevant configuration.\n> >\n> > This is with 8befc50c49e8a271fd3cd7fb34258fe88d1dfcad (also whatever\n> > version I used before, erm, probably\n> > de0db422782ddaf7754ac5b03fdc6dc5de1a9ae4), and possibly earlier\n> > versions---I've just started noticing now that the number of loose\n> > objects has started causing git gui to complain.\n> >\n> > (Hmm, I note that git gui reports a larger number of loose objects\n> > than git count-objects. Ah, OK, it really is just an approximation,\n> > so no surprise.)\n> \n> IIRC git gc only removes loose objects older than two weeks, if you\n> really want to remove them now, run git prune. But make sure no other\n> git process can be active when you run it, or it could possibly step\n> on something.\n\nTo clarify that a bit more: git gc keeps unreachable objects unpacked,\nso that git prune can drop them. And git gc invokes git prune so that\nonly unreachable objects older than 2 weeks are dropped.\n\nBjörn\n"},{"id":"97968","messageId":"20081215155610.GA11502@mit.edu","threadId":"16735","inReplyTo":"20081215140834.GA3684@atjola.homenet","subject":"Re: \"git gc\" doesn't seem to remove loose objects any more","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2008-12-15T15:56:10Z","receivedAt":"2008-12-15T15:56:10Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Mon, Dec 15, 2008 at 03:08:34PM +0100, Björn Steinbrink wrote:\n> To clarify that a bit more: git gc keeps unreachable objects unpacked,\n> so that git prune can drop them. And git gc invokes git prune so that\n> only unreachable objects older than 2 weeks are dropped.\n\nTo be even more explicit, \"git gc\" will **unpack** objects that have\nbecome unreachable and were currently in packs. As a result, the\namount of disk space used by a git repository can actually go **up**\ndramatically after a \"git gc\" operation, which could be surprising for\nsomeone who is running close to full on their filesystem, deletes a\nnumber of branches from a tracking repository, and then does a \"git\ngc\" may get a very unpleasant surprise.\n\nA really good repository which shows this is linux-next, since it is\nconstantly getting rewound, and old branches are reserved via a tag\nsuch as next-20081204. If you update the your local copy of the\nlinux-next repository every day, you will accumulate a large number of\nthese old branch tags. If you then delete a whole series of them, and\nrun git-gc, the operation will take quite a while, and the number of\nblocks and inodes used will grow significantly. They will disappear\nafter a \"git prune\", but when I do this housekeeping operation, I've\noften wished for a --yes-I-know-what-I-am-doing-and-it's-unsafe-but-\njust-drop-the-unreachable-objects-cause-this-is-just-a-tracking-repository\noption to \"git gc\".\n\n\t\t\t\t\t\t- Ted\n"},{"id":"97969","messageId":"20081215161212.GE31145@sirena.org.uk","threadId":"16735","inReplyTo":"20081215155610.GA11502@mit.edu","subject":"Re: \"git gc\" doesn't seem to remove loose objects any more","fromName":"Mark Brown","fromEmail":"broonie@sirena.org.uk","sentAt":"2008-12-15T16:12:13Z","receivedAt":"2008-12-15T16:12:13Z","isPatch":false,"sender":{"key":"broonie@sirena.org.uk","avatar":"https://gravatar.com/avatar/9e798c729a4a709279df497d9608ad68422755c9670e92434a5436f7e607cf86?d=mp&s=160"},"body":"On Mon, Dec 15, 2008 at 10:56:10AM -0500, Theodore Tso wrote:\n> On Mon, Dec 15, 2008 at 03:08:34PM +0100, Bj?rn Steinbrink wrote:\n\n> > To clarify that a bit more: git gc keeps unreachable objects unpacked,\n> > so that git prune can drop them. And git gc invokes git prune so that\n> > only unreachable objects older than 2 weeks are dropped.\n\n> To be even more explicit, \"git gc\" will **unpack** objects that have\n> become unreachable and were currently in packs. As a result, the\n> amount of disk space used by a git repository can actually go **up**\n> dramatically after a \"git gc\" operation, which could be surprising for\n> someone who is running close to full on their filesystem, deletes a\n> number of branches from a tracking repository, and then does a \"git\n> gc\" may get a very unpleasant surprise.\n\nIt can also cause things like the \"please repack\" warning in git gui to\ngo off. This is especially unhelpful since they tend to tell you to go\nand do a gc to resolve the problem.\n"},{"id":"97977","messageId":"200812151759.44420.johan@herland.net","threadId":"16735","inReplyTo":"20081215161212.GE31145@sirena.org.uk","subject":"Re: \"git gc\" doesn't seem to remove loose objects any more","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2008-12-15T16:59:43Z","receivedAt":"2008-12-15T16:59:43Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Monday 15 December 2008, Mark Brown wrote:\n> On Mon, Dec 15, 2008 at 10:56:10AM -0500, Theodore Tso wrote:\n> > On Mon, Dec 15, 2008 at 03:08:34PM +0100, Bj?rn Steinbrink wrote:\n> > > To clarify that a bit more: git gc keeps unreachable objects\n> > > unpacked, so that git prune can drop them. And git gc invokes git\n> > > prune so that only unreachable objects older than 2 weeks are\n> > > dropped.\n> >\n> > To be even more explicit, \"git gc\" will **unpack** objects that\n> > have become unreachable and were currently in packs. As a result,\n> > the amount of disk space used by a git repository can actually go\n> > **up** dramatically after a \"git gc\" operation, which could be\n> > surprising for someone who is running close to full on their\n> > filesystem, deletes a number of branches from a tracking\n> > repository, and then does a \"git gc\" may get a very unpleasant\n> > surprise.\n>\n> It can also cause things like the \"please repack\" warning in git gui\n> to go off. This is especially unhelpful since they tend to tell you\n> to go and do a gc to resolve the problem.\n\nInstead of exploding all unreachable objects into loose objects, does it \nmake sense to repack them into a separate pack? AFAICS, that would \nsolve both the disk usage problem and the git-gui-\"please repack\" \nproblem. Also, it might make git-prune's job much easier, since \nunreachable objects are now located in a single pack only?\n\n\nHave fun!\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"97976","messageId":"237967ef0812150859g2c506820y740a25be194e9754@mail.gmail.com","threadId":"16735","inReplyTo":"20081215161212.GE31145@sirena.org.uk","subject":"Re: \"git gc\" doesn't seem to remove loose objects any more","fromName":"Mikael Magnusson","fromEmail":"mikachu@gmail.com","sentAt":"2008-12-15T16:59:44Z","receivedAt":"2008-12-15T16:59:44Z","isPatch":false,"sender":{"key":"mikachu@gmail.com","avatar":null},"body":"2008/12/15 Mark Brown :\n> On Mon, Dec 15, 2008 at 10:56:10AM -0500, Theodore Tso wrote:\n>> On Mon, Dec 15, 2008 at 03:08:34PM +0100, Bj?rn Steinbrink wrote:\n>\n>> > To clarify that a bit more: git gc keeps unreachable objects unpacked,\n>> > so that git prune can drop them. And git gc invokes git prune so that\n>> > only unreachable objects older than 2 weeks are dropped.\n>\n>> To be even more explicit, \"git gc\" will **unpack** objects that have\n>> become unreachable and were currently in packs. As a result, the\n>> amount of disk space used by a git repository can actually go **up**\n>> dramatically after a \"git gc\" operation, which could be surprising for\n>> someone who is running close to full on their filesystem, deletes a\n>> number of branches from a tracking repository, and then does a \"git\n>> gc\" may get a very unpleasant surprise.\n>\n> It can also cause things like the \"please repack\" warning in git gui to\n> go off. This is especially unhelpful since they tend to tell you to go\n> and do a gc to resolve the problem.\n\nA thought that occurs to me is to add some sort of flag to git count-objects\nthat prints the number of objects older than some interval in a separate field.\nThat way git gui would give less (maybe no) false alarms.\n\n-- \nMikael Magnusson\n"},{"id":"97979","messageId":"m3vdtlcqp6.fsf@localhost.localdomain","threadId":"16735","inReplyTo":"20081215155610.GA11502@mit.edu","subject":"Re: \"git gc\" doesn't seem to remove loose objects any more","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-12-15T17:07:39Z","receivedAt":"2008-12-15T17:07:39Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Theodore Tso writes:\n> On Mon, Dec 15, 2008 at 03:08:34PM +0100, Björn Steinbrink wrote:\n\n> > To clarify that a bit more: git gc keeps unreachable objects unpacked,\n> > so that git prune can drop them. And git gc invokes git prune so that\n> > only unreachable objects older than 2 weeks are dropped.\n> \n> To be even more explicit, \"git gc\" will **unpack** objects that have\n> become unreachable and were currently in packs. As a result, the\n> amount of disk space used by a git repository can actually go **up**\n> dramatically after a \"git gc\" operation, which could be surprising for\n> someone who is running close to full on their filesystem, deletes a\n> number of branches from a tracking repository, and then does a \"git\n> gc\" may get a very unpleasant surprise.\n> \n> A really good repository which shows this is linux-next, since it is\n> constantly getting rewound, and old branches are reserved via a tag\n> such as next-20081204. If you update the your local copy of the\n> linux-next repository every day, you will accumulate a large number of\n> these old branch tags. If you then delete a whole series of them, and\n> run git-gc, the operation will take quite a while, and the number of\n> blocks and inodes used will grow significantly. They will disappear\n> after a \"git prune\", but when I do this housekeeping operation, I've\n> often wished for a --yes-I-know-what-I-am-doing-and-it's-unsafe-but-\n> just-drop-the-unreachable-objects-cause-this-is-just-a-tracking-repository\n> option to \"git gc\".\n\nThere was an idea to have \"git gc --prune\" run \"git prune\"\nunconditionally, i.e. without grace period for dangling loose objects.\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"97980","messageId":"flFRWuefszqSabGvIxamBeKJiW26VnyHEVC9IErJWJA@cipher.nrlssc.navy.mil","threadId":"16735","inReplyTo":"20081215155610.GA11502@mit.edu","subject":"Re: \"git gc\" doesn't seem to remove loose objects any more","fromName":"Brandon Casey","fromEmail":"casey@nrlssc.navy.mil","sentAt":"2008-12-15T17:11:44Z","receivedAt":"2008-12-15T17:11:44Z","isPatch":false,"sender":{"key":"drafnel@gmail.com","avatar":"https://avatars.githubusercontent.com/u/921167?v=4"},"body":"Theodore Tso wrote:\n> I've\n> often wished for a --yes-I-know-what-I-am-doing-and-it's-unsafe-but-\n> just-drop-the-unreachable-objects-cause-this-is-just-a-tracking-repository\n> option to \"git gc\".\n\nrepack -a -d -l\n\nNotice the lowercase 'a'.\n\ngit-gc calls repack with uppercase 'A' which is what causes the unreachable\nobjects to be unpacked. Little 'a', is for people who know what they are\ndoing, and want git to just drop unreachable objects.\n\n-brandon\n"},{"id":"97983","messageId":"alpine.LFD.2.00.0812151105100.30035@xanadu.home","threadId":"16735","inReplyTo":"20081215155610.GA11502@mit.edu","subject":"[PATCH] objects to be pruned immediately don't have to be loosened","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-12-15T17:38:02Z","receivedAt":"2008-12-15T17:38:02Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"\nWhen there is no grace period before pruning unreferenced objects, it is \npointless to push those objects in their loose form just to delete them \nright away.\n\nAlso be more explicit about the possibility of using \"now\" in the \ngc.pruneexpire config variable (needed for the above behavior to \nhappen).\n\nSigned-off-by: Nicolas Pitre \n---\n\nOn Mon, 15 Dec 2008, Theodore Tso wrote:\n\n> On Mon, Dec 15, 2008 at 03:08:34PM +0100, Björn Steinbrink wrote:\n> > To clarify that a bit more: git gc keeps unreachable objects unpacked,\n> > so that git prune can drop them. And git gc invokes git prune so that\n> > only unreachable objects older than 2 weeks are dropped.\n> \n> To be even more explicit, \"git gc\" will **unpack** objects that have\n> become unreachable and were currently in packs. As a result, the\n> amount of disk space used by a git repository can actually go **up**\n> dramatically after a \"git gc\" operation, which could be surprising for\n> someone who is running close to full on their filesystem, deletes a\n> number of branches from a tracking repository, and then does a \"git\n> gc\" may get a very unpleasant surprise.\n> \n> A really good repository which shows this is linux-next, since it is\n> constantly getting rewound, and old branches are reserved via a tag\n> such as next-20081204. If you update the your local copy of the\n> linux-next repository every day, you will accumulate a large number of\n> these old branch tags. If you then delete a whole series of them, and\n> run git-gc, the operation will take quite a while, and the number of\n> blocks and inodes used will grow significantly. They will disappear\n> after a \"git prune\", but when I do this housekeeping operation, I've\n> often wished for a --yes-I-know-what-I-am-doing-and-it's-unsafe-but-\n> just-drop-the-unreachable-objects-cause-this-is-just-a-tracking-repository\n> option to \"git gc\".\n\nWhat about this?\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 21ea165..ca45e71 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -702,7 +702,9 @@ gc.packrefs::\n \n gc.pruneexpire::\n \tWhen 'git-gc' is run, it will call 'prune --expire 2.weeks.ago'.\n-\tOverride the grace period with this config variable.\n+\tOverride the grace period with this config variable. The value\n+\t\"now\" may be used to disable this grace period and always prune\n+\tunreachable objects immediately.\n \n gc.reflogexpire::\n \t'git-reflog expire' removes reflog entries older than\ndiff --git a/builtin-gc.c b/builtin-gc.c\nindex 781df60..f8eae4a 100644\n--- a/builtin-gc.c\n+++ b/builtin-gc.c\n@@ -188,7 +188,9 @@ static int need_to_gc(void)\n \t * there is no need.\n \t */\n \tif (too_many_packs())\n-\t\tappend_option(argv_repack, \"-A\", MAX_ADD);\n+\t\tappend_option(argv_repack,\n+\t\t\t !strcmp(prune_expire, \"now\") ? \"-a\" : \"-A\",\n+\t\t\t MAX_ADD);\n \telse if (!too_many_loose_objects())\n \t\treturn 0;\n \n@@ -243,7 +245,9 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\t\"run \\\"git gc\\\" manually. See \"\n \t\t\t\"\\\"git help gc\\\" for more information.\\n\");\n \t} else\n-\t\tappend_option(argv_repack, \"-A\", MAX_ADD);\n+\t\tappend_option(argv_repack,\n+\t\t\t !strcmp(prune_expire, \"now\") ? \"-a\" : \"-A\",\n+\t\t\t MAX_ADD);\n \n \tif (pack_refs && run_command_v_opt(argv_pack_refs, RUN_GIT_CMD))\n \t\treturn error(FAILED_RUN, argv_pack_refs[0]);\n"},{"id":"97995","messageId":"20081215193837.GB11502@mit.edu","threadId":"16735","inReplyTo":"m3vdtlcqp6.fsf@localhost.localdomain","subject":"Re: \"git gc\" doesn't seem to remove loose objects any more","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2008-12-15T19:38:37Z","receivedAt":"2008-12-15T19:38:37Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Mon, Dec 15, 2008 at 09:07:39AM -0800, Jakub Narebski wrote:\n> \n> There was an idea to have \"git gc --prune\" run \"git prune\"\n> unconditionally, i.e. without grace period for dangling loose objects.\n> \n\nThat doesn't help that much, since (temporarily) you still need all of\nthe disk space for the exploded, unpacked objects. As Brandon Casey\npointed out, the key is \"git repack -a -d -l\" vs \"git repack -A -d\n-l\". If there is going to be a git-gc option, it would need to change\nthe options sent to git-repack. Or, I suppose the answer is to tell\npeople who run into this problem use a plumbing command, manually.\nThe question is how common is the use case of needing to gc a\nrepository like linux-next, I suppose.\n\n\t\t\t\t\t\t\t- Ted\n"}]}