{"thread":{"id":"29254","subject":"[PATCH] gc --auto: warn garbage collection happens soon","startedAt":"2011-12-27T13:45:34Z","lastAt":"2011-12-29T18:29:24Z","messageCount":8,"participants":["Nguyễn Thái Ngọc Duy","Junio C Hamano","Jeff King","Nguyen Thai Ngoc Duy","Mark Brown"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"181713","messageId":"1324993534-16307-1-git-send-email-pclouds@gmail.com","threadId":"29254","inReplyTo":null,"subject":"[PATCH] gc --auto: warn garbage collection happens soon","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2011-12-27T13:45:34Z","receivedAt":"2011-12-27T13:45:34Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This gives users a chance to run gc explicitly elsewhere if they do not\nwant gc to run suddenly in current terminal.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n v2 of a patch posted a few months ago. The warning limits are in\n percentage and configurable. I could have set the default limits to\n 100% (i.e. no warnings) to keep current behavior. However I think\n warning is better.\n\n May need rewording inn config.txt, I'm not sure I state it clearly.\n\n Documentation/config.txt | 12 ++++++++++++\n Documentation/git-gc.txt | 4 ++++\n builtin/gc.c | 41 +++++++++++++++++++++++++++++++++++++++--\n 3 files changed, 55 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 5a841da..c263496 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -965,12 +965,24 @@ gc.auto::\n \tlight-weight garbage collection from time to time. The\n \tdefault value is 6700. Setting this to 0 disables it.\n \n+gc.autowarn::\n+\tThe percentage of loose objects specified in `gc.auto`. If the\n+\tnumber of loose objects exceeds this limit, `git gc --auto`\n+\twill warn users garbage collection will happen soon. Default\n+\tvalue is 90. Setting this to 100 disables it.\n+\n gc.autopacklimit::\n \tWhen there are more than this many packs that are not\n \tmarked with `*.keep` file in the repository, `git gc\n \t--auto` consolidates them into one larger pack. The\n \tdefault\tvalue is 50. Setting this to 0 disables it.\n \n+gc.autopackwarn::\n+\tThe percentage of packs specified in `gc.autopacklimit`. If\n+\tthe number of packs exceeds this limit, `git gc --auto` will\n+\twarn users garbage collection will happen soon. Default value\n+\tis 90. Setting this to 100 disables it.\n+\n gc.packrefs::\n \tRunning `git pack-refs` in a repository renders it\n \tunclonable by Git versions prior to 1.5.1.2 over dumb\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 815afcb..937b3d6 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -59,6 +59,10 @@ then existing packs (except those marked with a `.keep` file)\n are consolidated into a single pack by using the `-A` option of\n 'git repack'. Setting `gc.autopacklimit` to 0 disables\n automatic consolidation of packs.\n++\n+`git gc --auto` will warn users when the number of loose objects or\n+packs is close to the limits. See `gc.autowarn` and `gc.autopackwarn`\n+for details.\n \n --prune=::\n \tPrune loose objects older than date (default is 2 weeks ago,\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 0498094..f3fa46d 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -28,6 +28,10 @@ static int gc_auto_threshold = 6700;\n static int gc_auto_pack_limit = 50;\n static const char *prune_expire = \"2.weeks.ago\";\n \n+/* numbers are in percent, to be converted to absolute later */\n+static int gc_warn_auto_threshold = 90;\n+static int gc_warn_auto_pack_limit = 90;\n+\n #define MAX_ADD 10\n static const char *argv_pack_refs[] = {\"pack-refs\", \"--all\", \"--prune\", NULL};\n static const char *argv_reflog[] = {\"reflog\", \"expire\", \"--all\", NULL};\n@@ -52,10 +56,26 @@ static int gc_config(const char *var, const char *value, void *cb)\n \t\tgc_auto_threshold = git_config_int(var, value);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(var, \"gc.autowarn\")) {\n+\t\tint percent = percent = git_config_int(var, value);\n+\t\tif (percent <= 0 || percent > 100)\n+\t\t\tdie(_(\"gc.autowarn %d%% does not make sense\"),\n+\t\t\t percent);\n+\t\tgc_warn_auto_threshold = percent;\n+\t\treturn 0;\n+\t}\n \tif (!strcmp(var, \"gc.autopacklimit\")) {\n \t\tgc_auto_pack_limit = git_config_int(var, value);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(var, \"gc.autopackwarn\")) {\n+\t\tint percent = percent = git_config_int(var, value);\n+\t\tif (percent <= 0 || percent > 100)\n+\t\t\tdie(_(\"gc.autopackwarn %d%% does not make sense\"),\n+\t\t\t percent);\n+\t\tgc_warn_auto_pack_limit = percent;\n+\t\treturn 0;\n+\t}\n \tif (!strcmp(var, \"gc.pruneexpire\")) {\n \t\tif (value && strcmp(value, \"now\")) {\n \t\t\tunsigned long now = approxidate(\"now\");\n@@ -118,7 +138,15 @@ static int too_many_loose_objects(void)\n \t\t}\n \t}\n \tclosedir(dir);\n-\treturn needed;\n+\tif (needed)\n+\t\treturn 1;\n+\n+\tauto_threshold = (gc_warn_auto_threshold + 255) / 256;\n+\tif (num_loose >= auto_threshold)\n+\t\twarning(_(\"Too many loose objects (current approx. %d, limit %d).\\n\"\n+\t\t\t \"\\\"git gc\\\" will soon run automatically\"),\n+\t\t\tnum_loose * 256, gc_auto_threshold);\n+\treturn 0;\n }\n \n static int too_many_packs(void)\n@@ -141,7 +169,14 @@ static int too_many_packs(void)\n \t\t */\n \t\tcnt++;\n \t}\n-\treturn gc_auto_pack_limit <= cnt;\n+\tif (gc_auto_pack_limit <= cnt)\n+\t\treturn 1;\n+\n+\tif (gc_warn_auto_pack_limit <= cnt)\n+\t\twarning(_(\"Too many packs (current %d, limit %d)\\n\"\n+\t\t\t \"\\\"git gc\\\" will soon run automatically.\"),\n+\t\t\tcnt, gc_auto_pack_limit);\n+\treturn 0;\n }\n \n static int need_to_gc(void)\n@@ -193,6 +228,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_gc_usage, builtin_gc_options);\n \n \tgit_config(gc_config, NULL);\n+\tgc_warn_auto_threshold = 0.01 * gc_auto_threshold * gc_warn_auto_threshold;\n+\tgc_warn_auto_pack_limit = 0.01 * gc_auto_pack_limit * gc_auto_pack_limit;\n \n \tif (pack_refs < 0)\n \t\tpack_refs = !is_bare_repository();\n-- \n1.7.8.36.g69ee2\n"},{"id":"181735","messageId":"7vpqf94r8c.fsf@alter.siamese.dyndns.org","threadId":"29254","inReplyTo":"1324993534-16307-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH] gc --auto: warn garbage collection happens soon","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-27T21:52:35Z","receivedAt":"2011-12-27T21:52:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> This gives users a chance to run gc explicitly elsewhere if they do not\n> want gc to run suddenly in current terminal.\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n\nAs I am still in a cheerly holiday mood, let's be a bit philosophical,\nstep back a bit and think.\n\nAfter this patch gets applied, will the users start feeling bothered by\nrepeated \"you will soon see auto-gc\" messages and will want \"you will soon\nstart seeing the you will soon see auto-gc messages\" warnings?\n\nAnd if the answer to that tongue-in-cheek question is no, what is the\nreason why the users will not find the messages disturbing, while loathing\nthe auto-gc?\n\nI suspect that is because auto-gc takes long time, making the user wait,\ncompared to the new message that may be noisy but quick. Perhaps the real\ncure for the disease is not to add the message but to make an auto-gc less\npainful, no?\n\nWhat are the things we could do to make auto-gc less painful?\n\nAre we doing something that is not necessary in auto-gc that takes time\nbut that we can live without doing?\n\nIt may be a better cure for the disease to force a full gc after\noperations that we know the users already know to take long time (e.g. a\nclone, a large fetch), so that the next auto-gc do not have to do much\nwork.\n"},{"id":"181754","messageId":"20111228184000.GA18780@sigill.intra.peff.net","threadId":"29254","inReplyTo":"7vpqf94r8c.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] gc --auto: warn garbage collection happens soon","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-12-28T18:40:00Z","receivedAt":"2011-12-28T18:40:00Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Dec 27, 2011 at 01:52:35PM -0800, Junio C Hamano wrote:\n\n> And if the answer to that tongue-in-cheek question is no, what is the\n> reason why the users will not find the messages disturbing, while loathing\n> the auto-gc?\n> \n> I suspect that is because auto-gc takes long time, making the user wait,\n> compared to the new message that may be noisy but quick. Perhaps the real\n> cure for the disease is not to add the message but to make an auto-gc less\n> painful, no?\n> \n> What are the things we could do to make auto-gc less painful?\n>\n> Are we doing something that is not necessary in auto-gc that takes time\n> but that we can live without doing?\n\nI don't personally find gc all that painful (though maybe that is\nbecause I tend to gc myself and rarely hit the auto-gc), but I have\nnoticed that git-prune takes by far the most time to run. If you are\njust doing an incremental pack, you might be packing only a few thousand\nobjects and not touching old history at all (and with many cores, the\ndelta compression flies by). But prune requires running \"git rev-list\n--objects --all\", which takes something like 45 seconds for linux-2.6 on\nmy fast-ish laptop (and about 23 seconds for git.git).\n\nWe could perhaps cut out pruning in the auto-gc case unless there are a\nlot of objects left over after the packing phase. It's not worth doing a\nfull prune to clean up a dozen objects[1]. It probably is if you have a\nthousand objects left after packing.\n\n-Peff\n\n[1] Actually, it's not just having objects. You may have just exploded\n unreachable objects from a pack, but they are still younger than the\n 2 week expiration period. Therefore trying to prune them is\n pointless, because even if they are unreachable, you won't delete\n them. So you really want to say \"how many actual candidate objects\n do we have for pruning?\"\n"},{"id":"181758","messageId":"7vfwg41n3p.fsf@alter.siamese.dyndns.org","threadId":"29254","inReplyTo":"20111228184000.GA18780@sigill.intra.peff.net","subject":"Re: [PATCH] gc --auto: warn garbage collection happens soon","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-28T20:02:18Z","receivedAt":"2011-12-28T20:02:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> [1] Actually, it's not just having objects. You may have just exploded\n> unreachable objects from a pack, but they are still younger than the\n> 2 week expiration period. Therefore trying to prune them is\n> pointless, because even if they are unreachable, you won't delete\n> them. So you really want to say \"how many actual candidate objects\n> do we have for pruning?\"\n\nAn obvious knee-jerk reaction is \"Ugh, if we have very recently repacked,\ndon't we know what are reachable and what are not already, and use that\nknowledge while pruning to avoid traversing everything again?\"\n\nMy memory around repack, fsck and prune needs refreshing, though, to tell\nif that suggestion is feasible.\n"},{"id":"181761","messageId":"20111228213018.GA22811@sigill.intra.peff.net","threadId":"29254","inReplyTo":"7vfwg41n3p.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] gc --auto: warn garbage collection happens soon","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-12-28T21:30:18Z","receivedAt":"2011-12-28T21:30:18Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 28, 2011 at 12:02:18PM -0800, Junio C Hamano wrote:\n\n> Jeff King writes:\n> \n> > [1] Actually, it's not just having objects. You may have just exploded\n> > unreachable objects from a pack, but they are still younger than the\n> > 2 week expiration period. Therefore trying to prune them is\n> > pointless, because even if they are unreachable, you won't delete\n> > them. So you really want to say \"how many actual candidate objects\n> > do we have for pruning?\"\n> \n> An obvious knee-jerk reaction is \"Ugh, if we have very recently repacked,\n> don't we know what are reachable and what are not already, and use that\n> knowledge while pruning to avoid traversing everything again?\"\n\nEspecially now that prune has learned about progress reporting, it's\neasy to see in \"git gc\" that the \"Counting objects\" phase of the repack\nand the connectivity search in prune are counting the same objects. It\nwould obviously be easy to just dump the set of sha1s in packed binary\nformat, and let git-prune reference that.\n\nBut it doesn't work in the general case. Running \"git gc\" will repack\neverything, and so it looks at all reachable objects. But \"git gc\n--auto\" will typically do an incremental pack (unless you have too many\npacks), which means its counting objects phase only looks at part of\nthe graph. So that result can't be used for object reachability, since\nmany objects won't be marked[1].\n\nSo yes, it's an optimization we can do, but it only works some of the\ntime. And worse, it works in the time we care less (when we are doing a\nfull repack anyway, so we are already spending more time counting\nobjects, and more I/O rewriting existing packed objects), but not when\nwe want it most (doing a few seconds of incremental repack during \"git\ngc --auto\", which balloons to a minute because of the prune time).\n\n-Peff\n\n[1] It's tempting to say \"well, we just repacked incrementally, so if\n something was referenced and not packed, we would have just packed\n it, right?\" But look at how incremental packing works. We do a\n traversal with \"--unpacked\", which means we don't dig down past\n commit objects that are already packed. And that's why its fast.\n\n But packs don't necessarily respect reachability. It's possible for\n you to have object X in a pack, but X^{tree} is not (or X^, or\n whatever)[2]. I believe using \"git repack\" would fail to actually\n pack that. But that's OK, because it almost never happens, and the\n worst case is that the object doesn't get packed until you do a full\n repack.\n\n But I'm not sure you would want the same level of shortcut for\n git-prune, which would actually be _deleting_ the object. We want to\n be very sure in that case.\n\n[2] The obvious way to get into this situation is to give weird rev-list\n parameters to pack-objects. But I think you could also do it\n accidentally by having commit X loose, then fetching history\n containing commit Y that builds on X. If the fetch is big enough,\n we'll keep the pack that we got from the other side. So X remains\n loose, but its ancestors are packed. Running an incremental repack\n will stop the traversal at Y and never consider X for packing.\n\n I didn't actually test this, but that's my reading of the code (see\n the revs->unpacked check in revision.c:get_commit_action).\n"},{"id":"181764","messageId":"CACsJy8BVWQHUfi3=fMiqFAfbFyTAV0LnY0yF0AbD_weT4bX6Hw@mail.gmail.com","threadId":"29254","inReplyTo":"7vpqf94r8c.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] gc --auto: warn garbage collection happens soon","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2011-12-28T21:50:49Z","receivedAt":"2011-12-28T21:50:49Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"2011/12/28 Junio C Hamano :\n> Nguyễn Thái Ngọc Duy   writes:\n>\n>> This gives users a chance to run gc explicitly elsewhere if they do not\n>> want gc to run suddenly in current terminal.\n>>\n>> Signed-off-by: Nguyễn Thái Ngọc Duy \n>\n> As I am still in a cheerly holiday mood, let's be a bit philosophical,\n> step back a bit and think.\n>\n> After this patch gets applied, will the users start feeling bothered by\n> repeated \"you will soon see auto-gc\" messages and will want \"you will soon\n> start seeing the you will soon see auto-gc messages\" warnings?\n\nThey should not for most of the time, given the default settings is\nwarnings at 90% limits. If they do feel bothered, they could turn it\noff or just run \"gc\".\n\n> And if the answer to that tongue-in-cheek question is no, what is the\n> reason why the users will not find the messages disturbing, while loathing\n> the auto-gc?\n>\n> I suspect that is because auto-gc takes long time, making the user wait,\n> compared to the new message that may be noisy but quick.  Perhaps the real\n> cure for the disease is not to add the message but to make an auto-gc less\n> painful, no?\n\nIt's something with expected run time of a command. When I'm about to\nrun \"commit\", I know the command is fast and I expect the shell prompt\nsoon. When I run \"fetch\", I know it may take a bit (or a lot) of time\nand I will be ready to make myself a cup of coffee while it's running.\n\nauto-gc is an unknown factor and may break my expectations. I would\nnot mind if auto-gc is extremely fast, e.g. a couple of seconds\nmaximum. But gc time seems to be proportional to repository size.\n\n> What are the things we could do to make auto-gc less painful?\n>\n> Are we doing something that is not necessary in auto-gc that takes time\n> but that we can live without doing?\n>\n> It may be a better cure for the disease to force a full gc after\n> operations that we know the users already know to take long time (e.g. a\n> clone, a large fetch), so that the next auto-gc do not have to do much\n> work.\n\ngit works best when everything is in one pack. So while we may be able\nto skip stuff and make auto-gc fast the first few times, eventually we\nneed to do something like \"git repack -ad\" as part of auto-gc. I don't\nsee any way to make that part complete in a few secs regardless repo\nsize (unless packv4 comes in time and speeds up revlist\nsignificantly). So the pain will be there in the end, it's just\ndelayed.\n\nThere's another possibility (but not sure if it's feasible): to make\nauto-gc use up to certain amount of time. If it runs out of allocated\ntime, it needs to save its state somewhere, somehow and resumes in\nnext auto-gc.\n-- \nDuy\n"},{"id":"181765","messageId":"CACsJy8BSewZuk-aoorqXRtD=nhPV=TY8YgjFj9uGMivv_GTQSA@mail.gmail.com","threadId":"29254","inReplyTo":"20111228213018.GA22811@sigill.intra.peff.net","subject":"Re: [PATCH] gc --auto: warn garbage collection happens soon","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2011-12-28T22:09:26Z","receivedAt":"2011-12-28T22:09:26Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"2011/12/29 Jeff King :\n> Especially now that prune has learned about progress reporting, it's\n> easy to see in \"git gc\" that the \"Counting objects\" phase of the repack\n> and the connectivity search in prune are counting the same objects.  It\n> would obviously be easy to just dump the set of sha1s in packed binary\n> format, and let git-prune reference that.\n>\n> But it doesn't work in the general case. Running \"git gc\" will repack\n> everything, and so it looks at all reachable objects. But \"git gc\n> --auto\" will typically do an incremental pack (unless you have too many\n> packs), which means its counting objects phase only looks at part of\n> the graph.  So that result can't be used for object reachability, since\n> many objects won't be marked[1].\n\nHmm.. I was thinking of sharing this \"counting objects\" part when\nrepack is rewritten in C. I guess I can drop the idea now.\n-- \nDuy\n"},{"id":"181783","messageId":"20111229182924.GA32392@sirena.org.uk","threadId":"29254","inReplyTo":"7vpqf94r8c.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] gc --auto: warn garbage collection happens soon","fromName":"Mark Brown","fromEmail":"broonie@opensource.wolfsonmicro.com","sentAt":"2011-12-29T18:29:24Z","receivedAt":"2011-12-29T18:29:24Z","isPatch":true,"sender":{"key":"broonie@opensource.wolfsonmicro.com","avatar":"https://gravatar.com/avatar/5fb25e4e0de3255caa21123e2b518c314d26245069221ff55910d5c6ba3343c4?d=mp&s=160"},"body":"On Tue, Dec 27, 2011 at 01:52:35PM -0800, Junio C Hamano wrote:\n\n> And if the answer to that tongue-in-cheek question is no, what is the\n> reason why the users will not find the messages disturbing, while loathing\n> the auto-gc?\n\nThe main problem I've noticed with the auto gc is that git gui seems to\nwant to do one at a much lower threashold than the command line tools\n(and far too aggressive), it seems that the logic that determines when\nto do one isn't quite in agreement within all the git tools.\n"}]}