{"thread":{"id":"38846","subject":"git pull & git gc","startedAt":"2015-03-18T13:53:49Z","lastAt":"2015-03-19T09:47:50Z","messageCount":21,"participants":["Дилян Палаузов","Duy Nguyen","John Keeping","Jeff King","Mike Hommey","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"257953","messageId":"5509836D.2020304@aegee.org","threadId":"38846","inReplyTo":null,"subject":"git pull & git gc","fromName":"Дилян Палаузов","fromEmail":"dilyan.palauzov@aegee.org","sentAt":"2015-03-18T13:53:49Z","receivedAt":"2015-03-18T13:53:49Z","isPatch":false,"sender":{"key":"dilyan.palauzov@aegee.org","avatar":null},"body":"Hello,\n\nI have a local folder with the git-repository (so that its .git/config \ncontains ([remote \"origin\"]\\n\turl = git://github.com/git/git.git\\nfetch \n= +refs/heads/*:refs/remotes/origin/* )\n\nI do there \"git pull\".\n\nUsually the output is\n Already up to date\n\nbut since today it prints\n Auto packing the repository in background for optimum performance.\n See \"git help gc\" for manual housekeeping.\n Already up-to-date.\n\nand starts in the background a \"git gc --auto\" process. This is all \nfine, however, when the \"git gc\" process finishes, and I do again \"git \npull\" I get the same message, as above (git gc is again started).\n\nMy understanding is, that \"git gc\" has to be occasionally run and then \nthe garbage collection is done for a while. In the concrete case, if \n\"git pull\" starts \"git gc\" in the background and prints a message on \nthis, it is all fine, but running \"git pull\" after a while, when the \ngarbage collection was recently done, where shall be neither message nor \nan action about \"git gc\".\n\nMy system-wide gitconfig contains \"[pack] threads = 1\".\n\nI have \"tar xJf\"'ed my local git repository and have put it under\n http://mail.aegee.org/dpa/v/git-repository.tar.xz\n\nThe question is:\n\nWhy does \"git pull\" every time when it is invoked today print \ninformation about \"git gc\"?\n\nI have git 2.3.3 adjusted with \"./configure --with-openssl \n--with-libpcre --with-curl --with-expat\".\n\nThanks in advance for your answer\n\nDilian\n"},{"id":"257954","messageId":"CACsJy8ChKgwr5OBRDG1p5nxwPC0vei7J=Lr3NefLYoReTMuhPA@mail.gmail.com","threadId":"38846","inReplyTo":"5509836D.2020304@aegee.org","subject":"Re: git pull & git gc","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-03-18T14:16:54Z","receivedAt":"2015-03-18T14:16:54Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Mar 18, 2015 at 8:53 PM, Дилян Палаузов\n wrote:\n> Hello,\n>\n> I have a local folder with the git-repository (so that its .git/config\n> contains ([remote \"origin\"]\\n url = git://github.com/git/git.git\\nfetch =\n> +refs/heads/*:refs/remotes/origin/* )\n>\n> I do there \"git pull\".\n>\n> Usually the output is\n> Already up to date\n>\n> but since today it prints\n> Auto packing the repository in background for optimum performance.\n> See \"git help gc\" for manual housekeeping.\n> Already up-to-date.\n>\n> and starts in the background a \"git gc --auto\" process. This is all fine,\n> however, when the \"git gc\" process finishes, and I do again \"git pull\" I get\n> the same message, as above (git gc is again started).\n\nSo if you do \"git gc --auto\" now, does it exit immediately or go\nthrough the garbage collection process again (it'll print something)?\nWhat does \"git count-objects -v\" show?\n-- \nDuy\n"},{"id":"257955","messageId":"55098A49.1020003@aegee.org","threadId":"38846","inReplyTo":"CACsJy8ChKgwr5OBRDG1p5nxwPC0vei7J=Lr3NefLYoReTMuhPA@mail.gmail.com","subject":"Re: git pull & git gc","fromName":"Дилян Палаузов","fromEmail":"dilyan.palauzov@aegee.org","sentAt":"2015-03-18T14:23:05Z","receivedAt":"2015-03-18T14:23:05Z","isPatch":false,"sender":{"key":"dilyan.palauzov@aegee.org","avatar":null},"body":"Hello,\n\n# git gc --auto\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\n\nand calls in the background:\n\n25618 1 0 32451 884 1 14:20 ? 00:00:00 git gc --auto\n25639 25618 51 49076 49428 0 14:20 ? 00:00:07 git prune \n--expire 2.weeks.ago\n\n# git count-objects -v\ncount: 6039\nsize: 65464\nin-pack: 185432\npacks: 1\nsize-pack: 46687\nprune-packable: 0\ngarbage: 0\nsize-garbage: 0\n\nRegards\n Dilian\n\n\nOn 18.03.2015 15:16, Duy Nguyen wrote:\n> On Wed, Mar 18, 2015 at 8:53 PM, Дилян Палаузов\n> wrote:\n>> Hello,\n>>\n>> I have a local folder with the git-repository (so that its .git/config\n>> contains ([remote \"origin\"]\\n url = git://github.com/git/git.git\\nfetch =\n>> +refs/heads/*:refs/remotes/origin/* )\n>>\n>> I do there \"git pull\".\n>>\n>> Usually the output is\n>> Already up to date\n>>\n>> but since today it prints\n>> Auto packing the repository in background for optimum performance.\n>> See \"git help gc\" for manual housekeeping.\n>> Already up-to-date.\n>>\n>> and starts in the background a \"git gc --auto\" process. This is all fine,\n>> however, when the \"git gc\" process finishes, and I do again \"git pull\" I get\n>> the same message, as above (git gc is again started).\n>\n> So if you do \"git gc --auto\" now, does it exit immediately or go\n> through the garbage collection process again (it'll print something)?\n> What does \"git count-objects -v\" show?\n>\n"},{"id":"257956","messageId":"CACsJy8AtoeSv7uZc3P2xXN0A15gpj-OUTRyNhMe__=HqtTkD2Q@mail.gmail.com","threadId":"38846","inReplyTo":"55098A49.1020003@aegee.org","subject":"Re: git pull & git gc","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-03-18T14:33:29Z","receivedAt":"2015-03-18T14:33:29Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Mar 18, 2015 at 9:23 PM, Дилян Палаузов\n wrote:\n> Hello,\n>\n> # git gc --auto\n> Auto packing the repository in background for optimum performance.\n> See \"git help gc\" for manual housekeeping.\n>\n> and calls in the background:\n>\n> 25618 1 0 32451 884 1 14:20 ? 00:00:00 git gc --auto\n> 25639 25618 51 49076 49428 0 14:20 ? 00:00:07 git prune --expire\n> 2.weeks.ago\n>\n> # git count-objects -v\n> count: 6039\n\nloose number threshold is 6700, unless you tweaked something. But\nthere's a tweak, we'll come back to this.\n\n> size: 65464\n> in-pack: 185432\n> packs: 1\n\nPack threshold is 50, You only have one pack, good\n\nOK back to the \"count 6039\" above. You have that many loose objects.\nBut 'git gc' is lazier than 'git count-objects'. It assume a flat\ndistribution, and only counts the number of objects in .git/objects/17\ndirectory only, then extrapolate for the total number.\n\nSo can you see how many files you have in this directory\n.git/objects/17? That number, multiplied by 256, should be greater\nthan 6700. If that's the case \"git gc\" laziness is the problem. If\nnot, I made some mistake in analyzing this and we'll start again.\n-- \nDuy\n"},{"id":"257957","messageId":"CACsJy8DCdF1YmfFF1aCuhMJ0v4+FW7EkDiwBnBF2OFhE=kcbFg@mail.gmail.com","threadId":"38846","inReplyTo":"CACsJy8AtoeSv7uZc3P2xXN0A15gpj-OUTRyNhMe__=HqtTkD2Q@mail.gmail.com","subject":"Re: git pull & git gc","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-03-18T14:41:59Z","receivedAt":"2015-03-18T14:41:59Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Mar 18, 2015 at 9:33 PM, Duy Nguyen wrote:\n> If not, I made some mistake in analyzing this and we'll start again.\n\nI did make one mistake, the first \"gc\" should have reduced the number\nof loose objects to zero. Why didn't it.? I'll come back to this\ntomorrow if nobody finds out first :)\n-- \nDuy\n"},{"id":"257958","messageId":"5509904A.2000606@aegee.org","threadId":"38846","inReplyTo":"CACsJy8AtoeSv7uZc3P2xXN0A15gpj-OUTRyNhMe__=HqtTkD2Q@mail.gmail.com","subject":"Re: git pull & git gc","fromName":"Дилян Палаузов","fromEmail":"dilyan.palauzov@aegee.org","sentAt":"2015-03-18T14:48:42Z","receivedAt":"2015-03-18T14:48:42Z","isPatch":false,"sender":{"key":"dilyan.palauzov@aegee.org","avatar":null},"body":"Hello Duy,\n\n#ls .git/objects/17/* | wc -l\n30\n\n30 * 256 = 7 680 > 6 700\n\nAnd now? Do I have to run git gc --aggressive ?\n\nKind regards\n Dilian\n\n\nOn 18.03.2015 15:33, Duy Nguyen wrote:\n> On Wed, Mar 18, 2015 at 9:23 PM, Дилян Палаузов\n> wrote:\n>> Hello,\n>>\n>> # git gc --auto\n>> Auto packing the repository in background for optimum performance.\n>> See \"git help gc\" for manual housekeeping.\n>>\n>> and calls in the background:\n>>\n>> 25618 1 0 32451 884 1 14:20 ? 00:00:00 git gc --auto\n>> 25639 25618 51 49076 49428 0 14:20 ? 00:00:07 git prune --expire\n>> 2.weeks.ago\n>>\n>> # git count-objects -v\n>> count: 6039\n>\n> loose number threshold is 6700, unless you tweaked something. But\n> there's a tweak, we'll come back to this.\n>\n>> size: 65464\n>> in-pack: 185432\n>> packs: 1\n>\n> Pack threshold is 50, You only have one pack, good\n>\n> OK back to the \"count 6039\" above. You have that many loose objects.\n> But 'git gc' is lazier than 'git count-objects'. It assume a flat\n> distribution, and only counts the number of objects in .git/objects/17\n> directory only, then extrapolate for the total number.\n>\n> So can you see how many files you have in this directory\n> .git/objects/17? That number, multiplied by 256, should be greater\n> than 6700. If that's the case \"git gc\" laziness is the problem. If\n> not, I made some mistake in analyzing this and we'll start again.\n>\n"},{"id":"257959","messageId":"20150318145815.GN1369@serenity.lan","threadId":"38846","inReplyTo":"CACsJy8DCdF1YmfFF1aCuhMJ0v4+FW7EkDiwBnBF2OFhE=kcbFg@mail.gmail.com","subject":"Re: git pull & git gc","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2015-03-18T14:58:15Z","receivedAt":"2015-03-18T14:58:15Z","isPatch":false,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Wed, Mar 18, 2015 at 09:41:59PM +0700, Duy Nguyen wrote:\n> On Wed, Mar 18, 2015 at 9:33 PM, Duy Nguyen wrote:\n> > If not, I made some mistake in analyzing this and we'll start again.\n> \n> I did make one mistake, the first \"gc\" should have reduced the number\n> of loose objects to zero. Why didn't it.? I'll come back to this\n> tomorrow if nobody finds out first :)\n\nMost likely they are not referenced by anything but are younger than 2\nweeks.\n\nI saw a similar issue with automatic gc triggering after every operation\nwhen I did something equivalent to:\n\n\tgit add \n\tgit commit\n\tgit reset --hard HEAD^\n\nwhich creates a log of unreachable objects which are not old enough to\nbe pruned.\n"},{"id":"257979","messageId":"20150318210458.GA3376@peff.net","threadId":"38846","inReplyTo":"20150318145815.GN1369@serenity.lan","subject":"Re: git pull & git gc","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-03-18T21:04:58Z","receivedAt":"2015-03-18T21:04:58Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 18, 2015 at 02:58:15PM +0000, John Keeping wrote:\n\n> On Wed, Mar 18, 2015 at 09:41:59PM +0700, Duy Nguyen wrote:\n> > On Wed, Mar 18, 2015 at 9:33 PM, Duy Nguyen wrote:\n> > > If not, I made some mistake in analyzing this and we'll start again.\n> > \n> > I did make one mistake, the first \"gc\" should have reduced the number\n> > of loose objects to zero. Why didn't it.? I'll come back to this\n> > tomorrow if nobody finds out first :)\n> \n> Most likely they are not referenced by anything but are younger than 2\n> weeks.\n> \n> I saw a similar issue with automatic gc triggering after every operation\n> when I did something equivalent to:\n> \n> \tgit add \n> \tgit commit\n> \tgit reset --hard HEAD^\n> \n> which creates a log of unreachable objects which are not old enough to\n> be pruned.\n\nYes, this is almost certainly the problem. Though to be pedantic, the\ncommand above will still have a reflog entry, so the objects will be\nreachable (and packed). But there are other variants that don't leave\nthe objects reachable from even reflogs.\n\nI don't know if there is an easy way around this. Auto-gc's object count\nis making the assumption that running the gc will reduce the number of\nobjects, but obviously it does not always do so. We could do a more\nthorough check and find the number of actual packable and prune-able\nobjects. The \"prune-able\" part of that is easy; just omit objects from\nthe count that are newer than 2 weeks. But \"packable\" is expensive. You\nwould have to compute reachability by walking from the tips. That can\ntake tens of seconds on a large repo.\n\nYou could perhaps cut off the walk early when you hit a packed commit\n(this does not strictly imply that all of the related objects are\npacked, but it would be good enough for a heuristic). But even that is\nprobably too expensive for \"gc --auto\".\n\n-Peff\n\nPS Note that in git v2.2.0 and up, prune will leave not only \"recent\"\n unreachable objects, but older objects which are reachable from those\n recent ones (so that we keep or prune whole chunks of history, rather\n than dropping part and leaving the rest broken). Technically this\n exacerbates the problem (we keep more objects), though I doubt it\n makes much difference in practice (most chunks of history were\n created at similar times, so the mtimes of the whole chunk will be\n close together).\n"},{"id":"257981","messageId":"20150318210706.GB3376@peff.net","threadId":"38846","inReplyTo":"5509904A.2000606@aegee.org","subject":"Re: git pull & git gc","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-03-18T21:07:06Z","receivedAt":"2015-03-18T21:07:06Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 18, 2015 at 03:48:42PM +0100, Дилян Палаузов wrote:\n\n> #ls .git/objects/17/* | wc -l\n> 30\n> \n> 30 * 256 = 7 680 > 6 700\n> \n> And now? Do I have to run git gc --aggressive ?\n\nNo, aggressive just controls the time we spend on repacking. If the\nguess is correct that the objects are kept because they are unreachable\nbut \"recent\", then shortening the prune expiration time would get rid of\nthem. E.g., \"git gc --prune=1.hour.ago\".\n\nThat does not solve the underlying problem discussed elsewhere in the\nthread, but it would make this particular instance of it go away. :)\n\n-Peff\n"},{"id":"257997","messageId":"CACsJy8AJfx8id+oHmhsLODBJeNLgRqD+5B295TtS+c-hrC5raQ@mail.gmail.com","threadId":"38846","inReplyTo":"20150318210458.GA3376@peff.net","subject":"Re: git pull & git gc","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-03-19T00:31:48Z","receivedAt":"2015-03-19T00:31:48Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Mar 19, 2015 at 4:04 AM, Jeff King wrote:\n> On Wed, Mar 18, 2015 at 02:58:15PM +0000, John Keeping wrote:\n>\n>> On Wed, Mar 18, 2015 at 09:41:59PM +0700, Duy Nguyen wrote:\n>> > On Wed, Mar 18, 2015 at 9:33 PM, Duy Nguyen wrote:\n>> > > If not, I made some mistake in analyzing this and we'll start again.\n>> >\n>> > I did make one mistake, the first \"gc\" should have reduced the number\n>> > of loose objects to zero. Why didn't it.? I'll come back to this\n>> > tomorrow if nobody finds out first :)\n>>\n>> Most likely they are not referenced by anything but are younger than 2\n>> weeks.\n>>\n>> I saw a similar issue with automatic gc triggering after every operation\n>> when I did something equivalent to:\n>>\n>> git add \n>> git commit\n>> git reset --hard HEAD^\n>>\n>> which creates a log of unreachable objects which are not old enough to\n>> be pruned.\n>\n> Yes, this is almost certainly the problem. Though to be pedantic, the\n> command above will still have a reflog entry, so the objects will be\n> reachable (and packed). But there are other variants that don't leave\n> the objects reachable from even reflogs.\n>\n> I don't know if there is an easy way around this. Auto-gc's object count\n> is making the assumption that running the gc will reduce the number of\n> objects, but obviously it does not always do so. We could do a more\n> thorough check and find the number of actual packable and prune-able\n> objects. The \"prune-able\" part of that is easy; just omit objects from\n> the count that are newer than 2 weeks. But \"packable\" is expensive. You\n> would have to compute reachability by walking from the tips. That can\n> take tens of seconds on a large repo.\n\nOr we could count/estimate the number of loose objects again after\nrepack/prune. Then we can maybe have a way to prevent the next gc that\nwe know will not improve the situation anyway. One option is pack\nunreachable objects in the second pack. This would stop the next gc,\nbut that would screw prune up because st_mtime info is gone.. Maybe we\njust save a file to tell gc to ignore the number of loose objects\nuntil after a specific date.\n-- \nDuy\n"},{"id":"257998","messageId":"20150319012722.GA26867@peff.net","threadId":"38846","inReplyTo":"CACsJy8AJfx8id+oHmhsLODBJeNLgRqD+5B295TtS+c-hrC5raQ@mail.gmail.com","subject":"Re: git pull & git gc","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-03-19T01:27:22Z","receivedAt":"2015-03-19T01:27:22Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 19, 2015 at 07:31:48AM +0700, Duy Nguyen wrote:\n\n> Or we could count/estimate the number of loose objects again after\n> repack/prune. Then we can maybe have a way to prevent the next gc that\n> we know will not improve the situation anyway. One option is pack\n> unreachable objects in the second pack. This would stop the next gc,\n> but that would screw prune up because st_mtime info is gone.. Maybe we\n> just save a file to tell gc to ignore the number of loose objects\n> until after a specific date.\n\nI don't think packing the unreachables is a good plan. They just end up\naccumulating then, and they never expire, because we keep refreshing\ntheir mtime at each pack (unless you pack them once and then leave them\nto expire, but then you end up with a large number of packs).\n\nKeeping a file that says \"I ran gc at time T, and there were still N\nobjects left over\" is probably the best bet. When the next \"gc --auto\"\nruns, if T is recent enough, subtract N from the estimated number of\nobjects. I'm not sure of the right value for \"recent enough\" there,\nthough. If it is too far back, you will not gc when you could. If it is\ntoo close, then you will end up running gc repeatedly, waiting for those\nobjects to leave the expiration window.\n\nI guess leaving a bunch of loose objects around longer than necessary\nisn't the end of the world. It wastes space, but it does not actively\nmake the rest of git slower (whereas having a large number of packs does\nimpact performance). So you could probably make \"recent enough\" be \"T <\nnow - gc.pruneExpire / 4\" or something. At most we would try to gc 4\ntimes before dropping unreachable objects, and for the default period,\nthat's only once every couple days.\n\n-Peff\n"},{"id":"257999","messageId":"20150319020117.GA31689@glandium.org","threadId":"38846","inReplyTo":"20150319012722.GA26867@peff.net","subject":"Re: git pull & git gc","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2015-03-19T02:01:17Z","receivedAt":"2015-03-19T02:01:17Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Wed, Mar 18, 2015 at 09:27:22PM -0400, Jeff King wrote:\n> On Thu, Mar 19, 2015 at 07:31:48AM +0700, Duy Nguyen wrote:\n> \n> > Or we could count/estimate the number of loose objects again after\n> > repack/prune. Then we can maybe have a way to prevent the next gc that\n> > we know will not improve the situation anyway. One option is pack\n> > unreachable objects in the second pack. This would stop the next gc,\n> > but that would screw prune up because st_mtime info is gone.. Maybe we\n> > just save a file to tell gc to ignore the number of loose objects\n> > until after a specific date.\n> \n> I don't think packing the unreachables is a good plan. They just end up\n> accumulating then, and they never expire, because we keep refreshing\n> their mtime at each pack (unless you pack them once and then leave them\n> to expire, but then you end up with a large number of packs).\n\nNote, sometimes I wish unreachables were packed. Recently, I ended up in\na situation where running gc created something like 3GB of data as per\ndu, because I suddenly had something like 600K unreachable objects, each\nof them, as a loose object, taking at least 4K on disk. This made my\n.git take 5GB instead of 2GB. That surely didn't feel like garbage\ncollection.\n\nMike\n"},{"id":"258000","messageId":"CAPc5daWmppS_PrvMurEUfvZ2c_bhMnLb-zmck0wzFpgJ4maxZw@mail.gmail.com","threadId":"38846","inReplyTo":"20150319012722.GA26867@peff.net","subject":"Re: git pull & git gc","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-03-19T02:27:46Z","receivedAt":"2015-03-19T02:27:46Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"On Wed, Mar 18, 2015 at 6:27 PM, Jeff King wrote:\n>\n> Keeping a file that says \"I ran gc at time T, and there were still N\n> objects left over\" is probably the best bet. When the next \"gc --auto\"\n> runs, if T is recent enough, subtract N from the estimated number of\n> objects. I'm not sure of the right value for \"recent enough\" there,\n> though. If it is too far back, you will not gc when you could. If it is\n> too close, then you will end up running gc repeatedly, waiting for those\n> objects to leave the expiration window.\n>\n> I guess leaving a bunch of loose objects around longer than necessary\n> isn't the end of the world. It wastes space, but it does not actively\n> make the rest of git slower (whereas having a large number of packs does\n> impact performance). So you could probably make \"recent enough\" be \"T <\n> now - gc.pruneExpire / 4\" or something. At most we would try to gc 4\n> times before dropping unreachable objects, and for the default period,\n> that's only once every couple days.\n\nWe could simply prune unreachables more aggressively, and it would\nsolve this issue at the root cause, no?\n\nWe do keep things reachable from reflogs, so the only thing you are\ngetting by leaving the unreachables around is for an expert to perform\nsome forensic analysis---especially if there are so many loose objects\nthat are all unreachable, nobody sane can go through them one by one\nand guess correctly if each of them is what they wished they kept if\ntheir ancient reflog entry extended a few weeks more.\n\nThat is, unless there is some tool to analyse the unreachable loose\nobjects, collect them into meaningful islands, and present them in\nsome way that the end user can make sense of, which I do not think\nexists (yet).\n"},{"id":"258001","messageId":"20150319040957.GA29437@peff.net","threadId":"38846","inReplyTo":"CAPc5daWmppS_PrvMurEUfvZ2c_bhMnLb-zmck0wzFpgJ4maxZw@mail.gmail.com","subject":"Re: git pull & git gc","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-03-19T04:09:57Z","receivedAt":"2015-03-19T04:09:57Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 18, 2015 at 07:27:46PM -0700, Junio C Hamano wrote:\n\n> > I guess leaving a bunch of loose objects around longer than necessary\n> > isn't the end of the world. It wastes space, but it does not actively\n> > make the rest of git slower (whereas having a large number of packs does\n> > impact performance). So you could probably make \"recent enough\" be \"T <\n> > now - gc.pruneExpire / 4\" or something. At most we would try to gc 4\n> > times before dropping unreachable objects, and for the default period,\n> > that's only once every couple days.\n> \n> We could simply prune unreachables more aggressively, and it would\n> solve this issue at the root cause, no?\n\nYes, but not too aggressively. You mentioned object archaeology, but my\nmain interest is avoiding corruption. The mtime check is the thing that\nprevents us from pruning objects being used for an operation-in-progress\nthat has not yet updated a ref. For some long-running operations, like\nadding files to a commit, we take into account references like a blob\nbeing mentioned in the index. But I do not know offhand if there are\nother long-running operations that would run into problems if we\nshortened the expiration time drastically. Anything building a\ntemporary index is potentially problematic.\n\nBut if we assume that operations like that tend to create and reference\ntheir objects within a reasonable time period (say, seconds to minutes)\nthen the current default of 2 weeks is absurd for this purpose. For\nraciness within a single operation, a few seconds is probably enough\n(e.g., we may write out a commit object and then update the ref a few\nmilliseconds later).\n\nThe potential for problems is exacerbated by the fact that object `X`\nmay exist in the filesystem with an old mtime, and then a new operation\nwants to reference it. That's made somewhat better by 33d4221\n(write_sha1_file: freshen existing objects, 2014-10-15), as before we\ncould silently turn a file write into a noop. But it's still racy to do:\n\n git cat-file -e $commit\n git update-ref refs/heads/foo $commit\n\nas we do not update the mtime for a read-only operation like cat-file\n(and even if we did, it's still somewhat racy as prune does not\natomically check the mtime and remove the file).\n\nSo I think there's definitely some possible danger with dropping the\ndefault prune expiration time.\n\nFor a long time GitHub ran with it as 1.hour.ago. We definitely saw some\noddities and corruption over the years that were apparently caused by\nover-aggressive pruning and/or raciness. I've fixed a number of bugs,\nand things did get better as a result. But I could not say whether all\nsuch problems are gone. These days we do our regular repacks with\n\"--keep-unreachable\" and almost never prune anything.\n\nIt's also not clear whether GitHub represents anything close to \"normal\"\nuse. We have a much smaller array of operations that we perform (most\nobjects are either from a push, or from a test-merge between a topic\nbranch and HEAD). But we also have busy repos that are frequently doing\ngc in the background (especially because we share object storage, so\nactivity on another fork can trigger a gc job that affects a whole\nrepository network). On workstations, I'd guess most git-gc jobs run\nduring a fairly quiescent period.\n\nAll of which is to say that I don't really know the answer, and there\nmay be dragons. I'd imagine that dropping the default expiration time\nfrom 2 weeks to 1 day would probably be fine. A good way to experiment\nwould be for some brave souls to set gc.pruneexpire themselves, run with\nit for a few weeks or months, and see if anything goes wrong.\n\n-Peff\n"},{"id":"258002","messageId":"20150319041453.GB29437@peff.net","threadId":"38846","inReplyTo":"20150319020117.GA31689@glandium.org","subject":"Re: git pull & git gc","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-03-19T04:14:53Z","receivedAt":"2015-03-19T04:14:53Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 19, 2015 at 11:01:17AM +0900, Mike Hommey wrote:\n\n> > I don't think packing the unreachables is a good plan. They just end up\n> > accumulating then, and they never expire, because we keep refreshing\n> > their mtime at each pack (unless you pack them once and then leave them\n> > to expire, but then you end up with a large number of packs).\n> \n> Note, sometimes I wish unreachables were packed. Recently, I ended up in\n> a situation where running gc created something like 3GB of data as per\n> du, because I suddenly had something like 600K unreachable objects, each\n> of them, as a loose object, taking at least 4K on disk. This made my\n> .git take 5GB instead of 2GB. That surely didn't feel like garbage\n> collection.\n\nThat's definitely a thing that happens, but it is a bit of a corner\ncase. It's unusual to have such a large number of unreferenced objects\nall at once.\n\nI don't suppose you happen to remember the details, but would a lower\nexpiration time (e.g., 1 day or 1 hour) have made all of those objects\ngo away? Or were they really from some extremely recent event (of\ncourse, \"event\" here might just have been \"I did a full repack right\nbefore rewriting history\" which would freshen the mtimes on everything\nin the pack).\n\nCertainly the \"loosening\" behavior for unreachable objects has corner\ncases like this, and they suck when you hit one. Leaving the objects\npacked would be better, but IMHO is not a viable alternative unless\nsomebody comes up with a plan for segregating the \"old\" objects in a way\nthat they actually expire eventually, and don't just keep getting\nrepacked and freshened over and over.\n\n-Peff\n"},{"id":"258003","messageId":"CACsJy8CUbe4-f4rpieAKYzHb4rpKg8JW+uXB5yA4c1HFG6r4dg@mail.gmail.com","threadId":"38846","inReplyTo":"20150319012722.GA26867@peff.net","subject":"Re: git pull & git gc","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-03-19T04:15:19Z","receivedAt":"2015-03-19T04:15:19Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Mar 19, 2015 at 8:27 AM, Jeff King wrote:\n> Keeping a file that says \"I ran gc at time T, and there were still N\n> objects left over\" is probably the best bet. When the next \"gc --auto\"\n> runs, if T is recent enough, subtract N from the estimated number of\n> objects. I'm not sure of the right value for \"recent enough\" there,\n> though. If it is too far back, you will not gc when you could. If it is\n> too close, then you will end up running gc repeatedly, waiting for those\n> objects to leave the expiration window.\n\nAnd would not be hard to implement either. git-gc is already prepared\nto deal with stale gc.pid, which would stop git-gc for a day or so\nbefore it deletes gc.pid and starts anyway. All we need to do is check\nat the end of git-gc, if we know for sure the next 'gc --auto' is a\nwaste, then leave gc.pid behind.\n-- \nDuy\n"},{"id":"258004","messageId":"20150319042050.GA29999@peff.net","threadId":"38846","inReplyTo":"CACsJy8CUbe4-f4rpieAKYzHb4rpKg8JW+uXB5yA4c1HFG6r4dg@mail.gmail.com","subject":"Re: git pull & git gc","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-03-19T04:20:50Z","receivedAt":"2015-03-19T04:20:50Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 19, 2015 at 11:15:19AM +0700, Duy Nguyen wrote:\n\n> On Thu, Mar 19, 2015 at 8:27 AM, Jeff King wrote:\n> > Keeping a file that says \"I ran gc at time T, and there were still N\n> > objects left over\" is probably the best bet. When the next \"gc --auto\"\n> > runs, if T is recent enough, subtract N from the estimated number of\n> > objects. I'm not sure of the right value for \"recent enough\" there,\n> > though. If it is too far back, you will not gc when you could. If it is\n> > too close, then you will end up running gc repeatedly, waiting for those\n> > objects to leave the expiration window.\n> \n> And would not be hard to implement either. git-gc is already prepared\n> to deal with stale gc.pid, which would stop git-gc for a day or so\n> before it deletes gc.pid and starts anyway. All we need to do is check\n> at the end of git-gc, if we know for sure the next 'gc --auto' is a\n> waste, then leave gc.pid behind.\n\nThat omits the \"N objects left over\" information. Which I think may be\nuseful, because otherwise the rule is basically \"don't do another gc at\nall for X time units\". That's OK for most use, but it has its own corner\ncases. E.g., imagine you are doing an SVN import that does an auto-gc\ncheck every 1000 commits. You have some unreferenced objects in your\nrepository. After the first 1000 commits, we do a gc, and then say \"wow,\nstill a lot of cruft; let's block gc for a day\". Five minutes later,\nafter another 1000 commits, we run \"gc --auto\" again. It doesn't run\nbecause of the cruft-check, even though there are a _huge_ number of new\npackable objects.\n\nIf the blocker file tells us \"7000 extra objects\" and we see that there\nare 17,000 in the repo, then we know it's still worth doing the gc\n(i.e., we know we that we'll probably end up ignoring the 7000 cruft\nthat didn't get cleaned up last time, but we also know that there are\n10,000 new objects).\n\n-Peff\n"},{"id":"258005","messageId":"20150319042617.GA1072@glandium.org","threadId":"38846","inReplyTo":"20150319041453.GB29437@peff.net","subject":"Re: git pull & git gc","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2015-03-19T04:26:17Z","receivedAt":"2015-03-19T04:26:17Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Thu, Mar 19, 2015 at 12:14:53AM -0400, Jeff King wrote:\n> On Thu, Mar 19, 2015 at 11:01:17AM +0900, Mike Hommey wrote:\n> \n> > > I don't think packing the unreachables is a good plan. They just end up\n> > > accumulating then, and they never expire, because we keep refreshing\n> > > their mtime at each pack (unless you pack them once and then leave them\n> > > to expire, but then you end up with a large number of packs).\n> > \n> > Note, sometimes I wish unreachables were packed. Recently, I ended up in\n> > a situation where running gc created something like 3GB of data as per\n> > du, because I suddenly had something like 600K unreachable objects, each\n> > of them, as a loose object, taking at least 4K on disk. This made my\n> > .git take 5GB instead of 2GB. That surely didn't feel like garbage\n> > collection.\n> \n> That's definitely a thing that happens, but it is a bit of a corner\n> case. It's unusual to have such a large number of unreferenced objects\n> all at once.\n> \n> I don't suppose you happen to remember the details, but would a lower\n> expiration time (e.g., 1 day or 1 hour) have made all of those objects\n> go away? Or were they really from some extremely recent event (of\n> course, \"event\" here might just have been \"I did a full repack right\n> before rewriting history\" which would freshen the mtimes on everything\n> in the pack).\n\nUnfortunately, I don't know the exact details. But yes, I guess a lower\nexpiration time might have helped.\n\n> Certainly the \"loosening\" behavior for unreachable objects has corner\n> cases like this, and they suck when you hit one. Leaving the objects\n> packed would be better, but IMHO is not a viable alternative unless\n> somebody comes up with a plan for segregating the \"old\" objects in a way\n> that they actually expire eventually, and don't just keep getting\n> repacked and freshened over and over.\n\nIt sure is a corner case, otoh, when it happens, every single git\noperation calls git gc --auto, which happily spends 5 minutes sucking\nCPU to end up doing nothing in practice. And add more salt on the\ninjury if you are on battery\n\n6700 loose objects seems easy to reach on a repo with 6M objects...\n\nMike\n"},{"id":"258006","messageId":"CACsJy8B3zyf7mCigQSGXOKLcR9s3=VKkxEe4PuSraS6qb-rjcA@mail.gmail.com","threadId":"38846","inReplyTo":"20150319042050.GA29999@peff.net","subject":"Re: git pull & git gc","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-03-19T04:29:57Z","receivedAt":"2015-03-19T04:29:57Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Mar 19, 2015 at 11:20 AM, Jeff King wrote:\n> On Thu, Mar 19, 2015 at 11:15:19AM +0700, Duy Nguyen wrote:\n>\n>> On Thu, Mar 19, 2015 at 8:27 AM, Jeff King wrote:\n>> > Keeping a file that says \"I ran gc at time T, and there were still N\n>> > objects left over\" is probably the best bet. When the next \"gc --auto\"\n>> > runs, if T is recent enough, subtract N from the estimated number of\n>> > objects. I'm not sure of the right value for \"recent enough\" there,\n>> > though. If it is too far back, you will not gc when you could. If it is\n>> > too close, then you will end up running gc repeatedly, waiting for those\n>> > objects to leave the expiration window.\n>>\n>> And would not be hard to implement either. git-gc is already prepared\n>> to deal with stale gc.pid, which would stop git-gc for a day or so\n>> before it deletes gc.pid and starts anyway. All we need to do is check\n>> at the end of git-gc, if we know for sure the next 'gc --auto' is a\n>> waste, then leave gc.pid behind.\n>\n> That omits the \"N objects left over\" information. Which I think may be\n> useful, because otherwise the rule is basically \"don't do another gc at\n> all for X time units\". That's OK for most use, but it has its own corner\n> cases.\n\nTrue. But saving \"N objects left over\" in a file also has a corner\ncase. If the user \"prune --expire=now\" manually, the next 'gc --auto'\nstill thinks we have that many leftovers and keeps delaying gc for\nsome more time. Unless we make 'prune' (or any other commands that\ndelete leftovers) to also delete this file. Yeah maybe saving this\ninfo in a file will work.\n\n> E.g., imagine you are doing an SVN import that does an auto-gc\n> check every 1000 commits. You have some unreferenced objects in your\n> repository. After the first 1000 commits, we do a gc, and then say \"wow,\n> still a lot of cruft; let's block gc for a day\". Five minutes later,\n> after another 1000 commits, we run \"gc --auto\" again. It doesn't run\n> because of the cruft-check, even though there are a _huge_ number of new\n> packable objects.\n>\n> If the blocker file tells us \"7000 extra objects\" and we see that there\n> are 17,000 in the repo, then we know it's still worth doing the gc\n> (i.e., we know we that we'll probably end up ignoring the 7000 cruft\n> that didn't get cleaned up last time, but we also know that there are\n> 10,000 new objects).\n-- \nDuy\n"},{"id":"258007","messageId":"20150319043437.GA30331@peff.net","threadId":"38846","inReplyTo":"CACsJy8B3zyf7mCigQSGXOKLcR9s3=VKkxEe4PuSraS6qb-rjcA@mail.gmail.com","subject":"Re: git pull & git gc","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-03-19T04:34:37Z","receivedAt":"2015-03-19T04:34:37Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 19, 2015 at 11:29:57AM +0700, Duy Nguyen wrote:\n\n> > That omits the \"N objects left over\" information. Which I think may be\n> > useful, because otherwise the rule is basically \"don't do another gc at\n> > all for X time units\". That's OK for most use, but it has its own corner\n> > cases.\n> \n> True. But saving \"N objects left over\" in a file also has a corner\n> case. If the user \"prune --expire=now\" manually, the next 'gc --auto'\n> still thinks we have that many leftovers and keeps delaying gc for\n> some more time. Unless we make 'prune' (or any other commands that\n> delete leftovers) to also delete this file. Yeah maybe saving this\n> info in a file will work.\n\nI assumed that the user would not run prune manually, but would run \"git\ngc --prune=now\". And yeah, definitely any time gc runs, it should update\nthe file (if there are fewer than `gc.auto` objects, I think it could\njust delete the file).\n\nWe could also apply that rule any run of \"git prune\", but my mental\nmodel is that \"git gc\" is the magical porcelain that will do this stuff\nfor you, and \"git prune\" is the plumbing that users shouldn't need to\ncall themselves. I don't know if that model is shared by users, though. :)\n\n-Peff\n"},{"id":"258019","messageId":"CACsJy8C9RF-PoJNs4PTOg_A0O=RVxZQRDp_UGxtJ_vQ-7h-L3A@mail.gmail.com","threadId":"38846","inReplyTo":"20150318145815.GN1369@serenity.lan","subject":"Re: git pull & git gc","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-03-19T09:47:50Z","receivedAt":"2015-03-19T09:47:50Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Mar 18, 2015 at 9:58 PM, John Keeping wrote:\n> On Wed, Mar 18, 2015 at 09:41:59PM +0700, Duy Nguyen wrote:\n>> On Wed, Mar 18, 2015 at 9:33 PM, Duy Nguyen wrote:\n>> > If not, I made some mistake in analyzing this and we'll start again.\n>>\n>> I did make one mistake, the first \"gc\" should have reduced the number\n>> of loose objects to zero. Why didn't it.? I'll come back to this\n>> tomorrow if nobody finds out first :)\n>\n> Most likely they are not referenced by anything but are younger than 2\n> weeks.\n>\n> I saw a similar issue with automatic gc triggering after every operation\n> when I did something equivalent to:\n>\n> git add \n> git commit\n> git reset --hard HEAD^\n>\n> which creates a log of unreachable objects which are not old enough to\n> be pruned.\n\nAnd there's another problem caused by background gc. If it's not run\nin background, it should print this\n\nwarning: There are too many unreachable loose objects; run 'git prune'\nto remove them.\n\nbut because background gc does not have access to stdout/stderr\nanymore, this is lost.\n-- \nDuy\n"}]}