{"thread":{"id":"34595","subject":"[BUG?] gc and impatience","startedAt":"2013-08-03T01:48:20Z","lastAt":"2013-08-06T02:14:00Z","messageCount":7,"participants":["Ramkumar Ramachandra","Duy Nguyen","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"224481","messageId":"CALkWK0k2B5OW9zMrw0jBDDWpufojYDJybXa7jCBdzftNUc7mxA@mail.gmail.com","threadId":"34595","inReplyTo":null,"subject":"[BUG?] gc and impatience","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-08-03T01:48:20Z","receivedAt":"2013-08-03T01:48:20Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi,\n\nI was pulling in some changes in the morning to find:\n\n Auto packing the repository for optimum performance. You may also\n run \"git gc\" manually. See \"git help gc\" for more information.\n\nBeing my usual impatient self, I opened another prompt and started\nmerging changes. After the checkout, it started running another gc\n(why!?), which I attempted to kill using ^C.\n\n Counting objects: 449291 x$\n\nIt didn't just fail to stop, but it kept writing output making my\nprompt completely unusable. I finally just killed the pane. Now, it's\nstruggling to update-index and update my cache (read: more waiting).\n\nWhy is gc not designed for impatient people, and what needs to be done\nto change this?\n\nRam\n"},{"id":"224483","messageId":"CACsJy8DRo9Akoydwn9-EPOSyJiPhqAf1=HncYjHvXZwS4S_d7A@mail.gmail.com","threadId":"34595","inReplyTo":"CALkWK0k2B5OW9zMrw0jBDDWpufojYDJybXa7jCBdzftNUc7mxA@mail.gmail.com","subject":"Re: [BUG?] gc and impatience","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-08-03T03:53:38Z","receivedAt":"2013-08-03T03:53:38Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Aug 3, 2013 at 8:48 AM, Ramkumar Ramachandra wrote:\n> Auto packing the repository for optimum performance. You may also\n> run \"git gc\" manually. See \"git help gc\" for more information.\n>\n> Being my usual impatient self, I opened another prompt and started\n> merging changes. After the checkout, it started running another gc\n> (why!?),\n\nGood point. I think that is because gc does not check if gc is already\nrunning. Adding such a check should not be too hard. I think gc could\nsave its pid in $GIT_DIR/auto-gc.pid. The next auto-gc checks this, if\nthe pid is valid, skip auto-gc.\n\n> Why is gc not designed for impatient people, and what needs to be done\n> to change this?\n\nSome improvements could be made in gc, for example warn users about\nupcoming gc so they can run it in background (of course the above bug\nshould be fixed)\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/197716/focus=197877\n\nor speed up repack by implementing pack-objects --merge-pack:\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/57672/focus=57943\n\nOr you could just make a cron job to gc all repos every week and the\nproblem goes away ;-)\n-- \nDuy\n"},{"id":"224485","messageId":"CAPc5daXi_ZG6GcK6pWafffyOY4MEZHUMkZxTxiRiU4BaFybqqg@mail.gmail.com","threadId":"34595","inReplyTo":"CACsJy8DRo9Akoydwn9-EPOSyJiPhqAf1=HncYjHvXZwS4S_d7A@mail.gmail.com","subject":"Re: [BUG?] gc and impatience","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-08-03T04:44:11Z","receivedAt":"2013-08-03T04:44:11Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"On Fri, Aug 2, 2013 at 8:53 PM, Duy Nguyen wrote:\n> Good point. I think that is because gc does not check if gc is already\n> running. Adding such a check should not be too hard. I think gc could\n> save its pid in $GIT_DIR/auto-gc.pid. The next auto-gc checks this, if\n> the pid is valid, skip auto-gc.\n\nDefining \"valid\" is a tricky business, though, as pid can and will\nwrap around, and the directory could be shared on multiple machines. A\npid written by a process on one machine has no relation to any pid on\nanother machine.\n"},{"id":"224487","messageId":"CACsJy8DO4VyCK_xDJDGVx6JLTqjKAf24AUOW3=kZEMEVjAUSVw@mail.gmail.com","threadId":"34595","inReplyTo":"CAPc5daXi_ZG6GcK6pWafffyOY4MEZHUMkZxTxiRiU4BaFybqqg@mail.gmail.com","subject":"Re: [BUG?] gc and impatience","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-08-03T05:25:11Z","receivedAt":"2013-08-03T05:25:11Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Aug 3, 2013 at 11:44 AM, Junio C Hamano wrote:\n> On Fri, Aug 2, 2013 at 8:53 PM, Duy Nguyen wrote:\n>> Good point. I think that is because gc does not check if gc is already\n>> running. Adding such a check should not be too hard. I think gc could\n>> save its pid in $GIT_DIR/auto-gc.pid. The next auto-gc checks this, if\n>> the pid is valid, skip auto-gc.\n>\n> Defining \"valid\" is a tricky business, though, as pid can and will\n> wrap around,\n\nYes there is a chance that the old pid is not used for another process\nand it could get worse when that process is a daemon and runs forever.\nIf we go the optimistic way, we could check mtime of auto-gc.pid. If\nit's older than a couple hours, ignore it and run gc anyway, assuming\ngc can't last longer than an hour or so. A more reliable way is save a\nunix socket instead of auto-gc.pid and send something over the socket\nto verify it's gc, but I think it's overkill.\n\n> and the directory could be shared on multiple machines. A\n> pid written by a process on one machine has no relation to any pid on\n> another machine.\n\nI worry less about this. It's not the right model to have two machines\nmodify the same shared repository (gc --auto is only triggered when we\nthink there are new objects) even though I think we support it. If\nit's two _scripts_ modifying the same repo, I don't care as this is\nmore about user interaction. If it's two people modifying the same\nrepo, it sounds like an insane setup and there may be more issues to\nworry about than gc --auto.\n-- \nDuy\n"},{"id":"224581","messageId":"7vk3k0not6.fsf@alter.siamese.dyndns.org","threadId":"34595","inReplyTo":"CACsJy8DO4VyCK_xDJDGVx6JLTqjKAf24AUOW3=kZEMEVjAUSVw@mail.gmail.com","subject":"Re: [BUG?] gc and impatience","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-08-05T15:24:21Z","receivedAt":"2013-08-05T15:24:21Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> I worry less about this. It's not the right model to have two machines\n> modify the same shared repository (gc --auto is only triggered when we\n> think there are new objects) even though I think we support it.\n\nI am a bit hesitant to dismiss with \"It's not the right model\", as\nthe original of accessing the repository from two terminals while\none clearly is being accessed busily by gc falls into the same\ncategory.\n\n> If\n> it's two _scripts_ modifying the same repo, I don't care as this is\n> more about user interaction.\n\nIt can very well be two terminals, one on one machine each, both\nwith the same human end-user interaction.\n"},{"id":"224588","messageId":"CALkWK0=fDaVUS7rSQ5LyS1UL2hD8JmA23BnpGeT6pC0EgEd9_w@mail.gmail.com","threadId":"34595","inReplyTo":"7vk3k0not6.fsf@alter.siamese.dyndns.org","subject":"Re: [BUG?] gc and impatience","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-08-05T15:54:29Z","receivedAt":"2013-08-05T15:54:29Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Junio C Hamano wrote:\n> I am a bit hesitant to dismiss with \"It's not the right model\", as\n> the original of accessing the repository from two terminals while\n> one clearly is being accessed busily by gc falls into the same\n> category.\n\nAs to why I think it makes sense: garbage collecting unreferenced\nobjects has nothing to do with updating refs, or checking out a\nworktree. Think about my earlier \"make push.default = current resolve\nHEAD early\"; why would the user want to update the ref that is being\npushed? She'd most likely want to continue working on another feature\non some other branch, and that's perfectly fine.\n\nIn long-running runtimes, garbage collection is absolutely essential\nto the performance. Often, stupidly written garbage collectors that\nstop-the-world (the execution of the program), compact the memory\nafter collection, and then restart the program, can cause the user to\nthrow that runtime out the window (Emacs has a really stupid one, by\nthe way). Most modern runtimes have concurrent garbage collectors\nthat are allocated very fine-grained slots by the scheduler: so, the\nprogram won't suddenly come to a grinding halt to do garbage\ncollection. The reason it's so hard to do concurrent gc is because\nthere can be races between data modification via variables (main\nprogram), and data being moved around in memory for compacting (gc).\n\nHaving said all this, the problem is highly simplified in git, because\nthe object store is a const-store. A particular key (sha-1) is\nguaranteed never to point to the wrong data. Frankly, even if there\nis concurrent access to the object store, the worst thing that can\nhappen is that the gc didn't collect some dangling objects that were\ncreated during the gc run.\n\nUnless you have some irrational fear of introducing some unexpected\nbehavior in some convoluted corner case, I really don't see what the\nproblem is. I'm sure server-side implementations have to do it all\nthe time: GitHub and Gerrit certainly doesn't say \"I'm gc'ing; please\npull after 10 mins\". Perhaps they're more conservative than the\nclient side about gc (space is cheap), but that's just a sane default.\n\n> It can very well be two terminals, one on one machine each, both\n> with the same human end-user interaction.\n\nSomeone does an SSH my machine to a submarine in Russia over a slow\nconnection. I remove an ordinary file, while she's trying to write to\nit. When did anyone make any guarantees about no races? What does git\ngc specifically have to do with this?\n\nFor the record, you can easily mess up your worktree by running two\ndifferent worktree updates (checkout/ merge) on two different\nterminals: nothing forbidding it. I don't see how _not_ forbidding gc\non two different terminals is better than forbidding it. This is quite\nan obscure feature for few super-impatient people, and we haven't even\nadvertised it in any documentation.\n\nUnless you can present an alternative now (patch-form, please), I\nthink you're being irrationally conservative about this.\n"},{"id":"224635","messageId":"CALkWK0nj+NS-OzT6faaNpgA6iQgLaCSX1=fohAnzEWt9BGzFYw@mail.gmail.com","threadId":"34595","inReplyTo":"CACsJy8DRo9Akoydwn9-EPOSyJiPhqAf1=HncYjHvXZwS4S_d7A@mail.gmail.com","subject":"Re: [BUG?] gc and impatience","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-08-06T02:14:00Z","receivedAt":"2013-08-06T02:14:00Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Duy Nguyen wrote:\n> Good point. I think that is because gc does not check if gc is already\n> running. Adding such a check should not be too hard. I think gc could\n> save its pid in $GIT_DIR/auto-gc.pid. The next auto-gc checks this, if\n> the pid is valid, skip auto-gc.\n\nCheck. I also talked about gc not catching SIGINT properly: I'm\nlooking the issue.\n\n> Or you could just make a cron job to gc all repos every week and the\n> problem goes away ;-)\n\nFundamentally, we need to fix these problems:\n\n1. Don't make the repo unusable when a gc is running: I don't expect\nanything more than minor annoyances after your patch is checked in.\n\n2. Improve the IO profile, so gc doesn't aggressively hammer out tiny\nfragmentations. For this, git-exproll.sh is definitely a step in the\nright direction.\n\n3. Improve how gc fundamentally works, so we can minimize rebuilds and\nCPU time. jc's git merge-pack is interesting, but I'm not very hopeful\nabout a naive incremental-packing. We need to keep the major\nundeltified objects near the top of the file, and build an idx sorted\nby SHA-1; mangling the offsets in the header after a packfile has been\nwritten is both complicated and dangerous (we might introduce subtle\nbugs corrupting the packfile), I think. I haven't thought about it\nhard enough though.\n\nWe'll tackle these problems bit by bit in future patches.\n\nThanks.\n"}]}