{"thread":{"id":"50821","subject":"Auto-gc in the background can take a long time to be put in the background","startedAt":"2019-03-25T23:22:29Z","lastAt":"2019-03-26T13:44:09Z","messageCount":5,"participants":["Mike Hommey","Jeff King","Ævar Arnfjörð Bjarmason","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"372485","messageId":"20190325232223.r72qtffyzn5qzoxc@glandium.org","threadId":"50821","inReplyTo":null,"subject":"Auto-gc in the background can take a long time to be put in the background","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2019-03-25T23:22:23Z","receivedAt":"2019-03-25T23:22:29Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"Hi,\n\nRecently, I've noticed that whenever the auto-gc message shows up about\nbeing spawned in the background, it still takes a while for git to\nreturn to the shell.\n\nI've finally looked at what it was stuck on, and it's \n`git reflog expire --all` taking more than 30s. I guess the question is\nwhether there's a reason this shouldn't run in the background? Another\nis whether there's something that makes this slower than it should be.\n\nMike\n"},{"id":"372486","messageId":"20190325233007.GA23728@sigill.intra.peff.net","threadId":"50821","inReplyTo":"20190325232223.r72qtffyzn5qzoxc@glandium.org","subject":"Re: Auto-gc in the background can take a long time to be put in the background","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-03-25T23:30:07Z","receivedAt":"2019-03-25T23:30:10Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Mar 26, 2019 at 08:22:23AM +0900, Mike Hommey wrote:\n\n> Recently, I've noticed that whenever the auto-gc message shows up about\n> being spawned in the background, it still takes a while for git to\n> return to the shell.\n> \n> I've finally looked at what it was stuck on, and it's \n> `git reflog expire --all` taking more than 30s. I guess the question is\n> whether there's a reason this shouldn't run in the background? Another\n> is whether there's something that makes this slower than it should be.\n\nThe reason is that it takes locks which can interfere with other\noperations; see 62aad1849f (gc --auto: do not lock refs in the\nbackground, 2014-05-25).\n\nUnfortunately making it faster is hard. To handle expiring unreachable\nitems, it has to know what's reachable. Which implies walking the commit\ngraph. I don't recall offhand whether setting unreachable-expiration to\n\"never\" would skip that part. But if not, that should be low-hanging\nfruit.\n\n(I also wonder whether there is really much valuable in keeping\nunreachable things for a shorter period of time, and the default should\nsimply be to just prune everything after 90 days, unreachable or not).\n\n-Peff\n"},{"id":"372499","messageId":"87k1gmb03v.fsf@evledraar.gmail.com","threadId":"50821","inReplyTo":"20190325233007.GA23728@sigill.intra.peff.net","subject":"Re: Auto-gc in the background can take a long time to be put in the background","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-03-26T06:50:28Z","receivedAt":"2019-03-26T06:50:33Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 26 2019, Jeff King wrote:\n\n> On Tue, Mar 26, 2019 at 08:22:23AM +0900, Mike Hommey wrote:\n>\n>> Recently, I've noticed that whenever the auto-gc message shows up about\n>> being spawned in the background, it still takes a while for git to\n>> return to the shell.\n>>\n>> I've finally looked at what it was stuck on, and it's\n>> `git reflog expire --all` taking more than 30s. I guess the question is\n>> whether there's a reason this shouldn't run in the background? Another\n>> is whether there's something that makes this slower than it should be.\n>\n> The reason is that it takes locks which can interfere with other\n> operations; see 62aad1849f (gc --auto: do not lock refs in the\n> background, 2014-05-25).\n\nEven assuming we can never improve this I think we should make this part\nconfigurable. It's assuming that the contention is otherwise going to be\nwith yourself in the same terminal, but it doesn't help if the primary\nsource of contention is going to be e.g. other concurrent processes in\nthe same repo.\n\n> Unfortunately making it faster is hard. To handle expiring unreachable\n> items, it has to know what's reachable. Which implies walking the commit\n> graph. I don't recall offhand whether setting unreachable-expiration to\n> \"never\" would skip that part. But if not, that should be low-hanging\n> fruit.\n\nI have a recently patch that does this that I need to re-roll:\nhttps://public-inbox.org/git/20190315155959.12390-8-avarab@gmail.com/\n\n> (I also wonder whether there is really much valuable in keeping\n> unreachable things for a shorter period of time, and the default should\n> simply be to just prune everything after 90 days, unreachable or not).\n\nDo you mean unify gc.reflogExpire & gc.pruneExpire (and other\nvariables). Would that be cheaper somehow?\n\nOr just blindly remove loose objects that are older than some mtime,\nassuming that if anyone cared they'd be in a pack already?\n\nThe latter of those would be very useful, but if not carefully handled\ncould lead to corruption.\n"},{"id":"372515","messageId":"20190326132512.GA22580@sigill.intra.peff.net","threadId":"50821","inReplyTo":"87k1gmb03v.fsf@evledraar.gmail.com","subject":"Re: Auto-gc in the background can take a long time to be put in the background","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-03-26T13:25:13Z","receivedAt":"2019-03-26T13:25:16Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Mar 26, 2019 at 07:50:28AM +0100, Ævar Arnfjörð Bjarmason wrote:\n\n> > Unfortunately making it faster is hard. To handle expiring unreachable\n> > items, it has to know what's reachable. Which implies walking the commit\n> > graph. I don't recall offhand whether setting unreachable-expiration to\n> > \"never\" would skip that part. But if not, that should be low-hanging\n> > fruit.\n> \n> I have a recently patch that does this that I need to re-roll:\n> https://public-inbox.org/git/20190315155959.12390-8-avarab@gmail.com/\n\nI think your patch skips calling git-reflog when both are set to\n\"never\". What I mean is that if regular expiration is set to 90 days,\nand unreachable expiration is set to 90 days (or greater), the there is\nno need for us to bother walking any history. An entry is either expired\nbased on time or it is not, regardless of reachability.\n\n> > (I also wonder whether there is really much valuable in keeping\n> > unreachable things for a shorter period of time, and the default should\n> > simply be to just prune everything after 90 days, unreachable or not).\n> \n> Do you mean unify gc.reflogExpire & gc.pruneExpire (and other\n> variables). Would that be cheaper somehow?\n\nYes, this. If we're just expiring based on the timestamp in the reflog,\nwe should be able to accomplish this with just a single pass over the\nreflog data, and never opening any objects at all.\n\n> Or just blindly remove loose objects that are older than some mtime,\n> assuming that if anyone cared they'd be in a pack already?\n\nNo, definitely not. We're expiring reflogs here, not objects.\n\n-Peff\n"},{"id":"372518","messageId":"nycvar.QRO.7.76.6.1903261443140.41@tvgsbejvaqbjf.bet","threadId":"50821","inReplyTo":"20190325232223.r72qtffyzn5qzoxc@glandium.org","subject":"Re: Auto-gc in the background can take a long time to be put in the background","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-03-26T13:43:44Z","receivedAt":"2019-03-26T13:44:09Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Mike,\n\nOn Tue, 26 Mar 2019, Mike Hommey wrote:\n\n> Recently, I've noticed that whenever the auto-gc message shows up about\n> being spawned in the background, it still takes a while for git to\n> return to the shell.\n>\n> I've finally looked at what it was stuck on, and it's\n> `git reflog expire --all` taking more than 30s. I guess the question is\n> whether there's a reason this shouldn't run in the background? Another\n> is whether there's something that makes this slower than it should be.\n\nThanks for tracking this down. I hit this problem yesterday and was too\nbusy with other things to dig into it.\n\nThank you!\nDscho\n"}]}