{"thread":{"id":"29070","subject":"git auto-repack is broken...","startedAt":"2011-12-02T16:22:00Z","lastAt":"2011-12-09T18:34:04Z","messageCount":18,"participants":["Linus Torvalds","Ævar Arnfjörð Bjarmason","Jeff King","Junio C Hamano","Brandon Casey","Nicolas Pitre"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"180254","messageId":"CA+55aFznj49hx6Ce6NhJ1rRd2nvNyOERseyyrC6SNcW-z9dyfg@mail.gmail.com","threadId":"29070","inReplyTo":null,"subject":"git auto-repack is broken...","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2011-12-02T16:22:00Z","receivedAt":"2011-12-02T16:22:00Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"I actually tend to repack things pretty religiously (ok, not really,\nbut I do \"git gc\" reasonably regularly, so I was surprised to see\nthig:\n\n Auto packing the repository for optimum performance. You may also\n run \"git gc\" manually. See \"git help gc\" for more information.\n\nfollowed by this pitiful effort:\n\n Counting objects: 8, done.\n Delta compression using up to 4 threads.\n Compressing objects: 100% (8/8), done.\n Writing objects: 100% (8/8), done.\n Total 8 (delta 0), reused 0 (delta 0)\n\nOk, those 8 objects will *not* help anything at all, and the\nautorepack is broken.\n\nSo what's going on? It turns out that I have a fair amount of\nunreachable objects in this repository, because I do things like\nfetching things without then merging them, etc. So the \"git gc --auto\"\nwill happily do \"git repack -A\" or whatever, and that in turn does\n*nothing* what-so-ever (or rather, it packs my latest merge commit\nlike the above and generates that pack of a whopping 8 objects).\n\nI can fix it with \"git gc --prune=now\", so it's not like I personally\nreally care, but since the whole point of \"git gc --auto\" is to allow\npeople who don't know what they are doing to ignore the whole issue of\nGC and pruning, I do think this is a real UI bug.\n\nI don't really have any suggestions for fixing it, though. Maybe we\nshould make \"git gc --auto\" remove any unreachable objects? That would\nbe potentially dangerous in shared repository situations, though. Or\nhave an extra option to \"git repack -A\" to also pack any loose objects\nit finds at the end (whether reachable or not)?\n\n Linus\n"},{"id":"180255","messageId":"CACBZZX7Q5qb1r_Oh0QfMiWh9UAM1c6QWBn4abv-xHpFBaKuyKg@mail.gmail.com","threadId":"29070","inReplyTo":"CA+55aFznj49hx6Ce6NhJ1rRd2nvNyOERseyyrC6SNcW-z9dyfg@mail.gmail.com","subject":"Re: git auto-repack is broken...","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2011-12-02T16:27:24Z","receivedAt":"2011-12-02T16:27:24Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Fri, Dec 2, 2011 at 17:22, Linus Torvalds\n wrote:\n\n> Maybe we should make \"git gc --auto\" remove any unreachable objects?\n\nWouldn't that mean that any loose commit objects you have lying around\nwould be removed by the automatic git gc?\n\nOne feature of git that I personally rely on is that I can liberally\nmove heads around / make commits on detached heads and not have those\ncommits gc'd unless I explicitly ask for it for a while.\n"},{"id":"180256","messageId":"CA+55aFyq28vmo9dk-5mVm+nNn86qSjNT6VJGc09iaJo=+OP1Sg@mail.gmail.com","threadId":"29070","inReplyTo":"CACBZZX7Q5qb1r_Oh0QfMiWh9UAM1c6QWBn4abv-xHpFBaKuyKg@mail.gmail.com","subject":"Re: git auto-repack is broken...","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2011-12-02T16:56:34Z","receivedAt":"2011-12-02T16:56:34Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"On Fri, Dec 2, 2011 at 8:27 AM, Ævar Arnfjörð Bjarmason\n wrote:\n>\n>> Maybe we should make \"git gc --auto\" remove any unreachable objects?\n>\n> Wouldn't that mean that any loose commit objects you have lying around\n> would be removed by the automatic git gc?\n>\n> One feature of git that I personally rely on is that I can liberally\n> move heads around / make commits on detached heads and not have those\n> commits gc'd unless I explicitly ask for it for a while.\n\nWell, with reflogs, you actually do have those objects reachable for\nquite a while (90 days by default).\n\nThe \"unreachable objects\" tends to happen when you do fetches without\never merging the result or actually remove branches (and/or expiring\nthe reflogs early etc). Not from the normal \"use 'git reset' and\nfriends to move heads around\".\n\nThat said, I do agree that removing loose objects is the much less\nsafe approach.\n\nOf course, repacking the objects results in problems too: now you've\nentirely lost the age information for that object, so now you cannot\nprune it based on age any more.\n\nBut leaving the loose objects around and basically failing auto-gc\nisn't good either.\n\n Linus\n"},{"id":"180258","messageId":"20111202171017.GB23447@sigill.intra.peff.net","threadId":"29070","inReplyTo":"CA+55aFyq28vmo9dk-5mVm+nNn86qSjNT6VJGc09iaJo=+OP1Sg@mail.gmail.com","subject":"Re: git auto-repack is broken...","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-12-02T17:10:17Z","receivedAt":"2011-12-02T17:10:17Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Dec 02, 2011 at 08:56:34AM -0800, Linus Torvalds wrote:\n\n> On Fri, Dec 2, 2011 at 8:27 AM, Ævar Arnfjörð Bjarmason\n> wrote:\n> >\n> >> Maybe we should make \"git gc --auto\" remove any unreachable objects?\n> >\n> > Wouldn't that mean that any loose commit objects you have lying around\n> > would be removed by the automatic git gc?\n> >\n> > One feature of git that I personally rely on is that I can liberally\n> > move heads around / make commits on detached heads and not have those\n> > commits gc'd unless I explicitly ask for it for a while.\n> \n> Well, with reflogs, you actually do have those objects reachable for\n> quite a while (90 days by default).\n> \n> The \"unreachable objects\" tends to happen when you do fetches without\n> ever merging the result or actually remove branches (and/or expiring\n> the reflogs early etc). Not from the normal \"use 'git reset' and\n> friends to move heads around\".\n> \n> That said, I do agree that removing loose objects is the much less\n> safe approach.\n\nWe do remove loose objects that are totally unreferenced, but there is\nstill a time-delay, because we don't want to prune something like an\nin-progress commit operation. The default delay for that is 2 weeks,\nwhich I think is an arbitrary number that was \"wow, if your git\noperation takes longer than this, you're way too patient\".\n\nAnd in general, it works OK because people don't tend to accumulate more\nthan the auto-gc number of objects within a 2 week period. So perhaps\nyou're just special in your usage patterns.\n\nOne solution is just dropping that \"2 weeks\" down to something smaller,\nbut still conservative (say, 3 days?).\n\nIf you still have the repo in question, what is the date breakdown on\nyour loose objects?\n\n> Of course, repacking the objects results in problems too: now you've\n> entirely lost the age information for that object, so now you cannot\n> prune it based on age any more.\n\nWhen the objects become unreferenced, we eject them from the pack into\nloose form again. If they don't become referenced in the 2-week window,\nthey get pruned then. So yes, you drop the age information, but they do\neventually go away.\n\n-Peff\n"},{"id":"180260","messageId":"7vobvqoozr.fsf@alter.siamese.dyndns.org","threadId":"29070","inReplyTo":"20111202171017.GB23447@sigill.intra.peff.net","subject":"Re: git auto-repack is broken...","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-02T17:35:52Z","receivedAt":"2011-12-02T17:35:52Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> When the objects become unreferenced, we eject them from the pack into\n> loose form again. If they don't become referenced in the 2-week window,\n> they get pruned then. So yes, you drop the age information, but they do\n> eventually go away.\n\nIf you update gc/repack -A to put them in a separate pack, then you would\nnever be able to get rid of them, no? You pack, then eject (which gives\nthem a fresher timestamp), then notice that you are within the 2-week window\nand pack them again,...\n"},{"id":"180261","messageId":"20111202174546.GA24093@sigill.intra.peff.net","threadId":"29070","inReplyTo":"7vobvqoozr.fsf@alter.siamese.dyndns.org","subject":"Re: git auto-repack is broken...","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-12-02T17:45:46Z","receivedAt":"2011-12-02T17:45:46Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Dec 02, 2011 at 09:35:52AM -0800, Junio C Hamano wrote:\n\n> Jeff King writes:\n> \n> > When the objects become unreferenced, we eject them from the pack into\n> > loose form again. If they don't become referenced in the 2-week window,\n> > they get pruned then. So yes, you drop the age information, but they do\n> > eventually go away.\n> \n> If you update gc/repack -A to put them in a separate pack, then you would\n> never be able to get rid of them, no? You pack, then eject (which gives\n> them a fresher timestamp), then notice that you are within the 2-week window\n> and pack them again,...\n\nBut we shouldn't be packing totally unreferenced objects. Barring bugs,\nthe life cycle of such an object should be something like:\n\n 1. Object X is created on branch 'foo'.\n\n 2. Branch 'foo' is deleted, but its commits are still in the HEAD\n reflog, referencing X.\n\n 3. 90 days pass (actually, I think this might be the 30-day\n expire-unreachable time)\n\n 4. \"git gc\" runs \"git repack -Ad\", which will eject X from the pack\n into a loose form (because it is not becoming part of the new pack\n we are writing).\n\n 5. Two weeks pass.\n\n 6. \"git gc\" runs \"git prune --expire=2.weeks.ago\", which removes the\n object.\n\n\"gc\" runs between (4) and (6) will not re-pack the object, because it\nremains unreferenced.\n\nI think things might be slowed somewhat by \"gc --auto\", which will not\ndo a \"repack -A\" until we have too many packs. So steps (3) and (4) are\nreally more like \"gc runs git-repack without -A\" 50 times, and then we\nfinally run \"git repack -A\".\n\n-Peff\n"},{"id":"180263","messageId":"7vd3c6onhs.fsf@alter.siamese.dyndns.org","threadId":"29070","inReplyTo":"20111202174546.GA24093@sigill.intra.peff.net","subject":"Re: git auto-repack is broken...","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-02T18:08:15Z","receivedAt":"2011-12-02T18:08:15Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> But we shouldn't be packing totally unreferenced objects.\n\nEverything you said is correct in today's Git and I obviously know it, but\nI was taking the \"Or have an extra option to...\" at the end of the OP's\nmessage in the thread into account, so...\n"},{"id":"180265","messageId":"20111202181358.GD24093@sigill.intra.peff.net","threadId":"29070","inReplyTo":"7vd3c6onhs.fsf@alter.siamese.dyndns.org","subject":"Re: git auto-repack is broken...","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-12-02T18:13:59Z","receivedAt":"2011-12-02T18:13:59Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Dec 02, 2011 at 10:08:15AM -0800, Junio C Hamano wrote:\n\n> Jeff King writes:\n> \n> > But we shouldn't be packing totally unreferenced objects.\n> \n> Everything you said is correct in today's Git and I obviously know it, but\n> I was taking the \"Or have an extra option to...\" at the end of the OP's\n> message in the thread into account, so...\n\nAh, sorry, I missed the subtlety of Linus's \"repacking the objects\nresults in problems...\" from his later message and thought he just meant\nrepacking in general. Yes, it's a bad idea to repack unreachable objects\nbecause then you could never prune anything.\n\nI think just shrinking the --expire window that we already use is a much\nmore reasonable bet. It's not about preventing the loss of old work\n(reflogs are there for that), but about avoiding hurting an actively\nrunning, about-to-reference-the-objects git process. And 2 weeks is\nquite conservative for that.\n\n-Peff\n"},{"id":"180283","messageId":"CA+sFfMdeVoz8XU5j4hNn1qCHHzaiDi0Bw=QbbuU3cwT9mMPZOA@mail.gmail.com","threadId":"29070","inReplyTo":"20111202174546.GA24093@sigill.intra.peff.net","subject":"Re: git auto-repack is broken...","fromName":"Brandon Casey","fromEmail":"drafnel@gmail.com","sentAt":"2011-12-03T19:42:22Z","receivedAt":"2011-12-03T19:42:22Z","isPatch":false,"sender":{"key":"drafnel@gmail.com","avatar":"https://avatars.githubusercontent.com/u/921167?v=4"},"body":"On Fri, Dec 2, 2011 at 11:45 AM, Jeff King wrote:\n> On Fri, Dec 02, 2011 at 09:35:52AM -0800, Junio C Hamano wrote:\n>\n>> Jeff King writes:\n>>\n>> > When the objects become unreferenced, we eject them from the pack into\n>> > loose form again. If they don't become referenced in the 2-week window,\n>> > they get pruned then. So yes, you drop the age information, but they do\n>> > eventually go away.\n>>\n>> If you update gc/repack -A to put them in a separate pack, then you would\n>> never be able to get rid of them, no? You pack, then eject (which gives\n>> them a fresher timestamp), then notice that you are within the 2-week window\n>> and pack them again,...\n>\n> But we shouldn't be packing totally unreferenced objects. Barring bugs,\n> the life cycle of such an object should be something like:\n>\n>  1. Object X is created on branch 'foo'.\n>\n>  2. Branch 'foo' is deleted, but its commits are still in the HEAD\n>     reflog, referencing X.\n>\n>  3. 90 days pass (actually, I think this might be the 30-day\n>     expire-unreachable time)\n>\n>  4. \"git gc\" runs \"git repack -Ad\", which will eject X from the pack\n>     into a loose form (because it is not becoming part of the new pack\n>     we are writing).\n\nActually, it is right here when the newly loosened unreferenced\nobjects will be deleted. Objects ejected from a pack _are_ given the\ntimestamp of the pack they were ejected from. So, if the pack is\nolder than two weeks (90 days in your example), then so will be the\nloosened objects, and git prune will delete them when called by git\ngc.\n\n>  5. Two weeks pass.\n>\n>  6. \"git gc\" runs \"git prune --expire=2.weeks.ago\", which removes the\n>     object.\n>\n> \"gc\" runs between (4) and (6) will not re-pack the object, because it\n> remains unreferenced.\n\nCorrect with the recognition that loose objects get pack mtime, so\nstep 5 may be less than two weeks.\n\n> I think things might be slowed somewhat by \"gc --auto\", which will not\n> do a \"repack -A\" until we have too many packs. So steps (3) and (4) are\n> really more like \"gc runs git-repack without -A\" 50 times, and then we\n> finally run \"git repack -A\".\n\nThis is correct. This should have the effect of increasing the age of\nunreferenced objects when they are finally loosened and make it more\nlikely that they are pruned during the same git gc operation that\nloosens them.\n\nLinus's scenario of fetching a lot of stuff that never actually makes\nit into the reflogs is still a valid problem. I'm not sure that\npeople who don't know what they are doing are going to run into this\nproblem though. Since he fetches a lot of stuff without ever checking\nit out or creating a branch from it, potentially many objects become\nunreferenced every time FETCH_HEAD changes. If he does this many\ntimes in a short period of time, he could reach the gc.autopacklimit\nand trigger gc --auto and produce more than gc.auto loose objects that\nare younger than gc.pruneExpire.\n\nDecreasing gc.pruneExpire as you suggested should make it much less\nlikely to run into this problem. I wonder if it is worth trying to\nlimit how often gc --auto is run to not be more often than\ngc.pruneExpire or something. If we modified the timestamp that is\nassigned to fetched packs, maybe we could use the pack timestamps as\nan indicator of how recently git gc has run.\n\n-Brandon\n"},{"id":"180534","messageId":"alpine.LFD.2.02.1112071709250.2907@xanadu.home","threadId":"29070","inReplyTo":"CA+sFfMdeVoz8XU5j4hNn1qCHHzaiDi0Bw=QbbuU3cwT9mMPZOA@mail.gmail.com","subject":"Re: git auto-repack is broken...","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2011-12-07T22:12:14Z","receivedAt":"2011-12-07T22:12:14Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sat, 3 Dec 2011, Brandon Casey wrote:\n\n> Linus's scenario of fetching a lot of stuff that never actually makes\n> it into the reflogs is still a valid problem. I'm not sure that\n> people who don't know what they are doing are going to run into this\n> problem though. Since he fetches a lot of stuff without ever checking\n> it out or creating a branch from it, potentially many objects become\n> unreferenced every time FETCH_HEAD changes.\n\nMaybe FETCH_HEAD should have a reflog too?\n\n\nNicolas\n"},{"id":"180540","messageId":"20111207225318.GA21852@sigill.intra.peff.net","threadId":"29070","inReplyTo":"alpine.LFD.2.02.1112071709250.2907@xanadu.home","subject":"Re: git auto-repack is broken...","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-12-07T22:53:19Z","receivedAt":"2011-12-07T22:53:19Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 07, 2011 at 05:12:14PM -0500, Nicolas Pitre wrote:\n\n> Maybe FETCH_HEAD should have a reflog too?\n\nThat might be nice. However, there is a complication, in that FETCH_HEAD\nmay contain many sha1s, but each reflog entry only has room for a single\nsha1 transition. You could obviously encode it as a series of reflog\nentries, but then \"git show FETCH_HEAD@{1}\" is not very meaningful.\n\n-Peff\n"},{"id":"180546","messageId":"alpine.LFD.2.02.1112071912570.2907@xanadu.home","threadId":"29070","inReplyTo":"20111207225318.GA21852@sigill.intra.peff.net","subject":"Re: git auto-repack is broken...","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2011-12-08T00:18:13Z","receivedAt":"2011-12-08T00:18:13Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 7 Dec 2011, Jeff King wrote:\n\n> On Wed, Dec 07, 2011 at 05:12:14PM -0500, Nicolas Pitre wrote:\n> \n> > Maybe FETCH_HEAD should have a reflog too?\n> \n> That might be nice. However, there is a complication, in that FETCH_HEAD\n> may contain many sha1s, but each reflog entry only has room for a single\n> sha1 transition. You could obviously encode it as a series of reflog\n> entries, but then \"git show FETCH_HEAD@{1}\" is not very meaningful.\n\nWhat does \"git show FETCH_HEAD\" do now? If it shows only one \n(presumably the first) SHA1 then its reflog doesn't have to be smarter, \nwhich would properly cover most cases already. I certainly never did a \nmulti-ref fetch myself.\n\n\nNicolas\n"},{"id":"180547","messageId":"20111208004515.GA23015@sigill.intra.peff.net","threadId":"29070","inReplyTo":"alpine.LFD.2.02.1112071912570.2907@xanadu.home","subject":"Re: git auto-repack is broken...","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-12-08T00:45:15Z","receivedAt":"2011-12-08T00:45:15Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 07, 2011 at 07:18:13PM -0500, Nicolas Pitre wrote:\n\n> > > Maybe FETCH_HEAD should have a reflog too?\n> > \n> > That might be nice. However, there is a complication, in that FETCH_HEAD\n> > may contain many sha1s, but each reflog entry only has room for a single\n> > sha1 transition. You could obviously encode it as a series of reflog\n> > entries, but then \"git show FETCH_HEAD@{1}\" is not very meaningful.\n> \n> What does \"git show FETCH_HEAD\" do now? If it shows only one\n> (presumably the first) SHA1 then its reflog doesn't have to be\n> smarter, which would properly cover most cases already.\n\nAre you proposing that it only store the first ref in the reflog, or\nthat we accept that a single fetch may write lots of reflog entries?\n\nIf the former, then you are missing the expiration/connectivity\nproperties.\n\nIf the latter, then it is not just \"we only show the first one for\nFETCH_HEAD@{1}\", but also \"the thing that used to be FETCH_HEAD@{1} does\nnot graduate to FETCH_HEAD@{2}, but rather FETCH_HEAD@{n} for some\nunknown n\". That may be an acceptable limitation; I just wanted to\nmention it in case somebody can think of some clever solution.\n\n> I certainly never did a multi-ref fetch myself.\n\nNot consciously, perhaps, but you do it all the time without realizing\nit:\n\n $ git clone git://git.kernel.org/pub/scm/git/git.git\n $ cd git\n $ git fetch -v origin\n = [up to date] maint -> origin/maint\n = [up to date] master -> origin/master\n = [up to date] next -> origin/next\n = [up to date] pu -> origin/pu\n = [up to date] todo -> origin/todo\n $ cat .git/FETCH_HEAD\n b1af9630d758e1728fc0008b3f18d90d8f87f4c5 not-for-merge branch 'maint' of git://git.kernel.org/pub/scm/git/git\n 4cb5d10b14dcbe0155bed9c45ccb94e83bd4c599 branch 'master' of git://git.kernel.org/pub/scm/git/git\n 03e5527c5df33d4550ccc1446d861c0aa5689d58 not-for-merge branch 'next' of git://git.kernel.org/pub/scm/git/git\n cc4e3f01fc6a5e09ae5bbdc464965981fae4cf39 not-for-merge branch 'pu' of git://git.kernel.org/pub/scm/git/git\n 7a02dba15bd28826344f9c14a5e2b5c57eeb7e50 not-for-merge branch 'todo' of git://git.kernel.org/pub/scm/git/git\n\n-Peff\n"},{"id":"180548","messageId":"20111208004931.GB23015@sigill.intra.peff.net","threadId":"29070","inReplyTo":"CA+sFfMdeVoz8XU5j4hNn1qCHHzaiDi0Bw=QbbuU3cwT9mMPZOA@mail.gmail.com","subject":"Re: git auto-repack is broken...","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-12-08T00:49:31Z","receivedAt":"2011-12-08T00:49:31Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Dec 03, 2011 at 01:42:22PM -0600, Brandon Casey wrote:\n\n> >  4. \"git gc\" runs \"git repack -Ad\", which will eject X from the pack\n> >     into a loose form (because it is not becoming part of the new pack\n> >     we are writing).\n> \n> Actually, it is right here when the newly loosened unreferenced\n> objects will be deleted. Objects ejected from a pack _are_ given the\n> timestamp of the pack they were ejected from. So, if the pack is\n> older than two weeks (90 days in your example), then so will be the\n> loosened objects, and git prune will delete them when called by git\n> gc.\n\nThanks, I didn't notice that when looking at the code.\n\n> Decreasing gc.pruneExpire as you suggested should make it much less\n> likely to run into this problem.\n\nI'd be more comfortable with that solution if we had data on what the\ntimestamps look like when it actually happens (e.g., an \"ls -lR\" listing\nof a repository that in practice is wanting to auto-gc too often).\n\n> I wonder if it is worth trying to limit how often gc --auto is run to\n> not be more often than gc.pruneExpire or something. If we modified\n> the timestamp that is assigned to fetched packs, maybe we could use\n> the pack timestamps as an indicator of how recently git gc has run.\n\nI'm worried you run into other corner cases, there. Like a repository\nwhich is generating new, referenced objects at a fast rate (e.g.,\nbecause you're importing something) should trigger auto-gc much sooner\nthan that, and this rule would prevent it.\n\n-Peff\n"},{"id":"180552","messageId":"alpine.LFD.2.02.1112072227510.2907@xanadu.home","threadId":"29070","inReplyTo":"20111208004515.GA23015@sigill.intra.peff.net","subject":"Re: git auto-repack is broken...","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2011-12-08T03:35:00Z","receivedAt":"2011-12-08T03:35:00Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 7 Dec 2011, Jeff King wrote:\n\n> On Wed, Dec 07, 2011 at 07:18:13PM -0500, Nicolas Pitre wrote:\n> \n> > I certainly never did a multi-ref fetch myself.\n> \n> Not consciously, perhaps, but you do it all the time without realizing\n> it:\n> \n> $ git clone git://git.kernel.org/pub/scm/git/git.git\n> $ cd git\n> $ git fetch -v origin\n> = [up to date] maint -> origin/maint\n> = [up to date] master -> origin/master\n> = [up to date] next -> origin/next\n> = [up to date] pu -> origin/pu\n> = [up to date] todo -> origin/todo\n> $ cat .git/FETCH_HEAD\n> b1af9630d758e1728fc0008b3f18d90d8f87f4c5 not-for-merge branch 'maint' of git://git.kernel.org/pub/scm/git/git\n> 4cb5d10b14dcbe0155bed9c45ccb94e83bd4c599 branch 'master' of git://git.kernel.org/pub/scm/git/git\n> 03e5527c5df33d4550ccc1446d861c0aa5689d58 not-for-merge branch 'next' of git://git.kernel.org/pub/scm/git/git\n> cc4e3f01fc6a5e09ae5bbdc464965981fae4cf39 not-for-merge branch 'pu' of git://git.kernel.org/pub/scm/git/git\n> 7a02dba15bd28826344f9c14a5e2b5c57eeb7e50 not-for-merge branch 'todo' of git://git.kernel.org/pub/scm/git/git\n\nOK, nevermind. I admitedly never have been close enough to the related \ncode.\n\nAnd I don't think this particular case is interesting anyway as the \nreflogs for the various branches alre already involved. I was thinking \nmore about the \"git fetch git://some.random.repo foobar\" case where the \nsummary also explicitly shows:\n\nFrom: git://some.random.repo\n ...... foobar -> FETCH_HEAD\n\nIn that case the only reference to the fetched branch is stored in \nFETCH_HEAD and that is what might be worthwile for a reflog.\n\n\nNicolas\n"},{"id":"180553","messageId":"20111208034038.GA2280@sigill.intra.peff.net","threadId":"29070","inReplyTo":"alpine.LFD.2.02.1112072227510.2907@xanadu.home","subject":"Re: git auto-repack is broken...","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-12-08T03:40:38Z","receivedAt":"2011-12-08T03:40:38Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 07, 2011 at 10:35:00PM -0500, Nicolas Pitre wrote:\n\n> And I don't think this particular case is interesting anyway as the \n> reflogs for the various branches alre already involved. I was thinking \n> more about the \"git fetch git://some.random.repo foobar\" case where the \n> summary also explicitly shows:\n> \n> From: git://some.random.repo\n> ...... foobar -> FETCH_HEAD\n> \n> In that case the only reference to the fetched branch is stored in \n> FETCH_HEAD and that is what might be worthwile for a reflog.\n\nI agree that is the interesting case. Perhaps we could just not bother\nwriting the other case into the reflog at all. So the reflog would be\nsensible and contain only the set of things they had fetched or pulled\nexplicitly by URL. If they really want to do a multi-ref one-off fetch\nfrom some URL, then we write multiple reflog entries. But at least the\nuser is very aware of what they've done, so they're not surprised by the\nreflog advancing by more than 1 entry.\n\n-Peff\n"},{"id":"180675","messageId":"7v4nx9hcmv.fsf@alter.siamese.dyndns.org","threadId":"29070","inReplyTo":"alpine.LFD.2.02.1112071709250.2907@xanadu.home","subject":"Re: git auto-repack is broken...","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-09T17:35:04Z","receivedAt":"2011-12-09T17:35:04Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre writes:\n\n> On Sat, 3 Dec 2011, Brandon Casey wrote:\n>\n>> Linus's scenario of fetching a lot of stuff that never actually makes\n>> it into the reflogs is still a valid problem. I'm not sure that\n>> people who don't know what they are doing are going to run into this\n>> problem though. Since he fetches a lot of stuff without ever checking\n>> it out or creating a branch from it, potentially many objects become\n>> unreferenced every time FETCH_HEAD changes.\n>\n> Maybe FETCH_HEAD should have a reflog too?\n\nIt is a feature that the objects that were fetched for a quick peek become\nimmediately unreferenced and eligible for early removal unless they are\nkept somewhere, e.g. remote tracking refs. What problem are we trying to\nsolve?\n\nI thought everybody agreed that the current expire window for unreachable\nobjects is way too conservative, especially given that the only purpose of\nthat window is to protect live objects from concurrent gcs. Perhaps the\nonly thing we need to do is to trim that window down to say 2 days or even\n8 hours?\n"},{"id":"180681","messageId":"alpine.LFD.2.02.1112091331370.2907@xanadu.home","threadId":"29070","inReplyTo":"7v4nx9hcmv.fsf@alter.siamese.dyndns.org","subject":"Re: git auto-repack is broken...","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2011-12-09T18:34:04Z","receivedAt":"2011-12-09T18:34:04Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 9 Dec 2011, Junio C Hamano wrote:\n\n> Nicolas Pitre writes:\n> \n> > On Sat, 3 Dec 2011, Brandon Casey wrote:\n> >\n> >> Linus's scenario of fetching a lot of stuff that never actually makes\n> >> it into the reflogs is still a valid problem. I'm not sure that\n> >> people who don't know what they are doing are going to run into this\n> >> problem though. Since he fetches a lot of stuff without ever checking\n> >> it out or creating a branch from it, potentially many objects become\n> >> unreferenced every time FETCH_HEAD changes.\n> >\n> > Maybe FETCH_HEAD should have a reflog too?\n> \n> It is a feature that the objects that were fetched for a quick peek become\n> immediately unreferenced and eligible for early removal unless they are\n> kept somewhere, e.g. remote tracking refs. What problem are we trying to\n> solve?\n\nThis is indeed a tangential observation to the expiration delay. I was \njust suggesting that having a reflog for FETCH_HEAD in the case when you \nfetch a branch with an explicit URL might be handy.\n\n\nNicolas\n"}]}