{"thread":{"id":"47597","subject":"git gc --auto yelling at users where a repo legitimately has >6700 loose objects","startedAt":"2018-01-11T21:33:26Z","lastAt":"2018-02-08T16:23:58Z","messageCount":9,"participants":["Ævar Arnfjörð Bjarmason","Duy Nguyen","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"336483","messageId":"87inc89j38.fsf@evledraar.gmail.com","threadId":"47597","inReplyTo":null,"subject":"git gc --auto yelling at users where a repo legitimately has >6700 loose objects","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-01-11T21:33:15Z","receivedAt":"2018-01-11T21:33:26Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"I recently disabled gc.auto=0 and my nightly aggressive repack script on\nour big monorepo across our infra, relying instead on git gc --auto in\nthe background to just do its thing.\n\nI didn't want users to wait for git-gc, and I'd written this nightly\ncronjob before git-gc learned to detach to the background.\n\nBut now I have git-gc on some servers yelling at users on every pull\ncommand:\n\n    warning: There are too many unreachable loose objects; run 'git prune' to remove them.\n\nThe reason is that I have all the values at git's default settings, and\nthere legitimately are >~6700 loose objects that were created in the\nlast 2 weeks.\n\nFor those rusty on git-gc's defaults, this is what it looks like in this\nscenario:\n\n 1. User runs \"git pull\"\n 2. git gc --auto is called, there are >6700 loose objects\n 3. it forks into the background, tries to prune and repack, objects\n    older than gc.pruneExpire (2.weeks.ago) are pruned.\n 4. At the end of all this, we check *again* if we have >6700 objects,\n    if we do we print \"run 'git prune'\" to .git/gc.log, and will just\n    emit that error for the next day before trying again, at which point\n    we unlink the gc.log and retry, see gc.logExpiry.\n\nRight now I've just worked around this by setting gc.pruneExpire to a\nlower value (4.days.ago). But there's a larger issue to be addressed\nhere, and I'm not sure how.\n\nWhen the warning was added in [1] it didn't know to detach to the\nbackground yet, that came in [2], shortly after came gc.log in [3].\n\nWe could add another gc.auto-like limit, which could be set at some\nhigher value than gc.auto. \"Hey if I have more than 6700 loose objects,\nprune the <2wks old ones, but if at the end there's still >6700 I don't\nwant to hear about it unless there's >6700*N\".\n\nI thought I'd just add that, but the details of how to pass that message\naround get nasty. With that solution we *also* don't want git gc to\nstart churning in the background once we reach >6700 objects, so we need\nsomething like gc.logExpiry which defers the gc until the next day. We\nmight need to create .git/gc-waitabit.marker, ew.\n\nMore generally, these hard limits seem contrary to what the user cares\nabout. E.g. I suspect that most of these loose objects come from\nbranches since deleted in upstream, whose objects could have a different\nretention policy.\n\nOr we could say \"I want 2 weeks of objects, but if that runs against the\n6700 limit just keep the latest 6700/2\".\n\n1. a087cc9819 (\"git-gc --auto: protect ourselves from accumulated\n   cruft\", 2007-09-17)\n2. 9f673f9477 (\"gc: config option for running --auto in background\",\n   2014-02-08)\n3. 329e6e8794 (\"gc: save log from daemonized gc --auto and print it next\n   time\", 2015-09-19)\n"},{"id":"336498","messageId":"CACsJy8AWO5Vk-Qz3VVBUezWL=oAd9YkeGq=_TXGSb0GSs5bLcg@mail.gmail.com","threadId":"47597","inReplyTo":"87inc89j38.fsf@evledraar.gmail.com","subject":"Re: git gc --auto yelling at users where a repo legitimately has >6700 loose objects","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-01-12T12:07:04Z","receivedAt":"2018-01-12T12:07:40Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Jan 12, 2018 at 4:33 AM, Ævar Arnfjörð Bjarmason\n<avarab@gmail.com> wrote:\n> For those rusty on git-gc's defaults, this is what it looks like in this\n> scenario:\n>\n>  1. User runs \"git pull\"\n>  2. git gc --auto is called, there are >6700 loose objects\n>  3. it forks into the background, tries to prune and repack, objects\n>     older than gc.pruneExpire (2.weeks.ago) are pruned.\n>  4. At the end of all this, we check *again* if we have >6700 objects,\n>     if we do we print \"run 'git prune'\" to .git/gc.log, and will just\n>     emit that error for the next day before trying again, at which point\n>     we unlink the gc.log and retry, see gc.logExpiry.\n>\n> Right now I've just worked around this by setting gc.pruneExpire to a\n> lower value (4.days.ago). But there's a larger issue to be addressed\n> here, and I'm not sure how.\n>\n> When the warning was added in [1] it didn't know to detach to the\n> background yet, that came in [2], shortly after came gc.log in [3].\n>\n> We could add another gc.auto-like limit, which could be set at some\n> higher value than gc.auto. \"Hey if I have more than 6700 loose objects,\n> prune the <2wks old ones, but if at the end there's still >6700 I don't\n> want to hear about it unless there's >6700*N\".\n\nYes it's about time we make too_many_loose_objects() more accurate and\ncomplain less, especially when the complaint is useless.\n\n> I thought I'd just add that, but the details of how to pass that message\n> around get nasty. With that solution we *also* don't want git gc to\n> start churning in the background once we reach >6700 objects, so we need\n> something like gc.logExpiry which defers the gc until the next day. We\n> might need to create .git/gc-waitabit.marker, ew.\n\nHmm.. could we save the info from the last run to help the next one?\nIf the last gc --auto (which does try to remove some loose objects)\nleaves 6700 objects still loose, then it's \"clear\" that the next run\nmay also leave those loose. If we save that number somewhere (gc.log\ntoo?) too_many_loose_objects() can read back and subtract it from the\nestimation and may decide not to do gc at all since the number of\nloose-and-prunable objects is below threshold.\n\nThe problem is of course these 6700 will gradually become prunable\nover time. We can't just substract the same constant forever. Perhaps\nwe can do something based on gc.pruneExpire?\n\nSay gc.pruneExpires specifies to keep objects in two weeks, we assume\nthese object create time is spread out equally over 14 days. So after\none day, 6700/14 objects are supposed to be prune-able and part of\ntoo_many_loose_objects estimation. The gc--auto that is run two weeks\nsince the first run would count all loose objects as prunable again.\n\n> More generally, these hard limits seem contrary to what the user cares\n> about. E.g. I suspect that most of these loose objects come from\n> branches since deleted in upstream, whose objects could have a different\n> retention policy.\n\nEr.. what retention policy? I think gc.pruneExpire is the only thing\nthat can keep loose objects around?\n\nBTW\n\n> But now I have git-gc on some servers yelling at users on every pull\n> command:\n>\n>    warning: There are too many unreachable loose objects; run 'git prune' to remove them.\n\nWhy do we yell at the users when some maintenance thing is supposed to\nbe done on the server side? If this is the case, should gc have some\nway to yell at the admin instead?\n-- \nDuy\n"},{"id":"336509","messageId":"CACsJy8CUYosOGK5tn0C=t=SkbS-fyaSxp536zx+9jh_O+WNaEQ@mail.gmail.com","threadId":"47597","inReplyTo":"CACsJy8AWO5Vk-Qz3VVBUezWL=oAd9YkeGq=_TXGSb0GSs5bLcg@mail.gmail.com","subject":"Re: git gc --auto yelling at users where a repo legitimately has >6700 loose objects","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-01-12T13:41:32Z","receivedAt":"2018-01-12T13:42:09Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Jan 12, 2018 at 7:07 PM, Duy Nguyen <pclouds@gmail.com> wrote:\n>> More generally, these hard limits seem contrary to what the user cares\n>> about. E.g. I suspect that most of these loose objects come from\n>> branches since deleted in upstream, whose objects could have a different\n>> retention policy.\n>\n> Er.. what retention policy? I think gc.pruneExpire is the only thing\n> that can keep loose objects around?\n\nEr... I think I know what you meant now. Loose objects can come from\nthree sources: worktree (git-hash-object and friends),\ngit-unpack-objects and unreachable objects in packs released back by\ngit-repack.\n\nThe last one could be a result of a branch deletion like you said.\nDepending on the branch size, you could release back a large amount of\nobjects in loose form at the same time. This really skews my \"create\ntime distributed equally\" model and the new estimation in\ntoo_many_objects() probably won't help you much either. If only we\nhave a way to count all these objects as \"one\"... but putting these\nback in a pack hurts obj lookup performance...\n-- \nDuy\n"},{"id":"336510","messageId":"20180112134609.GB7880@sigill.intra.peff.net","threadId":"47597","inReplyTo":"87inc89j38.fsf@evledraar.gmail.com","subject":"Re: git gc --auto yelling at users where a repo legitimately has >6700 loose objects","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-01-12T13:46:09Z","receivedAt":"2018-01-12T13:46:18Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 11, 2018 at 10:33:15PM +0100, Ævar Arnfjörð Bjarmason wrote:\n\n>  4. At the end of all this, we check *again* if we have >6700 objects,\n>     if we do we print \"run 'git prune'\" to .git/gc.log, and will just\n>     emit that error for the next day before trying again, at which point\n>     we unlink the gc.log and retry, see gc.logExpiry.\n> \n> Right now I've just worked around this by setting gc.pruneExpire to a\n> lower value (4.days.ago). But there's a larger issue to be addressed\n> here, and I'm not sure how.\n\nIMHO the right solution is to stop exploding loose objects, and instead\nwrite them all into a \"cruft\" pack. That's more efficient, to boot\n(since it doesn't waste inodes, and may even retain deltas between cruft\nobjects).\n\nBut there are some tricks around timestamps. I wrote up some thoughts\nin:\n\n  https://public-inbox.org/git/20170610080626.sjujpmgkli4muh7h@sigill.intra.peff.net/\n\nand downthread from there.\n\n-Peff\n"},{"id":"336511","messageId":"20180112142305.GA338@ash","threadId":"47597","inReplyTo":"20180112134609.GB7880@sigill.intra.peff.net","subject":"Re: git gc --auto yelling at users where a repo legitimately has >6700 loose objects","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-01-12T14:23:05Z","receivedAt":"2018-01-12T14:23:18Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Jan 12, 2018 at 08:46:09AM -0500, Jeff King wrote:\n> On Thu, Jan 11, 2018 at 10:33:15PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> \n> >  4. At the end of all this, we check *again* if we have >6700 objects,\n> >     if we do we print \"run 'git prune'\" to .git/gc.log, and will just\n> >     emit that error for the next day before trying again, at which point\n> >     we unlink the gc.log and retry, see gc.logExpiry.\n> > \n> > Right now I've just worked around this by setting gc.pruneExpire to a\n> > lower value (4.days.ago). But there's a larger issue to be addressed\n> > here, and I'm not sure how.\n> \n> IMHO the right solution is to stop exploding loose objects, and instead\n> write them all into a \"cruft\" pack. That's more efficient, to boot\n> (since it doesn't waste inodes, and may even retain deltas between cruft\n> objects).\n> \n> But there are some tricks around timestamps. I wrote up some thoughts\n> in:\n> \n>   https://public-inbox.org/git/20170610080626.sjujpmgkli4muh7h@sigill.intra.peff.net/\n> \n> and downthread from there.\n\nMy thoughts were moving towards that \"multiple cruft packs\" idea in\nyour last email of that thread [1]. I'll quote it here so people don't\nhave to open the link\n\n> > Why can't we generate a new cruft-pack on every gc run that\n> > detects too many unreachable objects? That would not be as\n> > efficient as a single cruft-pack but it should be way more\n> > efficient than the individual objects, no?\n> > \n> > Plus, chances are that the existing cruft-packs are purged with\n> > the next gc run anyways.\n> \n> Interesting idea. Here are some thoughts in random order.\n> \n> That loses some delta opportunities between the cruft packs, but\n> that's certainly no worse than the all-loose storage we have today.\n\nDoes it also affect deltas when we copy some objects to the new\nrepacked pack (e.g. some objects in the cruft pack getting referenced\nagain)? I remember we do reuse deltas sometimes but not in very\ndetail. I guess we probably won't suffer any suboptimal deltas ...\n\n> \n> One nice aspect is that it means cruft objects don't incur any I/O\n> cost during a repack.\n\nBut cruft packs do incur object lookup cost since we still go through\nall packs linearly. The multi-pack index being discussed recently\nwould help. But even without that, packs are sorted by mtime and old\ncruft packs won't affect as much I guess, as long as there aren't a\nzillion cruft packs around. Then even prepare_packed_git() is hit.\n\n> It doesn't really solve the \"too many loose objects after gc\"\n> problem.  It just punts it to \"too many packs after gc\". This is\n> likely to be better because the number of packs would scale with the\n> number of gc runs, rather than the number of crufty objects. But\n> there would still be corner cases if you run gc frequently. Maybe\n> that would be acceptable.\n> \n> I'm not sure how the pruning process would work, especially with\n> respect to objects reachable from other unreachable-but-recent\n> objects. Right now the repack-and-delete procedure is done by\n> git-repack, and is basically:\n> \n>   1. Get a list of all of the current packs.\n> \n>   2. Ask pack-objects to pack everything into a new pack. Normally this\n>      is reachable objects, but we also include recent objects and\n>      objects reachable from recent objects. And of course with \"-k\" all\n>      objects are kept.\n> \n>   3. Delete everything in the list from (1), under the assumption that\n>      anything worth keeping was repacked in step (2), and anything else\n>      is OK to drop.\n> \n> So if there are regular packs and cruft packs, we'd have to know in\n> step 3 which are which. We'd delete the regular ones, whose objects\n> have all been migrated to the new pack (either a \"real\" one or a\n> cruft one), but keep the crufty ones whose timestamps are still\n> fresh.\n> \n> That's a small change, and works except for one thing: the reachable\n> from recent objects. You can't just delete a whole cruft pack. Some\n> of its objects may be reachable from objects in other cruft packs\n> that we're keeping. In other words, you have cruft packs where you\n> want to keep half of the objects they contain. How do you do that?\n\nDo we have to? Those reachable from recent objects must have ended up\nin the new pack created at step 2, correct? Which means we can safely\nremove the whole pack.\n\nThose reachable from other cruft packs. I'm not sure if it's different\nfrom when these objects are loose. If a loose object A depends on B,\nbut B is much older than A, then B may get pruned anyway while A stays\n(does not sound right if A gets reused).\n\n> I think you'd have to make pack-objects aware of the concept of\n> cruft packs, and that it should include reachable-from-recent\n> objects in the new pack only if they're in a cruft pack that is\n> going to be deleted. So those objects would be \"rescued\" from the\n> cruft pack before it goes away and migrated to the new cruft\n> pack. That would effectively refresh their timestamp, but that's\n> fine. They're reachable from objects with that fresh timestamp\n> already, so effectively they couldn't be deleted until that\n> timestamp is hit.\n> \n> So I think it's do-able, but it is a little complicated.\n\n[1] https://public-inbox.org/git/20170620140837.fq3wxb63lnqay6xz@sigill.intra.peff.net/\n--\nDuy\n"},{"id":"336513","messageId":"87d12f9lx1.fsf@evledraar.gmail.com","threadId":"47597","inReplyTo":"CACsJy8AWO5Vk-Qz3VVBUezWL=oAd9YkeGq=_TXGSb0GSs5bLcg@mail.gmail.com","subject":"Re: git gc --auto yelling at users where a repo legitimately has >6700 loose objects","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-01-12T14:44:26Z","receivedAt":"2018-01-12T14:44:35Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jan 12 2018, Duy Nguyen jotted:\n\n> On Fri, Jan 12, 2018 at 4:33 AM, Ævar Arnfjörð Bjarmason\n> <avarab@gmail.com> wrote:\n>> For those rusty on git-gc's defaults, this is what it looks like in this\n>> scenario:\n>>\n>>  1. User runs \"git pull\"\n>>  2. git gc --auto is called, there are >6700 loose objects\n>>  3. it forks into the background, tries to prune and repack, objects\n>>     older than gc.pruneExpire (2.weeks.ago) are pruned.\n>>  4. At the end of all this, we check *again* if we have >6700 objects,\n>>     if we do we print \"run 'git prune'\" to .git/gc.log, and will just\n>>     emit that error for the next day before trying again, at which point\n>>     we unlink the gc.log and retry, see gc.logExpiry.\n>>\n>> Right now I've just worked around this by setting gc.pruneExpire to a\n>> lower value (4.days.ago). But there's a larger issue to be addressed\n>> here, and I'm not sure how.\n>>\n>> When the warning was added in [1] it didn't know to detach to the\n>> background yet, that came in [2], shortly after came gc.log in [3].\n>>\n>> We could add another gc.auto-like limit, which could be set at some\n>> higher value than gc.auto. \"Hey if I have more than 6700 loose objects,\n>> prune the <2wks old ones, but if at the end there's still >6700 I don't\n>> want to hear about it unless there's >6700*N\".\n>\n> Yes it's about time we make too_many_loose_objects() more accurate and\n> complain less, especially when the complaint is useless.\n>\n>> I thought I'd just add that, but the details of how to pass that message\n>> around get nasty. With that solution we *also* don't want git gc to\n>> start churning in the background once we reach >6700 objects, so we need\n>> something like gc.logExpiry which defers the gc until the next day. We\n>> might need to create .git/gc-waitabit.marker, ew.\n>\n> Hmm.. could we save the info from the last run to help the next one?\n> If the last gc --auto (which does try to remove some loose objects)\n> leaves 6700 objects still loose, then it's \"clear\" that the next run\n> may also leave those loose. If we save that number somewhere (gc.log\n> too?) too_many_loose_objects() can read back and subtract it from the\n> estimation and may decide not to do gc at all since the number of\n> loose-and-prunable objects is below threshold.\n>\n> The problem is of course these 6700 will gradually become prunable\n> over time. We can't just substract the same constant forever. Perhaps\n> we can do something based on gc.pruneExpire?\n>\n> Say gc.pruneExpires specifies to keep objects in two weeks, we assume\n> these object create time is spread out equally over 14 days. So after\n> one day, 6700/14 objects are supposed to be prune-able and part of\n> too_many_loose_objects estimation. The gc--auto that is run two weeks\n> since the first run would count all loose objects as prunable again.\n>\n>> More generally, these hard limits seem contrary to what the user cares\n>> about. E.g. I suspect that most of these loose objects come from\n>> branches since deleted in upstream, whose objects could have a different\n>> retention policy.\n>\n> Er.. what retention policy? I think gc.pruneExpire is the only thing\n> that can keep loose objects around?\n\nYou answered this yourself in\nCACsJy8CUYosOGK5tn0C=t=SkbS-fyaSxp536zx+9jh_O+WNaEQ@mail.gmail.com, yeah\nI mean loose objects from branch deletions.\n\nMore generally, the reason we even have the 2 week limit is to pick a\ngood trade-off between performance and not losing someone's work that\nthey e.g. \"git add\"-ed but never committed.\n\nI'm suggesting (but don't know if this is worth it, especially given\nJeff's comments) that one smarter approach might be to track where the\nobjects came from (e.g. by keeping reflogs for deleted upstream branches\nfor $expiry_time).\n\nThen we could immediately delete loose objects we got from upstream\nbranches (or delete them more aggressively), while treating objects that\nwere originally created in the local repository differently.\n\n>> But now I have git-gc on some servers yelling at users on every pull\n>> command:\n>>\n>>    warning: There are too many unreachable loose objects; run 'git prune' to remove them.\n>\n> Why do we yell at the users when some maintenance thing is supposed to\n> be done on the server side? If this is the case, should gc have some\n> way to yell at the admin instead?\n\nSorry I didn't clarify this, this is a shared server (rollout system\nwith staged checkouts) that users log into and stage/test a rollout from\nthe git repo, so not the git server.\n\nBecause it's a shared repo there's a lot more loose object churn, Mostly\ndue to pulling more often (and thus more branches that later get\ndeleted), but also from rebasing and whatnot in the rollout repo.\n"},{"id":"336552","messageId":"20180113095849.GA30511@sigill.intra.peff.net","threadId":"47597","inReplyTo":"20180112142305.GA338@ash","subject":"Re: git gc --auto yelling at users where a repo legitimately has >6700 loose objects","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-01-13T09:58:49Z","receivedAt":"2018-01-13T09:59:02Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 12, 2018 at 09:23:05PM +0700, Duy Nguyen wrote:\n\n> > > Why can't we generate a new cruft-pack on every gc run that\n> > > detects too many unreachable objects? That would not be as\n> > > efficient as a single cruft-pack but it should be way more\n> > > efficient than the individual objects, no?\n> > > \n> > > Plus, chances are that the existing cruft-packs are purged with\n> > > the next gc run anyways.\n> > \n> > Interesting idea. Here are some thoughts in random order.\n> > \n> > That loses some delta opportunities between the cruft packs, but\n> > that's certainly no worse than the all-loose storage we have today.\n> \n> Does it also affect deltas when we copy some objects to the new\n> repacked pack (e.g. some objects in the cruft pack getting referenced\n> again)? I remember we do reuse deltas sometimes but not in very\n> detail. I guess we probably won't suffer any suboptimal deltas ...\n\nWe always reuse deltas that are coming from one pack into another pack,\nunless the base isn't present in the new pack. So we'd retain existing\ndeltas. What you'd miss out on is just two versions of a file in two\nseparate cruft packs could not be delta'd together.\n\n> > One nice aspect is that it means cruft objects don't incur any I/O\n> > cost during a repack.\n> \n> But cruft packs do incur object lookup cost since we still go through\n> all packs linearly. The multi-pack index being discussed recently\n> would help. But even without that, packs are sorted by mtime and old\n> cruft packs won't affect as much I guess, as long as there aren't a\n> zillion cruft packs around. Then even prepare_packed_git() is hit.\n\nThe cruft packs should behave pretty well with the mru list. We'd never\nas for an object in such a pack during normal operations, so they'd end\nup at the end of the list (the big exception is abbreviation, which has\nto look in every single pack).\n\nI'm not sure how many cruft packs you'd end up with in practice. If it's\none per auto-gc, then probably you're only generating one every few\ndays, and cleaning up old ones as you go.\n\nI do still kind of favor having a single cruft pack, though, just\nbecause it makes it simpler to reason about these sorts of things (but\nthen you need to mark individual object timestamps).\n\n> > I'm not sure how the pruning process would work, especially with\n> > respect to objects reachable from other unreachable-but-recent\n> > objects. Right now the repack-and-delete procedure is done by\n> > git-repack, and is basically:\n> > \n> >   1. Get a list of all of the current packs.\n> > \n> >   2. Ask pack-objects to pack everything into a new pack. Normally this\n> >      is reachable objects, but we also include recent objects and\n> >      objects reachable from recent objects. And of course with \"-k\" all\n> >      objects are kept.\n> > \n> >   3. Delete everything in the list from (1), under the assumption that\n> >      anything worth keeping was repacked in step (2), and anything else\n> >      is OK to drop.\n> > \n> > So if there are regular packs and cruft packs, we'd have to know in\n> > step 3 which are which. We'd delete the regular ones, whose objects\n> > have all been migrated to the new pack (either a \"real\" one or a\n> > cruft one), but keep the crufty ones whose timestamps are still\n> > fresh.\n> > \n> > That's a small change, and works except for one thing: the reachable\n> > from recent objects. You can't just delete a whole cruft pack. Some\n> > of its objects may be reachable from objects in other cruft packs\n> > that we're keeping. In other words, you have cruft packs where you\n> > want to keep half of the objects they contain. How do you do that?\n> \n> Do we have to? Those reachable from recent objects must have ended up\n> in the new pack created at step 2, correct? Which means we can safely\n> remove the whole pack.\n\nNo, I think just I wrote (2) poorly. We repack the reachable objects,\nbut the recent ones (and things reachable only from them) are not\nactually packed, but turned loose.\n\nAnd of course in a cruft-packed world they'd end up in a cruft pack.\n\n> Those reachable from other cruft packs. I'm not sure if it's different\n> from when these objects are loose. If a loose object A depends on B,\n> but B is much older than A, then B may get pruned anyway while A stays\n> (does not sound right if A gets reused).\n\nHopefully not, after d3038d22f9 (prune: keep objects reachable from\nrecent objects, 2014-10-15). :)\n\n-Peff\n"},{"id":"336553","messageId":"20180113100734.GA30698@sigill.intra.peff.net","threadId":"47597","inReplyTo":"87d12f9lx1.fsf@evledraar.gmail.com","subject":"Re: git gc --auto yelling at users where a repo legitimately has >6700 loose objects","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-01-13T10:07:34Z","receivedAt":"2018-01-13T10:07:42Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 12, 2018 at 03:44:26PM +0100, Ævar Arnfjörð Bjarmason wrote:\n\n> More generally, the reason we even have the 2 week limit is to pick a\n> good trade-off between performance and not losing someone's work that\n> they e.g. \"git add\"-ed but never committed.\n> \n> I'm suggesting (but don't know if this is worth it, especially given\n> Jeff's comments) that one smarter approach might be to track where the\n> objects came from (e.g. by keeping reflogs for deleted upstream branches\n> for $expiry_time).\n\nI don't think reflogs would help here. We consider reflog'd objects as\n\"reachable\". So you'd still have something like this:\n\n  1. You delete a branch. Reflog still mentions its commits.\n\n  2. You run gc (or auto-gc). Those objects are still retained in the\n     main pack due to the reachability rule. This may happen multiple\n     times, and each time their \"timestamp\" is updated, because it is\n     really just the timestamp of the containing pack.\n\n  3. 30 days later, you run another gc. The reflog is now past its\n     expiration and is deleted, and now those objects are unreachable.\n     This gc turns them loose, but it still considers them \"recent\" as\n     of the last gc you ran, due to the timestamp thing above.\n\n-Peff\n"},{"id":"338736","messageId":"87eflvmovg.fsf@evledraar.gmail.com","threadId":"47597","inReplyTo":"87inc89j38.fsf@evledraar.gmail.com","subject":"Re: git gc --auto yelling at users where a repo legitimately has >6700 loose objects","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-02-08T16:23:47Z","receivedAt":"2018-02-08T16:23:58Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jan 11 2018, Ævar Arnfjörð Bjarmason jotted:\n\n> I recently disabled gc.auto=0 and my nightly aggressive repack script on\n> our big monorepo across our infra, relying instead on git gc --auto in\n> the background to just do its thing.\n>\n> I didn't want users to wait for git-gc, and I'd written this nightly\n> cronjob before git-gc learned to detach to the background.\n>\n> But now I have git-gc on some servers yelling at users on every pull\n> command:\n>\n>     warning: There are too many unreachable loose objects; run 'git prune' to remove them.\n>\n> The reason is that I have all the values at git's default settings, and\n> there legitimately are >~6700 loose objects that were created in the\n> last 2 weeks.\n>\n> For those rusty on git-gc's defaults, this is what it looks like in this\n> scenario:\n>\n>  1. User runs \"git pull\"\n>  2. git gc --auto is called, there are >6700 loose objects\n>  3. it forks into the background, tries to prune and repack, objects\n>     older than gc.pruneExpire (2.weeks.ago) are pruned.\n>  4. At the end of all this, we check *again* if we have >6700 objects,\n>     if we do we print \"run 'git prune'\" to .git/gc.log, and will just\n>     emit that error for the next day before trying again, at which point\n>     we unlink the gc.log and retry, see gc.logExpiry.\n>\n> Right now I've just worked around this by setting gc.pruneExpire to a\n> lower value (4.days.ago). But there's a larger issue to be addressed\n> here, and I'm not sure how.\n>\n> When the warning was added in [1] it didn't know to detach to the\n> background yet, that came in [2], shortly after came gc.log in [3].\n>\n> We could add another gc.auto-like limit, which could be set at some\n> higher value than gc.auto. \"Hey if I have more than 6700 loose objects,\n> prune the <2wks old ones, but if at the end there's still >6700 I don't\n> want to hear about it unless there's >6700*N\".\n>\n> I thought I'd just add that, but the details of how to pass that message\n> around get nasty. With that solution we *also* don't want git gc to\n> start churning in the background once we reach >6700 objects, so we need\n> something like gc.logExpiry which defers the gc until the next day. We\n> might need to create .git/gc-waitabit.marker, ew.\n>\n> More generally, these hard limits seem contrary to what the user cares\n> about. E.g. I suspect that most of these loose objects come from\n> branches since deleted in upstream, whose objects could have a different\n> retention policy.\n>\n> Or we could say \"I want 2 weeks of objects, but if that runs against the\n> 6700 limit just keep the latest 6700/2\".\n>\n> 1. a087cc9819 (\"git-gc --auto: protect ourselves from accumulated\n>    cruft\", 2007-09-17)\n> 2. 9f673f9477 (\"gc: config option for running --auto in background\",\n>    2014-02-08)\n> 3. 329e6e8794 (\"gc: save log from daemonized gc --auto and print it next\n>    time\", 2015-09-19)\n\nMy just-sent \"How to produce a loose ref+size explosion via pruning +\ngit-gc\", <87fu6bmr0j.fsf@evledraar.gmail.com>\n(https://public-inbox.org/git/87fu6bmr0j.fsf@evledraar.gmail.com/),\nshows an easy way to reproduce this.\n\nAfter the steps outlined there git-gc --auto will end up in a state\nwhere it'll start telling the user off for having too many loose\nobjects.\n"}]}