{"thread":{"id":"62992","subject":"'git gc auto' didn't trigger on large reflog","startedAt":"2025-02-22T23:00:11Z","lastAt":"2025-02-26T16:10:25Z","messageCount":5,"participants":["Markus Gerstel","Patrick Steinhardt","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"512877","messageId":"e650f4e4-e267-4f1f-bb3a-c71b1fe0b276@uxp.de","threadId":"62992","inReplyTo":null,"subject":"'git gc auto' didn't trigger on large reflog","fromName":"Markus Gerstel","fromEmail":"2025@uxp.de","sentAt":"2025-02-22T22:50:25Z","receivedAt":"2025-02-22T23:00:11Z","isPatch":false,"sender":{"key":"2025@uxp.de","avatar":null},"body":"Hi everyone,\n\nI was looking on a machine that does not normally get any attention. On \nthis machine a daily cronjob has been running\n\n     git checkout -q master && git fetch && git reset --hard \norigin/master && git gc --auto\n\nfor 6 years. The git directory now contains a .git/logs/HEAD file of \n180MB with 823921 lines.\n\nThe repo config contains\n\n[core]\n         repositoryformatversion = 0\n         filemode = true\n         bare = false\n         logallrefupdates = true\n\nand the system git version is 2.36.6.\n\nI can't change the git version -or install my own one- so I can't tell \nif this has been fixed since.\nA manual git gc fixed everything, so I amended the cronjob to just do \nthat instead.\n\nI was just slightly surprised (and amused) because I expected 'git gc \n--auto' to pick this up, so I thought I'd share this with you.\n\nThanks\n\n-Markus\n\n"},{"id":"512890","messageId":"Z7xQey4S0ewT5rs7@pks.im","threadId":"62992","inReplyTo":"e650f4e4-e267-4f1f-bb3a-c71b1fe0b276@uxp.de","subject":"Re: 'git gc auto' didn't trigger on large reflog","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-24T10:56:59Z","receivedAt":"2025-02-24T10:57:10Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, Feb 22, 2025 at 10:50:25PM +0000, Markus Gerstel wrote:\n> Hi everyone,\n> \n> I was looking on a machine that does not normally get any attention. On this\n> machine a daily cronjob has been running\n> \n>     git checkout -q master && git fetch && git reset --hard origin/master &&\n> git gc --auto\n> \n> for 6 years. The git directory now contains a .git/logs/HEAD file of 180MB\n> with 823921 lines.\n> \n> The repo config contains\n> \n> [core]\n>         repositoryformatversion = 0\n>         filemode = true\n>         bare = false\n>         logallrefupdates = true\n> \n> and the system git version is 2.36.6.\n> \n> I can't change the git version -or install my own one- so I can't tell\n> if this has been fixed since. A manual git gc fixed everything, so I\n> amended the cronjob to just do that instead.\n> \n> I was just slightly surprised (and amused) because I expected 'git gc\n> --auto' to pick this up, so I thought I'd share this with you.\n\nIt's a bit funny, but whether or not `git gc --auto` does anything\nsolely depends on the state of the object database. This is figured out\nin `need_to_gc()`, which returns a truish value if either:\n\n  - You have too many packfiles in the repository.\n\n  - You have too many loose objects in the repository.\n\nIf these prerequisites aren't met, then git-gc(1) will skip any other\nwork unrelated to objects, as well, including pruning reflogs.\n\nSo given your above sequence of commands:\n\n>     git checkout -q master && git fetch && git reset --hard origin/master &&\n> git gc --auto\n\nYou may hit an edge case, depending on whether or not git-fetch(1) ends\nup fetching changes. While git-checkout(1) won't write any reflogs if\nnothing changes, git-reset(1) writes a reflog entry regardless of\nwhether it performs an \"actual\" change. So if git-fetch(1) ends up never\nfetching anything you don't accumulate new loose objects or packfiles,\nbut do end up writing a new reflog entry every single time. The\nconditions mentioned above won't trigger, and thus the reflog is never\npruned, either.\n\nPatrick\n"},{"id":"512918","messageId":"xmqqeczn70pg.fsf@gitster.g","threadId":"62992","inReplyTo":"Z7xQey4S0ewT5rs7@pks.im","subject":"Re: 'git gc auto' didn't trigger on large reflog","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-02-24T16:43:23Z","receivedAt":"2025-02-24T16:43:27Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> It's a bit funny, but whether or not `git gc --auto` does anything\n> solely depends on the state of the object database.\n\nI guess after adding \"auto\", we haven't been careful enough to\nupdate the triggering condition as we added new kinds of \"garbage\"\nto collect?  Should we make an exhausitive and authoritative list of\ngc tasks, document them, and make sure \"--auto\" pays attention?\n\nOther than objects (packing loose ones, pruning unreferenced loose\nones or packing them into cruft packs), we seem to check reflog,\nworktree, and rerere database.\n\nI do not think there is a readily usable API to query how much stale\ndata is in reflogs that are more than N seconds old, without which\n\"gc --auto\" cannot make decisions.  I am reasonably sure rerere API\ndoes not give you such data, either.  I have no idea about the\ntriggering condition of \"worktree prune\".\n\n\n"},{"id":"513073","messageId":"Z779d7SnW5j8XcOb@pks.im","threadId":"62992","inReplyTo":"xmqqeczn70pg.fsf@gitster.g","subject":"Re: 'git gc auto' didn't trigger on large reflog","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-26T11:39:35Z","receivedAt":"2025-02-26T11:39:44Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Feb 24, 2025 at 08:43:23AM -0800, Junio C Hamano wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n> \n> > It's a bit funny, but whether or not `git gc --auto` does anything\n> > solely depends on the state of the object database.\n> \n> I guess after adding \"auto\", we haven't been careful enough to\n> update the triggering condition as we added new kinds of \"garbage\"\n> to collect?  Should we make an exhausitive and authoritative list of\n> gc tasks, document them, and make sure \"--auto\" pays attention?\n\nMaybe. But maybe a better solution would be to build this into\ngit-maintenance(1) instead, which is a lot more fine-grained. It already\nhas properly defined subtasks, and each of these subtasks has an\noptional callback function that makes it only run as-needed.\n\nSo from my perspective we should:\n\n  - Expand git-maintenance(1) to gain a new task for expiring reflogs.\n\n  - Adapt it to not use git-gc(1) anymore, but instead use the specific\n    subtasks.\n\nIt also allows us to iterate a lot more on the actual tasks run by the\ncommand and make them configurable. It would for example allow us to\neventually enable incremental repacking via multi-pack indices or\ngeometric repacking.\n\n> Other than objects (packing loose ones, pruning unreferenced loose\n> ones or packing them into cruft packs), we seem to check reflog,\n> worktree, and rerere database.\n> \n> I do not think there is a readily usable API to query how much stale\n> data is in reflogs that are more than N seconds old, without which\n> \"gc --auto\" cannot make decisions.  I am reasonably sure rerere API\n> does not give you such data, either.  I have no idea about the\n> triggering condition of \"worktree prune\".\n\nNo, there isn't, and computing it is also potentially expensive. You\nbasically have to iterate through each reflog and then also iterate\nthrough all of its reflog entries to figure out whether anything needs\ncleaning or not.\n\nBut probably we can come up with clever heuristics instead that don't\nrequire us to be this thorough. We could for example just read the\n\"HEAD\" reflog and figure out whether it contains reflog entries that\nwould be pruned.\n\nPatrick\n"},{"id":"513107","messageId":"xmqq4j0gg00h.fsf@gitster.g","threadId":"62992","inReplyTo":"Z779d7SnW5j8XcOb@pks.im","subject":"Re: 'git gc auto' didn't trigger on large reflog","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-02-26T16:10:22Z","receivedAt":"2025-02-26T16:10:25Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> No, there isn't, and computing it is also potentially expensive. You\n> basically have to iterate through each reflog and then also iterate\n> through all of its reflog entries to figure out whether anything needs\n> cleaning or not.\n>\n> But probably we can come up with clever heuristics instead that don't\n> require us to be this thorough. We could for example just read the\n> \"HEAD\" reflog and figure out whether it contains reflog entries that\n> would be pruned.\n\nAs we should be able to \"seek\" to implement HEAD@{2.months.ago}, I'd\nimagine that we should be able to ask \"give me the oldest entry in\nyour log\" to a ref.  Ask that question to a handful of refs that\nhave been most recently modified (with the theory that a ref that is\nmore often modified is also likely to have been touched in the\nrecent past---your HEAD heuristics is a good approximation), and we\nlearn fairly cheaply if it is likely that we have entries to be\nexpired.\n"}]}