{"thread":{"id":"62642","subject":"Consider adding pruning of refs to git maintenance","startedAt":"2024-12-16T12:33:40Z","lastAt":"2024-12-23T09:31:06Z","messageCount":11,"participants":["Shubham Kanodia","Patrick Steinhardt","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"509156","messageId":"CAG=Um+0v=BmmYjvBAXs4r4My6zYvpJvcE+0U6SAnxKUcd1-A4w@mail.gmail.com","threadId":"62642","inReplyTo":null,"subject":"Consider adding pruning of refs to git maintenance","fromName":"Shubham Kanodia","fromEmail":"shubham.kanodia10@gmail.com","sentAt":"2024-12-16T12:33:03Z","receivedAt":"2024-12-16T12:33:40Z","isPatch":false,"sender":{"key":"shubham.kanodia10@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8946207?v=4"},"body":"Remote-tracking refs accumulate quickly in large repositories as users\nmerge and delete their branches. While these branches are cleaned up\non the remote, local repositories may retain stale references to\ndeleted branches unless explicitly pruned. The number of local refs\ncan have an impact on git performance of several commands.\n\nGit currently provides two ways for orphan local refs to be cleaned up —\n1. Automated: `fetch.prune` and `fetch.pruneTags` configurations with\n`git fetch/pull`\n2. Manual: `git remote prune`\n\nHowever, both approaches have issues:\n- Full `git fetch/pull` operations are expensive on large\nrepositories, pulling thousands of irrelevant refs\n- Manual `git remote prune` requires user intervention\n\nProposal:\nAdd remote pruning to the daily `git-maintenance` task. This would\nclean stale refs automatically without requiring full fetches or\nmanual intervention.\n\nThis is especially useful for users who historically pulled all\nrefs/tags but now use targeted fetches. Moreover, it decouples the\ncleanup action (pruning) from the action to fetch more refs.\n\nHappy to submit on a patch for the same unless there's something\nobvious that I've missed here.\n"},{"id":"509205","messageId":"Z2Emh42DJkHFGWq7@pks.im","threadId":"62642","inReplyTo":"CAG=Um+0v=BmmYjvBAXs4r4My6zYvpJvcE+0U6SAnxKUcd1-A4w@mail.gmail.com","subject":"Re: Consider adding pruning of refs to git maintenance","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-12-17T07:21:54Z","receivedAt":"2024-12-17T07:22:14Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Dec 16, 2024 at 06:03:03PM +0530, Shubham Kanodia wrote:\n> Remote-tracking refs accumulate quickly in large repositories as users\n> merge and delete their branches. While these branches are cleaned up\n> on the remote, local repositories may retain stale references to\n> deleted branches unless explicitly pruned. The number of local refs\n> can have an impact on git performance of several commands.\n> \n> Git currently provides two ways for orphan local refs to be cleaned up —\n> 1. Automated: `fetch.prune` and `fetch.pruneTags` configurations with\n> `git fetch/pull`\n> 2. Manual: `git remote prune`\n> \n> However, both approaches have issues:\n> - Full `git fetch/pull` operations are expensive on large\n> repositories, pulling thousands of irrelevant refs\n> - Manual `git remote prune` requires user intervention\n\nFair. Neither of those issues feel insurmountable, but I can see why it\ncould make our users lifes easier.\n\n> Proposal:\n> Add remote pruning to the daily `git-maintenance` task. This would\n> clean stale refs automatically without requiring full fetches or\n> manual intervention.\n> \n> This is especially useful for users who historically pulled all\n> refs/tags but now use targeted fetches. Moreover, it decouples the\n> cleanup action (pruning) from the action to fetch more refs.\n\nI think we need to consider a couple of things:\n\n - It's somewhat awkward to have maintenance jobs that interact with a\n remote, as that may not work in contexts where you actually need to\n authenticate. But there is precedent with the \"prefetch\" task, so we\n have already opened that can of worms.\n\n - Maintenance tries to be as non-destructive as reasonably possible,\n and deleting refs certainly is a destructive operation.\n\n - We try to avoid bad interactions with a user that works concurrently\n in the repo that git-maintenance(1) runs in. This is the reason why\n the \"prefetch\" task does not fetch into `refs/remotes`, but into a\n separate ref namespace.\n\nIf we want to have such a feature I'd thus claim that it would be most\nsensible to make it opt-in rather than opt-out. I wouldn't want to be\nsurprised by remote refs vanishing after going to bed, but may be okay\nwith it when I explicitly ask for it.\n\nAt that point one has to raise the question whether it is still all that\nuseful compared to running `git remote prune` manually every now and\nthen. Mostly because explicitly configuring maintenance is probably\nsomething that only power users would do, and those power users would\nlikely know to prune manually.\n\nIn any case, that's just my 2c. I can see a usecase for your feature,\nbut think we should be careful with how it is introduced.\n\n> Happy to submit on a patch for the same unless there's something\n> obvious that I've missed here.\n\nI'm happy to have a look in case you decide to implement this feature.\n\nPatrick\n"},{"id":"509206","messageId":"xmqqcyhq3ge7.fsf@gitster.g","threadId":"62642","inReplyTo":"Z2Emh42DJkHFGWq7@pks.im","subject":"Re: Consider adding pruning of refs to git maintenance","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-17T07:41:52Z","receivedAt":"2024-12-17T07:41:55Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> If we want to have such a feature I'd thus claim that it would be most\n> sensible to make it opt-in rather than opt-out. I wouldn't want to be\n> surprised by remote refs vanishing after going to bed, but may be okay\n> with it when I explicitly ask for it.\n>\n> At that point one has to raise the question whether it is still all that\n> useful compared to running `git remote prune` manually every now and\n> then. Mostly because explicitly configuring maintenance is probably\n> something that only power users would do, and those power users would\n> likely know to prune manually.\n\nI am 100% in agreement with your reasoning. If this thing is to\nexist, it has to be opt-in. We also need to add ample warning why\ndoing this asynchronously behind user's back while the user could be\nworking in the same repository is prone to cause surprises in the\ndocumentation in big red letters.\n\nI however am OK with the idea of having it as an optional \"task\"\nthat is disabled by default in \"git maintenance\". \"git maintenance\"\ncan be viewed as a platform-neutral way to set up scheduled tasks.\n\nA user may be a Git expert who knows what `git remote prune` does,\nand understands and accepts the risk of confusion doing so without\nexplicit \"do it now\" from the end-user, but may not be a Linux or a\nmacOS or a Windows expert to know how to write crontab or its\nequivalents on various schemes to define scheduled tasks.\n\nThanks.\n"},{"id":"509211","messageId":"CAG=Um+1NwB=ymwg+oM62f_W8G=3Gt14UFGe+S2MM3gTOdUcuHg@mail.gmail.com","threadId":"62642","inReplyTo":"xmqqcyhq3ge7.fsf@gitster.g","subject":"Re: Consider adding pruning of refs to git maintenance","fromName":"Shubham Kanodia","fromEmail":"shubham.kanodia10@gmail.com","sentAt":"2024-12-17T11:21:32Z","receivedAt":"2024-12-17T11:22:09Z","isPatch":false,"sender":{"key":"shubham.kanodia10@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8946207?v=4"},"body":"On Tue, Dec 17, 2024 at 1:11 PM Junio C Hamano wrote:\n>\n> Patrick Steinhardt writes:\n>\n> > If we want to have such a feature I'd thus claim that it would be most\n> > sensible to make it opt-in rather than opt-out. I wouldn't want to be\n> > surprised by remote refs vanishing after going to bed, but may be okay\n> > with it when I explicitly ask for it.\n> >\n> > At that point one has to raise the question whether it is still all that\n> > useful compared to running `git remote prune` manually every now and\n> > then. Mostly because explicitly configuring maintenance is probably\n> > something that only power users would do, and those power users would\n> > likely know to prune manually.\n>\n> I am 100% in agreement with your reasoning. If this thing is to\n> exist, it has to be opt-in. We also need to add ample warning why\n> doing this asynchronously behind user's back while the user could be\n> working in the same repository is prone to cause surprises in the\n> documentation in big red letters.\n>\n> I however am OK with the idea of having it as an optional \"task\"\n> that is disabled by default in \"git maintenance\". \"git maintenance\"\n> can be viewed as a platform-neutral way to set up scheduled tasks.\n>\n> A user may be a Git expert who knows what `git remote prune` does,\n> and understands and accepts the risk of confusion doing so without\n> explicit \"do it now\" from the end-user, but may not be a Linux or a\n> macOS or a Windows expert to know how to write crontab or its\n> equivalents on various schemes to define scheduled tasks.\n>\n> Thanks.\n\nThanks for sharing your thoughts here. For some context — I currently\nlook after git performance for a very large repository (60k+ refs)\nwith a lot of active developers.\nI've observed that while power git users keep their repository tidy\neven without runnining maintenance, the majority of users infact don't\nknow (or execute these), and\nrun into performance issues. Easier fleet management was perhaps was\none of the motivations behind adding this to `git-maintenance`.\n\nWhile its pretty rare that someone relies on a `refs/remote/*`\nreference (without creating a local copy of it in `refs/heads`), I can\nsee why it can be surprising and an opt-in\nshould be okay for me.\n\nThanks,\nShubham K\n"},{"id":"509212","messageId":"CAG=Um+0yFYeBQGznkVG6TJeN-U+qySbt-0EbvM6Vd-BcvSCT4g@mail.gmail.com","threadId":"62642","inReplyTo":"CAG=Um+1NwB=ymwg+oM62f_W8G=3Gt14UFGe+S2MM3gTOdUcuHg@mail.gmail.com","subject":"Re: Consider adding pruning of refs to git maintenance","fromName":"Shubham Kanodia","fromEmail":"shubham.kanodia10@gmail.com","sentAt":"2024-12-17T11:24:46Z","receivedAt":"2024-12-17T11:25:23Z","isPatch":false,"sender":{"key":"shubham.kanodia10@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8946207?v=4"},"body":"On Tue, Dec 17, 2024 at 4:51 PM Shubham Kanodia\n wrote:\n>\n> On Tue, Dec 17, 2024 at 1:11 PM Junio C Hamano wrote:\n> >\n> > Patrick Steinhardt writes:\n> >\n> > > If we want to have such a feature I'd thus claim that it would be most\n> > > sensible to make it opt-in rather than opt-out. I wouldn't want to be\n> > > surprised by remote refs vanishing after going to bed, but may be okay\n> > > with it when I explicitly ask for it.\n> > >\n> > > At that point one has to raise the question whether it is still all that\n> > > useful compared to running `git remote prune` manually every now and\n> > > then. Mostly because explicitly configuring maintenance is probably\n> > > something that only power users would do, and those power users would\n> > > likely know to prune manually.\n> >\n> > I am 100% in agreement with your reasoning. If this thing is to\n> > exist, it has to be opt-in. We also need to add ample warning why\n> > doing this asynchronously behind user's back while the user could be\n> > working in the same repository is prone to cause surprises in the\n> > documentation in big red letters.\n> >\n> > I however am OK with the idea of having it as an optional \"task\"\n> > that is disabled by default in \"git maintenance\". \"git maintenance\"\n> > can be viewed as a platform-neutral way to set up scheduled tasks.\n> >\n> > A user may be a Git expert who knows what `git remote prune` does,\n> > and understands and accepts the risk of confusion doing so without\n> > explicit \"do it now\" from the end-user, but may not be a Linux or a\n> > macOS or a Windows expert to know how to write crontab or its\n> > equivalents on various schemes to define scheduled tasks.\n> >\n> > Thanks.\n>\n> Thanks for sharing your thoughts here. For some context — I currently\n> look after git performance for a very large repository (60k+ refs)\n> with a lot of active developers.\n> I've observed that while power git users keep their repository tidy\n> even without runnining maintenance, the majority of users infact don't\n> know (or execute these), and\n> run into performance issues. Easier fleet management was perhaps was\n> one of the motivations behind adding this to `git-maintenance`.\n>\n> While its pretty rare that someone relies on a `refs/remote/*`\n> reference (without creating a local copy of it in `refs/heads`), I can\n> see why it can be surprising and an opt-in\n> should be okay for me.\n>\n> Thanks,\n> Shubham K\n\nFrom aed935055aff819cce591eecbf6697f8e8e7b99e Mon Sep 17 00:00:00 2001\nFrom: Shubham Kanodia \nDate: Tue, 17 Dec 2024 16:05:16 +0530\nSubject: [PATCH] maintenance: add prune-remote-refs task\n\nRemote-tracking refs can accumulate in local repositories even as branches\nare deleted on remotes, impacting git performance. While git fetch --prune\nhelps clean these up, it requires a full fetch operation which is expensive\nfor large repositories.\n\nAdd a new maintenance task 'prune-remote-refs' that runs 'git remote prune'\nfor each configured remote. This provides an automated way to clean up stale\nremote-tracking refs without requiring full fetches or manual intervention.\nThis task is disabled by default.\n\nSigned-off-by: Shubham Kanodia \n---\n Documentation/git-maintenance.txt | 14 ++++++++++\n builtin/gc.c | 42 +++++++++++++++++++++++++++++\n t/t7900-maintenance.sh | 44 +++++++++++++++++++++++++++++++\n 3 files changed, 100 insertions(+)\n\ndiff --git a/Documentation/git-maintenance.txt\nb/Documentation/git-maintenance.txt\nindex 6e6651309d..3e0ca84de2 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -158,6 +158,20 @@ pack-refs::\n need to iterate across many references. See linkgit:git-pack-refs[1]\n for more information.\n\n+prune-remote-refs::\n+ The `prune-remote-refs` task runs `git remote prune` on each remote\n+ repository registered in the local repository. This task helps clean\n+ up deleted remote branches, improving the performance of operations\n+ that iterate through the refs. See linkgit:git-remote[1] for more\n+ information. This task is disabled by default and must be enabled\n+ manually.\n++\n+NOTE: This task is opt-in to prevent unexpected removal of remote refs\n+for users of git-maintenance. It is recommended for power users who\n+are comfortable with configuring maintenance tasks and understand the\n+implications of automatic pruning. For others, running `git remote prune`\n+manually might be more appropriate.\n+\n OPTIONS\n -------\n --auto::\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 4ae5196aed..9acf1d2989 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -20,6 +20,7 @@\n #include \"lockfile.h\"\n #include \"parse-options.h\"\n #include \"run-command.h\"\n+#include \"remote.h\"\n #include \"sigchain.h\"\n #include \"strvec.h\"\n #include \"commit.h\"\n@@ -913,6 +914,40 @@ static int maintenance_opt_schedule(const struct\noption *opt, const char *arg,\n return 0;\n }\n\n+static int collect_remote(struct remote *remote, void *cb_data)\n+{\n+ struct string_list *list = cb_data;\n+\n+ if (!remote->url.nr)\n+ return 0;\n+\n+ string_list_append(list, remote->name);\n+ return 0;\n+}\n+\n+static int maintenance_task_prune_remote(struct maintenance_run_opts\n*opts UNUSED,\n+ struct gc_config *cfg UNUSED)\n+{\n+ struct string_list_item *item;\n+ struct string_list remotes_list = STRING_LIST_INIT_NODUP;\n+ struct child_process child = CHILD_PROCESS_INIT;\n+ int result = 0;\n+\n+ for_each_remote(collect_remote, &remotes_list);\n+\n+ for_each_string_list_item (item, &remotes_list) {\n+ const char *remote_name = item->string;\n+ child.git_cmd = 1;\n+ strvec_pushl(&child.args, \"remote\", \"prune\", remote_name, NULL);\n+\n+ if (run_command(&child))\n+ result = error(_(\"failed to prune '%s'\"), remote_name);\n+ }\n+\n+ string_list_clear(&remotes_list, 0);\n+ return result;\n+}\n+\n /* Remember to update object flag allocation in object.h */\n #define SEEN (1u<<0)\n\n@@ -1375,6 +1410,7 @@ enum maintenance_task_label {\n TASK_GC,\n TASK_COMMIT_GRAPH,\n TASK_PACK_REFS,\n+ TASK_PRUNE_REMOTE_REFS,\n\n /* Leave as final value */\n TASK__COUNT\n@@ -1411,6 +1447,10 @@ static struct maintenance_task tasks[] = {\n maintenance_task_pack_refs,\n pack_refs_condition,\n },\n+ [TASK_PRUNE_REMOTE_REFS] = {\n+ \"prune-remote-refs\",\n+ maintenance_task_prune_remote,\n+ },\n };\n\n static int compare_tasks_by_selection(const void *a_, const void *b_)\n@@ -1505,6 +1545,8 @@ static void initialize_maintenance_strategy(void)\n tasks[TASK_LOOSE_OBJECTS].schedule = SCHEDULE_DAILY;\n tasks[TASK_PACK_REFS].enabled = 1;\n tasks[TASK_PACK_REFS].schedule = SCHEDULE_WEEKLY;\n+ tasks[TASK_PRUNE_REMOTE_REFS].enabled = 0;\n+ tasks[TASK_PRUNE_REMOTE_REFS].schedule = SCHEDULE_DAILY;\n }\n }\n\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 0ce4ba1cbe..60a0c3f835 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -446,6 +446,50 @@ test_expect_success 'pack-refs task' '\n test_subcommand git pack-refs --all --prune err &&\n+ test_subcommand ! git remote prune origin err &&\n test_grep \"at most one\" err\n-- \n2.46.2\n"},{"id":"509235","messageId":"xmqq7c7y13tc.fsf@gitster.g","threadId":"62642","inReplyTo":"CAG=Um+0yFYeBQGznkVG6TJeN-U+qySbt-0EbvM6Vd-BcvSCT4g@mail.gmail.com","subject":"Re: Consider adding pruning of refs to git maintenance","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-17T19:56:31Z","receivedAt":"2024-12-17T19:56:35Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Shubham Kanodia writes:\n\n> On Tue, Dec 17, 2024 at 4:51 PM Shubham Kanodia\n> wrote:\n>>\n>> On Tue, Dec 17, 2024 at 1:11 PM Junio C Hamano wrote:\n>> >\n>> > Patrick Steinhardt writes:\n>> >\n>> > > If we want to have such a feature I'd thus claim that it would be most\n>> > > sensible to make it opt-in rather than opt-out. I wouldn't want to be\n>> > > surprised by remote refs vanishing after going to bed, but may be okay\n>> > > with it when I explicitly ask for it.\n>> > >\n>> > > At that point one has to raise the question whether it is still all that\n>> > > useful compared to running `git remote prune` manually every now and\n>> > > then. Mostly because explicitly configuring maintenance is probably\n>> > > something that only power users would do, and those power users would\n>> > > likely know to prune manually.\n>> >\n>> > I am 100% in agreement with your reasoning. If this thing is to\n>> > exist, it has to be opt-in. We also need to add ample warning why\n>> > doing this asynchronously behind user's back while the user could be\n>> > working in the same repository is prone to cause surprises in the\n>> > documentation in big red letters.\n>> >\n>> > I however am OK with the idea of having it as an optional \"task\"\n>> > that is disabled by default in \"git maintenance\". \"git maintenance\"\n>> > can be viewed as a platform-neutral way to set up scheduled tasks.\n>> >\n>> > A user may be a Git expert who knows what `git remote prune` does,\n>> > and understands and accepts the risk of confusion doing so without\n>> > explicit \"do it now\" from the end-user, but may not be a Linux or a\n>> > macOS or a Windows expert to know how to write crontab or its\n>> > equivalents on various schemes to define scheduled tasks.\n>> >\n>> > Thanks.\n>>\n>> Thanks for sharing your thoughts here. For some context — I currently\n>> look after git performance for a very large repository (60k+ refs)\n>> with a lot of active developers.\n>> I've observed that while power git users keep their repository tidy\n>> even without runnining maintenance, the majority of users infact don't\n>> know (or execute these), and\n>> run into performance issues. Easier fleet management was perhaps was\n>> one of the motivations behind adding this to `git-maintenance`.\n>>\n>> While its pretty rare that someone relies on a `refs/remote/*`\n>> reference (without creating a local copy of it in `refs/heads`), I can\n>> see why it can be surprising and an opt-in\n>> should be okay for me.\n>>\n>> Thanks,\n>> Shubham K\n\nIf you want to mix a patch with other discussion, write a scissors\nline (e.g. \"--- >8 --- cut here --- >8 ---\" and nothing else on its\nown line) here before the patch begins, and tell readers that they\nneed to scroll 50+ lines before they can see the patch at the\nbeginning of your message if the leading part needs to be so long.\n\n> From aed935055aff819cce591eecbf6697f8e8e7b99e Mon Sep 17 00:00:00 2001\n\nNever include this line in the body of the message.\n\n> From: Shubham Kanodia \n\nHaving \"From:\" in the body of the message at the beginning of your\npatch is a good way to override the author identity and set it\ndifferently from what your MUA places on the From: header line,\nwhen you are forwarding somebody else's patch and want to record the\nauthor identity of the real author, but then this line would not\nhave your name-email. Another use case is when your MUA cannot be\nconvinced to place your author identity on its From: header, but the\nmessage I am responding to has the same name-address in the From:\nheader.\n\nSo in conclusion, the above line is unnecessary in the body of this\nmessage.\n\n> Date: Tue, 17 Dec 2024 16:05:16 +0530\n\nRarely it is a good idea to include this line in the body of the\nmessage. The Date: header records the time the general public saw\nthe patch for the first time, which is the author date as far as the\ngeneral public is concered, much better.\n\n> Subject: [PATCH] maintenance: add prune-remote-refs task\n\nOverriding \"Subject:\" header of the e-mail with this in-body header\nis almost always needed when you are responding to an ongoing\ndiscussion and including a patch after a scissors line.\n\nSo in short, \"From/From:/Date:\" should be dropped, \"Subject:\" kept,\nand insert a scissors line before it.\n\n> Remote-tracking refs can accumulate in local repositories even as branches\n> are deleted on remotes, impacting git performance.\n\nOK. It usually is better to be explicit about the direction of the\n\"impact\". E.g., \"impacting git performance negatively\".\n\nHave you considered using the reftable backend, instead of the files\nbackend, by the way? It is rumored to scale much better with the\nnumber of refs.\n\nIt is possible that the user deliberately kept stale remote-tracking\nbranches even after they are removed from the origin because they\nused them as the bases of their topic branches, in which case it\nwould not be a good idea to blindly remove them, but I do not think\nthat needs to be said here. These users would not be running\n\"git remote prune\".\n\n> While git fetch --prune\n> helps clean these up, it requires a full fetch operation which is expensive\n> for large repositories.\n\nIt is curious why \"git fetch --prune\" is mentioned here, especially\nwhen you know \"git remote prune\" is much more appropriate operation\nfor that purpose. The real justification this paragraph needed to\nmake is why it is a good idea to a new task in \"git maintenance\",\nrather than letting the user run \"git remote prune\" themselves,\nisn't it?\n\n> Add a new maintenance task 'prune-remote-refs' that runs 'git remote prune'\n> for each configured remote. This provides an automated way to clean up stale\n> remote-tracking refs without requiring full fetches or manual intervention.\n\nAgain, \"requiring full fetches or\" is probably out of place.\n\n> This task is disabled by default.\n\nIf we wanted to mention \"fetch --prune\", it is probably not as an\ninferiour alternative to what you invented here, but as a different\nway the user can work that makes the problem you are trying to solve\ngo away. Perhaps near the beginning, after the \"Remote-tracking\nrefs can accumulate and impact performance negatively\", say\nsomething like:\n\n ... negatively. The user can have the \"fetch.prune\"\n configuration variable set to true when interacting with the\n other side, and stale remote-tracking refs will automatically\n pruned at zero cost.\n\nOf course \"remote..prune\" is a more targetted way that works\nper remote, but your prune-remote-refs works on all remotes,\nfetch.prune is a better contrast.\n\nIn any case, stepping back a bit, for the population of user who\nbenefit from enabing the prune-remote-refs task, wouldn't it be an\neven better solution for them to set fetch.prune? You can tell them\nto run \"git remote prune\" just once, set that configuration\nvariable, and then the remote-tracking branches will stay clean from\nthen on. Any future interactions with the remote make sure stale\nremote-tracking branches will be removed automatically. Wouldn't\nthat be a much better option? I am sure I must be missing a use\ncase where fetch.prune (or remote..prune) is not a good idea\nbut background prune-remote-refs task works better.\n\nSo, if we want to still add this new maintenance task, we should\nexplain *that* use case after the proposed addition of \"you can do\nwith fetch.prune at zero cost\" to the first paragraph. IOW, before\nstarting \"Add a new maintenance task ...\", we need to say something\nlike:\n\n There however are cases where pruning stale remote-tracking refs\n at the \"fetch\" time is undesirable, yet keeping them indefinitely\n is also unacceptable. <>. To help such use cases, add a new\n maintenance task ...\n\nAnd if there is no such workflow where a background prune-remote-refs\ntask is needed and fetch.prune would not work well, then we do not\nneed a new task, do we?\n\nLet me review the change (assuming that there are cases where\nfetch.prune is not a good solution) below.\n\n> +prune-remote-refs::\n> + The `prune-remote-refs` task runs `git remote prune` on each remote\n> + repository registered in the local repository. This task helps clean\n> + up deleted remote branches, improving the performance of operations\n> + that iterate through the refs. See linkgit:git-remote[1] for more\n> + information. This task is disabled by default and must be enabled\n> + manually.\n> ++\n> +NOTE: This task is opt-in to prevent unexpected removal of remote refs\n> +for users of git-maintenance. It is recommended for power users who\n> +are comfortable with configuring maintenance tasks and understand the\n> +implications of automatic pruning. For others, running `git remote prune`\n> +manually might be more appropriate.\n\n\"and must be enabled\" is unnecessary---that is what \"disabled by\ndefault\" means. Also, we probably would want to add a sentence or\ntwo to contrast this with setting fetch.prune to highlight why users\nwould want to consider enabling this (instead of setting fetch.prune).\n\nOtherwise, nicely written.\n\n> OPTIONS\n> -------\n> --auto::\n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index 4ae5196aed..9acf1d2989 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -20,6 +20,7 @@\n> #include \"lockfile.h\"\n> #include \"parse-options.h\"\n> #include \"run-command.h\"\n> +#include \"remote.h\"\n> #include \"sigchain.h\"\n> #include \"strvec.h\"\n> #include \"commit.h\"\n> @@ -913,6 +914,40 @@ static int maintenance_opt_schedule(const struct\n> option *opt, const char *arg,\n> return 0;\n> }\n>\n> +static int collect_remote(struct remote *remote, void *cb_data)\n> +{\n> + struct string_list *list = cb_data;\n> +\n> + if (!remote->url.nr)\n> + return 0;\n> +\n> + string_list_append(list, remote->name);\n> + return 0;\n> +}\n> +\n> +static int maintenance_task_prune_remote(struct maintenance_run_opts\n> *opts UNUSED,\n> + struct gc_config *cfg UNUSED)\n> +{\n> + struct string_list_item *item;\n> + struct string_list remotes_list = STRING_LIST_INIT_NODUP;\n> + struct child_process child = CHILD_PROCESS_INIT;\n> + int result = 0;\n> +\n> + for_each_remote(collect_remote, &remotes_list);\n> +\n> + for_each_string_list_item (item, &remotes_list) {\n> + const char *remote_name = item->string;\n\nIs the patch whitespace-damaged? The indentation seems to be gone.\nI won't comment on misindented lines and blocks in this review (i.e.\neven if the original had such problems, I wouldn't be seeing them in\nthe message I am responding to).\n\n> + child.git_cmd = 1;\n> + strvec_pushl(&child.args, \"remote\", \"prune\", remote_name, NULL);\n> +\n> + if (run_command(&child))\n> + result = error(_(\"failed to prune '%s'\"), remote_name);\n> + }\n> +\n> + string_list_clear(&remotes_list, 0);\n> + return result;\n> +}\n\nOK.\n\n> /* Remember to update object flag allocation in object.h */\n> #define SEEN (1u<<0)\n>\n> @@ -1375,6 +1410,7 @@ enum maintenance_task_label {\n> TASK_GC,\n> TASK_COMMIT_GRAPH,\n> TASK_PACK_REFS,\n> + TASK_PRUNE_REMOTE_REFS,\n>\n> /* Leave as final value */\n> TASK__COUNT\n> @@ -1411,6 +1447,10 @@ static struct maintenance_task tasks[] = {\n> maintenance_task_pack_refs,\n> pack_refs_condition,\n> },\n> + [TASK_PRUNE_REMOTE_REFS] = {\n> + \"prune-remote-refs\",\n> + maintenance_task_prune_remote,\n> + },\n> };\n>\n> static int compare_tasks_by_selection(const void *a_, const void *b_)\n> @@ -1505,6 +1545,8 @@ static void initialize_maintenance_strategy(void)\n> tasks[TASK_LOOSE_OBJECTS].schedule = SCHEDULE_DAILY;\n> tasks[TASK_PACK_REFS].enabled = 1;\n> tasks[TASK_PACK_REFS].schedule = SCHEDULE_WEEKLY;\n> + tasks[TASK_PRUNE_REMOTE_REFS].enabled = 0;\n> + tasks[TASK_PRUNE_REMOTE_REFS].schedule = SCHEDULE_DAILY;\n> }\n> }\n\nOK.\n\n> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> index 0ce4ba1cbe..60a0c3f835 100755\n> --- a/t/t7900-maintenance.sh\n> +++ b/t/t7900-maintenance.sh\n> @@ -446,6 +446,50 @@ test_expect_success 'pack-refs task' '\n> test_subcommand git pack-refs --all --prune '\n>\n> +test_expect_success 'prune-remote-refs task not enabled by default' '\n> + git clone . prune-test &&\n> + (\n> + cd prune-test &&\n> + GIT_TRACE2_EVENT=\"$(pwd)/prune.txt\" git maintenance run 2>err &&\n> + test_subcommand ! git remote prune origin + )\n> +'\n\nOK.\n\n> +test_expect_success 'prune-remote-refs task cleans stale remote refs' '\n> + test_commit initial &&\n> +\n> + # Create two separate remote repos\n> + git clone . remote1 &&\n> + git clone . remote2 &&\n> +\n> + git clone . prune-test-clean &&\n> + (\n> + cd prune-test-clean &&\n> + git config maintenance.prune-remote-refs.enabled true &&\n> +\n> + # Add both remotes\n> + git remote add remote1 \"../remote1\" &&\n> + git remote add remote2 \"../remote2\" &&\n> +\n> + # Create and push branches to both remotes\n> + git branch -f side2 HEAD &&\n> + git push remote1 side2 &&\n> + git push remote2 side2 &&\n> +\n> + # Rename branches in each remote to simulate a stale branch\n> + git -C ../remote1 branch -m side2 side3 &&\n> + git -C ../remote2 branch -m side2 side4 &&\n> +\n> + GIT_TRACE2_EVENT=\"$(pwd)/prune.txt\" git maintenance run\n> --task=prune-remote-refs &&\n> +\n> + # Verify pruning happened for both remotes\n> + test_subcommand git remote prune remote1 + test_subcommand git remote prune remote2 + test_must_fail git rev-parse refs/remotes/remote1/side2 &&\n> + test_must_fail git rev-parse refs/remotes/remote2/side2\n> + )\n> +'\n\nOK.\n\nOther than that it is not clear if we want it, the change looks\ncleanly done and was a pleasant read (other than whitespace damages\non the patch text).\n\nThanks.\n"},{"id":"509280","messageId":"CAG=Um+0qGEf+pX0cjCA2Qti4NYwFeCb29zgS7k2Lu_0yfuEz-w@mail.gmail.com","threadId":"62642","inReplyTo":"xmqq7c7y13tc.fsf@gitster.g","subject":"Re: Consider adding pruning of refs to git maintenance","fromName":"Shubham Kanodia","fromEmail":"shubham.kanodia10@gmail.com","sentAt":"2024-12-18T08:30:03Z","receivedAt":"2024-12-18T08:30:41Z","isPatch":false,"sender":{"key":"shubham.kanodia10@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8946207?v=4"},"body":"On Wed, Dec 18, 2024 at 1:26 AM Junio C Hamano wrote:\n>\n> Shubham Kanodia writes:\n>\n> > On Tue, Dec 17, 2024 at 4:51 PM Shubham Kanodia\n> > wrote:\n> >>\n> >> On Tue, Dec 17, 2024 at 1:11 PM Junio C Hamano wrote:\n> >> >\n> >> > Patrick Steinhardt writes:\n> >> >\n> >> > > If we want to have such a feature I'd thus claim that it would be most\n> >> > > sensible to make it opt-in rather than opt-out. I wouldn't want to be\n> >> > > surprised by remote refs vanishing after going to bed, but may be okay\n> >> > > with it when I explicitly ask for it.\n> >> > >\n> >> > > At that point one has to raise the question whether it is still all that\n> >> > > useful compared to running `git remote prune` manually every now and\n> >> > > then. Mostly because explicitly configuring maintenance is probably\n> >> > > something that only power users would do, and those power users would\n> >> > > likely know to prune manually.\n> >> >\n> >> > I am 100% in agreement with your reasoning. If this thing is to\n> >> > exist, it has to be opt-in. We also need to add ample warning why\n> >> > doing this asynchronously behind user's back while the user could be\n> >> > working in the same repository is prone to cause surprises in the\n> >> > documentation in big red letters.\n> >> >\n> >> > I however am OK with the idea of having it as an optional \"task\"\n> >> > that is disabled by default in \"git maintenance\". \"git maintenance\"\n> >> > can be viewed as a platform-neutral way to set up scheduled tasks.\n> >> >\n> >> > A user may be a Git expert who knows what `git remote prune` does,\n> >> > and understands and accepts the risk of confusion doing so without\n> >> > explicit \"do it now\" from the end-user, but may not be a Linux or a\n> >> > macOS or a Windows expert to know how to write crontab or its\n> >> > equivalents on various schemes to define scheduled tasks.\n> >> >\n> >> > Thanks.\n> >>\n> >> Thanks for sharing your thoughts here. For some context — I currently\n> >> look after git performance for a very large repository (60k+ refs)\n> >> with a lot of active developers.\n> >> I've observed that while power git users keep their repository tidy\n> >> even without runnining maintenance, the majority of users infact don't\n> >> know (or execute these), and\n> >> run into performance issues. Easier fleet management was perhaps was\n> >> one of the motivations behind adding this to `git-maintenance`.\n> >>\n> >> While its pretty rare that someone relies on a `refs/remote/*`\n> >> reference (without creating a local copy of it in `refs/heads`), I can\n> >> see why it can be surprising and an opt-in\n> >> should be okay for me.\n> >>\n> >> Thanks,\n> >> Shubham K\n>\n> If you want to mix a patch with other discussion, write a scissors\n> line (e.g. \"--- >8 --- cut here --- >8 ---\" and nothing else on its\n> own line) here before the patch begins, and tell readers that they\n> need to scroll 50+ lines before they can see the patch at the\n> beginning of your message if the leading part needs to be so long.\n>\n> > From aed935055aff819cce591eecbf6697f8e8e7b99e Mon Sep 17 00:00:00 2001\n>\n> Never include this line in the body of the message.\n>\n> > From: Shubham Kanodia \n>\n> Having \"From:\" in the body of the message at the beginning of your\n> patch is a good way to override the author identity and set it\n> differently from what your MUA places on the From: header line,\n> when you are forwarding somebody else's patch and want to record the\n> author identity of the real author, but then this line would not\n> have your name-email. Another use case is when your MUA cannot be\n> convinced to place your author identity on its From: header, but the\n> message I am responding to has the same name-address in the From:\n> header.\n>\n> So in conclusion, the above line is unnecessary in the body of this\n> message.\n>\n> > Date: Tue, 17 Dec 2024 16:05:16 +0530\n>\n> Rarely it is a good idea to include this line in the body of the\n> message. The Date: header records the time the general public saw\n> the patch for the first time, which is the author date as far as the\n> general public is concered, much better.\n>\n> > Subject: [PATCH] maintenance: add prune-remote-refs task\n>\n> Overriding \"Subject:\" header of the e-mail with this in-body header\n> is almost always needed when you are responding to an ongoing\n> discussion and including a patch after a scissors line.\n>\n> So in short, \"From/From:/Date:\" should be dropped, \"Subject:\" kept,\n> and insert a scissors line before it.\n>\n> > Remote-tracking refs can accumulate in local repositories even as branches\n> > are deleted on remotes, impacting git performance.\n>\n> OK. It usually is better to be explicit about the direction of the\n> \"impact\". E.g., \"impacting git performance negatively\".\n>\n> Have you considered using the reftable backend, instead of the files\n> backend, by the way? It is rumored to scale much better with the\n> number of refs.\n>\n> It is possible that the user deliberately kept stale remote-tracking\n> branches even after they are removed from the origin because they\n> used them as the bases of their topic branches, in which case it\n> would not be a good idea to blindly remove them, but I do not think\n> that needs to be said here. These users would not be running\n> \"git remote prune\".\n>\n> > While git fetch --prune\n> > helps clean these up, it requires a full fetch operation which is expensive\n> > for large repositories.\n>\n> It is curious why \"git fetch --prune\" is mentioned here, especially\n> when you know \"git remote prune\" is much more appropriate operation\n> for that purpose. The real justification this paragraph needed to\n> make is why it is a good idea to a new task in \"git maintenance\",\n> rather than letting the user run \"git remote prune\" themselves,\n> isn't it?\n>\n> > Add a new maintenance task 'prune-remote-refs' that runs 'git remote prune'\n> > for each configured remote. This provides an automated way to clean up stale\n> > remote-tracking refs without requiring full fetches or manual intervention.\n>\n> Again, \"requiring full fetches or\" is probably out of place.\n>\n> > This task is disabled by default.\n>\n> If we wanted to mention \"fetch --prune\", it is probably not as an\n> inferiour alternative to what you invented here, but as a different\n> way the user can work that makes the problem you are trying to solve\n> go away. Perhaps near the beginning, after the \"Remote-tracking\n> refs can accumulate and impact performance negatively\", say\n> something like:\n>\n> ... negatively. The user can have the \"fetch.prune\"\n> configuration variable set to true when interacting with the\n> other side, and stale remote-tracking refs will automatically\n> pruned at zero cost.\n>\n> Of course \"remote..prune\" is a more targetted way that works\n> per remote, but your prune-remote-refs works on all remotes,\n> fetch.prune is a better contrast.\n>\n> In any case, stepping back a bit, for the population of user who\n> benefit from enabing the prune-remote-refs task, wouldn't it be an\n> even better solution for them to set fetch.prune? You can tell them\n> to run \"git remote prune\" just once, set that configuration\n> variable, and then the remote-tracking branches will stay clean from\n> then on. Any future interactions with the remote make sure stale\n> remote-tracking branches will be removed automatically. Wouldn't\n> that be a much better option? I am sure I must be missing a use\n> case where fetch.prune (or remote..prune) is not a good idea\n> but background prune-remote-refs task works better.\n>\n> So, if we want to still add this new maintenance task, we should\n> explain *that* use case after the proposed addition of \"you can do\n> with fetch.prune at zero cost\" to the first paragraph. IOW, before\n> starting \"Add a new maintenance task ...\", we need to say something\n> like:\n>\n> There however are cases where pruning stale remote-tracking refs\n> at the \"fetch\" time is undesirable, yet keeping them indefinitely\n> is also unacceptable. < SUCH CHARACTERISTICS>>. To help such use cases, add a new\n> maintenance task ...\n>\n> And if there is no such workflow where a background prune-remote-refs\n> task is needed and fetch.prune would not work well, then we do not\n> need a new task, do we?\n>\n> Let me review the change (assuming that there are cases where\n> fetch.prune is not a good solution) below.\n>\n> > +prune-remote-refs::\n> > + The `prune-remote-refs` task runs `git remote prune` on each remote\n> > + repository registered in the local repository. This task helps clean\n> > + up deleted remote branches, improving the performance of operations\n> > + that iterate through the refs. See linkgit:git-remote[1] for more\n> > + information. This task is disabled by default and must be enabled\n> > + manually.\n> > ++\n> > +NOTE: This task is opt-in to prevent unexpected removal of remote refs\n> > +for users of git-maintenance. It is recommended for power users who\n> > +are comfortable with configuring maintenance tasks and understand the\n> > +implications of automatic pruning. For others, running `git remote prune`\n> > +manually might be more appropriate.\n>\n> \"and must be enabled\" is unnecessary---that is what \"disabled by\n> default\" means. Also, we probably would want to add a sentence or\n> two to contrast this with setting fetch.prune to highlight why users\n> would want to consider enabling this (instead of setting fetch.prune).\n>\n> Otherwise, nicely written.\n>\n> > OPTIONS\n> > -------\n> > --auto::\n> > diff --git a/builtin/gc.c b/builtin/gc.c\n> > index 4ae5196aed..9acf1d2989 100644\n> > --- a/builtin/gc.c\n> > +++ b/builtin/gc.c\n> > @@ -20,6 +20,7 @@\n> > #include \"lockfile.h\"\n> > #include \"parse-options.h\"\n> > #include \"run-command.h\"\n> > +#include \"remote.h\"\n> > #include \"sigchain.h\"\n> > #include \"strvec.h\"\n> > #include \"commit.h\"\n> > @@ -913,6 +914,40 @@ static int maintenance_opt_schedule(const struct\n> > option *opt, const char *arg,\n> > return 0;\n> > }\n> >\n> > +static int collect_remote(struct remote *remote, void *cb_data)\n> > +{\n> > + struct string_list *list = cb_data;\n> > +\n> > + if (!remote->url.nr)\n> > + return 0;\n> > +\n> > + string_list_append(list, remote->name);\n> > + return 0;\n> > +}\n> > +\n> > +static int maintenance_task_prune_remote(struct maintenance_run_opts\n> > *opts UNUSED,\n> > + struct gc_config *cfg UNUSED)\n> > +{\n> > + struct string_list_item *item;\n> > + struct string_list remotes_list = STRING_LIST_INIT_NODUP;\n> > + struct child_process child = CHILD_PROCESS_INIT;\n> > + int result = 0;\n> > +\n> > + for_each_remote(collect_remote, &remotes_list);\n> > +\n> > + for_each_string_list_item (item, &remotes_list) {\n> > + const char *remote_name = item->string;\n>\n> Is the patch whitespace-damaged? The indentation seems to be gone.\n> I won't comment on misindented lines and blocks in this review (i.e.\n> even if the original had such problems, I wouldn't be seeing them in\n> the message I am responding to).\n>\n> > + child.git_cmd = 1;\n> > + strvec_pushl(&child.args, \"remote\", \"prune\", remote_name, NULL);\n> > +\n> > + if (run_command(&child))\n> > + result = error(_(\"failed to prune '%s'\"), remote_name);\n> > + }\n> > +\n> > + string_list_clear(&remotes_list, 0);\n> > + return result;\n> > +}\n>\n> OK.\n>\n> > /* Remember to update object flag allocation in object.h */\n> > #define SEEN (1u<<0)\n> >\n> > @@ -1375,6 +1410,7 @@ enum maintenance_task_label {\n> > TASK_GC,\n> > TASK_COMMIT_GRAPH,\n> > TASK_PACK_REFS,\n> > + TASK_PRUNE_REMOTE_REFS,\n> >\n> > /* Leave as final value */\n> > TASK__COUNT\n> > @@ -1411,6 +1447,10 @@ static struct maintenance_task tasks[] = {\n> > maintenance_task_pack_refs,\n> > pack_refs_condition,\n> > },\n> > + [TASK_PRUNE_REMOTE_REFS] = {\n> > + \"prune-remote-refs\",\n> > + maintenance_task_prune_remote,\n> > + },\n> > };\n> >\n> > static int compare_tasks_by_selection(const void *a_, const void *b_)\n> > @@ -1505,6 +1545,8 @@ static void initialize_maintenance_strategy(void)\n> > tasks[TASK_LOOSE_OBJECTS].schedule = SCHEDULE_DAILY;\n> > tasks[TASK_PACK_REFS].enabled = 1;\n> > tasks[TASK_PACK_REFS].schedule = SCHEDULE_WEEKLY;\n> > + tasks[TASK_PRUNE_REMOTE_REFS].enabled = 0;\n> > + tasks[TASK_PRUNE_REMOTE_REFS].schedule = SCHEDULE_DAILY;\n> > }\n> > }\n>\n> OK.\n>\n> > diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> > index 0ce4ba1cbe..60a0c3f835 100755\n> > --- a/t/t7900-maintenance.sh\n> > +++ b/t/t7900-maintenance.sh\n> > @@ -446,6 +446,50 @@ test_expect_success 'pack-refs task' '\n> > test_subcommand git pack-refs --all --prune > '\n> >\n> > +test_expect_success 'prune-remote-refs task not enabled by default' '\n> > + git clone . prune-test &&\n> > + (\n> > + cd prune-test &&\n> > + GIT_TRACE2_EVENT=\"$(pwd)/prune.txt\" git maintenance run 2>err &&\n> > + test_subcommand ! git remote prune origin > + )\n> > +'\n>\n> OK.\n>\n> > +test_expect_success 'prune-remote-refs task cleans stale remote refs' '\n> > + test_commit initial &&\n> > +\n> > + # Create two separate remote repos\n> > + git clone . remote1 &&\n> > + git clone . remote2 &&\n> > +\n> > + git clone . prune-test-clean &&\n> > + (\n> > + cd prune-test-clean &&\n> > + git config maintenance.prune-remote-refs.enabled true &&\n> > +\n> > + # Add both remotes\n> > + git remote add remote1 \"../remote1\" &&\n> > + git remote add remote2 \"../remote2\" &&\n> > +\n> > + # Create and push branches to both remotes\n> > + git branch -f side2 HEAD &&\n> > + git push remote1 side2 &&\n> > + git push remote2 side2 &&\n> > +\n> > + # Rename branches in each remote to simulate a stale branch\n> > + git -C ../remote1 branch -m side2 side3 &&\n> > + git -C ../remote2 branch -m side2 side4 &&\n> > +\n> > + GIT_TRACE2_EVENT=\"$(pwd)/prune.txt\" git maintenance run\n> > --task=prune-remote-refs &&\n> > +\n> > + # Verify pruning happened for both remotes\n> > + test_subcommand git remote prune remote1 > + test_subcommand git remote prune remote2 > + test_must_fail git rev-parse refs/remotes/remote1/side2 &&\n> > + test_must_fail git rev-parse refs/remotes/remote2/side2\n> > + )\n> > +'\n>\n> OK.\n>\n> Other than that it is not clear if we want it, the change looks\n> cleanly done and was a pleasant read (other than whitespace damages\n> on the patch text).\n>\n> Thanks.\n\nLet me address a few points here before I resend the patch in a more\nideally formatted manner.\n\n> Have you considered using the reftable backend, instead of the files\n> backend, by the way? It is rumored to scale much better with the\n> number of refs.\n\nI don't think we've done that yet, but thanks for the pointer. In any\ncase, till reftable is stable (including support for reflogs etc),\nwe'd want to avoid switching all clients to use that,\n\n> In any case, stepping back a bit, for the population of user who\n> benefit from enabing the prune-remote-refs task, wouldn't it be an\n> even better solution for them to set fetch.prune? You can tell them\n> to run \"git remote prune\" just once, set that configuration\n> variable, and then the remote-tracking branches will stay clean from\n> then on. Any future interactions with the remote make sure stale\n> remote-tracking branches will be removed automatically. Wouldn't\n> that be a much better option? I am sure I must be missing a use\n> case where fetch.prune (or remote..prune) is not a good idea\n> but background prune-remote-refs task works better.\n\nLet me expand on the context for suggesting this change:\n\nI work with a large repository that has over 50k refs, with about 4k\nnew ones added weekly.\nWe have maintenance scripts on our git server that clean up stale refs\n(unused older than N months).\n\nUsing `fetch.prune` with a normal git fetch isn't ideal because it\nwould cause git fetch to unnecessarily download many new refs that\nusers don't need. So we actively discourage that.\n\nIn theory, users could just run `git remote prune` once and carefully\navoid full fetches to keep their local ref count low.\nHowever, in practice, we've found that full fetches happen through\nvarious indirect means:\n\n- Shell plugins like zsh/pure\n- Git GUIs like Sourcetree\n- Code editors like VSCode\n\namong others.\n\nThese tools sometimes perform background fetches without users\nrealizing it, causing ref counts to gradually increase until users\nremember to run `git remote prune` again.\n\nTypically, most users care about:\n\n- Their local branches (refs/heads) - usually a few hundred in number\n- A small subset of remote branches they might work with from time to\ntime (>99% of active remote refs are irrelevant to any individual\ndeveloper in a large repo)\n\nThe goal here is to help users who can already run `git remote prune`\nto do so periodically.\n- If full `git fetch` is completely avoided, this will gradually\nreduce the local ref count from tens of thousands to just a few\nhundred active refs (even if the remote has 50k+ active refs) as old\nbranches on the remote expire with time.\n- Even if not —say, if an errant tool or the developer executes `git\nfetch` mistakenly, then the maintenance job ensures this doesn't\nbecome their permanent state until the next manual remote prune.\n\nUltimately, this is about providing a better default for large-scale\nrepositories (& users who choose to keep refs pruned).\n"},{"id":"509301","messageId":"xmqq34iluhqy.fsf@gitster.g","threadId":"62642","inReplyTo":"CAG=Um+0qGEf+pX0cjCA2Qti4NYwFeCb29zgS7k2Lu_0yfuEz-w@mail.gmail.com","subject":"Re: Consider adding pruning of refs to git maintenance","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-18T15:35:01Z","receivedAt":"2024-12-18T15:35:04Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Shubham Kanodia writes:\n\n>> ...\n>> Thanks.\n\n[administrivia: respond inline, trim out parts that do not have to\nbe read by bystanders to understand your response].\n\n>> In any case, stepping back a bit, for the population of user who\n>> benefit from enabing the prune-remote-refs task, wouldn't it be an\n>> even better solution for them to set fetch.prune? You can tell them\n>> to run \"git remote prune\" just once, set that configuration\n>> variable, and then the remote-tracking branches will stay clean from\n>> then on. Any future interactions with the remote make sure stale\n>> remote-tracking branches will be removed automatically. Wouldn't\n>> that be a much better option? I am sure I must be missing a use\n>> case where fetch.prune (or remote..prune) is not a good idea\n>> but background prune-remote-refs task works better.\n>\n> Let me expand on the context for suggesting this change:\n>\n> I work with a large repository that has over 50k refs, with about 4k\n> new ones added weekly.\n> We have maintenance scripts on our git server that clean up stale refs\n> (unused older than N months).\n\n> Using `fetch.prune` with a normal git fetch isn't ideal because it\n> would cause git fetch to unnecessarily download many new refs that\n> users don't need. So we actively discourage that.\n\nThis is what I did not quite understand. What do your users\nnormally do to update their repository from the remote to become in\nsync, if they are not running \"git fetch\"?\n\n\tSide note: it is very likely that your users are not\n\tdirectly be running \"git fetch\", but using various\n\tfront-ends like \"git pull\", \"git pull --rebase\", or even\n\t\"repo\", but they all at some point call \"git fetch\" to get\n\tthe new objects and update refs.\n\nAh, are they using \"git fetch origin +foo:refs/remotes/origin/foo\",\ni.e., only selectively fetch the thing that they use and nothing\nelse (again, their wrappers may supply the refspec to do the\nlimiting)? Now it slowly starts to make sense to me (sorry, I am\nslow, especially without caffeine in the morning).\n\nAm I following / guessing your set-up more or less correctly so far?\n\nIn any case, if your users are doing selective fetching, 50k refs or\n4k ref turnover per week on the other side does not really matter.\nYour users' desktop repositories won't see remote-tracking refs that\nthey didn't use and ask for.\n\nBut you are right that these selectively fetched refs will\naccumulate unless pruned, and fetch.prune would not prune anything\nwhen\n\n\tgit fetch origin +foo:refs/remotes/origin/foo\n\nbecause it will not prune what is outside the hierarchy the refspec\ncovers and this is a deliberate design decision.\n\nFor \"git fetch origin '+refs/heads/*:refs/remotes/origin/*'\", which\nis pretty much how \"git clone\" sets up the remotes, anything we have\nin refs/remotes/origin/ hierarchy that do not appear in the current\nrefs/heads/ hiearchy they have are pruned with fetch.prune=true.\nBut if you fetch selectively, either 'foo' exists (in which case it\nwon't be pruned), or 'foo' went away (in which case the fetch itself\nfails before even pruning what is on our end), so fetch.prune may\nnot help.\n\nAnd at least for a shorter term, periodically running \"remote prune\"\nwould be an acceptable workaround for such a workflow.\n\nIn the longer term, I suspect that we may want a new option that\nlets you more aggressively prune your remote-tracking refs, telling\nthe tool something like\n\n git fetch --prune-aggressive origin +refs/heads/foo:refs/remotes/origin/foo\n\nto mean \"I only am interested in getting the object to complete\ntheir current 'foo' branch, and get my remote-tracking ref for that\nbranch updated, BUT if you notice some ref in my refs/remotes/origin/*\nthat they do not have in their refs/heads/*, please prune it, even when\nthey are not 'foo' (which means normal --prune would not prune them)\",\nwould not be a terrible idea.\n\nIt would be more involved than running \"remote prune\" periodically,\nof course.\n\n> In theory, users could just run `git remote prune` once and carefully\n> avoid full fetches to keep their local ref count low.\n> However, in practice, we've found that full fetches happen through\n> various indirect means:\n>\n> - Shell plugins like zsh/pure\n> - Git GUIs like Sourcetree\n> - Code editors like VSCode\n>\n> among others.\n\nAnd do any of these bypass underlying \"git fetch\"? If not, then one\neasier solution is to accept that somebody will do the regular\n\n\trefs/heads/*:refs/remotes/origin/*\n\nfull-fetch *anyway*. Once we accept that to happen, we can tell\n\"git fetch\" to always prune. Then when these \"various indirect\nmeans\" attempt their full fetch, \"git fetch\" invoked by them would\nstill honor fetch.prune, so even though they would try to maintain\nthese 50k refs in-sync with the remote, which means you may see 4k\nnew refs per week, but plausibly the refs that got retired are seen\nas stale and get removed on the users' repositories.\n\n> - If full `git fetch` is completely avoided, this will gradually\n> reduce the local ref count from tens of thousands to just a few\n> hundred active refs (even if the remote has 50k+ active refs) as old\n> branches on the remote expire with time.\n\nYes. You'd somehow need to arrange these third-party tools not to\nfetch too much unneeded cruft.\n\n> - Even if not —say, if an errant tool or the developer executes `git\n> fetch` mistakenly, then the maintenance job ensures this doesn't\n> become their permanent state until the next manual remote prune.\n\nFor the latter case, fetch.prune=true would be the ideal solution, I\nwould imagine. The \"errant tool\"'s 'git fetch' would prune the\nstale ones.\n\nNow, the documentation should explain when this \"periodically running\nremote prune\" is an acceptable workaround and/or a useful solution,\nrelative to setting fetch.prune, as most parts of the existing\ndocumentation do assume that the users, intended audience of the\ndocument, are using the bog-standard \"git clone\" result, that copies\nall their branches to remote-tracking branches.\n\nThanks.\n"},{"id":"509498","messageId":"CAG=Um+2cdMcys2x492_i47-qdvx5aJ7k6r=xSH5a3JZtMZBEZw@mail.gmail.com","threadId":"62642","inReplyTo":"xmqq34iluhqy.fsf@gitster.g","subject":"Re: Consider adding pruning of refs to git maintenance","fromName":"Shubham Kanodia","fromEmail":"shubham.kanodia10@gmail.com","sentAt":"2024-12-22T12:19:35Z","receivedAt":"2024-12-22T12:20:12Z","isPatch":false,"sender":{"key":"shubham.kanodia10@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8946207?v=4"},"body":"On Wed, Dec 18, 2024 at 9:05 PM Junio C Hamano wrote:\n...\n\n> Ah, are they using \"git fetch origin +foo:refs/remotes/origin/foo\",\n> i.e., only selectively fetch the thing that they use and nothing\n> else (again, their wrappers may supply the refspec to do the\n> limiting)? Now it slowly starts to make sense to me (sorry, I am\n> slow, especially without caffeine in the morning).\n>\n> Am I following / guessing your set-up more or less correctly so far?\n\nYes. The root cause of the issue here is that the behaviour that `git\nfetch` / `git pull` fetches all refs by default is undesirable in\nlarge git repositories.\nIt's almost never what you want to do. We advice users to execute `git\nfetch ` when they want to run fetch (/pull) explicitly.\n\nIdeally, `git fetch` would only fetch the current branch when an\nexplicit branch is not specified, and `git fetch --all` would pull in\nall.\nNow I believe git does provide a way to configure the default fetch\nrefs via `remote..fetch`. So in theory I could just set —\n\n```\n[remote \"origin\"]\n fetch = +refs/heads/master:refs/remotes/origin/master\n```\n\nwhich would avoid someone pulling in all refs accidentally.\nHowever that has a side effect that now if you do want to fetch &\nstart working on a remote ref that you weren't previously tracking, a\ncommand like\n\n```\ngit fetch origin new-ref-branch-from-remote\n```\n\nno longer allows you to just start working on this new branch by doing\na `git checkout new-ref-branch-from-remote`.\n\nIf you wanted to be able to do that, you'd probably need to do —\n\n```\n git fetch origin new-ref-branch-from-remote\n--refmap=+refs/heads/new-ref-branch-from-remote:refs/remotes/origin/new-ref-branch-from-remote\n```\n\nwhich is pretty awkward to type everytime.\n\nNow to come back from this little digression, for now —\n- We ask users to set both `fetch.prune` and `fetch.pruneTags` to true\n(so that if a third party tool does do something, the damage is\nlimited and they don't have an ever-growing list of refs)\n- Setup this job that cleans stale remote refs on a periodic basis,\nwhich means that their ref counts heal over time (if they configure\nall third party tools right)\n\n\n> Now, the documentation should explain when this \"periodically running\n> remote prune\" is an acceptable workaround and/or a useful solution,\n> relative to setting fetch.prune, as most parts of the existing\n> documentation do assume that the users, intended audience of the\n> document, are using the bog-standard \"git clone\" result, that copies\n> all their branches to remote-tracking branches.\n\nAgreed, will update docs to include that.\n\nThanks,\nShubham K\n"},{"id":"509510","messageId":"xmqqwmfr112w.fsf@gitster.g","threadId":"62642","inReplyTo":"CAG=Um+2cdMcys2x492_i47-qdvx5aJ7k6r=xSH5a3JZtMZBEZw@mail.gmail.com","subject":"Re: Consider adding pruning of refs to git maintenance","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-23T04:21:27Z","receivedAt":"2024-12-23T04:21:30Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Shubham Kanodia writes:\n\n> If you wanted to be able to do that, you'd probably need to do —\n>\n> ```\n> git fetch origin new-ref-branch-from-remote\n> --refmap=+refs/heads/new-ref-branch-from-remote:refs/remotes/origin/new-ref-branch-from-remote\n> ```\n>\n> which is pretty awkward to type everytime.\n\nOr have this in your .git/config\n\n [remote \"origin\"]\n\tfetch = +refs/heads/new-ref-branch-from-remote:refs/remotes/origin/new-ref-branch-from-remote\n\nand run\n\n $ git fetch origin new-ref-branch-from-remote\n\nIf you plan to do the same for more random branches, you could\ninstead do\n\n [remote \"origin\"]\n\tfetch = +refs/heads/*:refs/remotes/origin/*\n\nas long as you promise that you never run \"git fetch\" without saying\nwhich exact refs to fetch. I wonder if it helps to introduce a new\nconfiguration remote.*.fetchmap\n\n [remote \"origin\"]\n\tfetchmap = +refs/heads/*:refs/remotes/origin/*\n\nthat only talks about how the mapping goes without saying what gets\nfetched by default, so that the lack of remote.origin.fetch would\ncause\n\n $ git fetch origin\n $ git fetch\n\nnot to grab everything (like when you had the same specification as\nthe value of remote.origin.fetch instead), but allows you to map\nwhatever you grab from there when you did\n\n $ git fetch origin foo bar\n\nIf the glob pattern given by the fetchmap participated in how\nfetch.prune behaves (i.e. I see refs/remotes/origin/baz here, but\nwhile trying to fetch refs/heads/{foo,bar}, I notice that they no\nlonger advertise refs/heads/baz---let me prune that stale one),\nthat would make it much simpler.\n\n> Now to come back from this little digression, for now —\n> - We ask users to set both `fetch.prune` and `fetch.pruneTags` to true\n> (so that if a third party tool does do something, the damage is\n> limited and they don't have an ever-growing list of refs)\n\nGood.\n\n> - Setup this job that cleans stale remote refs on a periodic basis,\n> which means that their ref counts heal over time (if they configure\n> all third party tools right)\n\nOK. And a scheduled task would make this part better.\n"},{"id":"509511","messageId":"CAG=Um+1wDY=3cxfXZxk_-vQcabogHgxGHaJ0utg+0NC2bmiXSA@mail.gmail.com","threadId":"62642","inReplyTo":"xmqqwmfr112w.fsf@gitster.g","subject":"Re: Consider adding pruning of refs to git maintenance","fromName":"Shubham Kanodia","fromEmail":"shubham.kanodia10@gmail.com","sentAt":"2024-12-23T09:30:29Z","receivedAt":"2024-12-23T09:31:06Z","isPatch":false,"sender":{"key":"shubham.kanodia10@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8946207?v=4"},"body":"On Mon, Dec 23, 2024 at 9:51 AM Junio C Hamano wrote:\n\n> as long as you promise that you never run \"git fetch\" without saying\n> which exact refs to fetch. I wonder if it helps to introduce a new\n> configuration remote.*.fetchmap\n>\n> [remote \"origin\"]\n> fetchmap = +refs/heads/*:refs/remotes/origin/*\n>\n> that only talks about how the mapping goes without saying what gets\n> fetched by default, so that the lack of remote.origin.fetch would\n> cause\n>\n> $ git fetch origin\n> $ git fetch\n>\n> not to grab everything (like when you had the same specification as\n> the value of remote.origin.fetch instead), but allows you to map\n> whatever you grab from there when you did\n\nThis would be perfect really. Adding a new refmap every time you need\nto pull in changes from someone else's branch is too cumbersome.\nI'd be happy to attempt this as a follow up patch where\n`remote..fetchmap` if you think that's possible to add.\n\nAlso to keep things cleaner — I'm going to re-submit this patch using\ngitgitgadget separately.\n\nThanks!\nShubham K\n"}]}