{"thread":{"id":"63724","subject":"[PATCH v3 0/2] fetch --prune performance problem","startedAt":"2025-07-02T01:08:10Z","lastAt":"2025-07-02T01:47:56Z","messageCount":5,"participants":["Phil Hord","Junio C Hamano"],"isPatch":true,"patchVersion":3,"patchTotal":2},"messages":[{"id":"521130","messageId":"20250702005837.2813893-2-phil.hord@gmail.com","threadId":"63724","inReplyTo":null,"subject":"[PATCH v3 0/2] fetch --prune performance problem","fromName":"Phil Hord","fromEmail":"phil.hord@gmail.com","sentAt":"2025-07-02T00:58:36Z","receivedAt":"2025-07-02T01:08:10Z","isPatch":true,"sender":{"key":"phil.hord@gmail.com","avatar":"https://avatars.githubusercontent.com/u/123908?v=4"},"body":"From: Phil Hord \n\n`git fetch --prune` runs in O(N^2) time normally. This happens because the code\niterates over each ref to be pruned to display its status. In a repo with\n174,000 refs, where I was pruning 15,000 refs, the current code made 2.6 billion\ncalls to strcmp and consumed 470 seconds of CPU. After this change, the same\noperation completes in under 1 second.\n\nThe loop looks like this:\n\n for p in prune_refs { for ref in all_refs { if p == ref { ... }}}\n\nThat loop runs only to check for and report newly dangling refs. A workaround to\navoid this slowness is to run with `-q` to bypass this check.\n\nThere is similar check/report functionality in `git remote prune`, but it uses a\nmore efficient method to check for dangling refs. prune_refs is first sorted, so\nit can be searched in O(logN), so this loop is O(N*logN).\n\n for ref in all_refs { if ref in prune_refs { ... }}\n\nWe can use that function instead, with some minor cleanup to the output to deal\nwith the ordering being changed.\n\nThis patch version only adds the deleted branch name to the output of the dangling\nsym refs since the ordering has changed. This is only a minor cleanup and was\nnot actually needed since, for example, `git origin prune` already did not\nmind losing track of this information in its output. But now it is improved\nto be more explicit.\n\nThis version (V3) has three changes from V2:\n - Removes a header declaration I forgot to move previously\n - Cleans up the refs_warn_dangling_symrefs API to be more sane\n - Drops the ref shortening that seems ill-advised in retrospect\n\nPhil Hord (2):\n refs: remove old refs_warn_dangling_symref\n clean up interface for refs_warn_dangling_symrefs\n\n builtin/fetch.c | 5 +----\n builtin/remote.c | 5 +----\n refs.c | 34 ++++++++++++----------------------\n refs.h | 5 ++---\n 4 files changed, 16 insertions(+), 33 deletions(-)\n\n-- \n2.50.0.149.g2f19833911.dirty\n\n"},{"id":"521131","messageId":"20250702005837.2813893-3-phil.hord@gmail.com","threadId":"63724","inReplyTo":"20250702005837.2813893-2-phil.hord@gmail.com","subject":"[PATCH v3 1/2] refs: remove old refs_warn_dangling_symref","fromName":"Phil Hord","fromEmail":"phil.hord@gmail.com","sentAt":"2025-07-02T00:58:37Z","receivedAt":"2025-07-02T01:08:13Z","isPatch":true,"sender":{"key":"phil.hord@gmail.com","avatar":"https://avatars.githubusercontent.com/u/123908?v=4"},"body":"From: Phil Hord \n\nThe dangling warning function that takes a single ref to search for\nis no longer used. Remove it.\n\nSigned-off-by: Phil Hord \n---\n refs.c | 17 +----------------\n refs.h | 2 --\n 2 files changed, 1 insertion(+), 18 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 651fb2d41299..07197c239e33 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -438,7 +438,6 @@ static int for_each_filter_refs(const char *refname, const char *referent,\n struct warn_if_dangling_data {\n \tstruct ref_store *refs;\n \tFILE *fp;\n-\tconst char *refname;\n \tconst struct string_list *refnames;\n \tconst char *msg_fmt;\n };\n@@ -455,9 +454,7 @@ static int warn_if_dangling_symref(const char *refname, const char *referent UNU\n \n \tresolves_to = refs_resolve_ref_unsafe(d->refs, refname, 0, NULL, NULL);\n \tif (!resolves_to\n-\t || (d->refname\n-\t\t? strcmp(resolves_to, d->refname)\n-\t\t: !string_list_has_string(d->refnames, resolves_to))) {\n+\t || !string_list_has_string(d->refnames, resolves_to)) {\n \t\treturn 0;\n \t}\n \n@@ -466,18 +463,6 @@ static int warn_if_dangling_symref(const char *refname, const char *referent UNU\n \treturn 0;\n }\n \n-void refs_warn_dangling_symref(struct ref_store *refs, FILE *fp,\n-\t\t\t const char *msg_fmt, const char *refname)\n-{\n-\tstruct warn_if_dangling_data data = {\n-\t\t.refs = refs,\n-\t\t.fp = fp,\n-\t\t.refname = refname,\n-\t\t.msg_fmt = msg_fmt,\n-\t};\n-\trefs_for_each_rawref(refs, warn_if_dangling_symref, &data);\n-}\n-\n void refs_warn_dangling_symrefs(struct ref_store *refs, FILE *fp,\n \t\t\t\tconst char *msg_fmt, const struct string_list *refnames)\n {\ndiff --git a/refs.h b/refs.h\nindex 46a6008e07f2..07f21824d480 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -452,8 +452,6 @@ static inline const char *has_glob_specials(const char *pattern)\n \treturn strpbrk(pattern, \"?*[\");\n }\n \n-void refs_warn_dangling_symref(struct ref_store *refs, FILE *fp,\n-\t\t\t const char *msg_fmt, const char *refname);\n void refs_warn_dangling_symrefs(struct ref_store *refs, FILE *fp,\n \t\t\t\tconst char *msg_fmt, const struct string_list *refnames);\n \n-- \n2.50.0.149.g2f19833911.dirty\n\n"},{"id":"521132","messageId":"20250702005837.2813893-4-phil.hord@gmail.com","threadId":"63724","inReplyTo":"20250702005837.2813893-2-phil.hord@gmail.com","subject":"[PATCH v3 2/2] clean up interface for refs_warn_dangling_symrefs","fromName":"Phil Hord","fromEmail":"phil.hord@gmail.com","sentAt":"2025-07-02T00:58:38Z","receivedAt":"2025-07-02T01:08:15Z","isPatch":true,"sender":{"key":"phil.hord@gmail.com","avatar":"https://avatars.githubusercontent.com/u/123908?v=4"},"body":"From: Phil Hord \n\nThe refs_warn_dangling_symrefs interface is a bit fragile as it passes\nin printf-formatting strings with expectations about the number of\narguments. This patch series made it worse by adding a 2nd positional\nargument. But there are only two call sites, and they both use almost\nidentical display options.\n\nMake this safer by moving the format strings into the function that uses\nthem to make it easier to see when the arguments don't match. Pass a\nprefix string and a dry_run flag so the decision logic can be handled\nwhere needed.\n\nSigned-off-by: Phil Hord \n---\n builtin/fetch.c | 5 +----\n builtin/remote.c | 5 +----\n refs.c | 17 +++++++++++------\n refs.h | 3 ++-\n 4 files changed, 15 insertions(+), 15 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 04d10c9e781a..fc72f2119c56 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1384,9 +1384,6 @@ static int prune_refs(struct display_state *display_state,\n \tstruct ref *ref, *stale_refs = get_stale_heads(rs, ref_map);\n \tstruct strbuf err = STRBUF_INIT;\n \tstruct string_list refnames = STRING_LIST_INIT_NODUP;\n-\tconst char *dangling_msg = dry_run\n-\t\t? _(\" %s will become dangling after %s is deleted\")\n-\t\t: _(\" %s has become dangling after %s was deleted\");\n \n \tfor (ref = stale_refs; ref; ref = ref->next)\n \t\tstring_list_append(&refnames, ref->name);\n@@ -1417,7 +1414,7 @@ static int prune_refs(struct display_state *display_state,\n \t\t}\n \t\tstring_list_sort(&refnames);\n \t\trefs_warn_dangling_symrefs(get_main_ref_store(the_repository),\n-\t\t\t\t\t stderr, dangling_msg, &refnames);\n+\t\t\t\t\t stderr, \" \", dry_run, &refnames);\n \t}\n \n cleanup:\ndiff --git a/builtin/remote.c b/builtin/remote.c\nindex 4de7dd373ae5..f672799e0d92 100644\n--- a/builtin/remote.c\n+++ b/builtin/remote.c\n@@ -1521,9 +1521,6 @@ static int prune_remote(const char *remote, int dry_run)\n \tstruct ref_states states = REF_STATES_INIT;\n \tstruct string_list refs_to_prune = STRING_LIST_INIT_NODUP;\n \tstruct string_list_item *item;\n-\tconst char *dangling_msg = dry_run\n-\t\t? _(\" %s will become dangling after %s is deleted!\")\n-\t\t: _(\" %s has become dangling after %s was deleted!\");\n \n \tget_remote_ref_states(remote, &states, GET_REF_STATES);\n \n@@ -1555,7 +1552,7 @@ static int prune_remote(const char *remote, int dry_run)\n \t}\n \n \trefs_warn_dangling_symrefs(get_main_ref_store(the_repository),\n-\t\t\t\t stdout, dangling_msg, &refs_to_prune);\n+\t\t\t\t stdout, \" \", dry_run, &refs_to_prune);\n \n \tstring_list_clear(&refs_to_prune, 0);\n \tfree_remote_ref_states(&states);\ndiff --git a/refs.c b/refs.c\nindex 07197c239e33..5602c18dbd5b 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -439,7 +439,8 @@ struct warn_if_dangling_data {\n \tstruct ref_store *refs;\n \tFILE *fp;\n \tconst struct string_list *refnames;\n-\tconst char *msg_fmt;\n+\tconst char *indent;\n+\tint dry_run;\n };\n \n static int warn_if_dangling_symref(const char *refname, const char *referent UNUSED,\n@@ -447,7 +448,7 @@ static int warn_if_dangling_symref(const char *refname, const char *referent UNU\n \t\t\t\t int flags, void *cb_data)\n {\n \tstruct warn_if_dangling_data *d = cb_data;\n-\tconst char *resolves_to;\n+\tconst char *resolves_to, *msg;\n \n \tif (!(flags & REF_ISSYMREF))\n \t\treturn 0;\n@@ -458,19 +459,23 @@ static int warn_if_dangling_symref(const char *refname, const char *referent UNU\n \t\treturn 0;\n \t}\n \n-\tfprintf(d->fp, d->msg_fmt, refname, resolves_to);\n-\tfputc('\\n', d->fp);\n+\tmsg = d->dry_run\n+\t\t? _(\"%s%s will become dangling after %s is deleted\\n\")\n+\t\t: _(\"%s%s has become dangling after %s was deleted\\n\");\n+\tfprintf(d->fp, msg, d->indent, refname, resolves_to);\n \treturn 0;\n }\n \n void refs_warn_dangling_symrefs(struct ref_store *refs, FILE *fp,\n-\t\t\t\tconst char *msg_fmt, const struct string_list *refnames)\n+\t\t\t\tconst char *indent, int dry_run,\n+\t\t\t\tconst struct string_list *refnames)\n {\n \tstruct warn_if_dangling_data data = {\n \t\t.refs = refs,\n \t\t.fp = fp,\n \t\t.refnames = refnames,\n-\t\t.msg_fmt = msg_fmt,\n+\t\t.indent = indent,\n+\t\t.dry_run = dry_run,\n \t};\n \trefs_for_each_rawref(refs, warn_if_dangling_symref, &data);\n }\ndiff --git a/refs.h b/refs.h\nindex 07f21824d480..25bed4d792e7 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -453,7 +453,8 @@ static inline const char *has_glob_specials(const char *pattern)\n }\n \n void refs_warn_dangling_symrefs(struct ref_store *refs, FILE *fp,\n-\t\t\t\tconst char *msg_fmt, const struct string_list *refnames);\n+\t\t\t\tconst char *indent, int dry_run,\n+\t\t\t\tconst struct string_list *refnames);\n \n /*\n * Flags for controlling behaviour of pack_refs()\n-- \n2.50.0.149.g2f19833911.dirty\n\n"},{"id":"521138","messageId":"xmqqms9nfjvz.fsf@gitster.g","threadId":"63724","inReplyTo":"20250702005837.2813893-2-phil.hord@gmail.com","subject":"Re: [PATCH v3 0/2] fetch --prune performance problem","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-02T01:42:40Z","receivedAt":"2025-07-02T01:42:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phil Hord writes:\n\n> This version (V3) has three changes from V2:\n> - Removes a header declaration I forgot to move previously\n> - Cleans up the refs_warn_dangling_symrefs API to be more sane\n> - Drops the ref shortening that seems ill-advised in retrospect\n>\n> Phil Hord (2):\n> refs: remove old refs_warn_dangling_symref\n> clean up interface for refs_warn_dangling_symrefs\n\nHmph. On top of which commit did you base these two patches?\nThe second one does not apply on top of applying 1/2 on top of\neither v2.48.1 (where I queued the last round), v2.50.0 (the obvious\nchoice for a new development), or 'master'.\n\n$ git am -s writes:\n\n> Phil Hord writes:\n>\n>> This version (V3) has three changes from V2:\n>> - Removes a header declaration I forgot to move previously\n>> - Cleans up the refs_warn_dangling_symrefs API to be more sane\n>> - Drops the ref shortening that seems ill-advised in retrospect\n>>\n>> Phil Hord (2):\n>> refs: remove old refs_warn_dangling_symref\n>> clean up interface for refs_warn_dangling_symrefs\n>\n> Hmph. On top of which commit did you base these two patches?\n> The second one does not apply on top of applying 1/2 on top of\n> either v2.48.1 (where I queued the last round), v2.50.0 (the obvious\n> choice for a new development), or 'master'.\n>\n> $ git am -s error: patch failed: builtin/fetch.c:1384\n> error: builtin/fetch.c: patch does not apply\n> error: patch failed: builtin/remote.c:1521\n> error: builtin/remote.c: patch does not apply\n> error: patch failed: refs.c:458\n> error: refs.c: patch does not apply\n>\n> Thanks.\n\nAh, nevermind. I'll discard your v3 and will take a look at your v4\ninstead later.\n\n"}]}