{"thread":{"id":"57329","subject":"[PATCH 0/2] fetch: speed up mirror-fetches with many refs","startedAt":"2022-01-28T10:15:46Z","lastAt":"2022-02-10T18:04:47Z","messageCount":16,"participants":["Patrick Steinhardt","Taylor Blau","Christian Couder","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"447181","messageId":"cover.1643364888.git.ps@pks.im","threadId":"57329","inReplyTo":null,"subject":"[PATCH 0/2] fetch: speed up mirror-fetches with many refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-01-28T10:15:34Z","receivedAt":"2022-01-28T10:15:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series aims to speed up mirror-fetches in repositories with\nhuge amounts of refs. It contains two patches which together speed up\ngit-fetch(1) in a repository with about 2,1 million references by\nroughly 30%.\n\nPatrick\n\nPatrick Steinhardt (2):\n  fetch-pack: use commit-graph when computing cutoff\n  fetch: skip computing output width when not printing anything\n\n builtin/fetch.c |  8 ++++++--\n fetch-pack.c    | 28 ++++++++++++++++------------\n 2 files changed, 22 insertions(+), 14 deletions(-)\n\n-- \n2.35.0\n\n"},{"id":"447183","messageId":"31cf8f87a149c0fc8013b869e0e30364f3c60e01.1643364888.git.ps@pks.im","threadId":"57329","inReplyTo":"cover.1643364888.git.ps@pks.im","subject":"[PATCH 1/2] fetch-pack: use commit-graph when computing cutoff","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-01-28T10:17:03Z","receivedAt":"2022-01-28T10:17:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"During packfile negotiation we iterate over all refs announced by the\nremote side to check whether their IDs refer to commits already known to\nus. If a commit is known to us already, then its date is a potential\ncutoff point for commits we have in common with the remote side.\n\nThere is potentially a lot of commits announced by the remote depending\non how many refs there are in the remote repository, and for every one\nof them we need to search for it in our object database and, if found,\nparse the corresponding object to find out whether it is a candidate for\nthe cutoff date. This can be sped up by trying to look up commits via\nthe commit-graph first, which is a lot more efficient.\n\nOne thing to keep in mind though is that the commit-graph corrects\ncommitter dates:\n\n    * A commit with at least one parent has corrected committer date\n      equal to the maximum of its commiter date and one more than the\n      largest corrected committer date among its parents.\n\nAs a result, it may be that the commit date we load via the commit graph\nis more recent than it would have been when loaded via the ODB, and as a\nresult we may also choose a more recent cutoff point. But as the code\ndocuments, this is only a heuristic and it is okay if we determine a\nwrong cutoff date. The worst that can happen is that we report more\ncommits as HAVEs to the server when using corrected dates.\n\nLoading commits via the commit-graph is typically much faster than\nloading commits via the object database. Benchmarks in a repository with\nabout 2,1 million refs and an up-to-date commit-graph show a 20% speedup\nwhen mirror-fetching:\n\n    Benchmark 1: git fetch --atomic +refs/*:refs/* (v2.35.0)\n      Time (mean ± σ):     75.264 s ±  1.115 s    [User: 68.199 s, System: 10.094 s]\n      Range (min … max):   74.145 s … 76.862 s    5 runs\n\n    Benchmark 2: git fetch --atomic +refs/*:refs/* (HEAD)\n      Time (mean ± σ):     62.350 s ±  0.854 s    [User: 55.412 s, System: 9.976 s]\n      Range (min … max):   61.224 s … 63.216 s    5 runs\n\n    Summary\n      'git fetch --atomic +refs/*:refs/* (HEAD)' ran\n        1.21 ± 0.02 times faster than 'git fetch --atomic +refs/*:refs/* (v2.35.0)'\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n fetch-pack.c | 28 ++++++++++++++++------------\n 1 file changed, 16 insertions(+), 12 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex dd6ec449f2..c5967e228e 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -696,26 +696,30 @@ static void mark_complete_and_common_ref(struct fetch_negotiator *negotiator,\n \n \ttrace2_region_enter(\"fetch-pack\", \"parse_remote_refs_and_find_cutoff\", NULL);\n \tfor (ref = *refs; ref; ref = ref->next) {\n-\t\tstruct object *o;\n+\t\tstruct commit *commit;\n \n-\t\tif (!has_object_file_with_flags(&ref->old_oid,\n+\t\tcommit = lookup_commit_in_graph(the_repository, &ref->old_oid);\n+\t\tif (!commit) {\n+\t\t\tstruct object *o;\n+\n+\t\t\tif (!has_object_file_with_flags(&ref->old_oid,\n \t\t\t\t\t\tOBJECT_INFO_QUICK |\n-\t\t\t\t\t\t\tOBJECT_INFO_SKIP_FETCH_OBJECT))\n-\t\t\tcontinue;\n-\t\to = parse_object(the_repository, &ref->old_oid);\n-\t\tif (!o)\n-\t\t\tcontinue;\n+\t\t\t\t\t\tOBJECT_INFO_SKIP_FETCH_OBJECT))\n+\t\t\t\tcontinue;\n+\t\t\to = parse_object(the_repository, &ref->old_oid);\n+\t\t\tif (!o || o->type != OBJ_COMMIT)\n+\t\t\t\tcontinue;\n+\n+\t\t\tcommit = (struct commit *)o;\n+\t\t}\n \n \t\t/*\n \t\t * We already have it -- which may mean that we were\n \t\t * in sync with the other side at some time after\n \t\t * that (it is OK if we guess wrong here).\n \t\t */\n-\t\tif (o->type == OBJ_COMMIT) {\n-\t\t\tstruct commit *commit = (struct commit *)o;\n-\t\t\tif (!cutoff || cutoff < commit->date)\n-\t\t\t\tcutoff = commit->date;\n-\t\t}\n+\t\tif (!cutoff || cutoff < commit->date)\n+\t\t\tcutoff = commit->date;\n \t}\n \ttrace2_region_leave(\"fetch-pack\", \"parse_remote_refs_and_find_cutoff\", NULL);\n \n-- \n2.35.0\n\n"},{"id":"447185","messageId":"5a3fd3232fd9e19e6f0054717a1f54c71bd8f272.1643364888.git.ps@pks.im","threadId":"57329","inReplyTo":"cover.1643364888.git.ps@pks.im","subject":"[PATCH 2/2] fetch: skip computing output width when not printing anything","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-01-28T10:19:30Z","receivedAt":"2022-01-28T10:19:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When updating references via git-fetch(1), then by default we report to\nthe user which references have been changed. This output is formatted in\na nice table such that the different columns are aligned. Because the\nfirst column contains abbreviated object IDs we thus need to iterate\nover all refs which have changed and compute the minimum length for\ntheir respective abbreviated hashes. While this effort makes sense in\nmost cases, it is wasteful when the user passes the `--quiet` flag: we\ndon't print the summary, but still compute the length.\n\nSkip computing the summary width when the user asked for us to be quiet.\nThis gives us a small speedup of nearly 10% when doing a dry-run\nmirror-fetch in a repository with thousands of references being updated:\n\n    Benchmark 1: git fetch --prune --dry-run +refs/*:refs/* (HEAD~)\n      Time (mean ± σ):     34.048 s ±  0.233 s    [User: 30.739 s, System: 4.640 s]\n      Range (min … max):   33.785 s … 34.296 s    5 runs\n\n    Benchmark 2: git fetch --prune --dry-run +refs/*:refs/* (HEAD)\n      Time (mean ± σ):     30.768 s ±  0.287 s    [User: 27.534 s, System: 4.565 s]\n      Range (min … max):   30.432 s … 31.181 s    5 runs\n\n    Summary\n      'git fetch --prune --dry-run +refs/*:refs/* (HEAD)' ran\n        1.11 ± 0.01 times faster than 'git fetch --prune --dry-run +refs/*:refs/* (HEAD~)'\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n\n[Resend with correct In-Reply-To header.]\n\n builtin/fetch.c | 8 ++++++--\n 1 file changed, 6 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 5f06b21f8e..ebbde5d56d 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1093,12 +1093,15 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \tstruct ref *rm;\n \tchar *url;\n \tint want_status;\n-\tint summary_width = transport_summary_width(ref_map);\n+\tint summary_width = 0;\n \n \trc = open_fetch_head(&fetch_head);\n \tif (rc)\n \t\treturn -1;\n \n+\tif (verbosity >= 0)\n+\t\tsummary_width = transport_summary_width(ref_map);\n+\n \tif (raw_url)\n \t\turl = transport_anonymize_url(raw_url);\n \telse\n@@ -1344,7 +1347,6 @@ static int prune_refs(struct refspec *rs, struct ref *ref_map,\n \tint url_len, i, result = 0;\n \tstruct ref *ref, *stale_refs = get_stale_heads(rs, ref_map);\n \tchar *url;\n-\tint summary_width = transport_summary_width(stale_refs);\n \tconst char *dangling_msg = dry_run\n \t\t? _(\"   (%s will become dangling)\")\n \t\t: _(\"   (%s has become dangling)\");\n@@ -1373,6 +1375,8 @@ static int prune_refs(struct refspec *rs, struct ref *ref_map,\n \t}\n \n \tif (verbosity >= 0) {\n+\t\tint summary_width = transport_summary_width(stale_refs);\n+\n \t\tfor (ref = stale_refs; ref; ref = ref->next) {\n \t\t\tstruct strbuf sb = STRBUF_INIT;\n \t\t\tif (!shown_url) {\n-- \n2.35.0\n\n"},{"id":"447378","messageId":"YfhoWk1LFnzbFaQf@nand.local","threadId":"57329","inReplyTo":"31cf8f87a149c0fc8013b869e0e30364f3c60e01.1643364888.git.ps@pks.im","subject":"Re: [PATCH 1/2] fetch-pack: use commit-graph when computing cutoff","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-01-31T22:53:14Z","receivedAt":"2022-01-31T22:53:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"(+cc Stolee, in case anything I'm saying here is wrong)\n\nOn Fri, Jan 28, 2022 at 11:17:03AM +0100, Patrick Steinhardt wrote:\n> One thing to keep in mind though is that the commit-graph corrects\n> committer dates:\n>\n>     * A commit with at least one parent has corrected committer date\n>       equal to the maximum of its commiter date and one more than the\n>       largest corrected committer date among its parents.\n\nThis snippet refers to how correct committer dates are computed, not how\nthe commit dates themselves are stored.\n\nIndeed, the corrected committer date is used to compute the corrected\ncommit date offset, which is the \"v2\" generation number scheme (as\nopposed to topological levels, which make up \"v1\").\n\nBut that is entirely separate from the committer dates stored by the\ncommit-graph file, which are faithful representations of the exact\ncommitter date attached to each commit.\n\nLooking at the very last few lines of the main loop in\nwrite_graph_chunk_data() (where the committer dates are stored):\n\n    if (sizeof((*list)->date) > 4)\n      packedDate[0] = htonl(((*list)->date >> 32) & 0x3);\n    else\n      packedDate[0] = 0;\n\n    packedDate[0] |= htonl(*topo_level_slab_at(ctx->topo_levels, *list) << 2);\n    packedDate[1] = htonl((*list)->date);\n    hashwrite(f, packedDate, 8);\n\nthe low-order 34 bits are used to store the commit's `->date` field, and\nthe remaining high-order 30 bits are used to store the generation\nnumber. (You can look in `fill_commit_graph_info()` to see that we only\nuse those 34 bits to write back the date field).\n\nSo I think this paragraph (and the ones related to it) about this being\nan approximation and that being OK since this is a heuristic can all go\naway.\n\nThanks,\nTaylor\n"},{"id":"447535","messageId":"Yfp9KsvvQa5yWuLE@ncase","threadId":"57329","inReplyTo":"YfhoWk1LFnzbFaQf@nand.local","subject":"Re: [PATCH 1/2] fetch-pack: use commit-graph when computing cutoff","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-02-02T12:46:34Z","receivedAt":"2022-02-02T12:46:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Jan 31, 2022 at 05:53:14PM -0500, Taylor Blau wrote:\n> (+cc Stolee, in case anything I'm saying here is wrong)\n> \n> On Fri, Jan 28, 2022 at 11:17:03AM +0100, Patrick Steinhardt wrote:\n> > One thing to keep in mind though is that the commit-graph corrects\n> > committer dates:\n> >\n> >     * A commit with at least one parent has corrected committer date\n> >       equal to the maximum of its commiter date and one more than the\n> >       largest corrected committer date among its parents.\n> \n> This snippet refers to how correct committer dates are computed, not how\n> the commit dates themselves are stored.\n> \n> Indeed, the corrected committer date is used to compute the corrected\n> commit date offset, which is the \"v2\" generation number scheme (as\n> opposed to topological levels, which make up \"v1\").\n> \n> But that is entirely separate from the committer dates stored by the\n> commit-graph file, which are faithful representations of the exact\n> committer date attached to each commit.\n> \n> Looking at the very last few lines of the main loop in\n> write_graph_chunk_data() (where the committer dates are stored):\n> \n>     if (sizeof((*list)->date) > 4)\n>       packedDate[0] = htonl(((*list)->date >> 32) & 0x3);\n>     else\n>       packedDate[0] = 0;\n> \n>     packedDate[0] |= htonl(*topo_level_slab_at(ctx->topo_levels, *list) << 2);\n>     packedDate[1] = htonl((*list)->date);\n>     hashwrite(f, packedDate, 8);\n> \n> the low-order 34 bits are used to store the commit's `->date` field, and\n> the remaining high-order 30 bits are used to store the generation\n> number. (You can look in `fill_commit_graph_info()` to see that we only\n> use those 34 bits to write back the date field).\n> \n> So I think this paragraph (and the ones related to it) about this being\n> an approximation and that being OK since this is a heuristic can all go\n> away.\n> \n> Thanks,\n> Taylor\n\nAha, that makes a lot of sense. Thanks a lot for correcting my\nmisconception! I'll send a v2 of this patch series with the corrected\ncommit message.\n\nPatrick\n"},{"id":"447536","messageId":"cover.1643806143.git.ps@pks.im","threadId":"57329","inReplyTo":"cover.1643364888.git.ps@pks.im","subject":"[PATCH v2 0/2] fetch: speed up mirror-fetches with many refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-02-02T12:51:43Z","receivedAt":"2022-02-02T12:51:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the second version of my patch series which aims to speed up\nmirror-fetches in repos with huge amounts of refs. The only change\ncompared to v1 is a fixed up commit message: Taylor has pointed out to\nme that commit dates retrieved from the commit-graph are not in fact the\ncorrected commit dates, which are stored separately.\n\nPatrick\n\nPatrick Steinhardt (2):\n  fetch-pack: use commit-graph when computing cutoff\n  fetch: skip computing output width when not printing anything\n\n builtin/fetch.c |  8 ++++++--\n fetch-pack.c    | 28 ++++++++++++++++------------\n 2 files changed, 22 insertions(+), 14 deletions(-)\n\nRange-diff against v1:\n1:  31cf8f87a1 ! 1:  6fac914f0f fetch-pack: use commit-graph when computing cutoff\n    @@ Commit message\n         the cutoff date. This can be sped up by trying to look up commits via\n         the commit-graph first, which is a lot more efficient.\n     \n    -    One thing to keep in mind though is that the commit-graph corrects\n    -    committer dates:\n    -\n    -        * A commit with at least one parent has corrected committer date\n    -          equal to the maximum of its commiter date and one more than the\n    -          largest corrected committer date among its parents.\n    -\n    -    As a result, it may be that the commit date we load via the commit graph\n    -    is more recent than it would have been when loaded via the ODB, and as a\n    -    result we may also choose a more recent cutoff point. But as the code\n    -    documents, this is only a heuristic and it is okay if we determine a\n    -    wrong cutoff date. The worst that can happen is that we report more\n    -    commits as HAVEs to the server when using corrected dates.\n    -\n    -    Loading commits via the commit-graph is typically much faster than\n    -    loading commits via the object database. Benchmarks in a repository with\n    -    about 2,1 million refs and an up-to-date commit-graph show a 20% speedup\n    -    when mirror-fetching:\n    +    Benchmarks in a repository with about 2,1 million refs and an up-to-date\n    +    commit-graph show a 20% speedup when mirror-fetching:\n     \n             Benchmark 1: git fetch --atomic +refs/*:refs/* (v2.35.0)\n               Time (mean ± σ):     75.264 s ±  1.115 s    [User: 68.199 s, System: 10.094 s]\n2:  5a3fd3232f = 2:  4b9bbcf795 fetch: skip computing output width when not printing anything\n-- \n2.35.1\n\n"},{"id":"447537","messageId":"6fac914f0fe77df4c3058340642bea2a45a850cd.1643806143.git.ps@pks.im","threadId":"57329","inReplyTo":"cover.1643806143.git.ps@pks.im","subject":"[PATCH v2 1/2] fetch-pack: use commit-graph when computing cutoff","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-02-02T12:51:48Z","receivedAt":"2022-02-02T12:51:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"During packfile negotiation we iterate over all refs announced by the\nremote side to check whether their IDs refer to commits already known to\nus. If a commit is known to us already, then its date is a potential\ncutoff point for commits we have in common with the remote side.\n\nThere is potentially a lot of commits announced by the remote depending\non how many refs there are in the remote repository, and for every one\nof them we need to search for it in our object database and, if found,\nparse the corresponding object to find out whether it is a candidate for\nthe cutoff date. This can be sped up by trying to look up commits via\nthe commit-graph first, which is a lot more efficient.\n\nBenchmarks in a repository with about 2,1 million refs and an up-to-date\ncommit-graph show a 20% speedup when mirror-fetching:\n\n    Benchmark 1: git fetch --atomic +refs/*:refs/* (v2.35.0)\n      Time (mean ± σ):     75.264 s ±  1.115 s    [User: 68.199 s, System: 10.094 s]\n      Range (min … max):   74.145 s … 76.862 s    5 runs\n\n    Benchmark 2: git fetch --atomic +refs/*:refs/* (HEAD)\n      Time (mean ± σ):     62.350 s ±  0.854 s    [User: 55.412 s, System: 9.976 s]\n      Range (min … max):   61.224 s … 63.216 s    5 runs\n\n    Summary\n      'git fetch --atomic +refs/*:refs/* (HEAD)' ran\n        1.21 ± 0.02 times faster than 'git fetch --atomic +refs/*:refs/* (v2.35.0)'\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n fetch-pack.c | 28 ++++++++++++++++------------\n 1 file changed, 16 insertions(+), 12 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex dd6ec449f2..c5967e228e 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -696,26 +696,30 @@ static void mark_complete_and_common_ref(struct fetch_negotiator *negotiator,\n \n \ttrace2_region_enter(\"fetch-pack\", \"parse_remote_refs_and_find_cutoff\", NULL);\n \tfor (ref = *refs; ref; ref = ref->next) {\n-\t\tstruct object *o;\n+\t\tstruct commit *commit;\n \n-\t\tif (!has_object_file_with_flags(&ref->old_oid,\n+\t\tcommit = lookup_commit_in_graph(the_repository, &ref->old_oid);\n+\t\tif (!commit) {\n+\t\t\tstruct object *o;\n+\n+\t\t\tif (!has_object_file_with_flags(&ref->old_oid,\n \t\t\t\t\t\tOBJECT_INFO_QUICK |\n-\t\t\t\t\t\t\tOBJECT_INFO_SKIP_FETCH_OBJECT))\n-\t\t\tcontinue;\n-\t\to = parse_object(the_repository, &ref->old_oid);\n-\t\tif (!o)\n-\t\t\tcontinue;\n+\t\t\t\t\t\tOBJECT_INFO_SKIP_FETCH_OBJECT))\n+\t\t\t\tcontinue;\n+\t\t\to = parse_object(the_repository, &ref->old_oid);\n+\t\t\tif (!o || o->type != OBJ_COMMIT)\n+\t\t\t\tcontinue;\n+\n+\t\t\tcommit = (struct commit *)o;\n+\t\t}\n \n \t\t/*\n \t\t * We already have it -- which may mean that we were\n \t\t * in sync with the other side at some time after\n \t\t * that (it is OK if we guess wrong here).\n \t\t */\n-\t\tif (o->type == OBJ_COMMIT) {\n-\t\t\tstruct commit *commit = (struct commit *)o;\n-\t\t\tif (!cutoff || cutoff < commit->date)\n-\t\t\t\tcutoff = commit->date;\n-\t\t}\n+\t\tif (!cutoff || cutoff < commit->date)\n+\t\t\tcutoff = commit->date;\n \t}\n \ttrace2_region_leave(\"fetch-pack\", \"parse_remote_refs_and_find_cutoff\", NULL);\n \n-- \n2.35.1\n\n"},{"id":"447538","messageId":"4b9bbcf7958da6db2be771f910974a6bffb2d94a.1643806143.git.ps@pks.im","threadId":"57329","inReplyTo":"cover.1643806143.git.ps@pks.im","subject":"[PATCH v2 2/2] fetch: skip computing output width when not printing anything","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-02-02T12:51:52Z","receivedAt":"2022-02-02T12:51:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When updating references via git-fetch(1), then by default we report to\nthe user which references have been changed. This output is formatted in\na nice table such that the different columns are aligned. Because the\nfirst column contains abbreviated object IDs we thus need to iterate\nover all refs which have changed and compute the minimum length for\ntheir respective abbreviated hashes. While this effort makes sense in\nmost cases, it is wasteful when the user passes the `--quiet` flag: we\ndon't print the summary, but still compute the length.\n\nSkip computing the summary width when the user asked for us to be quiet.\nThis gives us a small speedup of nearly 10% when doing a dry-run\nmirror-fetch in a repository with thousands of references being updated:\n\n    Benchmark 1: git fetch --prune --dry-run +refs/*:refs/* (HEAD~)\n      Time (mean ± σ):     34.048 s ±  0.233 s    [User: 30.739 s, System: 4.640 s]\n      Range (min … max):   33.785 s … 34.296 s    5 runs\n\n    Benchmark 2: git fetch --prune --dry-run +refs/*:refs/* (HEAD)\n      Time (mean ± σ):     30.768 s ±  0.287 s    [User: 27.534 s, System: 4.565 s]\n      Range (min … max):   30.432 s … 31.181 s    5 runs\n\n    Summary\n      'git fetch --prune --dry-run +refs/*:refs/* (HEAD)' ran\n        1.11 ± 0.01 times faster than 'git fetch --prune --dry-run +refs/*:refs/* (HEAD~)'\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/fetch.c | 8 ++++++--\n 1 file changed, 6 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 5f06b21f8e..ebbde5d56d 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1093,12 +1093,15 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \tstruct ref *rm;\n \tchar *url;\n \tint want_status;\n-\tint summary_width = transport_summary_width(ref_map);\n+\tint summary_width = 0;\n \n \trc = open_fetch_head(&fetch_head);\n \tif (rc)\n \t\treturn -1;\n \n+\tif (verbosity >= 0)\n+\t\tsummary_width = transport_summary_width(ref_map);\n+\n \tif (raw_url)\n \t\turl = transport_anonymize_url(raw_url);\n \telse\n@@ -1344,7 +1347,6 @@ static int prune_refs(struct refspec *rs, struct ref *ref_map,\n \tint url_len, i, result = 0;\n \tstruct ref *ref, *stale_refs = get_stale_heads(rs, ref_map);\n \tchar *url;\n-\tint summary_width = transport_summary_width(stale_refs);\n \tconst char *dangling_msg = dry_run\n \t\t? _(\"   (%s will become dangling)\")\n \t\t: _(\"   (%s has become dangling)\");\n@@ -1373,6 +1375,8 @@ static int prune_refs(struct refspec *rs, struct ref *ref_map,\n \t}\n \n \tif (verbosity >= 0) {\n+\t\tint summary_width = transport_summary_width(stale_refs);\n+\n \t\tfor (ref = stale_refs; ref; ref = ref->next) {\n \t\t\tstruct strbuf sb = STRBUF_INIT;\n \t\t\tif (!shown_url) {\n-- \n2.35.1\n\n"},{"id":"448059","messageId":"CAP8UFD2H-WfKRpwY7jNf=Dc0g1_Kc1adY4-c2OfkoMg3wHJCrA@mail.gmail.com","threadId":"57329","inReplyTo":"6fac914f0fe77df4c3058340642bea2a45a850cd.1643806143.git.ps@pks.im","subject":"Re: [PATCH v2 1/2] fetch-pack: use commit-graph when computing cutoff","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2022-02-09T18:01:54Z","receivedAt":"2022-02-09T18:02:08Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Feb 7, 2022 at 7:03 AM Patrick Steinhardt <ps@pks.im> wrote:\n\n> Benchmarks in a repository with about 2,1 million refs and an up-to-date\n> commit-graph show a 20% speedup when mirror-fetching:\n>\n>     Benchmark 1: git fetch --atomic +refs/*:refs/* (v2.35.0)\n>       Time (mean ± σ):     75.264 s ±  1.115 s    [User: 68.199 s, System: 10.094 s]\n>       Range (min … max):   74.145 s … 76.862 s    5 runs\n>\n>     Benchmark 2: git fetch --atomic +refs/*:refs/* (HEAD)\n>       Time (mean ± σ):     62.350 s ±  0.854 s    [User: 55.412 s, System: 9.976 s]\n>       Range (min … max):   61.224 s … 63.216 s    5 runs\n>\n>     Summary\n>       'git fetch --atomic +refs/*:refs/* (HEAD)' ran\n>         1.21 ± 0.02 times faster than 'git fetch --atomic +refs/*:refs/* (v2.35.0)'\n\nThe commit message and code make sense to me, but I wonder if there is\na reason why --atomic is used when fetching.\n"},{"id":"448062","messageId":"CAP8UFD0r2ZnGCAjkQcLfEZCvxHVFGSWeAC7vSat7V7-=rhiGeQ@mail.gmail.com","threadId":"57329","inReplyTo":"4b9bbcf7958da6db2be771f910974a6bffb2d94a.1643806143.git.ps@pks.im","subject":"Re: [PATCH v2 2/2] fetch: skip computing output width when not printing anything","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2022-02-09T18:10:38Z","receivedAt":"2022-02-09T18:10:53Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Feb 2, 2022 at 5:13 PM Patrick Steinhardt <ps@pks.im> wrote:\n\n> Skip computing the summary width when the user asked for us to be quiet.\n\nThere is a --quiet option for git fetch, so here we can expect that it\nwill be used to test this speedup...\n\n> This gives us a small speedup of nearly 10% when doing a dry-run\n> mirror-fetch in a repository with thousands of references being updated:\n>\n>     Benchmark 1: git fetch --prune --dry-run +refs/*:refs/* (HEAD~)\n>       Time (mean ± σ):     34.048 s ±  0.233 s    [User: 30.739 s, System: 4.640 s]\n>       Range (min … max):   33.785 s … 34.296 s    5 runs\n>\n>     Benchmark 2: git fetch --prune --dry-run +refs/*:refs/* (HEAD)\n>       Time (mean ± σ):     30.768 s ±  0.287 s    [User: 27.534 s, System: 4.565 s]\n>       Range (min … max):   30.432 s … 31.181 s    5 runs\n>\n>     Summary\n>       'git fetch --prune --dry-run +refs/*:refs/* (HEAD)' ran\n>         1.11 ± 0.01 times faster than 'git fetch --prune --dry-run +refs/*:refs/* (HEAD~)'\n\n...but --prune and --dry-run are used for testing. It would be nice if\nthis discrepancy was explained a bit.\n\nOtherwise the commit message and code look good to me.\n"},{"id":"448131","messageId":"YgT6WWsb1QaKIbsu@ncase","threadId":"57329","inReplyTo":"CAP8UFD2H-WfKRpwY7jNf=Dc0g1_Kc1adY4-c2OfkoMg3wHJCrA@mail.gmail.com","subject":"Re: [PATCH v2 1/2] fetch-pack: use commit-graph when computing cutoff","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-02-10T11:43:21Z","receivedAt":"2022-02-10T11:50:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Feb 09, 2022 at 07:01:54PM +0100, Christian Couder wrote:\n> On Mon, Feb 7, 2022 at 7:03 AM Patrick Steinhardt <ps@pks.im> wrote:\n> \n> > Benchmarks in a repository with about 2,1 million refs and an up-to-date\n> > commit-graph show a 20% speedup when mirror-fetching:\n> >\n> >     Benchmark 1: git fetch --atomic +refs/*:refs/* (v2.35.0)\n> >       Time (mean ± σ):     75.264 s ±  1.115 s    [User: 68.199 s, System: 10.094 s]\n> >       Range (min … max):   74.145 s … 76.862 s    5 runs\n> >\n> >     Benchmark 2: git fetch --atomic +refs/*:refs/* (HEAD)\n> >       Time (mean ± σ):     62.350 s ±  0.854 s    [User: 55.412 s, System: 9.976 s]\n> >       Range (min … max):   61.224 s … 63.216 s    5 runs\n> >\n> >     Summary\n> >       'git fetch --atomic +refs/*:refs/* (HEAD)' ran\n> >         1.21 ± 0.02 times faster than 'git fetch --atomic +refs/*:refs/* (v2.35.0)'\n> \n> The commit message and code make sense to me, but I wonder if there is\n> a reason why --atomic is used when fetching.\n\nThe repository that I was mirror-fetching into needs to update a big\nbunch of references, and doing that via `--atomic` is more efficient\nthan doing it without, and this shows in the benchmark. I did another\nbenchmarking run without `--atomic`, and it is indeed about 30 seconds\nslower for both cases. But interestingly the relative performance\nimprovement is still roughly the same:\n\n    Benchmark 1: git fetch +refs/*:refs/* (v2.35.0)\n      Time (mean ± σ):     115.587 s ±  2.009 s    [User: 109.874 s, System: 11.305 s]\n      Range (min … max):   113.584 s … 118.820 s    5 runs\n\n    Benchmark 2: git fetch +refs/*:refs/* (pks-fetch-pack-optim-v1~)\n      Time (mean ± σ):     96.859 s ±  0.624 s    [User: 91.948 s, System: 10.980 s]\n      Range (min … max):   96.180 s … 97.875 s    5 runs\n\n    Summary\n      'git fetch +refs/*:refs/* (pks-fetch-pack-optim-v1~)' ran\n        1.19 ± 0.02 times faster than 'git fetch +refs/*:refs/* (v2.35.0)'\n\nI'll update the commit message to just use this new benchmark so that\nthe `--atomic` flag doesn't cause any questions.\n\nPatrick\n"},{"id":"448132","messageId":"cover.1644495978.git.ps@pks.im","threadId":"57329","inReplyTo":"cover.1643806143.git.ps@pks.im","subject":"[PATCH v3 0/2] fetch: speed up mirror-fetches with many refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-02-10T12:28:01Z","receivedAt":"2022-02-10T12:28:08Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the third version of my patch series which aimn to speed up\nmirror-fetches in repos with huge amounts of refs. Again, the only\nchange compared to v2 is a change in commit messages: Chris has rightly\npointed out that the benchmarks were a bit confusing, so I've updated\nthem to hopefully be less so.\n\nThanks for your feedback!\n\nPatrick\n\nPatrick Steinhardt (2):\n  fetch-pack: use commit-graph when computing cutoff\n  fetch: skip computing output width when not printing anything\n\n builtin/fetch.c |  8 ++++++--\n fetch-pack.c    | 28 ++++++++++++++++------------\n 2 files changed, 22 insertions(+), 14 deletions(-)\n\nRange-diff against v2:\n1:  6fac914f0f ! 1:  077d06764c fetch-pack: use commit-graph when computing cutoff\n    @@ Commit message\n         the commit-graph first, which is a lot more efficient.\n     \n         Benchmarks in a repository with about 2,1 million refs and an up-to-date\n    -    commit-graph show a 20% speedup when mirror-fetching:\n    +    commit-graph show an almost 20% speedup when mirror-fetching:\n     \n    -        Benchmark 1: git fetch --atomic +refs/*:refs/* (v2.35.0)\n    -          Time (mean ± σ):     75.264 s ±  1.115 s    [User: 68.199 s, System: 10.094 s]\n    -          Range (min … max):   74.145 s … 76.862 s    5 runs\n    +        Benchmark 1: git fetch +refs/*:refs/* (v2.35.0)\n    +          Time (mean ± σ):     115.587 s ±  2.009 s    [User: 109.874 s, System: 11.305 s]\n    +          Range (min … max):   113.584 s … 118.820 s    5 runs\n     \n    -        Benchmark 2: git fetch --atomic +refs/*:refs/* (HEAD)\n    -          Time (mean ± σ):     62.350 s ±  0.854 s    [User: 55.412 s, System: 9.976 s]\n    -          Range (min … max):   61.224 s … 63.216 s    5 runs\n    +        Benchmark 2: git fetch +refs/*:refs/* (HEAD)\n    +          Time (mean ± σ):     96.859 s ±  0.624 s    [User: 91.948 s, System: 10.980 s]\n    +          Range (min … max):   96.180 s … 97.875 s    5 runs\n     \n             Summary\n    -          'git fetch --atomic +refs/*:refs/* (HEAD)' ran\n    -            1.21 ± 0.02 times faster than 'git fetch --atomic +refs/*:refs/* (v2.35.0)'\n    +          'git fetch +refs/*:refs/* (HEAD)' ran\n    +            1.19 ± 0.02 times faster than 'git fetch +refs/*:refs/* (v2.35.0)'\n     \n         Signed-off-by: Patrick Steinhardt <ps@pks.im>\n     \n2:  4b9bbcf795 ! 2:  ef1fd07be5 fetch: skip computing output width when not printing anything\n    @@ Commit message\n         don't print the summary, but still compute the length.\n     \n         Skip computing the summary width when the user asked for us to be quiet.\n    -    This gives us a small speedup of nearly 10% when doing a dry-run\n    -    mirror-fetch in a repository with thousands of references being updated:\n    +    This gives us a speedup of nearly 10% when doing a mirror-fetch in a\n    +    repository with thousands of references being updated:\n     \n    -        Benchmark 1: git fetch --prune --dry-run +refs/*:refs/* (HEAD~)\n    -          Time (mean ± σ):     34.048 s ±  0.233 s    [User: 30.739 s, System: 4.640 s]\n    -          Range (min … max):   33.785 s … 34.296 s    5 runs\n    +        Benchmark 1: git fetch --quiet +refs/*:refs/* (HEAD~)\n    +          Time (mean ± σ):     96.078 s ±  0.508 s    [User: 91.378 s, System: 10.870 s]\n    +          Range (min … max):   95.449 s … 96.760 s    5 runs\n     \n    -        Benchmark 2: git fetch --prune --dry-run +refs/*:refs/* (HEAD)\n    -          Time (mean ± σ):     30.768 s ±  0.287 s    [User: 27.534 s, System: 4.565 s]\n    -          Range (min … max):   30.432 s … 31.181 s    5 runs\n    +        Benchmark 2: git fetch --quiet +refs/*:refs/* (HEAD)\n    +          Time (mean ± σ):     88.214 s ±  0.192 s    [User: 83.274 s, System: 10.978 s]\n    +          Range (min … max):   87.998 s … 88.446 s    5 runs\n     \n             Summary\n    -          'git fetch --prune --dry-run +refs/*:refs/* (HEAD)' ran\n    -            1.11 ± 0.01 times faster than 'git fetch --prune --dry-run +refs/*:refs/* (HEAD~)'\n    +          'git fetch --quiet +refs/*:refs/* (HEAD)' ran\n    +            1.09 ± 0.01 times faster than 'git fetch --quiet +refs/*:refs/* (HEAD~)'\n     \n         Signed-off-by: Patrick Steinhardt <ps@pks.im>\n     \n-- \n2.35.1\n\n"},{"id":"448133","messageId":"077d06764cafde470ea0589072d387d7737bb91e.1644495978.git.ps@pks.im","threadId":"57329","inReplyTo":"cover.1644495978.git.ps@pks.im","subject":"[PATCH v3 1/2] fetch-pack: use commit-graph when computing cutoff","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-02-10T12:28:09Z","receivedAt":"2022-02-10T12:28:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"During packfile negotiation we iterate over all refs announced by the\nremote side to check whether their IDs refer to commits already known to\nus. If a commit is known to us already, then its date is a potential\ncutoff point for commits we have in common with the remote side.\n\nThere is potentially a lot of commits announced by the remote depending\non how many refs there are in the remote repository, and for every one\nof them we need to search for it in our object database and, if found,\nparse the corresponding object to find out whether it is a candidate for\nthe cutoff date. This can be sped up by trying to look up commits via\nthe commit-graph first, which is a lot more efficient.\n\nBenchmarks in a repository with about 2,1 million refs and an up-to-date\ncommit-graph show an almost 20% speedup when mirror-fetching:\n\n    Benchmark 1: git fetch +refs/*:refs/* (v2.35.0)\n      Time (mean ± σ):     115.587 s ±  2.009 s    [User: 109.874 s, System: 11.305 s]\n      Range (min … max):   113.584 s … 118.820 s    5 runs\n\n    Benchmark 2: git fetch +refs/*:refs/* (HEAD)\n      Time (mean ± σ):     96.859 s ±  0.624 s    [User: 91.948 s, System: 10.980 s]\n      Range (min … max):   96.180 s … 97.875 s    5 runs\n\n    Summary\n      'git fetch +refs/*:refs/* (HEAD)' ran\n        1.19 ± 0.02 times faster than 'git fetch +refs/*:refs/* (v2.35.0)'\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n fetch-pack.c | 28 ++++++++++++++++------------\n 1 file changed, 16 insertions(+), 12 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex dd6ec449f2..c5967e228e 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -696,26 +696,30 @@ static void mark_complete_and_common_ref(struct fetch_negotiator *negotiator,\n \n \ttrace2_region_enter(\"fetch-pack\", \"parse_remote_refs_and_find_cutoff\", NULL);\n \tfor (ref = *refs; ref; ref = ref->next) {\n-\t\tstruct object *o;\n+\t\tstruct commit *commit;\n \n-\t\tif (!has_object_file_with_flags(&ref->old_oid,\n+\t\tcommit = lookup_commit_in_graph(the_repository, &ref->old_oid);\n+\t\tif (!commit) {\n+\t\t\tstruct object *o;\n+\n+\t\t\tif (!has_object_file_with_flags(&ref->old_oid,\n \t\t\t\t\t\tOBJECT_INFO_QUICK |\n-\t\t\t\t\t\t\tOBJECT_INFO_SKIP_FETCH_OBJECT))\n-\t\t\tcontinue;\n-\t\to = parse_object(the_repository, &ref->old_oid);\n-\t\tif (!o)\n-\t\t\tcontinue;\n+\t\t\t\t\t\tOBJECT_INFO_SKIP_FETCH_OBJECT))\n+\t\t\t\tcontinue;\n+\t\t\to = parse_object(the_repository, &ref->old_oid);\n+\t\t\tif (!o || o->type != OBJ_COMMIT)\n+\t\t\t\tcontinue;\n+\n+\t\t\tcommit = (struct commit *)o;\n+\t\t}\n \n \t\t/*\n \t\t * We already have it -- which may mean that we were\n \t\t * in sync with the other side at some time after\n \t\t * that (it is OK if we guess wrong here).\n \t\t */\n-\t\tif (o->type == OBJ_COMMIT) {\n-\t\t\tstruct commit *commit = (struct commit *)o;\n-\t\t\tif (!cutoff || cutoff < commit->date)\n-\t\t\t\tcutoff = commit->date;\n-\t\t}\n+\t\tif (!cutoff || cutoff < commit->date)\n+\t\t\tcutoff = commit->date;\n \t}\n \ttrace2_region_leave(\"fetch-pack\", \"parse_remote_refs_and_find_cutoff\", NULL);\n \n-- \n2.35.1\n\n"},{"id":"448134","messageId":"ef1fd07be5e4c50fc096a4a373fbcdc36dd4cbea.1644495978.git.ps@pks.im","threadId":"57329","inReplyTo":"cover.1644495978.git.ps@pks.im","subject":"[PATCH v3 2/2] fetch: skip computing output width when not printing anything","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-02-10T12:28:16Z","receivedAt":"2022-02-10T12:28:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When updating references via git-fetch(1), then by default we report to\nthe user which references have been changed. This output is formatted in\na nice table such that the different columns are aligned. Because the\nfirst column contains abbreviated object IDs we thus need to iterate\nover all refs which have changed and compute the minimum length for\ntheir respective abbreviated hashes. While this effort makes sense in\nmost cases, it is wasteful when the user passes the `--quiet` flag: we\ndon't print the summary, but still compute the length.\n\nSkip computing the summary width when the user asked for us to be quiet.\nThis gives us a speedup of nearly 10% when doing a mirror-fetch in a\nrepository with thousands of references being updated:\n\n    Benchmark 1: git fetch --quiet +refs/*:refs/* (HEAD~)\n      Time (mean ± σ):     96.078 s ±  0.508 s    [User: 91.378 s, System: 10.870 s]\n      Range (min … max):   95.449 s … 96.760 s    5 runs\n\n    Benchmark 2: git fetch --quiet +refs/*:refs/* (HEAD)\n      Time (mean ± σ):     88.214 s ±  0.192 s    [User: 83.274 s, System: 10.978 s]\n      Range (min … max):   87.998 s … 88.446 s    5 runs\n\n    Summary\n      'git fetch --quiet +refs/*:refs/* (HEAD)' ran\n        1.09 ± 0.01 times faster than 'git fetch --quiet +refs/*:refs/* (HEAD~)'\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/fetch.c | 8 ++++++--\n 1 file changed, 6 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 5b3b18a72f..7ef305c66d 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1094,12 +1094,15 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \tstruct ref *rm;\n \tchar *url;\n \tint want_status;\n-\tint summary_width = transport_summary_width(ref_map);\n+\tint summary_width = 0;\n \n \trc = open_fetch_head(&fetch_head);\n \tif (rc)\n \t\treturn -1;\n \n+\tif (verbosity >= 0)\n+\t\tsummary_width = transport_summary_width(ref_map);\n+\n \tif (raw_url)\n \t\turl = transport_anonymize_url(raw_url);\n \telse\n@@ -1345,7 +1348,6 @@ static int prune_refs(struct refspec *rs, struct ref *ref_map,\n \tint url_len, i, result = 0;\n \tstruct ref *ref, *stale_refs = get_stale_heads(rs, ref_map);\n \tchar *url;\n-\tint summary_width = transport_summary_width(stale_refs);\n \tconst char *dangling_msg = dry_run\n \t\t? _(\"   (%s will become dangling)\")\n \t\t: _(\"   (%s has become dangling)\");\n@@ -1374,6 +1376,8 @@ static int prune_refs(struct refspec *rs, struct ref *ref_map,\n \t}\n \n \tif (verbosity >= 0) {\n+\t\tint summary_width = transport_summary_width(stale_refs);\n+\n \t\tfor (ref = stale_refs; ref; ref = ref->next) {\n \t\t\tstruct strbuf sb = STRBUF_INIT;\n \t\t\tif (!shown_url) {\n-- \n2.35.1\n\n"},{"id":"448135","messageId":"YgUDyLhO7UykfbF/@ncase","threadId":"57329","inReplyTo":"CAP8UFD0r2ZnGCAjkQcLfEZCvxHVFGSWeAC7vSat7V7-=rhiGeQ@mail.gmail.com","subject":"Re: [PATCH v2 2/2] fetch: skip computing output width when not printing anything","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-02-10T12:23:36Z","receivedAt":"2022-02-10T12:30:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Feb 09, 2022 at 07:10:38PM +0100, Christian Couder wrote:\n> On Wed, Feb 2, 2022 at 5:13 PM Patrick Steinhardt <ps@pks.im> wrote:\n> \n> > Skip computing the summary width when the user asked for us to be quiet.\n> \n> There is a --quiet option for git fetch, so here we can expect that it\n> will be used to test this speedup...\n> \n> > This gives us a small speedup of nearly 10% when doing a dry-run\n> > mirror-fetch in a repository with thousands of references being updated:\n> >\n> >     Benchmark 1: git fetch --prune --dry-run +refs/*:refs/* (HEAD~)\n> >       Time (mean ± σ):     34.048 s ±  0.233 s    [User: 30.739 s, System: 4.640 s]\n> >       Range (min … max):   33.785 s … 34.296 s    5 runs\n> >\n> >     Benchmark 2: git fetch --prune --dry-run +refs/*:refs/* (HEAD)\n> >       Time (mean ± σ):     30.768 s ±  0.287 s    [User: 27.534 s, System: 4.565 s]\n> >       Range (min … max):   30.432 s … 31.181 s    5 runs\n> >\n> >     Summary\n> >       'git fetch --prune --dry-run +refs/*:refs/* (HEAD)' ran\n> >         1.11 ± 0.01 times faster than 'git fetch --prune --dry-run +refs/*:refs/* (HEAD~)'\n> \n> ...but --prune and --dry-run are used for testing. It would be nice if\n> this discrepancy was explained a bit.\n> \n> Otherwise the commit message and code look good to me.\n\nYeah, I was hiding away the `--quiet` flag here by accident. I used\n`--prune` and `--dry-run` to trigger more lines to be printed to console\nand to not take into account the time it takes to update local refs and\nfetch objects. But doing so without these flags also demonstrates what I\nwant to:\n\n    Benchmark 1: git fetch --quiet +refs/*:refs/* (pks-fetch-pack-optim-v1~)\n      Time (mean ± σ):     96.078 s ±  0.508 s    [User: 91.378 s, System: 10.870 s]\n      Range (min … max):   95.449 s … 96.760 s    5 runs\n\n    Benchmark 2: git fetch --quiet +refs/*:refs/* (pks-fetch-pack-optim-v1)\n      Time (mean ± σ):     88.214 s ±  0.192 s    [User: 83.274 s, System: 10.978 s]\n      Range (min … max):   87.998 s … 88.446 s    5 runs\n\nSo again, I'll update the commit message.\n\nThanks for your feedback!\n\nPatrick\n"},{"id":"448176","messageId":"xmqqfsoqvbra.fsf@gitster.g","threadId":"57329","inReplyTo":"cover.1644495978.git.ps@pks.im","subject":"Re: [PATCH v3 0/2] fetch: speed up mirror-fetches with many refs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-10T18:04:41Z","receivedAt":"2022-02-10T18:04:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"1Patrick Steinhardt <ps@pks.im> writes:\n\n> this is the third version of my patch series which aimn to speed up\n> mirror-fetches in repos with huge amounts of refs. Again, the only\n> change compared to v2 is a change in commit messages: Chris has rightly\n> pointed out that the benchmarks were a bit confusing, so I've updated\n> them to hopefully be less so.\n>\n> Thanks for your feedback!\n\n> Patrick\n>\n> Patrick Steinhardt (2):\n>   fetch-pack: use commit-graph when computing cutoff\n>   fetch: skip computing output width when not printing anything\n\nBoth changes are based on quite sensible idea.  If we have\nprecomputed dates for each commit, it makes sense to look it up\nbefore parsing the commit.  If we are not preparing output, there is\nno point in computing the output width.\n\nVery simple and potentially effective.\n\nWill queue.  Thanks.\n\n"}]}