{"thread":{"id":"49275","subject":"[PATCH 0/2] commit-graph: add progress output","startedAt":"2018-09-04T20:27:47Z","lastAt":"2019-03-22T17:23:10Z","messageCount":133,"participants":["Ævar Arnfjörð Bjarmason","Eric Sunshine","Junio C Hamano","Derrick Stolee","Jeff King","Duy Nguyen","SZEDER Gábor","Martin Ågren"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"357342","messageId":"20180904202729.13900-1-avarab@gmail.com","threadId":"49275","inReplyTo":null,"subject":"[PATCH 0/2] commit-graph: add progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-09-04T20:27:27Z","receivedAt":"2018-09-04T20:27:47Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This series adds progress output to the commit-graph command, so that\nwhen it's called by \"git gc\" or \"git fsck\" we can see what's going on\nwith it.\n\nÆvar Arnfjörð Bjarmason (2):\n commit-graph write: add progress output\n commit-graph verify: add progress output\n\n commit-graph.c | 44 +++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 43 insertions(+), 1 deletion(-)\n\n-- \n2.19.0.rc1.350.ge57e33dbd1\n\n"},{"id":"357343","messageId":"20180904202729.13900-3-avarab@gmail.com","threadId":"49275","inReplyTo":"20180904202729.13900-1-avarab@gmail.com","subject":"[PATCH 2/2] commit-graph verify: add progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-09-04T20:27:29Z","receivedAt":"2018-09-04T20:27:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"For the reasons explained in the \"commit-graph write: add progress\noutput\" commit leading up to this one, emit progress on \"commit-graph\nverify\". Since e0fd51e1d7 (\"fsck: verify commit-graph\", 2018-06-27)\n\"git fsck\" has called this command if core.commitGraph=true, but\nthere's been no progress output to indicate that anything was\ndifferent. Now there is (on my tiny dotfiles.git repository):\n\n $ git -c core.commitGraph=true -C ~/ fsck\n Checking object directories: 100% (256/256), done.\n Checking objects: 100% (2821/2821), done.\n dangling blob 5b8bbdb9b788ed90459f505b0934619c17cc605b\n Verifying commits in commit graph: 100% (867/867), done.\n\nAnd on a larger repository, such as the 2015-04-03-1M-git.git test\nrepository:\n\n $ time git -c core.commitGraph=true -C ~/g/2015-04-03-1M-git/ commit-graph verify\n Verifying commits in commit graph: 100% (1000447/1000447), done.\n real 0m7.813s\n [...]\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 74889dc90a..1a02fe019a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -914,6 +914,7 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \tint generation_zero = 0;\n \tstruct hashfile *f;\n \tint devnull;\n+\tstruct progress *progress = NULL;\n \n \tif (!g) {\n \t\tgraph_report(\"no commit-graph file loaded\");\n@@ -981,11 +982,14 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \tif (verify_commit_graph_error & ~VERIFY_COMMIT_GRAPH_ERROR_HASH)\n \t\treturn verify_commit_graph_error;\n \n+\tprogress = start_progress(\"Verifying commits in commit graph\",\n+\t\t\t\t g->num_commits);\n \tfor (i = 0; i < g->num_commits; i++) {\n \t\tstruct commit *graph_commit, *odb_commit;\n \t\tstruct commit_list *graph_parents, *odb_parents;\n \t\tuint32_t max_generation = 0;\n \n+\t\tdisplay_progress(progress, i);\n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n \t\tgraph_commit = lookup_commit(r, &cur_oid);\n@@ -1062,6 +1066,8 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\t\t\t graph_commit->date,\n \t\t\t\t odb_commit->date);\n \t}\n+\tdisplay_progress(progress, i);\n+\tstop_progress(&progress);\n \n \treturn verify_commit_graph_error;\n }\n-- \n2.19.0.rc1.350.ge57e33dbd1\n\n"},{"id":"357344","messageId":"20180904202729.13900-2-avarab@gmail.com","threadId":"49275","inReplyTo":"20180904202729.13900-1-avarab@gmail.com","subject":"[PATCH 1/2] commit-graph write: add progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-09-04T20:27:28Z","receivedAt":"2018-09-04T20:27:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Before this change the \"commit-graph write\" command didn't report any\nprogress. On my machine this command takes more than 10 seconds to\nwrite the graph for linux.git, and around 1m30s on the\n2015-04-03-1M-git.git[1] test repository, which is a test case for\nlarger monorepos.\n\nFurthermore, since the gc.writeCommitGraph setting was added in\nd5d5d7b641 (\"gc: automatically write commit-graph files\", 2018-06-27),\nthere was no indication at all from a \"git gc\" run that anything was\ndifferent. This why one of the progress bars being added here uses\nstart_progress() instead of start_delayed_progress(), so that it's\nguaranteed to be seen. E.g. on my tiny 867 commit dotfiles.git\nrepository:\n\n $ git -c gc.writeCommitGraph=true gc\n Enumerating objects: 2821, done.\n [...]\n Total 2821 (delta 1670), reused 2821 (delta 1670)\n Computing commit graph generation numbers: 100% (867/867), done.\n\nOn larger repositories, such as linux.git the delayed progress bar(s)\nwill kick in, and we'll show what's going on instead of, as was\npreviously happening, printing nothing while we write the graph:\n\n $ git -c gc.writeCommitGraph=true gc\n Annotating commits in commit graph: 1565573, done.\n Computing commit graph generation numbers: 100% (782484/782484), done.\n\nNote that here we don't show \"Finding commits for commit graph\", this\nis because under \"git gc\" we seed the search with the commit\nreferences in the repository, and that set is too small to show any\nprogress, but would e.g. on a smaller repo such as git.git with\n--stdin-commits:\n\n $ git rev-list --all | git -c gc.writeCommitGraph=true write --stdin-commits\n Finding commits for commit graph: 100% (162576/162576), done.\n Computing commit graph generation numbers: 100% (162576/162576), done.\n\nWith --stdin-packs we don't show any estimation of how much is left to\ndo. This is because we might be processing more than one pack. We\ncould be less lazy here and show progress, either detect by detecting\nthat we're only processing one pack, or by first looping over the\npacks to discover how many commits they have. I don't see the point in\ndoing that work. So instead we get (on 2015-04-03-1M-git.git):\n\n $ echo pack-.idx | git -c gc.writeCommitGraph=true --exec-path=$PWD commit-graph write --stdin-packs\n Finding commits for commit graph: 13064614, done.\n Annotating commits in commit graph: 3001341, done.\n Computing commit graph generation numbers: 100% (1000447/1000447), done.\n\n1. https://github.com/avar/2015-04-03-1M-git\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 38 +++++++++++++++++++++++++++++++++++++-\n 1 file changed, 37 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8a1bec7b8a..74889dc90a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -13,6 +13,7 @@\n #include \"commit-graph.h\"\n #include \"object-store.h\"\n #include \"alloc.h\"\n+#include \"progress.h\"\n \n #define GRAPH_SIGNATURE 0x43475048 /* \"CGPH\" */\n #define GRAPH_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n@@ -548,6 +549,8 @@ struct packed_oid_list {\n \tstruct object_id *list;\n \tint nr;\n \tint alloc;\n+\tstruct progress *progress;\n+\tint progress_done;\n };\n \n static int add_packed_commits(const struct object_id *oid,\n@@ -560,6 +563,9 @@ static int add_packed_commits(const struct object_id *oid,\n \toff_t offset = nth_packed_object_offset(pack, pos);\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \n+\tif (list->progress)\n+\t\tdisplay_progress(list->progress, ++list->progress_done);\n+\n \toi.typep = &type;\n \tif (packed_object_info(the_repository, pack, offset, &oi) < 0)\n \t\tdie(_(\"unable to get type of object %s\"), oid_to_hex(oid));\n@@ -591,8 +597,13 @@ static void close_reachable(struct packed_oid_list *oids)\n {\n \tint i;\n \tstruct commit *commit;\n+\tstruct progress *progress = NULL;\n+\tint j = 0;\n \n+\tprogress = start_delayed_progress(\n+\t\t_(\"Annotating commits in commit graph\"), 0);\n \tfor (i = 0; i < oids->nr; i++) {\n+\t\tdisplay_progress(progress, ++j);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n \t\tif (commit)\n \t\t\tcommit->object.flags |= UNINTERESTING;\n@@ -604,6 +615,7 @@ static void close_reachable(struct packed_oid_list *oids)\n \t * closure.\n \t */\n \tfor (i = 0; i < oids->nr; i++) {\n+\t\tdisplay_progress(progress, ++j);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n \n \t\tif (commit && !parse_commit(commit))\n@@ -611,19 +623,25 @@ static void close_reachable(struct packed_oid_list *oids)\n \t}\n \n \tfor (i = 0; i < oids->nr; i++) {\n+\t\tdisplay_progress(progress, ++j);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n \n \t\tif (commit)\n \t\t\tcommit->object.flags &= ~UNINTERESTING;\n \t}\n+\tstop_progress(&progress);\n }\n \n static void compute_generation_numbers(struct packed_commit_list* commits)\n {\n \tint i;\n \tstruct commit_list *list = NULL;\n+\tstruct progress *progress = NULL;\n \n+\tprogress = start_progress(\n+\t\t_(\"Computing commit graph generation numbers\"), commits->nr);\n \tfor (i = 0; i < commits->nr; i++) {\n+\t\tdisplay_progress(progress, i);\n \t\tif (commits->list[i]->generation != GENERATION_NUMBER_INFINITY &&\n \t\t commits->list[i]->generation != GENERATION_NUMBER_ZERO)\n \t\t\tcontinue;\n@@ -655,6 +673,8 @@ static void compute_generation_numbers(struct packed_commit_list* commits)\n \t\t\t}\n \t\t}\n \t}\n+\tdisplay_progress(progress, i);\n+\tstop_progress(&progress);\n }\n \n static int add_ref_to_list(const char *refname,\n@@ -692,9 +712,12 @@ void write_commit_graph(const char *obj_dir,\n \tint num_chunks;\n \tint num_extra_edges;\n \tstruct commit_list *parent;\n+\tstruct progress *progress = NULL;\n \n \toids.nr = 0;\n \toids.alloc = approximate_object_count() / 4;\n+\toids.progress = NULL;\n+\toids.progress_done = 0;\n \n \tif (append) {\n \t\tprepare_commit_graph_one(the_repository, obj_dir);\n@@ -721,6 +744,9 @@ void write_commit_graph(const char *obj_dir,\n \t\tint dirlen;\n \t\tstrbuf_addf(&packname, \"%s/pack/\", obj_dir);\n \t\tdirlen = packname.len;\n+\t\toids.progress = start_delayed_progress(\n+\t\t\t_(\"Finding commits for commit graph\"), 0);\n+\t\toids.progress_done = 0;\n \t\tfor (i = 0; i < pack_indexes->nr; i++) {\n \t\t\tstruct packed_git *p;\n \t\t\tstrbuf_setlen(&packname, dirlen);\n@@ -733,15 +759,19 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tfor_each_object_in_pack(p, add_packed_commits, &oids, 0);\n \t\t\tclose_pack(p);\n \t\t}\n+\t\tstop_progress(&oids.progress);\n \t\tstrbuf_release(&packname);\n \t}\n \n \tif (commit_hex) {\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Finding commits for commit graph\"), commit_hex->nr);\n \t\tfor (i = 0; i < commit_hex->nr; i++) {\n \t\t\tconst char *end;\n \t\t\tstruct object_id oid;\n \t\t\tstruct commit *result;\n \n+\t\t\tdisplay_progress(progress, i);\n \t\t\tif (commit_hex->items[i].string &&\n \t\t\t parse_oid_hex(commit_hex->items[i].string, &oid, &end))\n \t\t\t\tcontinue;\n@@ -754,10 +784,16 @@ void write_commit_graph(const char *obj_dir,\n \t\t\t\toids.nr++;\n \t\t\t}\n \t\t}\n+\t\tdisplay_progress(progress, i);\n+\t\tstop_progress(&progress);\n \t}\n \n-\tif (!pack_indexes && !commit_hex)\n+\tif (!pack_indexes && !commit_hex) {\n+\t\toids.progress = start_delayed_progress(\n+\t\t\t_(\"Finding commits for commit graph\"), 0);\n \t\tfor_each_packed_object(add_packed_commits, &oids, 0);\n+\t\tstop_progress(&oids.progress);\n+\t}\n \n \tclose_reachable(&oids);\n \n-- \n2.19.0.rc1.350.ge57e33dbd1\n\n"},{"id":"357351","messageId":"CAPig+cR9Lcj3JaFdMU9EyjPStHRG1M4SQcHaR8BwXKsDAzG4kg@mail.gmail.com","threadId":"49275","inReplyTo":"20180904202729.13900-2-avarab@gmail.com","subject":"Re: [PATCH 1/2] commit-graph write: add progress output","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-09-04T21:16:30Z","receivedAt":"2018-09-04T21:16:43Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Sep 4, 2018 at 4:27 PM Ævar Arnfjörð Bjarmason wrote:\n> With --stdin-packs we don't show any estimation of how much is left to\n> do. This is because we might be processing more than one pack. We\n> could be less lazy here and show progress, either detect by detecting\n\ns/detect//\n\n> that we're only processing one pack, or by first looping over the\n> packs to discover how many commits they have. I don't see the point in\n> doing that work. So instead we get (on 2015-04-03-1M-git.git):\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n"},{"id":"357383","messageId":"xmqqin3k7w8b.fsf@gitster-ct.c.googlers.com","threadId":"49275","inReplyTo":"20180904202729.13900-2-avarab@gmail.com","subject":"Re: [PATCH 1/2] commit-graph write: add progress output","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-09-04T22:07:16Z","receivedAt":"2018-09-04T22:07:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> Before this change the \"commit-graph write\" command didn't report any\n\nPlease describe the pre-patch state in present tense without \"Before\nthis change\".\n\n> progress. On my machine this command takes more than 10 seconds to\n> write the graph for linux.git, and around 1m30s on the\n> 2015-04-03-1M-git.git[1] test repository, which is a test case for\n> larger monorepos.\n>\n> Furthermore, since the gc.writeCommitGraph setting was added in\n> d5d5d7b641 (\"gc: automatically write commit-graph files\", 2018-06-27),\n> there was no indication at all from a \"git gc\" run that anything was\n> different. This why one of the progress bars being added here uses\n\n\"This is why\", I guess.\n\n> start_progress() instead of start_delayed_progress(), so that it's\n> guaranteed to be seen. E.g. on my tiny 867 commit dotfiles.git\n> repository:\n>\n> $ git -c gc.writeCommitGraph=true gc\n> Enumerating objects: 2821, done.\n> [...]\n> Total 2821 (delta 1670), reused 2821 (delta 1670)\n> Computing commit graph generation numbers: 100% (867/867), done.\n>\n> On larger repositories, such as linux.git the delayed progress bar(s)\n\n\"such as linux.git, the delayed ...\"\n\n> With --stdin-packs we don't show any estimation of how much is left to\n> do. This is because we might be processing more than one pack. We\n> could be less lazy here and show progress, either detect by detecting\n> that we're only processing one pack, or by first looping over the\n> packs to discover how many commits they have. I don't see the point in\n\nI do not know if there is no point, but if we were to do it, I think\nslurping the list of packs and computing the number of objects is\nnot all that bad.\n\n> static void compute_generation_numbers(struct packed_commit_list* commits)\n> {\n> \tint i;\n> \tstruct commit_list *list = NULL;\n> +\tstruct progress *progress = NULL;\n> \n> +\tprogress = start_progress(\n> +\t\t_(\"Computing commit graph generation numbers\"), commits->nr);\n> \tfor (i = 0; i < commits->nr; i++) {\n> +\t\tdisplay_progress(progress, i);\n> \t\tif (commits->list[i]->generation != GENERATION_NUMBER_INFINITY &&\n> \t\t commits->list[i]->generation != GENERATION_NUMBER_ZERO)\n> \t\t\tcontinue;\n\nI am wondering if the progress call should be moved after this\nconditional continue; would we want to count the entry whose\ngeneration is already known here? Of course, as we give commits->nr\nas the 100% ceiling, we cannot avoid doing so, but it somehow smells\nwrong.\n"},{"id":"357384","messageId":"xmqqefe87w3q.fsf@gitster-ct.c.googlers.com","threadId":"49275","inReplyTo":"20180904202729.13900-3-avarab@gmail.com","subject":"Re: [PATCH 2/2] commit-graph verify: add progress output","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-09-04T22:10:01Z","receivedAt":"2018-09-04T22:10:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> For the reasons explained in the \"commit-graph write: add progress\n> output\" commit leading up to this one, emit progress on \"commit-graph\n> verify\". Since e0fd51e1d7 (\"fsck: verify commit-graph\", 2018-06-27)\n> \"git fsck\" has called this command if core.commitGraph=true, but\n> there's been no progress output to indicate that anything was\n> different. Now there is (on my tiny dotfiles.git repository):\n>\n> $ git -c core.commitGraph=true -C ~/ fsck\n> Checking object directories: 100% (256/256), done.\n> Checking objects: 100% (2821/2821), done.\n> dangling blob 5b8bbdb9b788ed90459f505b0934619c17cc605b\n> Verifying commits in commit graph: 100% (867/867), done.\n>\n> And on a larger repository, such as the 2015-04-03-1M-git.git test\n> repository:\n>\n> $ time git -c core.commitGraph=true -C ~/g/2015-04-03-1M-git/ commit-graph verify\n> Verifying commits in commit graph: 100% (1000447/1000447), done.\n> real 0m7.813s\n> [...]\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> commit-graph.c | 6 ++++++\n> 1 file changed, 6 insertions(+)\n\nYup. The verification side knows the total number of things, so it\nis much easier to give the percentage progress with a very simple\naddition like this, which is very nice.\n\n\n\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 74889dc90a..1a02fe019a 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -914,6 +914,7 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n> \tint generation_zero = 0;\n> \tstruct hashfile *f;\n> \tint devnull;\n> +\tstruct progress *progress = NULL;\n> \n> \tif (!g) {\n> \t\tgraph_report(\"no commit-graph file loaded\");\n> @@ -981,11 +982,14 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n> \tif (verify_commit_graph_error & ~VERIFY_COMMIT_GRAPH_ERROR_HASH)\n> \t\treturn verify_commit_graph_error;\n> \n> +\tprogress = start_progress(\"Verifying commits in commit graph\",\n> +\t\t\t\t g->num_commits);\n> \tfor (i = 0; i < g->num_commits; i++) {\n> \t\tstruct commit *graph_commit, *odb_commit;\n> \t\tstruct commit_list *graph_parents, *odb_parents;\n> \t\tuint32_t max_generation = 0;\n> \n> +\t\tdisplay_progress(progress, i);\n> \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n> \n> \t\tgraph_commit = lookup_commit(r, &cur_oid);\n> @@ -1062,6 +1066,8 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n> \t\t\t\t graph_commit->date,\n> \t\t\t\t odb_commit->date);\n> \t}\n> +\tdisplay_progress(progress, i);\n> +\tstop_progress(&progress);\n> \n> \treturn verify_commit_graph_error;\n> }\n"},{"id":"357414","messageId":"c6960252-c095-fb2b-e0bc-b1e6bb261614@gmail.com","threadId":"49275","inReplyTo":"xmqqin3k7w8b.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 1/2] commit-graph write: add progress output","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-09-05T11:58:12Z","receivedAt":"2018-09-05T11:58:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/4/2018 6:07 PM, Junio C Hamano wrote:\n> Ævar Arnfjörð Bjarmason writes:\n>\n>> With --stdin-packs we don't show any estimation of how much is left to\n>> do. This is because we might be processing more than one pack. We\n>> could be less lazy here and show progress, either detect by detecting\n>> that we're only processing one pack, or by first looping over the\n>> packs to discover how many commits they have. I don't see the point in\n> I do not know if there is no point, but if we were to do it, I think\n> slurping the list of packs and computing the number of objects is\n> not all that bad.\n\nIf you want to do that, I have nothing against it. However, I don't \nexpect users to use that option directly. That option is used by VFS for \nGit to compute the commit-graph in the background after receiving a pack \nof commits and trees, but not by 'git gc' which I expect is how most \nusers will compute commit-graphs.\n\n>> static void compute_generation_numbers(struct packed_commit_list* commits)\n>> {\n>> \tint i;\n>> \tstruct commit_list *list = NULL;\n>> +\tstruct progress *progress = NULL;\n>> \n>> +\tprogress = start_progress(\n>> +\t\t_(\"Computing commit graph generation numbers\"), commits->nr);\n>> \tfor (i = 0; i < commits->nr; i++) {\n>> +\t\tdisplay_progress(progress, i);\n>> \t\tif (commits->list[i]->generation != GENERATION_NUMBER_INFINITY &&\n>> \t\t commits->list[i]->generation != GENERATION_NUMBER_ZERO)\n>> \t\t\tcontinue;\n> I am wondering if the progress call should be moved after this\n> conditional continue; would we want to count the entry whose\n> generation is already known here? Of course, as we give commits->nr\n> as the 100% ceiling, we cannot avoid doing so, but it somehow smells\n> wrong.\n\nIf we wanted to be completely right, we would count the commits in the \nlist that do not have a generation number and report that as the 100% \nceiling.\n\nSomething like the diff below would work. I tested it in Linux by first \ndeleting my commit-graph and running the following:\n\nstolee@stolee-linux:~/linux$ rm .git/objects/info/commit-graph\nstolee@stolee-linux:~/linux$ git rev-parse v4.6 | ~/git/git commit-graph \nwrite --stdin-commits\nAnnotating commits in commit graph: 1180333, done.\nComputing commit graph generation numbers: 100% (590166/590166), done.\nstolee@stolee-linux:~/linux$ ~/git/git commit-graph write --reachable\nAnnotating commits in commit graph: 1564087, done.\nComputing commit graph generation numbers: 100% (191590/191590), done.\n\n-->8--\n\nFrom: Derrick Stolee \nDate: Wed, 5 Sep 2018 11:55:42 +0000\nSubject: [PATCH] fixup! commit-graph write: add progress output\n\nSigned-off-by: Derrick Stolee \n---\n  commit-graph.c | 15 +++++++++++----\n  1 file changed, 11 insertions(+), 4 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 1a02fe019a..b933bc9f00 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -634,14 +634,20 @@ static void close_reachable(struct packed_oid_list \n*oids)\n\n  static void compute_generation_numbers(struct packed_commit_list* commits)\n  {\n-       int i;\n+       int i, count_uncomputed = 0;\n         struct commit_list *list = NULL;\n         struct progress *progress = NULL;\n\n+       for (i = 0; i < commits->nr; i++)\n+               if (commits->list[i]->generation == \nGENERATION_NUMBER_INFINITY ||\n+                   commits->list[i]->generation == GENERATION_NUMBER_ZERO)\n+                       count_uncomputed++;\n+\n         progress = start_progress(\n-               _(\"Computing commit graph generation numbers\"), \ncommits->nr);\n+               _(\"Computing commit graph generation numbers\"), \ncount_uncomputed);\n+       count_uncomputed = 0;\n+\n         for (i = 0; i < commits->nr; i++) {\n-               display_progress(progress, i);\n                 if (commits->list[i]->generation != \nGENERATION_NUMBER_INFINITY &&\n                     commits->list[i]->generation != GENERATION_NUMBER_ZERO)\n                         continue;\n@@ -670,10 +676,11 @@ static void compute_generation_numbers(struct \npacked_commit_list* commits)\n\n                                 if (current->generation > \nGENERATION_NUMBER_MAX)\n                                         current->generation = \nGENERATION_NUMBER_MAX;\n+\n+                               display_progress(progress, \n++count_uncomputed);\n                         }\n                 }\n         }\n-       display_progress(progress, i);\n         stop_progress(&progress);\n  }\n\n--\n2.19.0.rc2\n\n"},{"id":"357415","messageId":"bd12692c-e169-31aa-03c8-4d229e85c925@gmail.com","threadId":"49275","inReplyTo":"20180904202729.13900-2-avarab@gmail.com","subject":"Re: [PATCH 1/2] commit-graph write: add progress output","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-09-05T12:06:21Z","receivedAt":"2018-09-05T12:06:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/4/2018 4:27 PM, Ævar Arnfjörð Bjarmason wrote:\n> @@ -591,8 +597,13 @@ static void close_reachable(struct packed_oid_list *oids)\n> {\n> \tint i;\n> \tstruct commit *commit;\n> +\tstruct progress *progress = NULL;\n> +\tint j = 0;\n\nThe change below over-counts the number of commits we are processing (by \nat least double, possibly triple).\n\n\n> +\tprogress = start_delayed_progress(\n> +\t\t_(\"Annotating commits in commit graph\"), 0);\n> \tfor (i = 0; i < oids->nr; i++) {\n> +\t\tdisplay_progress(progress, ++j);\n> \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n> \t\tif (commit)\n> \t\t\tcommit->object.flags |= UNINTERESTING;\nThis count is the number of oids given to the method. For 'git \ncommit-graph write --reachable', this will be the number of refs.\n> @@ -604,6 +615,7 @@ static void close_reachable(struct packed_oid_list *oids)\n> \t * closure.\n> \t */\n> \tfor (i = 0; i < oids->nr; i++) {\n> +\t\tdisplay_progress(progress, ++j);\n> \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n> \n> \t\tif (commit && !parse_commit(commit))\n\nThis is the important count, since we will be parsing commits and adding \ntheir parents to the list. The bulk of the work happens here.\n\n> @@ -611,19 +623,25 @@ static void close_reachable(struct packed_oid_list *oids)\n> \t}\n> \n> \tfor (i = 0; i < oids->nr; i++) {\n> +\t\tdisplay_progress(progress, ++j);\n> \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\nThis iterates through the commits a second time and removes the \nUNINTERESTING flag.\n> \n> \t\tif (commit)\n> \t\t\tcommit->object.flags &= ~UNINTERESTING;\n> \t}\n> +\tstop_progress(&progress);\n> }\n\nI think it is good to have the progress start before the first loop and \nend after the third loop, but the middle loop has the important count.\n\nI tried deleting the first and third display_progress() methods and \nre-ran the process on the Linux repo and did not notice a delay at the \n0% and 100% progress spots. The count matches the number of commits.\n\nThanks,\n\n-Stolee\n\n"},{"id":"357416","messageId":"5cf4d777-2536-cd6d-cc61-4671757306c1@gmail.com","threadId":"49275","inReplyTo":"20180904202729.13900-1-avarab@gmail.com","subject":"Re: [PATCH 0/2] commit-graph: add progress output","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-09-05T12:07:47Z","receivedAt":"2018-09-05T12:07:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/4/2018 4:27 PM, Ævar Arnfjörð Bjarmason wrote:\n> This series adds progress output to the commit-graph command, so that\n> when it's called by \"git gc\" or \"git fsck\" we can see what's going on\n> with it.\n>\n> Ævar Arnfjörð Bjarmason (2):\n> commit-graph write: add progress output\n> commit-graph verify: add progress output\n>\n> commit-graph.c | 44 +++++++++++++++++++++++++++++++++++++++++++-\n> 1 file changed, 43 insertions(+), 1 deletion(-)\n\nThanks for writing this, Ævar. I appreciate that you took the time to \nfill in an important UX gap.\n\nI had a couple of comments, but generally this is a good change.\n\n-Stolee\n\n"},{"id":"357417","messageId":"875zzkrvu2.fsf@evledraar.gmail.com","threadId":"49275","inReplyTo":"c6960252-c095-fb2b-e0bc-b1e6bb261614@gmail.com","subject":"Re: [PATCH 1/2] commit-graph write: add progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-09-05T12:07:49Z","receivedAt":"2018-09-05T12:07:54Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Sep 05 2018, Derrick Stolee wrote:\n\n> On 9/4/2018 6:07 PM, Junio C Hamano wrote:\n>> Ævar Arnfjörð Bjarmason writes:\n>>\n>>> With --stdin-packs we don't show any estimation of how much is left to\n>>> do. This is because we might be processing more than one pack. We\n>>> could be less lazy here and show progress, either detect by detecting\n>>> that we're only processing one pack, or by first looping over the\n>>> packs to discover how many commits they have. I don't see the point in\n>> I do not know if there is no point, but if we were to do it, I think\n>> slurping the list of packs and computing the number of objects is\n>> not all that bad.\n>\n> If you want to do that, I have nothing against it. However, I don't\n> expect users to use that option directly. That option is used by VFS\n> for Git to compute the commit-graph in the background after receiving\n> a pack of commits and trees, but not by 'git gc' which I expect is how\n> most users will compute commit-graphs.\n\nYeah, I suspected only one guy at Microsoft would potentially benefit\nfrom this, but added it just so we'd have progress regardless of entry\npoint :)\n\n>>> static void compute_generation_numbers(struct packed_commit_list* commits)\n>>> {\n>>> \tint i;\n>>> \tstruct commit_list *list = NULL;\n>>> +\tstruct progress *progress = NULL;\n>>> +\tprogress = start_progress(\n>>> +\t\t_(\"Computing commit graph generation numbers\"), commits->nr);\n>>> \tfor (i = 0; i < commits->nr; i++) {\n>>> +\t\tdisplay_progress(progress, i);\n>>> \t\tif (commits->list[i]->generation != GENERATION_NUMBER_INFINITY &&\n>>> \t\t commits->list[i]->generation != GENERATION_NUMBER_ZERO)\n>>> \t\t\tcontinue;\n>> I am wondering if the progress call should be moved after this\n>> conditional continue; would we want to count the entry whose\n>> generation is already known here? Of course, as we give commits->nr\n>> as the 100% ceiling, we cannot avoid doing so, but it somehow smells\n>> wrong.\n>\n> If we wanted to be completely right, we would count the commits in the\n> list that do not have a generation number and report that as the 100%\n> ceiling.\n>\n> Something like the diff below would work. I tested it in Linux by\n> first deleting my commit-graph and running the following:\n>\n> stolee@stolee-linux:~/linux$ rm .git/objects/info/commit-graph\n> stolee@stolee-linux:~/linux$ git rev-parse v4.6 | ~/git/git\n> commit-graph write --stdin-commits\n> Annotating commits in commit graph: 1180333, done.\n> Computing commit graph generation numbers: 100% (590166/590166), done.\n> stolee@stolee-linux:~/linux$ ~/git/git commit-graph write --reachable\n> Annotating commits in commit graph: 1564087, done.\n> Computing commit graph generation numbers: 100% (191590/191590), done.\n>\n> -->8--\n>\n> From: Derrick Stolee \n> Date: Wed, 5 Sep 2018 11:55:42 +0000\n> Subject: [PATCH] fixup! commit-graph write: add progress output\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 15 +++++++++++----\n> 1 file changed, 11 insertions(+), 4 deletions(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 1a02fe019a..b933bc9f00 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -634,14 +634,20 @@ static void close_reachable(struct\n> packed_oid_list *oids)\n>\n> static void compute_generation_numbers(struct packed_commit_list* commits)\n> {\n> - int i;\n> + int i, count_uncomputed = 0;\n> struct commit_list *list = NULL;\n> struct progress *progress = NULL;\n>\n> + for (i = 0; i < commits->nr; i++)\n> + if (commits->list[i]->generation ==\n> GENERATION_NUMBER_INFINITY ||\n> + commits->list[i]->generation == GENERATION_NUMBER_ZERO)\n> + count_uncomputed++;\n> +\n> progress = start_progress(\n> - _(\"Computing commit graph generation numbers\"),\n> commits->nr);\n> + _(\"Computing commit graph generation numbers\"),\n> count_uncomputed);\n> + count_uncomputed = 0;\n> +\n> for (i = 0; i < commits->nr; i++) {\n> - display_progress(progress, i);\n> if (commits->list[i]->generation !=\n> GENERATION_NUMBER_INFINITY &&\n> commits->list[i]->generation != GENERATION_NUMBER_ZERO)\n> continue;\n> @@ -670,10 +676,11 @@ static void compute_generation_numbers(struct\n> packed_commit_list* commits)\n>\n> if (current->generation >\n> GENERATION_NUMBER_MAX)\n> current->generation =\n> GENERATION_NUMBER_MAX;\n> +\n> + display_progress(progress,\n> ++count_uncomputed);\n> }\n> }\n> }\n> - display_progress(progress, i);\n> stop_progress(&progress);\n> }\n\nThanks! That looks good, and you obviously know this code a lot\nbetter. I'll squash this into v2 pending further feedback I'll need to\naddress.\n"},{"id":"357484","messageId":"xmqqtvn339dp.fsf@gitster-ct.c.googlers.com","threadId":"49275","inReplyTo":"c6960252-c095-fb2b-e0bc-b1e6bb261614@gmail.com","subject":"Re: [PATCH 1/2] commit-graph write: add progress output","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-09-05T21:46:42Z","receivedAt":"2018-09-05T21:46:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n>>> \tfor (i = 0; i < commits->nr; i++) {\n>>> +\t\tdisplay_progress(progress, i);\n>>> \t\tif (commits->list[i]->generation != GENERATION_NUMBER_INFINITY &&\n>>> \t\t commits->list[i]->generation != GENERATION_NUMBER_ZERO)\n>>> \t\t\tcontinue;\n>> I am wondering if the progress call should be moved after this\n>> conditional continue; would we want to count the entry whose\n>> generation is already known here? Of course, as we give commits->nr\n>> as the 100% ceiling, we cannot avoid doing so, but it somehow smells\n>> wrong.\n>\n> If we wanted to be completely right, we would count the commits in the\n> list that do not have a generation number and report that as the 100%\n> ceiling.\n\nYeah, but I realize that the definition of \"right\" really depends on\nwhat we consider a task being accomplished is in this loop. If we\ndefine the task to \"we have some number of commits that lack\ngeneration numbers and our task is to assign numbers to them.\", then\nyes counting the ones without generation number and culling the ones\nthat already have generation number is outside the work and we need\nanother loop to count them. But the position the posted patch takes\nis also a valid one: we have some commits and we are making sure\neach and every one of them has assigned a generation number.\n\nSo I do not think it is necessary to introduce another loop just for\ncounting.\n\nThanks.\n\n\n\n"},{"id":"357488","messageId":"8ea40192-ea3a-33e6-c561-ee7b4ab7f040@gmail.com","threadId":"49275","inReplyTo":"xmqqtvn339dp.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 1/2] commit-graph write: add progress output","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-09-05T22:12:53Z","receivedAt":"2018-09-05T22:12:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/5/2018 5:46 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n>>>> \tfor (i = 0; i < commits->nr; i++) {\n>>>> +\t\tdisplay_progress(progress, i);\n>>>> \t\tif (commits->list[i]->generation != GENERATION_NUMBER_INFINITY &&\n>>>> \t\t commits->list[i]->generation != GENERATION_NUMBER_ZERO)\n>>>> \t\t\tcontinue;\n>>> I am wondering if the progress call should be moved after this\n>>> conditional continue; would we want to count the entry whose\n>>> generation is already known here? Of course, as we give commits->nr\n>>> as the 100% ceiling, we cannot avoid doing so, but it somehow smells\n>>> wrong.\n>> If we wanted to be completely right, we would count the commits in the\n>> list that do not have a generation number and report that as the 100%\n>> ceiling.\n> Yeah, but I realize that the definition of \"right\" really depends on\n> what we consider a task being accomplished is in this loop. If we\n> define the task to \"we have some number of commits that lack\n> generation numbers and our task is to assign numbers to them.\", then\n> yes counting the ones without generation number and culling the ones\n> that already have generation number is outside the work and we need\n> another loop to count them. But the position the posted patch takes\n> is also a valid one: we have some commits and we are making sure\n> each and every one of them has assigned a generation number.\n>\n> So I do not think it is necessary to introduce another loop just for\n> counting.\n>\n> Thanks.\nMakes sense to me. Thanks!\n"},{"id":"357619","messageId":"87h8j1qy59.fsf@evledraar.gmail.com","threadId":"49275","inReplyTo":"20180904202729.13900-2-avarab@gmail.com","subject":"Re: [PATCH 1/2] commit-graph write: add progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-09-07T12:40:02Z","receivedAt":"2018-09-07T12:40:07Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Sep 04 2018, Ævar Arnfjörð Bjarmason wrote:\n\n> Before this change the \"commit-graph write\" command didn't report any\n> progress. On my machine this command takes more than 10 seconds to\n> write the graph for linux.git, and around 1m30s on the\n> 2015-04-03-1M-git.git[1] test repository, which is a test case for\n> larger monorepos.\n\nThere's a fun issue with this code that I'll fix, but thought was\ninformative to send a mail about.\n\nBecause the graph verification happens in the main \"git gc\" process, as\nopposed to everything else via external commands, so all this progress\noutput gets written to .git/gc.log.\n\nThen next time we do a \"gc --auto\" we vomit out a couple of KB of\nprogress bar output at the user, since spot that the gc.log isn't empty.\n"},{"id":"357622","messageId":"75c8f1f1-d836-7dda-b30b-6c2fec73b194@gmail.com","threadId":"49275","inReplyTo":"87h8j1qy59.fsf@evledraar.gmail.com","subject":"Re: [PATCH 1/2] commit-graph write: add progress output","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-09-07T13:12:12Z","receivedAt":"2018-09-07T13:12:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/7/2018 8:40 AM, Ævar Arnfjörð Bjarmason wrote:\n> On Tue, Sep 04 2018, Ævar Arnfjörð Bjarmason wrote:\n>\n>> Before this change the \"commit-graph write\" command didn't report any\n>> progress. On my machine this command takes more than 10 seconds to\n>> write the graph for linux.git, and around 1m30s on the\n>> 2015-04-03-1M-git.git[1] test repository, which is a test case for\n>> larger monorepos.\n> There's a fun issue with this code that I'll fix, but thought was\n> informative to send a mail about.\n>\n> Because the graph verification happens in the main \"git gc\" process, as\n> opposed to everything else via external commands, so all this progress\n> output gets written to .git/gc.log.\n>\n> Then next time we do a \"gc --auto\" we vomit out a couple of KB of\n> progress bar output at the user, since spot that the gc.log isn't empty.\nGood catch! (I do want to clarify that the graph _writing_ happens \nduring 'git gc' since 'git commit-graph verify' is a different thing.)\n"},{"id":"357626","messageId":"87ftylqr4z.fsf@evledraar.gmail.com","threadId":"49275","inReplyTo":"c6960252-c095-fb2b-e0bc-b1e6bb261614@gmail.com","subject":"Re: [PATCH 1/2] commit-graph write: add progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-09-07T15:11:24Z","receivedAt":"2018-09-07T15:11:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Sep 05 2018, Derrick Stolee wrote:\n\n> On 9/4/2018 6:07 PM, Junio C Hamano wrote:\n>> Ævar Arnfjörð Bjarmason writes:\n>>\n>>> With --stdin-packs we don't show any estimation of how much is left to\n>>> do. This is because we might be processing more than one pack. We\n>>> could be less lazy here and show progress, either detect by detecting\n>>> that we're only processing one pack, or by first looping over the\n>>> packs to discover how many commits they have. I don't see the point in\n>> I do not know if there is no point, but if we were to do it, I think\n>> slurping the list of packs and computing the number of objects is\n>> not all that bad.\n>\n> If you want to do that, I have nothing against it. However, I don't\n> expect users to use that option directly. That option is used by VFS\n> for Git to compute the commit-graph in the background after receiving\n> a pack of commits and trees, but not by 'git gc' which I expect is how\n> most users will compute commit-graphs.\n>\n>>> static void compute_generation_numbers(struct packed_commit_list* commits)\n>>> {\n>>> \tint i;\n>>> \tstruct commit_list *list = NULL;\n>>> +\tstruct progress *progress = NULL;\n>>> +\tprogress = start_progress(\n>>> +\t\t_(\"Computing commit graph generation numbers\"), commits->nr);\n>>> \tfor (i = 0; i < commits->nr; i++) {\n>>> +\t\tdisplay_progress(progress, i);\n>>> \t\tif (commits->list[i]->generation != GENERATION_NUMBER_INFINITY &&\n>>> \t\t commits->list[i]->generation != GENERATION_NUMBER_ZERO)\n>>> \t\t\tcontinue;\n>> I am wondering if the progress call should be moved after this\n>> conditional continue; would we want to count the entry whose\n>> generation is already known here? Of course, as we give commits->nr\n>> as the 100% ceiling, we cannot avoid doing so, but it somehow smells\n>> wrong.\n>\n> If we wanted to be completely right, we would count the commits in the\n> list that do not have a generation number and report that as the 100%\n> ceiling.\n>\n> Something like the diff below would work. I tested it in Linux by\n> first deleting my commit-graph and running the following:\n>\n> stolee@stolee-linux:~/linux$ rm .git/objects/info/commit-graph\n> stolee@stolee-linux:~/linux$ git rev-parse v4.6 | ~/git/git\n> commit-graph write --stdin-commits\n> Annotating commits in commit graph: 1180333, done.\n> Computing commit graph generation numbers: 100% (590166/590166), done.\n> stolee@stolee-linux:~/linux$ ~/git/git commit-graph write --reachable\n> Annotating commits in commit graph: 1564087, done.\n> Computing commit graph generation numbers: 100% (191590/191590), done.\n>\n> -->8--\n>\n> From: Derrick Stolee \n> Date: Wed, 5 Sep 2018 11:55:42 +0000\n> Subject: [PATCH] fixup! commit-graph write: add progress output\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 15 +++++++++++----\n> 1 file changed, 11 insertions(+), 4 deletions(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 1a02fe019a..b933bc9f00 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -634,14 +634,20 @@ static void close_reachable(struct\n> packed_oid_list *oids)\n>\n> static void compute_generation_numbers(struct packed_commit_list* commits)\n> {\n> - int i;\n> + int i, count_uncomputed = 0;\n> struct commit_list *list = NULL;\n> struct progress *progress = NULL;\n>\n> + for (i = 0; i < commits->nr; i++)\n> + if (commits->list[i]->generation ==\n> GENERATION_NUMBER_INFINITY ||\n> + commits->list[i]->generation == GENERATION_NUMBER_ZERO)\n> + count_uncomputed++;\n> +\n> progress = start_progress(\n> - _(\"Computing commit graph generation numbers\"),\n> commits->nr);\n> + _(\"Computing commit graph generation numbers\"),\n> count_uncomputed);\n> + count_uncomputed = 0;\n> +\n> for (i = 0; i < commits->nr; i++) {\n> - display_progress(progress, i);\n> if (commits->list[i]->generation !=\n> GENERATION_NUMBER_INFINITY &&\n> commits->list[i]->generation != GENERATION_NUMBER_ZERO)\n> continue;\n> @@ -670,10 +676,11 @@ static void compute_generation_numbers(struct\n> packed_commit_list* commits)\n>\n> if (current->generation >\n> GENERATION_NUMBER_MAX)\n> current->generation =\n> GENERATION_NUMBER_MAX;\n> +\n> + display_progress(progress,\n> ++count_uncomputed);\n> }\n> }\n> }\n> - display_progress(progress, i);\n> stop_progress(&progress);\n> }\n\nOne of the things I was trying to do with this series was to make sure\nthat whenever we run \"git gc\" there's always some indication that if you\nset gc.writeCommitGraph=true that it's actualy doing work.\n\nThis modifies that, which I think is actually fine, just something I\nwanted to note. I.e. if you run \"git commit-graph write\" twice in a row,\nthe second time will have no output.\n\nUnless that is, your repo is big enough that some of the delayed timers\nkick in. So e.g. on git.git we get no output the second time around, but\ndo get output the first time around, and on linux.git we always get\noutput.\n\nBut in the common case people aren't running this in a loop, and it's\nuseful to see how many new things are being added to the graph, so I\nthink this is better. Just wanted to note the behavior difference (and\nwill change the commit message).\n"},{"id":"357629","messageId":"87efe5qqks.fsf@evledraar.gmail.com","threadId":"49275","inReplyTo":"87ftylqr4z.fsf@evledraar.gmail.com","subject":"Re: [PATCH 1/2] commit-graph write: add progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-09-07T15:23:31Z","receivedAt":"2018-09-07T15:23:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Sep 07 2018, Ævar Arnfjörð Bjarmason wrote:\n\n> On Wed, Sep 05 2018, Derrick Stolee wrote:\n>\n>> On 9/4/2018 6:07 PM, Junio C Hamano wrote:\n>>> Ævar Arnfjörð Bjarmason writes:\n>>>\n>>>> With --stdin-packs we don't show any estimation of how much is left to\n>>>> do. This is because we might be processing more than one pack. We\n>>>> could be less lazy here and show progress, either detect by detecting\n>>>> that we're only processing one pack, or by first looping over the\n>>>> packs to discover how many commits they have. I don't see the point in\n>>> I do not know if there is no point, but if we were to do it, I think\n>>> slurping the list of packs and computing the number of objects is\n>>> not all that bad.\n>>\n>> If you want to do that, I have nothing against it. However, I don't\n>> expect users to use that option directly. That option is used by VFS\n>> for Git to compute the commit-graph in the background after receiving\n>> a pack of commits and trees, but not by 'git gc' which I expect is how\n>> most users will compute commit-graphs.\n>>\n>>>> static void compute_generation_numbers(struct packed_commit_list* commits)\n>>>> {\n>>>> \tint i;\n>>>> \tstruct commit_list *list = NULL;\n>>>> +\tstruct progress *progress = NULL;\n>>>> +\tprogress = start_progress(\n>>>> +\t\t_(\"Computing commit graph generation numbers\"), commits->nr);\n>>>> \tfor (i = 0; i < commits->nr; i++) {\n>>>> +\t\tdisplay_progress(progress, i);\n>>>> \t\tif (commits->list[i]->generation != GENERATION_NUMBER_INFINITY &&\n>>>> \t\t commits->list[i]->generation != GENERATION_NUMBER_ZERO)\n>>>> \t\t\tcontinue;\n>>> I am wondering if the progress call should be moved after this\n>>> conditional continue; would we want to count the entry whose\n>>> generation is already known here? Of course, as we give commits->nr\n>>> as the 100% ceiling, we cannot avoid doing so, but it somehow smells\n>>> wrong.\n>>\n>> If we wanted to be completely right, we would count the commits in the\n>> list that do not have a generation number and report that as the 100%\n>> ceiling.\n>>\n>> Something like the diff below would work. I tested it in Linux by\n>> first deleting my commit-graph and running the following:\n>>\n>> stolee@stolee-linux:~/linux$ rm .git/objects/info/commit-graph\n>> stolee@stolee-linux:~/linux$ git rev-parse v4.6 | ~/git/git\n>> commit-graph write --stdin-commits\n>> Annotating commits in commit graph: 1180333, done.\n>> Computing commit graph generation numbers: 100% (590166/590166), done.\n>> stolee@stolee-linux:~/linux$ ~/git/git commit-graph write --reachable\n>> Annotating commits in commit graph: 1564087, done.\n>> Computing commit graph generation numbers: 100% (191590/191590), done.\n>>\n>> -->8--\n>>\n>> From: Derrick Stolee \n>> Date: Wed, 5 Sep 2018 11:55:42 +0000\n>> Subject: [PATCH] fixup! commit-graph write: add progress output\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> commit-graph.c | 15 +++++++++++----\n>> 1 file changed, 11 insertions(+), 4 deletions(-)\n>>\n>> diff --git a/commit-graph.c b/commit-graph.c\n>> index 1a02fe019a..b933bc9f00 100644\n>> --- a/commit-graph.c\n>> +++ b/commit-graph.c\n>> @@ -634,14 +634,20 @@ static void close_reachable(struct\n>> packed_oid_list *oids)\n>>\n>> static void compute_generation_numbers(struct packed_commit_list* commits)\n>> {\n>> - int i;\n>> + int i, count_uncomputed = 0;\n>> struct commit_list *list = NULL;\n>> struct progress *progress = NULL;\n>>\n>> + for (i = 0; i < commits->nr; i++)\n>> + if (commits->list[i]->generation ==\n>> GENERATION_NUMBER_INFINITY ||\n>> + commits->list[i]->generation == GENERATION_NUMBER_ZERO)\n>> + count_uncomputed++;\n>> +\n>> progress = start_progress(\n>> - _(\"Computing commit graph generation numbers\"),\n>> commits->nr);\n>> + _(\"Computing commit graph generation numbers\"),\n>> count_uncomputed);\n>> + count_uncomputed = 0;\n>> +\n>> for (i = 0; i < commits->nr; i++) {\n>> - display_progress(progress, i);\n>> if (commits->list[i]->generation !=\n>> GENERATION_NUMBER_INFINITY &&\n>> commits->list[i]->generation != GENERATION_NUMBER_ZERO)\n>> continue;\n>> @@ -670,10 +676,11 @@ static void compute_generation_numbers(struct\n>> packed_commit_list* commits)\n>>\n>> if (current->generation >\n>> GENERATION_NUMBER_MAX)\n>> current->generation =\n>> GENERATION_NUMBER_MAX;\n>> +\n>> + display_progress(progress,\n>> ++count_uncomputed);\n>> }\n>> }\n>> }\n>> - display_progress(progress, i);\n>> stop_progress(&progress);\n>> }\n>\n> One of the things I was trying to do with this series was to make sure\n> that whenever we run \"git gc\" there's always some indication that if you\n> set gc.writeCommitGraph=true that it's actualy doing work.\n>\n> This modifies that, which I think is actually fine, just something I\n> wanted to note. I.e. if you run \"git commit-graph write\" twice in a row,\n> the second time will have no output.\n>\n> Unless that is, your repo is big enough that some of the delayed timers\n> kick in. So e.g. on git.git we get no output the second time around, but\n> do get output the first time around, and on linux.git we always get\n> output.\n>\n> But in the common case people aren't running this in a loop, and it's\n> useful to see how many new things are being added to the graph, so I\n> think this is better. Just wanted to note the behavior difference (and\n> will change the commit message).\n\nHrm, no. I spoke too soon because I was conflating \"commit-graph write\"\nv.s. \"gc\". For \"gc\" we're now with this change just e.g. spending 6\nseconds on 2015-04-03-1M-git displaying nothing, because we're looping\nthrough the commits and finding that we have no new work.\n\nSo I'm on the fence about this, but leaning towards just taking my\ninitial approch. I.e. it sucks if you're e.g. testing different \"git gc\"\noptions that we're churning in the background doing nothing, just\nbecause we're trying to report how many *new* things we added to the\ngraph.\n\nAfter all, the main point IMNSHO is not to show some diagnostic output\nof exactly how much work we're doing, that I have 200 new commits with\ngeneration numbers or whatever is just useless trivia, but rather to not\nleave the user thinking the command is hanging.\n\nSo I think I'll just do what I was doing to begin with and change the\nmessage to \"Refreshing commit graph generation numbers\" or something to\nindicate that it's a find/verify/compute operation, not just a compute\noperation.\n"},{"id":"357634","messageId":"20180907171548.GA11708@sigill.intra.peff.net","threadId":"49275","inReplyTo":"87efe5qqks.fsf@evledraar.gmail.com","subject":"Re: [PATCH 1/2] commit-graph write: add progress output","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-09-07T17:15:49Z","receivedAt":"2018-09-07T17:15:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Sep 07, 2018 at 05:23:31PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> Hrm, no. I spoke too soon because I was conflating \"commit-graph write\"\n> v.s. \"gc\". For \"gc\" we're now with this change just e.g. spending 6\n> seconds on 2015-04-03-1M-git displaying nothing, because we're looping\n> through the commits and finding that we have no new work.\n> \n> So I'm on the fence about this, but leaning towards just taking my\n> initial approch. I.e. it sucks if you're e.g. testing different \"git gc\"\n> options that we're churning in the background doing nothing, just\n> because we're trying to report how many *new* things we added to the\n> graph.\n> \n> After all, the main point IMNSHO is not to show some diagnostic output\n> of exactly how much work we're doing, that I have 200 new commits with\n> generation numbers or whatever is just useless trivia, but rather to not\n> leave the user thinking the command is hanging.\n\nI think there's some precedent for your view of things, too. For\nexample, \"writing objects\" counts _all_ of the objects, even though many\nof them are just copying bytes straight from disk, and some are actually\ngenerating a delta and/or zlib-deflating content.\n\nSo it's not the most precise measurement we could give, but it shows\nthere's activity, and the \"average\" movement over many objects tends to\nbe reasonably smooth.\n\n> So I think I'll just do what I was doing to begin with and change the\n> message to \"Refreshing commit graph generation numbers\" or something to\n> indicate that it's a find/verify/compute operation, not just a compute\n> operation.\n\nSo basically yes, I agree with this. :)\n\n-Peff\n"},{"id":"357637","messageId":"27b2e24c-4f96-b8d9-3c03-68195d447394@gmail.com","threadId":"49275","inReplyTo":"20180907171548.GA11708@sigill.intra.peff.net","subject":"Re: [PATCH 1/2] commit-graph write: add progress output","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-09-07T17:25:02Z","receivedAt":"2018-09-07T17:25:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/7/2018 1:15 PM, Jeff King wrote:\n> On Fri, Sep 07, 2018 at 05:23:31PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n>> Hrm, no. I spoke too soon because I was conflating \"commit-graph write\"\n>> v.s. \"gc\". For \"gc\" we're now with this change just e.g. spending 6\n>> seconds on 2015-04-03-1M-git displaying nothing, because we're looping\n>> through the commits and finding that we have no new work.\n>>\n>> So I'm on the fence about this, but leaning towards just taking my\n>> initial approch. I.e. it sucks if you're e.g. testing different \"git gc\"\n>> options that we're churning in the background doing nothing, just\n>> because we're trying to report how many *new* things we added to the\n>> graph.\n>>\n>> After all, the main point IMNSHO is not to show some diagnostic output\n>> of exactly how much work we're doing, that I have 200 new commits with\n>> generation numbers or whatever is just useless trivia, but rather to not\n>> leave the user thinking the command is hanging.\n> I think there's some precedent for your view of things, too. For\n> example, \"writing objects\" counts _all_ of the objects, even though many\n> of them are just copying bytes straight from disk, and some are actually\n> generating a delta and/or zlib-deflating content.\n>\n> So it's not the most precise measurement we could give, but it shows\n> there's activity, and the \"average\" movement over many objects tends to\n> be reasonably smooth.\n>\n>> So I think I'll just do what I was doing to begin with and change the\n>> message to \"Refreshing commit graph generation numbers\" or something to\n>> indicate that it's a find/verify/compute operation, not just a compute\n>> operation.\n> So basically yes, I agree with this. :)\n\nSame here. Thanks for the discussion.\n\n-Stolee\n\n"},{"id":"357644","messageId":"20180907182954.2413-1-avarab@gmail.com","threadId":"49275","inReplyTo":"20180904202729.13900-1-avarab@gmail.com","subject":"[PATCH v2 0/2] commit-graph: add progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-09-07T18:29:52Z","receivedAt":"2018-09-07T18:30:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Based on feedback on v1, and the \"this is yelling at my users through\ngc.log\" bug I found. Range-diff with v1:\n\n1: e0a09ad641 ! 1: b2dcfa0f55 commit-graph write: add progress output\n @@ -5,8 +5,8 @@\n Before this change the \"commit-graph write\" command didn't report any\n progress. On my machine this command takes more than 10 seconds to\n write the graph for linux.git, and around 1m30s on the\n - 2015-04-03-1M-git.git[1] test repository, which is a test case for\n - larger monorepos.\n + 2015-04-03-1M-git.git[1] test repository (a test case for a large\n + monorepository).\n \n Furthermore, since the gc.writeCommitGraph setting was added in\n d5d5d7b641 (\"gc: automatically write commit-graph files\", 2018-06-27),\n @@ -19,7 +19,6 @@\n $ git -c gc.writeCommitGraph=true gc\n Enumerating objects: 2821, done.\n [...]\n - Total 2821 (delta 1670), reused 2821 (delta 1670)\n Computing commit graph generation numbers: 100% (867/867), done.\n \n On larger repositories, such as linux.git the delayed progress bar(s)\n @@ -27,6 +26,7 @@\n previously happening, printing nothing while we write the graph:\n \n $ git -c gc.writeCommitGraph=true gc\n + [...]\n Annotating commits in commit graph: 1565573, done.\n Computing commit graph generation numbers: 100% (782484/782484), done.\n \n @@ -42,20 +42,90 @@\n \n With --stdin-packs we don't show any estimation of how much is left to\n do. This is because we might be processing more than one pack. We\n - could be less lazy here and show progress, either detect by detecting\n - that we're only processing one pack, or by first looping over the\n - packs to discover how many commits they have. I don't see the point in\n - doing that work. So instead we get (on 2015-04-03-1M-git.git):\n + could be less lazy here and show progress, either by detecting that\n + we're only processing one pack, or by first looping over the packs to\n + discover how many commits they have. I don't see the point in doing\n + that work. So instead we get (on 2015-04-03-1M-git.git):\n \n $ echo pack-.idx | git -c gc.writeCommitGraph=true --exec-path=$PWD commit-graph write --stdin-packs\n Finding commits for commit graph: 13064614, done.\n Annotating commits in commit graph: 3001341, done.\n Computing commit graph generation numbers: 100% (1000447/1000447), done.\n \n + No GC mode uses --stdin-packs. It's what they use at Microsoft to\n + manually compute the generation numbers for their collection of large\n + packs which are never coalesced.\n +\n + The reason we need a \"report_progress\" variable passed down from \"git\n + gc\" is so that we don't report this output when we're running in the\n + process \"git gc --auto\" detaches from the terminal.\n +\n + Since we write the commit graph from the \"git gc\" process itself (as\n + opposed to what we do with say the \"git repack\" phase), we'd end up\n + writing the output to .git/gc.log and reporting it to the user next\n + time as part of the \"The last gc run reported the following[...]\"\n + error, see 329e6e8794 (\"gc: save log from daemonized gc --auto and\n + print it next time\", 2015-09-19).\n +\n + So we must keep track of whether or not we're running in that\n + demonized mode, and if so print no progress.\n +\n + See [2] and subsequent replies for a discussion of an approach not\n + taken in compute_generation_numbers(). I.e. we're saying \"Computing\n + commit graph generation numbers\", even though on an established\n + history we're mostly skipping over all the work we did in the\n + past. This is similar to the white lie we tell in the \"Writing\n + objects\" phase (not all are objects being written).\n +\n + Always showing progress is considered more important than\n + accuracy. I.e. on a repository like 2015-04-03-1M-git.git we'd hang\n + for 6 seconds with no output on the second \"git gc\" if no changes were\n + made to any objects in the interim if we'd take the approach in [2].\n +\n 1. https://github.com/avar/2015-04-03-1M-git\n \n + 2. \n + (https://public-inbox.org/git/c6960252-c095-fb2b-e0bc-b1e6bb261614@gmail.com/)\n +\n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n + diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n + --- a/builtin/commit-graph.c\n + +++ b/builtin/commit-graph.c\n +@@\n + \t\topts.obj_dir = get_object_directory();\n + \n + \tif (opts.reachable) {\n +-\t\twrite_commit_graph_reachable(opts.obj_dir, opts.append);\n ++\t\twrite_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n + \t\treturn 0;\n + \t}\n + \n +@@\n + \twrite_commit_graph(opts.obj_dir,\n + \t\t\t pack_indexes,\n + \t\t\t commit_hex,\n +-\t\t\t opts.append);\n ++\t\t\t opts.append,\n ++\t\t\t 1);\n + \n + \tstring_list_clear(&lines, 0);\n + \treturn 0;\n +\n + diff --git a/builtin/gc.c b/builtin/gc.c\n + --- a/builtin/gc.c\n + +++ b/builtin/gc.c\n +@@\n + \t\tclean_pack_garbage();\n + \n + \tif (gc_write_commit_graph)\n +-\t\twrite_commit_graph_reachable(get_object_directory(), 0);\n ++\t\twrite_commit_graph_reachable(get_object_directory(), 0,\n ++\t\t\t\t\t !daemonized);\n + \n + \tif (auto_gc && too_many_loose_objects())\n + \t\twarning(_(\"There are too many unreachable loose objects; \"\n +\n diff --git a/commit-graph.c b/commit-graph.c\n --- a/commit-graph.c\n +++ b/commit-graph.c\n @@ -87,14 +157,20 @@\n \tif (packed_object_info(the_repository, pack, offset, &oi) < 0)\n \t\tdie(_(\"unable to get type of object %s\"), oid_to_hex(oid));\n @@\n + \t}\n + }\n + \n +-static void close_reachable(struct packed_oid_list *oids)\n ++static void close_reachable(struct packed_oid_list *oids, int report_progress)\n {\n \tint i;\n \tstruct commit *commit;\n +\tstruct progress *progress = NULL;\n +\tint j = 0;\n \n -+\tprogress = start_delayed_progress(\n -+\t\t_(\"Annotating commits in commit graph\"), 0);\n ++\tif (report_progress)\n ++\t\tprogress = start_delayed_progress(\n ++\t\t\t_(\"Annotating commits in commit graph\"), 0);\n \tfor (i = 0; i < oids->nr; i++) {\n +\t\tdisplay_progress(progress, ++j);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n @@ -121,16 +197,20 @@\n +\tstop_progress(&progress);\n }\n \n - static void compute_generation_numbers(struct packed_commit_list* commits)\n +-static void compute_generation_numbers(struct packed_commit_list* commits)\n ++static void compute_generation_numbers(struct packed_commit_list* commits, \n ++\t\t\t\t int report_progress)\n {\n \tint i;\n \tstruct commit_list *list = NULL;\n +\tstruct progress *progress = NULL;\n \n -+\tprogress = start_progress(\n -+\t\t_(\"Computing commit graph generation numbers\"), commits->nr);\n ++\tif (report_progress)\n ++\t\tprogress = start_progress(\n ++\t\t\t_(\"Computing commit graph generation numbers\"),\n ++\t\t\tcommits->nr);\n \tfor (i = 0; i < commits->nr; i++) {\n -+\t\tdisplay_progress(progress, i);\n ++\t\tdisplay_progress(progress, i + 1);\n \t\tif (commits->list[i]->generation != GENERATION_NUMBER_INFINITY &&\n \t\t commits->list[i]->generation != GENERATION_NUMBER_ZERO)\n \t\t\tcontinue;\n @@ -138,11 +218,34 @@\n \t\t\t}\n \t\t}\n \t}\n -+\tdisplay_progress(progress, i);\n +\tstop_progress(&progress);\n }\n \n static int add_ref_to_list(const char *refname,\n +@@\n + \treturn 0;\n + }\n + \n +-void write_commit_graph_reachable(const char *obj_dir, int append)\n ++void write_commit_graph_reachable(const char *obj_dir, int append,\n ++\t\t\t\t int report_progress)\n + {\n + \tstruct string_list list;\n + \n + \tstring_list_init(&list, 1);\n + \tfor_each_ref(add_ref_to_list, &list);\n +-\twrite_commit_graph(obj_dir, NULL, &list, append);\n ++\twrite_commit_graph(obj_dir, NULL, &list, append, report_progress);\n + }\n + \n + void write_commit_graph(const char *obj_dir,\n + \t\t\tstruct string_list *pack_indexes,\n + \t\t\tstruct string_list *commit_hex,\n +-\t\t\tint append)\n ++\t\t\tint append, int report_progress)\n + {\n + \tstruct packed_oid_list oids;\n + \tstruct packed_commit_list commits;\n @@\n \tint num_chunks;\n \tint num_extra_edges;\n @@ -160,9 +263,11 @@\n \t\tint dirlen;\n \t\tstrbuf_addf(&packname, \"%s/pack/\", obj_dir);\n \t\tdirlen = packname.len;\n -+\t\toids.progress = start_delayed_progress(\n -+\t\t\t_(\"Finding commits for commit graph\"), 0);\n -+\t\toids.progress_done = 0;\n ++\t\tif (report_progress) {\n ++\t\t\toids.progress = start_delayed_progress(\n ++\t\t\t\t_(\"Finding commits for commit graph\"), 0);\n ++\t\t\toids.progress_done = 0;\n ++\t\t}\n \t\tfor (i = 0; i < pack_indexes->nr; i++) {\n \t\t\tstruct packed_git *p;\n \t\t\tstrbuf_setlen(&packname, dirlen);\n @@ -175,14 +280,16 @@\n \t}\n \n \tif (commit_hex) {\n -+\t\tprogress = start_delayed_progress(\n -+\t\t\t_(\"Finding commits for commit graph\"), commit_hex->nr);\n ++\t\tif (report_progress)\n ++\t\t\tprogress = start_delayed_progress(\n ++\t\t\t\t_(\"Finding commits for commit graph\"),\n ++\t\t\t\tcommit_hex->nr);\n \t\tfor (i = 0; i < commit_hex->nr; i++) {\n \t\t\tconst char *end;\n \t\t\tstruct object_id oid;\n \t\t\tstruct commit *result;\n \n -+\t\t\tdisplay_progress(progress, i);\n ++\t\t\tdisplay_progress(progress, i + 1);\n \t\t\tif (commit_hex->items[i].string &&\n \t\t\t parse_oid_hex(commit_hex->items[i].string, &oid, &end))\n \t\t\t\tcontinue;\n @@ -190,17 +297,48 @@\n \t\t\t\toids.nr++;\n \t\t\t}\n \t\t}\n -+\t\tdisplay_progress(progress, i);\n +\t\tstop_progress(&progress);\n \t}\n \n -\tif (!pack_indexes && !commit_hex)\n +\tif (!pack_indexes && !commit_hex) {\n -+\t\toids.progress = start_delayed_progress(\n -+\t\t\t_(\"Finding commits for commit graph\"), 0);\n ++\t\tif (report_progress)\n ++\t\t\toids.progress = start_delayed_progress(\n ++\t\t\t\t_(\"Finding commits for commit graph\"), 0);\n \t\tfor_each_packed_object(add_packed_commits, &oids, 0);\n +\t\tstop_progress(&oids.progress);\n +\t}\n \n - \tclose_reachable(&oids);\n +-\tclose_reachable(&oids);\n ++\tclose_reachable(&oids, report_progress);\n + \n + \tQSORT(oids.list, oids.nr, commit_compare);\n + \n +@@\n + \tif (commits.nr >= GRAPH_PARENT_MISSING)\n + \t\tdie(_(\"too many commits to write graph\"));\n + \n +-\tcompute_generation_numbers(&commits);\n ++\tcompute_generation_numbers(&commits, report_progress);\n + \n + \tgraph_name = get_commit_graph_filename(obj_dir);\n + \tif (safe_create_leading_directories(graph_name))\n +\n + diff --git a/commit-graph.h b/commit-graph.h\n + --- a/commit-graph.h\n + +++ b/commit-graph.h\n +@@\n + \n + struct commit_graph *load_commit_graph_one(const char *graph_file);\n + \n +-void write_commit_graph_reachable(const char *obj_dir, int append);\n ++void write_commit_graph_reachable(const char *obj_dir, int append,\n ++\t\t\t\t int report_progress);\n + void write_commit_graph(const char *obj_dir,\n + \t\t\tstruct string_list *pack_indexes,\n + \t\t\tstruct string_list *commit_hex,\n +-\t\t\tint append);\n ++\t\t\tint append, int report_progress);\n + \n + int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \n2: a364297d15 ! 2: 775237cffb commit-graph verify: add progress output\n @@ -23,6 +23,10 @@\n real 0m7.813s\n [...]\n \n + Since the \"commit-graph verify\" subcommand is never called from \"git\n + gc\", we don't have to worry about passing some some \"report_progress\"\n + progress variable around for this codepath.\n +\n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n diff --git a/commit-graph.c b/commit-graph.c\n @@ -47,7 +51,7 @@\n \t\tstruct commit_list *graph_parents, *odb_parents;\n \t\tuint32_t max_generation = 0;\n \n -+\t\tdisplay_progress(progress, i);\n ++\t\tdisplay_progress(progress, i + 1);\n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n \t\tgraph_commit = lookup_commit(r, &cur_oid);\n @@ -55,7 +59,6 @@\n \t\t\t\t graph_commit->date,\n \t\t\t\t odb_commit->date);\n \t}\n -+\tdisplay_progress(progress, i);\n +\tstop_progress(&progress);\n \n \treturn verify_commit_graph_error;\n\nÆvar Arnfjörð Bjarmason (2):\n commit-graph write: add progress output\n commit-graph verify: add progress output\n\n builtin/commit-graph.c | 5 ++--\n builtin/gc.c | 3 +-\n commit-graph.c | 65 ++++++++++++++++++++++++++++++++++++------\n commit-graph.h | 5 ++--\n 4 files changed, 65 insertions(+), 13 deletions(-)\n\n-- \n2.19.0.rc1.350.ge57e33dbd1\n\n"},{"id":"357645","messageId":"20180907182954.2413-2-avarab@gmail.com","threadId":"49275","inReplyTo":"20180904202729.13900-1-avarab@gmail.com","subject":"[PATCH v2 1/2] commit-graph write: add progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-09-07T18:29:53Z","receivedAt":"2018-09-07T18:30:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Before this change the \"commit-graph write\" command didn't report any\nprogress. On my machine this command takes more than 10 seconds to\nwrite the graph for linux.git, and around 1m30s on the\n2015-04-03-1M-git.git[1] test repository (a test case for a large\nmonorepository).\n\nFurthermore, since the gc.writeCommitGraph setting was added in\nd5d5d7b641 (\"gc: automatically write commit-graph files\", 2018-06-27),\nthere was no indication at all from a \"git gc\" run that anything was\ndifferent. This why one of the progress bars being added here uses\nstart_progress() instead of start_delayed_progress(), so that it's\nguaranteed to be seen. E.g. on my tiny 867 commit dotfiles.git\nrepository:\n\n $ git -c gc.writeCommitGraph=true gc\n Enumerating objects: 2821, done.\n [...]\n Computing commit graph generation numbers: 100% (867/867), done.\n\nOn larger repositories, such as linux.git the delayed progress bar(s)\nwill kick in, and we'll show what's going on instead of, as was\npreviously happening, printing nothing while we write the graph:\n\n $ git -c gc.writeCommitGraph=true gc\n [...]\n Annotating commits in commit graph: 1565573, done.\n Computing commit graph generation numbers: 100% (782484/782484), done.\n\nNote that here we don't show \"Finding commits for commit graph\", this\nis because under \"git gc\" we seed the search with the commit\nreferences in the repository, and that set is too small to show any\nprogress, but would e.g. on a smaller repo such as git.git with\n--stdin-commits:\n\n $ git rev-list --all | git -c gc.writeCommitGraph=true write --stdin-commits\n Finding commits for commit graph: 100% (162576/162576), done.\n Computing commit graph generation numbers: 100% (162576/162576), done.\n\nWith --stdin-packs we don't show any estimation of how much is left to\ndo. This is because we might be processing more than one pack. We\ncould be less lazy here and show progress, either by detecting that\nwe're only processing one pack, or by first looping over the packs to\ndiscover how many commits they have. I don't see the point in doing\nthat work. So instead we get (on 2015-04-03-1M-git.git):\n\n $ echo pack-.idx | git -c gc.writeCommitGraph=true --exec-path=$PWD commit-graph write --stdin-packs\n Finding commits for commit graph: 13064614, done.\n Annotating commits in commit graph: 3001341, done.\n Computing commit graph generation numbers: 100% (1000447/1000447), done.\n\nNo GC mode uses --stdin-packs. It's what they use at Microsoft to\nmanually compute the generation numbers for their collection of large\npacks which are never coalesced.\n\nThe reason we need a \"report_progress\" variable passed down from \"git\ngc\" is so that we don't report this output when we're running in the\nprocess \"git gc --auto\" detaches from the terminal.\n\nSince we write the commit graph from the \"git gc\" process itself (as\nopposed to what we do with say the \"git repack\" phase), we'd end up\nwriting the output to .git/gc.log and reporting it to the user next\ntime as part of the \"The last gc run reported the following[...]\"\nerror, see 329e6e8794 (\"gc: save log from daemonized gc --auto and\nprint it next time\", 2015-09-19).\n\nSo we must keep track of whether or not we're running in that\ndemonized mode, and if so print no progress.\n\nSee [2] and subsequent replies for a discussion of an approach not\ntaken in compute_generation_numbers(). I.e. we're saying \"Computing\ncommit graph generation numbers\", even though on an established\nhistory we're mostly skipping over all the work we did in the\npast. This is similar to the white lie we tell in the \"Writing\nobjects\" phase (not all are objects being written).\n\nAlways showing progress is considered more important than\naccuracy. I.e. on a repository like 2015-04-03-1M-git.git we'd hang\nfor 6 seconds with no output on the second \"git gc\" if no changes were\nmade to any objects in the interim if we'd take the approach in [2].\n\n1. https://github.com/avar/2015-04-03-1M-git\n\n2. \n (https://public-inbox.org/git/c6960252-c095-fb2b-e0bc-b1e6bb261614@gmail.com/)\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/commit-graph.c | 5 ++--\n builtin/gc.c | 3 ++-\n commit-graph.c | 60 ++++++++++++++++++++++++++++++++++++------\n commit-graph.h | 5 ++--\n 4 files changed, 60 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 0bf0c48657..bc0fa9ba52 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -151,7 +151,7 @@ static int graph_write(int argc, const char **argv)\n \t\topts.obj_dir = get_object_directory();\n \n \tif (opts.reachable) {\n-\t\twrite_commit_graph_reachable(opts.obj_dir, opts.append);\n+\t\twrite_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n \t\treturn 0;\n \t}\n \n@@ -171,7 +171,8 @@ static int graph_write(int argc, const char **argv)\n \twrite_commit_graph(opts.obj_dir,\n \t\t\t pack_indexes,\n \t\t\t commit_hex,\n-\t\t\t opts.append);\n+\t\t\t opts.append,\n+\t\t\t 1);\n \n \tstring_list_clear(&lines, 0);\n \treturn 0;\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 57069442b0..06ddd3aea5 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -646,7 +646,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tclean_pack_garbage();\n \n \tif (gc_write_commit_graph)\n-\t\twrite_commit_graph_reachable(get_object_directory(), 0);\n+\t\twrite_commit_graph_reachable(get_object_directory(), 0,\n+\t\t\t\t\t !daemonized);\n \n \tif (auto_gc && too_many_loose_objects())\n \t\twarning(_(\"There are too many unreachable loose objects; \"\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8a1bec7b8a..2c5d996194 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -13,6 +13,7 @@\n #include \"commit-graph.h\"\n #include \"object-store.h\"\n #include \"alloc.h\"\n+#include \"progress.h\"\n \n #define GRAPH_SIGNATURE 0x43475048 /* \"CGPH\" */\n #define GRAPH_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n@@ -548,6 +549,8 @@ struct packed_oid_list {\n \tstruct object_id *list;\n \tint nr;\n \tint alloc;\n+\tstruct progress *progress;\n+\tint progress_done;\n };\n \n static int add_packed_commits(const struct object_id *oid,\n@@ -560,6 +563,9 @@ static int add_packed_commits(const struct object_id *oid,\n \toff_t offset = nth_packed_object_offset(pack, pos);\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \n+\tif (list->progress)\n+\t\tdisplay_progress(list->progress, ++list->progress_done);\n+\n \toi.typep = &type;\n \tif (packed_object_info(the_repository, pack, offset, &oi) < 0)\n \t\tdie(_(\"unable to get type of object %s\"), oid_to_hex(oid));\n@@ -587,12 +593,18 @@ static void add_missing_parents(struct packed_oid_list *oids, struct commit *com\n \t}\n }\n \n-static void close_reachable(struct packed_oid_list *oids)\n+static void close_reachable(struct packed_oid_list *oids, int report_progress)\n {\n \tint i;\n \tstruct commit *commit;\n+\tstruct progress *progress = NULL;\n+\tint j = 0;\n \n+\tif (report_progress)\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Annotating commits in commit graph\"), 0);\n \tfor (i = 0; i < oids->nr; i++) {\n+\t\tdisplay_progress(progress, ++j);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n \t\tif (commit)\n \t\t\tcommit->object.flags |= UNINTERESTING;\n@@ -604,6 +616,7 @@ static void close_reachable(struct packed_oid_list *oids)\n \t * closure.\n \t */\n \tfor (i = 0; i < oids->nr; i++) {\n+\t\tdisplay_progress(progress, ++j);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n \n \t\tif (commit && !parse_commit(commit))\n@@ -611,19 +624,28 @@ static void close_reachable(struct packed_oid_list *oids)\n \t}\n \n \tfor (i = 0; i < oids->nr; i++) {\n+\t\tdisplay_progress(progress, ++j);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n \n \t\tif (commit)\n \t\t\tcommit->object.flags &= ~UNINTERESTING;\n \t}\n+\tstop_progress(&progress);\n }\n \n-static void compute_generation_numbers(struct packed_commit_list* commits)\n+static void compute_generation_numbers(struct packed_commit_list* commits, \n+\t\t\t\t int report_progress)\n {\n \tint i;\n \tstruct commit_list *list = NULL;\n+\tstruct progress *progress = NULL;\n \n+\tif (report_progress)\n+\t\tprogress = start_progress(\n+\t\t\t_(\"Computing commit graph generation numbers\"),\n+\t\t\tcommits->nr);\n \tfor (i = 0; i < commits->nr; i++) {\n+\t\tdisplay_progress(progress, i + 1);\n \t\tif (commits->list[i]->generation != GENERATION_NUMBER_INFINITY &&\n \t\t commits->list[i]->generation != GENERATION_NUMBER_ZERO)\n \t\t\tcontinue;\n@@ -655,6 +677,7 @@ static void compute_generation_numbers(struct packed_commit_list* commits)\n \t\t\t}\n \t\t}\n \t}\n+\tstop_progress(&progress);\n }\n \n static int add_ref_to_list(const char *refname,\n@@ -667,19 +690,20 @@ static int add_ref_to_list(const char *refname,\n \treturn 0;\n }\n \n-void write_commit_graph_reachable(const char *obj_dir, int append)\n+void write_commit_graph_reachable(const char *obj_dir, int append,\n+\t\t\t\t int report_progress)\n {\n \tstruct string_list list;\n \n \tstring_list_init(&list, 1);\n \tfor_each_ref(add_ref_to_list, &list);\n-\twrite_commit_graph(obj_dir, NULL, &list, append);\n+\twrite_commit_graph(obj_dir, NULL, &list, append, report_progress);\n }\n \n void write_commit_graph(const char *obj_dir,\n \t\t\tstruct string_list *pack_indexes,\n \t\t\tstruct string_list *commit_hex,\n-\t\t\tint append)\n+\t\t\tint append, int report_progress)\n {\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n@@ -692,9 +716,12 @@ void write_commit_graph(const char *obj_dir,\n \tint num_chunks;\n \tint num_extra_edges;\n \tstruct commit_list *parent;\n+\tstruct progress *progress = NULL;\n \n \toids.nr = 0;\n \toids.alloc = approximate_object_count() / 4;\n+\toids.progress = NULL;\n+\toids.progress_done = 0;\n \n \tif (append) {\n \t\tprepare_commit_graph_one(the_repository, obj_dir);\n@@ -721,6 +748,11 @@ void write_commit_graph(const char *obj_dir,\n \t\tint dirlen;\n \t\tstrbuf_addf(&packname, \"%s/pack/\", obj_dir);\n \t\tdirlen = packname.len;\n+\t\tif (report_progress) {\n+\t\t\toids.progress = start_delayed_progress(\n+\t\t\t\t_(\"Finding commits for commit graph\"), 0);\n+\t\t\toids.progress_done = 0;\n+\t\t}\n \t\tfor (i = 0; i < pack_indexes->nr; i++) {\n \t\t\tstruct packed_git *p;\n \t\t\tstrbuf_setlen(&packname, dirlen);\n@@ -733,15 +765,21 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tfor_each_object_in_pack(p, add_packed_commits, &oids, 0);\n \t\t\tclose_pack(p);\n \t\t}\n+\t\tstop_progress(&oids.progress);\n \t\tstrbuf_release(&packname);\n \t}\n \n \tif (commit_hex) {\n+\t\tif (report_progress)\n+\t\t\tprogress = start_delayed_progress(\n+\t\t\t\t_(\"Finding commits for commit graph\"),\n+\t\t\t\tcommit_hex->nr);\n \t\tfor (i = 0; i < commit_hex->nr; i++) {\n \t\t\tconst char *end;\n \t\t\tstruct object_id oid;\n \t\t\tstruct commit *result;\n \n+\t\t\tdisplay_progress(progress, i + 1);\n \t\t\tif (commit_hex->items[i].string &&\n \t\t\t parse_oid_hex(commit_hex->items[i].string, &oid, &end))\n \t\t\t\tcontinue;\n@@ -754,12 +792,18 @@ void write_commit_graph(const char *obj_dir,\n \t\t\t\toids.nr++;\n \t\t\t}\n \t\t}\n+\t\tstop_progress(&progress);\n \t}\n \n-\tif (!pack_indexes && !commit_hex)\n+\tif (!pack_indexes && !commit_hex) {\n+\t\tif (report_progress)\n+\t\t\toids.progress = start_delayed_progress(\n+\t\t\t\t_(\"Finding commits for commit graph\"), 0);\n \t\tfor_each_packed_object(add_packed_commits, &oids, 0);\n+\t\tstop_progress(&oids.progress);\n+\t}\n \n-\tclose_reachable(&oids);\n+\tclose_reachable(&oids, report_progress);\n \n \tQSORT(oids.list, oids.nr, commit_compare);\n \n@@ -799,7 +843,7 @@ void write_commit_graph(const char *obj_dir,\n \tif (commits.nr >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"too many commits to write graph\"));\n \n-\tcompute_generation_numbers(&commits);\n+\tcompute_generation_numbers(&commits, report_progress);\n \n \tgraph_name = get_commit_graph_filename(obj_dir);\n \tif (safe_create_leading_directories(graph_name))\ndiff --git a/commit-graph.h b/commit-graph.h\nindex eea62f8c0e..f50712a973 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -52,11 +52,12 @@ struct commit_graph {\n \n struct commit_graph *load_commit_graph_one(const char *graph_file);\n \n-void write_commit_graph_reachable(const char *obj_dir, int append);\n+void write_commit_graph_reachable(const char *obj_dir, int append,\n+\t\t\t\t int report_progress);\n void write_commit_graph(const char *obj_dir,\n \t\t\tstruct string_list *pack_indexes,\n \t\t\tstruct string_list *commit_hex,\n-\t\t\tint append);\n+\t\t\tint append, int report_progress);\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \n-- \n2.19.0.rc1.350.ge57e33dbd1\n\n"},{"id":"357646","messageId":"20180907182954.2413-3-avarab@gmail.com","threadId":"49275","inReplyTo":"20180904202729.13900-1-avarab@gmail.com","subject":"[PATCH v2 2/2] commit-graph verify: add progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-09-07T18:29:54Z","receivedAt":"2018-09-07T18:30:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"For the reasons explained in the \"commit-graph write: add progress\noutput\" commit leading up to this one, emit progress on \"commit-graph\nverify\". Since e0fd51e1d7 (\"fsck: verify commit-graph\", 2018-06-27)\n\"git fsck\" has called this command if core.commitGraph=true, but\nthere's been no progress output to indicate that anything was\ndifferent. Now there is (on my tiny dotfiles.git repository):\n\n $ git -c core.commitGraph=true -C ~/ fsck\n Checking object directories: 100% (256/256), done.\n Checking objects: 100% (2821/2821), done.\n dangling blob 5b8bbdb9b788ed90459f505b0934619c17cc605b\n Verifying commits in commit graph: 100% (867/867), done.\n\nAnd on a larger repository, such as the 2015-04-03-1M-git.git test\nrepository:\n\n $ time git -c core.commitGraph=true -C ~/g/2015-04-03-1M-git/ commit-graph verify\n Verifying commits in commit graph: 100% (1000447/1000447), done.\n real 0m7.813s\n [...]\n\nSince the \"commit-graph verify\" subcommand is never called from \"git\ngc\", we don't have to worry about passing some some \"report_progress\"\nprogress variable around for this codepath.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 5 +++++\n 1 file changed, 5 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 2c5d996194..0bfb8c180e 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -922,6 +922,7 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \tint generation_zero = 0;\n \tstruct hashfile *f;\n \tint devnull;\n+\tstruct progress *progress = NULL;\n \n \tif (!g) {\n \t\tgraph_report(\"no commit-graph file loaded\");\n@@ -989,11 +990,14 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \tif (verify_commit_graph_error & ~VERIFY_COMMIT_GRAPH_ERROR_HASH)\n \t\treturn verify_commit_graph_error;\n \n+\tprogress = start_progress(\"Verifying commits in commit graph\",\n+\t\t\t\t g->num_commits);\n \tfor (i = 0; i < g->num_commits; i++) {\n \t\tstruct commit *graph_commit, *odb_commit;\n \t\tstruct commit_list *graph_parents, *odb_parents;\n \t\tuint32_t max_generation = 0;\n \n+\t\tdisplay_progress(progress, i + 1);\n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n \t\tgraph_commit = lookup_commit(r, &cur_oid);\n@@ -1070,6 +1074,7 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\t\t\t graph_commit->date,\n \t\t\t\t odb_commit->date);\n \t}\n+\tstop_progress(&progress);\n \n \treturn verify_commit_graph_error;\n }\n-- \n2.19.0.rc1.350.ge57e33dbd1\n\n"},{"id":"357908","messageId":"xmqqo9d3ok6a.fsf@gitster-ct.c.googlers.com","threadId":"49275","inReplyTo":"20180907182954.2413-1-avarab@gmail.com","subject":"Re: [PATCH v2 0/2] commit-graph: add progress output","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-09-11T20:26:05Z","receivedAt":"2018-09-11T20:26:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> Based on feedback on v1, and the \"this is yelling at my users through\n> gc.log\" bug I found.\n\nI notice that between 'master' and 'pu' there already is one new\ncallsite of the write_commit_graph_reachable() function; because I\nsuspect that we will discover more places that give us better\ntrade-off to have a call to the function, I would not be surprised\nif this will conflict with more in-flight topics soonish.\n\nLet me try to queue it and see what happens.\n\nThanks.\n\n"},{"id":"358223","messageId":"CACsJy8A5tFxAaD-OqNNvMmX+KnbmW=O7JCCBbY-5dZa8Ta7QYg@mail.gmail.com","threadId":"49275","inReplyTo":"20180907182954.2413-3-avarab@gmail.com","subject":"Re: [PATCH v2 2/2] commit-graph verify: add progress output","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-09-16T06:55:36Z","receivedAt":"2018-09-16T06:56:07Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Sep 7, 2018 at 8:30 PM Ævar Arnfjörð Bjarmason wrote:\n> @@ -989,11 +990,14 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n> if (verify_commit_graph_error & ~VERIFY_COMMIT_GRAPH_ERROR_HASH)\n> return verify_commit_graph_error;\n>\n> + progress = start_progress(\"Verifying commits in commit graph\",\n\n_()\n\n> + g->num_commits);\n-- \nDuy\n"},{"id":"358276","messageId":"20180917153336.2280-1-avarab@gmail.com","threadId":"49275","inReplyTo":"CACsJy8A5tFxAaD-OqNNvMmX+KnbmW=O7JCCBbY-5dZa8Ta7QYg@mail.gmail.com","subject":"[PATCH v3 0/2] commit-graph: add progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-09-17T15:33:34Z","receivedAt":"2018-09-17T15:33:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Micro change since v2: Missing _() in 2/2 pointed out by Duy.\n\nÆvar Arnfjörð Bjarmason (2):\n commit-graph write: add progress output\n commit-graph verify: add progress output\n\n builtin/commit-graph.c | 5 ++--\n builtin/gc.c | 3 +-\n commit-graph.c | 65 ++++++++++++++++++++++++++++++++++++------\n commit-graph.h | 5 ++--\n 4 files changed, 65 insertions(+), 13 deletions(-)\n\n-- \n2.19.0.rc2.392.g5ba43deb5a\n\n"},{"id":"358277","messageId":"20180917153336.2280-2-avarab@gmail.com","threadId":"49275","inReplyTo":"CACsJy8A5tFxAaD-OqNNvMmX+KnbmW=O7JCCBbY-5dZa8Ta7QYg@mail.gmail.com","subject":"[PATCH v3 1/2] commit-graph write: add progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-09-17T15:33:35Z","receivedAt":"2018-09-17T15:33:52Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Before this change the \"commit-graph write\" command didn't report any\nprogress. On my machine this command takes more than 10 seconds to\nwrite the graph for linux.git, and around 1m30s on the\n2015-04-03-1M-git.git[1] test repository (a test case for a large\nmonorepository).\n\nFurthermore, since the gc.writeCommitGraph setting was added in\nd5d5d7b641 (\"gc: automatically write commit-graph files\", 2018-06-27),\nthere was no indication at all from a \"git gc\" run that anything was\ndifferent. This why one of the progress bars being added here uses\nstart_progress() instead of start_delayed_progress(), so that it's\nguaranteed to be seen. E.g. on my tiny 867 commit dotfiles.git\nrepository:\n\n $ git -c gc.writeCommitGraph=true gc\n Enumerating objects: 2821, done.\n [...]\n Computing commit graph generation numbers: 100% (867/867), done.\n\nOn larger repositories, such as linux.git the delayed progress bar(s)\nwill kick in, and we'll show what's going on instead of, as was\npreviously happening, printing nothing while we write the graph:\n\n $ git -c gc.writeCommitGraph=true gc\n [...]\n Annotating commits in commit graph: 1565573, done.\n Computing commit graph generation numbers: 100% (782484/782484), done.\n\nNote that here we don't show \"Finding commits for commit graph\", this\nis because under \"git gc\" we seed the search with the commit\nreferences in the repository, and that set is too small to show any\nprogress, but would e.g. on a smaller repo such as git.git with\n--stdin-commits:\n\n $ git rev-list --all | git -c gc.writeCommitGraph=true write --stdin-commits\n Finding commits for commit graph: 100% (162576/162576), done.\n Computing commit graph generation numbers: 100% (162576/162576), done.\n\nWith --stdin-packs we don't show any estimation of how much is left to\ndo. This is because we might be processing more than one pack. We\ncould be less lazy here and show progress, either by detecting that\nwe're only processing one pack, or by first looping over the packs to\ndiscover how many commits they have. I don't see the point in doing\nthat work. So instead we get (on 2015-04-03-1M-git.git):\n\n $ echo pack-.idx | git -c gc.writeCommitGraph=true --exec-path=$PWD commit-graph write --stdin-packs\n Finding commits for commit graph: 13064614, done.\n Annotating commits in commit graph: 3001341, done.\n Computing commit graph generation numbers: 100% (1000447/1000447), done.\n\nNo GC mode uses --stdin-packs. It's what they use at Microsoft to\nmanually compute the generation numbers for their collection of large\npacks which are never coalesced.\n\nThe reason we need a \"report_progress\" variable passed down from \"git\ngc\" is so that we don't report this output when we're running in the\nprocess \"git gc --auto\" detaches from the terminal.\n\nSince we write the commit graph from the \"git gc\" process itself (as\nopposed to what we do with say the \"git repack\" phase), we'd end up\nwriting the output to .git/gc.log and reporting it to the user next\ntime as part of the \"The last gc run reported the following[...]\"\nerror, see 329e6e8794 (\"gc: save log from daemonized gc --auto and\nprint it next time\", 2015-09-19).\n\nSo we must keep track of whether or not we're running in that\ndemonized mode, and if so print no progress.\n\nSee [2] and subsequent replies for a discussion of an approach not\ntaken in compute_generation_numbers(). I.e. we're saying \"Computing\ncommit graph generation numbers\", even though on an established\nhistory we're mostly skipping over all the work we did in the\npast. This is similar to the white lie we tell in the \"Writing\nobjects\" phase (not all are objects being written).\n\nAlways showing progress is considered more important than\naccuracy. I.e. on a repository like 2015-04-03-1M-git.git we'd hang\nfor 6 seconds with no output on the second \"git gc\" if no changes were\nmade to any objects in the interim if we'd take the approach in [2].\n\n1. https://github.com/avar/2015-04-03-1M-git\n\n2. \n (https://public-inbox.org/git/c6960252-c095-fb2b-e0bc-b1e6bb261614@gmail.com/)\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n builtin/commit-graph.c | 5 ++--\n builtin/gc.c | 3 ++-\n commit-graph.c | 60 ++++++++++++++++++++++++++++++++++++------\n commit-graph.h | 5 ++--\n 4 files changed, 60 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 0bf0c48657..bc0fa9ba52 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -151,7 +151,7 @@ static int graph_write(int argc, const char **argv)\n \t\topts.obj_dir = get_object_directory();\n \n \tif (opts.reachable) {\n-\t\twrite_commit_graph_reachable(opts.obj_dir, opts.append);\n+\t\twrite_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n \t\treturn 0;\n \t}\n \n@@ -171,7 +171,8 @@ static int graph_write(int argc, const char **argv)\n \twrite_commit_graph(opts.obj_dir,\n \t\t\t pack_indexes,\n \t\t\t commit_hex,\n-\t\t\t opts.append);\n+\t\t\t opts.append,\n+\t\t\t 1);\n \n \tstring_list_clear(&lines, 0);\n \treturn 0;\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 57069442b0..06ddd3aea5 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -646,7 +646,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tclean_pack_garbage();\n \n \tif (gc_write_commit_graph)\n-\t\twrite_commit_graph_reachable(get_object_directory(), 0);\n+\t\twrite_commit_graph_reachable(get_object_directory(), 0,\n+\t\t\t\t\t !daemonized);\n \n \tif (auto_gc && too_many_loose_objects())\n \t\twarning(_(\"There are too many unreachable loose objects; \"\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8a1bec7b8a..2c5d996194 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -13,6 +13,7 @@\n #include \"commit-graph.h\"\n #include \"object-store.h\"\n #include \"alloc.h\"\n+#include \"progress.h\"\n \n #define GRAPH_SIGNATURE 0x43475048 /* \"CGPH\" */\n #define GRAPH_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n@@ -548,6 +549,8 @@ struct packed_oid_list {\n \tstruct object_id *list;\n \tint nr;\n \tint alloc;\n+\tstruct progress *progress;\n+\tint progress_done;\n };\n \n static int add_packed_commits(const struct object_id *oid,\n@@ -560,6 +563,9 @@ static int add_packed_commits(const struct object_id *oid,\n \toff_t offset = nth_packed_object_offset(pack, pos);\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \n+\tif (list->progress)\n+\t\tdisplay_progress(list->progress, ++list->progress_done);\n+\n \toi.typep = &type;\n \tif (packed_object_info(the_repository, pack, offset, &oi) < 0)\n \t\tdie(_(\"unable to get type of object %s\"), oid_to_hex(oid));\n@@ -587,12 +593,18 @@ static void add_missing_parents(struct packed_oid_list *oids, struct commit *com\n \t}\n }\n \n-static void close_reachable(struct packed_oid_list *oids)\n+static void close_reachable(struct packed_oid_list *oids, int report_progress)\n {\n \tint i;\n \tstruct commit *commit;\n+\tstruct progress *progress = NULL;\n+\tint j = 0;\n \n+\tif (report_progress)\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Annotating commits in commit graph\"), 0);\n \tfor (i = 0; i < oids->nr; i++) {\n+\t\tdisplay_progress(progress, ++j);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n \t\tif (commit)\n \t\t\tcommit->object.flags |= UNINTERESTING;\n@@ -604,6 +616,7 @@ static void close_reachable(struct packed_oid_list *oids)\n \t * closure.\n \t */\n \tfor (i = 0; i < oids->nr; i++) {\n+\t\tdisplay_progress(progress, ++j);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n \n \t\tif (commit && !parse_commit(commit))\n@@ -611,19 +624,28 @@ static void close_reachable(struct packed_oid_list *oids)\n \t}\n \n \tfor (i = 0; i < oids->nr; i++) {\n+\t\tdisplay_progress(progress, ++j);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n \n \t\tif (commit)\n \t\t\tcommit->object.flags &= ~UNINTERESTING;\n \t}\n+\tstop_progress(&progress);\n }\n \n-static void compute_generation_numbers(struct packed_commit_list* commits)\n+static void compute_generation_numbers(struct packed_commit_list* commits, \n+\t\t\t\t int report_progress)\n {\n \tint i;\n \tstruct commit_list *list = NULL;\n+\tstruct progress *progress = NULL;\n \n+\tif (report_progress)\n+\t\tprogress = start_progress(\n+\t\t\t_(\"Computing commit graph generation numbers\"),\n+\t\t\tcommits->nr);\n \tfor (i = 0; i < commits->nr; i++) {\n+\t\tdisplay_progress(progress, i + 1);\n \t\tif (commits->list[i]->generation != GENERATION_NUMBER_INFINITY &&\n \t\t commits->list[i]->generation != GENERATION_NUMBER_ZERO)\n \t\t\tcontinue;\n@@ -655,6 +677,7 @@ static void compute_generation_numbers(struct packed_commit_list* commits)\n \t\t\t}\n \t\t}\n \t}\n+\tstop_progress(&progress);\n }\n \n static int add_ref_to_list(const char *refname,\n@@ -667,19 +690,20 @@ static int add_ref_to_list(const char *refname,\n \treturn 0;\n }\n \n-void write_commit_graph_reachable(const char *obj_dir, int append)\n+void write_commit_graph_reachable(const char *obj_dir, int append,\n+\t\t\t\t int report_progress)\n {\n \tstruct string_list list;\n \n \tstring_list_init(&list, 1);\n \tfor_each_ref(add_ref_to_list, &list);\n-\twrite_commit_graph(obj_dir, NULL, &list, append);\n+\twrite_commit_graph(obj_dir, NULL, &list, append, report_progress);\n }\n \n void write_commit_graph(const char *obj_dir,\n \t\t\tstruct string_list *pack_indexes,\n \t\t\tstruct string_list *commit_hex,\n-\t\t\tint append)\n+\t\t\tint append, int report_progress)\n {\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n@@ -692,9 +716,12 @@ void write_commit_graph(const char *obj_dir,\n \tint num_chunks;\n \tint num_extra_edges;\n \tstruct commit_list *parent;\n+\tstruct progress *progress = NULL;\n \n \toids.nr = 0;\n \toids.alloc = approximate_object_count() / 4;\n+\toids.progress = NULL;\n+\toids.progress_done = 0;\n \n \tif (append) {\n \t\tprepare_commit_graph_one(the_repository, obj_dir);\n@@ -721,6 +748,11 @@ void write_commit_graph(const char *obj_dir,\n \t\tint dirlen;\n \t\tstrbuf_addf(&packname, \"%s/pack/\", obj_dir);\n \t\tdirlen = packname.len;\n+\t\tif (report_progress) {\n+\t\t\toids.progress = start_delayed_progress(\n+\t\t\t\t_(\"Finding commits for commit graph\"), 0);\n+\t\t\toids.progress_done = 0;\n+\t\t}\n \t\tfor (i = 0; i < pack_indexes->nr; i++) {\n \t\t\tstruct packed_git *p;\n \t\t\tstrbuf_setlen(&packname, dirlen);\n@@ -733,15 +765,21 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tfor_each_object_in_pack(p, add_packed_commits, &oids, 0);\n \t\t\tclose_pack(p);\n \t\t}\n+\t\tstop_progress(&oids.progress);\n \t\tstrbuf_release(&packname);\n \t}\n \n \tif (commit_hex) {\n+\t\tif (report_progress)\n+\t\t\tprogress = start_delayed_progress(\n+\t\t\t\t_(\"Finding commits for commit graph\"),\n+\t\t\t\tcommit_hex->nr);\n \t\tfor (i = 0; i < commit_hex->nr; i++) {\n \t\t\tconst char *end;\n \t\t\tstruct object_id oid;\n \t\t\tstruct commit *result;\n \n+\t\t\tdisplay_progress(progress, i + 1);\n \t\t\tif (commit_hex->items[i].string &&\n \t\t\t parse_oid_hex(commit_hex->items[i].string, &oid, &end))\n \t\t\t\tcontinue;\n@@ -754,12 +792,18 @@ void write_commit_graph(const char *obj_dir,\n \t\t\t\toids.nr++;\n \t\t\t}\n \t\t}\n+\t\tstop_progress(&progress);\n \t}\n \n-\tif (!pack_indexes && !commit_hex)\n+\tif (!pack_indexes && !commit_hex) {\n+\t\tif (report_progress)\n+\t\t\toids.progress = start_delayed_progress(\n+\t\t\t\t_(\"Finding commits for commit graph\"), 0);\n \t\tfor_each_packed_object(add_packed_commits, &oids, 0);\n+\t\tstop_progress(&oids.progress);\n+\t}\n \n-\tclose_reachable(&oids);\n+\tclose_reachable(&oids, report_progress);\n \n \tQSORT(oids.list, oids.nr, commit_compare);\n \n@@ -799,7 +843,7 @@ void write_commit_graph(const char *obj_dir,\n \tif (commits.nr >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"too many commits to write graph\"));\n \n-\tcompute_generation_numbers(&commits);\n+\tcompute_generation_numbers(&commits, report_progress);\n \n \tgraph_name = get_commit_graph_filename(obj_dir);\n \tif (safe_create_leading_directories(graph_name))\ndiff --git a/commit-graph.h b/commit-graph.h\nindex eea62f8c0e..f50712a973 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -52,11 +52,12 @@ struct commit_graph {\n \n struct commit_graph *load_commit_graph_one(const char *graph_file);\n \n-void write_commit_graph_reachable(const char *obj_dir, int append);\n+void write_commit_graph_reachable(const char *obj_dir, int append,\n+\t\t\t\t int report_progress);\n void write_commit_graph(const char *obj_dir,\n \t\t\tstruct string_list *pack_indexes,\n \t\t\tstruct string_list *commit_hex,\n-\t\t\tint append);\n+\t\t\tint append, int report_progress);\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \n-- \n2.19.0.rc2.392.g5ba43deb5a\n\n"},{"id":"358278","messageId":"20180917153336.2280-3-avarab@gmail.com","threadId":"49275","inReplyTo":"CACsJy8A5tFxAaD-OqNNvMmX+KnbmW=O7JCCBbY-5dZa8Ta7QYg@mail.gmail.com","subject":"[PATCH v3 2/2] commit-graph verify: add progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-09-17T15:33:36Z","receivedAt":"2018-09-17T15:33:54Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"For the reasons explained in the \"commit-graph write: add progress\noutput\" commit leading up to this one, emit progress on \"commit-graph\nverify\". Since e0fd51e1d7 (\"fsck: verify commit-graph\", 2018-06-27)\n\"git fsck\" has called this command if core.commitGraph=true, but\nthere's been no progress output to indicate that anything was\ndifferent. Now there is (on my tiny dotfiles.git repository):\n\n $ git -c core.commitGraph=true -C ~/ fsck\n Checking object directories: 100% (256/256), done.\n Checking objects: 100% (2821/2821), done.\n dangling blob 5b8bbdb9b788ed90459f505b0934619c17cc605b\n Verifying commits in commit graph: 100% (867/867), done.\n\nAnd on a larger repository, such as the 2015-04-03-1M-git.git test\nrepository:\n\n $ time git -c core.commitGraph=true -C ~/g/2015-04-03-1M-git/ commit-graph verify\n Verifying commits in commit graph: 100% (1000447/1000447), done.\n real 0m7.813s\n [...]\n\nSince the \"commit-graph verify\" subcommand is never called from \"git\ngc\", we don't have to worry about passing some some \"report_progress\"\nprogress variable around for this codepath.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 5 +++++\n 1 file changed, 5 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 2c5d996194..e6e4c03986 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -922,6 +922,7 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \tint generation_zero = 0;\n \tstruct hashfile *f;\n \tint devnull;\n+\tstruct progress *progress = NULL;\n \n \tif (!g) {\n \t\tgraph_report(\"no commit-graph file loaded\");\n@@ -989,11 +990,14 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \tif (verify_commit_graph_error & ~VERIFY_COMMIT_GRAPH_ERROR_HASH)\n \t\treturn verify_commit_graph_error;\n \n+\tprogress = start_progress(_(\"Verifying commits in commit graph\"),\n+\t\t\t\t g->num_commits);\n \tfor (i = 0; i < g->num_commits; i++) {\n \t\tstruct commit *graph_commit, *odb_commit;\n \t\tstruct commit_list *graph_parents, *odb_parents;\n \t\tuint32_t max_generation = 0;\n \n+\t\tdisplay_progress(progress, i + 1);\n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n \t\tgraph_commit = lookup_commit(r, &cur_oid);\n@@ -1070,6 +1074,7 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\t\t\t graph_commit->date,\n \t\t\t\t odb_commit->date);\n \t}\n+\tstop_progress(&progress);\n \n \treturn verify_commit_graph_error;\n }\n-- \n2.19.0.rc2.392.g5ba43deb5a\n\n"},{"id":"358643","messageId":"60aae3d6-35b2-94fb-afd7-6978e935a4f7@gmail.com","threadId":"49275","inReplyTo":"20180907182954.2413-2-avarab@gmail.com","subject":"Re: [PATCH v2 1/2] commit-graph write: add progress output","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-09-21T20:01:25Z","receivedAt":"2018-09-21T20:01:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/7/2018 2:29 PM, Ævar Arnfjörð Bjarmason wrote:\n> -void write_commit_graph_reachable(const char *obj_dir, int append);\n> +void write_commit_graph_reachable(const char *obj_dir, int append,\n> +\t\t\t\t int report_progress);\n> void write_commit_graph(const char *obj_dir,\n> \t\t\tstruct string_list *pack_indexes,\n> \t\t\tstruct string_list *commit_hex,\n> -\t\t\tint append);\n> +\t\t\tint append, int report_progress);\n> \n> int verify_commit_graph(struct repository *r, struct commit_graph *g);\n> \n\nJunio,\n\nThe above prototype change seems to have created a semantic conflict \nwith ds/commit-graph-tests (859fdc \"commit-graph: define \nGIT_TEST_COMMIT_GRAPH\") because when GIT_TEST_COMMIT_GRAPH is set, we \ncall write_commit_graph_reachable() but the final parameter was resolved \nto be \"1\" instead of \"0\".\n\nThis causes t3420-rebase-autostash.sh to fail, as that test watches the \nfull output of the rebase command, including commit runs. The following \npatch fixes the problem, but could probably be squashed into a merge or \nother commit.\n\nThanks,\n\n-Stolee\n\n-->8--\n\nFrom: Derrick Stolee \nDate: Fri, 21 Sep 2018 19:57:36 +0000\nSubject: [PATCH] commit: quietly write commit-graph in tests\n\nThe GIT_TEST_COMMIT_GRAPH environment variable causes git-commit to\nwrite a commit-graph file on every execution. Recently, we added\nprogress output when writing the commit-graph. This conflicts with\nsome expected output during some tests, so avoid writing progress\nif writing a commit-graph this way.\n\nSigned-off-by: Derrick Stolee \n---\n  builtin/commit.c | 2 +-\n  1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 2a49ab4917..764664d832 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1660,7 +1660,7 @@ int cmd_commit(int argc, const char **argv, const \nchar *prefix)\n                       \"not exceeded, and then \\\"git reset HEAD\\\" to \nrecover.\"));\n\n         if (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0))\n- write_commit_graph_reachable(get_object_directory(), 0, 1);\n+ write_commit_graph_reachable(get_object_directory(), 0, 0);\n\n         rerere(0);\n         run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);\n--\n2.19.0\n\n"},{"id":"358661","messageId":"xmqq36u2v85q.fsf@gitster-ct.c.googlers.com","threadId":"49275","inReplyTo":"60aae3d6-35b2-94fb-afd7-6978e935a4f7@gmail.com","subject":"Re: [PATCH v2 1/2] commit-graph write: add progress output","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-09-21T21:43:13Z","receivedAt":"2018-09-21T21:43:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 9/7/2018 2:29 PM, Ævar Arnfjörð Bjarmason wrote:\n>> -void write_commit_graph_reachable(const char *obj_dir, int append);\n>> +void write_commit_graph_reachable(const char *obj_dir, int append,\n>> +\t\t\t\t int report_progress);\n>> void write_commit_graph(const char *obj_dir,\n>> \t\t\tstruct string_list *pack_indexes,\n>> \t\t\tstruct string_list *commit_hex,\n>> -\t\t\tint append);\n>> +\t\t\tint append, int report_progress);\n>> int verify_commit_graph(struct repository *r, struct\n>> commit_graph *g);\n>> \n>\n> Junio,\n>\n> The above prototype change seems to have created a semantic conflict\n> with ds/commit-graph-tests (859fdc \"commit-graph: define\n> GIT_TEST_COMMIT_GRAPH\") because when GIT_TEST_COMMIT_GRAPH is set, we\n> call write_commit_graph_reachable() but the final parameter was\n> resolved to be \"1\" instead of \"0\".\n\nHmph. That's unfortunate.\n\nPerhaps one of the topics should have yielded and waited until the\nother one passes through.\n\nAs 859fdc0c (\"commit-graph: define GIT_TEST_COMMIT_GRAPH\",\n2018-08-29) already is in 'master', the other \"progress\" topic\nprobably should be corrected to match. The easiest and cleanest\nwould be to eject the ab/commit-graph-progress topic out of 'next'\nand have it rerolled on top of 'master', as we are going to rewind\nthe tip of 'next' anyway.\n\nWhile we are at it, I suspect that a saner evolution of the API into\nthe function would not append more parameters to the call, but would\nmake the \"do we append?\" bit into a flag word \"unsigned flags\" with\ntwo bits, and such a clean-up can be done as a preliminary change.\n\n> This causes t3420-rebase-autostash.sh to fail, as that test watches\n> the full output of the rebase command, including commit runs. The\n> following patch fixes the problem, but could probably be squashed into\n> a merge or other commit.\n>\n> Thanks,\n>\n> -Stolee\n>\n> -->8--\n>\n> From: Derrick Stolee \n> Date: Fri, 21 Sep 2018 19:57:36 +0000\n> Subject: [PATCH] commit: quietly write commit-graph in tests\n>\n> The GIT_TEST_COMMIT_GRAPH environment variable causes git-commit to\n> write a commit-graph file on every execution. Recently, we added\n> progress output when writing the commit-graph. This conflicts with\n> some expected output during some tests, so avoid writing progress\n> if writing a commit-graph this way.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n>  builtin/commit.c | 2 +-\n>  1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/builtin/commit.c b/builtin/commit.c\n> index 2a49ab4917..764664d832 100644\n> --- a/builtin/commit.c\n> +++ b/builtin/commit.c\n> @@ -1660,7 +1660,7 @@ int cmd_commit(int argc, const char **argv,\n> const char *prefix)\n>                       \"not exceeded, and then \\\"git reset HEAD\\\" to\n> recover.\"));\n>\n>         if (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0))\n> - write_commit_graph_reachable(get_object_directory(), 0, 1);\n> + write_commit_graph_reachable(get_object_directory(), 0, 0);\n>\n>         rerere(0);\n>         run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);\n> --\n> 2.19.0\n"},{"id":"358662","messageId":"xmqqy3butsxn.fsf@gitster-ct.c.googlers.com","threadId":"49275","inReplyTo":"xmqq36u2v85q.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 1/2] commit-graph write: add progress output","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-09-21T21:57:24Z","receivedAt":"2018-09-21T21:57:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> The above prototype change seems to have created a semantic conflict\n>> with ds/commit-graph-tests (859fdc \"commit-graph: define\n>> GIT_TEST_COMMIT_GRAPH\") because when GIT_TEST_COMMIT_GRAPH is set, we\n>> call write_commit_graph_reachable() but the final parameter was\n>> resolved to be \"1\" instead of \"0\".\n>\n> Hmph. That's unfortunate.\n>\n> Perhaps one of the topics should have yielded and waited until the\n> other one passes through.\n\nNah, I see where things went wrong. I'll queue a single-liner\n\"mismerge fix\" to 'next', and then correct the seed for the evil\nmerge kept in merge-fix/ab/commit-graph-progress, and rebuild 'pu'.\nThings will straighten out by themselves after that happens.\n\nThanks for noticing.\n"},{"id":"360065","messageId":"20181010203738.GE23446@szeder.dev","threadId":"49275","inReplyTo":"20180917153336.2280-2-avarab@gmail.com","subject":"Re: [PATCH v3 1/2] commit-graph write: add progress output","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-10T20:37:38Z","receivedAt":"2018-10-10T20:37:46Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Sep 17, 2018 at 03:33:35PM +0000, Ævar Arnfjörð Bjarmason wrote:\n> $ git -c gc.writeCommitGraph=true gc\n> [...]\n> Annotating commits in commit graph: 1565573, done.\n> Computing commit graph generation numbers: 100% (782484/782484), done.\n\nWhile poking around 'commit-graph.c' in my Bloom filter experiment, I\nsaw similar numbers like above, and was confused by the much higher\nthan expected number of annotated commits. It's about twice as much\nas the number of commits in the repository, or the number shown on the\nvery next line.\n\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 8a1bec7b8a..2c5d996194 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> -static void close_reachable(struct packed_oid_list *oids)\n> +static void close_reachable(struct packed_oid_list *oids, int report_progress)\n> {\n> \tint i;\n> \tstruct commit *commit;\n> +\tstruct progress *progress = NULL;\n> +\tint j = 0;\n> \n> +\tif (report_progress)\n> +\t\tprogress = start_delayed_progress(\n> +\t\t\t_(\"Annotating commits in commit graph\"), 0);\n> \tfor (i = 0; i < oids->nr; i++) {\n> +\t\tdisplay_progress(progress, ++j);\n> \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n> \t\tif (commit)\n> \t\t\tcommit->object.flags |= UNINTERESTING;\n> @@ -604,6 +616,7 @@ static void close_reachable(struct packed_oid_list *oids)\n> \t * closure.\n> \t */\n> \tfor (i = 0; i < oids->nr; i++) {\n> +\t\tdisplay_progress(progress, ++j);\n> \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n> \n> \t\tif (commit && !parse_commit(commit))\n> @@ -611,19 +624,28 @@ static void close_reachable(struct packed_oid_list *oids)\n> \t}\n\nThe above loops have already counted all the commits, and, more\nimportantly, did all the hard work that takes time and makes the\nprogress indicator useful.\n\n> \tfor (i = 0; i < oids->nr; i++) {\n> +\t\tdisplay_progress(progress, ++j);\n\nThis display_progress() call, however, doesn't seem to be necessary.\nFirst, it counts all commits for a second time, resulting in the ~2x\ndifference compared to the actual number of commits, and then causing\nmy confusion. Second, all what this loop is doing is setting a flag\nin commits that were already looked up and parsed in the above loops.\nIOW this loop is very fast, and the progress indicator jumps from\n~780k right to 1.5M, even on my tiny laptop, so it doesn't need a\nprogress indicator at all.\n\n> \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n> \n> \t\tif (commit)\n> \t\t\tcommit->object.flags &= ~UNINTERESTING;\n> \t}\n> +\tstop_progress(&progress);\n> }\n"},{"id":"360091","messageId":"87pnwhea8y.fsf@evledraar.gmail.com","threadId":"49275","inReplyTo":"20181010203738.GE23446@szeder.dev","subject":"Re: [PATCH v3 1/2] commit-graph write: add progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-10T21:56:45Z","receivedAt":"2018-10-10T21:56:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Oct 10 2018, SZEDER Gábor wrote:\n\n> On Mon, Sep 17, 2018 at 03:33:35PM +0000, Ævar Arnfjörð Bjarmason wrote:\n>> $ git -c gc.writeCommitGraph=true gc\n>> [...]\n>> Annotating commits in commit graph: 1565573, done.\n>> Computing commit graph generation numbers: 100% (782484/782484), done.\n>\n> While poking around 'commit-graph.c' in my Bloom filter experiment, I\n> saw similar numbers like above, and was confused by the much higher\n> than expected number of annotated commits. It's about twice as much\n> as the number of commits in the repository, or the number shown on the\n> very next line.\n>\n>> diff --git a/commit-graph.c b/commit-graph.c\n>> index 8a1bec7b8a..2c5d996194 100644\n>> --- a/commit-graph.c\n>> +++ b/commit-graph.c\n>> -static void close_reachable(struct packed_oid_list *oids)\n>> +static void close_reachable(struct packed_oid_list *oids, int report_progress)\n>> {\n>> \tint i;\n>> \tstruct commit *commit;\n>> +\tstruct progress *progress = NULL;\n>> +\tint j = 0;\n>>\n>> +\tif (report_progress)\n>> +\t\tprogress = start_delayed_progress(\n>> +\t\t\t_(\"Annotating commits in commit graph\"), 0);\n>> \tfor (i = 0; i < oids->nr; i++) {\n>> +\t\tdisplay_progress(progress, ++j);\n>> \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n>> \t\tif (commit)\n>> \t\t\tcommit->object.flags |= UNINTERESTING;\n>> @@ -604,6 +616,7 @@ static void close_reachable(struct packed_oid_list *oids)\n>> \t * closure.\n>> \t */\n>> \tfor (i = 0; i < oids->nr; i++) {\n>> +\t\tdisplay_progress(progress, ++j);\n>> \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n>>\n>> \t\tif (commit && !parse_commit(commit))\n>> @@ -611,19 +624,28 @@ static void close_reachable(struct packed_oid_list *oids)\n>> \t}\n>\n> The above loops have already counted all the commits, and, more\n> importantly, did all the hard work that takes time and makes the\n> progress indicator useful.\n>\n>> \tfor (i = 0; i < oids->nr; i++) {\n>> +\t\tdisplay_progress(progress, ++j);\n\n[...]\n\n> This display_progress() call, however, doesn't seem to be necessary.\n> First, it counts all commits for a second time, resulting in the ~2x\n> difference compared to the actual number of commits, and then causing\n> my confusion. Second, all what this loop is doing is setting a flag\n> in commits that were already looked up and parsed in the above loops.\n> IOW this loop is very fast, and the progress indicator jumps from\n> ~780k right to 1.5M, even on my tiny laptop, so it doesn't need a\n> progress indicator at all.\n\nYou're right, I tried this patch on top:\n\n diff --git a/commit-graph.c b/commit-graph.c\n index a686758603..cccd83de72 100644\n --- a/commit-graph.c\n +++ b/commit-graph.c\n @@ -655,12 +655,16 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n \t\tif (commit)\n \t\t\tcommit->object.flags |= UNINTERESTING;\n \t}\n +\tstop_progress(&progress); j = 0;\n\n \t/*\n \t * As this loop runs, oids->nr may grow, but not more\n \t * than the number of missing commits in the reachable\n \t * closure.\n \t */\n +\tif (report_progress)\n +\t\tprogress = start_delayed_progress(\n +\t\t\t_(\"Annotating commits in commit graph 2\"), 0);\n \tfor (i = 0; i < oids->nr; i++) {\n \t\tdisplay_progress(progress, ++j);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n @@ -668,7 +672,11 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n \t\tif (commit && !parse_commit(commit))\n \t\t\tadd_missing_parents(oids, commit);\n \t}\n +\tstop_progress(&progress); j = 0;\n\n +\tif (report_progress)\n +\t\tprogress = start_delayed_progress(\n +\t\t\t_(\"Annotating commits in commit graph 3\"), 0);\n \tfor (i = 0; i < oids->nr; i++) {\n \t\tdisplay_progress(progress, ++j);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n\nAnd on a large repo with around 3 million commits the 3rd progress bar\ndoesn't kick in.\n\nBut if I apply this on top:\n\n diff --git a/progress.c b/progress.c\n index 5a99c9fbf0..89cc705bf7 100644\n --- a/progress.c\n +++ b/progress.c\n @@ -58,8 +58,8 @@ static void set_progress_signal(void)\n \tsa.sa_flags = SA_RESTART;\n \tsigaction(SIGALRM, &sa, NULL);\n\n -\tv.it_interval.tv_sec = 1;\n -\tv.it_interval.tv_usec = 0;\n +\tv.it_interval.tv_sec = 0;\n +\tv.it_interval.tv_usec = 250000;\n \tv.it_value = v.it_interval;\n \tsetitimer(ITIMER_REAL, &v, NULL);\n }\n\nI.e. start the timer after 1/4 of a second instead of 1 second, I get\nthat progress bar.\n\nSo I'm inclined to keep it. It just needs to be 4x the size before it's\nnoticeably hanging for 1 second.\n\nThat repo isn't all that big compared to what we've heard about out\nthere, and inner loops like this have a tendency to accumulate some more\ncode over time without a re-visit of why we weren't monitoring progress\nthere.\n\nBut maybe we can fix the message. We say \"Annotating commits in commit\ngrap\", not \"Counting\" or whatever. I was trying to find something that\ndidn't imply that we were doing this once. One can annotate a thing more\nthan once, but maybe ther's a better way to explain this...\n\nWe had some more accurate progress reporting in close_reachable(),\ndiscussed in\nhttps://public-inbox.org/git/87efe5qqks.fsf@evledraar.gmail.com/ I still\nthink the *main* use-case for these things is to just report that we're\nnot hanging, so maybe the proper solution is to pick up Duy's patch to\ndisplay a spinner insted of a numeric progress.\n\n>> \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n>>\n>> \t\tif (commit)\n>> \t\t\tcommit->object.flags &= ~UNINTERESTING;\n>> \t}\n>> +\tstop_progress(&progress);\n>> }\n"},{"id":"360096","messageId":"20181010221951.GG23446@szeder.dev","threadId":"49275","inReplyTo":"87pnwhea8y.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 1/2] commit-graph write: add progress output","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-10T22:19:52Z","receivedAt":"2018-10-10T22:19:57Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Oct 10, 2018 at 11:56:45PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> On Wed, Oct 10 2018, SZEDER Gábor wrote:\n\n> >> \tfor (i = 0; i < oids->nr; i++) {\n> >> +\t\tdisplay_progress(progress, ++j);\n> \n> [...]\n> \n> > This display_progress() call, however, doesn't seem to be necessary.\n> > First, it counts all commits for a second time, resulting in the ~2x\n> > difference compared to the actual number of commits, and then causing\n> > my confusion. Second, all what this loop is doing is setting a flag\n> > in commits that were already looked up and parsed in the above loops.\n> > IOW this loop is very fast, and the progress indicator jumps from\n> > ~780k right to 1.5M, even on my tiny laptop, so it doesn't need a\n> > progress indicator at all.\n> \n> You're right, I tried this patch on top:\n\n[...] \n\n> And on a large repo with around 3 million commits the 3rd progress bar\n> doesn't kick in.\n> \n> But if I apply this on top:\n> \n[...]\n> \n> I.e. start the timer after 1/4 of a second instead of 1 second, I get\n> that progress bar.\n> \n> So I'm inclined to keep it. It just needs to be 4x the size before it's\n> noticeably hanging for 1 second.\n\nJust to clarify: are you worried about a 1 second hang in an approx. 12\nmillion commit repository? If so, then I'm unconvinced, that's not\neven a blip on the radar, and the misleading numbers are far worse.\n\n> That repo isn't all that big compared to what we've heard about out\n> there, and inner loops like this have a tendency to accumulate some more\n> code over time without a re-visit of why we weren't monitoring progress\n> there.\n> \n> But maybe we can fix the message. We say \"Annotating commits in commit\n> grap\", not \"Counting\" or whatever. I was trying to find something that\n> didn't imply that we were doing this once. One can annotate a thing more\n> than once, but maybe ther's a better way to explain this...\n\nIMO just remove it.\n\n"},{"id":"360101","messageId":"87murle8da.fsf@evledraar.gmail.com","threadId":"49275","inReplyTo":"20181010221951.GG23446@szeder.dev","subject":"Re: [PATCH v3 1/2] commit-graph write: add progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-10T22:37:21Z","receivedAt":"2018-10-10T22:37:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Oct 10 2018, SZEDER Gábor wrote:\n\n> On Wed, Oct 10, 2018 at 11:56:45PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> On Wed, Oct 10 2018, SZEDER Gábor wrote:\n>\n>> >> \tfor (i = 0; i < oids->nr; i++) {\n>> >> +\t\tdisplay_progress(progress, ++j);\n>>\n>> [...]\n>>\n>> > This display_progress() call, however, doesn't seem to be necessary.\n>> > First, it counts all commits for a second time, resulting in the ~2x\n>> > difference compared to the actual number of commits, and then causing\n>> > my confusion. Second, all what this loop is doing is setting a flag\n>> > in commits that were already looked up and parsed in the above loops.\n>> > IOW this loop is very fast, and the progress indicator jumps from\n>> > ~780k right to 1.5M, even on my tiny laptop, so it doesn't need a\n>> > progress indicator at all.\n>>\n>> You're right, I tried this patch on top:\n>\n> [...]\n>\n>> And on a large repo with around 3 million commits the 3rd progress bar\n>> doesn't kick in.\n>>\n>> But if I apply this on top:\n>>\n> [...]\n>>\n>> I.e. start the timer after 1/4 of a second instead of 1 second, I get\n>> that progress bar.\n>>\n>> So I'm inclined to keep it. It just needs to be 4x the size before it's\n>> noticeably hanging for 1 second.\n>\n> Just to clarify: are you worried about a 1 second hang in an approx. 12\n> million commit repository? If so, then I'm unconvinced, that's not\n> even a blip on the radar, and the misleading numbers are far worse.\n\nIt's not a blip on the runtime, but the point of these progress bars in\ngeneral is so we don't have a UI where there's no UI differnce between\ngit hanging and just doing work in some tight loop in the background,\nand even 1 second when you're watching something is noticeable if it\nstalls.\n\nAlso it's 1 second on a server where I had 128G of RAM. I think even a\n\"trivial\" flag change like this would very much change if e.g. the\nsystem was under memory pressure or was swapping.\n\nAnd as noted code like this tends to change over time, that loop might\nget more expensive, so let's future proof by having all the loops over N\ncall the progress code.\n\nWhen I wrote this the intent was just \"report progress\". So that it's\ncounting anything is just an implementation detail of how progress.c\nworks now.\n\nThis was the reference to Duy's patch, i.e. instead of spewing numbers\nat the user here let's just render a spinner. Then we no longer need to\nmake judgement calls about which loop over N is expensive right now, and\nwhich one isn't, and if any of them will result in reporting a 2N number\nwhile the user might be more familiar with or expecting N.\n\n>> That repo isn't all that big compared to what we've heard about out\n>> there, and inner loops like this have a tendency to accumulate some more\n>> code over time without a re-visit of why we weren't monitoring progress\n>> there.\n>>\n>> But maybe we can fix the message. We say \"Annotating commits in commit\n>> grap\", not \"Counting\" or whatever. I was trying to find something that\n>> didn't imply that we were doing this once. One can annotate a thing more\n>> than once, but maybe ther's a better way to explain this...\n>\n> IMO just remove it.\n"},{"id":"360198","messageId":"87a7nke5gq.fsf@evledraar.gmail.com","threadId":"49275","inReplyTo":"87murle8da.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 1/2] commit-graph write: add progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-11T17:52:21Z","receivedAt":"2018-10-11T17:52:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Oct 10 2018, Ævar Arnfjörð Bjarmason wrote:\n\n> On Wed, Oct 10 2018, SZEDER Gábor wrote:\n>\n>> On Wed, Oct 10, 2018 at 11:56:45PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>>> On Wed, Oct 10 2018, SZEDER Gábor wrote:\n>>\n>>> >> \tfor (i = 0; i < oids->nr; i++) {\n>>> >> +\t\tdisplay_progress(progress, ++j);\n>>>\n>>> [...]\n>>>\n>>> > This display_progress() call, however, doesn't seem to be necessary.\n>>> > First, it counts all commits for a second time, resulting in the ~2x\n>>> > difference compared to the actual number of commits, and then causing\n>>> > my confusion. Second, all what this loop is doing is setting a flag\n>>> > in commits that were already looked up and parsed in the above loops.\n>>> > IOW this loop is very fast, and the progress indicator jumps from\n>>> > ~780k right to 1.5M, even on my tiny laptop, so it doesn't need a\n>>> > progress indicator at all.\n>>>\n>>> You're right, I tried this patch on top:\n>>\n>> [...]\n>>\n>>> And on a large repo with around 3 million commits the 3rd progress bar\n>>> doesn't kick in.\n>>>\n>>> But if I apply this on top:\n>>>\n>> [...]\n>>>\n>>> I.e. start the timer after 1/4 of a second instead of 1 second, I get\n>>> that progress bar.\n>>>\n>>> So I'm inclined to keep it. It just needs to be 4x the size before it's\n>>> noticeably hanging for 1 second.\n>>\n>> Just to clarify: are you worried about a 1 second hang in an approx. 12\n>> million commit repository? If so, then I'm unconvinced, that's not\n>> even a blip on the radar, and the misleading numbers are far worse.\n>\n> It's not a blip on the runtime, but the point of these progress bars in\n> general is so we don't have a UI where there's no UI differnce between\n> git hanging and just doing work in some tight loop in the background,\n> and even 1 second when you're watching something is noticeable if it\n> stalls.\n>\n> Also it's 1 second on a server where I had 128G of RAM. I think even a\n> \"trivial\" flag change like this would very much change if e.g. the\n> system was under memory pressure or was swapping.\n>\n> And as noted code like this tends to change over time, that loop might\n> get more expensive, so let's future proof by having all the loops over N\n> call the progress code.\n>\n> When I wrote this the intent was just \"report progress\". So that it's\n> counting anything is just an implementation detail of how progress.c\n> works now.\n>\n> This was the reference to Duy's patch, i.e. instead of spewing numbers\n> at the user here let's just render a spinner. Then we no longer need to\n> make judgement calls about which loop over N is expensive right now, and\n> which one isn't, and if any of them will result in reporting a 2N number\n> while the user might be more familiar with or expecting N.\n>\n>>> That repo isn't all that big compared to what we've heard about out\n>>> there, and inner loops like this have a tendency to accumulate some more\n>>> code over time without a re-visit of why we weren't monitoring progress\n>>> there.\n>>>\n>>> But maybe we can fix the message. We say \"Annotating commits in commit\n>>> grap\", not \"Counting\" or whatever. I was trying to find something that\n>>> didn't imply that we were doing this once. One can annotate a thing more\n>>> than once, but maybe ther's a better way to explain this...\n>>\n>> IMO just remove it.\n\nHrm, actually reading this again your initial post says we end up with a\n2x difference v.s. the number of commits, but it's actually 3x. The loop\nthat has a rather trivial runtime comparatively is the 3x, but the 2x\nloop takes a notable amount of time. So e.g. on git.git:\n\n $ git rev-list --all | wc -l; ~/g/git/git commit-graph write\n 166678\n Annotating commits in commit graph: 518463, done.\n Computing commit graph generation numbers: 100% (172685/172685), done.\n"},{"id":"360275","messageId":"xmqqa7njd7bi.fsf@gitster-ct.c.googlers.com","threadId":"49275","inReplyTo":"20181010203738.GE23446@szeder.dev","subject":"Re: [PATCH v3 1/2] commit-graph write: add progress output","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-12T06:09:53Z","receivedAt":"2018-10-12T06:09:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"SZEDER Gábor writes:\n\n>> \tfor (i = 0; i < oids->nr; i++) {\n>> +\t\tdisplay_progress(progress, ++j);\n>> \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n>> \n>> \t\tif (commit && !parse_commit(commit))\n>> @@ -611,19 +624,28 @@ static void close_reachable(struct packed_oid_list *oids)\n>> \t}\n>\n> The above loops have already counted all the commits, and, more\n> importantly, did all the hard work that takes time and makes the\n> progress indicator useful.\n>\n>> \tfor (i = 0; i < oids->nr; i++) {\n>> +\t\tdisplay_progress(progress, ++j);\n>\n> This display_progress() call, however, doesn't seem to be necessary.\n> First, it counts all commits for a second time, resulting in the ~2x\n> difference compared to the actual number of commits, and then causing\n> my confusion. Second, all what this loop is doing is setting a flag\n> in commits that were already looked up and parsed in the above loops.\n> IOW this loop is very fast, and the progress indicator jumps from\n> ~780k right to 1.5M, even on my tiny laptop, so it doesn't need a\n> progress indicator at all.\n\nMakes sense. If this second iteration were also time consuming,\nthen it probably is a good idea to split these into two separate\nphases? \"Counting 1...N\" followed by \"Inspecting 1...N\" or\nsomething like that. Of course, if the latter does not take much\ntime, then doing without any progress indicator is also fine.\n"},{"id":"360318","messageId":"877eindx03.fsf@evledraar.gmail.com","threadId":"49275","inReplyTo":"xmqqa7njd7bi.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v3 1/2] commit-graph write: add progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-12T15:07:24Z","receivedAt":"2018-10-12T15:07:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Oct 12 2018, Junio C Hamano wrote:\n\n> SZEDER Gábor writes:\n>\n>>> \tfor (i = 0; i < oids->nr; i++) {\n>>> +\t\tdisplay_progress(progress, ++j);\n>>> \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n>>>\n>>> \t\tif (commit && !parse_commit(commit))\n>>> @@ -611,19 +624,28 @@ static void close_reachable(struct packed_oid_list *oids)\n>>> \t}\n>>\n>> The above loops have already counted all the commits, and, more\n>> importantly, did all the hard work that takes time and makes the\n>> progress indicator useful.\n>>\n>>> \tfor (i = 0; i < oids->nr; i++) {\n>>> +\t\tdisplay_progress(progress, ++j);\n>>\n>> This display_progress() call, however, doesn't seem to be necessary.\n>> First, it counts all commits for a second time, resulting in the ~2x\n>> difference compared to the actual number of commits, and then causing\n>> my confusion. Second, all what this loop is doing is setting a flag\n>> in commits that were already looked up and parsed in the above loops.\n>> IOW this loop is very fast, and the progress indicator jumps from\n>> ~780k right to 1.5M, even on my tiny laptop, so it doesn't need a\n>> progress indicator at all.\n>\n> Makes sense. If this second iteration were also time consuming,\n> then it probably is a good idea to split these into two separate\n> phases? \"Counting 1...N\" followed by \"Inspecting 1...N\" or\n> something like that. Of course, if the latter does not take much\n> time, then doing without any progress indicator is also fine.\n\nThat's a good point. Derrick: If the three loops in close_reachable()\nhad to be split up into different progress stages and given different\nnames what do you think they should be? Now it's \"Annotating commits in\ncommit graph\" for all of them.\n"},{"id":"360319","messageId":"f7a0cbee-863c-61d3-4959-5cec8b43c705@gmail.com","threadId":"49275","inReplyTo":"877eindx03.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 1/2] commit-graph write: add progress output","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-10-12T15:12:36Z","receivedAt":"2018-10-12T15:12:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/12/2018 11:07 AM, Ævar Arnfjörð Bjarmason wrote:\n> On Fri, Oct 12 2018, Junio C Hamano wrote:\n>\n>> Makes sense. If this second iteration were also time consuming,\n>> then it probably is a good idea to split these into two separate\n>> phases? \"Counting 1...N\" followed by \"Inspecting 1...N\" or\n>> something like that. Of course, if the latter does not take much\n>> time, then doing without any progress indicator is also fine.\n> That's a good point. Derrick: If the three loops in close_reachable()\n> had to be split up into different progress stages and given different\n> names what do you think they should be? Now it's \"Annotating commits in\n> commit graph\" for all of them.\n\nThe following is the best I can think of right now:\n\n1. Loading known commits.\n2. Expanding reachable commits.\n3. Clearing commit marks.\n\n-Stolee\n"},{"id":"360539","messageId":"20181015160545.GG19800@szeder.dev","threadId":"49275","inReplyTo":"87a7nke5gq.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 1/2] commit-graph write: add progress output","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-15T16:05:45Z","receivedAt":"2018-10-15T16:05:52Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Oct 11, 2018 at 07:52:21PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, Oct 10 2018, Ævar Arnfjörð Bjarmason wrote:\n> \n> > On Wed, Oct 10 2018, SZEDER Gábor wrote:\n> >\n> >> On Wed, Oct 10, 2018 at 11:56:45PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> >>> On Wed, Oct 10 2018, SZEDER Gábor wrote:\n> >>\n> >>> >> \tfor (i = 0; i < oids->nr; i++) {\n> >>> >> +\t\tdisplay_progress(progress, ++j);\n> >>>\n> >>> [...]\n> >>>\n> >>> > This display_progress() call, however, doesn't seem to be necessary.\n> >>> > First, it counts all commits for a second time, resulting in the ~2x\n> >>> > difference compared to the actual number of commits, and then causing\n> >>> > my confusion. Second, all what this loop is doing is setting a flag\n> >>> > in commits that were already looked up and parsed in the above loops.\n> >>> > IOW this loop is very fast, and the progress indicator jumps from\n> >>> > ~780k right to 1.5M, even on my tiny laptop, so it doesn't need a\n> >>> > progress indicator at all.\n\n> Hrm, actually reading this again your initial post says we end up with a\n> 2x difference v.s. the number of commits, but it's actually 3x.\n\nWell, it depends on how you create the commit-graph and on the repo as\nwell, I guess. I run 'git commit-graph write --reachable' in a repo\ncreated by 'git clone --single-branch ...', and in that case the\ndifference is only ~2x (the first loop in close_reachable() has as\nmany iterations as the number of refs). If the repo were to contain\ntwice as many refs as commits, then the difference could be as high as\n4x.\n\nHowever, I think I might have noticed an other progress counting issue\nas well, will get back to it later but first I have to get my numbers\nstraight.\n\n> The loop\n> that has a rather trivial runtime comparatively is the 3x, but the 2x\n> loop takes a notable amount of time. So e.g. on git.git:\n> \n> $ git rev-list --all | wc -l; ~/g/git/git commit-graph write\n> 166678\n> Annotating commits in commit graph: 518463, done.\n> Computing commit graph generation numbers: 100% (172685/172685), done.\n"},{"id":"360543","messageId":"20181015165447.GH19800@szeder.dev","threadId":"49275","inReplyTo":"20180917153336.2280-2-avarab@gmail.com","subject":"Re: [PATCH v3 1/2] commit-graph write: add progress output","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-15T16:54:47Z","receivedAt":"2018-10-15T16:54:54Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Sep 17, 2018 at 03:33:35PM +0000, Ævar Arnfjörð Bjarmason wrote:\n\n> @@ -560,6 +563,9 @@ static int add_packed_commits(const struct object_id *oid,\n> \toff_t offset = nth_packed_object_offset(pack, pos);\n> \tstruct object_info oi = OBJECT_INFO_INIT;\n> \n> +\tif (list->progress)\n> +\t\tdisplay_progress(list->progress, ++list->progress_done);\n\nNote that add_packed_commits() is used as a callback function for\nfor_each_object_in_pack() (with '--stdin-packs') or\nfor_each_packed_object() (no options), i.e. this will count the number\nof objects, not commits:\n\n $ git rev-list --all |wc -l\n 768524\n $ git rev-list --objects --all |wc -l\n 6130295\n # '--count --objects' together didn't work as expected.\n $ time ~/src/git/git commit-graph write\n Finding commits for commit graph: 6130295, done.\n Annotating commits in commit graph: 2305572, done.\n Computing commit graph generation numbers: 100% (768524/768524), done.\n\n(Now I also see the 3x difference in the \"Annotating commits\" counter\nthat you mentioned.)\n\nI see two options:\n\n - Provide a different title for this progress counter, e.g.\n \"Scanning objects for c-g\", or \"Processing objects...\", or\n something else that says \"objects\" instead of \"commits\".\n\n - Move this condition and display_progress() call to the end of the\n function, so it will only count commits, not any other objects.\n (As far as I understand both for_each_object_in_pack() and\n for_each_packed_object() iterate in pack .idx order, i.e. it's\n essentially random. This means that commit objects should be\n distributed evenly among other kinds of objects, so we don't have\n to worry about the counter stalling for a long stretch of\n consecutive non-commit objects. At least in theory.)\n\n\n\n"},{"id":"363645","messageId":"20181119160207.GU30222@szeder.dev","threadId":"49275","inReplyTo":"20181015165447.GH19800@szeder.dev","subject":"Re: [PATCH v3 1/2] commit-graph write: add progress output","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-11-19T16:02:07Z","receivedAt":"2018-11-19T16:02:13Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Ping?\n\nWe are at -rc0, this progress output is a new feature since v2.19.0,\nand the numbers shown are still way off.\n\n\nOn Mon, Oct 15, 2018 at 06:54:47PM +0200, SZEDER Gábor wrote:\n> On Mon, Sep 17, 2018 at 03:33:35PM +0000, Ævar Arnfjörð Bjarmason wrote:\n> \n> > @@ -560,6 +563,9 @@ static int add_packed_commits(const struct object_id *oid,\n> > \toff_t offset = nth_packed_object_offset(pack, pos);\n> > \tstruct object_info oi = OBJECT_INFO_INIT;\n> > \n> > +\tif (list->progress)\n> > +\t\tdisplay_progress(list->progress, ++list->progress_done);\n> \n> Note that add_packed_commits() is used as a callback function for\n> for_each_object_in_pack() (with '--stdin-packs') or\n> for_each_packed_object() (no options), i.e. this will count the number\n> of objects, not commits:\n> \n> $ git rev-list --all |wc -l\n> 768524\n> $ git rev-list --objects --all |wc -l\n> 6130295\n> # '--count --objects' together didn't work as expected.\n> $ time ~/src/git/git commit-graph write\n> Finding commits for commit graph: 6130295, done.\n> Annotating commits in commit graph: 2305572, done.\n> Computing commit graph generation numbers: 100% (768524/768524), done.\n> \n> (Now I also see the 3x difference in the \"Annotating commits\" counter\n> that you mentioned.)\n> \n> I see two options:\n> \n> - Provide a different title for this progress counter, e.g.\n> \"Scanning objects for c-g\", or \"Processing objects...\", or\n> something else that says \"objects\" instead of \"commits\".\n> \n> - Move this condition and display_progress() call to the end of the\n> function, so it will only count commits, not any other objects.\n> (As far as I understand both for_each_object_in_pack() and\n> for_each_packed_object() iterate in pack .idx order, i.e. it's\n> essentially random. This means that commit objects should be\n> distributed evenly among other kinds of objects, so we don't have\n> to worry about the counter stalling for a long stretch of\n> consecutive non-commit objects. At least in theory.)\n> \n> \n> \n"},{"id":"363667","messageId":"20181119202300.18670-1-avarab@gmail.com","threadId":"49275","inReplyTo":"20181119160207.GU30222@szeder.dev","subject":"[PATCH] commit-graph: split up close_reachable() progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-19T20:23:00Z","receivedAt":"2018-11-19T20:23:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Amend the progress output added in 7b0f229222 (\"commit-graph write:\nadd progress output\", 2018-09-17) so that the total numbers it reports\naren't higher than the total number of commits anymore. See [1] for a\nbug report pointing that out.\n\nWhen I added this I wasn't intending to provide an accurate count, but\njust have some progress output to show the user the command wasn't\nhanging[2]. But since we are showing numbers, let's make them\naccurate. The progress descriptions were suggested by Derrick Stolee\nin [3].\n\nAs noted in [2] we are unlikely to show anything except the \"Expanding\nreachable...\" message even on fairly large repositories such as\nlinux.git. On a test repository I have with north of 7 million commits\nall of these are displayed. Two of them don't show up for long, but as\nnoted in [5] future-proofing this for if the loops become more\nexpensive in the future makes sense.\n\n1. https://public-inbox.org/git/20181010203738.GE23446@szeder.dev/\n2. https://public-inbox.org/git/87pnwhea8y.fsf@evledraar.gmail.com/\n3. https://public-inbox.org/git/f7a0cbee-863c-61d3-4959-5cec8b43c705@gmail.com/\n4. https://public-inbox.org/git/20181015160545.GG19800@szeder.dev/\n5. https://public-inbox.org/git/87murle8da.fsf@evledraar.gmail.com/\n\nReported-by: SZEDER Gábor \nHelped-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n\nOn Mon, Nov 19 2018, SZEDER Gábor wrote:\n\n> Ping?\n>\n> We are at -rc0, this progress output is a new feature since v2.19.0,\n> and the numbers shown are still way off.\n\nI was under the impression after your\nhttps://public-inbox.org/git/20181015160545.GG19800@szeder.dev/ that\nyou were going to do some more digging & report back, so I put it on\nmy \"waiting for feedback\" list and then forgot about it.\n\nBut here's a patch that should address the issue you pointed out, but\nI don't know if it fixes whatever you were alluding to in the linked\nE-Mail above.\n\n commit-graph.c | 13 ++++++++++---\n 1 file changed, 10 insertions(+), 3 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 40c855f185..9c0d6914be 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -641,26 +641,29 @@ static void add_missing_parents(struct packed_oid_list *oids, struct commit *com\n \n static void close_reachable(struct packed_oid_list *oids, int report_progress)\n {\n-\tint i;\n+\tint i, j;\n \tstruct commit *commit;\n \tstruct progress *progress = NULL;\n-\tint j = 0;\n \n \tif (report_progress)\n \t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Annotating commits in commit graph\"), 0);\n+\t\t\t_(\"Loading known commits in commit graph\"), j = 0);\n \tfor (i = 0; i < oids->nr; i++) {\n \t\tdisplay_progress(progress, ++j);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n \t\tif (commit)\n \t\t\tcommit->object.flags |= UNINTERESTING;\n \t}\n+\tstop_progress(&progress);\n \n \t/*\n \t * As this loop runs, oids->nr may grow, but not more\n \t * than the number of missing commits in the reachable\n \t * closure.\n \t */\n+\tif (report_progress)\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Expanding reachable commits in commit graph\"), j = 0);\n \tfor (i = 0; i < oids->nr; i++) {\n \t\tdisplay_progress(progress, ++j);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n@@ -668,7 +671,11 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n \t\tif (commit && !parse_commit(commit))\n \t\t\tadd_missing_parents(oids, commit);\n \t}\n+\tstop_progress(&progress);\n \n+\tif (report_progress)\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Clearing commit marks in commit graph\"), j = 0);\n \tfor (i = 0; i < oids->nr; i++) {\n \t\tdisplay_progress(progress, ++j);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n-- \n2.19.1.1182.g4ecb1133ce\n\n"},{"id":"363670","messageId":"d7b3ca40-1f23-7264-4fd5-890abe908c2c@gmail.com","threadId":"49275","inReplyTo":"20181119202300.18670-1-avarab@gmail.com","subject":"Re: [PATCH] commit-graph: split up close_reachable() progress output","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-19T20:38:05Z","receivedAt":"2018-11-19T20:38:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/19/2018 3:23 PM, Ævar Arnfjörð Bjarmason wrote:\n> +\tif (report_progress)\n> +\t\tprogress = start_delayed_progress(\n> +\t\t\t_(\"Expanding reachable commits in commit graph\"), j = 0);\n\nThis should be the only one that shows up in all but the very largest of \nrepositories.\n\nLGTM.\n\nThanks,\n-Stolee\n"},{"id":"363681","messageId":"20181119225720.GV30222@szeder.dev","threadId":"49275","inReplyTo":"20181119202300.18670-1-avarab@gmail.com","subject":"Re: [PATCH] commit-graph: split up close_reachable() progress output","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-11-19T22:57:20Z","receivedAt":"2018-11-19T22:57:27Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Nov 19, 2018 at 08:23:00PM +0000, Ævar Arnfjörð Bjarmason wrote:\n> Amend the progress output added in 7b0f229222 (\"commit-graph write:\n> add progress output\", 2018-09-17) so that the total numbers it reports\n> aren't higher than the total number of commits anymore. See [1] for a\n> bug report pointing that out.\n\nPlease make the commit message more self-contained, i.e. describe the\nissue this patch fixes in more detail, so readers won't have to follow\nlinks to understand the problem.\n\n> When I added this I wasn't intending to provide an accurate count, but\n> just have some progress output to show the user the command wasn't\n> hanging[2]. But since we are showing numbers, let's make them\n> accurate. The progress descriptions were suggested by Derrick Stolee\n> in [3].\n> \n> As noted in [2] we are unlikely to show anything except the \"Expanding\n> reachable...\" message even on fairly large repositories such as\n> linux.git. On a test repository I have with north of 7 million commits\n> all of these are displayed. Two of them don't show up for long, but as\n> noted in [5] future-proofing this for if the loops become more\n> expensive in the future makes sense.\n\nIn my opinion this is rather one of those \"we'll cross that bridge\nwhen (or if ever) we get there\" situations.\n\n> 1. https://public-inbox.org/git/20181010203738.GE23446@szeder.dev/\n> 2. https://public-inbox.org/git/87pnwhea8y.fsf@evledraar.gmail.com/\n> 3. https://public-inbox.org/git/f7a0cbee-863c-61d3-4959-5cec8b43c705@gmail.com/\n> 4. https://public-inbox.org/git/20181015160545.GG19800@szeder.dev/\n> 5. https://public-inbox.org/git/87murle8da.fsf@evledraar.gmail.com/\n> \n> Reported-by: SZEDER Gábor \n> Helped-by: Derrick Stolee \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> \n> On Mon, Nov 19 2018, SZEDER Gábor wrote:\n> \n> > Ping?\n> >\n> > We are at -rc0, this progress output is a new feature since v2.19.0,\n> > and the numbers shown are still way off.\n> \n> I was under the impression after your\n> https://public-inbox.org/git/20181015160545.GG19800@szeder.dev/ that\n> you were going to do some more digging & report back, so I put it on\n> my \"waiting for feedback\" list and then forgot about it.\n\nNo, after I managed to \"get my numbers straight\" I sent another bug\nreport in\n\n https://public-inbox.org/git/20181015165447.GH19800@szeder.dev/\n\nbut as a reply to your original patch. Sorry about the confusion.\n\n> But here's a patch that should address the issue you pointed out, but\n> I don't know if it fixes whatever you were alluding to in the linked\n> E-Mail above.\n\nI'm afraid this patch doesn't address that issue, as it's limited to\nclose_reachable(), and that issue is related to the progress output in\nadd_packed_commits().\n\n> commit-graph.c | 13 ++++++++++---\n> 1 file changed, 10 insertions(+), 3 deletions(-)\n> \n> diff --git a/commit-graph.c b/commit-graph.c\n> index 40c855f185..9c0d6914be 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -641,26 +641,29 @@ static void add_missing_parents(struct packed_oid_list *oids, struct commit *com\n> \n> static void close_reachable(struct packed_oid_list *oids, int report_progress)\n> {\n> -\tint i;\n> +\tint i, j;\n> \tstruct commit *commit;\n> \tstruct progress *progress = NULL;\n> -\tint j = 0;\n> \n> \tif (report_progress)\n> \t\tprogress = start_delayed_progress(\n> -\t\t\t_(\"Annotating commits in commit graph\"), 0);\n> +\t\t\t_(\"Loading known commits in commit graph\"), j = 0);\n> \tfor (i = 0; i < oids->nr; i++) {\n> \t\tdisplay_progress(progress, ++j);\n> \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n> \t\tif (commit)\n> \t\t\tcommit->object.flags |= UNINTERESTING;\n> \t}\n> +\tstop_progress(&progress);\n> \n> \t/*\n> \t * As this loop runs, oids->nr may grow, but not more\n> \t * than the number of missing commits in the reachable\n> \t * closure.\n> \t */\n> +\tif (report_progress)\n> +\t\tprogress = start_delayed_progress(\n> +\t\t\t_(\"Expanding reachable commits in commit graph\"), j = 0);\n> \tfor (i = 0; i < oids->nr; i++) {\n> \t\tdisplay_progress(progress, ++j);\n> \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n> @@ -668,7 +671,11 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n> \t\tif (commit && !parse_commit(commit))\n> \t\t\tadd_missing_parents(oids, commit);\n> \t}\n> +\tstop_progress(&progress);\n> \n> +\tif (report_progress)\n> +\t\tprogress = start_delayed_progress(\n> +\t\t\t_(\"Clearing commit marks in commit graph\"), j = 0);\n> \tfor (i = 0; i < oids->nr; i++) {\n> \t\tdisplay_progress(progress, ++j);\n> \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n> -- \n> 2.19.1.1182.g4ecb1133ce\n> \n"},{"id":"363746","messageId":"20181120150443.12997-1-avarab@gmail.com","threadId":"49275","inReplyTo":"20181119225720.GV30222@szeder.dev","subject":"[PATCH 0/6] commit-graph write: progress output improvements","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-20T15:04:37Z","receivedAt":"2018-11-20T15:04:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This replaces my \"commit-graph: split up close_reachable() progress\noutput\". We could still do something like that, but I think this makes\nmore sense, and also plugs some missing holes in the progress\noutput. See 6/6 for what the end-state is.\n\nI believe this addresses SZEDER Gábor's concerns (thanks\nb.t.w.!). I.e. now it should be clear to the user at each step if\nwe're counting objects, or just as in the case of close_reachable()\ndoing some X amount of work without any particular relation to the\nnumber of objects or commits.\n\nÆvar Arnfjörð Bjarmason (6):\n commit-graph write: rephrase confusing progress output\n commit-graph write: add more progress output\n commit-graph write: show progress for object search\n commit-graph write: add more describing progress output\n commit-graph write: remove empty line for readability\n commit-graph write: add even more progress output\n\n commit-graph.c | 98 ++++++++++++++++++++++++++++++++++++++++----------\n 1 file changed, 79 insertions(+), 19 deletions(-)\n\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363747","messageId":"20181120150443.12997-2-avarab@gmail.com","threadId":"49275","inReplyTo":"20181119225720.GV30222@szeder.dev","subject":"[PATCH 1/6] commit-graph write: rephrase confusing progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-20T15:04:38Z","receivedAt":"2018-11-20T15:04:58Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Rephrase the title shown for the progress output emitted by\nclose_reachable(). The message I added in 7b0f229222 (\"commit-graph\nwrite: add progress output\", 2018-09-17) gave the impression that it\nwould count up to the number of commit objects.\n\nBut that's not what the number means. It just represents the work\nwe're doing in several for-loops to do various work before the graph\nis written out. So let's just say \"Annotating commit graph\", that\ntitle makes no such promises, and we can add other loops here in the\nfuture and still consistently show progress output.\n\nSee [1] for the initial bug report & subsequent discussion about other\napproaching to solving this.\n\n1. https://public-inbox.org/git/20181015165447.GH19800@szeder.dev/\n\nReported-by: SZEDER Gábor \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 40c855f185..e6d0d7722b 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -648,7 +648,7 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n \n \tif (report_progress)\n \t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Annotating commits in commit graph\"), 0);\n+\t\t\t_(\"Annotating commit graph\"), 0);\n \tfor (i = 0; i < oids->nr; i++) {\n \t\tdisplay_progress(progress, ++j);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363748","messageId":"20181120150443.12997-3-avarab@gmail.com","threadId":"49275","inReplyTo":"20181119225720.GV30222@szeder.dev","subject":"[PATCH 2/6] commit-graph write: add more progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-20T15:04:39Z","receivedAt":"2018-11-20T15:05:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add more progress output to the output already added in\n7b0f229222 (\"commit-graph write: add progress output\", 2018-09-17).\n\nAs noted in that commit most of the progress output isn't displayed on\nsmall repositories, but before this change we'd noticeably hang for\n2-3 seconds at the end on medium sized repositories such as linux.git.\n\nNow we'll instead show output like this, and have no human-observable\npoint at which we're not producing progress output:\n\n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph: 6418991, done.\n Computing commit graph generation numbers: 100% (797205/797205), done.\n Writing out commit graph chunks: 2399861, done.\n\nThis \"graph chunks\" number is not meant to be meaningful to the user,\nbut just to show that we're doing work and the command isn't\nhanging.\n\nOn a much larger in-house repository I have we'll show (note how we\nalso say \"Annotating[...]\"):\n\n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph: 48271163, done.\n Annotating commit graph: 21424536, done.\n Computing commit graph generation numbers: 100% (7141512/7141512), done.\n Writing out commit graph chunks: 21424913, done.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 47 ++++++++++++++++++++++++++++++++++++++---------\n 1 file changed, 38 insertions(+), 9 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex e6d0d7722b..afce20dd4d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -433,7 +433,9 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n \t\t\t\t struct commit **commits,\n-\t\t\t\t int nr_commits)\n+\t\t\t\t int nr_commits,\n+\t\t\t\t struct progress *progress,\n+\t\t\t\t uint64_t *progress_cnt)\n {\n \tint i, count = 0;\n \tstruct commit **list = commits;\n@@ -445,6 +447,8 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n \t */\n \tfor (i = 0; i < 256; i++) {\n \t\twhile (count < nr_commits) {\n+\t\t\tif (progress)\n+\t\t\t\tdisplay_progress(progress, ++*progress_cnt);\n \t\t\tif ((*list)->object.oid.hash[0] != i)\n \t\t\t\tbreak;\n \t\t\tcount++;\n@@ -456,12 +460,17 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n }\n \n static void write_graph_chunk_oids(struct hashfile *f, int hash_len,\n-\t\t\t\t struct commit **commits, int nr_commits)\n+\t\t\t\t struct commit **commits, int nr_commits,\n+\t\t\t\t struct progress *progress,\n+\t\t\t\t uint64_t *progress_cnt)\n {\n \tstruct commit **list = commits;\n \tint count;\n-\tfor (count = 0; count < nr_commits; count++, list++)\n+\tfor (count = 0; count < nr_commits; count++, list++) {\n+\t\tif (progress)\n+\t\t\tdisplay_progress(progress, ++*progress_cnt);\n \t\thashwrite(f, (*list)->object.oid.hash, (int)hash_len);\n+\t}\n }\n \n static const unsigned char *commit_to_sha1(size_t index, void *table)\n@@ -471,7 +480,9 @@ static const unsigned char *commit_to_sha1(size_t index, void *table)\n }\n \n static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n-\t\t\t\t struct commit **commits, int nr_commits)\n+\t\t\t\t struct commit **commits, int nr_commits,\n+\t\t\t\t struct progress *progress,\n+\t\t\t\t uint64_t *progress_cnt)\n {\n \tstruct commit **list = commits;\n \tstruct commit **last = commits + nr_commits;\n@@ -482,6 +493,9 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\tint edge_value;\n \t\tuint32_t packedDate[2];\n \n+\t\tif (progress)\n+\t\t\tdisplay_progress(progress, ++*progress_cnt);\n+\n \t\tparse_commit(*list);\n \t\thashwrite(f, get_commit_tree_oid(*list)->hash, hash_len);\n \n@@ -542,7 +556,9 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \n static void write_graph_chunk_large_edges(struct hashfile *f,\n \t\t\t\t\t struct commit **commits,\n-\t\t\t\t\t int nr_commits)\n+\t\t\t\t\t int nr_commits,\n+\t\t\t\t\t struct progress *progress,\n+\t\t\t\t\t uint64_t *progress_cnt)\n {\n \tstruct commit **list = commits;\n \tstruct commit **last = commits + nr_commits;\n@@ -566,6 +582,9 @@ static void write_graph_chunk_large_edges(struct hashfile *f,\n \t\t\t\t\t\t nr_commits,\n \t\t\t\t\t\t commit_to_sha1);\n \n+\t\t\tif (progress)\n+\t\t\t\tdisplay_progress(progress, ++*progress_cnt);\n+\n \t\t\tif (edge_value < 0)\n \t\t\t\tedge_value = GRAPH_PARENT_MISSING;\n \t\t\telse if (!parent->next)\n@@ -764,6 +783,7 @@ void write_commit_graph(const char *obj_dir,\n \tint num_extra_edges;\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n+\tuint64_t progress_cnt;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n@@ -937,10 +957,19 @@ void write_commit_graph(const char *obj_dir,\n \t\thashwrite(f, chunk_write, 12);\n \t}\n \n-\twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n-\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n-\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n-\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n+\tif (report_progress)\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Writing out commit graph chunks\"),\n+\t\t\tprogress_cnt = 0);\n+\twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress,\n+\t\t\t\t &progress_cnt);\n+\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr,\n+\t\t\t progress, &progress_cnt);\n+\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr,\n+\t\t\t progress, &progress_cnt);\n+\twrite_graph_chunk_large_edges(f, commits.list, commits.nr, progress,\n+\t\t\t\t &progress_cnt);\n+\tstop_progress(&progress);\n \n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363749","messageId":"20181120150443.12997-4-avarab@gmail.com","threadId":"49275","inReplyTo":"20181119225720.GV30222@szeder.dev","subject":"[PATCH 3/6] commit-graph write: show progress for object search","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-20T15:04:40Z","receivedAt":"2018-11-20T15:05:01Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Show the percentage progress for the \"Finding commits for commit\ngraph\" phase for the common case where we're operating on all packs in\nthe repository, as \"commit-graph write\" or \"gc\" will do.\n\nBefore we'd emit on e.g. linux.git with \"commit-graph write\":\n\n Finding commits for commit graph: 6418991, done.\n [...]\n\nAnd now:\n\n Finding commits for commit graph: 100% (6418991/6418991), done.\n [...]\n\nSince the commit graph only includes those commits that are\npacked (via for_each_packed_object(...)) the\napproximate_object_count() returns the actual number of objects we're\ngoing to process.\n\nStill, it is possible due to a race with \"gc\" or another process\nmaintaining packs that the number of objects we're going to process is\nlower than what approximate_object_count() reported. In that case we\ndon't want to stop the progress bar short of 100%. So let's make sure\nit snaps to 100% at the end.\n\nThe inverse case is also possible and more likely. I.e. that a new\npack has been added between approximate_object_count() and\nfor_each_packed_object(). In that case the percentage will go beyond\n100%, and we'll do nothing to snap it back to 100% at the end.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 9 +++++++--\n 1 file changed, 7 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex afce20dd4d..4d03f8aa7f 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -784,12 +784,14 @@ void write_commit_graph(const char *obj_dir,\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n \tuint64_t progress_cnt;\n+\tunsigned long approx_nr_objects;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n \n \toids.nr = 0;\n-\toids.alloc = approximate_object_count() / 32;\n+\tapprox_nr_objects = approximate_object_count();\n+\toids.alloc = approx_nr_objects / 32;\n \toids.progress = NULL;\n \toids.progress_done = 0;\n \n@@ -869,8 +871,11 @@ void write_commit_graph(const char *obj_dir,\n \tif (!pack_indexes && !commit_hex) {\n \t\tif (report_progress)\n \t\t\toids.progress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"), 0);\n+\t\t\t\t_(\"Finding commits for commit graph\"),\n+\t\t\t\tapprox_nr_objects);\n \t\tfor_each_packed_object(add_packed_commits, &oids, 0);\n+\t\tif (oids.progress_done < approx_nr_objects)\n+\t\t\tdisplay_progress(oids.progress, approx_nr_objects);\n \t\tstop_progress(&oids.progress);\n \t}\n \n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363751","messageId":"20181120150443.12997-5-avarab@gmail.com","threadId":"49275","inReplyTo":"20181119225720.GV30222@szeder.dev","subject":"[PATCH 4/6] commit-graph write: add more describing progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-20T15:04:41Z","receivedAt":"2018-11-20T15:05:04Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the progress output shown when we're searching for commits to\ninclude in the graph more descriptive. This amends code I added in\n7b0f229222 (\"commit-graph write: add progress output\", 2018-09-17).\n\nNow, on linux.git, we'll emit this sort of output in the various modes\nwe support:\n\n $ git commit-graph write\n Finding commits for commit graph among packed objects: 100% (6418991/6418991), done.\n [...]\n $ git for-each-ref --format='%(objectname)' | git commit-graph write --stdin-commits\n Finding commits for commit graph from 584 ref tips: 100% (584/584), done.\n [...]\n $ (cd .git/objects/pack/ && ls *idx) | git commit-graph write --stdin-pack\n Finding commits for commit graph in 4 packs: 6418991, done.\n [...]\n\nThe middle on of those is going to be the output users will most\ncommonly see, since it'll be emitted when they get the commit graph\nvia gc.writeCommitGraph=true.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 28 +++++++++++++++++++++-------\n 1 file changed, 21 insertions(+), 7 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 4d03f8aa7f..a0aea850f1 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -785,6 +785,7 @@ void write_commit_graph(const char *obj_dir,\n \tstruct progress *progress = NULL;\n \tuint64_t progress_cnt;\n \tunsigned long approx_nr_objects;\n+\tstruct strbuf progress_title = STRBUF_INIT;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n@@ -821,8 +822,12 @@ void write_commit_graph(const char *obj_dir,\n \t\tstrbuf_addf(&packname, \"%s/pack/\", obj_dir);\n \t\tdirlen = packname.len;\n \t\tif (report_progress) {\n-\t\t\toids.progress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"), 0);\n+\t\t\tstrbuf_addf(&progress_title,\n+\t\t\t\t Q_(\"Finding commits for commit graph in %d pack\",\n+\t\t\t\t \"Finding commits for commit graph in %d packs\",\n+\t\t\t\t pack_indexes->nr),\n+\t\t\t\t pack_indexes->nr);\n+\t\t\toids.progress = start_delayed_progress(progress_title.buf, 0);\n \t\t\toids.progress_done = 0;\n \t\t}\n \t\tfor (i = 0; i < pack_indexes->nr; i++) {\n@@ -839,14 +844,20 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tfree(p);\n \t\t}\n \t\tstop_progress(&oids.progress);\n+\t\tstrbuf_reset(&progress_title);\n \t\tstrbuf_release(&packname);\n \t}\n \n \tif (commit_hex) {\n-\t\tif (report_progress)\n-\t\t\tprogress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"),\n-\t\t\t\tcommit_hex->nr);\n+\t\tif (report_progress) {\n+\t\t\tstrbuf_addf(&progress_title,\n+\t\t\t\t Q_(\"Finding commits for commit graph from %d ref tip\",\n+\t\t\t\t \"Finding commits for commit graph from %d ref tips\",\n+\t\t\t\t commit_hex->nr),\n+\t\t\t\t commit_hex->nr);\n+\t\t\tprogress = start_delayed_progress(progress_title.buf,\n+\t\t\t\t\t\t\t commit_hex->nr);\n+\t\t}\n \t\tfor (i = 0; i < commit_hex->nr; i++) {\n \t\t\tconst char *end;\n \t\t\tstruct object_id oid;\n@@ -866,12 +877,13 @@ void write_commit_graph(const char *obj_dir,\n \t\t\t}\n \t\t}\n \t\tstop_progress(&progress);\n+\t\tstrbuf_reset(&progress_title);\n \t}\n \n \tif (!pack_indexes && !commit_hex) {\n \t\tif (report_progress)\n \t\t\toids.progress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"),\n+\t\t\t\t_(\"Finding commits for commit graph among packed objects\"),\n \t\t\t\tapprox_nr_objects);\n \t\tfor_each_packed_object(add_packed_commits, &oids, 0);\n \t\tif (oids.progress_done < approx_nr_objects)\n@@ -976,6 +988,8 @@ void write_commit_graph(const char *obj_dir,\n \t\t\t\t &progress_cnt);\n \tstop_progress(&progress);\n \n+\tstrbuf_release(&progress_title);\n+\n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n \tcommit_lock_file(&lk);\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363750","messageId":"20181120150443.12997-6-avarab@gmail.com","threadId":"49275","inReplyTo":"20181119225720.GV30222@szeder.dev","subject":"[PATCH 5/6] commit-graph write: remove empty line for readability","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-20T15:04:42Z","receivedAt":"2018-11-20T15:05:05Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Remove the empty line between a QSORT(...) and the subsequent oideq()\nfor-loop. This makes it clearer that the QSORT(...) is being done so\nthat we can run the oideq() loop on adjacent OIDs. Amends code added\nin 08fd81c9b6 (\"commit-graph: implement write_commit_graph()\",\n2018-04-02).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex a0aea850f1..d0961e89df 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -894,7 +894,6 @@ void write_commit_graph(const char *obj_dir,\n \tclose_reachable(&oids, report_progress);\n \n \tQSORT(oids.list, oids.nr, commit_compare);\n-\n \tcount_distinct = 1;\n \tfor (i = 1; i < oids.nr; i++) {\n \t\tif (!oideq(&oids.list[i - 1], &oids.list[i]))\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363752","messageId":"20181120150443.12997-7-avarab@gmail.com","threadId":"49275","inReplyTo":"20181119225720.GV30222@szeder.dev","subject":"[PATCH 6/6] commit-graph write: add even more progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-20T15:04:43Z","receivedAt":"2018-11-20T15:05:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add more progress output to sections of code that can collectively\ntake 5-10 seconds on a large enough repository. On a test repository\nwith 7141512 commits (see earlier patches for details) we'll now emit:\n\n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph among packed objects: 100% (50009986/50009986), done.\n Annotating commit graph: 21564240, done.\n Counting distinct commits in commit graph: 100% (7188080/7188080), done.\n Finding extra edges in commit graph: 100% (7188080/7188080), done.\n Computing commit graph generation numbers: 100% (7143635/7143635), done.\n Writing out commit graph chunks: 21431282, done.\n\nWhereas on a medium-sized repository such as linux.git we'll still\nemit output like:\n\n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph among packed objects: 100% (6365328/6365328), done.\n Annotating commit graph: 2391621, done.\n Computing commit graph generation numbers: 100% (797207/797207), done.\n Writing out commit graph chunks: 2399867, done.\n\nThe \"Counting distinct commits in commit graph\" phase will spend most\nof its time paused at \"0/*\" as we QSORT(...) the list. That's not\noptimal, but at least we don't seem to be stalling anymore.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 13 +++++++++++++\n 1 file changed, 13 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d0961e89df..2e2eaa24ca 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -893,12 +893,19 @@ void write_commit_graph(const char *obj_dir,\n \n \tclose_reachable(&oids, report_progress);\n \n+\tif (report_progress)\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Counting distinct commits in commit graph\"),\n+\t\t\toids.nr);\n+\tdisplay_progress(progress, 0); /* TODO: Measure QSORT() progress */\n \tQSORT(oids.list, oids.nr, commit_compare);\n \tcount_distinct = 1;\n \tfor (i = 1; i < oids.nr; i++) {\n+\t\tdisplay_progress(progress, i + 1);\n \t\tif (!oideq(&oids.list[i - 1], &oids.list[i]))\n \t\t\tcount_distinct++;\n \t}\n+\tstop_progress(&progress);\n \n \tif (count_distinct >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n@@ -908,8 +915,13 @@ void write_commit_graph(const char *obj_dir,\n \tALLOC_ARRAY(commits.list, commits.alloc);\n \n \tnum_extra_edges = 0;\n+\tif (report_progress)\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Finding extra edges in commit graph\"),\n+\t\t\toids.nr);\n \tfor (i = 0; i < oids.nr; i++) {\n \t\tint num_parents = 0;\n+\t\tdisplay_progress(progress, i + 1);\n \t\tif (i > 0 && oideq(&oids.list[i - 1], &oids.list[i]))\n \t\t\tcontinue;\n \n@@ -926,6 +938,7 @@ void write_commit_graph(const char *obj_dir,\n \t\tcommits.nr++;\n \t}\n \tnum_chunks = num_extra_edges ? 4 : 3;\n+\tstop_progress(&progress);\n \n \tif (commits.nr >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"too many commits to write graph\"));\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363756","messageId":"20181120165800.GB30222@szeder.dev","threadId":"49275","inReplyTo":"20181120150443.12997-3-avarab@gmail.com","subject":"Re: [PATCH 2/6] commit-graph write: add more progress output","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-11-20T16:58:00Z","receivedAt":"2018-11-20T16:58:08Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Nov 20, 2018 at 03:04:39PM +0000, Ævar Arnfjörð Bjarmason wrote:\n> Add more progress output to the output already added in\n> 7b0f229222 (\"commit-graph write: add progress output\", 2018-09-17).\n> \n> As noted in that commit most of the progress output isn't displayed on\n> small repositories, but before this change we'd noticeably hang for\n> 2-3 seconds at the end on medium sized repositories such as linux.git.\n> \n> Now we'll instead show output like this, and have no human-observable\n> point at which we're not producing progress output:\n> \n> $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n> Finding commits for commit graph: 6418991, done.\n> Computing commit graph generation numbers: 100% (797205/797205), done.\n> Writing out commit graph chunks: 2399861, done.\n> \n> This \"graph chunks\" number is not meant to be meaningful to the user,\n> but just to show that we're doing work and the command isn't\n> hanging.\n> \n> On a much larger in-house repository I have we'll show (note how we\n> also say \"Annotating[...]\"):\n> \n> $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n> Finding commits for commit graph: 48271163, done.\n> Annotating commit graph: 21424536, done.\n> Computing commit graph generation numbers: 100% (7141512/7141512), done.\n> Writing out commit graph chunks: 21424913, done.\n\nThat's a lot of chunks, but according to the specs, there are only 3\nor 4 chunks in a commit-graph file. More on this below.\n\n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> commit-graph.c | 47 ++++++++++++++++++++++++++++++++++++++---------\n> 1 file changed, 38 insertions(+), 9 deletions(-)\n> \n> diff --git a/commit-graph.c b/commit-graph.c\n> index e6d0d7722b..afce20dd4d 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -433,7 +433,9 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n> \n> static void write_graph_chunk_fanout(struct hashfile *f,\n> \t\t\t\t struct commit **commits,\n> -\t\t\t\t int nr_commits)\n> +\t\t\t\t int nr_commits,\n> +\t\t\t\t struct progress *progress,\n> +\t\t\t\t uint64_t *progress_cnt)\n> {\n> \tint i, count = 0;\n> \tstruct commit **list = commits;\n> @@ -445,6 +447,8 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n> \t */\n> \tfor (i = 0; i < 256; i++) {\n> \t\twhile (count < nr_commits) {\n> +\t\t\tif (progress)\n> +\t\t\t\tdisplay_progress(progress, ++*progress_cnt);\n\nThe condition is unnecessary, display_progress() is prepared to deal\nwith a NULL progress pointer. The same applies to all such calls in\nthis patch.\n\n> \t\t\tif ((*list)->object.oid.hash[0] != i)\n> \t\t\t\tbreak;\n> \t\t\tcount++;\n> @@ -456,12 +460,17 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n> }\n> \n> static void write_graph_chunk_oids(struct hashfile *f, int hash_len,\n> -\t\t\t\t struct commit **commits, int nr_commits)\n> +\t\t\t\t struct commit **commits, int nr_commits,\n> +\t\t\t\t struct progress *progress,\n> +\t\t\t\t uint64_t *progress_cnt)\n> {\n> \tstruct commit **list = commits;\n> \tint count;\n> -\tfor (count = 0; count < nr_commits; count++, list++)\n> +\tfor (count = 0; count < nr_commits; count++, list++) {\n> +\t\tif (progress)\n> +\t\t\tdisplay_progress(progress, ++*progress_cnt);\n> \t\thashwrite(f, (*list)->object.oid.hash, (int)hash_len);\n> +\t}\n> }\n> \n> static const unsigned char *commit_to_sha1(size_t index, void *table)\n> @@ -471,7 +480,9 @@ static const unsigned char *commit_to_sha1(size_t index, void *table)\n> }\n> \n> static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n> -\t\t\t\t struct commit **commits, int nr_commits)\n> +\t\t\t\t struct commit **commits, int nr_commits,\n> +\t\t\t\t struct progress *progress,\n> +\t\t\t\t uint64_t *progress_cnt)\n> {\n> \tstruct commit **list = commits;\n> \tstruct commit **last = commits + nr_commits;\n> @@ -482,6 +493,9 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n> \t\tint edge_value;\n> \t\tuint32_t packedDate[2];\n> \n> +\t\tif (progress)\n> +\t\t\tdisplay_progress(progress, ++*progress_cnt);\n> +\n> \t\tparse_commit(*list);\n> \t\thashwrite(f, get_commit_tree_oid(*list)->hash, hash_len);\n> \n> @@ -542,7 +556,9 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n> \n> static void write_graph_chunk_large_edges(struct hashfile *f,\n> \t\t\t\t\t struct commit **commits,\n> -\t\t\t\t\t int nr_commits)\n> +\t\t\t\t\t int nr_commits,\n> +\t\t\t\t\t struct progress *progress,\n> +\t\t\t\t\t uint64_t *progress_cnt)\n> {\n> \tstruct commit **list = commits;\n> \tstruct commit **last = commits + nr_commits;\n> @@ -566,6 +582,9 @@ static void write_graph_chunk_large_edges(struct hashfile *f,\n> \t\t\t\t\t\t nr_commits,\n> \t\t\t\t\t\t commit_to_sha1);\n> \n> +\t\t\tif (progress)\n> +\t\t\t\tdisplay_progress(progress, ++*progress_cnt);\n> +\n> \t\t\tif (edge_value < 0)\n> \t\t\t\tedge_value = GRAPH_PARENT_MISSING;\n> \t\t\telse if (!parent->next)\n> @@ -764,6 +783,7 @@ void write_commit_graph(const char *obj_dir,\n> \tint num_extra_edges;\n> \tstruct commit_list *parent;\n> \tstruct progress *progress = NULL;\n> +\tuint64_t progress_cnt;\n> \n> \tif (!commit_graph_compatible(the_repository))\n> \t\treturn;\n> @@ -937,10 +957,19 @@ void write_commit_graph(const char *obj_dir,\n> \t\thashwrite(f, chunk_write, 12);\n> \t}\n> \n> -\twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n> -\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n> -\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n> -\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n> +\tif (report_progress)\n> +\t\tprogress = start_delayed_progress(\n> +\t\t\t_(\"Writing out commit graph chunks\"),\n> +\t\t\tprogress_cnt = 0);\n\nFirst, this is an unusual place to set a variable.\n\nSecond, as mentioned above, there are only 3 or 4 chunks, therefore I\nthink this should only say \"Writing out commit graph\".\n\nFinally, each of the write_graph_chunk_*() functions called below\niterate over all commits, so we know and thus can show the total in\nadvance.\n\nSo how about something like the patch below on top? Note that I had\nto shift two display_progress() calls a couple of lines, because\notherwise the numbers didn't add up.\n\nJust to get you thinking and to have something to try out, but I saw a\nbit of weirdness while at it, and want to look into it, but now I've\ngot to go...\n\n\n> +\twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress,\n> +\t\t\t\t &progress_cnt);\n> +\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr,\n> +\t\t\t progress, &progress_cnt);\n> +\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr,\n> +\t\t\t progress, &progress_cnt);\n> +\twrite_graph_chunk_large_edges(f, commits.list, commits.nr, progress,\n> +\t\t\t\t &progress_cnt);\n> +\tstop_progress(&progress);\n> \n> \tclose_commit_graph(the_repository);\n> \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n\n\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 2e2eaa24ca..2f3417db32 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -447,10 +447,9 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n \t */\n \tfor (i = 0; i < 256; i++) {\n \t\twhile (count < nr_commits) {\n-\t\t\tif (progress)\n-\t\t\t\tdisplay_progress(progress, ++*progress_cnt);\n \t\t\tif ((*list)->object.oid.hash[0] != i)\n \t\t\t\tbreak;\n+\t\t\tdisplay_progress(progress, ++*progress_cnt);\n \t\t\tcount++;\n \t\t\tlist++;\n \t\t}\n@@ -467,8 +466,7 @@ static void write_graph_chunk_oids(struct hashfile *f, int hash_len,\n \tstruct commit **list = commits;\n \tint count;\n \tfor (count = 0; count < nr_commits; count++, list++) {\n-\t\tif (progress)\n-\t\t\tdisplay_progress(progress, ++*progress_cnt);\n+\t\tdisplay_progress(progress, ++*progress_cnt);\n \t\thashwrite(f, (*list)->object.oid.hash, (int)hash_len);\n \t}\n }\n@@ -493,8 +491,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\tint edge_value;\n \t\tuint32_t packedDate[2];\n \n-\t\tif (progress)\n-\t\t\tdisplay_progress(progress, ++*progress_cnt);\n+\t\tdisplay_progress(progress, ++*progress_cnt);\n \n \t\tparse_commit(*list);\n \t\thashwrite(f, get_commit_tree_oid(*list)->hash, hash_len);\n@@ -570,6 +567,8 @@ static void write_graph_chunk_large_edges(struct hashfile *f,\n \t\t parent = parent->next)\n \t\t\tnum_parents++;\n \n+\t\tdisplay_progress(progress, ++*progress_cnt);\n+\n \t\tif (num_parents <= 2) {\n \t\t\tlist++;\n \t\t\tcontinue;\n@@ -582,9 +581,6 @@ static void write_graph_chunk_large_edges(struct hashfile *f,\n \t\t\t\t\t\t nr_commits,\n \t\t\t\t\t\t commit_to_sha1);\n \n-\t\t\tif (progress)\n-\t\t\t\tdisplay_progress(progress, ++*progress_cnt);\n-\n \t\t\tif (edge_value < 0)\n \t\t\t\tedge_value = GRAPH_PARENT_MISSING;\n \t\t\telse if (!parent->next)\n@@ -986,10 +982,11 @@ void write_commit_graph(const char *obj_dir,\n \t\thashwrite(f, chunk_write, 12);\n \t}\n \n-\tif (report_progress)\n+\tif (report_progress) {\n \t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Writing out commit graph chunks\"),\n-\t\t\tprogress_cnt = 0);\n+\t\t\t_(\"Writing out commit graph\"), 4 * commits.nr);\n+\t\tprogress_cnt = 0;\n+\t}\n \twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress,\n \t\t\t\t &progress_cnt);\n \twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr,\n"},{"id":"363768","messageId":"20181120195027.3260-1-avarab@gmail.com","threadId":"49275","inReplyTo":"20181120165800.GB30222@szeder.dev","subject":"[PATCH v2 0/6] commit-graph write: progress output improvements","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-20T19:50:21Z","receivedAt":"2018-11-20T19:50:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fixes issues SZEDER raised with v1, except displaying an accurate ETA\nin write_graph_*(). As noted in 2/6 I don't think it's worth it, I\njust adjusted the message instead.\n\nÆvar Arnfjörð Bjarmason (6):\n commit-graph write: rephrase confusing progress output\n commit-graph write: add more progress output\n commit-graph write: show progress for object search\n commit-graph write: add more describing progress output\n commit-graph write: remove empty line for readability\n commit-graph write: add even more progress output\n\n commit-graph.c | 92 +++++++++++++++++++++++++++++++++++++++-----------\n 1 file changed, 73 insertions(+), 19 deletions(-)\n\nRange-diff:\n1: 751d3a7561 ! 1: 093c63e99f commit-graph write: add more progress output\n a => b | 0\n 1 file changed, 0 insertions(+), 0 deletions(-)\n \n @@ -13,22 +13,30 @@\n point at which we're not producing progress output:\n \n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n - Finding commits for commit graph: 6418991, done.\n - Computing commit graph generation numbers: 100% (797205/797205), done.\n - Writing out commit graph chunks: 2399861, done.\n + Finding commits for commit graph: 6365492, done.\n + Computing commit graph generation numbers: 100% (797222/797222), done.\n + Writing out commit graph: 2399912, done.\n \n - This \"graph chunks\" number is not meant to be meaningful to the user,\n + This \"writing out\" number is not meant to be meaningful to the user,\n but just to show that we're doing work and the command isn't\n hanging.\n \n + In the current implementation it's approximately 4x the number of\n + commits. As noted in on-list discussion[1] we could add the loops up\n + and show percentage progress here, but I don't think it's worth it. It\n + would make the implementation more complex and harder to maintain for\n + very little gain.\n +\n On a much larger in-house repository I have we'll show (note how we\n also say \"Annotating[...]\"):\n \n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n - Finding commits for commit graph: 48271163, done.\n - Annotating commit graph: 21424536, done.\n - Computing commit graph generation numbers: 100% (7141512/7141512), done.\n - Writing out commit graph chunks: 21424913, done.\n + Finding commits for commit graph: 50026015, done.\n + Annotating commit graph: 21567407, done.\n + Computing commit graph generation numbers: 100% (7144680/7144680), done.\n + Writing out commit graph: 21434417, done.\n +\n + 1. https://public-inbox.org/git/20181120165800.GB30222@szeder.dev/\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n @@ -50,8 +58,7 @@\n \t */\n \tfor (i = 0; i < 256; i++) {\n \t\twhile (count < nr_commits) {\n -+\t\t\tif (progress)\n -+\t\t\t\tdisplay_progress(progress, ++*progress_cnt);\n ++\t\t\tdisplay_progress(progress, ++*progress_cnt);\n \t\t\tif ((*list)->object.oid.hash[0] != i)\n \t\t\t\tbreak;\n \t\t\tcount++;\n @@ -68,8 +75,7 @@\n \tint count;\n -\tfor (count = 0; count < nr_commits; count++, list++)\n +\tfor (count = 0; count < nr_commits; count++, list++) {\n -+\t\tif (progress)\n -+\t\t\tdisplay_progress(progress, ++*progress_cnt);\n ++\t\tdisplay_progress(progress, ++*progress_cnt);\n \t\thashwrite(f, (*list)->object.oid.hash, (int)hash_len);\n +\t}\n }\n @@ -87,15 +93,13 @@\n \tstruct commit **list = commits;\n \tstruct commit **last = commits + nr_commits;\n @@\n + \t\tstruct commit_list *parent;\n \t\tint edge_value;\n \t\tuint32_t packedDate[2];\n ++\t\tdisplay_progress(progress, ++*progress_cnt);\n \n -+\t\tif (progress)\n -+\t\t\tdisplay_progress(progress, ++*progress_cnt);\n -+\n \t\tparse_commit(*list);\n \t\thashwrite(f, get_commit_tree_oid(*list)->hash, hash_len);\n - \n @@\n \n static void write_graph_chunk_large_edges(struct hashfile *f,\n @@ -108,20 +112,18 @@\n \tstruct commit **list = commits;\n \tstruct commit **last = commits + nr_commits;\n @@\n + \t\t\t\t\t\t commits,\n \t\t\t\t\t\t nr_commits,\n \t\t\t\t\t\t commit_to_sha1);\n ++\t\t\tdisplay_progress(progress, ++*progress_cnt);\n \n -+\t\t\tif (progress)\n -+\t\t\t\tdisplay_progress(progress, ++*progress_cnt);\n -+\n \t\t\tif (edge_value < 0)\n \t\t\t\tedge_value = GRAPH_PARENT_MISSING;\n - \t\t\telse if (!parent->next)\n @@\n \tint num_extra_edges;\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n -+\tuint64_t progress_cnt;\n ++\tuint64_t progress_cnt = 0;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n @@ -135,8 +137,8 @@\n -\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n +\tif (report_progress)\n +\t\tprogress = start_delayed_progress(\n -+\t\t\t_(\"Writing out commit graph chunks\"),\n -+\t\t\tprogress_cnt = 0);\n ++\t\t\t_(\"Writing out commit graph\"),\n ++\t\t\t0);\n +\twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress,\n +\t\t\t\t &progress_cnt);\n +\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr,\n2: d750f0dd16 ! 2: 6c71de9460 commit-graph write: show progress for object search\n a => b | 0\n 1 file changed, 0 insertions(+), 0 deletions(-)\n \n @@ -8,18 +8,17 @@\n \n Before we'd emit on e.g. linux.git with \"commit-graph write\":\n \n - Finding commits for commit graph: 6418991, done.\n + Finding commits for commit graph: 6365492, done.\n [...]\n \n And now:\n \n - Finding commits for commit graph: 100% (6418991/6418991), done.\n + Finding commits for commit graph: 100% (6365492/6365492), done.\n [...]\n \n - Since the commit graph only includes those commits that are\n - packed (via for_each_packed_object(...)) the\n - approximate_object_count() returns the actual number of objects we're\n - going to process.\n + Since the commit graph only includes those commits that are packed\n + (via for_each_packed_object(...)) the approximate_object_count()\n + returns the actual number of objects we're going to process.\n \n Still, it is possible due to a race with \"gc\" or another process\n maintaining packs that the number of objects we're going to process is\n @@ -40,7 +39,7 @@\n @@\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n - \tuint64_t progress_cnt;\n + \tuint64_t progress_cnt = 0;\n +\tunsigned long approx_nr_objects;\n \n \tif (!commit_graph_compatible(the_repository))\n3: a175ab49ff ! 3: c665dbdacb commit-graph write: add more describing progress output\n a => b | 0\n 1 file changed, 0 insertions(+), 0 deletions(-)\n \n @@ -10,18 +10,26 @@\n we support:\n \n $ git commit-graph write\n - Finding commits for commit graph among packed objects: 100% (6418991/6418991), done.\n + Finding commits for commit graph among packed objects: 100% (6365492/6365492), done.\n [...]\n +\n + # Actually we don't emit this since this takes almost no time at\n + # all. But if we did (s/_delayed//) we'd show:\n $ git for-each-ref --format='%(objectname)' | git commit-graph write --stdin-commits\n - Finding commits for commit graph from 584 ref tips: 100% (584/584), done.\n + Finding commits for commit graph from 584 refs: 100% (584/584), done.\n [...]\n +\n $ (cd .git/objects/pack/ && ls *idx) | git commit-graph write --stdin-pack\n - Finding commits for commit graph in 4 packs: 6418991, done.\n + Finding commits for commit graph in 3 packs: 6365492, done.\n [...]\n \n - The middle on of those is going to be the output users will most\n - commonly see, since it'll be emitted when they get the commit graph\n - via gc.writeCommitGraph=true.\n + The middle on of those is going to be the output users might see in\n + practice, since it'll be emitted when they get the commit graph via\n + gc.writeCommitGraph=true. But as noted above you need a really large\n + number of refs for this message to show. It'll show up on a test\n + repository I have with ~165k refs:\n +\n + Finding commits for commit graph from 165203 refs: 100% (165203/165203), done.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n @@ -30,7 +38,7 @@\n +++ b/commit-graph.c\n @@\n \tstruct progress *progress = NULL;\n - \tuint64_t progress_cnt;\n + \tuint64_t progress_cnt = 0;\n \tunsigned long approx_nr_objects;\n +\tstruct strbuf progress_title = STRBUF_INIT;\n \n @@ -66,8 +74,8 @@\n -\t\t\t\tcommit_hex->nr);\n +\t\tif (report_progress) {\n +\t\t\tstrbuf_addf(&progress_title,\n -+\t\t\t\t Q_(\"Finding commits for commit graph from %d ref tip\",\n -+\t\t\t\t \"Finding commits for commit graph from %d ref tips\",\n ++\t\t\t\t Q_(\"Finding commits for commit graph from %d ref\",\n ++\t\t\t\t \"Finding commits for commit graph from %d refs\",\n +\t\t\t\t commit_hex->nr),\n +\t\t\t\t commit_hex->nr);\n +\t\t\tprogress = start_delayed_progress(progress_title.buf,\n4: 4e11c8b2fd = 4: f70fc5045d commit-graph write: remove empty line for readability\n a => b | 0\n 1 file changed, 0 insertions(+), 0 deletions(-)\n \n5: 6fbba22fac ! 5: 2e943fa925 commit-graph write: add even more progress output\n a => b | 0\n 1 file changed, 0 insertions(+), 0 deletions(-)\n \n @@ -4,24 +4,26 @@\n \n Add more progress output to sections of code that can collectively\n take 5-10 seconds on a large enough repository. On a test repository\n - with 7141512 commits (see earlier patches for details) we'll now emit:\n + with I have with ~7 million commits and ~50 million objects we'll now\n + emit:\n \n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n - Finding commits for commit graph among packed objects: 100% (50009986/50009986), done.\n - Annotating commit graph: 21564240, done.\n - Counting distinct commits in commit graph: 100% (7188080/7188080), done.\n - Finding extra edges in commit graph: 100% (7188080/7188080), done.\n - Computing commit graph generation numbers: 100% (7143635/7143635), done.\n - Writing out commit graph chunks: 21431282, done.\n + Finding commits for commit graph among packed objects: 100% (50026015/50026015), done.\n + Annotating commit graph: 21567407, done.\n + Counting distinct commits in commit graph: 100% (7189147/7189147), done.\n + Finding extra edges in commit graph: 100% (7189147/7189147), done.\n + Computing commit graph generation numbers: 100% (7144680/7144680), done.\n + Writing out commit graph: 21434417, done.\n \n - Whereas on a medium-sized repository such as linux.git we'll still\n + Whereas on a medium-sized repository such as linux.git these new\n + progress bars won't have time to kick in and as before and we'll still\n emit output like:\n \n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n - Finding commits for commit graph among packed objects: 100% (6365328/6365328), done.\n - Annotating commit graph: 2391621, done.\n - Computing commit graph generation numbers: 100% (797207/797207), done.\n - Writing out commit graph chunks: 2399867, done.\n + Finding commits for commit graph among packed objects: 100% (6365492/6365492), done.\n + Annotating commit graph: 2391666, done.\n + Computing commit graph generation numbers: 100% (797222/797222), done.\n + Writing out commit graph: 2399912, done.\n \n The \"Counting distinct commits in commit graph\" phase will spend most\n of its time paused at \"0/*\" as we QSORT(...) the list. That's not\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363769","messageId":"20181120195027.3260-2-avarab@gmail.com","threadId":"49275","inReplyTo":"20181120165800.GB30222@szeder.dev","subject":"[PATCH v2 1/6] commit-graph write: rephrase confusing progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-20T19:50:22Z","receivedAt":"2018-11-20T19:50:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Rephrase the title shown for the progress output emitted by\nclose_reachable(). The message I added in 7b0f229222 (\"commit-graph\nwrite: add progress output\", 2018-09-17) gave the impression that it\nwould count up to the number of commit objects.\n\nBut that's not what the number means. It just represents the work\nwe're doing in several for-loops to do various work before the graph\nis written out. So let's just say \"Annotating commit graph\", that\ntitle makes no such promises, and we can add other loops here in the\nfuture and still consistently show progress output.\n\nSee [1] for the initial bug report & subsequent discussion about other\napproaching to solving this.\n\n1. https://public-inbox.org/git/20181015165447.GH19800@szeder.dev/\n\nReported-by: SZEDER Gábor \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 40c855f185..e6d0d7722b 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -648,7 +648,7 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n \n \tif (report_progress)\n \t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Annotating commits in commit graph\"), 0);\n+\t\t\t_(\"Annotating commit graph\"), 0);\n \tfor (i = 0; i < oids->nr; i++) {\n \t\tdisplay_progress(progress, ++j);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363770","messageId":"20181120195027.3260-3-avarab@gmail.com","threadId":"49275","inReplyTo":"20181120165800.GB30222@szeder.dev","subject":"[PATCH v2 2/6] commit-graph write: add more progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-20T19:50:23Z","receivedAt":"2018-11-20T19:50:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add more progress output to the output already added in\n7b0f229222 (\"commit-graph write: add progress output\", 2018-09-17).\n\nAs noted in that commit most of the progress output isn't displayed on\nsmall repositories, but before this change we'd noticeably hang for\n2-3 seconds at the end on medium sized repositories such as linux.git.\n\nNow we'll instead show output like this, and have no human-observable\npoint at which we're not producing progress output:\n\n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph: 6365492, done.\n Computing commit graph generation numbers: 100% (797222/797222), done.\n Writing out commit graph: 2399912, done.\n\nThis \"writing out\" number is not meant to be meaningful to the user,\nbut just to show that we're doing work and the command isn't\nhanging.\n\nIn the current implementation it's approximately 4x the number of\ncommits. As noted in on-list discussion[1] we could add the loops up\nand show percentage progress here, but I don't think it's worth it. It\nwould make the implementation more complex and harder to maintain for\nvery little gain.\n\nOn a much larger in-house repository I have we'll show (note how we\nalso say \"Annotating[...]\"):\n\n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph: 50026015, done.\n Annotating commit graph: 21567407, done.\n Computing commit graph generation numbers: 100% (7144680/7144680), done.\n Writing out commit graph: 21434417, done.\n\n1. https://public-inbox.org/git/20181120165800.GB30222@szeder.dev/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 41 ++++++++++++++++++++++++++++++++---------\n 1 file changed, 32 insertions(+), 9 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex e6d0d7722b..6f6409b292 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -433,7 +433,9 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n \t\t\t\t struct commit **commits,\n-\t\t\t\t int nr_commits)\n+\t\t\t\t int nr_commits,\n+\t\t\t\t struct progress *progress,\n+\t\t\t\t uint64_t *progress_cnt)\n {\n \tint i, count = 0;\n \tstruct commit **list = commits;\n@@ -445,6 +447,7 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n \t */\n \tfor (i = 0; i < 256; i++) {\n \t\twhile (count < nr_commits) {\n+\t\t\tdisplay_progress(progress, ++*progress_cnt);\n \t\t\tif ((*list)->object.oid.hash[0] != i)\n \t\t\t\tbreak;\n \t\t\tcount++;\n@@ -456,12 +459,16 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n }\n \n static void write_graph_chunk_oids(struct hashfile *f, int hash_len,\n-\t\t\t\t struct commit **commits, int nr_commits)\n+\t\t\t\t struct commit **commits, int nr_commits,\n+\t\t\t\t struct progress *progress,\n+\t\t\t\t uint64_t *progress_cnt)\n {\n \tstruct commit **list = commits;\n \tint count;\n-\tfor (count = 0; count < nr_commits; count++, list++)\n+\tfor (count = 0; count < nr_commits; count++, list++) {\n+\t\tdisplay_progress(progress, ++*progress_cnt);\n \t\thashwrite(f, (*list)->object.oid.hash, (int)hash_len);\n+\t}\n }\n \n static const unsigned char *commit_to_sha1(size_t index, void *table)\n@@ -471,7 +478,9 @@ static const unsigned char *commit_to_sha1(size_t index, void *table)\n }\n \n static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n-\t\t\t\t struct commit **commits, int nr_commits)\n+\t\t\t\t struct commit **commits, int nr_commits,\n+\t\t\t\t struct progress *progress,\n+\t\t\t\t uint64_t *progress_cnt)\n {\n \tstruct commit **list = commits;\n \tstruct commit **last = commits + nr_commits;\n@@ -481,6 +490,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\tstruct commit_list *parent;\n \t\tint edge_value;\n \t\tuint32_t packedDate[2];\n+\t\tdisplay_progress(progress, ++*progress_cnt);\n \n \t\tparse_commit(*list);\n \t\thashwrite(f, get_commit_tree_oid(*list)->hash, hash_len);\n@@ -542,7 +552,9 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \n static void write_graph_chunk_large_edges(struct hashfile *f,\n \t\t\t\t\t struct commit **commits,\n-\t\t\t\t\t int nr_commits)\n+\t\t\t\t\t int nr_commits,\n+\t\t\t\t\t struct progress *progress,\n+\t\t\t\t\t uint64_t *progress_cnt)\n {\n \tstruct commit **list = commits;\n \tstruct commit **last = commits + nr_commits;\n@@ -565,6 +577,7 @@ static void write_graph_chunk_large_edges(struct hashfile *f,\n \t\t\t\t\t\t commits,\n \t\t\t\t\t\t nr_commits,\n \t\t\t\t\t\t commit_to_sha1);\n+\t\t\tdisplay_progress(progress, ++*progress_cnt);\n \n \t\t\tif (edge_value < 0)\n \t\t\t\tedge_value = GRAPH_PARENT_MISSING;\n@@ -764,6 +777,7 @@ void write_commit_graph(const char *obj_dir,\n \tint num_extra_edges;\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n+\tuint64_t progress_cnt = 0;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n@@ -937,10 +951,19 @@ void write_commit_graph(const char *obj_dir,\n \t\thashwrite(f, chunk_write, 12);\n \t}\n \n-\twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n-\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n-\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n-\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n+\tif (report_progress)\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Writing out commit graph\"),\n+\t\t\t0);\n+\twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress,\n+\t\t\t\t &progress_cnt);\n+\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr,\n+\t\t\t progress, &progress_cnt);\n+\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr,\n+\t\t\t progress, &progress_cnt);\n+\twrite_graph_chunk_large_edges(f, commits.list, commits.nr, progress,\n+\t\t\t\t &progress_cnt);\n+\tstop_progress(&progress);\n \n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363771","messageId":"20181120195027.3260-4-avarab@gmail.com","threadId":"49275","inReplyTo":"20181120165800.GB30222@szeder.dev","subject":"[PATCH v2 3/6] commit-graph write: show progress for object search","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-20T19:50:24Z","receivedAt":"2018-11-20T19:50:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Show the percentage progress for the \"Finding commits for commit\ngraph\" phase for the common case where we're operating on all packs in\nthe repository, as \"commit-graph write\" or \"gc\" will do.\n\nBefore we'd emit on e.g. linux.git with \"commit-graph write\":\n\n Finding commits for commit graph: 6365492, done.\n [...]\n\nAnd now:\n\n Finding commits for commit graph: 100% (6365492/6365492), done.\n [...]\n\nSince the commit graph only includes those commits that are packed\n(via for_each_packed_object(...)) the approximate_object_count()\nreturns the actual number of objects we're going to process.\n\nStill, it is possible due to a race with \"gc\" or another process\nmaintaining packs that the number of objects we're going to process is\nlower than what approximate_object_count() reported. In that case we\ndon't want to stop the progress bar short of 100%. So let's make sure\nit snaps to 100% at the end.\n\nThe inverse case is also possible and more likely. I.e. that a new\npack has been added between approximate_object_count() and\nfor_each_packed_object(). In that case the percentage will go beyond\n100%, and we'll do nothing to snap it back to 100% at the end.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 9 +++++++--\n 1 file changed, 7 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 6f6409b292..7c1afa4704 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -778,12 +778,14 @@ void write_commit_graph(const char *obj_dir,\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n \tuint64_t progress_cnt = 0;\n+\tunsigned long approx_nr_objects;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n \n \toids.nr = 0;\n-\toids.alloc = approximate_object_count() / 32;\n+\tapprox_nr_objects = approximate_object_count();\n+\toids.alloc = approx_nr_objects / 32;\n \toids.progress = NULL;\n \toids.progress_done = 0;\n \n@@ -863,8 +865,11 @@ void write_commit_graph(const char *obj_dir,\n \tif (!pack_indexes && !commit_hex) {\n \t\tif (report_progress)\n \t\t\toids.progress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"), 0);\n+\t\t\t\t_(\"Finding commits for commit graph\"),\n+\t\t\t\tapprox_nr_objects);\n \t\tfor_each_packed_object(add_packed_commits, &oids, 0);\n+\t\tif (oids.progress_done < approx_nr_objects)\n+\t\t\tdisplay_progress(oids.progress, approx_nr_objects);\n \t\tstop_progress(&oids.progress);\n \t}\n \n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363772","messageId":"20181120195027.3260-5-avarab@gmail.com","threadId":"49275","inReplyTo":"20181120165800.GB30222@szeder.dev","subject":"[PATCH v2 4/6] commit-graph write: add more describing progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-20T19:50:25Z","receivedAt":"2018-11-20T19:50:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the progress output shown when we're searching for commits to\ninclude in the graph more descriptive. This amends code I added in\n7b0f229222 (\"commit-graph write: add progress output\", 2018-09-17).\n\nNow, on linux.git, we'll emit this sort of output in the various modes\nwe support:\n\n $ git commit-graph write\n Finding commits for commit graph among packed objects: 100% (6365492/6365492), done.\n [...]\n\n # Actually we don't emit this since this takes almost no time at\n # all. But if we did (s/_delayed//) we'd show:\n $ git for-each-ref --format='%(objectname)' | git commit-graph write --stdin-commits\n Finding commits for commit graph from 584 refs: 100% (584/584), done.\n [...]\n\n $ (cd .git/objects/pack/ && ls *idx) | git commit-graph write --stdin-pack\n Finding commits for commit graph in 3 packs: 6365492, done.\n [...]\n\nThe middle on of those is going to be the output users might see in\npractice, since it'll be emitted when they get the commit graph via\ngc.writeCommitGraph=true. But as noted above you need a really large\nnumber of refs for this message to show. It'll show up on a test\nrepository I have with ~165k refs:\n\n Finding commits for commit graph from 165203 refs: 100% (165203/165203), done.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 28 +++++++++++++++++++++-------\n 1 file changed, 21 insertions(+), 7 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 7c1afa4704..fd1fd61750 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -779,6 +779,7 @@ void write_commit_graph(const char *obj_dir,\n \tstruct progress *progress = NULL;\n \tuint64_t progress_cnt = 0;\n \tunsigned long approx_nr_objects;\n+\tstruct strbuf progress_title = STRBUF_INIT;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n@@ -815,8 +816,12 @@ void write_commit_graph(const char *obj_dir,\n \t\tstrbuf_addf(&packname, \"%s/pack/\", obj_dir);\n \t\tdirlen = packname.len;\n \t\tif (report_progress) {\n-\t\t\toids.progress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"), 0);\n+\t\t\tstrbuf_addf(&progress_title,\n+\t\t\t\t Q_(\"Finding commits for commit graph in %d pack\",\n+\t\t\t\t \"Finding commits for commit graph in %d packs\",\n+\t\t\t\t pack_indexes->nr),\n+\t\t\t\t pack_indexes->nr);\n+\t\t\toids.progress = start_delayed_progress(progress_title.buf, 0);\n \t\t\toids.progress_done = 0;\n \t\t}\n \t\tfor (i = 0; i < pack_indexes->nr; i++) {\n@@ -833,14 +838,20 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tfree(p);\n \t\t}\n \t\tstop_progress(&oids.progress);\n+\t\tstrbuf_reset(&progress_title);\n \t\tstrbuf_release(&packname);\n \t}\n \n \tif (commit_hex) {\n-\t\tif (report_progress)\n-\t\t\tprogress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"),\n-\t\t\t\tcommit_hex->nr);\n+\t\tif (report_progress) {\n+\t\t\tstrbuf_addf(&progress_title,\n+\t\t\t\t Q_(\"Finding commits for commit graph from %d ref\",\n+\t\t\t\t \"Finding commits for commit graph from %d refs\",\n+\t\t\t\t commit_hex->nr),\n+\t\t\t\t commit_hex->nr);\n+\t\t\tprogress = start_delayed_progress(progress_title.buf,\n+\t\t\t\t\t\t\t commit_hex->nr);\n+\t\t}\n \t\tfor (i = 0; i < commit_hex->nr; i++) {\n \t\t\tconst char *end;\n \t\t\tstruct object_id oid;\n@@ -860,12 +871,13 @@ void write_commit_graph(const char *obj_dir,\n \t\t\t}\n \t\t}\n \t\tstop_progress(&progress);\n+\t\tstrbuf_reset(&progress_title);\n \t}\n \n \tif (!pack_indexes && !commit_hex) {\n \t\tif (report_progress)\n \t\t\toids.progress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"),\n+\t\t\t\t_(\"Finding commits for commit graph among packed objects\"),\n \t\t\t\tapprox_nr_objects);\n \t\tfor_each_packed_object(add_packed_commits, &oids, 0);\n \t\tif (oids.progress_done < approx_nr_objects)\n@@ -970,6 +982,8 @@ void write_commit_graph(const char *obj_dir,\n \t\t\t\t &progress_cnt);\n \tstop_progress(&progress);\n \n+\tstrbuf_release(&progress_title);\n+\n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n \tcommit_lock_file(&lk);\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363773","messageId":"20181120195027.3260-6-avarab@gmail.com","threadId":"49275","inReplyTo":"20181120165800.GB30222@szeder.dev","subject":"[PATCH v2 5/6] commit-graph write: remove empty line for readability","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-20T19:50:26Z","receivedAt":"2018-11-20T19:50:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Remove the empty line between a QSORT(...) and the subsequent oideq()\nfor-loop. This makes it clearer that the QSORT(...) is being done so\nthat we can run the oideq() loop on adjacent OIDs. Amends code added\nin 08fd81c9b6 (\"commit-graph: implement write_commit_graph()\",\n2018-04-02).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex fd1fd61750..0e98679bce 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -888,7 +888,6 @@ void write_commit_graph(const char *obj_dir,\n \tclose_reachable(&oids, report_progress);\n \n \tQSORT(oids.list, oids.nr, commit_compare);\n-\n \tcount_distinct = 1;\n \tfor (i = 1; i < oids.nr; i++) {\n \t\tif (!oideq(&oids.list[i - 1], &oids.list[i]))\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363774","messageId":"20181120195027.3260-7-avarab@gmail.com","threadId":"49275","inReplyTo":"20181120165800.GB30222@szeder.dev","subject":"[PATCH v2 6/6] commit-graph write: add even more progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-20T19:50:27Z","receivedAt":"2018-11-20T19:50:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add more progress output to sections of code that can collectively\ntake 5-10 seconds on a large enough repository. On a test repository\nwith I have with ~7 million commits and ~50 million objects we'll now\nemit:\n\n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph among packed objects: 100% (50026015/50026015), done.\n Annotating commit graph: 21567407, done.\n Counting distinct commits in commit graph: 100% (7189147/7189147), done.\n Finding extra edges in commit graph: 100% (7189147/7189147), done.\n Computing commit graph generation numbers: 100% (7144680/7144680), done.\n Writing out commit graph: 21434417, done.\n\nWhereas on a medium-sized repository such as linux.git these new\nprogress bars won't have time to kick in and as before and we'll still\nemit output like:\n\n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph among packed objects: 100% (6365492/6365492), done.\n Annotating commit graph: 2391666, done.\n Computing commit graph generation numbers: 100% (797222/797222), done.\n Writing out commit graph: 2399912, done.\n\nThe \"Counting distinct commits in commit graph\" phase will spend most\nof its time paused at \"0/*\" as we QSORT(...) the list. That's not\noptimal, but at least we don't seem to be stalling anymore.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 13 +++++++++++++\n 1 file changed, 13 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 0e98679bce..1ad9000060 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -887,12 +887,19 @@ void write_commit_graph(const char *obj_dir,\n \n \tclose_reachable(&oids, report_progress);\n \n+\tif (report_progress)\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Counting distinct commits in commit graph\"),\n+\t\t\toids.nr);\n+\tdisplay_progress(progress, 0); /* TODO: Measure QSORT() progress */\n \tQSORT(oids.list, oids.nr, commit_compare);\n \tcount_distinct = 1;\n \tfor (i = 1; i < oids.nr; i++) {\n+\t\tdisplay_progress(progress, i + 1);\n \t\tif (!oideq(&oids.list[i - 1], &oids.list[i]))\n \t\t\tcount_distinct++;\n \t}\n+\tstop_progress(&progress);\n \n \tif (count_distinct >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n@@ -902,8 +909,13 @@ void write_commit_graph(const char *obj_dir,\n \tALLOC_ARRAY(commits.list, commits.alloc);\n \n \tnum_extra_edges = 0;\n+\tif (report_progress)\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Finding extra edges in commit graph\"),\n+\t\t\toids.nr);\n \tfor (i = 0; i < oids.nr; i++) {\n \t\tint num_parents = 0;\n+\t\tdisplay_progress(progress, i + 1);\n \t\tif (i > 0 && oideq(&oids.list[i - 1], &oids.list[i]))\n \t\t\tcontinue;\n \n@@ -920,6 +932,7 @@ void write_commit_graph(const char *obj_dir,\n \t\tcommits.nr++;\n \t}\n \tnum_chunks = num_extra_edges ? 4 : 3;\n+\tstop_progress(&progress);\n \n \tif (commits.nr >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"too many commits to write graph\"));\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363786","messageId":"20181120233840.GC30222@szeder.dev","threadId":"49275","inReplyTo":"20181120195027.3260-3-avarab@gmail.com","subject":"Re: [PATCH v2 2/6] commit-graph write: add more progress output","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-11-20T23:38:40Z","receivedAt":"2018-11-20T23:38:46Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Nov 20, 2018 at 07:50:23PM +0000, Ævar Arnfjörð Bjarmason wrote:\n> Add more progress output to the output already added in\n> 7b0f229222 (\"commit-graph write: add progress output\", 2018-09-17).\n> \n> As noted in that commit most of the progress output isn't displayed on\n> small repositories, but before this change we'd noticeably hang for\n> 2-3 seconds at the end on medium sized repositories such as linux.git.\n> \n> Now we'll instead show output like this, and have no human-observable\n> point at which we're not producing progress output:\n> \n> $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n> Finding commits for commit graph: 6365492, done.\n> Computing commit graph generation numbers: 100% (797222/797222), done.\n> Writing out commit graph: 2399912, done.\n> \n> This \"writing out\" number is not meant to be meaningful to the user,\n> but just to show that we're doing work and the command isn't\n> hanging.\n> \n> In the current implementation it's approximately 4x the number of\n> commits.\n\n\"approximately\" only, because the current implementation is buggy :)\nIf done right it's exactly 4x the number of commits.\n\n> As noted in on-list discussion[1] we could add the loops up\n> and show percentage progress here, but I don't think it's worth it. It\n> would make the implementation more complex and harder to maintain for\n> very little gain.\n\nI think that if we can cheaply and accurately figure out the total,\nthen we should display it, so onlooking users can guesstimate how much\nwork is still left to be done.\n\n> On a much larger in-house repository I have we'll show (note how we\n> also say \"Annotating[...]\"):\n> \n> $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n> Finding commits for commit graph: 50026015, done.\n> Annotating commit graph: 21567407, done.\n> Computing commit graph generation numbers: 100% (7144680/7144680), done.\n> Writing out commit graph: 21434417, done.\n> \n> 1. https://public-inbox.org/git/20181120165800.GB30222@szeder.dev/\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason \n> ---\n> commit-graph.c | 41 ++++++++++++++++++++++++++++++++---------\n> 1 file changed, 32 insertions(+), 9 deletions(-)\n> \n> diff --git a/commit-graph.c b/commit-graph.c\n> index e6d0d7722b..6f6409b292 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -433,7 +433,9 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n> \n> static void write_graph_chunk_fanout(struct hashfile *f,\n> \t\t\t\t struct commit **commits,\n> -\t\t\t\t int nr_commits)\n> +\t\t\t\t int nr_commits,\n> +\t\t\t\t struct progress *progress,\n> +\t\t\t\t uint64_t *progress_cnt)\n> {\n> \tint i, count = 0;\n> \tstruct commit **list = commits;\n> @@ -445,6 +447,7 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n> \t */\n> \tfor (i = 0; i < 256; i++) {\n> \t\twhile (count < nr_commits) {\n> +\t\t\tdisplay_progress(progress, ++*progress_cnt);\n\nI think this display_progress() should be places after the condition,\nso no one has to waste brain cycles on figuring out, why it always\ncounts 255 more than the number of commits.\n\n> \t\t\tif ((*list)->object.oid.hash[0] != i)\n> \t\t\t\tbreak;\n> \t\t\tcount++;\n> @@ -456,12 +459,16 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n> }\n> \n> static void write_graph_chunk_oids(struct hashfile *f, int hash_len,\n> -\t\t\t\t struct commit **commits, int nr_commits)\n> +\t\t\t\t struct commit **commits, int nr_commits,\n> +\t\t\t\t struct progress *progress,\n> +\t\t\t\t uint64_t *progress_cnt)\n> {\n> \tstruct commit **list = commits;\n> \tint count;\n> -\tfor (count = 0; count < nr_commits; count++, list++)\n> +\tfor (count = 0; count < nr_commits; count++, list++) {\n> +\t\tdisplay_progress(progress, ++*progress_cnt);\n> \t\thashwrite(f, (*list)->object.oid.hash, (int)hash_len);\n> +\t}\n> }\n> \n> static const unsigned char *commit_to_sha1(size_t index, void *table)\n> @@ -471,7 +478,9 @@ static const unsigned char *commit_to_sha1(size_t index, void *table)\n> }\n> \n> static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n> -\t\t\t\t struct commit **commits, int nr_commits)\n> +\t\t\t\t struct commit **commits, int nr_commits,\n> +\t\t\t\t struct progress *progress,\n> +\t\t\t\t uint64_t *progress_cnt)\n> {\n> \tstruct commit **list = commits;\n> \tstruct commit **last = commits + nr_commits;\n> @@ -481,6 +490,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n> \t\tstruct commit_list *parent;\n> \t\tint edge_value;\n> \t\tuint32_t packedDate[2];\n> +\t\tdisplay_progress(progress, ++*progress_cnt);\n> \n> \t\tparse_commit(*list);\n> \t\thashwrite(f, get_commit_tree_oid(*list)->hash, hash_len);\n> @@ -542,7 +552,9 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n> \n> static void write_graph_chunk_large_edges(struct hashfile *f,\n> \t\t\t\t\t struct commit **commits,\n> -\t\t\t\t\t int nr_commits)\n> +\t\t\t\t\t int nr_commits,\n> +\t\t\t\t\t struct progress *progress,\n> +\t\t\t\t\t uint64_t *progress_cnt)\n> {\n> \tstruct commit **list = commits;\n> \tstruct commit **last = commits + nr_commits;\n\n> @@ -565,6 +577,7 @@ static void write_graph_chunk_large_edges(struct hashfile *f,\n\n[Adding more before-context to this hunk here...]\n\n>\twhile (list < last) {\n\nThis loop iterates over all commits ...\n\n>\t\tint num_parents = 0;\n>\t\tfor (parent = (*list)->parents; num_parents < 3 && parent;\n>\t\t parent = parent->next)\n>\t\t\tnum_parents++;\n\n... counts the parents of the current commit ...\n\n>\n>\t\tif (num_parents <= 2) {\n>\t\t\tlist++;\n>\t\t\tcontinue;\n\n... and continues iterating unless it's an octopus merge.\n\n>\t\t}\n>\n>\t\t/* Since num_parents > 2, this initializer is safe. */\n>\t\tfor (parent = (*list)->parents->next; parent; parent = parent->next) {\n>\t\t\tint edge_value = sha1_pos(parent->item->object.oid.hash,\n> \t\t\t\t\t\t commits,\n> \t\t\t\t\t\t nr_commits,\n> \t\t\t\t\t\t commit_to_sha1);\n> +\t\t\tdisplay_progress(progress, ++*progress_cnt);\n\nSo this display_progress() call is in the wrong place, because it will\nonly be invoked on octopus merges, which only rarely occur in\npractice, thus it's entirely possible that it won't show any progress\nat all while the outer while loop iterates over the whole history.\n\nThis display_progress() should be places somewhere before that 'if\n(num_parents <= 2)' condition. And then this one, too, will count the\nnumber of commits.\n\n> \n> \t\t\tif (edge_value < 0)\n> \t\t\t\tedge_value = GRAPH_PARENT_MISSING;\n> @@ -764,6 +777,7 @@ void write_commit_graph(const char *obj_dir,\n> \tint num_extra_edges;\n> \tstruct commit_list *parent;\n> \tstruct progress *progress = NULL;\n> +\tuint64_t progress_cnt = 0;\n> \n> \tif (!commit_graph_compatible(the_repository))\n> \t\treturn;\n> @@ -937,10 +951,19 @@ void write_commit_graph(const char *obj_dir,\n> \t\thashwrite(f, chunk_write, 12);\n> \t}\n> \n> -\twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n> -\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n> -\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n> -\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n> +\tif (report_progress)\n> +\t\tprogress = start_delayed_progress(\n> +\t\t\t_(\"Writing out commit graph\"),\n> +\t\t\t0);\n> +\twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress,\n> +\t\t\t\t &progress_cnt);\n> +\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr,\n> +\t\t\t progress, &progress_cnt);\n> +\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr,\n> +\t\t\t progress, &progress_cnt);\n> +\twrite_graph_chunk_large_edges(f, commits.list, commits.nr, progress,\n> +\t\t\t\t &progress_cnt);\n> +\tstop_progress(&progress);\n> \n> \tclose_commit_graph(the_repository);\n> \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n> -- \n> 2.20.0.rc0.387.gc7a69e6b6c\n> \n"},{"id":"363795","messageId":"20181121012344.26711-1-szeder.dev@gmail.com","threadId":"49275","inReplyTo":"20181120165800.GB30222@szeder.dev","subject":"","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-11-21T01:23:42Z","receivedAt":"2018-11-21T01:23:58Z","isPatch":false,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Nov 20, 2018 at 05:58:00PM +0100, SZEDER Gábor wrote:\n> I saw a\n> bit of weirdness while at it, and want to look into it, but now I've\n> got to go...\n\nSo here are two simple patches that address the \"Huh?!\" moments I had\nwhile looking at the progress output during writing the commit graph\nfile. The first is a small cleanup to avoid confusion, but see the\nnotes attaches, while the second is a bit of an optimization.\n\nSZEDER Gábor (2):\n commit-graph: rename 'num_extra_edges' variable to 'num_large_edges'\n commit-graph: don't call write_graph_chunk_large_edges() unnecessarily\n\n commit-graph.c | 21 +++++++++++----------\n 1 file changed, 11 insertions(+), 10 deletions(-)\n\n-- \n2.20.0.rc0.134.gf0022f8e60\n\n"},{"id":"363796","messageId":"20181121012532.26878-1-szeder.dev@gmail.com","threadId":"49275","inReplyTo":"20181121012344.26711-1-szeder.dev@gmail.com","subject":"[PATCH 1/2] commit-graph: rename 'num_extra_edges' variable to 'num_large_edges'","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-11-21T01:25:32Z","receivedAt":"2018-11-21T01:25:52Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"The commit graph file format describes an optional 'Large Edge List'\nchunk, and the function writing out this chunk is called\nwrite_graph_chunk_large_edges(). Then there are two functions in\n'commit-graph.c', namely write_graph_chunk_data() and\nwrite_commit_graph(), which have a local variable called\n'num_extra_edges'.\n\nIt can be confusing on first sight whether large edges and extra edges\nrefer to the same thing or not, but they do, so let's rename those\nvariables to 'num_large_edges'.\n\nSigned-off-by: SZEDER Gábor \n---\n\nI rename these variables to 'num_large_edges', because the commit\ngraph file format speaks about the 'Large Edge List' chunk.\n\nHowever, I do find that the term 'extra' makes much more sense and\nfits the concept better (i.e. extra commit graph edges resulting from\nthe extra parents or octopus merges; after a s/extra/large/g the\nprevious phrase would make no sense), and notice that the term 'large'\ndoesn't come up in the file format itseld (the chunk's magic is {'E',\n'D', 'G', 'E'}, there is no 'L' in there), but only in the\nspecification text and a couple of variable and function names in the\ncode.\n\nWould it make sense to do the rename in the other direction?\n\n commit-graph.c | 18 +++++++++---------\n 1 file changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 40c855f185..7b4e3a02cf 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -475,7 +475,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n {\n \tstruct commit **list = commits;\n \tstruct commit **last = commits + nr_commits;\n-\tuint32_t num_extra_edges = 0;\n+\tuint32_t num_large_edges = 0;\n \n \twhile (list < last) {\n \t\tstruct commit_list *parent;\n@@ -507,7 +507,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\tif (!parent)\n \t\t\tedge_value = GRAPH_PARENT_NONE;\n \t\telse if (parent->next)\n-\t\t\tedge_value = GRAPH_OCTOPUS_EDGES_NEEDED | num_extra_edges;\n+\t\t\tedge_value = GRAPH_OCTOPUS_EDGES_NEEDED | num_large_edges;\n \t\telse {\n \t\t\tedge_value = sha1_pos(parent->item->object.oid.hash,\n \t\t\t\t\t commits,\n@@ -521,7 +521,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \n \t\tif (edge_value & GRAPH_OCTOPUS_EDGES_NEEDED) {\n \t\t\tdo {\n-\t\t\t\tnum_extra_edges++;\n+\t\t\t\tnum_large_edges++;\n \t\t\t\tparent = parent->next;\n \t\t\t} while (parent);\n \t\t}\n@@ -761,7 +761,7 @@ void write_commit_graph(const char *obj_dir,\n \tuint32_t chunk_ids[5];\n \tuint64_t chunk_offsets[5];\n \tint num_chunks;\n-\tint num_extra_edges;\n+\tint num_large_edges;\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n \n@@ -871,7 +871,7 @@ void write_commit_graph(const char *obj_dir,\n \tcommits.alloc = count_distinct;\n \tALLOC_ARRAY(commits.list, commits.alloc);\n \n-\tnum_extra_edges = 0;\n+\tnum_large_edges = 0;\n \tfor (i = 0; i < oids.nr; i++) {\n \t\tint num_parents = 0;\n \t\tif (i > 0 && oideq(&oids.list[i - 1], &oids.list[i]))\n@@ -885,11 +885,11 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tnum_parents++;\n \n \t\tif (num_parents > 2)\n-\t\t\tnum_extra_edges += num_parents - 1;\n+\t\t\tnum_large_edges += num_parents - 1;\n \n \t\tcommits.nr++;\n \t}\n-\tnum_chunks = num_extra_edges ? 4 : 3;\n+\tnum_chunks = num_large_edges ? 4 : 3;\n \n \tif (commits.nr >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"too many commits to write graph\"));\n@@ -916,7 +916,7 @@ void write_commit_graph(const char *obj_dir,\n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n \tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n-\tif (num_extra_edges)\n+\tif (num_large_edges)\n \t\tchunk_ids[3] = GRAPH_CHUNKID_LARGEEDGES;\n \telse\n \t\tchunk_ids[3] = 0;\n@@ -926,7 +926,7 @@ void write_commit_graph(const char *obj_dir,\n \tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n \tchunk_offsets[2] = chunk_offsets[1] + GRAPH_OID_LEN * commits.nr;\n \tchunk_offsets[3] = chunk_offsets[2] + (GRAPH_OID_LEN + 16) * commits.nr;\n-\tchunk_offsets[4] = chunk_offsets[3] + 4 * num_extra_edges;\n+\tchunk_offsets[4] = chunk_offsets[3] + 4 * num_large_edges;\n \n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tuint32_t chunk_write[3];\n-- \n2.20.0.rc0.134.gf0022f8e60\n\n"},{"id":"363797","messageId":"20181121012600.26951-1-szeder.dev@gmail.com","threadId":"49275","inReplyTo":"20181121012344.26711-1-szeder.dev@gmail.com","subject":"[PATCH 2/2] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-11-21T01:26:00Z","receivedAt":"2018-11-21T01:26:08Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"The optional 'Large Edge List' chunk of the commit graph file stores\nparent information for commits with more than two parents. Since the\nchunk is optional, write_commit_graph() looks through all commits to\nfind those with more than two parents, and then writes the commit\ngraph file header accordingly, i.e. if there are no such commits, then\nthere won't be a 'Large Edge List' chunk written, only the three\nmandatory chunks.\n\nHowever, when it comes to writing chunk data, write_commit_graph()\nunconditionally invokes write_graph_chunk_large_edges(), even when it\nwas decided earlier that that chunk won't be written. Strictly\nspeaking there is no bug here, because write_graph_chunk_large_edges()\nwon't write anything because it won't find any commits with more than\ntwo parents, but then it unnecessarily and in vain looks through all\ncommits once again in search for such commits.\n\nDon't call write_graph_chunk_large_edges() when that chunk won't be\nwritten to spare an unnecessary iteration over all commits.\n\nSigned-off-by: SZEDER Gábor \n---\n commit-graph.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 7b4e3a02cf..965eb23a7b 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -940,7 +940,8 @@ void write_commit_graph(const char *obj_dir,\n \twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n \twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n \twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n-\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n+\tif (num_large_edges)\n+\t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n \n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n-- \n2.20.0.rc0.134.gf0022f8e60\n\n"},{"id":"363802","messageId":"xmqqwop785wh.fsf@gitster-ct.c.googlers.com","threadId":"49275","inReplyTo":"20181121012532.26878-1-szeder.dev@gmail.com","subject":"Re: [PATCH 1/2] commit-graph: rename 'num_extra_edges' variable to 'num_large_edges'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-11-21T03:29:34Z","receivedAt":"2018-11-21T03:29:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"SZEDER Gábor writes:\n\n> I rename these variables to 'num_large_edges', because the commit\n> graph file format speaks about the 'Large Edge List' chunk.\n>\n> However, I do find that the term 'extra' makes much more sense and\n> fits the concept better (i.e. extra commit graph edges resulting from\n> the extra parents or octopus merges; after a s/extra/large/g the\n> previous phrase would make no sense), and notice that the term 'large'\n> doesn't come up in the file format itseld (the chunk's magic is {'E',\n> 'D', 'G', 'E'}, there is no 'L' in there), but only in the\n> specification text and a couple of variable and function names in the\n> code.\n>\n> Would it make sense to do the rename in the other direction?\n\nSo edges that are involved in octopus merges are counted with\nnum_extra_edges and written to the large edges table?\n\nI tend to agree that \"large edge\" is a misnomer. These edges that\npoint at third and subsequent parents are no larger than the edges\nthat point at the first or the second parents---they are the same\nsize. What is larger than usual is the size of the list of edges\n(i.e. the number of parents), because the commit has extra (compared\nto the majority of commits) number of edges. So from the point of\nview, I agree with you that \"extra\" makes a lot more sense than\n\"large\".\n\nAnd the magic number being \"EDGE\" without \"L\" is probably a good\nthing, as a graph whose commits are all without any extra edge does\nnot need the \"EDGE\" chunk, so presence of the chunk by itself is a\nsign that extra things are involved. Which means that there isn't\nany need to update the magic number, if we wanted to get rid of\n\"large\" and replace it with \"extra\". The only thing needed to\nupdate the documentation, variable names and in-code comment.\n\nAnd while at it, GRAPH_OCTOPUS_EDGES_NEEDED may also want to be\nrenamed with s/OCTOPUS/EXTRA/;\n\n> commit-graph.c | 18 +++++++++---------\n> 1 file changed, 9 insertions(+), 9 deletions(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 40c855f185..7b4e3a02cf 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -475,7 +475,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n> {\n> \tstruct commit **list = commits;\n> \tstruct commit **last = commits + nr_commits;\n> -\tuint32_t num_extra_edges = 0;\n> +\tuint32_t num_large_edges = 0;\n> \n> \twhile (list < last) {\n> \t\tstruct commit_list *parent;\n> @@ -507,7 +507,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n> \t\tif (!parent)\n> \t\t\tedge_value = GRAPH_PARENT_NONE;\n> \t\telse if (parent->next)\n> -\t\t\tedge_value = GRAPH_OCTOPUS_EDGES_NEEDED | num_extra_edges;\n> +\t\t\tedge_value = GRAPH_OCTOPUS_EDGES_NEEDED | num_large_edges;\n> \t\telse {\n> \t\t\tedge_value = sha1_pos(parent->item->object.oid.hash,\n> \t\t\t\t\t commits,\n> @@ -521,7 +521,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n> \n> \t\tif (edge_value & GRAPH_OCTOPUS_EDGES_NEEDED) {\n> \t\t\tdo {\n> -\t\t\t\tnum_extra_edges++;\n> +\t\t\t\tnum_large_edges++;\n> \t\t\t\tparent = parent->next;\n> \t\t\t} while (parent);\n> \t\t}\n> @@ -761,7 +761,7 @@ void write_commit_graph(const char *obj_dir,\n> \tuint32_t chunk_ids[5];\n> \tuint64_t chunk_offsets[5];\n> \tint num_chunks;\n> -\tint num_extra_edges;\n> +\tint num_large_edges;\n> \tstruct commit_list *parent;\n> \tstruct progress *progress = NULL;\n> \n> @@ -871,7 +871,7 @@ void write_commit_graph(const char *obj_dir,\n> \tcommits.alloc = count_distinct;\n> \tALLOC_ARRAY(commits.list, commits.alloc);\n> \n> -\tnum_extra_edges = 0;\n> +\tnum_large_edges = 0;\n> \tfor (i = 0; i < oids.nr; i++) {\n> \t\tint num_parents = 0;\n> \t\tif (i > 0 && oideq(&oids.list[i - 1], &oids.list[i]))\n> @@ -885,11 +885,11 @@ void write_commit_graph(const char *obj_dir,\n> \t\t\tnum_parents++;\n> \n> \t\tif (num_parents > 2)\n> -\t\t\tnum_extra_edges += num_parents - 1;\n> +\t\t\tnum_large_edges += num_parents - 1;\n> \n> \t\tcommits.nr++;\n> \t}\n> -\tnum_chunks = num_extra_edges ? 4 : 3;\n> +\tnum_chunks = num_large_edges ? 4 : 3;\n> \n> \tif (commits.nr >= GRAPH_PARENT_MISSING)\n> \t\tdie(_(\"too many commits to write graph\"));\n> @@ -916,7 +916,7 @@ void write_commit_graph(const char *obj_dir,\n> \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n> \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n> \tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n> -\tif (num_extra_edges)\n> +\tif (num_large_edges)\n> \t\tchunk_ids[3] = GRAPH_CHUNKID_LARGEEDGES;\n> \telse\n> \t\tchunk_ids[3] = 0;\n> @@ -926,7 +926,7 @@ void write_commit_graph(const char *obj_dir,\n> \tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n> \tchunk_offsets[2] = chunk_offsets[1] + GRAPH_OID_LEN * commits.nr;\n> \tchunk_offsets[3] = chunk_offsets[2] + (GRAPH_OID_LEN + 16) * commits.nr;\n> -\tchunk_offsets[4] = chunk_offsets[3] + 4 * num_extra_edges;\n> +\tchunk_offsets[4] = chunk_offsets[3] + 4 * num_large_edges;\n> \n> \tfor (i = 0; i <= num_chunks; i++) {\n> \t\tuint32_t chunk_write[3];\n"},{"id":"363816","messageId":"a82a2bec-07f5-ccd2-85ab-b9efd406929e@gmail.com","threadId":"49275","inReplyTo":"xmqqwop785wh.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 1/2] commit-graph: rename 'num_extra_edges' variable to 'num_large_edges'","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-21T11:32:19Z","receivedAt":"2018-11-21T11:32:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/20/2018 10:29 PM, Junio C Hamano wrote:\n> SZEDER Gábor writes:\n>\n>> I rename these variables to 'num_large_edges', because the commit\n>> graph file format speaks about the 'Large Edge List' chunk.\n>>\n>> However, I do find that the term 'extra' makes much more sense\n>>\n>> Would it make sense to do the rename in the other direction?\n> I tend to agree that \"large edge\" is a misnomer.\n\nI agree with you both. \"Extra\" is better.\n\nThanks,\n\n-Stolee\n\n"},{"id":"363817","messageId":"6d9ce874-0d96-1312-5b9f-71a8db20ce9b@gmail.com","threadId":"49275","inReplyTo":"20181121012600.26951-1-szeder.dev@gmail.com","subject":"Re: [PATCH 2/2] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-21T11:33:56Z","receivedAt":"2018-11-21T11:34:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/20/2018 8:26 PM, SZEDER Gábor wrote:\n> \twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n> -\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n> +\tif (num_large_edges)\n> +\t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n\nThis is clearly correct, and the tests in t5318-commit-graph.sh would \ncatch a dropped (or additional) large/extra edge chunk.\n\nThanks,\n\n-Stolee\n\n"},{"id":"363884","messageId":"20181122132823.9883-1-avarab@gmail.com","threadId":"49275","inReplyTo":"20181121012600.26951-1-szeder.dev@gmail.com","subject":"[PATCH v3 00/10] commit-graph write: progress output improvements","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-22T13:28:13Z","receivedAt":"2018-11-22T13:28:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This incorporates SZEDER's recent two-part series, rebases mine on\ntop, and fixes a few things while I'm at it. Now there's no progress\noutput where we don't show a completion percentage.\n\nSZEDER Gábor (2):\n commit-graph: rename 'num_extra_edges' variable to 'num_large_edges'\n commit-graph: don't call write_graph_chunk_large_edges() unnecessarily\n\nÆvar Arnfjörð Bjarmason (8):\n commit-graph write: rephrase confusing progress output\n commit-graph write: add \"Writing out\" progress output\n commit-graph write: more descriptive \"writing out\" output\n commit-graph write: show progress for object search\n commit-graph write: add more descriptive progress output\n commit-graph write: remove empty line for readability\n commit-graph write: add itermediate progress\n commit-graph write: emit a percentage for all progress\n\n commit-graph.c | 130 ++++++++++++++++++++++++++++++++++++++-----------\n 1 file changed, 102 insertions(+), 28 deletions(-)\n\nRange-diff:\n\nBy the way, is there any way to....\n\n [.. snipped lots of irrelevant commits...]\n -: ---------- > 14: 07d06c50c0 commit-graph: rename 'num_extra_edges' variable to 'num_large_edges'\n -: ---------- > 15: 904dda1e7a commit-graph: don't call write_graph_chunk_large_edges() unnecessarily\n\nPass the equivalent of \"git range-diff origin/master topic-2 topic-3\"\nto git-format-patch?\n\n 1: 9f7fb459bd = 16: 1126c7e29d commit-graph write: rephrase confusing progress output\n a => b | 0\n 1 file changed, 0 insertions(+), 0 deletions(-)\n \n 2: 093c63e99f ! 17: 2b52ad2284 commit-graph write: add more progress output\n a => b | 0\n 1 file changed, 0 insertions(+), 0 deletions(-)\n \n @@ -1,9 +1,10 @@\n Author: Ævar Arnfjörð Bjarmason \n \n - commit-graph write: add more progress output\n + commit-graph write: add \"Writing out\" progress output\n \n - Add more progress output to the output already added in\n - 7b0f229222 (\"commit-graph write: add progress output\", 2018-09-17).\n + Add progress output to be shown when we're writing out the\n + commit-graph, this adds to the output already added in 7b0f229222\n + (\"commit-graph write: add progress output\", 2018-09-17).\n \n As noted in that commit most of the progress output isn't displayed on\n small repositories, but before this change we'd noticeably hang for\n @@ -13,30 +14,13 @@\n point at which we're not producing progress output:\n \n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n - Finding commits for commit graph: 6365492, done.\n + Finding commits for commit graph: 6365442, done.\n Computing commit graph generation numbers: 100% (797222/797222), done.\n - Writing out commit graph: 2399912, done.\n + Writing out commit graph: 100% (3986110/3986110), done.\n \n - This \"writing out\" number is not meant to be meaningful to the user,\n - but just to show that we're doing work and the command isn't\n - hanging.\n -\n - In the current implementation it's approximately 4x the number of\n - commits. As noted in on-list discussion[1] we could add the loops up\n - and show percentage progress here, but I don't think it's worth it. It\n - would make the implementation more complex and harder to maintain for\n - very little gain.\n -\n - On a much larger in-house repository I have we'll show (note how we\n - also say \"Annotating[...]\"):\n -\n - $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n - Finding commits for commit graph: 50026015, done.\n - Annotating commit graph: 21567407, done.\n - Computing commit graph generation numbers: 100% (7144680/7144680), done.\n - Writing out commit graph: 21434417, done.\n -\n - 1. https://public-inbox.org/git/20181120165800.GB30222@szeder.dev/\n + This \"Writing out\" number is 4x or 5x the number of commits, depending\n + on the graph we're processing. A later change will make this explicit\n + to the user.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n @@ -55,13 +39,13 @@\n \tint i, count = 0;\n \tstruct commit **list = commits;\n @@\n - \t */\n - \tfor (i = 0; i < 256; i++) {\n \t\twhile (count < nr_commits) {\n -+\t\t\tdisplay_progress(progress, ++*progress_cnt);\n \t\t\tif ((*list)->object.oid.hash[0] != i)\n \t\t\t\tbreak;\n ++\t\t\tdisplay_progress(progress, ++*progress_cnt);\n \t\t\tcount++;\n + \t\t\tlist++;\n + \t\t}\n @@\n }\n \n @@ -112,15 +96,17 @@\n \tstruct commit **list = commits;\n \tstruct commit **last = commits + nr_commits;\n @@\n - \t\t\t\t\t\t commits,\n - \t\t\t\t\t\t nr_commits,\n - \t\t\t\t\t\t commit_to_sha1);\n -+\t\t\tdisplay_progress(progress, ++*progress_cnt);\n \n - \t\t\tif (edge_value < 0)\n - \t\t\t\tedge_value = GRAPH_PARENT_MISSING;\n + \twhile (list < last) {\n + \t\tint num_parents = 0;\n ++\n ++\t\tdisplay_progress(progress, ++*progress_cnt);\n ++\n + \t\tfor (parent = (*list)->parents; num_parents < 3 && parent;\n + \t\t parent = parent->next)\n + \t\t\tnum_parents++;\n @@\n - \tint num_extra_edges;\n + \tint num_large_edges;\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n +\tuint64_t progress_cnt = 0;\n @@ -134,19 +120,25 @@\n -\twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n -\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n -\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n --\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n -+\tif (report_progress)\n ++\tif (report_progress) {\n ++\t\t/*\n ++\t\t * Each of the write_graph_chunk_*() functions just\n ++\t\t * below loops over our N commits. This number must be\n ++\t\t * kept in sync with the number of passes we're doing.\n ++\t\t */\n ++\t\tint graph_passes = 4;\n ++\t\tif (num_large_edges)\n ++\t\t\tgraph_passes++;\n +\t\tprogress = start_delayed_progress(\n +\t\t\t_(\"Writing out commit graph\"),\n -+\t\t\t0);\n -+\twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress,\n -+\t\t\t\t &progress_cnt);\n -+\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr,\n -+\t\t\t progress, &progress_cnt);\n -+\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr,\n -+\t\t\t progress, &progress_cnt);\n -+\twrite_graph_chunk_large_edges(f, commits.list, commits.nr, progress,\n -+\t\t\t\t &progress_cnt);\n ++\t\t\tgraph_passes * commits.nr);\n ++\t}\n ++\twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress, &progress_cnt);\n ++\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr, progress, &progress_cnt);\n ++\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr, progress, &progress_cnt);\n + \tif (num_large_edges)\n +-\t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n ++\t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr, progress, &progress_cnt);\n +\tstop_progress(&progress);\n \n \tclose_commit_graph(the_repository);\n -: ---------- > 18: b1773677b1 commit-graph write: more descriptive \"writing out\" output\n 3: 6c71de9460 ! 19: 3138b00a2c commit-graph write: show progress for object search\n a => b | 0\n 1 file changed, 0 insertions(+), 0 deletions(-)\n \n @@ -37,9 +37,9 @@\n --- a/commit-graph.c\n +++ b/commit-graph.c\n @@\n - \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n \tuint64_t progress_cnt = 0;\n + \tstruct strbuf progress_title = STRBUF_INIT;\n +\tunsigned long approx_nr_objects;\n \n \tif (!commit_graph_compatible(the_repository))\n 4: c665dbdacb ! 20: f41e3b3eb3 commit-graph write: add more describing progress output\n a => b | 0\n 1 file changed, 0 insertions(+), 0 deletions(-)\n \n @@ -1,6 +1,6 @@\n Author: Ævar Arnfjörð Bjarmason \n \n - commit-graph write: add more describing progress output\n + commit-graph write: add more descriptive progress output\n \n Make the progress output shown when we're searching for commits to\n include in the graph more descriptive. This amends code I added in\n @@ -36,14 +36,6 @@\n diff --git a/commit-graph.c b/commit-graph.c\n --- a/commit-graph.c\n +++ b/commit-graph.c\n -@@\n - \tstruct progress *progress = NULL;\n - \tuint64_t progress_cnt = 0;\n - \tunsigned long approx_nr_objects;\n -+\tstruct strbuf progress_title = STRBUF_INIT;\n - \n - \tif (!commit_graph_compatible(the_repository))\n - \t\treturn;\n @@\n \t\tstrbuf_addf(&packname, \"%s/pack/\", obj_dir);\n \t\tdirlen = packname.len;\n @@ -99,12 +91,3 @@\n \t\t\t\tapprox_nr_objects);\n \t\tfor_each_packed_object(add_packed_commits, &oids, 0);\n \t\tif (oids.progress_done < approx_nr_objects)\n -@@\n - \t\t\t\t &progress_cnt);\n - \tstop_progress(&progress);\n - \n -+\tstrbuf_release(&progress_title);\n -+\n - \tclose_commit_graph(the_repository);\n - \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n - \tcommit_lock_file(&lk);\n 5: f70fc5045d = 21: 74037032d3 commit-graph write: remove empty line for readability\n a => b | 0\n 1 file changed, 0 insertions(+), 0 deletions(-)\n \n 6: 2e943fa925 ! 22: 502da68d14 commit-graph write: add even more progress output\n a => b | 0\n 1 file changed, 0 insertions(+), 0 deletions(-)\n \n @@ -1,11 +1,13 @@\n Author: Ævar Arnfjörð Bjarmason \n \n - commit-graph write: add even more progress output\n + commit-graph write: add itermediate progress\n \n - Add more progress output to sections of code that can collectively\n - take 5-10 seconds on a large enough repository. On a test repository\n - with I have with ~7 million commits and ~50 million objects we'll now\n - emit:\n + Add progress output to sections of code between \"Annotating[...]\" and\n + \"Computing[...]generation numbers\". This can collectively take 5-10\n + seconds on a large enough repository.\n +\n + On a test repository with I have with ~7 million commits and ~50\n + million objects we'll now emit:\n \n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph among packed objects: 100% (50026015/50026015), done.\n @@ -57,7 +59,7 @@\n @@\n \tALLOC_ARRAY(commits.list, commits.alloc);\n \n - \tnum_extra_edges = 0;\n + \tnum_large_edges = 0;\n +\tif (report_progress)\n +\t\tprogress = start_delayed_progress(\n +\t\t\t_(\"Finding extra edges in commit graph\"),\n @@ -71,7 +73,7 @@\n @@\n \t\tcommits.nr++;\n \t}\n - \tnum_chunks = num_extra_edges ? 4 : 3;\n + \tnum_chunks = num_large_edges ? 4 : 3;\n +\tstop_progress(&progress);\n \n \tif (commits.nr >= GRAPH_PARENT_MISSING)\n -: ---------- > 23: dfaf840983 commit-graph write: emit a percentage for all progress\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363885","messageId":"20181122132823.9883-2-avarab@gmail.com","threadId":"49275","inReplyTo":"20181121012600.26951-1-szeder.dev@gmail.com","subject":"[PATCH v3 01/10] commit-graph: rename 'num_extra_edges' variable to 'num_large_edges'","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-22T13:28:14Z","receivedAt":"2018-11-22T13:28:38Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: SZEDER Gábor \n\nThe commit graph file format describes an optional 'Large Edge List'\nchunk, and the function writing out this chunk is called\nwrite_graph_chunk_large_edges(). Then there are two functions in\n'commit-graph.c', namely write_graph_chunk_data() and\nwrite_commit_graph(), which have a local variable called\n'num_extra_edges'.\n\nIt can be confusing on first sight whether large edges and extra edges\nrefer to the same thing or not, but they do, so let's rename those\nvariables to 'num_large_edges'.\n\nSigned-off-by: SZEDER Gábor \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 18 +++++++++---------\n 1 file changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 40c855f185..7b4e3a02cf 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -475,7 +475,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n {\n \tstruct commit **list = commits;\n \tstruct commit **last = commits + nr_commits;\n-\tuint32_t num_extra_edges = 0;\n+\tuint32_t num_large_edges = 0;\n \n \twhile (list < last) {\n \t\tstruct commit_list *parent;\n@@ -507,7 +507,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\tif (!parent)\n \t\t\tedge_value = GRAPH_PARENT_NONE;\n \t\telse if (parent->next)\n-\t\t\tedge_value = GRAPH_OCTOPUS_EDGES_NEEDED | num_extra_edges;\n+\t\t\tedge_value = GRAPH_OCTOPUS_EDGES_NEEDED | num_large_edges;\n \t\telse {\n \t\t\tedge_value = sha1_pos(parent->item->object.oid.hash,\n \t\t\t\t\t commits,\n@@ -521,7 +521,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \n \t\tif (edge_value & GRAPH_OCTOPUS_EDGES_NEEDED) {\n \t\t\tdo {\n-\t\t\t\tnum_extra_edges++;\n+\t\t\t\tnum_large_edges++;\n \t\t\t\tparent = parent->next;\n \t\t\t} while (parent);\n \t\t}\n@@ -761,7 +761,7 @@ void write_commit_graph(const char *obj_dir,\n \tuint32_t chunk_ids[5];\n \tuint64_t chunk_offsets[5];\n \tint num_chunks;\n-\tint num_extra_edges;\n+\tint num_large_edges;\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n \n@@ -871,7 +871,7 @@ void write_commit_graph(const char *obj_dir,\n \tcommits.alloc = count_distinct;\n \tALLOC_ARRAY(commits.list, commits.alloc);\n \n-\tnum_extra_edges = 0;\n+\tnum_large_edges = 0;\n \tfor (i = 0; i < oids.nr; i++) {\n \t\tint num_parents = 0;\n \t\tif (i > 0 && oideq(&oids.list[i - 1], &oids.list[i]))\n@@ -885,11 +885,11 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tnum_parents++;\n \n \t\tif (num_parents > 2)\n-\t\t\tnum_extra_edges += num_parents - 1;\n+\t\t\tnum_large_edges += num_parents - 1;\n \n \t\tcommits.nr++;\n \t}\n-\tnum_chunks = num_extra_edges ? 4 : 3;\n+\tnum_chunks = num_large_edges ? 4 : 3;\n \n \tif (commits.nr >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"too many commits to write graph\"));\n@@ -916,7 +916,7 @@ void write_commit_graph(const char *obj_dir,\n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n \tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n-\tif (num_extra_edges)\n+\tif (num_large_edges)\n \t\tchunk_ids[3] = GRAPH_CHUNKID_LARGEEDGES;\n \telse\n \t\tchunk_ids[3] = 0;\n@@ -926,7 +926,7 @@ void write_commit_graph(const char *obj_dir,\n \tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n \tchunk_offsets[2] = chunk_offsets[1] + GRAPH_OID_LEN * commits.nr;\n \tchunk_offsets[3] = chunk_offsets[2] + (GRAPH_OID_LEN + 16) * commits.nr;\n-\tchunk_offsets[4] = chunk_offsets[3] + 4 * num_extra_edges;\n+\tchunk_offsets[4] = chunk_offsets[3] + 4 * num_large_edges;\n \n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tuint32_t chunk_write[3];\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363886","messageId":"20181122132823.9883-4-avarab@gmail.com","threadId":"49275","inReplyTo":"20181121012600.26951-1-szeder.dev@gmail.com","subject":"[PATCH v3 03/10] commit-graph write: rephrase confusing progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-22T13:28:16Z","receivedAt":"2018-11-22T13:28:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Rephrase the title shown for the progress output emitted by\nclose_reachable(). The message I added in 7b0f229222 (\"commit-graph\nwrite: add progress output\", 2018-09-17) gave the impression that it\nwould count up to the number of commit objects.\n\nBut that's not what the number means. It just represents the work\nwe're doing in several for-loops to do various work before the graph\nis written out. So let's just say \"Annotating commit graph\", that\ntitle makes no such promises, and we can add other loops here in the\nfuture and still consistently show progress output.\n\nSee [1] for the initial bug report & subsequent discussion about other\napproaching to solving this.\n\n1. https://public-inbox.org/git/20181015165447.GH19800@szeder.dev/\n\nReported-by: SZEDER Gábor \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 965eb23a7b..d11370a2b3 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -648,7 +648,7 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n \n \tif (report_progress)\n \t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Annotating commits in commit graph\"), 0);\n+\t\t\t_(\"Annotating commit graph\"), 0);\n \tfor (i = 0; i < oids->nr; i++) {\n \t\tdisplay_progress(progress, ++j);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363887","messageId":"20181122132823.9883-3-avarab@gmail.com","threadId":"49275","inReplyTo":"20181121012600.26951-1-szeder.dev@gmail.com","subject":"[PATCH v3 02/10] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-22T13:28:15Z","receivedAt":"2018-11-22T13:28:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: SZEDER Gábor \n\nThe optional 'Large Edge List' chunk of the commit graph file stores\nparent information for commits with more than two parents. Since the\nchunk is optional, write_commit_graph() looks through all commits to\nfind those with more than two parents, and then writes the commit\ngraph file header accordingly, i.e. if there are no such commits, then\nthere won't be a 'Large Edge List' chunk written, only the three\nmandatory chunks.\n\nHowever, when it comes to writing chunk data, write_commit_graph()\nunconditionally invokes write_graph_chunk_large_edges(), even when it\nwas decided earlier that that chunk won't be written. Strictly\nspeaking there is no bug here, because write_graph_chunk_large_edges()\nwon't write anything because it won't find any commits with more than\ntwo parents, but then it unnecessarily and in vain looks through all\ncommits once again in search for such commits.\n\nDon't call write_graph_chunk_large_edges() when that chunk won't be\nwritten to spare an unnecessary iteration over all commits.\n\nSigned-off-by: SZEDER Gábor \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 7b4e3a02cf..965eb23a7b 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -940,7 +940,8 @@ void write_commit_graph(const char *obj_dir,\n \twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n \twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n \twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n-\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n+\tif (num_large_edges)\n+\t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n \n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363888","messageId":"20181122132823.9883-5-avarab@gmail.com","threadId":"49275","inReplyTo":"20181121012600.26951-1-szeder.dev@gmail.com","subject":"[PATCH v3 04/10] commit-graph write: add \"Writing out\" progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-22T13:28:17Z","receivedAt":"2018-11-22T13:28:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add progress output to be shown when we're writing out the\ncommit-graph, this adds to the output already added in 7b0f229222\n(\"commit-graph write: add progress output\", 2018-09-17).\n\nAs noted in that commit most of the progress output isn't displayed on\nsmall repositories, but before this change we'd noticeably hang for\n2-3 seconds at the end on medium sized repositories such as linux.git.\n\nNow we'll instead show output like this, and have no human-observable\npoint at which we're not producing progress output:\n\n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph: 6365442, done.\n Computing commit graph generation numbers: 100% (797222/797222), done.\n Writing out commit graph: 100% (3986110/3986110), done.\n\nThis \"Writing out\" number is 4x or 5x the number of commits, depending\non the graph we're processing. A later change will make this explicit\nto the user.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 48 +++++++++++++++++++++++++++++++++++++++---------\n 1 file changed, 39 insertions(+), 9 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d11370a2b3..e32a5cc1bc 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -433,7 +433,9 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n \t\t\t\t struct commit **commits,\n-\t\t\t\t int nr_commits)\n+\t\t\t\t int nr_commits,\n+\t\t\t\t struct progress *progress,\n+\t\t\t\t uint64_t *progress_cnt)\n {\n \tint i, count = 0;\n \tstruct commit **list = commits;\n@@ -447,6 +449,7 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n \t\twhile (count < nr_commits) {\n \t\t\tif ((*list)->object.oid.hash[0] != i)\n \t\t\t\tbreak;\n+\t\t\tdisplay_progress(progress, ++*progress_cnt);\n \t\t\tcount++;\n \t\t\tlist++;\n \t\t}\n@@ -456,12 +459,16 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n }\n \n static void write_graph_chunk_oids(struct hashfile *f, int hash_len,\n-\t\t\t\t struct commit **commits, int nr_commits)\n+\t\t\t\t struct commit **commits, int nr_commits,\n+\t\t\t\t struct progress *progress,\n+\t\t\t\t uint64_t *progress_cnt)\n {\n \tstruct commit **list = commits;\n \tint count;\n-\tfor (count = 0; count < nr_commits; count++, list++)\n+\tfor (count = 0; count < nr_commits; count++, list++) {\n+\t\tdisplay_progress(progress, ++*progress_cnt);\n \t\thashwrite(f, (*list)->object.oid.hash, (int)hash_len);\n+\t}\n }\n \n static const unsigned char *commit_to_sha1(size_t index, void *table)\n@@ -471,7 +478,9 @@ static const unsigned char *commit_to_sha1(size_t index, void *table)\n }\n \n static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n-\t\t\t\t struct commit **commits, int nr_commits)\n+\t\t\t\t struct commit **commits, int nr_commits,\n+\t\t\t\t struct progress *progress,\n+\t\t\t\t uint64_t *progress_cnt)\n {\n \tstruct commit **list = commits;\n \tstruct commit **last = commits + nr_commits;\n@@ -481,6 +490,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\tstruct commit_list *parent;\n \t\tint edge_value;\n \t\tuint32_t packedDate[2];\n+\t\tdisplay_progress(progress, ++*progress_cnt);\n \n \t\tparse_commit(*list);\n \t\thashwrite(f, get_commit_tree_oid(*list)->hash, hash_len);\n@@ -542,7 +552,9 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \n static void write_graph_chunk_large_edges(struct hashfile *f,\n \t\t\t\t\t struct commit **commits,\n-\t\t\t\t\t int nr_commits)\n+\t\t\t\t\t int nr_commits,\n+\t\t\t\t\t struct progress *progress,\n+\t\t\t\t\t uint64_t *progress_cnt)\n {\n \tstruct commit **list = commits;\n \tstruct commit **last = commits + nr_commits;\n@@ -550,6 +562,9 @@ static void write_graph_chunk_large_edges(struct hashfile *f,\n \n \twhile (list < last) {\n \t\tint num_parents = 0;\n+\n+\t\tdisplay_progress(progress, ++*progress_cnt);\n+\n \t\tfor (parent = (*list)->parents; num_parents < 3 && parent;\n \t\t parent = parent->next)\n \t\t\tnum_parents++;\n@@ -764,6 +779,7 @@ void write_commit_graph(const char *obj_dir,\n \tint num_large_edges;\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n+\tuint64_t progress_cnt = 0;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n@@ -937,11 +953,25 @@ void write_commit_graph(const char *obj_dir,\n \t\thashwrite(f, chunk_write, 12);\n \t}\n \n-\twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n-\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n-\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n+\tif (report_progress) {\n+\t\t/*\n+\t\t * Each of the write_graph_chunk_*() functions just\n+\t\t * below loops over our N commits. This number must be\n+\t\t * kept in sync with the number of passes we're doing.\n+\t\t */\n+\t\tint graph_passes = 4;\n+\t\tif (num_large_edges)\n+\t\t\tgraph_passes++;\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Writing out commit graph\"),\n+\t\t\tgraph_passes * commits.nr);\n+\t}\n+\twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress, &progress_cnt);\n+\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr, progress, &progress_cnt);\n+\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr, progress, &progress_cnt);\n \tif (num_large_edges)\n-\t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n+\t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr, progress, &progress_cnt);\n+\tstop_progress(&progress);\n \n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363889","messageId":"20181122132823.9883-6-avarab@gmail.com","threadId":"49275","inReplyTo":"20181121012600.26951-1-szeder.dev@gmail.com","subject":"[PATCH v3 05/10] commit-graph write: more descriptive \"writing out\" output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-22T13:28:18Z","receivedAt":"2018-11-22T13:28:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the \"Writing out\" part of the progress output more\ndescriptive. Depending on the shape of the graph we either make 4 or 5\npasses over it.\n\nLet's present this information to the user in case they're wondering\nwhat this number, which is much larger than their number of commits,\nhas to do with writing out the commit graph. Now e.g. on linux.git we\nemit:\n\n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph: 6365442, done.\n Computing commit graph generation numbers: 100% (797222/797222), done.\n Writing out commit graph in 5 passes: 100% (3986110/3986110), done.\n\nA note on i18n: Why are we using the Q_() function and passing a\nnumber & English text for a singular which'll never be used? Because\nthe plural rules of translated languages may not match those of\nEnglish, and to use the plural function we need to use this format.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 10 +++++++++-\n 1 file changed, 9 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex e32a5cc1bc..8e5970f0b9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -780,6 +780,7 @@ void write_commit_graph(const char *obj_dir,\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n \tuint64_t progress_cnt = 0;\n+\tstruct strbuf progress_title = STRBUF_INIT;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n@@ -962,8 +963,13 @@ void write_commit_graph(const char *obj_dir,\n \t\tint graph_passes = 4;\n \t\tif (num_large_edges)\n \t\t\tgraph_passes++;\n+\t\tstrbuf_addf(&progress_title,\n+\t\t\t Q_(\"Writing out commit graph in %d pass\",\n+\t\t\t \"Writing out commit graph in %d passes\",\n+\t\t\t graph_passes),\n+\t\t\t graph_passes);\n \t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Writing out commit graph\"),\n+\t\t\tprogress_title.buf,\n \t\t\tgraph_passes * commits.nr);\n \t}\n \twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress, &progress_cnt);\n@@ -973,6 +979,8 @@ void write_commit_graph(const char *obj_dir,\n \t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr, progress, &progress_cnt);\n \tstop_progress(&progress);\n \n+\tstrbuf_release(&progress_title);\n+\n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n \tcommit_lock_file(&lk);\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363890","messageId":"20181122132823.9883-7-avarab@gmail.com","threadId":"49275","inReplyTo":"20181121012600.26951-1-szeder.dev@gmail.com","subject":"[PATCH v3 06/10] commit-graph write: show progress for object search","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-22T13:28:19Z","receivedAt":"2018-11-22T13:28:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Show the percentage progress for the \"Finding commits for commit\ngraph\" phase for the common case where we're operating on all packs in\nthe repository, as \"commit-graph write\" or \"gc\" will do.\n\nBefore we'd emit on e.g. linux.git with \"commit-graph write\":\n\n Finding commits for commit graph: 6365492, done.\n [...]\n\nAnd now:\n\n Finding commits for commit graph: 100% (6365492/6365492), done.\n [...]\n\nSince the commit graph only includes those commits that are packed\n(via for_each_packed_object(...)) the approximate_object_count()\nreturns the actual number of objects we're going to process.\n\nStill, it is possible due to a race with \"gc\" or another process\nmaintaining packs that the number of objects we're going to process is\nlower than what approximate_object_count() reported. In that case we\ndon't want to stop the progress bar short of 100%. So let's make sure\nit snaps to 100% at the end.\n\nThe inverse case is also possible and more likely. I.e. that a new\npack has been added between approximate_object_count() and\nfor_each_packed_object(). In that case the percentage will go beyond\n100%, and we'll do nothing to snap it back to 100% at the end.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 9 +++++++--\n 1 file changed, 7 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8e5970f0b9..d6166beb19 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -781,12 +781,14 @@ void write_commit_graph(const char *obj_dir,\n \tstruct progress *progress = NULL;\n \tuint64_t progress_cnt = 0;\n \tstruct strbuf progress_title = STRBUF_INIT;\n+\tunsigned long approx_nr_objects;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n \n \toids.nr = 0;\n-\toids.alloc = approximate_object_count() / 32;\n+\tapprox_nr_objects = approximate_object_count();\n+\toids.alloc = approx_nr_objects / 32;\n \toids.progress = NULL;\n \toids.progress_done = 0;\n \n@@ -866,8 +868,11 @@ void write_commit_graph(const char *obj_dir,\n \tif (!pack_indexes && !commit_hex) {\n \t\tif (report_progress)\n \t\t\toids.progress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"), 0);\n+\t\t\t\t_(\"Finding commits for commit graph\"),\n+\t\t\t\tapprox_nr_objects);\n \t\tfor_each_packed_object(add_packed_commits, &oids, 0);\n+\t\tif (oids.progress_done < approx_nr_objects)\n+\t\t\tdisplay_progress(oids.progress, approx_nr_objects);\n \t\tstop_progress(&oids.progress);\n \t}\n \n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363892","messageId":"20181122132823.9883-8-avarab@gmail.com","threadId":"49275","inReplyTo":"20181121012600.26951-1-szeder.dev@gmail.com","subject":"[PATCH v3 07/10] commit-graph write: add more descriptive progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-22T13:28:20Z","receivedAt":"2018-11-22T13:28:47Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the progress output shown when we're searching for commits to\ninclude in the graph more descriptive. This amends code I added in\n7b0f229222 (\"commit-graph write: add progress output\", 2018-09-17).\n\nNow, on linux.git, we'll emit this sort of output in the various modes\nwe support:\n\n $ git commit-graph write\n Finding commits for commit graph among packed objects: 100% (6365492/6365492), done.\n [...]\n\n # Actually we don't emit this since this takes almost no time at\n # all. But if we did (s/_delayed//) we'd show:\n $ git for-each-ref --format='%(objectname)' | git commit-graph write --stdin-commits\n Finding commits for commit graph from 584 refs: 100% (584/584), done.\n [...]\n\n $ (cd .git/objects/pack/ && ls *idx) | git commit-graph write --stdin-pack\n Finding commits for commit graph in 3 packs: 6365492, done.\n [...]\n\nThe middle on of those is going to be the output users might see in\npractice, since it'll be emitted when they get the commit graph via\ngc.writeCommitGraph=true. But as noted above you need a really large\nnumber of refs for this message to show. It'll show up on a test\nrepository I have with ~165k refs:\n\n Finding commits for commit graph from 165203 refs: 100% (165203/165203), done.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 25 ++++++++++++++++++-------\n 1 file changed, 18 insertions(+), 7 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d6166beb19..cb1aebeb79 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -818,8 +818,12 @@ void write_commit_graph(const char *obj_dir,\n \t\tstrbuf_addf(&packname, \"%s/pack/\", obj_dir);\n \t\tdirlen = packname.len;\n \t\tif (report_progress) {\n-\t\t\toids.progress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"), 0);\n+\t\t\tstrbuf_addf(&progress_title,\n+\t\t\t\t Q_(\"Finding commits for commit graph in %d pack\",\n+\t\t\t\t \"Finding commits for commit graph in %d packs\",\n+\t\t\t\t pack_indexes->nr),\n+\t\t\t\t pack_indexes->nr);\n+\t\t\toids.progress = start_delayed_progress(progress_title.buf, 0);\n \t\t\toids.progress_done = 0;\n \t\t}\n \t\tfor (i = 0; i < pack_indexes->nr; i++) {\n@@ -836,14 +840,20 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tfree(p);\n \t\t}\n \t\tstop_progress(&oids.progress);\n+\t\tstrbuf_reset(&progress_title);\n \t\tstrbuf_release(&packname);\n \t}\n \n \tif (commit_hex) {\n-\t\tif (report_progress)\n-\t\t\tprogress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"),\n-\t\t\t\tcommit_hex->nr);\n+\t\tif (report_progress) {\n+\t\t\tstrbuf_addf(&progress_title,\n+\t\t\t\t Q_(\"Finding commits for commit graph from %d ref\",\n+\t\t\t\t \"Finding commits for commit graph from %d refs\",\n+\t\t\t\t commit_hex->nr),\n+\t\t\t\t commit_hex->nr);\n+\t\t\tprogress = start_delayed_progress(progress_title.buf,\n+\t\t\t\t\t\t\t commit_hex->nr);\n+\t\t}\n \t\tfor (i = 0; i < commit_hex->nr; i++) {\n \t\t\tconst char *end;\n \t\t\tstruct object_id oid;\n@@ -863,12 +873,13 @@ void write_commit_graph(const char *obj_dir,\n \t\t\t}\n \t\t}\n \t\tstop_progress(&progress);\n+\t\tstrbuf_reset(&progress_title);\n \t}\n \n \tif (!pack_indexes && !commit_hex) {\n \t\tif (report_progress)\n \t\t\toids.progress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"),\n+\t\t\t\t_(\"Finding commits for commit graph among packed objects\"),\n \t\t\t\tapprox_nr_objects);\n \t\tfor_each_packed_object(add_packed_commits, &oids, 0);\n \t\tif (oids.progress_done < approx_nr_objects)\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363891","messageId":"20181122132823.9883-9-avarab@gmail.com","threadId":"49275","inReplyTo":"20181121012600.26951-1-szeder.dev@gmail.com","subject":"[PATCH v3 08/10] commit-graph write: remove empty line for readability","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-22T13:28:21Z","receivedAt":"2018-11-22T13:28:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Remove the empty line between a QSORT(...) and the subsequent oideq()\nfor-loop. This makes it clearer that the QSORT(...) is being done so\nthat we can run the oideq() loop on adjacent OIDs. Amends code added\nin 08fd81c9b6 (\"commit-graph: implement write_commit_graph()\",\n2018-04-02).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex cb1aebeb79..21751231e0 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -890,7 +890,6 @@ void write_commit_graph(const char *obj_dir,\n \tclose_reachable(&oids, report_progress);\n \n \tQSORT(oids.list, oids.nr, commit_compare);\n-\n \tcount_distinct = 1;\n \tfor (i = 1; i < oids.nr; i++) {\n \t\tif (!oideq(&oids.list[i - 1], &oids.list[i]))\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363894","messageId":"20181122132823.9883-10-avarab@gmail.com","threadId":"49275","inReplyTo":"20181121012600.26951-1-szeder.dev@gmail.com","subject":"[PATCH v3 09/10] commit-graph write: add itermediate progress","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-22T13:28:22Z","receivedAt":"2018-11-22T13:28:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add progress output to sections of code between \"Annotating[...]\" and\n\"Computing[...]generation numbers\". This can collectively take 5-10\nseconds on a large enough repository.\n\nOn a test repository with I have with ~7 million commits and ~50\nmillion objects we'll now emit:\n\n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph among packed objects: 100% (50026015/50026015), done.\n Annotating commit graph: 21567407, done.\n Counting distinct commits in commit graph: 100% (7189147/7189147), done.\n Finding extra edges in commit graph: 100% (7189147/7189147), done.\n Computing commit graph generation numbers: 100% (7144680/7144680), done.\n Writing out commit graph: 21434417, done.\n\nWhereas on a medium-sized repository such as linux.git these new\nprogress bars won't have time to kick in and as before and we'll still\nemit output like:\n\n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph among packed objects: 100% (6365492/6365492), done.\n Annotating commit graph: 2391666, done.\n Computing commit graph generation numbers: 100% (797222/797222), done.\n Writing out commit graph: 2399912, done.\n\nThe \"Counting distinct commits in commit graph\" phase will spend most\nof its time paused at \"0/*\" as we QSORT(...) the list. That's not\noptimal, but at least we don't seem to be stalling anymore.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 13 +++++++++++++\n 1 file changed, 13 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 21751231e0..a6e6eeb56b 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -889,12 +889,19 @@ void write_commit_graph(const char *obj_dir,\n \n \tclose_reachable(&oids, report_progress);\n \n+\tif (report_progress)\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Counting distinct commits in commit graph\"),\n+\t\t\toids.nr);\n+\tdisplay_progress(progress, 0); /* TODO: Measure QSORT() progress */\n \tQSORT(oids.list, oids.nr, commit_compare);\n \tcount_distinct = 1;\n \tfor (i = 1; i < oids.nr; i++) {\n+\t\tdisplay_progress(progress, i + 1);\n \t\tif (!oideq(&oids.list[i - 1], &oids.list[i]))\n \t\t\tcount_distinct++;\n \t}\n+\tstop_progress(&progress);\n \n \tif (count_distinct >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n@@ -904,8 +911,13 @@ void write_commit_graph(const char *obj_dir,\n \tALLOC_ARRAY(commits.list, commits.alloc);\n \n \tnum_large_edges = 0;\n+\tif (report_progress)\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Finding extra edges in commit graph\"),\n+\t\t\toids.nr);\n \tfor (i = 0; i < oids.nr; i++) {\n \t\tint num_parents = 0;\n+\t\tdisplay_progress(progress, i + 1);\n \t\tif (i > 0 && oideq(&oids.list[i - 1], &oids.list[i]))\n \t\t\tcontinue;\n \n@@ -922,6 +934,7 @@ void write_commit_graph(const char *obj_dir,\n \t\tcommits.nr++;\n \t}\n \tnum_chunks = num_large_edges ? 4 : 3;\n+\tstop_progress(&progress);\n \n \tif (commits.nr >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"too many commits to write graph\"));\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363893","messageId":"20181122132823.9883-11-avarab@gmail.com","threadId":"49275","inReplyTo":"20181121012600.26951-1-szeder.dev@gmail.com","subject":"[PATCH v3 10/10] commit-graph write: emit a percentage for all progress","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-22T13:28:23Z","receivedAt":"2018-11-22T13:28:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the \"Annotating commit graph\" progress output to show a\ncompletion percentage. I added this in 7b0f229222 (\"commit-graph\nwrite: add progress output\", 2018-09-17) and evidently didn't notice\nhow easy it was to add a completion percentage.\n\nNow for the very large test repository mentioned in previous commits\nwe'll emit (shows all progress output):\n\n Finding commits for commit graph among packed objects: 100% (48333911/48333911), done.\n Annotating commit graph: 100% (21435984/21435984), done.\n Counting distinct commits in commit graph: 100% (7145328/7145328), done.\n Finding extra edges in commit graph: 100% (7145328/7145328), done.\n Computing commit graph generation numbers: 100% (7145328/7145328), done.\n Writing out commit graph in 5 passes: 100% (35726640/35726640), done.\n\nAnd for linux.git:\n\n Finding commits for commit graph among packed objects: 100% (6365442/6365442), done.\n Annotating commit graph: 100% (2391666/2391666), done.\n Computing commit graph generation numbers: 100% (797222/797222), done.\n Writing out commit graph in 5 passes: 100% (3986110/3986110), done.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 9 ++++++++-\n 1 file changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex a6e6eeb56b..c893466042 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -660,10 +660,17 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n \tstruct commit *commit;\n \tstruct progress *progress = NULL;\n \tint j = 0;\n+\t/*\n+\t * We loop over the OIDs N times to close the graph\n+\t * below. This number must be kept in sync with the number of\n+\t * passes.\n+\t */\n+\tconst int oid_passes = 3;\n \n \tif (report_progress)\n \t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Annotating commit graph\"), 0);\n+\t\t\t_(\"Annotating commit graph\"),\n+\t\t\toid_passes * oids->nr);\n \tfor (i = 0; i < oids->nr; i++) {\n \t\tdisplay_progress(progress, ++j);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363895","messageId":"20181122153922.16912-1-avarab@gmail.com","threadId":"49275","inReplyTo":"20181122132823.9883-1-avarab@gmail.com","subject":"[PATCH v3 00/10] commit-graph write: progress output improvements","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-22T15:39:12Z","receivedAt":"2018-11-22T15:39:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The \"Writing out\" progress output was off-by-one because I'd screwed\nup a merge conflict. Fix that, and update the various progress output.\n\nOn my test setup the \"Annotating commit graph\" progress sometimes\nshows up on linux.git, sometimes not, it's right on that edge of\ntaking 1 second. So always show it in the commit message examples,\nthat's less confusing for the reader.\n\nSZEDER Gábor (2):\n commit-graph: rename 'num_extra_edges' variable to 'num_large_edges'\n commit-graph: don't call write_graph_chunk_large_edges() unnecessarily\n\nÆvar Arnfjörð Bjarmason (8):\n commit-graph write: rephrase confusing progress output\n commit-graph write: add \"Writing out\" progress output\n commit-graph write: more descriptive \"writing out\" output\n commit-graph write: show progress for object search\n commit-graph write: add more descriptive progress output\n commit-graph write: remove empty line for readability\n commit-graph write: add itermediate progress\n commit-graph write: emit a percentage for all progress\n\n commit-graph.c | 130 ++++++++++++++++++++++++++++++++++++++-----------\n 1 file changed, 102 insertions(+), 28 deletions(-)\n\nRange-diff:\n1: 2b52ad2284 ! 1: 9c17f56ed3 commit-graph write: add \"Writing out\" progress output\n a => b | 0\n 1 file changed, 0 insertions(+), 0 deletions(-)\n \n @@ -15,10 +15,11 @@\n \n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph: 6365442, done.\n + Annotating commit graph: 2391666, done.\n Computing commit graph generation numbers: 100% (797222/797222), done.\n - Writing out commit graph: 100% (3986110/3986110), done.\n + Writing out commit graph: 100% (3188888/3188888), done.\n \n - This \"Writing out\" number is 4x or 5x the number of commits, depending\n + This \"Writing out\" number is 3x or 4x the number of commits, depending\n on the graph we're processing. A later change will make this explicit\n to the user.\n \n @@ -126,7 +127,7 @@\n +\t\t * below loops over our N commits. This number must be\n +\t\t * kept in sync with the number of passes we're doing.\n +\t\t */\n -+\t\tint graph_passes = 4;\n ++\t\tint graph_passes = 3;\n +\t\tif (num_large_edges)\n +\t\t\tgraph_passes++;\n +\t\tprogress = start_delayed_progress(\n2: b1773677b1 ! 2: 79b0a467d9 commit-graph write: more descriptive \"writing out\" output\n a => b | 0\n 1 file changed, 0 insertions(+), 0 deletions(-)\n \n @@ -3,7 +3,7 @@\n commit-graph write: more descriptive \"writing out\" output\n \n Make the \"Writing out\" part of the progress output more\n - descriptive. Depending on the shape of the graph we either make 4 or 5\n + descriptive. Depending on the shape of the graph we either make 3 or 4\n passes over it.\n \n Let's present this information to the user in case they're wondering\n @@ -13,8 +13,9 @@\n \n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph: 6365442, done.\n + Annotating commit graph: 2391666, done.\n Computing commit graph generation numbers: 100% (797222/797222), done.\n - Writing out commit graph in 5 passes: 100% (3986110/3986110), done.\n + Writing out commit graph in 4 passes: 100% (3188888/3188888), done.\n \n A note on i18n: Why are we using the Q_() function and passing a\n number & English text for a singular which'll never be used? Because\n @@ -35,7 +36,7 @@\n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n @@\n - \t\tint graph_passes = 4;\n + \t\tint graph_passes = 3;\n \t\tif (num_large_edges)\n \t\t\tgraph_passes++;\n +\t\tstrbuf_addf(&progress_title,\n3: 3138b00a2c ! 3: b32be83b38 commit-graph write: show progress for object search\n a => b | 0\n 1 file changed, 0 insertions(+), 0 deletions(-)\n \n @@ -8,12 +8,12 @@\n \n Before we'd emit on e.g. linux.git with \"commit-graph write\":\n \n - Finding commits for commit graph: 6365492, done.\n + Finding commits for commit graph: 6365442, done.\n [...]\n \n And now:\n \n - Finding commits for commit graph: 100% (6365492/6365492), done.\n + Finding commits for commit graph: 100% (6365442/6365442), done.\n [...]\n \n Since the commit graph only includes those commits that are packed\n4: f41e3b3eb3 ! 4: 54276723c0 commit-graph write: add more descriptive progress output\n a => b | 0\n 1 file changed, 0 insertions(+), 0 deletions(-)\n \n @@ -10,7 +10,7 @@\n we support:\n \n $ git commit-graph write\n - Finding commits for commit graph among packed objects: 100% (6365492/6365492), done.\n + Finding commits for commit graph among packed objects: 100% (6365442/6365442), done.\n [...]\n \n # Actually we don't emit this since this takes almost no time at\n @@ -20,7 +20,7 @@\n [...]\n \n $ (cd .git/objects/pack/ && ls *idx) | git commit-graph write --stdin-pack\n - Finding commits for commit graph in 3 packs: 6365492, done.\n + Finding commits for commit graph in 2 packs: 6365442, done.\n [...]\n \n The middle on of those is going to be the output users might see in\n5: 74037032d3 = 5: 0e847366e1 commit-graph write: remove empty line for readability\n a => b | 0\n 1 file changed, 0 insertions(+), 0 deletions(-)\n \n6: 502da68d14 ! 6: c388aff73e commit-graph write: add itermediate progress\n a => b | 0\n 1 file changed, 0 insertions(+), 0 deletions(-)\n \n @@ -10,22 +10,22 @@\n million objects we'll now emit:\n \n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n - Finding commits for commit graph among packed objects: 100% (50026015/50026015), done.\n - Annotating commit graph: 21567407, done.\n - Counting distinct commits in commit graph: 100% (7189147/7189147), done.\n - Finding extra edges in commit graph: 100% (7189147/7189147), done.\n - Computing commit graph generation numbers: 100% (7144680/7144680), done.\n - Writing out commit graph: 21434417, done.\n + Finding commits for commit graph among packed objects: 100% (48333911/48333911), done.\n + Annotating commit graph: 21435984, done.\n + Counting distinct commits in commit graph: 100% (7145328/7145328), done.\n + Finding extra edges in commit graph: 100% (7145328/7145328), done.\n + Computing commit graph generation numbers: 100% (7145328/7145328), done.\n + Writing out commit graph in 4 passes: 100% (28581312/28581312), done.\n \n Whereas on a medium-sized repository such as linux.git these new\n progress bars won't have time to kick in and as before and we'll still\n emit output like:\n \n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n - Finding commits for commit graph among packed objects: 100% (6365492/6365492), done.\n + Finding commits for commit graph among packed objects: 100% (6365442/6365442), done.\n Annotating commit graph: 2391666, done.\n Computing commit graph generation numbers: 100% (797222/797222), done.\n - Writing out commit graph: 2399912, done.\n + Writing out commit graph in 4 passes: 100% (3188888/3188888), done.\n \n The \"Counting distinct commits in commit graph\" phase will spend most\n of its time paused at \"0/*\" as we QSORT(...) the list. That's not\n7: dfaf840983 ! 7: fd692499e0 commit-graph write: emit a percentage for all progress\n a => b | 0\n 1 file changed, 0 insertions(+), 0 deletions(-)\n \n @@ -7,22 +7,13 @@\n write: add progress output\", 2018-09-17) and evidently didn't notice\n how easy it was to add a completion percentage.\n \n - Now for the very large test repository mentioned in previous commits\n - we'll emit (shows all progress output):\n -\n - Finding commits for commit graph among packed objects: 100% (48333911/48333911), done.\n - Annotating commit graph: 100% (21435984/21435984), done.\n - Counting distinct commits in commit graph: 100% (7145328/7145328), done.\n - Finding extra edges in commit graph: 100% (7145328/7145328), done.\n - Computing commit graph generation numbers: 100% (7145328/7145328), done.\n - Writing out commit graph in 5 passes: 100% (35726640/35726640), done.\n -\n - And for linux.git:\n + Now for e.g. linux.git we'll emit:\n \n + ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph among packed objects: 100% (6365442/6365442), done.\n Annotating commit graph: 100% (2391666/2391666), done.\n Computing commit graph generation numbers: 100% (797222/797222), done.\n - Writing out commit graph in 5 passes: 100% (3986110/3986110), done.\n + Writing out commit graph in 4 passes: 100% (3188888/3188888), done.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363896","messageId":"20181122153922.16912-2-avarab@gmail.com","threadId":"49275","inReplyTo":"20181122132823.9883-1-avarab@gmail.com","subject":"[PATCH v4 01/10] commit-graph: rename 'num_extra_edges' variable to 'num_large_edges'","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-22T15:39:13Z","receivedAt":"2018-11-22T15:39:39Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: SZEDER Gábor \n\nThe commit graph file format describes an optional 'Large Edge List'\nchunk, and the function writing out this chunk is called\nwrite_graph_chunk_large_edges(). Then there are two functions in\n'commit-graph.c', namely write_graph_chunk_data() and\nwrite_commit_graph(), which have a local variable called\n'num_extra_edges'.\n\nIt can be confusing on first sight whether large edges and extra edges\nrefer to the same thing or not, but they do, so let's rename those\nvariables to 'num_large_edges'.\n\nSigned-off-by: SZEDER Gábor \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 18 +++++++++---------\n 1 file changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 40c855f185..7b4e3a02cf 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -475,7 +475,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n {\n \tstruct commit **list = commits;\n \tstruct commit **last = commits + nr_commits;\n-\tuint32_t num_extra_edges = 0;\n+\tuint32_t num_large_edges = 0;\n \n \twhile (list < last) {\n \t\tstruct commit_list *parent;\n@@ -507,7 +507,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\tif (!parent)\n \t\t\tedge_value = GRAPH_PARENT_NONE;\n \t\telse if (parent->next)\n-\t\t\tedge_value = GRAPH_OCTOPUS_EDGES_NEEDED | num_extra_edges;\n+\t\t\tedge_value = GRAPH_OCTOPUS_EDGES_NEEDED | num_large_edges;\n \t\telse {\n \t\t\tedge_value = sha1_pos(parent->item->object.oid.hash,\n \t\t\t\t\t commits,\n@@ -521,7 +521,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \n \t\tif (edge_value & GRAPH_OCTOPUS_EDGES_NEEDED) {\n \t\t\tdo {\n-\t\t\t\tnum_extra_edges++;\n+\t\t\t\tnum_large_edges++;\n \t\t\t\tparent = parent->next;\n \t\t\t} while (parent);\n \t\t}\n@@ -761,7 +761,7 @@ void write_commit_graph(const char *obj_dir,\n \tuint32_t chunk_ids[5];\n \tuint64_t chunk_offsets[5];\n \tint num_chunks;\n-\tint num_extra_edges;\n+\tint num_large_edges;\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n \n@@ -871,7 +871,7 @@ void write_commit_graph(const char *obj_dir,\n \tcommits.alloc = count_distinct;\n \tALLOC_ARRAY(commits.list, commits.alloc);\n \n-\tnum_extra_edges = 0;\n+\tnum_large_edges = 0;\n \tfor (i = 0; i < oids.nr; i++) {\n \t\tint num_parents = 0;\n \t\tif (i > 0 && oideq(&oids.list[i - 1], &oids.list[i]))\n@@ -885,11 +885,11 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tnum_parents++;\n \n \t\tif (num_parents > 2)\n-\t\t\tnum_extra_edges += num_parents - 1;\n+\t\t\tnum_large_edges += num_parents - 1;\n \n \t\tcommits.nr++;\n \t}\n-\tnum_chunks = num_extra_edges ? 4 : 3;\n+\tnum_chunks = num_large_edges ? 4 : 3;\n \n \tif (commits.nr >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"too many commits to write graph\"));\n@@ -916,7 +916,7 @@ void write_commit_graph(const char *obj_dir,\n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n \tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n-\tif (num_extra_edges)\n+\tif (num_large_edges)\n \t\tchunk_ids[3] = GRAPH_CHUNKID_LARGEEDGES;\n \telse\n \t\tchunk_ids[3] = 0;\n@@ -926,7 +926,7 @@ void write_commit_graph(const char *obj_dir,\n \tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n \tchunk_offsets[2] = chunk_offsets[1] + GRAPH_OID_LEN * commits.nr;\n \tchunk_offsets[3] = chunk_offsets[2] + (GRAPH_OID_LEN + 16) * commits.nr;\n-\tchunk_offsets[4] = chunk_offsets[3] + 4 * num_extra_edges;\n+\tchunk_offsets[4] = chunk_offsets[3] + 4 * num_large_edges;\n \n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tuint32_t chunk_write[3];\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363897","messageId":"20181122153922.16912-3-avarab@gmail.com","threadId":"49275","inReplyTo":"20181122132823.9883-1-avarab@gmail.com","subject":"[PATCH v4 02/10] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-22T15:39:14Z","receivedAt":"2018-11-22T15:39:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: SZEDER Gábor \n\nThe optional 'Large Edge List' chunk of the commit graph file stores\nparent information for commits with more than two parents. Since the\nchunk is optional, write_commit_graph() looks through all commits to\nfind those with more than two parents, and then writes the commit\ngraph file header accordingly, i.e. if there are no such commits, then\nthere won't be a 'Large Edge List' chunk written, only the three\nmandatory chunks.\n\nHowever, when it comes to writing chunk data, write_commit_graph()\nunconditionally invokes write_graph_chunk_large_edges(), even when it\nwas decided earlier that that chunk won't be written. Strictly\nspeaking there is no bug here, because write_graph_chunk_large_edges()\nwon't write anything because it won't find any commits with more than\ntwo parents, but then it unnecessarily and in vain looks through all\ncommits once again in search for such commits.\n\nDon't call write_graph_chunk_large_edges() when that chunk won't be\nwritten to spare an unnecessary iteration over all commits.\n\nSigned-off-by: SZEDER Gábor \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 7b4e3a02cf..965eb23a7b 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -940,7 +940,8 @@ void write_commit_graph(const char *obj_dir,\n \twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n \twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n \twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n-\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n+\tif (num_large_edges)\n+\t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n \n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363898","messageId":"20181122153922.16912-4-avarab@gmail.com","threadId":"49275","inReplyTo":"20181122132823.9883-1-avarab@gmail.com","subject":"[PATCH v4 03/10] commit-graph write: rephrase confusing progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-22T15:39:15Z","receivedAt":"2018-11-22T15:39:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Rephrase the title shown for the progress output emitted by\nclose_reachable(). The message I added in 7b0f229222 (\"commit-graph\nwrite: add progress output\", 2018-09-17) gave the impression that it\nwould count up to the number of commit objects.\n\nBut that's not what the number means. It just represents the work\nwe're doing in several for-loops to do various work before the graph\nis written out. So let's just say \"Annotating commit graph\", that\ntitle makes no such promises, and we can add other loops here in the\nfuture and still consistently show progress output.\n\nSee [1] for the initial bug report & subsequent discussion about other\napproaching to solving this.\n\n1. https://public-inbox.org/git/20181015165447.GH19800@szeder.dev/\n\nReported-by: SZEDER Gábor \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 965eb23a7b..d11370a2b3 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -648,7 +648,7 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n \n \tif (report_progress)\n \t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Annotating commits in commit graph\"), 0);\n+\t\t\t_(\"Annotating commit graph\"), 0);\n \tfor (i = 0; i < oids->nr; i++) {\n \t\tdisplay_progress(progress, ++j);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363899","messageId":"20181122153922.16912-6-avarab@gmail.com","threadId":"49275","inReplyTo":"20181122132823.9883-1-avarab@gmail.com","subject":"[PATCH v4 05/10] commit-graph write: more descriptive \"writing out\" output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-22T15:39:17Z","receivedAt":"2018-11-22T15:39:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the \"Writing out\" part of the progress output more\ndescriptive. Depending on the shape of the graph we either make 3 or 4\npasses over it.\n\nLet's present this information to the user in case they're wondering\nwhat this number, which is much larger than their number of commits,\nhas to do with writing out the commit graph. Now e.g. on linux.git we\nemit:\n\n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph: 6365442, done.\n Annotating commit graph: 2391666, done.\n Computing commit graph generation numbers: 100% (797222/797222), done.\n Writing out commit graph in 4 passes: 100% (3188888/3188888), done.\n\nA note on i18n: Why are we using the Q_() function and passing a\nnumber & English text for a singular which'll never be used? Because\nthe plural rules of translated languages may not match those of\nEnglish, and to use the plural function we need to use this format.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 10 +++++++++-\n 1 file changed, 9 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex dc57b8fedc..3de65bc2e9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -780,6 +780,7 @@ void write_commit_graph(const char *obj_dir,\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n \tuint64_t progress_cnt = 0;\n+\tstruct strbuf progress_title = STRBUF_INIT;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n@@ -962,8 +963,13 @@ void write_commit_graph(const char *obj_dir,\n \t\tint graph_passes = 3;\n \t\tif (num_large_edges)\n \t\t\tgraph_passes++;\n+\t\tstrbuf_addf(&progress_title,\n+\t\t\t Q_(\"Writing out commit graph in %d pass\",\n+\t\t\t \"Writing out commit graph in %d passes\",\n+\t\t\t graph_passes),\n+\t\t\t graph_passes);\n \t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Writing out commit graph\"),\n+\t\t\tprogress_title.buf,\n \t\t\tgraph_passes * commits.nr);\n \t}\n \twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress, &progress_cnt);\n@@ -973,6 +979,8 @@ void write_commit_graph(const char *obj_dir,\n \t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr, progress, &progress_cnt);\n \tstop_progress(&progress);\n \n+\tstrbuf_release(&progress_title);\n+\n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n \tcommit_lock_file(&lk);\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363900","messageId":"20181122153922.16912-5-avarab@gmail.com","threadId":"49275","inReplyTo":"20181122132823.9883-1-avarab@gmail.com","subject":"[PATCH v4 04/10] commit-graph write: add \"Writing out\" progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-22T15:39:16Z","receivedAt":"2018-11-22T15:39:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add progress output to be shown when we're writing out the\ncommit-graph, this adds to the output already added in 7b0f229222\n(\"commit-graph write: add progress output\", 2018-09-17).\n\nAs noted in that commit most of the progress output isn't displayed on\nsmall repositories, but before this change we'd noticeably hang for\n2-3 seconds at the end on medium sized repositories such as linux.git.\n\nNow we'll instead show output like this, and have no human-observable\npoint at which we're not producing progress output:\n\n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph: 6365442, done.\n Annotating commit graph: 2391666, done.\n Computing commit graph generation numbers: 100% (797222/797222), done.\n Writing out commit graph: 100% (3188888/3188888), done.\n\nThis \"Writing out\" number is 3x or 4x the number of commits, depending\non the graph we're processing. A later change will make this explicit\nto the user.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 48 +++++++++++++++++++++++++++++++++++++++---------\n 1 file changed, 39 insertions(+), 9 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d11370a2b3..dc57b8fedc 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -433,7 +433,9 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n \t\t\t\t struct commit **commits,\n-\t\t\t\t int nr_commits)\n+\t\t\t\t int nr_commits,\n+\t\t\t\t struct progress *progress,\n+\t\t\t\t uint64_t *progress_cnt)\n {\n \tint i, count = 0;\n \tstruct commit **list = commits;\n@@ -447,6 +449,7 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n \t\twhile (count < nr_commits) {\n \t\t\tif ((*list)->object.oid.hash[0] != i)\n \t\t\t\tbreak;\n+\t\t\tdisplay_progress(progress, ++*progress_cnt);\n \t\t\tcount++;\n \t\t\tlist++;\n \t\t}\n@@ -456,12 +459,16 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n }\n \n static void write_graph_chunk_oids(struct hashfile *f, int hash_len,\n-\t\t\t\t struct commit **commits, int nr_commits)\n+\t\t\t\t struct commit **commits, int nr_commits,\n+\t\t\t\t struct progress *progress,\n+\t\t\t\t uint64_t *progress_cnt)\n {\n \tstruct commit **list = commits;\n \tint count;\n-\tfor (count = 0; count < nr_commits; count++, list++)\n+\tfor (count = 0; count < nr_commits; count++, list++) {\n+\t\tdisplay_progress(progress, ++*progress_cnt);\n \t\thashwrite(f, (*list)->object.oid.hash, (int)hash_len);\n+\t}\n }\n \n static const unsigned char *commit_to_sha1(size_t index, void *table)\n@@ -471,7 +478,9 @@ static const unsigned char *commit_to_sha1(size_t index, void *table)\n }\n \n static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n-\t\t\t\t struct commit **commits, int nr_commits)\n+\t\t\t\t struct commit **commits, int nr_commits,\n+\t\t\t\t struct progress *progress,\n+\t\t\t\t uint64_t *progress_cnt)\n {\n \tstruct commit **list = commits;\n \tstruct commit **last = commits + nr_commits;\n@@ -481,6 +490,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\tstruct commit_list *parent;\n \t\tint edge_value;\n \t\tuint32_t packedDate[2];\n+\t\tdisplay_progress(progress, ++*progress_cnt);\n \n \t\tparse_commit(*list);\n \t\thashwrite(f, get_commit_tree_oid(*list)->hash, hash_len);\n@@ -542,7 +552,9 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \n static void write_graph_chunk_large_edges(struct hashfile *f,\n \t\t\t\t\t struct commit **commits,\n-\t\t\t\t\t int nr_commits)\n+\t\t\t\t\t int nr_commits,\n+\t\t\t\t\t struct progress *progress,\n+\t\t\t\t\t uint64_t *progress_cnt)\n {\n \tstruct commit **list = commits;\n \tstruct commit **last = commits + nr_commits;\n@@ -550,6 +562,9 @@ static void write_graph_chunk_large_edges(struct hashfile *f,\n \n \twhile (list < last) {\n \t\tint num_parents = 0;\n+\n+\t\tdisplay_progress(progress, ++*progress_cnt);\n+\n \t\tfor (parent = (*list)->parents; num_parents < 3 && parent;\n \t\t parent = parent->next)\n \t\t\tnum_parents++;\n@@ -764,6 +779,7 @@ void write_commit_graph(const char *obj_dir,\n \tint num_large_edges;\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n+\tuint64_t progress_cnt = 0;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n@@ -937,11 +953,25 @@ void write_commit_graph(const char *obj_dir,\n \t\thashwrite(f, chunk_write, 12);\n \t}\n \n-\twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n-\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n-\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n+\tif (report_progress) {\n+\t\t/*\n+\t\t * Each of the write_graph_chunk_*() functions just\n+\t\t * below loops over our N commits. This number must be\n+\t\t * kept in sync with the number of passes we're doing.\n+\t\t */\n+\t\tint graph_passes = 3;\n+\t\tif (num_large_edges)\n+\t\t\tgraph_passes++;\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Writing out commit graph\"),\n+\t\t\tgraph_passes * commits.nr);\n+\t}\n+\twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress, &progress_cnt);\n+\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr, progress, &progress_cnt);\n+\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr, progress, &progress_cnt);\n \tif (num_large_edges)\n-\t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n+\t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr, progress, &progress_cnt);\n+\tstop_progress(&progress);\n \n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363901","messageId":"20181122153922.16912-7-avarab@gmail.com","threadId":"49275","inReplyTo":"20181122132823.9883-1-avarab@gmail.com","subject":"[PATCH v4 06/10] commit-graph write: show progress for object search","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-22T15:39:18Z","receivedAt":"2018-11-22T15:39:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Show the percentage progress for the \"Finding commits for commit\ngraph\" phase for the common case where we're operating on all packs in\nthe repository, as \"commit-graph write\" or \"gc\" will do.\n\nBefore we'd emit on e.g. linux.git with \"commit-graph write\":\n\n Finding commits for commit graph: 6365442, done.\n [...]\n\nAnd now:\n\n Finding commits for commit graph: 100% (6365442/6365442), done.\n [...]\n\nSince the commit graph only includes those commits that are packed\n(via for_each_packed_object(...)) the approximate_object_count()\nreturns the actual number of objects we're going to process.\n\nStill, it is possible due to a race with \"gc\" or another process\nmaintaining packs that the number of objects we're going to process is\nlower than what approximate_object_count() reported. In that case we\ndon't want to stop the progress bar short of 100%. So let's make sure\nit snaps to 100% at the end.\n\nThe inverse case is also possible and more likely. I.e. that a new\npack has been added between approximate_object_count() and\nfor_each_packed_object(). In that case the percentage will go beyond\n100%, and we'll do nothing to snap it back to 100% at the end.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 9 +++++++--\n 1 file changed, 7 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 3de65bc2e9..42d8365f0d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -781,12 +781,14 @@ void write_commit_graph(const char *obj_dir,\n \tstruct progress *progress = NULL;\n \tuint64_t progress_cnt = 0;\n \tstruct strbuf progress_title = STRBUF_INIT;\n+\tunsigned long approx_nr_objects;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n \n \toids.nr = 0;\n-\toids.alloc = approximate_object_count() / 32;\n+\tapprox_nr_objects = approximate_object_count();\n+\toids.alloc = approx_nr_objects / 32;\n \toids.progress = NULL;\n \toids.progress_done = 0;\n \n@@ -866,8 +868,11 @@ void write_commit_graph(const char *obj_dir,\n \tif (!pack_indexes && !commit_hex) {\n \t\tif (report_progress)\n \t\t\toids.progress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"), 0);\n+\t\t\t\t_(\"Finding commits for commit graph\"),\n+\t\t\t\tapprox_nr_objects);\n \t\tfor_each_packed_object(add_packed_commits, &oids, 0);\n+\t\tif (oids.progress_done < approx_nr_objects)\n+\t\t\tdisplay_progress(oids.progress, approx_nr_objects);\n \t\tstop_progress(&oids.progress);\n \t}\n \n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363902","messageId":"20181122153922.16912-8-avarab@gmail.com","threadId":"49275","inReplyTo":"20181122132823.9883-1-avarab@gmail.com","subject":"[PATCH v4 07/10] commit-graph write: add more descriptive progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-22T15:39:19Z","receivedAt":"2018-11-22T15:39:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the progress output shown when we're searching for commits to\ninclude in the graph more descriptive. This amends code I added in\n7b0f229222 (\"commit-graph write: add progress output\", 2018-09-17).\n\nNow, on linux.git, we'll emit this sort of output in the various modes\nwe support:\n\n $ git commit-graph write\n Finding commits for commit graph among packed objects: 100% (6365442/6365442), done.\n [...]\n\n # Actually we don't emit this since this takes almost no time at\n # all. But if we did (s/_delayed//) we'd show:\n $ git for-each-ref --format='%(objectname)' | git commit-graph write --stdin-commits\n Finding commits for commit graph from 584 refs: 100% (584/584), done.\n [...]\n\n $ (cd .git/objects/pack/ && ls *idx) | git commit-graph write --stdin-pack\n Finding commits for commit graph in 2 packs: 6365442, done.\n [...]\n\nThe middle on of those is going to be the output users might see in\npractice, since it'll be emitted when they get the commit graph via\ngc.writeCommitGraph=true. But as noted above you need a really large\nnumber of refs for this message to show. It'll show up on a test\nrepository I have with ~165k refs:\n\n Finding commits for commit graph from 165203 refs: 100% (165203/165203), done.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 25 ++++++++++++++++++-------\n 1 file changed, 18 insertions(+), 7 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 42d8365f0d..43b15785f6 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -818,8 +818,12 @@ void write_commit_graph(const char *obj_dir,\n \t\tstrbuf_addf(&packname, \"%s/pack/\", obj_dir);\n \t\tdirlen = packname.len;\n \t\tif (report_progress) {\n-\t\t\toids.progress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"), 0);\n+\t\t\tstrbuf_addf(&progress_title,\n+\t\t\t\t Q_(\"Finding commits for commit graph in %d pack\",\n+\t\t\t\t \"Finding commits for commit graph in %d packs\",\n+\t\t\t\t pack_indexes->nr),\n+\t\t\t\t pack_indexes->nr);\n+\t\t\toids.progress = start_delayed_progress(progress_title.buf, 0);\n \t\t\toids.progress_done = 0;\n \t\t}\n \t\tfor (i = 0; i < pack_indexes->nr; i++) {\n@@ -836,14 +840,20 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tfree(p);\n \t\t}\n \t\tstop_progress(&oids.progress);\n+\t\tstrbuf_reset(&progress_title);\n \t\tstrbuf_release(&packname);\n \t}\n \n \tif (commit_hex) {\n-\t\tif (report_progress)\n-\t\t\tprogress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"),\n-\t\t\t\tcommit_hex->nr);\n+\t\tif (report_progress) {\n+\t\t\tstrbuf_addf(&progress_title,\n+\t\t\t\t Q_(\"Finding commits for commit graph from %d ref\",\n+\t\t\t\t \"Finding commits for commit graph from %d refs\",\n+\t\t\t\t commit_hex->nr),\n+\t\t\t\t commit_hex->nr);\n+\t\t\tprogress = start_delayed_progress(progress_title.buf,\n+\t\t\t\t\t\t\t commit_hex->nr);\n+\t\t}\n \t\tfor (i = 0; i < commit_hex->nr; i++) {\n \t\t\tconst char *end;\n \t\t\tstruct object_id oid;\n@@ -863,12 +873,13 @@ void write_commit_graph(const char *obj_dir,\n \t\t\t}\n \t\t}\n \t\tstop_progress(&progress);\n+\t\tstrbuf_reset(&progress_title);\n \t}\n \n \tif (!pack_indexes && !commit_hex) {\n \t\tif (report_progress)\n \t\t\toids.progress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"),\n+\t\t\t\t_(\"Finding commits for commit graph among packed objects\"),\n \t\t\t\tapprox_nr_objects);\n \t\tfor_each_packed_object(add_packed_commits, &oids, 0);\n \t\tif (oids.progress_done < approx_nr_objects)\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363903","messageId":"20181122153922.16912-9-avarab@gmail.com","threadId":"49275","inReplyTo":"20181122132823.9883-1-avarab@gmail.com","subject":"[PATCH v4 08/10] commit-graph write: remove empty line for readability","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-22T15:39:20Z","receivedAt":"2018-11-22T15:39:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Remove the empty line between a QSORT(...) and the subsequent oideq()\nfor-loop. This makes it clearer that the QSORT(...) is being done so\nthat we can run the oideq() loop on adjacent OIDs. Amends code added\nin 08fd81c9b6 (\"commit-graph: implement write_commit_graph()\",\n2018-04-02).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 43b15785f6..199155bd68 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -890,7 +890,6 @@ void write_commit_graph(const char *obj_dir,\n \tclose_reachable(&oids, report_progress);\n \n \tQSORT(oids.list, oids.nr, commit_compare);\n-\n \tcount_distinct = 1;\n \tfor (i = 1; i < oids.nr; i++) {\n \t\tif (!oideq(&oids.list[i - 1], &oids.list[i]))\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363904","messageId":"20181122153922.16912-10-avarab@gmail.com","threadId":"49275","inReplyTo":"20181122132823.9883-1-avarab@gmail.com","subject":"[PATCH v4 09/10] commit-graph write: add itermediate progress","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-22T15:39:21Z","receivedAt":"2018-11-22T15:39:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add progress output to sections of code between \"Annotating[...]\" and\n\"Computing[...]generation numbers\". This can collectively take 5-10\nseconds on a large enough repository.\n\nOn a test repository with I have with ~7 million commits and ~50\nmillion objects we'll now emit:\n\n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph among packed objects: 100% (48333911/48333911), done.\n Annotating commit graph: 21435984, done.\n Counting distinct commits in commit graph: 100% (7145328/7145328), done.\n Finding extra edges in commit graph: 100% (7145328/7145328), done.\n Computing commit graph generation numbers: 100% (7145328/7145328), done.\n Writing out commit graph in 4 passes: 100% (28581312/28581312), done.\n\nWhereas on a medium-sized repository such as linux.git these new\nprogress bars won't have time to kick in and as before and we'll still\nemit output like:\n\n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph among packed objects: 100% (6365442/6365442), done.\n Annotating commit graph: 2391666, done.\n Computing commit graph generation numbers: 100% (797222/797222), done.\n Writing out commit graph in 4 passes: 100% (3188888/3188888), done.\n\nThe \"Counting distinct commits in commit graph\" phase will spend most\nof its time paused at \"0/*\" as we QSORT(...) the list. That's not\noptimal, but at least we don't seem to be stalling anymore.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 13 +++++++++++++\n 1 file changed, 13 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 199155bd68..80f201adf4 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -889,12 +889,19 @@ void write_commit_graph(const char *obj_dir,\n \n \tclose_reachable(&oids, report_progress);\n \n+\tif (report_progress)\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Counting distinct commits in commit graph\"),\n+\t\t\toids.nr);\n+\tdisplay_progress(progress, 0); /* TODO: Measure QSORT() progress */\n \tQSORT(oids.list, oids.nr, commit_compare);\n \tcount_distinct = 1;\n \tfor (i = 1; i < oids.nr; i++) {\n+\t\tdisplay_progress(progress, i + 1);\n \t\tif (!oideq(&oids.list[i - 1], &oids.list[i]))\n \t\t\tcount_distinct++;\n \t}\n+\tstop_progress(&progress);\n \n \tif (count_distinct >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n@@ -904,8 +911,13 @@ void write_commit_graph(const char *obj_dir,\n \tALLOC_ARRAY(commits.list, commits.alloc);\n \n \tnum_large_edges = 0;\n+\tif (report_progress)\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Finding extra edges in commit graph\"),\n+\t\t\toids.nr);\n \tfor (i = 0; i < oids.nr; i++) {\n \t\tint num_parents = 0;\n+\t\tdisplay_progress(progress, i + 1);\n \t\tif (i > 0 && oideq(&oids.list[i - 1], &oids.list[i]))\n \t\t\tcontinue;\n \n@@ -922,6 +934,7 @@ void write_commit_graph(const char *obj_dir,\n \t\tcommits.nr++;\n \t}\n \tnum_chunks = num_large_edges ? 4 : 3;\n+\tstop_progress(&progress);\n \n \tif (commits.nr >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"too many commits to write graph\"));\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363905","messageId":"20181122153922.16912-11-avarab@gmail.com","threadId":"49275","inReplyTo":"20181122132823.9883-1-avarab@gmail.com","subject":"[PATCH v4 10/10] commit-graph write: emit a percentage for all progress","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-22T15:39:22Z","receivedAt":"2018-11-22T15:39:53Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the \"Annotating commit graph\" progress output to show a\ncompletion percentage. I added this in 7b0f229222 (\"commit-graph\nwrite: add progress output\", 2018-09-17) and evidently didn't notice\nhow easy it was to add a completion percentage.\n\nNow for e.g. linux.git we'll emit:\n\n ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph among packed objects: 100% (6365442/6365442), done.\n Annotating commit graph: 100% (2391666/2391666), done.\n Computing commit graph generation numbers: 100% (797222/797222), done.\n Writing out commit graph in 4 passes: 100% (3188888/3188888), done.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 9 ++++++++-\n 1 file changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 80f201adf4..6c6edc679b 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -660,10 +660,17 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n \tstruct commit *commit;\n \tstruct progress *progress = NULL;\n \tint j = 0;\n+\t/*\n+\t * We loop over the OIDs N times to close the graph\n+\t * below. This number must be kept in sync with the number of\n+\t * passes.\n+\t */\n+\tconst int oid_passes = 3;\n \n \tif (report_progress)\n \t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Annotating commit graph\"), 0);\n+\t\t\t_(\"Annotating commit graph\"),\n+\t\t\toid_passes * oids->nr);\n \tfor (i = 0; i < oids->nr; i++) {\n \t\tdisplay_progress(progress, ++j);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n-- \n2.20.0.rc0.387.gc7a69e6b6c\n\n"},{"id":"363923","messageId":"CAPig+cRCrbnoQoX72bmRcuYVP9b8W0gfrg0tL4kmuKfsYGa3bA@mail.gmail.com","threadId":"49275","inReplyTo":"20181122132823.9883-1-avarab@gmail.com","subject":"Re: [PATCH v3 00/10] commit-graph write: progress output improvements","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-11-22T18:59:22Z","receivedAt":"2018-11-22T18:59:36Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Nov 22, 2018 at 8:28 AM Ævar Arnfjörð Bjarmason\n wrote:\n> Range-diff:\n> By the way, is there any way to....\n> Pass the equivalent of \"git range-diff origin/master topic-2 topic-3\"\n> to git-format-patch?\n\ngit-range-diff documentations says that the three-argument form:\n\n git range-diff \n\nis equivalent to passing two ranges:\n\n git range-diff .. ..\n\ngit-format-patch synopsis shows:\n\n git format-patch --range-diff= \n\nwhere is the range of commits to format, and \ncan be a range specifying the previous version, so:\n\n git format-patch --range-diff=.. ..\n\nshould do what you ask.\n\nHowever, since the two versions in your example both derive from\norigin/master, you should be able to get by with the simpler:\n\n git format-patch --range-diff= ..\n\nwhich, if you were running git-range-diff manually, would be the equivalent of:\n\n git range-diff ...\n\nfor which the range-diff machinery figures out the common base\n(origin/master) automatically.\n"},{"id":"366816","messageId":"20190116132947.26025-1-avarab@gmail.com","threadId":"49275","inReplyTo":"20181122153922.16912-2-avarab@gmail.com","subject":"[PATCH v5 0/9] commit-graph write: progress output improvements","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-16T13:29:38Z","receivedAt":"2019-01-16T13:30:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This was reviewed & discussed shortly before 2.20 went out the door,\nbut the changes deemed too significant during the RC period, and\ninstead we had my much smaller 01ca387774 (\"commit-graph: split up\nclose_reachable() progress output\", 2018-11-19) as a fix.\n\nThat's now landed in master, and this version is rebased on top of\nthat.\n\nThe changes aren't significant, one patch was dropped because\n01ca387774 did another version of it already, and 09/09 is rewritten\nv.s. what 10/10 was before due to being on top of 01ca387774.\n\nSZEDER Gábor (2):\n commit-graph: rename 'num_extra_edges' variable to 'num_large_edges'\n commit-graph: don't call write_graph_chunk_large_edges() unnecessarily\n\nÆvar Arnfjörð Bjarmason (7):\n commit-graph write: add \"Writing out\" progress output\n commit-graph write: more descriptive \"writing out\" output\n commit-graph write: show progress for object search\n commit-graph write: add more descriptive progress output\n commit-graph write: remove empty line for readability\n commit-graph write: add itermediate progress\n commit-graph write: emit a percentage for all progress\n\n commit-graph.c | 134 ++++++++++++++++++++++++++++++++++++-------------\n 1 file changed, 100 insertions(+), 34 deletions(-)\n\nRange-diff:\n 1: 07d06c50c0 = 1: 8d56b1c5c1 commit-graph: rename 'num_extra_edges' variable to 'num_large_edges'\n 2: 904dda1e7a = 2: 8e33e12c7d commit-graph: don't call write_graph_chunk_large_edges() unnecessarily\n 3: 1126c7e29d < -: ---------- commit-graph write: rephrase confusing progress output\n 4: 9c17f56ed3 ! 3: fa48b4b4cd commit-graph write: add \"Writing out\" progress output\n @@ -10,14 +10,14 @@\n small repositories, but before this change we'd noticeably hang for\n 2-3 seconds at the end on medium sized repositories such as linux.git.\n \n - Now we'll instead show output like this, and have no human-observable\n - point at which we're not producing progress output:\n + Now we'll instead show output like this, and reduce the\n + human-observable times at which we're not producing progress output:\n \n - $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n - Finding commits for commit graph: 6365442, done.\n - Annotating commit graph: 2391666, done.\n - Computing commit graph generation numbers: 100% (797222/797222), done.\n - Writing out commit graph: 100% (3188888/3188888), done.\n + $ ~/g/git/git --exec-path=$HOME/g/git -C ~/g/2015-04-03-1M-git commit-graph write\n + Finding commits for commit graph: 13064614, done.\n + Expanding reachable commits in commit graph: 1000447, done.\n + Computing commit graph generation numbers: 100% (1000447/1000447), done.\n + Writing out commit graph: 100% (3001341/3001341), done.\n \n This \"Writing out\" number is 3x or 4x the number of commits, depending\n on the graph we're processing. A later change will make this explicit\n 5: 79b0a467d9 ! 4: 738074e802 commit-graph write: more descriptive \"writing out\" output\n @@ -11,11 +11,11 @@\n has to do with writing out the commit graph. Now e.g. on linux.git we\n emit:\n \n - $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n - Finding commits for commit graph: 6365442, done.\n - Annotating commit graph: 2391666, done.\n - Computing commit graph generation numbers: 100% (797222/797222), done.\n - Writing out commit graph in 4 passes: 100% (3188888/3188888), done.\n + $ ~/g/git/git --exec-path=$HOME/g/git -C ~/g/linux commit-graph write\n + Finding commits for commit graph: 6529159, done.\n + Expanding reachable commits in commit graph: 815990, done.\n + Computing commit graph generation numbers: 100% (815983/815983), done.\n + Writing out commit graph in 4 passes: 100% (3263932/3263932), done.\n \n A note on i18n: Why are we using the Q_() function and passing a\n number & English text for a singular which'll never be used? Because\n @@ -51,11 +51,10 @@\n \t}\n \twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress, &progress_cnt);\n @@\n + \tif (num_large_edges)\n \t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr, progress, &progress_cnt);\n \tstop_progress(&progress);\n - \n +\tstrbuf_release(&progress_title);\n -+\n + \n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n - \tcommit_lock_file(&lk);\n 6: b32be83b38 ! 5: b8b6f3ca84 commit-graph write: show progress for object search\n @@ -8,12 +8,12 @@\n \n Before we'd emit on e.g. linux.git with \"commit-graph write\":\n \n - Finding commits for commit graph: 6365442, done.\n + Finding commits for commit graph: 6529159, done.\n [...]\n \n And now:\n \n - Finding commits for commit graph: 100% (6365442/6365442), done.\n + Finding commits for commit graph: 100% (6529159/6529159), done.\n [...]\n \n Since the commit graph only includes those commits that are packed\n 7: 54276723c0 ! 6: 1fab1b3c8f commit-graph write: add more descriptive progress output\n @@ -10,17 +10,17 @@\n we support:\n \n $ git commit-graph write\n - Finding commits for commit graph among packed objects: 100% (6365442/6365442), done.\n + Finding commits for commit graph among packed objects: 100% (6529159/6529159), done.\n [...]\n \n # Actually we don't emit this since this takes almost no time at\n # all. But if we did (s/_delayed//) we'd show:\n $ git for-each-ref --format='%(objectname)' | git commit-graph write --stdin-commits\n - Finding commits for commit graph from 584 refs: 100% (584/584), done.\n + Finding commits for commit graph from 630 refs: 100% (630/630), done.\n [...]\n \n $ (cd .git/objects/pack/ && ls *idx) | git commit-graph write --stdin-pack\n - Finding commits for commit graph in 2 packs: 6365442, done.\n + Finding commits for commit graph in 3 packs: 6529159, done.\n [...]\n \n The middle on of those is going to be the output users might see in\n 8: 0e847366e1 = 7: e952c4b728 commit-graph write: remove empty line for readability\n 9: c388aff73e ! 8: 99ac6725cb commit-graph write: add itermediate progress\n @@ -10,26 +10,29 @@\n million objects we'll now emit:\n \n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n - Finding commits for commit graph among packed objects: 100% (48333911/48333911), done.\n - Annotating commit graph: 21435984, done.\n - Counting distinct commits in commit graph: 100% (7145328/7145328), done.\n - Finding extra edges in commit graph: 100% (7145328/7145328), done.\n - Computing commit graph generation numbers: 100% (7145328/7145328), done.\n - Writing out commit graph in 4 passes: 100% (28581312/28581312), done.\n + Finding commits for commit graph among packed objects: 100% (124763727/124763727), done.\n + Loading known commits in commit graph: 100% (18989461/18989461), done.\n + Expanding reachable commits in commit graph: 100% (18989507/18989461), done.\n + Clearing commit marks in commit graph: 100% (18989507/18989507), done.\n + Counting distinct commits in commit graph: 100% (18989507/18989507), done.\n + Finding extra edges in commit graph: 100% (18989507/18989507), done.\n + Computing commit graph generation numbers: 100% (7250302/7250302), done.\n + Writing out commit graph in 4 passes: 100% (29001208/29001208), done.\n \n Whereas on a medium-sized repository such as linux.git these new\n progress bars won't have time to kick in and as before and we'll still\n emit output like:\n \n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n - Finding commits for commit graph among packed objects: 100% (6365442/6365442), done.\n - Annotating commit graph: 2391666, done.\n - Computing commit graph generation numbers: 100% (797222/797222), done.\n - Writing out commit graph in 4 passes: 100% (3188888/3188888), done.\n + Finding commits for commit graph among packed objects: 100% (6529159/6529159), done.\n + Expanding reachable commits in commit graph: 815990, done.\n + Computing commit graph generation numbers: 100% (815983/815983), done.\n + Writing out commit graph in 4 passes: 100% (3263932/3263932), done.\n \n The \"Counting distinct commits in commit graph\" phase will spend most\n of its time paused at \"0/*\" as we QSORT(...) the list. That's not\n - optimal, but at least we don't seem to be stalling anymore.\n + optimal, but at least we don't seem to be stalling anymore most of the\n + time.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n10: fd692499e0 < -: ---------- commit-graph write: emit a percentage for all progress\n -: ---------- > 9: d69dbf81de commit-graph write: emit a percentage for all progress\n-- \n2.20.1.153.gd81d796ee0\n\n"},{"id":"366817","messageId":"20190116132947.26025-2-avarab@gmail.com","threadId":"49275","inReplyTo":"20181122153922.16912-2-avarab@gmail.com","subject":"[PATCH v5 1/9] commit-graph: rename 'num_extra_edges' variable to 'num_large_edges'","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-16T13:29:39Z","receivedAt":"2019-01-16T13:30:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: SZEDER Gábor \n\nThe commit graph file format describes an optional 'Large Edge List'\nchunk, and the function writing out this chunk is called\nwrite_graph_chunk_large_edges(). Then there are two functions in\n'commit-graph.c', namely write_graph_chunk_data() and\nwrite_commit_graph(), which have a local variable called\n'num_extra_edges'.\n\nIt can be confusing on first sight whether large edges and extra edges\nrefer to the same thing or not, but they do, so let's rename those\nvariables to 'num_large_edges'.\n\nSigned-off-by: SZEDER Gábor \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 18 +++++++++---------\n 1 file changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 5c8fb4b134..ab292e5d01 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -472,7 +472,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n {\n \tstruct commit **list = commits;\n \tstruct commit **last = commits + nr_commits;\n-\tuint32_t num_extra_edges = 0;\n+\tuint32_t num_large_edges = 0;\n \n \twhile (list < last) {\n \t\tstruct commit_list *parent;\n@@ -504,7 +504,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\tif (!parent)\n \t\t\tedge_value = GRAPH_PARENT_NONE;\n \t\telse if (parent->next)\n-\t\t\tedge_value = GRAPH_OCTOPUS_EDGES_NEEDED | num_extra_edges;\n+\t\t\tedge_value = GRAPH_OCTOPUS_EDGES_NEEDED | num_large_edges;\n \t\telse {\n \t\t\tedge_value = sha1_pos(parent->item->object.oid.hash,\n \t\t\t\t\t commits,\n@@ -518,7 +518,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \n \t\tif (edge_value & GRAPH_OCTOPUS_EDGES_NEEDED) {\n \t\t\tdo {\n-\t\t\t\tnum_extra_edges++;\n+\t\t\t\tnum_large_edges++;\n \t\t\t\tparent = parent->next;\n \t\t\t} while (parent);\n \t\t}\n@@ -765,7 +765,7 @@ void write_commit_graph(const char *obj_dir,\n \tuint32_t chunk_ids[5];\n \tuint64_t chunk_offsets[5];\n \tint num_chunks;\n-\tint num_extra_edges;\n+\tint num_large_edges;\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n \n@@ -875,7 +875,7 @@ void write_commit_graph(const char *obj_dir,\n \tcommits.alloc = count_distinct;\n \tALLOC_ARRAY(commits.list, commits.alloc);\n \n-\tnum_extra_edges = 0;\n+\tnum_large_edges = 0;\n \tfor (i = 0; i < oids.nr; i++) {\n \t\tint num_parents = 0;\n \t\tif (i > 0 && oideq(&oids.list[i - 1], &oids.list[i]))\n@@ -889,11 +889,11 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tnum_parents++;\n \n \t\tif (num_parents > 2)\n-\t\t\tnum_extra_edges += num_parents - 1;\n+\t\t\tnum_large_edges += num_parents - 1;\n \n \t\tcommits.nr++;\n \t}\n-\tnum_chunks = num_extra_edges ? 4 : 3;\n+\tnum_chunks = num_large_edges ? 4 : 3;\n \n \tif (commits.nr >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"too many commits to write graph\"));\n@@ -920,7 +920,7 @@ void write_commit_graph(const char *obj_dir,\n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n \tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n-\tif (num_extra_edges)\n+\tif (num_large_edges)\n \t\tchunk_ids[3] = GRAPH_CHUNKID_LARGEEDGES;\n \telse\n \t\tchunk_ids[3] = 0;\n@@ -930,7 +930,7 @@ void write_commit_graph(const char *obj_dir,\n \tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n \tchunk_offsets[2] = chunk_offsets[1] + GRAPH_OID_LEN * commits.nr;\n \tchunk_offsets[3] = chunk_offsets[2] + (GRAPH_OID_LEN + 16) * commits.nr;\n-\tchunk_offsets[4] = chunk_offsets[3] + 4 * num_extra_edges;\n+\tchunk_offsets[4] = chunk_offsets[3] + 4 * num_large_edges;\n \n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tuint32_t chunk_write[3];\n-- \n2.20.1.153.gd81d796ee0\n\n"},{"id":"366818","messageId":"20190116132947.26025-3-avarab@gmail.com","threadId":"49275","inReplyTo":"20181122153922.16912-2-avarab@gmail.com","subject":"[PATCH v5 2/9] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-16T13:29:40Z","receivedAt":"2019-01-16T13:30:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: SZEDER Gábor \n\nThe optional 'Large Edge List' chunk of the commit graph file stores\nparent information for commits with more than two parents. Since the\nchunk is optional, write_commit_graph() looks through all commits to\nfind those with more than two parents, and then writes the commit\ngraph file header accordingly, i.e. if there are no such commits, then\nthere won't be a 'Large Edge List' chunk written, only the three\nmandatory chunks.\n\nHowever, when it comes to writing chunk data, write_commit_graph()\nunconditionally invokes write_graph_chunk_large_edges(), even when it\nwas decided earlier that that chunk won't be written. Strictly\nspeaking there is no bug here, because write_graph_chunk_large_edges()\nwon't write anything because it won't find any commits with more than\ntwo parents, but then it unnecessarily and in vain looks through all\ncommits once again in search for such commits.\n\nDon't call write_graph_chunk_large_edges() when that chunk won't be\nwritten to spare an unnecessary iteration over all commits.\n\nSigned-off-by: SZEDER Gábor \nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex ab292e5d01..38b62051c2 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -944,7 +944,8 @@ void write_commit_graph(const char *obj_dir,\n \twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n \twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n \twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n-\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n+\tif (num_large_edges)\n+\t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n \n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n-- \n2.20.1.153.gd81d796ee0\n\n"},{"id":"366819","messageId":"20190116132947.26025-4-avarab@gmail.com","threadId":"49275","inReplyTo":"20181122153922.16912-2-avarab@gmail.com","subject":"[PATCH v5 3/9] commit-graph write: add \"Writing out\" progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-16T13:29:41Z","receivedAt":"2019-01-16T13:30:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add progress output to be shown when we're writing out the\ncommit-graph, this adds to the output already added in 7b0f229222\n(\"commit-graph write: add progress output\", 2018-09-17).\n\nAs noted in that commit most of the progress output isn't displayed on\nsmall repositories, but before this change we'd noticeably hang for\n2-3 seconds at the end on medium sized repositories such as linux.git.\n\nNow we'll instead show output like this, and reduce the\nhuman-observable times at which we're not producing progress output:\n\n $ ~/g/git/git --exec-path=$HOME/g/git -C ~/g/2015-04-03-1M-git commit-graph write\n Finding commits for commit graph: 13064614, done.\n Expanding reachable commits in commit graph: 1000447, done.\n Computing commit graph generation numbers: 100% (1000447/1000447), done.\n Writing out commit graph: 100% (3001341/3001341), done.\n\nThis \"Writing out\" number is 3x or 4x the number of commits, depending\non the graph we're processing. A later change will make this explicit\nto the user.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 48 +++++++++++++++++++++++++++++++++++++++---------\n 1 file changed, 39 insertions(+), 9 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 38b62051c2..0c3af28399 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -430,7 +430,9 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n \t\t\t\t struct commit **commits,\n-\t\t\t\t int nr_commits)\n+\t\t\t\t int nr_commits,\n+\t\t\t\t struct progress *progress,\n+\t\t\t\t uint64_t *progress_cnt)\n {\n \tint i, count = 0;\n \tstruct commit **list = commits;\n@@ -444,6 +446,7 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n \t\twhile (count < nr_commits) {\n \t\t\tif ((*list)->object.oid.hash[0] != i)\n \t\t\t\tbreak;\n+\t\t\tdisplay_progress(progress, ++*progress_cnt);\n \t\t\tcount++;\n \t\t\tlist++;\n \t\t}\n@@ -453,12 +456,16 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n }\n \n static void write_graph_chunk_oids(struct hashfile *f, int hash_len,\n-\t\t\t\t struct commit **commits, int nr_commits)\n+\t\t\t\t struct commit **commits, int nr_commits,\n+\t\t\t\t struct progress *progress,\n+\t\t\t\t uint64_t *progress_cnt)\n {\n \tstruct commit **list = commits;\n \tint count;\n-\tfor (count = 0; count < nr_commits; count++, list++)\n+\tfor (count = 0; count < nr_commits; count++, list++) {\n+\t\tdisplay_progress(progress, ++*progress_cnt);\n \t\thashwrite(f, (*list)->object.oid.hash, (int)hash_len);\n+\t}\n }\n \n static const unsigned char *commit_to_sha1(size_t index, void *table)\n@@ -468,7 +475,9 @@ static const unsigned char *commit_to_sha1(size_t index, void *table)\n }\n \n static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n-\t\t\t\t struct commit **commits, int nr_commits)\n+\t\t\t\t struct commit **commits, int nr_commits,\n+\t\t\t\t struct progress *progress,\n+\t\t\t\t uint64_t *progress_cnt)\n {\n \tstruct commit **list = commits;\n \tstruct commit **last = commits + nr_commits;\n@@ -478,6 +487,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\tstruct commit_list *parent;\n \t\tint edge_value;\n \t\tuint32_t packedDate[2];\n+\t\tdisplay_progress(progress, ++*progress_cnt);\n \n \t\tparse_commit(*list);\n \t\thashwrite(f, get_commit_tree_oid(*list)->hash, hash_len);\n@@ -539,7 +549,9 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \n static void write_graph_chunk_large_edges(struct hashfile *f,\n \t\t\t\t\t struct commit **commits,\n-\t\t\t\t\t int nr_commits)\n+\t\t\t\t\t int nr_commits,\n+\t\t\t\t\t struct progress *progress,\n+\t\t\t\t\t uint64_t *progress_cnt)\n {\n \tstruct commit **list = commits;\n \tstruct commit **last = commits + nr_commits;\n@@ -547,6 +559,9 @@ static void write_graph_chunk_large_edges(struct hashfile *f,\n \n \twhile (list < last) {\n \t\tint num_parents = 0;\n+\n+\t\tdisplay_progress(progress, ++*progress_cnt);\n+\n \t\tfor (parent = (*list)->parents; num_parents < 3 && parent;\n \t\t parent = parent->next)\n \t\t\tnum_parents++;\n@@ -768,6 +783,7 @@ void write_commit_graph(const char *obj_dir,\n \tint num_large_edges;\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n+\tuint64_t progress_cnt = 0;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n@@ -941,11 +957,25 @@ void write_commit_graph(const char *obj_dir,\n \t\thashwrite(f, chunk_write, 12);\n \t}\n \n-\twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n-\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n-\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n+\tif (report_progress) {\n+\t\t/*\n+\t\t * Each of the write_graph_chunk_*() functions just\n+\t\t * below loops over our N commits. This number must be\n+\t\t * kept in sync with the number of passes we're doing.\n+\t\t */\n+\t\tint graph_passes = 3;\n+\t\tif (num_large_edges)\n+\t\t\tgraph_passes++;\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Writing out commit graph\"),\n+\t\t\tgraph_passes * commits.nr);\n+\t}\n+\twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress, &progress_cnt);\n+\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr, progress, &progress_cnt);\n+\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr, progress, &progress_cnt);\n \tif (num_large_edges)\n-\t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n+\t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr, progress, &progress_cnt);\n+\tstop_progress(&progress);\n \n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n-- \n2.20.1.153.gd81d796ee0\n\n"},{"id":"366820","messageId":"20190116132947.26025-5-avarab@gmail.com","threadId":"49275","inReplyTo":"20181122153922.16912-2-avarab@gmail.com","subject":"[PATCH v5 4/9] commit-graph write: more descriptive \"writing out\" output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-16T13:29:42Z","receivedAt":"2019-01-16T13:30:16Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the \"Writing out\" part of the progress output more\ndescriptive. Depending on the shape of the graph we either make 3 or 4\npasses over it.\n\nLet's present this information to the user in case they're wondering\nwhat this number, which is much larger than their number of commits,\nhas to do with writing out the commit graph. Now e.g. on linux.git we\nemit:\n\n $ ~/g/git/git --exec-path=$HOME/g/git -C ~/g/linux commit-graph write\n Finding commits for commit graph: 6529159, done.\n Expanding reachable commits in commit graph: 815990, done.\n Computing commit graph generation numbers: 100% (815983/815983), done.\n Writing out commit graph in 4 passes: 100% (3263932/3263932), done.\n\nA note on i18n: Why are we using the Q_() function and passing a\nnumber & English text for a singular which'll never be used? Because\nthe plural rules of translated languages may not match those of\nEnglish, and to use the plural function we need to use this format.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 9 ++++++++-\n 1 file changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 0c3af28399..7653c4e9d6 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -784,6 +784,7 @@ void write_commit_graph(const char *obj_dir,\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n \tuint64_t progress_cnt = 0;\n+\tstruct strbuf progress_title = STRBUF_INIT;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n@@ -966,8 +967,13 @@ void write_commit_graph(const char *obj_dir,\n \t\tint graph_passes = 3;\n \t\tif (num_large_edges)\n \t\t\tgraph_passes++;\n+\t\tstrbuf_addf(&progress_title,\n+\t\t\t Q_(\"Writing out commit graph in %d pass\",\n+\t\t\t \"Writing out commit graph in %d passes\",\n+\t\t\t graph_passes),\n+\t\t\t graph_passes);\n \t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Writing out commit graph\"),\n+\t\t\tprogress_title.buf,\n \t\t\tgraph_passes * commits.nr);\n \t}\n \twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress, &progress_cnt);\n@@ -976,6 +982,7 @@ void write_commit_graph(const char *obj_dir,\n \tif (num_large_edges)\n \t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr, progress, &progress_cnt);\n \tstop_progress(&progress);\n+\tstrbuf_release(&progress_title);\n \n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n-- \n2.20.1.153.gd81d796ee0\n\n"},{"id":"366821","messageId":"20190116132947.26025-6-avarab@gmail.com","threadId":"49275","inReplyTo":"20181122153922.16912-2-avarab@gmail.com","subject":"[PATCH v5 5/9] commit-graph write: show progress for object search","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-16T13:29:43Z","receivedAt":"2019-01-16T13:30:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Show the percentage progress for the \"Finding commits for commit\ngraph\" phase for the common case where we're operating on all packs in\nthe repository, as \"commit-graph write\" or \"gc\" will do.\n\nBefore we'd emit on e.g. linux.git with \"commit-graph write\":\n\n Finding commits for commit graph: 6529159, done.\n [...]\n\nAnd now:\n\n Finding commits for commit graph: 100% (6529159/6529159), done.\n [...]\n\nSince the commit graph only includes those commits that are packed\n(via for_each_packed_object(...)) the approximate_object_count()\nreturns the actual number of objects we're going to process.\n\nStill, it is possible due to a race with \"gc\" or another process\nmaintaining packs that the number of objects we're going to process is\nlower than what approximate_object_count() reported. In that case we\ndon't want to stop the progress bar short of 100%. So let's make sure\nit snaps to 100% at the end.\n\nThe inverse case is also possible and more likely. I.e. that a new\npack has been added between approximate_object_count() and\nfor_each_packed_object(). In that case the percentage will go beyond\n100%, and we'll do nothing to snap it back to 100% at the end.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 9 +++++++--\n 1 file changed, 7 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 7653c4e9d6..f0bcb3c690 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -785,12 +785,14 @@ void write_commit_graph(const char *obj_dir,\n \tstruct progress *progress = NULL;\n \tuint64_t progress_cnt = 0;\n \tstruct strbuf progress_title = STRBUF_INIT;\n+\tunsigned long approx_nr_objects;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n \n \toids.nr = 0;\n-\toids.alloc = approximate_object_count() / 32;\n+\tapprox_nr_objects = approximate_object_count();\n+\toids.alloc = approx_nr_objects / 32;\n \toids.progress = NULL;\n \toids.progress_done = 0;\n \n@@ -870,8 +872,11 @@ void write_commit_graph(const char *obj_dir,\n \tif (!pack_indexes && !commit_hex) {\n \t\tif (report_progress)\n \t\t\toids.progress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"), 0);\n+\t\t\t\t_(\"Finding commits for commit graph\"),\n+\t\t\t\tapprox_nr_objects);\n \t\tfor_each_packed_object(add_packed_commits, &oids, 0);\n+\t\tif (oids.progress_done < approx_nr_objects)\n+\t\t\tdisplay_progress(oids.progress, approx_nr_objects);\n \t\tstop_progress(&oids.progress);\n \t}\n \n-- \n2.20.1.153.gd81d796ee0\n\n"},{"id":"366822","messageId":"20190116132947.26025-7-avarab@gmail.com","threadId":"49275","inReplyTo":"20181122153922.16912-2-avarab@gmail.com","subject":"[PATCH v5 6/9] commit-graph write: add more descriptive progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-16T13:29:44Z","receivedAt":"2019-01-16T13:30:22Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the progress output shown when we're searching for commits to\ninclude in the graph more descriptive. This amends code I added in\n7b0f229222 (\"commit-graph write: add progress output\", 2018-09-17).\n\nNow, on linux.git, we'll emit this sort of output in the various modes\nwe support:\n\n $ git commit-graph write\n Finding commits for commit graph among packed objects: 100% (6529159/6529159), done.\n [...]\n\n # Actually we don't emit this since this takes almost no time at\n # all. But if we did (s/_delayed//) we'd show:\n $ git for-each-ref --format='%(objectname)' | git commit-graph write --stdin-commits\n Finding commits for commit graph from 630 refs: 100% (630/630), done.\n [...]\n\n $ (cd .git/objects/pack/ && ls *idx) | git commit-graph write --stdin-pack\n Finding commits for commit graph in 3 packs: 6529159, done.\n [...]\n\nThe middle on of those is going to be the output users might see in\npractice, since it'll be emitted when they get the commit graph via\ngc.writeCommitGraph=true. But as noted above you need a really large\nnumber of refs for this message to show. It'll show up on a test\nrepository I have with ~165k refs:\n\n Finding commits for commit graph from 165203 refs: 100% (165203/165203), done.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 25 ++++++++++++++++++-------\n 1 file changed, 18 insertions(+), 7 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f0bcb3c690..c6f8e73980 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -822,8 +822,12 @@ void write_commit_graph(const char *obj_dir,\n \t\tstrbuf_addf(&packname, \"%s/pack/\", obj_dir);\n \t\tdirlen = packname.len;\n \t\tif (report_progress) {\n-\t\t\toids.progress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"), 0);\n+\t\t\tstrbuf_addf(&progress_title,\n+\t\t\t\t Q_(\"Finding commits for commit graph in %d pack\",\n+\t\t\t\t \"Finding commits for commit graph in %d packs\",\n+\t\t\t\t pack_indexes->nr),\n+\t\t\t\t pack_indexes->nr);\n+\t\t\toids.progress = start_delayed_progress(progress_title.buf, 0);\n \t\t\toids.progress_done = 0;\n \t\t}\n \t\tfor (i = 0; i < pack_indexes->nr; i++) {\n@@ -840,14 +844,20 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tfree(p);\n \t\t}\n \t\tstop_progress(&oids.progress);\n+\t\tstrbuf_reset(&progress_title);\n \t\tstrbuf_release(&packname);\n \t}\n \n \tif (commit_hex) {\n-\t\tif (report_progress)\n-\t\t\tprogress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"),\n-\t\t\t\tcommit_hex->nr);\n+\t\tif (report_progress) {\n+\t\t\tstrbuf_addf(&progress_title,\n+\t\t\t\t Q_(\"Finding commits for commit graph from %d ref\",\n+\t\t\t\t \"Finding commits for commit graph from %d refs\",\n+\t\t\t\t commit_hex->nr),\n+\t\t\t\t commit_hex->nr);\n+\t\t\tprogress = start_delayed_progress(progress_title.buf,\n+\t\t\t\t\t\t\t commit_hex->nr);\n+\t\t}\n \t\tfor (i = 0; i < commit_hex->nr; i++) {\n \t\t\tconst char *end;\n \t\t\tstruct object_id oid;\n@@ -867,12 +877,13 @@ void write_commit_graph(const char *obj_dir,\n \t\t\t}\n \t\t}\n \t\tstop_progress(&progress);\n+\t\tstrbuf_reset(&progress_title);\n \t}\n \n \tif (!pack_indexes && !commit_hex) {\n \t\tif (report_progress)\n \t\t\toids.progress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"),\n+\t\t\t\t_(\"Finding commits for commit graph among packed objects\"),\n \t\t\t\tapprox_nr_objects);\n \t\tfor_each_packed_object(add_packed_commits, &oids, 0);\n \t\tif (oids.progress_done < approx_nr_objects)\n-- \n2.20.1.153.gd81d796ee0\n\n"},{"id":"366823","messageId":"20190116132947.26025-8-avarab@gmail.com","threadId":"49275","inReplyTo":"20181122153922.16912-2-avarab@gmail.com","subject":"[PATCH v5 7/9] commit-graph write: remove empty line for readability","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-16T13:29:45Z","receivedAt":"2019-01-16T13:30:25Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Remove the empty line between a QSORT(...) and the subsequent oideq()\nfor-loop. This makes it clearer that the QSORT(...) is being done so\nthat we can run the oideq() loop on adjacent OIDs. Amends code added\nin 08fd81c9b6 (\"commit-graph: implement write_commit_graph()\",\n2018-04-02).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex c6f8e73980..7138bbd02f 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -894,7 +894,6 @@ void write_commit_graph(const char *obj_dir,\n \tclose_reachable(&oids, report_progress);\n \n \tQSORT(oids.list, oids.nr, commit_compare);\n-\n \tcount_distinct = 1;\n \tfor (i = 1; i < oids.nr; i++) {\n \t\tif (!oideq(&oids.list[i - 1], &oids.list[i]))\n-- \n2.20.1.153.gd81d796ee0\n\n"},{"id":"366824","messageId":"20190116132947.26025-9-avarab@gmail.com","threadId":"49275","inReplyTo":"20181122153922.16912-2-avarab@gmail.com","subject":"[PATCH v5 8/9] commit-graph write: add itermediate progress","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-16T13:29:46Z","receivedAt":"2019-01-16T13:30:28Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add progress output to sections of code between \"Annotating[...]\" and\n\"Computing[...]generation numbers\". This can collectively take 5-10\nseconds on a large enough repository.\n\nOn a test repository with I have with ~7 million commits and ~50\nmillion objects we'll now emit:\n\n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph among packed objects: 100% (124763727/124763727), done.\n Loading known commits in commit graph: 100% (18989461/18989461), done.\n Expanding reachable commits in commit graph: 100% (18989507/18989461), done.\n Clearing commit marks in commit graph: 100% (18989507/18989507), done.\n Counting distinct commits in commit graph: 100% (18989507/18989507), done.\n Finding extra edges in commit graph: 100% (18989507/18989507), done.\n Computing commit graph generation numbers: 100% (7250302/7250302), done.\n Writing out commit graph in 4 passes: 100% (29001208/29001208), done.\n\nWhereas on a medium-sized repository such as linux.git these new\nprogress bars won't have time to kick in and as before and we'll still\nemit output like:\n\n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph among packed objects: 100% (6529159/6529159), done.\n Expanding reachable commits in commit graph: 815990, done.\n Computing commit graph generation numbers: 100% (815983/815983), done.\n Writing out commit graph in 4 passes: 100% (3263932/3263932), done.\n\nThe \"Counting distinct commits in commit graph\" phase will spend most\nof its time paused at \"0/*\" as we QSORT(...) the list. That's not\noptimal, but at least we don't seem to be stalling anymore most of the\ntime.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 13 +++++++++++++\n 1 file changed, 13 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 7138bbd02f..60ec7c1b22 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -893,12 +893,19 @@ void write_commit_graph(const char *obj_dir,\n \n \tclose_reachable(&oids, report_progress);\n \n+\tif (report_progress)\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Counting distinct commits in commit graph\"),\n+\t\t\toids.nr);\n+\tdisplay_progress(progress, 0); /* TODO: Measure QSORT() progress */\n \tQSORT(oids.list, oids.nr, commit_compare);\n \tcount_distinct = 1;\n \tfor (i = 1; i < oids.nr; i++) {\n+\t\tdisplay_progress(progress, i + 1);\n \t\tif (!oideq(&oids.list[i - 1], &oids.list[i]))\n \t\t\tcount_distinct++;\n \t}\n+\tstop_progress(&progress);\n \n \tif (count_distinct >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n@@ -908,8 +915,13 @@ void write_commit_graph(const char *obj_dir,\n \tALLOC_ARRAY(commits.list, commits.alloc);\n \n \tnum_large_edges = 0;\n+\tif (report_progress)\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Finding extra edges in commit graph\"),\n+\t\t\toids.nr);\n \tfor (i = 0; i < oids.nr; i++) {\n \t\tint num_parents = 0;\n+\t\tdisplay_progress(progress, i + 1);\n \t\tif (i > 0 && oideq(&oids.list[i - 1], &oids.list[i]))\n \t\t\tcontinue;\n \n@@ -926,6 +938,7 @@ void write_commit_graph(const char *obj_dir,\n \t\tcommits.nr++;\n \t}\n \tnum_chunks = num_large_edges ? 4 : 3;\n+\tstop_progress(&progress);\n \n \tif (commits.nr >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"too many commits to write graph\"));\n-- \n2.20.1.153.gd81d796ee0\n\n"},{"id":"366825","messageId":"20190116132947.26025-10-avarab@gmail.com","threadId":"49275","inReplyTo":"20181122153922.16912-2-avarab@gmail.com","subject":"[PATCH v5 9/9] commit-graph write: emit a percentage for all progress","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-16T13:29:47Z","receivedAt":"2019-01-16T13:30:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Follow-up 01ca387774 (\"commit-graph: split up close_reachable()\nprogress output\", 2018-11-19) by making the progress bars in\nclose_reachable() report a completion percentage. This fixes the last\noccurrence where in the commit graph writing where we didn't report\nthat.\n\nThe change in 01ca387774 split up the 1x progress bar in\nclose_reachable() into 3x, but left them as dumb counters without a\npercentage completion. Fixing that is easy, and the only reason it\nwasn't done already is because that commit was rushed in during the\nv2.20.0 RC period to fix the unrelated issue of over-reporting commit\nnumbers. See [1] and follow-ups for ML activity at the time and [2]\nfor an alternative approach where the progress bars weren't split up.\n\nNow for e.g. linux.git we'll emit:\n\n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph among packed objects: 100% (6529159/6529159), done.\n Expanding reachable commits in commit graph: 100% (815990/815980), done.\n Computing commit graph generation numbers: 100% (815983/815983), done.\n Writing out commit graph in 4 passes: 100% (3263932/3263932), done.\n\n1. https://public-inbox.org/git/20181119202300.18670-1-avarab@gmail.com/\n2. https://public-inbox.org/git/20181122153922.16912-11-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 60ec7c1b22..b6a074c80d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -653,15 +653,15 @@ static void add_missing_parents(struct packed_oid_list *oids, struct commit *com\n \n static void close_reachable(struct packed_oid_list *oids, int report_progress)\n {\n-\tint i, j;\n+\tint i;\n \tstruct commit *commit;\n \tstruct progress *progress = NULL;\n \n \tif (report_progress)\n \t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Loading known commits in commit graph\"), j = 0);\n+\t\t\t_(\"Loading known commits in commit graph\"), oids->nr);\n \tfor (i = 0; i < oids->nr; i++) {\n-\t\tdisplay_progress(progress, ++j);\n+\t\tdisplay_progress(progress, i + 1);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n \t\tif (commit)\n \t\t\tcommit->object.flags |= UNINTERESTING;\n@@ -675,9 +675,9 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n \t */\n \tif (report_progress)\n \t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Expanding reachable commits in commit graph\"), j = 0);\n+\t\t\t_(\"Expanding reachable commits in commit graph\"), oids->nr);\n \tfor (i = 0; i < oids->nr; i++) {\n-\t\tdisplay_progress(progress, ++j);\n+\t\tdisplay_progress(progress, i + 1);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n \n \t\tif (commit && !parse_commit(commit))\n@@ -687,9 +687,9 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n \n \tif (report_progress)\n \t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Clearing commit marks in commit graph\"), j = 0);\n+\t\t\t_(\"Clearing commit marks in commit graph\"), oids->nr);\n \tfor (i = 0; i < oids->nr; i++) {\n-\t\tdisplay_progress(progress, ++j);\n+\t\tdisplay_progress(progress, i + 1);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n \n \t\tif (commit)\n-- \n2.20.1.153.gd81d796ee0\n\n"},{"id":"366988","messageId":"20190117132345.29791-1-avarab@gmail.com","threadId":"49275","inReplyTo":"20190116132947.26025-10-avarab@gmail.com","subject":"[PATCH] commit-graph write: use pack order when finding commits","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-17T13:23:45Z","receivedAt":"2019-01-17T13:24:09Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Slightly optimize the \"commit-graph write\" step by using\nFOR_EACH_OBJECT_PACK_ORDER with for_each_object_in_pack(). See commit\n[1] and [2] for the facility and a similar optimization for \"cat-file\".\n\nOn Linux it is around 5% slower to run:\n\n echo 1 >/proc/sys/vm/drop_caches &&\n cat .git/objects/pack/* >/dev/null &&\n git cat-file --batch-all-objects --batch-check --unordered\n\nThan the same thing with the \"cat\" omitted. This is as expected, since\nwe're iterating in pack order and the \"cat\" is extra work.\n\nBefore this change the opposite was true of \"commit-graph write\". We\nwere 6% faster if we first ran \"cat\" to efficiently populate the FS\ncache for our sole big pack on linux.git than if we had it populated\nvia for_each_object_in_pack(). Now we're 3% faster without the \"cat\"\ninstead.\n\nThe tests were done on an unloaded Linux 3.10 system with 10 runs for\neach.\n\n1. 736eb88fdc (\"for_each_packed_object: support iterating in\n pack-order\", 2018-08-10)\n\n2. 0750bb5b51 (\"cat-file: support \"unordered\" output for\n --batch-all-objects\", 2018-08-10)\n---\n\nThis is conceptually different from my \"commit-graph write: progress\noutput improvements\" series but conflicts with it, so it's based on\ntop of it. This patch goes after its 9/9.\n\n commit-graph.c | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex b6a074c80d..68124e8385 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -839,7 +839,8 @@ void write_commit_graph(const char *obj_dir,\n \t\t\t\tdie(_(\"error adding pack %s\"), packname.buf);\n \t\t\tif (open_pack_index(p))\n \t\t\t\tdie(_(\"error opening index for %s\"), packname.buf);\n-\t\t\tfor_each_object_in_pack(p, add_packed_commits, &oids, 0);\n+\t\t\tfor_each_object_in_pack(p, add_packed_commits, &oids,\n+\t\t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n \t\t\tclose_pack(p);\n \t\t\tfree(p);\n \t\t}\n@@ -885,7 +886,8 @@ void write_commit_graph(const char *obj_dir,\n \t\t\toids.progress = start_delayed_progress(\n \t\t\t\t_(\"Finding commits for commit graph among packed objects\"),\n \t\t\t\tapprox_nr_objects);\n-\t\tfor_each_packed_object(add_packed_commits, &oids, 0);\n+\t\tfor_each_packed_object(add_packed_commits, &oids,\n+\t\t\t\t FOR_EACH_OBJECT_PACK_ORDER);\n \t\tif (oids.progress_done < approx_nr_objects)\n \t\t\tdisplay_progress(oids.progress, approx_nr_objects);\n \t\tstop_progress(&oids.progress);\n-- \n2.20.1.153.gd81d796ee0\n\n"},{"id":"366995","messageId":"5b1146a1-9ec6-f30e-80e9-cb7cd4dc1a58@gmail.com","threadId":"49275","inReplyTo":"20190117132345.29791-1-avarab@gmail.com","subject":"Re: [PATCH] commit-graph write: use pack order when finding commits","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-01-17T15:09:18Z","receivedAt":"2019-01-17T15:09:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/17/2019 8:23 AM, Ævar Arnfjörð Bjarmason wrote:\n> Slightly optimize the \"commit-graph write\" step by using\n> FOR_EACH_OBJECT_PACK_ORDER with for_each_object_in_pack(). See commit\n> [1] and [2] for the facility and a similar optimization for \"cat-file\".\n>\n> On Linux it is around 5% slower to run:\n>\n> echo 1 >/proc/sys/vm/drop_caches &&\n> cat .git/objects/pack/* >/dev/null &&\n> git cat-file --batch-all-objects --batch-check --unordered\n>\n> Than the same thing with the \"cat\" omitted. This is as expected, since\n> we're iterating in pack order and the \"cat\" is extra work.\n>\n> Before this change the opposite was true of \"commit-graph write\". We\n> were 6% faster if we first ran \"cat\" to efficiently populate the FS\n> cache for our sole big pack on linux.git than if we had it populated\n> via for_each_object_in_pack(). Now we're 3% faster without the \"cat\"\n> instead.\n>\n> The tests were done on an unloaded Linux 3.10 system with 10 runs for\n> each.\n>\n> 1. 736eb88fdc (\"for_each_packed_object: support iterating in\n> pack-order\", 2018-08-10)\n>\n> 2. 0750bb5b51 (\"cat-file: support \"unordered\" output for\n> --batch-all-objects\", 2018-08-10)\nThanks, Aevar! I plan to give this a test on the Windows repository, as \nthis is the way we generate the commit-graph in VFS for Git.\n\nI created a PR on microsoft/git [1] and am generating the installers now [2]\n\n[1] https://github.com/Microsoft/git/pull/110\n\n[2] https://dev.azure.com/gvfs/ci/_build/results?buildId=6114\n\nThe code itself looks good, I just need to double-check the performance \nnumbers.\n\nThanks,\n-Stolee\n"},{"id":"367001","messageId":"f71fa868-25e8-a9c9-46a6-611b987f1a8f@gmail.com","threadId":"49275","inReplyTo":"5b1146a1-9ec6-f30e-80e9-cb7cd4dc1a58@gmail.com","subject":"Re: [PATCH] commit-graph write: use pack order when finding commits","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-01-17T16:35:00Z","receivedAt":"2019-01-17T16:43:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/17/2019 10:09 AM, Derrick Stolee wrote:\n>\n> The code itself looks good, I just need to double-check the \n> performance numbers.\n\nI tested this patch on our VFS-specific version using five \"prefetch \npacks\".\n\nBefore: 22.61 s (avg)\nAfter: 22.18 s (avg)\n\nImprovement: 2%\n\nThe error rate among the samples is very low (~.15 s) so this result is \nsignificant.\n\nConsidering that we are writing 4.4 million commits -- many more than \nthe objects in the packs that we are reading -- this is an excellent \nimprovement!\n\nThanks!\n-Stolee\n"},{"id":"367111","messageId":"20190118170549.30403-1-szeder.dev@gmail.com","threadId":"49275","inReplyTo":"a82a2bec-07f5-ccd2-85ab-b9efd406929e@gmail.com","subject":"[PATCH v2 0/2] commit-graph: minor cleanup and optimization","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-18T17:05:47Z","receivedAt":"2019-01-18T17:06:01Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Nov 21, 2018 at 06:32:19AM -0500, Derrick Stolee wrote:\n> >>I rename these variables to 'num_large_edges', because the commit\n> >>graph file format speaks about the 'Large Edge List' chunk.\n> >>\n> >>However, I do find that the term 'extra' makes much more sense\n> >>\n> >>Would it make sense to do the rename in the other direction?\n> >I tend to agree that \"large edge\" is a misnomer.\n> \n> I agree with you both. \"Extra\" is better.\n\nOK, so here are the two patches with the rename done in the other\ndirection.\n\nThese should replace the bottom two patches of the topic\n'ab/commit-graph-write-progress'.\n\n\nSZEDER Gábor (2):\n commit-graph: rename \"large edges\" to \"extra edges\"\n commit-graph: don't call write_graph_chunk_large_edges() unnecessarily\n\n .../technical/commit-graph-format.txt | 4 +--\n builtin/commit-graph.c | 4 +--\n commit-graph.c | 25 ++++++++++---------\n commit-graph.h | 2 +-\n t/t5318-commit-graph.sh | 14 +++++------\n 5 files changed, 25 insertions(+), 24 deletions(-)\n\n-- \n2.20.1.499.gf60de1223c\n\n"},{"id":"367112","messageId":"20190118170549.30403-2-szeder.dev@gmail.com","threadId":"49275","inReplyTo":"20190118170549.30403-1-szeder.dev@gmail.com","subject":"[PATCH v2 1/2] commit-graph: rename \"large edges\" to \"extra edges\"","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-18T17:05:48Z","receivedAt":"2019-01-18T17:06:06Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"The optional 'Large Edge List' chunk of the commit graph file stores\nparent information for commits with more than two parents, and the\nnames of most of the macros, variables, struct fields, and functions\nrelated to this chunk contain the term \"large edges\", e.g.\nwrite_graph_chunk_large_edges(). However, it's not a really great\nterm, as the edges to the second and subsequent parents stored in this\nchunk are not any larger than the edges to the first and second\nparents stored in the \"main\" 'Commit Data' chunk. It's the number of\nedges, IOW number of parents, that is larger compared to non-merge and\n\"regular\" two-parent merge commits. And indeed, two functions in\n'commit-graph.c' have a local variable called 'num_extra_edges' that\nrefer to the same thing, and this \"extra edges\" term is much better at\ndescribing these edges.\n\nSo let's rename all these references to \"large edges\" in macro,\nvariable, function, etc. names to \"extra edges\". There is a\nGRAPH_OCTOPUS_EDGES_NEEDED macro as well; for the sake of consistency\nrename it to GRAPH_EXTRA_EDGES_NEEDED.\n\nWe can do so safely without causing any incompatibility issues,\nbecause the term \"large edges\" doesn't come up in the file format\nitself in any form (the chunk's magic is {'E', 'D', 'G', 'E'}, there\nis no 'L' in there), but only in the specification text. The string\n\"large edges\", however, does come up in the output of 'git\ncommit-graph read' and in tests looking at its input, but that command\nis explicitly documented as debugging aid, so we can change its output\nand the affected tests safely.\n\nSigned-off-by: SZEDER Gábor \n---\n .../technical/commit-graph-format.txt | 4 ++--\n builtin/commit-graph.c | 4 ++--\n commit-graph.c | 24 +++++++++----------\n commit-graph.h | 2 +-\n t/t5318-commit-graph.sh | 14 +++++------\n 5 files changed, 24 insertions(+), 24 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex cc0474ba3e..16452a0504 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -76,7 +76,7 @@ CHUNK DATA:\n of the ith commit. Stores value 0x7000000 if no parent in that\n position. If there are more than two parents, the second value\n has its most-significant bit on and the other bits store an array\n- position into the Large Edge List chunk.\n+ position into the Extra Edge List chunk.\n * The next 8 bytes store the generation number of the commit and\n the commit time in seconds since EPOCH. The generation number\n uses the higher 30 bits of the first 4 bytes, while the commit\n@@ -84,7 +84,7 @@ CHUNK DATA:\n 2 bits of the lowest byte, storing the 33rd and 34th bit of the\n commit time.\n \n- Large Edge List (ID: {'E', 'D', 'G', 'E'}) [Optional]\n+ Extra Edge List (ID: {'E', 'D', 'G', 'E'}) [Optional]\n This list of 4-byte values store the second through nth parents for\n all octopus merges. The second parent value in the commit data stores\n an array position within this list along with the most-significant bit\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex c02a3f1221..4ae502754c 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -110,8 +110,8 @@ static int graph_read(int argc, const char **argv)\n \t\tprintf(\" oid_lookup\");\n \tif (graph->chunk_commit_data)\n \t\tprintf(\" commit_metadata\");\n-\tif (graph->chunk_large_edges)\n-\t\tprintf(\" large_edges\");\n+\tif (graph->chunk_extra_edges)\n+\t\tprintf(\" extra_edges\");\n \tprintf(\"\\n\");\n \n \tUNLEAK(graph);\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 40c855f185..544d6b8b4a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -21,7 +21,7 @@\n #define GRAPH_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n #define GRAPH_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define GRAPH_CHUNKID_DATA 0x43444154 /* \"CDAT\" */\n-#define GRAPH_CHUNKID_LARGEEDGES 0x45444745 /* \"EDGE\" */\n+#define GRAPH_CHUNKID_EXTRAEDGES 0x45444745 /* \"EDGE\" */\n \n #define GRAPH_DATA_WIDTH 36\n \n@@ -33,7 +33,7 @@\n #define GRAPH_OID_VERSION GRAPH_OID_VERSION_SHA1\n #define GRAPH_OID_LEN GRAPH_OID_LEN_SHA1\n \n-#define GRAPH_OCTOPUS_EDGES_NEEDED 0x80000000\n+#define GRAPH_EXTRA_EDGES_NEEDED 0x80000000\n #define GRAPH_PARENT_MISSING 0x7fffffff\n #define GRAPH_EDGE_LAST_MASK 0x7fffffff\n #define GRAPH_PARENT_NONE 0x70000000\n@@ -177,11 +177,11 @@ struct commit_graph *load_commit_graph_one(const char *graph_file)\n \t\t\t\tgraph->chunk_commit_data = data + chunk_offset;\n \t\t\tbreak;\n \n-\t\tcase GRAPH_CHUNKID_LARGEEDGES:\n-\t\t\tif (graph->chunk_large_edges)\n+\t\tcase GRAPH_CHUNKID_EXTRAEDGES:\n+\t\t\tif (graph->chunk_extra_edges)\n \t\t\t\tchunk_repeated = 1;\n \t\t\telse\n-\t\t\t\tgraph->chunk_large_edges = data + chunk_offset;\n+\t\t\t\tgraph->chunk_extra_edges = data + chunk_offset;\n \t\t\tbreak;\n \t\t}\n \n@@ -346,12 +346,12 @@ static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \tedge_value = get_be32(commit_data + g->hash_len + 4);\n \tif (edge_value == GRAPH_PARENT_NONE)\n \t\treturn 1;\n-\tif (!(edge_value & GRAPH_OCTOPUS_EDGES_NEEDED)) {\n+\tif (!(edge_value & GRAPH_EXTRA_EDGES_NEEDED)) {\n \t\tpptr = insert_parent_or_die(g, edge_value, pptr);\n \t\treturn 1;\n \t}\n \n-\tparent_data_ptr = (uint32_t*)(g->chunk_large_edges +\n+\tparent_data_ptr = (uint32_t*)(g->chunk_extra_edges +\n \t\t\t 4 * (uint64_t)(edge_value & GRAPH_EDGE_LAST_MASK));\n \tdo {\n \t\tedge_value = get_be32(parent_data_ptr);\n@@ -507,7 +507,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\tif (!parent)\n \t\t\tedge_value = GRAPH_PARENT_NONE;\n \t\telse if (parent->next)\n-\t\t\tedge_value = GRAPH_OCTOPUS_EDGES_NEEDED | num_extra_edges;\n+\t\t\tedge_value = GRAPH_EXTRA_EDGES_NEEDED | num_extra_edges;\n \t\telse {\n \t\t\tedge_value = sha1_pos(parent->item->object.oid.hash,\n \t\t\t\t\t commits,\n@@ -519,7 +519,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \n \t\thashwrite_be32(f, edge_value);\n \n-\t\tif (edge_value & GRAPH_OCTOPUS_EDGES_NEEDED) {\n+\t\tif (edge_value & GRAPH_EXTRA_EDGES_NEEDED) {\n \t\t\tdo {\n \t\t\t\tnum_extra_edges++;\n \t\t\t\tparent = parent->next;\n@@ -540,7 +540,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t}\n }\n \n-static void write_graph_chunk_large_edges(struct hashfile *f,\n+static void write_graph_chunk_extra_edges(struct hashfile *f,\n \t\t\t\t\t struct commit **commits,\n \t\t\t\t\t int nr_commits)\n {\n@@ -917,7 +917,7 @@ void write_commit_graph(const char *obj_dir,\n \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n \tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n \tif (num_extra_edges)\n-\t\tchunk_ids[3] = GRAPH_CHUNKID_LARGEEDGES;\n+\t\tchunk_ids[3] = GRAPH_CHUNKID_EXTRAEDGES;\n \telse\n \t\tchunk_ids[3] = 0;\n \tchunk_ids[4] = 0;\n@@ -940,7 +940,7 @@ void write_commit_graph(const char *obj_dir,\n \twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n \twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n \twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n-\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n+\twrite_graph_chunk_extra_edges(f, commits.list, commits.nr);\n \n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 9db40b4d3a..e6aff2c2e1 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -49,7 +49,7 @@ struct commit_graph {\n \tconst uint32_t *chunk_oid_fanout;\n \tconst unsigned char *chunk_oid_lookup;\n \tconst unsigned char *chunk_commit_data;\n-\tconst unsigned char *chunk_large_edges;\n+\tconst unsigned char *chunk_extra_edges;\n };\n \n struct commit_graph *load_commit_graph_one(const char *graph_file);\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 5fe21db99f..f4deb13b1d 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -122,7 +122,7 @@ test_expect_success 'write graph with merges' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit commit-graph write &&\n \ttest_path_is_file $objdir/info/commit-graph &&\n-\tgraph_read_expect \"10\" \"large_edges\"\n+\tgraph_read_expect \"10\" \"extra_edges\"\n '\n \n graph_git_behavior 'merge 1 vs 2' full merge/1 merge/2\n@@ -157,7 +157,7 @@ test_expect_success 'write graph with new commit' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit commit-graph write &&\n \ttest_path_is_file $objdir/info/commit-graph &&\n-\tgraph_read_expect \"11\" \"large_edges\"\n+\tgraph_read_expect \"11\" \"extra_edges\"\n '\n \n graph_git_behavior 'full graph, commit 8 vs merge 1' full commits/8 merge/1\n@@ -167,7 +167,7 @@ test_expect_success 'write graph with nothing new' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit commit-graph write &&\n \ttest_path_is_file $objdir/info/commit-graph &&\n-\tgraph_read_expect \"11\" \"large_edges\"\n+\tgraph_read_expect \"11\" \"extra_edges\"\n '\n \n graph_git_behavior 'cleared graph, commit 8 vs merge 1' full commits/8 merge/1\n@@ -177,7 +177,7 @@ test_expect_success 'build graph from latest pack with closure' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tcat new-idx | git commit-graph write --stdin-packs &&\n \ttest_path_is_file $objdir/info/commit-graph &&\n-\tgraph_read_expect \"9\" \"large_edges\"\n+\tgraph_read_expect \"9\" \"extra_edges\"\n '\n \n graph_git_behavior 'graph from pack, commit 8 vs merge 1' full commits/8 merge/1\n@@ -200,7 +200,7 @@ test_expect_success 'build graph from commits with append' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit rev-parse merge/3 | git commit-graph write --stdin-commits --append &&\n \ttest_path_is_file $objdir/info/commit-graph &&\n-\tgraph_read_expect \"10\" \"large_edges\"\n+\tgraph_read_expect \"10\" \"extra_edges\"\n '\n \n graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n@@ -210,7 +210,7 @@ test_expect_success 'build graph using --reachable' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit commit-graph write --reachable &&\n \ttest_path_is_file $objdir/info/commit-graph &&\n-\tgraph_read_expect \"11\" \"large_edges\"\n+\tgraph_read_expect \"11\" \"extra_edges\"\n '\n \n graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n@@ -231,7 +231,7 @@ test_expect_success 'write graph in bare repo' '\n \tcd \"$TRASH_DIRECTORY/bare\" &&\n \tgit commit-graph write &&\n \ttest_path_is_file $baredir/info/commit-graph &&\n-\tgraph_read_expect \"11\" \"large_edges\"\n+\tgraph_read_expect \"11\" \"extra_edges\"\n '\n \n graph_git_behavior 'bare repo with graph, commit 8 vs merge 1' bare commits/8 merge/1\n-- \n2.20.1.499.gf60de1223c\n\n"},{"id":"367113","messageId":"20190118170549.30403-3-szeder.dev@gmail.com","threadId":"49275","inReplyTo":"20190118170549.30403-1-szeder.dev@gmail.com","subject":"[PATCH v2 2/2] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-18T17:05:49Z","receivedAt":"2019-01-18T17:06:08Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"The optional 'Large Edge List' chunk of the commit graph file stores\nparent information for commits with more than two parents. Since the\nchunk is optional, write_commit_graph() looks through all commits to\nfind those with more than two parents, and then writes the commit\ngraph file header accordingly, i.e. if there are no such commits, then\nthere won't be a 'Large Edge List' chunk written, only the three\nmandatory chunks.\n\nHowever, when it later comes to writing actual chunk data,\nwrite_commit_graph() unconditionally invokes\nwrite_graph_chunk_large_edges(), even when it was decided earlier that\nthat chunk won't be written. Strictly speaking there is no bug here,\nbecause write_graph_chunk_large_edges() won't write anything if it\ndoesn't find any commits with more than two parents, but then it\nunnecessarily and in vain looks through all commits once again in\nsearch for such commits.\n\nDon't call write_graph_chunk_large_edges() when that chunk won't be\nwritten to spare an unnecessary iteration over all commits.\n\nSigned-off-by: SZEDER Gábor \n---\n commit-graph.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 544d6b8b4a..bf9eb44ca9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -940,7 +940,8 @@ void write_commit_graph(const char *obj_dir,\n \twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n \twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n \twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n-\twrite_graph_chunk_extra_edges(f, commits.list, commits.nr);\n+\tif (num_extra_edges)\n+\t\twrite_graph_chunk_extra_edges(f, commits.list, commits.nr);\n \n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n-- \n2.20.1.499.gf60de1223c\n\n"},{"id":"367114","messageId":"20190118171605.GM840@szeder.dev","threadId":"49275","inReplyTo":"20190116132947.26025-4-avarab@gmail.com","subject":"Re: [PATCH v5 3/9] commit-graph write: add \"Writing out\" progress output","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-18T17:16:05Z","receivedAt":"2019-01-18T17:16:11Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Jan 16, 2019 at 02:29:41PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> @@ -941,11 +957,25 @@ void write_commit_graph(const char *obj_dir,\n> \t\thashwrite(f, chunk_write, 12);\n> \t}\n> \n> -\twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n> -\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n> -\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n> +\tif (report_progress) {\n> +\t\t/*\n> +\t\t * Each of the write_graph_chunk_*() functions just\n> +\t\t * below loops over our N commits. This number must be\n> +\t\t * kept in sync with the number of passes we're doing.\n> +\t\t */\n> +\t\tint graph_passes = 3;\n> +\t\tif (num_large_edges)\n> +\t\t\tgraph_passes++;\n\nwrite_commit_graph() already has the 'num_chunks' variable, which\nholds exactly the right value.\n\n> +\t\tprogress = start_delayed_progress(\n> +\t\t\t_(\"Writing out commit graph\"),\n> +\t\t\tgraph_passes * commits.nr);\n> +\t}\n> +\twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress, &progress_cnt);\n> +\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr, progress, &progress_cnt);\n> +\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr, progress, &progress_cnt);\n> \tif (num_large_edges)\n> -\t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n> +\t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr, progress, &progress_cnt);\n> +\tstop_progress(&progress);\n> \n> \tclose_commit_graph(the_repository);\n> \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n> -- \n> 2.20.1.153.gd81d796ee0\n> \n"},{"id":"367135","messageId":"xmqqimylkb3n.fsf@gitster-ct.c.googlers.com","threadId":"49275","inReplyTo":"20190118170549.30403-1-szeder.dev@gmail.com","subject":"Re: [PATCH v2 0/2] commit-graph: minor cleanup and optimization","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-18T19:41:00Z","receivedAt":"2019-01-18T19:41:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"SZEDER Gábor writes:\n\n> On Wed, Nov 21, 2018 at 06:32:19AM -0500, Derrick Stolee wrote:\n>> >>I rename these variables to 'num_large_edges', because the commit\n>> >>graph file format speaks about the 'Large Edge List' chunk.\n>> >>\n>> >>However, I do find that the term 'extra' makes much more sense\n>> >>\n>> >>Would it make sense to do the rename in the other direction?\n>> >I tend to agree that \"large edge\" is a misnomer.\n>> \n>> I agree with you both. \"Extra\" is better.\n>\n> OK, so here are the two patches with the rename done in the other\n> direction.\n>\n> These should replace the bottom two patches of the topic\n> 'ab/commit-graph-write-progress'.\n\nTechnically it is \"replace the bottom two\", but it involves a bit\nmore. The effect of this cascades down, so patch 3 and later need\nto be rebased on top, while doing some more s/large/extra/.\n\nI've done all that on a detached head, and the result compiles, but\nI'd rather see Ævar involved in the rebasing process, so I won't use\nthe result of _my_ rebasing (besides, the \"enumerate in pack order\"\noptimization also needs rebasing on top, that is another chore).\n\nThanks. I'd just consider these two patches as you giving feedback\nto Ævar's series and expect them to be included in a reroll of it.\n\n\n> SZEDER Gábor (2):\n> commit-graph: rename \"large edges\" to \"extra edges\"\n> commit-graph: don't call write_graph_chunk_large_edges() unnecessarily\n>\n> .../technical/commit-graph-format.txt | 4 +--\n> builtin/commit-graph.c | 4 +--\n> commit-graph.c | 25 ++++++++++---------\n> commit-graph.h | 2 +-\n> t/t5318-commit-graph.sh | 14 +++++------\n> 5 files changed, 25 insertions(+), 24 deletions(-)\n"},{"id":"367166","messageId":"xmqq8szhigj6.fsf@gitster-ct.c.googlers.com","threadId":"49275","inReplyTo":"20190116132947.26025-1-avarab@gmail.com","subject":"Re: [PATCH v5 0/9] commit-graph write: progress output improvements","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-19T01:26:37Z","receivedAt":"2019-01-19T01:26:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> This was reviewed & discussed shortly before 2.20 went out the door,\n> but the changes deemed too significant during the RC period, and\n> instead we had my much smaller 01ca387774 (\"commit-graph: split up\n> close_reachable() progress output\", 2018-11-19) as a fix.\n>\n> That's now landed in master, and this version is rebased on top of\n> that.\n>\n> The changes aren't significant, one patch was dropped because\n> 01ca387774 did another version of it already, and 09/09 is rewritten\n> v.s. what 10/10 was before due to being on top of 01ca387774.\n\nThanks for a well-written concise background and summary.\n"},{"id":"367172","messageId":"CAN0heSrAHGU0OE97XyVVajbNcA+Ed=gRr-3+opF7OPg6_FiBLg@mail.gmail.com","threadId":"49275","inReplyTo":"20190118170549.30403-3-szeder.dev@gmail.com","subject":"Re: [PATCH v2 2/2] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2019-01-19T09:32:36Z","receivedAt":"2019-01-19T09:32:52Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Fri, 18 Jan 2019 at 18:23, SZEDER Gábor wrote:\n> write_commit_graph() unconditionally invokes\n> write_graph_chunk_large_edges(), even when it was decided earlier that\n> that chunk won't be written. Strictly speaking there is no bug here,\n> because write_graph_chunk_large_edges() won't write anything if it\n\n> Don't call write_graph_chunk_large_edges() when that chunk won't be\n> written to spare an unnecessary iteration over all commits.\n\nThis commit message (including the one-line subject) needs some\ns/_large_/_extra_/.\n\n> - write_graph_chunk_extra_edges(f, commits.list, commits.nr);\n> + if (num_extra_edges)\n> + write_graph_chunk_extra_edges(f, commits.list, commits.nr);\n\nMartin\n"},{"id":"367181","messageId":"20190119202121.3590-1-avarab@gmail.com","threadId":"49275","inReplyTo":"20190116132947.26025-1-avarab@gmail.com","subject":"[PATCH v6 00/10] commit-graph write: progress output improvements","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-19T20:21:11Z","receivedAt":"2019-01-19T20:21:38Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Improvements since v6:\n\n * Integrate my \"commit-graph write: use pack order when finding\n commits\" patch, and per Junio's suggestion put it at the start so\n it's easier to split the two apart.\n\n * A signed-off-by of mine was missing on that patch. Fixed.\n\n * Replace SZEDER's two patches with his re-roll at\n https://public-inbox.org/git/20190118170549.30403-1-szeder.dev@gmail.com/\n for convenienc, since mine needed rebasing on top of his.\n\n * SZEDER rightly pointed out that I was doing some work for nothing\n in https://public-inbox.org/git/20190118171605.GM840@szeder.dev/;\n fixed.\n\nSZEDER Gábor (2):\n commit-graph: rename \"large edges\" to \"extra edges\"\n commit-graph: don't call write_graph_chunk_large_edges() unnecessarily\n\nÆvar Arnfjörð Bjarmason (8):\n commit-graph write: use pack order when finding commits\n commit-graph write: add \"Writing out\" progress output\n commit-graph write: more descriptive \"writing out\" output\n commit-graph write: show progress for object search\n commit-graph write: add more descriptive progress output\n commit-graph write: remove empty line for readability\n commit-graph write: add itermediate progress\n commit-graph write: emit a percentage for all progress\n\n .../technical/commit-graph-format.txt | 4 +-\n builtin/commit-graph.c | 4 +-\n commit-graph.c | 136 +++++++++++++-----\n commit-graph.h | 2 +-\n t/t5318-commit-graph.sh | 14 +-\n 5 files changed, 110 insertions(+), 50 deletions(-)\n\nRange-diff:\n 1: 07d06c50c0 < -: ---------- commit-graph: rename 'num_extra_edges' variable to 'num_large_edges'\n -: ---------- > 1: 9ca0b5573b commit-graph write: use pack order when finding commits\n -: ---------- > 2: 4370ea7591 commit-graph: rename \"large edges\" to \"extra edges\"\n 2: 904dda1e7a ! 3: 3468e30c89 commit-graph: don't call write_graph_chunk_large_edges() unnecessarily\n @@ -10,19 +10,19 @@\n there won't be a 'Large Edge List' chunk written, only the three\n mandatory chunks.\n \n - However, when it comes to writing chunk data, write_commit_graph()\n - unconditionally invokes write_graph_chunk_large_edges(), even when it\n - was decided earlier that that chunk won't be written. Strictly\n - speaking there is no bug here, because write_graph_chunk_large_edges()\n - won't write anything because it won't find any commits with more than\n - two parents, but then it unnecessarily and in vain looks through all\n - commits once again in search for such commits.\n + However, when it later comes to writing actual chunk data,\n + write_commit_graph() unconditionally invokes\n + write_graph_chunk_large_edges(), even when it was decided earlier that\n + that chunk won't be written. Strictly speaking there is no bug here,\n + because write_graph_chunk_large_edges() won't write anything if it\n + doesn't find any commits with more than two parents, but then it\n + unnecessarily and in vain looks through all commits once again in\n + search for such commits.\n \n Don't call write_graph_chunk_large_edges() when that chunk won't be\n written to spare an unnecessary iteration over all commits.\n \n Signed-off-by: SZEDER Gábor \n - Signed-off-by: Ævar Arnfjörð Bjarmason \n \n diff --git a/commit-graph.c b/commit-graph.c\n --- a/commit-graph.c\n @@ -31,9 +31,9 @@\n \twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n \twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n \twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n --\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n -+\tif (num_large_edges)\n -+\t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n +-\twrite_graph_chunk_extra_edges(f, commits.list, commits.nr);\n ++\tif (num_extra_edges)\n ++\t\twrite_graph_chunk_extra_edges(f, commits.list, commits.nr);\n \n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n 3: 1126c7e29d < -: ---------- commit-graph write: rephrase confusing progress output\n 4: 9c17f56ed3 ! 4: 1860fa606e commit-graph write: add \"Writing out\" progress output\n @@ -10,14 +10,14 @@\n small repositories, but before this change we'd noticeably hang for\n 2-3 seconds at the end on medium sized repositories such as linux.git.\n \n - Now we'll instead show output like this, and have no human-observable\n - point at which we're not producing progress output:\n + Now we'll instead show output like this, and reduce the\n + human-observable times at which we're not producing progress output:\n \n - $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n - Finding commits for commit graph: 6365442, done.\n - Annotating commit graph: 2391666, done.\n - Computing commit graph generation numbers: 100% (797222/797222), done.\n - Writing out commit graph: 100% (3188888/3188888), done.\n + $ ~/g/git/git --exec-path=$HOME/g/git -C ~/g/2015-04-03-1M-git commit-graph write\n + Finding commits for commit graph: 13064614, done.\n + Expanding reachable commits in commit graph: 1000447, done.\n + Computing commit graph generation numbers: 100% (1000447/1000447), done.\n + Writing out commit graph: 100% (3001341/3001341), done.\n \n This \"Writing out\" number is 3x or 4x the number of commits, depending\n on the graph we're processing. A later change will make this explicit\n @@ -87,7 +87,7 @@\n \t\thashwrite(f, get_commit_tree_oid(*list)->hash, hash_len);\n @@\n \n - static void write_graph_chunk_large_edges(struct hashfile *f,\n + static void write_graph_chunk_extra_edges(struct hashfile *f,\n \t\t\t\t\t struct commit **commits,\n -\t\t\t\t\t int nr_commits)\n +\t\t\t\t\t int nr_commits,\n @@ -107,7 +107,7 @@\n \t\t parent = parent->next)\n \t\t\tnum_parents++;\n @@\n - \tint num_large_edges;\n + \tint num_extra_edges;\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n +\tuint64_t progress_cnt = 0;\n @@ -121,25 +121,16 @@\n -\twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n -\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n -\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n -+\tif (report_progress) {\n -+\t\t/*\n -+\t\t * Each of the write_graph_chunk_*() functions just\n -+\t\t * below loops over our N commits. This number must be\n -+\t\t * kept in sync with the number of passes we're doing.\n -+\t\t */\n -+\t\tint graph_passes = 3;\n -+\t\tif (num_large_edges)\n -+\t\t\tgraph_passes++;\n ++\tif (report_progress)\n +\t\tprogress = start_delayed_progress(\n +\t\t\t_(\"Writing out commit graph\"),\n -+\t\t\tgraph_passes * commits.nr);\n -+\t}\n ++\t\t\tnum_chunks * commits.nr);\n +\twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress, &progress_cnt);\n +\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr, progress, &progress_cnt);\n +\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr, progress, &progress_cnt);\n - \tif (num_large_edges)\n --\t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n -+\t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr, progress, &progress_cnt);\n + \tif (num_extra_edges)\n +-\t\twrite_graph_chunk_extra_edges(f, commits.list, commits.nr);\n ++\t\twrite_graph_chunk_extra_edges(f, commits.list, commits.nr, progress, &progress_cnt);\n +\tstop_progress(&progress);\n \n \tclose_commit_graph(the_repository);\n 5: 79b0a467d9 ! 5: cc36909add commit-graph write: more descriptive \"writing out\" output\n @@ -11,11 +11,11 @@\n has to do with writing out the commit graph. Now e.g. on linux.git we\n emit:\n \n - $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n - Finding commits for commit graph: 6365442, done.\n - Annotating commit graph: 2391666, done.\n - Computing commit graph generation numbers: 100% (797222/797222), done.\n - Writing out commit graph in 4 passes: 100% (3188888/3188888), done.\n + $ ~/g/git/git --exec-path=$HOME/g/git -C ~/g/linux commit-graph write\n + Finding commits for commit graph: 6529159, done.\n + Expanding reachable commits in commit graph: 815990, done.\n + Computing commit graph generation numbers: 100% (815983/815983), done.\n + Writing out commit graph in 4 passes: 100% (3263932/3263932), done.\n \n A note on i18n: Why are we using the Q_() function and passing a\n number & English text for a singular which'll never be used? Because\n @@ -36,26 +36,28 @@\n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n @@\n - \t\tint graph_passes = 3;\n - \t\tif (num_large_edges)\n - \t\t\tgraph_passes++;\n + \t\thashwrite(f, chunk_write, 12);\n + \t}\n + \n +-\tif (report_progress)\n ++\tif (report_progress) {\n +\t\tstrbuf_addf(&progress_title,\n +\t\t\t Q_(\"Writing out commit graph in %d pass\",\n +\t\t\t \"Writing out commit graph in %d passes\",\n -+\t\t\t graph_passes),\n -+\t\t\t graph_passes);\n ++\t\t\t num_chunks),\n ++\t\t\t num_chunks);\n \t\tprogress = start_delayed_progress(\n -\t\t\t_(\"Writing out commit graph\"),\n +\t\t\tprogress_title.buf,\n - \t\t\tgraph_passes * commits.nr);\n - \t}\n + \t\t\tnum_chunks * commits.nr);\n ++\t}\n \twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress, &progress_cnt);\n -@@\n - \t\twrite_graph_chunk_large_edges(f, commits.list, commits.nr, progress, &progress_cnt);\n + \twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr, progress, &progress_cnt);\n + \twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr, progress, &progress_cnt);\n + \tif (num_extra_edges)\n + \t\twrite_graph_chunk_extra_edges(f, commits.list, commits.nr, progress, &progress_cnt);\n \tstop_progress(&progress);\n - \n +\tstrbuf_release(&progress_title);\n -+\n + \n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n - \tcommit_lock_file(&lk);\n 6: b32be83b38 ! 6: 9a4aec16ca commit-graph write: show progress for object search\n @@ -8,12 +8,12 @@\n \n Before we'd emit on e.g. linux.git with \"commit-graph write\":\n \n - Finding commits for commit graph: 6365442, done.\n + Finding commits for commit graph: 6529159, done.\n [...]\n \n And now:\n \n - Finding commits for commit graph: 100% (6365442/6365442), done.\n + Finding commits for commit graph: 100% (6529159/6529159), done.\n [...]\n \n Since the commit graph only includes those commits that are packed\n @@ -59,7 +59,8 @@\n -\t\t\t\t_(\"Finding commits for commit graph\"), 0);\n +\t\t\t\t_(\"Finding commits for commit graph\"),\n +\t\t\t\tapprox_nr_objects);\n - \t\tfor_each_packed_object(add_packed_commits, &oids, 0);\n + \t\tfor_each_packed_object(add_packed_commits, &oids,\n + \t\t\t\t FOR_EACH_OBJECT_PACK_ORDER);\n +\t\tif (oids.progress_done < approx_nr_objects)\n +\t\t\tdisplay_progress(oids.progress, approx_nr_objects);\n \t\tstop_progress(&oids.progress);\n 7: 54276723c0 ! 7: 6b523d0f0d commit-graph write: add more descriptive progress output\n @@ -10,17 +10,17 @@\n we support:\n \n $ git commit-graph write\n - Finding commits for commit graph among packed objects: 100% (6365442/6365442), done.\n + Finding commits for commit graph among packed objects: 100% (6529159/6529159), done.\n [...]\n \n # Actually we don't emit this since this takes almost no time at\n # all. But if we did (s/_delayed//) we'd show:\n $ git for-each-ref --format='%(objectname)' | git commit-graph write --stdin-commits\n - Finding commits for commit graph from 584 refs: 100% (584/584), done.\n + Finding commits for commit graph from 630 refs: 100% (630/630), done.\n [...]\n \n $ (cd .git/objects/pack/ && ls *idx) | git commit-graph write --stdin-pack\n - Finding commits for commit graph in 2 packs: 6365442, done.\n + Finding commits for commit graph in 3 packs: 6529159, done.\n [...]\n \n The middle on of those is going to be the output users might see in\n @@ -89,5 +89,5 @@\n -\t\t\t\t_(\"Finding commits for commit graph\"),\n +\t\t\t\t_(\"Finding commits for commit graph among packed objects\"),\n \t\t\t\tapprox_nr_objects);\n - \t\tfor_each_packed_object(add_packed_commits, &oids, 0);\n - \t\tif (oids.progress_done < approx_nr_objects)\n + \t\tfor_each_packed_object(add_packed_commits, &oids,\n + \t\t\t\t FOR_EACH_OBJECT_PACK_ORDER);\n 8: 0e847366e1 = 8: 582f6f477f commit-graph write: remove empty line for readability\n 9: c388aff73e ! 9: 2c9fa68210 commit-graph write: add itermediate progress\n @@ -10,26 +10,29 @@\n million objects we'll now emit:\n \n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n - Finding commits for commit graph among packed objects: 100% (48333911/48333911), done.\n - Annotating commit graph: 21435984, done.\n - Counting distinct commits in commit graph: 100% (7145328/7145328), done.\n - Finding extra edges in commit graph: 100% (7145328/7145328), done.\n - Computing commit graph generation numbers: 100% (7145328/7145328), done.\n - Writing out commit graph in 4 passes: 100% (28581312/28581312), done.\n + Finding commits for commit graph among packed objects: 100% (124763727/124763727), done.\n + Loading known commits in commit graph: 100% (18989461/18989461), done.\n + Expanding reachable commits in commit graph: 100% (18989507/18989461), done.\n + Clearing commit marks in commit graph: 100% (18989507/18989507), done.\n + Counting distinct commits in commit graph: 100% (18989507/18989507), done.\n + Finding extra edges in commit graph: 100% (18989507/18989507), done.\n + Computing commit graph generation numbers: 100% (7250302/7250302), done.\n + Writing out commit graph in 4 passes: 100% (29001208/29001208), done.\n \n Whereas on a medium-sized repository such as linux.git these new\n progress bars won't have time to kick in and as before and we'll still\n emit output like:\n \n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n - Finding commits for commit graph among packed objects: 100% (6365442/6365442), done.\n - Annotating commit graph: 2391666, done.\n - Computing commit graph generation numbers: 100% (797222/797222), done.\n - Writing out commit graph in 4 passes: 100% (3188888/3188888), done.\n + Finding commits for commit graph among packed objects: 100% (6529159/6529159), done.\n + Expanding reachable commits in commit graph: 815990, done.\n + Computing commit graph generation numbers: 100% (815983/815983), done.\n + Writing out commit graph in 4 passes: 100% (3263932/3263932), done.\n \n The \"Counting distinct commits in commit graph\" phase will spend most\n of its time paused at \"0/*\" as we QSORT(...) the list. That's not\n - optimal, but at least we don't seem to be stalling anymore.\n + optimal, but at least we don't seem to be stalling anymore most of the\n + time.\n \n Signed-off-by: Ævar Arnfjörð Bjarmason \n \n @@ -59,7 +62,7 @@\n @@\n \tALLOC_ARRAY(commits.list, commits.alloc);\n \n - \tnum_large_edges = 0;\n + \tnum_extra_edges = 0;\n +\tif (report_progress)\n +\t\tprogress = start_delayed_progress(\n +\t\t\t_(\"Finding extra edges in commit graph\"),\n @@ -73,7 +76,7 @@\n @@\n \t\tcommits.nr++;\n \t}\n - \tnum_chunks = num_large_edges ? 4 : 3;\n + \tnum_chunks = num_extra_edges ? 4 : 3;\n +\tstop_progress(&progress);\n \n \tif (commits.nr >= GRAPH_PARENT_MISSING)\n10: fd692499e0 < -: ---------- commit-graph write: emit a percentage for all progress\n -: ---------- > 10: e69bb1be2c commit-graph write: emit a percentage for all progress\n-- \n2.20.1.153.gd81d796ee0\n\n"},{"id":"367182","messageId":"20190119202121.3590-2-avarab@gmail.com","threadId":"49275","inReplyTo":"20190116132947.26025-1-avarab@gmail.com","subject":"[PATCH v6 01/10] commit-graph write: use pack order when finding commits","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-19T20:21:12Z","receivedAt":"2019-01-19T20:21:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Slightly optimize the \"commit-graph write\" step by using\nFOR_EACH_OBJECT_PACK_ORDER with for_each_object_in_pack(). See commit\n[1] and [2] for the facility and a similar optimization for \"cat-file\".\n\nOn Linux it is around 5% slower to run:\n\n echo 1 >/proc/sys/vm/drop_caches &&\n cat .git/objects/pack/* >/dev/null &&\n git cat-file --batch-all-objects --batch-check --unordered\n\nThan the same thing with the \"cat\" omitted. This is as expected, since\nwe're iterating in pack order and the \"cat\" is extra work.\n\nBefore this change the opposite was true of \"commit-graph write\". We\nwere 6% faster if we first ran \"cat\" to efficiently populate the FS\ncache for our sole big pack on linux.git, than if we had populated it\nvia for_each_object_in_pack(). Now we're 3% faster without the \"cat\"\ninstead.\n\nMy tests were done on an unloaded Linux 3.10 system with 10 runs for\neach. Derrick Stolee did his own tests on Windows[3] showing a 2%\nimprovement with a high degree of accuracy.\n\n1. 736eb88fdc (\"for_each_packed_object: support iterating in\n pack-order\", 2018-08-10)\n\n2. 0750bb5b51 (\"cat-file: support \"unordered\" output for\n --batch-all-objects\", 2018-08-10)\n\n3. https://public-inbox.org/git/f71fa868-25e8-a9c9-46a6-611b987f1a8f@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 5c8fb4b134..981faf0465 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -816,7 +816,8 @@ void write_commit_graph(const char *obj_dir,\n \t\t\t\tdie(_(\"error adding pack %s\"), packname.buf);\n \t\t\tif (open_pack_index(p))\n \t\t\t\tdie(_(\"error opening index for %s\"), packname.buf);\n-\t\t\tfor_each_object_in_pack(p, add_packed_commits, &oids, 0);\n+\t\t\tfor_each_object_in_pack(p, add_packed_commits, &oids,\n+\t\t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n \t\t\tclose_pack(p);\n \t\t\tfree(p);\n \t\t}\n@@ -854,7 +855,8 @@ void write_commit_graph(const char *obj_dir,\n \t\tif (report_progress)\n \t\t\toids.progress = start_delayed_progress(\n \t\t\t\t_(\"Finding commits for commit graph\"), 0);\n-\t\tfor_each_packed_object(add_packed_commits, &oids, 0);\n+\t\tfor_each_packed_object(add_packed_commits, &oids,\n+\t\t\t\t FOR_EACH_OBJECT_PACK_ORDER);\n \t\tstop_progress(&oids.progress);\n \t}\n \n-- \n2.20.1.153.gd81d796ee0\n\n"},{"id":"367183","messageId":"20190119202121.3590-3-avarab@gmail.com","threadId":"49275","inReplyTo":"20190116132947.26025-1-avarab@gmail.com","subject":"[PATCH v6 02/10] commit-graph: rename \"large edges\" to \"extra edges\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-19T20:21:13Z","receivedAt":"2019-01-19T20:21:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: SZEDER Gábor \n\nThe optional 'Large Edge List' chunk of the commit graph file stores\nparent information for commits with more than two parents, and the\nnames of most of the macros, variables, struct fields, and functions\nrelated to this chunk contain the term \"large edges\", e.g.\nwrite_graph_chunk_large_edges(). However, it's not a really great\nterm, as the edges to the second and subsequent parents stored in this\nchunk are not any larger than the edges to the first and second\nparents stored in the \"main\" 'Commit Data' chunk. It's the number of\nedges, IOW number of parents, that is larger compared to non-merge and\n\"regular\" two-parent merge commits. And indeed, two functions in\n'commit-graph.c' have a local variable called 'num_extra_edges' that\nrefer to the same thing, and this \"extra edges\" term is much better at\ndescribing these edges.\n\nSo let's rename all these references to \"large edges\" in macro,\nvariable, function, etc. names to \"extra edges\". There is a\nGRAPH_OCTOPUS_EDGES_NEEDED macro as well; for the sake of consistency\nrename it to GRAPH_EXTRA_EDGES_NEEDED.\n\nWe can do so safely without causing any incompatibility issues,\nbecause the term \"large edges\" doesn't come up in the file format\nitself in any form (the chunk's magic is {'E', 'D', 'G', 'E'}, there\nis no 'L' in there), but only in the specification text. The string\n\"large edges\", however, does come up in the output of 'git\ncommit-graph read' and in tests looking at its input, but that command\nis explicitly documented as debugging aid, so we can change its output\nand the affected tests safely.\n\nSigned-off-by: SZEDER Gábor \n---\n .../technical/commit-graph-format.txt | 4 ++--\n builtin/commit-graph.c | 4 ++--\n commit-graph.c | 24 +++++++++----------\n commit-graph.h | 2 +-\n t/t5318-commit-graph.sh | 14 +++++------\n 5 files changed, 24 insertions(+), 24 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex cc0474ba3e..16452a0504 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -76,7 +76,7 @@ CHUNK DATA:\n of the ith commit. Stores value 0x7000000 if no parent in that\n position. If there are more than two parents, the second value\n has its most-significant bit on and the other bits store an array\n- position into the Large Edge List chunk.\n+ position into the Extra Edge List chunk.\n * The next 8 bytes store the generation number of the commit and\n the commit time in seconds since EPOCH. The generation number\n uses the higher 30 bits of the first 4 bytes, while the commit\n@@ -84,7 +84,7 @@ CHUNK DATA:\n 2 bits of the lowest byte, storing the 33rd and 34th bit of the\n commit time.\n \n- Large Edge List (ID: {'E', 'D', 'G', 'E'}) [Optional]\n+ Extra Edge List (ID: {'E', 'D', 'G', 'E'}) [Optional]\n This list of 4-byte values store the second through nth parents for\n all octopus merges. The second parent value in the commit data stores\n an array position within this list along with the most-significant bit\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex c02a3f1221..4ae502754c 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -110,8 +110,8 @@ static int graph_read(int argc, const char **argv)\n \t\tprintf(\" oid_lookup\");\n \tif (graph->chunk_commit_data)\n \t\tprintf(\" commit_metadata\");\n-\tif (graph->chunk_large_edges)\n-\t\tprintf(\" large_edges\");\n+\tif (graph->chunk_extra_edges)\n+\t\tprintf(\" extra_edges\");\n \tprintf(\"\\n\");\n \n \tUNLEAK(graph);\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 981faf0465..c5c6ab5367 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -21,7 +21,7 @@\n #define GRAPH_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n #define GRAPH_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define GRAPH_CHUNKID_DATA 0x43444154 /* \"CDAT\" */\n-#define GRAPH_CHUNKID_LARGEEDGES 0x45444745 /* \"EDGE\" */\n+#define GRAPH_CHUNKID_EXTRAEDGES 0x45444745 /* \"EDGE\" */\n \n #define GRAPH_DATA_WIDTH 36\n \n@@ -33,7 +33,7 @@\n #define GRAPH_OID_VERSION GRAPH_OID_VERSION_SHA1\n #define GRAPH_OID_LEN GRAPH_OID_LEN_SHA1\n \n-#define GRAPH_OCTOPUS_EDGES_NEEDED 0x80000000\n+#define GRAPH_EXTRA_EDGES_NEEDED 0x80000000\n #define GRAPH_PARENT_MISSING 0x7fffffff\n #define GRAPH_EDGE_LAST_MASK 0x7fffffff\n #define GRAPH_PARENT_NONE 0x70000000\n@@ -177,11 +177,11 @@ struct commit_graph *load_commit_graph_one(const char *graph_file)\n \t\t\t\tgraph->chunk_commit_data = data + chunk_offset;\n \t\t\tbreak;\n \n-\t\tcase GRAPH_CHUNKID_LARGEEDGES:\n-\t\t\tif (graph->chunk_large_edges)\n+\t\tcase GRAPH_CHUNKID_EXTRAEDGES:\n+\t\t\tif (graph->chunk_extra_edges)\n \t\t\t\tchunk_repeated = 1;\n \t\t\telse\n-\t\t\t\tgraph->chunk_large_edges = data + chunk_offset;\n+\t\t\t\tgraph->chunk_extra_edges = data + chunk_offset;\n \t\t\tbreak;\n \t\t}\n \n@@ -343,12 +343,12 @@ static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \tedge_value = get_be32(commit_data + g->hash_len + 4);\n \tif (edge_value == GRAPH_PARENT_NONE)\n \t\treturn 1;\n-\tif (!(edge_value & GRAPH_OCTOPUS_EDGES_NEEDED)) {\n+\tif (!(edge_value & GRAPH_EXTRA_EDGES_NEEDED)) {\n \t\tpptr = insert_parent_or_die(g, edge_value, pptr);\n \t\treturn 1;\n \t}\n \n-\tparent_data_ptr = (uint32_t*)(g->chunk_large_edges +\n+\tparent_data_ptr = (uint32_t*)(g->chunk_extra_edges +\n \t\t\t 4 * (uint64_t)(edge_value & GRAPH_EDGE_LAST_MASK));\n \tdo {\n \t\tedge_value = get_be32(parent_data_ptr);\n@@ -504,7 +504,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\tif (!parent)\n \t\t\tedge_value = GRAPH_PARENT_NONE;\n \t\telse if (parent->next)\n-\t\t\tedge_value = GRAPH_OCTOPUS_EDGES_NEEDED | num_extra_edges;\n+\t\t\tedge_value = GRAPH_EXTRA_EDGES_NEEDED | num_extra_edges;\n \t\telse {\n \t\t\tedge_value = sha1_pos(parent->item->object.oid.hash,\n \t\t\t\t\t commits,\n@@ -516,7 +516,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \n \t\thashwrite_be32(f, edge_value);\n \n-\t\tif (edge_value & GRAPH_OCTOPUS_EDGES_NEEDED) {\n+\t\tif (edge_value & GRAPH_EXTRA_EDGES_NEEDED) {\n \t\t\tdo {\n \t\t\t\tnum_extra_edges++;\n \t\t\t\tparent = parent->next;\n@@ -537,7 +537,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t}\n }\n \n-static void write_graph_chunk_large_edges(struct hashfile *f,\n+static void write_graph_chunk_extra_edges(struct hashfile *f,\n \t\t\t\t\t struct commit **commits,\n \t\t\t\t\t int nr_commits)\n {\n@@ -923,7 +923,7 @@ void write_commit_graph(const char *obj_dir,\n \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n \tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n \tif (num_extra_edges)\n-\t\tchunk_ids[3] = GRAPH_CHUNKID_LARGEEDGES;\n+\t\tchunk_ids[3] = GRAPH_CHUNKID_EXTRAEDGES;\n \telse\n \t\tchunk_ids[3] = 0;\n \tchunk_ids[4] = 0;\n@@ -946,7 +946,7 @@ void write_commit_graph(const char *obj_dir,\n \twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n \twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n \twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n-\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n+\twrite_graph_chunk_extra_edges(f, commits.list, commits.nr);\n \n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 9db40b4d3a..e6aff2c2e1 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -49,7 +49,7 @@ struct commit_graph {\n \tconst uint32_t *chunk_oid_fanout;\n \tconst unsigned char *chunk_oid_lookup;\n \tconst unsigned char *chunk_commit_data;\n-\tconst unsigned char *chunk_large_edges;\n+\tconst unsigned char *chunk_extra_edges;\n };\n \n struct commit_graph *load_commit_graph_one(const char *graph_file);\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 5fe21db99f..f4deb13b1d 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -122,7 +122,7 @@ test_expect_success 'write graph with merges' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit commit-graph write &&\n \ttest_path_is_file $objdir/info/commit-graph &&\n-\tgraph_read_expect \"10\" \"large_edges\"\n+\tgraph_read_expect \"10\" \"extra_edges\"\n '\n \n graph_git_behavior 'merge 1 vs 2' full merge/1 merge/2\n@@ -157,7 +157,7 @@ test_expect_success 'write graph with new commit' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit commit-graph write &&\n \ttest_path_is_file $objdir/info/commit-graph &&\n-\tgraph_read_expect \"11\" \"large_edges\"\n+\tgraph_read_expect \"11\" \"extra_edges\"\n '\n \n graph_git_behavior 'full graph, commit 8 vs merge 1' full commits/8 merge/1\n@@ -167,7 +167,7 @@ test_expect_success 'write graph with nothing new' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit commit-graph write &&\n \ttest_path_is_file $objdir/info/commit-graph &&\n-\tgraph_read_expect \"11\" \"large_edges\"\n+\tgraph_read_expect \"11\" \"extra_edges\"\n '\n \n graph_git_behavior 'cleared graph, commit 8 vs merge 1' full commits/8 merge/1\n@@ -177,7 +177,7 @@ test_expect_success 'build graph from latest pack with closure' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tcat new-idx | git commit-graph write --stdin-packs &&\n \ttest_path_is_file $objdir/info/commit-graph &&\n-\tgraph_read_expect \"9\" \"large_edges\"\n+\tgraph_read_expect \"9\" \"extra_edges\"\n '\n \n graph_git_behavior 'graph from pack, commit 8 vs merge 1' full commits/8 merge/1\n@@ -200,7 +200,7 @@ test_expect_success 'build graph from commits with append' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit rev-parse merge/3 | git commit-graph write --stdin-commits --append &&\n \ttest_path_is_file $objdir/info/commit-graph &&\n-\tgraph_read_expect \"10\" \"large_edges\"\n+\tgraph_read_expect \"10\" \"extra_edges\"\n '\n \n graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n@@ -210,7 +210,7 @@ test_expect_success 'build graph using --reachable' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit commit-graph write --reachable &&\n \ttest_path_is_file $objdir/info/commit-graph &&\n-\tgraph_read_expect \"11\" \"large_edges\"\n+\tgraph_read_expect \"11\" \"extra_edges\"\n '\n \n graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n@@ -231,7 +231,7 @@ test_expect_success 'write graph in bare repo' '\n \tcd \"$TRASH_DIRECTORY/bare\" &&\n \tgit commit-graph write &&\n \ttest_path_is_file $baredir/info/commit-graph &&\n-\tgraph_read_expect \"11\" \"large_edges\"\n+\tgraph_read_expect \"11\" \"extra_edges\"\n '\n \n graph_git_behavior 'bare repo with graph, commit 8 vs merge 1' bare commits/8 merge/1\n-- \n2.20.1.153.gd81d796ee0\n\n"},{"id":"367184","messageId":"20190119202121.3590-4-avarab@gmail.com","threadId":"49275","inReplyTo":"20190116132947.26025-1-avarab@gmail.com","subject":"[PATCH v6 03/10] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-19T20:21:14Z","receivedAt":"2019-01-19T20:21:47Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: SZEDER Gábor \n\nThe optional 'Large Edge List' chunk of the commit graph file stores\nparent information for commits with more than two parents. Since the\nchunk is optional, write_commit_graph() looks through all commits to\nfind those with more than two parents, and then writes the commit\ngraph file header accordingly, i.e. if there are no such commits, then\nthere won't be a 'Large Edge List' chunk written, only the three\nmandatory chunks.\n\nHowever, when it later comes to writing actual chunk data,\nwrite_commit_graph() unconditionally invokes\nwrite_graph_chunk_large_edges(), even when it was decided earlier that\nthat chunk won't be written. Strictly speaking there is no bug here,\nbecause write_graph_chunk_large_edges() won't write anything if it\ndoesn't find any commits with more than two parents, but then it\nunnecessarily and in vain looks through all commits once again in\nsearch for such commits.\n\nDon't call write_graph_chunk_large_edges() when that chunk won't be\nwritten to spare an unnecessary iteration over all commits.\n\nSigned-off-by: SZEDER Gábor \n---\n commit-graph.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex c5c6ab5367..e733ba180a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -946,7 +946,8 @@ void write_commit_graph(const char *obj_dir,\n \twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n \twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n \twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n-\twrite_graph_chunk_extra_edges(f, commits.list, commits.nr);\n+\tif (num_extra_edges)\n+\t\twrite_graph_chunk_extra_edges(f, commits.list, commits.nr);\n \n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n-- \n2.20.1.153.gd81d796ee0\n\n"},{"id":"367185","messageId":"20190119202121.3590-5-avarab@gmail.com","threadId":"49275","inReplyTo":"20190116132947.26025-1-avarab@gmail.com","subject":"[PATCH v6 04/10] commit-graph write: add \"Writing out\" progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-19T20:21:15Z","receivedAt":"2019-01-19T20:21:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add progress output to be shown when we're writing out the\ncommit-graph, this adds to the output already added in 7b0f229222\n(\"commit-graph write: add progress output\", 2018-09-17).\n\nAs noted in that commit most of the progress output isn't displayed on\nsmall repositories, but before this change we'd noticeably hang for\n2-3 seconds at the end on medium sized repositories such as linux.git.\n\nNow we'll instead show output like this, and reduce the\nhuman-observable times at which we're not producing progress output:\n\n $ ~/g/git/git --exec-path=$HOME/g/git -C ~/g/2015-04-03-1M-git commit-graph write\n Finding commits for commit graph: 13064614, done.\n Expanding reachable commits in commit graph: 1000447, done.\n Computing commit graph generation numbers: 100% (1000447/1000447), done.\n Writing out commit graph: 100% (3001341/3001341), done.\n\nThis \"Writing out\" number is 3x or 4x the number of commits, depending\non the graph we're processing. A later change will make this explicit\nto the user.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 39 ++++++++++++++++++++++++++++++---------\n 1 file changed, 30 insertions(+), 9 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex e733ba180a..a407d5bff4 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -430,7 +430,9 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n \t\t\t\t struct commit **commits,\n-\t\t\t\t int nr_commits)\n+\t\t\t\t int nr_commits,\n+\t\t\t\t struct progress *progress,\n+\t\t\t\t uint64_t *progress_cnt)\n {\n \tint i, count = 0;\n \tstruct commit **list = commits;\n@@ -444,6 +446,7 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n \t\twhile (count < nr_commits) {\n \t\t\tif ((*list)->object.oid.hash[0] != i)\n \t\t\t\tbreak;\n+\t\t\tdisplay_progress(progress, ++*progress_cnt);\n \t\t\tcount++;\n \t\t\tlist++;\n \t\t}\n@@ -453,12 +456,16 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n }\n \n static void write_graph_chunk_oids(struct hashfile *f, int hash_len,\n-\t\t\t\t struct commit **commits, int nr_commits)\n+\t\t\t\t struct commit **commits, int nr_commits,\n+\t\t\t\t struct progress *progress,\n+\t\t\t\t uint64_t *progress_cnt)\n {\n \tstruct commit **list = commits;\n \tint count;\n-\tfor (count = 0; count < nr_commits; count++, list++)\n+\tfor (count = 0; count < nr_commits; count++, list++) {\n+\t\tdisplay_progress(progress, ++*progress_cnt);\n \t\thashwrite(f, (*list)->object.oid.hash, (int)hash_len);\n+\t}\n }\n \n static const unsigned char *commit_to_sha1(size_t index, void *table)\n@@ -468,7 +475,9 @@ static const unsigned char *commit_to_sha1(size_t index, void *table)\n }\n \n static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n-\t\t\t\t struct commit **commits, int nr_commits)\n+\t\t\t\t struct commit **commits, int nr_commits,\n+\t\t\t\t struct progress *progress,\n+\t\t\t\t uint64_t *progress_cnt)\n {\n \tstruct commit **list = commits;\n \tstruct commit **last = commits + nr_commits;\n@@ -478,6 +487,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\tstruct commit_list *parent;\n \t\tint edge_value;\n \t\tuint32_t packedDate[2];\n+\t\tdisplay_progress(progress, ++*progress_cnt);\n \n \t\tparse_commit(*list);\n \t\thashwrite(f, get_commit_tree_oid(*list)->hash, hash_len);\n@@ -539,7 +549,9 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \n static void write_graph_chunk_extra_edges(struct hashfile *f,\n \t\t\t\t\t struct commit **commits,\n-\t\t\t\t\t int nr_commits)\n+\t\t\t\t\t int nr_commits,\n+\t\t\t\t\t struct progress *progress,\n+\t\t\t\t\t uint64_t *progress_cnt)\n {\n \tstruct commit **list = commits;\n \tstruct commit **last = commits + nr_commits;\n@@ -547,6 +559,9 @@ static void write_graph_chunk_extra_edges(struct hashfile *f,\n \n \twhile (list < last) {\n \t\tint num_parents = 0;\n+\n+\t\tdisplay_progress(progress, ++*progress_cnt);\n+\n \t\tfor (parent = (*list)->parents; num_parents < 3 && parent;\n \t\t parent = parent->next)\n \t\t\tnum_parents++;\n@@ -768,6 +783,7 @@ void write_commit_graph(const char *obj_dir,\n \tint num_extra_edges;\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n+\tuint64_t progress_cnt = 0;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n@@ -943,11 +959,16 @@ void write_commit_graph(const char *obj_dir,\n \t\thashwrite(f, chunk_write, 12);\n \t}\n \n-\twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n-\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n-\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n+\tif (report_progress)\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Writing out commit graph\"),\n+\t\t\tnum_chunks * commits.nr);\n+\twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress, &progress_cnt);\n+\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr, progress, &progress_cnt);\n+\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr, progress, &progress_cnt);\n \tif (num_extra_edges)\n-\t\twrite_graph_chunk_extra_edges(f, commits.list, commits.nr);\n+\t\twrite_graph_chunk_extra_edges(f, commits.list, commits.nr, progress, &progress_cnt);\n+\tstop_progress(&progress);\n \n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n-- \n2.20.1.153.gd81d796ee0\n\n"},{"id":"367186","messageId":"20190119202121.3590-6-avarab@gmail.com","threadId":"49275","inReplyTo":"20190116132947.26025-1-avarab@gmail.com","subject":"[PATCH v6 05/10] commit-graph write: more descriptive \"writing out\" output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-19T20:21:16Z","receivedAt":"2019-01-19T20:21:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the \"Writing out\" part of the progress output more\ndescriptive. Depending on the shape of the graph we either make 3 or 4\npasses over it.\n\nLet's present this information to the user in case they're wondering\nwhat this number, which is much larger than their number of commits,\nhas to do with writing out the commit graph. Now e.g. on linux.git we\nemit:\n\n $ ~/g/git/git --exec-path=$HOME/g/git -C ~/g/linux commit-graph write\n Finding commits for commit graph: 6529159, done.\n Expanding reachable commits in commit graph: 815990, done.\n Computing commit graph generation numbers: 100% (815983/815983), done.\n Writing out commit graph in 4 passes: 100% (3263932/3263932), done.\n\nA note on i18n: Why are we using the Q_() function and passing a\nnumber & English text for a singular which'll never be used? Because\nthe plural rules of translated languages may not match those of\nEnglish, and to use the plural function we need to use this format.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 12 ++++++++++--\n 1 file changed, 10 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex a407d5bff4..7c639c6a65 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -784,6 +784,7 @@ void write_commit_graph(const char *obj_dir,\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n \tuint64_t progress_cnt = 0;\n+\tstruct strbuf progress_title = STRBUF_INIT;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n@@ -959,16 +960,23 @@ void write_commit_graph(const char *obj_dir,\n \t\thashwrite(f, chunk_write, 12);\n \t}\n \n-\tif (report_progress)\n+\tif (report_progress) {\n+\t\tstrbuf_addf(&progress_title,\n+\t\t\t Q_(\"Writing out commit graph in %d pass\",\n+\t\t\t \"Writing out commit graph in %d passes\",\n+\t\t\t num_chunks),\n+\t\t\t num_chunks);\n \t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Writing out commit graph\"),\n+\t\t\tprogress_title.buf,\n \t\t\tnum_chunks * commits.nr);\n+\t}\n \twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress, &progress_cnt);\n \twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr, progress, &progress_cnt);\n \twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr, progress, &progress_cnt);\n \tif (num_extra_edges)\n \t\twrite_graph_chunk_extra_edges(f, commits.list, commits.nr, progress, &progress_cnt);\n \tstop_progress(&progress);\n+\tstrbuf_release(&progress_title);\n \n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n-- \n2.20.1.153.gd81d796ee0\n\n"},{"id":"367187","messageId":"20190119202121.3590-7-avarab@gmail.com","threadId":"49275","inReplyTo":"20190116132947.26025-1-avarab@gmail.com","subject":"[PATCH v6 06/10] commit-graph write: show progress for object search","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-19T20:21:17Z","receivedAt":"2019-01-19T20:21:53Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Show the percentage progress for the \"Finding commits for commit\ngraph\" phase for the common case where we're operating on all packs in\nthe repository, as \"commit-graph write\" or \"gc\" will do.\n\nBefore we'd emit on e.g. linux.git with \"commit-graph write\":\n\n Finding commits for commit graph: 6529159, done.\n [...]\n\nAnd now:\n\n Finding commits for commit graph: 100% (6529159/6529159), done.\n [...]\n\nSince the commit graph only includes those commits that are packed\n(via for_each_packed_object(...)) the approximate_object_count()\nreturns the actual number of objects we're going to process.\n\nStill, it is possible due to a race with \"gc\" or another process\nmaintaining packs that the number of objects we're going to process is\nlower than what approximate_object_count() reported. In that case we\ndon't want to stop the progress bar short of 100%. So let's make sure\nit snaps to 100% at the end.\n\nThe inverse case is also possible and more likely. I.e. that a new\npack has been added between approximate_object_count() and\nfor_each_packed_object(). In that case the percentage will go beyond\n100%, and we'll do nothing to snap it back to 100% at the end.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 9 +++++++--\n 1 file changed, 7 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 7c639c6a65..3124dfae13 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -785,12 +785,14 @@ void write_commit_graph(const char *obj_dir,\n \tstruct progress *progress = NULL;\n \tuint64_t progress_cnt = 0;\n \tstruct strbuf progress_title = STRBUF_INIT;\n+\tunsigned long approx_nr_objects;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n \n \toids.nr = 0;\n-\toids.alloc = approximate_object_count() / 32;\n+\tapprox_nr_objects = approximate_object_count();\n+\toids.alloc = approx_nr_objects / 32;\n \toids.progress = NULL;\n \toids.progress_done = 0;\n \n@@ -871,9 +873,12 @@ void write_commit_graph(const char *obj_dir,\n \tif (!pack_indexes && !commit_hex) {\n \t\tif (report_progress)\n \t\t\toids.progress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"), 0);\n+\t\t\t\t_(\"Finding commits for commit graph\"),\n+\t\t\t\tapprox_nr_objects);\n \t\tfor_each_packed_object(add_packed_commits, &oids,\n \t\t\t\t FOR_EACH_OBJECT_PACK_ORDER);\n+\t\tif (oids.progress_done < approx_nr_objects)\n+\t\t\tdisplay_progress(oids.progress, approx_nr_objects);\n \t\tstop_progress(&oids.progress);\n \t}\n \n-- \n2.20.1.153.gd81d796ee0\n\n"},{"id":"367188","messageId":"20190119202121.3590-9-avarab@gmail.com","threadId":"49275","inReplyTo":"20190116132947.26025-1-avarab@gmail.com","subject":"[PATCH v6 08/10] commit-graph write: remove empty line for readability","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-19T20:21:19Z","receivedAt":"2019-01-19T20:21:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Remove the empty line between a QSORT(...) and the subsequent oideq()\nfor-loop. This makes it clearer that the QSORT(...) is being done so\nthat we can run the oideq() loop on adjacent OIDs. Amends code added\nin 08fd81c9b6 (\"commit-graph: implement write_commit_graph()\",\n2018-04-02).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 936119b1f9..d4a7280fa1 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -896,7 +896,6 @@ void write_commit_graph(const char *obj_dir,\n \tclose_reachable(&oids, report_progress);\n \n \tQSORT(oids.list, oids.nr, commit_compare);\n-\n \tcount_distinct = 1;\n \tfor (i = 1; i < oids.nr; i++) {\n \t\tif (!oideq(&oids.list[i - 1], &oids.list[i]))\n-- \n2.20.1.153.gd81d796ee0\n\n"},{"id":"367189","messageId":"20190119202121.3590-8-avarab@gmail.com","threadId":"49275","inReplyTo":"20190116132947.26025-1-avarab@gmail.com","subject":"[PATCH v6 07/10] commit-graph write: add more descriptive progress output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-19T20:21:18Z","receivedAt":"2019-01-19T20:21:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make the progress output shown when we're searching for commits to\ninclude in the graph more descriptive. This amends code I added in\n7b0f229222 (\"commit-graph write: add progress output\", 2018-09-17).\n\nNow, on linux.git, we'll emit this sort of output in the various modes\nwe support:\n\n $ git commit-graph write\n Finding commits for commit graph among packed objects: 100% (6529159/6529159), done.\n [...]\n\n # Actually we don't emit this since this takes almost no time at\n # all. But if we did (s/_delayed//) we'd show:\n $ git for-each-ref --format='%(objectname)' | git commit-graph write --stdin-commits\n Finding commits for commit graph from 630 refs: 100% (630/630), done.\n [...]\n\n $ (cd .git/objects/pack/ && ls *idx) | git commit-graph write --stdin-pack\n Finding commits for commit graph in 3 packs: 6529159, done.\n [...]\n\nThe middle on of those is going to be the output users might see in\npractice, since it'll be emitted when they get the commit graph via\ngc.writeCommitGraph=true. But as noted above you need a really large\nnumber of refs for this message to show. It'll show up on a test\nrepository I have with ~165k refs:\n\n Finding commits for commit graph from 165203 refs: 100% (165203/165203), done.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 25 ++++++++++++++++++-------\n 1 file changed, 18 insertions(+), 7 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 3124dfae13..936119b1f9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -822,8 +822,12 @@ void write_commit_graph(const char *obj_dir,\n \t\tstrbuf_addf(&packname, \"%s/pack/\", obj_dir);\n \t\tdirlen = packname.len;\n \t\tif (report_progress) {\n-\t\t\toids.progress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"), 0);\n+\t\t\tstrbuf_addf(&progress_title,\n+\t\t\t\t Q_(\"Finding commits for commit graph in %d pack\",\n+\t\t\t\t \"Finding commits for commit graph in %d packs\",\n+\t\t\t\t pack_indexes->nr),\n+\t\t\t\t pack_indexes->nr);\n+\t\t\toids.progress = start_delayed_progress(progress_title.buf, 0);\n \t\t\toids.progress_done = 0;\n \t\t}\n \t\tfor (i = 0; i < pack_indexes->nr; i++) {\n@@ -841,14 +845,20 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tfree(p);\n \t\t}\n \t\tstop_progress(&oids.progress);\n+\t\tstrbuf_reset(&progress_title);\n \t\tstrbuf_release(&packname);\n \t}\n \n \tif (commit_hex) {\n-\t\tif (report_progress)\n-\t\t\tprogress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"),\n-\t\t\t\tcommit_hex->nr);\n+\t\tif (report_progress) {\n+\t\t\tstrbuf_addf(&progress_title,\n+\t\t\t\t Q_(\"Finding commits for commit graph from %d ref\",\n+\t\t\t\t \"Finding commits for commit graph from %d refs\",\n+\t\t\t\t commit_hex->nr),\n+\t\t\t\t commit_hex->nr);\n+\t\t\tprogress = start_delayed_progress(progress_title.buf,\n+\t\t\t\t\t\t\t commit_hex->nr);\n+\t\t}\n \t\tfor (i = 0; i < commit_hex->nr; i++) {\n \t\t\tconst char *end;\n \t\t\tstruct object_id oid;\n@@ -868,12 +878,13 @@ void write_commit_graph(const char *obj_dir,\n \t\t\t}\n \t\t}\n \t\tstop_progress(&progress);\n+\t\tstrbuf_reset(&progress_title);\n \t}\n \n \tif (!pack_indexes && !commit_hex) {\n \t\tif (report_progress)\n \t\t\toids.progress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph\"),\n+\t\t\t\t_(\"Finding commits for commit graph among packed objects\"),\n \t\t\t\tapprox_nr_objects);\n \t\tfor_each_packed_object(add_packed_commits, &oids,\n \t\t\t\t FOR_EACH_OBJECT_PACK_ORDER);\n-- \n2.20.1.153.gd81d796ee0\n\n"},{"id":"367190","messageId":"20190119202121.3590-10-avarab@gmail.com","threadId":"49275","inReplyTo":"20190116132947.26025-1-avarab@gmail.com","subject":"[PATCH v6 09/10] commit-graph write: add itermediate progress","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-19T20:21:20Z","receivedAt":"2019-01-19T20:21:58Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add progress output to sections of code between \"Annotating[...]\" and\n\"Computing[...]generation numbers\". This can collectively take 5-10\nseconds on a large enough repository.\n\nOn a test repository with I have with ~7 million commits and ~50\nmillion objects we'll now emit:\n\n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph among packed objects: 100% (124763727/124763727), done.\n Loading known commits in commit graph: 100% (18989461/18989461), done.\n Expanding reachable commits in commit graph: 100% (18989507/18989461), done.\n Clearing commit marks in commit graph: 100% (18989507/18989507), done.\n Counting distinct commits in commit graph: 100% (18989507/18989507), done.\n Finding extra edges in commit graph: 100% (18989507/18989507), done.\n Computing commit graph generation numbers: 100% (7250302/7250302), done.\n Writing out commit graph in 4 passes: 100% (29001208/29001208), done.\n\nWhereas on a medium-sized repository such as linux.git these new\nprogress bars won't have time to kick in and as before and we'll still\nemit output like:\n\n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph among packed objects: 100% (6529159/6529159), done.\n Expanding reachable commits in commit graph: 815990, done.\n Computing commit graph generation numbers: 100% (815983/815983), done.\n Writing out commit graph in 4 passes: 100% (3263932/3263932), done.\n\nThe \"Counting distinct commits in commit graph\" phase will spend most\nof its time paused at \"0/*\" as we QSORT(...) the list. That's not\noptimal, but at least we don't seem to be stalling anymore most of the\ntime.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 13 +++++++++++++\n 1 file changed, 13 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d4a7280fa1..889cdefc49 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -895,12 +895,19 @@ void write_commit_graph(const char *obj_dir,\n \n \tclose_reachable(&oids, report_progress);\n \n+\tif (report_progress)\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Counting distinct commits in commit graph\"),\n+\t\t\toids.nr);\n+\tdisplay_progress(progress, 0); /* TODO: Measure QSORT() progress */\n \tQSORT(oids.list, oids.nr, commit_compare);\n \tcount_distinct = 1;\n \tfor (i = 1; i < oids.nr; i++) {\n+\t\tdisplay_progress(progress, i + 1);\n \t\tif (!oideq(&oids.list[i - 1], &oids.list[i]))\n \t\t\tcount_distinct++;\n \t}\n+\tstop_progress(&progress);\n \n \tif (count_distinct >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n@@ -910,8 +917,13 @@ void write_commit_graph(const char *obj_dir,\n \tALLOC_ARRAY(commits.list, commits.alloc);\n \n \tnum_extra_edges = 0;\n+\tif (report_progress)\n+\t\tprogress = start_delayed_progress(\n+\t\t\t_(\"Finding extra edges in commit graph\"),\n+\t\t\toids.nr);\n \tfor (i = 0; i < oids.nr; i++) {\n \t\tint num_parents = 0;\n+\t\tdisplay_progress(progress, i + 1);\n \t\tif (i > 0 && oideq(&oids.list[i - 1], &oids.list[i]))\n \t\t\tcontinue;\n \n@@ -928,6 +940,7 @@ void write_commit_graph(const char *obj_dir,\n \t\tcommits.nr++;\n \t}\n \tnum_chunks = num_extra_edges ? 4 : 3;\n+\tstop_progress(&progress);\n \n \tif (commits.nr >= GRAPH_PARENT_MISSING)\n \t\tdie(_(\"too many commits to write graph\"));\n-- \n2.20.1.153.gd81d796ee0\n\n"},{"id":"367191","messageId":"20190119202121.3590-11-avarab@gmail.com","threadId":"49275","inReplyTo":"20190116132947.26025-1-avarab@gmail.com","subject":"[PATCH v6 10/10] commit-graph write: emit a percentage for all progress","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-19T20:21:21Z","receivedAt":"2019-01-19T20:21:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Follow-up 01ca387774 (\"commit-graph: split up close_reachable()\nprogress output\", 2018-11-19) by making the progress bars in\nclose_reachable() report a completion percentage. This fixes the last\noccurrence where in the commit graph writing where we didn't report\nthat.\n\nThe change in 01ca387774 split up the 1x progress bar in\nclose_reachable() into 3x, but left them as dumb counters without a\npercentage completion. Fixing that is easy, and the only reason it\nwasn't done already is because that commit was rushed in during the\nv2.20.0 RC period to fix the unrelated issue of over-reporting commit\nnumbers. See [1] and follow-ups for ML activity at the time and [2]\nfor an alternative approach where the progress bars weren't split up.\n\nNow for e.g. linux.git we'll emit:\n\n $ ~/g/git/git --exec-path=$HOME/g/git commit-graph write\n Finding commits for commit graph among packed objects: 100% (6529159/6529159), done.\n Expanding reachable commits in commit graph: 100% (815990/815980), done.\n Computing commit graph generation numbers: 100% (815983/815983), done.\n Writing out commit graph in 4 passes: 100% (3263932/3263932), done.\n\n1. https://public-inbox.org/git/20181119202300.18670-1-avarab@gmail.com/\n2. https://public-inbox.org/git/20181122153922.16912-11-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n commit-graph.c | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 889cdefc49..017225ccea 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -653,15 +653,15 @@ static void add_missing_parents(struct packed_oid_list *oids, struct commit *com\n \n static void close_reachable(struct packed_oid_list *oids, int report_progress)\n {\n-\tint i, j;\n+\tint i;\n \tstruct commit *commit;\n \tstruct progress *progress = NULL;\n \n \tif (report_progress)\n \t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Loading known commits in commit graph\"), j = 0);\n+\t\t\t_(\"Loading known commits in commit graph\"), oids->nr);\n \tfor (i = 0; i < oids->nr; i++) {\n-\t\tdisplay_progress(progress, ++j);\n+\t\tdisplay_progress(progress, i + 1);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n \t\tif (commit)\n \t\t\tcommit->object.flags |= UNINTERESTING;\n@@ -675,9 +675,9 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n \t */\n \tif (report_progress)\n \t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Expanding reachable commits in commit graph\"), j = 0);\n+\t\t\t_(\"Expanding reachable commits in commit graph\"), oids->nr);\n \tfor (i = 0; i < oids->nr; i++) {\n-\t\tdisplay_progress(progress, ++j);\n+\t\tdisplay_progress(progress, i + 1);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n \n \t\tif (commit && !parse_commit(commit))\n@@ -687,9 +687,9 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n \n \tif (report_progress)\n \t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Clearing commit marks in commit graph\"), j = 0);\n+\t\t\t_(\"Clearing commit marks in commit graph\"), oids->nr);\n \tfor (i = 0; i < oids->nr; i++) {\n-\t\tdisplay_progress(progress, ++j);\n+\t\tdisplay_progress(progress, i + 1);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n \n \t\tif (commit)\n-- \n2.20.1.153.gd81d796ee0\n\n"},{"id":"367337","messageId":"09f8fc01-898c-58f7-f14c-7a843e2e34b9@gmail.com","threadId":"49275","inReplyTo":"20190119202121.3590-1-avarab@gmail.com","subject":"Re: [PATCH v6 00/10] commit-graph write: progress output improvements","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-01-22T18:30:07Z","receivedAt":"2019-01-22T18:30:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/19/2019 3:21 PM, Ævar Arnfjörð Bjarmason wrote:\n> Improvements since v6:\n>\n> * Integrate my \"commit-graph write: use pack order when finding\n> commits\" patch, and per Junio's suggestion put it at the start so\n> it's easier to split the two apart.\n>\n> * A signed-off-by of mine was missing on that patch. Fixed.\n>\n> * Replace SZEDER's two patches with his re-roll at\n> https://public-inbox.org/git/20190118170549.30403-1-szeder.dev@gmail.com/\n> for convenienc, since mine needed rebasing on top of his.\n>\n> * SZEDER rightly pointed out that I was doing some work for nothing\n> in https://public-inbox.org/git/20190118171605.GM840@szeder.dev/;\n> fixed.\n>\n> SZEDER Gábor (2):\n> commit-graph: rename \"large edges\" to \"extra edges\"\n> commit-graph: don't call write_graph_chunk_large_edges() unnecessarily\n>\n> Ævar Arnfjörð Bjarmason (8):\n> commit-graph write: use pack order when finding commits\n> commit-graph write: add \"Writing out\" progress output\n> commit-graph write: more descriptive \"writing out\" output\n> commit-graph write: show progress for object search\n> commit-graph write: add more descriptive progress output\n> commit-graph write: remove empty line for readability\n> commit-graph write: add itermediate progress\n> commit-graph write: emit a percentage for all progress\n\nI'm a few days late, but I took another look through the patches and \nthey look good to me. Thanks!\n\n-Stolee\n\n"},{"id":"367345","messageId":"xmqqsgxkfpqa.fsf@gitster-ct.c.googlers.com","threadId":"49275","inReplyTo":"20190119202121.3590-1-avarab@gmail.com","subject":"Re: [PATCH v6 00/10] commit-graph write: progress output improvements","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-22T19:37:33Z","receivedAt":"2019-01-22T19:37:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> Improvements since v6:\n>\n> * Integrate my \"commit-graph write: use pack order when finding\n> commits\" patch, and per Junio's suggestion put it at the start so\n> it's easier to split the two apart.\n>\n> * A signed-off-by of mine was missing on that patch. Fixed.\n>\n> * Replace SZEDER's two patches with his re-roll at\n> https://public-inbox.org/git/20190118170549.30403-1-szeder.dev@gmail.com/\n> for convenienc, since mine needed rebasing on top of his.\n>\n> * SZEDER rightly pointed out that I was doing some work for nothing\n> in https://public-inbox.org/git/20190118171605.GM840@szeder.dev/;\n> fixed.\n\nReading the whole diff from the previous round, aside from\n\"large\" vs \"extra\" rename, and removal of the extra \"graph_passes\"\nvariable, this looks identical to the previous one.\n\nLooking good. Let's merge it to 'next' soonish.\n\nThanks.\n\n"},{"id":"367444","messageId":"20190123175122.30649-1-szeder.dev@gmail.com","threadId":"49275","inReplyTo":"20190119202121.3590-4-avarab@gmail.com","subject":"[PATCH v6.1 03/10] commit-graph: don't call write_graph_chunk_extra_edges() unnecessarily","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-23T17:51:22Z","receivedAt":"2019-01-23T17:51:36Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"The optional 'Extra Edge List' chunk of the commit graph file stores\nparent information for commits with more than two parents. Since the\nchunk is optional, write_commit_graph() looks through all commits to\nfind those with more than two parents, and then writes the commit\ngraph file header accordingly, i.e. if there are no such commits, then\nthere won't be a 'Extra Edge List' chunk written, only the three\nmandatory chunks.\n\nHowever, when it later comes to writing actual chunk data,\nwrite_commit_graph() unconditionally invokes\nwrite_graph_chunk_extra_edges(), even when it was decided earlier that\nthat chunk won't be written. Strictly speaking there is no bug here,\nbecause write_graph_chunk_extra_edges() won't write anything if it\ndoesn't find any commits with more than two parents, but then it\nunnecessarily and in vain looks through all commits once again in\nsearch for such commits.\n\nDon't call write_graph_chunk_extra_edges() when that chunk won't be\nwritten to spare an unnecessary iteration over all commits.\n\nSigned-off-by: SZEDER Gábor \n---\n\nThis is basically a s/large/extra/ in the commit message, as pointed\nout by Martin somewhere upthread. The patch is unchanged.\n\n commit-graph.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex c5c6ab5367..e733ba180a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -946,7 +946,8 @@ void write_commit_graph(const char *obj_dir,\n \twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n \twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n \twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n-\twrite_graph_chunk_extra_edges(f, commits.list, commits.nr);\n+\tif (num_extra_edges)\n+\t\twrite_graph_chunk_extra_edges(f, commits.list, commits.nr);\n \n \tclose_commit_graph(the_repository);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n-- \n2.20.1.642.gc55a771460\n\n"},{"id":"367446","messageId":"xmqqd0onclci.fsf@gitster-ct.c.googlers.com","threadId":"49275","inReplyTo":"09f8fc01-898c-58f7-f14c-7a843e2e34b9@gmail.com","subject":"Re: [PATCH v6 00/10] commit-graph write: progress output improvements","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-23T17:52:45Z","receivedAt":"2019-01-23T17:52:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 1/19/2019 3:21 PM, Ævar Arnfjörð Bjarmason wrote:\n>> Improvements since v6:\n>>\n>> * Integrate my \"commit-graph write: use pack order when finding\n>> commits\" patch, and per Junio's suggestion put it at the start so\n>> it's easier to split the two apart.\n>>\n>> * A signed-off-by of mine was missing on that patch. Fixed.\n>>\n>> * Replace SZEDER's two patches with his re-roll at\n>> https://public-inbox.org/git/20190118170549.30403-1-szeder.dev@gmail.com/\n>> for convenienc, since mine needed rebasing on top of his.\n>>\n>> * SZEDER rightly pointed out that I was doing some work for nothing\n>> in https://public-inbox.org/git/20190118171605.GM840@szeder.dev/;\n>> fixed.\n>>\n>> SZEDER Gábor (2):\n>> commit-graph: rename \"large edges\" to \"extra edges\"\n>> commit-graph: don't call write_graph_chunk_large_edges() unnecessarily\n>>\n>> Ævar Arnfjörð Bjarmason (8):\n>> commit-graph write: use pack order when finding commits\n>> commit-graph write: add \"Writing out\" progress output\n>> commit-graph write: more descriptive \"writing out\" output\n>> commit-graph write: show progress for object search\n>> commit-graph write: add more descriptive progress output\n>> commit-graph write: remove empty line for readability\n>> commit-graph write: add itermediate progress\n>> commit-graph write: emit a percentage for all progress\n>\n> I'm a few days late, but I took another look through the patches and\n> they look good to me. Thanks!\n\nThanks, all.\n"},{"id":"372233","messageId":"20190322102817.19708-1-szeder.dev@gmail.com","threadId":"49275","inReplyTo":"20190119202121.3590-11-avarab@gmail.com","subject":"[PATCH] commit-graph: don't show progress percentages while expanding reachable commits","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-03-22T10:28:17Z","receivedAt":"2019-03-22T10:28:59Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Commit 49bbc57a57 (commit-graph write: emit a percentage for all\nprogress, 2019-01-19) was a bit overeager when it added progress\npercentages to the \"Expanding reachable commits in commit graph\" phase\nas well, because most of the time the number of commits that phase has\nto iterate over is not known in advance and grows significantly, and,\nconsequently, we end up with nonsensical numbers:\n\n $ git commit-graph write --reachable\n Expanding reachable commits in commit graph: 138606% (824706/595), done.\n [...]\n\n $ git rev-parse v5.0 | git commit-graph write --stdin-commits\n Expanding reachable commits in commit graph: 81264400% (812644/1), done.\n [...]\n\nTherefore, don't show progress percentages in the \"Expanding reachable\ncommits in commit graph\" phase.\n\nSigned-off-by: SZEDER Gábor \n---\n commit-graph.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 017225ccea..60c06ce58f 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -672,13 +672,13 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n \t * As this loop runs, oids->nr may grow, but not more\n \t * than the number of missing commits in the reachable\n \t * closure.\n \t */\n \tif (report_progress)\n \t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Expanding reachable commits in commit graph\"), oids->nr);\n+\t\t\t_(\"Expanding reachable commits in commit graph\"), 0);\n \tfor (i = 0; i < oids->nr; i++) {\n \t\tdisplay_progress(progress, i + 1);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n \n \t\tif (commit && !parse_commit(commit))\n \t\t\tadd_missing_parents(oids, commit);\n-- \n2.21.0.539.g07239c3a71.dirty\n\n"},{"id":"372235","messageId":"87lg17cgf5.fsf@evledraar.gmail.com","threadId":"49275","inReplyTo":"20190322102817.19708-1-szeder.dev@gmail.com","subject":"Re: [PATCH] commit-graph: don't show progress percentages while expanding reachable commits","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-03-22T11:11:26Z","receivedAt":"2019-03-22T11:11:31Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Mar 22 2019, SZEDER Gábor wrote:\n\n> Commit 49bbc57a57 (commit-graph write: emit a percentage for all\n> progress, 2019-01-19) was a bit overeager when it added progress\n> percentages to the \"Expanding reachable commits in commit graph\" phase\n> as well, because most of the time the number of commits that phase has\n> to iterate over is not known in advance and grows significantly, and,\n> consequently, we end up with nonsensical numbers:\n>\n> $ git commit-graph write --reachable\n> Expanding reachable commits in commit graph: 138606% (824706/595), done.\n> [...]\n>\n> $ git rev-parse v5.0 | git commit-graph write --stdin-commits\n> Expanding reachable commits in commit graph: 81264400% (812644/1), done.\n> [...]\n>\n> Therefore, don't show progress percentages in the \"Expanding reachable\n> commits in commit graph\" phase.\n\nThere's indeed a bug here as your examples show, but there *are* cases\nwhere it's correct, as the commit message for my patch on \"master\" shows\nthere's cases where we correctly.\n\nSo this \"fixes\" things by always removing the progress, why not instead\npass down the state to close_reachable() about what we're walking over,\nso we can always show progress, or at least in some cases?\n\n> Signed-off-by: SZEDER Gábor \n> ---\n> commit-graph.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 017225ccea..60c06ce58f 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -672,13 +672,13 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n> \t * As this loop runs, oids->nr may grow, but not more\n> \t * than the number of missing commits in the reachable\n> \t * closure.\n> \t */\n> \tif (report_progress)\n> \t\tprogress = start_delayed_progress(\n> -\t\t\t_(\"Expanding reachable commits in commit graph\"), oids->nr);\n> +\t\t\t_(\"Expanding reachable commits in commit graph\"), 0);\n> \tfor (i = 0; i < oids->nr; i++) {\n> \t\tdisplay_progress(progress, i + 1);\n> \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n>\n> \t\tif (commit && !parse_commit(commit))\n> \t\t\tadd_missing_parents(oids, commit);\n"},{"id":"372236","messageId":"20190322111829.GC22459@szeder.dev","threadId":"49275","inReplyTo":"87lg17cgf5.fsf@evledraar.gmail.com","subject":"Re: [PATCH] commit-graph: don't show progress percentages while expanding reachable commits","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-03-22T11:18:29Z","receivedAt":"2019-03-22T11:18:36Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Fri, Mar 22, 2019 at 12:11:26PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Fri, Mar 22 2019, SZEDER Gábor wrote:\n> \n> > Commit 49bbc57a57 (commit-graph write: emit a percentage for all\n> > progress, 2019-01-19) was a bit overeager when it added progress\n> > percentages to the \"Expanding reachable commits in commit graph\" phase\n> > as well, because most of the time the number of commits that phase has\n> > to iterate over is not known in advance and grows significantly, and,\n> > consequently, we end up with nonsensical numbers:\n> >\n> > $ git commit-graph write --reachable\n> > Expanding reachable commits in commit graph: 138606% (824706/595), done.\n> > [...]\n> >\n> > $ git rev-parse v5.0 | git commit-graph write --stdin-commits\n> > Expanding reachable commits in commit graph: 81264400% (812644/1), done.\n> > [...]\n> >\n> > Therefore, don't show progress percentages in the \"Expanding reachable\n> > commits in commit graph\" phase.\n> \n> There's indeed a bug here as your examples show, but there *are* cases\n> where it's correct, as the commit message for my patch on \"master\" shows\n> there's cases where we correctly.\n> \n> So this \"fixes\" things by always removing the progress, why not instead\n> pass down the state to close_reachable() about what we're walking over,\n> so we can always show progress, or at least in some cases?\n\nThe cases where it does display correct percentages are exceptional,\nand doesn't worth the effort to try to find out whether ther current\noperation happens to be such a case.\n\n> > Signed-off-by: SZEDER Gábor \n> > ---\n> > commit-graph.c | 2 +-\n> > 1 file changed, 1 insertion(+), 1 deletion(-)\n> >\n> > diff --git a/commit-graph.c b/commit-graph.c\n> > index 017225ccea..60c06ce58f 100644\n> > --- a/commit-graph.c\n> > +++ b/commit-graph.c\n> > @@ -672,13 +672,13 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n> > \t * As this loop runs, oids->nr may grow, but not more\n> > \t * than the number of missing commits in the reachable\n> > \t * closure.\n> > \t */\n> > \tif (report_progress)\n> > \t\tprogress = start_delayed_progress(\n> > -\t\t\t_(\"Expanding reachable commits in commit graph\"), oids->nr);\n> > +\t\t\t_(\"Expanding reachable commits in commit graph\"), 0);\n> > \tfor (i = 0; i < oids->nr; i++) {\n> > \t\tdisplay_progress(progress, i + 1);\n> > \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n> >\n> > \t\tif (commit && !parse_commit(commit))\n> > \t\t\tadd_missing_parents(oids, commit);\n"},{"id":"372246","messageId":"87k1grc7al.fsf@evledraar.gmail.com","threadId":"49275","inReplyTo":"20190322111829.GC22459@szeder.dev","subject":"Re: [PATCH] commit-graph: don't show progress percentages while expanding reachable commits","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-03-22T14:28:34Z","receivedAt":"2019-03-22T14:28:40Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Mar 22 2019, SZEDER Gábor wrote:\n\n> On Fri, Mar 22, 2019 at 12:11:26PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>>\n>> On Fri, Mar 22 2019, SZEDER Gábor wrote:\n>>\n>> > Commit 49bbc57a57 (commit-graph write: emit a percentage for all\n>> > progress, 2019-01-19) was a bit overeager when it added progress\n>> > percentages to the \"Expanding reachable commits in commit graph\" phase\n>> > as well, because most of the time the number of commits that phase has\n>> > to iterate over is not known in advance and grows significantly, and,\n>> > consequently, we end up with nonsensical numbers:\n>> >\n>> > $ git commit-graph write --reachable\n>> > Expanding reachable commits in commit graph: 138606% (824706/595), done.\n>> > [...]\n>> >\n>> > $ git rev-parse v5.0 | git commit-graph write --stdin-commits\n>> > Expanding reachable commits in commit graph: 81264400% (812644/1), done.\n>> > [...]\n>> >\n>> > Therefore, don't show progress percentages in the \"Expanding reachable\n>> > commits in commit graph\" phase.\n>>\n>> There's indeed a bug here as your examples show, but there *are* cases\n>> where it's correct, as the commit message for my patch on \"master\" shows\n>> there's cases where we correctly.\n>>\n>> So this \"fixes\" things by always removing the progress, why not instead\n>> pass down the state to close_reachable() about what we're walking over,\n>> so we can always show progress, or at least in some cases?\n>\n> The cases where it does display correct percentages are exceptional,\n> and doesn't worth the effort to try to find out whether ther current\n> operation happens to be such a case.\n\nIt's the \"write\" entry point without arguments that displays the correct\nprogress. So not exceptional, but yeah, it's not what we use on \"gc\".\n\nIn any case, the problem is that sometimes we've walked the full set of\ncommits already, and some other times we haven't.\n\nSo in cases where we have we can show progress, and as a TODO (I think\nthis came up in previous discussions), we could do better if we had a\napproximate_commit_count().\n\nIn any case, the below fix seems correct to me, but I haven't poked it\nmuch. It *does* suffer from a theoretical race with the progress bar\nsimilar to d9b1b309cf (\"commit-graph write: show progress for object\nsearch\", 2019-01-19), but I work around it in the same way:\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 47e9be0a3a..0fab3d8b2b 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -693,7 +693,8 @@ static void add_missing_parents(struct packed_oid_list *oids, struct commit *com\n \t}\n }\n \n-static void close_reachable(struct packed_oid_list *oids, int report_progress)\n+static void close_reachable(struct packed_oid_list *oids, int report_progress,\n+\t\t\t uint64_t oids_count_for_progress)\n {\n \tint i;\n \tstruct commit *commit;\n@@ -717,7 +718,8 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n \t */\n \tif (report_progress)\n \t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Expanding reachable commits in commit graph\"), oids->nr);\n+\t\t\t_(\"Expanding reachable commits in commit graph\"),\n+\t\t\toids_count_for_progress);\n \tfor (i = 0; i < oids->nr; i++) {\n \t\tdisplay_progress(progress, i + 1);\n \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n@@ -725,6 +727,8 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n \t\tif (commit && !parse_commit(commit))\n \t\t\tadd_missing_parents(oids, commit);\n \t}\n+\tif (oids->nr < oids_count_for_progress)\n+\t\tdisplay_progress(progress, oids_count_for_progress);\n \tstop_progress(&progress);\n \n \tif (report_progress)\n@@ -829,6 +833,7 @@ void write_commit_graph(const char *obj_dir,\n \tuint64_t progress_cnt = 0;\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tunsigned long approx_nr_objects;\n+\tuint64_t oids_count_for_progress = 0;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n@@ -934,9 +939,10 @@ void write_commit_graph(const char *obj_dir,\n \t\tif (oids.progress_done < approx_nr_objects)\n \t\t\tdisplay_progress(oids.progress, approx_nr_objects);\n \t\tstop_progress(&oids.progress);\n+\t\toids_count_for_progress = oids.nr;\n \t}\n \n-\tclose_reachable(&oids, report_progress);\n+\tclose_reachable(&oids, report_progress, oids_count_for_progress);\n \n \tif (report_progress)\n \t\tprogress = start_delayed_progress(\n\n"},{"id":"372248","messageId":"87imwbc6x8.fsf@evledraar.gmail.com","threadId":"49275","inReplyTo":"87k1grc7al.fsf@evledraar.gmail.com","subject":"Re: [PATCH] commit-graph: don't show progress percentages while expanding reachable commits","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-03-22T14:36:35Z","receivedAt":"2019-03-22T14:36:40Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Mar 22 2019, Ævar Arnfjörð Bjarmason wrote:\n\n> On Fri, Mar 22 2019, SZEDER Gábor wrote:\n>\n>> On Fri, Mar 22, 2019 at 12:11:26PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>>>\n>>> On Fri, Mar 22 2019, SZEDER Gábor wrote:\n>>>\n>>> > Commit 49bbc57a57 (commit-graph write: emit a percentage for all\n>>> > progress, 2019-01-19) was a bit overeager when it added progress\n>>> > percentages to the \"Expanding reachable commits in commit graph\" phase\n>>> > as well, because most of the time the number of commits that phase has\n>>> > to iterate over is not known in advance and grows significantly, and,\n>>> > consequently, we end up with nonsensical numbers:\n>>> >\n>>> > $ git commit-graph write --reachable\n>>> > Expanding reachable commits in commit graph: 138606% (824706/595), done.\n>>> > [...]\n>>> >\n>>> > $ git rev-parse v5.0 | git commit-graph write --stdin-commits\n>>> > Expanding reachable commits in commit graph: 81264400% (812644/1), done.\n>>> > [...]\n>>> >\n>>> > Therefore, don't show progress percentages in the \"Expanding reachable\n>>> > commits in commit graph\" phase.\n>>>\n>>> There's indeed a bug here as your examples show, but there *are* cases\n>>> where it's correct, as the commit message for my patch on \"master\" shows\n>>> there's cases where we correctly.\n>>>\n>>> So this \"fixes\" things by always removing the progress, why not instead\n>>> pass down the state to close_reachable() about what we're walking over,\n>>> so we can always show progress, or at least in some cases?\n>>\n>> The cases where it does display correct percentages are exceptional,\n>> and doesn't worth the effort to try to find out whether ther current\n>> operation happens to be such a case.\n>\n> It's the \"write\" entry point without arguments that displays the correct\n> progress. So not exceptional, but yeah, it's not what we use on \"gc\".\n>\n> In any case, the problem is that sometimes we've walked the full set of\n> commits already, and some other times we haven't.\n>\n> So in cases where we have we can show progress, and as a TODO (I think\n> this came up in previous discussions), we could do better if we had a\n> approximate_commit_count().\n>\n> In any case, the below fix seems correct to me, but I haven't poked it\n> much. It *does* suffer from a theoretical race with the progress bar\n> similar to d9b1b309cf (\"commit-graph write: show progress for object\n> search\", 2019-01-19), but I work around it in the same way:\n\nSo double brainfart there, I meant \"race with new objects being\nadded/removed\", but ....\n\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 47e9be0a3a..0fab3d8b2b 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -693,7 +693,8 @@ static void add_missing_parents(struct packed_oid_list *oids, struct commit *com\n> \t}\n> }\n>\n> -static void close_reachable(struct packed_oid_list *oids, int report_progress)\n> +static void close_reachable(struct packed_oid_list *oids, int report_progress,\n> +\t\t\t uint64_t oids_count_for_progress)\n> {\n> \tint i;\n> \tstruct commit *commit;\n> @@ -717,7 +718,8 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n> \t */\n> \tif (report_progress)\n> \t\tprogress = start_delayed_progress(\n> -\t\t\t_(\"Expanding reachable commits in commit graph\"), oids->nr);\n> +\t\t\t_(\"Expanding reachable commits in commit graph\"),\n> +\t\t\toids_count_for_progress);\n> \tfor (i = 0; i < oids->nr; i++) {\n> \t\tdisplay_progress(progress, i + 1);\n> \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n> @@ -725,6 +727,8 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n> \t\tif (commit && !parse_commit(commit))\n> \t\t\tadd_missing_parents(oids, commit);\n> \t}\n> +\tif (oids->nr < oids_count_for_progress)\n> +\t\tdisplay_progress(progress, oids_count_for_progress);\n\n\n...that race exists for the approximate_object_count case in d9b1b309cf\nbecause in the time between approximate_object_count() and our second\nwalk via for_each_packed_object() the packs may have changed\n(e.g. concurrent \"git push\").\n\nBut this won't happen in this case, because we already know the commits\nwe want to seed from, and the only thing that will \"change\" is that we\nhaven't walked the parents of some of those commits when we e.g. seed\nfrom just HEAD.\n\nBut when oids_count_for_progress is >0 here we've walked all the commits\nwe found in the relevant packs already, it's not possible that they've\ngained new parents in the interim, so this \"oids->nr <\noids_count_for_progress\" case here is pointless cargo-culting.\n\n> \tstop_progress(&progress);\n>\n> \tif (report_progress)\n> @@ -829,6 +833,7 @@ void write_commit_graph(const char *obj_dir,\n> \tuint64_t progress_cnt = 0;\n> \tstruct strbuf progress_title = STRBUF_INIT;\n> \tunsigned long approx_nr_objects;\n> +\tuint64_t oids_count_for_progress = 0;\n>\n> \tif (!commit_graph_compatible(the_repository))\n> \t\treturn;\n> @@ -934,9 +939,10 @@ void write_commit_graph(const char *obj_dir,\n> \t\tif (oids.progress_done < approx_nr_objects)\n> \t\t\tdisplay_progress(oids.progress, approx_nr_objects);\n> \t\tstop_progress(&oids.progress);\n> +\t\toids_count_for_progress = oids.nr;\n> \t}\n>\n> -\tclose_reachable(&oids, report_progress);\n> +\tclose_reachable(&oids, report_progress, oids_count_for_progress);\n>\n> \tif (report_progress)\n> \t\tprogress = start_delayed_progress(\n"},{"id":"372250","messageId":"20190322145550.GE22459@szeder.dev","threadId":"49275","inReplyTo":"87k1grc7al.fsf@evledraar.gmail.com","subject":"Re: [PATCH] commit-graph: don't show progress percentages while expanding reachable commits","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-03-22T14:55:50Z","receivedAt":"2019-03-22T14:55:56Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Fri, Mar 22, 2019 at 03:28:34PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Fri, Mar 22 2019, SZEDER Gábor wrote:\n> \n> > On Fri, Mar 22, 2019 at 12:11:26PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> >>\n> >> On Fri, Mar 22 2019, SZEDER Gábor wrote:\n> >>\n> >> > Commit 49bbc57a57 (commit-graph write: emit a percentage for all\n> >> > progress, 2019-01-19) was a bit overeager when it added progress\n> >> > percentages to the \"Expanding reachable commits in commit graph\" phase\n> >> > as well, because most of the time the number of commits that phase has\n> >> > to iterate over is not known in advance and grows significantly, and,\n> >> > consequently, we end up with nonsensical numbers:\n> >> >\n> >> > $ git commit-graph write --reachable\n> >> > Expanding reachable commits in commit graph: 138606% (824706/595), done.\n> >> > [...]\n> >> >\n> >> > $ git rev-parse v5.0 | git commit-graph write --stdin-commits\n> >> > Expanding reachable commits in commit graph: 81264400% (812644/1), done.\n> >> > [...]\n> >> >\n> >> > Therefore, don't show progress percentages in the \"Expanding reachable\n> >> > commits in commit graph\" phase.\n> >>\n> >> There's indeed a bug here as your examples show, but there *are* cases\n> >> where it's correct, as the commit message for my patch on \"master\" shows\n> >> there's cases where we correctly.\n> >>\n> >> So this \"fixes\" things by always removing the progress, why not instead\n> >> pass down the state to close_reachable() about what we're walking over,\n> >> so we can always show progress, or at least in some cases?\n> >\n> > The cases where it does display correct percentages are exceptional,\n> > and doesn't worth the effort to try to find out whether ther current\n> > operation happens to be such a case.\n> \n> It's the \"write\" entry point without arguments that displays the correct\n> progress. So not exceptional, but yeah, it's not what we use on \"gc\".\n\nBit it displays the correct number only if all the reachable commits\nare in packfiles, which is not necessarily the case (e.g. unpacked\nsmall packs during 'git fetch').\n\n> In any case, the problem is that sometimes we've walked the full set of\n> commits already, and some other times we haven't.\n\n... and that we can't really be sure whether we've walked the full set\nof commits until after this loop.\n\n> So in cases where we have we can show progress, and as a TODO (I think\n> this came up in previous discussions), we could do better if we had a\n> approximate_commit_count().\n> \n> In any case, the below fix seems correct to me, but I haven't poked it\n> much. It *does* suffer from a theoretical race with the progress bar\n> similar to d9b1b309cf (\"commit-graph write: show progress for object\n> search\", 2019-01-19), but I work around it in the same way:\n> \n> diff --git a/commit-graph.c b/commit-graph.c\n> index 47e9be0a3a..0fab3d8b2b 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -693,7 +693,8 @@ static void add_missing_parents(struct packed_oid_list *oids, struct commit *com\n> \t}\n> }\n> \n> -static void close_reachable(struct packed_oid_list *oids, int report_progress)\n> +static void close_reachable(struct packed_oid_list *oids, int report_progress,\n> +\t\t\t uint64_t oids_count_for_progress)\n> {\n> \tint i;\n> \tstruct commit *commit;\n> @@ -717,7 +718,8 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n> \t */\n> \tif (report_progress)\n> \t\tprogress = start_delayed_progress(\n> -\t\t\t_(\"Expanding reachable commits in commit graph\"), oids->nr);\n> +\t\t\t_(\"Expanding reachable commits in commit graph\"),\n> +\t\t\toids_count_for_progress);\n> \tfor (i = 0; i < oids->nr; i++) {\n> \t\tdisplay_progress(progress, i + 1);\n> \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n> @@ -725,6 +727,8 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n> \t\tif (commit && !parse_commit(commit))\n> \t\t\tadd_missing_parents(oids, commit);\n> \t}\n> +\tif (oids->nr < oids_count_for_progress)\n> +\t\tdisplay_progress(progress, oids_count_for_progress);\n> \tstop_progress(&progress);\n> \n> \tif (report_progress)\n> @@ -829,6 +833,7 @@ void write_commit_graph(const char *obj_dir,\n> \tuint64_t progress_cnt = 0;\n> \tstruct strbuf progress_title = STRBUF_INIT;\n> \tunsigned long approx_nr_objects;\n> +\tuint64_t oids_count_for_progress = 0;\n> \n> \tif (!commit_graph_compatible(the_repository))\n> \t\treturn;\n> @@ -934,9 +939,10 @@ void write_commit_graph(const char *obj_dir,\n> \t\tif (oids.progress_done < approx_nr_objects)\n> \t\t\tdisplay_progress(oids.progress, approx_nr_objects);\n> \t\tstop_progress(&oids.progress);\n> +\t\toids_count_for_progress = oids.nr;\n> \t}\n> \n> -\tclose_reachable(&oids, report_progress);\n> +\tclose_reachable(&oids, report_progress, oids_count_for_progress);\n> \n> \tif (report_progress)\n> \t\tprogress = start_delayed_progress(\n> \n"},{"id":"372251","messageId":"87ftrfc5b7.fsf@evledraar.gmail.com","threadId":"49275","inReplyTo":"20190322145550.GE22459@szeder.dev","subject":"Re: [PATCH] commit-graph: don't show progress percentages while expanding reachable commits","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-03-22T15:11:24Z","receivedAt":"2019-03-22T15:11:30Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Mar 22 2019, SZEDER Gábor wrote:\n\n> On Fri, Mar 22, 2019 at 03:28:34PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>>\n>> On Fri, Mar 22 2019, SZEDER Gábor wrote:\n>>\n>> > On Fri, Mar 22, 2019 at 12:11:26PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>> >>\n>> >> On Fri, Mar 22 2019, SZEDER Gábor wrote:\n>> >>\n>> >> > Commit 49bbc57a57 (commit-graph write: emit a percentage for all\n>> >> > progress, 2019-01-19) was a bit overeager when it added progress\n>> >> > percentages to the \"Expanding reachable commits in commit graph\" phase\n>> >> > as well, because most of the time the number of commits that phase has\n>> >> > to iterate over is not known in advance and grows significantly, and,\n>> >> > consequently, we end up with nonsensical numbers:\n>> >> >\n>> >> > $ git commit-graph write --reachable\n>> >> > Expanding reachable commits in commit graph: 138606% (824706/595), done.\n>> >> > [...]\n>> >> >\n>> >> > $ git rev-parse v5.0 | git commit-graph write --stdin-commits\n>> >> > Expanding reachable commits in commit graph: 81264400% (812644/1), done.\n>> >> > [...]\n>> >> >\n>> >> > Therefore, don't show progress percentages in the \"Expanding reachable\n>> >> > commits in commit graph\" phase.\n>> >>\n>> >> There's indeed a bug here as your examples show, but there *are* cases\n>> >> where it's correct, as the commit message for my patch on \"master\" shows\n>> >> there's cases where we correctly.\n>> >>\n>> >> So this \"fixes\" things by always removing the progress, why not instead\n>> >> pass down the state to close_reachable() about what we're walking over,\n>> >> so we can always show progress, or at least in some cases?\n>> >\n>> > The cases where it does display correct percentages are exceptional,\n>> > and doesn't worth the effort to try to find out whether ther current\n>> > operation happens to be such a case.\n>>\n>> It's the \"write\" entry point without arguments that displays the correct\n>> progress. So not exceptional, but yeah, it's not what we use on \"gc\".\n>\n> Bit it displays the correct number only if all the reachable commits\n> are in packfiles, which is not necessarily the case (e.g. unpacked\n> small packs during 'git fetch').\n\nNo, argument-less \"write\" only considers packed commits.\n\n>> In any case, the problem is that sometimes we've walked the full set of\n>> commits already, and some other times we haven't.\n>\n> ... and that we can't really be sure whether we've walked the full set\n> of commits until after this loop.\n\nI'm fairly sure we can when we start with a full walk. See my\nexplanation in <87imwbc6x8.fsf@evledraar.gmail.com>, but I may have\nmissed something.\n\n>> So in cases where we have we can show progress, and as a TODO (I think\n>> this came up in previous discussions), we could do better if we had a\n>> approximate_commit_count().\n>>\n>> In any case, the below fix seems correct to me, but I haven't poked it\n>> much. It *does* suffer from a theoretical race with the progress bar\n>> similar to d9b1b309cf (\"commit-graph write: show progress for object\n>> search\", 2019-01-19), but I work around it in the same way:\n>>\n>> diff --git a/commit-graph.c b/commit-graph.c\n>> index 47e9be0a3a..0fab3d8b2b 100644\n>> --- a/commit-graph.c\n>> +++ b/commit-graph.c\n>> @@ -693,7 +693,8 @@ static void add_missing_parents(struct packed_oid_list *oids, struct commit *com\n>> \t}\n>> }\n>>\n>> -static void close_reachable(struct packed_oid_list *oids, int report_progress)\n>> +static void close_reachable(struct packed_oid_list *oids, int report_progress,\n>> +\t\t\t uint64_t oids_count_for_progress)\n>> {\n>> \tint i;\n>> \tstruct commit *commit;\n>> @@ -717,7 +718,8 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n>> \t */\n>> \tif (report_progress)\n>> \t\tprogress = start_delayed_progress(\n>> -\t\t\t_(\"Expanding reachable commits in commit graph\"), oids->nr);\n>> +\t\t\t_(\"Expanding reachable commits in commit graph\"),\n>> +\t\t\toids_count_for_progress);\n>> \tfor (i = 0; i < oids->nr; i++) {\n>> \t\tdisplay_progress(progress, i + 1);\n>> \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n>> @@ -725,6 +727,8 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n>> \t\tif (commit && !parse_commit(commit))\n>> \t\t\tadd_missing_parents(oids, commit);\n>> \t}\n>> +\tif (oids->nr < oids_count_for_progress)\n>> +\t\tdisplay_progress(progress, oids_count_for_progress);\n>> \tstop_progress(&progress);\n>>\n>> \tif (report_progress)\n>> @@ -829,6 +833,7 @@ void write_commit_graph(const char *obj_dir,\n>> \tuint64_t progress_cnt = 0;\n>> \tstruct strbuf progress_title = STRBUF_INIT;\n>> \tunsigned long approx_nr_objects;\n>> +\tuint64_t oids_count_for_progress = 0;\n>>\n>> \tif (!commit_graph_compatible(the_repository))\n>> \t\treturn;\n>> @@ -934,9 +939,10 @@ void write_commit_graph(const char *obj_dir,\n>> \t\tif (oids.progress_done < approx_nr_objects)\n>> \t\t\tdisplay_progress(oids.progress, approx_nr_objects);\n>> \t\tstop_progress(&oids.progress);\n>> +\t\toids_count_for_progress = oids.nr;\n>> \t}\n>>\n>> -\tclose_reachable(&oids, report_progress);\n>> +\tclose_reachable(&oids, report_progress, oids_count_for_progress);\n>>\n>> \tif (report_progress)\n>> \t\tprogress = start_delayed_progress(\n>>\n"},{"id":"372255","messageId":"20190322154943.GF22459@szeder.dev","threadId":"49275","inReplyTo":"87ftrfc5b7.fsf@evledraar.gmail.com","subject":"Re: [PATCH] commit-graph: don't show progress percentages while expanding reachable commits","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-03-22T15:49:43Z","receivedAt":"2019-03-22T15:49:51Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Fri, Mar 22, 2019 at 04:11:24PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Fri, Mar 22 2019, SZEDER Gábor wrote:\n> \n> > On Fri, Mar 22, 2019 at 03:28:34PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> >>\n> >> On Fri, Mar 22 2019, SZEDER Gábor wrote:\n> >>\n> >> > On Fri, Mar 22, 2019 at 12:11:26PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> >> >>\n> >> >> On Fri, Mar 22 2019, SZEDER Gábor wrote:\n> >> >>\n> >> >> > Commit 49bbc57a57 (commit-graph write: emit a percentage for all\n> >> >> > progress, 2019-01-19) was a bit overeager when it added progress\n> >> >> > percentages to the \"Expanding reachable commits in commit graph\" phase\n> >> >> > as well, because most of the time the number of commits that phase has\n> >> >> > to iterate over is not known in advance and grows significantly, and,\n> >> >> > consequently, we end up with nonsensical numbers:\n> >> >> >\n> >> >> > $ git commit-graph write --reachable\n> >> >> > Expanding reachable commits in commit graph: 138606% (824706/595), done.\n> >> >> > [...]\n> >> >> >\n> >> >> > $ git rev-parse v5.0 | git commit-graph write --stdin-commits\n> >> >> > Expanding reachable commits in commit graph: 81264400% (812644/1), done.\n> >> >> > [...]\n> >> >> >\n> >> >> > Therefore, don't show progress percentages in the \"Expanding reachable\n> >> >> > commits in commit graph\" phase.\n> >> >>\n> >> >> There's indeed a bug here as your examples show, but there *are* cases\n> >> >> where it's correct, as the commit message for my patch on \"master\" shows\n> >> >> there's cases where we correctly.\n> >> >>\n> >> >> So this \"fixes\" things by always removing the progress, why not instead\n> >> >> pass down the state to close_reachable() about what we're walking over,\n> >> >> so we can always show progress, or at least in some cases?\n> >> >\n> >> > The cases where it does display correct percentages are exceptional,\n> >> > and doesn't worth the effort to try to find out whether ther current\n> >> > operation happens to be such a case.\n> >>\n> >> It's the \"write\" entry point without arguments that displays the correct\n> >> progress. So not exceptional, but yeah, it's not what we use on \"gc\".\n> >\n> > Bit it displays the correct number only if all the reachable commits\n> > are in packfiles, which is not necessarily the case (e.g. unpacked\n> > small packs during 'git fetch').\n> \n> No, argument-less \"write\" only considers packed commits.\n\nNo, it considers packed commits as starting points, and then expands\nto all reachable commits, that's what that loop in question is about.\n\n $ git init\n Initialized empty Git repository in /tmp/test/.git/\n $ echo >file\n $ git add file\n $ git commit -q -m initial\n $ echo 1 >file\n $ git commit -q -m 1 file\n $ git rev-parse HEAD | git pack-objects\n .git/objects/pack/pack\n Enumerating objects: 1, done.\n Counting objects: 100% (1/1), done.\n ece3ff72952af2b28e048fa5c58db88c44312876\n Writing objects: 100% (1/1), done.\n Total 1 (delta 0), reused 0 (delta 0)\n $ git commit-graph write\n Computing commit graph generation numbers: 100% (2/2), done.\n\n\n> >> In any case, the problem is that sometimes we've walked the full set of\n> >> commits already, and some other times we haven't.\n> >\n> > ... and that we can't really be sure whether we've walked the full set\n> > of commits until after this loop.\n> \n> I'm fairly sure we can when we start with a full walk. See my\n> explanation in <87imwbc6x8.fsf@evledraar.gmail.com>, but I may have\n> missed something.\n> \n> >> So in cases where we have we can show progress, and as a TODO (I think\n> >> this came up in previous discussions), we could do better if we had a\n> >> approximate_commit_count().\n> >>\n> >> In any case, the below fix seems correct to me, but I haven't poked it\n> >> much. It *does* suffer from a theoretical race with the progress bar\n> >> similar to d9b1b309cf (\"commit-graph write: show progress for object\n> >> search\", 2019-01-19), but I work around it in the same way:\n> >>\n> >> diff --git a/commit-graph.c b/commit-graph.c\n> >> index 47e9be0a3a..0fab3d8b2b 100644\n> >> --- a/commit-graph.c\n> >> +++ b/commit-graph.c\n> >> @@ -693,7 +693,8 @@ static void add_missing_parents(struct packed_oid_list *oids, struct commit *com\n> >> \t}\n> >> }\n> >>\n> >> -static void close_reachable(struct packed_oid_list *oids, int report_progress)\n> >> +static void close_reachable(struct packed_oid_list *oids, int report_progress,\n> >> +\t\t\t uint64_t oids_count_for_progress)\n> >> {\n> >> \tint i;\n> >> \tstruct commit *commit;\n> >> @@ -717,7 +718,8 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n> >> \t */\n> >> \tif (report_progress)\n> >> \t\tprogress = start_delayed_progress(\n> >> -\t\t\t_(\"Expanding reachable commits in commit graph\"), oids->nr);\n> >> +\t\t\t_(\"Expanding reachable commits in commit graph\"),\n> >> +\t\t\toids_count_for_progress);\n> >> \tfor (i = 0; i < oids->nr; i++) {\n> >> \t\tdisplay_progress(progress, i + 1);\n> >> \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n> >> @@ -725,6 +727,8 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n> >> \t\tif (commit && !parse_commit(commit))\n> >> \t\t\tadd_missing_parents(oids, commit);\n> >> \t}\n> >> +\tif (oids->nr < oids_count_for_progress)\n> >> +\t\tdisplay_progress(progress, oids_count_for_progress);\n> >> \tstop_progress(&progress);\n> >>\n> >> \tif (report_progress)\n> >> @@ -829,6 +833,7 @@ void write_commit_graph(const char *obj_dir,\n> >> \tuint64_t progress_cnt = 0;\n> >> \tstruct strbuf progress_title = STRBUF_INIT;\n> >> \tunsigned long approx_nr_objects;\n> >> +\tuint64_t oids_count_for_progress = 0;\n> >>\n> >> \tif (!commit_graph_compatible(the_repository))\n> >> \t\treturn;\n> >> @@ -934,9 +939,10 @@ void write_commit_graph(const char *obj_dir,\n> >> \t\tif (oids.progress_done < approx_nr_objects)\n> >> \t\t\tdisplay_progress(oids.progress, approx_nr_objects);\n> >> \t\tstop_progress(&oids.progress);\n> >> +\t\toids_count_for_progress = oids.nr;\n> >> \t}\n> >>\n> >> -\tclose_reachable(&oids, report_progress);\n> >> +\tclose_reachable(&oids, report_progress, oids_count_for_progress);\n> >>\n> >> \tif (report_progress)\n> >> \t\tprogress = start_delayed_progress(\n> >>\n"},{"id":"372258","messageId":"20190322165228.GG22459@szeder.dev","threadId":"49275","inReplyTo":"20190322154943.GF22459@szeder.dev","subject":"Re: [PATCH] commit-graph: don't show progress percentages while expanding reachable commits","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-03-22T16:52:28Z","receivedAt":"2019-03-22T16:52:36Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Fri, Mar 22, 2019 at 04:49:43PM +0100, SZEDER Gábor wrote:\n> On Fri, Mar 22, 2019 at 04:11:24PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> > \n> > On Fri, Mar 22 2019, SZEDER Gábor wrote:\n> > \n> > > On Fri, Mar 22, 2019 at 03:28:34PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> > >>\n> > >> On Fri, Mar 22 2019, SZEDER Gábor wrote:\n> > >>\n> > >> > On Fri, Mar 22, 2019 at 12:11:26PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> > >> >>\n> > >> >> On Fri, Mar 22 2019, SZEDER Gábor wrote:\n> > >> >>\n> > >> >> > Commit 49bbc57a57 (commit-graph write: emit a percentage for all\n> > >> >> > progress, 2019-01-19) was a bit overeager when it added progress\n> > >> >> > percentages to the \"Expanding reachable commits in commit graph\" phase\n> > >> >> > as well, because most of the time the number of commits that phase has\n> > >> >> > to iterate over is not known in advance and grows significantly, and,\n> > >> >> > consequently, we end up with nonsensical numbers:\n> > >> >> >\n> > >> >> > $ git commit-graph write --reachable\n> > >> >> > Expanding reachable commits in commit graph: 138606% (824706/595), done.\n> > >> >> > [...]\n> > >> >> >\n> > >> >> > $ git rev-parse v5.0 | git commit-graph write --stdin-commits\n> > >> >> > Expanding reachable commits in commit graph: 81264400% (812644/1), done.\n> > >> >> > [...]\n> > >> >> >\n> > >> >> > Therefore, don't show progress percentages in the \"Expanding reachable\n> > >> >> > commits in commit graph\" phase.\n> > >> >>\n> > >> >> There's indeed a bug here as your examples show, but there *are* cases\n> > >> >> where it's correct, as the commit message for my patch on \"master\" shows\n> > >> >> there's cases where we correctly.\n> > >> >>\n> > >> >> So this \"fixes\" things by always removing the progress, why not instead\n> > >> >> pass down the state to close_reachable() about what we're walking over,\n> > >> >> so we can always show progress, or at least in some cases?\n> > >> >\n> > >> > The cases where it does display correct percentages are exceptional,\n> > >> > and doesn't worth the effort to try to find out whether ther current\n> > >> > operation happens to be such a case.\n> > >>\n> > >> It's the \"write\" entry point without arguments that displays the correct\n> > >> progress. So not exceptional, but yeah, it's not what we use on \"gc\".\n> > >\n> > > Bit it displays the correct number only if all the reachable commits\n> > > are in packfiles, which is not necessarily the case (e.g. unpacked\n> > > small packs during 'git fetch').\n> > \n> > No, argument-less \"write\" only considers packed commits.\n> \n> No, it considers packed commits as starting points, and then expands\n> to all reachable commits, that's what that loop in question is about.\n> \n> $ git init\n> Initialized empty Git repository in /tmp/test/.git/\n> $ echo >file\n> $ git add file\n> $ git commit -q -m initial\n> $ echo 1 >file\n> $ git commit -q -m 1 file\n> $ git rev-parse HEAD | git pack-objects\n> .git/objects/pack/pack\n> Enumerating objects: 1, done.\n> Counting objects: 100% (1/1), done.\n> ece3ff72952af2b28e048fa5c58db88c44312876\n> Writing objects: 100% (1/1), done.\n> Total 1 (delta 0), reused 0 (delta 0)\n> $ git commit-graph write\n> Computing commit graph generation numbers: 100% (2/2), done.\n> \n> \n> > >> In any case, the problem is that sometimes we've walked the full set of\n> > >> commits already, and some other times we haven't.\n> > >\n> > > ... and that we can't really be sure whether we've walked the full set\n> > > of commits until after this loop.\n> > \n> > I'm fairly sure we can when we start with a full walk. See my\n> > explanation in <87imwbc6x8.fsf@evledraar.gmail.com>, but I may have\n> > missed something.\n\nBTW, if we know for _sure_ that we started with the full set of\ncommits, then we don't have to iterate over all the history in this\n\"Expanding reachable commits...\" phase in the first place, because\nthere is nowhere to expand anyway. Maybe we wouldn't even have to\ncall close_reachable() in that case.\n\n"},{"id":"372263","messageId":"87ef6ydds8.fsf@evledraar.gmail.com","threadId":"49275","inReplyTo":"20190322154943.GF22459@szeder.dev","subject":"Re: [PATCH] commit-graph: don't show progress percentages while expanding reachable commits","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-03-22T17:23:03Z","receivedAt":"2019-03-22T17:23:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Mar 22 2019, SZEDER Gábor wrote:\n\n> On Fri, Mar 22, 2019 at 04:11:24PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>>\n>> On Fri, Mar 22 2019, SZEDER Gábor wrote:\n>>\n>> > On Fri, Mar 22, 2019 at 03:28:34PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>> >>\n>> >> On Fri, Mar 22 2019, SZEDER Gábor wrote:\n>> >>\n>> >> > On Fri, Mar 22, 2019 at 12:11:26PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>> >> >>\n>> >> >> On Fri, Mar 22 2019, SZEDER Gábor wrote:\n>> >> >>\n>> >> >> > Commit 49bbc57a57 (commit-graph write: emit a percentage for all\n>> >> >> > progress, 2019-01-19) was a bit overeager when it added progress\n>> >> >> > percentages to the \"Expanding reachable commits in commit graph\" phase\n>> >> >> > as well, because most of the time the number of commits that phase has\n>> >> >> > to iterate over is not known in advance and grows significantly, and,\n>> >> >> > consequently, we end up with nonsensical numbers:\n>> >> >> >\n>> >> >> > $ git commit-graph write --reachable\n>> >> >> > Expanding reachable commits in commit graph: 138606% (824706/595), done.\n>> >> >> > [...]\n>> >> >> >\n>> >> >> > $ git rev-parse v5.0 | git commit-graph write --stdin-commits\n>> >> >> > Expanding reachable commits in commit graph: 81264400% (812644/1), done.\n>> >> >> > [...]\n>> >> >> >\n>> >> >> > Therefore, don't show progress percentages in the \"Expanding reachable\n>> >> >> > commits in commit graph\" phase.\n>> >> >>\n>> >> >> There's indeed a bug here as your examples show, but there *are* cases\n>> >> >> where it's correct, as the commit message for my patch on \"master\" shows\n>> >> >> there's cases where we correctly.\n>> >> >>\n>> >> >> So this \"fixes\" things by always removing the progress, why not instead\n>> >> >> pass down the state to close_reachable() about what we're walking over,\n>> >> >> so we can always show progress, or at least in some cases?\n>> >> >\n>> >> > The cases where it does display correct percentages are exceptional,\n>> >> > and doesn't worth the effort to try to find out whether ther current\n>> >> > operation happens to be such a case.\n>> >>\n>> >> It's the \"write\" entry point without arguments that displays the correct\n>> >> progress. So not exceptional, but yeah, it's not what we use on \"gc\".\n>> >\n>> > Bit it displays the correct number only if all the reachable commits\n>> > are in packfiles, which is not necessarily the case (e.g. unpacked\n>> > small packs during 'git fetch').\n>>\n>> No, argument-less \"write\" only considers packed commits.\n>\n> No, it considers packed commits as starting points, and then expands\n> to all reachable commits, that's what that loop in question is about.\n>\n> $ git init\n> Initialized empty Git repository in /tmp/test/.git/\n> $ echo >file\n> $ git add file\n> $ git commit -q -m initial\n> $ echo 1 >file\n> $ git commit -q -m 1 file\n> $ git rev-parse HEAD | git pack-objects\n> .git/objects/pack/pack\n> Enumerating objects: 1, done.\n> Counting objects: 100% (1/1), done.\n> ece3ff72952af2b28e048fa5c58db88c44312876\n> Writing objects: 100% (1/1), done.\n> Total 1 (delta 0), reused 0 (delta 0)\n> $ git commit-graph write\n> Computing commit graph generation numbers: 100% (2/2), done.\n\nAh, indeed. I think in practice it'll be unlikely to happen in practice\nexcept on servers due to receive.unpackLimit, and then it won't be off\nby much.\n\nSo I think it's best to do something like my WIP patch upthread + the\n\"snap to 100%\" at the end behavior. I'll try to clean that up / test it\n/ submit that sometime soon.\n\n>> >> In any case, the problem is that sometimes we've walked the full set of\n>> >> commits already, and some other times we haven't.\n>> >\n>> > ... and that we can't really be sure whether we've walked the full set\n>> > of commits until after this loop.\n>>\n>> I'm fairly sure we can when we start with a full walk. See my\n>> explanation in <87imwbc6x8.fsf@evledraar.gmail.com>, but I may have\n>> missed something.\n>>\n>> >> So in cases where we have we can show progress, and as a TODO (I think\n>> >> this came up in previous discussions), we could do better if we had a\n>> >> approximate_commit_count().\n>> >>\n>> >> In any case, the below fix seems correct to me, but I haven't poked it\n>> >> much. It *does* suffer from a theoretical race with the progress bar\n>> >> similar to d9b1b309cf (\"commit-graph write: show progress for object\n>> >> search\", 2019-01-19), but I work around it in the same way:\n>> >>\n>> >> diff --git a/commit-graph.c b/commit-graph.c\n>> >> index 47e9be0a3a..0fab3d8b2b 100644\n>> >> --- a/commit-graph.c\n>> >> +++ b/commit-graph.c\n>> >> @@ -693,7 +693,8 @@ static void add_missing_parents(struct packed_oid_list *oids, struct commit *com\n>> >> \t}\n>> >> }\n>> >>\n>> >> -static void close_reachable(struct packed_oid_list *oids, int report_progress)\n>> >> +static void close_reachable(struct packed_oid_list *oids, int report_progress,\n>> >> +\t\t\t uint64_t oids_count_for_progress)\n>> >> {\n>> >> \tint i;\n>> >> \tstruct commit *commit;\n>> >> @@ -717,7 +718,8 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n>> >> \t */\n>> >> \tif (report_progress)\n>> >> \t\tprogress = start_delayed_progress(\n>> >> -\t\t\t_(\"Expanding reachable commits in commit graph\"), oids->nr);\n>> >> +\t\t\t_(\"Expanding reachable commits in commit graph\"),\n>> >> +\t\t\toids_count_for_progress);\n>> >> \tfor (i = 0; i < oids->nr; i++) {\n>> >> \t\tdisplay_progress(progress, i + 1);\n>> >> \t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n>> >> @@ -725,6 +727,8 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)\n>> >> \t\tif (commit && !parse_commit(commit))\n>> >> \t\t\tadd_missing_parents(oids, commit);\n>> >> \t}\n>> >> +\tif (oids->nr < oids_count_for_progress)\n>> >> +\t\tdisplay_progress(progress, oids_count_for_progress);\n>> >> \tstop_progress(&progress);\n>> >>\n>> >> \tif (report_progress)\n>> >> @@ -829,6 +833,7 @@ void write_commit_graph(const char *obj_dir,\n>> >> \tuint64_t progress_cnt = 0;\n>> >> \tstruct strbuf progress_title = STRBUF_INIT;\n>> >> \tunsigned long approx_nr_objects;\n>> >> +\tuint64_t oids_count_for_progress = 0;\n>> >>\n>> >> \tif (!commit_graph_compatible(the_repository))\n>> >> \t\treturn;\n>> >> @@ -934,9 +939,10 @@ void write_commit_graph(const char *obj_dir,\n>> >> \t\tif (oids.progress_done < approx_nr_objects)\n>> >> \t\t\tdisplay_progress(oids.progress, approx_nr_objects);\n>> >> \t\tstop_progress(&oids.progress);\n>> >> +\t\toids_count_for_progress = oids.nr;\n>> >> \t}\n>> >>\n>> >> -\tclose_reachable(&oids, report_progress);\n>> >> +\tclose_reachable(&oids, report_progress, oids_count_for_progress);\n>> >>\n>> >> \tif (report_progress)\n>> >> \t\tprogress = start_delayed_progress(\n>> >>\n"}]}