{"thread":{"id":"55972","subject":"[PATCH 0/7] progress: verify progress counters in the test suite","startedAt":"2021-06-20T20:03:15Z","lastAt":"2022-02-03T23:39:50Z","messageCount":197,"participants":["SZEDER Gábor","Ævar Arnfjörð Bjarmason","René Scharfe","Taylor Blau","Randall S. Becker","Andrei Rybak","Felipe Contreras","Junio C Hamano","Phillip Wood","Emily Shaffer","Johannes Sixt","Carlo Arenas"],"isPatch":true,"patchVersion":1,"patchTotal":7},"messages":[{"id":"427995","messageId":"20210620200303.2328957-1-szeder.dev@gmail.com","threadId":"55972","inReplyTo":null,"subject":"[PATCH 0/7] progress: verify progress counters in the test suite","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-06-20T20:02:56Z","receivedAt":"2021-06-20T20:03:15Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Splitting off from:\n\n  https://public-inbox.org/git/cover-0.2-0000000000-20210607T144206Z-avarab@gmail.com/T/#me5d3176914d4268fd9f2a96fc63f4e41beb26bd6\n\nOn Tue, Jun 08, 2021 at 06:14:42PM +0200, René Scharfe wrote:\n> I wonder (only in a semi-curious way, though) if we can detect\n> off-by-one errors by adding an assertion to display_progress() that\n> requires the first update to have the value 0, and in stop_progress()\n> one that requires the previous display_progress() call to have a value\n> equal to the total number of work items.  Not sure it'd be worth the\n> hassle..\n\nI fixed and reported a number of bogus progress lines in the past, the\nlast one during v2.31.0-rc phase, so I've looked into whether progress\ncounters could be automatically validated in our tests, and came up\nwith these patches a few months ago.  It turned out that progress\ncounters can be checked easily and transparently in case of progress\nlines that are shown in the tests, i.e. that are shown even when\nstderr is not a terminal or are forced with '--progress'.  (In other\ncases it's still fairly easy but not quite transparent, as I think we\nneed changes to the progress API; more on that later in a separate\nseries.)\n\nThese checks did uncover a couple of buggy progress lines which are\nfixed in this series as well, but I'm not sure that the fix presented\nin patch 6 is the right approach, hence the RFC.\n\n\nSZEDER Gábor (7):\n  progress: introduce GIT_TEST_CHECK_PROGRESS to verify progress\n    counters\n  progress: catch nested/overlapping progresses with\n    GIT_TEST_CHECK_PROGRESS\n  progress: catch backwards counting with GIT_TEST_CHECK_PROGRESS\n  commit-graph: fix bogus counter in \"Scanning merged commits\" progress\n    line\n  entry: show finer-grained counter in \"Filtering content\" progress line\n  [RFC] entry: don't show \"Filtering content: ... done.\" line in case of\n    errors\n  test-lib: enable GIT_TEST_CHECK_PROGRESS by default\n\n commit-graph.c              |  2 +-\n entry.c                     | 10 +++---\n progress.c                  | 29 ++++++++++++++--\n t/t0500-progress-display.sh | 69 +++++++++++++++++++++++++------------\n t/test-lib.sh               |  6 ++++\n 5 files changed, 86 insertions(+), 30 deletions(-)\n\n-- \n2.32.0.289.g44fbea0957\n\n"},{"id":"427996","messageId":"20210620200303.2328957-2-szeder.dev@gmail.com","threadId":"55972","inReplyTo":"20210620200303.2328957-1-szeder.dev@gmail.com","subject":"[PATCH 1/7] progress: introduce GIT_TEST_CHECK_PROGRESS to verify progress counters","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-06-20T20:02:57Z","receivedAt":"2021-06-20T20:03:23Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"We had to fix a couple of buggy progress lines in the past, where the\nprogress counter's final value didn't match the expected total [1],\ne.g.:\n\n  Expanding reachable commits in commit graph: 138606% (824706/595), done.\n  Writing out commit graph in 3 passes: 166% (4187845/2512707), done.\n\nLet's do better, and, instead of waiting for someone to notice such\nissues by mere chance, start verifying progress counters in the test\nsuite: introduce the GIT_TEST_CHECK_PROGRESS knob to automatically\ncheck that the final value of each progress counter matches the\nexpected total upon calling stop_progress(), and trigger a BUG() if it\ndoesn't.\n\nThis check should cover progress lines that are too fast to be shown,\nbecause the repositories used in our tests are tiny and most of our\nprogress lines are delayed.  However, in case of a delayed progress\nline the variable holding the value of the progress counter\n('progress->last_value') is only updated after that delay is up, and,\nconsequently, we can't compare the progress counter with the expected\ntotal in stop_progress() in these cases.\n\nSo let's update 'progress->last_value' already during the initial\ndelay as well.  This doesn't affect the visible behavior of progress\nlines, though it results in additional invocations of the internal\ndisplay() function during the initial delay, but those don't make any\ndifference, because display() returns early without displaying\nanything until the delay is up anyway.\n\nNote that this can only check progress lines that are actually\nstarted, i.e. that are shown by default even when standard error is\nnot a terminal, or that are forced to show with the '--progress'\noption of whichever Git command displaying them.\n\nNonetheless, running the test suite with this new knob enabled results\nin failures in 't0021-conversion.sh' and 't5510-fetch.sh', revealing\ntwo more progress lines whose counter doesn't reach the expected\ntotal.  These will be fixed in later patches in this series, and after\nthat GIT_TEST_CHECK_PROGRESS will be enabled by default in the test\nsuite.\n\n[1] c4ff24bbb3 (commit-graph.c: display correct number of chunks when\n                writing, 2021-02-24)\n    1cbdbf3bef (commit-graph: drop count_distinct_commits() function,\n                2020-12-07), though this didn't actually fixed, but\n                instead removed a buggy progress line.\n    150cd3b61d (commit-graph: fix \"Writing out commit graph\" progress\n                counter, 2020-07-09)\n    67fa6aac5a (commit-graph: don't show progress percentages while\n                expanding reachable commits, 2019-09-07)\n    531e6daa03 (prune-packed: advanced progress even for non-existing\n                fan-out directories, 2009-04-27)\n\nSigned-off-by: SZEDER Gábor <szeder.dev@gmail.com>\n---\n progress.c                  | 16 ++++++++++++++--\n t/t0500-progress-display.sh | 34 ++++++++++++++++++++++++++++++++++\n 2 files changed, 48 insertions(+), 2 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 680c6a8bf9..255995406f 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -47,6 +47,8 @@ struct progress {\n \n static volatile sig_atomic_t progress_update;\n \n+static int test_check_progress;\n+\n /*\n  * These are only intended for testing the progress output, i.e. exclusively\n  * for 'test-tool progress'.\n@@ -111,10 +113,11 @@ static void display(struct progress *progress, uint64_t n, const char *done)\n \tint show_update = 0;\n \tint last_count_len = counters_sb->len;\n \n+\tprogress->last_value = n;\n+\n \tif (progress->delay && (!progress_update || --progress->delay))\n \t\treturn;\n \n-\tprogress->last_value = n;\n \ttp = (progress->throughput) ? progress->throughput->display.buf : \"\";\n \tif (progress->total) {\n \t\tunsigned percent = n * 100 / progress->total;\n@@ -252,7 +255,11 @@ void display_progress(struct progress *progress, uint64_t n)\n static struct progress *start_progress_delay(const char *title, uint64_t total,\n \t\t\t\t\t     unsigned delay, unsigned sparse)\n {\n-\tstruct progress *progress = xmalloc(sizeof(*progress));\n+\tstruct progress *progress;\n+\n+\ttest_check_progress = git_env_bool(\"GIT_TEST_CHECK_PROGRESS\", 0);\n+\n+\tprogress = xmalloc(sizeof(*progress));\n \tprogress->title = title;\n \tprogress->total = total;\n \tprogress->last_value = -1;\n@@ -349,6 +356,11 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)\n \tprogress = *p_progress;\n \tif (!progress)\n \t\treturn;\n+\tif (test_check_progress && progress->total &&\n+\t    progress->total != progress->last_value)\n+\t\tBUG(\"total progress does not match for \\\"%s\\\": expected: %\"PRIuMAX\" got: %\"PRIuMAX,\n+\t\t    progress->title, (uintmax_t)progress->total,\n+\t\t    (uintmax_t)progress->last_value);\n \t*p_progress = NULL;\n \tif (progress->last_value != -1) {\n \t\t/* Force the last update */\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 22058b503a..641fa0964e 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -308,4 +308,38 @@ test_expect_success 'progress generates traces' '\n \tgrep \"\\\"key\\\":\\\"total_bytes\\\",\\\"value\\\":\\\"409600\\\"\" trace.event\n '\n \n+test_expect_success 'GIT_TEST_CHECK_PROGRESS catches non-matching total' '\n+\tcat >in <<-\\EOF &&\n+\tprogress 1\n+\tprogress 2\n+\tprogress 4\n+\tEOF\n+\n+\ttest_must_fail env GIT_TEST_CHECK_PROGRESS=1 \\\n+\t\ttest-tool progress --total=3 \"Not enough\" <in 2>stderr &&\n+\tgrep \"BUG:.*total progress does not match\" stderr &&\n+\n+\ttest_must_fail env GIT_TEST_CHECK_PROGRESS=1 \\\n+\t\ttest-tool progress --total=5 \"Too much\" <in 2>stderr &&\n+\tgrep \"BUG:.*total progress does not match\" stderr\n+'\n+\n+test_expect_success 'tolerate bogus progress without GIT_TEST_CHECK_PROGRESS' '\n+\tcat >expect <<-\\EOF &&\n+\tWorking hard:  33% (1/3)<CR>\n+\tWorking hard:  33% (1/3), done.\n+\tEOF\n+\n+\tcat >in <<-\\EOF &&\n+\tprogress 1\n+\tEOF\n+\t(\n+\t\tsane_unset GIT_TEST_CHECK_PROGRESS &&\n+\t\ttest-tool progress --total=3 \"Working hard\" <in 2>stderr\n+\t) &&\n+\n+\tshow_cr <stderr >out &&\n+\ttest_cmp expect out\n+'\n+\n test_done\n-- \n2.32.0.289.g44fbea0957\n\n"},{"id":"427997","messageId":"20210620200303.2328957-3-szeder.dev@gmail.com","threadId":"55972","inReplyTo":"20210620200303.2328957-1-szeder.dev@gmail.com","subject":"[PATCH 2/7] progress: catch nested/overlapping progresses with GIT_TEST_CHECK_PROGRESS","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-06-20T20:02:58Z","receivedAt":"2021-06-20T20:03:28Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"We had to fix two buggy progress lines in the past, where\nstop_progress calls were added at the wrong place [1], resulting in\n\"done\" progress lines appearing in the wrong order.\n\nExtend GIT_TEST_CHECK_PROGRESS to catch these cases as well, i.e.\ntrigger a BUG() when a progress has already been running when\nstart_progress() or one of its variants is called to start a new one.\n\nRunning the test suite with GIT_TEST_CHECK_PROGRESS enabled doesn't\nreveal any new issues [2].\n\nNote that this will trigger even in cases where the output is not\nvisibly wrong, e.g. consider this simplified sequence of calls:\n\n  progress1 = start_delayed_progress();\n  progress2 = start_delayed_progress();\n  for (i = 0; ...)\n      display_progress(progress2, i + 1);\n  stop_progres(&progress2);\n  for (j = 0; ...)\n      display_progress(progress1, j + 1);\n  stop_progres(&progress1);\n\nThis doesn't produce bogus output like what is shown in those two\nfixes [1], because 'progress2' is already \"done\" before the first\ndisplay_progress(progress1, ...) call.  Btw, this is not just a\npathological example, we do have two progress lines arranged like\nthis, but they are only shown when standard error is a terminal, and\nthus aren't caught by GIT_TEST_CHECK_PROGRESS in its current form.\n\n[1] 6f9d5f2fda (commit-graph: fix progress of reachable commits,\n                2020-07-09)\n    862aead24e (commit-graph: fix \"Collecting commits from input\"\n                progress line, 2020-07-10)\n\n[2] This patch series applies with a minor conflict on top of\n    6f9d5f2fda^, and makes 37 tests fail because of that bug.\n\nSigned-off-by: SZEDER Gábor <szeder.dev@gmail.com>\n---\n progress.c | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/progress.c b/progress.c\nindex 255995406f..549e8d1fe7 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -48,6 +48,8 @@ struct progress {\n static volatile sig_atomic_t progress_update;\n \n static int test_check_progress;\n+/* Used to catch nested/overlapping progresses with GIT_TEST_CHECK_PROGRESS. */\n+static struct progress *current_progress = NULL;\n \n /*\n  * These are only intended for testing the progress output, i.e. exclusively\n@@ -258,8 +260,12 @@ static struct progress *start_progress_delay(const char *title, uint64_t total,\n \tstruct progress *progress;\n \n \ttest_check_progress = git_env_bool(\"GIT_TEST_CHECK_PROGRESS\", 0);\n+\tif (test_check_progress && current_progress)\n+\t\tBUG(\"progress \\\"%s\\\" is still active when starting new progress \\\"%s\\\"\",\n+\t\t    current_progress->title, title);\n \n \tprogress = xmalloc(sizeof(*progress));\n+\tcurrent_progress = progress;\n \tprogress->title = title;\n \tprogress->total = total;\n \tprogress->last_value = -1;\n@@ -383,6 +389,7 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)\n \tstrbuf_release(&progress->counters_sb);\n \tif (progress->throughput)\n \t\tstrbuf_release(&progress->throughput->display);\n+\tcurrent_progress = NULL;\n \tfree(progress->throughput);\n \tfree(progress);\n }\n-- \n2.32.0.289.g44fbea0957\n\n"},{"id":"427998","messageId":"20210620200303.2328957-4-szeder.dev@gmail.com","threadId":"55972","inReplyTo":"20210620200303.2328957-1-szeder.dev@gmail.com","subject":"[PATCH 3/7] progress: catch backwards counting with GIT_TEST_CHECK_PROGRESS","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-06-20T20:02:59Z","receivedAt":"2021-06-20T20:03:31Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"We had to fix a buggy progress line recently, where the progress\ncounter counted backwards, see 8e118e8490 (pack-objects: update\n\"nr_seen\" progress based on pack-reused count, 2021-04-11).\n\nExtend GIT_TEST_CHECK_PROGRESS to catch these cases as well, i.e.\ntrigger a BUG() when the counter passed to display_progress() is\nsmaller than the previous value.\n\nNote that we allow subsequent display_progress() calls with the same\ncounter value, because:\n\n  - Strictly speaking, it's not wrong to do so.\n\n  - Forbidding it might make the code calling display_progress() more\n    complex; I suspect that would be the case with e.g. the \"Updating\n    index flags\" progress line in 'unpack-trees.c', where the counter\n    is increased in recursive function calls.\n\n  - We would need to special case the internal display() call in\n    stop_progress_msg(), because it uses the same counter value as the\n    last display_progress() call, which would trigger this BUG().\n\n't0500-progress-display.sh' countains a few tests that check how\nshortened progress lines are covered up, and one of them ('progress\nshortens - crazy caller') shortens the progress line by counting\nbackwards.  From now on that test would trigger this BUG(), so remove\nit; the other test cases cover shortening progress lines sufficiently.\n\nSigned-off-by: SZEDER Gábor <szeder.dev@gmail.com>\n---\n progress.c                  |  6 ++++++\n t/t0500-progress-display.sh | 35 +++++++++++++----------------------\n 2 files changed, 19 insertions(+), 22 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 549e8d1fe7..034d50cd6b 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -115,6 +115,12 @@ static void display(struct progress *progress, uint64_t n, const char *done)\n \tint show_update = 0;\n \tint last_count_len = counters_sb->len;\n \n+\tif (test_check_progress && progress->last_value != -1 &&\n+\t    n < progress->last_value)\n+\t\tBUG(\"progress \\\"%s\\\" counts backwards %\"PRIuMAX\" -> %\"PRIuMAX,\n+\t\t    progress->title, (uintmax_t)progress->last_value,\n+\t\t    (uintmax_t)n);\n+\n \tprogress->last_value = n;\n \n \tif (progress->delay && (!progress_update || --progress->delay))\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 641fa0964e..a73dd45153 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -153,28 +153,6 @@ EOF\n \ttest_cmp expect out\n '\n \n-# Progress counter goes backwards, this should not happen in practice.\n-test_expect_success 'progress shortens - crazy caller' '\n-\tcat >expect <<-\\EOF &&\n-\tWorking hard:  10% (100/1000)<CR>\n-\tWorking hard:  20% (200/1000)<CR>\n-\tWorking hard:   0% (1/1000)  <CR>\n-\tWorking hard: 100% (1000/1000)<CR>\n-\tWorking hard: 100% (1000/1000), done.\n-\tEOF\n-\n-\tcat >in <<-\\EOF &&\n-\tprogress 100\n-\tprogress 200\n-\tprogress 1\n-\tprogress 1000\n-\tEOF\n-\ttest-tool progress --total=1000 \"Working hard\" <in 2>stderr &&\n-\n-\tshow_cr <stderr >out &&\n-\ttest_cmp expect out\n-'\n-\n test_expect_success 'progress display with throughput' '\n \tcat >expect <<-\\EOF &&\n \tWorking hard: 10<CR>\n@@ -324,13 +302,26 @@ test_expect_success 'GIT_TEST_CHECK_PROGRESS catches non-matching total' '\n \tgrep \"BUG:.*total progress does not match\" stderr\n '\n \n+test_expect_success 'GIT_TEST_CHECK_PROGRESS catches backwards counting' '\n+\tcat >in <<-\\EOF &&\n+\tprogress 2\n+\tprogress 1\n+\tEOF\n+\n+\ttest_must_fail env GIT_TEST_CHECK_PROGRESS=1 \\\n+\t\ttest-tool progress --total=3 \"Working hard\" <in 2>stderr &&\n+\tgrep \"BUG:.*counts backwards\" stderr\n+'\n+\n test_expect_success 'tolerate bogus progress without GIT_TEST_CHECK_PROGRESS' '\n \tcat >expect <<-\\EOF &&\n+\tWorking hard:  66% (2/3)<CR>\n \tWorking hard:  33% (1/3)<CR>\n \tWorking hard:  33% (1/3), done.\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tprogress 2\n \tprogress 1\n \tEOF\n \t(\n-- \n2.32.0.289.g44fbea0957\n\n"},{"id":"427999","messageId":"20210620200303.2328957-5-szeder.dev@gmail.com","threadId":"55972","inReplyTo":"20210620200303.2328957-1-szeder.dev@gmail.com","subject":"[PATCH 4/7] commit-graph: fix bogus counter in \"Scanning merged commits\" progress line","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-06-20T20:03:00Z","receivedAt":"2021-06-20T20:03:32Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"The final value of the counter of the \"Scanning merged commits\"\nprogress line is always one less than its expected total, e.g.:\n\n  Scanning merged commits:  83% (5/6), done.\n\nThis happens because while iterating over an array the loop variable\nis passed to display_progress() as-is, but while C arrays (and thus\nthe loop variable) start at 0 and end at N-1, the progress counter\nmust end at N.  This causes the failures of the tests\n'fetch.writeCommitGraph' and 'fetch.writeCommitGraph with submodules'\nin 't5510-fetch.sh' when run with GIT_TEST_CHECK_PROGRESS=1.\n\nFix this by passing 'i + 1' to display_progress(), like most other\ncallsites do.\n\nSigned-off-by: SZEDER Gábor <szeder.dev@gmail.com>\n---\n commit-graph.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 2bcb4e0f89..3181906368 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -2096,7 +2096,7 @@ static void sort_and_scan_merged_commits(struct write_commit_graph_context *ctx)\n \n \tctx->num_extra_edges = 0;\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n-\t\tdisplay_progress(ctx->progress, i);\n+\t\tdisplay_progress(ctx->progress, i + 1);\n \n \t\tif (i && oideq(&ctx->commits.list[i - 1]->object.oid,\n \t\t\t  &ctx->commits.list[i]->object.oid)) {\n-- \n2.32.0.289.g44fbea0957\n\n"},{"id":"428000","messageId":"20210620200303.2328957-6-szeder.dev@gmail.com","threadId":"55972","inReplyTo":"20210620200303.2328957-1-szeder.dev@gmail.com","subject":"[PATCH 5/7] entry: show finer-grained counter in \"Filtering content\" progress line","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-06-20T20:03:01Z","receivedAt":"2021-06-20T20:03:32Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"The \"Filtering content\" progress in entry.c:finish_delayed_checkout()\nis unusual because of how it calculates the progress count and because\nit shows the progress of a nested loop.  It works basically like this:\n\n  start_delayed_progress(p, nr_of_paths_to_filter)\n  for_each_filter {\n      display_progress(p, nr_of_paths_to_filter - nr_of_paths_still_left_to_filter)\n      for_each_path_handled_by_the_current_filter {\n          checkout_entry()\n      }\n  }\n  stop_progress(p)\n\nThere are two issues with this approach:\n\n  - The work done by the last filter (or the only filter if there is\n    only one) is never counted, so if the last filter still has some\n    paths to process, then the counter shown in the \"done\" progress\n    line will not match the expected total.\n\n    This is, in part, responsible for the failures of the tests\n    'missing file in delayed checkout' and 'invalid file in delayed\n    checkout' in 't0021-conversion.sh' when run with\n    GIT_TEST_CHECK_PROGRESS=1, because both use only one filter.  (The\n    test 'delayed checkout in process filter' uses two filters but the\n    first one does all the work, so that test already happens to\n    succeed even with GIT_TEST_CHECK_PROGRESS=1.)\n\n  - The progress counter is updated only once per filter, not once per\n    processed path, so if a filter has a lot of paths to process, then\n    the counter might stay unchanged for a long while and then make a\n    big jump (though the user still gets a sense of progress, because\n    we call display_throughput() after each processed path to show the\n    amount of processed data).\n\nMove the display_progress() call to the inner loop, right next to that\ncheckout_entry() call that does the hard work for each path, and use a\ndedicated counter variable that is incremented upon processing each\npath.\n\nAfter this change the 'invalid file in delayed checkout' in\n't0021-conversion.sh' will succeed with GIT_TEST_CHECK_PROGRESS=1, but\nthe 'missing file in delayed checkout' test will still fail, because\nits purposefully buggy filter doesn't process any paths, so we won't\nexecute that inner loop at all (this will be fixed in the next patch).\n\nSigned-off-by: SZEDER Gábor <szeder.dev@gmail.com>\n---\n entry.c | 7 +++----\n 1 file changed, 3 insertions(+), 4 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex 711ee0693c..bc4b8fcc98 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -162,7 +162,7 @@ static int remove_available_paths(struct string_list_item *item, void *cb_data)\n int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n {\n \tint errs = 0;\n-\tunsigned delayed_object_count;\n+\tunsigned processed_paths = 0;\n \toff_t filtered_bytes = 0;\n \tstruct string_list_item *filter, *path;\n \tstruct progress *progress;\n@@ -172,12 +172,10 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n \t\treturn errs;\n \n \tdco->state = CE_RETRY;\n-\tdelayed_object_count = dco->paths.nr;\n-\tprogress = start_delayed_progress(_(\"Filtering content\"), delayed_object_count);\n+\tprogress = start_delayed_progress(_(\"Filtering content\"), dco->paths.nr);\n \twhile (dco->filters.nr > 0) {\n \t\tfor_each_string_list_item(filter, &dco->filters) {\n \t\t\tstruct string_list available_paths = STRING_LIST_INIT_NODUP;\n-\t\t\tdisplay_progress(progress, delayed_object_count - dco->paths.nr);\n \n \t\t\tif (!async_query_available_blobs(filter->string, &available_paths)) {\n \t\t\t\t/* Filter reported an error */\n@@ -224,6 +222,7 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n \t\t\t\tce = index_file_exists(state->istate, path->string,\n \t\t\t\t\t\t       strlen(path->string), 0);\n \t\t\t\tif (ce) {\n+\t\t\t\t\tdisplay_progress(progress, ++processed_paths);\n \t\t\t\t\terrs |= checkout_entry(ce, state, NULL, nr_checkouts);\n \t\t\t\t\tfiltered_bytes += ce->ce_stat_data.sd_size;\n \t\t\t\t\tdisplay_throughput(progress, filtered_bytes);\n-- \n2.32.0.289.g44fbea0957\n\n"},{"id":"428001","messageId":"20210620200303.2328957-7-szeder.dev@gmail.com","threadId":"55972","inReplyTo":"20210620200303.2328957-1-szeder.dev@gmail.com","subject":"[PATCH 6/7] [RFC] entry: don't show \"Filtering content: ... done.\" line in case of errors","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-06-20T20:03:02Z","receivedAt":"2021-06-20T20:03:34Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"The test 'missing file in delayed checkout' in 't0021-conversion.sh'\nfails when run with GIT_TEST_CHECK_PROGRESS=1, because the final value\nof the \"Filtering content\" progress counter doesn't match the expected\ntotal, triggering BUG().  This is not caused by a bug in how we count\nprogress, but because the test involves a purposefully buggy filter\nprocess that doesn't process any paths, so the progress counter\ndoesn't have a chance to reach the expected total.\n\nArguably, it is wrong to show \"done\" at the end of the progress\nline when not all work was done.\n\nSo let's check whether there were any errors while processing or that\nthere are still unprocessed paths at the end (which a few lines later\nwill in fact be considered as error) and don't show the final \"done\"\nline, i.e. don't call stop_progress(), if there were any.  And if we\ndon't call stop_progress(), then we won't verify that the progress\ncounter matches the expected total, won't trigger BUG() on mismatch,\nand t0021 will succeed even with GIT_TEST_CHECK_PROGRESS=1.\n\nAfter this change the test suite passes with\nGIT_TEST_CHECK_PROGRESS=1.\n\nRFC!!  Alas, not calling stop_progress() on error has drawbacks:\n\n  - All memory allocated for the progress bar is leaked.\n  - This progress line remains \"active\", in the sense that if we were\n    to start a new progress later in the same git process, then with\n    GIT_TEST_CHECK_PROGRESS it would trigger the other BUG() catching\n    nested/overlapping progresses.\n\nDo we care?!  TBH I don't :)\nAnyway, if we do, then we might need some sort of an abort_progress()\nfunction...\n\nSigned-off-by: SZEDER Gábor <szeder.dev@gmail.com>\n---\n entry.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/entry.c b/entry.c\nindex bc4b8fcc98..38baefe22a 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -232,7 +232,8 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n \t\t}\n \t\tstring_list_remove_empty_items(&dco->filters, 0);\n \t}\n-\tstop_progress(&progress);\n+\tif (!errs && !dco->paths.nr)\n+\t\tstop_progress(&progress);\n \tstring_list_clear(&dco->filters, 0);\n \n \t/* At this point we should not have any delayed paths anymore. */\n-- \n2.32.0.289.g44fbea0957\n\n"},{"id":"428002","messageId":"20210620200303.2328957-8-szeder.dev@gmail.com","threadId":"55972","inReplyTo":"20210620200303.2328957-1-szeder.dev@gmail.com","subject":"[PATCH 7/7] test-lib: enable GIT_TEST_CHECK_PROGRESS by default","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-06-20T20:03:03Z","receivedAt":"2021-06-20T20:03:35Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Let's enable GIT_TEST_CHECK_PROGRESS by default, in the hope that it\nwill effectively prevent buggy progress line counters and nested\nprogress lines from entering our codebase in the future.\n\nSigned-off-by: SZEDER Gábor <szeder.dev@gmail.com>\n---\n t/test-lib.sh | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex adaf03543e..ae2dd6d0d2 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1502,6 +1502,12 @@ then\n \texport GIT_TEST_CHECK_CACHE_TREE\n fi\n \n+if test -z \"$GIT_TEST_CHECK_PROGRESS\"\n+then\n+\tGIT_TEST_CHECK_PROGRESS=true\n+\texport GIT_TEST_CHECK_PROGRESS\n+fi\n+\n test_lazy_prereq PIPE '\n \t# test whether the filesystem supports FIFOs\n \ttest_have_prereq !MINGW,!CYGWIN &&\n-- \n2.32.0.289.g44fbea0957\n\n"},{"id":"428006","messageId":"87a6nk41wy.fsf@evledraar.gmail.com","threadId":"55972","inReplyTo":"20210620200303.2328957-5-szeder.dev@gmail.com","subject":"Re: [PATCH 4/7] commit-graph: fix bogus counter in \"Scanning merged commits\" progress line","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-20T22:13:59Z","receivedAt":"2021-06-20T22:16:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Jun 20 2021, SZEDER Gábor wrote:\n\n> The final value of the counter of the \"Scanning merged commits\"\n> progress line is always one less than its expected total, e.g.:\n>\n>   Scanning merged commits:  83% (5/6), done.\n>\n> This happens because while iterating over an array the loop variable\n> is passed to display_progress() as-is, but while C arrays (and thus\n> the loop variable) start at 0 and end at N-1, the progress counter\n> must end at N.  This causes the failures of the tests\n> 'fetch.writeCommitGraph' and 'fetch.writeCommitGraph with submodules'\n> in 't5510-fetch.sh' when run with GIT_TEST_CHECK_PROGRESS=1.\n>\n> Fix this by passing 'i + 1' to display_progress(), like most other\n> callsites do.\n>\n> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>\n> ---\n>  commit-graph.c | 2 +-\n>  1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 2bcb4e0f89..3181906368 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -2096,7 +2096,7 @@ static void sort_and_scan_merged_commits(struct write_commit_graph_context *ctx)\n>  \n>  \tctx->num_extra_edges = 0;\n>  \tfor (i = 0; i < ctx->commits.nr; i++) {\n> -\t\tdisplay_progress(ctx->progress, i);\n> +\t\tdisplay_progress(ctx->progress, i + 1);\n>  \n>  \t\tif (i && oideq(&ctx->commits.list[i - 1]->object.oid,\n>  \t\t\t  &ctx->commits.list[i]->object.oid)) {\n\nI think this fix makes sense, but FWIW there's a large thread starting\nat [1] where René disagrees with me, and thinks the fix for this sort of\nthing would be to display_progress(..., i + 1) at the end of that\nfor-loop, or just before the stop_progress().\n\nI don't agree, but just noting the disagreement, and that if that\nargument wins then a patch like this would involve changing the other\n20-some calls to display_progress() in commit-graph.c to work\ndifferently (and to be more complex, we'd need to deal with loop\nbreak/continue etc.).\n\n1. https://lore.kernel.org/git/patch-2.2-042f598826-20210607T144206Z-avarab@gmail.com/ \n"},{"id":"428008","messageId":"874kds3tdn.fsf@evledraar.gmail.com","threadId":"55972","inReplyTo":"20210620200303.2328957-1-szeder.dev@gmail.com","subject":"Re: [PATCH 0/7] progress: verify progress counters in the test suite","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-21T00:59:53Z","receivedAt":"2021-06-21T01:21:13Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Jun 20 2021, SZEDER Gábor wrote:\n\n> Splitting off from:\n>\n>   https://public-inbox.org/git/cover-0.2-0000000000-20210607T144206Z-avarab@gmail.com/T/#me5d3176914d4268fd9f2a96fc63f4e41beb26bd6\n>\n> On Tue, Jun 08, 2021 at 06:14:42PM +0200, René Scharfe wrote:\n>> I wonder (only in a semi-curious way, though) if we can detect\n>> off-by-one errors by adding an assertion to display_progress() that\n>> requires the first update to have the value 0, and in stop_progress()\n>> one that requires the previous display_progress() call to have a value\n>> equal to the total number of work items.  Not sure it'd be worth the\n>> hassle..\n>\n> I fixed and reported a number of bogus progress lines in the past, the\n> last one during v2.31.0-rc phase, so I've looked into whether progress\n> counters could be automatically validated in our tests, and came up\n> with these patches a few months ago.  It turned out that progress\n> counters can be checked easily and transparently in case of progress\n> lines that are shown in the tests, i.e. that are shown even when\n> stderr is not a terminal or are forced with '--progress'.  (In other\n> cases it's still fairly easy but not quite transparent, as I think we\n> need changes to the progress API; more on that later in a separate\n> series.)\n\nI've also been working on some progress.[ch] patches that are mostly\nfinished, and I'm some 20 patches in at the moment. I wasn't sure about\nwhether to send an alternate 20-patch \"let's do this (mostly) instead?\"\nseries, hence this message.\n\nMuch of what you're doing here becomes easier after that series,\ne.g. your global process struct in 2/7 is something I ended up\nimplementing as part of a general feature to allow progress to be driven\nby either display_progress() *or* the signal handler itself.\n\nThus we can show a \"stalled\" message if we run start(), but hang before\nwe ever call display_progress(), as we do on e.g. git.git in gc's\n\"Enumerating Objects\" phase (at least on my laptop).\n\nSo e.g. your 2/7 becomes a general hard assertion, not some test-only\nmode.\n\nAfter that I use the same facility to implement a mode where any signal\ncan update a new \"spinner\" part of the progress bar. So let's say you're\nhanging on item 1/3 and not calling display_progress() at all, we'll\nupdate a spinner on each signal to show the user that git itself isn't\nhanging, just working.\n\nI could also rebase on yours, but much of it would be rewriting the\ntest-only code to be more generalized, perhaps it's easier if we start\ngoing for the more generalized solution first.\n\nPer some of what I mentioned in the thread you linked to I'm a bit\nuncomfortable with the direction in your 1/7. I seems it works in-tree\nfor now, but I'd like to take the progress.c API in the direction of a\nmore generally useful API, not just something that narrowly fits the\nexact set of current use-cases.\n\nThere's a lot of potential uses in-tree where the total not matching at\nthe end is just something that happens due to real-world fuzzyness,\ne.g. the unlink() example here:\nhttps://public-inbox.org/git/87lf7k2bem.fsf@evledraar.gmail.com/\n\nPerhaps we can just have it BUG() for now as you're doing and cross that\nbridge when we come to it. I just wonder if we can't catch potential\nbugs in a more gentle way somehow.\n\n> These checks did uncover a couple of buggy progress lines which are\n> fixed in this series as well, but I'm not sure that the fix presented\n> in patch 6 is the right approach, hence the RFC.\n\nThe approach in 6/7 will also have the effect of not balancing a trace2\nstart/stop region. Quoting a line from its commit message:\n\n    > Arguably, it is wrong to show \"done\" at the end of the progress\n    > line when not all work was done.\n\nI think for a more general API it makes sense to think of \"done\" as a\ndifferent state than \"we have reached == total\". The target may change\nas in the unlink() example, or we may simply decide to abort and \"be\ndone early\".\n"},{"id":"428021","messageId":"87y2b33d7y.fsf@evledraar.gmail.com","threadId":"55972","inReplyTo":"20210620200303.2328957-2-szeder.dev@gmail.com","subject":"Re: [PATCH 1/7] progress: introduce GIT_TEST_CHECK_PROGRESS to verify progress counters","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-21T07:09:39Z","receivedAt":"2021-06-21T07:10:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Jun 20 2021, SZEDER Gábor wrote:\n\n> @@ -252,7 +255,11 @@ void display_progress(struct progress *progress, uint64_t n)\n>  static struct progress *start_progress_delay(const char *title, uint64_t total,\n>  \t\t\t\t\t     unsigned delay, unsigned sparse)\n>  {\n> -\tstruct progress *progress = xmalloc(sizeof(*progress));\n> +\tstruct progress *progress;\n> +\n> +\ttest_check_progress = git_env_bool(\"GIT_TEST_CHECK_PROGRESS\", 0);\n> +\n> +\tprogress = xmalloc(sizeof(*progress));\n\nIs this simply an unrelated cleanup/refactoring? I don't see how this\nre-arrangement is needed for adding the git_env_bool() call.\n"},{"id":"428083","messageId":"f580c5c6-304c-24f4-535c-35025b6b4a80@web.de","threadId":"55972","inReplyTo":"87a6nk41wy.fsf@evledraar.gmail.com","subject":"Re: [PATCH 4/7] commit-graph: fix bogus counter in \"Scanning merged commits\" progress line","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-06-21T18:32:53Z","receivedAt":"2021-06-21T18:32:57Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 21.06.21 um 00:13 schrieb Ævar Arnfjörð Bjarmason:\n>\n> On Sun, Jun 20 2021, SZEDER Gábor wrote:\n>\n>> The final value of the counter of the \"Scanning merged commits\"\n>> progress line is always one less than its expected total, e.g.:\n>>\n>>   Scanning merged commits:  83% (5/6), done.\n>>\n>> This happens because while iterating over an array the loop variable\n>> is passed to display_progress() as-is, but while C arrays (and thus\n>> the loop variable) start at 0 and end at N-1, the progress counter\n>> must end at N.  This causes the failures of the tests\n>> 'fetch.writeCommitGraph' and 'fetch.writeCommitGraph with submodules'\n>> in 't5510-fetch.sh' when run with GIT_TEST_CHECK_PROGRESS=1.\n>>\n>> Fix this by passing 'i + 1' to display_progress(), like most other\n>> callsites do.\n>>\n>> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>\n>> ---\n>>  commit-graph.c | 2 +-\n>>  1 file changed, 1 insertion(+), 1 deletion(-)\n>>\n>> diff --git a/commit-graph.c b/commit-graph.c\n>> index 2bcb4e0f89..3181906368 100644\n>> --- a/commit-graph.c\n>> +++ b/commit-graph.c\n>> @@ -2096,7 +2096,7 @@ static void sort_and_scan_merged_commits(struct write_commit_graph_context *ctx)\n>>\n>>  \tctx->num_extra_edges = 0;\n>>  \tfor (i = 0; i < ctx->commits.nr; i++) {\n>> -\t\tdisplay_progress(ctx->progress, i);\n>> +\t\tdisplay_progress(ctx->progress, i + 1);\n>>\n>>  \t\tif (i && oideq(&ctx->commits.list[i - 1]->object.oid,\n>>  \t\t\t  &ctx->commits.list[i]->object.oid)) {\n>\n> I think this fix makes sense, but FWIW there's a large thread starting\n> at [1] where René disagrees with me, and thinks the fix for this sort of\n> thing would be to display_progress(..., i + 1) at the end of that\n> for-loop, or just before the stop_progress().\n>\n> I don't agree, but just noting the disagreement, and that if that\n> argument wins then a patch like this would involve changing the other\n> 20-some calls to display_progress() in commit-graph.c to work\n> differently (and to be more complex, we'd need to deal with loop\n> break/continue etc.).\n>\n> 1. https://lore.kernel.org/git/patch-2.2-042f598826-20210607T144206Z-avarab@gmail.com/\n\n*sigh*  (And sorry, Ævar.)\n\nBefore an item is done, it should be reported as not done.  After an\nitem is done, it should be reported as done.  One loop iteration\nfinishes one item.  Thus the number of items to report at the bottom of\nthe loop is one higher than at the top.  i is the correct number to\nreport at the top of a zero-based loop, i+1 at the bottom.\n\nThere is another place: In the loop header.  It's a weird place for a\nfunction call, but it gets triggered before, between and after all\nitems, just as we need it:\n\n\tfor (i = 0; display_progress(ctx->progress), i < ctx->commits.nr; i++) {\n\nWe could hide this unseemly sight in a macro:\n\n  #define progress_foreach(index, count, progress) \\\n  for (index = 0; display_progress(progress, index), index < count; index++)\n\nHmm?\n\nRené\n"},{"id":"428084","messageId":"85ff4cb1-f3e3-4fcb-d9d8-fd294e0df451@web.de","threadId":"55972","inReplyTo":"20210620200303.2328957-7-szeder.dev@gmail.com","subject":"Re: [PATCH 6/7] [RFC] entry: don't show \"Filtering content: ... done.\" line in case of errors","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-06-21T18:32:56Z","receivedAt":"2021-06-21T18:33:00Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 20.06.21 um 22:03 schrieb SZEDER Gábor:\n> RFC!!  Alas, not calling stop_progress() on error has drawbacks:\n>\n>   - All memory allocated for the progress bar is leaked.\n>   - This progress line remains \"active\", in the sense that if we were\n>     to start a new progress later in the same git process, then with\n>     GIT_TEST_CHECK_PROGRESS it would trigger the other BUG() catching\n>     nested/overlapping progresses.\n>\n> Do we care?!  TBH I don't :)\n> Anyway, if we do, then we might need some sort of an abort_progress()\n> function...\n\nI think the abort_progress() idea makes sense; to clean up allocations,\ntell the user what happened and avoid the BUG().  Showing just\n\"aborted\" instead of \"done\" should suffice here -- the explanation is\ngiven a few lines later (\"'foo' was not filtered properly\").\n\nIt could be a cheesy stop_progress_msg() wrapper that temporarily sets\ntest_check_progress to zero..\n\nRené\n"},{"id":"428099","messageId":"87h7hr2cx0.fsf@evledraar.gmail.com","threadId":"55972","inReplyTo":"f580c5c6-304c-24f4-535c-35025b6b4a80@web.de","subject":"Re: [PATCH 4/7] commit-graph: fix bogus counter in \"Scanning merged commits\" progress line","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-21T20:08:50Z","receivedAt":"2021-06-21T20:14:23Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jun 21 2021, René Scharfe wrote:\n\n> Am 21.06.21 um 00:13 schrieb Ævar Arnfjörð Bjarmason:\n>>\n>> On Sun, Jun 20 2021, SZEDER Gábor wrote:\n>>\n>>> The final value of the counter of the \"Scanning merged commits\"\n>>> progress line is always one less than its expected total, e.g.:\n>>>\n>>>   Scanning merged commits:  83% (5/6), done.\n>>>\n>>> This happens because while iterating over an array the loop variable\n>>> is passed to display_progress() as-is, but while C arrays (and thus\n>>> the loop variable) start at 0 and end at N-1, the progress counter\n>>> must end at N.  This causes the failures of the tests\n>>> 'fetch.writeCommitGraph' and 'fetch.writeCommitGraph with submodules'\n>>> in 't5510-fetch.sh' when run with GIT_TEST_CHECK_PROGRESS=1.\n>>>\n>>> Fix this by passing 'i + 1' to display_progress(), like most other\n>>> callsites do.\n>>>\n>>> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>\n>>> ---\n>>>  commit-graph.c | 2 +-\n>>>  1 file changed, 1 insertion(+), 1 deletion(-)\n>>>\n>>> diff --git a/commit-graph.c b/commit-graph.c\n>>> index 2bcb4e0f89..3181906368 100644\n>>> --- a/commit-graph.c\n>>> +++ b/commit-graph.c\n>>> @@ -2096,7 +2096,7 @@ static void sort_and_scan_merged_commits(struct write_commit_graph_context *ctx)\n>>>\n>>>  \tctx->num_extra_edges = 0;\n>>>  \tfor (i = 0; i < ctx->commits.nr; i++) {\n>>> -\t\tdisplay_progress(ctx->progress, i);\n>>> +\t\tdisplay_progress(ctx->progress, i + 1);\n>>>\n>>>  \t\tif (i && oideq(&ctx->commits.list[i - 1]->object.oid,\n>>>  \t\t\t  &ctx->commits.list[i]->object.oid)) {\n>>\n>> I think this fix makes sense, but FWIW there's a large thread starting\n>> at [1] where René disagrees with me, and thinks the fix for this sort of\n>> thing would be to display_progress(..., i + 1) at the end of that\n>> for-loop, or just before the stop_progress().\n>>\n>> I don't agree, but just noting the disagreement, and that if that\n>> argument wins then a patch like this would involve changing the other\n>> 20-some calls to display_progress() in commit-graph.c to work\n>> differently (and to be more complex, we'd need to deal with loop\n>> break/continue etc.).\n>>\n>> 1. https://lore.kernel.org/git/patch-2.2-042f598826-20210607T144206Z-avarab@gmail.com/\n>\n> *sigh*  (And sorry, Ævar.)\n>\n> Before an item is done, it should be reported as not done.  After an\n> item is done, it should be reported as done.  One loop iteration\n> finishes one item.  Thus the number of items to report at the bottom of\n> the loop is one higher than at the top.  i is the correct number to\n> report at the top of a zero-based loop, i+1 at the bottom.\n>\n> There is another place: In the loop header.  It's a weird place for a\n> function call, but it gets triggered before, between and after all\n> items, just as we need it:\n>\n> \tfor (i = 0; display_progress(ctx->progress), i < ctx->commits.nr; i++) {\n>\n> We could hide this unseemly sight in a macro:\n>\n>   #define progress_foreach(index, count, progress) \\\n>   for (index = 0; display_progress(progress, index), index < count; index++)\n\nAnyone with more time than sense can go and read over our linked back &\nforth thread where we're disagreeing on that point :). I think the pattern\nin commit-graph.c makes sense, you don't.\n\nAnyway, aside from that. I think, and I really would be advocating this\ntoo, even if our respective positions were reversed, that *in this case*\nit makes sense to just take something like SZEDER's patch here\nas-is. Because in that file there's some dozen occurrences of that exact\npattern.\n\nLet's just bring this one case in line with the rest, if we then want to\nargue that one or the other use of the progress.c API is wrong as a\ngeneral thing, I think it makes more sense to discuss that as some\nfollow-up series that changes these various API uses en-masse than\nholding back isolated fixes that leave the state of the progress bar it\n!= 100%.\n"},{"id":"428204","messageId":"YNIH6FD9NKLc9zYl@nand.local","threadId":"55972","inReplyTo":"20210620200303.2328957-2-szeder.dev@gmail.com","subject":"Re: [PATCH 1/7] progress: introduce GIT_TEST_CHECK_PROGRESS to verify progress counters","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-22T15:55:20Z","receivedAt":"2021-06-22T15:55:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Jun 20, 2021 at 10:02:57PM +0200, SZEDER Gábor wrote:\n> +\tprogress->last_value = n;\n> +\n>  \tif (progress->delay && (!progress_update || --progress->delay))\n>  \t\treturn;\n>\n> -\tprogress->last_value = n;\n\nMakes sense, and thanks for explaining it explicitly in the patch\nmessage.\n\n>  \ttp = (progress->throughput) ? progress->throughput->display.buf : \"\";\n>  \tif (progress->total) {\n>  \t\tunsigned percent = n * 100 / progress->total;\n> @@ -252,7 +255,11 @@ void display_progress(struct progress *progress, uint64_t n)\n>  static struct progress *start_progress_delay(const char *title, uint64_t total,\n>  \t\t\t\t\t     unsigned delay, unsigned sparse)\n>  {\n> -\tstruct progress *progress = xmalloc(sizeof(*progress));\n> +\tstruct progress *progress;\n> +\n> +\ttest_check_progress = git_env_bool(\"GIT_TEST_CHECK_PROGRESS\", 0);\n> +\n> +\tprogress = xmalloc(sizeof(*progress));\n\nÆvar noted below, I think, but this cleanup to move the xmalloc() call\nto after reading $GIT_TEST_CHECK_PROGRESS is unnecessary.\n\n> +test_expect_success 'GIT_TEST_CHECK_PROGRESS catches non-matching total' '\n> +\tcat >in <<-\\EOF &&\n> +\tprogress 1\n> +\tprogress 2\n> +\tprogress 4\n> +\tEOF\n> +\n> +\ttest_must_fail env GIT_TEST_CHECK_PROGRESS=1 \\\n> +\t\ttest-tool progress --total=3 \"Not enough\" <in 2>stderr &&\n> +\tgrep \"BUG:.*total progress does not match\" stderr &&\n> +\n> +\ttest_must_fail env GIT_TEST_CHECK_PROGRESS=1 \\\n> +\t\ttest-tool progress --total=5 \"Too much\" <in 2>stderr &&\n> +\tgrep \"BUG:.*total progress does not match\" stderr\n> +'\n\nThis and the below test are both good to see. I wondered briefly whether\nor not it would be worth adding a test to check that the \"progress does\nnot match\" triggers even when we have a non-zero delay, like:\n\n    test_must_fail env GIT_PROGRESS_DELAY=100 GIT_TEST_CHECK_PROGRESS=1 \\\n      test-tool progress --total=5 \"Too much\" <in 2>stderr &&\n    grep \"BUG:.*total progress does not match\" stderr\n\nBut it's not helpful, because GIT_PROGRESS_DELAY is already 2 by\ndefault, and we unset GIT_* environment variables (including\nGIT_PROGRESS_DELAY) except a few which are left alone.\n\nSo we are already testing this case implicitly. It may be worth making\nit explicit, and/or testing the case where GIT_PROGRESS_DELAY=0, but I\ndo not feel strongly about it. Besides, I would much rather err on the\nside of testing cases we feel are legitimately interesting, rather than\nfilling in a grid of all possible combinations, including uninteresting\nones.\n\nThanks,\nTaylor\n"},{"id":"428205","messageId":"YNIJB8uYp7gzLC6v@nand.local","threadId":"55972","inReplyTo":"20210620200303.2328957-3-szeder.dev@gmail.com","subject":"Re: [PATCH 2/7] progress: catch nested/overlapping progresses with GIT_TEST_CHECK_PROGRESS","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-22T16:00:07Z","receivedAt":"2021-06-22T16:00:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Jun 20, 2021 at 10:02:58PM +0200, SZEDER Gábor wrote:\n> Note that this will trigger even in cases where the output is not\n> visibly wrong, e.g. consider this simplified sequence of calls:\n>\n>   progress1 = start_delayed_progress();\n>   progress2 = start_delayed_progress();\n>   for (i = 0; ...)\n>       display_progress(progress2, i + 1);\n>   stop_progres(&progress2);\n>   for (j = 0; ...)\n>       display_progress(progress1, j + 1);\n>   stop_progres(&progress1);\n\ns/stop_progres/&s, but no big deal. Everything else here looks good.\n\n> diff --git a/progress.c b/progress.c\n> index 255995406f..549e8d1fe7 100644\n> --- a/progress.c\n> +++ b/progress.c\n> @@ -48,6 +48,8 @@ struct progress {\n>  static volatile sig_atomic_t progress_update;\n>\n>  static int test_check_progress;\n> +/* Used to catch nested/overlapping progresses with GIT_TEST_CHECK_PROGRESS. */\n> +static struct progress *current_progress = NULL;\n>\n>  /*\n>   * These are only intended for testing the progress output, i.e. exclusively\n> @@ -258,8 +260,12 @@ static struct progress *start_progress_delay(const char *title, uint64_t total,\n>  \tstruct progress *progress;\n>\n>  \ttest_check_progress = git_env_bool(\"GIT_TEST_CHECK_PROGRESS\", 0);\n> +\tif (test_check_progress && current_progress)\n> +\t\tBUG(\"progress \\\"%s\\\" is still active when starting new progress \\\"%s\\\"\",\n> +\t\t    current_progress->title, title);\n>\n>  \tprogress = xmalloc(sizeof(*progress));\n\nAh. This is why you moved the allocation down further, since we don't\nhave to free anything up when calling BUG() if it wasn't allocated in\nthe first place (and we had no such conditional that would cause us to\nabort early before).\n\nFor what it's worth, I probably would have preferred to see that change\nfrom the previous patch included in this one rather than in the first of\nthe series, since it's much clearer here than it is in the first patch.\n\nThanks,\nTaylor\n"},{"id":"428244","messageId":"YNKT4Fr/LOGnx9/o@nand.local","threadId":"55972","inReplyTo":"85ff4cb1-f3e3-4fcb-d9d8-fd294e0df451@web.de","subject":"Re: [PATCH 6/7] [RFC] entry: don't show \"Filtering content: ... done.\" line in case of errors","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-23T01:52:32Z","receivedAt":"2021-06-23T01:52:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jun 21, 2021 at 08:32:56PM +0200, René Scharfe wrote:\n> Am 20.06.21 um 22:03 schrieb SZEDER Gábor:\n> > RFC!!  Alas, not calling stop_progress() on error has drawbacks:\n> >\n> >   - All memory allocated for the progress bar is leaked.\n> >   - This progress line remains \"active\", in the sense that if we were\n> >     to start a new progress later in the same git process, then with\n> >     GIT_TEST_CHECK_PROGRESS it would trigger the other BUG() catching\n> >     nested/overlapping progresses.\n> >\n> > Do we care?!  TBH I don't :)\n> > Anyway, if we do, then we might need some sort of an abort_progress()\n> > function...\n>\n> I think the abort_progress() idea makes sense; to clean up allocations,\n> tell the user what happened and avoid the BUG().  Showing just\n> \"aborted\" instead of \"done\" should suffice here -- the explanation is\n> given a few lines later (\"'foo' was not filtered properly\").\n\nVery well put. I concur that having an abort_progress() API makes sense\nfor all of the reasons that you suggest, but also because we shouldn't\nencourage not using what seems like an appropriate API in order to not\nfail tests when GIT_TEST_CHECK_PROGRESS is set.\n\nThanks,\nTaylor\n"},{"id":"428245","messageId":"YNKWsTsQgB2Ijxu7@nand.local","threadId":"55972","inReplyTo":"874kds3tdn.fsf@evledraar.gmail.com","subject":"Re: [PATCH 0/7] progress: verify progress counters in the test suite","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-23T02:04:33Z","receivedAt":"2021-06-23T02:04:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jun 21, 2021 at 02:59:53AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n> On Sun, Jun 20 2021, SZEDER Gábor wrote:\n>\n> > Splitting off from:\n> >\n> >   https://public-inbox.org/git/cover-0.2-0000000000-20210607T144206Z-avarab@gmail.com/T/#me5d3176914d4268fd9f2a96fc63f4e41beb26bd6\n> >\n> > On Tue, Jun 08, 2021 at 06:14:42PM +0200, René Scharfe wrote:\n> >> I wonder (only in a semi-curious way, though) if we can detect\n> >> off-by-one errors by adding an assertion to display_progress() that\n> >> requires the first update to have the value 0, and in stop_progress()\n> >> one that requires the previous display_progress() call to have a value\n> >> equal to the total number of work items.  Not sure it'd be worth the\n> >> hassle..\n> >\n> > I fixed and reported a number of bogus progress lines in the past, the\n> > last one during v2.31.0-rc phase, so I've looked into whether progress\n> > counters could be automatically validated in our tests, and came up\n> > with these patches a few months ago.  It turned out that progress\n> > counters can be checked easily and transparently in case of progress\n> > lines that are shown in the tests, i.e. that are shown even when\n> > stderr is not a terminal or are forced with '--progress'.  (In other\n> > cases it's still fairly easy but not quite transparent, as I think we\n> > need changes to the progress API; more on that later in a separate\n> > series.)\n>\n> I've also been working on some progress.[ch] patches that are mostly\n> finished, and I'm some 20 patches in at the moment. I wasn't sure about\n> whether to send an alternate 20-patch \"let's do this (mostly) instead?\"\n> series, hence this message.\n>\n> Much of what you're doing here becomes easier after that series,\n> e.g. your global process struct in 2/7 is something I ended up\n> implementing as part of a general feature to allow progress to be driven\n> by either display_progress() *or* the signal handler itself.\n\nIt's difficult to know who should rebase onto who without seeing one\nhalf of the patches. I couldn't find a link to them anywhere (even if\nthey are only available in your fork in a pre-polished state) despite\nlooking, but my apologies if they are available and I'm just missing\nthem.\n\nIn general, I think that these patches are clear and are helpful in\npinning down issues with the progress API (which I have made a hadnful\nof times in the past), so I would be happy to see them picked up.\n\n> I could also rebase on yours, but much of it would be rewriting the\n> test-only code to be more generalized, perhaps it's easier if we start\n> going for the more generalized solution first.\n\nAgain, without knowing the substance of your patches it's hard to\ncomment for sure, but I don't have a problem with a simple and direct\napproach here.\n\n> Perhaps we can just have it BUG() for now as you're doing and cross that\n> bridge when we come to it. I just wonder if we can't catch potential\n> bugs in a more gentle way somehow.\n\nI think there are compelling reasons to feel that the new mode should\nonly be enabled during tests, as well as compelling reasons to feel that\nit should be enabled all of the time.\n\nOne way to think about it is that we do not want users to have a BUG()\nabort their program just because a progress meter went rogue. So in that\nsense, it makes sense that we would only see that happen during tests,\nso that those tests could tell us where the bug is, and we could fix it.\n\nOn the other hand, since we make sure that our tests pass at each patch,\nthere's no point in having a separate mode (and instead, remove the\nconditionals on GIT_TEST_PROGRESS_CHECK), since successfully running the\ntests tells us that there are no rogue progress meters that we exercise\nin our (hopefully) complete set of tests.\n\nI could go either way, I think both lines of reasoning are quite\nreasonable. But, I think we are generally more lax about having the\nwhole ci/run-build-and-tests.sh script pass at every commit, and that it\nseems we care more about having the tip of each series pass CI when\nintegrated into 'seen'.\n\nSo I don't think that hiding this new mode behind an environment\nvariable is giving us as much confidence as we'd like, because it\ndoesn't add anything in \"make test\".\n\nTo me, I think a reasonable direction to take would be to *always*\nexport GIT_TEST_PROGRESS_CHECK when running tests, not just in\nci/run-build-and-tests.sh. That means we'll catch incorrect uses of the\nprogress API during tests, without worrying that incomplete coverage\nwill cause user-visible breakage.\n\nThanks,\nTaylor\n"},{"id":"428281","messageId":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"YNKWsTsQgB2Ijxu7@nand.local","subject":"[PATCH 00/25] progress.c: various fixes + SZEDER's RFC code","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:00Z","receivedAt":"2021-06-23T17:48:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"> On Mon, Jun 21, 2021 at 02:59:53AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>>\n>> On Sun, Jun 20 2021, SZEDER Gábor wrote:\n>>\n>> > Splitting off from:\n>> >\n>> >   https://public-inbox.org/git/cover-0.2-0000000000-20210607T144206Z-avarab@gmail.com/T/#me5d3176914d4268fd9f2a96fc63f4e41beb26bd6\n>> >\n>> > On Tue, Jun 08, 2021 at 06:14:42PM +0200, René Scharfe wrote:\n>> >> I wonder (only in a semi-curious way, though) if we can detect\n>> >> off-by-one errors by adding an assertion to display_progress() that\n>> >> requires the first update to have the value 0, and in stop_progress()\n>> >> one that requires the previous display_progress() call to have a value\n>> >> equal to the total number of work items.  Not sure it'd be worth the\n>> >> hassle..\n>> >\n>> > I fixed and reported a number of bogus progress lines in the past, the\n>> > last one during v2.31.0-rc phase, so I've looked into whether progress\n>> > counters could be automatically validated in our tests, and came up\n>> > with these patches a few months ago.  It turned out that progress\n>> > counters can be checked easily and transparently in case of progress\n>> > lines that are shown in the tests, i.e. that are shown even when\n>> > stderr is not a terminal or are forced with '--progress'.  (In other\n>> > cases it's still fairly easy but not quite transparent, as I think we\n>> > need changes to the progress API; more on that later in a separate\n>> > series.)\n>>\n>> I've also been working on some progress.[ch] patches that are mostly\n>> finished, and I'm some 20 patches in at the moment. I wasn't sure about\n>> whether to send an alternate 20-patch \"let's do this (mostly) instead?\"\n>> series, hence this message.\n>>\n>> Much of what you're doing here becomes easier after that series,\n>> e.g. your global process struct in 2/7 is something I ended up\n>> implementing as part of a general feature to allow progress to be driven\n>> by either display_progress() *or* the signal handler itself.\n>\n> It's difficult to know who should rebase onto who without seeing one\n> half of the patches.\n\nI was sort of hoping he'd take me word for it, but here it is. Don't\nsay I didn't warn you :)\n\n> I couldn't find a link to them anywhere (even if\n> they are only available in your fork in a pre-polished state) despite\n> looking, but my apologies if they are available and I'm just missing\n> them.\n\nFWIW it's avar-szeder/progress-bar-assertions in\nhttps://github.com/avar/git.git, that repo contains various\nfunctioning and not-so-functioning code.\n\nhttps://github.com/avar/git/tree/meta/ is my version of the crappy\nscripts we probably all have some version of for building my own git,\nthings that are uncommented in series.conf is what I build my own git\nfrom.\n\n> In general, I think that these patches are clear and are helpful in\n> pinning down issues with the progress API (which I have made a hadnful\n> of times in the past), so I would be happy to see them picked up.\n\nHere's all 25 patches (well, around 20 before) that I had queued up\nlocally and fixed up a bit.\n\nThe 01/25 is something I submitted already as\nhttps://lore.kernel.org/git/patch-1.1-cba5d88ca35-20210621T070114Z-avarab@gmail.com;\nhoping to get this in incrementally.\n\nThe 12/25 is my own version of that \"global progress struct, 11/25 is\nthe first of many bugs SZEDER missed in his :)\n\n18/25 is the first step of the UI I was going for, the signal handler\ncan now drive the progress bar, so e.g. during \"git gc\" we show (at\nleast for me, on git.git), a \"stalled\" message just before we start\nthe actual count of \"Enumerating Objects\".\n\nAfter that was in I was planning on adding config-driven support to\nshow a \"spinner\" when we stalled in that way, config-driven because\nyou could just scrape\ne.g. https://github.com/sindresorhus/cli-spinners/blob/main/spinners.json\ninto your own config. See\nhttps://jsfiddle.net/sindresorhus/2eLtsbey/embedded/result/ :)\n\n19-23/25 is my grabbing of SZEDER's patches that I'm comfortable\nlabeling as \"PATCH\", I think they work, but no BUG() assertions yet. I\nleft out the GIT_TEST_CHECK_PROGRESS parts, since my earlier works set\nthings up to do any BUG() we trust by default.\n\n22/25 is what I think we should do instead of SZEDER's 6/7\n(http://lore.kernel.org/git/20210620200303.2328957-7-szeder.dev@gmail.com)\nI don't think this \"our total doesn't match at the end\" is something\nwe should always BUG() on, for reasons explained there.\n\nI am sympathetic to doing it by default though, hence the\nstop_progress_early() API, that's there to allow select callers to\nbypass his BUG(...) assertion.\n\n24/25 and 25/25 are \"RFC\" and a rebased+modified version of SZEDER's\nBUG(...) assertions.\n\nHis series passes the test suite, but actually severely break things\nthings. It'll make e.g. \"git commit-graph write\" BUG(...) out. The\nreason the tests don't catch it is because we have a blind spot in the\ntests.\n\nNamely, that most things that use the progress bar API use isatty() to\ncheck if they should start_progress(). If you run the tests as\ne.g. (better ways to do this, especially in parallel, most welcome):\n\n    for t in t[0-9]*.sh; do if ! ./$t -vixd; then echo $t bad; break; fi; done\n\nYou can discover various things that his series BUG()'s on, I fixed a\ncouple of those myself, it's an early part of this series.\n\nBut we'll still have various untested for BUG()'s even then, this is\nbecause you *also* have to have the test actually emit a \"naked\"\nprogress bar on stderr, if the test itself e.g. pipes fd 2 to a file\nit won't work.\n\nI created a shitty-and-mostly-broken throwaway change to\nsearch-replace all the guards of \"start_progress(...)\" to run\nunconditionally, and convert all the \"delayed\" to the non-delayed\nversion. That'll find even more BUG()'s where SZEDER's series still\nneeds to be fixed (and also some unrelated segfaults, I gave up on it\nsoon after).\n\nEven if we fix that I wouldn't trust it, because a lot of the progress\nbars we have depend on the size and shape of the data we're\nprocessing, e.g. the bug I fixed in 11/25. If people find this BUG()\napproach worth pursuing I think it would be better to make it an\nopt-in flag we convert one caller at a time to.\n\nFor some it's really clear that we could assert it, for others such as\nthe commit-graph it's much more subtle, we're in some callback after\nsetting a \"total\", that callback does a \"break\", \"continue\" etc. in\nvarious places, all depending on repository data.\n\nIt's not easy to reason about that and be certain that we can hold to\nthe estimate. If we get it wrong someone's repo in the wild won't\nfully GC because of the overly eager BUG().\n\nIf SZEDER wants to pursue it I think it'll be easier on top of this\nseries, but personally I really don't see the point of spending effort\non it.\n\nWe should really be going in the other direction, of having more fuzzy\nETAs, not less.\n\nE.g. we often have enough data at the start of \"Enumerating Objects\"\nto give a good-enough target value, that it's 5-10% off isn't really\nthe point, but that the user looking at it sees something better than\na dumb count-up, and can instead see that they'll probably be looking\nat it for about a minute. Now our API is to give no ETA/target if\nwe're not 100% sure, it's not good UX.\n\nSo trying to get the current exact count/exact percentage right seems\nlike a distraction to me in the longer term. If anything we should\njust be rounding those numbers, showing fuzzy ETAs instead of\npercentages if we can etc.\n\nSZEDER Gábor (4):\n  commit-graph: fix bogus counter in \"Scanning merged commits\" progress\n    line\n  entry: show finer-grained counter in \"Filtering content\" progress line\n  progress: assert last update in stop_progress()\n  progress: assert counting upwards in display()\n\nÆvar Arnfjörð Bjarmason (21):\n  progress.c tests: fix breakage with COLUMNS != 80\n  progress.c tests: make start/stop verbs on stdin\n  progress.c tests: test some invalid usage\n  progress.c tests: add a \"signal\" verb\n  progress.c: move signal handler functions lower\n  progress.c: call progress_interval() from progress_test_force_update()\n  progress.c: stop eagerly fflush(stderr) when not a terminal\n  progress.c: add temporary variable from progress struct\n  midx perf: add a perf test for multi-pack-index\n  progress.c: remove the \"sparse\" mode nano-optimization\n  pack-bitmap-write.c: add a missing stop_progress()\n  progress.c: add & assert a \"global_progress\" variable\n  progress.[ch]: move the \"struct progress\" to the header\n  progress.[ch]: move test-only code away from \"extern\" variables\n  progress.c: pass \"is done?\" (again) to display()\n  progress.[ch]: convert \"title\" to \"struct strbuf\"\n  progress.c: refactor display() for less confusion, and fix bug\n  progress.c: emit progress on first signal, show \"stalled\"\n  midx: don't provide a total for QSORT() progress\n  progress.c: add a stop_progress_early() function\n  entry: deal with unexpected \"Filtering content\" total\n\n cache.h                          |   1 -\n commit-graph.c                   |   2 +-\n csum-file.h                      |   2 -\n entry.c                          |  12 +-\n midx.c                           |  25 +-\n pack-bitmap-write.c              |   1 +\n pack.h                           |   1 -\n parallel-checkout.h              |   1 -\n progress.c                       | 391 ++++++++++++++++++-------------\n progress.h                       |  50 +++-\n reachable.h                      |   1 -\n t/helper/test-progress.c         |  54 +++--\n t/perf/p5319-multi-pack-index.sh |  21 ++\n t/t0500-progress-display.sh      | 247 ++++++++++++++-----\n 14 files changed, 537 insertions(+), 272 deletions(-)\n create mode 100755 t/perf/p5319-multi-pack-index.sh\n\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428282","messageId":"patch-01.25-cba5d88ca35-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 01/25] progress.c tests: fix breakage with COLUMNS != 80","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:01Z","receivedAt":"2021-06-23T17:48:38Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The tests added in 2bb74b53a49 (Test the progress display, 2019-09-16)\nbroke under anything except COLUMNS=80, i.e. when running them under\nthe \"-v\" mode under a differently sized terminal.\n\nLet's set the expected number of COLUMNS at the start of the test to\nfix that bug. It's handy not do do this in test-progress.c itself, in\ncase we'd like to test for a different number of COLUMNS, either\nmanually or in a future test.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/t0500-progress-display.sh | 5 +++++\n 1 file changed, 5 insertions(+)\n\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 22058b503ac..66c092a0fe3 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -8,6 +8,11 @@ show_cr () {\n \ttr '\\015' Q | sed -e \"s/Q/<CR>\\\\$LF/g\"\n }\n \n+test_expect_success 'setup COLUMNS' '\n+\tCOLUMNS=80 &&\n+\texport COLUMNS\n+'\n+\n test_expect_success 'simple progress display' '\n \tcat >expect <<-\\EOF &&\n \tWorking hard: 1<CR>\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428283","messageId":"patch-05.25-fa6359da171-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 05/25] progress.c: move signal handler functions lower","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:05Z","receivedAt":"2021-06-23T17:48:40Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the signal handler functions to just before the\nstart_progress_delay() where they'll be referenced, instead of having\nthem at the top of the file.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 92 ++++++++++++++++++++++++++++--------------------------\n 1 file changed, 48 insertions(+), 44 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 680c6a8bf93..893cb0fe56f 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -53,50 +53,6 @@ static volatile sig_atomic_t progress_update;\n  */\n int progress_testing;\n uint64_t progress_test_ns = 0;\n-void progress_test_force_update(void)\n-{\n-\tprogress_update = 1;\n-}\n-\n-\n-static void progress_interval(int signum)\n-{\n-\tprogress_update = 1;\n-}\n-\n-static void set_progress_signal(void)\n-{\n-\tstruct sigaction sa;\n-\tstruct itimerval v;\n-\n-\tif (progress_testing)\n-\t\treturn;\n-\n-\tprogress_update = 0;\n-\n-\tmemset(&sa, 0, sizeof(sa));\n-\tsa.sa_handler = progress_interval;\n-\tsigemptyset(&sa.sa_mask);\n-\tsa.sa_flags = SA_RESTART;\n-\tsigaction(SIGALRM, &sa, NULL);\n-\n-\tv.it_interval.tv_sec = 1;\n-\tv.it_interval.tv_usec = 0;\n-\tv.it_value = v.it_interval;\n-\tsetitimer(ITIMER_REAL, &v, NULL);\n-}\n-\n-static void clear_progress_signal(void)\n-{\n-\tstruct itimerval v = {{0,},};\n-\n-\tif (progress_testing)\n-\t\treturn;\n-\n-\tsetitimer(ITIMER_REAL, &v, NULL);\n-\tsignal(SIGALRM, SIG_IGN);\n-\tprogress_update = 0;\n-}\n \n static int is_foreground_fd(int fd)\n {\n@@ -249,6 +205,54 @@ void display_progress(struct progress *progress, uint64_t n)\n \t\tdisplay(progress, n, NULL);\n }\n \n+static void progress_interval(int signum)\n+{\n+\tprogress_update = 1;\n+}\n+\n+/*\n+ * The progress_test_force_update() function is intended for testing\n+ * the progress output, i.e. exclusively for 'test-tool progress'.\n+ */\n+void progress_test_force_update(void)\n+{\n+\tprogress_update = 1;\n+}\n+\n+static void set_progress_signal(void)\n+{\n+\tstruct sigaction sa;\n+\tstruct itimerval v;\n+\n+\tif (progress_testing)\n+\t\treturn;\n+\n+\tprogress_update = 0;\n+\n+\tmemset(&sa, 0, sizeof(sa));\n+\tsa.sa_handler = progress_interval;\n+\tsigemptyset(&sa.sa_mask);\n+\tsa.sa_flags = SA_RESTART;\n+\tsigaction(SIGALRM, &sa, NULL);\n+\n+\tv.it_interval.tv_sec = 1;\n+\tv.it_interval.tv_usec = 0;\n+\tv.it_value = v.it_interval;\n+\tsetitimer(ITIMER_REAL, &v, NULL);\n+}\n+\n+static void clear_progress_signal(void)\n+{\n+\tstruct itimerval v = {{0,},};\n+\n+\tif (progress_testing)\n+\t\treturn;\n+\n+\tsetitimer(ITIMER_REAL, &v, NULL);\n+\tsignal(SIGALRM, SIG_IGN);\n+\tprogress_update = 0;\n+}\n+\n static struct progress *start_progress_delay(const char *title, uint64_t total,\n \t\t\t\t\t     unsigned delay, unsigned sparse)\n {\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428284","messageId":"patch-02.25-75e949e21d7-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 02/25] progress.c tests: make start/stop verbs on stdin","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:02Z","receivedAt":"2021-06-23T17:48:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the usage of the \"test-tool progress\" introduced in\n2bb74b53a49 (Test the progress display, 2019-09-16) to take command\nlike \"start\" and \"stop\" on stdin, instead of running them implicitly.\n\nThis makes for tests that are easier to read, since the recipe will\nmirror the API usage, and allows for easily testing invalid usage that\nwould yield (or should yield) a BUG(), e.g. providing two \"start\"\ncalls in a row. A subsequent commit will add such stress tests.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/helper/test-progress.c    | 45 ++++++++++++++++++++--------\n t/t0500-progress-display.sh | 59 +++++++++++++++++++++++--------------\n 2 files changed, 69 insertions(+), 35 deletions(-)\n\ndiff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\nindex 5d05cbe7894..eb925d591e1 100644\n--- a/t/helper/test-progress.c\n+++ b/t/helper/test-progress.c\n@@ -3,6 +3,9 @@\n  *\n  * Reads instructions from standard input, one instruction per line:\n  *\n+ *   \"start[ <total>[ <title>]]\" - Call start_progress(title, total),\n+ *                                 when \"start\" use a title of\n+ *                                 \"Working hard\" with a total of 0.\n  *   \"progress <items>\" - Call display_progress() with the given item count\n  *                        as parameter.\n  *   \"throughput <bytes> <millis> - Call display_throughput() with the given\n@@ -10,6 +13,7 @@\n  *                                  specify the time elapsed since the\n  *                                  start_progress() call.\n  *   \"update\" - Set the 'progress_update' flag.\n+ *   \"stop\" - Call stop_progress().\n  *\n  * See 't0500-progress-display.sh' for examples.\n  */\n@@ -22,31 +26,42 @@\n \n int cmd__progress(int argc, const char **argv)\n {\n-\tint total = 0;\n-\tconst char *title;\n+\tconst char *default_title = \"Working hard\";\n+\tchar *detached_title = NULL;\n \tstruct strbuf line = STRBUF_INIT;\n-\tstruct progress *progress;\n+\tstruct progress *progress = NULL;\n \n \tconst char *usage[] = {\n-\t\t\"test-tool progress [--total=<n>] <progress-title>\",\n+\t\t\"test-tool progress <stdin\",\n \t\tNULL\n \t};\n \tstruct option options[] = {\n-\t\tOPT_INTEGER(0, \"total\", &total, \"total number of items\"),\n \t\tOPT_END(),\n \t};\n \n \targc = parse_options(argc, argv, NULL, options, usage, 0);\n-\tif (argc != 1)\n-\t\tdie(\"need a title for the progress output\");\n-\ttitle = argv[0];\n+\tif (argc)\n+\t\tusage_with_options(usage, options);\n \n \tprogress_testing = 1;\n-\tprogress = start_progress(title, total);\n \twhile (strbuf_getline(&line, stdin) != EOF) {\n \t\tchar *end;\n \n-\t\tif (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n+\t\tif (!strcmp(line.buf, \"start\")) {\n+\t\t\tprogress = start_progress(default_title, 0);\n+\t\t} else if (skip_prefix(line.buf, \"start \", (const char **) &end)) {\n+\t\t\tuint64_t total = strtoull(end, &end, 10);\n+\t\t\tif (*end == '\\0') {\n+\t\t\t\tprogress = start_progress(default_title, total);\n+\t\t\t} else if (*end == ' ') {\n+\t\t\t\tif (detached_title)\n+\t\t\t\t\tfree(detached_title);\n+\t\t\t\tdetached_title = strbuf_detach(&line, NULL);\n+\t\t\t\tprogress = start_progress(end + 1, total);\n+\t\t\t} else {\n+\t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n+\t\t\t}\n+\t\t} else if (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n \t\t\tuint64_t item_count = strtoull(end, &end, 10);\n \t\t\tif (*end != '\\0')\n \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n@@ -63,12 +78,16 @@ int cmd__progress(int argc, const char **argv)\n \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n \t\t\tprogress_test_ns = test_ms * 1000 * 1000;\n \t\t\tdisplay_throughput(progress, byte_count);\n-\t\t} else if (!strcmp(line.buf, \"update\"))\n+\t\t} else if (!strcmp(line.buf, \"update\")) {\n \t\t\tprogress_test_force_update();\n-\t\telse\n+\t\t} else if (!strcmp(line.buf, \"stop\")) {\n+\t\t\tstop_progress(&progress);\n+\t\t} else {\n \t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n+\t\t}\n \t}\n-\tstop_progress(&progress);\n+\tif (detached_title)\n+\t\tfree(detached_title);\n \n \treturn 0;\n }\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 66c092a0fe3..ce6c3434673 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -22,6 +22,7 @@ test_expect_success 'simple progress display' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tupdate\n \tprogress 1\n \tupdate\n@@ -30,8 +31,9 @@ test_expect_success 'simple progress display' '\n \tprogress 4\n \tupdate\n \tprogress 5\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -46,11 +48,13 @@ test_expect_success 'progress display with total' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 3\n \tprogress 1\n \tprogress 2\n \tprogress 3\n+\tstop\n \tEOF\n-\ttest-tool progress --total=3 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -67,14 +71,14 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tprogress 100\n \tprogress 1000\n \tprogress 10000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -93,16 +97,15 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n-\tupdate\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tprogress 1\n \tupdate\n \tprogress 2\n \tprogress 10000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -121,14 +124,14 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tprogress 25000\n \tprogress 50000\n \tprogress 75000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -145,14 +148,14 @@ Working hard.......2.........3.........4.........5.........6.........7.........:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6.........7.........\n \tprogress 25000\n \tprogress 50000\n \tprogress 75000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6.........7.........\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -169,12 +172,14 @@ test_expect_success 'progress shortens - crazy caller' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 1000\n \tprogress 100\n \tprogress 200\n \tprogress 1\n \tprogress 1000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=1000 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -190,6 +195,7 @@ test_expect_success 'progress display with throughput' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart\n \tthroughput 102400 1000\n \tupdate\n \tprogress 10\n@@ -202,8 +208,9 @@ test_expect_success 'progress display with throughput' '\n \tthroughput 409600 4000\n \tupdate\n \tprogress 40\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -219,6 +226,7 @@ test_expect_success 'progress display with throughput and total' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 40\n \tthroughput 102400 1000\n \tprogress 10\n \tthroughput 204800 2000\n@@ -227,8 +235,9 @@ test_expect_success 'progress display with throughput and total' '\n \tprogress 30\n \tthroughput 409600 4000\n \tprogress 40\n+\tstop\n \tEOF\n-\ttest-tool progress --total=40 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -244,6 +253,7 @@ test_expect_success 'cover up after throughput shortens' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart\n \tthroughput 409600 1000\n \tupdate\n \tprogress 1\n@@ -256,8 +266,9 @@ test_expect_success 'cover up after throughput shortens' '\n \tthroughput 1638400 4000\n \tupdate\n \tprogress 4\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -272,6 +283,7 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart\n \tthroughput 1 1000\n \tupdate\n \tprogress 1\n@@ -281,8 +293,9 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \tthroughput 3145728 3000\n \tupdate\n \tprogress 3\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -290,6 +303,7 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \n test_expect_success 'progress generates traces' '\n \tcat >in <<-\\EOF &&\n+\tstart 40\n \tthroughput 102400 1000\n \tupdate\n \tprogress 10\n@@ -302,10 +316,11 @@ test_expect_success 'progress generates traces' '\n \tthroughput 409600 4000\n \tupdate\n \tprogress 40\n+\tstop\n \tEOF\n \n-\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" test-tool progress --total=40 \\\n-\t\t\"Working hard\" <in 2>stderr &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n \n \t# t0212/parse_events.perl intentionally omits regions and data.\n \ttest_region progress \"Working hard\" trace.event &&\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428285","messageId":"patch-03.25-397291e0f65-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 03/25] progress.c tests: test some invalid usage","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:03Z","receivedAt":"2021-06-23T17:48:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Test what happens when we \"stop\" without a \"start\", omit the \"stop\"\nafter a \"start\", or try to start two concurrent progress bars. This\nextends the trace2 tests added in 98a13647408 (trace2: log progress\ntime and throughput, 2020-05-12).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/t0500-progress-display.sh | 33 +++++++++++++++++++++++++++++++++\n 1 file changed, 33 insertions(+)\n\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex ce6c3434673..50eced31f03 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -328,4 +328,37 @@ test_expect_success 'progress generates traces' '\n \tgrep \"\\\"key\\\":\\\"total_bytes\\\",\\\"value\\\":\\\"409600\\\"\" trace.event\n '\n \n+test_expect_success 'progress generates traces: stop / start' '\n+\tcat >in <<-\\EOF &&\n+\tstart\n+\tstop\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-startstop.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n+\ttest_region progress \"Working hard\" trace-startstop.event\n+'\n+\n+test_expect_success 'progress generates traces: start without stop' '\n+\tcat >in <<-\\EOF &&\n+\tstart\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-start.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n+\tgrep region_enter.*progress trace-start.event &&\n+\t! grep region_leave.*progress trace-start.event\n+'\n+\n+test_expect_success 'progress generates traces: stop without start' '\n+\tcat >in <<-\\EOF &&\n+\tstop\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-stop.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n+\t! grep region_enter.*progress trace-stop.event &&\n+\t! grep region_leave.*progress trace-stop.event\n+'\n+\n test_done\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428286","messageId":"patch-04.25-900eeb0b115-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 04/25] progress.c tests: add a \"signal\" verb","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:04Z","receivedAt":"2021-06-23T17:48:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a \"signal\" synonym for \"update\". It is not typical of the\nprogress.c API to encounter a scenario where we do an update before\nthe first display_progress(), let's indicate this explicitly by\ncalling such instances \"signal\".\n\nIt's just a synonym for \"update\", but we can imagine than the\nfollowing \"update\" calls could elide many \"progress\" calls, and the\nprogress bar output will generally be of the same type, whereas the\noutput where we're asked to emit an update before we've received any\ndata is a special case.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/helper/test-progress.c    |  6 +++++-\n t/t0500-progress-display.sh | 10 +++++-----\n 2 files changed, 10 insertions(+), 6 deletions(-)\n\ndiff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\nindex eb925d591e1..7ca58a3ee78 100644\n--- a/t/helper/test-progress.c\n+++ b/t/helper/test-progress.c\n@@ -13,6 +13,9 @@\n  *                                  specify the time elapsed since the\n  *                                  start_progress() call.\n  *   \"update\" - Set the 'progress_update' flag.\n+ *   \"signal\" - Synonym for \"update\", used for self-documenting tests,\n+ *              i.e. \"expect signal here due to hanging (\"signal\")\n+ *              v.s. it was time to update (\"update\").\n  *   \"stop\" - Call stop_progress().\n  *\n  * See 't0500-progress-display.sh' for examples.\n@@ -78,7 +81,8 @@ int cmd__progress(int argc, const char **argv)\n \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n \t\t\tprogress_test_ns = test_ms * 1000 * 1000;\n \t\t\tdisplay_throughput(progress, byte_count);\n-\t\t} else if (!strcmp(line.buf, \"update\")) {\n+\t\t} else if (!strcmp(line.buf, \"update\") ||\n+\t\t\t   !strcmp(line.buf, \"signal\")) {\n \t\t\tprogress_test_force_update();\n \t\t} else if (!strcmp(line.buf, \"stop\")) {\n \t\t\tstop_progress(&progress);\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 50eced31f03..66c1989b176 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -23,7 +23,7 @@ test_expect_success 'simple progress display' '\n \n \tcat >in <<-\\EOF &&\n \tstart 0\n-\tupdate\n+\tsignal\n \tprogress 1\n \tupdate\n \tprogress 2\n@@ -197,7 +197,7 @@ test_expect_success 'progress display with throughput' '\n \tcat >in <<-\\EOF &&\n \tstart\n \tthroughput 102400 1000\n-\tupdate\n+\tsignal\n \tprogress 10\n \tthroughput 204800 2000\n \tupdate\n@@ -255,7 +255,7 @@ test_expect_success 'cover up after throughput shortens' '\n \tcat >in <<-\\EOF &&\n \tstart\n \tthroughput 409600 1000\n-\tupdate\n+\tsignal\n \tprogress 1\n \tthroughput 819200 2000\n \tupdate\n@@ -285,7 +285,7 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \tcat >in <<-\\EOF &&\n \tstart\n \tthroughput 1 1000\n-\tupdate\n+\tsignal\n \tprogress 1\n \tthroughput 1024000 2000\n \tupdate\n@@ -305,7 +305,7 @@ test_expect_success 'progress generates traces' '\n \tcat >in <<-\\EOF &&\n \tstart 40\n \tthroughput 102400 1000\n-\tupdate\n+\tsignal\n \tprogress 10\n \tthroughput 204800 2000\n \tupdate\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428287","messageId":"patch-07.25-02e6edd4b7a-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 07/25] progress.c: stop eagerly fflush(stderr) when not a terminal","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:07Z","receivedAt":"2021-06-23T17:48:47Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"It's the clear intention of the combination of 137a0d0ef56 (Flush\nprogress message buffer in display()., 2007-11-19) and\n85cb8906f0e (progress: no progress in background, 2015-04-13) to call\nfflush(stderr) when we have a stderr in the foreground, but we ended\nup always calling fflush(stderr) seemingly by omission. Let's not.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 7fcc513717a..1fade5808de 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -91,7 +91,8 @@ static void display(struct progress *progress, uint64_t n, const char *done)\n \t}\n \n \tif (show_update) {\n-\t\tif (is_foreground_fd(fileno(stderr)) || done) {\n+\t\tint stderr_is_foreground_fd = is_foreground_fd(fileno(stderr));\n+\t\tif (stderr_is_foreground_fd || done) {\n \t\t\tconst char *eol = done ? done : \"\\r\";\n \t\t\tsize_t clear_len = counters_sb->len < last_count_len ?\n \t\t\t\t\tlast_count_len - counters_sb->len + 1 :\n@@ -115,7 +116,8 @@ static void display(struct progress *progress, uint64_t n, const char *done)\n \t\t\t\tfprintf(stderr, \"%s: %s%*s\", progress->title,\n \t\t\t\t\tcounters_sb->buf, (int) clear_len, eol);\n \t\t\t}\n-\t\t\tfflush(stderr);\n+\t\t\tif (stderr_is_foreground_fd)\n+\t\t\t\tfflush(stderr);\n \t\t}\n \t\tprogress_update = 0;\n \t}\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428288","messageId":"patch-06.25-8261d2977e0-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 06/25] progress.c: call progress_interval() from progress_test_force_update()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:06Z","receivedAt":"2021-06-23T17:48:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Define the progress_test_force_update() function in terms of\nprogress_interval(). For documentation purposes these two functions\nhave the same body, but different names. Let's just define the test\nfunction by calling progress_interval() with SIGALRM ourselves.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/progress.c b/progress.c\nindex 893cb0fe56f..7fcc513717a 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -216,7 +216,7 @@ static void progress_interval(int signum)\n  */\n void progress_test_force_update(void)\n {\n-\tprogress_update = 1;\n+\tprogress_interval(SIGALRM);\n }\n \n static void set_progress_signal(void)\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428289","messageId":"patch-08.25-75b72929191-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 08/25] progress.c: add temporary variable from progress struct","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:08Z","receivedAt":"2021-06-23T17:48:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a temporary \"progress\" variable for the dereferenced p_progress\npointer to a \"struct progress *\". Before 98a13647408 (trace2: log\nprogress time and throughput, 2020-05-12) we didn't dereference\n\"p_progress\" in this function, now that we do it's easier to read the\ncode if we work with a \"progress\" struct pointer like everywhere else,\ninstead of a pointer to a pointer.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 1fade5808de..1ab7d19deb8 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -331,15 +331,16 @@ void stop_progress(struct progress **p_progress)\n \tfinish_if_sparse(*p_progress);\n \n \tif (*p_progress) {\n+\t\tstruct progress *progress = *p_progress;\n \t\ttrace2_data_intmax(\"progress\", the_repository, \"total_objects\",\n \t\t\t\t   (*p_progress)->total);\n \n \t\tif ((*p_progress)->throughput)\n \t\t\ttrace2_data_intmax(\"progress\", the_repository,\n \t\t\t\t\t   \"total_bytes\",\n-\t\t\t\t\t   (*p_progress)->throughput->curr_total);\n+\t\t\t\t\t   progress->throughput->curr_total);\n \n-\t\ttrace2_region_leave(\"progress\", (*p_progress)->title, the_repository);\n+\t\ttrace2_region_leave(\"progress\", progress->title, the_repository);\n \t}\n \n \tstop_progress_msg(p_progress, _(\"done\"));\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428290","messageId":"patch-09.25-c7a51927cad-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 09/25] midx perf: add a perf test for multi-pack-index","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:09Z","receivedAt":"2021-06-23T17:48:52Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a basic write and verify performance test for the multi-pack-index\ncommand.\n\nThe reason for doing the \"write\" also in a \"test_expect_success\" is to\nbe friendly to skipping the \"write\" test as a perf test (which would\nrun N times) but still being guaranteed to have a midx to verify by\nthe time we get to the \"verify\" test.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/perf/p5319-multi-pack-index.sh | 21 +++++++++++++++++++++\n 1 file changed, 21 insertions(+)\n create mode 100755 t/perf/p5319-multi-pack-index.sh\n\ndiff --git a/t/perf/p5319-multi-pack-index.sh b/t/perf/p5319-multi-pack-index.sh\nnew file mode 100755\nindex 00000000000..39769602ab7\n--- /dev/null\n+++ b/t/perf/p5319-multi-pack-index.sh\n@@ -0,0 +1,21 @@\n+#!/bin/sh\n+\n+test_description='Test midx performance'\n+\n+. ./perf-lib.sh\n+\n+test_perf_large_repo\n+\n+test_expect_success 'setup multi-pack-index' '\n+\tgit multi-pack-index write\n+'\n+\n+test_perf 'midx write' '\n+\tgit multi-pack-index write\n+'\n+\n+test_perf 'midx verify' '\n+\tgit multi-pack-index verify\n+'\n+\n+test_done\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428291","messageId":"patch-10.25-66e4909580b-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 10/25] progress.c: remove the \"sparse\" mode nano-optimization","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:10Z","receivedAt":"2021-06-23T17:48:52Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Revert the code added in 9d81ecb52b5 (progress: add sparse mode to\nforce 100% complete message, 2019-03-21) for the \"sparse\" progress\nmode, and change its only user added in 430efb8a74b (midx: add\nprogress indicators in multi-pack-index verify, 2019-03-21) to use the\nnormal non-sparse progress.c API instead.\n\nThe reason for checking the SPARSE_PROGRESS_INTERVAL for every 2^12\nobjects is to improve performance. It does that, but only in an\nisolated and artificial benchmark. In the case of the\n\"verify_midx_file\" user we're in a loop doing various other OID/object\nwork, the cost of calling display_progress() is entirely lost in the\nnoise.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n midx.c     | 26 +++++++-------------------\n progress.c | 38 +++-----------------------------------\n 2 files changed, 10 insertions(+), 54 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 21d6a05e887..d80e68998b8 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1186,18 +1186,6 @@ static int compare_pair_pos_vs_id(const void *_a, const void *_b)\n \treturn b->pack_int_id - a->pack_int_id;\n }\n \n-/*\n- * Limit calls to display_progress() for performance reasons.\n- * The interval here was arbitrarily chosen.\n- */\n-#define SPARSE_PROGRESS_INTERVAL (1 << 12)\n-#define midx_display_sparse_progress(progress, n) \\\n-\tdo { \\\n-\t\tuint64_t _n = (n); \\\n-\t\tif ((_n & (SPARSE_PROGRESS_INTERVAL - 1)) == 0) \\\n-\t\t\tdisplay_progress(progress, _n); \\\n-\t} while (0)\n-\n int verify_midx_file(struct repository *r, const char *object_dir, unsigned flags)\n {\n \tstruct pair_pos_vs_id *pairs = NULL;\n@@ -1248,8 +1236,8 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \t}\n \n \tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_sparse_progress(_(\"Verifying OID order in multi-pack-index\"),\n-\t\t\t\t\t\t m->num_objects - 1);\n+\t\tprogress = start_progress(_(\"Verifying OID order in multi-pack-index\"),\n+\t\t\t\t\t  m->num_objects - 1);\n \tfor (i = 0; i < m->num_objects - 1; i++) {\n \t\tstruct object_id oid1, oid2;\n \n@@ -1260,7 +1248,7 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \t\t\tmidx_report(_(\"oid lookup out of order: oid[%d] = %s >= %s = oid[%d]\"),\n \t\t\t\t    i, oid_to_hex(&oid1), oid_to_hex(&oid2), i + 1);\n \n-\t\tmidx_display_sparse_progress(progress, i + 1);\n+\t\tdisplay_progress(progress, i + 1);\n \t}\n \tstop_progress(&progress);\n \n@@ -1277,14 +1265,14 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \t}\n \n \tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_sparse_progress(_(\"Sorting objects by packfile\"),\n-\t\t\t\t\t\t m->num_objects);\n+\t\tprogress = start_progress(_(\"Sorting objects by packfile\"),\n+\t\t\t\t\t  m->num_objects);\n \tdisplay_progress(progress, 0); /* TODO: Measure QSORT() progress */\n \tQSORT(pairs, m->num_objects, compare_pair_pos_vs_id);\n \tstop_progress(&progress);\n \n \tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_sparse_progress(_(\"Verifying object offsets\"), m->num_objects);\n+\t\tprogress = start_progress(_(\"Verifying object offsets\"), m->num_objects);\n \tfor (i = 0; i < m->num_objects; i++) {\n \t\tstruct object_id oid;\n \t\tstruct pack_entry e;\n@@ -1318,7 +1306,7 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \t\t\tmidx_report(_(\"incorrect object offset for oid[%d] = %s: %\"PRIx64\" != %\"PRIx64),\n \t\t\t\t    pairs[i].pos, oid_to_hex(&oid), m_offset, p_offset);\n \n-\t\tmidx_display_sparse_progress(progress, i + 1);\n+\t\tdisplay_progress(progress, i + 1);\n \t}\n \tstop_progress(&progress);\n \ndiff --git a/progress.c b/progress.c\nindex 1ab7d19deb8..912edd4c818 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -37,7 +37,6 @@ struct progress {\n \tuint64_t total;\n \tunsigned last_percent;\n \tunsigned delay;\n-\tunsigned sparse;\n \tstruct throughput *throughput;\n \tuint64_t start_ns;\n \tstruct strbuf counters_sb;\n@@ -256,7 +255,7 @@ static void clear_progress_signal(void)\n }\n \n static struct progress *start_progress_delay(const char *title, uint64_t total,\n-\t\t\t\t\t     unsigned delay, unsigned sparse)\n+\t\t\t\t\t     unsigned delay)\n {\n \tstruct progress *progress = xmalloc(sizeof(*progress));\n \tprogress->title = title;\n@@ -264,7 +263,6 @@ static struct progress *start_progress_delay(const char *title, uint64_t total,\n \tprogress->last_value = -1;\n \tprogress->last_percent = -1;\n \tprogress->delay = delay;\n-\tprogress->sparse = sparse;\n \tprogress->throughput = NULL;\n \tprogress->start_ns = getnanotime();\n \tstrbuf_init(&progress->counters_sb, 0);\n@@ -287,40 +285,12 @@ static int get_default_delay(void)\n \n struct progress *start_delayed_progress(const char *title, uint64_t total)\n {\n-\treturn start_progress_delay(title, total, get_default_delay(), 0);\n+\treturn start_progress_delay(title, total, get_default_delay());\n }\n \n struct progress *start_progress(const char *title, uint64_t total)\n {\n-\treturn start_progress_delay(title, total, 0, 0);\n-}\n-\n-/*\n- * Here \"sparse\" means that the caller might use some sampling criteria to\n- * decide when to call display_progress() rather than calling it for every\n- * integer value in[0 .. total).  In particular, the caller might not call\n- * display_progress() for the last value in the range.\n- *\n- * When \"sparse\" is set, stop_progress() will automatically force the done\n- * message to show 100%.\n- */\n-struct progress *start_sparse_progress(const char *title, uint64_t total)\n-{\n-\treturn start_progress_delay(title, total, 0, 1);\n-}\n-\n-struct progress *start_delayed_sparse_progress(const char *title,\n-\t\t\t\t\t       uint64_t total)\n-{\n-\treturn start_progress_delay(title, total, get_default_delay(), 1);\n-}\n-\n-static void finish_if_sparse(struct progress *progress)\n-{\n-\tif (progress &&\n-\t    progress->sparse &&\n-\t    progress->last_value != progress->total)\n-\t\tdisplay_progress(progress, progress->total);\n+\treturn start_progress_delay(title, total, 0);\n }\n \n void stop_progress(struct progress **p_progress)\n@@ -328,8 +298,6 @@ void stop_progress(struct progress **p_progress)\n \tif (!p_progress)\n \t\tBUG(\"don't provide NULL to stop_progress\");\n \n-\tfinish_if_sparse(*p_progress);\n-\n \tif (*p_progress) {\n \t\tstruct progress *progress = *p_progress;\n \t\ttrace2_data_intmax(\"progress\", the_repository, \"total_objects\",\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428292","messageId":"patch-11.25-7f5b71fe1be-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 11/25] pack-bitmap-write.c: add a missing stop_progress()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:11Z","receivedAt":"2021-06-23T17:48:53Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a bug that's been here since 7cc8f971085 (pack-objects: implement\nbitmap writing, 2013-12-21), we did not call stop_progress() if we\nreached the early exit in this function. This will matter in a\nsubsequent commit where we BUG(...) out if this happens, and matters\nnow e.g. because we don't have a corresponding \"region_end\" for the\nprogress trace2 event.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n pack-bitmap-write.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 88d9e696a54..6e110e41ea4 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -550,6 +550,7 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \tif (indexed_commits_nr < 100) {\n \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n \t\t\tpush_bitmapped_commit(indexed_commits[i]);\n+\t\tstop_progress(&writer.progress);\n \t\treturn;\n \t}\n \n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428293","messageId":"patch-12.25-c4e7179e957-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 12/25] progress.c: add & assert a \"global_progress\" variable","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:12Z","receivedAt":"2021-06-23T17:48:54Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The progress.c code makes a hard assumption that only one progress bar\nbe active at a time (see [1] for a bug where this wasn't the case),\nbut nothing has asserted that that's the case. Let's add a BUG()\nthat'll trigger if two progress bars are active at the same time.\n\nThere's an alternate test-only approach to doing the same thing[2],\nbut by doing this for all progress bars we'll have a canary to check\nif we have any unexpected interaction between the \"sig_atomic_t\nprogress_update\" variable and this global struct.\n\nI am then planning on using this scaffolding in the future to fix a\nlimitation in the progress output, namely the current limitation of\nthe progress.c bar code that any update must pro-actively go through\nthe likes of display_progress().\n\nIf we e.g. hang forever before the first display_progress(), or in the\nmiddle of a loop that would call display_progress() the user will only\nsee either no output, or output frozen at the last display_progress()\nthat would have done an update (e.g. in cases where progress_update\nwas \"1\" due to an earlier signal).\n\nThis change does not fix that, but sets up the structure for solving\nthat and other related problems by juggling this \"global_progress\"\nstruct. Later changes will make more use of the \"global_progress\" than\nonly using it for these assertions.\n\n1. 6f9d5f2fda1 (commit-graph: fix progress of reachable commits, 2020-07-09)\n2. https://lore.kernel.org/git/20210620200303.2328957-3-szeder.dev@gmail.com\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c                  | 17 +++++++++++++----\n t/t0500-progress-display.sh | 11 +++++++++++\n 2 files changed, 24 insertions(+), 4 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 912edd4c818..e1b50ef7882 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -45,6 +45,7 @@ struct progress {\n };\n \n static volatile sig_atomic_t progress_update;\n+static struct progress *global_progress;\n \n /*\n  * These are only intended for testing the progress output, i.e. exclusively\n@@ -220,11 +221,15 @@ void progress_test_force_update(void)\n \tprogress_interval(SIGALRM);\n }\n \n-static void set_progress_signal(void)\n+static void set_progress_signal(struct progress *progress)\n {\n \tstruct sigaction sa;\n \tstruct itimerval v;\n \n+\tif (global_progress)\n+\t\tBUG(\"should have no global_progress in set_progress_signal()\");\n+\tglobal_progress = progress;\n+\n \tif (progress_testing)\n \t\treturn;\n \n@@ -242,10 +247,14 @@ static void set_progress_signal(void)\n \tsetitimer(ITIMER_REAL, &v, NULL);\n }\n \n-static void clear_progress_signal(void)\n+static void clear_progress_signal(struct progress *progress)\n {\n \tstruct itimerval v = {{0,},};\n \n+\tif (!global_progress)\n+\t\tBUG(\"should have a global_progress in clear_progress_signal()\");\n+\tglobal_progress = NULL;\n+\n \tif (progress_testing)\n \t\treturn;\n \n@@ -268,7 +277,7 @@ static struct progress *start_progress_delay(const char *title, uint64_t total,\n \tstrbuf_init(&progress->counters_sb, 0);\n \tprogress->title_len = utf8_strwidth(title);\n \tprogress->split = 0;\n-\tset_progress_signal();\n+\tset_progress_signal(progress);\n \ttrace2_region_enter(\"progress\", title, the_repository);\n \treturn progress;\n }\n@@ -342,7 +351,7 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)\n \t\tdisplay(progress, progress->last_value, buf);\n \t\tfree(buf);\n \t}\n-\tclear_progress_signal();\n+\tclear_progress_signal(progress);\n \tstrbuf_release(&progress->counters_sb);\n \tif (progress->throughput)\n \t\tstrbuf_release(&progress->throughput->display);\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 66c1989b176..476a31222a3 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -361,4 +361,15 @@ test_expect_success 'progress generates traces: stop without start' '\n \t! grep region_leave.*progress trace-stop.event\n '\n \n+test_expect_success 'BUG: start two concurrent progress bars' '\n+\tcat >in <<-\\EOF &&\n+\tstart 0 one\n+\tstart 0 two\n+\tEOF\n+\n+\ttest_must_fail test-tool progress \\\n+\t\t<in 2>stderr &&\n+\tgrep -E \"^BUG: .*: should have no global_progress in set_progress_signal\\(\\)$\" stderr\n+'\n+\n test_done\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428294","messageId":"patch-14.25-88d864c373a-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 14/25] progress.[ch]: move test-only code away from \"extern\" variables","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:14Z","receivedAt":"2021-06-23T17:48:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Since the test-only support code was added in 2bb74b53a49 (Test the\nprogress display, 2019-09-16) we've had to define\nGIT_TEST_PROGRESS_ONLY more widely as part of the bugfix in\n3cacb9aaf46 (progress.c: silence cgcc suggestion about internal\nlinkage, 2020-04-27).\n\nSo the only thing we were getting out of this indirection was keeping\nGIT_TEST_PROGRESS_ONLY from being defined in progress.h itself,\ni.e. so the likes of csum-file.h wouldn't have access to them, we'd\nstill compile them in progress.o.\n\nLet's just always define and compile them without this needless slight\nof hand, the linking and strip step will take care of removing these\nunused symbols, if needed.\n\nWe now expose a start_progress_testing() function instead, which'll\nset a \"test_mode\" member, which the test of the code can check.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c               | 34 ++++++++++++++--------------------\n progress.h               | 21 ++++++++++++++-------\n t/helper/test-progress.c | 11 +++++------\n 3 files changed, 33 insertions(+), 33 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex aff9af9ee8b..39d7f6bd86b 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -8,7 +8,6 @@\n  * published by the Free Software Foundation.\n  */\n \n-#define GIT_TEST_PROGRESS_ONLY\n #include \"cache.h\"\n #include \"gettext.h\"\n #include \"progress.h\"\n@@ -20,13 +19,6 @@\n static volatile sig_atomic_t progress_update;\n static struct progress *global_progress;\n \n-/*\n- * These are only intended for testing the progress output, i.e. exclusively\n- * for 'test-tool progress'.\n- */\n-int progress_testing;\n-uint64_t progress_test_ns = 0;\n-\n static int is_foreground_fd(int fd)\n {\n \tint tpgrp = tcgetpgrp(fd);\n@@ -108,8 +100,8 @@ static void throughput_string(struct strbuf *buf, uint64_t total,\n \n static uint64_t progress_getnanotime(struct progress *progress)\n {\n-\tif (progress_testing)\n-\t\treturn progress->start_ns + progress_test_ns;\n+\tif (progress->test_getnanotime)\n+\t\treturn progress->start_ns + progress->test_getnanotime;\n \telse\n \t\treturn getnanotime();\n }\n@@ -185,11 +177,7 @@ static void progress_interval(int signum)\n \tprogress_update = 1;\n }\n \n-/*\n- * The progress_test_force_update() function is intended for testing\n- * the progress output, i.e. exclusively for 'test-tool progress'.\n- */\n-void progress_test_force_update(void)\n+void test_progress_force_update(void)\n {\n \tprogress_interval(SIGALRM);\n }\n@@ -203,7 +191,7 @@ static void set_progress_signal(struct progress *progress)\n \t\tBUG(\"should have no global_progress in set_progress_signal()\");\n \tglobal_progress = progress;\n \n-\tif (progress_testing)\n+\tif (progress->test_mode)\n \t\treturn;\n \n \tprogress_update = 0;\n@@ -228,7 +216,7 @@ static void clear_progress_signal(struct progress *progress)\n \t\tBUG(\"should have a global_progress in clear_progress_signal()\");\n \tglobal_progress = NULL;\n \n-\tif (progress_testing)\n+\tif (progress->test_mode)\n \t\treturn;\n \n \tsetitimer(ITIMER_REAL, &v, NULL);\n@@ -237,7 +225,7 @@ static void clear_progress_signal(struct progress *progress)\n }\n \n static struct progress *start_progress_delay(const char *title, uint64_t total,\n-\t\t\t\t\t     unsigned delay)\n+\t\t\t\t\t     unsigned delay, int testing)\n {\n \tstruct progress *progress = xmalloc(sizeof(*progress));\n \tprogress->title = title;\n@@ -250,11 +238,17 @@ static struct progress *start_progress_delay(const char *title, uint64_t total,\n \tstrbuf_init(&progress->counters_sb, 0);\n \tprogress->title_len = utf8_strwidth(title);\n \tprogress->split = 0;\n+\tprogress->test_mode = testing;\n \tset_progress_signal(progress);\n \ttrace2_region_enter(\"progress\", title, the_repository);\n \treturn progress;\n }\n \n+struct progress *start_progress_testing(const char *title, uint64_t total)\n+{\n+\treturn start_progress_delay(title, total, 0, 1);\n+}\n+\n static int get_default_delay(void)\n {\n \tstatic int delay_in_secs = -1;\n@@ -267,12 +261,12 @@ static int get_default_delay(void)\n \n struct progress *start_delayed_progress(const char *title, uint64_t total)\n {\n-\treturn start_progress_delay(title, total, get_default_delay());\n+\treturn start_progress_delay(title, total, get_default_delay(), 0);\n }\n \n struct progress *start_progress(const char *title, uint64_t total)\n {\n-\treturn start_progress_delay(title, total, 0);\n+\treturn start_progress_delay(title, total, 0, 0);\n }\n \n void stop_progress(struct progress **p_progress)\ndiff --git a/progress.h b/progress.h\nindex 4fb2b483d36..4693dddb6c5 100644\n--- a/progress.h\n+++ b/progress.h\n@@ -27,15 +27,22 @@ struct progress {\n \tstruct strbuf counters_sb;\n \tint title_len;\n \tint split;\n-};\n-\n-#ifdef GIT_TEST_PROGRESS_ONLY\n \n-extern int progress_testing;\n-extern uint64_t progress_test_ns;\n-void progress_test_force_update(void);\n+\t/*\n+\t * The test_* members are are only intended for testing the\n+\t * progress output, i.e. exclusively for 'test-tool progress'.\n+\t */\n+\tint test_mode;\n+\tuint64_t test_getnanotime;\n+};\n \n-#endif\n+/*\n+ * *_testing() functions are only for use in\n+ * t/helper/test-progress.c. Do not use them elsewhere!\n+ */\n+void test_progress_force_update(void);\n+struct progress *start_progress_testing(const char *title, uint64_t total);\n+void test_progress_setnanotime(struct progress *progress, uint64_t time);\n \n void display_throughput(struct progress *progress, uint64_t total);\n void display_progress(struct progress *progress, uint64_t n);\ndiff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\nindex 7ca58a3ee78..40dbacb0557 100644\n--- a/t/helper/test-progress.c\n+++ b/t/helper/test-progress.c\n@@ -46,21 +46,20 @@ int cmd__progress(int argc, const char **argv)\n \tif (argc)\n \t\tusage_with_options(usage, options);\n \n-\tprogress_testing = 1;\n \twhile (strbuf_getline(&line, stdin) != EOF) {\n \t\tchar *end;\n \n \t\tif (!strcmp(line.buf, \"start\")) {\n-\t\t\tprogress = start_progress(default_title, 0);\n+\t\t\tprogress = start_progress_testing(default_title, 0);\n \t\t} else if (skip_prefix(line.buf, \"start \", (const char **) &end)) {\n \t\t\tuint64_t total = strtoull(end, &end, 10);\n \t\t\tif (*end == '\\0') {\n-\t\t\t\tprogress = start_progress(default_title, total);\n+\t\t\t\tprogress = start_progress_testing(default_title, total);\n \t\t\t} else if (*end == ' ') {\n \t\t\t\tif (detached_title)\n \t\t\t\t\tfree(detached_title);\n \t\t\t\tdetached_title = strbuf_detach(&line, NULL);\n-\t\t\t\tprogress = start_progress(end + 1, total);\n+\t\t\t\tprogress = start_progress_testing(end + 1, total);\n \t\t\t} else {\n \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n \t\t\t}\n@@ -79,11 +78,11 @@ int cmd__progress(int argc, const char **argv)\n \t\t\ttest_ms = strtoull(end + 1, &end, 10);\n \t\t\tif (*end != '\\0')\n \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n-\t\t\tprogress_test_ns = test_ms * 1000 * 1000;\n+\t\t\tprogress->test_getnanotime = test_ms * 1000 * 1000;\n \t\t\tdisplay_throughput(progress, byte_count);\n \t\t} else if (!strcmp(line.buf, \"update\") ||\n \t\t\t   !strcmp(line.buf, \"signal\")) {\n-\t\t\tprogress_test_force_update();\n+\t\t\ttest_progress_force_update();\n \t\t} else if (!strcmp(line.buf, \"stop\")) {\n \t\t\tstop_progress(&progress);\n \t\t} else {\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428295","messageId":"patch-13.25-f2e84e3a3f2-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 13/25] progress.[ch]: move the \"struct progress\" to the header","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:13Z","receivedAt":"2021-06-23T17:48:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the definition of the \"struct progress\" to the progress.h\nheader. Even though its contents are meant to be \"private\" this\npattern has resulted in forward declarations of it in various places,\nas other functions have a need to pass it around.\n\nLet's just define it in the header instead. It's part of our own\ninternal code, so we're not at much risk of someone tweaking the\ninternal fields manually. While doing that rename the \"TP_IDX_MAX\"\nmacro to the more clearly namespaced \"PROGRESS_THROUGHPUT_IDX_MAX\".\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n cache.h             |  1 -\n csum-file.h         |  2 --\n pack.h              |  1 -\n parallel-checkout.h |  1 -\n progress.c          | 29 +----------------------------\n progress.h          | 28 +++++++++++++++++++++++++++-\n reachable.h         |  1 -\n 7 files changed, 28 insertions(+), 35 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex ba04ff8bd36..7e03a181f68 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -308,7 +308,6 @@ static inline unsigned int canon_mode(unsigned int mode)\n \n struct split_index;\n struct untracked_cache;\n-struct progress;\n struct pattern_list;\n \n struct index_state {\ndiff --git a/csum-file.h b/csum-file.h\nindex 3044bd19ab6..3de0de653e8 100644\n--- a/csum-file.h\n+++ b/csum-file.h\n@@ -3,8 +3,6 @@\n \n #include \"hash.h\"\n \n-struct progress;\n-\n /* A SHA1-protected file */\n struct hashfile {\n \tint fd;\ndiff --git a/pack.h b/pack.h\nindex fa139545262..8df04f4937a 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -77,7 +77,6 @@ struct pack_idx_entry {\n };\n \n \n-struct progress;\n /* Note, the data argument could be NULL if object type is blob */\n typedef int (*verify_fn)(const struct object_id *, enum object_type, unsigned long, void*, int*);\n \ndiff --git a/parallel-checkout.h b/parallel-checkout.h\nindex 80f539bcb77..193f76398d6 100644\n--- a/parallel-checkout.h\n+++ b/parallel-checkout.h\n@@ -5,7 +5,6 @@\n \n struct cache_entry;\n struct checkout;\n-struct progress;\n \n /****************************************************************\n  * Users of parallel checkout\ndiff --git a/progress.c b/progress.c\nindex e1b50ef7882..aff9af9ee8b 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -17,33 +17,6 @@\n #include \"utf8.h\"\n #include \"config.h\"\n \n-#define TP_IDX_MAX      8\n-\n-struct throughput {\n-\toff_t curr_total;\n-\toff_t prev_total;\n-\tuint64_t prev_ns;\n-\tunsigned int avg_bytes;\n-\tunsigned int avg_misecs;\n-\tunsigned int last_bytes[TP_IDX_MAX];\n-\tunsigned int last_misecs[TP_IDX_MAX];\n-\tunsigned int idx;\n-\tstruct strbuf display;\n-};\n-\n-struct progress {\n-\tconst char *title;\n-\tuint64_t last_value;\n-\tuint64_t total;\n-\tunsigned last_percent;\n-\tunsigned delay;\n-\tstruct throughput *throughput;\n-\tuint64_t start_ns;\n-\tstruct strbuf counters_sb;\n-\tint title_len;\n-\tint split;\n-};\n-\n static volatile sig_atomic_t progress_update;\n static struct progress *global_progress;\n \n@@ -194,7 +167,7 @@ void display_throughput(struct progress *progress, uint64_t total)\n \ttp->avg_misecs -= tp->last_misecs[tp->idx];\n \ttp->last_bytes[tp->idx] = count;\n \ttp->last_misecs[tp->idx] = misecs;\n-\ttp->idx = (tp->idx + 1) % TP_IDX_MAX;\n+\ttp->idx = (tp->idx + 1) % PROGRESS_THROUGHPUT_IDX_MAX;\n \n \tthroughput_string(&tp->display, total, rate);\n \tif (progress->last_value != -1 && progress_update)\ndiff --git a/progress.h b/progress.h\nindex f1913acf73f..4fb2b483d36 100644\n--- a/progress.h\n+++ b/progress.h\n@@ -1,7 +1,33 @@\n #ifndef PROGRESS_H\n #define PROGRESS_H\n+#include \"strbuf.h\"\n \n-struct progress;\n+#define PROGRESS_THROUGHPUT_IDX_MAX      8\n+\n+struct throughput {\n+\toff_t curr_total;\n+\toff_t prev_total;\n+\tuint64_t prev_ns;\n+\tunsigned int avg_bytes;\n+\tunsigned int avg_misecs;\n+\tunsigned int last_bytes[PROGRESS_THROUGHPUT_IDX_MAX];\n+\tunsigned int last_misecs[PROGRESS_THROUGHPUT_IDX_MAX];\n+\tunsigned int idx;\n+\tstruct strbuf display;\n+};\n+\n+struct progress {\n+\tconst char *title;\n+\tuint64_t last_value;\n+\tuint64_t total;\n+\tunsigned last_percent;\n+\tunsigned delay;\n+\tstruct throughput *throughput;\n+\tuint64_t start_ns;\n+\tstruct strbuf counters_sb;\n+\tint title_len;\n+\tint split;\n+};\n \n #ifdef GIT_TEST_PROGRESS_ONLY\n \ndiff --git a/reachable.h b/reachable.h\nindex 5df932ad8f5..7e1ddddbc63 100644\n--- a/reachable.h\n+++ b/reachable.h\n@@ -1,7 +1,6 @@\n #ifndef REACHEABLE_H\n #define REACHEABLE_H\n \n-struct progress;\n struct rev_info;\n \n int add_unseen_recent_objects_to_traversal(struct rev_info *revs,\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428296","messageId":"patch-16.25-2161d9cbef3-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 16/25] progress.[ch]: convert \"title\" to \"struct strbuf\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:16Z","receivedAt":"2021-06-23T17:49:01Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Covert the \"title\" for the progress bar to a \"struct strbuf\", as with\nthe existing \"counters_sb\". Let's also rename the \"counters_sb\" to\nmerely \"status\", as we'll soon start using it not just to count, but\nfor any other arbitrary messaging after our fixed \"title\".\n\nThis makes the emitting the output more consistent, and allows us to\nhave both a UTF-8 progress bar, and a \"status\" portion. We won't be\nmaking use of the latter just let, but let's not close the door to it\nby relying on a strbuf with a len for one, and a char * for the other.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 63 ++++++++++++++++++++++++++++++++----------------------\n progress.h |  9 +++++---\n 2 files changed, 44 insertions(+), 28 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 44479f65921..e17490964c4 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -29,9 +29,8 @@ static void display(struct progress *progress, uint64_t n,\n \t\t    const char *update_msg, int last_update)\n {\n \tconst char *tp;\n-\tstruct strbuf *counters_sb = &progress->counters_sb;\n \tint show_update = 0;\n-\tint last_count_len = counters_sb->len;\n+\tsize_t last_count_len = progress->status_len_utf8;\n \n \tif (progress->delay && (!progress_update || --progress->delay))\n \t\treturn;\n@@ -43,47 +42,57 @@ static void display(struct progress *progress, uint64_t n,\n \t\tif (percent != progress->last_percent || progress_update) {\n \t\t\tprogress->last_percent = percent;\n \n-\t\t\tstrbuf_reset(counters_sb);\n-\t\t\tstrbuf_addf(counters_sb,\n+\t\t\tstrbuf_reset(&progress->status);\n+\t\t\tstrbuf_addf(&progress->status,\n \t\t\t\t    \"%3u%% (%\"PRIuMAX\"/%\"PRIuMAX\")%s\", percent,\n \t\t\t\t    (uintmax_t)n, (uintmax_t)progress->total,\n \t\t\t\t    tp);\n \t\t\tshow_update = 1;\n \t\t}\n \t} else if (progress_update) {\n-\t\tstrbuf_reset(counters_sb);\n-\t\tstrbuf_addf(counters_sb, \"%\"PRIuMAX\"%s\", (uintmax_t)n, tp);\n+\t\tstrbuf_reset(&progress->status);\n+\t\tstrbuf_addf(&progress->status, \"%\"PRIuMAX\"%s\", (uintmax_t)n, tp);\n \t\tshow_update = 1;\n \t}\n \n \tif (show_update && update_msg)\n-\t\tstrbuf_addf(counters_sb, \", %s.\", update_msg);\n+\t\tstrbuf_addf(&progress->status, \", %s.\", update_msg);\n \n \tif (show_update) {\n \t\tint stderr_is_foreground_fd = is_foreground_fd(fileno(stderr));\n \t\tif (stderr_is_foreground_fd || update_msg) {\n \t\t\tconst char *eol = last_update ? \"\\n\" : \"\\r\";\n-\t\t\tsize_t clear_len = counters_sb->len < last_count_len ?\n-\t\t\t\t\tlast_count_len - counters_sb->len + 1 :\n+\t\t\tsize_t clear_len = progress->status.len < last_count_len ?\n+\t\t\t\t\tlast_count_len - progress->status.len + 1 :\n \t\t\t\t\t0;\n \t\t\t/* The \"+ 2\" accounts for the \": \". */\n-\t\t\tsize_t progress_line_len = progress->title_len +\n-\t\t\t\t\t\tcounters_sb->len + 2;\n+\t\t\tsize_t progress_line_len = progress->title_len_utf8 +\n+\t\t\t\t\t\tprogress->status.len + 2;\n \t\t\tint cols = term_columns();\n+\t\t\tprogress->status_len_utf8 = utf8_strwidth(progress->status.buf);\n \n \t\t\tif (progress->split) {\n-\t\t\t\tfprintf(stderr, \"  %s%*s\", counters_sb->buf,\n-\t\t\t\t\t(int) clear_len, eol);\n+\t\t\t\tfprintf(stderr, \"  %*s%*s\",\n+\t\t\t\t\t(int)progress->status_len_utf8,\n+\t\t\t\t\tprogress->status.buf,\n+\t\t\t\t\t(int)clear_len, eol);\n \t\t\t} else if (!update_msg && cols < progress_line_len) {\n-\t\t\t\tclear_len = progress->title_len + 1 < cols ?\n-\t\t\t\t\t    cols - progress->title_len - 1 : 0;\n-\t\t\t\tfprintf(stderr, \"%s:%*s\\n  %s%s\",\n-\t\t\t\t\tprogress->title, (int) clear_len, \"\",\n-\t\t\t\t\tcounters_sb->buf, eol);\n+\t\t\t\tclear_len = progress->title_len_utf8 + 1 < cols ?\n+\t\t\t\t\t    cols - progress->title_len_utf8 - 1 : 0;\n+\t\t\t\tfprintf(stderr, \"%*s:%*s\\n  %*s%s\",\n+\t\t\t\t\t(int)progress->title_len_utf8,\n+\t\t\t\t\tprogress->title.buf,\n+\t\t\t\t\t(int)clear_len, \"\",\n+\t\t\t\t\t(int)progress->status_len_utf8,\n+\t\t\t\t\tprogress->status.buf, eol);\n \t\t\t\tprogress->split = 1;\n \t\t\t} else {\n-\t\t\t\tfprintf(stderr, \"%s: %s%*s\", progress->title,\n-\t\t\t\t\tcounters_sb->buf, (int) clear_len, eol);\n+\t\t\t\tfprintf(stderr, \"%*s: %*s%*s\",\n+\t\t\t\t\t(int)progress->title_len_utf8,\n+\t\t\t\t\tprogress->title.buf,\n+\t\t\t\t\t(int)progress->status_len_utf8,\n+\t\t\t\t\tprogress->status.buf,\n+\t\t\t\t\t(int)clear_len, eol);\n \t\t\t}\n \t\t\tif (stderr_is_foreground_fd)\n \t\t\t\tfflush(stderr);\n@@ -232,15 +241,18 @@ static struct progress *start_progress_delay(const char *title, uint64_t total,\n \t\t\t\t\t     unsigned delay, int testing)\n {\n \tstruct progress *progress = xmalloc(sizeof(*progress));\n-\tprogress->title = title;\n+\tstrbuf_init(&progress->title, 0);\n+\tstrbuf_addstr(&progress->title, title);\n+\tprogress->title_len_utf8 = utf8_strwidth(title);\n+\tstrbuf_init(&progress->status, 0);\n+\tprogress->status_len_utf8 = 0;\n+\n \tprogress->total = total;\n \tprogress->last_value = -1;\n \tprogress->last_percent = -1;\n \tprogress->delay = delay;\n \tprogress->throughput = NULL;\n \tprogress->start_ns = getnanotime();\n-\tstrbuf_init(&progress->counters_sb, 0);\n-\tprogress->title_len = utf8_strwidth(title);\n \tprogress->split = 0;\n \tprogress->test_mode = testing;\n \tset_progress_signal(progress);\n@@ -288,7 +300,7 @@ void stop_progress(struct progress **p_progress)\n \t\t\t\t\t   \"total_bytes\",\n \t\t\t\t\t   progress->throughput->curr_total);\n \n-\t\ttrace2_region_leave(\"progress\", progress->title, the_repository);\n+\t\ttrace2_region_leave(\"progress\", progress->title.buf, the_repository);\n \t}\n \n \tstop_progress_msg(p_progress, _(\"done\"));\n@@ -320,7 +332,8 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)\n \t\tdisplay(progress, progress->last_value, msg, 1);\n \t}\n \tclear_progress_signal(progress);\n-\tstrbuf_release(&progress->counters_sb);\n+\tstrbuf_release(&progress->title);\n+\tstrbuf_release(&progress->status);\n \tif (progress->throughput)\n \t\tstrbuf_release(&progress->throughput->display);\n \tfree(progress->throughput);\ndiff --git a/progress.h b/progress.h\nindex 4693dddb6c5..ba38447d104 100644\n--- a/progress.h\n+++ b/progress.h\n@@ -17,15 +17,18 @@ struct throughput {\n };\n \n struct progress {\n-\tconst char *title;\n+\tstruct strbuf title;\n+\tsize_t title_len_utf8;\n+\n+\tstruct strbuf status;\n+\tsize_t status_len_utf8;\n+\n \tuint64_t last_value;\n \tuint64_t total;\n \tunsigned last_percent;\n \tunsigned delay;\n \tstruct throughput *throughput;\n \tuint64_t start_ns;\n-\tstruct strbuf counters_sb;\n-\tint title_len;\n \tint split;\n \n \t/*\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428297","messageId":"patch-17.25-8f2ba566aae-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 17/25] progress.c: refactor display() for less confusion, and fix bug","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:17Z","receivedAt":"2021-06-23T17:49:01Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"As tested for in 2bb74b53a49 (Test the progress display, 2019-09-16)\nwe would redundantly emit extra spaces to clear output we never\nemitted under the split mode. Now we'll always clear precisely as many\ncolumns as we need, and no more.\n\nThe root cause of that issue is that since the progress code was\noriginally written we've grown support for various new features, and\nended up with a function where we didn't build the output we were\nabout to emit once, and then emitted it.\n\nWe thus couldn't easily track the length of the output we really did\nemit, with everything going downhill from there.\n\nThe alternative approach is longer (largely due to added comments),\nbut I think much clearer.\n\nWe no longer rely on magic constants like \"2\" for \": \" or \"\n\" (although we do still rely on the two separators being the same\nlength, but now have a related BUG(...) assertion).\n\nWe don't update \"status_len_utf8\" (or rather, the now-gone\n\"last_count_len\") or \"progress->last_value\" until after we've emitted\nall the output.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c                  | 137 +++++++++++++++++++++++++++---------\n t/t0500-progress-display.sh |   8 +--\n 2 files changed, 104 insertions(+), 41 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex e17490964c4..6c4038df791 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -25,17 +25,24 @@ static int is_foreground_fd(int fd)\n \treturn tpgrp < 0 || tpgrp == getpgid(0);\n }\n \n+static const char *counter_prefix(int split)\n+{\n+\tswitch (split) {\n+\tcase 1: return \"  \";\n+\tcase 0: return \": \";\n+\tdefault: BUG(\"unknown split value\");\n+\t}\n+}\n+\n static void display(struct progress *progress, uint64_t n,\n \t\t    const char *update_msg, int last_update)\n {\n \tconst char *tp;\n \tint show_update = 0;\n-\tsize_t last_count_len = progress->status_len_utf8;\n \n \tif (progress->delay && (!progress_update || --progress->delay))\n \t\treturn;\n \n-\tprogress->last_value = n;\n \ttp = (progress->throughput) ? progress->throughput->display.buf : \"\";\n \tif (progress->total) {\n \t\tunsigned percent = n * 100 / progress->total;\n@@ -44,61 +51,121 @@ static void display(struct progress *progress, uint64_t n,\n \n \t\t\tstrbuf_reset(&progress->status);\n \t\t\tstrbuf_addf(&progress->status,\n-\t\t\t\t    \"%3u%% (%\"PRIuMAX\"/%\"PRIuMAX\")%s\", percent,\n+\t\t\t\t    \"%s%3u%% (%\"PRIuMAX\"/%\"PRIuMAX\")%s\",\n+\t\t\t\t    counter_prefix(progress->split), percent,\n \t\t\t\t    (uintmax_t)n, (uintmax_t)progress->total,\n \t\t\t\t    tp);\n \t\t\tshow_update = 1;\n \t\t}\n \t} else if (progress_update) {\n \t\tstrbuf_reset(&progress->status);\n-\t\tstrbuf_addf(&progress->status, \"%\"PRIuMAX\"%s\", (uintmax_t)n, tp);\n+\t\tstrbuf_addf(&progress->status, \"%s%\"PRIuMAX\"%s\", counter_prefix(progress->split),\n+\t\t\t    (uintmax_t)n, tp);\n \t\tshow_update = 1;\n \t}\n \n \tif (show_update && update_msg)\n-\t\tstrbuf_addf(&progress->status, \", %s.\", update_msg);\n+\t\tstrbuf_addstr(&progress->status, update_msg);\n \n \tif (show_update) {\n \t\tint stderr_is_foreground_fd = is_foreground_fd(fileno(stderr));\n \t\tif (stderr_is_foreground_fd || update_msg) {\n \t\t\tconst char *eol = last_update ? \"\\n\" : \"\\r\";\n-\t\t\tsize_t clear_len = progress->status.len < last_count_len ?\n-\t\t\t\t\tlast_count_len - progress->status.len + 1 :\n-\t\t\t\t\t0;\n-\t\t\t/* The \"+ 2\" accounts for the \": \". */\n-\t\t\tsize_t progress_line_len = progress->title_len_utf8 +\n-\t\t\t\t\t\tprogress->status.len + 2;\n-\t\t\tint cols = term_columns();\n-\t\t\tprogress->status_len_utf8 = utf8_strwidth(progress->status.buf);\n-\n-\t\t\tif (progress->split) {\n-\t\t\t\tfprintf(stderr, \"  %*s%*s\",\n-\t\t\t\t\t(int)progress->status_len_utf8,\n-\t\t\t\t\tprogress->status.buf,\n-\t\t\t\t\t(int)clear_len, eol);\n-\t\t\t} else if (!update_msg && cols < progress_line_len) {\n-\t\t\t\tclear_len = progress->title_len_utf8 + 1 < cols ?\n-\t\t\t\t\t    cols - progress->title_len_utf8 - 1 : 0;\n-\t\t\t\tfprintf(stderr, \"%*s:%*s\\n  %*s%s\",\n-\t\t\t\t\t(int)progress->title_len_utf8,\n-\t\t\t\t\tprogress->title.buf,\n-\t\t\t\t\t(int)clear_len, \"\",\n-\t\t\t\t\t(int)progress->status_len_utf8,\n-\t\t\t\t\tprogress->status.buf, eol);\n+\t\t\tsize_t status_len_utf8 = utf8_strwidth(progress->status.buf);\n+\t\t\tsize_t progress_line_len = progress->title_len_utf8 + status_len_utf8;\n+\n+\t\t\t/*\n+\t\t\t * We're back at the beginning, so we'll\n+\t\t\t * always print out the title, unless we're\n+\t\t\t * already split, then the title is on an\n+\t\t\t * earlier line.\n+\t\t\t */\n+\t\t\tif (!progress->split)\n+\t\t\t\tfprintf(stderr, \"%*s\",\n+\t\t\t\t\t(int)(progress->title_len_utf8),\n+\t\t\t\t\tprogress->title.buf);\n+\n+\t\t\t/*\n+\t\t\t * Did the user resize the terminal and we're\n+\t\t\t * splitting this progress bar? Clear previous\n+\t\t\t * \": (X/Y) [msg]\"\n+\t\t\t */\n+\t\t\tif (!progress->split &&\n+\t\t\t    term_columns() < progress_line_len) {\n+\t\t\t\tconst char *split_prefix = counter_prefix(0);\n+\t\t\t\tconst char *unsplit_prefix = counter_prefix(1);\n+\t\t\t\tconst char *split_colon = \":\";\n \t\t\t\tprogress->split = 1;\n+\n+\t\t\t\tif (progress->last_value == -1) {\n+\t\t\t\t\t/*\n+\t\t\t\t\t * We've got no previous\n+\t\t\t\t\t * output whatsoever, so we\n+\t\t\t\t\t * were \"always split\". No\n+\t\t\t\t\t * previous status output to\n+\t\t\t\t\t * erase.\n+\t\t\t\t\t */\n+\t\t\t\t\tfprintf(stderr, \"%s\\n\", split_colon);\n+\t\t\t\t} else {\n+\t\t\t\t\tconst char *split_colon = \":\";\n+\t\t\t\t\tconst size_t split_colon_len = strlen(split_colon);\n+\n+\t\t\t\t\t/*\n+\t\t\t\t\t * Erase whatever we had, adding a\n+\t\t\t\t\t * trailing \":\" (not \": \") to indicate\n+\t\t\t\t\t * the progress on the next line.\n+\t\t\t\t\t */\n+\t\t\t\t\tfprintf(stderr, \"%s%*s\\n\", split_colon,\n+\t\t\t\t\t\t(int)(progress->status_len_utf8 - split_colon_len),\n+\t\t\t\t\t\t\"\");\n+\t\t\t\t}\n+\n+\t\t\t\t/*\n+\t\t\t\t * For the one-off switching from\n+\t\t\t\t * \"!progress->split\" to\n+\t\t\t\t * \"progress->split\" fake up the\n+\t\t\t\t * expected strbuf and replace the \":\n+\t\t\t\t * \" with a \" \".\n+\t\t\t\t *\n+\t\t\t\t * The length of the two delimiters\n+\t\t\t\t * must be the same for this trick to\n+\t\t\t\t * work.\n+\t\t\t\t */\n+\t\t\t\tif (!starts_with(progress->status.buf, split_prefix))\n+\t\t\t\t\tBUG(\"switching from already true split mode to split mode?\");\n+\n+\t\t\t\tstrbuf_splice(&progress->status, 0,\n+\t\t\t\t\t      strlen(split_prefix),\n+\t\t\t\t\t      unsplit_prefix,\n+\t\t\t\t\t      strlen(unsplit_prefix));\n+\n+\t\t\t\tfprintf(stderr, \"%*s%s\", (int)status_len_utf8,\n+\t\t\t\t\tprogress->status.buf, eol);\n \t\t\t} else {\n-\t\t\t\tfprintf(stderr, \"%*s: %*s%*s\",\n-\t\t\t\t\t(int)progress->title_len_utf8,\n-\t\t\t\t\tprogress->title.buf,\n-\t\t\t\t\t(int)progress->status_len_utf8,\n-\t\t\t\t\tprogress->status.buf,\n-\t\t\t\t\t(int)clear_len, eol);\n+\t\t\t\t/*\n+\t\t\t\t * Our current\n+\t\t\t\t * message may be larger or smaller than the\n+\t\t\t\t * last one. Either the progress bar went\n+\t\t\t\t * backards (smaller numbers), or we went back\n+\t\t\t\t * and forth with a status message.\n+\t\t\t\t */\n+\t\t\t\tsize_t clear_len = progress->status_len_utf8 > status_len_utf8\n+\t\t\t\t\t? progress->status_len_utf8 - status_len_utf8\n+\t\t\t\t\t: 0;\n+\t\t\t\tfprintf(stderr, \"%*s%*s%s\",\n+\t\t\t\t\t(int) status_len_utf8, progress->status.buf,\n+\t\t\t\t\t(int) clear_len, \"\",\n+\t\t\t\t\teol);\n \t\t\t}\n+\t\t\tprogress->status_len_utf8 = status_len_utf8;\n+\n \t\t\tif (stderr_is_foreground_fd)\n \t\t\t\tfflush(stderr);\n \t\t}\n \t\tprogress_update = 0;\n \t}\n+\tprogress->last_value = n;\n+\n }\n \n static void throughput_string(struct strbuf *buf, uint64_t total,\n@@ -303,7 +370,7 @@ void stop_progress(struct progress **p_progress)\n \t\ttrace2_region_leave(\"progress\", progress->title.buf, the_repository);\n \t}\n \n-\tstop_progress_msg(p_progress, _(\"done\"));\n+\tstop_progress_msg(p_progress, _(\", done.\"));\n }\n \n void stop_progress_msg(struct progress **p_progress, const char *msg)\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 476a31222a3..883e044fe64 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -85,12 +85,10 @@ EOF\n '\n \n test_expect_success 'progress display breaks long lines #2' '\n-\t# Note: we do not need that many spaces after the title to cover up\n-\t# the last line before breaking the progress line.\n \tsed -e \"s/Z$//\" >expect <<\\EOF &&\n Working hard.......2.........3.........4.........5.........6:   0% (1/100000)<CR>\n Working hard.......2.........3.........4.........5.........6:   0% (2/100000)<CR>\n-Working hard.......2.........3.........4.........5.........6:                   Z\n+Working hard.......2.........3.........4.........5.........6:                Z\n    10% (10000/100000)<CR>\n   100% (100000/100000)<CR>\n   100% (100000/100000), done.\n@@ -112,10 +110,8 @@ EOF\n '\n \n test_expect_success 'progress display breaks long lines #3 - even the first is too long' '\n-\t# Note: we do not actually need any spaces at the end of the title\n-\t# line, because there is no previous progress line to cover up.\n \tsed -e \"s/Z$//\" >expect <<\\EOF &&\n-Working hard.......2.........3.........4.........5.........6:                   Z\n+Working hard.......2.........3.........4.........5.........6:\n    25% (25000/100000)<CR>\n    50% (50000/100000)<CR>\n    75% (75000/100000)<CR>\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428298","messageId":"patch-15.25-160c5fe04c3-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 15/25] progress.c: pass \"is done?\" (again) to display()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:15Z","receivedAt":"2021-06-23T17:49:03Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Go back to passing a \"are we done?\" state variable to the display()\nfunction, instead of passing a string that happens to end in a newline\nfor the \", done\\n\" special-case in stop_progress().\n\nThis doesn't matter now, but is needed to display an arbitrary message\nearlier in the progress display, not just at the very end.\n\nIn a984a06a07c (nicer display of thin pack completion, 2007-11-08)\nthis code worked like this, but later on in 42e18fbf5f9 (more compact\nprogress display, 2007-10-16) we ended up with the \"const\nchar *done\". Then in d53ba841d4f (progress: assemble percentage and\ncounters in a strbuf before printing, 2019-04-05) we ended up with the\ncurrent code structure around the \"counters_sb\" strbuf.\n\nThe \"counters_sb\" is needed because when we emit a line like:\n\n    Title (1/10)<CR>\n\nWe need to know how many characters the \" (1/10)\" variable part is, so\nthat we'll emit the appropriate number of spaces to \"clear\" the line.\n\nIf we want to emit output like:\n\n    Title (1/10), some message<CR>\n\nWe'll need to stick the whole \" (1/10), some message\" part into the\nstrbuf, so that if we want to clear the message we'll know to emit:\n\n    Title (1/10), some message<CR>\n    Title (2/10)              <CR>\n\nThis didn't matter for the \", done\\n\" case because we were ending the\nprocess anyway, but in preparation for the above let's star treating\nit like any other line, and pass an \"int last_update\" to decide\nwhether the line ends with a \"\\r\" or a \"\\n\".\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 21 +++++++++++----------\n 1 file changed, 11 insertions(+), 10 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 39d7f6bd86b..44479f65921 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -25,7 +25,8 @@ static int is_foreground_fd(int fd)\n \treturn tpgrp < 0 || tpgrp == getpgid(0);\n }\n \n-static void display(struct progress *progress, uint64_t n, const char *done)\n+static void display(struct progress *progress, uint64_t n,\n+\t\t    const char *update_msg, int last_update)\n {\n \tconst char *tp;\n \tstruct strbuf *counters_sb = &progress->counters_sb;\n@@ -55,10 +56,13 @@ static void display(struct progress *progress, uint64_t n, const char *done)\n \t\tshow_update = 1;\n \t}\n \n+\tif (show_update && update_msg)\n+\t\tstrbuf_addf(counters_sb, \", %s.\", update_msg);\n+\n \tif (show_update) {\n \t\tint stderr_is_foreground_fd = is_foreground_fd(fileno(stderr));\n-\t\tif (stderr_is_foreground_fd || done) {\n-\t\t\tconst char *eol = done ? done : \"\\r\";\n+\t\tif (stderr_is_foreground_fd || update_msg) {\n+\t\t\tconst char *eol = last_update ? \"\\n\" : \"\\r\";\n \t\t\tsize_t clear_len = counters_sb->len < last_count_len ?\n \t\t\t\t\tlast_count_len - counters_sb->len + 1 :\n \t\t\t\t\t0;\n@@ -70,7 +74,7 @@ static void display(struct progress *progress, uint64_t n, const char *done)\n \t\t\tif (progress->split) {\n \t\t\t\tfprintf(stderr, \"  %s%*s\", counters_sb->buf,\n \t\t\t\t\t(int) clear_len, eol);\n-\t\t\t} else if (!done && cols < progress_line_len) {\n+\t\t\t} else if (!update_msg && cols < progress_line_len) {\n \t\t\t\tclear_len = progress->title_len + 1 < cols ?\n \t\t\t\t\t    cols - progress->title_len - 1 : 0;\n \t\t\t\tfprintf(stderr, \"%s:%*s\\n  %s%s\",\n@@ -163,13 +167,13 @@ void display_throughput(struct progress *progress, uint64_t total)\n \n \tthroughput_string(&tp->display, total, rate);\n \tif (progress->last_value != -1 && progress_update)\n-\t\tdisplay(progress, progress->last_value, NULL);\n+\t\tdisplay(progress, progress->last_value, NULL, 0);\n }\n \n void display_progress(struct progress *progress, uint64_t n)\n {\n \tif (progress)\n-\t\tdisplay(progress, n, NULL);\n+\t\tdisplay(progress, n, NULL, 0);\n }\n \n static void progress_interval(int signum)\n@@ -303,7 +307,6 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)\n \t*p_progress = NULL;\n \tif (progress->last_value != -1) {\n \t\t/* Force the last update */\n-\t\tchar *buf;\n \t\tstruct throughput *tp = progress->throughput;\n \n \t\tif (tp) {\n@@ -314,9 +317,7 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)\n \t\t\tthroughput_string(&tp->display, tp->curr_total, rate);\n \t\t}\n \t\tprogress_update = 1;\n-\t\tbuf = xstrfmt(\", %s.\\n\", msg);\n-\t\tdisplay(progress, progress->last_value, buf);\n-\t\tfree(buf);\n+\t\tdisplay(progress, progress->last_value, msg, 1);\n \t}\n \tclear_progress_signal(progress);\n \tstrbuf_release(&progress->counters_sb);\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428299","messageId":"patch-18.25-e21fc66623f-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 18/25] progress.c: emit progress on first signal, show \"stalled\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:18Z","receivedAt":"2021-06-23T17:49:05Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Ever since the progress.c code was added in 96a02f8f6d2 (common\nprogress display support, 2007-04-18) we have been driven purely by\ncalls to the display() function (via the public display_progress()),\nor via stop_progress(). Even though we got a signal and invoked\nprogress_interval() that function would not actually emit progress\noutput for us.\n\nThus in cases like \"git gc\" we don't emit any \"Enumerating Objects\"\noutput until we get past the setup code, and start enumerating\nobjects, we'll now (at least on my laptop) show output earlier, and\nemit a \"stalled\" message before we start the count.\n\nBut more generally, this is a first step towards never showing a\nhanging progress bar from the user's perspective. If we're truly\ntaking a very long time with one item we can show some spinner that we\nupdate every time we get a signal. We don't right now, and only\nspecial-case the most common case of hanging before we get to the\nfirst item.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c                  |  7 +++++\n t/t0500-progress-display.sh | 63 ++++++++++++++++++++++++++++++++++---\n 2 files changed, 66 insertions(+), 4 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 6c4038df791..35847d3a7f2 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -255,6 +255,13 @@ void display_progress(struct progress *progress, uint64_t n)\n static void progress_interval(int signum)\n {\n \tprogress_update = 1;\n+\n+\tif (global_progress->last_value != -1)\n+\t\treturn;\n+\n+\tdisplay(global_progress, 0, _(\", stalled.\"), 0);\n+\tprogress_update = 1;\n+\treturn;\n }\n \n void test_progress_force_update(void)\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 883e044fe64..bc458cfc28b 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -15,7 +15,8 @@ test_expect_success 'setup COLUMNS' '\n \n test_expect_success 'simple progress display' '\n \tcat >expect <<-\\EOF &&\n-\tWorking hard: 1<CR>\n+\tWorking hard: 0, stalled.<CR>\n+\tWorking hard: 1          <CR>\n \tWorking hard: 2<CR>\n \tWorking hard: 5<CR>\n \tWorking hard: 5, done.\n@@ -60,6 +61,57 @@ test_expect_success 'progress display with total' '\n \ttest_cmp expect out\n '\n \n+test_expect_success 'stalled progress display' '\n+\tcat >expect <<-\\EOF &&\n+\tWorking hard:   0% (0/3), stalled.<CR>\n+\tWorking hard:  33% (1/3)          <CR>\n+\tWorking hard:  66% (2/3)<CR>\n+\tWorking hard: 100% (3/3)<CR>\n+\tWorking hard: 100% (3/3), done.\n+\tEOF\n+\n+\tcat >in <<-\\EOF &&\n+\tstart 3\n+\tsignal\n+\tsignal\n+\tsignal\n+\tprogress 1\n+\tsignal\n+\tupdate\n+\tsignal\n+\tprogress 2\n+\tupdate\n+\tprogress 3\n+\tstop\n+\tEOF\n+\tSTALLED=1 test-tool progress <in 2>stderr &&\n+\n+\tshow_cr <stderr >out &&\n+\ttest_cmp expect out\n+'\n+\n+test_expect_success 'progress display breaks long lines #0, stalled' '\n+\tsed -e \"s/Z$//\" >expect <<\\EOF &&\n+Working hard.......2.........3.........4.........5.........6.........7:\n+    0% (0/100), stalled.<CR>\n+    1% (1/100)          <CR>\n+   50% (50/100)<CR>\n+   50% (50/100), done.\n+EOF\n+\n+\tcat >in <<-\\EOF &&\n+\tstart 100 Working hard.......2.........3.........4.........5.........6.........7\n+\tsignal\n+\tprogress 1\n+\tprogress 50\n+\tstop\n+\tEOF\n+\ttest-tool progress <in 2>stderr &&\n+\n+\tshow_cr <stderr >out &&\n+\ttest_cmp expect out\n+'\n+\n test_expect_success 'progress display breaks long lines #1' '\n \tsed -e \"s/Z$//\" >expect <<\\EOF &&\n Working hard.......2.........3.........4.........5.........6:   0% (100/100000)<CR>\n@@ -183,7 +235,8 @@ test_expect_success 'progress shortens - crazy caller' '\n \n test_expect_success 'progress display with throughput' '\n \tcat >expect <<-\\EOF &&\n-\tWorking hard: 10<CR>\n+\tWorking hard: 0, stalled.<CR>\n+\tWorking hard: 10         <CR>\n \tWorking hard: 20, 200.00 KiB | 100.00 KiB/s<CR>\n \tWorking hard: 30, 300.00 KiB | 100.00 KiB/s<CR>\n \tWorking hard: 40, 400.00 KiB | 100.00 KiB/s<CR>\n@@ -241,7 +294,8 @@ test_expect_success 'progress display with throughput and total' '\n \n test_expect_success 'cover up after throughput shortens' '\n \tcat >expect <<-\\EOF &&\n-\tWorking hard: 1<CR>\n+\tWorking hard: 0, stalled.<CR>\n+\tWorking hard: 1          <CR>\n \tWorking hard: 2, 800.00 KiB | 400.00 KiB/s<CR>\n \tWorking hard: 3, 1.17 MiB | 400.00 KiB/s  <CR>\n \tWorking hard: 4, 1.56 MiB | 400.00 KiB/s<CR>\n@@ -272,7 +326,8 @@ test_expect_success 'cover up after throughput shortens' '\n \n test_expect_success 'cover up after throughput shortens a lot' '\n \tcat >expect <<-\\EOF &&\n-\tWorking hard: 1<CR>\n+\tWorking hard: 0, stalled.<CR>\n+\tWorking hard: 1          <CR>\n \tWorking hard: 2, 1000.00 KiB | 1000.00 KiB/s<CR>\n \tWorking hard: 3, 3.00 MiB | 1.50 MiB/s      <CR>\n \tWorking hard: 3, 3.00 MiB | 1024.00 KiB/s, done.\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428300","messageId":"patch-20.25-80a991d019a-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 20/25] midx: don't provide a total for QSORT() progress","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:20Z","receivedAt":"2021-06-23T17:49:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The quicksort algorithm can be anywhere between O(n) and O(n^2), so\nproviding a \"num objects\" as a total means that in some cases we're\ngoing to go past 100%.\n\nThis fixes a logic error in 5ae18df9d8e (midx: during verify group\nobjects by packfile to speed verification, 2019-03-21), which in turn\nseems to have been diligently copied from my own logic error in the\ncommit-graph.c code, see 890226ccb57 (commit-graph write: add\nitermediate progress, 2019-01-19).\n\nThat commit-graph code of mine was removed in\n1cbdbf3bef7 (commit-graph: drop count_distinct_commits() function,\n2020-12-07), so we don't need to fix that too.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n midx.c | 3 +--\n 1 file changed, 1 insertion(+), 2 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex d80e68998b8..9f1b4018c1c 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1265,8 +1265,7 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \t}\n \n \tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_progress(_(\"Sorting objects by packfile\"),\n-\t\t\t\t\t  m->num_objects);\n+\t\tprogress = start_progress(_(\"Sorting objects by packfile\"), 0);\n \tdisplay_progress(progress, 0); /* TODO: Measure QSORT() progress */\n \tQSORT(pairs, m->num_objects, compare_pair_pos_vs_id);\n \tstop_progress(&progress);\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428301","messageId":"patch-19.25-26a6e93717e-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 19/25] commit-graph: fix bogus counter in \"Scanning merged commits\" progress line","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:19Z","receivedAt":"2021-06-23T17:49:11Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: SZEDER Gábor <szeder.dev@gmail.com>\n\nThe final value of the counter of the \"Scanning merged commits\"\nprogress line is always one less than its expected total, e.g.:\n\n  Scanning merged commits:  83% (5/6), done.\n\nThis happens because while iterating over an array the loop variable\nis passed to display_progress() as-is, but while C arrays (and thus\nthe loop variable) start at 0 and end at N-1, the progress counter\nmust end at N.  This causes the failures of the tests\n'fetch.writeCommitGraph' and 'fetch.writeCommitGraph with submodules'\nin 't5510-fetch.sh' when run with GIT_TEST_CHECK_PROGRESS=1.\n\nFix this by passing 'i + 1' to display_progress(), like most other\ncallsites do.\n\nSigned-off-by: SZEDER Gábor <szeder.dev@gmail.com>\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n commit-graph.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 2bcb4e0f89e..3181906368d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -2096,7 +2096,7 @@ static void sort_and_scan_merged_commits(struct write_commit_graph_context *ctx)\n \n \tctx->num_extra_edges = 0;\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n-\t\tdisplay_progress(ctx->progress, i);\n+\t\tdisplay_progress(ctx->progress, i + 1);\n \n \t\tif (i && oideq(&ctx->commits.list[i - 1]->object.oid,\n \t\t\t  &ctx->commits.list[i]->object.oid)) {\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428302","messageId":"patch-22.25-e7f8314867d-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 22/25] progress.c: add a stop_progress_early() function","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:22Z","receivedAt":"2021-06-23T17:49:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"In cases where we error out during processing or otherwise miss\ninitial \"total\" estimate we'll still want to show a \"done\" message and\nend our trace2 region, but it won't be true that our total ==\nlast_update at the end.\n\nSo let's add a \"last_update\" and this stop_progress_early() function\nto handle that edge case, this will be used in a subsequent commit.\n\nWe could also use a total=0 in such cases, but that would make the\nprogress output worse for the common non-erroring case. Let's instead\nnote that we didn't reach the total count, and snap the progress bar\nto \"100%, done\" at the end.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 20 ++++++++++++++++++++\n progress.h |  2 ++\n 2 files changed, 22 insertions(+)\n\ndiff --git a/progress.c b/progress.c\nindex 35847d3a7f2..c1cb01ba975 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -40,6 +40,8 @@ static void display(struct progress *progress, uint64_t n,\n \tconst char *tp;\n \tint show_update = 0;\n \n+\tprogress->last_update = n;\n+\n \tif (progress->delay && (!progress_update || --progress->delay))\n \t\treturn;\n \n@@ -413,3 +415,21 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)\n \tfree(progress->throughput);\n \tfree(progress);\n }\n+\n+void stop_progress_early(struct progress **p_progress)\n+{\n+\tstruct progress *progress;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n+\tif (!p_progress)\n+\t\tBUG(\"don't provide NULL to stop_progress_early\");\n+\tprogress = *p_progress;\n+\tif (!progress)\n+\t\treturn;\n+\n+\tstrbuf_addf(&sb, _(\", done at %\"PRIuMAX\" items, expected %\"PRIuMAX\".\"),\n+\t\t    progress->total, progress->last_update);\n+\tprogress->total = progress->last_update;\n+\tstop_progress_msg(p_progress, sb.buf);\n+\tstrbuf_release(&sb);\n+}\ndiff --git a/progress.h b/progress.h\nindex ba38447d104..5c5d027d1a0 100644\n--- a/progress.h\n+++ b/progress.h\n@@ -23,6 +23,7 @@ struct progress {\n \tstruct strbuf status;\n \tsize_t status_len_utf8;\n \n+\tuint64_t last_update;\n \tuint64_t last_value;\n \tuint64_t total;\n \tunsigned last_percent;\n@@ -56,5 +57,6 @@ struct progress *start_delayed_sparse_progress(const char *title,\n \t\t\t\t\t       uint64_t total);\n void stop_progress(struct progress **progress);\n void stop_progress_msg(struct progress **progress, const char *msg);\n+void stop_progress_early(struct progress **p_progress);\n \n #endif\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428303","messageId":"patch-21.25-9ed1d5a39c2-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 21/25] entry: show finer-grained counter in \"Filtering content\" progress line","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:21Z","receivedAt":"2021-06-23T17:49:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: SZEDER Gábor <szeder.dev@gmail.com>\n\nThe \"Filtering content\" progress in entry.c:finish_delayed_checkout()\nis unusual because of how it calculates the progress count and because\nit shows the progress of a nested loop.  It works basically like this:\n\n  start_delayed_progress(p, nr_of_paths_to_filter)\n  for_each_filter {\n      display_progress(p, nr_of_paths_to_filter - nr_of_paths_still_left_to_filter)\n      for_each_path_handled_by_the_current_filter {\n          checkout_entry()\n      }\n  }\n  stop_progress(p)\n\nThere are two issues with this approach:\n\n  - The work done by the last filter (or the only filter if there is\n    only one) is never counted, so if the last filter still has some\n    paths to process, then the counter shown in the \"done\" progress\n    line will not match the expected total.\n\n    This would cause a BUG() in an upcoming change that adds an\n    assertion checking if the \"total\" at the end matches the last\n    progress bar update..\n\n    This is because both use only one filter.  (The test 'delayed\n    checkout in process filter' uses two filters but the first one\n    does all the work, so that test already happens to succeed even\n    with such an assertion.)\n\n  - The progress counter is updated only once per filter, not once per\n    processed path, so if a filter has a lot of paths to process, then\n    the counter might stay unchanged for a long while and then make a\n    big jump (though the user still gets a sense of progress, because\n    we call display_throughput() after each processed path to show the\n    amount of processed data).\n\nMove the display_progress() call to the inner loop, right next to that\ncheckout_entry() call that does the hard work for each path, and use a\ndedicated counter variable that is incremented upon processing each\npath.\n\nAfter this change the 'invalid file in delayed checkout' in\n't0021-conversion.sh' would succeed with the future BUG() assertion\ndiscussed above but the 'missing file in delayed checkout' test would\nstill fail, because its purposefully buggy filter doesn't process any\npaths, so we won't execute that inner loop at all (this will be fixed\nin a subsequent commit).\n\nSigned-off-by: SZEDER Gábor <szeder.dev@gmail.com>\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n entry.c | 7 +++----\n 1 file changed, 3 insertions(+), 4 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex 711ee0693c7..bc4b8fcc980 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -162,7 +162,7 @@ static int remove_available_paths(struct string_list_item *item, void *cb_data)\n int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n {\n \tint errs = 0;\n-\tunsigned delayed_object_count;\n+\tunsigned processed_paths = 0;\n \toff_t filtered_bytes = 0;\n \tstruct string_list_item *filter, *path;\n \tstruct progress *progress;\n@@ -172,12 +172,10 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n \t\treturn errs;\n \n \tdco->state = CE_RETRY;\n-\tdelayed_object_count = dco->paths.nr;\n-\tprogress = start_delayed_progress(_(\"Filtering content\"), delayed_object_count);\n+\tprogress = start_delayed_progress(_(\"Filtering content\"), dco->paths.nr);\n \twhile (dco->filters.nr > 0) {\n \t\tfor_each_string_list_item(filter, &dco->filters) {\n \t\t\tstruct string_list available_paths = STRING_LIST_INIT_NODUP;\n-\t\t\tdisplay_progress(progress, delayed_object_count - dco->paths.nr);\n \n \t\t\tif (!async_query_available_blobs(filter->string, &available_paths)) {\n \t\t\t\t/* Filter reported an error */\n@@ -224,6 +222,7 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n \t\t\t\tce = index_file_exists(state->istate, path->string,\n \t\t\t\t\t\t       strlen(path->string), 0);\n \t\t\t\tif (ce) {\n+\t\t\t\t\tdisplay_progress(progress, ++processed_paths);\n \t\t\t\t\terrs |= checkout_entry(ce, state, NULL, nr_checkouts);\n \t\t\t\t\tfiltered_bytes += ce->ce_stat_data.sd_size;\n \t\t\t\t\tdisplay_throughput(progress, filtered_bytes);\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428304","messageId":"patch-23.25-e0317d7c780-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[PATCH 23/25] entry: deal with unexpected \"Filtering content\" total","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:23Z","receivedAt":"2021-06-23T17:49:16Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The \"Filtering content\" end total does not match the expected total in\ncases such as the 'missing file in delayed checkout' test in\n't0021-conversion.sh'.\n\nIf we encounter errors we can't accurately estimate the end state of\nthe progress bar. This is because the test involves a purposefully\nbuggy filter process that doesn't process any paths, so the progress\ncounter doesn't have a chance to reach the expected total.\n\nSee the preceding commit for why we'd want a stop_progress_early() in\nthis case, as opposed to leaking memory here, or not providing a\n\"total\" estimate to begin with.\n\nSigned-off-by: SZEDER Gábor <szeder.dev@gmail.com>\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n entry.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/entry.c b/entry.c\nindex bc4b8fcc980..e79a13daa51 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -232,7 +232,10 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n \t\t}\n \t\tstring_list_remove_empty_items(&dco->filters, 0);\n \t}\n-\tstop_progress(&progress);\n+\tif (!errs && !dco->paths.nr)\n+\t\tstop_progress(&progress);\n+\telse\n+\t\tstop_progress_early(&progress);\n \tstring_list_clear(&dco->filters, 0);\n \n \t/* At this point we should not have any delayed paths anymore. */\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428305","messageId":"patch-24.25-37b3d1e4d9d-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[RFC/PATCH 24/25] progress: assert last update in stop_progress()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:24Z","receivedAt":"2021-06-23T17:49:19Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: SZEDER Gábor <szeder.dev@gmail.com>\n\nWe had to fix a couple of buggy progress lines in the past, where the\nprogress counter's final value didn't match the expected total [1],\ne.g.:\n\n  Expanding reachable commits in commit graph: 138606% (824706/595), done.\n  Writing out commit graph in 3 passes: 166% (4187845/2512707), done.\n\nLet's do better, and, instead of waiting for someone to notice such\nissues by mere chance, start verifying progress counters in the test\nsuite. Let's track what the last display_progress() value was, and if\nit doesn't match the total at the end invoke BUG().\n\nWe need to introduce a \"last_update\" distinct from \"last_value\" for\nthis, since the \"last_value\" really means \"last displayed value\", and\nthe logic in display() relies on it having those semantics.\n\nUsing the \"last_value\" would also leave us with a subtle case where\nthis assertion wouldn't catch broken API uses, as an earlier version\nof this change did.\n\nEven if that was not the case we couldn't rely on it for the purposes\nof this assertion. In the case of a delayed progress the variable\nholding the value of the progress counter\n('progress->last_value') is only updated after that delay is up, and,\nconsequently, we can't compare the progress counter with the expected\ntotal in stop_progress() in these cases. Thus this check will cover\nprogress lines that are too fast to be shown, because the repositories\nused in our tests are tiny and most of our progress lines are delayed.\n\nWhat it can't cover is code that doesn't start the progress bar at\nall, e.g. due to its own isatty() check, so progress that is only\nstarted and shown when standard error is not a terminal won't be\ncovered by our tests.\n\n[1] c4ff24bbb3 (commit-graph.c: display correct number of chunks when\n                writing, 2021-02-24)\n    1cbdbf3bef (commit-graph: drop count_distinct_commits() function,\n                2020-12-07), though this didn't actually fixed, but\n                instead removed a buggy progress line.\n    150cd3b61d (commit-graph: fix \"Writing out commit graph\" progress\n                counter, 2020-07-09)\n    67fa6aac5a (commit-graph: don't show progress percentages while\n                expanding reachable commits, 2019-09-07)\n    531e6daa03 (prune-packed: advanced progress even for non-existing\n                fan-out directories, 2009-04-27)\n\nSigned-off-by: SZEDER Gábor <szeder.dev@gmail.com>\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n\nWARNING: I belive this is subtly buggy, see the discussion in the\ncover letter. It needs more fixes of the progress.c API usage in\nvarious places before being ready.\n\n progress.c                  |  8 ++++++++\n t/t0500-progress-display.sh | 30 +++++++++++++++++++++++++++++-\n 2 files changed, 37 insertions(+), 1 deletion(-)\n\ndiff --git a/progress.c b/progress.c\nindex c1cb01ba975..40043bf6601 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -325,6 +325,7 @@ static struct progress *start_progress_delay(const char *title, uint64_t total,\n \n \tprogress->total = total;\n \tprogress->last_value = -1;\n+\tprogress->last_update = -1;\n \tprogress->last_percent = -1;\n \tprogress->delay = delay;\n \tprogress->throughput = NULL;\n@@ -393,6 +394,13 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)\n \tif (!progress)\n \t\treturn;\n \t*p_progress = NULL;\n+\n+\tif (progress->total &&\n+\t    progress->total != progress->last_update)\n+\t\tBUG(\"total progress does not match for \\\"%*s\\\": expected: %\"PRIuMAX\" got: %\"PRIuMAX,\n+\t\t    (int)(progress->status_len_utf8), progress->title.buf,\n+\t\t    (uintmax_t)progress->total,\n+\t\t    (uintmax_t)progress->last_update);\n \tif (progress->last_value != -1) {\n \t\t/* Force the last update */\n \t\tstruct throughput *tp = progress->throughput;\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex bc458cfc28b..3f00e52ce46 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -96,7 +96,8 @@ Working hard.......2.........3.........4.........5.........6.........7:\n     0% (0/100), stalled.<CR>\n     1% (1/100)          <CR>\n    50% (50/100)<CR>\n-   50% (50/100), done.\n+  100% (100/100)<CR>\n+  100% (100/100), done.\n EOF\n \n \tcat >in <<-\\EOF &&\n@@ -104,6 +105,7 @@ EOF\n \tsignal\n \tprogress 1\n \tprogress 50\n+\tprogress 100\n \tstop\n \tEOF\n \ttest-tool progress <in 2>stderr &&\n@@ -423,4 +425,30 @@ test_expect_success 'BUG: start two concurrent progress bars' '\n \tgrep -E \"^BUG: .*: should have no global_progress in set_progress_signal\\(\\)$\" stderr\n '\n \n+test_expect_success 'BUG: display_progress() goes past declared \"total\"' '\n+\tcat >in <<-\\EOF &&\n+\tstart 3\n+\tprogress 1\n+\tprogress 2\n+\tprogress 4\n+\tstop\n+\tEOF\n+\n+\ttest_must_fail test-tool progress <in 2>stderr &&\n+\tgrep \"BUG:.*total progress does not match\" stderr\n+'\n+\n+test_expect_success 'BUG: display_progress() does not reach declared \"total\"' '\n+\tcat >in <<-\\EOF &&\n+\tstart 5\n+\tprogress 1\n+\tprogress 2\n+\tprogress 4\n+\tstop\n+\tEOF\n+\n+\ttest_must_fail test-tool progress <in 2>stderr &&\n+\tgrep \"BUG:.*total progress does not match\" stderr\n+'\n+\n test_done\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428306","messageId":"patch-25.25-099e9c019a9-20210623T155626Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"[RFC/PATCH 25/25] progress: assert counting upwards in display()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T17:48:25Z","receivedAt":"2021-06-23T17:49:19Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: SZEDER Gábor <szeder.dev@gmail.com>\n\nWe had to fix a buggy progress line recently, where the progress\ncounter counted backwards, see 8e118e8490 (pack-objects: update\n\"nr_seen\" progress based on pack-reused count, 2021-04-11).\n\nLet's add a BUG(...) assertion that makes use of the \"last_update\"\nvalue to make sure this doesn't happen again, i.e.  trigger a BUG()\nwhen the counter passed to display_progress() is smaller than the\nprevious value.\n\nNote that we allow subsequent display_progress() calls with the same\ncounter value, because:\n\n  - Strictly speaking, it's not wrong to do so.\n\n  - Forbidding it might make the code calling display_progress() more\n    complex; I suspect that would be the case with e.g. the \"Updating\n    index flags\" progress line in 'unpack-trees.c', where the counter\n    is increased in recursive function calls.\n\n  - We would need to special case the internal display() call in\n    stop_progress_msg(), because it uses the same counter value as the\n    last display_progress() call, which would trigger this BUG().\n\n't0500-progress-display.sh' countains a few tests that check how\nshortened progress lines are covered up, and one of them ('progress\nshortens - crazy caller') shortens the progress line by counting\nbackwards.  From now on that test would trigger this BUG(), so remove\nit; the other test cases cover shortening progress lines sufficiently.\n\nSigned-off-by: SZEDER Gábor <szeder.dev@gmail.com>\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n\nWARNING: I belive this is subtly buggy, see the discussion in the\ncover letter. It needs more fixes of the progress.c API usage in\nvarious places before being ready.\n\n progress.c                  |  2 ++\n t/t0500-progress-display.sh | 36 ++++++++++++------------------------\n 2 files changed, 14 insertions(+), 24 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 40043bf6601..7b59006c7c4 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -40,6 +40,8 @@ static void display(struct progress *progress, uint64_t n,\n \tconst char *tp;\n \tint show_update = 0;\n \n+\tif (progress->last_update != -1 && n < progress->last_update)\n+\t\tBUG(\"counting backwards with display_progress()\");\n \tprogress->last_update = n;\n \n \tif (progress->delay && (!progress_update || --progress->delay))\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 3f00e52ce46..de59a757f86 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -211,30 +211,6 @@ EOF\n \ttest_cmp expect out\n '\n \n-# Progress counter goes backwards, this should not happen in practice.\n-test_expect_success 'progress shortens - crazy caller' '\n-\tcat >expect <<-\\EOF &&\n-\tWorking hard:  10% (100/1000)<CR>\n-\tWorking hard:  20% (200/1000)<CR>\n-\tWorking hard:   0% (1/1000)  <CR>\n-\tWorking hard: 100% (1000/1000)<CR>\n-\tWorking hard: 100% (1000/1000), done.\n-\tEOF\n-\n-\tcat >in <<-\\EOF &&\n-\tstart 1000\n-\tprogress 100\n-\tprogress 200\n-\tprogress 1\n-\tprogress 1000\n-\tstop\n-\tEOF\n-\ttest-tool progress <in 2>stderr &&\n-\n-\tshow_cr <stderr >out &&\n-\ttest_cmp expect out\n-'\n-\n test_expect_success 'progress display with throughput' '\n \tcat >expect <<-\\EOF &&\n \tWorking hard: 0, stalled.<CR>\n@@ -451,4 +427,16 @@ test_expect_success 'BUG: display_progress() does not reach declared \"total\"' '\n \tgrep \"BUG:.*total progress does not match\" stderr\n '\n \n+test_expect_success 'BUG: display_progres() counting backwards' '\n+\tcat >in <<-\\EOF &&\n+\tstart 3\n+\tprogress 1\n+\tprogress 2\n+\tprogress 1\n+\tEOF\n+\n+\ttest_must_fail test-tool progress <in 2>stderr &&\n+\tgrep \"BUG:.*counting backwards\" stderr\n+'\n+\n test_done\n-- \n2.32.0.599.g3967b4fa4ac\n\n"},{"id":"428307","messageId":"00fb01d76859$8a6ebc50$9f4c34f0$@nexbridge.com","threadId":"55972","inReplyTo":"cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com","subject":"RE: [PATCH 00/25] progress.c: various fixes + SZEDER's RFC code","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2021-06-23T17:59:35Z","receivedAt":"2021-06-23T17:59:49Z","isPatch":true,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On June 23, 2021 1:48 PM, Ævar Arnfjörð Bjarmason wrote:\n>> On Mon, Jun 21, 2021 at 02:59:53AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>>>\n>>> On Sun, Jun 20 2021, SZEDER Gábor wrote:\n>>>\n>>> > Splitting off from:\n>>> >\n>>> >\n>>> > https://public-inbox.org/git/cover-0.2-0000000000-20210607T144206Z-\n>>> > avarab@gmail.com/T/#me5d3176914d4268fd9f2a96fc63f4e41beb26bd6\n>>> >\n>>> > On Tue, Jun 08, 2021 at 06:14:42PM +0200, René Scharfe wrote:\n>>> >> I wonder (only in a semi-curious way, though) if we can detect\n>>> >> off-by-one errors by adding an assertion to display_progress()\n>>> >> that requires the first update to have the value 0, and in\n>>> >> stop_progress() one that requires the previous display_progress()\n>>> >> call to have a value equal to the total number of work items.  Not\n>>> >> sure it'd be worth the hassle..\n>>> >\n>>> > I fixed and reported a number of bogus progress lines in the past,\n>>> > the last one during v2.31.0-rc phase, so I've looked into whether\n>>> > progress counters could be automatically validated in our tests,\n>>> > and came up with these patches a few months ago.  It turned out\n>>> > that progress counters can be checked easily and transparently in\n>>> > case of progress lines that are shown in the tests, i.e. that are\n>>> > shown even when stderr is not a terminal or are forced with\n>>> > '--progress'.  (In other cases it's still fairly easy but not quite\n>>> > transparent, as I think we need changes to the progress API; more\n>>> > on that later in a separate\n>>> > series.)\n>>>\n>>> I've also been working on some progress.[ch] patches that are mostly\n>>> finished, and I'm some 20 patches in at the moment. I wasn't sure\n>>> about whether to send an alternate 20-patch \"let's do this (mostly) instead?\"\n>>> series, hence this message.\n>>>\n>>> Much of what you're doing here becomes easier after that series, e.g.\n>>> your global process struct in 2/7 is something I ended up\n>>> implementing as part of a general feature to allow progress to be\n>>> driven by either display_progress() *or* the signal handler itself.\n>>\n>> It's difficult to know who should rebase onto who without seeing one\n>> half of the patches.\n>\n>I was sort of hoping he'd take me word for it, but here it is. Don't say I didn't warn you :)\n>\n>> I couldn't find a link to them anywhere (even if they are only\n>> available in your fork in a pre-polished state) despite looking, but\n>> my apologies if they are available and I'm just missing them.\n>\n>FWIW it's avar-szeder/progress-bar-assertions in https://github.com/avar/git.git, that repo contains various functioning and not-so-\n>functioning code.\n>\n>https://github.com/avar/git/tree/meta/ is my version of the crappy scripts we probably all have some version of for building my own git,\n>things that are uncommented in series.conf is what I build my own git from.\n>\n>> In general, I think that these patches are clear and are helpful in\n>> pinning down issues with the progress API (which I have made a hadnful\n>> of times in the past), so I would be happy to see them picked up.\n>\n>Here's all 25 patches (well, around 20 before) that I had queued up locally and fixed up a bit.\n>\n>The 01/25 is something I submitted already as https://lore.kernel.org/git/patch-1.1-cba5d88ca35-20210621T070114Z-avarab@gmail.com;\n>hoping to get this in incrementally.\n>\n>The 12/25 is my own version of that \"global progress struct, 11/25 is the first of many bugs SZEDER missed in his :)\n>\n>18/25 is the first step of the UI I was going for, the signal handler can now drive the progress bar, so e.g. during \"git gc\" we show (at least\n>for me, on git.git), a \"stalled\" message just before we start the actual count of \"Enumerating Objects\".\n>\n>After that was in I was planning on adding config-driven support to show a \"spinner\" when we stalled in that way, config-driven because\n>you could just scrape e.g. https://github.com/sindresorhus/cli-spinners/blob/main/spinners.json\n>into your own config. See\n>https://jsfiddle.net/sindresorhus/2eLtsbey/embedded/result/ :)\n>\n>19-23/25 is my grabbing of SZEDER's patches that I'm comfortable labeling as \"PATCH\", I think they work, but no BUG() assertions yet. I\n>left out the GIT_TEST_CHECK_PROGRESS parts, since my earlier works set things up to do any BUG() we trust by default.\n>\n>22/25 is what I think we should do instead of SZEDER's 6/7\n>(http://lore.kernel.org/git/20210620200303.2328957-7-szeder.dev@gmail.com)\n>I don't think this \"our total doesn't match at the end\" is something we should always BUG() on, for reasons explained there.\n>\n>I am sympathetic to doing it by default though, hence the\n>stop_progress_early() API, that's there to allow select callers to bypass his BUG(...) assertion.\n>\n>24/25 and 25/25 are \"RFC\" and a rebased+modified version of SZEDER's\n>BUG(...) assertions.\n>\n>His series passes the test suite, but actually severely break things things. It'll make e.g. \"git commit-graph write\" BUG(...) out. The reason\n>the tests don't catch it is because we have a blind spot in the tests.\n>\n>Namely, that most things that use the progress bar API use isatty() to check if they should start_progress(). If you run the tests as e.g.\n>(better ways to do this, especially in parallel, most welcome):\n>\n>    for t in t[0-9]*.sh; do if ! ./$t -vixd; then echo $t bad; break; fi; done\n>\n>You can discover various things that his series BUG()'s on, I fixed a couple of those myself, it's an early part of this series.\n>\n>But we'll still have various untested for BUG()'s even then, this is because you *also* have to have the test actually emit a \"naked\"\n>progress bar on stderr, if the test itself e.g. pipes fd 2 to a file it won't work.\n>\n>I created a shitty-and-mostly-broken throwaway change to search-replace all the guards of \"start_progress(...)\" to run unconditionally, and\n>convert all the \"delayed\" to the non-delayed version. That'll find even more BUG()'s where SZEDER's series still needs to be fixed (and also\n>some unrelated segfaults, I gave up on it soon after).\n>\n>Even if we fix that I wouldn't trust it, because a lot of the progress bars we have depend on the size and shape of the data we're\n>processing, e.g. the bug I fixed in 11/25. If people find this BUG() approach worth pursuing I think it would be better to make it an opt-in\n>flag we convert one caller at a time to.\n>\n>For some it's really clear that we could assert it, for others such as the commit-graph it's much more subtle, we're in some callback after\n>setting a \"total\", that callback does a \"break\", \"continue\" etc. in various places, all depending on repository data.\n>\n>It's not easy to reason about that and be certain that we can hold to the estimate. If we get it wrong someone's repo in the wild won't fully\n>GC because of the overly eager BUG().\n>\n>If SZEDER wants to pursue it I think it'll be easier on top of this series, but personally I really don't see the point of spending effort on it.\n>\n>We should really be going in the other direction, of having more fuzzy ETAs, not less.\n>\n>E.g. we often have enough data at the start of \"Enumerating Objects\"\n>to give a good-enough target value, that it's 5-10% off isn't really the point, but that the user looking at it sees something better than a\n>dumb count-up, and can instead see that they'll probably be looking at it for about a minute. Now our API is to give no ETA/target if we're\n>not 100% sure, it's not good UX.\n>\n>So trying to get the current exact count/exact percentage right seems like a distraction to me in the longer term. If anything we should\n>just be rounding those numbers, showing fuzzy ETAs instead of percentages if we can etc.\n>\n>SZEDER Gábor (4):\n>  commit-graph: fix bogus counter in \"Scanning merged commits\" progress\n>    line\n>  entry: show finer-grained counter in \"Filtering content\" progress line\n>  progress: assert last update in stop_progress()\n>  progress: assert counting upwards in display()\n>\n>Ævar Arnfjörð Bjarmason (21):\n>  progress.c tests: fix breakage with COLUMNS != 80\n>  progress.c tests: make start/stop verbs on stdin\n>  progress.c tests: test some invalid usage\n>  progress.c tests: add a \"signal\" verb\n>  progress.c: move signal handler functions lower\n>  progress.c: call progress_interval() from progress_test_force_update()\n>  progress.c: stop eagerly fflush(stderr) when not a terminal\n>  progress.c: add temporary variable from progress struct\n>  midx perf: add a perf test for multi-pack-index\n>  progress.c: remove the \"sparse\" mode nano-optimization\n>  pack-bitmap-write.c: add a missing stop_progress()\n>  progress.c: add & assert a \"global_progress\" variable\n>  progress.[ch]: move the \"struct progress\" to the header\n>  progress.[ch]: move test-only code away from \"extern\" variables\n>  progress.c: pass \"is done?\" (again) to display()\n>  progress.[ch]: convert \"title\" to \"struct strbuf\"\n>  progress.c: refactor display() for less confusion, and fix bug\n>  progress.c: emit progress on first signal, show \"stalled\"\n>  midx: don't provide a total for QSORT() progress\n>  progress.c: add a stop_progress_early() function\n>  entry: deal with unexpected \"Filtering content\" total\n>\n> cache.h                          |   1 -\n> commit-graph.c                   |   2 +-\n> csum-file.h                      |   2 -\n> entry.c                          |  12 +-\n> midx.c                           |  25 +-\n> pack-bitmap-write.c              |   1 +\n> pack.h                           |   1 -\n> parallel-checkout.h              |   1 -\n> progress.c                       | 391 ++++++++++++++++++-------------\n> progress.h                       |  50 +++-\n> reachable.h                      |   1 -\n> t/helper/test-progress.c         |  54 +++--\n> t/perf/p5319-multi-pack-index.sh |  21 ++\n> t/t0500-progress-display.sh      | 247 ++++++++++++++-----\n> 14 files changed, 537 insertions(+), 272 deletions(-)  create mode 100755 t/perf/p5319-multi-pack-index.sh\n\nIs there provision for disabling progress on a per-command basis? My use case is specifically in a CI/CD script, being able to suppress progress handling. The current Jenkins plugin does not appear to have provision for hooking into a mechanism, which makes things get a bit wonky when a job runs with a pseudo-tty (as provided by Jenkins through SSH/RMI).\n-Randall\n\n"},{"id":"428325","messageId":"87lf701h6k.fsf@evledraar.gmail.com","threadId":"55972","inReplyTo":"00fb01d76859$8a6ebc50$9f4c34f0$@nexbridge.com","subject":"Re: [PATCH 00/25] progress.c: various fixes + SZEDER's RFC code","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-23T20:01:55Z","receivedAt":"2021-06-23T20:04:25Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jun 23 2021, Randall S. Becker wrote:\n\n> On June 23, 2021 1:48 PM, Ævar Arnfjörð Bjarmason wrote:\n>>> On Mon, Jun 21, 2021 at 02:59:53AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>>>>\n>>>> On Sun, Jun 20 2021, SZEDER Gábor wrote:\n>>>>\n>>>> > Splitting off from:\n>>>> >\n>>>> >\n>>>> > https://public-inbox.org/git/cover-0.2-0000000000-20210607T144206Z-\n>>>> > avarab@gmail.com/T/#me5d3176914d4268fd9f2a96fc63f4e41beb26bd6\n>>>> >\n>>>> > On Tue, Jun 08, 2021 at 06:14:42PM +0200, René Scharfe wrote:\n>>>> >> I wonder (only in a semi-curious way, though) if we can detect\n>>>> >> off-by-one errors by adding an assertion to display_progress()\n>>>> >> that requires the first update to have the value 0, and in\n>>>> >> stop_progress() one that requires the previous display_progress()\n>>>> >> call to have a value equal to the total number of work items.  Not\n>>>> >> sure it'd be worth the hassle..\n>>>> >\n>>>> > I fixed and reported a number of bogus progress lines in the past,\n>>>> > the last one during v2.31.0-rc phase, so I've looked into whether\n>>>> > progress counters could be automatically validated in our tests,\n>>>> > and came up with these patches a few months ago.  It turned out\n>>>> > that progress counters can be checked easily and transparently in\n>>>> > case of progress lines that are shown in the tests, i.e. that are\n>>>> > shown even when stderr is not a terminal or are forced with\n>>>> > '--progress'.  (In other cases it's still fairly easy but not quite\n>>>> > transparent, as I think we need changes to the progress API; more\n>>>> > on that later in a separate\n>>>> > series.)\n>>>>\n>>>> I've also been working on some progress.[ch] patches that are mostly\n>>>> finished, and I'm some 20 patches in at the moment. I wasn't sure\n>>>> about whether to send an alternate 20-patch \"let's do this (mostly) instead?\"\n>>>> series, hence this message.\n>>>>\n>>>> Much of what you're doing here becomes easier after that series, e.g.\n>>>> your global process struct in 2/7 is something I ended up\n>>>> implementing as part of a general feature to allow progress to be\n>>>> driven by either display_progress() *or* the signal handler itself.\n>>>\n>>> It's difficult to know who should rebase onto who without seeing one\n>>> half of the patches.\n>>\n>>I was sort of hoping he'd take me word for it, but here it is. Don't say I didn't warn you :)\n>>\n>>> I couldn't find a link to them anywhere (even if they are only\n>>> available in your fork in a pre-polished state) despite looking, but\n>>> my apologies if they are available and I'm just missing them.\n>>\n>>FWIW it's avar-szeder/progress-bar-assertions in https://github.com/avar/git.git, that repo contains various functioning and not-so-\n>>functioning code.\n>>\n>>https://github.com/avar/git/tree/meta/ is my version of the crappy scripts we probably all have some version of for building my own git,\n>>things that are uncommented in series.conf is what I build my own git from.\n>>\n>>> In general, I think that these patches are clear and are helpful in\n>>> pinning down issues with the progress API (which I have made a hadnful\n>>> of times in the past), so I would be happy to see them picked up.\n>>\n>>Here's all 25 patches (well, around 20 before) that I had queued up locally and fixed up a bit.\n>>\n>>The 01/25 is something I submitted already as https://lore.kernel.org/git/patch-1.1-cba5d88ca35-20210621T070114Z-avarab@gmail.com;\n>>hoping to get this in incrementally.\n>>\n>>The 12/25 is my own version of that \"global progress struct, 11/25 is the first of many bugs SZEDER missed in his :)\n>>\n>>18/25 is the first step of the UI I was going for, the signal handler can now drive the progress bar, so e.g. during \"git gc\" we show (at least\n>>for me, on git.git), a \"stalled\" message just before we start the actual count of \"Enumerating Objects\".\n>>\n>>After that was in I was planning on adding config-driven support to show a \"spinner\" when we stalled in that way, config-driven because\n>>you could just scrape e.g. https://github.com/sindresorhus/cli-spinners/blob/main/spinners.json\n>>into your own config. See\n>>https://jsfiddle.net/sindresorhus/2eLtsbey/embedded/result/ :)\n>>\n>>19-23/25 is my grabbing of SZEDER's patches that I'm comfortable labeling as \"PATCH\", I think they work, but no BUG() assertions yet. I\n>>left out the GIT_TEST_CHECK_PROGRESS parts, since my earlier works set things up to do any BUG() we trust by default.\n>>\n>>22/25 is what I think we should do instead of SZEDER's 6/7\n>>(http://lore.kernel.org/git/20210620200303.2328957-7-szeder.dev@gmail.com)\n>>I don't think this \"our total doesn't match at the end\" is something we should always BUG() on, for reasons explained there.\n>>\n>>I am sympathetic to doing it by default though, hence the\n>>stop_progress_early() API, that's there to allow select callers to bypass his BUG(...) assertion.\n>>\n>>24/25 and 25/25 are \"RFC\" and a rebased+modified version of SZEDER's\n>>BUG(...) assertions.\n>>\n>>His series passes the test suite, but actually severely break things things. It'll make e.g. \"git commit-graph write\" BUG(...) out. The reason\n>>the tests don't catch it is because we have a blind spot in the tests.\n>>\n>>Namely, that most things that use the progress bar API use isatty() to check if they should start_progress(). If you run the tests as e.g.\n>>(better ways to do this, especially in parallel, most welcome):\n>>\n>>    for t in t[0-9]*.sh; do if ! ./$t -vixd; then echo $t bad; break; fi; done\n>>\n>>You can discover various things that his series BUG()'s on, I fixed a couple of those myself, it's an early part of this series.\n>>\n>>But we'll still have various untested for BUG()'s even then, this is because you *also* have to have the test actually emit a \"naked\"\n>>progress bar on stderr, if the test itself e.g. pipes fd 2 to a file it won't work.\n>>\n>>I created a shitty-and-mostly-broken throwaway change to search-replace all the guards of \"start_progress(...)\" to run unconditionally, and\n>>convert all the \"delayed\" to the non-delayed version. That'll find even more BUG()'s where SZEDER's series still needs to be fixed (and also\n>>some unrelated segfaults, I gave up on it soon after).\n>>\n>>Even if we fix that I wouldn't trust it, because a lot of the progress bars we have depend on the size and shape of the data we're\n>>processing, e.g. the bug I fixed in 11/25. If people find this BUG() approach worth pursuing I think it would be better to make it an opt-in\n>>flag we convert one caller at a time to.\n>>\n>>For some it's really clear that we could assert it, for others such as the commit-graph it's much more subtle, we're in some callback after\n>>setting a \"total\", that callback does a \"break\", \"continue\" etc. in various places, all depending on repository data.\n>>\n>>It's not easy to reason about that and be certain that we can hold to the estimate. If we get it wrong someone's repo in the wild won't fully\n>>GC because of the overly eager BUG().\n>>\n>>If SZEDER wants to pursue it I think it'll be easier on top of this series, but personally I really don't see the point of spending effort on it.\n>>\n>>We should really be going in the other direction, of having more fuzzy ETAs, not less.\n>>\n>>E.g. we often have enough data at the start of \"Enumerating Objects\"\n>>to give a good-enough target value, that it's 5-10% off isn't really the point, but that the user looking at it sees something better than a\n>>dumb count-up, and can instead see that they'll probably be looking at it for about a minute. Now our API is to give no ETA/target if we're\n>>not 100% sure, it's not good UX.\n>>\n>>So trying to get the current exact count/exact percentage right seems like a distraction to me in the longer term. If anything we should\n>>just be rounding those numbers, showing fuzzy ETAs instead of percentages if we can etc.\n>>\n>>SZEDER Gábor (4):\n>>  commit-graph: fix bogus counter in \"Scanning merged commits\" progress\n>>    line\n>>  entry: show finer-grained counter in \"Filtering content\" progress line\n>>  progress: assert last update in stop_progress()\n>>  progress: assert counting upwards in display()\n>>\n>>Ævar Arnfjörð Bjarmason (21):\n>>  progress.c tests: fix breakage with COLUMNS != 80\n>>  progress.c tests: make start/stop verbs on stdin\n>>  progress.c tests: test some invalid usage\n>>  progress.c tests: add a \"signal\" verb\n>>  progress.c: move signal handler functions lower\n>>  progress.c: call progress_interval() from progress_test_force_update()\n>>  progress.c: stop eagerly fflush(stderr) when not a terminal\n>>  progress.c: add temporary variable from progress struct\n>>  midx perf: add a perf test for multi-pack-index\n>>  progress.c: remove the \"sparse\" mode nano-optimization\n>>  pack-bitmap-write.c: add a missing stop_progress()\n>>  progress.c: add & assert a \"global_progress\" variable\n>>  progress.[ch]: move the \"struct progress\" to the header\n>>  progress.[ch]: move test-only code away from \"extern\" variables\n>>  progress.c: pass \"is done?\" (again) to display()\n>>  progress.[ch]: convert \"title\" to \"struct strbuf\"\n>>  progress.c: refactor display() for less confusion, and fix bug\n>>  progress.c: emit progress on first signal, show \"stalled\"\n>>  midx: don't provide a total for QSORT() progress\n>>  progress.c: add a stop_progress_early() function\n>>  entry: deal with unexpected \"Filtering content\" total\n>>\n>> cache.h                          |   1 -\n>> commit-graph.c                   |   2 +-\n>> csum-file.h                      |   2 -\n>> entry.c                          |  12 +-\n>> midx.c                           |  25 +-\n>> pack-bitmap-write.c              |   1 +\n>> pack.h                           |   1 -\n>> parallel-checkout.h              |   1 -\n>> progress.c                       | 391 ++++++++++++++++++-------------\n>> progress.h                       |  50 +++-\n>> reachable.h                      |   1 -\n>> t/helper/test-progress.c         |  54 +++--\n>> t/perf/p5319-multi-pack-index.sh |  21 ++\n>> t/t0500-progress-display.sh      | 247 ++++++++++++++-----\n>> 14 files changed, 537 insertions(+), 272 deletions(-)  create mode 100755 t/perf/p5319-multi-pack-index.sh\n>\n> Is there provision for disabling progress on a per-command basis? My\n> use case is specifically in a CI/CD script, being able to suppress\n> progress handling. The current Jenkins plugin does not appear to have\n> provision for hooking into a mechanism, which makes things get a bit\n> wonky when a job runs with a pseudo-tty (as provided by Jenkins\n> through SSH/RMI).\n> -Randall\n\nThere isn't, some commands support --no-progress, but it's hit and miss.\n\nYou can then set the undocumented GIT_PROGRESS_DELAY=99999999 (or some\nreally big number) to suppress more of them.\n\nWe could just add it as a top-level \"git --no-progress\" option I\nsuppose...\n\nProbably better would be to detect such not-a-terminals somehow, I think\nat some point our own gc.log was a victim of this.\n"},{"id":"428326","messageId":"011401d7686d$f9b658c0$ed230a40$@nexbridge.com","threadId":"55972","inReplyTo":"87lf701h6k.fsf@evledraar.gmail.com","subject":"RE: [PATCH 00/25] progress.c: various fixes + SZEDER's RFC code","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2021-06-23T20:25:52Z","receivedAt":"2021-06-23T20:26:08Z","isPatch":true,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On June 23, 2021 4:02 PM, Ævar Arnfjörð Bjarmason wrote:\n>On Wed, Jun 23 2021, Randall S. Becker wrote:\n>> On June 23, 2021 1:48 PM, Ævar Arnfjörð Bjarmason wrote:\n>>>> On Mon, Jun 21, 2021 at 02:59:53AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>>>>>\n>>>>> On Sun, Jun 20 2021, SZEDER Gábor wrote:\n>>>>>\n>>>>> > Splitting off from:\n>>>>> >\n>>>>> >\n>>>>> > https://public-inbox.org/git/cover-0.2-0000000000-20210607T144206\n>>>>> > Z-\n>>>>> > avarab@gmail.com/T/#me5d3176914d4268fd9f2a96fc63f4e41beb26bd6\n>>>>> >\n>>>>> > On Tue, Jun 08, 2021 at 06:14:42PM +0200, René Scharfe wrote:\n>>>>> >> I wonder (only in a semi-curious way, though) if we can detect\n>>>>> >> off-by-one errors by adding an assertion to display_progress()\n>>>>> >> that requires the first update to have the value 0, and in\n>>>>> >> stop_progress() one that requires the previous\n>>>>> >> display_progress() call to have a value equal to the total\n>>>>> >> number of work items.  Not sure it'd be worth the hassle..\n>>>>> >\n>>>>> > I fixed and reported a number of bogus progress lines in the\n>>>>> > past, the last one during v2.31.0-rc phase, so I've looked into\n>>>>> > whether progress counters could be automatically validated in our\n>>>>> > tests, and came up with these patches a few months ago.  It\n>>>>> > turned out that progress counters can be checked easily and\n>>>>> > transparently in case of progress lines that are shown in the\n>>>>> > tests, i.e. that are shown even when stderr is not a terminal or\n>>>>> > are forced with '--progress'.  (In other cases it's still fairly\n>>>>> > easy but not quite transparent, as I think we need changes to the\n>>>>> > progress API; more on that later in a separate\n>>>>> > series.)\n>>>>>\n>>>>> I've also been working on some progress.[ch] patches that are\n>>>>> mostly finished, and I'm some 20 patches in at the moment. I wasn't\n>>>>> sure about whether to send an alternate 20-patch \"let's do this (mostly) instead?\"\n>>>>> series, hence this message.\n>>>>>\n>>>>> Much of what you're doing here becomes easier after that series, e.g.\n>>>>> your global process struct in 2/7 is something I ended up\n>>>>> implementing as part of a general feature to allow progress to be\n>>>>> driven by either display_progress() *or* the signal handler itself.\n>>>>\n>>>> It's difficult to know who should rebase onto who without seeing one\n>>>> half of the patches.\n>>>\n>>>I was sort of hoping he'd take me word for it, but here it is. Don't\n>>>say I didn't warn you :)\n>>>\n>>>> I couldn't find a link to them anywhere (even if they are only\n>>>> available in your fork in a pre-polished state) despite looking, but\n>>>> my apologies if they are available and I'm just missing them.\n>>>\n>>>FWIW it's avar-szeder/progress-bar-assertions in\n>>>https://github.com/avar/git.git, that repo contains various functioning and not-so- functioning code.\n>>>\n>>>https://github.com/avar/git/tree/meta/ is my version of the crappy\n>>>scripts we probably all have some version of for building my own git, things that are uncommented in series.conf is what I build my own\n>git from.\n>>>\n>>>> In general, I think that these patches are clear and are helpful in\n>>>> pinning down issues with the progress API (which I have made a\n>>>> hadnful of times in the past), so I would be happy to see them picked up.\n>>>\n>>>Here's all 25 patches (well, around 20 before) that I had queued up locally and fixed up a bit.\n>>>\n>>>The 01/25 is something I submitted already as\n>>>https://lore.kernel.org/git/patch-1.1-cba5d88ca35-20210621T070114Z-ava\n>>>rab@gmail.com;\n>>>hoping to get this in incrementally.\n>>>\n>>>The 12/25 is my own version of that \"global progress struct, 11/25 is\n>>>the first of many bugs SZEDER missed in his :)\n>>>\n>>>18/25 is the first step of the UI I was going for, the signal handler\n>>>can now drive the progress bar, so e.g. during \"git gc\" we show (at least for me, on git.git), a \"stalled\" message just before we start the\n>actual count of \"Enumerating Objects\".\n>>>\n>>>After that was in I was planning on adding config-driven support to\n>>>show a \"spinner\" when we stalled in that way, config-driven because\n>>>you could just scrape e.g.\n>>>https://github.com/sindresorhus/cli-spinners/blob/main/spinners.json\n>>>into your own config. See\n>>>https://jsfiddle.net/sindresorhus/2eLtsbey/embedded/result/ :)\n>>>\n>>>19-23/25 is my grabbing of SZEDER's patches that I'm comfortable\n>>>labeling as \"PATCH\", I think they work, but no BUG() assertions yet. I left out the GIT_TEST_CHECK_PROGRESS parts, since my earlier\n>works set things up to do any BUG() we trust by default.\n>>>\n>>>22/25 is what I think we should do instead of SZEDER's 6/7\n>>>(http://lore.kernel.org/git/20210620200303.2328957-7-szeder.dev@gmail.\n>>>com) I don't think this \"our total doesn't match at the end\" is\n>>>something we should always BUG() on, for reasons explained there.\n>>>\n>>>I am sympathetic to doing it by default though, hence the\n>>>stop_progress_early() API, that's there to allow select callers to bypass his BUG(...) assertion.\n>>>\n>>>24/25 and 25/25 are \"RFC\" and a rebased+modified version of SZEDER's\n>>>BUG(...) assertions.\n>>>\n>>>His series passes the test suite, but actually severely break things\n>>>things. It'll make e.g. \"git commit-graph write\" BUG(...) out. The reason the tests don't catch it is because we have a blind spot in the\n>tests.\n>>>\n>>>Namely, that most things that use the progress bar API use isatty() to check if they should start_progress(). If you run the tests as e.g.\n>>>(better ways to do this, especially in parallel, most welcome):\n>>>\n>>>    for t in t[0-9]*.sh; do if ! ./$t -vixd; then echo $t bad; break;\n>>> fi; done\n>>>\n>>>You can discover various things that his series BUG()'s on, I fixed a couple of those myself, it's an early part of this series.\n>>>\n>>>But we'll still have various untested for BUG()'s even then, this is because you *also* have to have the test actually emit a \"naked\"\n>>>progress bar on stderr, if the test itself e.g. pipes fd 2 to a file it won't work.\n>>>\n>>>I created a shitty-and-mostly-broken throwaway change to\n>>>search-replace all the guards of \"start_progress(...)\" to run\n>>>unconditionally, and convert all the \"delayed\" to the non-delayed version. That'll find even more BUG()'s where SZEDER's series still\n>needs to be fixed (and also some unrelated segfaults, I gave up on it soon after).\n>>>\n>>>Even if we fix that I wouldn't trust it, because a lot of the progress\n>>>bars we have depend on the size and shape of the data we're\n>>>processing, e.g. the bug I fixed in 11/25. If people find this BUG() approach worth pursuing I think it would be better to make it an opt-in\n>flag we convert one caller at a time to.\n>>>\n>>>For some it's really clear that we could assert it, for others such as\n>>>the commit-graph it's much more subtle, we're in some callback after setting a \"total\", that callback does a \"break\", \"continue\" etc. in\n>various places, all depending on repository data.\n>>>\n>>>It's not easy to reason about that and be certain that we can hold to\n>>>the estimate. If we get it wrong someone's repo in the wild won't fully GC because of the overly eager BUG().\n>>>\n>>>If SZEDER wants to pursue it I think it'll be easier on top of this series, but personally I really don't see the point of spending effort on it.\n>>>\n>>>We should really be going in the other direction, of having more fuzzy ETAs, not less.\n>>>\n>>>E.g. we often have enough data at the start of \"Enumerating Objects\"\n>>>to give a good-enough target value, that it's 5-10% off isn't really\n>>>the point, but that the user looking at it sees something better than\n>>>a dumb count-up, and can instead see that they'll probably be looking at it for about a minute. Now our API is to give no ETA/target if\n>we're not 100% sure, it's not good UX.\n>>>\n>>>So trying to get the current exact count/exact percentage right seems\n>>>like a distraction to me in the longer term. If anything we should just be rounding those numbers, showing fuzzy ETAs instead of\n>percentages if we can etc.\n>>>\n>>>SZEDER Gábor (4):\n>>>  commit-graph: fix bogus counter in \"Scanning merged commits\" progress\n>>>    line\n>>>  entry: show finer-grained counter in \"Filtering content\" progress\n>>>line\n>>>  progress: assert last update in stop_progress()\n>>>  progress: assert counting upwards in display()\n>>>\n>>>Ævar Arnfjörð Bjarmason (21):\n>>>  progress.c tests: fix breakage with COLUMNS != 80\n>>>  progress.c tests: make start/stop verbs on stdin\n>>>  progress.c tests: test some invalid usage\n>>>  progress.c tests: add a \"signal\" verb\n>>>  progress.c: move signal handler functions lower\n>>>  progress.c: call progress_interval() from\n>>>progress_test_force_update()\n>>>  progress.c: stop eagerly fflush(stderr) when not a terminal\n>>>  progress.c: add temporary variable from progress struct\n>>>  midx perf: add a perf test for multi-pack-index\n>>>  progress.c: remove the \"sparse\" mode nano-optimization\n>>>  pack-bitmap-write.c: add a missing stop_progress()\n>>>  progress.c: add & assert a \"global_progress\" variable\n>>>  progress.[ch]: move the \"struct progress\" to the header\n>>>  progress.[ch]: move test-only code away from \"extern\" variables\n>>>  progress.c: pass \"is done?\" (again) to display()\n>>>  progress.[ch]: convert \"title\" to \"struct strbuf\"\n>>>  progress.c: refactor display() for less confusion, and fix bug\n>>>  progress.c: emit progress on first signal, show \"stalled\"\n>>>  midx: don't provide a total for QSORT() progress\n>>>  progress.c: add a stop_progress_early() function\n>>>  entry: deal with unexpected \"Filtering content\" total\n>>>\n>>> cache.h                          |   1 -\n>>> commit-graph.c                   |   2 +-\n>>> csum-file.h                      |   2 -\n>>> entry.c                          |  12 +-\n>>> midx.c                           |  25 +-\n>>> pack-bitmap-write.c              |   1 +\n>>> pack.h                           |   1 -\n>>> parallel-checkout.h              |   1 -\n>>> progress.c                       | 391 ++++++++++++++++++-------------\n>>> progress.h                       |  50 +++-\n>>> reachable.h                      |   1 -\n>>> t/helper/test-progress.c         |  54 +++--\n>>> t/perf/p5319-multi-pack-index.sh |  21 ++\n>>> t/t0500-progress-display.sh      | 247 ++++++++++++++-----\n>>> 14 files changed, 537 insertions(+), 272 deletions(-)  create mode\n>>> 100755 t/perf/p5319-multi-pack-index.sh\n>>\n>> Is there provision for disabling progress on a per-command basis? My\n>> use case is specifically in a CI/CD script, being able to suppress\n>> progress handling. The current Jenkins plugin does not appear to have\n>> provision for hooking into a mechanism, which makes things get a bit\n>> wonky when a job runs with a pseudo-tty (as provided by Jenkins\n>> through SSH/RMI).\n>> -Randall\n>\n>There isn't, some commands support --no-progress, but it's hit and miss.\n>\n>You can then set the undocumented GIT_PROGRESS_DELAY=99999999 (or some really big number) to suppress more of them.\n>\n>We could just add it as a top-level \"git --no-progress\" option I suppose...\n>\n>Probably better would be to detect such not-a-terminals somehow, I think at some point our own gc.log was a victim of this.\n\nI think a global not-a-terminal would be best here. It does not make a lot of sense to dump progress on a device that does not handle Control-M. I think I recall someone recently saying that we should be detecting this.\n\n"},{"id":"428330","messageId":"20210623215736.8279-1-szeder.dev@gmail.com","threadId":"55972","inReplyTo":"20210620200303.2328957-1-szeder.dev@gmail.com","subject":"[PATCH 0/4] WIP/POC check isatty(2)-protected progress lines","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-06-23T21:57:32Z","receivedAt":"2021-06-23T21:57:46Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Sun, Jun 20, 2021 at 10:02:56PM +0200, SZEDER Gábor wrote:\n> It turned out that progress\n> counters can be checked easily and transparently in case of progress\n> lines that are shown in the tests, i.e. that are shown even when\n> stderr is not a terminal or are forced with '--progress'.  (In other\n> cases it's still fairly easy but not quite transparent, as I think we\n> need changes to the progress API; more on that later in a separate\n> series.)\n\nSo, the first patch in this WIP/POC series is my attempt at checking\neven those progress counters that are not shown in our test suite,\neither because stderr is not a terminal or because of an explicit\n'--no-progress' option.  There are no usable commit messages yet, I\njust wanted to see whether it's possible to check all progress lines\nand whether it uncovers any more bugs; and the answer is yes to both.\n\nAnyway, the basic idea is that instead of checking isatty(2) in the\ncaller, let's perform that check in start_progress() and let callers\noverride it through an extra function parameter (e.g. when\n'--(no-)progress', '-v' or '--quiet' was given).  This way\nstart_progress() will always be called and it would then return NULL\nif the progress line should not be shown.  Or, if\nGIT_TEST_CHECK_PROGRESS=1, then it would return a valid non-NULL\nprogress instance even when the progress line should not be shown, but\nwith the new 'progress->hidden' flag set, so subsequent\ndisplay_progress() and stop_progress() calls won't print anything but\nwill be able to perform all the checks and trigger BUG() if one is\nviolated.\n\nHowever, after Ævar pointed out upthread that progress also generates\ntrace2 regions, I think that it would be better if start_progress()\nalways returned a valid progress instance, even without\nGIT_TEST_CHECK_PROGRESS but with 'progress->hidden' set as necessary,\nbecause that way we would always get that trace2 output, even with\n'--no-progress' or 'git cmd 2>log'.\n\nThe first patch also converts a good couple of progress lines to this\nnew approach, and the subsequent patches fix most of the uncovered\nbuggy progress lines.\n\n\nSZEDER Gábor (4):\n  WIP progress, isatty(2), hidden progress lnies for\n    GIT_TEST_CHECK_PROGRESS\n  blame: fix progress total with line ranges\n  read-cache: avoid overlapping progress lines\n  preload-index: fix \"Refreshing index\" progress line\n\n builtin/blame.c          |  8 ++++----\n builtin/fsck.c           | 10 +++-------\n builtin/index-pack.c     | 18 +++++++++---------\n builtin/log.c            |  4 ++--\n builtin/prune.c          |  5 +----\n builtin/unpack-objects.c |  6 +++---\n preload-index.c          | 10 +++++-----\n progress.c               | 26 +++++++++++++++++++-------\n progress.h               |  6 ++++--\n read-cache.c             |  9 +++++----\n 10 files changed, 55 insertions(+), 47 deletions(-)\n\n-- \n2.32.0.289.g44fbea0957\n\n"},{"id":"428331","messageId":"20210623215736.8279-2-szeder.dev@gmail.com","threadId":"55972","inReplyTo":"20210623215736.8279-1-szeder.dev@gmail.com","subject":"[PATCH 1/4] WIP progress, isatty(2), hidden progress lnies for GIT_TEST_CHECK_PROGRESS","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-06-23T21:57:33Z","receivedAt":"2021-06-23T21:57:47Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"---\n builtin/blame.c          |  6 ++----\n builtin/fsck.c           | 10 +++-------\n builtin/index-pack.c     | 18 +++++++++---------\n builtin/log.c            |  4 ++--\n builtin/prune.c          |  5 +----\n builtin/unpack-objects.c |  6 +++---\n preload-index.c          |  7 +++----\n progress.c               | 26 +++++++++++++++++++-------\n progress.h               |  6 ++++--\n read-cache.c             |  6 +++---\n 10 files changed, 49 insertions(+), 45 deletions(-)\n\ndiff --git a/builtin/blame.c b/builtin/blame.c\nindex 641523ff9a..5efb920dd4 100644\n--- a/builtin/blame.c\n+++ b/builtin/blame.c\n@@ -944,8 +944,7 @@ int cmd_blame(int argc, const char **argv, const char *prefix)\n \t\tif (show_progress > 0)\n \t\t\tdie(_(\"--progress can't be used with --incremental or porcelain formats\"));\n \t\tshow_progress = 0;\n-\t} else if (show_progress < 0)\n-\t\tshow_progress = isatty(2);\n+\t}\n \n \tif (0 < abbrev && abbrev < hexsz)\n \t\t/* one more abbrev length is needed for the boundary commit */\n@@ -1153,8 +1152,7 @@ int cmd_blame(int argc, const char **argv, const char *prefix)\n \n \tsb.found_guilty_entry = &found_guilty_entry;\n \tsb.found_guilty_entry_data = &pi;\n-\tif (show_progress)\n-\t\tpi.progress = start_delayed_progress(_(\"Blaming lines\"), sb.num_lines);\n+\tpi.progress = start_delayed_progress_if_tty(_(\"Blaming lines\"), sb.num_lines, show_progress);\n \n \tassign_blame(&sb, opt);\n \ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex b42b6fe21f..78e799f748 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -185,8 +185,7 @@ static int traverse_reachable(void)\n \tstruct progress *progress = NULL;\n \tunsigned int nr = 0;\n \tint result = 0;\n-\tif (show_progress)\n-\t\tprogress = start_delayed_progress(_(\"Checking connectivity\"), 0);\n+\tprogress = start_delayed_progress_if_tty(_(\"Checking connectivity\"), 0, show_progress);\n \twhile (pending.nr) {\n \t\tresult |= traverse_one_object(object_array_pop(&pending));\n \t\tdisplay_progress(progress, ++nr);\n@@ -653,8 +652,7 @@ static void fsck_object_dir(const char *path)\n \tif (verbose)\n \t\tfprintf_ln(stderr, _(\"Checking object directory\"));\n \n-\tif (show_progress)\n-\t\tprogress = start_progress(_(\"Checking object directories\"), 256);\n+\tprogress = start_progress_if_tty(_(\"Checking object directories\"), 256, show_progress);\n \n \tfor_each_loose_file_in_objdir(path, fsck_loose, fsck_cruft, fsck_subdir,\n \t\t\t\t      progress);\n@@ -789,8 +787,6 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \tif (check_strict)\n \t\tfsck_obj_options.strict = 1;\n \n-\tif (show_progress == -1)\n-\t\tshow_progress = isatty(2);\n \tif (verbose)\n \t\tshow_progress = 0;\n \n@@ -825,7 +821,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t\t\t\t\ttotal += p->num_objects;\n \t\t\t\t}\n \n-\t\t\t\tprogress = start_progress(_(\"Checking objects\"), total);\n+\t\t\t\tprogress = start_progress_if_tty(_(\"Checking objects\"), total, show_progress);\n \t\t\t}\n \t\t\tfor (p = get_all_packs(the_repository); p;\n \t\t\t     p = p->next) {\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 3fbc5d7077..0caabe237e 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -258,8 +258,8 @@ static unsigned check_objects(void)\n \n \tmax = get_max_object_index();\n \n-\tif (verbose)\n-\t\tprogress = start_delayed_progress(_(\"Checking objects\"), max);\n+\tprogress = start_delayed_progress_if_tty(_(\"Checking objects\"), max,\n+\t\t\t\t\t\t verbose ? 1 : 0);\n \n \tfor (i = 0; i < max; i++) {\n \t\tforeign_nr += check_object(get_indexed_object(i));\n@@ -1157,10 +1157,9 @@ static void parse_pack_objects(unsigned char *hash)\n \tstruct object_id ref_delta_oid;\n \tstruct stat st;\n \n-\tif (verbose)\n-\t\tprogress = start_progress(\n-\t\t\t\tfrom_stdin ? _(\"Receiving objects\") : _(\"Indexing objects\"),\n-\t\t\t\tnr_objects);\n+\tprogress = start_progress_if_tty(\n+\t\t\tfrom_stdin ? _(\"Receiving objects\") : _(\"Indexing objects\"),\n+\t\t\tnr_objects, verbose ? 1 : 0);\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tstruct object_entry *obj = &objects[i];\n \t\tvoid *data = unpack_raw_entry(obj, &ofs_delta->offset,\n@@ -1235,9 +1234,10 @@ static void resolve_deltas(void)\n \tQSORT(ofs_deltas, nr_ofs_deltas, compare_ofs_delta_entry);\n \tQSORT(ref_deltas, nr_ref_deltas, compare_ref_delta_entry);\n \n-\tif (verbose || show_resolving_progress)\n-\t\tprogress = start_progress(_(\"Resolving deltas\"),\n-\t\t\t\t\t  nr_ref_deltas + nr_ofs_deltas);\n+\t/* TODO: breaks 5309.3 and .4 */\n+\tprogress = start_progress_if_tty(_(\"Resolving deltas\"),\n+\t\t\t\t\t nr_ref_deltas + nr_ofs_deltas,\n+\t\t\t\t\t verbose || show_resolving_progress ? 1 : 0);\n \n \tnr_dispatched = 0;\n \tbase_cache_limit = delta_base_cache_limit * nr_threads;\ndiff --git a/builtin/log.c b/builtin/log.c\nindex 6102893fcc..41bcd4d0fb 100644\n--- a/builtin/log.c\n+++ b/builtin/log.c\n@@ -2154,8 +2154,8 @@ int cmd_format_patch(int argc, const char **argv, const char *prefix)\n \t}\n \trev.add_signoff = do_signoff;\n \n-\tif (show_progress)\n-\t\tprogress = start_delayed_progress(_(\"Generating patches\"), total);\n+\tprogress = start_delayed_progress_if_tty(_(\"Generating patches\"), total,\n+\t\t\t\t\t\t show_progress);\n \twhile (0 <= --nr) {\n \t\tint shown;\n \t\tdisplay_progress(progress, total - nr);\ndiff --git a/builtin/prune.c b/builtin/prune.c\nindex 02c6ab7cba..2ee1baf40d 100644\n--- a/builtin/prune.c\n+++ b/builtin/prune.c\n@@ -41,8 +41,7 @@ static void perform_reachability_traversal(struct rev_info *revs)\n \tif (initialized)\n \t\treturn;\n \n-\tif (show_progress)\n-\t\tprogress = start_delayed_progress(_(\"Checking connectivity\"), 0);\n+\tprogress = start_delayed_progress_if_tty(_(\"Checking connectivity\"), 0, show_progress);\n \tmark_reachable_objects(revs, 1, expire, progress);\n \tstop_progress(&progress);\n \tinitialized = 1;\n@@ -164,8 +163,6 @@ int cmd_prune(int argc, const char **argv, const char *prefix)\n \t\t\tdie(\"unrecognized argument: %s\", name);\n \t}\n \n-\tif (show_progress == -1)\n-\t\tshow_progress = isatty(2);\n \tif (exclude_promisor_objects) {\n \t\tfetch_if_missing = 0;\n \t\trevs.exclude_promisor_objects = 1;\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex 4a9466295b..8517522a31 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -14,7 +14,7 @@\n #include \"decorate.h\"\n #include \"fsck.h\"\n \n-static int dry_run, quiet, recover, has_errors, strict;\n+static int dry_run, quiet = -1, recover, has_errors, strict;\n static const char unpack_usage[] = \"git unpack-objects [-n] [-q] [-r] [--strict]\";\n \n /* We always read in 4kB chunks. */\n@@ -500,8 +500,8 @@ static void unpack_all(void)\n \t\t\tntohl(hdr->hdr_version));\n \tuse(sizeof(struct pack_header));\n \n-\tif (!quiet)\n-\t\tprogress = start_progress(_(\"Unpacking objects\"), nr_objects);\n+\tprogress = start_progress_if_tty(_(\"Unpacking objects\"), nr_objects,\n+\t\t\t\t\t quiet ? 0 : -1);\n \tCALLOC_ARRAY(obj_list, nr_objects);\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tunpack_one(i);\ndiff --git a/preload-index.c b/preload-index.c\nindex e5529a5863..aae6e4a042 100644\n--- a/preload-index.c\n+++ b/preload-index.c\n@@ -121,10 +121,9 @@ void preload_index(struct index_state *index,\n \tmemset(&data, 0, sizeof(data));\n \n \tmemset(&pd, 0, sizeof(pd));\n-\tif (refresh_flags & REFRESH_PROGRESS && isatty(2)) {\n-\t\tpd.progress = start_delayed_progress(_(\"Refreshing index\"), index->cache_nr);\n-\t\tpthread_mutex_init(&pd.mutex, NULL);\n-\t}\n+\tpd.progress = start_delayed_progress_if_tty(_(\"Refreshing index\"),index->cache_nr,\n+\t\t\t\t\t\t   refresh_flags & REFRESH_PROGRESS ? -1 : 0);\n+\tpthread_mutex_init(&pd.mutex, NULL);\n \n \tfor (i = 0; i < threads; i++) {\n \t\tstruct thread_data *p = data+i;\ndiff --git a/progress.c b/progress.c\nindex 034d50cd6b..99e130f1eb 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -43,6 +43,7 @@ struct progress {\n \tstruct strbuf counters_sb;\n \tint title_len;\n \tint split;\n+\tint hidden;\n };\n \n static volatile sig_atomic_t progress_update;\n@@ -123,6 +124,8 @@ static void display(struct progress *progress, uint64_t n, const char *done)\n \n \tprogress->last_value = n;\n \n+\tif (progress->hidden)\n+\t\treturn;\n \tif (progress->delay && (!progress_update || --progress->delay))\n \t\treturn;\n \n@@ -261,15 +264,23 @@ void display_progress(struct progress *progress, uint64_t n)\n }\n \n static struct progress *start_progress_delay(const char *title, uint64_t total,\n-\t\t\t\t\t     unsigned delay, unsigned sparse)\n+\t\t\t\t\t     unsigned delay, unsigned sparse,\n+\t\t\t\t\t     int show)\n {\n \tstruct progress *progress;\n \n \ttest_check_progress = git_env_bool(\"GIT_TEST_CHECK_PROGRESS\", 0);\n+\n+\tif (show == -1)\n+\t\tshow = isatty(STDERR_FILENO);\n+\n \tif (test_check_progress && current_progress)\n \t\tBUG(\"progress \\\"%s\\\" is still active when starting new progress \\\"%s\\\"\",\n \t\t    current_progress->title, title);\n \n+\tif (!show && !test_check_progress)\n+\t\treturn NULL;\n+\n \tprogress = xmalloc(sizeof(*progress));\n \tcurrent_progress = progress;\n \tprogress->title = title;\n@@ -283,6 +294,7 @@ static struct progress *start_progress_delay(const char *title, uint64_t total,\n \tstrbuf_init(&progress->counters_sb, 0);\n \tprogress->title_len = utf8_strwidth(title);\n \tprogress->split = 0;\n+\tprogress->hidden = !show;\n \tset_progress_signal();\n \ttrace2_region_enter(\"progress\", title, the_repository);\n \treturn progress;\n@@ -298,14 +310,14 @@ static int get_default_delay(void)\n \treturn delay_in_secs;\n }\n \n-struct progress *start_delayed_progress(const char *title, uint64_t total)\n+struct progress *start_delayed_progress_if_tty(const char *title, uint64_t total, int show)\n {\n-\treturn start_progress_delay(title, total, get_default_delay(), 0);\n+\treturn start_progress_delay(title, total, get_default_delay(), 0, show);\n }\n \n-struct progress *start_progress(const char *title, uint64_t total)\n+struct progress *start_progress_if_tty(const char *title, uint64_t total, int show)\n {\n-\treturn start_progress_delay(title, total, 0, 0);\n+\treturn start_progress_delay(title, total, 0, 0, show);\n }\n \n /*\n@@ -319,13 +331,13 @@ struct progress *start_progress(const char *title, uint64_t total)\n  */\n struct progress *start_sparse_progress(const char *title, uint64_t total)\n {\n-\treturn start_progress_delay(title, total, 0, 1);\n+\treturn start_progress_delay(title, total, 0, 1, 1);\n }\n \n struct progress *start_delayed_sparse_progress(const char *title,\n \t\t\t\t\t       uint64_t total)\n {\n-\treturn start_progress_delay(title, total, get_default_delay(), 1);\n+\treturn start_progress_delay(title, total, get_default_delay(), 1, 1);\n }\n \n static void finish_if_sparse(struct progress *progress)\ndiff --git a/progress.h b/progress.h\nindex f1913acf73..7c3bdd3d63 100644\n--- a/progress.h\n+++ b/progress.h\n@@ -13,9 +13,11 @@ void progress_test_force_update(void);\n \n void display_throughput(struct progress *progress, uint64_t total);\n void display_progress(struct progress *progress, uint64_t n);\n-struct progress *start_progress(const char *title, uint64_t total);\n+#define start_progress(title, total) start_progress_if_tty((title), (total), 1)\n+struct progress *start_progress_if_tty(const char *title, uint64_t total, int show);\n struct progress *start_sparse_progress(const char *title, uint64_t total);\n-struct progress *start_delayed_progress(const char *title, uint64_t total);\n+#define start_delayed_progress(title, total) start_delayed_progress_if_tty((title), (total), 1)\n+struct progress *start_delayed_progress_if_tty(const char *title, uint64_t total, int show);\n struct progress *start_delayed_sparse_progress(const char *title,\n \t\t\t\t\t       uint64_t total);\n void stop_progress(struct progress **progress);\ndiff --git a/read-cache.c b/read-cache.c\nindex 1b3c2eb408..c3fc797639 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1567,9 +1567,9 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \tint t2_sum_lstat = 0;\n \tint t2_sum_scan = 0;\n \n-\tif (flags & REFRESH_PROGRESS && isatty(2))\n-\t\tprogress = start_delayed_progress(_(\"Refresh index\"),\n-\t\t\t\t\t\t  istate->cache_nr);\n+\tprogress = start_delayed_progress_if_tty(_(\"Refresh index\"),\n+\t\t\t\t\t\t istate->cache_nr,\n+\t\t\t\t\t\t flags & REFRESH_PROGRESS ? -1 : 0);\n \n \ttrace_performance_enter();\n \tmodified_fmt   = in_porcelain ? \"M\\t%s\\n\" : \"%s: needs update\\n\";\n-- \n2.32.0.289.g44fbea0957\n\n"},{"id":"428332","messageId":"20210623215736.8279-3-szeder.dev@gmail.com","threadId":"55972","inReplyTo":"20210623215736.8279-1-szeder.dev@gmail.com","subject":"[PATCH 2/4] blame: fix progress total with line ranges","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-06-23T21:57:34Z","receivedAt":"2021-06-23T21:57:50Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"When not blaming a whole file but only a subset of its lines using the\n'-L<start>,<end>' option, then the \"Blaming lines\" progress counter\ncan be way off, because the counter only counts the actually processed\nlines in the line range(s) while the expected total wrongly shows the\nnumber of lines in the given file:\n\n  $ wc -l git.c\n  932 git.c\n  $ GIT_PROGRESS_DELAY=0 git blame -L10,20 git.c\n  Blaming lines:   1% (11/932), done.\n  <...>\n\nLet's sum up the number of lines in all (sorted and merged) line\nranges and specify the resulting number as expected total.  (Note:\nwhen blaming the whole file, then we (implicitly) have one line range\nencompassing all its lines, so this approach works even when no line\nrange was given as option.)\n\nSigned-off-by: SZEDER Gábor <szeder.dev@gmail.com>\n---\n builtin/blame.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/blame.c b/builtin/blame.c\nindex 5efb920dd4..7d29f5dc61 100644\n--- a/builtin/blame.c\n+++ b/builtin/blame.c\n@@ -1121,9 +1121,11 @@ int cmd_blame(int argc, const char **argv, const char *prefix)\n \t}\n \tsort_and_merge_range_set(&ranges);\n \n+\tlno = 0;\n \tfor (range_i = ranges.nr; range_i > 0; --range_i) {\n \t\tconst struct range *r = &ranges.ranges[range_i - 1];\n \t\tent = blame_entry_prepend(ent, r->start, r->end, o);\n+\t\tlno += r->end - r->start;\n \t}\n \n \to->suspects = ent;\n@@ -1152,7 +1154,7 @@ int cmd_blame(int argc, const char **argv, const char *prefix)\n \n \tsb.found_guilty_entry = &found_guilty_entry;\n \tsb.found_guilty_entry_data = &pi;\n-\tpi.progress = start_delayed_progress_if_tty(_(\"Blaming lines\"), sb.num_lines, show_progress);\n+\tpi.progress = start_delayed_progress_if_tty(_(\"Blaming lines\"), lno, show_progress);\n \n \tassign_blame(&sb, opt);\n \n-- \n2.32.0.289.g44fbea0957\n\n"},{"id":"428333","messageId":"20210623215736.8279-4-szeder.dev@gmail.com","threadId":"55972","inReplyTo":"20210623215736.8279-1-szeder.dev@gmail.com","subject":"[PATCH 3/4] read-cache: avoid overlapping progress lines","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-06-23T21:57:35Z","receivedAt":"2021-06-23T21:57:52Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"\"Refresh index\" in refresh_index() in 'read-cache.c' vs. \"Refreshing\nindex\" in preload_index() in 'preload-index.c'.\n---\n read-cache.c | 9 +++++----\n 1 file changed, 5 insertions(+), 4 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex c3fc797639..692a69f2db 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1567,10 +1567,6 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \tint t2_sum_lstat = 0;\n \tint t2_sum_scan = 0;\n \n-\tprogress = start_delayed_progress_if_tty(_(\"Refresh index\"),\n-\t\t\t\t\t\t istate->cache_nr,\n-\t\t\t\t\t\t flags & REFRESH_PROGRESS ? -1 : 0);\n-\n \ttrace_performance_enter();\n \tmodified_fmt   = in_porcelain ? \"M\\t%s\\n\" : \"%s: needs update\\n\";\n \tdeleted_fmt    = in_porcelain ? \"D\\t%s\\n\" : \"%s: needs update\\n\";\n@@ -1583,6 +1579,11 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t * we only have to do the special cases that are left.\n \t */\n \tpreload_index(istate, pathspec, 0);\n+\n+\tprogress = start_delayed_progress_if_tty(_(\"Refresh index\"),\n+\t\t\t\t\t\t istate->cache_nr,\n+\t\t\t\t\t\t flags & REFRESH_PROGRESS ? -1 : 0);\n+\n \ttrace2_region_enter(\"index\", \"refresh\", NULL);\n \t/* TODO: audit for interaction with sparse-index. */\n \tensure_full_index(istate);\n-- \n2.32.0.289.g44fbea0957\n\n"},{"id":"428334","messageId":"20210623215736.8279-5-szeder.dev@gmail.com","threadId":"55972","inReplyTo":"20210623215736.8279-1-szeder.dev@gmail.com","subject":"[PATCH 4/4] preload-index: fix \"Refreshing index\" progress line","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-06-23T21:57:36Z","receivedAt":"2021-06-23T21:57:54Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"---\n preload-index.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/preload-index.c b/preload-index.c\nindex aae6e4a042..757dbeced6 100644\n--- a/preload-index.c\n+++ b/preload-index.c\n@@ -86,7 +86,8 @@ static void *preload_thread(void *_data)\n \t\tstruct progress_data *pd = p->progress;\n \n \t\tpthread_mutex_lock(&pd->mutex);\n-\t\tdisplay_progress(pd->progress, pd->n + last_nr);\n+\t\tpd->n += last_nr;\n+\t\tdisplay_progress(pd->progress, pd->n);\n \t\tpthread_mutex_unlock(&pd->mutex);\n \t}\n \tcache_def_clear(&cache);\n-- \n2.32.0.289.g44fbea0957\n\n"},{"id":"428335","messageId":"20210623221152.GF6312@szeder.dev","threadId":"55972","inReplyTo":"20210623215736.8279-1-szeder.dev@gmail.com","subject":"Re: [PATCH 0/4] WIP/POC check isatty(2)-protected progress lines","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-06-23T22:11:52Z","receivedAt":"2021-06-23T22:15:27Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Jun 23, 2021 at 11:57:32PM +0200, SZEDER Gábor wrote:\n> I just wanted to see whether it's possible to check all progress lines\n> and whether it uncovers any more bugs; and the answer is yes to both.\n\nOh, and there is another one:\n\ntest_expect_success 'test' '\n\tgit commit --allow-empty -m 1 &&\n\tgit commit --allow-empty -m 2 &&\n\tgit commit --allow-empty -m 3 &&\n\tGIT_PROGRESS_DELAY=0 \\\n\tgit commit-graph write --progress --reachable --split &&\n\tgit commit --allow-empty -m 4 &&\n\tGIT_PROGRESS_DELAY=0 \\\n\tgit commit-graph write --progress --reachable --split\n'\n\nThe last command's progress output ends with:\n\n  Writing out commit graph in 5 passes:  80% (4/5), done.\n\nThis is because since 53035c4f0b (commit-graph write: add \"Writing\nout\" progress output, 2019-01-19) we have assumed that the work done\nwhile writing each chunk is proportional to the number of commits in\nthe graph, but with the arrival of split commit graphs and the BASE\nchunk in 118bd57002 (commit-graph: add base graphs chunk, 2019-06-18)\nthat's not longer the case.\n\n"},{"id":"428367","messageId":"87fsx71re3.fsf@evledraar.gmail.com","threadId":"55972","inReplyTo":"patch-22.25-e7f8314867d-20210623T155626Z-avarab@gmail.com","subject":"Re: [PATCH 22/25] progress.c: add a stop_progress_early() function","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T10:35:32Z","receivedAt":"2021-06-24T10:36:09Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jun 23 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> +\tstrbuf_addf(&sb, _(\", done at %\"PRIuMAX\" items, expected %\"PRIuMAX\".\"),\n> +\t\t    progress->total, progress->last_update);\n\nThese two need a (uintmax_t) cast like the rest of such sprintfs in the\nfile, as I discovered with the OSX CI.\n"},{"id":"428370","messageId":"87a6nf1r10.fsf@evledraar.gmail.com","threadId":"55972","inReplyTo":"20210623221152.GF6312@szeder.dev","subject":"Re: [PATCH 0/4] WIP/POC check isatty(2)-protected progress lines","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T10:43:30Z","receivedAt":"2021-06-24T10:43:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jun 24 2021, SZEDER Gábor wrote:\n\n> On Wed, Jun 23, 2021 at 11:57:32PM +0200, SZEDER Gábor wrote:\n>> I just wanted to see whether it's possible to check all progress lines\n>> and whether it uncovers any more bugs; and the answer is yes to both.\n>\n> Oh, and there is another one:\n>\n> test_expect_success 'test' '\n> \tgit commit --allow-empty -m 1 &&\n> \tgit commit --allow-empty -m 2 &&\n> \tgit commit --allow-empty -m 3 &&\n> \tGIT_PROGRESS_DELAY=0 \\\n> \tgit commit-graph write --progress --reachable --split &&\n> \tgit commit --allow-empty -m 4 &&\n> \tGIT_PROGRESS_DELAY=0 \\\n> \tgit commit-graph write --progress --reachable --split\n> '\n>\n> The last command's progress output ends with:\n>\n>   Writing out commit graph in 5 passes:  80% (4/5), done.\n>\n> This is because since 53035c4f0b (commit-graph write: add \"Writing\n> out\" progress output, 2019-01-19) we have assumed that the work done\n> while writing each chunk is proportional to the number of commits in\n> the graph, but with the arrival of split commit graphs and the BASE\n> chunk in 118bd57002 (commit-graph: add base graphs chunk, 2019-06-18)\n> that's not longer the case.\n\nAh, I encountered the off-by-something in that \"writing in N passes\" but\ndidn't find the root cause, thanks.\n"},{"id":"428371","messageId":"877dij1pzv.fsf@evledraar.gmail.com","threadId":"55972","inReplyTo":"20210623215736.8279-1-szeder.dev@gmail.com","subject":"Re: [PATCH 0/4] WIP/POC check isatty(2)-protected progress lines","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-24T10:45:25Z","receivedAt":"2021-06-24T11:06:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jun 23 2021, SZEDER Gábor wrote:\n\n> On Sun, Jun 20, 2021 at 10:02:56PM +0200, SZEDER Gábor wrote:\n>> It turned out that progress\n>> counters can be checked easily and transparently in case of progress\n>> lines that are shown in the tests, i.e. that are shown even when\n>> stderr is not a terminal or are forced with '--progress'.  (In other\n>> cases it's still fairly easy but not quite transparent, as I think we\n>> need changes to the progress API; more on that later in a separate\n>> series.)\n>\n> So, the first patch in this WIP/POC series is my attempt at checking\n> even those progress counters that are not shown in our test suite,\n> either because stderr is not a terminal or because of an explicit\n> '--no-progress' option.  There are no usable commit messages yet, I\n> just wanted to see whether it's possible to check all progress lines\n> and whether it uncovers any more bugs; and the answer is yes to both.\n>\n> Anyway, the basic idea is that instead of checking isatty(2) in the\n> caller, let's perform that check in start_progress() and let callers\n> override it through an extra function parameter (e.g. when\n> '--(no-)progress', '-v' or '--quiet' was given).  This way\n> start_progress() will always be called and it would then return NULL\n> if the progress line should not be shown.  Or, if\n> GIT_TEST_CHECK_PROGRESS=1, then it would return a valid non-NULL\n> progress instance even when the progress line should not be shown, but\n> with the new 'progress->hidden' flag set, so subsequent\n> display_progress() and stop_progress() calls won't print anything but\n> will be able to perform all the checks and trigger BUG() if one is\n> violated.\n>\n> However, after Ævar pointed out upthread that progress also generates\n> trace2 regions, I think that it would be better if start_progress()\n> always returned a valid progress instance, even without\n> GIT_TEST_CHECK_PROGRESS but with 'progress->hidden' set as necessary,\n> because that way we would always get that trace2 output, even with\n> '--no-progress' or 'git cmd 2>log'.\n>\n> The first patch also converts a good couple of progress lines to this\n> new approach, and the subsequent patches fix most of the uncovered\n> buggy progress lines.\n\nThanks, I skimmed over it and this sort of approach is definitely what\nwe'll need to address my \"But we'll still have various untested for\nBUG()[...]\" in\nhttps://lore.kernel.org/git/cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com/\n\nAnd as you point out we'll get the benefit of consistent trace2 regions,\non the one hand it's a bit weird to have this UI code drive a trace2\nregion when we don't have a TTY, but I think it's useful. We could\ne.g. eventually record some stats about min/max/avg/percentile\nprocessing per-item while we're at it, that's unlikely to be worth it if\nwe need another API like display_progress(), but since we have that one\nwe can piggy-back on it quite easily.\n\nJust some implementation nits: I for one would prefer \"static inline\"\nwrappers instead of macros in progress.h, makes it easier to\nconsistently set breakpoints in gdb.\n\nIt's more work up-front, but I think Re Randall's question in\nhttps://lore.kernel.org/git/00fb01d76859$8a6ebc50$9f4c34f0$@nexbridge.com\nthat instead of s/start_delayed_progress/start_delayed_progress_if_tty/\nit would be better to just leave the \"start_delayed_progress\", and have\nit by default do the TTY check, and also check for --progress and/or\n--verbose/--quiet etc. itself.\n\nWe'd probably have some special-cases left even then, but I think most\nof them can be handled with an isatty() check and the \"standard\" options\nof --progress etc.\n\nI.e. we have OPT__VERBOSE now, but no OPT__PROGRESS (we just use\nOPT_BOOL). If we made the various common parse-options flags that impact\nit callbacks that would munge a global variable we could then pick that\nup in progress.c, and handle the common case of \"git some-command\n--no-progress\" directly.\n\nIt would also make it easy to just move that over to git.c, so we could\nhave \"git --no-progress some-command\", which I think for --progress,\n--object-format and other \"global-y\" options it we should have them to\n\"git\" directly, not per-command, especially with us hopefully soon\nmoving 100% away from dashed built-ins.\n\nIsn't the most common general rule just:\n\n    int want_progress = progress ? 1 : verbose ? 1 : quiet ? 0 : isatty(2);\n\nWell, that and a version that handles --no-progress distinct from \"did\nnot provide it\", so we need some \"-1\" checks in there. Maybe:\n\n    /* Earlier */\n    if (quiet != -1 && verbose != -1)\n        die(\"--quiet and --verbose?\");\n\n    /* In progress.c after getopt */\n    int enable = -1;\n    if (opt_progress != -1) enable = opt_progress;\n    if (enable == -1 && opt_verbose != -1) enable = opt_verbose;\n    if (enable == -1 && opt_quiet != -1) enable = !opt_quiet;\n    if (enable == -1) enable = isatty(2);\n\nIn any case, I think moving that to one place so it's consistently\nchecked would make sense.\n\nSome things like builtin/multi-pack-index.c set \"progress\" as a bitflag\nfor IMO (this was discussed on-list before) no good reason. I.e. the\nbuiltin should handle it with a bool, maybe the library wants a flag,\nbut in any case if we can do what I proposed above such libraries won't\nneed a flag at all.\n"},{"id":"428452","messageId":"0e317e88-03fc-9d5b-6fa4-eb5bc3422f78@gmail.com","threadId":"55972","inReplyTo":"patch-22.25-e7f8314867d-20210623T155626Z-avarab@gmail.com","subject":"Re: [PATCH 22/25] progress.c: add a stop_progress_early() function","fromName":"Andrei Rybak","fromEmail":"rybak.a.v@gmail.com","sentAt":"2021-06-25T01:24:18Z","receivedAt":"2021-06-25T01:24:25Z","isPatch":true,"sender":{"key":"rybak.a.v@gmail.com","avatar":"https://avatars.githubusercontent.com/u/624072?v=4"},"body":"On 23/06/2021 19:48, Ævar Arnfjörð Bjarmason wrote:\n> In cases where we error out during processing or otherwise miss\n> initial \"total\" estimate we'll still want to show a \"done\" message and\n> end our trace2 region, but it won't be true that our total ==\n> last_update at the end.\n> \n> So let's add a \"last_update\" and this stop_progress_early() function\n> to handle that edge case, this will be used in a subsequent commit.\n> \n> We could also use a total=0 in such cases, but that would make the\n> progress output worse for the common non-erroring case. Let's instead\n> note that we didn't reach the total count, and snap the progress bar\n> to \"100%, done\" at the end.\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>   progress.c | 20 ++++++++++++++++++++\n>   progress.h |  2 ++\n>   2 files changed, 22 insertions(+)\n> \n> diff --git a/progress.c b/progress.c\n> index 35847d3a7f2..c1cb01ba975 100644\n> --- a/progress.c\n> +++ b/progress.c\n> @@ -40,6 +40,8 @@ static void display(struct progress *progress, uint64_t n,\n>   \tconst char *tp;\n>   \tint show_update = 0;\n>   \n> +\tprogress->last_update = n;\n> +\n>   \tif (progress->delay && (!progress_update || --progress->delay))\n>   \t\treturn;\n>   \n> @@ -413,3 +415,21 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)\n>   \tfree(progress->throughput);\n>   \tfree(progress);\n>   }\n> +\n> +void stop_progress_early(struct progress **p_progress)\n> +{\n> +\tstruct progress *progress;\n> +\tstruct strbuf sb = STRBUF_INIT;\n> +\n> +\tif (!p_progress)\n> +\t\tBUG(\"don't provide NULL to stop_progress_early\");\n> +\tprogress = *p_progress;\n> +\tif (!progress)\n> +\t\treturn;\n> +\n> +\tstrbuf_addf(&sb, _(\", done at %\"PRIuMAX\" items, expected %\"PRIuMAX\".\"),\n> +\t\t    progress->total, progress->last_update);\n\nIt seems that these two arguments to strbuf_addf should be swapped\naround.  Done at progress->last_update, expected progress->total.\n\n> +\tprogress->total = progress->last_update;\n> +\tstop_progress_msg(p_progress, sb.buf);\n> +\tstrbuf_release(&sb);\n> +}\n> diff --git a/progress.h b/progress.h\n> index ba38447d104..5c5d027d1a0 100644\n> --- a/progress.h\n> +++ b/progress.h\n> @@ -23,6 +23,7 @@ struct progress {\n>   \tstruct strbuf status;\n>   \tsize_t status_len_utf8;\n>   \n> +\tuint64_t last_update;\n>   \tuint64_t last_value;\n>   \tuint64_t total;\n>   \tunsigned last_percent;\n> @@ -56,5 +57,6 @@ struct progress *start_delayed_sparse_progress(const char *title,\n>   \t\t\t\t\t       uint64_t total);\n>   void stop_progress(struct progress **progress);\n>   void stop_progress_msg(struct progress **progress, const char *msg);\n> +void stop_progress_early(struct progress **p_progress);\n>   \n>   #endif\n> \n\n"},{"id":"428514","messageId":"c1c5316a-6a43-a377-69d5-531d226463c8@web.de","threadId":"55972","inReplyTo":"87h7hr2cx0.fsf@evledraar.gmail.com","subject":"Re: [PATCH 4/7] commit-graph: fix bogus counter in \"Scanning merged commits\" progress line","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-06-26T08:27:23Z","receivedAt":"2021-06-26T08:27:27Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 21.06.21 um 22:08 schrieb Ævar Arnfjörð Bjarmason:\n>\n> On Mon, Jun 21 2021, René Scharfe wrote:\n>\n>> Am 21.06.21 um 00:13 schrieb Ævar Arnfjörð Bjarmason:\n>>>\n>>> On Sun, Jun 20 2021, SZEDER Gábor wrote:\n>>>\n>>>> The final value of the counter of the \"Scanning merged commits\"\n>>>> progress line is always one less than its expected total, e.g.:\n>>>>\n>>>>   Scanning merged commits:  83% (5/6), done.\n>>>>\n>>>> This happens because while iterating over an array the loop variable\n>>>> is passed to display_progress() as-is, but while C arrays (and thus\n>>>> the loop variable) start at 0 and end at N-1, the progress counter\n>>>> must end at N.  This causes the failures of the tests\n>>>> 'fetch.writeCommitGraph' and 'fetch.writeCommitGraph with submodules'\n>>>> in 't5510-fetch.sh' when run with GIT_TEST_CHECK_PROGRESS=1.\n>>>>\n>>>> Fix this by passing 'i + 1' to display_progress(), like most other\n>>>> callsites do.\n>>>>\n>>>> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>\n>>>> ---\n>>>>  commit-graph.c | 2 +-\n>>>>  1 file changed, 1 insertion(+), 1 deletion(-)\n>>>>\n>>>> diff --git a/commit-graph.c b/commit-graph.c\n>>>> index 2bcb4e0f89..3181906368 100644\n>>>> --- a/commit-graph.c\n>>>> +++ b/commit-graph.c\n>>>> @@ -2096,7 +2096,7 @@ static void sort_and_scan_merged_commits(struct write_commit_graph_context *ctx)\n>>>>\n>>>>  \tctx->num_extra_edges = 0;\n>>>>  \tfor (i = 0; i < ctx->commits.nr; i++) {\n>>>> -\t\tdisplay_progress(ctx->progress, i);\n>>>> +\t\tdisplay_progress(ctx->progress, i + 1);\n>>>>\n>>>>  \t\tif (i && oideq(&ctx->commits.list[i - 1]->object.oid,\n>>>>  \t\t\t  &ctx->commits.list[i]->object.oid)) {\n>>>\n>>> I think this fix makes sense, but FWIW there's a large thread starting\n>>> at [1] where René disagrees with me, and thinks the fix for this sort of\n>>> thing would be to display_progress(..., i + 1) at the end of that\n>>> for-loop, or just before the stop_progress().\n>>>\n>>> I don't agree, but just noting the disagreement, and that if that\n>>> argument wins then a patch like this would involve changing the other\n>>> 20-some calls to display_progress() in commit-graph.c to work\n>>> differently (and to be more complex, we'd need to deal with loop\n>>> break/continue etc.).\n>>>\n>>> 1. https://lore.kernel.org/git/patch-2.2-042f598826-20210607T144206Z-avarab@gmail.com/\n>>\n>> *sigh*  (And sorry, Ævar.)\n>>\n>> Before an item is done, it should be reported as not done.  After an\n>> item is done, it should be reported as done.  One loop iteration\n>> finishes one item.  Thus the number of items to report at the bottom of\n>> the loop is one higher than at the top.  i is the correct number to\n>> report at the top of a zero-based loop, i+1 at the bottom.\n\n> Anyone with more time than sense can go and read over our linked back &\n> forth thread where we're disagreeing on that point :). I think the pattern\n> in commit-graph.c makes sense, you don't.\n\nThanks for this comment, I think I got it now: Work doesn't count in the\ncommit-graph.c model of measuring progress, literally.  I.e. progress is\nthe same before and after one item of work.  Instead it counts the\nnumber of loop iterations.  The model I describe above counts finished\nwork items instead.  The results of the two models differ by at most one\ndespite their inverted axiom regarding the value of work.\n\nPhew, that took me a while.\n\n> Anyway, aside from that. I think, and I really would be advocating this\n> too, even if our respective positions were reversed, that *in this case*\n> it makes sense to just take something like SZEDER's patch here\n> as-is. Because in that file there's some dozen occurrences of that exact\n> pattern.\n\nThe code without the patch either forgets to report the last work item\nin the count-work-items model or is one short in the count-iterations\nmodel, so a fix is needed either way.\n\nThe number of the other occurrences wouldn't matter if they were\nbuggy, but in this case they indicate that Stolee consistently used\nthe count-iterations model.  Thus using it in the patch as well makes\nsense.\n\n> Let's just bring this one case in line with the rest, if we then want to\n> argue that one or the other use of the progress.c API is wrong as a\n> general thing, I think it makes more sense to discuss that as some\n> follow-up series that changes these various API uses en-masse than\n> holding back isolated fixes that leave the state of the progress bar it\n> != 100%.\n\nAgreed.\n\nRené\n"},{"id":"428520","messageId":"87a6ncy9aq.fsf@evledraar.gmail.com","threadId":"55972","inReplyTo":"c1c5316a-6a43-a377-69d5-531d226463c8@web.de","subject":"Re: [PATCH 4/7] commit-graph: fix bogus counter in \"Scanning merged commits\" progress line","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-26T14:11:31Z","receivedAt":"2021-06-26T14:45:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Jun 26 2021, René Scharfe wrote:\n\n> Am 21.06.21 um 22:08 schrieb Ævar Arnfjörð Bjarmason:\n>>\n>> On Mon, Jun 21 2021, René Scharfe wrote:\n>>\n>>> Am 21.06.21 um 00:13 schrieb Ævar Arnfjörð Bjarmason:\n>>>>\n>>>> On Sun, Jun 20 2021, SZEDER Gábor wrote:\n>>>>\n>>>>> The final value of the counter of the \"Scanning merged commits\"\n>>>>> progress line is always one less than its expected total, e.g.:\n>>>>>\n>>>>>   Scanning merged commits:  83% (5/6), done.\n>>>>>\n>>>>> This happens because while iterating over an array the loop variable\n>>>>> is passed to display_progress() as-is, but while C arrays (and thus\n>>>>> the loop variable) start at 0 and end at N-1, the progress counter\n>>>>> must end at N.  This causes the failures of the tests\n>>>>> 'fetch.writeCommitGraph' and 'fetch.writeCommitGraph with submodules'\n>>>>> in 't5510-fetch.sh' when run with GIT_TEST_CHECK_PROGRESS=1.\n>>>>>\n>>>>> Fix this by passing 'i + 1' to display_progress(), like most other\n>>>>> callsites do.\n>>>>>\n>>>>> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>\n>>>>> ---\n>>>>>  commit-graph.c | 2 +-\n>>>>>  1 file changed, 1 insertion(+), 1 deletion(-)\n>>>>>\n>>>>> diff --git a/commit-graph.c b/commit-graph.c\n>>>>> index 2bcb4e0f89..3181906368 100644\n>>>>> --- a/commit-graph.c\n>>>>> +++ b/commit-graph.c\n>>>>> @@ -2096,7 +2096,7 @@ static void sort_and_scan_merged_commits(struct write_commit_graph_context *ctx)\n>>>>>\n>>>>>  \tctx->num_extra_edges = 0;\n>>>>>  \tfor (i = 0; i < ctx->commits.nr; i++) {\n>>>>> -\t\tdisplay_progress(ctx->progress, i);\n>>>>> +\t\tdisplay_progress(ctx->progress, i + 1);\n>>>>>\n>>>>>  \t\tif (i && oideq(&ctx->commits.list[i - 1]->object.oid,\n>>>>>  \t\t\t  &ctx->commits.list[i]->object.oid)) {\n>>>>\n>>>> I think this fix makes sense, but FWIW there's a large thread starting\n>>>> at [1] where René disagrees with me, and thinks the fix for this sort of\n>>>> thing would be to display_progress(..., i + 1) at the end of that\n>>>> for-loop, or just before the stop_progress().\n>>>>\n>>>> I don't agree, but just noting the disagreement, and that if that\n>>>> argument wins then a patch like this would involve changing the other\n>>>> 20-some calls to display_progress() in commit-graph.c to work\n>>>> differently (and to be more complex, we'd need to deal with loop\n>>>> break/continue etc.).\n>>>>\n>>>> 1. https://lore.kernel.org/git/patch-2.2-042f598826-20210607T144206Z-avarab@gmail.com/\n>>>\n>>> *sigh*  (And sorry, Ævar.)\n>>>\n>>> Before an item is done, it should be reported as not done.  After an\n>>> item is done, it should be reported as done.  One loop iteration\n>>> finishes one item.  Thus the number of items to report at the bottom of\n>>> the loop is one higher than at the top.  i is the correct number to\n>>> report at the top of a zero-based loop, i+1 at the bottom.\n>\n>> Anyone with more time than sense can go and read over our linked back &\n>> forth thread where we're disagreeing on that point :). I think the pattern\n>> in commit-graph.c makes sense, you don't.\n>\n> Thanks for this comment, I think I got it now: Work doesn't count in the\n> commit-graph.c model of measuring progress, literally.  I.e. progress is\n> the same before and after one item of work.\n\nThe progress isn't the same, we update the count. Or do you mean in the\ntime it takes us to go from the end of the for-loop & jump to the start\nof it and update the count?\n\n> Instead it counts the number of loop iterations.  The model I describe\n> above counts finished work items instead.  The results of the two\n> models differ by at most one despite their inverted axiom regarding\n> the value of work.\n>\n> Phew, that took me a while.\n\nFor what it's worth I had some extensive examples in our initial\nthread[1][2] (search for \"apple\" and \"throughput\", respectively), that\nyou cut out when replying to the relevant E-Mails. I'd think we could\nprobably have gotten here earlier :)\n\nI'm a bit confused about this \"value of work\" comment.\n\nIf you pick up a copy of say a video game like Mario Kart you'll find\nthat for a 3-lap race you start at 1/3, and still have an entire lap to\ngo when the count is at 3/3.\n\nSo it's just a question of whether you report progress on item N or work\nfinished on item N, not whether laps in a race have more or less\nvalue.\n\nTo reference my earlier E-Mail[1] are you eating the first apple or the\nzeroeth apple? I don't think one is more or less right in the\nmathematical sense, I just think for UX aimed at people counting \"laps\"\nmakes more sense than counting completed items.\n\n>> Anyway, aside from that. I think, and I really would be advocating this\n>> too, even if our respective positions were reversed, that *in this case*\n>> it makes sense to just take something like SZEDER's patch here\n>> as-is. Because in that file there's some dozen occurrences of that exact\n>> pattern.\n>\n> The code without the patch either forgets to report the last work item\n> in the count-work-items model or is one short in the count-iterations\n> model, so a fix is needed either way.\n\nIt won't be one short, for a loop of 2 items we'll go from:\n\n     0/2\n     1/2\n     1/2, done\n\nTo:\n\n     1/2\n     2/2\n     2/2, done\n\nJust like the rest of the uses of the progress API in that file.\n\nWhich is one of the two reasons I prefer this pattern, i.e. this is less\nverbose:\n\n    start_progress()\n    for i in (0..X-1):\n        display_progress(i+1)\n        work()\n    stop_progress()\n\nThan one of these, which AFAICT would be your recommendation:\n\n    # Simplest, but stalls on work()\n    start_progress()\n    for i in (0..X-1):\n        work()\n        display_progress(i+1)\n    stop_progress()\n\n    # More verbose, but doesn't:\n    start_progress()\n    for i in (0..X-1):\n        display_progress(i)\n        work()\n        display_progress(i+1)\n    stop_progress()\n\n    # Ditto:\n    start_progress()\n    display_progress(0)\n    for i in (0..X-1):\n        work()\n        display_progress(i+1)\n    stop_progress()\n\nAnd of course if your loop continues or whatever you'll need a last\n\"display_progress(X)\" before the \"stop_progress()\".\n\nThe other is that if you count laps you can have your progress bar\noptionally show progress on that item. E.g. we could if we stall show\nseconds spend that we're hung on that item, or '3/3 ETA 40s\". I have a\npatch[3] that takes an initial step towards that, with some more queued\nlocally.\n\n> The number of the other occurrences wouldn't matter if they were\n> buggy, but in this case they indicate that Stolee consistently used\n> the count-iterations model.  Thus using it in the patch as well makes\n> sense.\n\n>> Let's just bring this one case in line with the rest, if we then want to\n>> argue that one or the other use of the progress.c API is wrong as a\n>> general thing, I think it makes more sense to discuss that as some\n>> follow-up series that changes these various API uses en-masse than\n>> holding back isolated fixes that leave the state of the progress bar it\n>> != 100%.\n>\n> Agreed.\n\nSorry to go on about this again :)\n\n1. https://lore.kernel.org/git/87lf7k2bem.fsf@evledraar.gmail.com/\n2. https://lore.kernel.org/git/87o8c8z105.fsf@evledraar.gmail.com/\n3. https://lore.kernel.org/git/patch-18.25-e21fc66623f-20210623T155626Z-avarab@gmail.com/\n"},{"id":"428532","messageId":"177c59bf-98e1-ff55-1b75-ea89c0de976a@web.de","threadId":"55972","inReplyTo":"87a6ncy9aq.fsf@evledraar.gmail.com","subject":"Re: [PATCH 4/7] commit-graph: fix bogus counter in \"Scanning merged commits\" progress line","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-06-26T20:22:11Z","receivedAt":"2021-06-26T20:22:15Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 26.06.21 um 16:11 schrieb Ævar Arnfjörð Bjarmason:\n>\n> On Sat, Jun 26 2021, René Scharfe wrote:\n>\n>> Am 21.06.21 um 22:08 schrieb Ævar Arnfjörð Bjarmason:\n>>>\n>>> On Mon, Jun 21 2021, René Scharfe wrote:\n>>>\n>>>> Before an item is done, it should be reported as not done.  After an\n>>>> item is done, it should be reported as done.  One loop iteration\n>>>> finishes one item.  Thus the number of items to report at the bottom of\n>>>> the loop is one higher than at the top.  i is the correct number to\n>>>> report at the top of a zero-based loop, i+1 at the bottom.\n>>\n>>> Anyone with more time than sense can go and read over our linked back &\n>>> forth thread where we're disagreeing on that point :). I think the pattern\n>>> in commit-graph.c makes sense, you don't.\n>>\n>> Thanks for this comment, I think I got it now: Work doesn't count in the\n>> commit-graph.c model of measuring progress, literally.  I.e. progress is\n>> the same before and after one item of work.\n>\n> The progress isn't the same, we update the count. Or do you mean in the\n> time it takes us to go from the end of the for-loop & jump to the start\n> of it and update the count?\n>\n>> Instead it counts the number of loop iterations.  The model I describe\n>> above counts finished work items instead.  The results of the two\n>> models differ by at most one despite their inverted axiom regarding\n>> the value of work.\n>>\n>> Phew, that took me a while.\n>\n> For what it's worth I had some extensive examples in our initial\n> thread[1][2] (search for \"apple\" and \"throughput\", respectively), that\n> you cut out when replying to the relevant E-Mails. I'd think we could\n> probably have gotten here earlier :)\n\nPerhaps, but the key point for me was to invert my basic assumption that\na work item has value, and for that I had to realize and state it first\n(done above).  A mathematician would have done that in an instant, I\nguess (\"Invert, always invert\").\n\n> I'm a bit confused about this \"value of work\" comment.\n\nProgress is a counter.  The difference of the counter before and after\na work item is done is one in the count-work model, but zero in the\ncount-iterations model.\n\n> If you pick up a copy of say a video game like Mario Kart you'll find\n> that for a 3-lap race you start at 1/3, and still have an entire lap to\n> go when the count is at 3/3.\n>\n> So it's just a question of whether you report progress on item N or work\n> finished on item N, not whether laps in a race have more or less\n> value.\n\nThese are linked.  If you want to know which lap you are in, the answer\nwon't change until you start a new lap:\n\n\tfor (i = 0; i < 3; i++) {\n\t\tdisplay_progress(p, i + 1);\n\t\tdrive_one_lap();\n\t\tdisplay_progress(p, i + 1);\n\t}\n\nIf you want for know how many laps you finished, the answer will\nincrease after a lap is done:\n\n\tfor (i = 0; i < 3; i++) {\n\t\tdisplay_progress(p, i);\n\t\tdrive_one_lap();\n\t\tdisplay_progress(p, i + 1);\n\t}\n\n> To reference my earlier E-Mail[1] are you eating the first apple or the\n> zeroeth apple? I don't think one is more or less right in the\n> mathematical sense, I just think for UX aimed at people counting \"laps\"\n> makes more sense than counting completed items.\n\nThe difference between counting iterations and work items vanishes as\ntheir numbers increase.  The most pronounced difference is observed when\nthere is only a single item of work.  The count-iterations model shows\n1/1 from start to finish.  The count-work model shows 0/1 initially and\n1/1 after the work is done.\n\nAs a user I prefer the second one.  If presented with just a number and\na percentage then I assume 100% means all work is done and would cancel\nthe program if that status is shown for too long.  With Git I have\nlearned that only the final \", done\" really means done in some cases,\nbut that's an unnecessary lesson and still surprising to me.\n\n>>> Anyway, aside from that. I think, and I really would be advocating this\n>>> too, even if our respective positions were reversed, that *in this case*\n>>> it makes sense to just take something like SZEDER's patch here\n>>> as-is. Because in that file there's some dozen occurrences of that exact\n>>> pattern.\n>>\n>> The code without the patch either forgets to report the last work item\n>> in the count-work-items model or is one short in the count-iterations\n>> model, so a fix is needed either way.\n>\n> It won't be one short, for a loop of 2 items we'll go from:\n>\n>      0/2\n>      1/2\n>      1/2, done\n>\n> To:\n>\n>      1/2\n>      2/2\n>      2/2, done\n>\n> Just like the rest of the uses of the progress API in that file.\n\nYes, just like I wrote -- the old code is one short compared to the\ncorrect output of the count-iterations method.\n\nFor completeness' sake, the correct output of the count-work method\nwould be:\n\n\t0/2\n\t1/2\n\t2/2\n\t2/2, done\n\n> Which is one of the two reasons I prefer this pattern, i.e. this is less\n> verbose:\n>\n>     start_progress()\n>     for i in (0..X-1):\n>         display_progress(i+1)\n>         work()\n>     stop_progress()\n>\n> Than one of these, which AFAICT would be your recommendation:\n>\n>     # Simplest, but stalls on work()\n>     start_progress()\n>     for i in (0..X-1):\n>         work()\n>         display_progress(i+1)\n>     stop_progress()\n>\n>     # More verbose, but doesn't:\n>     start_progress()\n>     for i in (0..X-1):\n>         display_progress(i)\n>         work()\n>         display_progress(i+1)\n>     stop_progress()\n>\n>     # Ditto:\n>     start_progress()\n>     display_progress(0)\n>     for i in (0..X-1):\n>         work()\n>         display_progress(i+1)\n>     stop_progress()\n>\n> And of course if your loop continues or whatever you'll need a last\n> \"display_progress(X)\" before the \"stop_progress()\".\n\nThe count-work model needs one more progress update than the\ncount-iteration model.  We could do all updates in the loop header,\nwhich is evaluated just the right number of times.  But I think that we\nrather should choose between the models based on their results.\n\nIf each work item finishes within a progress display update period\n(half a second) then there won't be any user-visible difference and\nboth models would do.\n\n> The other is that if you count laps you can have your progress bar\n> optionally show progress on that item. E.g. we could if we stall show\n> seconds spend that we're hung on that item, or '3/3 ETA 40s\". I have a\n> patch[3] that takes an initial step towards that, with some more queued\n> locally.\n\nA time estimate for the whole operation (until \", done\") would be nice.\nIt can help with the decision to go for a break or to keep staring at\nthe screen.  I guess we just need to remember when start_progress() was\ncalled and can then estimate the remaining time once the first item is\ndone.  Stalling items would push the estimate further into the future.\n\nA time estimate per item wouldn't help me much.  I'd have to subtract\nto get the number of unfinished items, catch the maximum estimated\nduration and multiply those values.  OK, by the time I manage that Git\nis probably done -- but I'd rather like to leave arithmetic tasks to\nthe computer..\n\nSeconds spent for the current item can be shown with both models.  The\nprogress value is not sufficient to identify the problem case in most\ncases.  An ID of some kind (e.g. a file name or hash) would have to be\nshown as well for that.  But how would I use that information?\n\nRené\n"},{"id":"428533","messageId":"874kdkxot4.fsf@evledraar.gmail.com","threadId":"55972","inReplyTo":"177c59bf-98e1-ff55-1b75-ea89c0de976a@web.de","subject":"Re: [PATCH 4/7] commit-graph: fix bogus counter in \"Scanning merged commits\" progress line","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-26T21:38:38Z","receivedAt":"2021-06-26T22:07:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Jun 26 2021, René Scharfe wrote:\n\n> Am 26.06.21 um 16:11 schrieb Ævar Arnfjörð Bjarmason:\n>> [...]\n>> To reference my earlier E-Mail[1] are you eating the first apple or the\n>> zeroeth apple? I don't think one is more or less right in the\n>> mathematical sense, I just think for UX aimed at people counting \"laps\"\n>> makes more sense than counting completed items.\n>\n> The difference between counting iterations and work items vanishes as\n> their numbers increase.  The most pronounced difference is observed when\n> there is only a single item of work.  The count-iterations model shows\n> 1/1 from start to finish.  The count-work model shows 0/1 initially and\n> 1/1 after the work is done.\n>\n> As a user I prefer the second one.  If presented with just a number and\n> a percentage then I assume 100% means all work is done and would cancel\n> the program if that status is shown for too long.  With Git I have\n> learned that only the final \", done\" really means done in some cases,\n> but that's an unnecessary lesson and still surprising to me.\n\nWhat progress bar of ours goes slow enough that the difference matters\nfor you in either case?\n\nThe only one I know of is \"Enumerating objects\", which notably stalls at\nthe start, and which I'm proposing changing the output of in:\nhttps://lore.kernel.org/git/patch-18.25-e21fc66623f-20210623T155626Z-avarab@gmail.com/\n\n>> [...]\n>> Which is one of the two reasons I prefer this pattern, i.e. this is less\n>> verbose:\n>>\n>>     start_progress()\n>>     for i in (0..X-1):\n>>         display_progress(i+1)\n>>         work()\n>>     stop_progress()\n>>\n>> Than one of these, which AFAICT would be your recommendation:\n>>\n>>     # Simplest, but stalls on work()\n>>     start_progress()\n>>     for i in (0..X-1):\n>>         work()\n>>         display_progress(i+1)\n>>     stop_progress()\n>>\n>>     # More verbose, but doesn't:\n>>     start_progress()\n>>     for i in (0..X-1):\n>>         display_progress(i)\n>>         work()\n>>         display_progress(i+1)\n>>     stop_progress()\n>>\n>>     # Ditto:\n>>     start_progress()\n>>     display_progress(0)\n>>     for i in (0..X-1):\n>>         work()\n>>         display_progress(i+1)\n>>     stop_progress()\n>>\n>> And of course if your loop continues or whatever you'll need a last\n>> \"display_progress(X)\" before the \"stop_progress()\".\n>\n> The count-work model needs one more progress update than the\n> count-iteration model.  We could do all updates in the loop header,\n> which is evaluated just the right number of times.  But I think that we\n> rather should choose between the models based on their results.\n\nI think we should be more biased towards API convenience here than\nanything else, because for most of these they'll go so fast that users\nwon't see the difference. I just also happen to think that the easy way\nto do it is also more correct.\n\nAlso, because for these cases that you're focusing on where we count up\nto exactly 100% and we therefore expect N calls to display_progress()\n(igroning the rare but allowed duplicate calls with the same number,\nwhich most callers don't use). We could just have a convenience API of:\n\n    start_progress()\n    for i in (0..X-1):\n        progress_update() /* passing \"i\" not needed, we increment internally */\n        work()\n    stop_progress()\n\nThen we could even make showing 0/N or 1/N the first time configuable,\nbut we could only do both if we use the API as I'm suggesting, not as\nyou want to use it.\n\nYou also sort of can get me what I want with with what you're\nsuggesting, but you'd conflate \"setup\" work with the first item, which\nmatters e.g. for \"Enumerating objects\" and my \"stalled\" patch. It's also\nmore verbose at the code level, and complex (need to deal with \"break\",\n\"continue\"), so why would you?\n\nWhich I think is the main point of our not so much disagreement but I\nthink a bit of talking past one another.\n\nI.e. I think you're narrowly focused on what I think of as a display\nissue of the current progress bars we show, I'm mainly interested in how\nwe use the API, and we should pick a way to use it that allows us to do\nmore with displaying progress better in the future.\n\n> If each work item finishes within a progress display update period\n> (half a second) then there won't be any user-visible difference and\n> both models would do.\n\nA trivial point, but don't you mean a second? AFAICT for \"delayed\" we\ndisplay after 2 seconds, then update every 1 seconds, it's only if we\nhave display_throughput() that we do every 0.5s.\n\n>> The other is that if you count laps you can have your progress bar\n>> optionally show progress on that item. E.g. we could if we stall show\n>> seconds spend that we're hung on that item, or '3/3 ETA 40s\". I have a\n>> patch[3] that takes an initial step towards that, with some more queued\n>> locally.\n>\n> A time estimate for the whole operation (until \", done\") would be nice.\n> It can help with the decision to go for a break or to keep staring at\n> the screen.  I guess we just need to remember when start_progress() was\n> called and can then estimate the remaining time once the first item is\n> done.  Stalling items would push the estimate further into the future.\n>\n> A time estimate per item wouldn't help me much.  I'd have to subtract\n> to get the number of unfinished items, catch the maximum estimated\n> duration and multiply those values.  OK, by the time I manage that Git\n> is probably done -- but I'd rather like to leave arithmetic tasks to\n> the computer..\n>\n> Seconds spent for the current item can be shown with both models.  The\n> progress value is not sufficient to identify the problem case in most\n> cases.  An ID of some kind (e.g. a file name or hash) would have to be\n> shown as well for that.  But how would I use that information?\n\nIf we're spending enough time on one item to update progress for it N\ntimes we probably want to show throughput/progress/ETA mainly for that\nitem, not the work as a whole.\n\nIf we do run into those cases and want to convert them to show some\nintra-item progress we'd need to first migrate them over to suggested\nway of using the API if we picked yours first, with my suggested use we\nonly need to add new API calls (display_throughput(), and similar future\ncalls/implicit display).\n\nConsider e.g. using the packfile-uri response to ask the user to\ndownload N number of URLs, just because we grab one at 1MB/s that\nprobably won't do much to inform our estimate of the next one (which may\nbe on a different CDN etc.).\n\nThe throughput API was intended (and mainly used) for the estimate for\nthe whole batch, I just wonder if as we use it more widely whether that\nuse-case won't be the exception.\n"},{"id":"428572","messageId":"60d8b5f080160_ba5a220852@natae.notmuch","threadId":"55972","inReplyTo":"177c59bf-98e1-ff55-1b75-ea89c0de976a@web.de","subject":"Re: [PATCH 4/7] commit-graph: fix bogus counter in \"Scanning merged commits\" progress line","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2021-06-27T17:31:28Z","receivedAt":"2021-06-27T17:31:34Z","isPatch":true,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"René Scharfe wrote:\n> Am 26.06.21 um 16:11 schrieb Ævar Arnfjörð Bjarmason:\n\n> > For what it's worth I had some extensive examples in our initial\n> > thread[1][2] (search for \"apple\" and \"throughput\", respectively), that\n> > you cut out when replying to the relevant E-Mails. I'd think we could\n> > probably have gotten here earlier :)\n> \n> Perhaps, but the key point for me was to invert my basic assumption that\n> a work item has value, and for that I had to realize and state it first\n> (done above).  A mathematician would have done that in an instant, I\n> guess (\"Invert, always invert\").\n\nWhen you get down to it, numbers almost never mean what most people\nthink they mean.\n\nIf work is a continuum, the probabilty that you would land exactly at\n1/3 is 0 P(X=1/3). What you want is the probability of less than 1/3\nP(X<=1/3), and that includes 0.\n\nSo, anything from 0 to 1/3 is part of the first chunk of work.\n\n-- \nFelipe Contreras"},{"id":"429164","messageId":"ca3eba46-7c83-51fd-2d65-c55309ac7182@web.de","threadId":"55972","inReplyTo":"874kdkxot4.fsf@evledraar.gmail.com","subject":"Re: [PATCH 4/7] commit-graph: fix bogus counter in \"Scanning merged commits\" progress line","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-07-04T12:15:34Z","receivedAt":"2021-07-04T12:15:38Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 26.06.21 um 23:38 schrieb Ævar Arnfjörð Bjarmason:\n>\n> On Sat, Jun 26 2021, René Scharfe wrote:\n>\n>> Am 26.06.21 um 16:11 schrieb Ævar Arnfjörð Bjarmason:\n>>> [...]\n>>> To reference my earlier E-Mail[1] are you eating the first apple or the\n>>> zeroeth apple? I don't think one is more or less right in the\n>>> mathematical sense, I just think for UX aimed at people counting \"laps\"\n>>> makes more sense than counting completed items.\n>>\n>> The difference between counting iterations and work items vanishes as\n>> their numbers increase.  The most pronounced difference is observed when\n>> there is only a single item of work.  The count-iterations model shows\n>> 1/1 from start to finish.  The count-work model shows 0/1 initially and\n>> 1/1 after the work is done.\n>>\n>> As a user I prefer the second one.  If presented with just a number and\n>> a percentage then I assume 100% means all work is done and would cancel\n>> the program if that status is shown for too long.  With Git I have\n>> learned that only the final \", done\" really means done in some cases,\n>> but that's an unnecessary lesson and still surprising to me.\n>\n> What progress bar of ours goes slow enough that the difference matters\n> for you in either case?\n\nI don't have an example -- Git, network and SSD are quick enough for my\nsmall use cases.\n\nThe advantage of the count-work method is that the question doesn't even\ncome up.\n\n> The only one I know of is \"Enumerating objects\", which notably stalls at\n> the start, and which I'm proposing changing the output of in:\n> https://lore.kernel.org/git/patch-18.25-e21fc66623f-20210623T155626Z-avarab@gmail.com/\n\nThat's annoying, but the first number I see there has five or six digits,\nso it's not an example of the issue mentioned above for me.\n\nYour patch shows \", stalled.\" while pack-objects starts up.  I'm not sure\nthis helps.  Perhaps there are cases when it gets stuck, but it's hard to\ndetermine by the clock alone.  When I run gc, it just needs a few seconds\nto prepare something and then starts visibly counting objects.  A more\nfine-grained report of the preparation steps would help, but seeing\n\"stalled\" would just scare me.\n\n>> The count-work model needs one more progress update than the\n>> count-iteration model.  We could do all updates in the loop header,\n>> which is evaluated just the right number of times.  But I think that we\n>> rather should choose between the models based on their results.\n>\n> I think we should be more biased towards API convenience here than\n> anything else, because for most of these they'll go so fast that users\n> won't see the difference. I just also happen to think that the easy way\n> to do it is also more correct.\n\nThe convenience of having one less display_progress() call is only a\nslight advantage.\n\nCorrectness is a matter of definitions.  Recently I learned that in Arabic\na person's age is given using the count-iterations model.  I.e. on the day\nof your birth your age is one.  That causes trouble if you deal with\nstate officials that use the count-work, err, count-completed-years model,\nwhere your age is one only after living through a full year.\n\nThe solution around here is to avoid ambiguity by not using terms like\n\"age\" in laws, regulations and forms, but to state explicitly \"full years\nsince birth\" or so.\n\n\"2/3 (33%)\" means something else to me than to you by default.  So a\nsolution could be to state the model explicitly.  I.e. \"2/3 (66%) done\"\nor \"working on 2/3 (66%)\", but the percentage doesn't quite fit in the\nlatter case.  Thoughts?\n\n> Also, because for these cases that you're focusing on where we count up\n> to exactly 100% and we therefore expect N calls to display_progress()\n> (igroning the rare but allowed duplicate calls with the same number,\n> which most callers don't use). We could just have a convenience API of:\n>\n>     start_progress()\n>     for i in (0..X-1):\n>         progress_update() /* passing \"i\" not needed, we increment internally */\n>         work()\n>     stop_progress()\n>\n> Then we could even make showing 0/N or 1/N the first time configuable,\n> but we could only do both if we use the API as I'm suggesting, not as\n> you want to use it.\n\nA function that increments the progress number relatively can be used\nwith both models.  It's more useful for the count-iterations model,\nthough, as in the count-work model you can piggy-back on the loop\ncounter check:\n\n\tfor (i = 0; display_progress(p, i), i < X; i++)\n\t\twork();\n\n> You also sort of can get me what I want with with what you're\n> suggesting, but you'd conflate \"setup\" work with the first item, which\n> matters e.g. for \"Enumerating objects\" and my \"stalled\" patch. It's also\n> more verbose at the code level, and complex (need to deal with \"break\",\n> \"continue\"), so why would you?\n\nIt's not complicated, just slightly odd, because function calls are\nseldomly put into the loop counter check.\n\n>> If each work item finishes within a progress display update period\n>> (half a second) then there won't be any user-visible difference and\n>> both models would do.\n>\n> A trivial point, but don't you mean a second? AFAICT for \"delayed\" we\n> display after 2 seconds, then update every 1 seconds, it's only if we\n> have display_throughput() that we do every 0.5s.\n\nRight, I mixed those up.\n\n>>> The other is that if you count laps you can have your progress bar\n>>> optionally show progress on that item. E.g. we could if we stall show\n>>> seconds spend that we're hung on that item, or '3/3 ETA 40s\". I have a\n>>> patch[3] that takes an initial step towards that, with some more queued\n>>> locally.\n>>\n>> A time estimate for the whole operation (until \", done\") would be nice.\n>> It can help with the decision to go for a break or to keep staring at\n>> the screen.  I guess we just need to remember when start_progress() was\n>> called and can then estimate the remaining time once the first item is\n>> done.  Stalling items would push the estimate further into the future.\n>>\n>> A time estimate per item wouldn't help me much.  I'd have to subtract\n>> to get the number of unfinished items, catch the maximum estimated\n>> duration and multiply those values.  OK, by the time I manage that Git\n>> is probably done -- but I'd rather like to leave arithmetic tasks to\n>> the computer..\n>>\n>> Seconds spent for the current item can be shown with both models.  The\n>> progress value is not sufficient to identify the problem case in most\n>> cases.  An ID of some kind (e.g. a file name or hash) would have to be\n>> shown as well for that.  But how would I use that information?\n>\n> If we're spending enough time on one item to update progress for it N\n> times we probably want to show throughput/progress/ETA mainly for that\n> item, not the work as a whole.\n\nThroughput is shown for the last time period.  It is independent of the\nitem or items being worked on during that period.  If one item takes\nmultiple periods to finish then indeed only its current progress is\nshown automatically, as you want.\n\nShowing intra-item progress requires some kind of hierarchical API to\nkeep track of both parent and child progress and show them in some\nreadable way.  Perhaps appending another progress display would suffice?\n\"Files 1/3 (33%) Bytes 17kB/9GB (0%)\".  Not sure.\n\nCalculating the ETA of a single item seems hard.  It does require intra-\nitem progress to be reported by the work code.\n\n> If we do run into those cases and want to convert them to show some\n> intra-item progress we'd need to first migrate them over to suggested\n> way of using the API if we picked yours first, with my suggested use we\n> only need to add new API calls (display_throughput(), and similar future\n> calls/implicit display).\n\nI don't see why.  The intra-item progress numbers need to be reported in\nany case if they are to be shown somehow.  If the model is clear then we\ncan show unambiguous output.\n\n> Consider e.g. using the packfile-uri response to ask the user to\n> download N number of URLs, just because we grab one at 1MB/s that\n> probably won't do much to inform our estimate of the next one (which may\n> be on a different CDN etc.).\n\nSure, if the speed of work items varies wildly then estimates will be\nunreliable.\n\nI can vaguely imagine that it would be kind of useful to know the\nthroughput of different data sources, to allow e.g. use a different\nmirror next time.  The current API doesn't distinguish work items in a\nmeaningful way, though.  They only have numbers.  I'd need a name (e.g.\nthe URL) for intra-item progress numbers to mean something.\n\nRené\n"},{"id":"429241","messageId":"xmqqtul8n97x.fsf@gitster.g","threadId":"55972","inReplyTo":"ca3eba46-7c83-51fd-2d65-c55309ac7182@web.de","subject":"Re: [PATCH 4/7] commit-graph: fix bogus counter in \"Scanning merged commits\" progress line","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-05T14:09:22Z","receivedAt":"2021-07-05T14:09:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe <l.s.r@web.de> writes:\n\n> ...  A more\n> fine-grained report of the preparation steps would help, but seeing\n> \"stalled\" would just scare me.\n\nTrue.\n\n> The convenience of having one less display_progress() call is only a\n> slight advantage.\n\nTrue, too.\n\n> \"2/3 (33%)\" means something else to me than to you by default.  So a\n> solution could be to state the model explicitly.  I.e. \"2/3 (66%) done\"\n> or \"working on 2/3 (66%)\", but the percentage doesn't quite fit in the\n> latter case.  Thoughts?\n\n\nI still see \"2/3 done\" is how we should look at it, but either way,\nthat's a good way to view at the problem.\n\nThanks.\n\n\n[Unrelated Tangent]\n\n> ...  Recently I learned that in Arabic a person's age is given\n> using the count-iterations model.  I.e. on the day of your birth\n> your age is one.\n\nEast Asign age reckoning is shared among EA countries and works the\nsame way, not just Arabic.\n"},{"id":"429271","messageId":"87y2ak5noa.fsf@evledraar.gmail.com","threadId":"55972","inReplyTo":"ca3eba46-7c83-51fd-2d65-c55309ac7182@web.de","subject":"Re: [PATCH 4/7] commit-graph: fix bogus counter in \"Scanning merged commits\" progress line","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-05T23:28:44Z","receivedAt":"2021-07-05T23:46:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Jul 04 2021, René Scharfe wrote:\n\n> Am 26.06.21 um 23:38 schrieb Ævar Arnfjörð Bjarmason:\n>>\n>> On Sat, Jun 26 2021, René Scharfe wrote:\n>>\n>>> Am 26.06.21 um 16:11 schrieb Ævar Arnfjörð Bjarmason:\n>>>> [...]\n>>>> To reference my earlier E-Mail[1] are you eating the first apple or the\n>>>> zeroeth apple? I don't think one is more or less right in the\n>>>> mathematical sense, I just think for UX aimed at people counting \"laps\"\n>>>> makes more sense than counting completed items.\n>>>\n>>> The difference between counting iterations and work items vanishes as\n>>> their numbers increase.  The most pronounced difference is observed when\n>>> there is only a single item of work.  The count-iterations model shows\n>>> 1/1 from start to finish.  The count-work model shows 0/1 initially and\n>>> 1/1 after the work is done.\n>>>\n>>> As a user I prefer the second one.  If presented with just a number and\n>>> a percentage then I assume 100% means all work is done and would cancel\n>>> the program if that status is shown for too long.  With Git I have\n>>> learned that only the final \", done\" really means done in some cases,\n>>> but that's an unnecessary lesson and still surprising to me.\n>>\n>> What progress bar of ours goes slow enough that the difference matters\n>> for you in either case?\n>\n> I don't have an example -- Git, network and SSD are quick enough for my\n> small use cases.\n>\n> The advantage of the count-work method is that the question doesn't even\n> come up.\n>\n>> The only one I know of is \"Enumerating objects\", which notably stalls at\n>> the start, and which I'm proposing changing the output of in:\n>> https://lore.kernel.org/git/patch-18.25-e21fc66623f-20210623T155626Z-avarab@gmail.com/\n>\n> That's annoying, but the first number I see there has five or six digits,\n> so it's not an example of the issue mentioned above for me.\n\nBecause it stalls and shows nothing, but with my patch it'll show\nsomething while stalling, FWIW on linux.git from a cold cache it took\n5-10s before showing anything.\n\n> Your patch shows \", stalled.\" while pack-objects starts up.  I'm not sure\n> this helps.  Perhaps there are cases when it gets stuck, but it's hard to\n> determine by the clock alone.  When I run gc, it just needs a few seconds\n> to prepare something and then starts visibly counting objects.  A more\n> fine-grained report of the preparation steps would help, but seeing\n> \"stalled\" would just scare me.\n\nFair enough, I have other patches to have it show a spinner. Again, API\nv.s. UI. The idea is that we show something before we start the loop.\n\n>>> The count-work model needs one more progress update than the\n>>> count-iteration model.  We could do all updates in the loop header,\n>>> which is evaluated just the right number of times.  But I think that we\n>>> rather should choose between the models based on their results.\n>>\n>> I think we should be more biased towards API convenience here than\n>> anything else, because for most of these they'll go so fast that users\n>> won't see the difference. I just also happen to think that the easy way\n>> to do it is also more correct.\n>\n> The convenience of having one less display_progress() call is only a\n> slight advantage.\n>\n> Correctness is a matter of definitions.  Recently I learned that in Arabic\n> a person's age is given using the count-iterations model.  I.e. on the day\n> of your birth your age is one.  That causes trouble if you deal with\n> state officials that use the count-work, err, count-completed-years model,\n> where your age is one only after living through a full year.\n>\n> The solution around here is to avoid ambiguity by not using terms like\n> \"age\" in laws, regulations and forms, but to state explicitly \"full years\n> since birth\" or so.\n>\n> \"2/3 (33%)\" means something else to me than to you by default.  So a\n> solution could be to state the model explicitly.  I.e. \"2/3 (66%) done\"\n> or \"working on 2/3 (66%)\", but the percentage doesn't quite fit in the\n> latter case.  Thoughts?\n\nOK, UI again.\n\n>> Also, because for these cases that you're focusing on where we count up\n>> to exactly 100% and we therefore expect N calls to display_progress()\n>> (igroning the rare but allowed duplicate calls with the same number,\n>> which most callers don't use). We could just have a convenience API of:\n>>\n>>     start_progress()\n>>     for i in (0..X-1):\n>>         progress_update() /* passing \"i\" not needed, we increment internally */\n>>         work()\n>>     stop_progress()\n>>\n>> Then we could even make showing 0/N or 1/N the first time configuable,\n>> but we could only do both if we use the API as I'm suggesting, not as\n>> you want to use it.\n>\n> A function that increments the progress number relatively can be used\n> with both models.  It's more useful for the count-iterations model,\n> though, as in the count-work model you can piggy-back on the loop\n> counter check:\n>\n> \tfor (i = 0; display_progress(p, i), i < X; i++)\n> \t\twork();\n\nAside from this whole progress API discussion I find sticking stuff like\nthat in the for-loop body to be less readable.\n\nBut no, that can't be used with both models, because it conflates the 0\nof the 1st iteration with 0 of doing prep work. I.e.:\n\n    p = start_progress();\n    display_progress(p, 0);\n    prep_work();\n    for (i = 0; i < 100; i++)\n        display_progress(p, i + 1);\n\nWhich is implicitly how that \"stalled\" patch views the world, i.e. our\ncount is -1 is at start_progress() (that's already the case in\nprogress.c).\n\nIf you set it to 0 you're not working on the 1st item yet, but\nexplicitly doing setup. \n\nThen at n=1 you're starting work on the 1st item.\n\n>> You also sort of can get me what I want with with what you're\n>> suggesting, but you'd conflate \"setup\" work with the first item, which\n>> matters e.g. for \"Enumerating objects\" and my \"stalled\" patch. It's also\n>> more verbose at the code level, and complex (need to deal with \"break\",\n>> \"continue\"), so why would you?\n>\n> It's not complicated, just slightly odd, because function calls are\n> seldomly put into the loop counter check.\n\nFWIW the \"complicated\" here was referring to dealing with break/continue.\n\nYes I'll grant you that there's cases where the uglyness/oddity of that\nfor-loop trick is going to be better than dealing with that, but there's\nalso while loops doing progress, callbacks etc.\n\nPicking an API pattern that works with all of that makes sense, since\nthe UI can render the count one way or the other.\n\n>>> If each work item finishes within a progress display update period\n>>> (half a second) then there won't be any user-visible difference and\n>>> both models would do.\n>>\n>> A trivial point, but don't you mean a second? AFAICT for \"delayed\" we\n>> display after 2 seconds, then update every 1 seconds, it's only if we\n>> have display_throughput() that we do every 0.5s.\n>\n> Right, I mixed those up.\n>\n>>>> The other is that if you count laps you can have your progress bar\n>>>> optionally show progress on that item. E.g. we could if we stall show\n>>>> seconds spend that we're hung on that item, or '3/3 ETA 40s\". I have a\n>>>> patch[3] that takes an initial step towards that, with some more queued\n>>>> locally.\n>>>\n>>> A time estimate for the whole operation (until \", done\") would be nice.\n>>> It can help with the decision to go for a break or to keep staring at\n>>> the screen.  I guess we just need to remember when start_progress() was\n>>> called and can then estimate the remaining time once the first item is\n>>> done.  Stalling items would push the estimate further into the future.\n>>>\n>>> A time estimate per item wouldn't help me much.  I'd have to subtract\n>>> to get the number of unfinished items, catch the maximum estimated\n>>> duration and multiply those values.  OK, by the time I manage that Git\n>>> is probably done -- but I'd rather like to leave arithmetic tasks to\n>>> the computer..\n>>>\n>>> Seconds spent for the current item can be shown with both models.  The\n>>> progress value is not sufficient to identify the problem case in most\n>>> cases.  An ID of some kind (e.g. a file name or hash) would have to be\n>>> shown as well for that.  But how would I use that information?\n>>\n>> If we're spending enough time on one item to update progress for it N\n>> times we probably want to show throughput/progress/ETA mainly for that\n>> item, not the work as a whole.\n>\n> Throughput is shown for the last time period.  It is independent of the\n> item or items being worked on during that period.  If one item takes\n> multiple periods to finish then indeed only its current progress is\n> shown automatically, as you want.\n>\n> Showing intra-item progress requires some kind of hierarchical API to\n> keep track of both parent and child progress and show them in some\n> readable way.  Perhaps appending another progress display would suffice?\n> \"Files 1/3 (33%) Bytes 17kB/9GB (0%)\".  Not sure.\n\nYes, this is another thing I'm heading for with the patches I posted.\n\nFor now I just fixed bugs in the state machine of how many characters we\nerase, now we always reset exactly as much as we need to, and pass\nthings like \", done\" around, not \", done\\n\" or \", done\\r\" (i.e. the\noutput we're emitting isn't conflacted with whether we're clearing the\nline, or creating a new line.\n\nIt's a relatively straightforward change from there to have N progress\nstructs that each track/emit their part of a larger progress bar,\ne.g. something like the progress prove(1) shows you (test status for\neach concurrent test you're running).\n\nYou just need a \"parent\" progress struct that has the \"title\" (or none),\nand receives the signal, and to have N registered sub-progress structs.\n\n> Calculating the ETA of a single item seems hard.  It does require intra-\n> item progress to be reported by the work code.\n>\n>> If we do run into those cases and want to convert them to show some\n>> intra-item progress we'd need to first migrate them over to suggested\n>> way of using the API if we picked yours first, with my suggested use we\n>> only need to add new API calls (display_throughput(), and similar future\n>> calls/implicit display).\n>\n> I don't see why.  The intra-item progress numbers need to be reported in\n> any case if they are to be shown somehow.  If the model is clear then we\n> can show unambiguous output.\n\nBecause you want to show:\n\n    Files 1/3 (33%) Bytes 17kB/9GB (0%)\n\nNot:\n\n    Files 0/3 (33%) Bytes 17kB/9GB (0%)\n\nYou're downloading the 1st file, not the 0th file, so the code is a\nfor-loop (or equivalent) with a display_progress(p, i + 1) for that\nfile, not display_progress(p, i).\n\nThis is the main reason I prefer the API and UI of reporting \"what item\nam I on?\" v.s. \"how many items are done?\", because it's easy to add\nintra-item state to the former.\n\n>> Consider e.g. using the packfile-uri response to ask the user to\n>> download N number of URLs, just because we grab one at 1MB/s that\n>> probably won't do much to inform our estimate of the next one (which may\n>> be on a different CDN etc.).\n>\n> Sure, if the speed of work items varies wildly then estimates will be\n> unreliable.\n>\n> I can vaguely imagine that it would be kind of useful to know the\n> throughput of different data sources, to allow e.g. use a different\n> mirror next time.  The current API doesn't distinguish work items in a\n> meaningful way, though.  They only have numbers.  I'd need a name (e.g.\n> the URL) for intra-item progress numbers to mean something.\n\nSure, anyway, let's assume all those numbers are magically known and\nconstant. The point was that as noted above you're downloading the 1st\nfile, not the 0th file, and want to show throughput/ETA etc. for that\nfile.\n"},{"id":"429298","messageId":"0a03bae6-53cf-98e5-635a-6f61583f5dd1@web.de","threadId":"55972","inReplyTo":"87y2ak5noa.fsf@evledraar.gmail.com","subject":"Re: [PATCH 4/7] commit-graph: fix bogus counter in \"Scanning merged commits\" progress line","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-07-06T16:02:32Z","receivedAt":"2021-07-06T16:02:35Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 06.07.21 um 01:28 schrieb Ævar Arnfjörð Bjarmason:\n>\n> On Sun, Jul 04 2021, René Scharfe wrote:\n>\n>> Am 26.06.21 um 23:38 schrieb Ævar Arnfjörð Bjarmason:\n>>>\n>>> On Sat, Jun 26 2021, René Scharfe wrote:\n>>>\n>>>> Am 26.06.21 um 16:11 schrieb Ævar Arnfjörð Bjarmason:\n>>> The only one I know of is \"Enumerating objects\", which notably stalls at\n>>> the start, and which I'm proposing changing the output of in:\n>>> https://lore.kernel.org/git/patch-18.25-e21fc66623f-20210623T155626Z-avarab@gmail.com/\n>>\n>> That's annoying, but the first number I see there has five or six digits,\n>> so it's not an example of the issue mentioned above for me.\n>\n> Because it stalls and shows nothing, but with my patch it'll show\n> something while stalling, FWIW on linux.git from a cold cache it took\n> 5-10s before showing anything.\n>\n>> Your patch shows \", stalled.\" while pack-objects starts up.  I'm not sure\n>> this helps.  Perhaps there are cases when it gets stuck, but it's hard to\n>> determine by the clock alone.  When I run gc, it just needs a few seconds\n>> to prepare something and then starts visibly counting objects.  A more\n>> fine-grained report of the preparation steps would help, but seeing\n>> \"stalled\" would just scare me.\n>\n> Fair enough, I have other patches to have it show a spinner. Again, API\n> v.s. UI. The idea is that we show something before we start the loop.\n\nA spinner would be nicer, but I would be more interested to see what it is\nactually spending all that time on.  A separate progress line might be\njustified here.\n\n>>> Also, because for these cases that you're focusing on where we count up\n>>> to exactly 100% and we therefore expect N calls to display_progress()\n>>> (igroning the rare but allowed duplicate calls with the same number,\n>>> which most callers don't use). We could just have a convenience API of:\n>>>\n>>>     start_progress()\n>>>     for i in (0..X-1):\n>>>         progress_update() /* passing \"i\" not needed, we increment internally */\n>>>         work()\n>>>     stop_progress()\n>>>\n>>> Then we could even make showing 0/N or 1/N the first time configuable,\n>>> but we could only do both if we use the API as I'm suggesting, not as\n>>> you want to use it.\n>>\n>> A function that increments the progress number relatively can be used\n>> with both models.  It's more useful for the count-iterations model,\n>> though, as in the count-work model you can piggy-back on the loop\n>> counter check:\n>>\n>> \tfor (i = 0; display_progress(p, i), i < X; i++)\n>> \t\twork();\n>\n> Aside from this whole progress API discussion I find sticking stuff like\n> that in the for-loop body to be less readable.\n>\n> But no, that can't be used with both models, because it conflates the 0\n> of the 1st iteration with 0 of doing prep work. I.e.:\n>\n>     p = start_progress();\n>     display_progress(p, 0);\n>     prep_work();\n>     for (i = 0; i < 100; i++)\n>         display_progress(p, i + 1);\n>\n> Which is implicitly how that \"stalled\" patch views the world, i.e. our\n> count is -1 is at start_progress() (that's already the case in\n> progress.c).\n>\n> If you set it to 0 you're not working on the 1st item yet, but\n> explicitly doing setup.\n>\n> Then at n=1 you're starting work on the 1st item.\n\nA distinct preparation phase feels like an extension to the progress\nAPI.  A symmetric cleanup phase at the end may make sense as well then.\n\nI assume that preparations would be done between the start_progress call\nand the first display_progress (no matter what number it reports).  And\ncleanup would be done between the last display_progress call and the\nstop_progress call.\n\nIn the count-iterations model this might report the time taken fro the\nfirst or last item as preparation or cleanup depending on the placement\nof the display_progress call.  That shouldn't be much of a problem,\nthough, as the value of one work item is zero in that model.\n\n> FWIW the \"complicated\" here was referring to dealing with break/continue.\n>\n> Yes I'll grant you that there's cases where the uglyness/oddity of that\n> for-loop trick is going to be better than dealing with that, but there's\n> also while loops doing progress, callbacks etc.\n\nwhile loops can easily be converted to for loops, of course.\n\nCallbacks are a different matter.  I think we should use them less in\ngeneral (they force different operations to use the same set of\nparameters, which is worked around with context structs).  A function\nto increment progress would help them because then they wouldn't need\nto keep track of the item/iteration count themselves in a context\nvariable.\n\nHowever, in some cases display_progress calls are rate-limited, e.g.\nmidx_display_sparse_progress does that for performance reasons.  I\nwonder why, and whether this is a problem that needs to be addressed\nfor all callers.  We don't want the progress API to delay the actual\nprogress significantly!  Currently display_progress avoids updating\nthe progress counter; an increment function would need to write an\nupdated value at each call.\n\n> Picking an API pattern that works with all of that makes sense, since\n> the UI can render the count one way or the other.\n\nRight.\n\n>>> If we do run into those cases and want to convert them to show some\n>>> intra-item progress we'd need to first migrate them over to suggested\n>>> way of using the API if we picked yours first, with my suggested use we\n>>> only need to add new API calls (display_throughput(), and similar future\n>>> calls/implicit display).\n>>\n>> I don't see why.  The intra-item progress numbers need to be reported in\n>> any case if they are to be shown somehow.  If the model is clear then we\n>> can show unambiguous output.\n>\n> Because you want to show:\n>\n>     Files 1/3 (33%) Bytes 17kB/9GB (0%)\n>\n> Not:\n>\n>     Files 0/3 (33%) Bytes 17kB/9GB (0%)\n>\n> You're downloading the 1st file, not the 0th file, so the code is a\n> for-loop (or equivalent) with a display_progress(p, i + 1) for that\n> file, not display_progress(p, i).\n>\n> This is the main reason I prefer the API and UI of reporting \"what item\n> am I on?\" v.s. \"how many items are done?\", because it's easy to add\n> intra-item state to the former.\n\nBoth look confusing.  If I'd care enough about one of the files or each\nof them that I'd like to know their individual progress then I'd\ncertainly would want to see their names instead of some random number.\n\nAnd as you write above: The display part can easily add or subtract one\nto convert the number between models.\n\n> Sure, anyway, let's assume all those numbers are magically known and\n> constant. The point was that as noted above you're downloading the 1st\n> file, not the 0th file, and want to show throughput/ETA etc. for that\n> file.\n\nOK, but still some kind of indication would have to be given that the\nBytes relate to a particular File instead of being the total for this\nactivity.  Perhaps like this, but it's a bit cluttered:\n\n   File 1 (Bytes 17kB/9GB, 0% done) of 3 (0% done in total)\n\nRené\n"},{"id":"430914","messageId":"cover-0.3-0000000000-20210722T121801Z-avarab@gmail.com","threadId":"55972","inReplyTo":"20210620200303.2328957-1-szeder.dev@gmail.com","subject":"[PATCH 0/3] progress.c API users: fix bogus counting","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-22T12:20:14Z","receivedAt":"2021-07-22T12:20:23Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"As a split-off from the larger topic these were submitted as part of\n[1] and which didn't get picked up. As I pointed out in [2] that\nlarger topic had some hidden untested-for flaws.\n\nBut these patches are just fixes to bogus progress bar output from\nthat topic. Let's consider them in isolation...\n\n1. https://lore.kernel.org/git/20210620200303.2328957-1-szeder.dev@gmail.com/\n2. https://lore.kernel.org/git/cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com/\n\nSZEDER Gábor (2):\n  commit-graph: fix bogus counter in \"Scanning merged commits\" progress\n    line\n  entry: show finer-grained counter in \"Filtering content\" progress line\n\nÆvar Arnfjörð Bjarmason (1):\n  midx: don't provide a total for QSORT() progress\n\n commit-graph.c | 2 +-\n entry.c        | 7 +++----\n midx.c         | 2 +-\n 3 files changed, 5 insertions(+), 6 deletions(-)\n\n-- \n2.32.0.957.gd9e39d72fe6\n\n"},{"id":"430915","messageId":"patch-1.3-832a6c1f78-20210722T121801Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-0.3-0000000000-20210722T121801Z-avarab@gmail.com","subject":"[PATCH 1/3] commit-graph: fix bogus counter in \"Scanning merged commits\" progress line","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-22T12:20:15Z","receivedAt":"2021-07-22T12:20:24Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: SZEDER Gábor <szeder.dev@gmail.com>\n\nThe final value of the counter of the \"Scanning merged commits\"\nprogress line is always one less than its expected total, e.g.:\n\n  Scanning merged commits:  83% (5/6), done.\n\nThis happens because while iterating over an array the loop variable\nis passed to display_progress() as-is, but while C arrays (and thus\nthe loop variable) start at 0 and end at N-1, the progress counter\nmust end at N.  This causes the failures of the tests\n'fetch.writeCommitGraph' and 'fetch.writeCommitGraph with submodules'\nin 't5510-fetch.sh' when run with GIT_TEST_CHECK_PROGRESS=1.\n\nFix this by passing 'i + 1' to display_progress(), like most other\ncallsites do.\n\nSigned-off-by: SZEDER Gábor <szeder.dev@gmail.com>\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n commit-graph.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 1a2602da61..918061f207 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -2096,7 +2096,7 @@ static void sort_and_scan_merged_commits(struct write_commit_graph_context *ctx)\n \n \tctx->num_extra_edges = 0;\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n-\t\tdisplay_progress(ctx->progress, i);\n+\t\tdisplay_progress(ctx->progress, i + 1);\n \n \t\tif (i && oideq(&ctx->commits.list[i - 1]->object.oid,\n \t\t\t  &ctx->commits.list[i]->object.oid)) {\n-- \n2.32.0.957.gd9e39d72fe6\n\n"},{"id":"430916","messageId":"patch-2.3-3411fe0515-20210722T121801Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-0.3-0000000000-20210722T121801Z-avarab@gmail.com","subject":"[PATCH 2/3] midx: don't provide a total for QSORT() progress","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-22T12:20:16Z","receivedAt":"2021-07-22T12:20:28Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The quicksort algorithm can be anywhere between O(n) and O(n^2), so\nproviding a \"num objects\" as a total means that in some cases we're\ngoing to go past 100%.\n\nThis fixes a logic error in 5ae18df9d8e (midx: during verify group\nobjects by packfile to speed verification, 2019-03-21), which in turn\nseems to have been diligently copied from my own logic error in the\ncommit-graph.c code, see 890226ccb57 (commit-graph write: add\nitermediate progress, 2019-01-19).\n\nThat commit-graph code of mine was removed in\n1cbdbf3bef7 (commit-graph: drop count_distinct_commits() function,\n2020-12-07), so we don't need to fix that too.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n midx.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 9a35b0255d..eaae75ab19 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1291,7 +1291,7 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \n \tif (flags & MIDX_PROGRESS)\n \t\tprogress = start_sparse_progress(_(\"Sorting objects by packfile\"),\n-\t\t\t\t\t\t m->num_objects);\n+\t\t\t\t\t\t 0);\n \tdisplay_progress(progress, 0); /* TODO: Measure QSORT() progress */\n \tQSORT(pairs, m->num_objects, compare_pair_pos_vs_id);\n \tstop_progress(&progress);\n-- \n2.32.0.957.gd9e39d72fe6\n\n"},{"id":"430917","messageId":"patch-3.3-f65001627a-20210722T121801Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-0.3-0000000000-20210722T121801Z-avarab@gmail.com","subject":"[PATCH 3/3] entry: show finer-grained counter in \"Filtering content\" progress line","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-22T12:20:17Z","receivedAt":"2021-07-22T12:20:30Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: SZEDER Gábor <szeder.dev@gmail.com>\n\nThe \"Filtering content\" progress in entry.c:finish_delayed_checkout()\nis unusual because of how it calculates the progress count and because\nit shows the progress of a nested loop.  It works basically like this:\n\n  start_delayed_progress(p, nr_of_paths_to_filter)\n  for_each_filter {\n      display_progress(p, nr_of_paths_to_filter - nr_of_paths_still_left_to_filter)\n      for_each_path_handled_by_the_current_filter {\n          checkout_entry()\n      }\n  }\n  stop_progress(p)\n\nThere are two issues with this approach:\n\n  - The work done by the last filter (or the only filter if there is\n    only one) is never counted, so if the last filter still has some\n    paths to process, then the counter shown in the \"done\" progress\n    line will not match the expected total.\n\n    This would cause a BUG() in an upcoming change that adds an\n    assertion checking if the \"total\" at the end matches the last\n    progress bar update..\n\n    This is because both use only one filter.  (The test 'delayed\n    checkout in process filter' uses two filters but the first one\n    does all the work, so that test already happens to succeed even\n    with such an assertion.)\n\n  - The progress counter is updated only once per filter, not once per\n    processed path, so if a filter has a lot of paths to process, then\n    the counter might stay unchanged for a long while and then make a\n    big jump (though the user still gets a sense of progress, because\n    we call display_throughput() after each processed path to show the\n    amount of processed data).\n\nMove the display_progress() call to the inner loop, right next to that\ncheckout_entry() call that does the hard work for each path, and use a\ndedicated counter variable that is incremented upon processing each\npath.\n\nAfter this change the 'invalid file in delayed checkout' in\n't0021-conversion.sh' would succeed with the future BUG() assertion\ndiscussed above but the 'missing file in delayed checkout' test would\nstill fail, because its purposefully buggy filter doesn't process any\npaths, so we won't execute that inner loop at all (this will be fixed\nin a subsequent commit).\n\nSigned-off-by: SZEDER Gábor <szeder.dev@gmail.com>\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n entry.c | 7 +++----\n 1 file changed, 3 insertions(+), 4 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex 125fabdbd5..d92dd020b3 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -162,7 +162,7 @@ static int remove_available_paths(struct string_list_item *item, void *cb_data)\n int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n {\n \tint errs = 0;\n-\tunsigned delayed_object_count;\n+\tunsigned processed_paths = 0;\n \toff_t filtered_bytes = 0;\n \tstruct string_list_item *filter, *path;\n \tstruct progress *progress;\n@@ -172,12 +172,10 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n \t\treturn errs;\n \n \tdco->state = CE_RETRY;\n-\tdelayed_object_count = dco->paths.nr;\n-\tprogress = start_delayed_progress(_(\"Filtering content\"), delayed_object_count);\n+\tprogress = start_delayed_progress(_(\"Filtering content\"), dco->paths.nr);\n \twhile (dco->filters.nr > 0) {\n \t\tfor_each_string_list_item(filter, &dco->filters) {\n \t\t\tstruct string_list available_paths = STRING_LIST_INIT_NODUP;\n-\t\t\tdisplay_progress(progress, delayed_object_count - dco->paths.nr);\n \n \t\t\tif (!async_query_available_blobs(filter->string, &available_paths)) {\n \t\t\t\t/* Filter reported an error */\n@@ -224,6 +222,7 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n \t\t\t\tce = index_file_exists(state->istate, path->string,\n \t\t\t\t\t\t       strlen(path->string), 0);\n \t\t\t\tif (ce) {\n+\t\t\t\t\tdisplay_progress(progress, ++processed_paths);\n \t\t\t\t\terrs |= checkout_entry(ce, state, NULL, nr_checkouts);\n \t\t\t\t\tfiltered_bytes += ce->ce_stat_data.sd_size;\n \t\t\t\t\tdisplay_throughput(progress, filtered_bytes);\n-- \n2.32.0.957.gd9e39d72fe6\n\n"},{"id":"430919","messageId":"cover-0.8-0000000000-20210722T125012Z-avarab@gmail.com","threadId":"55972","inReplyTo":"20210620200303.2328957-1-szeder.dev@gmail.com","subject":"[PATCH 0/8] progress: assert \"global_progress\" + test fixes / cleanup","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-22T12:54:58Z","receivedAt":"2021-07-22T12:55:13Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"These patches were originally submitted as part of a much larger topic\nat [1]. The add a \"global_progress\" \"struct progress *\" which we\nassign/clear to as we start/stop progress bars.\n\nThis will become imporant for some new progress features I have\nplanend, but right now is just used to assert that we don't start two\nprogress bars at the same time. 7/8 fixes an existing bug where we did\nthat.\n\nTo get there I fixed up the test helper to be able to test this, moved\nsome code around, and fixes a couple of existing nits in 5/8 and 6/8..\n\nSee also [2] which is a re-submission of that larger topic, but the\ntwo can proceed independently.\n\n1. https://lore.kernel.org/git/cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com/\n2. https://lore.kernel.org/git/cover-0.3-0000000000-20210722T121801Z-avarab@gmail.com/\n\nÆvar Arnfjörð Bjarmason (8):\n  progress.c tests: make start/stop verbs on stdin\n  progress.c tests: test some invalid usage\n  progress.c: move signal handler functions lower\n  progress.c: call progress_interval() from progress_test_force_update()\n  progress.c: stop eagerly fflush(stderr) when not a terminal\n  progress.c: add temporary variable from progress struct\n  pack-bitmap-write.c: add a missing stop_progress()\n  progress.c: add & assert a \"global_progress\" variable\n\n pack-bitmap-write.c         |   1 +\n progress.c                  | 116 ++++++++++++++++++++----------------\n t/helper/test-progress.c    |  43 +++++++++----\n t/t0500-progress-display.sh | 103 +++++++++++++++++++++++++-------\n 4 files changed, 178 insertions(+), 85 deletions(-)\n\n-- \n2.32.0.957.gd9e39d72fe6\n\n"},{"id":"430920","messageId":"patch-1.8-0a137d9af3-20210722T125012Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-0.8-0000000000-20210722T125012Z-avarab@gmail.com","subject":"[PATCH 1/8] progress.c tests: make start/stop verbs on stdin","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-22T12:54:59Z","receivedAt":"2021-07-22T12:55:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the usage of the \"test-tool progress\" introduced in\n2bb74b53a49 (Test the progress display, 2019-09-16) to take command\nlike \"start\" and \"stop\" on stdin, instead of running them implicitly.\n\nThis makes for tests that are easier to read, since the recipe will\nmirror the API usage, and allows for easily testing invalid usage that\nwould yield (or should yield) a BUG(), e.g. providing two \"start\"\ncalls in a row. A subsequent commit will add such stress tests.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/helper/test-progress.c    | 43 +++++++++++++++++++--------\n t/t0500-progress-display.sh | 59 +++++++++++++++++++++++--------------\n 2 files changed, 67 insertions(+), 35 deletions(-)\n\ndiff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\nindex 5d05cbe789..685c0a7c49 100644\n--- a/t/helper/test-progress.c\n+++ b/t/helper/test-progress.c\n@@ -3,6 +3,9 @@\n  *\n  * Reads instructions from standard input, one instruction per line:\n  *\n+ *   \"start[ <total>[ <title>]]\" - Call start_progress(title, total),\n+ *                                 when \"start\" use a title of\n+ *                                 \"Working hard\" with a total of 0.\n  *   \"progress <items>\" - Call display_progress() with the given item count\n  *                        as parameter.\n  *   \"throughput <bytes> <millis> - Call display_throughput() with the given\n@@ -10,6 +13,7 @@\n  *                                  specify the time elapsed since the\n  *                                  start_progress() call.\n  *   \"update\" - Set the 'progress_update' flag.\n+ *   \"stop\" - Call stop_progress().\n  *\n  * See 't0500-progress-display.sh' for examples.\n  */\n@@ -22,31 +26,41 @@\n \n int cmd__progress(int argc, const char **argv)\n {\n-\tint total = 0;\n-\tconst char *title;\n+\tconst char *default_title = \"Working hard\";\n+\tchar *detached_title = NULL;\n \tstruct strbuf line = STRBUF_INIT;\n-\tstruct progress *progress;\n+\tstruct progress *progress = NULL;\n \n \tconst char *usage[] = {\n-\t\t\"test-tool progress [--total=<n>] <progress-title>\",\n+\t\t\"test-tool progress <stdin\",\n \t\tNULL\n \t};\n \tstruct option options[] = {\n-\t\tOPT_INTEGER(0, \"total\", &total, \"total number of items\"),\n \t\tOPT_END(),\n \t};\n \n \targc = parse_options(argc, argv, NULL, options, usage, 0);\n-\tif (argc != 1)\n-\t\tdie(\"need a title for the progress output\");\n-\ttitle = argv[0];\n+\tif (argc)\n+\t\tusage_with_options(usage, options);\n \n \tprogress_testing = 1;\n-\tprogress = start_progress(title, total);\n \twhile (strbuf_getline(&line, stdin) != EOF) {\n \t\tchar *end;\n \n-\t\tif (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n+\t\tif (!strcmp(line.buf, \"start\")) {\n+\t\t\tprogress = start_progress(default_title, 0);\n+\t\t} else if (skip_prefix(line.buf, \"start \", (const char **) &end)) {\n+\t\t\tuint64_t total = strtoull(end, &end, 10);\n+\t\t\tif (*end == '\\0') {\n+\t\t\t\tprogress = start_progress(default_title, total);\n+\t\t\t} else if (*end == ' ') {\n+\t\t\t\tfree(detached_title);\n+\t\t\t\tdetached_title = strbuf_detach(&line, NULL);\n+\t\t\t\tprogress = start_progress(end + 1, total);\n+\t\t\t} else {\n+\t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n+\t\t\t}\n+\t\t} else if (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n \t\t\tuint64_t item_count = strtoull(end, &end, 10);\n \t\t\tif (*end != '\\0')\n \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n@@ -63,12 +77,15 @@ int cmd__progress(int argc, const char **argv)\n \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n \t\t\tprogress_test_ns = test_ms * 1000 * 1000;\n \t\t\tdisplay_throughput(progress, byte_count);\n-\t\t} else if (!strcmp(line.buf, \"update\"))\n+\t\t} else if (!strcmp(line.buf, \"update\")) {\n \t\t\tprogress_test_force_update();\n-\t\telse\n+\t\t} else if (!strcmp(line.buf, \"stop\")) {\n+\t\t\tstop_progress(&progress);\n+\t\t} else {\n \t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n+\t\t}\n \t}\n-\tstop_progress(&progress);\n+\tfree(detached_title);\n \n \treturn 0;\n }\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 22058b503a..ca96ac1fa5 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -17,6 +17,7 @@ test_expect_success 'simple progress display' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tupdate\n \tprogress 1\n \tupdate\n@@ -25,8 +26,9 @@ test_expect_success 'simple progress display' '\n \tprogress 4\n \tupdate\n \tprogress 5\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -41,11 +43,13 @@ test_expect_success 'progress display with total' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 3\n \tprogress 1\n \tprogress 2\n \tprogress 3\n+\tstop\n \tEOF\n-\ttest-tool progress --total=3 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -62,14 +66,14 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tprogress 100\n \tprogress 1000\n \tprogress 10000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -88,16 +92,15 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n-\tupdate\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tprogress 1\n \tupdate\n \tprogress 2\n \tprogress 10000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -116,14 +119,14 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tprogress 25000\n \tprogress 50000\n \tprogress 75000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -140,14 +143,14 @@ Working hard.......2.........3.........4.........5.........6.........7.........:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6.........7.........\n \tprogress 25000\n \tprogress 50000\n \tprogress 75000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6.........7.........\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -164,12 +167,14 @@ test_expect_success 'progress shortens - crazy caller' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 1000\n \tprogress 100\n \tprogress 200\n \tprogress 1\n \tprogress 1000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=1000 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -185,6 +190,7 @@ test_expect_success 'progress display with throughput' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart\n \tthroughput 102400 1000\n \tupdate\n \tprogress 10\n@@ -197,8 +203,9 @@ test_expect_success 'progress display with throughput' '\n \tthroughput 409600 4000\n \tupdate\n \tprogress 40\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -214,6 +221,7 @@ test_expect_success 'progress display with throughput and total' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 40\n \tthroughput 102400 1000\n \tprogress 10\n \tthroughput 204800 2000\n@@ -222,8 +230,9 @@ test_expect_success 'progress display with throughput and total' '\n \tprogress 30\n \tthroughput 409600 4000\n \tprogress 40\n+\tstop\n \tEOF\n-\ttest-tool progress --total=40 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -239,6 +248,7 @@ test_expect_success 'cover up after throughput shortens' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart\n \tthroughput 409600 1000\n \tupdate\n \tprogress 1\n@@ -251,8 +261,9 @@ test_expect_success 'cover up after throughput shortens' '\n \tthroughput 1638400 4000\n \tupdate\n \tprogress 4\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -267,6 +278,7 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart\n \tthroughput 1 1000\n \tupdate\n \tprogress 1\n@@ -276,8 +288,9 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \tthroughput 3145728 3000\n \tupdate\n \tprogress 3\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -285,6 +298,7 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \n test_expect_success 'progress generates traces' '\n \tcat >in <<-\\EOF &&\n+\tstart 40\n \tthroughput 102400 1000\n \tupdate\n \tprogress 10\n@@ -297,10 +311,11 @@ test_expect_success 'progress generates traces' '\n \tthroughput 409600 4000\n \tupdate\n \tprogress 40\n+\tstop\n \tEOF\n \n-\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" test-tool progress --total=40 \\\n-\t\t\"Working hard\" <in 2>stderr &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n \n \t# t0212/parse_events.perl intentionally omits regions and data.\n \ttest_region progress \"Working hard\" trace.event &&\n-- \n2.32.0.957.gd9e39d72fe6\n\n"},{"id":"430921","messageId":"patch-2.8-2dc0d255c8-20210722T125012Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-0.8-0000000000-20210722T125012Z-avarab@gmail.com","subject":"[PATCH 2/8] progress.c tests: test some invalid usage","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-22T12:55:00Z","receivedAt":"2021-07-22T12:55:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Test what happens when we \"stop\" without a \"start\", omit the \"stop\"\nafter a \"start\", or try to start two concurrent progress bars. This\nextends the trace2 tests added in 98a13647408 (trace2: log progress\ntime and throughput, 2020-05-12).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/t0500-progress-display.sh | 33 +++++++++++++++++++++++++++++++++\n 1 file changed, 33 insertions(+)\n\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex ca96ac1fa5..ffa819ca1d 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -323,4 +323,37 @@ test_expect_success 'progress generates traces' '\n \tgrep \"\\\"key\\\":\\\"total_bytes\\\",\\\"value\\\":\\\"409600\\\"\" trace.event\n '\n \n+test_expect_success 'progress generates traces: stop / start' '\n+\tcat >in <<-\\EOF &&\n+\tstart\n+\tstop\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-startstop.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n+\ttest_region progress \"Working hard\" trace-startstop.event\n+'\n+\n+test_expect_success 'progress generates traces: start without stop' '\n+\tcat >in <<-\\EOF &&\n+\tstart\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-start.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n+\tgrep region_enter.*progress trace-start.event &&\n+\t! grep region_leave.*progress trace-start.event\n+'\n+\n+test_expect_success 'progress generates traces: stop without start' '\n+\tcat >in <<-\\EOF &&\n+\tstop\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-stop.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n+\t! grep region_enter.*progress trace-stop.event &&\n+\t! grep region_leave.*progress trace-stop.event\n+'\n+\n test_done\n-- \n2.32.0.957.gd9e39d72fe6\n\n"},{"id":"430922","messageId":"patch-3.8-b91e37fdb5-20210722T125012Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-0.8-0000000000-20210722T125012Z-avarab@gmail.com","subject":"[PATCH 3/8] progress.c: move signal handler functions lower","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-22T12:55:01Z","receivedAt":"2021-07-22T12:55:18Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the signal handler functions to just before the\nstart_progress_delay() where they'll be referenced, instead of having\nthem at the top of the file.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 92 ++++++++++++++++++++++++++++--------------------------\n 1 file changed, 48 insertions(+), 44 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 680c6a8bf9..893cb0fe56 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -53,50 +53,6 @@ static volatile sig_atomic_t progress_update;\n  */\n int progress_testing;\n uint64_t progress_test_ns = 0;\n-void progress_test_force_update(void)\n-{\n-\tprogress_update = 1;\n-}\n-\n-\n-static void progress_interval(int signum)\n-{\n-\tprogress_update = 1;\n-}\n-\n-static void set_progress_signal(void)\n-{\n-\tstruct sigaction sa;\n-\tstruct itimerval v;\n-\n-\tif (progress_testing)\n-\t\treturn;\n-\n-\tprogress_update = 0;\n-\n-\tmemset(&sa, 0, sizeof(sa));\n-\tsa.sa_handler = progress_interval;\n-\tsigemptyset(&sa.sa_mask);\n-\tsa.sa_flags = SA_RESTART;\n-\tsigaction(SIGALRM, &sa, NULL);\n-\n-\tv.it_interval.tv_sec = 1;\n-\tv.it_interval.tv_usec = 0;\n-\tv.it_value = v.it_interval;\n-\tsetitimer(ITIMER_REAL, &v, NULL);\n-}\n-\n-static void clear_progress_signal(void)\n-{\n-\tstruct itimerval v = {{0,},};\n-\n-\tif (progress_testing)\n-\t\treturn;\n-\n-\tsetitimer(ITIMER_REAL, &v, NULL);\n-\tsignal(SIGALRM, SIG_IGN);\n-\tprogress_update = 0;\n-}\n \n static int is_foreground_fd(int fd)\n {\n@@ -249,6 +205,54 @@ void display_progress(struct progress *progress, uint64_t n)\n \t\tdisplay(progress, n, NULL);\n }\n \n+static void progress_interval(int signum)\n+{\n+\tprogress_update = 1;\n+}\n+\n+/*\n+ * The progress_test_force_update() function is intended for testing\n+ * the progress output, i.e. exclusively for 'test-tool progress'.\n+ */\n+void progress_test_force_update(void)\n+{\n+\tprogress_update = 1;\n+}\n+\n+static void set_progress_signal(void)\n+{\n+\tstruct sigaction sa;\n+\tstruct itimerval v;\n+\n+\tif (progress_testing)\n+\t\treturn;\n+\n+\tprogress_update = 0;\n+\n+\tmemset(&sa, 0, sizeof(sa));\n+\tsa.sa_handler = progress_interval;\n+\tsigemptyset(&sa.sa_mask);\n+\tsa.sa_flags = SA_RESTART;\n+\tsigaction(SIGALRM, &sa, NULL);\n+\n+\tv.it_interval.tv_sec = 1;\n+\tv.it_interval.tv_usec = 0;\n+\tv.it_value = v.it_interval;\n+\tsetitimer(ITIMER_REAL, &v, NULL);\n+}\n+\n+static void clear_progress_signal(void)\n+{\n+\tstruct itimerval v = {{0,},};\n+\n+\tif (progress_testing)\n+\t\treturn;\n+\n+\tsetitimer(ITIMER_REAL, &v, NULL);\n+\tsignal(SIGALRM, SIG_IGN);\n+\tprogress_update = 0;\n+}\n+\n static struct progress *start_progress_delay(const char *title, uint64_t total,\n \t\t\t\t\t     unsigned delay, unsigned sparse)\n {\n-- \n2.32.0.957.gd9e39d72fe6\n\n"},{"id":"430923","messageId":"patch-5.8-dea3df8ba6-20210722T125012Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-0.8-0000000000-20210722T125012Z-avarab@gmail.com","subject":"[PATCH 5/8] progress.c: stop eagerly fflush(stderr) when not a terminal","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-22T12:55:03Z","receivedAt":"2021-07-22T12:55:19Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"It's the clear intention of the combination of 137a0d0ef56 (Flush\nprogress message buffer in display()., 2007-11-19) and\n85cb8906f0e (progress: no progress in background, 2015-04-13) to call\nfflush(stderr) when we have a stderr in the foreground, but we ended\nup always calling fflush(stderr) seemingly by omission. Let's not.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 7fcc513717..1fade5808d 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -91,7 +91,8 @@ static void display(struct progress *progress, uint64_t n, const char *done)\n \t}\n \n \tif (show_update) {\n-\t\tif (is_foreground_fd(fileno(stderr)) || done) {\n+\t\tint stderr_is_foreground_fd = is_foreground_fd(fileno(stderr));\n+\t\tif (stderr_is_foreground_fd || done) {\n \t\t\tconst char *eol = done ? done : \"\\r\";\n \t\t\tsize_t clear_len = counters_sb->len < last_count_len ?\n \t\t\t\t\tlast_count_len - counters_sb->len + 1 :\n@@ -115,7 +116,8 @@ static void display(struct progress *progress, uint64_t n, const char *done)\n \t\t\t\tfprintf(stderr, \"%s: %s%*s\", progress->title,\n \t\t\t\t\tcounters_sb->buf, (int) clear_len, eol);\n \t\t\t}\n-\t\t\tfflush(stderr);\n+\t\t\tif (stderr_is_foreground_fd)\n+\t\t\t\tfflush(stderr);\n \t\t}\n \t\tprogress_update = 0;\n \t}\n-- \n2.32.0.957.gd9e39d72fe6\n\n"},{"id":"430924","messageId":"patch-4.8-ac07681f24-20210722T125012Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-0.8-0000000000-20210722T125012Z-avarab@gmail.com","subject":"[PATCH 4/8] progress.c: call progress_interval() from progress_test_force_update()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-22T12:55:02Z","receivedAt":"2021-07-22T12:55:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Define the progress_test_force_update() function in terms of\nprogress_interval(). For documentation purposes these two functions\nhave the same body, but different names. Let's just define the test\nfunction by calling progress_interval() with SIGALRM ourselves.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/progress.c b/progress.c\nindex 893cb0fe56..7fcc513717 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -216,7 +216,7 @@ static void progress_interval(int signum)\n  */\n void progress_test_force_update(void)\n {\n-\tprogress_update = 1;\n+\tprogress_interval(SIGALRM);\n }\n \n static void set_progress_signal(void)\n-- \n2.32.0.957.gd9e39d72fe6\n\n"},{"id":"430925","messageId":"patch-6.8-b99bb60dca-20210722T125012Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-0.8-0000000000-20210722T125012Z-avarab@gmail.com","subject":"[PATCH 6/8] progress.c: add temporary variable from progress struct","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-22T12:55:04Z","receivedAt":"2021-07-22T12:55:21Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a temporary \"progress\" variable for the dereferenced p_progress\npointer to a \"struct progress *\". Before 98a13647408 (trace2: log\nprogress time and throughput, 2020-05-12) we didn't dereference\n\"p_progress\" in this function, now that we do it's easier to read the\ncode if we work with a \"progress\" struct pointer like everywhere else,\ninstead of a pointer to a pointer.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 1fade5808d..1ab7d19deb 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -331,15 +331,16 @@ void stop_progress(struct progress **p_progress)\n \tfinish_if_sparse(*p_progress);\n \n \tif (*p_progress) {\n+\t\tstruct progress *progress = *p_progress;\n \t\ttrace2_data_intmax(\"progress\", the_repository, \"total_objects\",\n \t\t\t\t   (*p_progress)->total);\n \n \t\tif ((*p_progress)->throughput)\n \t\t\ttrace2_data_intmax(\"progress\", the_repository,\n \t\t\t\t\t   \"total_bytes\",\n-\t\t\t\t\t   (*p_progress)->throughput->curr_total);\n+\t\t\t\t\t   progress->throughput->curr_total);\n \n-\t\ttrace2_region_leave(\"progress\", (*p_progress)->title, the_repository);\n+\t\ttrace2_region_leave(\"progress\", progress->title, the_repository);\n \t}\n \n \tstop_progress_msg(p_progress, _(\"done\"));\n-- \n2.32.0.957.gd9e39d72fe6\n\n"},{"id":"430926","messageId":"patch-7.8-eb63b4ba6a-20210722T125012Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-0.8-0000000000-20210722T125012Z-avarab@gmail.com","subject":"[PATCH 7/8] pack-bitmap-write.c: add a missing stop_progress()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-22T12:55:05Z","receivedAt":"2021-07-22T12:55:26Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a bug that's been here since 7cc8f971085 (pack-objects: implement\nbitmap writing, 2013-12-21), we did not call stop_progress() if we\nreached the early exit in this function. This will matter in a\nsubsequent commit where we BUG(...) out if this happens, and matters\nnow e.g. because we don't have a corresponding \"region_end\" for the\nprogress trace2 event.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n pack-bitmap-write.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 88d9e696a5..6e110e41ea 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -550,6 +550,7 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \tif (indexed_commits_nr < 100) {\n \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n \t\t\tpush_bitmapped_commit(indexed_commits[i]);\n+\t\tstop_progress(&writer.progress);\n \t\treturn;\n \t}\n \n-- \n2.32.0.957.gd9e39d72fe6\n\n"},{"id":"430927","messageId":"patch-8.8-1a14176bee-20210722T125012Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-0.8-0000000000-20210722T125012Z-avarab@gmail.com","subject":"[PATCH 8/8] progress.c: add & assert a \"global_progress\" variable","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-07-22T12:55:06Z","receivedAt":"2021-07-22T12:55:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The progress.c code makes a hard assumption that only one progress bar\nbe active at a time (see [1] for a bug where this wasn't the case),\nbut nothing has asserted that that's the case. Let's add a BUG()\nthat'll trigger if two progress bars are active at the same time.\n\nThere's an alternate test-only approach to doing the same thing[2],\nbut by doing this for all progress bars we'll have a canary to check\nif we have any unexpected interaction between the \"sig_atomic_t\nprogress_update\" variable and this global struct.\n\nI am then planning on using this scaffolding in the future to fix a\nlimitation in the progress output, namely the current limitation of\nthe progress.c bar code that any update must pro-actively go through\nthe likes of display_progress().\n\nIf we e.g. hang forever before the first display_progress(), or in the\nmiddle of a loop that would call display_progress() the user will only\nsee either no output, or output frozen at the last display_progress()\nthat would have done an update (e.g. in cases where progress_update\nwas \"1\" due to an earlier signal).\n\nThis change does not fix that, but sets up the structure for solving\nthat and other related problems by juggling this \"global_progress\"\nstruct. Later changes will make more use of the \"global_progress\" than\nonly using it for these assertions.\n\n1. 6f9d5f2fda1 (commit-graph: fix progress of reachable commits, 2020-07-09)\n2. https://lore.kernel.org/git/20210620200303.2328957-3-szeder.dev@gmail.com\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c                  | 17 +++++++++++++----\n t/t0500-progress-display.sh | 11 +++++++++++\n 2 files changed, 24 insertions(+), 4 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 1ab7d19deb..14a023f4b4 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -46,6 +46,7 @@ struct progress {\n };\n \n static volatile sig_atomic_t progress_update;\n+static struct progress *global_progress;\n \n /*\n  * These are only intended for testing the progress output, i.e. exclusively\n@@ -221,11 +222,15 @@ void progress_test_force_update(void)\n \tprogress_interval(SIGALRM);\n }\n \n-static void set_progress_signal(void)\n+static void set_progress_signal(struct progress *progress)\n {\n \tstruct sigaction sa;\n \tstruct itimerval v;\n \n+\tif (global_progress)\n+\t\tBUG(\"should have no global_progress in set_progress_signal()\");\n+\tglobal_progress = progress;\n+\n \tif (progress_testing)\n \t\treturn;\n \n@@ -243,10 +248,14 @@ static void set_progress_signal(void)\n \tsetitimer(ITIMER_REAL, &v, NULL);\n }\n \n-static void clear_progress_signal(void)\n+static void clear_progress_signal(struct progress *progress)\n {\n \tstruct itimerval v = {{0,},};\n \n+\tif (!global_progress)\n+\t\tBUG(\"should have a global_progress in clear_progress_signal()\");\n+\tglobal_progress = NULL;\n+\n \tif (progress_testing)\n \t\treturn;\n \n@@ -270,7 +279,7 @@ static struct progress *start_progress_delay(const char *title, uint64_t total,\n \tstrbuf_init(&progress->counters_sb, 0);\n \tprogress->title_len = utf8_strwidth(title);\n \tprogress->split = 0;\n-\tset_progress_signal();\n+\tset_progress_signal(progress);\n \ttrace2_region_enter(\"progress\", title, the_repository);\n \treturn progress;\n }\n@@ -374,7 +383,7 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)\n \t\tdisplay(progress, progress->last_value, buf);\n \t\tfree(buf);\n \t}\n-\tclear_progress_signal();\n+\tclear_progress_signal(progress);\n \tstrbuf_release(&progress->counters_sb);\n \tif (progress->throughput)\n \t\tstrbuf_release(&progress->throughput->display);\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex ffa819ca1d..124d33c96b 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -296,6 +296,17 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \ttest_cmp expect out\n '\n \n+test_expect_success 'BUG: start two concurrent progress bars' '\n+\tcat >in <<-\\EOF &&\n+\tstart 0 one\n+\tstart 0 two\n+\tEOF\n+\n+\ttest_must_fail test-tool progress \\\n+\t\t<in 2>stderr &&\n+\tgrep -E \"^BUG: .*: should have no global_progress in set_progress_signal\\(\\)$\" stderr\n+'\n+\n test_expect_success 'progress generates traces' '\n \tcat >in <<-\\EOF &&\n \tstart 40\n-- \n2.32.0.957.gd9e39d72fe6\n\n"},{"id":"431101","messageId":"xmqqk0lglmpe.fsf@gitster.g","threadId":"55972","inReplyTo":"patch-1.3-832a6c1f78-20210722T121801Z-avarab@gmail.com","subject":"Re: [PATCH 1/3] commit-graph: fix bogus counter in \"Scanning merged commits\" progress line","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-23T21:55:25Z","receivedAt":"2021-07-23T21:55:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:\n\n> From: SZEDER Gábor <szeder.dev@gmail.com>\n>\n> The final value of the counter of the \"Scanning merged commits\"\n> progress line is always one less than its expected total, e.g.:\n>\n>   Scanning merged commits:  83% (5/6), done.\n>\n> This happens because while iterating over an array the loop variable\n> is passed to display_progress() as-is, but while C arrays (and thus\n> the loop variable) start at 0 and end at N-1, the progress counter\n> must end at N.  This causes the failures of the tests\n> 'fetch.writeCommitGraph' and 'fetch.writeCommitGraph with submodules'\n> in 't5510-fetch.sh' when run with GIT_TEST_CHECK_PROGRESS=1.\n>\n> Fix this by passing 'i + 1' to display_progress(), like most other\n> callsites do.\n\nSensible, I guess.\n\n>\n> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>\n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>  commit-graph.c | 2 +-\n>  1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 1a2602da61..918061f207 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -2096,7 +2096,7 @@ static void sort_and_scan_merged_commits(struct write_commit_graph_context *ctx)\n>  \n>  \tctx->num_extra_edges = 0;\n>  \tfor (i = 0; i < ctx->commits.nr; i++) {\n> -\t\tdisplay_progress(ctx->progress, i);\n> +\t\tdisplay_progress(ctx->progress, i + 1);\n>  \n>  \t\tif (i && oideq(&ctx->commits.list[i - 1]->object.oid,\n>  \t\t\t  &ctx->commits.list[i]->object.oid)) {\n\n"},{"id":"431102","messageId":"xmqqfsw4lmo5.fsf@gitster.g","threadId":"55972","inReplyTo":"patch-2.3-3411fe0515-20210722T121801Z-avarab@gmail.com","subject":"Re: [PATCH 2/3] midx: don't provide a total for QSORT() progress","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-23T21:56:10Z","receivedAt":"2021-07-23T21:56:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:\n\n> The quicksort algorithm can be anywhere between O(n) and O(n^2), so\n> providing a \"num objects\" as a total means that in some cases we're\n> going to go past 100%.\n>\n> This fixes a logic error in 5ae18df9d8e (midx: during verify group\n> objects by packfile to speed verification, 2019-03-21), which in turn\n> seems to have been diligently copied from my own logic error in the\n> commit-graph.c code, see 890226ccb57 (commit-graph write: add\n> itermediate progress, 2019-01-19).\n\nInteresting.\n\n>\n> That commit-graph code of mine was removed in\n> 1cbdbf3bef7 (commit-graph: drop count_distinct_commits() function,\n> 2020-12-07), so we don't need to fix that too.\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>  midx.c | 2 +-\n>  1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/midx.c b/midx.c\n> index 9a35b0255d..eaae75ab19 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -1291,7 +1291,7 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n>  \n>  \tif (flags & MIDX_PROGRESS)\n>  \t\tprogress = start_sparse_progress(_(\"Sorting objects by packfile\"),\n> -\t\t\t\t\t\t m->num_objects);\n> +\t\t\t\t\t\t 0);\n>  \tdisplay_progress(progress, 0); /* TODO: Measure QSORT() progress */\n>  \tQSORT(pairs, m->num_objects, compare_pair_pos_vs_id);\n>  \tstop_progress(&progress);\n"},{"id":"431104","messageId":"xmqqbl6slmer.fsf@gitster.g","threadId":"55972","inReplyTo":"patch-3.3-f65001627a-20210722T121801Z-avarab@gmail.com","subject":"Re: [PATCH 3/3] entry: show finer-grained counter in \"Filtering content\" progress line","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-23T22:01:48Z","receivedAt":"2021-07-23T22:01:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:\n\n> From: SZEDER Gábor <szeder.dev@gmail.com>\n>\n> The \"Filtering content\" progress in entry.c:finish_delayed_checkout()\n> is unusual because of how it calculates the progress count and because\n> it shows the progress of a nested loop.  It works basically like this:\n>\n>   start_delayed_progress(p, nr_of_paths_to_filter)\n>   for_each_filter {\n>       display_progress(p, nr_of_paths_to_filter - nr_of_paths_still_left_to_filter)\n>       for_each_path_handled_by_the_current_filter {\n>           checkout_entry()\n>       }\n>   }\n>   stop_progress(p)\n>\n> There are two issues with this approach:\n>\n>   - The work done by the last filter (or the only filter if there is\n>     only one) is never counted, so if the last filter still has some\n>     paths to process, then the counter shown in the \"done\" progress\n>     line will not match the expected total.\n>\n>     This would cause a BUG() in an upcoming change that adds an\n>     assertion checking if the \"total\" at the end matches the last\n>     progress bar update..\n\nSo the other series will semantically depend on this 3-patch series?\nJust making sure that is the intended topic structure.\n\n> diff --git a/entry.c b/entry.c\n> index 125fabdbd5..d92dd020b3 100644\n> --- a/entry.c\n> +++ b/entry.c\n> @@ -162,7 +162,7 @@ static int remove_available_paths(struct string_list_item *item, void *cb_data)\n>  int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n>  {\n>  \tint errs = 0;\n> -\tunsigned delayed_object_count;\n> +\tunsigned processed_paths = 0;\n>  \toff_t filtered_bytes = 0;\n>  \tstruct string_list_item *filter, *path;\n>  \tstruct progress *progress;\n> @@ -172,12 +172,10 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n>  \t\treturn errs;\n>  \n>  \tdco->state = CE_RETRY;\n> -\tdelayed_object_count = dco->paths.nr;\n> -\tprogress = start_delayed_progress(_(\"Filtering content\"), delayed_object_count);\n> +\tprogress = start_delayed_progress(_(\"Filtering content\"), dco->paths.nr);\n>  \twhile (dco->filters.nr > 0) {\n>  \t\tfor_each_string_list_item(filter, &dco->filters) {\n>  \t\t\tstruct string_list available_paths = STRING_LIST_INIT_NODUP;\n> -\t\t\tdisplay_progress(progress, delayed_object_count - dco->paths.nr);\n>  \n>  \t\t\tif (!async_query_available_blobs(filter->string, &available_paths)) {\n>  \t\t\t\t/* Filter reported an error */\n> @@ -224,6 +222,7 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n>  \t\t\t\tce = index_file_exists(state->istate, path->string,\n>  \t\t\t\t\t\t       strlen(path->string), 0);\n>  \t\t\t\tif (ce) {\n> +\t\t\t\t\tdisplay_progress(progress, ++processed_paths);\n>  \t\t\t\t\terrs |= checkout_entry(ce, state, NULL, nr_checkouts);\n>  \t\t\t\t\tfiltered_bytes += ce->ce_stat_data.sd_size;\n>  \t\t\t\t\tdisplay_throughput(progress, filtered_bytes);\n\nHmph.  A missing cache entries will not increment processed; would\nthat cause stop_progress() to see at the end the counter that is\nsmaller than dco->paths.nr we saw at the beginning?\n\n"},{"id":"431105","messageId":"xmqq7dhglmcv.fsf@gitster.g","threadId":"55972","inReplyTo":"cover-0.8-0000000000-20210722T125012Z-avarab@gmail.com","subject":"Re: [PATCH 0/8] progress: assert \"global_progress\" + test fixes / cleanup","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-23T22:02:56Z","receivedAt":"2021-07-23T22:03:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:\n\n> These patches were originally submitted as part of a much larger topic\n> at [1]. The add a \"global_progress\" \"struct progress *\" which we\n> assign/clear to as we start/stop progress bars.\n>\n> This will become imporant for some new progress features I have\n> planend, but right now is just used to assert that we don't start two\n> progress bars at the same time. 7/8 fixes an existing bug where we did\n> that.\n>\n> To get there I fixed up the test helper to be able to test this, moved\n> some code around, and fixes a couple of existing nits in 5/8 and 6/8..\n>\n> See also [2] which is a re-submission of that larger topic, but the\n> two can proceed independently.\n\nOK.\n\n>\n> 1. https://lore.kernel.org/git/cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com/\n> 2. https://lore.kernel.org/git/cover-0.3-0000000000-20210722T121801Z-avarab@gmail.com/\n>\n> Ævar Arnfjörð Bjarmason (8):\n>   progress.c tests: make start/stop verbs on stdin\n>   progress.c tests: test some invalid usage\n>   progress.c: move signal handler functions lower\n>   progress.c: call progress_interval() from progress_test_force_update()\n>   progress.c: stop eagerly fflush(stderr) when not a terminal\n>   progress.c: add temporary variable from progress struct\n>   pack-bitmap-write.c: add a missing stop_progress()\n>   progress.c: add & assert a \"global_progress\" variable\n>\n>  pack-bitmap-write.c         |   1 +\n>  progress.c                  | 116 ++++++++++++++++++++----------------\n>  t/helper/test-progress.c    |  43 +++++++++----\n>  t/t0500-progress-display.sh | 103 +++++++++++++++++++++++++-------\n>  4 files changed, 178 insertions(+), 85 deletions(-)\n"},{"id":"431762","messageId":"20210802210759.GD23408@szeder.dev","threadId":"55972","inReplyTo":"patch-1.3-832a6c1f78-20210722T121801Z-avarab@gmail.com","subject":"Re: [PATCH 1/3] commit-graph: fix bogus counter in \"Scanning merged commits\" progress line","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-08-02T21:07:59Z","receivedAt":"2021-08-02T21:08:05Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Jul 22, 2021 at 02:20:15PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> From: SZEDER Gábor <szeder.dev@gmail.com>\n> \n> The final value of the counter of the \"Scanning merged commits\"\n> progress line is always one less than its expected total, e.g.:\n> \n>   Scanning merged commits:  83% (5/6), done.\n> \n> This happens because while iterating over an array the loop variable\n> is passed to display_progress() as-is, but while C arrays (and thus\n> the loop variable) start at 0 and end at N-1, the progress counter\n> must end at N.  This causes the failures of the tests\n> 'fetch.writeCommitGraph' and 'fetch.writeCommitGraph with submodules'\n> in 't5510-fetch.sh' when run with GIT_TEST_CHECK_PROGRESS=1.\n\nThere is no GIT_TEST_CHECK_PROGRESS in this patch series.\n\n> Fix this by passing 'i + 1' to display_progress(), like most other\n> callsites do.\n> \n> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>\n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>  commit-graph.c | 2 +-\n>  1 file changed, 1 insertion(+), 1 deletion(-)\n> \n> diff --git a/commit-graph.c b/commit-graph.c\n> index 1a2602da61..918061f207 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -2096,7 +2096,7 @@ static void sort_and_scan_merged_commits(struct write_commit_graph_context *ctx)\n>  \n>  \tctx->num_extra_edges = 0;\n>  \tfor (i = 0; i < ctx->commits.nr; i++) {\n> -\t\tdisplay_progress(ctx->progress, i);\n> +\t\tdisplay_progress(ctx->progress, i + 1);\n>  \n>  \t\tif (i && oideq(&ctx->commits.list[i - 1]->object.oid,\n>  \t\t\t  &ctx->commits.list[i]->object.oid)) {\n> -- \n> 2.32.0.957.gd9e39d72fe6\n> \n"},{"id":"431766","messageId":"20210802214827.GE23408@szeder.dev","threadId":"55972","inReplyTo":"patch-3.3-f65001627a-20210722T121801Z-avarab@gmail.com","subject":"Re: [PATCH 3/3] entry: show finer-grained counter in \"Filtering content\" progress line","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-08-02T21:48:27Z","receivedAt":"2021-08-02T21:48:33Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Jul 22, 2021 at 02:20:17PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> From: SZEDER Gábor <szeder.dev@gmail.com>\n> \n> The \"Filtering content\" progress in entry.c:finish_delayed_checkout()\n> is unusual because of how it calculates the progress count and because\n> it shows the progress of a nested loop.  It works basically like this:\n> \n>   start_delayed_progress(p, nr_of_paths_to_filter)\n>   for_each_filter {\n>       display_progress(p, nr_of_paths_to_filter - nr_of_paths_still_left_to_filter)\n>       for_each_path_handled_by_the_current_filter {\n>           checkout_entry()\n>       }\n>   }\n>   stop_progress(p)\n> \n> There are two issues with this approach:\n> \n>   - The work done by the last filter (or the only filter if there is\n>     only one) is never counted, so if the last filter still has some\n>     paths to process, then the counter shown in the \"done\" progress\n>     line will not match the expected total.\n> \n>     This would cause a BUG() in an upcoming change that adds an\n>     assertion checking if the \"total\" at the end matches the last\n>     progress bar update..\n> \n>     This is because both\n\n\"Both\" what?\n\n>     use only one filter.  (The test 'delayed\n>     checkout in process filter' uses two filters but the first one\n>     does all the work, so that test already happens to succeed even\n>     with such an assertion.)\n> \n>   - The progress counter is updated only once per filter, not once per\n>     processed path, so if a filter has a lot of paths to process, then\n>     the counter might stay unchanged for a long while and then make a\n>     big jump (though the user still gets a sense of progress, because\n>     we call display_throughput() after each processed path to show the\n>     amount of processed data).\n> \n> Move the display_progress() call to the inner loop, right next to that\n> checkout_entry() call that does the hard work for each path, and use a\n> dedicated counter variable that is incremented upon processing each\n> path.\n> \n> After this change the 'invalid file in delayed checkout' in\n> 't0021-conversion.sh' would succeed with the future BUG() assertion\n> discussed above but the 'missing file in delayed checkout' test would\n> still fail, because its purposefully buggy filter doesn't process any\n> paths, so we won't execute that inner loop at all (this will be fixed\n> in a subsequent commit).\n\nI don't like how the updates to the commit message keeps referring to\nsome future BUG().\n\nA benefit of my original submission is that all those checks are added\nat the beginning of the patch series, so when looking at these later\nbugfix patches reviewers can easily run the problematic tests with\nGIT_TEST_CHECK_PROGRESS=1 to see the failure themselves and to confirm\nthat the fix indeed works.  Without those checks the next best thing\nis applying the patch below and then looking at the verbose log of\nthose tests:\n\n\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex b5749f327d..93a67f2f1f 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -955,7 +955,8 @@ test_expect_success PERL 'missing file in delayed checkout' '\n \t) &&\n \n \trm -rf repo-cloned &&\n-\ttest_must_fail git clone repo repo-cloned 2>git-stderr.log &&\n+\ttest_must_fail env GIT_PROGRESS_DELAY=0 git clone repo repo-cloned 2>git-stderr.log &&\n+\tcat git-stderr.log &&\n \tgrep \"error: .missing-delay\\.a. was not filtered properly\" git-stderr.log\n '\n \n@@ -976,7 +977,8 @@ test_expect_success PERL 'invalid file in delayed checkout' '\n \t) &&\n \n \trm -rf repo-cloned &&\n-\ttest_must_fail git clone repo repo-cloned 2>git-stderr.log &&\n+\ttest_must_fail env GIT_PROGRESS_DELAY=0 git clone repo repo-cloned 2>git-stderr.log &&\n+\tcat git-stderr.log &&\n \tgrep \"error: external filter .* signaled that .unfiltered. is now available although it has not been delayed earlier\" git-stderr.log\n '\n \n\n> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>\n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>  entry.c | 7 +++----\n>  1 file changed, 3 insertions(+), 4 deletions(-)\n> \n> diff --git a/entry.c b/entry.c\n> index 125fabdbd5..d92dd020b3 100644\n> --- a/entry.c\n> +++ b/entry.c\n> @@ -162,7 +162,7 @@ static int remove_available_paths(struct string_list_item *item, void *cb_data)\n>  int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n>  {\n>  \tint errs = 0;\n> -\tunsigned delayed_object_count;\n> +\tunsigned processed_paths = 0;\n>  \toff_t filtered_bytes = 0;\n>  \tstruct string_list_item *filter, *path;\n>  \tstruct progress *progress;\n> @@ -172,12 +172,10 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n>  \t\treturn errs;\n>  \n>  \tdco->state = CE_RETRY;\n> -\tdelayed_object_count = dco->paths.nr;\n> -\tprogress = start_delayed_progress(_(\"Filtering content\"), delayed_object_count);\n> +\tprogress = start_delayed_progress(_(\"Filtering content\"), dco->paths.nr);\n>  \twhile (dco->filters.nr > 0) {\n>  \t\tfor_each_string_list_item(filter, &dco->filters) {\n>  \t\t\tstruct string_list available_paths = STRING_LIST_INIT_NODUP;\n> -\t\t\tdisplay_progress(progress, delayed_object_count - dco->paths.nr);\n>  \n>  \t\t\tif (!async_query_available_blobs(filter->string, &available_paths)) {\n>  \t\t\t\t/* Filter reported an error */\n> @@ -224,6 +222,7 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n>  \t\t\t\tce = index_file_exists(state->istate, path->string,\n>  \t\t\t\t\t\t       strlen(path->string), 0);\n>  \t\t\t\tif (ce) {\n> +\t\t\t\t\tdisplay_progress(progress, ++processed_paths);\n>  \t\t\t\t\terrs |= checkout_entry(ce, state, NULL, nr_checkouts);\n>  \t\t\t\t\tfiltered_bytes += ce->ce_stat_data.sd_size;\n>  \t\t\t\t\tdisplay_throughput(progress, filtered_bytes);\n> -- \n> 2.32.0.957.gd9e39d72fe6\n> \n"},{"id":"431770","messageId":"20210802220506.GF23408@szeder.dev","threadId":"55972","inReplyTo":"xmqqbl6slmer.fsf@gitster.g","subject":"Re: [PATCH 3/3] entry: show finer-grained counter in \"Filtering content\" progress line","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-08-02T22:05:06Z","receivedAt":"2021-08-02T22:05:11Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Fri, Jul 23, 2021 at 03:01:48PM -0700, Junio C Hamano wrote:\n> Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:\n> \n> > From: SZEDER Gábor <szeder.dev@gmail.com>\n> >\n> > The \"Filtering content\" progress in entry.c:finish_delayed_checkout()\n> > is unusual because of how it calculates the progress count and because\n> > it shows the progress of a nested loop.  It works basically like this:\n> >\n> >   start_delayed_progress(p, nr_of_paths_to_filter)\n> >   for_each_filter {\n> >       display_progress(p, nr_of_paths_to_filter - nr_of_paths_still_left_to_filter)\n> >       for_each_path_handled_by_the_current_filter {\n> >           checkout_entry()\n> >       }\n> >   }\n> >   stop_progress(p)\n> >\n> > There are two issues with this approach:\n> >\n> >   - The work done by the last filter (or the only filter if there is\n> >     only one) is never counted, so if the last filter still has some\n> >     paths to process, then the counter shown in the \"done\" progress\n> >     line will not match the expected total.\n> >\n> >     This would cause a BUG() in an upcoming change that adds an\n> >     assertion checking if the \"total\" at the end matches the last\n> >     progress bar update..\n> \n> So the other series will semantically depend on this 3-patch series?\n> Just making sure that is the intended topic structure.\n> \n> > diff --git a/entry.c b/entry.c\n> > index 125fabdbd5..d92dd020b3 100644\n> > --- a/entry.c\n> > +++ b/entry.c\n> > @@ -162,7 +162,7 @@ static int remove_available_paths(struct string_list_item *item, void *cb_data)\n> >  int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n> >  {\n> >  \tint errs = 0;\n> > -\tunsigned delayed_object_count;\n> > +\tunsigned processed_paths = 0;\n> >  \toff_t filtered_bytes = 0;\n> >  \tstruct string_list_item *filter, *path;\n> >  \tstruct progress *progress;\n> > @@ -172,12 +172,10 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n> >  \t\treturn errs;\n> >  \n> >  \tdco->state = CE_RETRY;\n> > -\tdelayed_object_count = dco->paths.nr;\n> > -\tprogress = start_delayed_progress(_(\"Filtering content\"), delayed_object_count);\n> > +\tprogress = start_delayed_progress(_(\"Filtering content\"), dco->paths.nr);\n> >  \twhile (dco->filters.nr > 0) {\n> >  \t\tfor_each_string_list_item(filter, &dco->filters) {\n> >  \t\t\tstruct string_list available_paths = STRING_LIST_INIT_NODUP;\n> > -\t\t\tdisplay_progress(progress, delayed_object_count - dco->paths.nr);\n> >  \n> >  \t\t\tif (!async_query_available_blobs(filter->string, &available_paths)) {\n> >  \t\t\t\t/* Filter reported an error */\n> > @@ -224,6 +222,7 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n> >  \t\t\t\tce = index_file_exists(state->istate, path->string,\n> >  \t\t\t\t\t\t       strlen(path->string), 0);\n> >  \t\t\t\tif (ce) {\n> > +\t\t\t\t\tdisplay_progress(progress, ++processed_paths);\n> >  \t\t\t\t\terrs |= checkout_entry(ce, state, NULL, nr_checkouts);\n> >  \t\t\t\t\tfiltered_bytes += ce->ce_stat_data.sd_size;\n> >  \t\t\t\t\tdisplay_throughput(progress, filtered_bytes);\n> \n> Hmph.  A missing cache entries will not increment processed; would\n> that cause stop_progress() to see at the end the counter that is\n> smaller than dco->paths.nr we saw at the beginning?\n\nYes, but this 'if (ce)' condition has an 'else errs = 1;' branch as\nwell, i.e. a missing cache entry is considered an error.  This patch\nfixes issues with this progress bar in the case when all went well,\ni.e. when there were no errors.  In my original submission it is\nfollowed up by another patch that attempts to fix this progress line\nwhen there are errors, arguing that it's wrong to show \"done\" at the\nend when not all work was done because of said errors...  although\n\"fix\" is not quite the right word for the approach taken in that\npatch, it's more like \"papering over\" ;)\n\n  https://public-inbox.org/git/20210620200303.2328957-7-szeder.dev@gmail.com/\n\n"},{"id":"432088","messageId":"cover-v2-0.3-0000000000-20210805T105720Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-0.3-0000000000-20210722T121801Z-avarab@gmail.com","subject":"[PATCH v2 0/3] progress.c API users: fix bogus counting","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-05T11:01:10Z","receivedAt":"2021-08-05T11:01:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"As a split-off from the larger topic these were submitted as part of\n[1] and which didn't get picked up. As I pointed out in [2] that\nlarger topic had some hidden untested-for flaws.\n\nBut these patches are just fixes to bogus progress bar output from\nthat topic. Let's consider them in isolation...\n\nSince v1 the only changes are to the commit messages, in response to\nSZEDER's feedback at\nhttps://lore.kernel.org/git/20210802210759.GD23408@szeder.dev/ and\nhttps://lore.kernel.org/git/20210802214827.GE23408@szeder.dev/;\nHopefully this update addresses all of those outstanding comments.\n\n1. https://lore.kernel.org/git/20210620200303.2328957-1-szeder.dev@gmail.com/\n2. https://lore.kernel.org/git/cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com/\n\nSZEDER Gábor (2):\n  commit-graph: fix bogus counter in \"Scanning merged commits\" progress\n    line\n  entry: show finer-grained counter in \"Filtering content\" progress line\n\nÆvar Arnfjörð Bjarmason (1):\n  midx: don't provide a total for QSORT() progress\n\n commit-graph.c | 2 +-\n entry.c        | 7 +++----\n midx.c         | 2 +-\n 3 files changed, 5 insertions(+), 6 deletions(-)\n\nRange-diff against v1:\n1:  832a6c1f78 ! 1:  bcb13be500 commit-graph: fix bogus counter in \"Scanning merged commits\" progress line\n    @@ Commit message\n         This happens because while iterating over an array the loop variable\n         is passed to display_progress() as-is, but while C arrays (and thus\n         the loop variable) start at 0 and end at N-1, the progress counter\n    -    must end at N.  This causes the failures of the tests\n    -    'fetch.writeCommitGraph' and 'fetch.writeCommitGraph with submodules'\n    -    in 't5510-fetch.sh' when run with GIT_TEST_CHECK_PROGRESS=1.\n    +    must end at N. Fix this by passing 'i + 1' to display_progress(), like\n    +    most other callsites do.\n     \n    -    Fix this by passing 'i + 1' to display_progress(), like most other\n    -    callsites do.\n    +    There's an RFC series to add a GIT_TEST_CHECK_PROGRESS=1 mode[1] which\n    +    catches this issue in the 'fetch.writeCommitGraph' and\n    +    'fetch.writeCommitGraph with submodules' tests in\n    +    't5510-fetch.sh'. The GIT_TEST_CHECK_PROGRESS=1 mode is not part of\n    +    this series, but future changes to progress.c may add it or similar\n    +    assertions to catch this and similar bugs elsewhere.\n    +\n    +    1. https://lore.kernel.org/git/20210620200303.2328957-1-szeder.dev@gmail.com/\n     \n         Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>\n         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n2:  3411fe0515 = 2:  8e67712c48 midx: don't provide a total for QSORT() progress\n3:  f65001627a ! 3:  c70e554e46 entry: show finer-grained counter in \"Filtering content\" progress line\n    @@ Commit message\n             paths to process, then the counter shown in the \"done\" progress\n             line will not match the expected total.\n     \n    -        This would cause a BUG() in an upcoming change that adds an\n    -        assertion checking if the \"total\" at the end matches the last\n    -        progress bar update..\n    -\n    -        This is because both use only one filter.  (The test 'delayed\n    -        checkout in process filter' uses two filters but the first one\n    -        does all the work, so that test already happens to succeed even\n    -        with such an assertion.)\n    +        The partially-RFC series to add a GIT_TEST_CHECK_PROGRESS=1\n    +        mode[1] helps spot this issue. Under it the 'missing file in\n    +        delayed checkout' and 'invalid file in delayed checkout' tests in\n    +        't0021-conversion.sh' fail, because both use only one\n    +        filter.  (The test 'delayed checkout in process filter' uses two\n    +        filters but the first one does all the work, so that test already\n    +        happens to succeed even with GIT_TEST_CHECK_PROGRESS=1.)\n     \n           - The progress counter is updated only once per filter, not once per\n             processed path, so if a filter has a lot of paths to process, then\n    @@ Commit message\n         path.\n     \n         After this change the 'invalid file in delayed checkout' in\n    -    't0021-conversion.sh' would succeed with the future BUG() assertion\n    -    discussed above but the 'missing file in delayed checkout' test would\n    -    still fail, because its purposefully buggy filter doesn't process any\n    -    paths, so we won't execute that inner loop at all (this will be fixed\n    -    in a subsequent commit).\n    +    't0021-conversion.sh' would succeed with the GIT_TEST_CHECK_PROGRESS=1\n    +    assertion discussed above, but the 'missing file in delayed checkout'\n    +    test would still fail.\n    +\n    +    It'll fail because its purposefully buggy filter doesn't process any\n    +    paths, so we won't execute that inner loop at all, see [2] for how to\n    +    spot that issue without GIT_TEST_CHECK_PROGRESS=1. It's not\n    +    straightforward to fix it with the current progress.c library (see [3]\n    +    for an attempt), so let's leave it for now.\n    +\n    +    1. https://lore.kernel.org/git/20210620200303.2328957-1-szeder.dev@gmail.com/\n    +    2. http://lore.kernel.org/git/20210802214827.GE23408@szeder.dev\n    +    3. https://lore.kernel.org/git/20210620200303.2328957-7-szeder.dev@gmail.com/\n     \n         Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>\n         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n-- \n2.33.0.rc0.635.g0ab9d6d3b5a\n\n"},{"id":"432089","messageId":"patch-v2-1.3-bcb13be500-20210805T105720Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v2-0.3-0000000000-20210805T105720Z-avarab@gmail.com","subject":"[PATCH v2 1/3] commit-graph: fix bogus counter in \"Scanning merged commits\" progress line","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-05T11:01:11Z","receivedAt":"2021-08-05T11:01:22Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: SZEDER Gábor <szeder.dev@gmail.com>\n\nThe final value of the counter of the \"Scanning merged commits\"\nprogress line is always one less than its expected total, e.g.:\n\n  Scanning merged commits:  83% (5/6), done.\n\nThis happens because while iterating over an array the loop variable\nis passed to display_progress() as-is, but while C arrays (and thus\nthe loop variable) start at 0 and end at N-1, the progress counter\nmust end at N. Fix this by passing 'i + 1' to display_progress(), like\nmost other callsites do.\n\nThere's an RFC series to add a GIT_TEST_CHECK_PROGRESS=1 mode[1] which\ncatches this issue in the 'fetch.writeCommitGraph' and\n'fetch.writeCommitGraph with submodules' tests in\n't5510-fetch.sh'. The GIT_TEST_CHECK_PROGRESS=1 mode is not part of\nthis series, but future changes to progress.c may add it or similar\nassertions to catch this and similar bugs elsewhere.\n\n1. https://lore.kernel.org/git/20210620200303.2328957-1-szeder.dev@gmail.com/\n\nSigned-off-by: SZEDER Gábor <szeder.dev@gmail.com>\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n commit-graph.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 3860a0d847..9d18c1d87d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -2096,7 +2096,7 @@ static void sort_and_scan_merged_commits(struct write_commit_graph_context *ctx)\n \n \tctx->num_extra_edges = 0;\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n-\t\tdisplay_progress(ctx->progress, i);\n+\t\tdisplay_progress(ctx->progress, i + 1);\n \n \t\tif (i && oideq(&ctx->commits.list[i - 1]->object.oid,\n \t\t\t  &ctx->commits.list[i]->object.oid)) {\n-- \n2.33.0.rc0.635.g0ab9d6d3b5a\n\n"},{"id":"432090","messageId":"patch-v2-2.3-8e67712c48-20210805T105720Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v2-0.3-0000000000-20210805T105720Z-avarab@gmail.com","subject":"[PATCH v2 2/3] midx: don't provide a total for QSORT() progress","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-05T11:01:12Z","receivedAt":"2021-08-05T11:01:23Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The quicksort algorithm can be anywhere between O(n) and O(n^2), so\nproviding a \"num objects\" as a total means that in some cases we're\ngoing to go past 100%.\n\nThis fixes a logic error in 5ae18df9d8e (midx: during verify group\nobjects by packfile to speed verification, 2019-03-21), which in turn\nseems to have been diligently copied from my own logic error in the\ncommit-graph.c code, see 890226ccb57 (commit-graph write: add\nitermediate progress, 2019-01-19).\n\nThat commit-graph code of mine was removed in\n1cbdbf3bef7 (commit-graph: drop count_distinct_commits() function,\n2020-12-07), so we don't need to fix that too.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n midx.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/midx.c b/midx.c\nindex 321c6fdd2f..cad78d71fc 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -1292,7 +1292,7 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \n \tif (flags & MIDX_PROGRESS)\n \t\tprogress = start_sparse_progress(_(\"Sorting objects by packfile\"),\n-\t\t\t\t\t\t m->num_objects);\n+\t\t\t\t\t\t 0);\n \tdisplay_progress(progress, 0); /* TODO: Measure QSORT() progress */\n \tQSORT(pairs, m->num_objects, compare_pair_pos_vs_id);\n \tstop_progress(&progress);\n-- \n2.33.0.rc0.635.g0ab9d6d3b5a\n\n"},{"id":"432091","messageId":"patch-v2-3.3-c70e554e46-20210805T105720Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v2-0.3-0000000000-20210805T105720Z-avarab@gmail.com","subject":"[PATCH v2 3/3] entry: show finer-grained counter in \"Filtering content\" progress line","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-05T11:01:13Z","receivedAt":"2021-08-05T11:01:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: SZEDER Gábor <szeder.dev@gmail.com>\n\nThe \"Filtering content\" progress in entry.c:finish_delayed_checkout()\nis unusual because of how it calculates the progress count and because\nit shows the progress of a nested loop.  It works basically like this:\n\n  start_delayed_progress(p, nr_of_paths_to_filter)\n  for_each_filter {\n      display_progress(p, nr_of_paths_to_filter - nr_of_paths_still_left_to_filter)\n      for_each_path_handled_by_the_current_filter {\n          checkout_entry()\n      }\n  }\n  stop_progress(p)\n\nThere are two issues with this approach:\n\n  - The work done by the last filter (or the only filter if there is\n    only one) is never counted, so if the last filter still has some\n    paths to process, then the counter shown in the \"done\" progress\n    line will not match the expected total.\n\n    The partially-RFC series to add a GIT_TEST_CHECK_PROGRESS=1\n    mode[1] helps spot this issue. Under it the 'missing file in\n    delayed checkout' and 'invalid file in delayed checkout' tests in\n    't0021-conversion.sh' fail, because both use only one\n    filter.  (The test 'delayed checkout in process filter' uses two\n    filters but the first one does all the work, so that test already\n    happens to succeed even with GIT_TEST_CHECK_PROGRESS=1.)\n\n  - The progress counter is updated only once per filter, not once per\n    processed path, so if a filter has a lot of paths to process, then\n    the counter might stay unchanged for a long while and then make a\n    big jump (though the user still gets a sense of progress, because\n    we call display_throughput() after each processed path to show the\n    amount of processed data).\n\nMove the display_progress() call to the inner loop, right next to that\ncheckout_entry() call that does the hard work for each path, and use a\ndedicated counter variable that is incremented upon processing each\npath.\n\nAfter this change the 'invalid file in delayed checkout' in\n't0021-conversion.sh' would succeed with the GIT_TEST_CHECK_PROGRESS=1\nassertion discussed above, but the 'missing file in delayed checkout'\ntest would still fail.\n\nIt'll fail because its purposefully buggy filter doesn't process any\npaths, so we won't execute that inner loop at all, see [2] for how to\nspot that issue without GIT_TEST_CHECK_PROGRESS=1. It's not\nstraightforward to fix it with the current progress.c library (see [3]\nfor an attempt), so let's leave it for now.\n\n1. https://lore.kernel.org/git/20210620200303.2328957-1-szeder.dev@gmail.com/\n2. http://lore.kernel.org/git/20210802214827.GE23408@szeder.dev\n3. https://lore.kernel.org/git/20210620200303.2328957-7-szeder.dev@gmail.com/\n\nSigned-off-by: SZEDER Gábor <szeder.dev@gmail.com>\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n entry.c | 7 +++----\n 1 file changed, 3 insertions(+), 4 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex 125fabdbd5..d92dd020b3 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -162,7 +162,7 @@ static int remove_available_paths(struct string_list_item *item, void *cb_data)\n int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n {\n \tint errs = 0;\n-\tunsigned delayed_object_count;\n+\tunsigned processed_paths = 0;\n \toff_t filtered_bytes = 0;\n \tstruct string_list_item *filter, *path;\n \tstruct progress *progress;\n@@ -172,12 +172,10 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n \t\treturn errs;\n \n \tdco->state = CE_RETRY;\n-\tdelayed_object_count = dco->paths.nr;\n-\tprogress = start_delayed_progress(_(\"Filtering content\"), delayed_object_count);\n+\tprogress = start_delayed_progress(_(\"Filtering content\"), dco->paths.nr);\n \twhile (dco->filters.nr > 0) {\n \t\tfor_each_string_list_item(filter, &dco->filters) {\n \t\t\tstruct string_list available_paths = STRING_LIST_INIT_NODUP;\n-\t\t\tdisplay_progress(progress, delayed_object_count - dco->paths.nr);\n \n \t\t\tif (!async_query_available_blobs(filter->string, &available_paths)) {\n \t\t\t\t/* Filter reported an error */\n@@ -224,6 +222,7 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n \t\t\t\tce = index_file_exists(state->istate, path->string,\n \t\t\t\t\t\t       strlen(path->string), 0);\n \t\t\t\tif (ce) {\n+\t\t\t\t\tdisplay_progress(progress, ++processed_paths);\n \t\t\t\t\terrs |= checkout_entry(ce, state, NULL, nr_checkouts);\n \t\t\t\t\tfiltered_bytes += ce->ce_stat_data.sd_size;\n \t\t\t\t\tdisplay_throughput(progress, filtered_bytes);\n-- \n2.33.0.rc0.635.g0ab9d6d3b5a\n\n"},{"id":"432103","messageId":"5cf55741-d06a-f8f3-5027-02c5e1dfa672@gmail.com","threadId":"55972","inReplyTo":"patch-2.3-3411fe0515-20210722T121801Z-avarab@gmail.com","subject":"Re: [PATCH 2/3] midx: don't provide a total for QSORT() progress","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2021-08-05T15:07:14Z","receivedAt":"2021-08-05T15:07:22Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 22/07/2021 13:20, Ævar Arnfjörð Bjarmason wrote:\n> The quicksort algorithm can be anywhere between O(n) and O(n^2), so\n> providing a \"num objects\" as a total means that in some cases we're\n> going to go past 100%.\n\nBeing pedantic QSORT() is not necessarily a quicksort, for example \ncompact/qsort_s.c implements a merge sort.\n\nI'm confused as to how we go past 100% when we only call \ndisplay_progress(progress, 0) and then stop_progress(). If my reading of \nprogress.c is correct then this change means we'll stop displaying a \npercentage as progress->total will be zero in\n\nstatic void display(struct progress *progress, uint64_t n, const char *done)\n{\n\t/* ... */\n\tif (progress->total) {\n\t\tunsigned percent = n * 100 / progress->total;\n\t\tif (percent != progress->last_percent || progress_update) {\n\t\t\tprogress->last_percent = percent;\n\n\t\t\tstrbuf_reset(counters_sb);\n\t\t\tstrbuf_addf(counters_sb,\n\t\t\t\t    \"%3u%% (%\"PRIuMAX\"/%\"PRIuMAX\")%s\", percent,\n\t\t\t\t    (uintmax_t)n, (uintmax_t)progress->total,\n\t\t\t\t    tp);\n\t\t\tshow_update = 1;\n\t\t}\n\t} else if (progress_update) {\n\t\tstrbuf_reset(counters_sb);\n\t\tstrbuf_addf(counters_sb, \"%\"PRIuMAX\"%s\", (uintmax_t)n, tp);\n\t\tshow_update = 1;\n\t}\n\t/* ... */\n}\n\nBest Wishes\n\nPhillip\n\n> This fixes a logic error in 5ae18df9d8e (midx: during verify group\n> objects by packfile to speed verification, 2019-03-21), which in turn\n> seems to have been diligently copied from my own logic error in the\n> commit-graph.c code, see 890226ccb57 (commit-graph write: add\n> itermediate progress, 2019-01-19).\n> \n> That commit-graph code of mine was removed in\n> 1cbdbf3bef7 (commit-graph: drop count_distinct_commits() function,\n> 2020-12-07), so we don't need to fix that too.\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>   midx.c | 2 +-\n>   1 file changed, 1 insertion(+), 1 deletion(-)\n> \n> diff --git a/midx.c b/midx.c\n> index 9a35b0255d..eaae75ab19 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -1291,7 +1291,7 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n>   \n>   \tif (flags & MIDX_PROGRESS)\n>   \t\tprogress = start_sparse_progress(_(\"Sorting objects by packfile\"),\n> -\t\t\t\t\t\t m->num_objects);\n> +\t\t\t\t\t\t 0);\n>   \tdisplay_progress(progress, 0); /* TODO: Measure QSORT() progress */\n>   \tQSORT(pairs, m->num_objects, compare_pair_pos_vs_id);\n>   \tstop_progress(&progress);\n> \n"},{"id":"432126","messageId":"87v94jzoxj.fsf@evledraar.gmail.com","threadId":"55972","inReplyTo":"5cf55741-d06a-f8f3-5027-02c5e1dfa672@gmail.com","subject":"Re: [PATCH 2/3] midx: don't provide a total for QSORT() progress","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-05T19:07:36Z","receivedAt":"2021-08-05T19:12:18Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Aug 05 2021, Phillip Wood wrote:\n\n> On 22/07/2021 13:20, Ævar Arnfjörð Bjarmason wrote:\n>> The quicksort algorithm can be anywhere between O(n) and O(n^2), so\n>> providing a \"num objects\" as a total means that in some cases we're\n>> going to go past 100%.\n>\n> Being pedantic QSORT() is not necessarily a quicksort, for example\n> compact/qsort_s.c implements a merge sort.\n>\n> I'm confused as to how we go past 100% when we only call\n> display_progress(progress, 0) and then stop_progress(). If my reading\n> of progress.c is correct then this change means we'll stop displaying\n> a percentage as progress->total will be zero in\n>\n> static void display(struct progress *progress, uint64_t n, const char *done)\n> {\n> \t/* ... */\n> \tif (progress->total) {\n> \t\tunsigned percent = n * 100 / progress->total;\n> \t\tif (percent != progress->last_percent || progress_update) {\n> \t\t\tprogress->last_percent = percent;\n>\n> \t\t\tstrbuf_reset(counters_sb);\n> \t\t\tstrbuf_addf(counters_sb,\n> \t\t\t\t    \"%3u%% (%\"PRIuMAX\"/%\"PRIuMAX\")%s\", percent,\n> \t\t\t\t    (uintmax_t)n, (uintmax_t)progress->total,\n> \t\t\t\t    tp);\n> \t\t\tshow_update = 1;\n> \t\t}\n> \t} else if (progress_update) {\n> \t\tstrbuf_reset(counters_sb);\n> \t\tstrbuf_addf(counters_sb, \"%\"PRIuMAX\"%s\", (uintmax_t)n, tp);\n> \t\tshow_update = 1;\n> \t}\n> \t/* ... */\n> }\n\nHrm, now I'm also confused. Yes you're right. This whole patch doesn't\nmake sense.\n\nI.e. it's perfectly fine to provide a target of <num> for QSORT and then\nuse display_progress() just to bump the progress bar's display like\nthis, and we have other code in commit-graph.c that does the same.\n\nI think I might have (badly) extracted this patch from some WIP work\nwhere I was giving qsort() calls progress of some sort, or maybe I was\njust being stupid...\n\n(I have a few local experiments in progress bar output, one of those is\nthat you can give the API a <num> target, and not be sure if it'll be\nO(n), O(log(n)), O(n^2) etc, and we'll show progress output\nappropriately, i.e. \"still working, at X items of work for N, so north\nof O(N)...\".)\n"},{"id":"433360","messageId":"cover-v3-0.2-00000000000-20210823T102722Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v2-0.3-0000000000-20210805T105720Z-avarab@gmail.com","subject":"[PATCH v3 0/2] progress.c API users: fix bogus counting","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T10:29:39Z","receivedAt":"2021-08-23T10:29:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fixes up some users of the progress.c API. See\nhttps://lore.kernel.org/git/cover-v2-0.3-0000000000-20210805T105720Z-avarab@gmail.com\nfor v2.\n\nIn v3 the old 2/3 is ejected per feedback from Phillip Wood, see:\nhttps://lore.kernel.org/git/87v94jzoxj.fsf@evledraar.gmail.com/\n\nSZEDER Gábor (2):\n  commit-graph: fix bogus counter in \"Scanning merged commits\" progress\n    line\n  entry: show finer-grained counter in \"Filtering content\" progress line\n\n commit-graph.c | 2 +-\n entry.c        | 7 +++----\n 2 files changed, 4 insertions(+), 5 deletions(-)\n\nRange-diff against v2:\n1:  bcb13be5006 = 1:  443374551ad commit-graph: fix bogus counter in \"Scanning merged commits\" progress line\n2:  8e67712c480 < -:  ----------- midx: don't provide a total for QSORT() progress\n3:  c70e554e461 = 2:  71c93f624ec entry: show finer-grained counter in \"Filtering content\" progress line\n-- \n2.33.0.632.g78310755cd0\n\n"},{"id":"433361","messageId":"patch-v3-1.2-443374551ad-20210823T102722Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v3-0.2-00000000000-20210823T102722Z-avarab@gmail.com","subject":"[PATCH v3 1/2] commit-graph: fix bogus counter in \"Scanning merged commits\" progress line","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T10:29:40Z","receivedAt":"2021-08-23T10:29:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: SZEDER Gábor <szeder.dev@gmail.com>\n\nThe final value of the counter of the \"Scanning merged commits\"\nprogress line is always one less than its expected total, e.g.:\n\n  Scanning merged commits:  83% (5/6), done.\n\nThis happens because while iterating over an array the loop variable\nis passed to display_progress() as-is, but while C arrays (and thus\nthe loop variable) start at 0 and end at N-1, the progress counter\nmust end at N. Fix this by passing 'i + 1' to display_progress(), like\nmost other callsites do.\n\nThere's an RFC series to add a GIT_TEST_CHECK_PROGRESS=1 mode[1] which\ncatches this issue in the 'fetch.writeCommitGraph' and\n'fetch.writeCommitGraph with submodules' tests in\n't5510-fetch.sh'. The GIT_TEST_CHECK_PROGRESS=1 mode is not part of\nthis series, but future changes to progress.c may add it or similar\nassertions to catch this and similar bugs elsewhere.\n\n1. https://lore.kernel.org/git/20210620200303.2328957-1-szeder.dev@gmail.com/\n\nSigned-off-by: SZEDER Gábor <szeder.dev@gmail.com>\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n commit-graph.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 3860a0d8477..9d18c1d87d9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -2096,7 +2096,7 @@ static void sort_and_scan_merged_commits(struct write_commit_graph_context *ctx)\n \n \tctx->num_extra_edges = 0;\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n-\t\tdisplay_progress(ctx->progress, i);\n+\t\tdisplay_progress(ctx->progress, i + 1);\n \n \t\tif (i && oideq(&ctx->commits.list[i - 1]->object.oid,\n \t\t\t  &ctx->commits.list[i]->object.oid)) {\n-- \n2.33.0.632.g78310755cd0\n\n"},{"id":"433362","messageId":"patch-v3-2.2-71c93f624ec-20210823T102722Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v3-0.2-00000000000-20210823T102722Z-avarab@gmail.com","subject":"[PATCH v3 2/2] entry: show finer-grained counter in \"Filtering content\" progress line","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-23T10:29:41Z","receivedAt":"2021-08-23T10:29:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: SZEDER Gábor <szeder.dev@gmail.com>\n\nThe \"Filtering content\" progress in entry.c:finish_delayed_checkout()\nis unusual because of how it calculates the progress count and because\nit shows the progress of a nested loop.  It works basically like this:\n\n  start_delayed_progress(p, nr_of_paths_to_filter)\n  for_each_filter {\n      display_progress(p, nr_of_paths_to_filter - nr_of_paths_still_left_to_filter)\n      for_each_path_handled_by_the_current_filter {\n          checkout_entry()\n      }\n  }\n  stop_progress(p)\n\nThere are two issues with this approach:\n\n  - The work done by the last filter (or the only filter if there is\n    only one) is never counted, so if the last filter still has some\n    paths to process, then the counter shown in the \"done\" progress\n    line will not match the expected total.\n\n    The partially-RFC series to add a GIT_TEST_CHECK_PROGRESS=1\n    mode[1] helps spot this issue. Under it the 'missing file in\n    delayed checkout' and 'invalid file in delayed checkout' tests in\n    't0021-conversion.sh' fail, because both use only one\n    filter.  (The test 'delayed checkout in process filter' uses two\n    filters but the first one does all the work, so that test already\n    happens to succeed even with GIT_TEST_CHECK_PROGRESS=1.)\n\n  - The progress counter is updated only once per filter, not once per\n    processed path, so if a filter has a lot of paths to process, then\n    the counter might stay unchanged for a long while and then make a\n    big jump (though the user still gets a sense of progress, because\n    we call display_throughput() after each processed path to show the\n    amount of processed data).\n\nMove the display_progress() call to the inner loop, right next to that\ncheckout_entry() call that does the hard work for each path, and use a\ndedicated counter variable that is incremented upon processing each\npath.\n\nAfter this change the 'invalid file in delayed checkout' in\n't0021-conversion.sh' would succeed with the GIT_TEST_CHECK_PROGRESS=1\nassertion discussed above, but the 'missing file in delayed checkout'\ntest would still fail.\n\nIt'll fail because its purposefully buggy filter doesn't process any\npaths, so we won't execute that inner loop at all, see [2] for how to\nspot that issue without GIT_TEST_CHECK_PROGRESS=1. It's not\nstraightforward to fix it with the current progress.c library (see [3]\nfor an attempt), so let's leave it for now.\n\n1. https://lore.kernel.org/git/20210620200303.2328957-1-szeder.dev@gmail.com/\n2. http://lore.kernel.org/git/20210802214827.GE23408@szeder.dev\n3. https://lore.kernel.org/git/20210620200303.2328957-7-szeder.dev@gmail.com/\n\nSigned-off-by: SZEDER Gábor <szeder.dev@gmail.com>\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n entry.c | 7 +++----\n 1 file changed, 3 insertions(+), 4 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex 125fabdbd52..d92dd020b3d 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -162,7 +162,7 @@ static int remove_available_paths(struct string_list_item *item, void *cb_data)\n int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n {\n \tint errs = 0;\n-\tunsigned delayed_object_count;\n+\tunsigned processed_paths = 0;\n \toff_t filtered_bytes = 0;\n \tstruct string_list_item *filter, *path;\n \tstruct progress *progress;\n@@ -172,12 +172,10 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n \t\treturn errs;\n \n \tdco->state = CE_RETRY;\n-\tdelayed_object_count = dco->paths.nr;\n-\tprogress = start_delayed_progress(_(\"Filtering content\"), delayed_object_count);\n+\tprogress = start_delayed_progress(_(\"Filtering content\"), dco->paths.nr);\n \twhile (dco->filters.nr > 0) {\n \t\tfor_each_string_list_item(filter, &dco->filters) {\n \t\t\tstruct string_list available_paths = STRING_LIST_INIT_NODUP;\n-\t\t\tdisplay_progress(progress, delayed_object_count - dco->paths.nr);\n \n \t\t\tif (!async_query_available_blobs(filter->string, &available_paths)) {\n \t\t\t\t/* Filter reported an error */\n@@ -224,6 +222,7 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n \t\t\t\tce = index_file_exists(state->istate, path->string,\n \t\t\t\t\t\t       strlen(path->string), 0);\n \t\t\t\tif (ce) {\n+\t\t\t\t\tdisplay_progress(progress, ++processed_paths);\n \t\t\t\t\terrs |= checkout_entry(ce, state, NULL, nr_checkouts);\n \t\t\t\t\tfiltered_bytes += ce->ce_stat_data.sd_size;\n \t\t\t\t\tdisplay_throughput(progress, filtered_bytes);\n-- \n2.33.0.632.g78310755cd0\n\n"},{"id":"434174","messageId":"20210830211528.GK23408@szeder.dev","threadId":"55972","inReplyTo":"YNIJB8uYp7gzLC6v@nand.local","subject":"Re: [PATCH 2/7] progress: catch nested/overlapping progresses with GIT_TEST_CHECK_PROGRESS","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-08-30T21:15:28Z","receivedAt":"2021-08-30T21:15:33Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Jun 22, 2021 at 12:00:07PM -0400, Taylor Blau wrote:\n> On Sun, Jun 20, 2021 at 10:02:58PM +0200, SZEDER Gábor wrote:\n> > Note that this will trigger even in cases where the output is not\n> > visibly wrong, e.g. consider this simplified sequence of calls:\n> >\n> >   progress1 = start_delayed_progress();\n> >   progress2 = start_delayed_progress();\n> >   for (i = 0; ...)\n> >       display_progress(progress2, i + 1);\n> >   stop_progres(&progress2);\n> >   for (j = 0; ...)\n> >       display_progress(progress1, j + 1);\n> >   stop_progres(&progress1);\n> \n> s/stop_progres/&s, but no big deal. Everything else here looks good.\n\nWell, at least I was consistent :)\n\n> > diff --git a/progress.c b/progress.c\n> > index 255995406f..549e8d1fe7 100644\n> > --- a/progress.c\n> > +++ b/progress.c\n> > @@ -48,6 +48,8 @@ struct progress {\n> >  static volatile sig_atomic_t progress_update;\n> >\n> >  static int test_check_progress;\n> > +/* Used to catch nested/overlapping progresses with GIT_TEST_CHECK_PROGRESS. */\n> > +static struct progress *current_progress = NULL;\n> >\n> >  /*\n> >   * These are only intended for testing the progress output, i.e. exclusively\n> > @@ -258,8 +260,12 @@ static struct progress *start_progress_delay(const char *title, uint64_t total,\n> >  \tstruct progress *progress;\n> >\n> >  \ttest_check_progress = git_env_bool(\"GIT_TEST_CHECK_PROGRESS\", 0);\n> > +\tif (test_check_progress && current_progress)\n> > +\t\tBUG(\"progress \\\"%s\\\" is still active when starting new progress \\\"%s\\\"\",\n> > +\t\t    current_progress->title, title);\n> >\n> >  \tprogress = xmalloc(sizeof(*progress));\n> \n> Ah. This is why you moved the allocation down further, since we don't\n> have to free anything up when calling BUG() if it wasn't allocated in\n> the first place (and we had no such conditional that would cause us to\n> abort early before).\n> \n> For what it's worth, I probably would have preferred to see that change\n> from the previous patch included in this one rather than in the first of\n> the series, since it's much clearer here than it is in the first patch.\n\nYeah.  It must have been a rebase mishap.  (I started working on this\nafter I reported yet another commit-graph related progress bug around\nv2.31.0-rc0, and I had the first two checks on the same evening.  But\nthen some time later Peff came along and found a backwards counting\nprogress line, so I decided to add a check for that as well, which\nnecessitated a bit of refactoring in the other two checks, and then a\nhunk somehow ended up in the wrong patch.)\n\n"},{"id":"434175","messageId":"20210830211730.GL23408@szeder.dev","threadId":"55972","inReplyTo":"YNKT4Fr/LOGnx9/o@nand.local","subject":"Re: [PATCH 6/7] [RFC] entry: don't show \"Filtering content: ... done.\" line in case of errors","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-08-30T21:17:30Z","receivedAt":"2021-08-30T21:17:35Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Jun 22, 2021 at 09:52:32PM -0400, Taylor Blau wrote:\n> On Mon, Jun 21, 2021 at 08:32:56PM +0200, René Scharfe wrote:\n> > Am 20.06.21 um 22:03 schrieb SZEDER Gábor:\n> > > RFC!!  Alas, not calling stop_progress() on error has drawbacks:\n> > >\n> > >   - All memory allocated for the progress bar is leaked.\n> > >   - This progress line remains \"active\", in the sense that if we were\n> > >     to start a new progress later in the same git process, then with\n> > >     GIT_TEST_CHECK_PROGRESS it would trigger the other BUG() catching\n> > >     nested/overlapping progresses.\n> > >\n> > > Do we care?!  TBH I don't :)\n> > > Anyway, if we do, then we might need some sort of an abort_progress()\n> > > function...\n> >\n> > I think the abort_progress() idea makes sense; to clean up allocations,\n> > tell the user what happened and avoid the BUG().  Showing just\n> > \"aborted\" instead of \"done\" should suffice here -- the explanation is\n> > given a few lines later (\"'foo' was not filtered properly\").\n> \n> Very well put. I concur that having an abort_progress() API makes sense\n> for all of the reasons that you suggest, but also because we shouldn't\n> encourage not using what seems like an appropriate API in order to not\n> fail tests when GIT_TEST_CHECK_PROGRESS is set.\n\nAh, damn, I was hoping that I can get away with it :)\n\n"},{"id":"435192","messageId":"cover-v4-0.2-00000000000-20210909T010722Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v3-0.2-00000000000-20210823T102722Z-avarab@gmail.com","subject":"[PATCH v4 0/2] progress.c API users: fix bogus counting","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-09T01:10:10Z","receivedAt":"2021-09-09T01:10:18Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix bad uses of the progress.c API. See\nhttps://lore.kernel.org/git/cover-v3-0.2-00000000000-20210823T102722Z-avarab@gmail.com\nfor the v3.\n\nThis re-roll is on top of a merge conflict in v3 with 7a132c628e5\n(checkout: make delayed checkout respect --quiet and --no-progress,\n2021-08-26), i.e. the mt/quiet-with-delayed-checkout topic.\n\nSZEDER Gábor (2):\n  commit-graph: fix bogus counter in \"Scanning merged commits\" progress\n    line\n  entry: show finer-grained counter in \"Filtering content\" progress line\n\n commit-graph.c |  2 +-\n entry.c        | 12 +++++-------\n 2 files changed, 6 insertions(+), 8 deletions(-)\n\nRange-diff against v3:\n1:  443374551ad = 1:  4cc3923089d commit-graph: fix bogus counter in \"Scanning merged commits\" progress line\n2:  71c93f624ec ! 2:  54a09b5b883 entry: show finer-grained counter in \"Filtering content\" progress line\n    @@ Commit message\n         straightforward to fix it with the current progress.c library (see [3]\n         for an attempt), so let's leave it for now.\n     \n    +    Let's also initialize the *progress to \"NULL\" while we're at it. Since\n    +    7a132c628e5 (checkout: make delayed checkout respect --quiet and\n    +    --no-progress, 2021-08-26) we have had progress conditional on\n    +    \"show_progress\", usually we use the idiom of a \"NULL\" initialization\n    +    of the \"*progress\", rather than the more verbose ternary added in\n    +    7a132c628e5.\n    +\n         1. https://lore.kernel.org/git/20210620200303.2328957-1-szeder.dev@gmail.com/\n         2. http://lore.kernel.org/git/20210802214827.GE23408@szeder.dev\n         3. https://lore.kernel.org/git/20210620200303.2328957-7-szeder.dev@gmail.com/\n    @@ Commit message\n         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n     \n      ## entry.c ##\n    -@@ entry.c: static int remove_available_paths(struct string_list_item *item, void *cb_data)\n    - int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n    +@@ entry.c: int finish_delayed_checkout(struct checkout *state, int *nr_checkouts,\n    + \t\t\t    int show_progress)\n      {\n      \tint errs = 0;\n     -\tunsigned delayed_object_count;\n     +\tunsigned processed_paths = 0;\n      \toff_t filtered_bytes = 0;\n      \tstruct string_list_item *filter, *path;\n    - \tstruct progress *progress;\n    -@@ entry.c: int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n    +-\tstruct progress *progress;\n    ++\tstruct progress *progress = NULL;\n    + \tstruct delayed_checkout *dco = state->delayed_checkout;\n    + \n    + \tif (!state->delayed_checkout)\n      \t\treturn errs;\n      \n      \tdco->state = CE_RETRY;\n     -\tdelayed_object_count = dco->paths.nr;\n    --\tprogress = start_delayed_progress(_(\"Filtering content\"), delayed_object_count);\n    -+\tprogress = start_delayed_progress(_(\"Filtering content\"), dco->paths.nr);\n    +-\tprogress = show_progress\n    +-\t\t? start_delayed_progress(_(\"Filtering content\"), delayed_object_count)\n    +-\t\t: NULL;\n    ++\tif (show_progress)\n    ++\t\tprogress = start_delayed_progress(_(\"Filtering content\"), dco->paths.nr);\n      \twhile (dco->filters.nr > 0) {\n      \t\tfor_each_string_list_item(filter, &dco->filters) {\n      \t\t\tstruct string_list available_paths = STRING_LIST_INIT_NODUP;\n    @@ entry.c: int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n      \n      \t\t\tif (!async_query_available_blobs(filter->string, &available_paths)) {\n      \t\t\t\t/* Filter reported an error */\n    -@@ entry.c: int finish_delayed_checkout(struct checkout *state, int *nr_checkouts)\n    +@@ entry.c: int finish_delayed_checkout(struct checkout *state, int *nr_checkouts,\n      \t\t\t\tce = index_file_exists(state->istate, path->string,\n      \t\t\t\t\t\t       strlen(path->string), 0);\n      \t\t\t\tif (ce) {\n-- \n2.33.0.825.gdc3f7a2a6c7\n\n"},{"id":"435193","messageId":"patch-v4-1.2-4cc3923089d-20210909T010722Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v4-0.2-00000000000-20210909T010722Z-avarab@gmail.com","subject":"[PATCH v4 1/2] commit-graph: fix bogus counter in \"Scanning merged commits\" progress line","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-09T01:10:11Z","receivedAt":"2021-09-09T01:10:19Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: SZEDER Gábor <szeder.dev@gmail.com>\n\nThe final value of the counter of the \"Scanning merged commits\"\nprogress line is always one less than its expected total, e.g.:\n\n  Scanning merged commits:  83% (5/6), done.\n\nThis happens because while iterating over an array the loop variable\nis passed to display_progress() as-is, but while C arrays (and thus\nthe loop variable) start at 0 and end at N-1, the progress counter\nmust end at N. Fix this by passing 'i + 1' to display_progress(), like\nmost other callsites do.\n\nThere's an RFC series to add a GIT_TEST_CHECK_PROGRESS=1 mode[1] which\ncatches this issue in the 'fetch.writeCommitGraph' and\n'fetch.writeCommitGraph with submodules' tests in\n't5510-fetch.sh'. The GIT_TEST_CHECK_PROGRESS=1 mode is not part of\nthis series, but future changes to progress.c may add it or similar\nassertions to catch this and similar bugs elsewhere.\n\n1. https://lore.kernel.org/git/20210620200303.2328957-1-szeder.dev@gmail.com/\n\nSigned-off-by: SZEDER Gábor <szeder.dev@gmail.com>\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n commit-graph.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 00614acd65d..46170592204 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -2125,7 +2125,7 @@ static void sort_and_scan_merged_commits(struct write_commit_graph_context *ctx)\n \n \tctx->num_extra_edges = 0;\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n-\t\tdisplay_progress(ctx->progress, i);\n+\t\tdisplay_progress(ctx->progress, i + 1);\n \n \t\tif (i && oideq(&ctx->commits.list[i - 1]->object.oid,\n \t\t\t  &ctx->commits.list[i]->object.oid)) {\n-- \n2.33.0.825.gdc3f7a2a6c7\n\n"},{"id":"435194","messageId":"patch-v4-2.2-54a09b5b883-20210909T010722Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v4-0.2-00000000000-20210909T010722Z-avarab@gmail.com","subject":"[PATCH v4 2/2] entry: show finer-grained counter in \"Filtering content\" progress line","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-09T01:10:12Z","receivedAt":"2021-09-09T01:10:19Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: SZEDER Gábor <szeder.dev@gmail.com>\n\nThe \"Filtering content\" progress in entry.c:finish_delayed_checkout()\nis unusual because of how it calculates the progress count and because\nit shows the progress of a nested loop.  It works basically like this:\n\n  start_delayed_progress(p, nr_of_paths_to_filter)\n  for_each_filter {\n      display_progress(p, nr_of_paths_to_filter - nr_of_paths_still_left_to_filter)\n      for_each_path_handled_by_the_current_filter {\n          checkout_entry()\n      }\n  }\n  stop_progress(p)\n\nThere are two issues with this approach:\n\n  - The work done by the last filter (or the only filter if there is\n    only one) is never counted, so if the last filter still has some\n    paths to process, then the counter shown in the \"done\" progress\n    line will not match the expected total.\n\n    The partially-RFC series to add a GIT_TEST_CHECK_PROGRESS=1\n    mode[1] helps spot this issue. Under it the 'missing file in\n    delayed checkout' and 'invalid file in delayed checkout' tests in\n    't0021-conversion.sh' fail, because both use only one\n    filter.  (The test 'delayed checkout in process filter' uses two\n    filters but the first one does all the work, so that test already\n    happens to succeed even with GIT_TEST_CHECK_PROGRESS=1.)\n\n  - The progress counter is updated only once per filter, not once per\n    processed path, so if a filter has a lot of paths to process, then\n    the counter might stay unchanged for a long while and then make a\n    big jump (though the user still gets a sense of progress, because\n    we call display_throughput() after each processed path to show the\n    amount of processed data).\n\nMove the display_progress() call to the inner loop, right next to that\ncheckout_entry() call that does the hard work for each path, and use a\ndedicated counter variable that is incremented upon processing each\npath.\n\nAfter this change the 'invalid file in delayed checkout' in\n't0021-conversion.sh' would succeed with the GIT_TEST_CHECK_PROGRESS=1\nassertion discussed above, but the 'missing file in delayed checkout'\ntest would still fail.\n\nIt'll fail because its purposefully buggy filter doesn't process any\npaths, so we won't execute that inner loop at all, see [2] for how to\nspot that issue without GIT_TEST_CHECK_PROGRESS=1. It's not\nstraightforward to fix it with the current progress.c library (see [3]\nfor an attempt), so let's leave it for now.\n\nLet's also initialize the *progress to \"NULL\" while we're at it. Since\n7a132c628e5 (checkout: make delayed checkout respect --quiet and\n--no-progress, 2021-08-26) we have had progress conditional on\n\"show_progress\", usually we use the idiom of a \"NULL\" initialization\nof the \"*progress\", rather than the more verbose ternary added in\n7a132c628e5.\n\n1. https://lore.kernel.org/git/20210620200303.2328957-1-szeder.dev@gmail.com/\n2. http://lore.kernel.org/git/20210802214827.GE23408@szeder.dev\n3. https://lore.kernel.org/git/20210620200303.2328957-7-szeder.dev@gmail.com/\n\nSigned-off-by: SZEDER Gábor <szeder.dev@gmail.com>\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n entry.c | 12 +++++-------\n 1 file changed, 5 insertions(+), 7 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex 044e8ec92c6..9b0f968a70c 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -163,24 +163,21 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts,\n \t\t\t    int show_progress)\n {\n \tint errs = 0;\n-\tunsigned delayed_object_count;\n+\tunsigned processed_paths = 0;\n \toff_t filtered_bytes = 0;\n \tstruct string_list_item *filter, *path;\n-\tstruct progress *progress;\n+\tstruct progress *progress = NULL;\n \tstruct delayed_checkout *dco = state->delayed_checkout;\n \n \tif (!state->delayed_checkout)\n \t\treturn errs;\n \n \tdco->state = CE_RETRY;\n-\tdelayed_object_count = dco->paths.nr;\n-\tprogress = show_progress\n-\t\t? start_delayed_progress(_(\"Filtering content\"), delayed_object_count)\n-\t\t: NULL;\n+\tif (show_progress)\n+\t\tprogress = start_delayed_progress(_(\"Filtering content\"), dco->paths.nr);\n \twhile (dco->filters.nr > 0) {\n \t\tfor_each_string_list_item(filter, &dco->filters) {\n \t\t\tstruct string_list available_paths = STRING_LIST_INIT_NODUP;\n-\t\t\tdisplay_progress(progress, delayed_object_count - dco->paths.nr);\n \n \t\t\tif (!async_query_available_blobs(filter->string, &available_paths)) {\n \t\t\t\t/* Filter reported an error */\n@@ -227,6 +224,7 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts,\n \t\t\t\tce = index_file_exists(state->istate, path->string,\n \t\t\t\t\t\t       strlen(path->string), 0);\n \t\t\t\tif (ce) {\n+\t\t\t\t\tdisplay_progress(progress, ++processed_paths);\n \t\t\t\t\terrs |= checkout_entry(ce, state, NULL, nr_checkouts);\n \t\t\t\t\tfiltered_bytes += ce->ce_stat_data.sd_size;\n \t\t\t\t\tdisplay_throughput(progress, filtered_bytes);\n-- \n2.33.0.825.gdc3f7a2a6c7\n\n"},{"id":"435327","messageId":"xmqqee9xy0t1.fsf@gitster.g","threadId":"55972","inReplyTo":"cover-v4-0.2-00000000000-20210909T010722Z-avarab@gmail.com","subject":"Re: [PATCH v4 0/2] progress.c API users: fix bogus counting","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-09T20:02:50Z","receivedAt":"2021-09-09T20:02:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:\n\n> Fix bad uses of the progress.c API. See\n> https://lore.kernel.org/git/cover-v3-0.2-00000000000-20210823T102722Z-avarab@gmail.com\n> for the v3.\n>\n> This re-roll is on top of a merge conflict in v3 with 7a132c628e5\n> (checkout: make delayed checkout respect --quiet and --no-progress,\n> 2021-08-26), i.e. the mt/quiet-with-delayed-checkout topic.\n\nThanks, as that commit makes the call to progress code conditional,\nwith a new variable involved in the decision, it is understandable\nthat this needs to be adjusted for that newer codebase.\n\nVery much appreciated.\n"},{"id":"436143","messageId":"20210916183137.GD76263@szeder.dev","threadId":"55972","inReplyTo":"patch-12.25-c4e7179e957-20210623T155626Z-avarab@gmail.com","subject":"Re: [PATCH 12/25] progress.c: add & assert a \"global_progress\" variable","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-09-16T18:31:37Z","receivedAt":"2021-09-16T19:00:53Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Jun 23, 2021 at 07:48:12PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> The progress.c code makes a hard assumption that only one progress bar\n> be active at a time (see [1] for a bug where this wasn't the case),\n> but nothing has asserted that that's the case. Let's add a BUG()\n> that'll trigger if two progress bars are active at the same time.\n\nI very much dislike the idea of any BUG() in the progress code that\ncan trigger outside of the test suite.\n\nAs the number of progress-related fixes clearly show, we often misuse\nthe progress API, and, arguably, a bug is a bug is a bug, so strictly\nspeaking a BUG() is not wrong here.\n\nHowever, the progress line is merely a UI gimmick, not a crucial part\nof Git, and none of those progress bugs affected the correctness of\nthe operation itself.  Worse, calling BUG() during some operations\n(e.g. 'git commit-graph write', the worst offender when it comes to\nprogress bugs) can leave a lockfile behind, resulting in scary errors\nand requiring manual cleanup in the .git directory, which is a much\nworse UX than showing some bogus values or out of order progress\nlines.\n\n"},{"id":"436144","messageId":"20210916183711.GE76263@szeder.dev","threadId":"55972","inReplyTo":"patch-18.25-e21fc66623f-20210623T155626Z-avarab@gmail.com","subject":"Re: [PATCH 18/25] progress.c: emit progress on first signal, show \"stalled\"","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-09-16T18:37:11Z","receivedAt":"2021-09-16T19:03:33Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Jun 23, 2021 at 07:48:18PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Ever since the progress.c code was added in 96a02f8f6d2 (common\n> progress display support, 2007-04-18) we have been driven purely by\n> calls to the display() function (via the public display_progress()),\n> or via stop_progress(). Even though we got a signal and invoked\n> progress_interval() that function would not actually emit progress\n> output for us.\n> \n> Thus in cases like \"git gc\" we don't emit any \"Enumerating Objects\"\n> output until we get past the setup code, and start enumerating\n> objects, we'll now (at least on my laptop) show output earlier, and\n> emit a \"stalled\" message before we start the count.\n> \n> But more generally, this is a first step towards never showing a\n> hanging progress bar from the user's perspective. If we're truly\n> taking a very long time with one item we can show some spinner that we\n> update every time we get a signal. We don't right now, and only\n> special-case the most common case of hanging before we get to the\n> first item.\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>  progress.c                  |  7 +++++\n>  t/t0500-progress-display.sh | 63 ++++++++++++++++++++++++++++++++++---\n>  2 files changed, 66 insertions(+), 4 deletions(-)\n> \n> diff --git a/progress.c b/progress.c\n> index 6c4038df791..35847d3a7f2 100644\n> --- a/progress.c\n> +++ b/progress.c\n> @@ -255,6 +255,13 @@ void display_progress(struct progress *progress, uint64_t n)\n>  static void progress_interval(int signum)\n>  {\n>  \tprogress_update = 1;\n> +\n> +\tif (global_progress->last_value != -1)\n> +\t\treturn;\n> +\n> +\tdisplay(global_progress, 0, _(\", stalled.\"), 0);\n\nWe have a few progress lines that are updated from multiple threads,\nand to prevent concurrency issues those threads call display() while\nholding a mutex.  This call without synchronization causes undefined\nbehavior.\n\n"},{"id":"436149","messageId":"20210916194243.GA2118053@szeder.dev","threadId":"55972","inReplyTo":"patch-13.25-f2e84e3a3f2-20210623T155626Z-avarab@gmail.com","subject":"Re: [PATCH 13/25] progress.[ch]: move the \"struct progress\" to the header","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-09-16T19:42:43Z","receivedAt":"2021-09-16T19:42:53Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Jun 23, 2021 at 07:48:13PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Move the definition of the \"struct progress\" to the progress.h\n> header. Even though its contents are meant to be \"private\" this\n> pattern has resulted in forward declarations of it in various places,\n> as other functions have a need to pass it around.\n> \n> Let's just define it in the header instead. \n\nThis is not a good excuse to move the definition of 'struct progress'\nto the header file.  Defining a struct in a C source file and\ndeclaring it in header files is C's well-established way to create\nan opaque data type and to hide implementation details, so there is\nnothing wrong with those forward declarations, and keeping 'struct\nprogress' private to 'progress.c' is a good thing.\n\nHaving said that, we can simply remove all those forward declarations\nwithout moving the definition of 'struct progress' to 'progress.h',\nand still successfully build git.  The reason is that in 'cache.h':\n\n  struct index_state {\n    [...]\n    struct progress *progress;\n    [...]\n  };\n\ndoes count as a forward declaration of 'struct progress', and\n'cache.h' is the first header included in just about all our C source\nfiles, rendering the other forward declaration unnecessary.\n\n\n> It's part of our own\n> internal code, so we're not at much risk of someone tweaking the\n> internal fields manually. While doing that rename the \"TP_IDX_MAX\"\n> macro to the more clearly namespaced \"PROGRESS_THROUGHPUT_IDX_MAX\".\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>  cache.h             |  1 -\n>  csum-file.h         |  2 --\n>  pack.h              |  1 -\n>  parallel-checkout.h |  1 -\n>  progress.c          | 29 +----------------------------\n>  progress.h          | 28 +++++++++++++++++++++++++++-\n>  reachable.h         |  1 -\n>  7 files changed, 28 insertions(+), 35 deletions(-)\n> \n> diff --git a/cache.h b/cache.h\n> index ba04ff8bd36..7e03a181f68 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -308,7 +308,6 @@ static inline unsigned int canon_mode(unsigned int mode)\n>  \n>  struct split_index;\n>  struct untracked_cache;\n> -struct progress;\n>  struct pattern_list;\n>  \n>  struct index_state {\n> diff --git a/csum-file.h b/csum-file.h\n> index 3044bd19ab6..3de0de653e8 100644\n> --- a/csum-file.h\n> +++ b/csum-file.h\n> @@ -3,8 +3,6 @@\n>  \n>  #include \"hash.h\"\n>  \n> -struct progress;\n> -\n>  /* A SHA1-protected file */\n>  struct hashfile {\n>  \tint fd;\n> diff --git a/pack.h b/pack.h\n> index fa139545262..8df04f4937a 100644\n> --- a/pack.h\n> +++ b/pack.h\n> @@ -77,7 +77,6 @@ struct pack_idx_entry {\n>  };\n>  \n>  \n> -struct progress;\n>  /* Note, the data argument could be NULL if object type is blob */\n>  typedef int (*verify_fn)(const struct object_id *, enum object_type, unsigned long, void*, int*);\n>  \n> diff --git a/parallel-checkout.h b/parallel-checkout.h\n> index 80f539bcb77..193f76398d6 100644\n> --- a/parallel-checkout.h\n> +++ b/parallel-checkout.h\n> @@ -5,7 +5,6 @@\n>  \n>  struct cache_entry;\n>  struct checkout;\n> -struct progress;\n>  \n>  /****************************************************************\n>   * Users of parallel checkout\n> diff --git a/progress.c b/progress.c\n> index e1b50ef7882..aff9af9ee8b 100644\n> --- a/progress.c\n> +++ b/progress.c\n> @@ -17,33 +17,6 @@\n>  #include \"utf8.h\"\n>  #include \"config.h\"\n>  \n> -#define TP_IDX_MAX      8\n> -\n> -struct throughput {\n> -\toff_t curr_total;\n> -\toff_t prev_total;\n> -\tuint64_t prev_ns;\n> -\tunsigned int avg_bytes;\n> -\tunsigned int avg_misecs;\n> -\tunsigned int last_bytes[TP_IDX_MAX];\n> -\tunsigned int last_misecs[TP_IDX_MAX];\n> -\tunsigned int idx;\n> -\tstruct strbuf display;\n> -};\n> -\n> -struct progress {\n> -\tconst char *title;\n> -\tuint64_t last_value;\n> -\tuint64_t total;\n> -\tunsigned last_percent;\n> -\tunsigned delay;\n> -\tstruct throughput *throughput;\n> -\tuint64_t start_ns;\n> -\tstruct strbuf counters_sb;\n> -\tint title_len;\n> -\tint split;\n> -};\n> -\n>  static volatile sig_atomic_t progress_update;\n>  static struct progress *global_progress;\n>  \n> @@ -194,7 +167,7 @@ void display_throughput(struct progress *progress, uint64_t total)\n>  \ttp->avg_misecs -= tp->last_misecs[tp->idx];\n>  \ttp->last_bytes[tp->idx] = count;\n>  \ttp->last_misecs[tp->idx] = misecs;\n> -\ttp->idx = (tp->idx + 1) % TP_IDX_MAX;\n> +\ttp->idx = (tp->idx + 1) % PROGRESS_THROUGHPUT_IDX_MAX;\n>  \n>  \tthroughput_string(&tp->display, total, rate);\n>  \tif (progress->last_value != -1 && progress_update)\n> diff --git a/progress.h b/progress.h\n> index f1913acf73f..4fb2b483d36 100644\n> --- a/progress.h\n> +++ b/progress.h\n> @@ -1,7 +1,33 @@\n>  #ifndef PROGRESS_H\n>  #define PROGRESS_H\n> +#include \"strbuf.h\"\n>  \n> -struct progress;\n> +#define PROGRESS_THROUGHPUT_IDX_MAX      8\n> +\n> +struct throughput {\n> +\toff_t curr_total;\n> +\toff_t prev_total;\n> +\tuint64_t prev_ns;\n> +\tunsigned int avg_bytes;\n> +\tunsigned int avg_misecs;\n> +\tunsigned int last_bytes[PROGRESS_THROUGHPUT_IDX_MAX];\n> +\tunsigned int last_misecs[PROGRESS_THROUGHPUT_IDX_MAX];\n> +\tunsigned int idx;\n> +\tstruct strbuf display;\n> +};\n> +\n> +struct progress {\n> +\tconst char *title;\n> +\tuint64_t last_value;\n> +\tuint64_t total;\n> +\tunsigned last_percent;\n> +\tunsigned delay;\n> +\tstruct throughput *throughput;\n> +\tuint64_t start_ns;\n> +\tstruct strbuf counters_sb;\n> +\tint title_len;\n> +\tint split;\n> +};\n>  \n>  #ifdef GIT_TEST_PROGRESS_ONLY\n>  \n> diff --git a/reachable.h b/reachable.h\n> index 5df932ad8f5..7e1ddddbc63 100644\n> --- a/reachable.h\n> +++ b/reachable.h\n> @@ -1,7 +1,6 @@\n>  #ifndef REACHEABLE_H\n>  #define REACHEABLE_H\n>  \n> -struct progress;\n>  struct rev_info;\n>  \n>  int add_unseen_recent_objects_to_traversal(struct rev_info *revs,\n> -- \n> 2.32.0.599.g3967b4fa4ac\n> \n"},{"id":"436205","messageId":"20210917051448.GB2118053@szeder.dev","threadId":"55972","inReplyTo":"patch-11.25-7f5b71fe1be-20210623T155626Z-avarab@gmail.com","subject":"Re: [PATCH 11/25] pack-bitmap-write.c: add a missing stop_progress()","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-09-17T05:14:48Z","receivedAt":"2021-09-17T05:14:55Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Jun 23, 2021 at 07:48:11PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Fix a bug that's been here since 7cc8f971085 (pack-objects: implement\n> bitmap writing, 2013-12-21), we did not call stop_progress() if we\n> reached the early exit in this function. This will matter in a\n> subsequent commit where we BUG(...) out if this happens, and matters\n> now e.g. because we don't have a corresponding \"region_end\" for the\n> progress trace2 event.\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>  pack-bitmap-write.c | 1 +\n>  1 file changed, 1 insertion(+)\n> \n> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> index 88d9e696a54..6e110e41ea4 100644\n> --- a/pack-bitmap-write.c\n> +++ b/pack-bitmap-write.c\n> @@ -550,6 +550,7 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n>  \tif (indexed_commits_nr < 100) {\n>  \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n>  \t\t\tpush_bitmapped_commit(indexed_commits[i]);\n> +\t\tstop_progress(&writer.progress);\n>  \t\treturn;\n>  \t}\n\nWhen I found this bug I fixed it differently: with your patch there\nare no display() calls at all between start_progress() and this new\nstop_progress(), indicating that a stop_progress() is not missing but\nrather the start_progress is in the wrong place:\n\n  ---  >8  ---\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 88d9e696a5..f0b4044e2b 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -544,15 +544,15 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \n \tQSORT(indexed_commits, indexed_commits_nr, date_compare);\n \n-\tif (writer.show_progress)\n-\t\twriter.progress = start_progress(\"Selecting bitmap commits\", 0);\n-\n \tif (indexed_commits_nr < 100) {\n \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n \t\t\tpush_bitmapped_commit(indexed_commits[i]);\n \t\treturn;\n \t}\n \n+\tif (writer.show_progress)\n+\t\twriter.progress = start_progress(\"Selecting bitmap commits\", 0);\n+\n \tfor (;;) {\n \t\tstruct commit *chosen = NULL;\n \n  ---  8<  ---\n\nAnd I don't think it's worth adding display() calls to that loop,\nbecause it has so few iterations and it does barely anything per\niteration.\n\n"},{"id":"436207","messageId":"877dffg37n.fsf@evledraar.gmail.com","threadId":"55972","inReplyTo":"patch-8.8-1a14176bee-20210722T125012Z-avarab@gmail.com","subject":"Re: [PATCH 12/25] progress.c: add & assert a \"global_progress\" variable","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-16T21:34:56Z","receivedAt":"2021-09-17T05:50:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nI've adjusted the In-reply-to header here.\n\nThis is really in reply to <20210916183137.GD76263@szeder.dev>, but\nthat's a reply to a previous & longer RFC-ish iteration of the\nseries. https://lore.kernel.org/git/cover-0.8-0000000000-20210722T125012Z-avarab@gmail.com/\nis what's currently queued as ab/only-single-progress-at-once in \"seen\".\n\nLet's continue the discussion relevant to the currently proposed patches\nin this thread.\n\nOn Thu, Sep 16 2021, SZEDER Gábor wrote:\n\n> On Wed, Jun 23, 2021 at 07:48:12PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> The progress.c code makes a hard assumption that only one progress bar\n>> be active at a time (see [1] for a bug where this wasn't the case),\n>> but nothing has asserted that that's the case. Let's add a BUG()\n>> that'll trigger if two progress bars are active at the same time.\n>\n> I very much dislike the idea of any BUG() in the progress code that\n> can trigger outside of the test suite.\n\nFirst regarding the state of this series. I'd understood your:\n\n    Please don't advance this to next yet.  I've found some issues with\n    it, but not the time to raise them.\n\nIn https://lore.kernel.org/git/20210901050406.GB76263@szeder.dev/ to\nmean (, by perhaps overly reading between the lines out of paranoia,)\nthat you'd found some case where we'd hit the BUG() here.\n\nAm I correct that you haven't, but are concerned that we've left some\ncase undiscovered where we might?\n\nI'm wondering given your replies there if you thought that the series to\nbe merged down was that 25 patch one. I agree (e.g. because of your\nhttps://lore.kernel.org/git/20210916183711.GE76263@szeder.dev/) that\nmerging that one down wouldn't be a good idea in its current state.\n\nOf course you may still have valid concerns etc., just trying to clarify\nif there's some specific known outstanding issue or not with the current\n8-patch series in \"seen\".\n\n> As the number of progress-related fixes clearly show, we often misuse\n> the progress API, and, arguably, a bug is a bug is a bug, so strictly\n> speaking a BUG() is not wrong here.\n>\n> However, the progress line is merely a UI gimmick, not a crucial part\n> of Git, and none of those progress bugs affected the correctness of\n> the operation itself.  Worse, calling BUG() during some operations\n> (e.g. 'git commit-graph write', the worst offender when it comes to\n> progress bugs) can leave a lockfile behind, resulting in scary errors\n> and requiring manual cleanup in the .git directory, which is a much\n> worse UX than showing some bogus values or out of order progress\n> lines.\n\nYes, I agree that actually hitting this BUG() would absolutely suck, and\nthat we shouldn't consider this patch if we weren't certain, or near\nenough, that we wouldn't hit it.\n\nAs covered in the cover letter of the earlier series I sent (at\nhttps://lore.kernel.org/git/cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com/)\nI share your concerns that it's hard to uncover if we've reached\nsufficient coverage to be certain that we should add certain assertions,\ni.e. some of the ones your initial series added around the actual\nprogress counting that we do.\n\nIn this case though, I do think we can safely add this. Maybe you have\nrun into the BUG() and I'm about to get some egg on my face, but here's\nwhy I think we won't hit it.\n\nThis is a *much* narrower case than the general thread quagmire around\ndisplay_progress() etc., since this only covers the start/stop progress\ncalls. All of the multi-threaded code we have does the equivalent of:\n\n    p = start_progress();\n    start_threads();\n    /* do stuff in threads, including with \"p\" */\n    stop_threads();\n    stop_progress(p);\n\nDo you agree that if this is the pattern everywhere that this patch\nwould be safe in its current form?\n\nTo assert that that's the case (I'd read the code before) I instrumented\nthe tests to BUG() out if we ever start() or stop() where getpid() !=\ngettid(), which on Linux means you're inside a pthread. The diff-on-top\nis at the end of this E-Mail.\n\nWith that and running the tests with:\n\n    GIT_TEST_BUG_START=1 \\\n    GIT_TEST_BUG_STOP=1\n\nWe pass all tests, i.e. there's no current callers that call\nstart_progress() or start_progress() and do so in anything but the main\nprogram thread.\n\nWell, there could be, per the concerns I had in the CL linked above,\ni.e. sometimes that start_progress() is guarded by a preceding isatty()\nor whatever, but having looked at those / grepped the union of pthread +\nprogress in the source I think there's no such cases. That's *much*\neasier to eyeball than \"does all this control flow around\ndisplay_progress() make sense?\".\n\nGetting back on topic, we pass all tests with that, but we'll fail with:\n\n    GIT_TEST_BUG_DISPLAY=1 \\\n    GIT_TEST_BUG_DISPLAY_HARDER=1\n\nWhy the second variable? Because I marked up the two callers that call\ndisplay_progress() within threads, it's just index-pack and\npack-objects. If you don't set GIT_TEST_BUG_DISPLAY_HARDER=1 we'll\nwhitelist those two, and all tests pass.\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 6cc48902170..05c82dc6e6d 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1229,6 +1229,7 @@ static void resolve_deltas(void)\n \tQSORT(ofs_deltas, nr_ofs_deltas, compare_ofs_delta_entry);\n \tQSORT(ref_deltas, nr_ref_deltas, compare_ref_delta_entry);\n \n+\tno_progress_bug();\n \tif (verbose || show_resolving_progress)\n \t\tprogress = start_progress(_(\"Resolving deltas\"),\n \t\t\t\t\t  nr_ref_deltas + nr_ofs_deltas);\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex ec8503563a6..97ad321f67c 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3072,6 +3072,7 @@ static void prepare_pack(int window, int depth)\n \tif (nr_deltas && n > 1) {\n \t\tunsigned nr_done = 0;\n \n+\t\tno_progress_bug();\n \t\tif (progress)\n \t\t\tprogress_state = start_progress(_(\"Compressing objects\"),\n \t\t\t\t\t\t\tnr_deltas);\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex f3aba5d6cbb..efa1e5e5bdf 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -5,6 +5,9 @@\n \n . ${0%/*}/lib.sh\n \n+export GIT_TEST_BUG_START=true\n+export GIT_TEST_BUG_STOP=true\n+\n case \"$CI_OS_NAME\" in\n windows*) cmd //c mklink //j t\\\\.prove \"$(cygpath -aw \"$cache_dir/.prove\")\";;\n *) ln -s \"$cache_dir/.prove\" t/.prove;;\ndiff --git a/progress.c b/progress.c\nindex 14a023f4b43..bdebf100ad8 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -16,6 +16,7 @@\n #include \"trace.h\"\n #include \"utf8.h\"\n #include \"config.h\"\n+#include <unistd.h>\n \n #define TP_IDX_MAX      8\n \n@@ -202,8 +203,18 @@ void display_throughput(struct progress *progress, uint64_t total)\n \t\tdisplay(progress, progress->last_value, NULL);\n }\n \n+static int no_bug_please;\n+void no_progress_bug(void)\n+{\n+\tno_bug_please = 1;\n+}\n+\n void display_progress(struct progress *progress, uint64_t n)\n {\n+\tif (getpid() != gettid() && getenv(\"GIT_TEST_BUG_DISPLAY\") &&\n+\t    (getenv(\"GIT_TEST_BUG_DISPLAY_HARDER\") || !no_bug_please))\n+\t\tBUG(\"display: pid = %d, tid = %d: %s\\n\", getpid(), gettid(),\n+\t\t    progress ? progress->title : \"N/A\");\n \tif (progress)\n \t\tdisplay(progress, n, NULL);\n }\n@@ -281,6 +292,11 @@ static struct progress *start_progress_delay(const char *title, uint64_t total,\n \tprogress->split = 0;\n \tset_progress_signal(progress);\n \ttrace2_region_enter(\"progress\", title, the_repository);\n+\n+\tif (getpid() != gettid() && getenv(\"GIT_TEST_BUG_START\"))\n+\t\tBUG(\"start: pid = %d, tid = %d: %s\\n\", getpid(), gettid(),\n+\t\t    title ? title : \"N/A\");\n+\n \treturn progress;\n }\n \n@@ -334,6 +350,11 @@ static void finish_if_sparse(struct progress *progress)\n \n void stop_progress(struct progress **p_progress)\n {\n+\tno_bug_please = 0;\n+\tif (getpid() != gettid() && getenv(\"GIT_TEST_BUG_STOP\"))\n+\t\tBUG(\"stop: pid = %d, tid = %d: %s\\n\", getpid(), gettid(),\n+\t\t    ((p_progress && *p_progress) ? (*p_progress)->title : \"N/A\"));\n+\n \tif (!p_progress)\n \t\tBUG(\"don't provide NULL to stop_progress\");\n \ndiff --git a/progress.h b/progress.h\nindex f1913acf73f..2ebb1da2666 100644\n--- a/progress.h\n+++ b/progress.h\n@@ -20,5 +20,6 @@ struct progress *start_delayed_sparse_progress(const char *title,\n \t\t\t\t\t       uint64_t total);\n void stop_progress(struct progress **progress);\n void stop_progress_msg(struct progress **progress, const char *msg);\n+void no_progress_bug(void);\n \n #endif\n"},{"id":"436208","messageId":"8735q3g2rg.fsf@evledraar.gmail.com","threadId":"55972","inReplyTo":"20210917051448.GB2118053@szeder.dev","subject":"Re: [PATCH 11/25] pack-bitmap-write.c: add a missing stop_progress()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-17T05:56:48Z","receivedAt":"2021-09-17T06:00:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Sep 17 2021, SZEDER Gábor wrote:\n\n> On Wed, Jun 23, 2021 at 07:48:11PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> Fix a bug that's been here since 7cc8f971085 (pack-objects: implement\n>> bitmap writing, 2013-12-21), we did not call stop_progress() if we\n>> reached the early exit in this function. This will matter in a\n>> subsequent commit where we BUG(...) out if this happens, and matters\n>> now e.g. because we don't have a corresponding \"region_end\" for the\n>> progress trace2 event.\n>> \n>> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n>> ---\n>>  pack-bitmap-write.c | 1 +\n>>  1 file changed, 1 insertion(+)\n>> \n>> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n>> index 88d9e696a54..6e110e41ea4 100644\n>> --- a/pack-bitmap-write.c\n>> +++ b/pack-bitmap-write.c\n>> @@ -550,6 +550,7 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n>>  \tif (indexed_commits_nr < 100) {\n>>  \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n>>  \t\t\tpush_bitmapped_commit(indexed_commits[i]);\n>> +\t\tstop_progress(&writer.progress);\n>>  \t\treturn;\n>>  \t}\n>\n> When I found this bug I fixed it differently: with your patch there\n\nIs that patch on-list somewhere or something you have locally?\n\n> are no display() calls at all between start_progress() and this new\n> stop_progress(), indicating that a stop_progress() is not missing but\n> rather the start_progress is in the wrong place:\n\n*Nod*, I'll see about fixing it differenty depending on the above / any\nother comments.\n\nNote that while this comment is current to\nhttps://lore.kernel.org/git/patch-7.8-eb63b4ba6a-20210722T125012Z-avarab@gmail.com/\nas well, as noted in\nhttps://lore.kernel.org/git/877dffg37n.fsf@evledraar.gmail.com/ you've\nhad several comments on the 25 patch series not currently queued in\n\"seen\".\n\nStill very useful as I'd had some of it planned for after that 8-patch\nseries, just noting it for context.\n"},{"id":"436269","messageId":"20210917213815.GD2118053@szeder.dev","threadId":"55972","inReplyTo":"8735q3g2rg.fsf@evledraar.gmail.com","subject":"Re: [PATCH 11/25] pack-bitmap-write.c: add a missing stop_progress()","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-09-17T21:38:15Z","receivedAt":"2021-09-17T21:38:23Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Fri, Sep 17, 2021 at 07:56:48AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Fri, Sep 17 2021, SZEDER Gábor wrote:\n> \n> > On Wed, Jun 23, 2021 at 07:48:11PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> >> Fix a bug that's been here since 7cc8f971085 (pack-objects: implement\n> >> bitmap writing, 2013-12-21), we did not call stop_progress() if we\n> >> reached the early exit in this function. This will matter in a\n> >> subsequent commit where we BUG(...) out if this happens, and matters\n> >> now e.g. because we don't have a corresponding \"region_end\" for the\n> >> progress trace2 event.\n> >> \n> >> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> >> ---\n> >>  pack-bitmap-write.c | 1 +\n> >>  1 file changed, 1 insertion(+)\n> >> \n> >> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> >> index 88d9e696a54..6e110e41ea4 100644\n> >> --- a/pack-bitmap-write.c\n> >> +++ b/pack-bitmap-write.c\n> >> @@ -550,6 +550,7 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n> >>  \tif (indexed_commits_nr < 100) {\n> >>  \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n> >>  \t\t\tpush_bitmapped_commit(indexed_commits[i]);\n> >> +\t\tstop_progress(&writer.progress);\n> >>  \t\treturn;\n> >>  \t}\n> >\n> > When I found this bug I fixed it differently: with your patch there\n> \n> Is that patch on-list somewhere or something you have locally?\n\nYes, it is between the scissors lines in the email you responded to :)\nIt was another fallout from my isatty(2) vs. progress PoC starting at\n\n  https://public-inbox.org/git/20210623215736.8279-1-szeder.dev@gmail.com/\n\nbut it was not included there, because I fixed it some days after\nsending those patches.\n\n> > are no display() calls at all between start_progress() and this new\n> > stop_progress(), indicating that a stop_progress() is not missing but\n> > rather the start_progress is in the wrong place:\n> \n> *Nod*, I'll see about fixing it differenty depending on the above / any\n> other comments.\n> \n> Note that while this comment is current to\n> https://lore.kernel.org/git/patch-7.8-eb63b4ba6a-20210722T125012Z-avarab@gmail.com/\n> as well, as noted in\n> https://lore.kernel.org/git/877dffg37n.fsf@evledraar.gmail.com/ you've\n> had several comments on the 25 patch series not currently queued in\n> \"seen\".\n\nOh, right, I commented on the wrong patch series.  Gaah.\n\n> Still very useful as I'd had some of it planned for after that 8-patch\n> series, just noting it for context.\n"},{"id":"436511","messageId":"cover-v2-0.8-00000000000-20210920T225701Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-0.8-0000000000-20210722T125012Z-avarab@gmail.com","subject":"[PATCH v2 0/8] progress: assert \"global_progress\" + test fixes / cleanup","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T23:09:21Z","receivedAt":"2021-09-21T02:21:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"These patches improve the progress.c tests, fix a couple of\nmiscellaneous nits in 5/8 and 6/8, and 7/8 and 8/8/ then fix what I\nbelieve is the last in a class of bug that 8/8 adds a new BUG() assert\nfor: We should not be starting two progress bars at the same time.\n\n\nThis series has been held off since September 1st on SZEDER's comment\nthat he \"found some issues with it\" in[1].\n\nHe's been relatively inactive on list recently, but I belive based on\na recent discussion in the thread-at-large that those comments didn't\nrefer to any unreported bug, but a general concern that we might be\ngetting things wrong and had cases where the BUG() might trigger that\nwe haven't thought of.\n\nI think my [2] and the updated commit message of 8/8 cover in detail\nwhy I think this is safe to do. SZEDER didn't reply to my [2] yet, so\nperhaps there really is some specific issue I'm not aware of (i.e. the\nBUG() being hit), but I don't think there is based on the information\nI have now.\n\nSZEDER also had comments rightly pointing out some issues[3] in the\nearlier 25-patch series I'd submitted[2]. Those will need to be\naddressed or fixed if I re-submit those, but they're not part of this\nseries.\n\n1. https://lore.kernel.org/git/20210901050406.GB76263@szeder.dev/\n2. https://lore.kernel.org/git/877dffg37n.fsf@evledraar.gmail.com/\n3. https://lore.kernel.org/git/20210916183711.GE76263@szeder.dev/\n\nÆvar Arnfjörð Bjarmason (8):\n  progress.c tests: make start/stop verbs on stdin\n  progress.c tests: test some invalid usage\n  progress.c: move signal handler functions lower\n  progress.c: call progress_interval() from progress_test_force_update()\n  progress.c: stop eagerly fflush(stderr) when not a terminal\n  progress.c: add temporary variable from progress struct\n  pack-bitmap-write.c: add a missing stop_progress()\n  progress.c: add & assert a \"global_progress\" variable\n\n pack-bitmap-write.c         |   1 +\n progress.c                  | 116 ++++++++++++++++++++----------------\n t/helper/test-progress.c    |  43 +++++++++----\n t/t0500-progress-display.sh | 103 +++++++++++++++++++++++++-------\n 4 files changed, 178 insertions(+), 85 deletions(-)\n\nRange-diff against v1:\n1:  e0a294eb479 = 1:  e0a294eb479 progress.c tests: make start/stop verbs on stdin\n2:  7b1220b641e = 2:  7b1220b641e progress.c tests: test some invalid usage\n3:  f1b8bf1dbde = 3:  f1b8bf1dbde progress.c: move signal handler functions lower\n4:  74057b0046a = 4:  74057b0046a progress.c: call progress_interval() from progress_test_force_update()\n5:  250e50667c2 = 5:  250e50667c2 progress.c: stop eagerly fflush(stderr) when not a terminal\n6:  d4e9ff1de73 = 6:  d4e9ff1de73 progress.c: add temporary variable from progress struct\n7:  a3f133ca7ad = 7:  a3f133ca7ad pack-bitmap-write.c: add a missing stop_progress()\n8:  4fd2754caeb ! 8:  1bd285eba0d progress.c: add & assert a \"global_progress\" variable\n    @@ Commit message\n         progress.c: add & assert a \"global_progress\" variable\n     \n         The progress.c code makes a hard assumption that only one progress bar\n    -    be active at a time (see [1] for a bug where this wasn't the case),\n    -    but nothing has asserted that that's the case. Let's add a BUG()\n    -    that'll trigger if two progress bars are active at the same time.\n    -\n    -    There's an alternate test-only approach to doing the same thing[2],\n    -    but by doing this for all progress bars we'll have a canary to check\n    -    if we have any unexpected interaction between the \"sig_atomic_t\n    -    progress_update\" variable and this global struct.\n    -\n    -    I am then planning on using this scaffolding in the future to fix a\n    -    limitation in the progress output, namely the current limitation of\n    -    the progress.c bar code that any update must pro-actively go through\n    -    the likes of display_progress().\n    -\n    -    If we e.g. hang forever before the first display_progress(), or in the\n    -    middle of a loop that would call display_progress() the user will only\n    -    see either no output, or output frozen at the last display_progress()\n    -    that would have done an update (e.g. in cases where progress_update\n    -    was \"1\" due to an earlier signal).\n    -\n    -    This change does not fix that, but sets up the structure for solving\n    -    that and other related problems by juggling this \"global_progress\"\n    -    struct. Later changes will make more use of the \"global_progress\" than\n    -    only using it for these assertions.\n    +    be active at a time (see [1] for a bug where this wasn't the\n    +    case). Add a BUG() that'll trigger if we ever regress on that promise\n    +    and have two progress bars active at the same time.\n    +\n    +    There was an alternative test-only approach to doing the same\n    +    thing[2], but by doing this outside of a GIT_TEST_* mode we'll know\n    +    we've put a hard stop to this particular API misuse.\n    +\n    +    It will also establish scaffolding to address current fundamental\n    +    limitations in the progress output: The current output must be\n    +    \"driven\" by calls to the likes of display_progress(). Once we have a\n    +    global current progress object we'll be able to update that object via\n    +    SIGALRM. See [3] for early code to do that.\n    +\n    +    It's conceivable that this change will hit the BUG() condition in some\n    +    scenario that we don't currently have tests for, this would be very\n    +    bad. If that happened we'd die just because we couldn't emit some\n    +    pretty output.\n    +\n    +    See [4] for a discussion of why our test coverage is lacking; our\n    +    progress display is hidden behind isatty(2) checks in many cases, so\n    +    the test suite doesn't cover it unless individual tests are run in\n    +    \"--verbose\" mode, we might also have multi-threaded use of the API, so\n    +    two progress bars stopping and starting would only be visible due to a\n    +    race condition.\n    +\n    +    Despite that, I think that this change won't introduce such\n    +    regressions, because:\n    +\n    +     1. I've read all the code using the progress API (and have modified a\n    +        large part of it in some WIP code I have). Almost all of it is really\n    +        simple, the parts that aren't[5] are complex in the display_progress() part,\n    +        not in starting or stopping the progress bar.\n    +\n    +     2. The entire test suite passes when instrumented with an ad-hoc\n    +        Linux-specific mode (it uses gettid()) to die if progress bars are\n    +        ever started or stopped on anything but the main thread[6].\n    +\n    +        Extending that to die if display_progress() is called in a thread\n    +        reveals that we have exactly two users of the progress bar under\n    +        threaded conditions, \"git index-pack\" and \"git pack-objects\". Both\n    +        uses are straightforward, and they don't start/stop the progress\n    +        bar when threads are active.\n    +\n    +     3. I've likewise done an ad-hoc test to force progress bars to be\n    +        displayed with:\n    +\n    +            perl -pi -e 's[isatty\\((?:STDERR_FILENO|2)\\)][1]g' $(git grep -l 'isatty\\((STDERR_FILENO|2)\\)')\n    +\n    +        I.e. to replace all checks (not just for progress) of checking\n    +        whether STDERR is connected to a TTY, and then monkeypatching\n    +        is_foreground_fd() in progress.c to always \"return 1\". Running the\n    +        tests with those applied, interactively and under -V reveals via:\n    +\n    +            $ grep -e set_progress_signal -e clear_progress_signal test-results/*out\n    +\n    +        That nothing our tests cover hits the BUG conditions added here,\n    +        except the expected \"BUG: start two concurrent progress bars\" test\n    +        being added here.\n    +\n    +        That isn't entirely true since we won't be getting 100% coverage\n    +        due to cascading failures from tests that expected no progress\n    +        output on stderr. To make sure I covered 100% I also tried making\n    +        the display() function in progress.c a NOOP on top of that (it's\n    +        the calls to start_progress_delay() and stop_progress()) that\n    +        matter.\n    +\n    +        That doesn't hit the BUG() either. Some tests fail in that mode\n    +        due to a combination of the overzealous isatty(2) munging noted\n    +        above, and the tests that are testing that the progress output\n    +        itself is present (but for testing I'd made display() a NOOP).\n    +\n    +    Between those three points I think it's safe to go ahead with this\n    +    change.\n     \n         1. 6f9d5f2fda1 (commit-graph: fix progress of reachable commits, 2020-07-09)\n         2. https://lore.kernel.org/git/20210620200303.2328957-3-szeder.dev@gmail.com\n    +    3. https://lore.kernel.org/git/patch-18.25-e21fc66623f-20210623T155626Z-avarab@gmail.com/\n    +    4. https://lore.kernel.org/git/cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com/\n    +    5. b50c37aa44d (Merge branch 'ab/progress-users-adjust-counters' into\n    +       next, 2021-09-10)\n    +    6. https://lore.kernel.org/git/877dffg37n.fsf@evledraar.gmail.com/\n     \n         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n     \n-- \n2.33.0.1098.gf02a64c1a2d\n\n"},{"id":"436513","messageId":"patch-v2-1.8-e0a294eb479-20210920T225701Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v2-0.8-00000000000-20210920T225701Z-avarab@gmail.com","subject":"[PATCH v2 1/8] progress.c tests: make start/stop verbs on stdin","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T23:09:22Z","receivedAt":"2021-09-21T02:21:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the usage of the \"test-tool progress\" introduced in\n2bb74b53a49 (Test the progress display, 2019-09-16) to take command\nlike \"start\" and \"stop\" on stdin, instead of running them implicitly.\n\nThis makes for tests that are easier to read, since the recipe will\nmirror the API usage, and allows for easily testing invalid usage that\nwould yield (or should yield) a BUG(), e.g. providing two \"start\"\ncalls in a row. A subsequent commit will add such stress tests.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/helper/test-progress.c    | 43 +++++++++++++++++++--------\n t/t0500-progress-display.sh | 59 +++++++++++++++++++++++--------------\n 2 files changed, 67 insertions(+), 35 deletions(-)\n\ndiff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\nindex 5d05cbe7894..685c0a7c49a 100644\n--- a/t/helper/test-progress.c\n+++ b/t/helper/test-progress.c\n@@ -3,6 +3,9 @@\n  *\n  * Reads instructions from standard input, one instruction per line:\n  *\n+ *   \"start[ <total>[ <title>]]\" - Call start_progress(title, total),\n+ *                                 when \"start\" use a title of\n+ *                                 \"Working hard\" with a total of 0.\n  *   \"progress <items>\" - Call display_progress() with the given item count\n  *                        as parameter.\n  *   \"throughput <bytes> <millis> - Call display_throughput() with the given\n@@ -10,6 +13,7 @@\n  *                                  specify the time elapsed since the\n  *                                  start_progress() call.\n  *   \"update\" - Set the 'progress_update' flag.\n+ *   \"stop\" - Call stop_progress().\n  *\n  * See 't0500-progress-display.sh' for examples.\n  */\n@@ -22,31 +26,41 @@\n \n int cmd__progress(int argc, const char **argv)\n {\n-\tint total = 0;\n-\tconst char *title;\n+\tconst char *default_title = \"Working hard\";\n+\tchar *detached_title = NULL;\n \tstruct strbuf line = STRBUF_INIT;\n-\tstruct progress *progress;\n+\tstruct progress *progress = NULL;\n \n \tconst char *usage[] = {\n-\t\t\"test-tool progress [--total=<n>] <progress-title>\",\n+\t\t\"test-tool progress <stdin\",\n \t\tNULL\n \t};\n \tstruct option options[] = {\n-\t\tOPT_INTEGER(0, \"total\", &total, \"total number of items\"),\n \t\tOPT_END(),\n \t};\n \n \targc = parse_options(argc, argv, NULL, options, usage, 0);\n-\tif (argc != 1)\n-\t\tdie(\"need a title for the progress output\");\n-\ttitle = argv[0];\n+\tif (argc)\n+\t\tusage_with_options(usage, options);\n \n \tprogress_testing = 1;\n-\tprogress = start_progress(title, total);\n \twhile (strbuf_getline(&line, stdin) != EOF) {\n \t\tchar *end;\n \n-\t\tif (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n+\t\tif (!strcmp(line.buf, \"start\")) {\n+\t\t\tprogress = start_progress(default_title, 0);\n+\t\t} else if (skip_prefix(line.buf, \"start \", (const char **) &end)) {\n+\t\t\tuint64_t total = strtoull(end, &end, 10);\n+\t\t\tif (*end == '\\0') {\n+\t\t\t\tprogress = start_progress(default_title, total);\n+\t\t\t} else if (*end == ' ') {\n+\t\t\t\tfree(detached_title);\n+\t\t\t\tdetached_title = strbuf_detach(&line, NULL);\n+\t\t\t\tprogress = start_progress(end + 1, total);\n+\t\t\t} else {\n+\t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n+\t\t\t}\n+\t\t} else if (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n \t\t\tuint64_t item_count = strtoull(end, &end, 10);\n \t\t\tif (*end != '\\0')\n \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n@@ -63,12 +77,15 @@ int cmd__progress(int argc, const char **argv)\n \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n \t\t\tprogress_test_ns = test_ms * 1000 * 1000;\n \t\t\tdisplay_throughput(progress, byte_count);\n-\t\t} else if (!strcmp(line.buf, \"update\"))\n+\t\t} else if (!strcmp(line.buf, \"update\")) {\n \t\t\tprogress_test_force_update();\n-\t\telse\n+\t\t} else if (!strcmp(line.buf, \"stop\")) {\n+\t\t\tstop_progress(&progress);\n+\t\t} else {\n \t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n+\t\t}\n \t}\n-\tstop_progress(&progress);\n+\tfree(detached_title);\n \n \treturn 0;\n }\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 22058b503ac..ca96ac1fa55 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -17,6 +17,7 @@ test_expect_success 'simple progress display' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tupdate\n \tprogress 1\n \tupdate\n@@ -25,8 +26,9 @@ test_expect_success 'simple progress display' '\n \tprogress 4\n \tupdate\n \tprogress 5\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -41,11 +43,13 @@ test_expect_success 'progress display with total' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 3\n \tprogress 1\n \tprogress 2\n \tprogress 3\n+\tstop\n \tEOF\n-\ttest-tool progress --total=3 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -62,14 +66,14 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tprogress 100\n \tprogress 1000\n \tprogress 10000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -88,16 +92,15 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n-\tupdate\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tprogress 1\n \tupdate\n \tprogress 2\n \tprogress 10000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -116,14 +119,14 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tprogress 25000\n \tprogress 50000\n \tprogress 75000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -140,14 +143,14 @@ Working hard.......2.........3.........4.........5.........6.........7.........:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6.........7.........\n \tprogress 25000\n \tprogress 50000\n \tprogress 75000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6.........7.........\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -164,12 +167,14 @@ test_expect_success 'progress shortens - crazy caller' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 1000\n \tprogress 100\n \tprogress 200\n \tprogress 1\n \tprogress 1000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=1000 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -185,6 +190,7 @@ test_expect_success 'progress display with throughput' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart\n \tthroughput 102400 1000\n \tupdate\n \tprogress 10\n@@ -197,8 +203,9 @@ test_expect_success 'progress display with throughput' '\n \tthroughput 409600 4000\n \tupdate\n \tprogress 40\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -214,6 +221,7 @@ test_expect_success 'progress display with throughput and total' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 40\n \tthroughput 102400 1000\n \tprogress 10\n \tthroughput 204800 2000\n@@ -222,8 +230,9 @@ test_expect_success 'progress display with throughput and total' '\n \tprogress 30\n \tthroughput 409600 4000\n \tprogress 40\n+\tstop\n \tEOF\n-\ttest-tool progress --total=40 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -239,6 +248,7 @@ test_expect_success 'cover up after throughput shortens' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart\n \tthroughput 409600 1000\n \tupdate\n \tprogress 1\n@@ -251,8 +261,9 @@ test_expect_success 'cover up after throughput shortens' '\n \tthroughput 1638400 4000\n \tupdate\n \tprogress 4\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -267,6 +278,7 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart\n \tthroughput 1 1000\n \tupdate\n \tprogress 1\n@@ -276,8 +288,9 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \tthroughput 3145728 3000\n \tupdate\n \tprogress 3\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -285,6 +298,7 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \n test_expect_success 'progress generates traces' '\n \tcat >in <<-\\EOF &&\n+\tstart 40\n \tthroughput 102400 1000\n \tupdate\n \tprogress 10\n@@ -297,10 +311,11 @@ test_expect_success 'progress generates traces' '\n \tthroughput 409600 4000\n \tupdate\n \tprogress 40\n+\tstop\n \tEOF\n \n-\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" test-tool progress --total=40 \\\n-\t\t\"Working hard\" <in 2>stderr &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n \n \t# t0212/parse_events.perl intentionally omits regions and data.\n \ttest_region progress \"Working hard\" trace.event &&\n-- \n2.33.0.1098.gf02a64c1a2d\n\n"},{"id":"436514","messageId":"patch-v2-2.8-7b1220b641e-20210920T225701Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v2-0.8-00000000000-20210920T225701Z-avarab@gmail.com","subject":"[PATCH v2 2/8] progress.c tests: test some invalid usage","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T23:09:23Z","receivedAt":"2021-09-21T02:21:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Test what happens when we \"stop\" without a \"start\", omit the \"stop\"\nafter a \"start\", or try to start two concurrent progress bars. This\nextends the trace2 tests added in 98a13647408 (trace2: log progress\ntime and throughput, 2020-05-12).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/t0500-progress-display.sh | 33 +++++++++++++++++++++++++++++++++\n 1 file changed, 33 insertions(+)\n\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex ca96ac1fa55..ffa819ca1db 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -323,4 +323,37 @@ test_expect_success 'progress generates traces' '\n \tgrep \"\\\"key\\\":\\\"total_bytes\\\",\\\"value\\\":\\\"409600\\\"\" trace.event\n '\n \n+test_expect_success 'progress generates traces: stop / start' '\n+\tcat >in <<-\\EOF &&\n+\tstart\n+\tstop\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-startstop.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n+\ttest_region progress \"Working hard\" trace-startstop.event\n+'\n+\n+test_expect_success 'progress generates traces: start without stop' '\n+\tcat >in <<-\\EOF &&\n+\tstart\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-start.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n+\tgrep region_enter.*progress trace-start.event &&\n+\t! grep region_leave.*progress trace-start.event\n+'\n+\n+test_expect_success 'progress generates traces: stop without start' '\n+\tcat >in <<-\\EOF &&\n+\tstop\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-stop.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n+\t! grep region_enter.*progress trace-stop.event &&\n+\t! grep region_leave.*progress trace-stop.event\n+'\n+\n test_done\n-- \n2.33.0.1098.gf02a64c1a2d\n\n"},{"id":"436515","messageId":"patch-v2-4.8-74057b0046a-20210920T225701Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v2-0.8-00000000000-20210920T225701Z-avarab@gmail.com","subject":"[PATCH v2 4/8] progress.c: call progress_interval() from progress_test_force_update()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T23:09:25Z","receivedAt":"2021-09-21T02:22:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Define the progress_test_force_update() function in terms of\nprogress_interval(). For documentation purposes these two functions\nhave the same body, but different names. Let's just define the test\nfunction by calling progress_interval() with SIGALRM ourselves.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/progress.c b/progress.c\nindex 893cb0fe56f..7fcc513717a 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -216,7 +216,7 @@ static void progress_interval(int signum)\n  */\n void progress_test_force_update(void)\n {\n-\tprogress_update = 1;\n+\tprogress_interval(SIGALRM);\n }\n \n static void set_progress_signal(void)\n-- \n2.33.0.1098.gf02a64c1a2d\n\n"},{"id":"436516","messageId":"patch-v2-3.8-f1b8bf1dbde-20210920T225701Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v2-0.8-00000000000-20210920T225701Z-avarab@gmail.com","subject":"[PATCH v2 3/8] progress.c: move signal handler functions lower","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T23:09:24Z","receivedAt":"2021-09-21T02:22:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the signal handler functions to just before the\nstart_progress_delay() where they'll be referenced, instead of having\nthem at the top of the file.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 92 ++++++++++++++++++++++++++++--------------------------\n 1 file changed, 48 insertions(+), 44 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 680c6a8bf93..893cb0fe56f 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -53,50 +53,6 @@ static volatile sig_atomic_t progress_update;\n  */\n int progress_testing;\n uint64_t progress_test_ns = 0;\n-void progress_test_force_update(void)\n-{\n-\tprogress_update = 1;\n-}\n-\n-\n-static void progress_interval(int signum)\n-{\n-\tprogress_update = 1;\n-}\n-\n-static void set_progress_signal(void)\n-{\n-\tstruct sigaction sa;\n-\tstruct itimerval v;\n-\n-\tif (progress_testing)\n-\t\treturn;\n-\n-\tprogress_update = 0;\n-\n-\tmemset(&sa, 0, sizeof(sa));\n-\tsa.sa_handler = progress_interval;\n-\tsigemptyset(&sa.sa_mask);\n-\tsa.sa_flags = SA_RESTART;\n-\tsigaction(SIGALRM, &sa, NULL);\n-\n-\tv.it_interval.tv_sec = 1;\n-\tv.it_interval.tv_usec = 0;\n-\tv.it_value = v.it_interval;\n-\tsetitimer(ITIMER_REAL, &v, NULL);\n-}\n-\n-static void clear_progress_signal(void)\n-{\n-\tstruct itimerval v = {{0,},};\n-\n-\tif (progress_testing)\n-\t\treturn;\n-\n-\tsetitimer(ITIMER_REAL, &v, NULL);\n-\tsignal(SIGALRM, SIG_IGN);\n-\tprogress_update = 0;\n-}\n \n static int is_foreground_fd(int fd)\n {\n@@ -249,6 +205,54 @@ void display_progress(struct progress *progress, uint64_t n)\n \t\tdisplay(progress, n, NULL);\n }\n \n+static void progress_interval(int signum)\n+{\n+\tprogress_update = 1;\n+}\n+\n+/*\n+ * The progress_test_force_update() function is intended for testing\n+ * the progress output, i.e. exclusively for 'test-tool progress'.\n+ */\n+void progress_test_force_update(void)\n+{\n+\tprogress_update = 1;\n+}\n+\n+static void set_progress_signal(void)\n+{\n+\tstruct sigaction sa;\n+\tstruct itimerval v;\n+\n+\tif (progress_testing)\n+\t\treturn;\n+\n+\tprogress_update = 0;\n+\n+\tmemset(&sa, 0, sizeof(sa));\n+\tsa.sa_handler = progress_interval;\n+\tsigemptyset(&sa.sa_mask);\n+\tsa.sa_flags = SA_RESTART;\n+\tsigaction(SIGALRM, &sa, NULL);\n+\n+\tv.it_interval.tv_sec = 1;\n+\tv.it_interval.tv_usec = 0;\n+\tv.it_value = v.it_interval;\n+\tsetitimer(ITIMER_REAL, &v, NULL);\n+}\n+\n+static void clear_progress_signal(void)\n+{\n+\tstruct itimerval v = {{0,},};\n+\n+\tif (progress_testing)\n+\t\treturn;\n+\n+\tsetitimer(ITIMER_REAL, &v, NULL);\n+\tsignal(SIGALRM, SIG_IGN);\n+\tprogress_update = 0;\n+}\n+\n static struct progress *start_progress_delay(const char *title, uint64_t total,\n \t\t\t\t\t     unsigned delay, unsigned sparse)\n {\n-- \n2.33.0.1098.gf02a64c1a2d\n\n"},{"id":"436517","messageId":"patch-v2-6.8-d4e9ff1de73-20210920T225701Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v2-0.8-00000000000-20210920T225701Z-avarab@gmail.com","subject":"[PATCH v2 6/8] progress.c: add temporary variable from progress struct","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T23:09:27Z","receivedAt":"2021-09-21T02:22:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a temporary \"progress\" variable for the dereferenced p_progress\npointer to a \"struct progress *\". Before 98a13647408 (trace2: log\nprogress time and throughput, 2020-05-12) we didn't dereference\n\"p_progress\" in this function, now that we do it's easier to read the\ncode if we work with a \"progress\" struct pointer like everywhere else,\ninstead of a pointer to a pointer.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 1fade5808de..1ab7d19deb8 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -331,15 +331,16 @@ void stop_progress(struct progress **p_progress)\n \tfinish_if_sparse(*p_progress);\n \n \tif (*p_progress) {\n+\t\tstruct progress *progress = *p_progress;\n \t\ttrace2_data_intmax(\"progress\", the_repository, \"total_objects\",\n \t\t\t\t   (*p_progress)->total);\n \n \t\tif ((*p_progress)->throughput)\n \t\t\ttrace2_data_intmax(\"progress\", the_repository,\n \t\t\t\t\t   \"total_bytes\",\n-\t\t\t\t\t   (*p_progress)->throughput->curr_total);\n+\t\t\t\t\t   progress->throughput->curr_total);\n \n-\t\ttrace2_region_leave(\"progress\", (*p_progress)->title, the_repository);\n+\t\ttrace2_region_leave(\"progress\", progress->title, the_repository);\n \t}\n \n \tstop_progress_msg(p_progress, _(\"done\"));\n-- \n2.33.0.1098.gf02a64c1a2d\n\n"},{"id":"436518","messageId":"patch-v2-7.8-a3f133ca7ad-20210920T225701Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v2-0.8-00000000000-20210920T225701Z-avarab@gmail.com","subject":"[PATCH v2 7/8] pack-bitmap-write.c: add a missing stop_progress()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T23:09:28Z","receivedAt":"2021-09-21T02:22:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a bug that's been here since 7cc8f971085 (pack-objects: implement\nbitmap writing, 2013-12-21), we did not call stop_progress() if we\nreached the early exit in this function. This will matter in a\nsubsequent commit where we BUG(...) out if this happens, and matters\nnow e.g. because we don't have a corresponding \"region_end\" for the\nprogress trace2 event.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n pack-bitmap-write.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 88d9e696a54..6e110e41ea4 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -550,6 +550,7 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \tif (indexed_commits_nr < 100) {\n \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n \t\t\tpush_bitmapped_commit(indexed_commits[i]);\n+\t\tstop_progress(&writer.progress);\n \t\treturn;\n \t}\n \n-- \n2.33.0.1098.gf02a64c1a2d\n\n"},{"id":"436519","messageId":"patch-v2-5.8-250e50667c2-20210920T225701Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v2-0.8-00000000000-20210920T225701Z-avarab@gmail.com","subject":"[PATCH v2 5/8] progress.c: stop eagerly fflush(stderr) when not a terminal","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T23:09:26Z","receivedAt":"2021-09-21T02:22:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"It's the clear intention of the combination of 137a0d0ef56 (Flush\nprogress message buffer in display()., 2007-11-19) and\n85cb8906f0e (progress: no progress in background, 2015-04-13) to call\nfflush(stderr) when we have a stderr in the foreground, but we ended\nup always calling fflush(stderr) seemingly by omission. Let's not.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 7fcc513717a..1fade5808de 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -91,7 +91,8 @@ static void display(struct progress *progress, uint64_t n, const char *done)\n \t}\n \n \tif (show_update) {\n-\t\tif (is_foreground_fd(fileno(stderr)) || done) {\n+\t\tint stderr_is_foreground_fd = is_foreground_fd(fileno(stderr));\n+\t\tif (stderr_is_foreground_fd || done) {\n \t\t\tconst char *eol = done ? done : \"\\r\";\n \t\t\tsize_t clear_len = counters_sb->len < last_count_len ?\n \t\t\t\t\tlast_count_len - counters_sb->len + 1 :\n@@ -115,7 +116,8 @@ static void display(struct progress *progress, uint64_t n, const char *done)\n \t\t\t\tfprintf(stderr, \"%s: %s%*s\", progress->title,\n \t\t\t\t\tcounters_sb->buf, (int) clear_len, eol);\n \t\t\t}\n-\t\t\tfflush(stderr);\n+\t\t\tif (stderr_is_foreground_fd)\n+\t\t\t\tfflush(stderr);\n \t\t}\n \t\tprogress_update = 0;\n \t}\n-- \n2.33.0.1098.gf02a64c1a2d\n\n"},{"id":"436522","messageId":"patch-v2-8.8-1bd285eba0d-20210920T225701Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v2-0.8-00000000000-20210920T225701Z-avarab@gmail.com","subject":"[PATCH v2 8/8] progress.c: add & assert a \"global_progress\" variable","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-20T23:09:29Z","receivedAt":"2021-09-21T02:22:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The progress.c code makes a hard assumption that only one progress bar\nbe active at a time (see [1] for a bug where this wasn't the\ncase). Add a BUG() that'll trigger if we ever regress on that promise\nand have two progress bars active at the same time.\n\nThere was an alternative test-only approach to doing the same\nthing[2], but by doing this outside of a GIT_TEST_* mode we'll know\nwe've put a hard stop to this particular API misuse.\n\nIt will also establish scaffolding to address current fundamental\nlimitations in the progress output: The current output must be\n\"driven\" by calls to the likes of display_progress(). Once we have a\nglobal current progress object we'll be able to update that object via\nSIGALRM. See [3] for early code to do that.\n\nIt's conceivable that this change will hit the BUG() condition in some\nscenario that we don't currently have tests for, this would be very\nbad. If that happened we'd die just because we couldn't emit some\npretty output.\n\nSee [4] for a discussion of why our test coverage is lacking; our\nprogress display is hidden behind isatty(2) checks in many cases, so\nthe test suite doesn't cover it unless individual tests are run in\n\"--verbose\" mode, we might also have multi-threaded use of the API, so\ntwo progress bars stopping and starting would only be visible due to a\nrace condition.\n\nDespite that, I think that this change won't introduce such\nregressions, because:\n\n 1. I've read all the code using the progress API (and have modified a\n    large part of it in some WIP code I have). Almost all of it is really\n    simple, the parts that aren't[5] are complex in the display_progress() part,\n    not in starting or stopping the progress bar.\n\n 2. The entire test suite passes when instrumented with an ad-hoc\n    Linux-specific mode (it uses gettid()) to die if progress bars are\n    ever started or stopped on anything but the main thread[6].\n\n    Extending that to die if display_progress() is called in a thread\n    reveals that we have exactly two users of the progress bar under\n    threaded conditions, \"git index-pack\" and \"git pack-objects\". Both\n    uses are straightforward, and they don't start/stop the progress\n    bar when threads are active.\n\n 3. I've likewise done an ad-hoc test to force progress bars to be\n    displayed with:\n\n        perl -pi -e 's[isatty\\((?:STDERR_FILENO|2)\\)][1]g' $(git grep -l 'isatty\\((STDERR_FILENO|2)\\)')\n\n    I.e. to replace all checks (not just for progress) of checking\n    whether STDERR is connected to a TTY, and then monkeypatching\n    is_foreground_fd() in progress.c to always \"return 1\". Running the\n    tests with those applied, interactively and under -V reveals via:\n\n        $ grep -e set_progress_signal -e clear_progress_signal test-results/*out\n\n    That nothing our tests cover hits the BUG conditions added here,\n    except the expected \"BUG: start two concurrent progress bars\" test\n    being added here.\n\n    That isn't entirely true since we won't be getting 100% coverage\n    due to cascading failures from tests that expected no progress\n    output on stderr. To make sure I covered 100% I also tried making\n    the display() function in progress.c a NOOP on top of that (it's\n    the calls to start_progress_delay() and stop_progress()) that\n    matter.\n\n    That doesn't hit the BUG() either. Some tests fail in that mode\n    due to a combination of the overzealous isatty(2) munging noted\n    above, and the tests that are testing that the progress output\n    itself is present (but for testing I'd made display() a NOOP).\n\nBetween those three points I think it's safe to go ahead with this\nchange.\n\n1. 6f9d5f2fda1 (commit-graph: fix progress of reachable commits, 2020-07-09)\n2. https://lore.kernel.org/git/20210620200303.2328957-3-szeder.dev@gmail.com\n3. https://lore.kernel.org/git/patch-18.25-e21fc66623f-20210623T155626Z-avarab@gmail.com/\n4. https://lore.kernel.org/git/cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com/\n5. b50c37aa44d (Merge branch 'ab/progress-users-adjust-counters' into\n   next, 2021-09-10)\n6. https://lore.kernel.org/git/877dffg37n.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c                  | 17 +++++++++++++----\n t/t0500-progress-display.sh | 11 +++++++++++\n 2 files changed, 24 insertions(+), 4 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 1ab7d19deb8..14a023f4b43 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -46,6 +46,7 @@ struct progress {\n };\n \n static volatile sig_atomic_t progress_update;\n+static struct progress *global_progress;\n \n /*\n  * These are only intended for testing the progress output, i.e. exclusively\n@@ -221,11 +222,15 @@ void progress_test_force_update(void)\n \tprogress_interval(SIGALRM);\n }\n \n-static void set_progress_signal(void)\n+static void set_progress_signal(struct progress *progress)\n {\n \tstruct sigaction sa;\n \tstruct itimerval v;\n \n+\tif (global_progress)\n+\t\tBUG(\"should have no global_progress in set_progress_signal()\");\n+\tglobal_progress = progress;\n+\n \tif (progress_testing)\n \t\treturn;\n \n@@ -243,10 +248,14 @@ static void set_progress_signal(void)\n \tsetitimer(ITIMER_REAL, &v, NULL);\n }\n \n-static void clear_progress_signal(void)\n+static void clear_progress_signal(struct progress *progress)\n {\n \tstruct itimerval v = {{0,},};\n \n+\tif (!global_progress)\n+\t\tBUG(\"should have a global_progress in clear_progress_signal()\");\n+\tglobal_progress = NULL;\n+\n \tif (progress_testing)\n \t\treturn;\n \n@@ -270,7 +279,7 @@ static struct progress *start_progress_delay(const char *title, uint64_t total,\n \tstrbuf_init(&progress->counters_sb, 0);\n \tprogress->title_len = utf8_strwidth(title);\n \tprogress->split = 0;\n-\tset_progress_signal();\n+\tset_progress_signal(progress);\n \ttrace2_region_enter(\"progress\", title, the_repository);\n \treturn progress;\n }\n@@ -374,7 +383,7 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)\n \t\tdisplay(progress, progress->last_value, buf);\n \t\tfree(buf);\n \t}\n-\tclear_progress_signal();\n+\tclear_progress_signal(progress);\n \tstrbuf_release(&progress->counters_sb);\n \tif (progress->throughput)\n \t\tstrbuf_release(&progress->throughput->display);\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex ffa819ca1db..124d33c96b3 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -296,6 +296,17 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \ttest_cmp expect out\n '\n \n+test_expect_success 'BUG: start two concurrent progress bars' '\n+\tcat >in <<-\\EOF &&\n+\tstart 0 one\n+\tstart 0 two\n+\tEOF\n+\n+\ttest_must_fail test-tool progress \\\n+\t\t<in 2>stderr &&\n+\tgrep -E \"^BUG: .*: should have no global_progress in set_progress_signal\\(\\)$\" stderr\n+'\n+\n test_expect_success 'progress generates traces' '\n \tcat >in <<-\\EOF &&\n \tstart 40\n-- \n2.33.0.1098.gf02a64c1a2d\n\n"},{"id":"438258","messageId":"YV++bhK2sscQt5bc@google.com","threadId":"55972","inReplyTo":"patch-v2-1.8-e0a294eb479-20210920T225701Z-avarab@gmail.com","subject":"Re: [PATCH v2 1/8] progress.c tests: make start/stop verbs on stdin","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-10-08T03:43:42Z","receivedAt":"2021-10-08T03:43:53Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Tue, Sep 21, 2021 at 01:09:22AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> Change the usage of the \"test-tool progress\" introduced in\n> 2bb74b53a49 (Test the progress display, 2019-09-16) to take command\n> like \"start\" and \"stop\" on stdin, instead of running them implicitly.\n> \n> This makes for tests that are easier to read, since the recipe will\n> mirror the API usage, and allows for easily testing invalid usage that\n> would yield (or should yield) a BUG(), e.g. providing two \"start\"\n> calls in a row. A subsequent commit will add such stress tests.\n\nOk. So this is just a readability change, and not a functional change to\nthe helper, for now.\n\nOr so I thought, but I was surprised to see the usage changing and the\ntotal count moving to stdin. I don't think that's a bad change but the\ncommit message doesn't mention it in a way that I expected to see it in\nthe diff.\n\n> diff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\n> index 5d05cbe7894..685c0a7c49a 100644\n> --- a/t/helper/test-progress.c\n> +++ b/t/helper/test-progress.c\n> @@ -22,31 +26,41 @@\n>  \n>  int cmd__progress(int argc, const char **argv)\n>  {\n> -\tint total = 0;\n> -\tconst char *title;\n> +\tconst char *default_title = \"Working hard\";\n> +\tchar *detached_title = NULL;\n>  \tstruct strbuf line = STRBUF_INIT;\n> -\tstruct progress *progress;\n> +\tstruct progress *progress = NULL;\n>  \n>  \tconst char *usage[] = {\n> -\t\t\"test-tool progress [--total=<n>] <progress-title>\",\n> +\t\t\"test-tool progress <stdin\",\n>  \t\tNULL\n>  \t};\n>  \tstruct option options[] = {\n> -\t\tOPT_INTEGER(0, \"total\", &total, \"total number of items\"),\n>  \t\tOPT_END(),\n>  \t};\n>  \n>  \targc = parse_options(argc, argv, NULL, options, usage, 0);\n> -\tif (argc != 1)\n> -\t\tdie(\"need a title for the progress output\");\n> -\ttitle = argv[0];\n> +\tif (argc)\n> +\t\tusage_with_options(usage, options);\n\nOk. We lose the args entirely, moving them to stdin lines, and that\ncleans up the usage() check. Nice.\n\n>  \tprogress_testing = 1;\n> -\tprogress = start_progress(title, total);\n\nGetting rid of the implied start. Ok.\n\n>  \twhile (strbuf_getline(&line, stdin) != EOF) {\n>  \t\tchar *end;\n>  \n> -\t\tif (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n> +\t\tif (!strcmp(line.buf, \"start\")) {\n> +\t\t\tprogress = start_progress(default_title, 0);\n'start' with no args...\n> +\t\t} else if (skip_prefix(line.buf, \"start \", (const char **) &end)) {\n'start 1234'...\n\nWould it be more readable to use strbuf_split_buf() here instead? Maybe\nit doesn't fix the need for strtoull() but it could make the parsing\nclearer. I did have to think about this one for a bit.\n\n> +\t\t\tuint64_t total = strtoull(end, &end, 10);\n> +\t\t\tif (*end == '\\0') {\n> +\t\t\t\tprogress = start_progress(default_title, total);\n> +\t\t\t} else if (*end == ' ') {\n'start 1234 lorem ipsum dolor'. Ok.\n> +\t\t\t\tfree(detached_title);\n> +\t\t\t\tdetached_title = strbuf_detach(&line, NULL);\n> +\t\t\t\tprogress = start_progress(end + 1, total);\n> +\t\t\t} else {\n> +\t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n> +\t\t\t}\n\nI wondered why we had to do all this title parsing from scratch now when\nwe didn't before, but I guess it's because we don't get a nicely\nallocated argv[0]. Ok.\n\n> +\t\t} else if (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n>  \t\t\tuint64_t item_count = strtoull(end, &end, 10);\n>  \t\t\tif (*end != '\\0')\n>  \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n> @@ -63,12 +77,15 @@ int cmd__progress(int argc, const char **argv)\n>  \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n>  \t\t\tprogress_test_ns = test_ms * 1000 * 1000;\n>  \t\t\tdisplay_throughput(progress, byte_count);\n> -\t\t} else if (!strcmp(line.buf, \"update\"))\n> +\t\t} else if (!strcmp(line.buf, \"update\")) {\n>  \t\t\tprogress_test_force_update();\n> -\t\telse\n> +\t\t} else if (!strcmp(line.buf, \"stop\")) {\n> +\t\t\tstop_progress(&progress);\n> +\t\t} else {\n\nAnd 'stop' doesn't take any args. Ok. Do you need the {}?\n\n>  \t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n> +\t\t}\n>  \t}\n> -\tstop_progress(&progress);\n> +\tfree(detached_title);\n>  \n>  \treturn 0;\n>  }\n> diff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\n> index 22058b503ac..ca96ac1fa55 100755\n> --- a/t/t0500-progress-display.sh\n> +++ b/t/t0500-progress-display.sh\n> @@ -17,6 +17,7 @@ test_expect_success 'simple progress display' '\n>  \tEOF\n>  \n>  \tcat >in <<-\\EOF &&\n> +\tstart 0\n\nDoes it need the total arg?\n\n>  \tupdate\n>  \tprogress 1\n>  \tupdate\n> @@ -88,16 +92,15 @@ Working hard.......2.........3.........4.........5.........6:\n>  EOF\n>  \n>  \tcat >in <<-\\EOF &&\n> -\tupdate\nWas it intended to drop the 'update' line here? Does this not change the\ncontent of the test?\n> +\tstart 100000 Working hard.......2.........3.........4.........5.........6\n>  \tprogress 1\n>  \tupdate\n>  \tprogress 2\n>  \tprogress 10000\n>  \tprogress 100000\n> +\tstop\n>  \tEOF\n> -\ttest-tool progress --total=100000 \\\n> -\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n> -\t\t<in 2>stderr &&\n> +\ttest-tool progress <in 2>stderr &&\n>  \n>  \tshow_cr <stderr >out &&\n>  \ttest_cmp expect out\n\nWith whichever nits seem appropriate,\n\nReviewed-by: Emily Shaffer <emilyshaffer@google.com>\n"},{"id":"438259","messageId":"YV/Ap9wg6pn00nA1@google.com","threadId":"55972","inReplyTo":"patch-v2-2.8-7b1220b641e-20210920T225701Z-avarab@gmail.com","subject":"Re: [PATCH v2 2/8] progress.c tests: test some invalid usage","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-10-08T03:53:11Z","receivedAt":"2021-10-08T03:53:19Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Tue, Sep 21, 2021 at 01:09:23AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> Test what happens when we \"stop\" without a \"start\", omit the \"stop\"\n> after a \"start\", or try to start two concurrent progress bars. This\n> extends the trace2 tests added in 98a13647408 (trace2: log progress\n> time and throughput, 2020-05-12).\n\nI wondered whether these tests were more testing the helper, rather than\ntesting the API, but I think this is a good change - you're correct that\nhaving the helper assume correct usage by automatically\nstart_progress()ing and stop_progress()ing was an oversight. Thanks.\n\nDiff is pretty straightforward.\n\nReviewed-by: Emily Shaffer <emilyshaffer@google.com>\n"},{"id":"438261","messageId":"YV/CNSdntvm1QetC@google.com","threadId":"55972","inReplyTo":"patch-v2-5.8-250e50667c2-20210920T225701Z-avarab@gmail.com","subject":"Re: [PATCH v2 5/8] progress.c: stop eagerly fflush(stderr) when not a terminal","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-10-08T03:59:49Z","receivedAt":"2021-10-08T03:59:56Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Tue, Sep 21, 2021 at 01:09:26AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> It's the clear intention of the combination of 137a0d0ef56 (Flush\n> progress message buffer in display()., 2007-11-19) and\n> 85cb8906f0e (progress: no progress in background, 2015-04-13) to call\n> fflush(stderr) when we have a stderr in the foreground, but we ended\n> up always calling fflush(stderr) seemingly by omission. Let's not.\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>  progress.c | 6 ++++--\n>  1 file changed, 4 insertions(+), 2 deletions(-)\n> \n> diff --git a/progress.c b/progress.c\n> index 7fcc513717a..1fade5808de 100644\n> --- a/progress.c\n> +++ b/progress.c\n> @@ -91,7 +91,8 @@ static void display(struct progress *progress, uint64_t n, const char *done)\n>  \t}\n>  \n>  \tif (show_update) {\n> -\t\tif (is_foreground_fd(fileno(stderr)) || done) {\n> +\t\tint stderr_is_foreground_fd = is_foreground_fd(fileno(stderr));\n> +\t\tif (stderr_is_foreground_fd || done) {\n>  \t\t\tconst char *eol = done ? done : \"\\r\";\n>  \t\t\tsize_t clear_len = counters_sb->len < last_count_len ?\n>  \t\t\t\t\tlast_count_len - counters_sb->len + 1 :\n> @@ -115,7 +116,8 @@ static void display(struct progress *progress, uint64_t n, const char *done)\n>  \t\t\t\tfprintf(stderr, \"%s: %s%*s\", progress->title,\n>  \t\t\t\t\tcounters_sb->buf, (int) clear_len, eol);\n>  \t\t\t}\n> -\t\t\tfflush(stderr);\n> +\t\t\tif (stderr_is_foreground_fd)\n> +\t\t\t\tfflush(stderr);\n\nLooks like a straightforward refactor, although I wonder what's the\ndifference between is_foreground_fd(fileno(stderr)) and isatty() in\npractice.\n\nReviewed-by: Emily Shaffer <emilyshaffer@google.com>\n\n>  \t\t}\n>  \t\tprogress_update = 0;\n>  \t}\n> -- \n> 2.33.0.1098.gf02a64c1a2d\n> \n"},{"id":"438262","messageId":"YV/C3SBBDWiZCAq1@google.com","threadId":"55972","inReplyTo":"patch-v2-6.8-d4e9ff1de73-20210920T225701Z-avarab@gmail.com","subject":"Re: [PATCH v2 6/8] progress.c: add temporary variable from progress struct","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-10-08T04:02:37Z","receivedAt":"2021-10-08T04:02:45Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Tue, Sep 21, 2021 at 01:09:27AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> Add a temporary \"progress\" variable for the dereferenced p_progress\n> pointer to a \"struct progress *\". Before 98a13647408 (trace2: log\n> progress time and throughput, 2020-05-12) we didn't dereference\n> \"p_progress\" in this function, now that we do it's easier to read the\n> code if we work with a \"progress\" struct pointer like everywhere else,\n> instead of a pointer to a pointer.\n\nThanks, this looks much nicer :)\n\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n\nReviewed-by: Emily Shaffer <emilyshaffer@google.com>\n"},{"id":"438263","messageId":"YV/DV95BkT85a2xl@google.com","threadId":"55972","inReplyTo":"patch-v2-7.8-a3f133ca7ad-20210920T225701Z-avarab@gmail.com","subject":"Re: [PATCH v2 7/8] pack-bitmap-write.c: add a missing stop_progress()","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-10-08T04:04:39Z","receivedAt":"2021-10-08T04:04:47Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Tue, Sep 21, 2021 at 01:09:28AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> Fix a bug that's been here since 7cc8f971085 (pack-objects: implement\n> bitmap writing, 2013-12-21), we did not call stop_progress() if we\n> reached the early exit in this function. This will matter in a\n> subsequent commit where we BUG(...) out if this happens, and matters\n> now e.g. because we don't have a corresponding \"region_end\" for the\n> progress trace2 event.\n\nSounds like this was the only place we were calling start_progress()\nwithout a stop_progress(), then? Or at least the only place that is\nexercised by the test suite. Wow. I'm proud of Git contributor base :)\n\nReviewed-by: Emily Shaffer <emilyshaffer@google.com>\n\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>  pack-bitmap-write.c | 1 +\n>  1 file changed, 1 insertion(+)\n> \n> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> index 88d9e696a54..6e110e41ea4 100644\n> --- a/pack-bitmap-write.c\n> +++ b/pack-bitmap-write.c\n> @@ -550,6 +550,7 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n>  \tif (indexed_commits_nr < 100) {\n>  \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n>  \t\t\tpush_bitmapped_commit(indexed_commits[i]);\n> +\t\tstop_progress(&writer.progress);\n>  \t\treturn;\n>  \t}\n>  \n> -- \n> 2.33.0.1098.gf02a64c1a2d\n> \n"},{"id":"438267","messageId":"YV/Gndf9e4MbFgYM@google.com","threadId":"55972","inReplyTo":"patch-v2-8.8-1bd285eba0d-20210920T225701Z-avarab@gmail.com","subject":"Re: [PATCH v2 8/8] progress.c: add & assert a \"global_progress\" variable","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-10-08T04:18:37Z","receivedAt":"2021-10-08T04:18:47Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Tue, Sep 21, 2021 at 01:09:29AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> The progress.c code makes a hard assumption that only one progress bar\n> be active at a time (see [1] for a bug where this wasn't the\n> case). Add a BUG() that'll trigger if we ever regress on that promise\n> and have two progress bars active at the same time.\n> \n> There was an alternative test-only approach to doing the same\n> thing[2], but by doing this outside of a GIT_TEST_* mode we'll know\n> we've put a hard stop to this particular API misuse.\n> \n> It will also establish scaffolding to address current fundamental\n> limitations in the progress output: The current output must be\n> \"driven\" by calls to the likes of display_progress(). Once we have a\n> global current progress object we'll be able to update that object via\n> SIGALRM. See [3] for early code to do that.\n> \n> It's conceivable that this change will hit the BUG() condition in some\n> scenario that we don't currently have tests for, this would be very\n> bad. If that happened we'd die just because we couldn't emit some\n> pretty output.\n> \n> See [4] for a discussion of why our test coverage is lacking; our\n> progress display is hidden behind isatty(2) checks in many cases, so\n> the test suite doesn't cover it unless individual tests are run in\n> \"--verbose\" mode, we might also have multi-threaded use of the API, so\n> two progress bars stopping and starting would only be visible due to a\n> race condition.\n> \n> Despite that, I think that this change won't introduce such\n> regressions, because:\n> \n>  1. I've read all the code using the progress API (and have modified a\n>     large part of it in some WIP code I have). Almost all of it is really\n>     simple, the parts that aren't[5] are complex in the display_progress() part,\n>     not in starting or stopping the progress bar.\n> \n>  2. The entire test suite passes when instrumented with an ad-hoc\n>     Linux-specific mode (it uses gettid()) to die if progress bars are\n>     ever started or stopped on anything but the main thread[6].\n> \n>     Extending that to die if display_progress() is called in a thread\n>     reveals that we have exactly two users of the progress bar under\n>     threaded conditions, \"git index-pack\" and \"git pack-objects\". Both\n>     uses are straightforward, and they don't start/stop the progress\n>     bar when threads are active.\n> \n>  3. I've likewise done an ad-hoc test to force progress bars to be\n>     displayed with:\n> \n>         perl -pi -e 's[isatty\\((?:STDERR_FILENO|2)\\)][1]g' $(git grep -l 'isatty\\((STDERR_FILENO|2)\\)')\n\nI think your ad-hoc test might be a little more compelling if it was\neasier to understand, which is to say, maybe if your Perl oneliner was\non more than one line, or had comments, or was in a different language.\nAlthough you explain it right after, we kind of have to take your word\nfor it.\n\n> \n>     I.e. to replace all checks (not just for progress) of checking\n>     whether STDERR is connected to a TTY, and then monkeypatching\n>     is_foreground_fd() in progress.c to always \"return 1\". Running the\n>     tests with those applied, interactively and under -V reveals via:\n> \n>         $ grep -e set_progress_signal -e clear_progress_signal test-results/*out\n> \n>     That nothing our tests cover hits the BUG conditions added here,\n>     except the expected \"BUG: start two concurrent progress bars\" test\n>     being added here.\n> \n>     That isn't entirely true since we won't be getting 100% coverage\n>     due to cascading failures from tests that expected no progress\n>     output on stderr. To make sure I covered 100% I also tried making\n>     the display() function in progress.c a NOOP on top of that (it's\n>     the calls to start_progress_delay() and stop_progress()) that\n>     matter.\n> \n>     That doesn't hit the BUG() either. Some tests fail in that mode\n>     due to a combination of the overzealous isatty(2) munging noted\n>     above, and the tests that are testing that the progress output\n>     itself is present (but for testing I'd made display() a NOOP).\n> \n> Between those three points I think it's safe to go ahead with this\n> change.\n\nOne worry I had was that we might be painting ourselves into a corner\nhere if we did want to support the ability to do multiple progress bars\nsimultaneously (for example if we want to pull from multiple CDNs at the\nsame time when we're using promisor packfiles, and we expect those packs\nto be large enough that we'd need to show a progress bar for each one).\nHowever, I think the pattern - hang onto a pointer to the progress\nobjects, and complain if we get a signal and there are any still valid -\nstill holds well enough, so I'm ok with this change.\n\nThere are a couple patches in the middle which I didn't reply to, but I\ndid read them, and they were so tiny and mechanical that I did not have\nuseful comments to add.\n\nThanks, it's nice to see progress here (ha ha ha).\n\nPreferably with the BUG() message nit below,\nReviewed-by: Emily Shaffer <emilyshaffer@google.com>\n\n> \n> 1. 6f9d5f2fda1 (commit-graph: fix progress of reachable commits, 2020-07-09)\n> 2. https://lore.kernel.org/git/20210620200303.2328957-3-szeder.dev@gmail.com\n> 3. https://lore.kernel.org/git/patch-18.25-e21fc66623f-20210623T155626Z-avarab@gmail.com/\n> 4. https://lore.kernel.org/git/cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com/\n> 5. b50c37aa44d (Merge branch 'ab/progress-users-adjust-counters' into\n>    next, 2021-09-10)\n> 6. https://lore.kernel.org/git/877dffg37n.fsf@evledraar.gmail.com/\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>  progress.c                  | 17 +++++++++++++----\n>  t/t0500-progress-display.sh | 11 +++++++++++\n>  2 files changed, 24 insertions(+), 4 deletions(-)\n> \n> diff --git a/progress.c b/progress.c\n> index 1ab7d19deb8..14a023f4b43 100644\n> --- a/progress.c\n> +++ b/progress.c\n> @@ -46,6 +46,7 @@ struct progress {\n>  };\n>  \n>  static volatile sig_atomic_t progress_update;\n> +static struct progress *global_progress;\n>  \n>  /*\n>   * These are only intended for testing the progress output, i.e. exclusively\n> @@ -221,11 +222,15 @@ void progress_test_force_update(void)\n>  \tprogress_interval(SIGALRM);\n>  }\n>  \n> -static void set_progress_signal(void)\n> +static void set_progress_signal(struct progress *progress)\n>  {\n>  \tstruct sigaction sa;\n>  \tstruct itimerval v;\n>  \n> +\tif (global_progress)\n> +\t\tBUG(\"should have no global_progress in set_progress_signal()\");\n> +\tglobal_progress = progress;\n\nCan we make the BUG() message a little clearer? Even in the context of\nthe code, it's not clear that what this BUG() really means is \"hey, you\nforgot to call stop_progress on something\" or \"hey, you can't have two\nprogress bars at the same time\". Even if you were to change the name of\n'global_progress' to 'existing_progress_bar' or something, I think that\nwould make the message more understandable.\n\n> +\n>  \tif (progress_testing)\n>  \t\treturn;\n>  \n> @@ -243,10 +248,14 @@ static void set_progress_signal(void)\n>  \tsetitimer(ITIMER_REAL, &v, NULL);\n>  }\n>  \n> -static void clear_progress_signal(void)\n> +static void clear_progress_signal(struct progress *progress)\n>  {\n>  \tstruct itimerval v = {{0,},};\n>  \n> +\tif (!global_progress)\n> +\t\tBUG(\"should have a global_progress in clear_progress_signal()\");\n> +\tglobal_progress = NULL;\n> +\n>  \tif (progress_testing)\n>  \t\treturn;\n>  \n> @@ -270,7 +279,7 @@ static struct progress *start_progress_delay(const char *title, uint64_t total,\n>  \tstrbuf_init(&progress->counters_sb, 0);\n>  \tprogress->title_len = utf8_strwidth(title);\n>  \tprogress->split = 0;\n> -\tset_progress_signal();\n> +\tset_progress_signal(progress);\n>  \ttrace2_region_enter(\"progress\", title, the_repository);\n>  \treturn progress;\n>  }\n> @@ -374,7 +383,7 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)\n>  \t\tdisplay(progress, progress->last_value, buf);\n>  \t\tfree(buf);\n>  \t}\n> -\tclear_progress_signal();\n> +\tclear_progress_signal(progress);\n>  \tstrbuf_release(&progress->counters_sb);\n>  \tif (progress->throughput)\n>  \t\tstrbuf_release(&progress->throughput->display);\n> diff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\n> index ffa819ca1db..124d33c96b3 100755\n> --- a/t/t0500-progress-display.sh\n> +++ b/t/t0500-progress-display.sh\n> @@ -296,6 +296,17 @@ test_expect_success 'cover up after throughput shortens a lot' '\n>  \ttest_cmp expect out\n>  '\n>  \n> +test_expect_success 'BUG: start two concurrent progress bars' '\n> +\tcat >in <<-\\EOF &&\n> +\tstart 0 one\n> +\tstart 0 two\n> +\tEOF\n> +\n> +\ttest_must_fail test-tool progress \\\n> +\t\t<in 2>stderr &&\n> +\tgrep -E \"^BUG: .*: should have no global_progress in set_progress_signal\\(\\)$\" stderr\n> +'\n> +\n>  test_expect_success 'progress generates traces' '\n>  \tcat >in <<-\\EOF &&\n>  \tstart 40\n> -- \n> 2.33.0.1098.gf02a64c1a2d\n> \n"},{"id":"438276","messageId":"8735pcgf8c.fsf@evledraar.gmail.com","threadId":"55972","inReplyTo":"YV/CNSdntvm1QetC@google.com","subject":"Re: [PATCH v2 5/8] progress.c: stop eagerly fflush(stderr) when not a terminal","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-08T07:01:43Z","receivedAt":"2021-10-08T07:14:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Oct 07 2021, Emily Shaffer wrote:\n\n> On Tue, Sep 21, 2021 at 01:09:26AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> It's the clear intention of the combination of 137a0d0ef56 (Flush\n>> progress message buffer in display()., 2007-11-19) and\n>> 85cb8906f0e (progress: no progress in background, 2015-04-13) to call\n>> fflush(stderr) when we have a stderr in the foreground, but we ended\n>> up always calling fflush(stderr) seemingly by omission. Let's not.\n>> \n>> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n>> ---\n>>  progress.c | 6 ++++--\n>>  1 file changed, 4 insertions(+), 2 deletions(-)\n>> \n>> diff --git a/progress.c b/progress.c\n>> index 7fcc513717a..1fade5808de 100644\n>> --- a/progress.c\n>> +++ b/progress.c\n>> @@ -91,7 +91,8 @@ static void display(struct progress *progress, uint64_t n, const char *done)\n>>  \t}\n>>  \n>>  \tif (show_update) {\n>> -\t\tif (is_foreground_fd(fileno(stderr)) || done) {\n>> +\t\tint stderr_is_foreground_fd = is_foreground_fd(fileno(stderr));\n>> +\t\tif (stderr_is_foreground_fd || done) {\n>>  \t\t\tconst char *eol = done ? done : \"\\r\";\n>>  \t\t\tsize_t clear_len = counters_sb->len < last_count_len ?\n>>  \t\t\t\t\tlast_count_len - counters_sb->len + 1 :\n>> @@ -115,7 +116,8 @@ static void display(struct progress *progress, uint64_t n, const char *done)\n>>  \t\t\t\tfprintf(stderr, \"%s: %s%*s\", progress->title,\n>>  \t\t\t\t\tcounters_sb->buf, (int) clear_len, eol);\n>>  \t\t\t}\n>> -\t\t\tfflush(stderr);\n>> +\t\t\tif (stderr_is_foreground_fd)\n>> +\t\t\t\tfflush(stderr);\n>\n> Looks like a straightforward refactor, although I wonder what's the\n> difference between is_foreground_fd(fileno(stderr)) and isatty() in\n> practice.\n\nGood question. Whether you have a TTY is different from if it's in the\nforeground. In this case we don't want progress bars to display their\nfull output if they're not in the foreground, just the summary line.\n\nI.e.:\n    \n    perl -MPOSIX=tcgetpgrp,isatty,getpgrp -wE '\n            say \"TTY: \", isatty(1) ? \"yes\" : \"no\";\n            open my $tty, \"/dev/tty\";\n            my $tpgrp = tcgetpgrp(fileno($tty));\n            my $pgrp = getpgrp();\n            say \"Foreground?: \",  $tpgrp == $pgrp ? \"yes\" : \"no\"\n    '\n    \nThen:\n    \n    $ <that>\n    TTY: yes\n    Foreground?: yes\n    $ <that> &\n    TTY: yes\n    Foreground?: no\n    $ <that> >f && cat f\n    TTY: no\n    Foreground?: yes\n    $ (<that> >f &); sleep 1; cat f;\n    TTY: no\n    Foreground?: no\n\nBut having written that I can see that this commit of mine is buggy,\nbecause when I wrote it I conflated the two. I.e. we don't want to defer\neager flushing in that \"&\" case. I.e. to have our line-buffered summary\nline be held up by I/O buffered flushing.\n\n> Reviewed-by: Emily Shaffer <emilyshaffer@google.com>\n>\n>>  \t\t}\n>>  \t\tprogress_update = 0;\n>>  \t}\n>> -- \n>> 2.33.0.1098.gf02a64c1a2d\n>> \n\n"},{"id":"438277","messageId":"87y274f0ck.fsf@evledraar.gmail.com","threadId":"55972","inReplyTo":"YV/Gndf9e4MbFgYM@google.com","subject":"Re: [PATCH v2 8/8] progress.c: add & assert a \"global_progress\" variable","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-08T07:15:19Z","receivedAt":"2021-10-08T07:21:35Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Oct 07 2021, Emily Shaffer wrote:\n\n> On Tue, Sep 21, 2021 at 01:09:29AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> The progress.c code makes a hard assumption that only one progress bar\n>> be active at a time (see [1] for a bug where this wasn't the\n>> case). Add a BUG() that'll trigger if we ever regress on that promise\n>> and have two progress bars active at the same time.\n>> \n>> There was an alternative test-only approach to doing the same\n>> thing[2], but by doing this outside of a GIT_TEST_* mode we'll know\n>> we've put a hard stop to this particular API misuse.\n>> \n>> It will also establish scaffolding to address current fundamental\n>> limitations in the progress output: The current output must be\n>> \"driven\" by calls to the likes of display_progress(). Once we have a\n>> global current progress object we'll be able to update that object via\n>> SIGALRM. See [3] for early code to do that.\n>> \n>> It's conceivable that this change will hit the BUG() condition in some\n>> scenario that we don't currently have tests for, this would be very\n>> bad. If that happened we'd die just because we couldn't emit some\n>> pretty output.\n>> \n>> See [4] for a discussion of why our test coverage is lacking; our\n>> progress display is hidden behind isatty(2) checks in many cases, so\n>> the test suite doesn't cover it unless individual tests are run in\n>> \"--verbose\" mode, we might also have multi-threaded use of the API, so\n>> two progress bars stopping and starting would only be visible due to a\n>> race condition.\n>> \n>> Despite that, I think that this change won't introduce such\n>> regressions, because:\n>> \n>>  1. I've read all the code using the progress API (and have modified a\n>>     large part of it in some WIP code I have). Almost all of it is really\n>>     simple, the parts that aren't[5] are complex in the display_progress() part,\n>>     not in starting or stopping the progress bar.\n>> \n>>  2. The entire test suite passes when instrumented with an ad-hoc\n>>     Linux-specific mode (it uses gettid()) to die if progress bars are\n>>     ever started or stopped on anything but the main thread[6].\n>> \n>>     Extending that to die if display_progress() is called in a thread\n>>     reveals that we have exactly two users of the progress bar under\n>>     threaded conditions, \"git index-pack\" and \"git pack-objects\". Both\n>>     uses are straightforward, and they don't start/stop the progress\n>>     bar when threads are active.\n>> \n>>  3. I've likewise done an ad-hoc test to force progress bars to be\n>>     displayed with:\n>> \n>>         perl -pi -e 's[isatty\\((?:STDERR_FILENO|2)\\)][1]g' $(git grep -l 'isatty\\((STDERR_FILENO|2)\\)')\n>\n> I think your ad-hoc test might be a little more compelling if it was\n> easier to understand, which is to say, maybe if your Perl oneliner was\n> on more than one line, or had comments, or was in a different language.\n> Although you explain it right after, we kind of have to take your word\n> for it.\n\nI'll see if I can use sed or something, which is easy enough in this\ncase. I just write Perl out of habit for this sort of thing\n(e.g. balanced braces & Perl-regexes make it much nicer).\n\n>> \n>>     I.e. to replace all checks (not just for progress) of checking\n>>     whether STDERR is connected to a TTY, and then monkeypatching\n>>     is_foreground_fd() in progress.c to always \"return 1\". Running the\n>>     tests with those applied, interactively and under -V reveals via:\n>> \n>>         $ grep -e set_progress_signal -e clear_progress_signal test-results/*out\n>> \n>>     That nothing our tests cover hits the BUG conditions added here,\n>>     except the expected \"BUG: start two concurrent progress bars\" test\n>>     being added here.\n>> \n>>     That isn't entirely true since we won't be getting 100% coverage\n>>     due to cascading failures from tests that expected no progress\n>>     output on stderr. To make sure I covered 100% I also tried making\n>>     the display() function in progress.c a NOOP on top of that (it's\n>>     the calls to start_progress_delay() and stop_progress()) that\n>>     matter.\n>> \n>>     That doesn't hit the BUG() either. Some tests fail in that mode\n>>     due to a combination of the overzealous isatty(2) munging noted\n>>     above, and the tests that are testing that the progress output\n>>     itself is present (but for testing I'd made display() a NOOP).\n>> \n>> Between those three points I think it's safe to go ahead with this\n>> change.\n>\n> One worry I had was that we might be painting ourselves into a corner\n> here if we did want to support the ability to do multiple progress bars\n> simultaneously (for example if we want to pull from multiple CDNs at the\n> same time when we're using promisor packfiles, and we expect those packs\n> to be large enough that we'd need to show a progress bar for each one).\n> However, I think the pattern - hang onto a pointer to the progress\n> objects, and complain if we get a signal and there are any still valid -\n> still holds well enough, so I'm ok with this change.\n\nYes, and that's a thing I'd really like the progress code to be able to\ndo too, and I've got some follow-up patches, but (somewhat\nparadoxically) in order to display multiple progress bars you need to\nfirst have a step like this to ensure that there is only ever one\nprogress bar.\n\nThe user only has one terminal, so we'll need to serialize our N\nprogress bars to one \"emitter\", we'll need to teach the progress\naccounting to either have N parallel progress lines, or to simply make N\nnumber of \"slave\" \"struct progress *\" hang off it. I'm leaning towards\nthe latter.\n\n> There are a couple patches in the middle which I didn't reply to, but I\n> did read them, and they were so tiny and mechanical that I did not have\n> useful comments to add.\n>\n> Thanks, it's nice to see progress here (ha ha ha).\n\n:)\n\n> Preferably with the BUG() message nit below,\n> Reviewed-by: Emily Shaffer <emilyshaffer@google.com>\n>\n>> \n>> 1. 6f9d5f2fda1 (commit-graph: fix progress of reachable commits, 2020-07-09)\n>> 2. https://lore.kernel.org/git/20210620200303.2328957-3-szeder.dev@gmail.com\n>> 3. https://lore.kernel.org/git/patch-18.25-e21fc66623f-20210623T155626Z-avarab@gmail.com/\n>> 4. https://lore.kernel.org/git/cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com/\n>> 5. b50c37aa44d (Merge branch 'ab/progress-users-adjust-counters' into\n>>    next, 2021-09-10)\n>> 6. https://lore.kernel.org/git/877dffg37n.fsf@evledraar.gmail.com/\n>> \n>> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n>> ---\n>>  progress.c                  | 17 +++++++++++++----\n>>  t/t0500-progress-display.sh | 11 +++++++++++\n>>  2 files changed, 24 insertions(+), 4 deletions(-)\n>> \n>> diff --git a/progress.c b/progress.c\n>> index 1ab7d19deb8..14a023f4b43 100644\n>> --- a/progress.c\n>> +++ b/progress.c\n>> @@ -46,6 +46,7 @@ struct progress {\n>>  };\n>>  \n>>  static volatile sig_atomic_t progress_update;\n>> +static struct progress *global_progress;\n>>  \n>>  /*\n>>   * These are only intended for testing the progress output, i.e. exclusively\n>> @@ -221,11 +222,15 @@ void progress_test_force_update(void)\n>>  \tprogress_interval(SIGALRM);\n>>  }\n>>  \n>> -static void set_progress_signal(void)\n>> +static void set_progress_signal(struct progress *progress)\n>>  {\n>>  \tstruct sigaction sa;\n>>  \tstruct itimerval v;\n>>  \n>> +\tif (global_progress)\n>> +\t\tBUG(\"should have no global_progress in set_progress_signal()\");\n>> +\tglobal_progress = progress;\n>\n> Can we make the BUG() message a little clearer? Even in the context of\n> the code, it's not clear that what this BUG() really means is \"hey, you\n> forgot to call stop_progress on something\" or \"hey, you can't have two\n> progress bars at the same time\". Even if you were to change the name of\n> 'global_progress' to 'existing_progress_bar' or something, I think that\n> would make the message more understandable.\n\nWilldo, thanks.\n\n>> +\n>>  \tif (progress_testing)\n>>  \t\treturn;\n>>  \n>> @@ -243,10 +248,14 @@ static void set_progress_signal(void)\n>>  \tsetitimer(ITIMER_REAL, &v, NULL);\n>>  }\n>>  \n>> -static void clear_progress_signal(void)\n>> +static void clear_progress_signal(struct progress *progress)\n>>  {\n>>  \tstruct itimerval v = {{0,},};\n>>  \n>> +\tif (!global_progress)\n>> +\t\tBUG(\"should have a global_progress in clear_progress_signal()\");\n>> +\tglobal_progress = NULL;\n>> +\n>>  \tif (progress_testing)\n>>  \t\treturn;\n>>  \n>> @@ -270,7 +279,7 @@ static struct progress *start_progress_delay(const char *title, uint64_t total,\n>>  \tstrbuf_init(&progress->counters_sb, 0);\n>>  \tprogress->title_len = utf8_strwidth(title);\n>>  \tprogress->split = 0;\n>> -\tset_progress_signal();\n>> +\tset_progress_signal(progress);\n>>  \ttrace2_region_enter(\"progress\", title, the_repository);\n>>  \treturn progress;\n>>  }\n>> @@ -374,7 +383,7 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)\n>>  \t\tdisplay(progress, progress->last_value, buf);\n>>  \t\tfree(buf);\n>>  \t}\n>> -\tclear_progress_signal();\n>> +\tclear_progress_signal(progress);\n>>  \tstrbuf_release(&progress->counters_sb);\n>>  \tif (progress->throughput)\n>>  \t\tstrbuf_release(&progress->throughput->display);\n>> diff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\n>> index ffa819ca1db..124d33c96b3 100755\n>> --- a/t/t0500-progress-display.sh\n>> +++ b/t/t0500-progress-display.sh\n>> @@ -296,6 +296,17 @@ test_expect_success 'cover up after throughput shortens a lot' '\n>>  \ttest_cmp expect out\n>>  '\n>>  \n>> +test_expect_success 'BUG: start two concurrent progress bars' '\n>> +\tcat >in <<-\\EOF &&\n>> +\tstart 0 one\n>> +\tstart 0 two\n>> +\tEOF\n>> +\n>> +\ttest_must_fail test-tool progress \\\n>> +\t\t<in 2>stderr &&\n>> +\tgrep -E \"^BUG: .*: should have no global_progress in set_progress_signal\\(\\)$\" stderr\n>> +'\n>> +\n>>  test_expect_success 'progress generates traces' '\n>>  \tcat >in <<-\\EOF &&\n>>  \tstart 40\n>> -- \n>> 2.33.0.1098.gf02a64c1a2d\n>> \n\n"},{"id":"438292","messageId":"87czofogq9.fsf@evledraar.gmail.com","threadId":"55972","inReplyTo":"YV/DV95BkT85a2xl@google.com","subject":"Re: [PATCH v2 7/8] pack-bitmap-write.c: add a missing stop_progress()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-08T12:14:43Z","receivedAt":"2021-10-08T12:15:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Oct 07 2021, Emily Shaffer wrote:\n\n> On Tue, Sep 21, 2021 at 01:09:28AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> Fix a bug that's been here since 7cc8f971085 (pack-objects: implement\n>> bitmap writing, 2013-12-21), we did not call stop_progress() if we\n>> reached the early exit in this function. This will matter in a\n>> subsequent commit where we BUG(...) out if this happens, and matters\n>> now e.g. because we don't have a corresponding \"region_end\" for the\n>> progress trace2 event.\n>\n> Sounds like this was the only place we were calling start_progress()\n> without a stop_progress(), then? Or at least the only place that is\n> exercised by the test suite. Wow. I'm proud of Git contributor base :)\n>\n> Reviewed-by: Emily Shaffer <emilyshaffer@google.com>\n\nYes! Will clarify that, there were some fixes to other bugs in the area,\nbut this one was the last one. Might squash it actually...\n\n>> \n>> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n>> ---\n>>  pack-bitmap-write.c | 1 +\n>>  1 file changed, 1 insertion(+)\n>> \n>> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n>> index 88d9e696a54..6e110e41ea4 100644\n>> --- a/pack-bitmap-write.c\n>> +++ b/pack-bitmap-write.c\n>> @@ -550,6 +550,7 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n>>  \tif (indexed_commits_nr < 100) {\n>>  \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n>>  \t\t\tpush_bitmapped_commit(indexed_commits[i]);\n>> +\t\tstop_progress(&writer.progress);\n>>  \t\treturn;\n>>  \t}\n>>  \n>> -- \n>> 2.33.0.1098.gf02a64c1a2d\n>> \n\n"},{"id":"438429","messageId":"20211010212949.GC571180@szeder.dev","threadId":"55972","inReplyTo":"patch-v2-7.8-a3f133ca7ad-20210920T225701Z-avarab@gmail.com","subject":"Re: [PATCH v2 7/8] pack-bitmap-write.c: add a missing stop_progress()","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-10-10T21:29:49Z","receivedAt":"2021-10-10T21:29:54Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Sep 21, 2021 at 01:09:28AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Fix a bug that's been here since 7cc8f971085 (pack-objects: implement\n> bitmap writing, 2013-12-21), we did not call stop_progress() if we\n> reached the early exit in this function. This will matter in a\n> subsequent commit where we BUG(...) out if this happens, and matters\n> now e.g. because we don't have a corresponding \"region_end\" for the\n> progress trace2 event.\n\nThe stop_progress() is not missing, but rather the start_progress() is\nin the wrong place.\n\n  https://public-inbox.org/git/20210917051448.GB2118053@szeder.dev/\n\n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>  pack-bitmap-write.c | 1 +\n>  1 file changed, 1 insertion(+)\n> \n> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> index 88d9e696a54..6e110e41ea4 100644\n> --- a/pack-bitmap-write.c\n> +++ b/pack-bitmap-write.c\n> @@ -550,6 +550,7 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n>  \tif (indexed_commits_nr < 100) {\n>  \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n>  \t\t\tpush_bitmapped_commit(indexed_commits[i]);\n> +\t\tstop_progress(&writer.progress);\n>  \t\treturn;\n>  \t}\n>  \n> -- \n> 2.33.0.1098.gf02a64c1a2d\n> \n"},{"id":"438702","messageId":"cover-v3-00.10-00000000000-20211013T222329Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v2-0.8-00000000000-20210920T225701Z-avarab@gmail.com","subject":"[PATCH v3 00/10] progress: assert \"global_progress\" + test fixes / cleanup","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-13T22:28:16Z","receivedAt":"2021-10-13T22:28:33Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This series fixes various issues in and related to progress.c, and\nadds a BUG() assertion for us not starting two progress bars at the\nsame time. Those changes are needed for subsequent changes that do\nmore interesting things with this new global progress bar.\n\nThis v3 hopefully addresses all the feedback on the v2, thanks\nall. Changes:\n\n * Fix a memory leak in 1/10, and make the progress tests use the\n   SANITIZE=leak test mode.\n\n * Simplified some of the test-progress.c code (no more \"start\"\n   handling, the \"total\" count is mandatory now.\n\n * Split out a formatting change into 2/10 to make 3/10 easier to\n   read.\n\n * A new 9/10 makes an ad-hoc test recipie in 10/10 easier to explain\n   (in response to Emily's comment).\n\n * The BUG() assertion in 10/10 now has a much better message, we dump\n   the title of the two progress bars in play if we have a bug where\n   we started two at the same time.\n\n\nÆvar Arnfjörð Bjarmason (10):\n  leak tests: fix a memory leaks in \"test-progress\" helper\n  progress.c test helper: add missing braces\n  progress.c tests: make start/stop verbs on stdin\n  progress.c tests: test some invalid usage\n  progress.c: move signal handler functions lower\n  progress.c: call progress_interval() from progress_test_force_update()\n  progress.c: add temporary variable from progress struct\n  pack-bitmap-write.c: don't return without stop_progress()\n  various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO)\n  progress.c: add & assert a \"global_progress\" variable\n\n builtin/bisect--helper.c    |   2 +-\n builtin/bundle.c            |   2 +-\n compat/mingw.c              |   2 +-\n pack-bitmap-write.c         |   6 +-\n progress.c                  | 111 ++++++++++++++++++++----------------\n t/helper/test-progress.c    |  43 +++++++++-----\n t/t0500-progress-display.sh | 105 +++++++++++++++++++++++++++-------\n 7 files changed, 183 insertions(+), 88 deletions(-)\n\nRange-diff against v2:\n -:  ----------- >  1:  40f7c438a1e leak tests: fix a memory leaks in \"test-progress\" helper\n -:  ----------- >  2:  ee177d253a8 progress.c test helper: add missing braces\n 1:  e0a294eb479 !  3:  045d58d8201 progress.c tests: make start/stop verbs on stdin\n    @@ Commit message\n         This makes for tests that are easier to read, since the recipe will\n         mirror the API usage, and allows for easily testing invalid usage that\n         would yield (or should yield) a BUG(), e.g. providing two \"start\"\n    -    calls in a row. A subsequent commit will add such stress tests.\n    +    calls in a row. A subsequent commit will add such tests.\n     \n         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n     \n    @@ t/helper/test-progress.c\n       *\n       * Reads instructions from standard input, one instruction per line:\n       *\n    -+ *   \"start[ <total>[ <title>]]\" - Call start_progress(title, total),\n    -+ *                                 when \"start\" use a title of\n    -+ *                                 \"Working hard\" with a total of 0.\n    ++ *   \"start <total>[ <title>]\" - Call start_progress(title, total),\n    ++ *                               Uses the default title of \"Working hard\"\n    ++ *                               if the \" <title>\" is omitted.\n       *   \"progress <items>\" - Call display_progress() with the given item count\n       *                        as parameter.\n       *   \"throughput <bytes> <millis> - Call display_throughput() with the given\n    @@ t/helper/test-progress.c\n       * See 't0500-progress-display.sh' for examples.\n       */\n     @@\n    + #include \"parse-options.h\"\n    + #include \"progress.h\"\n    + #include \"strbuf.h\"\n    ++#include \"string-list.h\"\n      \n      int cmd__progress(int argc, const char **argv)\n      {\n     -\tint total = 0;\n     -\tconst char *title;\n    -+\tconst char *default_title = \"Working hard\";\n    -+\tchar *detached_title = NULL;\n    ++\tconst char *const default_title = \"Working hard\";\n    ++\tstruct string_list list = STRING_LIST_INIT_DUP;\n    ++\tconst struct string_list_item *item;\n      \tstruct strbuf line = STRBUF_INIT;\n     -\tstruct progress *progress;\n     +\tstruct progress *progress = NULL;\n    @@ t/helper/test-progress.c\n      \t\tchar *end;\n      \n     -\t\tif (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n    -+\t\tif (!strcmp(line.buf, \"start\")) {\n    -+\t\t\tprogress = start_progress(default_title, 0);\n    -+\t\t} else if (skip_prefix(line.buf, \"start \", (const char **) &end)) {\n    ++\t\tif (skip_prefix(line.buf, \"start \", (const char **) &end)) {\n     +\t\t\tuint64_t total = strtoull(end, &end, 10);\n     +\t\t\tif (*end == '\\0') {\n     +\t\t\t\tprogress = start_progress(default_title, total);\n     +\t\t\t} else if (*end == ' ') {\n    -+\t\t\t\tfree(detached_title);\n    -+\t\t\t\tdetached_title = strbuf_detach(&line, NULL);\n    -+\t\t\t\tprogress = start_progress(end + 1, total);\n    ++\t\t\t\titem = string_list_insert(&list, end + 1);\n    ++\t\t\t\tprogress = start_progress(item->string, total);\n     +\t\t\t} else {\n     +\t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n     +\t\t\t}\n    @@ t/helper/test-progress.c\n      \t\t\tif (*end != '\\0')\n      \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n     @@ t/helper/test-progress.c: int cmd__progress(int argc, const char **argv)\n    - \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n    - \t\t\tprogress_test_ns = test_ms * 1000 * 1000;\n      \t\t\tdisplay_throughput(progress, byte_count);\n    --\t\t} else if (!strcmp(line.buf, \"update\"))\n    -+\t\t} else if (!strcmp(line.buf, \"update\")) {\n    + \t\t} else if (!strcmp(line.buf, \"update\")) {\n      \t\t\tprogress_test_force_update();\n    --\t\telse\n     +\t\t} else if (!strcmp(line.buf, \"stop\")) {\n     +\t\t\tstop_progress(&progress);\n    -+\t\t} else {\n    + \t\t} else {\n      \t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n    -+\t\t}\n    + \t\t}\n      \t}\n     -\tstop_progress(&progress);\n    -+\tfree(detached_title);\n    + \tstrbuf_release(&line);\n    ++\tstring_list_clear(&list, 0);\n      \n      \treturn 0;\n      }\n    @@ t/t0500-progress-display.sh: Working hard.......2.........3.........4.........5.\n      EOF\n      \n      \tcat >in <<-\\EOF &&\n    --\tupdate\n     +\tstart 100000 Working hard.......2.........3.........4.........5.........6\n    + \tupdate\n      \tprogress 1\n      \tupdate\n      \tprogress 2\n    @@ t/t0500-progress-display.sh: test_expect_success 'progress display with throughp\n      \tEOF\n      \n      \tcat >in <<-\\EOF &&\n    -+\tstart\n    ++\tstart 0\n      \tthroughput 102400 1000\n      \tupdate\n      \tprogress 10\n    @@ t/t0500-progress-display.sh: test_expect_success 'cover up after throughput shor\n      \tEOF\n      \n      \tcat >in <<-\\EOF &&\n    -+\tstart\n    ++\tstart 0\n      \tthroughput 409600 1000\n      \tupdate\n      \tprogress 1\n    @@ t/t0500-progress-display.sh: test_expect_success 'cover up after throughput shor\n      \tEOF\n      \n      \tcat >in <<-\\EOF &&\n    -+\tstart\n    ++\tstart 0\n      \tthroughput 1 1000\n      \tupdate\n      \tprogress 1\n 2:  7b1220b641e !  4:  efc0ec360cc progress.c tests: test some invalid usage\n    @@ Commit message\n         extends the trace2 tests added in 98a13647408 (trace2: log progress\n         time and throughput, 2020-05-12).\n     \n    +    These tests are not merely testing the helper, but invalid API usage\n    +    that can happen if the progress.c API is misused.\n    +\n    +    The \"without stop\" test will leak under SANITIZE=leak, since this\n    +    buggy use of the API will leak memory. But let's not skip it entirely,\n    +    or use the \"!SANITIZE_LEAK\" prerequisite check as we'd do with tests\n    +    that we're skipping due to leaks we haven't fixed yet. Instead\n    +    annotate the specific command that should skip leak checking with\n    +    custom $LSAN_OPTIONS[1].\n    +\n    +    1. https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer\n    +\n         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n     \n      ## t/t0500-progress-display.sh ##\n    @@ t/t0500-progress-display.sh: test_expect_success 'progress generates traces' '\n      \n     +test_expect_success 'progress generates traces: stop / start' '\n     +\tcat >in <<-\\EOF &&\n    -+\tstart\n    ++\tstart 0\n     +\tstop\n     +\tEOF\n     +\n    @@ t/t0500-progress-display.sh: test_expect_success 'progress generates traces' '\n     +\n     +test_expect_success 'progress generates traces: start without stop' '\n     +\tcat >in <<-\\EOF &&\n    -+\tstart\n    ++\tstart 0\n     +\tEOF\n     +\n    -+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-start.event\" test-tool progress \\\n    ++\tGIT_TRACE2_EVENT=\"$(pwd)/trace-start.event\" \\\n    ++\tLSAN_OPTIONS=detect_leaks=0 \\\n    ++\ttest-tool progress \\\n     +\t\t<in 2>stderr &&\n     +\tgrep region_enter.*progress trace-start.event &&\n     +\t! grep region_leave.*progress trace-start.event\n 3:  f1b8bf1dbde =  5:  9e36f03de46 progress.c: move signal handler functions lower\n 4:  74057b0046a =  6:  c7c3843564e progress.c: call progress_interval() from progress_test_force_update()\n 5:  250e50667c2 <  -:  ----------- progress.c: stop eagerly fflush(stderr) when not a terminal\n 6:  d4e9ff1de73 =  7:  cd2d27b1626 progress.c: add temporary variable from progress struct\n 7:  a3f133ca7ad !  8:  e0a3510dd88 pack-bitmap-write.c: add a missing stop_progress()\n    @@ Metadata\n     Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n     \n      ## Commit message ##\n    -    pack-bitmap-write.c: add a missing stop_progress()\n    +    pack-bitmap-write.c: don't return without stop_progress()\n     \n         Fix a bug that's been here since 7cc8f971085 (pack-objects: implement\n         bitmap writing, 2013-12-21), we did not call stop_progress() if we\n    -    reached the early exit in this function. This will matter in a\n    -    subsequent commit where we BUG(...) out if this happens, and matters\n    -    now e.g. because we don't have a corresponding \"region_end\" for the\n    -    progress trace2 event.\n    +    reached the early exit in this function.\n     \n    +    We could call stop_progress() before we return, but better yet is to\n    +    defer calling start_progress() until we need it.\n    +\n    +    This will matter in a subsequent commit where we BUG(...) out if this\n    +    happens, and matters now e.g. because we don't have a corresponding\n    +    \"region_end\" for the progress trace2 event.\n    +\n    +    Suggested-by: SZEDER Gábor <szeder.dev@gmail.com>\n         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n     \n      ## pack-bitmap-write.c ##\n     @@ pack-bitmap-write.c: void bitmap_writer_select_commits(struct commit **indexed_commits,\n    + \n    + \tQSORT(indexed_commits, indexed_commits_nr, date_compare);\n    + \n    +-\tif (writer.show_progress)\n    +-\t\twriter.progress = start_progress(\"Selecting bitmap commits\", 0);\n    +-\n      \tif (indexed_commits_nr < 100) {\n      \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n      \t\t\tpush_bitmapped_commit(indexed_commits[i]);\n    -+\t\tstop_progress(&writer.progress);\n      \t\treturn;\n      \t}\n      \n    ++\tif (writer.show_progress)\n    ++\t\twriter.progress = start_progress(\"Selecting bitmap commits\", 0);\n    ++\n    + \tfor (;;) {\n    + \t\tstruct commit *chosen = NULL;\n    + \n -:  ----------- >  9:  2cf14881ecf various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO)\n 8:  1bd285eba0d ! 10:  01d5bbfce76 progress.c: add & assert a \"global_progress\" variable\n    @@ Commit message\n          3. I've likewise done an ad-hoc test to force progress bars to be\n             displayed with:\n     \n    -            perl -pi -e 's[isatty\\((?:STDERR_FILENO|2)\\)][1]g' $(git grep -l 'isatty\\((STDERR_FILENO|2)\\)')\n    +            perl -pi -e 's[isatty\\(2\\)][1]g' $(git grep -l -F 'isatty(2)')\n     \n             I.e. to replace all checks (not just for progress) of checking\n             whether STDERR is connected to a TTY, and then monkeypatching\n    @@ progress.c: void progress_test_force_update(void)\n      \tstruct itimerval v;\n      \n     +\tif (global_progress)\n    -+\t\tBUG(\"should have no global_progress in set_progress_signal()\");\n    ++\t\tBUG(\"'%s' progress still active when trying to start '%s'\",\n    ++\t\t    global_progress->title, progress->title);\n     +\tglobal_progress = progress;\n     +\n      \tif (progress_testing)\n    @@ progress.c: static void set_progress_signal(void)\n      \tstruct itimerval v = {{0,},};\n      \n     +\tif (!global_progress)\n    -+\t\tBUG(\"should have a global_progress in clear_progress_signal()\");\n    ++\t\tBUG(\"should have active global_progress when cleaning up\");\n     +\tglobal_progress = NULL;\n     +\n      \tif (progress_testing)\n    @@ t/t0500-progress-display.sh: test_expect_success 'cover up after throughput shor\n     +\n     +\ttest_must_fail test-tool progress \\\n     +\t\t<in 2>stderr &&\n    -+\tgrep -E \"^BUG: .*: should have no global_progress in set_progress_signal\\(\\)$\" stderr\n    ++\tgrep \"^BUG: .*'\\''one'\\'' progress still active when trying to start '\\''two'\\''$\" stderr\n     +'\n     +\n      test_expect_success 'progress generates traces' '\n-- \n2.33.1.1346.g48288c3c089\n\n"},{"id":"438703","messageId":"patch-v3-01.10-40f7c438a1e-20211013T222329Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v3-00.10-00000000000-20211013T222329Z-avarab@gmail.com","subject":"[PATCH v3 01/10] leak tests: fix a memory leaks in \"test-progress\" helper","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-13T22:28:17Z","receivedAt":"2021-10-13T22:28:34Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a memory leak in the test-progress helper, and mark the\ncorresponding \"t0500-progress-display.sh\" test as being leak-free\nunder SANITIZE=leak. This fixes a leak added in 2bb74b53a4 (Test the\nprogress display, 2019-09-16).\n\nMy 48f68715b14 (tr2: stop leaking \"thread_name\" memory, 2021-08-27)\nhad fixed another memory leak in this test (as it did some trace2\ntesting).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/helper/test-progress.c    | 1 +\n t/t0500-progress-display.sh | 1 +\n 2 files changed, 2 insertions(+)\n\ndiff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\nindex 5d05cbe7894..9265e6ab7cf 100644\n--- a/t/helper/test-progress.c\n+++ b/t/helper/test-progress.c\n@@ -69,6 +69,7 @@ int cmd__progress(int argc, const char **argv)\n \t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n \t}\n \tstop_progress(&progress);\n+\tstrbuf_release(&line);\n \n \treturn 0;\n }\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 22058b503ac..f37cf2eb9c9 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -2,6 +2,7 @@\n \n test_description='progress display'\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n show_cr () {\n-- \n2.33.1.1346.g48288c3c089\n\n"},{"id":"438704","messageId":"patch-v3-02.10-ee177d253a8-20211013T222329Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v3-00.10-00000000000-20211013T222329Z-avarab@gmail.com","subject":"[PATCH v3 02/10] progress.c test helper: add missing braces","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-13T22:28:18Z","receivedAt":"2021-10-13T22:28:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"If we have braces on one arm of an if/else all of them should have it,\nper the CodingGuidelines's \"When there are multiple arms to a\nconditional[...]\" advice. This formatting change makes a subsequent\ncommit smaller.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/helper/test-progress.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\nindex 9265e6ab7cf..50fd3be3dad 100644\n--- a/t/helper/test-progress.c\n+++ b/t/helper/test-progress.c\n@@ -63,10 +63,11 @@ int cmd__progress(int argc, const char **argv)\n \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n \t\t\tprogress_test_ns = test_ms * 1000 * 1000;\n \t\t\tdisplay_throughput(progress, byte_count);\n-\t\t} else if (!strcmp(line.buf, \"update\"))\n+\t\t} else if (!strcmp(line.buf, \"update\")) {\n \t\t\tprogress_test_force_update();\n-\t\telse\n+\t\t} else {\n \t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n+\t\t}\n \t}\n \tstop_progress(&progress);\n \tstrbuf_release(&line);\n-- \n2.33.1.1346.g48288c3c089\n\n"},{"id":"438705","messageId":"patch-v3-03.10-045d58d8201-20211013T222329Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v3-00.10-00000000000-20211013T222329Z-avarab@gmail.com","subject":"[PATCH v3 03/10] progress.c tests: make start/stop verbs on stdin","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-13T22:28:19Z","receivedAt":"2021-10-13T22:28:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the usage of the \"test-tool progress\" introduced in\n2bb74b53a49 (Test the progress display, 2019-09-16) to take command\nlike \"start\" and \"stop\" on stdin, instead of running them implicitly.\n\nThis makes for tests that are easier to read, since the recipe will\nmirror the API usage, and allows for easily testing invalid usage that\nwould yield (or should yield) a BUG(), e.g. providing two \"start\"\ncalls in a row. A subsequent commit will add such tests.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/helper/test-progress.c    | 37 ++++++++++++++++-------\n t/t0500-progress-display.sh | 58 +++++++++++++++++++++++--------------\n 2 files changed, 63 insertions(+), 32 deletions(-)\n\ndiff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\nindex 50fd3be3dad..45ccbafa9da 100644\n--- a/t/helper/test-progress.c\n+++ b/t/helper/test-progress.c\n@@ -3,6 +3,9 @@\n  *\n  * Reads instructions from standard input, one instruction per line:\n  *\n+ *   \"start <total>[ <title>]\" - Call start_progress(title, total),\n+ *                               Uses the default title of \"Working hard\"\n+ *                               if the \" <title>\" is omitted.\n  *   \"progress <items>\" - Call display_progress() with the given item count\n  *                        as parameter.\n  *   \"throughput <bytes> <millis> - Call display_throughput() with the given\n@@ -10,6 +13,7 @@\n  *                                  specify the time elapsed since the\n  *                                  start_progress() call.\n  *   \"update\" - Set the 'progress_update' flag.\n+ *   \"stop\" - Call stop_progress().\n  *\n  * See 't0500-progress-display.sh' for examples.\n  */\n@@ -19,34 +23,43 @@\n #include \"parse-options.h\"\n #include \"progress.h\"\n #include \"strbuf.h\"\n+#include \"string-list.h\"\n \n int cmd__progress(int argc, const char **argv)\n {\n-\tint total = 0;\n-\tconst char *title;\n+\tconst char *const default_title = \"Working hard\";\n+\tstruct string_list list = STRING_LIST_INIT_DUP;\n+\tconst struct string_list_item *item;\n \tstruct strbuf line = STRBUF_INIT;\n-\tstruct progress *progress;\n+\tstruct progress *progress = NULL;\n \n \tconst char *usage[] = {\n-\t\t\"test-tool progress [--total=<n>] <progress-title>\",\n+\t\t\"test-tool progress <stdin\",\n \t\tNULL\n \t};\n \tstruct option options[] = {\n-\t\tOPT_INTEGER(0, \"total\", &total, \"total number of items\"),\n \t\tOPT_END(),\n \t};\n \n \targc = parse_options(argc, argv, NULL, options, usage, 0);\n-\tif (argc != 1)\n-\t\tdie(\"need a title for the progress output\");\n-\ttitle = argv[0];\n+\tif (argc)\n+\t\tusage_with_options(usage, options);\n \n \tprogress_testing = 1;\n-\tprogress = start_progress(title, total);\n \twhile (strbuf_getline(&line, stdin) != EOF) {\n \t\tchar *end;\n \n-\t\tif (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n+\t\tif (skip_prefix(line.buf, \"start \", (const char **) &end)) {\n+\t\t\tuint64_t total = strtoull(end, &end, 10);\n+\t\t\tif (*end == '\\0') {\n+\t\t\t\tprogress = start_progress(default_title, total);\n+\t\t\t} else if (*end == ' ') {\n+\t\t\t\titem = string_list_insert(&list, end + 1);\n+\t\t\t\tprogress = start_progress(item->string, total);\n+\t\t\t} else {\n+\t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n+\t\t\t}\n+\t\t} else if (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n \t\t\tuint64_t item_count = strtoull(end, &end, 10);\n \t\t\tif (*end != '\\0')\n \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n@@ -65,12 +78,14 @@ int cmd__progress(int argc, const char **argv)\n \t\t\tdisplay_throughput(progress, byte_count);\n \t\t} else if (!strcmp(line.buf, \"update\")) {\n \t\t\tprogress_test_force_update();\n+\t\t} else if (!strcmp(line.buf, \"stop\")) {\n+\t\t\tstop_progress(&progress);\n \t\t} else {\n \t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n \t\t}\n \t}\n-\tstop_progress(&progress);\n \tstrbuf_release(&line);\n+\tstring_list_clear(&list, 0);\n \n \treturn 0;\n }\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex f37cf2eb9c9..27ab4218b01 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -18,6 +18,7 @@ test_expect_success 'simple progress display' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tupdate\n \tprogress 1\n \tupdate\n@@ -26,8 +27,9 @@ test_expect_success 'simple progress display' '\n \tprogress 4\n \tupdate\n \tprogress 5\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -42,11 +44,13 @@ test_expect_success 'progress display with total' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 3\n \tprogress 1\n \tprogress 2\n \tprogress 3\n+\tstop\n \tEOF\n-\ttest-tool progress --total=3 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -63,14 +67,14 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tprogress 100\n \tprogress 1000\n \tprogress 10000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -89,16 +93,16 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tupdate\n \tprogress 1\n \tupdate\n \tprogress 2\n \tprogress 10000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -117,14 +121,14 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tprogress 25000\n \tprogress 50000\n \tprogress 75000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -141,14 +145,14 @@ Working hard.......2.........3.........4.........5.........6.........7.........:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6.........7.........\n \tprogress 25000\n \tprogress 50000\n \tprogress 75000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6.........7.........\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -165,12 +169,14 @@ test_expect_success 'progress shortens - crazy caller' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 1000\n \tprogress 100\n \tprogress 200\n \tprogress 1\n \tprogress 1000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=1000 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -186,6 +192,7 @@ test_expect_success 'progress display with throughput' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tthroughput 102400 1000\n \tupdate\n \tprogress 10\n@@ -198,8 +205,9 @@ test_expect_success 'progress display with throughput' '\n \tthroughput 409600 4000\n \tupdate\n \tprogress 40\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -215,6 +223,7 @@ test_expect_success 'progress display with throughput and total' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 40\n \tthroughput 102400 1000\n \tprogress 10\n \tthroughput 204800 2000\n@@ -223,8 +232,9 @@ test_expect_success 'progress display with throughput and total' '\n \tprogress 30\n \tthroughput 409600 4000\n \tprogress 40\n+\tstop\n \tEOF\n-\ttest-tool progress --total=40 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -240,6 +250,7 @@ test_expect_success 'cover up after throughput shortens' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tthroughput 409600 1000\n \tupdate\n \tprogress 1\n@@ -252,8 +263,9 @@ test_expect_success 'cover up after throughput shortens' '\n \tthroughput 1638400 4000\n \tupdate\n \tprogress 4\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -268,6 +280,7 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tthroughput 1 1000\n \tupdate\n \tprogress 1\n@@ -277,8 +290,9 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \tthroughput 3145728 3000\n \tupdate\n \tprogress 3\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -286,6 +300,7 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \n test_expect_success 'progress generates traces' '\n \tcat >in <<-\\EOF &&\n+\tstart 40\n \tthroughput 102400 1000\n \tupdate\n \tprogress 10\n@@ -298,10 +313,11 @@ test_expect_success 'progress generates traces' '\n \tthroughput 409600 4000\n \tupdate\n \tprogress 40\n+\tstop\n \tEOF\n \n-\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" test-tool progress --total=40 \\\n-\t\t\"Working hard\" <in 2>stderr &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n \n \t# t0212/parse_events.perl intentionally omits regions and data.\n \ttest_region progress \"Working hard\" trace.event &&\n-- \n2.33.1.1346.g48288c3c089\n\n"},{"id":"438706","messageId":"patch-v3-04.10-efc0ec360cc-20211013T222329Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v3-00.10-00000000000-20211013T222329Z-avarab@gmail.com","subject":"[PATCH v3 04/10] progress.c tests: test some invalid usage","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-13T22:28:20Z","receivedAt":"2021-10-13T22:28:38Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Test what happens when we \"stop\" without a \"start\", omit the \"stop\"\nafter a \"start\", or try to start two concurrent progress bars. This\nextends the trace2 tests added in 98a13647408 (trace2: log progress\ntime and throughput, 2020-05-12).\n\nThese tests are not merely testing the helper, but invalid API usage\nthat can happen if the progress.c API is misused.\n\nThe \"without stop\" test will leak under SANITIZE=leak, since this\nbuggy use of the API will leak memory. But let's not skip it entirely,\nor use the \"!SANITIZE_LEAK\" prerequisite check as we'd do with tests\nthat we're skipping due to leaks we haven't fixed yet. Instead\nannotate the specific command that should skip leak checking with\ncustom $LSAN_OPTIONS[1].\n\n1. https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/t0500-progress-display.sh | 35 +++++++++++++++++++++++++++++++++++\n 1 file changed, 35 insertions(+)\n\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 27ab4218b01..59e9f226ea4 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -325,4 +325,39 @@ test_expect_success 'progress generates traces' '\n \tgrep \"\\\"key\\\":\\\"total_bytes\\\",\\\"value\\\":\\\"409600\\\"\" trace.event\n '\n \n+test_expect_success 'progress generates traces: stop / start' '\n+\tcat >in <<-\\EOF &&\n+\tstart 0\n+\tstop\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-startstop.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n+\ttest_region progress \"Working hard\" trace-startstop.event\n+'\n+\n+test_expect_success 'progress generates traces: start without stop' '\n+\tcat >in <<-\\EOF &&\n+\tstart 0\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-start.event\" \\\n+\tLSAN_OPTIONS=detect_leaks=0 \\\n+\ttest-tool progress \\\n+\t\t<in 2>stderr &&\n+\tgrep region_enter.*progress trace-start.event &&\n+\t! grep region_leave.*progress trace-start.event\n+'\n+\n+test_expect_success 'progress generates traces: stop without start' '\n+\tcat >in <<-\\EOF &&\n+\tstop\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-stop.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n+\t! grep region_enter.*progress trace-stop.event &&\n+\t! grep region_leave.*progress trace-stop.event\n+'\n+\n test_done\n-- \n2.33.1.1346.g48288c3c089\n\n"},{"id":"438707","messageId":"patch-v3-05.10-9e36f03de46-20211013T222329Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v3-00.10-00000000000-20211013T222329Z-avarab@gmail.com","subject":"[PATCH v3 05/10] progress.c: move signal handler functions lower","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-13T22:28:21Z","receivedAt":"2021-10-13T22:28:39Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the signal handler functions to just before the\nstart_progress_delay() where they'll be referenced, instead of having\nthem at the top of the file.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 92 ++++++++++++++++++++++++++++--------------------------\n 1 file changed, 48 insertions(+), 44 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 680c6a8bf93..893cb0fe56f 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -53,50 +53,6 @@ static volatile sig_atomic_t progress_update;\n  */\n int progress_testing;\n uint64_t progress_test_ns = 0;\n-void progress_test_force_update(void)\n-{\n-\tprogress_update = 1;\n-}\n-\n-\n-static void progress_interval(int signum)\n-{\n-\tprogress_update = 1;\n-}\n-\n-static void set_progress_signal(void)\n-{\n-\tstruct sigaction sa;\n-\tstruct itimerval v;\n-\n-\tif (progress_testing)\n-\t\treturn;\n-\n-\tprogress_update = 0;\n-\n-\tmemset(&sa, 0, sizeof(sa));\n-\tsa.sa_handler = progress_interval;\n-\tsigemptyset(&sa.sa_mask);\n-\tsa.sa_flags = SA_RESTART;\n-\tsigaction(SIGALRM, &sa, NULL);\n-\n-\tv.it_interval.tv_sec = 1;\n-\tv.it_interval.tv_usec = 0;\n-\tv.it_value = v.it_interval;\n-\tsetitimer(ITIMER_REAL, &v, NULL);\n-}\n-\n-static void clear_progress_signal(void)\n-{\n-\tstruct itimerval v = {{0,},};\n-\n-\tif (progress_testing)\n-\t\treturn;\n-\n-\tsetitimer(ITIMER_REAL, &v, NULL);\n-\tsignal(SIGALRM, SIG_IGN);\n-\tprogress_update = 0;\n-}\n \n static int is_foreground_fd(int fd)\n {\n@@ -249,6 +205,54 @@ void display_progress(struct progress *progress, uint64_t n)\n \t\tdisplay(progress, n, NULL);\n }\n \n+static void progress_interval(int signum)\n+{\n+\tprogress_update = 1;\n+}\n+\n+/*\n+ * The progress_test_force_update() function is intended for testing\n+ * the progress output, i.e. exclusively for 'test-tool progress'.\n+ */\n+void progress_test_force_update(void)\n+{\n+\tprogress_update = 1;\n+}\n+\n+static void set_progress_signal(void)\n+{\n+\tstruct sigaction sa;\n+\tstruct itimerval v;\n+\n+\tif (progress_testing)\n+\t\treturn;\n+\n+\tprogress_update = 0;\n+\n+\tmemset(&sa, 0, sizeof(sa));\n+\tsa.sa_handler = progress_interval;\n+\tsigemptyset(&sa.sa_mask);\n+\tsa.sa_flags = SA_RESTART;\n+\tsigaction(SIGALRM, &sa, NULL);\n+\n+\tv.it_interval.tv_sec = 1;\n+\tv.it_interval.tv_usec = 0;\n+\tv.it_value = v.it_interval;\n+\tsetitimer(ITIMER_REAL, &v, NULL);\n+}\n+\n+static void clear_progress_signal(void)\n+{\n+\tstruct itimerval v = {{0,},};\n+\n+\tif (progress_testing)\n+\t\treturn;\n+\n+\tsetitimer(ITIMER_REAL, &v, NULL);\n+\tsignal(SIGALRM, SIG_IGN);\n+\tprogress_update = 0;\n+}\n+\n static struct progress *start_progress_delay(const char *title, uint64_t total,\n \t\t\t\t\t     unsigned delay, unsigned sparse)\n {\n-- \n2.33.1.1346.g48288c3c089\n\n"},{"id":"438708","messageId":"patch-v3-06.10-c7c3843564e-20211013T222329Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v3-00.10-00000000000-20211013T222329Z-avarab@gmail.com","subject":"[PATCH v3 06/10] progress.c: call progress_interval() from progress_test_force_update()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-13T22:28:22Z","receivedAt":"2021-10-13T22:28:40Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Define the progress_test_force_update() function in terms of\nprogress_interval(). For documentation purposes these two functions\nhave the same body, but different names. Let's just define the test\nfunction by calling progress_interval() with SIGALRM ourselves.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/progress.c b/progress.c\nindex 893cb0fe56f..7fcc513717a 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -216,7 +216,7 @@ static void progress_interval(int signum)\n  */\n void progress_test_force_update(void)\n {\n-\tprogress_update = 1;\n+\tprogress_interval(SIGALRM);\n }\n \n static void set_progress_signal(void)\n-- \n2.33.1.1346.g48288c3c089\n\n"},{"id":"438709","messageId":"patch-v3-07.10-cd2d27b1626-20211013T222329Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v3-00.10-00000000000-20211013T222329Z-avarab@gmail.com","subject":"[PATCH v3 07/10] progress.c: add temporary variable from progress struct","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-13T22:28:23Z","receivedAt":"2021-10-13T22:28:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a temporary \"progress\" variable for the dereferenced p_progress\npointer to a \"struct progress *\". Before 98a13647408 (trace2: log\nprogress time and throughput, 2020-05-12) we didn't dereference\n\"p_progress\" in this function, now that we do it's easier to read the\ncode if we work with a \"progress\" struct pointer like everywhere else,\ninstead of a pointer to a pointer.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 7fcc513717a..b9369e9a264 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -329,15 +329,16 @@ void stop_progress(struct progress **p_progress)\n \tfinish_if_sparse(*p_progress);\n \n \tif (*p_progress) {\n+\t\tstruct progress *progress = *p_progress;\n \t\ttrace2_data_intmax(\"progress\", the_repository, \"total_objects\",\n \t\t\t\t   (*p_progress)->total);\n \n \t\tif ((*p_progress)->throughput)\n \t\t\ttrace2_data_intmax(\"progress\", the_repository,\n \t\t\t\t\t   \"total_bytes\",\n-\t\t\t\t\t   (*p_progress)->throughput->curr_total);\n+\t\t\t\t\t   progress->throughput->curr_total);\n \n-\t\ttrace2_region_leave(\"progress\", (*p_progress)->title, the_repository);\n+\t\ttrace2_region_leave(\"progress\", progress->title, the_repository);\n \t}\n \n \tstop_progress_msg(p_progress, _(\"done\"));\n-- \n2.33.1.1346.g48288c3c089\n\n"},{"id":"438710","messageId":"patch-v3-08.10-e0a3510dd88-20211013T222329Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v3-00.10-00000000000-20211013T222329Z-avarab@gmail.com","subject":"[PATCH v3 08/10] pack-bitmap-write.c: don't return without stop_progress()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-13T22:28:24Z","receivedAt":"2021-10-13T22:28:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a bug that's been here since 7cc8f971085 (pack-objects: implement\nbitmap writing, 2013-12-21), we did not call stop_progress() if we\nreached the early exit in this function.\n\nWe could call stop_progress() before we return, but better yet is to\ndefer calling start_progress() until we need it.\n\nThis will matter in a subsequent commit where we BUG(...) out if this\nhappens, and matters now e.g. because we don't have a corresponding\n\"region_end\" for the progress trace2 event.\n\nSuggested-by: SZEDER Gábor <szeder.dev@gmail.com>\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n pack-bitmap-write.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 9c55c1531e1..cab3eaa2acd 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -575,15 +575,15 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \n \tQSORT(indexed_commits, indexed_commits_nr, date_compare);\n \n-\tif (writer.show_progress)\n-\t\twriter.progress = start_progress(\"Selecting bitmap commits\", 0);\n-\n \tif (indexed_commits_nr < 100) {\n \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n \t\t\tpush_bitmapped_commit(indexed_commits[i]);\n \t\treturn;\n \t}\n \n+\tif (writer.show_progress)\n+\t\twriter.progress = start_progress(\"Selecting bitmap commits\", 0);\n+\n \tfor (;;) {\n \t\tstruct commit *chosen = NULL;\n \n-- \n2.33.1.1346.g48288c3c089\n\n"},{"id":"438711","messageId":"patch-v3-09.10-2cf14881ecf-20211013T222329Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v3-00.10-00000000000-20211013T222329Z-avarab@gmail.com","subject":"[PATCH v3 09/10] various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-13T22:28:25Z","receivedAt":"2021-10-13T22:28:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"We have over 50 uses of \"isatty(1)\" and \"isatty(2)\" in the codebase,\nonly these two used the stdlib.h macros to refer to them.\n\nLet's change these for consistency, and because a subsequent commit's\ncommit message outlines a recipe to change all of these for ad-hoc\ntesting, not needing to match these with that ad-hoc regex will make\nthings easier to explain.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n builtin/bisect--helper.c | 2 +-\n builtin/bundle.c         | 2 +-\n compat/mingw.c           | 2 +-\n 3 files changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/bisect--helper.c b/builtin/bisect--helper.c\nindex 28a2e6a5750..1727cb051fb 100644\n--- a/builtin/bisect--helper.c\n+++ b/builtin/bisect--helper.c\n@@ -830,7 +830,7 @@ static int bisect_autostart(struct bisect_terms *terms)\n \tfprintf_ln(stderr, _(\"You need to start by \\\"git bisect \"\n \t\t\t  \"start\\\"\\n\"));\n \n-\tif (!isatty(STDIN_FILENO))\n+\tif (!isatty(1))\n \t\treturn -1;\n \n \t/*\ndiff --git a/builtin/bundle.c b/builtin/bundle.c\nindex 5a85d7cd0fe..df69c651753 100644\n--- a/builtin/bundle.c\n+++ b/builtin/bundle.c\n@@ -56,7 +56,7 @@ static int parse_options_cmd_bundle(int argc,\n \n static int cmd_bundle_create(int argc, const char **argv, const char *prefix) {\n \tint all_progress_implied = 0;\n-\tint progress = isatty(STDERR_FILENO);\n+\tint progress = isatty(2);\n \tstruct strvec pack_opts;\n \tint version = -1;\n \tint ret;\ndiff --git a/compat/mingw.c b/compat/mingw.c\nindex 9e0cd1e097f..0f545c1a7d1 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw.c\n@@ -2374,7 +2374,7 @@ int mingw_raise(int sig)\n \tswitch (sig) {\n \tcase SIGALRM:\n \t\tif (timer_fn == SIG_DFL) {\n-\t\t\tif (isatty(STDERR_FILENO))\n+\t\t\tif (isatty(2))\n \t\t\t\tfputs(\"Alarm clock\\n\", stderr);\n \t\t\texit(128 + SIGALRM);\n \t\t} else if (timer_fn != SIG_IGN)\n-- \n2.33.1.1346.g48288c3c089\n\n"},{"id":"438712","messageId":"patch-v3-10.10-01d5bbfce76-20211013T222329Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v3-00.10-00000000000-20211013T222329Z-avarab@gmail.com","subject":"[PATCH v3 10/10] progress.c: add & assert a \"global_progress\" variable","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-13T22:28:26Z","receivedAt":"2021-10-13T22:28:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The progress.c code makes a hard assumption that only one progress bar\nbe active at a time (see [1] for a bug where this wasn't the\ncase). Add a BUG() that'll trigger if we ever regress on that promise\nand have two progress bars active at the same time.\n\nThere was an alternative test-only approach to doing the same\nthing[2], but by doing this outside of a GIT_TEST_* mode we'll know\nwe've put a hard stop to this particular API misuse.\n\nIt will also establish scaffolding to address current fundamental\nlimitations in the progress output: The current output must be\n\"driven\" by calls to the likes of display_progress(). Once we have a\nglobal current progress object we'll be able to update that object via\nSIGALRM. See [3] for early code to do that.\n\nIt's conceivable that this change will hit the BUG() condition in some\nscenario that we don't currently have tests for, this would be very\nbad. If that happened we'd die just because we couldn't emit some\npretty output.\n\nSee [4] for a discussion of why our test coverage is lacking; our\nprogress display is hidden behind isatty(2) checks in many cases, so\nthe test suite doesn't cover it unless individual tests are run in\n\"--verbose\" mode, we might also have multi-threaded use of the API, so\ntwo progress bars stopping and starting would only be visible due to a\nrace condition.\n\nDespite that, I think that this change won't introduce such\nregressions, because:\n\n 1. I've read all the code using the progress API (and have modified a\n    large part of it in some WIP code I have). Almost all of it is really\n    simple, the parts that aren't[5] are complex in the display_progress() part,\n    not in starting or stopping the progress bar.\n\n 2. The entire test suite passes when instrumented with an ad-hoc\n    Linux-specific mode (it uses gettid()) to die if progress bars are\n    ever started or stopped on anything but the main thread[6].\n\n    Extending that to die if display_progress() is called in a thread\n    reveals that we have exactly two users of the progress bar under\n    threaded conditions, \"git index-pack\" and \"git pack-objects\". Both\n    uses are straightforward, and they don't start/stop the progress\n    bar when threads are active.\n\n 3. I've likewise done an ad-hoc test to force progress bars to be\n    displayed with:\n\n        perl -pi -e 's[isatty\\(2\\)][1]g' $(git grep -l -F 'isatty(2)')\n\n    I.e. to replace all checks (not just for progress) of checking\n    whether STDERR is connected to a TTY, and then monkeypatching\n    is_foreground_fd() in progress.c to always \"return 1\". Running the\n    tests with those applied, interactively and under -V reveals via:\n\n        $ grep -e set_progress_signal -e clear_progress_signal test-results/*out\n\n    That nothing our tests cover hits the BUG conditions added here,\n    except the expected \"BUG: start two concurrent progress bars\" test\n    being added here.\n\n    That isn't entirely true since we won't be getting 100% coverage\n    due to cascading failures from tests that expected no progress\n    output on stderr. To make sure I covered 100% I also tried making\n    the display() function in progress.c a NOOP on top of that (it's\n    the calls to start_progress_delay() and stop_progress()) that\n    matter.\n\n    That doesn't hit the BUG() either. Some tests fail in that mode\n    due to a combination of the overzealous isatty(2) munging noted\n    above, and the tests that are testing that the progress output\n    itself is present (but for testing I'd made display() a NOOP).\n\nBetween those three points I think it's safe to go ahead with this\nchange.\n\n1. 6f9d5f2fda1 (commit-graph: fix progress of reachable commits, 2020-07-09)\n2. https://lore.kernel.org/git/20210620200303.2328957-3-szeder.dev@gmail.com\n3. https://lore.kernel.org/git/patch-18.25-e21fc66623f-20210623T155626Z-avarab@gmail.com/\n4. https://lore.kernel.org/git/cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com/\n5. b50c37aa44d (Merge branch 'ab/progress-users-adjust-counters' into\n   next, 2021-09-10)\n6. https://lore.kernel.org/git/877dffg37n.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c                  | 18 ++++++++++++++----\n t/t0500-progress-display.sh | 11 +++++++++++\n 2 files changed, 25 insertions(+), 4 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex b9369e9a264..a31500f8b2b 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -46,6 +46,7 @@ struct progress {\n };\n \n static volatile sig_atomic_t progress_update;\n+static struct progress *global_progress;\n \n /*\n  * These are only intended for testing the progress output, i.e. exclusively\n@@ -219,11 +220,16 @@ void progress_test_force_update(void)\n \tprogress_interval(SIGALRM);\n }\n \n-static void set_progress_signal(void)\n+static void set_progress_signal(struct progress *progress)\n {\n \tstruct sigaction sa;\n \tstruct itimerval v;\n \n+\tif (global_progress)\n+\t\tBUG(\"'%s' progress still active when trying to start '%s'\",\n+\t\t    global_progress->title, progress->title);\n+\tglobal_progress = progress;\n+\n \tif (progress_testing)\n \t\treturn;\n \n@@ -241,10 +247,14 @@ static void set_progress_signal(void)\n \tsetitimer(ITIMER_REAL, &v, NULL);\n }\n \n-static void clear_progress_signal(void)\n+static void clear_progress_signal(struct progress *progress)\n {\n \tstruct itimerval v = {{0,},};\n \n+\tif (!global_progress)\n+\t\tBUG(\"should have active global_progress when cleaning up\");\n+\tglobal_progress = NULL;\n+\n \tif (progress_testing)\n \t\treturn;\n \n@@ -268,7 +278,7 @@ static struct progress *start_progress_delay(const char *title, uint64_t total,\n \tstrbuf_init(&progress->counters_sb, 0);\n \tprogress->title_len = utf8_strwidth(title);\n \tprogress->split = 0;\n-\tset_progress_signal();\n+\tset_progress_signal(progress);\n \ttrace2_region_enter(\"progress\", title, the_repository);\n \treturn progress;\n }\n@@ -372,7 +382,7 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)\n \t\tdisplay(progress, progress->last_value, buf);\n \t\tfree(buf);\n \t}\n-\tclear_progress_signal();\n+\tclear_progress_signal(progress);\n \tstrbuf_release(&progress->counters_sb);\n \tif (progress->throughput)\n \t\tstrbuf_release(&progress->throughput->display);\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 59e9f226ea4..867fdace3f2 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -298,6 +298,17 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \ttest_cmp expect out\n '\n \n+test_expect_success 'BUG: start two concurrent progress bars' '\n+\tcat >in <<-\\EOF &&\n+\tstart 0 one\n+\tstart 0 two\n+\tEOF\n+\n+\ttest_must_fail test-tool progress \\\n+\t\t<in 2>stderr &&\n+\tgrep \"^BUG: .*'\\''one'\\'' progress still active when trying to start '\\''two'\\''$\" stderr\n+'\n+\n test_expect_success 'progress generates traces' '\n \tcat >in <<-\\EOF &&\n \tstart 40\n-- \n2.33.1.1346.g48288c3c089\n\n"},{"id":"439296","messageId":"20211021223044.GA3615@szeder.dev","threadId":"55972","inReplyTo":"patch-v3-03.10-045d58d8201-20211013T222329Z-avarab@gmail.com","subject":"Re: [PATCH v3 03/10] progress.c tests: make start/stop verbs on stdin","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-10-21T22:30:44Z","receivedAt":"2021-10-21T22:30:53Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Oct 14, 2021 at 12:28:19AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Subject: [PATCH v3 03/10] progress.c tests: make start/stop verbs on stdin\n\ns/verbs/commands/ or instructions.\n\nPlease call them what they are in the context of Git in general, or in\na test script in particular, instead of what part of speech they would\nbe if they were to appear in a sentence.\n\n> Change the usage of the \"test-tool progress\" introduced in\n> 2bb74b53a49 (Test the progress display, 2019-09-16) to take command\n> like \"start\" and \"stop\" on stdin, instead of running them implicitly.\n> \n> This makes for tests that are easier to read, since the recipe will\n> mirror the API usage, and allows for easily testing invalid usage that\n> would yield (or should yield) a BUG(), e.g. providing two \"start\"\n> calls in a row. A subsequent commit will add such tests.\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>  t/helper/test-progress.c    | 37 ++++++++++++++++-------\n>  t/t0500-progress-display.sh | 58 +++++++++++++++++++++++--------------\n>  2 files changed, 63 insertions(+), 32 deletions(-)\n> \n> diff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\n> index 50fd3be3dad..45ccbafa9da 100644\n> --- a/t/helper/test-progress.c\n> +++ b/t/helper/test-progress.c\n> @@ -3,6 +3,9 @@\n>   *\n>   * Reads instructions from standard input, one instruction per line:\n>   *\n> + *   \"start <total>[ <title>]\" - Call start_progress(title, total),\n> + *                               Uses the default title of \"Working hard\"\n> + *                               if the \" <title>\" is omitted.\n>   *   \"progress <items>\" - Call display_progress() with the given item count\n>   *                        as parameter.\n>   *   \"throughput <bytes> <millis> - Call display_throughput() with the given\n> @@ -10,6 +13,7 @@\n>   *                                  specify the time elapsed since the\n>   *                                  start_progress() call.\n>   *   \"update\" - Set the 'progress_update' flag.\n> + *   \"stop\" - Call stop_progress().\n>   *\n>   * See 't0500-progress-display.sh' for examples.\n>   */\n> @@ -19,34 +23,43 @@\n>  #include \"parse-options.h\"\n>  #include \"progress.h\"\n>  #include \"strbuf.h\"\n> +#include \"string-list.h\"\n>  \n>  int cmd__progress(int argc, const char **argv)\n>  {\n> -\tint total = 0;\n> -\tconst char *title;\n> +\tconst char *const default_title = \"Working hard\";\n> +\tstruct string_list list = STRING_LIST_INIT_DUP;\n> +\tconst struct string_list_item *item;\n>  \tstruct strbuf line = STRBUF_INIT;\n> -\tstruct progress *progress;\n> +\tstruct progress *progress = NULL;\n>  \n>  \tconst char *usage[] = {\n> -\t\t\"test-tool progress [--total=<n>] <progress-title>\",\n> +\t\t\"test-tool progress <stdin\",\n>  \t\tNULL\n>  \t};\n>  \tstruct option options[] = {\n> -\t\tOPT_INTEGER(0, \"total\", &total, \"total number of items\"),\n>  \t\tOPT_END(),\n>  \t};\n>  \n>  \targc = parse_options(argc, argv, NULL, options, usage, 0);\n> -\tif (argc != 1)\n> -\t\tdie(\"need a title for the progress output\");\n> -\ttitle = argv[0];\n> +\tif (argc)\n> +\t\tusage_with_options(usage, options);\n>  \n>  \tprogress_testing = 1;\n> -\tprogress = start_progress(title, total);\n>  \twhile (strbuf_getline(&line, stdin) != EOF) {\n>  \t\tchar *end;\n>  \n> -\t\tif (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n> +\t\tif (skip_prefix(line.buf, \"start \", (const char **) &end)) {\n> +\t\t\tuint64_t total = strtoull(end, &end, 10);\n> +\t\t\tif (*end == '\\0') {\n> +\t\t\t\tprogress = start_progress(default_title, total);\n> +\t\t\t} else if (*end == ' ') {\n> +\t\t\t\titem = string_list_insert(&list, end + 1);\n> +\t\t\t\tprogress = start_progress(item->string, total);\n\nWhy is it necessary to use a string_list here?  This is the only place\nwhere it is used, so I don't understand why we should store anything\nin it.\n\n> +\t\t\t} else {\n> +\t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n> +\t\t\t}\n> +\t\t} else if (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n>  \t\t\tuint64_t item_count = strtoull(end, &end, 10);\n>  \t\t\tif (*end != '\\0')\n>  \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n> @@ -65,12 +78,14 @@ int cmd__progress(int argc, const char **argv)\n>  \t\t\tdisplay_throughput(progress, byte_count);\n>  \t\t} else if (!strcmp(line.buf, \"update\")) {\n>  \t\t\tprogress_test_force_update();\n> +\t\t} else if (!strcmp(line.buf, \"stop\")) {\n> +\t\t\tstop_progress(&progress);\n>  \t\t} else {\n>  \t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n>  \t\t}\n>  \t}\n> -\tstop_progress(&progress);\n>  \tstrbuf_release(&line);\n> +\tstring_list_clear(&list, 0);\n>  \n>  \treturn 0;\n>  }\n> diff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\n> index f37cf2eb9c9..27ab4218b01 100755\n> --- a/t/t0500-progress-display.sh\n> +++ b/t/t0500-progress-display.sh\n> @@ -18,6 +18,7 @@ test_expect_success 'simple progress display' '\n>  \tEOF\n>  \n>  \tcat >in <<-\\EOF &&\n> +\tstart 0\n>  \tupdate\n>  \tprogress 1\n>  \tupdate\n> @@ -26,8 +27,9 @@ test_expect_success 'simple progress display' '\n>  \tprogress 4\n>  \tupdate\n>  \tprogress 5\n> +\tstop\n>  \tEOF\n> -\ttest-tool progress \"Working hard\" <in 2>stderr &&\n> +\ttest-tool progress <in 2>stderr &&\n>  \n>  \tshow_cr <stderr >out &&\n>  \ttest_cmp expect out\n> @@ -42,11 +44,13 @@ test_expect_success 'progress display with total' '\n>  \tEOF\n>  \n>  \tcat >in <<-\\EOF &&\n> +\tstart 3\n>  \tprogress 1\n>  \tprogress 2\n>  \tprogress 3\n> +\tstop\n>  \tEOF\n> -\ttest-tool progress --total=3 \"Working hard\" <in 2>stderr &&\n> +\ttest-tool progress <in 2>stderr &&\n>  \n>  \tshow_cr <stderr >out &&\n>  \ttest_cmp expect out\n> @@ -63,14 +67,14 @@ Working hard.......2.........3.........4.........5.........6:\n>  EOF\n>  \n>  \tcat >in <<-\\EOF &&\n> +\tstart 100000 Working hard.......2.........3.........4.........5.........6\n>  \tprogress 100\n>  \tprogress 1000\n>  \tprogress 10000\n>  \tprogress 100000\n> +\tstop\n>  \tEOF\n> -\ttest-tool progress --total=100000 \\\n> -\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n> -\t\t<in 2>stderr &&\n> +\ttest-tool progress <in 2>stderr &&\n>  \n>  \tshow_cr <stderr >out &&\n>  \ttest_cmp expect out\n> @@ -89,16 +93,16 @@ Working hard.......2.........3.........4.........5.........6:\n>  EOF\n>  \n>  \tcat >in <<-\\EOF &&\n> +\tstart 100000 Working hard.......2.........3.........4.........5.........6\n>  \tupdate\n>  \tprogress 1\n>  \tupdate\n>  \tprogress 2\n>  \tprogress 10000\n>  \tprogress 100000\n> +\tstop\n>  \tEOF\n> -\ttest-tool progress --total=100000 \\\n> -\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n> -\t\t<in 2>stderr &&\n> +\ttest-tool progress <in 2>stderr &&\n>  \n>  \tshow_cr <stderr >out &&\n>  \ttest_cmp expect out\n> @@ -117,14 +121,14 @@ Working hard.......2.........3.........4.........5.........6:\n>  EOF\n>  \n>  \tcat >in <<-\\EOF &&\n> +\tstart 100000 Working hard.......2.........3.........4.........5.........6\n>  \tprogress 25000\n>  \tprogress 50000\n>  \tprogress 75000\n>  \tprogress 100000\n> +\tstop\n>  \tEOF\n> -\ttest-tool progress --total=100000 \\\n> -\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n> -\t\t<in 2>stderr &&\n> +\ttest-tool progress <in 2>stderr &&\n>  \n>  \tshow_cr <stderr >out &&\n>  \ttest_cmp expect out\n> @@ -141,14 +145,14 @@ Working hard.......2.........3.........4.........5.........6.........7.........:\n>  EOF\n>  \n>  \tcat >in <<-\\EOF &&\n> +\tstart 100000 Working hard.......2.........3.........4.........5.........6.........7.........\n>  \tprogress 25000\n>  \tprogress 50000\n>  \tprogress 75000\n>  \tprogress 100000\n> +\tstop\n>  \tEOF\n> -\ttest-tool progress --total=100000 \\\n> -\t\t\"Working hard.......2.........3.........4.........5.........6.........7.........\" \\\n> -\t\t<in 2>stderr &&\n> +\ttest-tool progress <in 2>stderr &&\n>  \n>  \tshow_cr <stderr >out &&\n>  \ttest_cmp expect out\n> @@ -165,12 +169,14 @@ test_expect_success 'progress shortens - crazy caller' '\n>  \tEOF\n>  \n>  \tcat >in <<-\\EOF &&\n> +\tstart 1000\n>  \tprogress 100\n>  \tprogress 200\n>  \tprogress 1\n>  \tprogress 1000\n> +\tstop\n>  \tEOF\n> -\ttest-tool progress --total=1000 \"Working hard\" <in 2>stderr &&\n> +\ttest-tool progress <in 2>stderr &&\n>  \n>  \tshow_cr <stderr >out &&\n>  \ttest_cmp expect out\n> @@ -186,6 +192,7 @@ test_expect_success 'progress display with throughput' '\n>  \tEOF\n>  \n>  \tcat >in <<-\\EOF &&\n> +\tstart 0\n>  \tthroughput 102400 1000\n>  \tupdate\n>  \tprogress 10\n> @@ -198,8 +205,9 @@ test_expect_success 'progress display with throughput' '\n>  \tthroughput 409600 4000\n>  \tupdate\n>  \tprogress 40\n> +\tstop\n>  \tEOF\n> -\ttest-tool progress \"Working hard\" <in 2>stderr &&\n> +\ttest-tool progress <in 2>stderr &&\n>  \n>  \tshow_cr <stderr >out &&\n>  \ttest_cmp expect out\n> @@ -215,6 +223,7 @@ test_expect_success 'progress display with throughput and total' '\n>  \tEOF\n>  \n>  \tcat >in <<-\\EOF &&\n> +\tstart 40\n>  \tthroughput 102400 1000\n>  \tprogress 10\n>  \tthroughput 204800 2000\n> @@ -223,8 +232,9 @@ test_expect_success 'progress display with throughput and total' '\n>  \tprogress 30\n>  \tthroughput 409600 4000\n>  \tprogress 40\n> +\tstop\n>  \tEOF\n> -\ttest-tool progress --total=40 \"Working hard\" <in 2>stderr &&\n> +\ttest-tool progress <in 2>stderr &&\n>  \n>  \tshow_cr <stderr >out &&\n>  \ttest_cmp expect out\n> @@ -240,6 +250,7 @@ test_expect_success 'cover up after throughput shortens' '\n>  \tEOF\n>  \n>  \tcat >in <<-\\EOF &&\n> +\tstart 0\n>  \tthroughput 409600 1000\n>  \tupdate\n>  \tprogress 1\n> @@ -252,8 +263,9 @@ test_expect_success 'cover up after throughput shortens' '\n>  \tthroughput 1638400 4000\n>  \tupdate\n>  \tprogress 4\n> +\tstop\n>  \tEOF\n> -\ttest-tool progress \"Working hard\" <in 2>stderr &&\n> +\ttest-tool progress <in 2>stderr &&\n>  \n>  \tshow_cr <stderr >out &&\n>  \ttest_cmp expect out\n> @@ -268,6 +280,7 @@ test_expect_success 'cover up after throughput shortens a lot' '\n>  \tEOF\n>  \n>  \tcat >in <<-\\EOF &&\n> +\tstart 0\n>  \tthroughput 1 1000\n>  \tupdate\n>  \tprogress 1\n> @@ -277,8 +290,9 @@ test_expect_success 'cover up after throughput shortens a lot' '\n>  \tthroughput 3145728 3000\n>  \tupdate\n>  \tprogress 3\n> +\tstop\n>  \tEOF\n> -\ttest-tool progress \"Working hard\" <in 2>stderr &&\n> +\ttest-tool progress <in 2>stderr &&\n>  \n>  \tshow_cr <stderr >out &&\n>  \ttest_cmp expect out\n> @@ -286,6 +300,7 @@ test_expect_success 'cover up after throughput shortens a lot' '\n>  \n>  test_expect_success 'progress generates traces' '\n>  \tcat >in <<-\\EOF &&\n> +\tstart 40\n>  \tthroughput 102400 1000\n>  \tupdate\n>  \tprogress 10\n> @@ -298,10 +313,11 @@ test_expect_success 'progress generates traces' '\n>  \tthroughput 409600 4000\n>  \tupdate\n>  \tprogress 40\n> +\tstop\n>  \tEOF\n>  \n> -\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" test-tool progress --total=40 \\\n> -\t\t\"Working hard\" <in 2>stderr &&\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" test-tool progress \\\n> +\t\t<in 2>stderr &&\n>  \n>  \t# t0212/parse_events.perl intentionally omits regions and data.\n>  \ttest_region progress \"Working hard\" trace.event &&\n> -- \n> 2.33.1.1346.g48288c3c089\n> \n"},{"id":"439330","messageId":"YXIxLHJa6uzYAQZU@google.com","threadId":"55972","inReplyTo":"patch-v3-03.10-045d58d8201-20211013T222329Z-avarab@gmail.com","subject":"Re: [PATCH v3 03/10] progress.c tests: make start/stop verbs on stdin","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-10-22T03:34:04Z","receivedAt":"2021-10-22T03:34:16Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Thu, Oct 14, 2021 at 12:28:19AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> Change the usage of the \"test-tool progress\" introduced in\n> 2bb74b53a49 (Test the progress display, 2019-09-16) to take command\n> like \"start\" and \"stop\" on stdin, instead of running them implicitly.\n> \n> This makes for tests that are easier to read, since the recipe will\n> mirror the API usage, and allows for easily testing invalid usage that\n> would yield (or should yield) a BUG(), e.g. providing two \"start\"\n> calls in a row. A subsequent commit will add such tests.\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>  t/helper/test-progress.c    | 37 ++++++++++++++++-------\n>  t/t0500-progress-display.sh | 58 +++++++++++++++++++++++--------------\n>  2 files changed, 63 insertions(+), 32 deletions(-)\n> \n> diff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\n> index 50fd3be3dad..45ccbafa9da 100644\n> --- a/t/helper/test-progress.c\n> +++ b/t/helper/test-progress.c\n> @@ -19,34 +23,43 @@\n>  #include \"parse-options.h\"\n>  #include \"progress.h\"\n>  #include \"strbuf.h\"\n> +#include \"string-list.h\"\n>  \n>  int cmd__progress(int argc, const char **argv)\n>  {\n> -\tint total = 0;\n> -\tconst char *title;\n> +\tconst char *const default_title = \"Working hard\";\n> +\tstruct string_list list = STRING_LIST_INIT_DUP;\n> +\tconst struct string_list_item *item;\n\nI suspect the string_list is there to enable multiple progress lines\nlater, no? I saw SZEDER ask about it in another reply... If it's for\nlater, is there a reason not to add the extra structure alongside the\nseries adding multiple progress lines?\n\n> diff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\n> index f37cf2eb9c9..27ab4218b01 100755\n> --- a/t/t0500-progress-display.sh\n> +++ b/t/t0500-progress-display.sh\n> @@ -18,6 +18,7 @@ test_expect_success 'simple progress display' '\n>  \tEOF\n>  \n>  \tcat >in <<-\\EOF &&\n> +\tstart 0\n\nSeems better to me, more explicit for the test reader. Thanks for the\nchange.\n\n - Emily\n"},{"id":"439331","messageId":"YXIx/8Hl+BG7HFV9@google.com","threadId":"55972","inReplyTo":"patch-v3-08.10-e0a3510dd88-20211013T222329Z-avarab@gmail.com","subject":"Re: [PATCH v3 08/10] pack-bitmap-write.c: don't return without stop_progress()","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-10-22T03:37:35Z","receivedAt":"2021-10-22T03:37:43Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Thu, Oct 14, 2021 at 12:28:24AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> Fix a bug that's been here since 7cc8f971085 (pack-objects: implement\n> bitmap writing, 2013-12-21), we did not call stop_progress() if we\n> reached the early exit in this function.\n> \n> We could call stop_progress() before we return, but better yet is to\n> defer calling start_progress() until we need it.\n> \n> This will matter in a subsequent commit where we BUG(...) out if this\n> happens, and matters now e.g. because we don't have a corresponding\n> \"region_end\" for the progress trace2 event.\n> \n> Suggested-by: SZEDER Gábor <szeder.dev@gmail.com>\n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\nVery nice redo.\nReviewed-by: Emily Shaffer <emilyshaffer@google.com>\n"},{"id":"439332","messageId":"YXIyc7+ipDVxj7Ff@google.com","threadId":"55972","inReplyTo":"patch-v3-09.10-2cf14881ecf-20211013T222329Z-avarab@gmail.com","subject":"Re: [PATCH v3 09/10] various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO)","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-10-22T03:39:31Z","receivedAt":"2021-10-22T03:39:40Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Thu, Oct 14, 2021 at 12:28:25AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> We have over 50 uses of \"isatty(1)\" and \"isatty(2)\" in the codebase,\n> only these two used the stdlib.h macros to refer to them.\n> \n> Let's change these for consistency, and because a subsequent commit's\n> commit message outlines a recipe to change all of these for ad-hoc\n> testing, not needing to match these with that ad-hoc regex will make\n> things easier to explain.\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n\nUsually I'd really rather see this go the other way and reduce the\nnumber of \"magic numbers\" in the codebase, but I think file handles are\npretty well-known, so I guess for the sake of brevity it's ok...\n\nReviewed-by: Emily Shaffer <emilyshaffer@google.com>\n\n> ---\n>  builtin/bisect--helper.c | 2 +-\n>  builtin/bundle.c         | 2 +-\n>  compat/mingw.c           | 2 +-\n>  3 files changed, 3 insertions(+), 3 deletions(-)\n> \n> diff --git a/builtin/bisect--helper.c b/builtin/bisect--helper.c\n> index 28a2e6a5750..1727cb051fb 100644\n> --- a/builtin/bisect--helper.c\n> +++ b/builtin/bisect--helper.c\n> @@ -830,7 +830,7 @@ static int bisect_autostart(struct bisect_terms *terms)\n>  \tfprintf_ln(stderr, _(\"You need to start by \\\"git bisect \"\n>  \t\t\t  \"start\\\"\\n\"));\n>  \n> -\tif (!isatty(STDIN_FILENO))\n> +\tif (!isatty(1))\n>  \t\treturn -1;\n>  \n>  \t/*\n> diff --git a/builtin/bundle.c b/builtin/bundle.c\n> index 5a85d7cd0fe..df69c651753 100644\n> --- a/builtin/bundle.c\n> +++ b/builtin/bundle.c\n> @@ -56,7 +56,7 @@ static int parse_options_cmd_bundle(int argc,\n>  \n>  static int cmd_bundle_create(int argc, const char **argv, const char *prefix) {\n>  \tint all_progress_implied = 0;\n> -\tint progress = isatty(STDERR_FILENO);\n> +\tint progress = isatty(2);\n>  \tstruct strvec pack_opts;\n>  \tint version = -1;\n>  \tint ret;\n> diff --git a/compat/mingw.c b/compat/mingw.c\n> index 9e0cd1e097f..0f545c1a7d1 100644\n> --- a/compat/mingw.c\n> +++ b/compat/mingw.c\n> @@ -2374,7 +2374,7 @@ int mingw_raise(int sig)\n>  \tswitch (sig) {\n>  \tcase SIGALRM:\n>  \t\tif (timer_fn == SIG_DFL) {\n> -\t\t\tif (isatty(STDERR_FILENO))\n> +\t\t\tif (isatty(2))\n>  \t\t\t\tfputs(\"Alarm clock\\n\", stderr);\n>  \t\t\texit(128 + SIGALRM);\n>  \t\t} else if (timer_fn != SIG_IGN)\n> -- \n> 2.33.1.1346.g48288c3c089\n> \n"},{"id":"439333","messageId":"YXIzMFI62fBsgjMM@google.com","threadId":"55972","inReplyTo":"patch-v3-10.10-01d5bbfce76-20211013T222329Z-avarab@gmail.com","subject":"Re: [PATCH v3 10/10] progress.c: add & assert a \"global_progress\" variable","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-10-22T03:42:40Z","receivedAt":"2021-10-22T03:42:48Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Thu, Oct 14, 2021 at 12:28:26AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> The progress.c code makes a hard assumption that only one progress bar\n> be active at a time (see [1] for a bug where this wasn't the\n> case). Add a BUG() that'll trigger if we ever regress on that promise\n> and have two progress bars active at the same time.\n> \n> There was an alternative test-only approach to doing the same\n> thing[2], but by doing this outside of a GIT_TEST_* mode we'll know\n> we've put a hard stop to this particular API misuse.\n> \n> It will also establish scaffolding to address current fundamental\n> limitations in the progress output: The current output must be\n> \"driven\" by calls to the likes of display_progress(). Once we have a\n> global current progress object we'll be able to update that object via\n> SIGALRM. See [3] for early code to do that.\n> \n> It's conceivable that this change will hit the BUG() condition in some\n> scenario that we don't currently have tests for, this would be very\n> bad. If that happened we'd die just because we couldn't emit some\n> pretty output.\n> \n> See [4] for a discussion of why our test coverage is lacking; our\n> progress display is hidden behind isatty(2) checks in many cases, so\n> the test suite doesn't cover it unless individual tests are run in\n> \"--verbose\" mode, we might also have multi-threaded use of the API, so\n> two progress bars stopping and starting would only be visible due to a\n> race condition.\n> \n> Despite that, I think that this change won't introduce such\n> regressions, because:\n> \n>  1. I've read all the code using the progress API (and have modified a\n>     large part of it in some WIP code I have). Almost all of it is really\n>     simple, the parts that aren't[5] are complex in the display_progress() part,\n>     not in starting or stopping the progress bar.\n> \n>  2. The entire test suite passes when instrumented with an ad-hoc\n>     Linux-specific mode (it uses gettid()) to die if progress bars are\n>     ever started or stopped on anything but the main thread[6].\n> \n>     Extending that to die if display_progress() is called in a thread\n>     reveals that we have exactly two users of the progress bar under\n>     threaded conditions, \"git index-pack\" and \"git pack-objects\". Both\n>     uses are straightforward, and they don't start/stop the progress\n>     bar when threads are active.\n> \n>  3. I've likewise done an ad-hoc test to force progress bars to be\n>     displayed with:\n> \n>         perl -pi -e 's[isatty\\(2\\)][1]g' $(git grep -l -F 'isatty(2)')\n> \n>     I.e. to replace all checks (not just for progress) of checking\n>     whether STDERR is connected to a TTY, and then monkeypatching\n>     is_foreground_fd() in progress.c to always \"return 1\". Running the\n>     tests with those applied, interactively and under -V reveals via:\n> \n>         $ grep -e set_progress_signal -e clear_progress_signal test-results/*out\n> \n>     That nothing our tests cover hits the BUG conditions added here,\n>     except the expected \"BUG: start two concurrent progress bars\" test\n>     being added here.\n> \n>     That isn't entirely true since we won't be getting 100% coverage\n>     due to cascading failures from tests that expected no progress\n>     output on stderr. To make sure I covered 100% I also tried making\n>     the display() function in progress.c a NOOP on top of that (it's\n>     the calls to start_progress_delay() and stop_progress()) that\n>     matter.\n> \n>     That doesn't hit the BUG() either. Some tests fail in that mode\n>     due to a combination of the overzealous isatty(2) munging noted\n>     above, and the tests that are testing that the progress output\n>     itself is present (but for testing I'd made display() a NOOP).\n> \n> Between those three points I think it's safe to go ahead with this\n> change.\n> \n> 1. 6f9d5f2fda1 (commit-graph: fix progress of reachable commits, 2020-07-09)\n> 2. https://lore.kernel.org/git/20210620200303.2328957-3-szeder.dev@gmail.com\n> 3. https://lore.kernel.org/git/patch-18.25-e21fc66623f-20210623T155626Z-avarab@gmail.com/\n> 4. https://lore.kernel.org/git/cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com/\n> 5. b50c37aa44d (Merge branch 'ab/progress-users-adjust-counters' into\n>    next, 2021-09-10)\n> 6. https://lore.kernel.org/git/877dffg37n.fsf@evledraar.gmail.com/\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n\nI find it much nicer to understand now, thanks. The BUG() change in\nparticular is excellent.\n\nReviewed-by: Emily Shaffer <emilyshaffer@google.com>\n"},{"id":"439334","messageId":"YXIzitjDUpJcFOcS@google.com","threadId":"55972","inReplyTo":"cover-v3-00.10-00000000000-20211013T222329Z-avarab@gmail.com","subject":"Re: [PATCH v3 00/10] progress: assert \"global_progress\" + test fixes / cleanup","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-10-22T03:44:10Z","receivedAt":"2021-10-22T03:44:18Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Thu, Oct 14, 2021 at 12:28:16AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> This series fixes various issues in and related to progress.c, and\n> adds a BUG() assertion for us not starting two progress bars at the\n> same time. Those changes are needed for subsequent changes that do\n> more interesting things with this new global progress bar.\n> \n> This v3 hopefully addresses all the feedback on the v2, thanks\n> all. Changes:\n> \n>  * Fix a memory leak in 1/10, and make the progress tests use the\n>    SANITIZE=leak test mode.\n> \n>  * Simplified some of the test-progress.c code (no more \"start\"\n>    handling, the \"total\" count is mandatory now.\n> \n>  * Split out a formatting change into 2/10 to make 3/10 easier to\n>    read.\n> \n>  * A new 9/10 makes an ad-hoc test recipie in 10/10 easier to explain\n>    (in response to Emily's comment).\n> \n>  * The BUG() assertion in 10/10 now has a much better message, we dump\n>    the title of the two progress bars in play if we have a bug where\n>    we started two at the same time.\n\nOne thing I noticed (and so did SZEDER) was the addition of string_list\nto the helper; I made a comment accordingly in the patch where it was\nadded. It kind of seemed like an oversight - but I remember you were\nwanting to head in that direction (multiple progress bar support) so\nmaybe it was paving the way for that?\n\nAnyways, with or without it the series looks OK to me in this round too.\n\n - Emily\n"},{"id":"439373","messageId":"211022.864k99kune.gmgdl@evledraar.gmail.com","threadId":"55972","inReplyTo":"20211021223044.GA3615@szeder.dev","subject":"Re: [PATCH v3 03/10] progress.c tests: make start/stop verbs on stdin","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-22T14:18:16Z","receivedAt":"2021-10-22T14:21:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Oct 22 2021, SZEDER Gábor wrote:\n\n> On Thu, Oct 14, 2021 at 12:28:19AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> Subject: [PATCH v3 03/10] progress.c tests: make start/stop verbs on stdin\n>\n> s/verbs/commands/ or instructions.\n>\n> Please call them what they are in the context of Git in general, or in\n> a test script in particular, instead of what part of speech they would\n> be if they were to appear in a sentence.\n\n*nod*, although barring further issues I don't think I'll re-roll with\njust that commit message adjustment...\n\n>>  \tprogress_testing = 1;\n>> -\tprogress = start_progress(title, total);\n>>  \twhile (strbuf_getline(&line, stdin) != EOF) {\n>>  \t\tchar *end;\n>>  \n>> -\t\tif (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n>> +\t\tif (skip_prefix(line.buf, \"start \", (const char **) &end)) {\n>> +\t\t\tuint64_t total = strtoull(end, &end, 10);\n>> +\t\t\tif (*end == '\\0') {\n>> +\t\t\t\tprogress = start_progress(default_title, total);\n>> +\t\t\t} else if (*end == ' ') {\n>> +\t\t\t\titem = string_list_insert(&list, end + 1);\n>> +\t\t\t\tprogress = start_progress(item->string, total);\n>\n> Why is it necessary to use a string_list here?  This is the only place\n> where it is used, so I don't understand why we should store anything\n> in it.\n\nThe progress.c API doesn't xstrdup() the title you give it, so you can't\nfree() it after while it's alive.\n\nHere we're re-setting the same strbuf in a loop, so if we hand a \"title\"\nto progress.c we need to keep it around, when we later add a BUG\nassertion in 10/10:\n\n+\tif (global_progress)\n+\t\tBUG(\"'%s' progress still active when trying to start '%s'\",\n+\t\t    global_progress->title, progress->title);\n+\tglobal_progress = progress;\n\nThe \"old\" title would point to already-free'd or to a nonsensical value\nif we didn't do that.\n"},{"id":"439374","messageId":"211022.86zgr1jg0b.gmgdl@evledraar.gmail.com","threadId":"55972","inReplyTo":"YXIzitjDUpJcFOcS@google.com","subject":"Re: [PATCH v3 00/10] progress: assert \"global_progress\" + test fixes / cleanup","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-22T14:21:29Z","receivedAt":"2021-10-22T14:22:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Oct 21 2021, Emily Shaffer wrote:\n\n> On Thu, Oct 14, 2021 at 12:28:16AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> This series fixes various issues in and related to progress.c, and\n>> adds a BUG() assertion for us not starting two progress bars at the\n>> same time. Those changes are needed for subsequent changes that do\n>> more interesting things with this new global progress bar.\n>> \n>> This v3 hopefully addresses all the feedback on the v2, thanks\n>> all. Changes:\n>> \n>>  * Fix a memory leak in 1/10, and make the progress tests use the\n>>    SANITIZE=leak test mode.\n>> \n>>  * Simplified some of the test-progress.c code (no more \"start\"\n>>    handling, the \"total\" count is mandatory now.\n>> \n>>  * Split out a formatting change into 2/10 to make 3/10 easier to\n>>    read.\n>> \n>>  * A new 9/10 makes an ad-hoc test recipie in 10/10 easier to explain\n>>    (in response to Emily's comment).\n>> \n>>  * The BUG() assertion in 10/10 now has a much better message, we dump\n>>    the title of the two progress bars in play if we have a bug where\n>>    we started two at the same time.\n>\n> One thing I noticed (and so did SZEDER) was the addition of string_list\n> to the helper; I made a comment accordingly in the patch where it was\n> added. It kind of seemed like an oversight - but I remember you were\n> wanting to head in that direction (multiple progress bar support) so\n> maybe it was paving the way for that?\n>\n> Anyways, with or without it the series looks OK to me in this round too.\n\nReplied to in the thread with SZEDER at:\nhttps://lore.kernel.org/git/211022.864k99kune.gmgdl@evledraar.gmail.com/\n\nI.e. it's not for multi-progress-bar support, but because the API\ndoesn't xstrdup() the title. The loop handing it the title needs to keep\nthe string it gave to it alive.\n"},{"id":"439428","messageId":"YXM3ynt1UafUHfMQ@nand.local","threadId":"55972","inReplyTo":"211022.864k99kune.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v3 03/10] progress.c tests: make start/stop verbs on stdin","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-10-22T22:14:34Z","receivedAt":"2021-10-22T22:14:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Oct 22, 2021 at 04:18:16PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> >>  \tprogress_testing = 1;\n> >> -\tprogress = start_progress(title, total);\n> >>  \twhile (strbuf_getline(&line, stdin) != EOF) {\n> >>  \t\tchar *end;\n> >>\n> >> -\t\tif (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n> >> +\t\tif (skip_prefix(line.buf, \"start \", (const char **) &end)) {\n> >> +\t\t\tuint64_t total = strtoull(end, &end, 10);\n> >> +\t\t\tif (*end == '\\0') {\n> >> +\t\t\t\tprogress = start_progress(default_title, total);\n> >> +\t\t\t} else if (*end == ' ') {\n> >> +\t\t\t\titem = string_list_insert(&list, end + 1);\n> >> +\t\t\t\tprogress = start_progress(item->string, total);\n> >\n> > Why is it necessary to use a string_list here?  This is the only place\n> > where it is used, so I don't understand why we should store anything\n> > in it.\n>\n> The progress.c API doesn't xstrdup() the title you give it, so you can't\n> free() it after while it's alive.\n>\n> Here we're re-setting the same strbuf in a loop, so if we hand a \"title\"\n> to progress.c we need to keep it around, when we later add a BUG\n> assertion in 10/10:\n>\n> +\tif (global_progress)\n> +\t\tBUG(\"'%s' progress still active when trying to start '%s'\",\n> +\t\t    global_progress->title, progress->title);\n> +\tglobal_progress = progress;\n>\n> The \"old\" title would point to already-free'd or to a nonsensical value\n> if we didn't do that.\n\n*nod*. I figured as much, but a comment indicating that may be helpful\nfor reviewers and future readers of this code.\n\nThanks,\nTaylor\n"},{"id":"439430","messageId":"YXM7G22L+te1k3eV@nand.local","threadId":"55972","inReplyTo":"patch-v3-06.10-c7c3843564e-20211013T222329Z-avarab@gmail.com","subject":"Re: [PATCH v3 06/10] progress.c: call progress_interval() from progress_test_force_update()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-10-22T22:28:43Z","receivedAt":"2021-10-22T22:28:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Oct 14, 2021 at 12:28:22AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Define the progress_test_force_update() function in terms of\n> progress_interval(). For documentation purposes these two functions\n> have the same body, but different names. Let's just define the test\n> function by calling progress_interval() with SIGALRM ourselves.\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>  progress.c | 2 +-\n>  1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/progress.c b/progress.c\n> index 893cb0fe56f..7fcc513717a 100644\n> --- a/progress.c\n> +++ b/progress.c\n> @@ -216,7 +216,7 @@ static void progress_interval(int signum)\n>   */\n>  void progress_test_force_update(void)\n>  {\n> -\tprogress_update = 1;\n> +\tprogress_interval(SIGALRM);\n>  }\n\nI agree that the pre- and post-image of this patch do the same thing.\n(And not that it matters, but the value for 'signum' could have been\nwhatever you want, since progress_interval() ignores it).\n\nIs there a reason to make this change other than to make clear that the\ntwo do the same thing? It may be worth calling that out explicitly in\nyour patch message if so. You kind of do this, but I'd be as explicit\nas:\n\n  \"To make it clear that progress_test_force_update() is a synonym for\n  progress_interval(), define for the former in terms of a single call\n  to the latter.\"\n\nI think your second sentence says basically that, but it took me a few\nattempts to discover.\n\nThanks,\nTaylor\n"},{"id":"439431","messageId":"YXM9KVUousJXgRup@nand.local","threadId":"55972","inReplyTo":"patch-v3-07.10-cd2d27b1626-20211013T222329Z-avarab@gmail.com","subject":"Re: [PATCH v3 07/10] progress.c: add temporary variable from progress struct","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-10-22T22:37:29Z","receivedAt":"2021-10-22T22:37:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Oct 14, 2021 at 12:28:23AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Add a temporary \"progress\" variable for the dereferenced p_progress\n> pointer to a \"struct progress *\". Before 98a13647408 (trace2: log\n> progress time and throughput, 2020-05-12) we didn't dereference\n> \"p_progress\" in this function, now that we do it's easier to read the\n> code if we work with a \"progress\" struct pointer like everywhere else,\n> instead of a pointer to a pointer.\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>  progress.c | 5 +++--\n>  1 file changed, 3 insertions(+), 2 deletions(-)\n>\n> diff --git a/progress.c b/progress.c\n> index 7fcc513717a..b9369e9a264 100644\n> --- a/progress.c\n> +++ b/progress.c\n> @@ -329,15 +329,16 @@ void stop_progress(struct progress **p_progress)\n>  \tfinish_if_sparse(*p_progress);\n\nThis spot also wants to dereference p_progress, but doesn't access any\nof the fields by dereferencing the second pointer. So this could\ncertainly have passed `progress` (had you scoped its declaration to the\nwhole function).\n\nI don't think it hurts readability, and I definitely don't have a strong\nopinion about it. Just an idle thought while reading this patch...\n\n>  \tif (*p_progress) {\n> +\t\tstruct progress *progress = *p_progress;\n>  \t\ttrace2_data_intmax(\"progress\", the_repository, \"total_objects\",\n>  \t\t\t\t   (*p_progress)->total);\n>\n>  \t\tif ((*p_progress)->throughput)\n>  \t\t\ttrace2_data_intmax(\"progress\", the_repository,\n>  \t\t\t\t\t   \"total_bytes\",\n> -\t\t\t\t\t   (*p_progress)->throughput->curr_total);\n> +\t\t\t\t\t   progress->throughput->curr_total);\n>\n> -\t\ttrace2_region_leave(\"progress\", (*p_progress)->title, the_repository);\n> +\t\ttrace2_region_leave(\"progress\", progress->title, the_repository);\n\nThis all looks much better to me. Thanks.\n\nThanks,\nTaylor\n"},{"id":"439433","messageId":"YXM/dw2llyrWA370@nand.local","threadId":"55972","inReplyTo":"patch-v3-08.10-e0a3510dd88-20211013T222329Z-avarab@gmail.com","subject":"Re: [PATCH v3 08/10] pack-bitmap-write.c: don't return without stop_progress()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-10-22T22:47:19Z","receivedAt":"2021-10-22T22:47:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Oct 14, 2021 at 12:28:24AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> Fix a bug that's been here since 7cc8f971085 (pack-objects: implement\n> bitmap writing, 2013-12-21), we did not call stop_progress() if we\n> reached the early exit in this function.\n>\n> We could call stop_progress() before we return, but better yet is to\n> defer calling start_progress() until we need it.\n>\n> This will matter in a subsequent commit where we BUG(...) out if this\n> happens, and matters now e.g. because we don't have a corresponding\n> \"region_end\" for the progress trace2 event.\n>\n> Suggested-by: SZEDER Gábor <szeder.dev@gmail.com>\n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>  pack-bitmap-write.c | 6 +++---\n>  1 file changed, 3 insertions(+), 3 deletions(-)\n>\n> diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\n> index 9c55c1531e1..cab3eaa2acd 100644\n> --- a/pack-bitmap-write.c\n> +++ b/pack-bitmap-write.c\n> @@ -575,15 +575,15 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n>\n>  \tQSORT(indexed_commits, indexed_commits_nr, date_compare);\n>\n> -\tif (writer.show_progress)\n> -\t\twriter.progress = start_progress(\"Selecting bitmap commits\", 0);\n> -\n>  \tif (indexed_commits_nr < 100) {\n>  \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n>  \t\t\tpush_bitmapped_commit(indexed_commits[i]);\n>  \t\treturn;\n>  \t}\n>\n> +\tif (writer.show_progress)\n> +\t\twriter.progress = start_progress(\"Selecting bitmap commits\", 0);\n> +\n\nAh, this is the change that you and I discovered independently. For what\nit's worth, I wrote something slightly differently, which was to jump to\na \"cleanup\" label and then call stop_progress() there instead of waiting\nto call start_progress().\n\nBut thinking on it more, I think that difference highlights a compelling\nreason to get rid of this progress meter entirely. That's because this\nrule of \"if there are fewer than 100 commits, select all of them\"\ndoesn't even update the progress meter!\n\nAnd the loop that it is timing below is so fast that we end up just\nshowing the final count, which doesn't provide any useful information\nbeyond how many commits *could* have been selected.\n\nSo I think that your fix is OK, and I'd be happy to see this get picked\nup as-is, but I'd be equally happy to drop this progress meter\naltogether.\n\n>  \tfor (;;) {\n>  \t\tstruct commit *chosen = NULL;\n>\n> --\n> 2.33.1.1346.g48288c3c089\n>\nThanks,\nTaylor\n"},{"id":"439504","messageId":"20211024201000.GB2101@szeder.dev","threadId":"55972","inReplyTo":"YXM3ynt1UafUHfMQ@nand.local","subject":"Re: [PATCH v3 03/10] progress.c tests: make start/stop verbs on stdin","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-10-24T20:10:00Z","receivedAt":"2021-10-24T20:10:11Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Fri, Oct 22, 2021 at 06:14:34PM -0400, Taylor Blau wrote:\n> On Fri, Oct 22, 2021 at 04:18:16PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> > >>  \tprogress_testing = 1;\n> > >> -\tprogress = start_progress(title, total);\n> > >>  \twhile (strbuf_getline(&line, stdin) != EOF) {\n> > >>  \t\tchar *end;\n> > >>\n> > >> -\t\tif (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n> > >> +\t\tif (skip_prefix(line.buf, \"start \", (const char **) &end)) {\n> > >> +\t\t\tuint64_t total = strtoull(end, &end, 10);\n> > >> +\t\t\tif (*end == '\\0') {\n> > >> +\t\t\t\tprogress = start_progress(default_title, total);\n> > >> +\t\t\t} else if (*end == ' ') {\n> > >> +\t\t\t\titem = string_list_insert(&list, end + 1);\n> > >> +\t\t\t\tprogress = start_progress(item->string, total);\n> > >\n> > > Why is it necessary to use a string_list here?  This is the only place\n> > > where it is used, so I don't understand why we should store anything\n> > > in it.\n> >\n> > The progress.c API doesn't xstrdup() the title you give it, so you can't\n> > free() it after while it's alive.\n> >\n> > Here we're re-setting the same strbuf in a loop, so if we hand a \"title\"\n> > to progress.c we need to keep it around, when we later add a BUG\n> > assertion in 10/10:\n> >\n> > +\tif (global_progress)\n> > +\t\tBUG(\"'%s' progress still active when trying to start '%s'\",\n> > +\t\t    global_progress->title, progress->title);\n> > +\tglobal_progress = progress;\n> >\n> > The \"old\" title would point to already-free'd or to a nonsensical value\n> > if we didn't do that.\n> \n> *nod*. I figured as much, but a comment indicating that may be helpful\n> for reviewers and future readers of this code.\n\nYeah, a comment would be nice; though perhaps giving that list\nvariable a descriptive name, e.g. 'titles' (plural!) would have been\nenough to set me on the right track.\n\n"},{"id":"439512","messageId":"20211025050202.GC2101@szeder.dev","threadId":"55972","inReplyTo":"patch-v3-10.10-01d5bbfce76-20211013T222329Z-avarab@gmail.com","subject":"Re: [PATCH v3 10/10] progress.c: add & assert a \"global_progress\" variable","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-10-25T05:02:02Z","receivedAt":"2021-10-25T05:02:09Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Oct 14, 2021 at 12:28:26AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> The progress.c code makes a hard assumption that only one progress bar\n> be active at a time (see [1] for a bug where this wasn't the\n> case). Add a BUG() that'll trigger if we ever regress on that promise\n> and have two progress bars active at the same time.\n\nI still very much dislike the idea of a BUG() in the progress code\nthat can trigger outside of the test suite, because the progress line\nis only a UI gimmick and not a crucial part of any Git operation, and\neven though a progress line might be buggy, the underlying Git\noperation is not affected by it and would still finish successfully,\nas was the case with the dozen of so progress line bugs in the past.\n\n> There was an alternative test-only approach to doing the same\n> thing[2], but by doing this outside of a GIT_TEST_* mode we'll know\n> we've put a hard stop to this particular API misuse.\n> \n> It will also establish scaffolding to address current fundamental\n> limitations in the progress output: The current output must be\n> \"driven\" by calls to the likes of display_progress().\n\nPlease elaborate why that is a \"fundamental limitation\"; I don't see\nany drawback of the current approach.\n\n> Once we have a\n> global current progress object we'll be able to update that object via\n> SIGALRM.\n\nWhat are the supposed benefits of doing that?  I do see its drawbacks,\nconsidering that we have progress lines that are updated from multiple\nthreads.\n\n> See [3] for early code to do that.\n> \n> It's conceivable that this change will hit the BUG() condition in some\n> scenario that we don't currently have tests for, this would be very\n> bad. If that happened we'd die just because we couldn't emit some\n> pretty output.\n> \n> See [4] for a discussion of why our test coverage is lacking; our\n> progress display is hidden behind isatty(2) checks in many cases, so\n> the test suite doesn't cover it unless individual tests are run in\n> \"--verbose\" mode, we might also have multi-threaded use of the API, so\n> two progress bars stopping and starting would only be visible due to a\n> race condition.\n> \n> Despite that, I think that this change won't introduce such\n> regressions, because:\n> \n>  1. I've read all the code using the progress API (and have modified a\n>     large part of it in some WIP code I have). Almost all of it is really\n>     simple, the parts that aren't[5] are complex in the display_progress() part,\n>     not in starting or stopping the progress bar.\n> \n>  2. The entire test suite passes when instrumented with an ad-hoc\n>     Linux-specific mode (it uses gettid()) to die if progress bars are\n>     ever started or stopped on anything but the main thread[6].\n> \n>     Extending that to die if display_progress() is called in a thread\n>     reveals that we have exactly two users of the progress bar under\n>     threaded conditions, \"git index-pack\" and \"git pack-objects\". Both\n>     uses are straightforward, and they don't start/stop the progress\n>     bar when threads are active.\n> \n>  3. I've likewise done an ad-hoc test to force progress bars to be\n>     displayed with:\n> \n>         perl -pi -e 's[isatty\\(2\\)][1]g' $(git grep -l -F 'isatty(2)')\n> \n>     I.e. to replace all checks (not just for progress) of checking\n>     whether STDERR is connected to a TTY, and then monkeypatching\n>     is_foreground_fd() in progress.c to always \"return 1\". Running the\n>     tests with those applied, interactively and under -V reveals via:\n> \n>         $ grep -e set_progress_signal -e clear_progress_signal test-results/*out\n> \n>     That nothing our tests cover hits the BUG conditions added here,\n>     except the expected \"BUG: start two concurrent progress bars\" test\n>     being added here.\n> \n>     That isn't entirely true since we won't be getting 100% coverage\n>     due to cascading failures from tests that expected no progress\n>     output on stderr. To make sure I covered 100% I also tried making\n>     the display() function in progress.c a NOOP on top of that (it's\n>     the calls to start_progress_delay() and stop_progress()) that\n>     matter.\n> \n>     That doesn't hit the BUG() either. Some tests fail in that mode\n>     due to a combination of the overzealous isatty(2) munging noted\n>     above, and the tests that are testing that the progress output\n>     itself is present (but for testing I'd made display() a NOOP).\n> \n> Between those three points I think it's safe to go ahead with this\n> change.\n\nThe above analysis only considers _our_ _current_ codebase.\n\nHowever, even though this might be safe now, it doesn't mean that it\nwill remain safe in the future, as we might add new progress lines\nthat lack test coverage (though hopefully won't), and would hit that\nBUG() at a user.\n\nFurthermore, even though this might be safe in our codebase, it\ndoesn't mean that it is safe in some 20+k forks of Git that exist on\nGitHub alone (I for one have a git command or two with in my fork\nwhich output progress lines but, sadly, have zero test coverage).\n\nBut more importantly, even though it might be safe to do so, that\ndoesn't mean that it's a good idea to do so.  The commit message does\nlittle to justify why it is conceptually a good idea to add this BUG()\nto the progress code in a way that it can trigger outside of the test\nsuite.\n\n\n> 1. 6f9d5f2fda1 (commit-graph: fix progress of reachable commits, 2020-07-09)\n> 2. https://lore.kernel.org/git/20210620200303.2328957-3-szeder.dev@gmail.com\n> 3. https://lore.kernel.org/git/patch-18.25-e21fc66623f-20210623T155626Z-avarab@gmail.com/\n> 4. https://lore.kernel.org/git/cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com/\n> 5. b50c37aa44d (Merge branch 'ab/progress-users-adjust-counters' into\n>    next, 2021-09-10)\n> 6. https://lore.kernel.org/git/877dffg37n.fsf@evledraar.gmail.com/\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>  progress.c                  | 18 ++++++++++++++----\n>  t/t0500-progress-display.sh | 11 +++++++++++\n>  2 files changed, 25 insertions(+), 4 deletions(-)\n> \n> diff --git a/progress.c b/progress.c\n> index b9369e9a264..a31500f8b2b 100644\n> --- a/progress.c\n> +++ b/progress.c\n> @@ -46,6 +46,7 @@ struct progress {\n>  };\n>  \n>  static volatile sig_atomic_t progress_update;\n> +static struct progress *global_progress;\n>  \n>  /*\n>   * These are only intended for testing the progress output, i.e. exclusively\n> @@ -219,11 +220,16 @@ void progress_test_force_update(void)\n>  \tprogress_interval(SIGALRM);\n>  }\n>  \n> -static void set_progress_signal(void)\n> +static void set_progress_signal(struct progress *progress)\n>  {\n>  \tstruct sigaction sa;\n>  \tstruct itimerval v;\n>  \n> +\tif (global_progress)\n> +\t\tBUG(\"'%s' progress still active when trying to start '%s'\",\n> +\t\t    global_progress->title, progress->title);\n> +\tglobal_progress = progress;\n\nThis function is called set_progress_signal(), so checking and setting\n'global_progress' feels out of place here; it would be better to do\nthat in start_progress_delay().\n\n> +\n>  \tif (progress_testing)\n>  \t\treturn;\n>  \n> @@ -241,10 +247,14 @@ static void set_progress_signal(void)\n>  \tsetitimer(ITIMER_REAL, &v, NULL);\n>  }\n>  \n> -static void clear_progress_signal(void)\n> +static void clear_progress_signal(struct progress *progress)\n>  {\n>  \tstruct itimerval v = {{0,},};\n>  \n> +\tif (!global_progress)\n> +\t\tBUG(\"should have active global_progress when cleaning up\");\n> +\tglobal_progress = NULL;\n\nLikewise.\n\n>  \tif (progress_testing)\n>  \t\treturn;\n>  \n> @@ -268,7 +278,7 @@ static struct progress *start_progress_delay(const char *title, uint64_t total,\n>  \tstrbuf_init(&progress->counters_sb, 0);\n>  \tprogress->title_len = utf8_strwidth(title);\n>  \tprogress->split = 0;\n> -\tset_progress_signal();\n> +\tset_progress_signal(progress);\n>  \ttrace2_region_enter(\"progress\", title, the_repository);\n>  \treturn progress;\n>  }\n> @@ -372,7 +382,7 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)\n>  \t\tdisplay(progress, progress->last_value, buf);\n>  \t\tfree(buf);\n>  \t}\n> -\tclear_progress_signal();\n> +\tclear_progress_signal(progress);\n>  \tstrbuf_release(&progress->counters_sb);\n>  \tif (progress->throughput)\n>  \t\tstrbuf_release(&progress->throughput->display);\n> diff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\n> index 59e9f226ea4..867fdace3f2 100755\n> --- a/t/t0500-progress-display.sh\n> +++ b/t/t0500-progress-display.sh\n> @@ -298,6 +298,17 @@ test_expect_success 'cover up after throughput shortens a lot' '\n>  \ttest_cmp expect out\n>  '\n>  \n> +test_expect_success 'BUG: start two concurrent progress bars' '\n> +\tcat >in <<-\\EOF &&\n> +\tstart 0 one\n> +\tstart 0 two\n> +\tEOF\n> +\n> +\ttest_must_fail test-tool progress \\\n> +\t\t<in 2>stderr &&\n> +\tgrep \"^BUG: .*'\\''one'\\'' progress still active when trying to start '\\''two'\\''$\" stderr\n> +'\n> +\n>  test_expect_success 'progress generates traces' '\n>  \tcat >in <<-\\EOF &&\n>  \tstart 40\n> -- \n> 2.33.1.1346.g48288c3c089\n> \n"},{"id":"439522","messageId":"xmqq35op4f7n.fsf@gitster.g","threadId":"55972","inReplyTo":"20211025050202.GC2101@szeder.dev","subject":"Re: [PATCH v3 10/10] progress.c: add & assert a \"global_progress\" variable","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-25T09:38:04Z","receivedAt":"2021-10-25T09:38:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"SZEDER Gábor <szeder.dev@gmail.com> writes:\n\n> I still very much dislike the idea of a BUG() in the progress code\n> that can trigger outside of the test suite, because the progress line\n> is only a UI gimmick and not a crucial part of any Git operation, and\n> even though a progress line might be buggy, the underlying Git\n> operation is not affected by it and would still finish successfully,\n> as was the case with the dozen of so progress line bugs in the past.\n\nI too recall that we have fixed numerous bugs in the past year in\nthe area, but weren't they kind of obvious ones _once_ they are\npointed out at you (e.g. progress never reaching to 100%)?  Yet the\ndevelopers have failed to catch them because their eyes would coast\nover without paying attention to them, exactly because the progress\nbar is merely a UI gimmick.\n\nI haven't formed a firm opinion on this yet, but I think the idea\nbehind these BUG() is to help such problems be caught while they are\nstill in the lab.  You may not notice when your live progress bar\nbehaved a bit funny, but if you hit a BUG(), that would be squarely\nin your face and you cannot ignore it.\n"},{"id":"439527","messageId":"211025.867de1icks.gmgdl@evledraar.gmail.com","threadId":"55972","inReplyTo":"20211025050202.GC2101@szeder.dev","subject":"Re: [PATCH v3 10/10] progress.c: add & assert a \"global_progress\" variable","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-25T11:06:03Z","receivedAt":"2021-10-25T11:11:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Oct 25 2021, SZEDER Gábor wrote:\n\n> On Thu, Oct 14, 2021 at 12:28:26AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> The progress.c code makes a hard assumption that only one progress bar\n>> be active at a time (see [1] for a bug where this wasn't the\n>> case). Add a BUG() that'll trigger if we ever regress on that promise\n>> and have two progress bars active at the same time.\n>\n> I still very much dislike the idea of a BUG() in the progress code\n> that can trigger outside of the test suite, because the progress line\n> is only a UI gimmick and not a crucial part of any Git operation, and\n> even though a progress line might be buggy, the underlying Git\n> operation is not affected by it and would still finish successfully,\n> as was the case with the dozen of so progress line bugs in the past.\n>\n>> There was an alternative test-only approach to doing the same\n>> thing[2], but by doing this outside of a GIT_TEST_* mode we'll know\n>> we've put a hard stop to this particular API misuse.\n>> \n>> It will also establish scaffolding to address current fundamental\n>> limitations in the progress output: The current output must be\n>> \"driven\" by calls to the likes of display_progress().\n>\n> Please elaborate why that is a \"fundamental limitation\"; I don't see\n> any drawback of the current approach.\n>\n>> Once we have a\n>> global current progress object we'll be able to update that object via\n>> SIGALRM.\n>\n> What are the supposed benefits of doing that?  I do see its drawbacks,\n> considering that we have progress lines that are updated from multiple\n> threads.\n\nI've updated the commit messages in a re-roll I have incoming to\nhopefully clear this up.\n\n>> See [3] for early code to do that.\n>> \n>> It's conceivable that this change will hit the BUG() condition in some\n>> scenario that we don't currently have tests for, this would be very\n>> bad. If that happened we'd die just because we couldn't emit some\n>> pretty output.\n>> \n>> See [4] for a discussion of why our test coverage is lacking; our\n>> progress display is hidden behind isatty(2) checks in many cases, so\n>> the test suite doesn't cover it unless individual tests are run in\n>> \"--verbose\" mode, we might also have multi-threaded use of the API, so\n>> two progress bars stopping and starting would only be visible due to a\n>> race condition.\n>> \n>> Despite that, I think that this change won't introduce such\n>> regressions, because:\n>> \n>>  1. I've read all the code using the progress API (and have modified a\n>>     large part of it in some WIP code I have). Almost all of it is really\n>>     simple, the parts that aren't[5] are complex in the display_progress() part,\n>>     not in starting or stopping the progress bar.\n>> \n>>  2. The entire test suite passes when instrumented with an ad-hoc\n>>     Linux-specific mode (it uses gettid()) to die if progress bars are\n>>     ever started or stopped on anything but the main thread[6].\n>> \n>>     Extending that to die if display_progress() is called in a thread\n>>     reveals that we have exactly two users of the progress bar under\n>>     threaded conditions, \"git index-pack\" and \"git pack-objects\". Both\n>>     uses are straightforward, and they don't start/stop the progress\n>>     bar when threads are active.\n>> \n>>  3. I've likewise done an ad-hoc test to force progress bars to be\n>>     displayed with:\n>> \n>>         perl -pi -e 's[isatty\\(2\\)][1]g' $(git grep -l -F 'isatty(2)')\n>> \n>>     I.e. to replace all checks (not just for progress) of checking\n>>     whether STDERR is connected to a TTY, and then monkeypatching\n>>     is_foreground_fd() in progress.c to always \"return 1\". Running the\n>>     tests with those applied, interactively and under -V reveals via:\n>> \n>>         $ grep -e set_progress_signal -e clear_progress_signal test-results/*out\n>> \n>>     That nothing our tests cover hits the BUG conditions added here,\n>>     except the expected \"BUG: start two concurrent progress bars\" test\n>>     being added here.\n>> \n>>     That isn't entirely true since we won't be getting 100% coverage\n>>     due to cascading failures from tests that expected no progress\n>>     output on stderr. To make sure I covered 100% I also tried making\n>>     the display() function in progress.c a NOOP on top of that (it's\n>>     the calls to start_progress_delay() and stop_progress()) that\n>>     matter.\n>> \n>>     That doesn't hit the BUG() either. Some tests fail in that mode\n>>     due to a combination of the overzealous isatty(2) munging noted\n>>     above, and the tests that are testing that the progress output\n>>     itself is present (but for testing I'd made display() a NOOP).\n>> \n>> Between those three points I think it's safe to go ahead with this\n>> change.\n>\n> The above analysis only considers _our_ _current_ codebase.\n>\n> However, even though this might be safe now, it doesn't mean that it\n> will remain safe in the future, as we might add new progress lines\n> that lack test coverage (though hopefully won't), and would hit that\n> BUG() at a user.\n>\n> Furthermore, even though this might be safe in our codebase, it\n> doesn't mean that it is safe in some 20+k forks of Git that exist on\n> GitHub alone (I for one have a git command or two with in my fork\n> which output progress lines but, sadly, have zero test coverage).\n>\n> But more importantly, even though it might be safe to do so, that\n> doesn't mean that it's a good idea to do so.  The commit message does\n> little to justify why it is conceptually a good idea to add this BUG()\n> to the progress code in a way that it can trigger outside of the test\n> suite.\n\nI think partially I've addressed this above (i.e. in the incoming\nre-roll's update commit message), but I might not for the question of\nwhether this is worth it overall. I'll update the commit message to\naddress this specific case, which was missing from it.\n\n>> 1. 6f9d5f2fda1 (commit-graph: fix progress of reachable commits, 2020-07-09)\n>> 2. https://lore.kernel.org/git/20210620200303.2328957-3-szeder.dev@gmail.com\n>> 3. https://lore.kernel.org/git/patch-18.25-e21fc66623f-20210623T155626Z-avarab@gmail.com/\n>> 4. https://lore.kernel.org/git/cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com/\n>> 5. b50c37aa44d (Merge branch 'ab/progress-users-adjust-counters' into\n>>    next, 2021-09-10)\n>> 6. https://lore.kernel.org/git/877dffg37n.fsf@evledraar.gmail.com/\n>> \n>> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n>> ---\n>>  progress.c                  | 18 ++++++++++++++----\n>>  t/t0500-progress-display.sh | 11 +++++++++++\n>>  2 files changed, 25 insertions(+), 4 deletions(-)\n>> \n>> diff --git a/progress.c b/progress.c\n>> index b9369e9a264..a31500f8b2b 100644\n>> --- a/progress.c\n>> +++ b/progress.c\n>> @@ -46,6 +46,7 @@ struct progress {\n>>  };\n>>  \n>>  static volatile sig_atomic_t progress_update;\n>> +static struct progress *global_progress;\n>>  \n>>  /*\n>>   * These are only intended for testing the progress output, i.e. exclusively\n>> @@ -219,11 +220,16 @@ void progress_test_force_update(void)\n>>  \tprogress_interval(SIGALRM);\n>>  }\n>>  \n>> -static void set_progress_signal(void)\n>> +static void set_progress_signal(struct progress *progress)\n>>  {\n>>  \tstruct sigaction sa;\n>>  \tstruct itimerval v;\n>>  \n>> +\tif (global_progress)\n>> +\t\tBUG(\"'%s' progress still active when trying to start '%s'\",\n>> +\t\t    global_progress->title, progress->title);\n>> +\tglobal_progress = progress;\n>\n> This function is called set_progress_signal(), so checking and setting\n> 'global_progress' feels out of place here; it would be better to do\n> that in start_progress_delay().\n>\n>> +\n>>  \tif (progress_testing)\n>>  \t\treturn;\n>>  \n>> @@ -241,10 +247,14 @@ static void set_progress_signal(void)\n>>  \tsetitimer(ITIMER_REAL, &v, NULL);\n>>  }\n>>  \n>> -static void clear_progress_signal(void)\n>> +static void clear_progress_signal(struct progress *progress)\n>>  {\n>>  \tstruct itimerval v = {{0,},};\n>>  \n>> +\tif (!global_progress)\n>> +\t\tBUG(\"should have active global_progress when cleaning up\");\n>> +\tglobal_progress = NULL;\n>\n> Likewise.\n\n*Nod* cleaned up this whole part, which became much simpler overall as a\nresult, thank you.\n"},{"id":"439528","messageId":"cover-v4-0.8-00000000000-20211025T111915Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v3-00.10-00000000000-20211013T222329Z-avarab@gmail.com","subject":"[PATCH v4 0/8] progress: assert \"global_progress\" + test fixes / cleanup","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-25T11:24:59Z","receivedAt":"2021-10-25T11:25:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This series fixes various issues in and related to progress.c, and\nadds a BUG() assertion for us not starting two progress bars at the\nsame time. Those changes are needed for subsequent changes that do\nmore interesting things with this new global progress bar.\n\nI think this should address all the feedback on v3[1]. Changes:\n\n * Clarified the whole string_list_insert() in test-progress.c. I\n   added a wrapper function with a comment to make that code flow\n   clearer, and as a bonus could drop braces and a temporary variable\n   from the main function as a result.\n\n * I elaborated on the \"limitations in the progress output\" part of\n   the commit message, and tried to address the safety concerns SZEDER\n   expressed in [2]. I think we might still disagree on whether 8/8 is\n   worth it overall, but hopefully this addreses any outstanding\n   questions about the approach & why it's being taken.\n\n * SZEDER pointed out that I should move the BUG() assert out of the\n   signal handlers, there's now a couple of small helpers to do that,\n   which also allowed for ejecting a couple of earlier refactoring\n   commits (code moves as they were \"static\").\n\n1. https://lore.kernel.org/git/cover-v3-00.10-00000000000-20211013T222329Z-avarab@gmail.com/\n2. https://lore.kernel.org/git/20211025050202.GC2101@szeder.dev/\n\nÆvar Arnfjörð Bjarmason (8):\n  leak tests: fix a memory leaks in \"test-progress\" helper\n  progress.c test helper: add missing braces\n  progress.c tests: make start/stop commands on stdin\n  progress.c tests: test some invalid usage\n  progress.c: add temporary variable from progress struct\n  pack-bitmap-write.c: don't return without stop_progress()\n  various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO)\n  progress.c: add & assert a \"global_progress\" variable\n\n builtin/bisect--helper.c    |   2 +-\n builtin/bundle.c            |   2 +-\n compat/mingw.c              |   2 +-\n pack-bitmap-write.c         |   6 +--\n progress.c                  |  23 +++++++-\n t/helper/test-progress.c    |  52 +++++++++++++-----\n t/t0500-progress-display.sh | 105 ++++++++++++++++++++++++++++--------\n 7 files changed, 150 insertions(+), 42 deletions(-)\n\nRange-diff against v3:\n 1:  40f7c438a1e =  1:  a3bd032d1eb leak tests: fix a memory leaks in \"test-progress\" helper\n 2:  ee177d253a8 =  2:  e441cfea7c5 progress.c test helper: add missing braces\n 3:  045d58d8201 !  3:  1c5f9bdfe6d progress.c tests: make start/stop verbs on stdin\n    @@ Metadata\n     Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n     \n      ## Commit message ##\n    -    progress.c tests: make start/stop verbs on stdin\n    +    progress.c tests: make start/stop commands on stdin\n     \n         Change the usage of the \"test-tool progress\" introduced in\n         2bb74b53a49 (Test the progress display, 2019-09-16) to take command\n    @@ t/helper/test-progress.c\n      #include \"progress.h\"\n      #include \"strbuf.h\"\n     +#include \"string-list.h\"\n    ++\n    ++/*\n    ++ * We can't use \"end + 1\" as an argument to start_progress() below, it\n    ++ * doesn't xstrdup() its \"title\" argument. We need to hold onto a\n    ++ * valid \"char *\" for it until the end.\n    ++ */\n    ++static char *dup_title(struct string_list *titles, const char *title)\n    ++{\n    ++\treturn string_list_insert(titles, title)->string;\n    ++}\n      \n      int cmd__progress(int argc, const char **argv)\n      {\n     -\tint total = 0;\n     -\tconst char *title;\n     +\tconst char *const default_title = \"Working hard\";\n    -+\tstruct string_list list = STRING_LIST_INIT_DUP;\n    -+\tconst struct string_list_item *item;\n    ++\tstruct string_list titles = STRING_LIST_INIT_DUP;\n      \tstruct strbuf line = STRBUF_INIT;\n     -\tstruct progress *progress;\n     +\tstruct progress *progress = NULL;\n    @@ t/helper/test-progress.c\n     -\t\tif (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n     +\t\tif (skip_prefix(line.buf, \"start \", (const char **) &end)) {\n     +\t\t\tuint64_t total = strtoull(end, &end, 10);\n    -+\t\t\tif (*end == '\\0') {\n    ++\t\t\tif (*end == '\\0')\n     +\t\t\t\tprogress = start_progress(default_title, total);\n    -+\t\t\t} else if (*end == ' ') {\n    -+\t\t\t\titem = string_list_insert(&list, end + 1);\n    -+\t\t\t\tprogress = start_progress(item->string, total);\n    -+\t\t\t} else {\n    ++\t\t\telse if (*end == ' ')\n    ++\t\t\t\tprogress = start_progress(dup_title(&titles,\n    ++\t\t\t\t\t\t\t\t    end + 1),\n    ++\t\t\t\t\t\t\t  total);\n    ++\t\t\telse\n     +\t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n    -+\t\t\t}\n     +\t\t} else if (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n      \t\t\tuint64_t item_count = strtoull(end, &end, 10);\n      \t\t\tif (*end != '\\0')\n    @@ t/helper/test-progress.c: int cmd__progress(int argc, const char **argv)\n      \t}\n     -\tstop_progress(&progress);\n      \tstrbuf_release(&line);\n    -+\tstring_list_clear(&list, 0);\n    ++\tstring_list_clear(&titles, 0);\n      \n      \treturn 0;\n      }\n 4:  efc0ec360cc =  4:  474ce31f9d2 progress.c tests: test some invalid usage\n 5:  9e36f03de46 <  -:  ----------- progress.c: move signal handler functions lower\n 6:  c7c3843564e <  -:  ----------- progress.c: call progress_interval() from progress_test_force_update()\n 7:  cd2d27b1626 =  5:  ff039742148 progress.c: add temporary variable from progress struct\n 8:  e0a3510dd88 =  6:  3dfe31decff pack-bitmap-write.c: don't return without stop_progress()\n 9:  2cf14881ecf =  7:  8a18eb40fae various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO)\n10:  01d5bbfce76 !  8:  06124e8ac5e progress.c: add & assert a \"global_progress\" variable\n    @@ Commit message\n         limitations in the progress output: The current output must be\n         \"driven\" by calls to the likes of display_progress(). Once we have a\n         global current progress object we'll be able to update that object via\n    -    SIGALRM. See [3] for early code to do that.\n    +    SIGALRM, this will cover cases where we're busy, but either haven't\n    +    invoked our first display_progress() yet, or the time between\n    +    display_progress() is too long. See [3] for early code to do that.\n     \n         It's conceivable that this change will hit the BUG() condition in some\n         scenario that we don't currently have tests for, this would be very\n    @@ Commit message\n             above, and the tests that are testing that the progress output\n             itself is present (but for testing I'd made display() a NOOP).\n     \n    -    Between those three points I think it's safe to go ahead with this\n    +    This doesn't address any currently out-of-tree user of progress.c,\n    +    i.e. WIP patches, or progress output that's a part of forks of\n    +    git.git. Those hopefully have test coverage that would expose the\n    +    BUG().\n    +\n    +    If they don't they'll either run into it in code that displays more\n    +    than one progress bar for the lifetime of the progress, or which calls\n    +    stop_progress() with a non-NULL \"progress\" without a corresponding\n    +    start_progress(). Both of those cases are less likely than the general\n    +    cases of progress.c API misuse.\n    +\n    +    Between those three points above and the discussion of how this could\n    +    impact out-of-tree users I think it's safe to go ahead with this\n         change.\n     \n         1. 6f9d5f2fda1 (commit-graph: fix progress of reachable commits, 2020-07-09)\n    @@ progress.c: struct progress {\n      \n      /*\n       * These are only intended for testing the progress output, i.e. exclusively\n    -@@ progress.c: void progress_test_force_update(void)\n    - \tprogress_interval(SIGALRM);\n    +@@ progress.c: void display_progress(struct progress *progress, uint64_t n)\n    + \t\tdisplay(progress, n, NULL);\n      }\n      \n    --static void set_progress_signal(void)\n    -+static void set_progress_signal(struct progress *progress)\n    - {\n    - \tstruct sigaction sa;\n    - \tstruct itimerval v;\n    - \n    ++static void set_global_progress(struct progress *progress)\n    ++{\n     +\tif (global_progress)\n     +\t\tBUG(\"'%s' progress still active when trying to start '%s'\",\n     +\t\t    global_progress->title, progress->title);\n     +\tglobal_progress = progress;\n    ++}\n     +\n    - \tif (progress_testing)\n    - \t\treturn;\n    - \n    -@@ progress.c: static void set_progress_signal(void)\n    - \tsetitimer(ITIMER_REAL, &v, NULL);\n    - }\n    - \n    --static void clear_progress_signal(void)\n    -+static void clear_progress_signal(struct progress *progress)\n    + static struct progress *start_progress_delay(const char *title, uint64_t total,\n    + \t\t\t\t\t     unsigned delay, unsigned sparse)\n      {\n    - \tstruct itimerval v = {{0,},};\n    - \n    -+\tif (!global_progress)\n    -+\t\tBUG(\"should have active global_progress when cleaning up\");\n    -+\tglobal_progress = NULL;\n    -+\n    - \tif (progress_testing)\n    - \t\treturn;\n    - \n     @@ progress.c: static struct progress *start_progress_delay(const char *title, uint64_t total,\n      \tstrbuf_init(&progress->counters_sb, 0);\n      \tprogress->title_len = utf8_strwidth(title);\n      \tprogress->split = 0;\n    --\tset_progress_signal();\n    -+\tset_progress_signal(progress);\n    ++\tset_global_progress(progress);\n    + \tset_progress_signal();\n      \ttrace2_region_enter(\"progress\", title, the_repository);\n      \treturn progress;\n    +@@ progress.c: void stop_progress(struct progress **p_progress)\n    + \tstop_progress_msg(p_progress, _(\"done\"));\n      }\n    + \n    ++static void unset_global_progress(void)\n    ++{\n    ++\tif (!global_progress)\n    ++\t\tBUG(\"should have active global_progress when cleaning up\");\n    ++\tglobal_progress = NULL;\n    ++}\n    ++\n    + void stop_progress_msg(struct progress **p_progress, const char *msg)\n    + {\n    + \tstruct progress *progress;\n     @@ progress.c: void stop_progress_msg(struct progress **p_progress, const char *msg)\n    - \t\tdisplay(progress, progress->last_value, buf);\n      \t\tfree(buf);\n      \t}\n    --\tclear_progress_signal();\n    -+\tclear_progress_signal(progress);\n    + \tclear_progress_signal();\n    ++\tunset_global_progress();\n      \tstrbuf_release(&progress->counters_sb);\n      \tif (progress->throughput)\n      \t\tstrbuf_release(&progress->throughput->display);\n-- \n2.33.1.1510.ge5c82eefb93\n\n"},{"id":"439529","messageId":"patch-v4-1.8-a3bd032d1eb-20211025T111915Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v4-0.8-00000000000-20211025T111915Z-avarab@gmail.com","subject":"[PATCH v4 1/8] leak tests: fix a memory leaks in \"test-progress\" helper","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-25T11:25:00Z","receivedAt":"2021-10-25T11:25:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a memory leak in the test-progress helper, and mark the\ncorresponding \"t0500-progress-display.sh\" test as being leak-free\nunder SANITIZE=leak. This fixes a leak added in 2bb74b53a4 (Test the\nprogress display, 2019-09-16).\n\nMy 48f68715b14 (tr2: stop leaking \"thread_name\" memory, 2021-08-27)\nhad fixed another memory leak in this test (as it did some trace2\ntesting).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/helper/test-progress.c    | 1 +\n t/t0500-progress-display.sh | 1 +\n 2 files changed, 2 insertions(+)\n\ndiff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\nindex 5d05cbe7894..9265e6ab7cf 100644\n--- a/t/helper/test-progress.c\n+++ b/t/helper/test-progress.c\n@@ -69,6 +69,7 @@ int cmd__progress(int argc, const char **argv)\n \t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n \t}\n \tstop_progress(&progress);\n+\tstrbuf_release(&line);\n \n \treturn 0;\n }\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 22058b503ac..f37cf2eb9c9 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -2,6 +2,7 @@\n \n test_description='progress display'\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n show_cr () {\n-- \n2.33.1.1510.ge5c82eefb93\n\n"},{"id":"439530","messageId":"patch-v4-2.8-e441cfea7c5-20211025T111915Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v4-0.8-00000000000-20211025T111915Z-avarab@gmail.com","subject":"[PATCH v4 2/8] progress.c test helper: add missing braces","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-25T11:25:01Z","receivedAt":"2021-10-25T11:25:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"If we have braces on one arm of an if/else all of them should have it,\nper the CodingGuidelines's \"When there are multiple arms to a\nconditional[...]\" advice. This formatting change makes a subsequent\ncommit smaller.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/helper/test-progress.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\nindex 9265e6ab7cf..50fd3be3dad 100644\n--- a/t/helper/test-progress.c\n+++ b/t/helper/test-progress.c\n@@ -63,10 +63,11 @@ int cmd__progress(int argc, const char **argv)\n \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n \t\t\tprogress_test_ns = test_ms * 1000 * 1000;\n \t\t\tdisplay_throughput(progress, byte_count);\n-\t\t} else if (!strcmp(line.buf, \"update\"))\n+\t\t} else if (!strcmp(line.buf, \"update\")) {\n \t\t\tprogress_test_force_update();\n-\t\telse\n+\t\t} else {\n \t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n+\t\t}\n \t}\n \tstop_progress(&progress);\n \tstrbuf_release(&line);\n-- \n2.33.1.1510.ge5c82eefb93\n\n"},{"id":"439531","messageId":"patch-v4-3.8-1c5f9bdfe6d-20211025T111915Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v4-0.8-00000000000-20211025T111915Z-avarab@gmail.com","subject":"[PATCH v4 3/8] progress.c tests: make start/stop commands on stdin","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-25T11:25:02Z","receivedAt":"2021-10-25T11:25:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the usage of the \"test-tool progress\" introduced in\n2bb74b53a49 (Test the progress display, 2019-09-16) to take command\nlike \"start\" and \"stop\" on stdin, instead of running them implicitly.\n\nThis makes for tests that are easier to read, since the recipe will\nmirror the API usage, and allows for easily testing invalid usage that\nwould yield (or should yield) a BUG(), e.g. providing two \"start\"\ncalls in a row. A subsequent commit will add such tests.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/helper/test-progress.c    | 46 ++++++++++++++++++++++-------\n t/t0500-progress-display.sh | 58 +++++++++++++++++++++++--------------\n 2 files changed, 72 insertions(+), 32 deletions(-)\n\ndiff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\nindex 50fd3be3dad..1435c28e950 100644\n--- a/t/helper/test-progress.c\n+++ b/t/helper/test-progress.c\n@@ -3,6 +3,9 @@\n  *\n  * Reads instructions from standard input, one instruction per line:\n  *\n+ *   \"start <total>[ <title>]\" - Call start_progress(title, total),\n+ *                               Uses the default title of \"Working hard\"\n+ *                               if the \" <title>\" is omitted.\n  *   \"progress <items>\" - Call display_progress() with the given item count\n  *                        as parameter.\n  *   \"throughput <bytes> <millis> - Call display_throughput() with the given\n@@ -10,6 +13,7 @@\n  *                                  specify the time elapsed since the\n  *                                  start_progress() call.\n  *   \"update\" - Set the 'progress_update' flag.\n+ *   \"stop\" - Call stop_progress().\n  *\n  * See 't0500-progress-display.sh' for examples.\n  */\n@@ -19,34 +23,52 @@\n #include \"parse-options.h\"\n #include \"progress.h\"\n #include \"strbuf.h\"\n+#include \"string-list.h\"\n+\n+/*\n+ * We can't use \"end + 1\" as an argument to start_progress() below, it\n+ * doesn't xstrdup() its \"title\" argument. We need to hold onto a\n+ * valid \"char *\" for it until the end.\n+ */\n+static char *dup_title(struct string_list *titles, const char *title)\n+{\n+\treturn string_list_insert(titles, title)->string;\n+}\n \n int cmd__progress(int argc, const char **argv)\n {\n-\tint total = 0;\n-\tconst char *title;\n+\tconst char *const default_title = \"Working hard\";\n+\tstruct string_list titles = STRING_LIST_INIT_DUP;\n \tstruct strbuf line = STRBUF_INIT;\n-\tstruct progress *progress;\n+\tstruct progress *progress = NULL;\n \n \tconst char *usage[] = {\n-\t\t\"test-tool progress [--total=<n>] <progress-title>\",\n+\t\t\"test-tool progress <stdin\",\n \t\tNULL\n \t};\n \tstruct option options[] = {\n-\t\tOPT_INTEGER(0, \"total\", &total, \"total number of items\"),\n \t\tOPT_END(),\n \t};\n \n \targc = parse_options(argc, argv, NULL, options, usage, 0);\n-\tif (argc != 1)\n-\t\tdie(\"need a title for the progress output\");\n-\ttitle = argv[0];\n+\tif (argc)\n+\t\tusage_with_options(usage, options);\n \n \tprogress_testing = 1;\n-\tprogress = start_progress(title, total);\n \twhile (strbuf_getline(&line, stdin) != EOF) {\n \t\tchar *end;\n \n-\t\tif (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n+\t\tif (skip_prefix(line.buf, \"start \", (const char **) &end)) {\n+\t\t\tuint64_t total = strtoull(end, &end, 10);\n+\t\t\tif (*end == '\\0')\n+\t\t\t\tprogress = start_progress(default_title, total);\n+\t\t\telse if (*end == ' ')\n+\t\t\t\tprogress = start_progress(dup_title(&titles,\n+\t\t\t\t\t\t\t\t    end + 1),\n+\t\t\t\t\t\t\t  total);\n+\t\t\telse\n+\t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n+\t\t} else if (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n \t\t\tuint64_t item_count = strtoull(end, &end, 10);\n \t\t\tif (*end != '\\0')\n \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n@@ -65,12 +87,14 @@ int cmd__progress(int argc, const char **argv)\n \t\t\tdisplay_throughput(progress, byte_count);\n \t\t} else if (!strcmp(line.buf, \"update\")) {\n \t\t\tprogress_test_force_update();\n+\t\t} else if (!strcmp(line.buf, \"stop\")) {\n+\t\t\tstop_progress(&progress);\n \t\t} else {\n \t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n \t\t}\n \t}\n-\tstop_progress(&progress);\n \tstrbuf_release(&line);\n+\tstring_list_clear(&titles, 0);\n \n \treturn 0;\n }\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex f37cf2eb9c9..27ab4218b01 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -18,6 +18,7 @@ test_expect_success 'simple progress display' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tupdate\n \tprogress 1\n \tupdate\n@@ -26,8 +27,9 @@ test_expect_success 'simple progress display' '\n \tprogress 4\n \tupdate\n \tprogress 5\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -42,11 +44,13 @@ test_expect_success 'progress display with total' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 3\n \tprogress 1\n \tprogress 2\n \tprogress 3\n+\tstop\n \tEOF\n-\ttest-tool progress --total=3 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -63,14 +67,14 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tprogress 100\n \tprogress 1000\n \tprogress 10000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -89,16 +93,16 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tupdate\n \tprogress 1\n \tupdate\n \tprogress 2\n \tprogress 10000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -117,14 +121,14 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tprogress 25000\n \tprogress 50000\n \tprogress 75000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -141,14 +145,14 @@ Working hard.......2.........3.........4.........5.........6.........7.........:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6.........7.........\n \tprogress 25000\n \tprogress 50000\n \tprogress 75000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6.........7.........\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -165,12 +169,14 @@ test_expect_success 'progress shortens - crazy caller' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 1000\n \tprogress 100\n \tprogress 200\n \tprogress 1\n \tprogress 1000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=1000 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -186,6 +192,7 @@ test_expect_success 'progress display with throughput' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tthroughput 102400 1000\n \tupdate\n \tprogress 10\n@@ -198,8 +205,9 @@ test_expect_success 'progress display with throughput' '\n \tthroughput 409600 4000\n \tupdate\n \tprogress 40\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -215,6 +223,7 @@ test_expect_success 'progress display with throughput and total' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 40\n \tthroughput 102400 1000\n \tprogress 10\n \tthroughput 204800 2000\n@@ -223,8 +232,9 @@ test_expect_success 'progress display with throughput and total' '\n \tprogress 30\n \tthroughput 409600 4000\n \tprogress 40\n+\tstop\n \tEOF\n-\ttest-tool progress --total=40 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -240,6 +250,7 @@ test_expect_success 'cover up after throughput shortens' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tthroughput 409600 1000\n \tupdate\n \tprogress 1\n@@ -252,8 +263,9 @@ test_expect_success 'cover up after throughput shortens' '\n \tthroughput 1638400 4000\n \tupdate\n \tprogress 4\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -268,6 +280,7 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tthroughput 1 1000\n \tupdate\n \tprogress 1\n@@ -277,8 +290,9 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \tthroughput 3145728 3000\n \tupdate\n \tprogress 3\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -286,6 +300,7 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \n test_expect_success 'progress generates traces' '\n \tcat >in <<-\\EOF &&\n+\tstart 40\n \tthroughput 102400 1000\n \tupdate\n \tprogress 10\n@@ -298,10 +313,11 @@ test_expect_success 'progress generates traces' '\n \tthroughput 409600 4000\n \tupdate\n \tprogress 40\n+\tstop\n \tEOF\n \n-\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" test-tool progress --total=40 \\\n-\t\t\"Working hard\" <in 2>stderr &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n \n \t# t0212/parse_events.perl intentionally omits regions and data.\n \ttest_region progress \"Working hard\" trace.event &&\n-- \n2.33.1.1510.ge5c82eefb93\n\n"},{"id":"439532","messageId":"patch-v4-4.8-474ce31f9d2-20211025T111915Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v4-0.8-00000000000-20211025T111915Z-avarab@gmail.com","subject":"[PATCH v4 4/8] progress.c tests: test some invalid usage","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-25T11:25:03Z","receivedAt":"2021-10-25T11:25:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Test what happens when we \"stop\" without a \"start\", omit the \"stop\"\nafter a \"start\", or try to start two concurrent progress bars. This\nextends the trace2 tests added in 98a13647408 (trace2: log progress\ntime and throughput, 2020-05-12).\n\nThese tests are not merely testing the helper, but invalid API usage\nthat can happen if the progress.c API is misused.\n\nThe \"without stop\" test will leak under SANITIZE=leak, since this\nbuggy use of the API will leak memory. But let's not skip it entirely,\nor use the \"!SANITIZE_LEAK\" prerequisite check as we'd do with tests\nthat we're skipping due to leaks we haven't fixed yet. Instead\nannotate the specific command that should skip leak checking with\ncustom $LSAN_OPTIONS[1].\n\n1. https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/t0500-progress-display.sh | 35 +++++++++++++++++++++++++++++++++++\n 1 file changed, 35 insertions(+)\n\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 27ab4218b01..59e9f226ea4 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -325,4 +325,39 @@ test_expect_success 'progress generates traces' '\n \tgrep \"\\\"key\\\":\\\"total_bytes\\\",\\\"value\\\":\\\"409600\\\"\" trace.event\n '\n \n+test_expect_success 'progress generates traces: stop / start' '\n+\tcat >in <<-\\EOF &&\n+\tstart 0\n+\tstop\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-startstop.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n+\ttest_region progress \"Working hard\" trace-startstop.event\n+'\n+\n+test_expect_success 'progress generates traces: start without stop' '\n+\tcat >in <<-\\EOF &&\n+\tstart 0\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-start.event\" \\\n+\tLSAN_OPTIONS=detect_leaks=0 \\\n+\ttest-tool progress \\\n+\t\t<in 2>stderr &&\n+\tgrep region_enter.*progress trace-start.event &&\n+\t! grep region_leave.*progress trace-start.event\n+'\n+\n+test_expect_success 'progress generates traces: stop without start' '\n+\tcat >in <<-\\EOF &&\n+\tstop\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-stop.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n+\t! grep region_enter.*progress trace-stop.event &&\n+\t! grep region_leave.*progress trace-stop.event\n+'\n+\n test_done\n-- \n2.33.1.1510.ge5c82eefb93\n\n"},{"id":"439533","messageId":"patch-v4-6.8-3dfe31decff-20211025T111915Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v4-0.8-00000000000-20211025T111915Z-avarab@gmail.com","subject":"[PATCH v4 6/8] pack-bitmap-write.c: don't return without stop_progress()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-25T11:25:05Z","receivedAt":"2021-10-25T11:25:23Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a bug that's been here since 7cc8f971085 (pack-objects: implement\nbitmap writing, 2013-12-21), we did not call stop_progress() if we\nreached the early exit in this function.\n\nWe could call stop_progress() before we return, but better yet is to\ndefer calling start_progress() until we need it.\n\nThis will matter in a subsequent commit where we BUG(...) out if this\nhappens, and matters now e.g. because we don't have a corresponding\n\"region_end\" for the progress trace2 event.\n\nSuggested-by: SZEDER Gábor <szeder.dev@gmail.com>\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n pack-bitmap-write.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 9c55c1531e1..cab3eaa2acd 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -575,15 +575,15 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \n \tQSORT(indexed_commits, indexed_commits_nr, date_compare);\n \n-\tif (writer.show_progress)\n-\t\twriter.progress = start_progress(\"Selecting bitmap commits\", 0);\n-\n \tif (indexed_commits_nr < 100) {\n \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n \t\t\tpush_bitmapped_commit(indexed_commits[i]);\n \t\treturn;\n \t}\n \n+\tif (writer.show_progress)\n+\t\twriter.progress = start_progress(\"Selecting bitmap commits\", 0);\n+\n \tfor (;;) {\n \t\tstruct commit *chosen = NULL;\n \n-- \n2.33.1.1510.ge5c82eefb93\n\n"},{"id":"439534","messageId":"patch-v4-5.8-ff039742148-20211025T111915Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v4-0.8-00000000000-20211025T111915Z-avarab@gmail.com","subject":"[PATCH v4 5/8] progress.c: add temporary variable from progress struct","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-25T11:25:04Z","receivedAt":"2021-10-25T11:25:24Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a temporary \"progress\" variable for the dereferenced p_progress\npointer to a \"struct progress *\". Before 98a13647408 (trace2: log\nprogress time and throughput, 2020-05-12) we didn't dereference\n\"p_progress\" in this function, now that we do it's easier to read the\ncode if we work with a \"progress\" struct pointer like everywhere else,\ninstead of a pointer to a pointer.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 680c6a8bf93..76a95cb7322 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -325,15 +325,16 @@ void stop_progress(struct progress **p_progress)\n \tfinish_if_sparse(*p_progress);\n \n \tif (*p_progress) {\n+\t\tstruct progress *progress = *p_progress;\n \t\ttrace2_data_intmax(\"progress\", the_repository, \"total_objects\",\n \t\t\t\t   (*p_progress)->total);\n \n \t\tif ((*p_progress)->throughput)\n \t\t\ttrace2_data_intmax(\"progress\", the_repository,\n \t\t\t\t\t   \"total_bytes\",\n-\t\t\t\t\t   (*p_progress)->throughput->curr_total);\n+\t\t\t\t\t   progress->throughput->curr_total);\n \n-\t\ttrace2_region_leave(\"progress\", (*p_progress)->title, the_repository);\n+\t\ttrace2_region_leave(\"progress\", progress->title, the_repository);\n \t}\n \n \tstop_progress_msg(p_progress, _(\"done\"));\n-- \n2.33.1.1510.ge5c82eefb93\n\n"},{"id":"439535","messageId":"patch-v4-7.8-8a18eb40fae-20211025T111915Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v4-0.8-00000000000-20211025T111915Z-avarab@gmail.com","subject":"[PATCH v4 7/8] various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-25T11:25:06Z","receivedAt":"2021-10-25T11:25:26Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"We have over 50 uses of \"isatty(1)\" and \"isatty(2)\" in the codebase,\nonly these two used the stdlib.h macros to refer to them.\n\nLet's change these for consistency, and because a subsequent commit's\ncommit message outlines a recipe to change all of these for ad-hoc\ntesting, not needing to match these with that ad-hoc regex will make\nthings easier to explain.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n builtin/bisect--helper.c | 2 +-\n builtin/bundle.c         | 2 +-\n compat/mingw.c           | 2 +-\n 3 files changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/bisect--helper.c b/builtin/bisect--helper.c\nindex 28a2e6a5750..1727cb051fb 100644\n--- a/builtin/bisect--helper.c\n+++ b/builtin/bisect--helper.c\n@@ -830,7 +830,7 @@ static int bisect_autostart(struct bisect_terms *terms)\n \tfprintf_ln(stderr, _(\"You need to start by \\\"git bisect \"\n \t\t\t  \"start\\\"\\n\"));\n \n-\tif (!isatty(STDIN_FILENO))\n+\tif (!isatty(1))\n \t\treturn -1;\n \n \t/*\ndiff --git a/builtin/bundle.c b/builtin/bundle.c\nindex 5a85d7cd0fe..df69c651753 100644\n--- a/builtin/bundle.c\n+++ b/builtin/bundle.c\n@@ -56,7 +56,7 @@ static int parse_options_cmd_bundle(int argc,\n \n static int cmd_bundle_create(int argc, const char **argv, const char *prefix) {\n \tint all_progress_implied = 0;\n-\tint progress = isatty(STDERR_FILENO);\n+\tint progress = isatty(2);\n \tstruct strvec pack_opts;\n \tint version = -1;\n \tint ret;\ndiff --git a/compat/mingw.c b/compat/mingw.c\nindex 9e0cd1e097f..0f545c1a7d1 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw.c\n@@ -2374,7 +2374,7 @@ int mingw_raise(int sig)\n \tswitch (sig) {\n \tcase SIGALRM:\n \t\tif (timer_fn == SIG_DFL) {\n-\t\t\tif (isatty(STDERR_FILENO))\n+\t\t\tif (isatty(2))\n \t\t\t\tfputs(\"Alarm clock\\n\", stderr);\n \t\t\texit(128 + SIGALRM);\n \t\t} else if (timer_fn != SIG_IGN)\n-- \n2.33.1.1510.ge5c82eefb93\n\n"},{"id":"439536","messageId":"patch-v4-8.8-06124e8ac5e-20211025T111915Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v4-0.8-00000000000-20211025T111915Z-avarab@gmail.com","subject":"[PATCH v4 8/8] progress.c: add & assert a \"global_progress\" variable","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-25T11:25:07Z","receivedAt":"2021-10-25T11:25:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The progress.c code makes a hard assumption that only one progress bar\nbe active at a time (see [1] for a bug where this wasn't the\ncase). Add a BUG() that'll trigger if we ever regress on that promise\nand have two progress bars active at the same time.\n\nThere was an alternative test-only approach to doing the same\nthing[2], but by doing this outside of a GIT_TEST_* mode we'll know\nwe've put a hard stop to this particular API misuse.\n\nIt will also establish scaffolding to address current fundamental\nlimitations in the progress output: The current output must be\n\"driven\" by calls to the likes of display_progress(). Once we have a\nglobal current progress object we'll be able to update that object via\nSIGALRM, this will cover cases where we're busy, but either haven't\ninvoked our first display_progress() yet, or the time between\ndisplay_progress() is too long. See [3] for early code to do that.\n\nIt's conceivable that this change will hit the BUG() condition in some\nscenario that we don't currently have tests for, this would be very\nbad. If that happened we'd die just because we couldn't emit some\npretty output.\n\nSee [4] for a discussion of why our test coverage is lacking; our\nprogress display is hidden behind isatty(2) checks in many cases, so\nthe test suite doesn't cover it unless individual tests are run in\n\"--verbose\" mode, we might also have multi-threaded use of the API, so\ntwo progress bars stopping and starting would only be visible due to a\nrace condition.\n\nDespite that, I think that this change won't introduce such\nregressions, because:\n\n 1. I've read all the code using the progress API (and have modified a\n    large part of it in some WIP code I have). Almost all of it is really\n    simple, the parts that aren't[5] are complex in the display_progress() part,\n    not in starting or stopping the progress bar.\n\n 2. The entire test suite passes when instrumented with an ad-hoc\n    Linux-specific mode (it uses gettid()) to die if progress bars are\n    ever started or stopped on anything but the main thread[6].\n\n    Extending that to die if display_progress() is called in a thread\n    reveals that we have exactly two users of the progress bar under\n    threaded conditions, \"git index-pack\" and \"git pack-objects\". Both\n    uses are straightforward, and they don't start/stop the progress\n    bar when threads are active.\n\n 3. I've likewise done an ad-hoc test to force progress bars to be\n    displayed with:\n\n        perl -pi -e 's[isatty\\(2\\)][1]g' $(git grep -l -F 'isatty(2)')\n\n    I.e. to replace all checks (not just for progress) of checking\n    whether STDERR is connected to a TTY, and then monkeypatching\n    is_foreground_fd() in progress.c to always \"return 1\". Running the\n    tests with those applied, interactively and under -V reveals via:\n\n        $ grep -e set_progress_signal -e clear_progress_signal test-results/*out\n\n    That nothing our tests cover hits the BUG conditions added here,\n    except the expected \"BUG: start two concurrent progress bars\" test\n    being added here.\n\n    That isn't entirely true since we won't be getting 100% coverage\n    due to cascading failures from tests that expected no progress\n    output on stderr. To make sure I covered 100% I also tried making\n    the display() function in progress.c a NOOP on top of that (it's\n    the calls to start_progress_delay() and stop_progress()) that\n    matter.\n\n    That doesn't hit the BUG() either. Some tests fail in that mode\n    due to a combination of the overzealous isatty(2) munging noted\n    above, and the tests that are testing that the progress output\n    itself is present (but for testing I'd made display() a NOOP).\n\nThis doesn't address any currently out-of-tree user of progress.c,\ni.e. WIP patches, or progress output that's a part of forks of\ngit.git. Those hopefully have test coverage that would expose the\nBUG().\n\nIf they don't they'll either run into it in code that displays more\nthan one progress bar for the lifetime of the progress, or which calls\nstop_progress() with a non-NULL \"progress\" without a corresponding\nstart_progress(). Both of those cases are less likely than the general\ncases of progress.c API misuse.\n\nBetween those three points above and the discussion of how this could\nimpact out-of-tree users I think it's safe to go ahead with this\nchange.\n\n1. 6f9d5f2fda1 (commit-graph: fix progress of reachable commits, 2020-07-09)\n2. https://lore.kernel.org/git/20210620200303.2328957-3-szeder.dev@gmail.com\n3. https://lore.kernel.org/git/patch-18.25-e21fc66623f-20210623T155626Z-avarab@gmail.com/\n4. https://lore.kernel.org/git/cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com/\n5. b50c37aa44d (Merge branch 'ab/progress-users-adjust-counters' into\n   next, 2021-09-10)\n6. https://lore.kernel.org/git/877dffg37n.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c                  | 18 ++++++++++++++++++\n t/t0500-progress-display.sh | 11 +++++++++++\n 2 files changed, 29 insertions(+)\n\ndiff --git a/progress.c b/progress.c\nindex 76a95cb7322..7483aec2e2a 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -46,6 +46,7 @@ struct progress {\n };\n \n static volatile sig_atomic_t progress_update;\n+static struct progress *global_progress;\n \n /*\n  * These are only intended for testing the progress output, i.e. exclusively\n@@ -249,6 +250,14 @@ void display_progress(struct progress *progress, uint64_t n)\n \t\tdisplay(progress, n, NULL);\n }\n \n+static void set_global_progress(struct progress *progress)\n+{\n+\tif (global_progress)\n+\t\tBUG(\"'%s' progress still active when trying to start '%s'\",\n+\t\t    global_progress->title, progress->title);\n+\tglobal_progress = progress;\n+}\n+\n static struct progress *start_progress_delay(const char *title, uint64_t total,\n \t\t\t\t\t     unsigned delay, unsigned sparse)\n {\n@@ -264,6 +273,7 @@ static struct progress *start_progress_delay(const char *title, uint64_t total,\n \tstrbuf_init(&progress->counters_sb, 0);\n \tprogress->title_len = utf8_strwidth(title);\n \tprogress->split = 0;\n+\tset_global_progress(progress);\n \tset_progress_signal();\n \ttrace2_region_enter(\"progress\", title, the_repository);\n \treturn progress;\n@@ -340,6 +350,13 @@ void stop_progress(struct progress **p_progress)\n \tstop_progress_msg(p_progress, _(\"done\"));\n }\n \n+static void unset_global_progress(void)\n+{\n+\tif (!global_progress)\n+\t\tBUG(\"should have active global_progress when cleaning up\");\n+\tglobal_progress = NULL;\n+}\n+\n void stop_progress_msg(struct progress **p_progress, const char *msg)\n {\n \tstruct progress *progress;\n@@ -369,6 +386,7 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)\n \t\tfree(buf);\n \t}\n \tclear_progress_signal();\n+\tunset_global_progress();\n \tstrbuf_release(&progress->counters_sb);\n \tif (progress->throughput)\n \t\tstrbuf_release(&progress->throughput->display);\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 59e9f226ea4..867fdace3f2 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -298,6 +298,17 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \ttest_cmp expect out\n '\n \n+test_expect_success 'BUG: start two concurrent progress bars' '\n+\tcat >in <<-\\EOF &&\n+\tstart 0 one\n+\tstart 0 two\n+\tEOF\n+\n+\ttest_must_fail test-tool progress \\\n+\t\t<in 2>stderr &&\n+\tgrep \"^BUG: .*'\\''one'\\'' progress still active when trying to start '\\''two'\\''$\" stderr\n+'\n+\n test_expect_success 'progress generates traces' '\n \tcat >in <<-\\EOF &&\n \tstart 40\n-- \n2.33.1.1510.ge5c82eefb93\n\n"},{"id":"439537","messageId":"c65c0a54-b66f-a043-01d9-192f9ca1efde@kdbg.org","threadId":"55972","inReplyTo":"patch-v4-8.8-06124e8ac5e-20211025T111915Z-avarab@gmail.com","subject":"Re: [PATCH v4 8/8] progress.c: add & assert a \"global_progress\" variable","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2021-10-25T11:53:45Z","receivedAt":"2021-10-25T11:53:50Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 25.10.21 um 13:25 schrieb Ævar Arnfjörð Bjarmason:\n> Once we have a\n> global current progress object we'll be able to update that object via\n> SIGALRM, this will cover cases where we're busy, but either haven't\n> invoked our first display_progress() yet, or the time between\n> display_progress() is too long. See [3] for early code to do that.\n\nThis can't be a goal. You cannot do serious stuff in a signal handler.\nDon't proliferate complicated signal handlers.\n\nOn top of that, our SIGALRM handler on Windows runs in its own thread\nand can't do more than set a volatile sigatomic_t (and even that is\nundefined behavior, strictly speaking).\n\n-- Hannes\n"},{"id":"439539","messageId":"211025.8635opi8om.gmgdl@evledraar.gmail.com","threadId":"55972","inReplyTo":"c65c0a54-b66f-a043-01d9-192f9ca1efde@kdbg.org","subject":"Re: [PATCH v4 8/8] progress.c: add & assert a \"global_progress\" variable","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-25T12:29:36Z","receivedAt":"2021-10-25T12:35:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Oct 25 2021, Johannes Sixt wrote:\n\n> Am 25.10.21 um 13:25 schrieb Ævar Arnfjörð Bjarmason:\n>> Once we have a\n>> global current progress object we'll be able to update that object via\n>> SIGALRM, this will cover cases where we're busy, but either haven't\n>> invoked our first display_progress() yet, or the time between\n>> display_progress() is too long. See [3] for early code to do that.\n>\n> This can't be a goal. You cannot do serious stuff in a signal handler.\n> Don't proliferate complicated signal handlers.\n\nI'd missed some of the nuances with signal safety when I hacked that up\ninitially. For the purposes of this patch the commit message is\ndiscussing a POC that suggests an interesting direction to go in.\n\nMy current implementation of this which I've run with for some time\nrelies on undefined behavior and glibc being forgiving, but for any\nfuture submission I'll make sure it's signal-safe.\n\nIn particular to format a progress line etc. we we need sprintf() and\nthe like, but to have a \"stalled\" message we can pre-format that at\nstart_progress() time, and inside the signal handler all we'll need is a\nwrite(), which should be safe in most places...\n\n> On top of that, our SIGALRM handler on Windows runs in its own thread\n> and can't do more than set a volatile sigatomic_t (and even that is\n> undefined behavior, strictly speaking).\n\n... but we may need to disable it entirely on some platforms, I didn't\nknow about that Windows edge case.\n\nAnyway, for this patch having the BUG() is useful even if we never went\nfor that.\n"},{"id":"440245","messageId":"cover-v5-0.8-00000000000-20211101T190630Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v4-0.8-00000000000-20211025T111915Z-avarab@gmail.com","subject":"[PATCH v5 0/8] progress: assert \"global_progress\" + test fixes / cleanup","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-01T19:09:06Z","receivedAt":"2021-11-01T19:09:21Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This is a commit-message only update of v4[1]. It was pointed out on\nv4 that linked-to POC code discussed in the commit message wasn't safe\nfor running inside a signal handler.\n\nThat's correct, but achieving the same goals would be possible while\nretaining signal safety on some platforms, so update the commit\nmessage discussion to note that.\n\nIn any case that part of the commit message is discussing future\nside-benefits of the change, the change as it is isn't doing anything\nnovel with signal handlers in the progress.c code.\n\n1. https://lore.kernel.org/git/cover-v4-0.8-00000000000-20211025T111915Z-avarab@gmail.com/\n2. https://lore.kernel.org/git/c65c0a54-b66f-a043-01d9-192f9ca1efde@kdbg.org/\n\nÆvar Arnfjörð Bjarmason (8):\n  leak tests: fix a memory leaks in \"test-progress\" helper\n  progress.c test helper: add missing braces\n  progress.c tests: make start/stop commands on stdin\n  progress.c tests: test some invalid usage\n  progress.c: add temporary variable from progress struct\n  pack-bitmap-write.c: don't return without stop_progress()\n  various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO)\n  progress.c: add & assert a \"global_progress\" variable\n\n builtin/bisect--helper.c    |   2 +-\n builtin/bundle.c            |   2 +-\n compat/mingw.c              |   2 +-\n pack-bitmap-write.c         |   6 +--\n progress.c                  |  23 +++++++-\n t/helper/test-progress.c    |  52 +++++++++++++-----\n t/t0500-progress-display.sh | 105 ++++++++++++++++++++++++++++--------\n 7 files changed, 150 insertions(+), 42 deletions(-)\n\nRange-diff against v4:\n1:  a3bd032d1eb = 1:  0c0e5bfc656 leak tests: fix a memory leaks in \"test-progress\" helper\n2:  e441cfea7c5 = 2:  0ae86e40e16 progress.c test helper: add missing braces\n3:  1c5f9bdfe6d = 3:  767aa010026 progress.c tests: make start/stop commands on stdin\n4:  474ce31f9d2 = 4:  9bcc8cfc9d2 progress.c tests: test some invalid usage\n5:  ff039742148 = 5:  3a95613cd04 progress.c: add temporary variable from progress struct\n6:  3dfe31decff = 6:  0300b802dcb pack-bitmap-write.c: don't return without stop_progress()\n7:  8a18eb40fae = 7:  4795d4835b0 various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO)\n8:  06124e8ac5e ! 8:  e58f4a0c1b9 progress.c: add & assert a \"global_progress\" variable\n    @@ Commit message\n     \n         It will also establish scaffolding to address current fundamental\n         limitations in the progress output: The current output must be\n    -    \"driven\" by calls to the likes of display_progress(). Once we have a\n    -    global current progress object we'll be able to update that object via\n    -    SIGALRM, this will cover cases where we're busy, but either haven't\n    -    invoked our first display_progress() yet, or the time between\n    -    display_progress() is too long. See [3] for early code to do that.\n    +    \"driven\" by calls to the likes of display_progress().\n    +\n    +    Once we have a global current progress object we'll be able to update\n    +    that object via SIGALRM, this will cover cases where we're busy, but\n    +    either haven't invoked our first display_progress() yet, or the time\n    +    between display_progress() is too long. See [3] for early code to do\n    +    that.\n    +\n    +    The linked code in [3] is WIP and not signal-safe since among other\n    +    things it calls sprintf() from within a signal handler, see e.g. \"man\n    +    7 signal-safety\". But on some platforms a real implementation of it\n    +    would be able to write() out a prepared-formatted progress update from\n    +    within a signal handler. That would be sufficient to e.g. show that\n    +    we're \"stalled\", or to display something like a simple pre-formatted\n    +    \"spinner\".\n     \n         It's conceivable that this change will hit the BUG() condition in some\n         scenario that we don't currently have tests for, this would be very\n-- \n2.33.1.1570.g069344fdd45\n\n"},{"id":"440246","messageId":"patch-v5-1.8-0c0e5bfc656-20211101T190630Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v5-0.8-00000000000-20211101T190630Z-avarab@gmail.com","subject":"[PATCH v5 1/8] leak tests: fix a memory leaks in \"test-progress\" helper","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-01T19:09:07Z","receivedAt":"2021-11-01T19:09:23Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a memory leak in the test-progress helper, and mark the\ncorresponding \"t0500-progress-display.sh\" test as being leak-free\nunder SANITIZE=leak. This fixes a leak added in 2bb74b53a4 (Test the\nprogress display, 2019-09-16).\n\nMy 48f68715b14 (tr2: stop leaking \"thread_name\" memory, 2021-08-27)\nhad fixed another memory leak in this test (as it did some trace2\ntesting).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/helper/test-progress.c    | 1 +\n t/t0500-progress-display.sh | 1 +\n 2 files changed, 2 insertions(+)\n\ndiff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\nindex 5d05cbe7894..9265e6ab7cf 100644\n--- a/t/helper/test-progress.c\n+++ b/t/helper/test-progress.c\n@@ -69,6 +69,7 @@ int cmd__progress(int argc, const char **argv)\n \t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n \t}\n \tstop_progress(&progress);\n+\tstrbuf_release(&line);\n \n \treturn 0;\n }\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 22058b503ac..f37cf2eb9c9 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -2,6 +2,7 @@\n \n test_description='progress display'\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n show_cr () {\n-- \n2.33.1.1570.g069344fdd45\n\n"},{"id":"440247","messageId":"patch-v5-2.8-0ae86e40e16-20211101T190630Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v5-0.8-00000000000-20211101T190630Z-avarab@gmail.com","subject":"[PATCH v5 2/8] progress.c test helper: add missing braces","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-01T19:09:08Z","receivedAt":"2021-11-01T19:09:28Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"If we have braces on one arm of an if/else all of them should have it,\nper the CodingGuidelines's \"When there are multiple arms to a\nconditional[...]\" advice. This formatting change makes a subsequent\ncommit smaller.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/helper/test-progress.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\nindex 9265e6ab7cf..50fd3be3dad 100644\n--- a/t/helper/test-progress.c\n+++ b/t/helper/test-progress.c\n@@ -63,10 +63,11 @@ int cmd__progress(int argc, const char **argv)\n \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n \t\t\tprogress_test_ns = test_ms * 1000 * 1000;\n \t\t\tdisplay_throughput(progress, byte_count);\n-\t\t} else if (!strcmp(line.buf, \"update\"))\n+\t\t} else if (!strcmp(line.buf, \"update\")) {\n \t\t\tprogress_test_force_update();\n-\t\telse\n+\t\t} else {\n \t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n+\t\t}\n \t}\n \tstop_progress(&progress);\n \tstrbuf_release(&line);\n-- \n2.33.1.1570.g069344fdd45\n\n"},{"id":"440248","messageId":"patch-v5-3.8-767aa010026-20211101T190630Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v5-0.8-00000000000-20211101T190630Z-avarab@gmail.com","subject":"[PATCH v5 3/8] progress.c tests: make start/stop commands on stdin","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-01T19:09:09Z","receivedAt":"2021-11-01T19:09:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the usage of the \"test-tool progress\" introduced in\n2bb74b53a49 (Test the progress display, 2019-09-16) to take command\nlike \"start\" and \"stop\" on stdin, instead of running them implicitly.\n\nThis makes for tests that are easier to read, since the recipe will\nmirror the API usage, and allows for easily testing invalid usage that\nwould yield (or should yield) a BUG(), e.g. providing two \"start\"\ncalls in a row. A subsequent commit will add such tests.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/helper/test-progress.c    | 46 ++++++++++++++++++++++-------\n t/t0500-progress-display.sh | 58 +++++++++++++++++++++++--------------\n 2 files changed, 72 insertions(+), 32 deletions(-)\n\ndiff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\nindex 50fd3be3dad..1435c28e950 100644\n--- a/t/helper/test-progress.c\n+++ b/t/helper/test-progress.c\n@@ -3,6 +3,9 @@\n  *\n  * Reads instructions from standard input, one instruction per line:\n  *\n+ *   \"start <total>[ <title>]\" - Call start_progress(title, total),\n+ *                               Uses the default title of \"Working hard\"\n+ *                               if the \" <title>\" is omitted.\n  *   \"progress <items>\" - Call display_progress() with the given item count\n  *                        as parameter.\n  *   \"throughput <bytes> <millis> - Call display_throughput() with the given\n@@ -10,6 +13,7 @@\n  *                                  specify the time elapsed since the\n  *                                  start_progress() call.\n  *   \"update\" - Set the 'progress_update' flag.\n+ *   \"stop\" - Call stop_progress().\n  *\n  * See 't0500-progress-display.sh' for examples.\n  */\n@@ -19,34 +23,52 @@\n #include \"parse-options.h\"\n #include \"progress.h\"\n #include \"strbuf.h\"\n+#include \"string-list.h\"\n+\n+/*\n+ * We can't use \"end + 1\" as an argument to start_progress() below, it\n+ * doesn't xstrdup() its \"title\" argument. We need to hold onto a\n+ * valid \"char *\" for it until the end.\n+ */\n+static char *dup_title(struct string_list *titles, const char *title)\n+{\n+\treturn string_list_insert(titles, title)->string;\n+}\n \n int cmd__progress(int argc, const char **argv)\n {\n-\tint total = 0;\n-\tconst char *title;\n+\tconst char *const default_title = \"Working hard\";\n+\tstruct string_list titles = STRING_LIST_INIT_DUP;\n \tstruct strbuf line = STRBUF_INIT;\n-\tstruct progress *progress;\n+\tstruct progress *progress = NULL;\n \n \tconst char *usage[] = {\n-\t\t\"test-tool progress [--total=<n>] <progress-title>\",\n+\t\t\"test-tool progress <stdin\",\n \t\tNULL\n \t};\n \tstruct option options[] = {\n-\t\tOPT_INTEGER(0, \"total\", &total, \"total number of items\"),\n \t\tOPT_END(),\n \t};\n \n \targc = parse_options(argc, argv, NULL, options, usage, 0);\n-\tif (argc != 1)\n-\t\tdie(\"need a title for the progress output\");\n-\ttitle = argv[0];\n+\tif (argc)\n+\t\tusage_with_options(usage, options);\n \n \tprogress_testing = 1;\n-\tprogress = start_progress(title, total);\n \twhile (strbuf_getline(&line, stdin) != EOF) {\n \t\tchar *end;\n \n-\t\tif (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n+\t\tif (skip_prefix(line.buf, \"start \", (const char **) &end)) {\n+\t\t\tuint64_t total = strtoull(end, &end, 10);\n+\t\t\tif (*end == '\\0')\n+\t\t\t\tprogress = start_progress(default_title, total);\n+\t\t\telse if (*end == ' ')\n+\t\t\t\tprogress = start_progress(dup_title(&titles,\n+\t\t\t\t\t\t\t\t    end + 1),\n+\t\t\t\t\t\t\t  total);\n+\t\t\telse\n+\t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n+\t\t} else if (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n \t\t\tuint64_t item_count = strtoull(end, &end, 10);\n \t\t\tif (*end != '\\0')\n \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n@@ -65,12 +87,14 @@ int cmd__progress(int argc, const char **argv)\n \t\t\tdisplay_throughput(progress, byte_count);\n \t\t} else if (!strcmp(line.buf, \"update\")) {\n \t\t\tprogress_test_force_update();\n+\t\t} else if (!strcmp(line.buf, \"stop\")) {\n+\t\t\tstop_progress(&progress);\n \t\t} else {\n \t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n \t\t}\n \t}\n-\tstop_progress(&progress);\n \tstrbuf_release(&line);\n+\tstring_list_clear(&titles, 0);\n \n \treturn 0;\n }\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex f37cf2eb9c9..27ab4218b01 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -18,6 +18,7 @@ test_expect_success 'simple progress display' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tupdate\n \tprogress 1\n \tupdate\n@@ -26,8 +27,9 @@ test_expect_success 'simple progress display' '\n \tprogress 4\n \tupdate\n \tprogress 5\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -42,11 +44,13 @@ test_expect_success 'progress display with total' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 3\n \tprogress 1\n \tprogress 2\n \tprogress 3\n+\tstop\n \tEOF\n-\ttest-tool progress --total=3 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -63,14 +67,14 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tprogress 100\n \tprogress 1000\n \tprogress 10000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -89,16 +93,16 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tupdate\n \tprogress 1\n \tupdate\n \tprogress 2\n \tprogress 10000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -117,14 +121,14 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tprogress 25000\n \tprogress 50000\n \tprogress 75000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -141,14 +145,14 @@ Working hard.......2.........3.........4.........5.........6.........7.........:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6.........7.........\n \tprogress 25000\n \tprogress 50000\n \tprogress 75000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6.........7.........\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -165,12 +169,14 @@ test_expect_success 'progress shortens - crazy caller' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 1000\n \tprogress 100\n \tprogress 200\n \tprogress 1\n \tprogress 1000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=1000 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -186,6 +192,7 @@ test_expect_success 'progress display with throughput' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tthroughput 102400 1000\n \tupdate\n \tprogress 10\n@@ -198,8 +205,9 @@ test_expect_success 'progress display with throughput' '\n \tthroughput 409600 4000\n \tupdate\n \tprogress 40\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -215,6 +223,7 @@ test_expect_success 'progress display with throughput and total' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 40\n \tthroughput 102400 1000\n \tprogress 10\n \tthroughput 204800 2000\n@@ -223,8 +232,9 @@ test_expect_success 'progress display with throughput and total' '\n \tprogress 30\n \tthroughput 409600 4000\n \tprogress 40\n+\tstop\n \tEOF\n-\ttest-tool progress --total=40 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -240,6 +250,7 @@ test_expect_success 'cover up after throughput shortens' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tthroughput 409600 1000\n \tupdate\n \tprogress 1\n@@ -252,8 +263,9 @@ test_expect_success 'cover up after throughput shortens' '\n \tthroughput 1638400 4000\n \tupdate\n \tprogress 4\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -268,6 +280,7 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tthroughput 1 1000\n \tupdate\n \tprogress 1\n@@ -277,8 +290,9 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \tthroughput 3145728 3000\n \tupdate\n \tprogress 3\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -286,6 +300,7 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \n test_expect_success 'progress generates traces' '\n \tcat >in <<-\\EOF &&\n+\tstart 40\n \tthroughput 102400 1000\n \tupdate\n \tprogress 10\n@@ -298,10 +313,11 @@ test_expect_success 'progress generates traces' '\n \tthroughput 409600 4000\n \tupdate\n \tprogress 40\n+\tstop\n \tEOF\n \n-\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" test-tool progress --total=40 \\\n-\t\t\"Working hard\" <in 2>stderr &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n \n \t# t0212/parse_events.perl intentionally omits regions and data.\n \ttest_region progress \"Working hard\" trace.event &&\n-- \n2.33.1.1570.g069344fdd45\n\n"},{"id":"440249","messageId":"patch-v5-4.8-9bcc8cfc9d2-20211101T190630Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v5-0.8-00000000000-20211101T190630Z-avarab@gmail.com","subject":"[PATCH v5 4/8] progress.c tests: test some invalid usage","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-01T19:09:10Z","receivedAt":"2021-11-01T19:09:30Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Test what happens when we \"stop\" without a \"start\", omit the \"stop\"\nafter a \"start\", or try to start two concurrent progress bars. This\nextends the trace2 tests added in 98a13647408 (trace2: log progress\ntime and throughput, 2020-05-12).\n\nThese tests are not merely testing the helper, but invalid API usage\nthat can happen if the progress.c API is misused.\n\nThe \"without stop\" test will leak under SANITIZE=leak, since this\nbuggy use of the API will leak memory. But let's not skip it entirely,\nor use the \"!SANITIZE_LEAK\" prerequisite check as we'd do with tests\nthat we're skipping due to leaks we haven't fixed yet. Instead\nannotate the specific command that should skip leak checking with\ncustom $LSAN_OPTIONS[1].\n\n1. https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/t0500-progress-display.sh | 35 +++++++++++++++++++++++++++++++++++\n 1 file changed, 35 insertions(+)\n\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 27ab4218b01..59e9f226ea4 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -325,4 +325,39 @@ test_expect_success 'progress generates traces' '\n \tgrep \"\\\"key\\\":\\\"total_bytes\\\",\\\"value\\\":\\\"409600\\\"\" trace.event\n '\n \n+test_expect_success 'progress generates traces: stop / start' '\n+\tcat >in <<-\\EOF &&\n+\tstart 0\n+\tstop\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-startstop.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n+\ttest_region progress \"Working hard\" trace-startstop.event\n+'\n+\n+test_expect_success 'progress generates traces: start without stop' '\n+\tcat >in <<-\\EOF &&\n+\tstart 0\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-start.event\" \\\n+\tLSAN_OPTIONS=detect_leaks=0 \\\n+\ttest-tool progress \\\n+\t\t<in 2>stderr &&\n+\tgrep region_enter.*progress trace-start.event &&\n+\t! grep region_leave.*progress trace-start.event\n+'\n+\n+test_expect_success 'progress generates traces: stop without start' '\n+\tcat >in <<-\\EOF &&\n+\tstop\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-stop.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n+\t! grep region_enter.*progress trace-stop.event &&\n+\t! grep region_leave.*progress trace-stop.event\n+'\n+\n test_done\n-- \n2.33.1.1570.g069344fdd45\n\n"},{"id":"440250","messageId":"patch-v5-5.8-3a95613cd04-20211101T190630Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v5-0.8-00000000000-20211101T190630Z-avarab@gmail.com","subject":"[PATCH v5 5/8] progress.c: add temporary variable from progress struct","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-01T19:09:11Z","receivedAt":"2021-11-01T19:09:33Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a temporary \"progress\" variable for the dereferenced p_progress\npointer to a \"struct progress *\". Before 98a13647408 (trace2: log\nprogress time and throughput, 2020-05-12) we didn't dereference\n\"p_progress\" in this function, now that we do it's easier to read the\ncode if we work with a \"progress\" struct pointer like everywhere else,\ninstead of a pointer to a pointer.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 680c6a8bf93..76a95cb7322 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -325,15 +325,16 @@ void stop_progress(struct progress **p_progress)\n \tfinish_if_sparse(*p_progress);\n \n \tif (*p_progress) {\n+\t\tstruct progress *progress = *p_progress;\n \t\ttrace2_data_intmax(\"progress\", the_repository, \"total_objects\",\n \t\t\t\t   (*p_progress)->total);\n \n \t\tif ((*p_progress)->throughput)\n \t\t\ttrace2_data_intmax(\"progress\", the_repository,\n \t\t\t\t\t   \"total_bytes\",\n-\t\t\t\t\t   (*p_progress)->throughput->curr_total);\n+\t\t\t\t\t   progress->throughput->curr_total);\n \n-\t\ttrace2_region_leave(\"progress\", (*p_progress)->title, the_repository);\n+\t\ttrace2_region_leave(\"progress\", progress->title, the_repository);\n \t}\n \n \tstop_progress_msg(p_progress, _(\"done\"));\n-- \n2.33.1.1570.g069344fdd45\n\n"},{"id":"440251","messageId":"patch-v5-6.8-0300b802dcb-20211101T190630Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v5-0.8-00000000000-20211101T190630Z-avarab@gmail.com","subject":"[PATCH v5 6/8] pack-bitmap-write.c: don't return without stop_progress()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-01T19:09:12Z","receivedAt":"2021-11-01T19:09:40Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a bug that's been here since 7cc8f971085 (pack-objects: implement\nbitmap writing, 2013-12-21), we did not call stop_progress() if we\nreached the early exit in this function.\n\nWe could call stop_progress() before we return, but better yet is to\ndefer calling start_progress() until we need it.\n\nThis will matter in a subsequent commit where we BUG(...) out if this\nhappens, and matters now e.g. because we don't have a corresponding\n\"region_end\" for the progress trace2 event.\n\nSuggested-by: SZEDER Gábor <szeder.dev@gmail.com>\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n pack-bitmap-write.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 9c55c1531e1..cab3eaa2acd 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -575,15 +575,15 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \n \tQSORT(indexed_commits, indexed_commits_nr, date_compare);\n \n-\tif (writer.show_progress)\n-\t\twriter.progress = start_progress(\"Selecting bitmap commits\", 0);\n-\n \tif (indexed_commits_nr < 100) {\n \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n \t\t\tpush_bitmapped_commit(indexed_commits[i]);\n \t\treturn;\n \t}\n \n+\tif (writer.show_progress)\n+\t\twriter.progress = start_progress(\"Selecting bitmap commits\", 0);\n+\n \tfor (;;) {\n \t\tstruct commit *chosen = NULL;\n \n-- \n2.33.1.1570.g069344fdd45\n\n"},{"id":"440252","messageId":"patch-v5-8.8-e58f4a0c1b9-20211101T190630Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v5-0.8-00000000000-20211101T190630Z-avarab@gmail.com","subject":"[PATCH v5 8/8] progress.c: add & assert a \"global_progress\" variable","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-01T19:09:14Z","receivedAt":"2021-11-01T19:09:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The progress.c code makes a hard assumption that only one progress bar\nbe active at a time (see [1] for a bug where this wasn't the\ncase). Add a BUG() that'll trigger if we ever regress on that promise\nand have two progress bars active at the same time.\n\nThere was an alternative test-only approach to doing the same\nthing[2], but by doing this outside of a GIT_TEST_* mode we'll know\nwe've put a hard stop to this particular API misuse.\n\nIt will also establish scaffolding to address current fundamental\nlimitations in the progress output: The current output must be\n\"driven\" by calls to the likes of display_progress().\n\nOnce we have a global current progress object we'll be able to update\nthat object via SIGALRM, this will cover cases where we're busy, but\neither haven't invoked our first display_progress() yet, or the time\nbetween display_progress() is too long. See [3] for early code to do\nthat.\n\nThe linked code in [3] is WIP and not signal-safe since among other\nthings it calls sprintf() from within a signal handler, see e.g. \"man\n7 signal-safety\". But on some platforms a real implementation of it\nwould be able to write() out a prepared-formatted progress update from\nwithin a signal handler. That would be sufficient to e.g. show that\nwe're \"stalled\", or to display something like a simple pre-formatted\n\"spinner\".\n\nIt's conceivable that this change will hit the BUG() condition in some\nscenario that we don't currently have tests for, this would be very\nbad. If that happened we'd die just because we couldn't emit some\npretty output.\n\nSee [4] for a discussion of why our test coverage is lacking; our\nprogress display is hidden behind isatty(2) checks in many cases, so\nthe test suite doesn't cover it unless individual tests are run in\n\"--verbose\" mode, we might also have multi-threaded use of the API, so\ntwo progress bars stopping and starting would only be visible due to a\nrace condition.\n\nDespite that, I think that this change won't introduce such\nregressions, because:\n\n 1. I've read all the code using the progress API (and have modified a\n    large part of it in some WIP code I have). Almost all of it is really\n    simple, the parts that aren't[5] are complex in the display_progress() part,\n    not in starting or stopping the progress bar.\n\n 2. The entire test suite passes when instrumented with an ad-hoc\n    Linux-specific mode (it uses gettid()) to die if progress bars are\n    ever started or stopped on anything but the main thread[6].\n\n    Extending that to die if display_progress() is called in a thread\n    reveals that we have exactly two users of the progress bar under\n    threaded conditions, \"git index-pack\" and \"git pack-objects\". Both\n    uses are straightforward, and they don't start/stop the progress\n    bar when threads are active.\n\n 3. I've likewise done an ad-hoc test to force progress bars to be\n    displayed with:\n\n        perl -pi -e 's[isatty\\(2\\)][1]g' $(git grep -l -F 'isatty(2)')\n\n    I.e. to replace all checks (not just for progress) of checking\n    whether STDERR is connected to a TTY, and then monkeypatching\n    is_foreground_fd() in progress.c to always \"return 1\". Running the\n    tests with those applied, interactively and under -V reveals via:\n\n        $ grep -e set_progress_signal -e clear_progress_signal test-results/*out\n\n    That nothing our tests cover hits the BUG conditions added here,\n    except the expected \"BUG: start two concurrent progress bars\" test\n    being added here.\n\n    That isn't entirely true since we won't be getting 100% coverage\n    due to cascading failures from tests that expected no progress\n    output on stderr. To make sure I covered 100% I also tried making\n    the display() function in progress.c a NOOP on top of that (it's\n    the calls to start_progress_delay() and stop_progress()) that\n    matter.\n\n    That doesn't hit the BUG() either. Some tests fail in that mode\n    due to a combination of the overzealous isatty(2) munging noted\n    above, and the tests that are testing that the progress output\n    itself is present (but for testing I'd made display() a NOOP).\n\nThis doesn't address any currently out-of-tree user of progress.c,\ni.e. WIP patches, or progress output that's a part of forks of\ngit.git. Those hopefully have test coverage that would expose the\nBUG().\n\nIf they don't they'll either run into it in code that displays more\nthan one progress bar for the lifetime of the progress, or which calls\nstop_progress() with a non-NULL \"progress\" without a corresponding\nstart_progress(). Both of those cases are less likely than the general\ncases of progress.c API misuse.\n\nBetween those three points above and the discussion of how this could\nimpact out-of-tree users I think it's safe to go ahead with this\nchange.\n\n1. 6f9d5f2fda1 (commit-graph: fix progress of reachable commits, 2020-07-09)\n2. https://lore.kernel.org/git/20210620200303.2328957-3-szeder.dev@gmail.com\n3. https://lore.kernel.org/git/patch-18.25-e21fc66623f-20210623T155626Z-avarab@gmail.com/\n4. https://lore.kernel.org/git/cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com/\n5. b50c37aa44d (Merge branch 'ab/progress-users-adjust-counters' into\n   next, 2021-09-10)\n6. https://lore.kernel.org/git/877dffg37n.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c                  | 18 ++++++++++++++++++\n t/t0500-progress-display.sh | 11 +++++++++++\n 2 files changed, 29 insertions(+)\n\ndiff --git a/progress.c b/progress.c\nindex 76a95cb7322..7483aec2e2a 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -46,6 +46,7 @@ struct progress {\n };\n \n static volatile sig_atomic_t progress_update;\n+static struct progress *global_progress;\n \n /*\n  * These are only intended for testing the progress output, i.e. exclusively\n@@ -249,6 +250,14 @@ void display_progress(struct progress *progress, uint64_t n)\n \t\tdisplay(progress, n, NULL);\n }\n \n+static void set_global_progress(struct progress *progress)\n+{\n+\tif (global_progress)\n+\t\tBUG(\"'%s' progress still active when trying to start '%s'\",\n+\t\t    global_progress->title, progress->title);\n+\tglobal_progress = progress;\n+}\n+\n static struct progress *start_progress_delay(const char *title, uint64_t total,\n \t\t\t\t\t     unsigned delay, unsigned sparse)\n {\n@@ -264,6 +273,7 @@ static struct progress *start_progress_delay(const char *title, uint64_t total,\n \tstrbuf_init(&progress->counters_sb, 0);\n \tprogress->title_len = utf8_strwidth(title);\n \tprogress->split = 0;\n+\tset_global_progress(progress);\n \tset_progress_signal();\n \ttrace2_region_enter(\"progress\", title, the_repository);\n \treturn progress;\n@@ -340,6 +350,13 @@ void stop_progress(struct progress **p_progress)\n \tstop_progress_msg(p_progress, _(\"done\"));\n }\n \n+static void unset_global_progress(void)\n+{\n+\tif (!global_progress)\n+\t\tBUG(\"should have active global_progress when cleaning up\");\n+\tglobal_progress = NULL;\n+}\n+\n void stop_progress_msg(struct progress **p_progress, const char *msg)\n {\n \tstruct progress *progress;\n@@ -369,6 +386,7 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)\n \t\tfree(buf);\n \t}\n \tclear_progress_signal();\n+\tunset_global_progress();\n \tstrbuf_release(&progress->counters_sb);\n \tif (progress->throughput)\n \t\tstrbuf_release(&progress->throughput->display);\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 59e9f226ea4..867fdace3f2 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -298,6 +298,17 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \ttest_cmp expect out\n '\n \n+test_expect_success 'BUG: start two concurrent progress bars' '\n+\tcat >in <<-\\EOF &&\n+\tstart 0 one\n+\tstart 0 two\n+\tEOF\n+\n+\ttest_must_fail test-tool progress \\\n+\t\t<in 2>stderr &&\n+\tgrep \"^BUG: .*'\\''one'\\'' progress still active when trying to start '\\''two'\\''$\" stderr\n+'\n+\n test_expect_success 'progress generates traces' '\n \tcat >in <<-\\EOF &&\n \tstart 40\n-- \n2.33.1.1570.g069344fdd45\n\n"},{"id":"440253","messageId":"patch-v5-7.8-4795d4835b0-20211101T190630Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v5-0.8-00000000000-20211101T190630Z-avarab@gmail.com","subject":"[PATCH v5 7/8] various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-01T19:09:13Z","receivedAt":"2021-11-01T19:09:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"We have over 50 uses of \"isatty(1)\" and \"isatty(2)\" in the codebase,\nonly these two used the stdlib.h macros to refer to them.\n\nLet's change these for consistency, and because a subsequent commit's\ncommit message outlines a recipe to change all of these for ad-hoc\ntesting, not needing to match these with that ad-hoc regex will make\nthings easier to explain.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n builtin/bisect--helper.c | 2 +-\n builtin/bundle.c         | 2 +-\n compat/mingw.c           | 2 +-\n 3 files changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/bisect--helper.c b/builtin/bisect--helper.c\nindex 28a2e6a5750..1727cb051fb 100644\n--- a/builtin/bisect--helper.c\n+++ b/builtin/bisect--helper.c\n@@ -830,7 +830,7 @@ static int bisect_autostart(struct bisect_terms *terms)\n \tfprintf_ln(stderr, _(\"You need to start by \\\"git bisect \"\n \t\t\t  \"start\\\"\\n\"));\n \n-\tif (!isatty(STDIN_FILENO))\n+\tif (!isatty(1))\n \t\treturn -1;\n \n \t/*\ndiff --git a/builtin/bundle.c b/builtin/bundle.c\nindex 5a85d7cd0fe..df69c651753 100644\n--- a/builtin/bundle.c\n+++ b/builtin/bundle.c\n@@ -56,7 +56,7 @@ static int parse_options_cmd_bundle(int argc,\n \n static int cmd_bundle_create(int argc, const char **argv, const char *prefix) {\n \tint all_progress_implied = 0;\n-\tint progress = isatty(STDERR_FILENO);\n+\tint progress = isatty(2);\n \tstruct strvec pack_opts;\n \tint version = -1;\n \tint ret;\ndiff --git a/compat/mingw.c b/compat/mingw.c\nindex 9e0cd1e097f..0f545c1a7d1 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw.c\n@@ -2374,7 +2374,7 @@ int mingw_raise(int sig)\n \tswitch (sig) {\n \tcase SIGALRM:\n \t\tif (timer_fn == SIG_DFL) {\n-\t\t\tif (isatty(STDERR_FILENO))\n+\t\t\tif (isatty(2))\n \t\t\t\tfputs(\"Alarm clock\\n\", stderr);\n \t\t\texit(128 + SIGALRM);\n \t\t} else if (timer_fn != SIG_IGN)\n-- \n2.33.1.1570.g069344fdd45\n\n"},{"id":"440284","messageId":"CAPUEspg1wFhQviN652E0riDOuicxF_gS=hguk4+Dd7BLoci90g@mail.gmail.com","threadId":"55972","inReplyTo":"patch-v5-7.8-4795d4835b0-20211101T190630Z-avarab@gmail.com","subject":"Re: [PATCH v5 7/8] various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO)","fromName":"Carlo Arenas","fromEmail":"carenas@gmail.com","sentAt":"2021-11-02T00:58:39Z","receivedAt":"2021-11-02T00:58:52Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"On Mon, Nov 1, 2021 at 12:10 PM Ævar Arnfjörð Bjarmason\n<avarab@gmail.com> wrote:\n>\n> We have over 50 uses of \"isatty(1)\" and \"isatty(2)\" in the codebase,\n> only these two used the stdlib.h macros to refer to them.\n>\n> Let's change these for consistency, and because a subsequent commit's\n> commit message outlines a recipe to change all of these for ad-hoc\n> testing, not needing to match these with that ad-hoc regex will make\n> things easier to explain.\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>  builtin/bisect--helper.c | 2 +-\n>  builtin/bundle.c         | 2 +-\n>  compat/mingw.c           | 2 +-\n>  3 files changed, 3 insertions(+), 3 deletions(-)\n>\n> diff --git a/builtin/bisect--helper.c b/builtin/bisect--helper.c\n> index 28a2e6a5750..1727cb051fb 100644\n> --- a/builtin/bisect--helper.c\n> +++ b/builtin/bisect--helper.c\n> @@ -830,7 +830,7 @@ static int bisect_autostart(struct bisect_terms *terms)\n>         fprintf_ln(stderr, _(\"You need to start by \\\"git bisect \"\n>                           \"start\\\"\\n\"));\n>\n> -       if (!isatty(STDIN_FILENO))\n> +       if (!isatty(1))\n\nSTDIN_FILENO -> 0\n\nCarlo\n"},{"id":"440293","messageId":"cover-v6-0.8-00000000000-20211102T122507Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v5-0.8-00000000000-20211101T190630Z-avarab@gmail.com","subject":"[PATCH v6 0/8] progress: assert \"global_progress\" + test fixes / cleanup","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-02T12:27:22Z","receivedAt":"2021-11-02T12:27:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"A minor but important fixup of v5[1], as pointed out by Carlo Arenas\nthe STDIN_FILENO I added in response to feedback in an earlier re-roll\nhad a pretty bad off-by-one bug.\n\nAs an aside that we have no tests for this \"git bisect\" code at all is\npretty bad, I'll try to address that separately from this series.\n\nhttps://lore.kernel.org/git/cover-v5-0.8-00000000000-20211101T190630Z-avarab@gmail.com/\n\nÆvar Arnfjörð Bjarmason (8):\n  leak tests: fix a memory leaks in \"test-progress\" helper\n  progress.c test helper: add missing braces\n  progress.c tests: make start/stop commands on stdin\n  progress.c tests: test some invalid usage\n  progress.c: add temporary variable from progress struct\n  pack-bitmap-write.c: don't return without stop_progress()\n  various *.c: use isatty(0|2), not isatty(STDIN_FILENO|STDERR_FILENO)\n  progress.c: add & assert a \"global_progress\" variable\n\n builtin/bisect--helper.c    |   2 +-\n builtin/bundle.c            |   2 +-\n compat/mingw.c              |   2 +-\n pack-bitmap-write.c         |   6 +--\n progress.c                  |  23 +++++++-\n t/helper/test-progress.c    |  52 +++++++++++++-----\n t/t0500-progress-display.sh | 105 ++++++++++++++++++++++++++++--------\n 7 files changed, 150 insertions(+), 42 deletions(-)\n\nRange-diff against v5:\n1:  0c0e5bfc656 = 1:  88d89652831 leak tests: fix a memory leaks in \"test-progress\" helper\n2:  0ae86e40e16 = 2:  2fa92e7db83 progress.c test helper: add missing braces\n3:  767aa010026 = 3:  400b491449e progress.c tests: make start/stop commands on stdin\n4:  9bcc8cfc9d2 = 4:  7557975a122 progress.c tests: test some invalid usage\n5:  3a95613cd04 = 5:  61c8da31aeb progress.c: add temporary variable from progress struct\n6:  0300b802dcb = 6:  f6a76b80e91 pack-bitmap-write.c: don't return without stop_progress()\n7:  4795d4835b0 ! 7:  1a2eadf28d0 various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO)\n    @@ Metadata\n     Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n     \n      ## Commit message ##\n    -    various *.c: use isatty(1|2), not isatty(STDIN_FILENO|STDERR_FILENO)\n    +    various *.c: use isatty(0|2), not isatty(STDIN_FILENO|STDERR_FILENO)\n     \n         We have over 50 uses of \"isatty(1)\" and \"isatty(2)\" in the codebase,\n    -    only these two used the stdlib.h macros to refer to them.\n    +    and around 10 \"isatty(0)\", but these used the\n    +    {STDIN_FILENO,STD{OUT,ERR}_FILENO} macros in \"stdlib.h\" to refer to\n    +    them.\n     \n         Let's change these for consistency, and because a subsequent commit's\n         commit message outlines a recipe to change all of these for ad-hoc\n    @@ builtin/bisect--helper.c: static int bisect_autostart(struct bisect_terms *terms\n      \t\t\t  \"start\\\"\\n\"));\n      \n     -\tif (!isatty(STDIN_FILENO))\n    -+\tif (!isatty(1))\n    ++\tif (!isatty(0))\n      \t\treturn -1;\n      \n      \t/*\n8:  e58f4a0c1b9 = 8:  bff919994b5 progress.c: add & assert a \"global_progress\" variable\n-- \n2.33.1.1570.g069344fdd45\n\n"},{"id":"440294","messageId":"patch-v6-1.8-88d89652831-20211102T122507Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v6-0.8-00000000000-20211102T122507Z-avarab@gmail.com","subject":"[PATCH v6 1/8] leak tests: fix a memory leaks in \"test-progress\" helper","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-02T12:27:23Z","receivedAt":"2021-11-02T12:27:38Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a memory leak in the test-progress helper, and mark the\ncorresponding \"t0500-progress-display.sh\" test as being leak-free\nunder SANITIZE=leak. This fixes a leak added in 2bb74b53a4 (Test the\nprogress display, 2019-09-16).\n\nMy 48f68715b14 (tr2: stop leaking \"thread_name\" memory, 2021-08-27)\nhad fixed another memory leak in this test (as it did some trace2\ntesting).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/helper/test-progress.c    | 1 +\n t/t0500-progress-display.sh | 1 +\n 2 files changed, 2 insertions(+)\n\ndiff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\nindex 5d05cbe7894..9265e6ab7cf 100644\n--- a/t/helper/test-progress.c\n+++ b/t/helper/test-progress.c\n@@ -69,6 +69,7 @@ int cmd__progress(int argc, const char **argv)\n \t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n \t}\n \tstop_progress(&progress);\n+\tstrbuf_release(&line);\n \n \treturn 0;\n }\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 22058b503ac..f37cf2eb9c9 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -2,6 +2,7 @@\n \n test_description='progress display'\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n show_cr () {\n-- \n2.33.1.1570.g069344fdd45\n\n"},{"id":"440295","messageId":"patch-v6-2.8-2fa92e7db83-20211102T122507Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v6-0.8-00000000000-20211102T122507Z-avarab@gmail.com","subject":"[PATCH v6 2/8] progress.c test helper: add missing braces","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-02T12:27:24Z","receivedAt":"2021-11-02T12:27:39Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"If we have braces on one arm of an if/else all of them should have it,\nper the CodingGuidelines's \"When there are multiple arms to a\nconditional[...]\" advice. This formatting change makes a subsequent\ncommit smaller.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/helper/test-progress.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\nindex 9265e6ab7cf..50fd3be3dad 100644\n--- a/t/helper/test-progress.c\n+++ b/t/helper/test-progress.c\n@@ -63,10 +63,11 @@ int cmd__progress(int argc, const char **argv)\n \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n \t\t\tprogress_test_ns = test_ms * 1000 * 1000;\n \t\t\tdisplay_throughput(progress, byte_count);\n-\t\t} else if (!strcmp(line.buf, \"update\"))\n+\t\t} else if (!strcmp(line.buf, \"update\")) {\n \t\t\tprogress_test_force_update();\n-\t\telse\n+\t\t} else {\n \t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n+\t\t}\n \t}\n \tstop_progress(&progress);\n \tstrbuf_release(&line);\n-- \n2.33.1.1570.g069344fdd45\n\n"},{"id":"440296","messageId":"patch-v6-3.8-400b491449e-20211102T122507Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v6-0.8-00000000000-20211102T122507Z-avarab@gmail.com","subject":"[PATCH v6 3/8] progress.c tests: make start/stop commands on stdin","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-02T12:27:25Z","receivedAt":"2021-11-02T12:27:40Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the usage of the \"test-tool progress\" introduced in\n2bb74b53a49 (Test the progress display, 2019-09-16) to take command\nlike \"start\" and \"stop\" on stdin, instead of running them implicitly.\n\nThis makes for tests that are easier to read, since the recipe will\nmirror the API usage, and allows for easily testing invalid usage that\nwould yield (or should yield) a BUG(), e.g. providing two \"start\"\ncalls in a row. A subsequent commit will add such tests.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/helper/test-progress.c    | 46 ++++++++++++++++++++++-------\n t/t0500-progress-display.sh | 58 +++++++++++++++++++++++--------------\n 2 files changed, 72 insertions(+), 32 deletions(-)\n\ndiff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\nindex 50fd3be3dad..1435c28e950 100644\n--- a/t/helper/test-progress.c\n+++ b/t/helper/test-progress.c\n@@ -3,6 +3,9 @@\n  *\n  * Reads instructions from standard input, one instruction per line:\n  *\n+ *   \"start <total>[ <title>]\" - Call start_progress(title, total),\n+ *                               Uses the default title of \"Working hard\"\n+ *                               if the \" <title>\" is omitted.\n  *   \"progress <items>\" - Call display_progress() with the given item count\n  *                        as parameter.\n  *   \"throughput <bytes> <millis> - Call display_throughput() with the given\n@@ -10,6 +13,7 @@\n  *                                  specify the time elapsed since the\n  *                                  start_progress() call.\n  *   \"update\" - Set the 'progress_update' flag.\n+ *   \"stop\" - Call stop_progress().\n  *\n  * See 't0500-progress-display.sh' for examples.\n  */\n@@ -19,34 +23,52 @@\n #include \"parse-options.h\"\n #include \"progress.h\"\n #include \"strbuf.h\"\n+#include \"string-list.h\"\n+\n+/*\n+ * We can't use \"end + 1\" as an argument to start_progress() below, it\n+ * doesn't xstrdup() its \"title\" argument. We need to hold onto a\n+ * valid \"char *\" for it until the end.\n+ */\n+static char *dup_title(struct string_list *titles, const char *title)\n+{\n+\treturn string_list_insert(titles, title)->string;\n+}\n \n int cmd__progress(int argc, const char **argv)\n {\n-\tint total = 0;\n-\tconst char *title;\n+\tconst char *const default_title = \"Working hard\";\n+\tstruct string_list titles = STRING_LIST_INIT_DUP;\n \tstruct strbuf line = STRBUF_INIT;\n-\tstruct progress *progress;\n+\tstruct progress *progress = NULL;\n \n \tconst char *usage[] = {\n-\t\t\"test-tool progress [--total=<n>] <progress-title>\",\n+\t\t\"test-tool progress <stdin\",\n \t\tNULL\n \t};\n \tstruct option options[] = {\n-\t\tOPT_INTEGER(0, \"total\", &total, \"total number of items\"),\n \t\tOPT_END(),\n \t};\n \n \targc = parse_options(argc, argv, NULL, options, usage, 0);\n-\tif (argc != 1)\n-\t\tdie(\"need a title for the progress output\");\n-\ttitle = argv[0];\n+\tif (argc)\n+\t\tusage_with_options(usage, options);\n \n \tprogress_testing = 1;\n-\tprogress = start_progress(title, total);\n \twhile (strbuf_getline(&line, stdin) != EOF) {\n \t\tchar *end;\n \n-\t\tif (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n+\t\tif (skip_prefix(line.buf, \"start \", (const char **) &end)) {\n+\t\t\tuint64_t total = strtoull(end, &end, 10);\n+\t\t\tif (*end == '\\0')\n+\t\t\t\tprogress = start_progress(default_title, total);\n+\t\t\telse if (*end == ' ')\n+\t\t\t\tprogress = start_progress(dup_title(&titles,\n+\t\t\t\t\t\t\t\t    end + 1),\n+\t\t\t\t\t\t\t  total);\n+\t\t\telse\n+\t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n+\t\t} else if (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n \t\t\tuint64_t item_count = strtoull(end, &end, 10);\n \t\t\tif (*end != '\\0')\n \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n@@ -65,12 +87,14 @@ int cmd__progress(int argc, const char **argv)\n \t\t\tdisplay_throughput(progress, byte_count);\n \t\t} else if (!strcmp(line.buf, \"update\")) {\n \t\t\tprogress_test_force_update();\n+\t\t} else if (!strcmp(line.buf, \"stop\")) {\n+\t\t\tstop_progress(&progress);\n \t\t} else {\n \t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n \t\t}\n \t}\n-\tstop_progress(&progress);\n \tstrbuf_release(&line);\n+\tstring_list_clear(&titles, 0);\n \n \treturn 0;\n }\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex f37cf2eb9c9..27ab4218b01 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -18,6 +18,7 @@ test_expect_success 'simple progress display' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tupdate\n \tprogress 1\n \tupdate\n@@ -26,8 +27,9 @@ test_expect_success 'simple progress display' '\n \tprogress 4\n \tupdate\n \tprogress 5\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -42,11 +44,13 @@ test_expect_success 'progress display with total' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 3\n \tprogress 1\n \tprogress 2\n \tprogress 3\n+\tstop\n \tEOF\n-\ttest-tool progress --total=3 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -63,14 +67,14 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tprogress 100\n \tprogress 1000\n \tprogress 10000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -89,16 +93,16 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tupdate\n \tprogress 1\n \tupdate\n \tprogress 2\n \tprogress 10000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -117,14 +121,14 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tprogress 25000\n \tprogress 50000\n \tprogress 75000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -141,14 +145,14 @@ Working hard.......2.........3.........4.........5.........6.........7.........:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6.........7.........\n \tprogress 25000\n \tprogress 50000\n \tprogress 75000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6.........7.........\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -165,12 +169,14 @@ test_expect_success 'progress shortens - crazy caller' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 1000\n \tprogress 100\n \tprogress 200\n \tprogress 1\n \tprogress 1000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=1000 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -186,6 +192,7 @@ test_expect_success 'progress display with throughput' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tthroughput 102400 1000\n \tupdate\n \tprogress 10\n@@ -198,8 +205,9 @@ test_expect_success 'progress display with throughput' '\n \tthroughput 409600 4000\n \tupdate\n \tprogress 40\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -215,6 +223,7 @@ test_expect_success 'progress display with throughput and total' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 40\n \tthroughput 102400 1000\n \tprogress 10\n \tthroughput 204800 2000\n@@ -223,8 +232,9 @@ test_expect_success 'progress display with throughput and total' '\n \tprogress 30\n \tthroughput 409600 4000\n \tprogress 40\n+\tstop\n \tEOF\n-\ttest-tool progress --total=40 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -240,6 +250,7 @@ test_expect_success 'cover up after throughput shortens' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tthroughput 409600 1000\n \tupdate\n \tprogress 1\n@@ -252,8 +263,9 @@ test_expect_success 'cover up after throughput shortens' '\n \tthroughput 1638400 4000\n \tupdate\n \tprogress 4\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -268,6 +280,7 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tthroughput 1 1000\n \tupdate\n \tprogress 1\n@@ -277,8 +290,9 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \tthroughput 3145728 3000\n \tupdate\n \tprogress 3\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -286,6 +300,7 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \n test_expect_success 'progress generates traces' '\n \tcat >in <<-\\EOF &&\n+\tstart 40\n \tthroughput 102400 1000\n \tupdate\n \tprogress 10\n@@ -298,10 +313,11 @@ test_expect_success 'progress generates traces' '\n \tthroughput 409600 4000\n \tupdate\n \tprogress 40\n+\tstop\n \tEOF\n \n-\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" test-tool progress --total=40 \\\n-\t\t\"Working hard\" <in 2>stderr &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n \n \t# t0212/parse_events.perl intentionally omits regions and data.\n \ttest_region progress \"Working hard\" trace.event &&\n-- \n2.33.1.1570.g069344fdd45\n\n"},{"id":"440297","messageId":"patch-v6-4.8-7557975a122-20211102T122507Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v6-0.8-00000000000-20211102T122507Z-avarab@gmail.com","subject":"[PATCH v6 4/8] progress.c tests: test some invalid usage","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-02T12:27:26Z","receivedAt":"2021-11-02T12:27:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Test what happens when we \"stop\" without a \"start\", omit the \"stop\"\nafter a \"start\", or try to start two concurrent progress bars. This\nextends the trace2 tests added in 98a13647408 (trace2: log progress\ntime and throughput, 2020-05-12).\n\nThese tests are not merely testing the helper, but invalid API usage\nthat can happen if the progress.c API is misused.\n\nThe \"without stop\" test will leak under SANITIZE=leak, since this\nbuggy use of the API will leak memory. But let's not skip it entirely,\nor use the \"!SANITIZE_LEAK\" prerequisite check as we'd do with tests\nthat we're skipping due to leaks we haven't fixed yet. Instead\nannotate the specific command that should skip leak checking with\ncustom $LSAN_OPTIONS[1].\n\n1. https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/t0500-progress-display.sh | 35 +++++++++++++++++++++++++++++++++++\n 1 file changed, 35 insertions(+)\n\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 27ab4218b01..59e9f226ea4 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -325,4 +325,39 @@ test_expect_success 'progress generates traces' '\n \tgrep \"\\\"key\\\":\\\"total_bytes\\\",\\\"value\\\":\\\"409600\\\"\" trace.event\n '\n \n+test_expect_success 'progress generates traces: stop / start' '\n+\tcat >in <<-\\EOF &&\n+\tstart 0\n+\tstop\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-startstop.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n+\ttest_region progress \"Working hard\" trace-startstop.event\n+'\n+\n+test_expect_success 'progress generates traces: start without stop' '\n+\tcat >in <<-\\EOF &&\n+\tstart 0\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-start.event\" \\\n+\tLSAN_OPTIONS=detect_leaks=0 \\\n+\ttest-tool progress \\\n+\t\t<in 2>stderr &&\n+\tgrep region_enter.*progress trace-start.event &&\n+\t! grep region_leave.*progress trace-start.event\n+'\n+\n+test_expect_success 'progress generates traces: stop without start' '\n+\tcat >in <<-\\EOF &&\n+\tstop\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-stop.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n+\t! grep region_enter.*progress trace-stop.event &&\n+\t! grep region_leave.*progress trace-stop.event\n+'\n+\n test_done\n-- \n2.33.1.1570.g069344fdd45\n\n"},{"id":"440298","messageId":"patch-v6-5.8-61c8da31aeb-20211102T122507Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v6-0.8-00000000000-20211102T122507Z-avarab@gmail.com","subject":"[PATCH v6 5/8] progress.c: add temporary variable from progress struct","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-02T12:27:27Z","receivedAt":"2021-11-02T12:27:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a temporary \"progress\" variable for the dereferenced p_progress\npointer to a \"struct progress *\". Before 98a13647408 (trace2: log\nprogress time and throughput, 2020-05-12) we didn't dereference\n\"p_progress\" in this function, now that we do it's easier to read the\ncode if we work with a \"progress\" struct pointer like everywhere else,\ninstead of a pointer to a pointer.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 680c6a8bf93..76a95cb7322 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -325,15 +325,16 @@ void stop_progress(struct progress **p_progress)\n \tfinish_if_sparse(*p_progress);\n \n \tif (*p_progress) {\n+\t\tstruct progress *progress = *p_progress;\n \t\ttrace2_data_intmax(\"progress\", the_repository, \"total_objects\",\n \t\t\t\t   (*p_progress)->total);\n \n \t\tif ((*p_progress)->throughput)\n \t\t\ttrace2_data_intmax(\"progress\", the_repository,\n \t\t\t\t\t   \"total_bytes\",\n-\t\t\t\t\t   (*p_progress)->throughput->curr_total);\n+\t\t\t\t\t   progress->throughput->curr_total);\n \n-\t\ttrace2_region_leave(\"progress\", (*p_progress)->title, the_repository);\n+\t\ttrace2_region_leave(\"progress\", progress->title, the_repository);\n \t}\n \n \tstop_progress_msg(p_progress, _(\"done\"));\n-- \n2.33.1.1570.g069344fdd45\n\n"},{"id":"440299","messageId":"patch-v6-6.8-f6a76b80e91-20211102T122507Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v6-0.8-00000000000-20211102T122507Z-avarab@gmail.com","subject":"[PATCH v6 6/8] pack-bitmap-write.c: don't return without stop_progress()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-02T12:27:28Z","receivedAt":"2021-11-02T12:27:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a bug that's been here since 7cc8f971085 (pack-objects: implement\nbitmap writing, 2013-12-21), we did not call stop_progress() if we\nreached the early exit in this function.\n\nWe could call stop_progress() before we return, but better yet is to\ndefer calling start_progress() until we need it.\n\nThis will matter in a subsequent commit where we BUG(...) out if this\nhappens, and matters now e.g. because we don't have a corresponding\n\"region_end\" for the progress trace2 event.\n\nSuggested-by: SZEDER Gábor <szeder.dev@gmail.com>\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n pack-bitmap-write.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 9c55c1531e1..cab3eaa2acd 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -575,15 +575,15 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \n \tQSORT(indexed_commits, indexed_commits_nr, date_compare);\n \n-\tif (writer.show_progress)\n-\t\twriter.progress = start_progress(\"Selecting bitmap commits\", 0);\n-\n \tif (indexed_commits_nr < 100) {\n \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n \t\t\tpush_bitmapped_commit(indexed_commits[i]);\n \t\treturn;\n \t}\n \n+\tif (writer.show_progress)\n+\t\twriter.progress = start_progress(\"Selecting bitmap commits\", 0);\n+\n \tfor (;;) {\n \t\tstruct commit *chosen = NULL;\n \n-- \n2.33.1.1570.g069344fdd45\n\n"},{"id":"440300","messageId":"patch-v6-7.8-1a2eadf28d0-20211102T122507Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v6-0.8-00000000000-20211102T122507Z-avarab@gmail.com","subject":"[PATCH v6 7/8] various *.c: use isatty(0|2), not isatty(STDIN_FILENO|STDERR_FILENO)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-02T12:27:29Z","receivedAt":"2021-11-02T12:27:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"We have over 50 uses of \"isatty(1)\" and \"isatty(2)\" in the codebase,\nand around 10 \"isatty(0)\", but these used the\n{STDIN_FILENO,STD{OUT,ERR}_FILENO} macros in \"stdlib.h\" to refer to\nthem.\n\nLet's change these for consistency, and because a subsequent commit's\ncommit message outlines a recipe to change all of these for ad-hoc\ntesting, not needing to match these with that ad-hoc regex will make\nthings easier to explain.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n builtin/bisect--helper.c | 2 +-\n builtin/bundle.c         | 2 +-\n compat/mingw.c           | 2 +-\n 3 files changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/bisect--helper.c b/builtin/bisect--helper.c\nindex 28a2e6a5750..21360a4e70b 100644\n--- a/builtin/bisect--helper.c\n+++ b/builtin/bisect--helper.c\n@@ -830,7 +830,7 @@ static int bisect_autostart(struct bisect_terms *terms)\n \tfprintf_ln(stderr, _(\"You need to start by \\\"git bisect \"\n \t\t\t  \"start\\\"\\n\"));\n \n-\tif (!isatty(STDIN_FILENO))\n+\tif (!isatty(0))\n \t\treturn -1;\n \n \t/*\ndiff --git a/builtin/bundle.c b/builtin/bundle.c\nindex 5a85d7cd0fe..df69c651753 100644\n--- a/builtin/bundle.c\n+++ b/builtin/bundle.c\n@@ -56,7 +56,7 @@ static int parse_options_cmd_bundle(int argc,\n \n static int cmd_bundle_create(int argc, const char **argv, const char *prefix) {\n \tint all_progress_implied = 0;\n-\tint progress = isatty(STDERR_FILENO);\n+\tint progress = isatty(2);\n \tstruct strvec pack_opts;\n \tint version = -1;\n \tint ret;\ndiff --git a/compat/mingw.c b/compat/mingw.c\nindex 9e0cd1e097f..0f545c1a7d1 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw.c\n@@ -2374,7 +2374,7 @@ int mingw_raise(int sig)\n \tswitch (sig) {\n \tcase SIGALRM:\n \t\tif (timer_fn == SIG_DFL) {\n-\t\t\tif (isatty(STDERR_FILENO))\n+\t\t\tif (isatty(2))\n \t\t\t\tfputs(\"Alarm clock\\n\", stderr);\n \t\t\texit(128 + SIGALRM);\n \t\t} else if (timer_fn != SIG_IGN)\n-- \n2.33.1.1570.g069344fdd45\n\n"},{"id":"440301","messageId":"patch-v6-8.8-bff919994b5-20211102T122507Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v6-0.8-00000000000-20211102T122507Z-avarab@gmail.com","subject":"[PATCH v6 8/8] progress.c: add & assert a \"global_progress\" variable","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-02T12:27:30Z","receivedAt":"2021-11-02T12:28:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"The progress.c code makes a hard assumption that only one progress bar\nbe active at a time (see [1] for a bug where this wasn't the\ncase). Add a BUG() that'll trigger if we ever regress on that promise\nand have two progress bars active at the same time.\n\nThere was an alternative test-only approach to doing the same\nthing[2], but by doing this outside of a GIT_TEST_* mode we'll know\nwe've put a hard stop to this particular API misuse.\n\nIt will also establish scaffolding to address current fundamental\nlimitations in the progress output: The current output must be\n\"driven\" by calls to the likes of display_progress().\n\nOnce we have a global current progress object we'll be able to update\nthat object via SIGALRM, this will cover cases where we're busy, but\neither haven't invoked our first display_progress() yet, or the time\nbetween display_progress() is too long. See [3] for early code to do\nthat.\n\nThe linked code in [3] is WIP and not signal-safe since among other\nthings it calls sprintf() from within a signal handler, see e.g. \"man\n7 signal-safety\". But on some platforms a real implementation of it\nwould be able to write() out a prepared-formatted progress update from\nwithin a signal handler. That would be sufficient to e.g. show that\nwe're \"stalled\", or to display something like a simple pre-formatted\n\"spinner\".\n\nIt's conceivable that this change will hit the BUG() condition in some\nscenario that we don't currently have tests for, this would be very\nbad. If that happened we'd die just because we couldn't emit some\npretty output.\n\nSee [4] for a discussion of why our test coverage is lacking; our\nprogress display is hidden behind isatty(2) checks in many cases, so\nthe test suite doesn't cover it unless individual tests are run in\n\"--verbose\" mode, we might also have multi-threaded use of the API, so\ntwo progress bars stopping and starting would only be visible due to a\nrace condition.\n\nDespite that, I think that this change won't introduce such\nregressions, because:\n\n 1. I've read all the code using the progress API (and have modified a\n    large part of it in some WIP code I have). Almost all of it is really\n    simple, the parts that aren't[5] are complex in the display_progress() part,\n    not in starting or stopping the progress bar.\n\n 2. The entire test suite passes when instrumented with an ad-hoc\n    Linux-specific mode (it uses gettid()) to die if progress bars are\n    ever started or stopped on anything but the main thread[6].\n\n    Extending that to die if display_progress() is called in a thread\n    reveals that we have exactly two users of the progress bar under\n    threaded conditions, \"git index-pack\" and \"git pack-objects\". Both\n    uses are straightforward, and they don't start/stop the progress\n    bar when threads are active.\n\n 3. I've likewise done an ad-hoc test to force progress bars to be\n    displayed with:\n\n        perl -pi -e 's[isatty\\(2\\)][1]g' $(git grep -l -F 'isatty(2)')\n\n    I.e. to replace all checks (not just for progress) of checking\n    whether STDERR is connected to a TTY, and then monkeypatching\n    is_foreground_fd() in progress.c to always \"return 1\". Running the\n    tests with those applied, interactively and under -V reveals via:\n\n        $ grep -e set_progress_signal -e clear_progress_signal test-results/*out\n\n    That nothing our tests cover hits the BUG conditions added here,\n    except the expected \"BUG: start two concurrent progress bars\" test\n    being added here.\n\n    That isn't entirely true since we won't be getting 100% coverage\n    due to cascading failures from tests that expected no progress\n    output on stderr. To make sure I covered 100% I also tried making\n    the display() function in progress.c a NOOP on top of that (it's\n    the calls to start_progress_delay() and stop_progress()) that\n    matter.\n\n    That doesn't hit the BUG() either. Some tests fail in that mode\n    due to a combination of the overzealous isatty(2) munging noted\n    above, and the tests that are testing that the progress output\n    itself is present (but for testing I'd made display() a NOOP).\n\nThis doesn't address any currently out-of-tree user of progress.c,\ni.e. WIP patches, or progress output that's a part of forks of\ngit.git. Those hopefully have test coverage that would expose the\nBUG().\n\nIf they don't they'll either run into it in code that displays more\nthan one progress bar for the lifetime of the progress, or which calls\nstop_progress() with a non-NULL \"progress\" without a corresponding\nstart_progress(). Both of those cases are less likely than the general\ncases of progress.c API misuse.\n\nBetween those three points above and the discussion of how this could\nimpact out-of-tree users I think it's safe to go ahead with this\nchange.\n\n1. 6f9d5f2fda1 (commit-graph: fix progress of reachable commits, 2020-07-09)\n2. https://lore.kernel.org/git/20210620200303.2328957-3-szeder.dev@gmail.com\n3. https://lore.kernel.org/git/patch-18.25-e21fc66623f-20210623T155626Z-avarab@gmail.com/\n4. https://lore.kernel.org/git/cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com/\n5. b50c37aa44d (Merge branch 'ab/progress-users-adjust-counters' into\n   next, 2021-09-10)\n6. https://lore.kernel.org/git/877dffg37n.fsf@evledraar.gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c                  | 18 ++++++++++++++++++\n t/t0500-progress-display.sh | 11 +++++++++++\n 2 files changed, 29 insertions(+)\n\ndiff --git a/progress.c b/progress.c\nindex 76a95cb7322..7483aec2e2a 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -46,6 +46,7 @@ struct progress {\n };\n \n static volatile sig_atomic_t progress_update;\n+static struct progress *global_progress;\n \n /*\n  * These are only intended for testing the progress output, i.e. exclusively\n@@ -249,6 +250,14 @@ void display_progress(struct progress *progress, uint64_t n)\n \t\tdisplay(progress, n, NULL);\n }\n \n+static void set_global_progress(struct progress *progress)\n+{\n+\tif (global_progress)\n+\t\tBUG(\"'%s' progress still active when trying to start '%s'\",\n+\t\t    global_progress->title, progress->title);\n+\tglobal_progress = progress;\n+}\n+\n static struct progress *start_progress_delay(const char *title, uint64_t total,\n \t\t\t\t\t     unsigned delay, unsigned sparse)\n {\n@@ -264,6 +273,7 @@ static struct progress *start_progress_delay(const char *title, uint64_t total,\n \tstrbuf_init(&progress->counters_sb, 0);\n \tprogress->title_len = utf8_strwidth(title);\n \tprogress->split = 0;\n+\tset_global_progress(progress);\n \tset_progress_signal();\n \ttrace2_region_enter(\"progress\", title, the_repository);\n \treturn progress;\n@@ -340,6 +350,13 @@ void stop_progress(struct progress **p_progress)\n \tstop_progress_msg(p_progress, _(\"done\"));\n }\n \n+static void unset_global_progress(void)\n+{\n+\tif (!global_progress)\n+\t\tBUG(\"should have active global_progress when cleaning up\");\n+\tglobal_progress = NULL;\n+}\n+\n void stop_progress_msg(struct progress **p_progress, const char *msg)\n {\n \tstruct progress *progress;\n@@ -369,6 +386,7 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)\n \t\tfree(buf);\n \t}\n \tclear_progress_signal();\n+\tunset_global_progress();\n \tstrbuf_release(&progress->counters_sb);\n \tif (progress->throughput)\n \t\tstrbuf_release(&progress->throughput->display);\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 59e9f226ea4..867fdace3f2 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -298,6 +298,17 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \ttest_cmp expect out\n '\n \n+test_expect_success 'BUG: start two concurrent progress bars' '\n+\tcat >in <<-\\EOF &&\n+\tstart 0 one\n+\tstart 0 two\n+\tEOF\n+\n+\ttest_must_fail test-tool progress \\\n+\t\t<in 2>stderr &&\n+\tgrep \"^BUG: .*'\\''one'\\'' progress still active when trying to start '\\''two'\\''$\" stderr\n+'\n+\n test_expect_success 'progress generates traces' '\n \tcat >in <<-\\EOF &&\n \tstart 40\n-- \n2.33.1.1570.g069344fdd45\n\n"},{"id":"442919","messageId":"20211202231421.GA624717@szeder.dev","threadId":"55972","inReplyTo":"xmqq35op4f7n.fsf@gitster.g","subject":"Re: [PATCH v3 10/10] progress.c: add & assert a \"global_progress\" variable","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-12-02T23:14:21Z","receivedAt":"2021-12-02T23:14:30Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Oct 25, 2021 at 02:38:04AM -0700, Junio C Hamano wrote:\n> SZEDER Gábor <szeder.dev@gmail.com> writes:\n> \n> > I still very much dislike the idea of a BUG() in the progress code\n> > that can trigger outside of the test suite, because the progress line\n> > is only a UI gimmick and not a crucial part of any Git operation, and\n> > even though a progress line might be buggy, the underlying Git\n> > operation is not affected by it and would still finish successfully,\n> > as was the case with the dozen of so progress line bugs in the past.\n> \n> I too recall that we have fixed numerous bugs in the past year in\n> the area, but weren't they kind of obvious ones _once_ they are\n> pointed out at you (e.g. progress never reaching to 100%)?  Yet the\n> developers have failed to catch them because their eyes would coast\n> over without paying attention to them, exactly because the progress\n> bar is merely a UI gimmick.\n> \n> I haven't formed a firm opinion on this yet, but I think the idea\n> behind these BUG() is to help such problems be caught while they are\n> still in the lab.  You may not notice when your live progress bar\n> behaved a bit funny, but if you hit a BUG(), that would be squarely\n> in your face and you cannot ignore it.\n\nThe \"outside of the test suite\" part is important.  Running the test\nsuite with a GIT_TEST_CHECK_PROGRESS knob is sufficient to catch these\nissues as my inital two patch series have shown at the very beginning\nof this thread before Ævar hijacked them.  Interested Git developers\ncan even enable it in their .profile if they wish; I did so with\nGIT_TEST_SPLIT_INDEX for a while to expose some of my patches to more\nreal-world use.\n\nHowever, I think it's conceptually a bad idea to abort with a BUG() an\notherwise successful Git operation by default, when that bug happens\nto be in such an ancillary component as the progress display, and find\nthe justifications given in the commit message unconvincing.\n\n"},{"id":"442967","messageId":"211203.868rx2t0hv.gmgdl@evledraar.gmail.com","threadId":"55972","inReplyTo":"20211202231421.GA624717@szeder.dev","subject":"Re: [PATCH v3 10/10] progress.c: add & assert a \"global_progress\" variable","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-12-03T10:29:26Z","receivedAt":"2021-12-03T11:02:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Dec 03 2021, SZEDER Gábor wrote:\n\n> On Mon, Oct 25, 2021 at 02:38:04AM -0700, Junio C Hamano wrote:\n>> SZEDER Gábor <szeder.dev@gmail.com> writes:\n>> \n>> > I still very much dislike the idea of a BUG() in the progress code\n>> > that can trigger outside of the test suite, because the progress line\n>> > is only a UI gimmick and not a crucial part of any Git operation, and\n>> > even though a progress line might be buggy, the underlying Git\n>> > operation is not affected by it and would still finish successfully,\n>> > as was the case with the dozen of so progress line bugs in the past.\n>> \n>> I too recall that we have fixed numerous bugs in the past year in\n>> the area, but weren't they kind of obvious ones _once_ they are\n>> pointed out at you (e.g. progress never reaching to 100%)?  Yet the\n>> developers have failed to catch them because their eyes would coast\n>> over without paying attention to them, exactly because the progress\n>> bar is merely a UI gimmick.\n>> \n>> I haven't formed a firm opinion on this yet, but I think the idea\n>> behind these BUG() is to help such problems be caught while they are\n>> still in the lab.  You may not notice when your live progress bar\n>> behaved a bit funny, but if you hit a BUG(), that would be squarely\n>> in your face and you cannot ignore it.\n>\n> The \"outside of the test suite\" part is important.  Running the test\n> suite with a GIT_TEST_CHECK_PROGRESS knob is sufficient to catch these\n> issues as my inital two patch series have shown at the very beginning\n> of this thread before Ævar hijacked them.\n\nFWIW I didn't hijack and adjust your patches to be BUG() instead of a\ntest mode. We came up with more-or-less the same thing\nindependently around the same time.\n\nYou submitted yours on the 20th of June, but I hacked up mine on top af\non already-WIP series I had ~5 days earlier in response to another\ndiscussion about progress.c[1]. You submitter yours first, and I then\nreplied to the effect of \"here's a similar thing I had locally, RFC\"[2].\n\nOr do you mean the smaller \"fix bogus counting\" fixed that are now\nlanded[3]? I just wanted to push that forward since it seemed you\nweren't per[4]. Sorry if I stepped on any toes there...\n\n> Interested Git developers\n> can even enable it in their .profile if they wish; I did so with\n> GIT_TEST_SPLIT_INDEX for a while to expose some of my patches to more\n> real-world use.\n>\n> However, I think it's conceptually a bad idea to abort with a BUG() an\n> otherwise successful Git operation by default, when that bug happens\n> to be in such an ancillary component as the progress display, and find\n> the justifications given in the commit message unconvincing.\n\nI think it's important to get to the root of what we really disagree\nabout here.\n\nWe hard die on all sorts of trivial things in BUG() already, as paging\nthrough this shows:\n\n    git grep -w -F 'BUG('\n\nI.e. the general criteria for BUG() isn't whether we could\nhypothetically recover from the scenario.\n\nWhich, if you adjust the code around many existing BUG()s to trigger the\n\"bad thing\" (as it were..) is the case. I haven't done any exhaustive\nchecking, but I'd bet it's more common than not, or at least something\nlike 1/4 or 1/8 of them.\n\nRather it's a combination of:\n\n A. This should absolutely not happen\n B. We are sure (or are sure as we can be) that it doesn't\n C. We think if it does, we'll know/really want to know via tests/early integration testing.\n D. We mark it with a BUG() because we're as confident as we reasonably can be in A and B\n\nI took your earlier comments to mean that you agreed on \"A\", but you\nweren't so sure about \"B\". I agree that without confidence about \"B\"\nthat having \"D\" would be overzealous.\n\nBut here with your mention of \"conceptually a bad idea to[...]\" I'm not\nso sure, i.e. it seems like you're saying that you're categorically\nopposed to such use of BUG(). I think that's a fair stance to take if\nthat's what you're going for, but per the above I also think it's true\nto say that a lot of our existing use of that API doesn't match that\nworldview. I.e. it's not an UNRECOVARABLE_BUG(), but\nSHOULD_NOT_ESCAPE_DEVELOPMENT_BUG().\n\nI think that in this case we should be confident that we've got \"B\", as\nexplained in detail in the upthread patch[5].\n\nI would be exceedingly paranoid about more exhaustive checks, e.g. I\ndon't think we've got enough coverage to have BUG() checks for some of\nthe things you added test-only asserts (which would be good) for in your\ninitial series.\n\nI.e. asserting whether we hit 99% and not 100%, whether\ndisplay_progress() \"misses\" updates, or if we miss a call to\nstop_progress(), or whatever else one might assert.\n\nI've been running my personal git with many of those sorts of pedantic\nassertions for a while now, but I'm still not confident those changes\nare OK to submit for inclusion. I very much share your concerns in that\narea.\n\nBut I think (and [5] argues) that it's *much* easier to be confident\nabout the narrow BUG() assertion being added here.\n\nI.e. it's trivial to discover that almost all callers of progress.c are\nnever going to start more than one progress bar ever, so we can discard\nthose out of hand. For the rest they're rather easily auditable.\n\nBut clearly you disagree, and I thinks it helps neither of us to just\nrepeat our points, which is not what I'm trying to do here, but to get\nto the following:\n\n - Can you cite specifically what about [5] you think wouldn't catch any\n   cases where we couldn't be certain about the BUG(). I.e. it outlines\n   the sort of testing I did, which involved forcing all hidden progress bars\n   to ignore their existing isatty() checks, adding more pedantic asserts\n   to narrow down potentially affected callers etc.\n\n   Do you think that testing could have missed something? How & why?\n\n - Related to that, are there cases where you would agree that we've got \"B\",\n   as opposed to others where we don't?\n\n   As I've argued I do think these patches are ready as-is, but one alternative\n   way forward with them would be to add the BUG() to everything, but whitelist\n   those current callers that have >1 progress bar. Then as a (slow) follow-up\n   proceed to un-whitelist those one at a time.\n\nI'm not familiar with the GIT_TEST_SPLIT_INDEX changes you're\nmentioning, but I'd think that those cases would have been more complex\n(the code flow around index-related stuff) than the progress API users.\n\n1. https://lore.kernel.org/git/87o8c8z105.fsf@evledraar.gmail.com/\n2. https://lore.kernel.org/git/cover-00.25-00000000000-20210623T155626Z-avarab@gmail.com/\n3. https://lore.kernel.org/git/cover-v4-0.2-00000000000-20210909T010722Z-avarab@gmail.com/\n4. https://lore.kernel.org/git/cover-0.3-0000000000-20210722T121801Z-avarab@gmail.com/\n5. https://lore.kernel.org/git/patch-v6-8.8-bff919994b5-20211102T122507Z-avarab@gmail.com/\n"},{"id":"447659","messageId":"patch-v9-1.9-dc304df9468-20220203T213350Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v9-0.9-00000000000-20220203T213350Z-avarab@gmail.com","subject":"[PATCH v9 1/9] leak tests: fix a memory leak in \"test-progress\" helper","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-03T21:40:11Z","receivedAt":"2022-02-03T21:40:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a memory leak in the test-progress helper, and mark the\ncorresponding \"t0500-progress-display.sh\" test as being leak-free\nunder SANITIZE=leak. This fixes a leak added in 2bb74b53a4 (Test the\nprogress display, 2019-09-16).\n\nMy 48f68715b14 (tr2: stop leaking \"thread_name\" memory, 2021-08-27)\nhad fixed another memory leak in this test (as it did some trace2\ntesting).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/helper/test-progress.c    | 1 +\n t/t0500-progress-display.sh | 1 +\n 2 files changed, 2 insertions(+)\n\ndiff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\nindex 5d05cbe7894..9265e6ab7cf 100644\n--- a/t/helper/test-progress.c\n+++ b/t/helper/test-progress.c\n@@ -69,6 +69,7 @@ int cmd__progress(int argc, const char **argv)\n \t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n \t}\n \tstop_progress(&progress);\n+\tstrbuf_release(&line);\n \n \treturn 0;\n }\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 22058b503ac..f37cf2eb9c9 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -2,6 +2,7 @@\n \n test_description='progress display'\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n show_cr () {\n-- \n2.35.1.939.g42bf83caa3d\n\n"},{"id":"447660","messageId":"cover-v9-0.9-00000000000-20220203T213350Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v6-0.8-00000000000-20211102T122507Z-avarab@gmail.com","subject":"[PATCH v9 0/9] progress: test fixes / cleanup","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-03T21:40:10Z","receivedAt":"2022-02-03T21:40:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"A re-roll of this series of various basic fixes to the progress API.\n\nThis addresses all the comments brought up on v8, thanks a lot to René\nand Johannes Altmanninger. It just took me about to a month to get\naround to re-rolling this.\n\nThere's various small fixes early in the series. I amended some commit\nmessages, added a mising promised test etc.\n\nBut most significantly as René pointed out stop_progress() and\nstop_progress_msg() should really be unified. This series does that,\nwhich fixes a long-standing bug with the trace2 region not being ended\nif the stop_progress_msg() variant was called.\n\nThis is a reply to the v6, which is the thread this was originally\nin[1]. In v7[2] I didn't add In-Reply-T, and in v8[3] I ended up\nreplying to the wrong thread! Sorry about that.\n\nI then ejected the isatty() change. That was done for a final addition\nof a BUG() assertion that's now gone, but which I plan to submit after\nthis series. I can re-visit that if and when I re-roll that.\n\n1. https://lore.kernel.org/git/cover-v6-0.8-00000000000-20211102T122507Z-avarab@gmail.com\n2. https://lore.kernel.org/git/cover-v7-0.7-00000000000-20211217T041945Z-avarab@gmail.com\n3. https://lore.kernel.org/git/cover-v8-0.7-00000000000-20211228T150728Z-avarab@gmail.com\n\nÆvar Arnfjörð Bjarmason (9):\n  leak tests: fix a memory leak in \"test-progress\" helper\n  progress.c test helper: add missing braces\n  progress.c tests: make start/stop commands on stdin\n  progress.c tests: test some invalid usage\n  progress.h: format and be consistent with progress.c naming\n  progress.c: use dereferenced \"progress\" variable, not \"(*p_progress)\"\n  progress.c: refactor stop_progress{,_msg}() to use helpers\n  progress API: unify stop_progress{,_msg}(), fix trace2 bug\n  pack-bitmap-write.c: don't return without stop_progress()\n\n pack-bitmap-write.c         |   6 +-\n progress.c                  |  66 ++++++++++------------\n progress.h                  |   9 ++-\n t/helper/test-progress.c    |  50 ++++++++++++-----\n t/t0500-progress-display.sh | 109 +++++++++++++++++++++++++++++-------\n t/t5316-pack-delta-depth.sh |   6 +-\n 6 files changed, 170 insertions(+), 76 deletions(-)\n\nRange-diff against v8:\n 1:  aa08dab654d =  1:  dc304df9468 leak tests: fix a memory leak in \"test-progress\" helper\n 2:  3ecdab074b6 =  2:  6951c059aa9 progress.c test helper: add missing braces\n 3:  271f6d7ec3b !  3:  4548144a0b4 progress.c tests: make start/stop commands on stdin\n    @@ t/helper/test-progress.c\n     +\t\tif (skip_prefix(line.buf, \"start \", (const char **) &end)) {\n     +\t\t\tuint64_t total = strtoull(end, &end, 10);\n     +\t\t\tconst char *title;\n    -+\t\t\tconst char *str;\n     +\n     +\t\t\t/*\n     +\t\t\t * We can't use \"end + 1\" as an argument to\n    @@ t/helper/test-progress.c\n     +\t\t\telse\n     +\t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n     +\n    -+\t\t\tstr = title ? title : default_title;\n    -+\t\t\tprogress = start_progress(str, total);\n    ++\t\t\tprogress = start_progress(title, total);\n     +\t\t} else if (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n      \t\t\tuint64_t item_count = strtoull(end, &end, 10);\n      \t\t\tif (*end != '\\0')\n 4:  7c1b8b287c5 !  4:  965d4ba7b54 progress.c tests: test some invalid usage\n    @@ Commit message\n         progress.c tests: test some invalid usage\n     \n         Test what happens when we \"stop\" without a \"start\", omit the \"stop\"\n    -    after a \"start\", or try to start two concurrent progress bars. This\n    +    after a \"start\", or start two concurrent progress bars. This\n         extends the trace2 tests added in 98a13647408 (trace2: log progress\n         time and throughput, 2020-05-12).\n     \n    @@ t/t0500-progress-display.sh: test_expect_success 'progress generates traces' '\n     +\tstop\n     +\tEOF\n     +\n    -+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-startstop.event\" test-tool progress \\\n    ++\tGIT_TRACE2_EVENT=\"$PWD/trace-startstop.event\" test-tool progress \\\n     +\t\t<in 2>stderr &&\n     +\ttest_region progress \"Working hard\" trace-startstop.event\n     +'\n    @@ t/t0500-progress-display.sh: test_expect_success 'progress generates traces' '\n     +\tstart 0\n     +\tEOF\n     +\n    -+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-start.event\" \\\n    ++\tGIT_TRACE2_EVENT=\"$PWD/trace-start.event\" \\\n     +\tLSAN_OPTIONS=detect_leaks=0 \\\n     +\ttest-tool progress \\\n     +\t\t<in 2>stderr &&\n    @@ t/t0500-progress-display.sh: test_expect_success 'progress generates traces' '\n     +\tstop\n     +\tEOF\n     +\n    -+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-stop.event\" test-tool progress \\\n    ++\tGIT_TRACE2_EVENT=\"$PWD/trace-stop.event\" test-tool progress \\\n     +\t\t<in 2>stderr &&\n     +\t! grep region_enter.*progress trace-stop.event &&\n     +\t! grep region_leave.*progress trace-stop.event\n     +'\n    ++\n    ++test_expect_success 'progress generates traces: start with active progress bar (no stops)' '\n    ++\tcat >in <<-\\EOF &&\n    ++\tstart 0 One\n    ++\tstart 0 Two\n    ++\tEOF\n    ++\n    ++\tGIT_TRACE2_EVENT=\"$PWD/trace-2start.event\" \\\n    ++\tLSAN_OPTIONS=detect_leaks=0 \\\n    ++\ttest-tool progress \\\n    ++\t\t<in 2>stderr &&\n    ++\tgrep region_enter.*progress.*One trace-2start.event &&\n    ++\tgrep region_enter.*progress.*Two trace-2start.event &&\n    ++\t! grep region_leave trace-2start.event\n    ++'\n     +\n      test_done\n -:  ----------- >  5:  4ab4eb3a20a progress.h: format and be consistent with progress.c naming\n 5:  72a31bd7191 !  6:  ab24cb78d73 progress.c: add temporary variable from progress struct\n    @@ Metadata\n     Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n     \n      ## Commit message ##\n    -    progress.c: add temporary variable from progress struct\n    +    progress.c: use dereferenced \"progress\" variable, not \"(*p_progress)\"\n     \n         Since 98a13647408 (trace2: log progress time and throughput,\n         2020-05-12) stop_progress() dereferences a \"struct progress **\"\n    -    parameter in several places. Extract a dereferenced variable (like in\n    -    stop_progress_msg()) to reduce clutter and make it clearer who needs\n    -    to write to this parameter.\n    +    parameter in several places. Extract a dereferenced variable to reduce\n    +    clutter and make it clearer who needs to write to this parameter.\n     \n         Now instead of using \"*p_progress\" several times in stop_progress() we\n         check it once for NULL and then use a dereferenced \"progress\" variable\n    -    thereafter. This continues the same pattern used in the above\n    -    stop_progress() function, see ac900fddb7f (progress: don't dereference\n    -    before checking for NULL, 2020-08-10).\n    +    thereafter. This uses the same pattern as the adjacent\n    +    stop_progress_msg() function, see ac900fddb7f (progress: don't\n    +    dereference before checking for NULL, 2020-08-10).\n     \n         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n     \n    @@ progress.c: static void finish_if_sparse(struct progress *progress)\n      void stop_progress(struct progress **p_progress)\n      {\n     +\tstruct progress *progress;\n    ++\n      \tif (!p_progress)\n      \t\tBUG(\"don't provide NULL to stop_progress\");\n     +\tprogress = *p_progress;\n -:  ----------- >  7:  3406d71b499 progress.c: refactor stop_progress{,_msg}() to use helpers\n -:  ----------- >  8:  62a93bb98b0 progress API: unify stop_progress{,_msg}(), fix trace2 bug\n 6:  0bd08e1b018 =  9:  0b2248a2d74 pack-bitmap-write.c: don't return without stop_progress()\n 7:  060483fb5ce <  -:  ----------- *.c: use isatty(0|2), not isatty(STDIN_FILENO|STDERR_FILENO)\n-- \n2.35.1.939.g42bf83caa3d\n\n"},{"id":"447661","messageId":"patch-v9-2.9-6951c059aa9-20220203T213350Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v9-0.9-00000000000-20220203T213350Z-avarab@gmail.com","subject":"[PATCH v9 2/9] progress.c test helper: add missing braces","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-03T21:40:12Z","receivedAt":"2022-02-03T21:40:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"If we have braces on one arm of an if/else all of them should have it,\nper the CodingGuidelines's \"When there are multiple arms to a\nconditional[...]\" advice. This formatting change makes a subsequent\ncommit smaller.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/helper/test-progress.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\nindex 9265e6ab7cf..50fd3be3dad 100644\n--- a/t/helper/test-progress.c\n+++ b/t/helper/test-progress.c\n@@ -63,10 +63,11 @@ int cmd__progress(int argc, const char **argv)\n \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n \t\t\tprogress_test_ns = test_ms * 1000 * 1000;\n \t\t\tdisplay_throughput(progress, byte_count);\n-\t\t} else if (!strcmp(line.buf, \"update\"))\n+\t\t} else if (!strcmp(line.buf, \"update\")) {\n \t\t\tprogress_test_force_update();\n-\t\telse\n+\t\t} else {\n \t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n+\t\t}\n \t}\n \tstop_progress(&progress);\n \tstrbuf_release(&line);\n-- \n2.35.1.939.g42bf83caa3d\n\n"},{"id":"447662","messageId":"patch-v9-4.9-965d4ba7b54-20220203T213350Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v9-0.9-00000000000-20220203T213350Z-avarab@gmail.com","subject":"[PATCH v9 4/9] progress.c tests: test some invalid usage","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-03T21:40:14Z","receivedAt":"2022-02-03T21:40:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Test what happens when we \"stop\" without a \"start\", omit the \"stop\"\nafter a \"start\", or start two concurrent progress bars. This\nextends the trace2 tests added in 98a13647408 (trace2: log progress\ntime and throughput, 2020-05-12).\n\nThese tests are not merely testing the helper, but invalid API usage\nthat can happen if the progress.c API is misused.\n\nThe \"without stop\" test will leak under SANITIZE=leak, since this\nbuggy use of the API will leak memory. But let's not skip it entirely,\nor use the \"!SANITIZE_LEAK\" prerequisite check as we'd do with tests\nthat we're skipping due to leaks we haven't fixed yet. Instead\nannotate the specific command that should skip leak checking with\ncustom $LSAN_OPTIONS[1].\n\n1. https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/t0500-progress-display.sh | 50 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 50 insertions(+)\n\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex 27ab4218b01..1eb3a8306ba 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -325,4 +325,54 @@ test_expect_success 'progress generates traces' '\n \tgrep \"\\\"key\\\":\\\"total_bytes\\\",\\\"value\\\":\\\"409600\\\"\" trace.event\n '\n \n+test_expect_success 'progress generates traces: stop / start' '\n+\tcat >in <<-\\EOF &&\n+\tstart 0\n+\tstop\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/trace-startstop.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n+\ttest_region progress \"Working hard\" trace-startstop.event\n+'\n+\n+test_expect_success 'progress generates traces: start without stop' '\n+\tcat >in <<-\\EOF &&\n+\tstart 0\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/trace-start.event\" \\\n+\tLSAN_OPTIONS=detect_leaks=0 \\\n+\ttest-tool progress \\\n+\t\t<in 2>stderr &&\n+\tgrep region_enter.*progress trace-start.event &&\n+\t! grep region_leave.*progress trace-start.event\n+'\n+\n+test_expect_success 'progress generates traces: stop without start' '\n+\tcat >in <<-\\EOF &&\n+\tstop\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/trace-stop.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n+\t! grep region_enter.*progress trace-stop.event &&\n+\t! grep region_leave.*progress trace-stop.event\n+'\n+\n+test_expect_success 'progress generates traces: start with active progress bar (no stops)' '\n+\tcat >in <<-\\EOF &&\n+\tstart 0 One\n+\tstart 0 Two\n+\tEOF\n+\n+\tGIT_TRACE2_EVENT=\"$PWD/trace-2start.event\" \\\n+\tLSAN_OPTIONS=detect_leaks=0 \\\n+\ttest-tool progress \\\n+\t\t<in 2>stderr &&\n+\tgrep region_enter.*progress.*One trace-2start.event &&\n+\tgrep region_enter.*progress.*Two trace-2start.event &&\n+\t! grep region_leave trace-2start.event\n+'\n+\n test_done\n-- \n2.35.1.939.g42bf83caa3d\n\n"},{"id":"447663","messageId":"patch-v9-8.9-62a93bb98b0-20220203T213350Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v9-0.9-00000000000-20220203T213350Z-avarab@gmail.com","subject":"[PATCH v9 8/9] progress API: unify stop_progress{,_msg}(), fix trace2 bug","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-03T21:40:18Z","receivedAt":"2022-02-03T21:40:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a bug that's been with us ever since 98a13647408 (trace2: log\nprogress time and throughput, 2020-05-12), when the\nstop_progress_msg() API was used we didn't log a \"region_leave\" for\nthe \"region_enter\" we start in \"start_progress_delay()\".\n\nThe only user of the \"stop_progress_msg()\" function is\n\"index-pack\". Let's add a previously failing test to check that we\nhave the same number of \"region_enter\" and \"region_leave\" events, with\n\"-v\" we'll log progress even in the test environment.\n\nIn addition to that we've had a submarine bug here introduced with\n9d81ecb52b5 (progress: add sparse mode to force 100% complete message,\n2019-03-21). The \"start_sparse_progress()\" API would only do the right\nthing if the progress was ended with \"stop_progress()\", not\n\"stop_progress_msg()\".\n\nThe only user of that API uses \"stop_progress()\", but let's still fix\nthat along with the trace2 issue by making \"stop_progress()\" a trivial\nwrapper for \"stop_progress_msg()\".\n\nWe can also drop the \"if (progress)\" test from\n\"finish_if_sparse()\". It's now a helper for the small\n\"stop_progress_msg()\" function. We'll already have returned from it if\n\"progress\" is \"NULL\".\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c                  | 21 +++------------------\n progress.h                  |  6 +++++-\n t/t5316-pack-delta-depth.sh |  6 +++++-\n 3 files changed, 13 insertions(+), 20 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 6cc7f902f5e..0cdd875d37f 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -311,8 +311,7 @@ struct progress *start_delayed_sparse_progress(const char *title,\n \n static void finish_if_sparse(struct progress *progress)\n {\n-\tif (progress &&\n-\t    progress->sparse &&\n+\tif (progress->sparse &&\n \t    progress->last_value != progress->total)\n \t\tdisplay_progress(progress, progress->total);\n }\n@@ -347,22 +346,6 @@ static void log_trace2(struct progress *progress)\n \ttrace2_region_leave(\"progress\", progress->title, the_repository);\n }\n \n-void stop_progress(struct progress **p_progress)\n-{\n-\tstruct progress *progress;\n-\n-\tif (!p_progress)\n-\t\tBUG(\"don't provide NULL to stop_progress\");\n-\tprogress = *p_progress;\n-\n-\tfinish_if_sparse(progress);\n-\n-\tif (progress)\n-\t\tlog_trace2(*p_progress);\n-\n-\tstop_progress_msg(p_progress, _(\"done\"));\n-}\n-\n void stop_progress_msg(struct progress **p_progress, const char *msg)\n {\n \tstruct progress *progress;\n@@ -375,8 +358,10 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)\n \t\treturn;\n \t*p_progress = NULL;\n \n+\tfinish_if_sparse(progress);\n \tif (progress->last_value != -1)\n \t\tforce_last_update(progress, msg);\n+\tlog_trace2(progress);\n \n \tclear_progress_signal();\n \tstrbuf_release(&progress->counters_sb);\ndiff --git a/progress.h b/progress.h\nindex 4f6806904a8..3a945637c81 100644\n--- a/progress.h\n+++ b/progress.h\n@@ -1,5 +1,6 @@\n #ifndef PROGRESS_H\n #define PROGRESS_H\n+#include \"gettext.h\"\n \n struct progress;\n \n@@ -19,5 +20,8 @@ struct progress *start_delayed_progress(const char *title, uint64_t total);\n struct progress *start_delayed_sparse_progress(const char *title,\n \t\t\t\t\t       uint64_t total);\n void stop_progress_msg(struct progress **p_progress, const char *msg);\n-void stop_progress(struct progress **p_progress);\n+static inline void stop_progress(struct progress **p_progress)\n+{\n+\tstop_progress_msg(p_progress, _(\"done\"));\n+}\n #endif\ndiff --git a/t/t5316-pack-delta-depth.sh b/t/t5316-pack-delta-depth.sh\nindex df524f7b6dd..e9045009a11 100755\n--- a/t/t5316-pack-delta-depth.sh\n+++ b/t/t5316-pack-delta-depth.sh\n@@ -64,7 +64,11 @@ test_expect_success 'create series of packs' '\n \t\t\techo $cur &&\n \t\t\techo \"$(git rev-parse :file) file\"\n \t\t} | git pack-objects --stdout >tmp &&\n-\t\tgit index-pack --stdin --fix-thin <tmp || return 1\n+\t\tGIT_TRACE2_EVENT=$PWD/trace \\\n+\t\tgit index-pack -v --stdin --fix-thin <tmp || return 1 &&\n+\t\tgrep -c region_enter.*progress trace >enter &&\n+\t\tgrep -c region_leave.*progress trace >leave &&\n+\t\ttest_cmp enter leave &&\n \t\tprev=$cur\n \tdone\n '\n-- \n2.35.1.939.g42bf83caa3d\n\n"},{"id":"447664","messageId":"patch-v9-9.9-0b2248a2d74-20220203T213350Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v9-0.9-00000000000-20220203T213350Z-avarab@gmail.com","subject":"[PATCH v9 9/9] pack-bitmap-write.c: don't return without stop_progress()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-03T21:40:19Z","receivedAt":"2022-02-03T21:40:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix a bug that's been here since 7cc8f971085 (pack-objects: implement\nbitmap writing, 2013-12-21), we did not call stop_progress() if we\nreached the early exit in this function.\n\nWe could call stop_progress() before we return, but better yet is to\ndefer calling start_progress() until we need it. For now this only\nmatters in practice because we'd previously omit the \"region_leave\"\nfor the progress trace2 event.\n\nSuggested-by: SZEDER Gábor <szeder.dev@gmail.com>\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n pack-bitmap-write.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/pack-bitmap-write.c b/pack-bitmap-write.c\nindex 9c55c1531e1..cab3eaa2acd 100644\n--- a/pack-bitmap-write.c\n+++ b/pack-bitmap-write.c\n@@ -575,15 +575,15 @@ void bitmap_writer_select_commits(struct commit **indexed_commits,\n \n \tQSORT(indexed_commits, indexed_commits_nr, date_compare);\n \n-\tif (writer.show_progress)\n-\t\twriter.progress = start_progress(\"Selecting bitmap commits\", 0);\n-\n \tif (indexed_commits_nr < 100) {\n \t\tfor (i = 0; i < indexed_commits_nr; ++i)\n \t\t\tpush_bitmapped_commit(indexed_commits[i]);\n \t\treturn;\n \t}\n \n+\tif (writer.show_progress)\n+\t\twriter.progress = start_progress(\"Selecting bitmap commits\", 0);\n+\n \tfor (;;) {\n \t\tstruct commit *chosen = NULL;\n \n-- \n2.35.1.939.g42bf83caa3d\n\n"},{"id":"447665","messageId":"patch-v9-3.9-4548144a0b4-20220203T213350Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v9-0.9-00000000000-20220203T213350Z-avarab@gmail.com","subject":"[PATCH v9 3/9] progress.c tests: make start/stop commands on stdin","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-03T21:40:13Z","receivedAt":"2022-02-03T21:40:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the usage of the \"test-tool progress\" introduced in\n2bb74b53a49 (Test the progress display, 2019-09-16) to take command\nlike \"start\" and \"stop\" on stdin, instead of running them implicitly.\n\nThis makes for tests that are easier to read, since the recipe will\nmirror the API usage, and allows for easily testing invalid usage that\nwould yield (or should yield) a BUG(), e.g. providing two \"start\"\ncalls in a row. A subsequent commit will add such tests.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n t/helper/test-progress.c    | 44 +++++++++++++++++++++-------\n t/t0500-progress-display.sh | 58 +++++++++++++++++++++++--------------\n 2 files changed, 70 insertions(+), 32 deletions(-)\n\ndiff --git a/t/helper/test-progress.c b/t/helper/test-progress.c\nindex 50fd3be3dad..6cc9735b601 100644\n--- a/t/helper/test-progress.c\n+++ b/t/helper/test-progress.c\n@@ -3,6 +3,9 @@\n  *\n  * Reads instructions from standard input, one instruction per line:\n  *\n+ *   \"start <total>[ <title>]\" - Call start_progress(title, total),\n+ *                               Uses the default title of \"Working hard\"\n+ *                               if the \" <title>\" is omitted.\n  *   \"progress <items>\" - Call display_progress() with the given item count\n  *                        as parameter.\n  *   \"throughput <bytes> <millis> - Call display_throughput() with the given\n@@ -10,6 +13,7 @@\n  *                                  specify the time elapsed since the\n  *                                  start_progress() call.\n  *   \"update\" - Set the 'progress_update' flag.\n+ *   \"stop\" - Call stop_progress().\n  *\n  * See 't0500-progress-display.sh' for examples.\n  */\n@@ -19,34 +23,50 @@\n #include \"parse-options.h\"\n #include \"progress.h\"\n #include \"strbuf.h\"\n+#include \"string-list.h\"\n \n int cmd__progress(int argc, const char **argv)\n {\n-\tint total = 0;\n-\tconst char *title;\n+\tconst char *const default_title = \"Working hard\";\n+\tstruct string_list titles = STRING_LIST_INIT_DUP;\n \tstruct strbuf line = STRBUF_INIT;\n-\tstruct progress *progress;\n+\tstruct progress *progress = NULL;\n \n \tconst char *usage[] = {\n-\t\t\"test-tool progress [--total=<n>] <progress-title>\",\n+\t\t\"test-tool progress <stdin\",\n \t\tNULL\n \t};\n \tstruct option options[] = {\n-\t\tOPT_INTEGER(0, \"total\", &total, \"total number of items\"),\n \t\tOPT_END(),\n \t};\n \n \targc = parse_options(argc, argv, NULL, options, usage, 0);\n-\tif (argc != 1)\n-\t\tdie(\"need a title for the progress output\");\n-\ttitle = argv[0];\n+\tif (argc)\n+\t\tusage_with_options(usage, options);\n \n \tprogress_testing = 1;\n-\tprogress = start_progress(title, total);\n \twhile (strbuf_getline(&line, stdin) != EOF) {\n \t\tchar *end;\n \n-\t\tif (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n+\t\tif (skip_prefix(line.buf, \"start \", (const char **) &end)) {\n+\t\t\tuint64_t total = strtoull(end, &end, 10);\n+\t\t\tconst char *title;\n+\n+\t\t\t/*\n+\t\t\t * We can't use \"end + 1\" as an argument to\n+\t\t\t * start_progress(), it doesn't xstrdup() its\n+\t\t\t * \"title\" argument. We need to hold onto a\n+\t\t\t * valid \"char *\" for it until the end.\n+\t\t\t */\n+\t\t\tif (!*end)\n+\t\t\t\ttitle = default_title;\n+\t\t\telse if (*end == ' ')\n+\t\t\t\ttitle = string_list_insert(&titles, end + 1)->string;\n+\t\t\telse\n+\t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n+\n+\t\t\tprogress = start_progress(title, total);\n+\t\t} else if (skip_prefix(line.buf, \"progress \", (const char **) &end)) {\n \t\t\tuint64_t item_count = strtoull(end, &end, 10);\n \t\t\tif (*end != '\\0')\n \t\t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n@@ -65,12 +85,14 @@ int cmd__progress(int argc, const char **argv)\n \t\t\tdisplay_throughput(progress, byte_count);\n \t\t} else if (!strcmp(line.buf, \"update\")) {\n \t\t\tprogress_test_force_update();\n+\t\t} else if (!strcmp(line.buf, \"stop\")) {\n+\t\t\tstop_progress(&progress);\n \t\t} else {\n \t\t\tdie(\"invalid input: '%s'\\n\", line.buf);\n \t\t}\n \t}\n-\tstop_progress(&progress);\n \tstrbuf_release(&line);\n+\tstring_list_clear(&titles, 0);\n \n \treturn 0;\n }\ndiff --git a/t/t0500-progress-display.sh b/t/t0500-progress-display.sh\nindex f37cf2eb9c9..27ab4218b01 100755\n--- a/t/t0500-progress-display.sh\n+++ b/t/t0500-progress-display.sh\n@@ -18,6 +18,7 @@ test_expect_success 'simple progress display' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tupdate\n \tprogress 1\n \tupdate\n@@ -26,8 +27,9 @@ test_expect_success 'simple progress display' '\n \tprogress 4\n \tupdate\n \tprogress 5\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -42,11 +44,13 @@ test_expect_success 'progress display with total' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 3\n \tprogress 1\n \tprogress 2\n \tprogress 3\n+\tstop\n \tEOF\n-\ttest-tool progress --total=3 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -63,14 +67,14 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tprogress 100\n \tprogress 1000\n \tprogress 10000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -89,16 +93,16 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tupdate\n \tprogress 1\n \tupdate\n \tprogress 2\n \tprogress 10000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -117,14 +121,14 @@ Working hard.......2.........3.........4.........5.........6:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6\n \tprogress 25000\n \tprogress 50000\n \tprogress 75000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -141,14 +145,14 @@ Working hard.......2.........3.........4.........5.........6.........7.........:\n EOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 100000 Working hard.......2.........3.........4.........5.........6.........7.........\n \tprogress 25000\n \tprogress 50000\n \tprogress 75000\n \tprogress 100000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=100000 \\\n-\t\t\"Working hard.......2.........3.........4.........5.........6.........7.........\" \\\n-\t\t<in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -165,12 +169,14 @@ test_expect_success 'progress shortens - crazy caller' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 1000\n \tprogress 100\n \tprogress 200\n \tprogress 1\n \tprogress 1000\n+\tstop\n \tEOF\n-\ttest-tool progress --total=1000 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -186,6 +192,7 @@ test_expect_success 'progress display with throughput' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tthroughput 102400 1000\n \tupdate\n \tprogress 10\n@@ -198,8 +205,9 @@ test_expect_success 'progress display with throughput' '\n \tthroughput 409600 4000\n \tupdate\n \tprogress 40\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -215,6 +223,7 @@ test_expect_success 'progress display with throughput and total' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 40\n \tthroughput 102400 1000\n \tprogress 10\n \tthroughput 204800 2000\n@@ -223,8 +232,9 @@ test_expect_success 'progress display with throughput and total' '\n \tprogress 30\n \tthroughput 409600 4000\n \tprogress 40\n+\tstop\n \tEOF\n-\ttest-tool progress --total=40 \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -240,6 +250,7 @@ test_expect_success 'cover up after throughput shortens' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tthroughput 409600 1000\n \tupdate\n \tprogress 1\n@@ -252,8 +263,9 @@ test_expect_success 'cover up after throughput shortens' '\n \tthroughput 1638400 4000\n \tupdate\n \tprogress 4\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -268,6 +280,7 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \tEOF\n \n \tcat >in <<-\\EOF &&\n+\tstart 0\n \tthroughput 1 1000\n \tupdate\n \tprogress 1\n@@ -277,8 +290,9 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \tthroughput 3145728 3000\n \tupdate\n \tprogress 3\n+\tstop\n \tEOF\n-\ttest-tool progress \"Working hard\" <in 2>stderr &&\n+\ttest-tool progress <in 2>stderr &&\n \n \tshow_cr <stderr >out &&\n \ttest_cmp expect out\n@@ -286,6 +300,7 @@ test_expect_success 'cover up after throughput shortens a lot' '\n \n test_expect_success 'progress generates traces' '\n \tcat >in <<-\\EOF &&\n+\tstart 40\n \tthroughput 102400 1000\n \tupdate\n \tprogress 10\n@@ -298,10 +313,11 @@ test_expect_success 'progress generates traces' '\n \tthroughput 409600 4000\n \tupdate\n \tprogress 40\n+\tstop\n \tEOF\n \n-\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" test-tool progress --total=40 \\\n-\t\t\"Working hard\" <in 2>stderr &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" test-tool progress \\\n+\t\t<in 2>stderr &&\n \n \t# t0212/parse_events.perl intentionally omits regions and data.\n \ttest_region progress \"Working hard\" trace.event &&\n-- \n2.35.1.939.g42bf83caa3d\n\n"},{"id":"447666","messageId":"patch-v9-5.9-4ab4eb3a20a-20220203T213350Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v9-0.9-00000000000-20220203T213350Z-avarab@gmail.com","subject":"[PATCH v9 5/9] progress.h: format and be consistent with progress.c naming","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-03T21:40:15Z","receivedAt":"2022-02-03T21:42:35Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix an inconsistency introduced in dc6a0757c4f (make struct progress\nan opaque type, 2007-10-30) and rename the \"progress\" parameters to\nstop_progress{,_msg}() to \"p_progress\". Now these match the\ncorresponding parameters in the *.c code.\n\nWhile we're at it let's move the definition of the former below the\nlatter, a subsequent change will start defining stop_progress() in\nterms of stop_progress_msg(). Let's also remove the excess whitespace\nat the end of the file.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.h | 5 ++---\n 1 file changed, 2 insertions(+), 3 deletions(-)\n\ndiff --git a/progress.h b/progress.h\nindex f1913acf73f..4f6806904a8 100644\n--- a/progress.h\n+++ b/progress.h\n@@ -18,7 +18,6 @@ struct progress *start_sparse_progress(const char *title, uint64_t total);\n struct progress *start_delayed_progress(const char *title, uint64_t total);\n struct progress *start_delayed_sparse_progress(const char *title,\n \t\t\t\t\t       uint64_t total);\n-void stop_progress(struct progress **progress);\n-void stop_progress_msg(struct progress **progress, const char *msg);\n-\n+void stop_progress_msg(struct progress **p_progress, const char *msg);\n+void stop_progress(struct progress **p_progress);\n #endif\n-- \n2.35.1.939.g42bf83caa3d\n\n"},{"id":"447667","messageId":"patch-v9-7.9-3406d71b499-20220203T213350Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v9-0.9-00000000000-20220203T213350Z-avarab@gmail.com","subject":"[PATCH v9 7/9] progress.c: refactor stop_progress{,_msg}() to use helpers","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-03T21:40:17Z","receivedAt":"2022-02-03T21:42:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Create two new static helpers for the stop_progress() and\nstop_progress_msg() functions.\n\nAs we'll see in the subsequent commit having those two split up\ndoesn't make much sense, and results in a bug in how we log to\ntrace2. This narrow preparatory change makes the diff for that\nsubsequent change smaller.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 64 ++++++++++++++++++++++++++++++------------------------\n 1 file changed, 36 insertions(+), 28 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 6e7daa3f8a2..6cc7f902f5e 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -317,6 +317,36 @@ static void finish_if_sparse(struct progress *progress)\n \t\tdisplay_progress(progress, progress->total);\n }\n \n+static void force_last_update(struct progress *progress, const char *msg)\n+{\n+\tchar *buf;\n+\tstruct throughput *tp = progress->throughput;\n+\n+\tif (tp) {\n+\t\tuint64_t now_ns = progress_getnanotime(progress);\n+\t\tunsigned int misecs, rate;\n+\t\tmisecs = ((now_ns - progress->start_ns) * 4398) >> 32;\n+\t\trate = tp->curr_total / (misecs ? misecs : 1);\n+\t\tthroughput_string(&tp->display, tp->curr_total, rate);\n+\t}\n+\tprogress_update = 1;\n+\tbuf = xstrfmt(\", %s.\\n\", msg);\n+\tdisplay(progress, progress->last_value, buf);\n+\tfree(buf);\n+}\n+\n+static void log_trace2(struct progress *progress)\n+{\n+\ttrace2_data_intmax(\"progress\", the_repository, \"total_objects\",\n+\t\t\t   progress->total);\n+\n+\tif (progress->throughput)\n+\t\ttrace2_data_intmax(\"progress\", the_repository, \"total_bytes\",\n+\t\t\t\t   progress->throughput->curr_total);\n+\n+\ttrace2_region_leave(\"progress\", progress->title, the_repository);\n+}\n+\n void stop_progress(struct progress **p_progress)\n {\n \tstruct progress *progress;\n@@ -327,17 +357,8 @@ void stop_progress(struct progress **p_progress)\n \n \tfinish_if_sparse(progress);\n \n-\tif (progress) {\n-\t\ttrace2_data_intmax(\"progress\", the_repository, \"total_objects\",\n-\t\t\t\t   progress->total);\n-\n-\t\tif (progress->throughput)\n-\t\t\ttrace2_data_intmax(\"progress\", the_repository,\n-\t\t\t\t\t   \"total_bytes\",\n-\t\t\t\t\t   progress->throughput->curr_total);\n-\n-\t\ttrace2_region_leave(\"progress\", progress->title, the_repository);\n-\t}\n+\tif (progress)\n+\t\tlog_trace2(*p_progress);\n \n \tstop_progress_msg(p_progress, _(\"done\"));\n }\n@@ -353,23 +374,10 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)\n \tif (!progress)\n \t\treturn;\n \t*p_progress = NULL;\n-\tif (progress->last_value != -1) {\n-\t\t/* Force the last update */\n-\t\tchar *buf;\n-\t\tstruct throughput *tp = progress->throughput;\n-\n-\t\tif (tp) {\n-\t\t\tuint64_t now_ns = progress_getnanotime(progress);\n-\t\t\tunsigned int misecs, rate;\n-\t\t\tmisecs = ((now_ns - progress->start_ns) * 4398) >> 32;\n-\t\t\trate = tp->curr_total / (misecs ? misecs : 1);\n-\t\t\tthroughput_string(&tp->display, tp->curr_total, rate);\n-\t\t}\n-\t\tprogress_update = 1;\n-\t\tbuf = xstrfmt(\", %s.\\n\", msg);\n-\t\tdisplay(progress, progress->last_value, buf);\n-\t\tfree(buf);\n-\t}\n+\n+\tif (progress->last_value != -1)\n+\t\tforce_last_update(progress, msg);\n+\n \tclear_progress_signal();\n \tstrbuf_release(&progress->counters_sb);\n \tif (progress->throughput)\n-- \n2.35.1.939.g42bf83caa3d\n\n"},{"id":"447668","messageId":"patch-v9-6.9-ab24cb78d73-20220203T213350Z-avarab@gmail.com","threadId":"55972","inReplyTo":"cover-v9-0.9-00000000000-20220203T213350Z-avarab@gmail.com","subject":"[PATCH v9 6/9] progress.c: use dereferenced \"progress\" variable, not \"(*p_progress)\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-03T21:40:16Z","receivedAt":"2022-02-03T21:42:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Since 98a13647408 (trace2: log progress time and throughput,\n2020-05-12) stop_progress() dereferences a \"struct progress **\"\nparameter in several places. Extract a dereferenced variable to reduce\nclutter and make it clearer who needs to write to this parameter.\n\nNow instead of using \"*p_progress\" several times in stop_progress() we\ncheck it once for NULL and then use a dereferenced \"progress\" variable\nthereafter. This uses the same pattern as the adjacent\nstop_progress_msg() function, see ac900fddb7f (progress: don't\ndereference before checking for NULL, 2020-08-10).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n progress.c | 15 +++++++++------\n 1 file changed, 9 insertions(+), 6 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 680c6a8bf93..6e7daa3f8a2 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -319,21 +319,24 @@ static void finish_if_sparse(struct progress *progress)\n \n void stop_progress(struct progress **p_progress)\n {\n+\tstruct progress *progress;\n+\n \tif (!p_progress)\n \t\tBUG(\"don't provide NULL to stop_progress\");\n+\tprogress = *p_progress;\n \n-\tfinish_if_sparse(*p_progress);\n+\tfinish_if_sparse(progress);\n \n-\tif (*p_progress) {\n+\tif (progress) {\n \t\ttrace2_data_intmax(\"progress\", the_repository, \"total_objects\",\n-\t\t\t\t   (*p_progress)->total);\n+\t\t\t\t   progress->total);\n \n-\t\tif ((*p_progress)->throughput)\n+\t\tif (progress->throughput)\n \t\t\ttrace2_data_intmax(\"progress\", the_repository,\n \t\t\t\t\t   \"total_bytes\",\n-\t\t\t\t\t   (*p_progress)->throughput->curr_total);\n+\t\t\t\t\t   progress->throughput->curr_total);\n \n-\t\ttrace2_region_leave(\"progress\", (*p_progress)->title, the_repository);\n+\t\ttrace2_region_leave(\"progress\", progress->title, the_repository);\n \t}\n \n \tstop_progress_msg(p_progress, _(\"done\"));\n-- \n2.35.1.939.g42bf83caa3d\n\n"},{"id":"447683","messageId":"xmqqtudfplj0.fsf@gitster.g","threadId":"55972","inReplyTo":"cover-v9-0.9-00000000000-20220203T213350Z-avarab@gmail.com","subject":"Re: [PATCH v9 0/9] progress: test fixes / cleanup","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-03T23:39:47Z","receivedAt":"2022-02-03T23:39:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:\n\n> But most significantly as René pointed out stop_progress() and\n> stop_progress_msg() should really be unified. This series does that,\n> which fixes a long-standing bug with the trace2 region not being ended\n> if the stop_progress_msg() variant was called.\n\n;-)\n\nYes, it is very pleasing to see it fixed.\n\nWill take a look and queue but after finishing the week's \"What's\ncooking\" report.\n\nThanks.\n"}]}