threads / patch / 59965

patch, 6 partsfsck: squelch progress output with `--no-progress`

Subject: [PATCH 0/6] fsck: squelch progress output with `--no-progress`

## tl;dr

8 messages between Jul 8, 2023 and Jul 10, 2023. Diffs are folded; open one to read it.

replies: 7people: 2as markdown or json

Taylor Blau· Jul 8, 2023, 00:31 UTC · lore

This short series addresses a pair of issues where the `commit-graph verify` and `multi-pack-index verify` steps of `git fsck` produce output regardless of whether or not `fsck` was invoked with the `--no-progress` option or not.

The first two patches address the commit-graph and MIDX issues respectively. The final four patches further clean up the output of `git commit-graph verify --progress` when verifying multi-layer graphs to produce a single progress meter instead of one per graph layer.

Before, the output of `git commit-graph verify` on a repository with a commit-graph chain with two layers looked like:

    $ git.compile commit-graph verify
    Verifying commits in commit graph: 100% (4356/4356), done.
    Verifying commits in commit graph: 100% (131912/131912), done.
After this patch series, the output instead looks as follows:
    $ git.compile commit-graph verify
    Verifying commits in commit graph: 100% (136268/136268), done.
Thanks in advance for your review.
Taylor Blau (6):
  fsck: suppress commit-graph output with `--no-progress`
  fsck: suppress MIDX output with `--no-progress`
  commit-graph.c: extract `verify_one_commit_graph()`
  commit-graph.c: iteratively verify commit-graph chains
  commit-graph.c: pass progress to `verify_one_commit_graph()`
  commit-graph.c: avoid duplicated progress output during `verify`
 builtin/fsck.c                |  8 ++++++
 commit-graph.c                | 53 +++++++++++++++++++++++------------
 t/t5318-commit-graph.sh       | 10 +++++++
 t/t5319-multi-pack-index.sh   | 12 ++++++++
 t/t5324-split-commit-graph.sh |  3 +-
 5 files changed, 67 insertions(+), 19 deletions(-)
-- 
2.41.0.242.g6eec849fa5a
Taylor Blau· Jul 8, 2023, 00:31 UTC · re: Taylor Blau · lore

[PATCH 1/6] fsck: suppress commit-graph output with `--no-progress`

Since e0fd51e1d7 (fsck: verify commit-graph, 2018-06-27), `fsck` runs `git commit-graph verify` to check the integrity of any commit-graph(s).

Originally, the `git commit-graph verify` step would always print to stdout/stderr, regardless of whether or not `fsck` was invoked with `--[no-]progress` or not. But in 7371612255 (commit-graph: add --[no-]progress to write and verify, 2019-08-26), the commit-graph machinery learned the `--[no-]progress` option, though `fsck` was not updated to pass this new flag (or not).

This led to seeing output from running `git fsck`, even with `--no-progress` on repositories that have a commit-graph:

    $ git.compile fsck --connectivity-only --no-progress --no-dangling
    Verifying commits in commit graph: 100% (4356/4356), done.
    Verifying commits in commit graph: 100% (131912/131912), done.

Ensure that `fsck` passes `--[no-]progress` as appropriate when calling `git commit-graph verify`.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 builtin/fsck.c          |  4 ++++
 t/t5318-commit-graph.sh | 10 ++++++++++
 2 files changed, 14 insertions(+)
Show changes to 2 files +14 −0

builtin/fsck.c, t/t5318-commit-graph.sh

diff --git a/builtin/fsck.c b/builtin/fsck.c
index fa26462337a..e6473ecabc7 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -1074,6 +1074,10 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 			commit_graph_verify.git_cmd = 1;
 			strvec_pushl(&commit_graph_verify.args, "commit-graph",
 				     "verify", "--object-dir", odb->path, NULL);
+			if (show_progress)
+				strvec_push(&commit_graph_verify.args, "--progress");
+			else
+				strvec_push(&commit_graph_verify.args, "--no-progress");
 			if (run_command(&commit_graph_verify))
 				errors_found |= ERROR_COMMIT_GRAPH;
 		}
diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh
index b6e12115786..bf8a92317b3 100755
--- a/t/t5318-commit-graph.sh
+++ b/t/t5318-commit-graph.sh
@@ -684,6 +684,16 @@ test_expect_success 'git fsck (checks commit-graph when config unset)' '
 	test_must_fail git fsck
 '
 
+test_expect_success 'git fsck shows commit-graph output with --progress' '
+	git -C "$TRASH_DIRECTORY/full" fsck --progress 2>err &&
+	grep "Verifying commits in commit graph" err
+'
+
+test_expect_success 'git fsck suppresses commit-graph output with --no-progress' '
+	git -C "$TRASH_DIRECTORY/full" fsck --no-progress 2>err &&
+	! grep "Verifying commits in commit graph" err
+'
+
 test_expect_success 'setup non-the_repository tests' '
 	rm -rf repo &&
 	git init repo &&
-- 
2.41.0.242.g6eec849fa5a
Taylor Blau· Jul 8, 2023, 00:31 UTC · re: Taylor Blau · lore

[PATCH 2/6] fsck: suppress MIDX output with `--no-progress`

In a similar spirit as the previous commit, address a bug where `git fsck` produces output when calling `git multi-pack-index verify` even when invoked with `--no-progress`.

    $ git.compile fsck --connectivity-only --no-progress --no-dangling
    Verifying OID order in multi-pack-index: 100% (605677/605677), done.
    Sorting objects by packfile: 100% (605678/605678), done.
    Verifying object offsets: 100% (605678/605678), done.

The three lines produced by `git fsck` come from `git multi-pack-index verify`, but should be squelched due to `--no-progress`.

The MIDX machinery learned to generate these progress messages as early as 430efb8a74b (midx: add progress indicators in multi-pack-index verify, 2019-03-21), but did not respect `--progress` or `--no-progress` until ad60096d1c8 (midx: honor the MIDX_PROGRESS flag in verify_midx_file, 2019-10-21).

But the `git multi-pack-index verify` step was added to fsck in 66ec0390e75 (fsck: verify multi-pack-index, 2018-09-13), pre-dating any of the above patches.

Pass `--[no-]progress` as appropriate to ensure that we don't produce output when told not to.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 builtin/fsck.c              |  4 ++++
 t/t5319-multi-pack-index.sh | 12 ++++++++++++
 2 files changed, 16 insertions(+)
Show changes to 2 files +16 −0

builtin/fsck.c, t/t5319-multi-pack-index.sh

diff --git a/builtin/fsck.c b/builtin/fsck.c
index e6473ecabc7..8959b3c7b87 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -1092,6 +1092,10 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 			midx_verify.git_cmd = 1;
 			strvec_pushl(&midx_verify.args, "multi-pack-index",
 				     "verify", "--object-dir", odb->path, NULL);
+			if (show_progress)
+				strvec_push(&midx_verify.args, "--progress");
+			else
+				strvec_push(&midx_verify.args, "--no-progress");
 			if (run_command(&midx_verify))
 				errors_found |= ERROR_MULTI_PACK_INDEX;
 		}
diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh
index 0883c7c6bd9..1bcc02004d7 100755
--- a/t/t5319-multi-pack-index.sh
+++ b/t/t5319-multi-pack-index.sh
@@ -485,6 +485,18 @@ test_expect_success 'git-fsck incorrect offset' '
 	git -c core.multiPackIndex=false fsck
 '
 
+test_expect_success 'git fsck shows MIDX output with --progress' '
+	git fsck --progress 2>err &&
+	grep "Verifying OID order in multi-pack-index" err &&
+	grep "Verifying object offsets" err
+'
+
+test_expect_success 'git fsck suppresses MIDX output with --no-progress' '
+	git fsck --no-progress 2>err &&
+	! grep "Verifying OID order in multi-pack-index" err &&
+	! grep "Verifying object offsets" err
+'
+
 test_expect_success 'corrupt MIDX is not reused' '
 	corrupt_midx_and_verify $MIDX_BYTE_OFFSET "\377" $objdir \
 		"incorrect object offset" &&
-- 
2.41.0.242.g6eec849fa5a
Taylor Blau· Jul 8, 2023, 00:31 UTC · re: Taylor Blau · lore

[PATCH 3/6] commit-graph.c: extract `verify_one_commit_graph()`

When the `verify_commit_graph()` function was extended to support commit-graph chains via 3da4b609bb1 (commit-graph: verify chains with --shallow mode, 2019-06-18), it did so by recursively calling itself on each layer of the commit-graph chain.

In practice this poses no issues, since commit-graph chains do not loop, and there are few enough of them that adding additional frames to the stack is not a problem.

A future commit will consolidate the progress output from `git commit-graph verify` when verifying chained commit-graphs to print a single line instead of one progress meter per commit-graph layer. Prepare for this by extracting a routine to verify a single layer of a commit-graph.

Note that `verify_commit_graph()` is still recursive after this patch, but this will change in the subsequent patch.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 commit-graph.c | 24 ++++++++++++++++--------
 1 file changed, 16 insertions(+), 8 deletions(-)
Show changes to commit-graph.c +16 −8
diff --git a/commit-graph.c b/commit-graph.c
index f70afccada4..3d7cc11927d 100644
--- a/commit-graph.c
+++ b/commit-graph.c
@@ -2542,18 +2542,14 @@ static int commit_graph_checksum_valid(struct commit_graph *g)
 	return hashfile_checksum_valid(g->data, g->data_len);
 }
 
-int verify_commit_graph(struct repository *r, struct commit_graph *g, int flags)
+static int verify_one_commit_graph(struct repository *r,
+				   struct commit_graph *g,
+				   int flags)
 {
 	uint32_t i, cur_fanout_pos = 0;
 	struct object_id prev_oid, cur_oid;
 	int generation_zero = 0;
 	struct progress *progress = NULL;
-	int local_error = 0;
-
-	if (!g) {
-		graph_report("no commit-graph file loaded");
-		return 1;
-	}
 
 	verify_commit_graph_error = verify_commit_graph_lite(g);
 	if (verify_commit_graph_error)
@@ -2699,7 +2695,19 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g, int flags)
 	}
 	stop_progress(&progress);
 
-	local_error = verify_commit_graph_error;
+	return verify_commit_graph_error;
+}
+
+int verify_commit_graph(struct repository *r, struct commit_graph *g, int flags)
+{
+	int local_error = 0;
+
+	if (!g) {
+		graph_report("no commit-graph file loaded");
+		return 1;
+	}
+
+	local_error = verify_one_commit_graph(r, g, flags);
 
 	if (!(flags & COMMIT_GRAPH_VERIFY_SHALLOW) && g->base_graph)
 		local_error |= verify_commit_graph(r, g->base_graph, flags);
-- 
2.41.0.242.g6eec849fa5a
Taylor Blau· Jul 8, 2023, 00:31 UTC · re: Taylor Blau · lore

[PATCH 4/6] commit-graph.c: iteratively verify commit-graph chains

Now that we have a function which can verify a single layer of a commit-graph chain, implement `verify_commit_graph()` in terms of iterating over commit-graphs along their `->base_graph` pointers.

This further prepares us to consolidate the progress output of `git commit-graph verify`.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 commit-graph.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
Show changes to commit-graph.c +5 −4
diff --git a/commit-graph.c b/commit-graph.c
index 3d7cc11927d..3c29ea7c706 100644
--- a/commit-graph.c
+++ b/commit-graph.c
@@ -2707,10 +2707,11 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g, int flags)
 		return 1;
 	}
 
-	local_error = verify_one_commit_graph(r, g, flags);
-
-	if (!(flags & COMMIT_GRAPH_VERIFY_SHALLOW) && g->base_graph)
-		local_error |= verify_commit_graph(r, g->base_graph, flags);
+	for (; g; g = g->base_graph) {
+		local_error |= verify_one_commit_graph(r, g, flags);
+		if (flags & COMMIT_GRAPH_VERIFY_SHALLOW)
+			break;
+	}
 
 	return local_error;
 }
-- 
2.41.0.242.g6eec849fa5a
Taylor Blau· Jul 8, 2023, 00:31 UTC · re: Taylor Blau · lore

[PATCH 5/6] commit-graph.c: pass progress to `verify_one_commit_graph()`

This is the final step to prepare for consolidating the output of `git commit-graph verify`. Instead of having each call to `verify_one_commit_graph()` initialize its own progress struct, have the caller pass one in instead.

This patch does not alter the output of `git commit-graph verify`, but the next commit will consolidate the output.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 commit-graph.c | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)
Show changes to commit-graph.c +9 −8
diff --git a/commit-graph.c b/commit-graph.c
index 3c29ea7c706..65dd4edf0e1 100644
--- a/commit-graph.c
+++ b/commit-graph.c
@@ -2544,12 +2544,11 @@ static int commit_graph_checksum_valid(struct commit_graph *g)
 
 static int verify_one_commit_graph(struct repository *r,
 				   struct commit_graph *g,
-				   int flags)
+				   struct progress *progress)
 {
 	uint32_t i, cur_fanout_pos = 0;
 	struct object_id prev_oid, cur_oid;
 	int generation_zero = 0;
-	struct progress *progress = NULL;
 
 	verify_commit_graph_error = verify_commit_graph_lite(g);
 	if (verify_commit_graph_error)
@@ -2600,10 +2599,6 @@ static int verify_one_commit_graph(struct repository *r,
 	if (verify_commit_graph_error & ~VERIFY_COMMIT_GRAPH_ERROR_HASH)
 		return verify_commit_graph_error;
 
-	if (flags & COMMIT_GRAPH_WRITE_PROGRESS)
-		progress = start_progress(_("Verifying commits in commit graph"),
-					g->num_commits);
-
 	for (i = 0; i < g->num_commits; i++) {
 		struct commit *graph_commit, *odb_commit;
 		struct commit_list *graph_parents, *odb_parents;
@@ -2693,7 +2688,6 @@ static int verify_one_commit_graph(struct repository *r,
 				     graph_commit->date,
 				     odb_commit->date);
 	}
-	stop_progress(&progress);
 
 	return verify_commit_graph_error;
 }
@@ -2708,9 +2702,16 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g, int flags)
 	}
 
 	for (; g; g = g->base_graph) {
-		local_error |= verify_one_commit_graph(r, g, flags);
+		struct progress *progress = NULL;
+		if (flags & COMMIT_GRAPH_WRITE_PROGRESS)
+			progress = start_progress(_("Verifying commits in commit graph"),
+						g->num_commits);
+
+		local_error |= verify_one_commit_graph(r, g, progress);
 		if (flags & COMMIT_GRAPH_VERIFY_SHALLOW)
 			break;
+
+		stop_progress(&progress);
 	}
 
 	return local_error;
-- 
2.41.0.242.g6eec849fa5a
Taylor Blau· Jul 8, 2023, 00:31 UTC · re: Taylor Blau · lore

[PATCH 6/6] commit-graph.c: avoid duplicated progress output during `verify`

When `git commit-graph verify` was taught how to verify commit-graph chains in 3da4b609bb1 (commit-graph: verify chains with --shallow mode, 2019-06-18), it produced one line of progress per layer of the commit-graph chain.

    $ git.compile commit-graph verify
    Verifying commits in commit graph: 100% (4356/4356), done.
    Verifying commits in commit graph: 100% (131912/131912), done.

This could be somewhat confusing to users, who may wonder why there are multiple occurrences of "Verifying commits in commit graph".

There are likely good arguments on whether or not there should be one line of progress output per commit-graph layer. On the one hand, the existing output shows us verifying each individual layer of the chain. But on the other hand, the fact that a commit-graph may be stored among multiple layers is an implementation detail that the caller need not be aware of.

Clarify this by showing a single progress meter regardless of the number of layers in the commit-graph chain. After this patch, the output reflects the logical contents of a commit-graph chain, instead of showing one line of output per commit-graph layer:

    $ git.compile commit-graph verify
    Verifying commits in commit graph: 100% (136268/136268), done.
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 commit-graph.c                | 27 +++++++++++++++++----------
 t/t5324-split-commit-graph.sh |  3 ++-
 2 files changed, 19 insertions(+), 11 deletions(-)
Show changes to 2 files +19 −11

commit-graph.c, t/t5324-split-commit-graph.sh

diff --git a/commit-graph.c b/commit-graph.c
index 65dd4edf0e1..ffcf548223f 100644
--- a/commit-graph.c
+++ b/commit-graph.c
@@ -2544,7 +2544,8 @@ static int commit_graph_checksum_valid(struct commit_graph *g)
 
 static int verify_one_commit_graph(struct repository *r,
 				   struct commit_graph *g,
-				   struct progress *progress)
+				   struct progress *progress,
+				   uint64_t *seen)
 {
 	uint32_t i, cur_fanout_pos = 0;
 	struct object_id prev_oid, cur_oid;
@@ -2605,7 +2606,7 @@ static int verify_one_commit_graph(struct repository *r,
 		timestamp_t max_generation = 0;
 		timestamp_t generation;
 
-		display_progress(progress, i + 1);
+		display_progress(progress, ++(*seen));
 		oidread(&cur_oid, g->chunk_oid_lookup + g->hash_len * i);
 
 		graph_commit = lookup_commit(r, &cur_oid);
@@ -2694,26 +2695,32 @@ static int verify_one_commit_graph(struct repository *r,
 
 int verify_commit_graph(struct repository *r, struct commit_graph *g, int flags)
 {
+	struct progress *progress = NULL;
 	int local_error = 0;
+	uint64_t seen = 0;
 
 	if (!g) {
 		graph_report("no commit-graph file loaded");
 		return 1;
 	}
 
+	if (flags & COMMIT_GRAPH_WRITE_PROGRESS) {
+		uint64_t total = g->num_commits;
+		if (!(flags & COMMIT_GRAPH_VERIFY_SHALLOW))
+			total += g->num_commits_in_base;
+
+		progress = start_progress(_("Verifying commits in commit graph"),
+					  total);
+	}
+
 	for (; g; g = g->base_graph) {
-		struct progress *progress = NULL;
-		if (flags & COMMIT_GRAPH_WRITE_PROGRESS)
-			progress = start_progress(_("Verifying commits in commit graph"),
-						g->num_commits);
-
-		local_error |= verify_one_commit_graph(r, g, progress);
+		local_error |= verify_one_commit_graph(r, g, progress, &seen);
 		if (flags & COMMIT_GRAPH_VERIFY_SHALLOW)
 			break;
-
-		stop_progress(&progress);
 	}
 
+	stop_progress(&progress);
+
 	return local_error;
 }
 
diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh
index 669ddc645fa..36c4141e67b 100755
--- a/t/t5324-split-commit-graph.sh
+++ b/t/t5324-split-commit-graph.sh
@@ -351,7 +351,8 @@ test_expect_success 'add octopus merge' '
 	git branch merge/octopus &&
 	git commit-graph write --reachable --split &&
 	git commit-graph verify --progress 2>err &&
-	test_line_count = 3 err &&
+	test_line_count = 1 err &&
+	grep "Verifying commits in commit graph: 100% (18/18)" err &&
 	test_i18ngrep ! warning err &&
 	test_line_count = 3 $graphdir/commit-graph-chain
 '
-- 
2.41.0.242.g6eec849fa5a
Derrick Stolee· Jul 10, 2023, 15:55 UTC · re: Taylor Blau · lore

Re: [PATCH 0/6] fsck: squelch progress output with `--no-progress`

On 7/7/2023 6:31 PM, Taylor Blau wrote:
Show 21 quoted lines
>This short series addresses a pair of issues where the `commit-graph
>verify` and `multi-pack-index verify` steps of `git fsck` produce output
>regardless of whether or not `fsck`was invoked with the
>`--no-progress`option or not.
>
>The first two patches address the commit-graph and MIDX issues
>respectively. The final four patches further clean up the output of
>`git commit-graph verify --progress`when verifying multi-layer graphs
>to produce a single progress meter instead of one per graph layer.
>
>Before, the output of `git commit-graph verify`on a repository with a
>commit-graph chain with two layers looked like:
>
>    $ git.compile commit-graph verify
>    Verifying commits in commit graph: 100% (4356/4356), done.
>    Verifying commits in commit graph: 100% (131912/131912), done.
>
>After this patch series, the output instead looks as follows:
>
>    $ git.compile commit-graph verify
>    Verifying commits in commit graph: 100% (136268/136268), done.

Thanks for noticing and fixing these issues. The patches are structured well and make the refactoring easy to follow. This is especially tricky when swapping from recursive to iterative, but the patches break this down nicely. v1 LGTM. Thanks, -Stolee

← back to recent threads