[PATCH 30/30] refs: skip hashing when writing packed-refs v2

From: Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>
Date: Nov 7, 2022, 18:36 UTC
Message-ID: <37fb4e73ca711f642351d10e1db51c330a1544f1.1667846165.git.gitgitgadget@gmail.com>
In-Reply-To: <pull.1408.git.1667846164.gitgitgadget@gmail.com>

From: Derrick Stolee <derrickstolee@github.com>

The 'skip_hash' option in 'struct hashfile' indicates that we want to use the hashfile API as a buffered writer, and not use the hash function to create a trailing hash. We still write a trailing null hash to indicate that we do not have a checksum at the end. This feature is enabled for index writes using the 'index.computeHash' config key.

Create a similar (currently hidden) option for the packed-refs v2 file format: refs.hashPackedRefs. This defaults to false because performance is compared to the packed-refs v1 file format which does have a checksum anywhere.

This change results in improvements to p1401 when using a repository with a 42 MB packed-refs file (600,000+ refs).

Test HEAD~1 HEAD -------------------------------------------------------------------- 1401.1: git pack-refs (v1) 0.38(0.31+0.52) 0.37(0.28+0.52) -2.6% 1401.5: git pack-refs (v2) 0.39(0.33+0.52) 0.30(0.28+0.46) -23.1%

Note that these tests update a ref and then repack the packed-refs file. The following benchmarks are from a hyperfine experiment that only ran the 'git pack-refs --all' command for the two formats, but also compared the effect when refs.hashPackedRefs=true.

Benchmark 1: v1
  Time (mean ± σ):     163.5 ms ±  18.1 ms    [User: 117.8 ms, System: 38.1 ms]
  Range (min … max):   131.3 ms … 190.4 ms    50 runs

Benchmark 2: v2-no-hash
  Time (mean ± σ):      95.8 ms ±  15.1 ms    [User: 72.5 ms, System: 23.0 ms]
  Range (min … max):    82.9 ms … 131.2 ms    50 runs

Benchmark 3: v2-hashing
  Time (mean ± σ):     100.8 ms ±  16.4 ms    [User: 77.2 ms, System: 23.1 ms]
  Range (min … max):    83.0 ms … 131.1 ms    50 runs

Summary
  'v2-no-hash' ran
    1.05 ± 0.24 times faster than 'v2-hashing'
    1.71 ± 0.33 times faster than 'v1'

In this case of repeatedly rewriting the same refs seems to demonstrate a smaller improvement than the p1401 test. However, the overall reduction from v1 matches the expected reduction in file size. In my tests, the 42 MB packed-refs (v1) file was compacted to 28 MB in the v2 format.

Signed-off-by: Derrick Stolee <derrickstolee@github.com>
---
 refs/packed-format-v2.c        | 7 +++++++
 t/perf/p1401-ref-operations.sh | 5 +++++
 2 files changed, 12 insertions(+)

diff --git a/refs/packed-format-v2.c b/refs/packed-format-v2.c
index 2cd45a5987a..ada34bf9bf0 100644
--- a/refs/packed-format-v2.c
+++ b/refs/packed-format-v2.c
@@ -417,6 +417,7 @@ struct write_packed_refs_v2_context *create_v2_context(struct packed_ref_store *
 						       struct strbuf *err)
 {
 	struct write_packed_refs_v2_context *ctx;
+	int do_skip_hash;
 	CALLOC_ARRAY(ctx, 1);
 
 	ctx->refs = refs;
@@ -430,6 +431,12 @@ struct write_packed_refs_v2_context *create_v2_context(struct packed_ref_store *
 	}
 
 	ctx->f = hashfd(refs->tempfile->fd, refs->tempfile->filename.buf);
+
+	/* Default to true, so skip_hash if not set. */
+	if (git_config_get_maybe_bool("refs.hashpackedrefs", &do_skip_hash) ||
+	    do_skip_hash)
+		ctx->f->skip_hash = 1;
+
 	ctx->cf = init_chunkfile(ctx->f);
 
 	return ctx;
diff --git a/t/perf/p1401-ref-operations.sh b/t/perf/p1401-ref-operations.sh
index 1c372ba0ee8..0b88a2f531a 100755
--- a/t/perf/p1401-ref-operations.sh
+++ b/t/perf/p1401-ref-operations.sh
@@ -36,6 +36,11 @@ test_perf 'git pack-refs (v2)' '
 	git pack-refs --all
 '
 
+test_perf 'git pack-refs (v2;hashing)' '
+	git commit --allow-empty -m "change one ref" &&
+	git -c refs.hashPackedRefs=true pack-refs --all
+'
+
 test_perf 'git for-each-ref (v2)' '
 	git for-each-ref --format="%(refname)" >/dev/null
 '

-- 
gitgitgadget