git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 3/7] midx.c: respect 'pack.writeBitmapHashcache' when writing bitmaps

From
Taylor Blau <me@ttaylorr.com>
Date
Sep 17, 2021, 21:21 UTC
Message-ID
<f0d8f106c22dae64eaf4b77d1ecf67ef79c4515c.1631913631.git.me@ttaylorr.com>
In-Reply-To
<cover.1631913631.git.me@ttaylorr.com>

In the previous commit, the bitmap writing code learned to propagate values from an existing hash-cache extension into the bitmap that it is writing.

Now that that functionality exists, let's expose it by teaching the 'git multi-pack-index' builtin to respect the `pack.writeBitmapHashCache` option so that the hash-cache may be written at all.

Two minor points worth noting here:
  - The 'git multi-pack-index write' sub-command didn't previously read
    any configuration (instead this is handled in the base command). A
    separate handler is added here to respect this write-specific
    config option.
  - I briefly considered adding a 'bitmap_flags' field to the static
    options struct, but decided against it since it would require
    plumbing through a new parameter to the write_midx_file() function.
    Instead, a new MIDX-specific flag is added, which is translated to
    the corresponding bitmap one.
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 Documentation/config/pack.txt |  4 ++++
 builtin/multi-pack-index.c    | 21 +++++++++++++++++++++
 midx.c                        |  6 +++++-
 midx.h                        |  1 +
 4 files changed, 31 insertions(+), 1 deletion(-)
diff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt
index 763f7af7c4..ad7f73a1ea 100644
--- a/Documentation/config/pack.txt
+++ b/Documentation/config/pack.txt
@@ -159,6 +159,10 @@ pack.writeBitmapHashCache::
 	between an older, bitmapped pack and objects that have been
 	pushed since the last gc). The downside is that it consumes 4
 	bytes per object of disk space. Defaults to true.
++
+When writing a multi-pack reachability bitmap, no new namehashes are
+computed; instead, any namehashes stored in an existing bitmap are
+permuted into their appropriate location when writing a new bitmap.
 
 pack.writeReverseIndex::
 	When true, git will write a corresponding .rev file (see:
diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c
index 73c0113b48..578ffea6e8 100644
--- a/builtin/multi-pack-index.c
+++ b/builtin/multi-pack-index.c
@@ -61,6 +61,23 @@ static struct option *add_common_options(struct option *prev)
 	return parse_options_concat(common_opts, prev);
 }
 
+static int git_multi_pack_index_write_config(const char *var, const char *value,
+					     void *cb)
+{
+	if (!strcmp(var, "pack.writebitmaphashcache")) {
+		if (git_config_bool(var, value))
+			opts.flags |= MIDX_WRITE_BITMAP_HASH_CACHE;
+		else
+			opts.flags &= ~MIDX_WRITE_BITMAP_HASH_CACHE;
+	}
+
+	/*
+	 * We should never make a fall-back call to 'git_default_config', since
+	 * this was already called in 'cmd_multi_pack_index()'.
+	 */
+	return 0;
+}
+
 static int cmd_multi_pack_index_write(int argc, const char **argv)
 {
 	struct option *options;
@@ -73,6 +90,10 @@ static int cmd_multi_pack_index_write(int argc, const char **argv)
 		OPT_END(),
 	};
 
+	opts.flags |= MIDX_WRITE_BITMAP_HASH_CACHE;
+
+	git_config(git_multi_pack_index_write_config, NULL);
+
 	options = add_common_options(builtin_multi_pack_index_write_options);
 
 	trace2_cmd_mode(argv[0]);
diff --git a/midx.c b/midx.c
index 864034a6ad..38c8600458 100644
--- a/midx.c
+++ b/midx.c
@@ -1008,9 +1008,13 @@ static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,
 	struct pack_idx_entry **index;
 	struct commit **commits = NULL;
 	uint32_t i, commits_nr;
+	uint16_t options = 0;
 	char *bitmap_name = xstrfmt("%s-%s.bitmap", midx_name, hash_to_hex(midx_hash));
 	int ret;
 
+	if (flags & MIDX_WRITE_BITMAP_HASH_CACHE)
+		options |= BITMAP_OPT_HASH_CACHE;
+
 	prepare_midx_packing_data(&pdata, ctx);
 
 	commits = find_commits_for_midx_bitmap(&commits_nr, ctx);
@@ -1049,7 +1053,7 @@ static int write_midx_bitmap(char *midx_name, unsigned char *midx_hash,
 		goto cleanup;
 
 	bitmap_writer_set_checksum(midx_hash);
-	bitmap_writer_finish(index, pdata.nr_objects, bitmap_name, 0);
+	bitmap_writer_finish(index, pdata.nr_objects, bitmap_name, options);
 
 cleanup:
 	free(index);
diff --git a/midx.h b/midx.h
index aa3da557bb..541d9ac728 100644
--- a/midx.h
+++ b/midx.h
@@ -44,6 +44,7 @@ struct multi_pack_index {
 #define MIDX_PROGRESS     (1 << 0)
 #define MIDX_WRITE_REV_INDEX (1 << 1)
 #define MIDX_WRITE_BITMAP (1 << 2)
+#define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3)
 
 const unsigned char *get_midx_checksum(struct multi_pack_index *m);
 char *get_midx_filename(const char *object_dir);
-- 
2.33.0.96.g73915697e6
Previous: Taylor BlauNext: Taylor Blau
Message 50 of 55 in “pack-bitmap: permute existing namehash values”
  1. 0/4 pack-bitmap: permute existing namehash valuesTaylor Blau, Sep 7, 2021
  2. 1/4 t/helper/test-bitmap.c: add 'dump-hashes' modeTaylor Blau, Sep 7, 2021
  3. Ævar Arnfjörð BjarmasonSep 8, 2021
  4. Taylor BlauSep 8, 2021
  5. 2/4 pack-bitmap.c: propagate namehash values from existing bitmapsTaylor Blau, Sep 7, 2021
  6. 3/4 midx.c: respect 'pack.writeBitmapHashcache' when writing bitmapsTaylor Blau, Sep 7, 2021
  7. Ævar Arnfjörð BjarmasonSep 8, 2021
  8. Taylor BlauSep 8, 2021
  9. Ævar Arnfjörð BjarmasonSep 9, 2021
  10. Ævar Arnfjörð BjarmasonSep 9, 2021
  11. Taylor BlauSep 9, 2021
  12. Ævar Arnfjörð BjarmasonSep 9, 2021
  13. Taylor BlauSep 9, 2021
  14. Taylor BlauSep 9, 2021
  15. Junio C HamanoSep 13, 2021
  16. Taylor BlauSep 14, 2021
  17. 4/4 t5326: test propagating hashcache valuesTaylor Blau, Sep 7, 2021
  18. Ævar Arnfjörð BjarmasonSep 8, 2021
  19. Taylor BlauSep 8, 2021
  20. Ævar Arnfjörð BjarmasonSep 17, 2021
  21. Taylor BlauSep 17, 2021
  22. Ævar Arnfjörð BjarmasonSep 17, 2021
  23. Junio C HamanoSep 13, 2021
  24. Taylor BlauSep 14, 2021
  25. Junio C HamanoSep 14, 2021
  26. Taylor BlauSep 14, 2021
  27. Taylor BlauSep 14, 2021
  28. Jeff KingSep 14, 2021
  29. Taylor BlauSep 14, 2021
  30. Jeff KingSep 14, 2021
  31. Junio C HamanoSep 14, 2021
  32. 0/7 pack-bitmap: permute existing namehash valuesTaylor Blau, Sep 14, 2021
  33. 1/7 t/helper/test-bitmap.c: add 'dump-hashes' modeTaylor Blau, Sep 14, 2021
  34. 2/7 pack-bitmap.c: propagate namehash values from existing bitmapsTaylor Blau, Sep 14, 2021
  35. 3/7 midx.c: respect 'pack.writeBitmapHashcache' when writing bitmapsTaylor Blau, Sep 14, 2021
  36. 4/7 p5326: create missing 'perf-tag' tagTaylor Blau, Sep 14, 2021
  37. Jeff KingSep 16, 2021
  38. Taylor BlauSep 17, 2021
  39. 5/7 p5326: don't set core.multiPackIndex unnecessarilyTaylor Blau, Sep 14, 2021
  40. Jeff KingSep 16, 2021
  41. 6/7 p5326: generate pack bitmaps before writing the MIDX bitmapTaylor Blau, Sep 14, 2021
  42. Jeff KingSep 16, 2021
  43. Taylor BlauSep 17, 2021
  44. 7/7 t5326: test propagating hashcache valuesTaylor Blau, Sep 14, 2021
  45. Jeff KingSep 16, 2021
  46. Jeff KingSep 16, 2021
  47. 0/7 pack-bitmap: permute existing namehash valuesTaylor Blau, Sep 17, 2021
  48. 1/7 t/helper/test-bitmap.c: add 'dump-hashes' modeTaylor Blau, Sep 17, 2021
  49. 2/7 pack-bitmap.c: propagate namehash values from existing bitmapsTaylor Blau, Sep 17, 2021
  50. 3/7 midx.c: respect 'pack.writeBitmapHashcache' when writing bitmapsTaylor Blau, Sep 17, 2021
  51. 4/7 p5326: create missing 'perf-tag' tagTaylor Blau, Sep 17, 2021
  52. 5/7 p5326: don't set core.multiPackIndex unnecessarilyTaylor Blau, Sep 17, 2021
  53. 6/7 p5326: generate pack bitmaps before writing the MIDX bitmapTaylor Blau, Sep 17, 2021
  54. 7/7 t5326: test propagating hashcache valuesTaylor Blau, Sep 17, 2021
  55. Jeff KingSep 17, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.