git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 12/24] pack-bitmap: return multiple packs via `reuse_partial_packfile_from_bitmap()`

From
Taylor Blau <me@ttaylorr.com>
Date
Nov 28, 2023, 19:08 UTC
Message-ID
<36b794d9e1eaa413b3b87eb7e5a0d82098d05dca.1701198172.git.me@ttaylorr.com>
In-Reply-To
<cover.1701198172.git.me@ttaylorr.com>

Further prepare for enabling verbatim pack-reuse over multiple packfiles by changing the signature of reuse_partial_packfile_from_bitmap() to populate an array of `struct bitmapped_pack *`'s instead of a pointer to a single packfile.

Since the array we're filling out is sized dynamically[^1], add an additional `size_t *` parameter which will hold the number of reusable packs (equal to the number of elements in the array).

Note that since we still have not implemented true multi-pack reuse, these changes aren't propagated out to the rest of the caller in builtin/pack-objects.c.

In the interim state, we expect that the array has a single element, and we use that element to fill out the static `reuse_packfile` variable (which is a bog-standard `struct packed_git *`). Future commits will continue to push this change further out through the pack-objects code.

[^1]: That is, even though we know the number of packs which are
  candidates for pack-reuse, we do not know how many of those
  candidates we can actually reuse.
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 builtin/pack-objects.c | 9 +++++++--
 pack-bitmap.c          | 6 ++++--
 pack-bitmap.h          | 5 +++--
 3 files changed, 14 insertions(+), 6 deletions(-)
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 2bb1b64e8f..89de23f39a 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -3943,14 +3943,19 @@ static int pack_options_allow_reuse(void)
 
 static int get_object_list_from_bitmap(struct rev_info *revs)
 {
+	struct bitmapped_pack *packs = NULL;
+	size_t packs_nr = 0;
+
 	if (!(bitmap_git = prepare_bitmap_walk(revs, 0)))
 		return -1;
 
 	if (pack_options_allow_reuse())
-		reuse_partial_packfile_from_bitmap(bitmap_git, &reuse_packfile,
+		reuse_partial_packfile_from_bitmap(bitmap_git, &packs,
+						   &packs_nr,
 						   &reuse_packfile_bitmap);
 
-	if (reuse_packfile) {
+	if (packs) {
+		reuse_packfile = packs[0].p;
 		reuse_packfile_objects = bitmap_popcount(reuse_packfile_bitmap);
 		if (!reuse_packfile_objects)
 			BUG("expected non-empty reuse bitmap");
diff --git a/pack-bitmap.c b/pack-bitmap.c
index 614fc09a4e..670deec909 100644
--- a/pack-bitmap.c
+++ b/pack-bitmap.c
@@ -1989,7 +1989,8 @@ static void reuse_partial_packfile_from_bitmap_1(struct bitmap_index *bitmap_git
 }
 
 void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,
-					struct packed_git **packfile_out,
+					struct bitmapped_pack **packs_out,
+					size_t *packs_nr_out,
 					struct bitmap **reuse_out)
 {
 	struct repository *r = the_repository;
@@ -2056,7 +2057,8 @@ void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,
 	 * need to be handled separately.
 	 */
 	bitmap_and_not(result, reuse);
-	*packfile_out = packs[0].p;
+	*packs_out = packs;
+	*packs_nr_out = packs_nr;
 	*reuse_out = reuse;
 }
 
diff --git a/pack-bitmap.h b/pack-bitmap.h
index 5bc1ca5b65..901a3b86ed 100644
--- a/pack-bitmap.h
+++ b/pack-bitmap.h
@@ -78,8 +78,9 @@ int test_bitmap_hashes(struct repository *r);
 struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,
 					 int filter_provided_objects);
 uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git);
-void reuse_partial_packfile_from_bitmap(struct bitmap_index *,
-					struct packed_git **packfile,
+void reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git,
+					struct bitmapped_pack **packs_out,
+					size_t *packs_nr_out,
 					struct bitmap **reuse_out);
 int rebuild_existing_bitmaps(struct bitmap_index *, struct packing_data *mapping,
 			     kh_oid_map_t *reused_bitmaps, int show_progress);
-- 
2.43.0.24.g980b318f98
Previous: Taylor BlauNext: Taylor Blau
Message 49 of 107 in “pack-objects: multi-pack verbatim reuse”
  1. 00/24 pack-objects: multi-pack verbatim reuseTaylor Blau, Nov 28, 2023
  2. 01/24 pack-objects: free packing_data in more placesTaylor Blau, Nov 28, 2023
  3. Patrick SteinhardtNov 30, 2023
  4. Taylor BlauNov 30, 2023
  5. 02/24 pack-bitmap-write: deep-clear the `bb_commit` slabTaylor Blau, Nov 28, 2023
  6. Patrick SteinhardtNov 30, 2023
  7. Taylor BlauNov 30, 2023
  8. Jeff KingDec 12, 2023
  9. Taylor BlauDec 12, 2023
  10. 03/24 pack-bitmap: plug leak in find_objects()Taylor Blau, Nov 28, 2023
  11. Jeff KingDec 12, 2023
  12. 04/24 midx: factor out `fill_pack_info()`Taylor Blau, Nov 28, 2023
  13. Patrick SteinhardtNov 30, 2023
  14. Taylor BlauNov 30, 2023
  15. 05/24 midx: implement `DISP` chunkTaylor Blau, Nov 28, 2023
  16. Patrick SteinhardtNov 30, 2023
  17. Taylor BlauNov 30, 2023
  18. Junio C HamanoDec 3, 2023
  19. Taylor BlauDec 5, 2023
  20. Junio C HamanoDec 9, 2023
  21. Taylor BlauDec 9, 2023
  22. Jeff KingDec 12, 2023
  23. Taylor BlauDec 13, 2023
  24. Junio C HamanoDec 13, 2023
  25. 06/24 midx: implement `midx_locate_pack()`Taylor Blau, Nov 28, 2023
  26. 07/24 midx: implement `--retain-disjoint` modeTaylor Blau, Nov 28, 2023
  27. Patrick SteinhardtNov 30, 2023
  28. Taylor BlauNov 30, 2023
  29. Patrick SteinhardtDec 1, 2023
  30. 08/24 pack-objects: implement `--ignore-disjoint` modeTaylor Blau, Nov 28, 2023
  31. Patrick SteinhardtNov 30, 2023
  32. Taylor BlauNov 30, 2023
  33. Patrick SteinhardtDec 1, 2023
  34. Taylor BlauDec 1, 2023
  35. 09/24 repack: implement `--extend-disjoint` modeTaylor Blau, Nov 28, 2023
  36. Patrick SteinhardtDec 7, 2023
  37. Taylor BlauDec 7, 2023
  38. Patrick SteinhardtDec 8, 2023
  39. Taylor BlauDec 8, 2023
  40. Patrick SteinhardtDec 11, 2023
  41. Taylor BlauDec 11, 2023
  42. 10/24 pack-bitmap: pass `bitmapped_pack` struct to pack-reuse functionsTaylor Blau, Nov 28, 2023
  43. Patrick SteinhardtDec 7, 2023
  44. Taylor BlauDec 7, 2023
  45. Patrick SteinhardtDec 8, 2023
  46. 11/24 pack-bitmap: simplify `reuse_partial_packfile_from_bitmap()` signatureTaylor Blau, Nov 28, 2023
  47. Patrick SteinhardtDec 7, 2023
  48. Taylor BlauDec 7, 2023
  49. 12/24 pack-bitmap: return multiple packs via `reuse_partial_packfile_from_bitmap()`Taylor Blau, Nov 28, 2023
  50. 13/24 pack-objects: parameterize pack-reuse routines over a single packTaylor Blau, Nov 28, 2023
  51. 14/24 pack-objects: keep track of `pack_start` for each reuse packTaylor Blau, Nov 28, 2023
  52. Patrick SteinhardtDec 7, 2023
  53. Taylor BlauDec 7, 2023
  54. 15/24 pack-objects: pass `bitmapped_pack`'s to pack-reuse functionsTaylor Blau, Nov 28, 2023
  55. 16/24 pack-objects: prepare `write_reused_pack()` for multi-pack reuseTaylor Blau, Nov 28, 2023
  56. Patrick SteinhardtDec 7, 2023
  57. Taylor BlauDec 7, 2023
  58. 17/24 pack-objects: prepare `write_reused_pack_verbatim()` for multi-pack reuseTaylor Blau, Nov 28, 2023
  59. 18/24 pack-objects: include number of packs reused in outputTaylor Blau, Nov 28, 2023
  60. 19/24 pack-bitmap: prepare to mark objects from multiple packs for reuseTaylor Blau, Nov 28, 2023
  61. 20/24 pack-objects: add tracing for various packfile metricsTaylor Blau, Nov 28, 2023
  62. 21/24 t/test-lib-functions.sh: implement `test_trace2_data` helperTaylor Blau, Nov 28, 2023
  63. 22/24 pack-objects: allow setting `pack.allowPackReuse` to "single"Taylor Blau, Nov 28, 2023
  64. 23/24 pack-bitmap: reuse objects from all disjoint packsTaylor Blau, Nov 28, 2023
  65. 24/24 t/perf: add performance tests for multi-pack reuseTaylor Blau, Nov 28, 2023
  66. Patrick SteinhardtNov 30, 2023
  67. Taylor BlauNov 30, 2023
  68. Patrick SteinhardtDec 1, 2023
  69. Taylor BlauDec 1, 2023
  70. Patrick SteinhardtDec 4, 2023
  71. Jeff KingDec 12, 2023
  72. Taylor BlauDec 15, 2023
  73. Jeff KingDec 21, 2023
  74. Taylor BlauJan 4, 2024
  75. 00/26 pack-objects: multi-pack verbatim reuseTaylor Blau, Dec 14, 2023
  76. 01/26 pack-objects: free packing_data in more placesTaylor Blau, Dec 14, 2023
  77. 02/26 pack-bitmap-write: deep-clear the `bb_commit` slabTaylor Blau, Dec 14, 2023
  78. 03/26 pack-bitmap: plug leak in find_objects()Taylor Blau, Dec 14, 2023
  79. 04/26 midx: factor out `fill_pack_info()`Taylor Blau, Dec 14, 2023
  80. 05/26 midx: implement `BTMP` chunkTaylor Blau, Dec 14, 2023
  81. 06/26 midx: implement `midx_locate_pack()`Taylor Blau, Dec 14, 2023
  82. 07/26 pack-bitmap: pass `bitmapped_pack` struct to pack-reuse functionsTaylor Blau, Dec 14, 2023
  83. 08/26 ewah: implement `bitmap_is_empty()`Taylor Blau, Dec 14, 2023
  84. 09/26 pack-bitmap: simplify `reuse_partial_packfile_from_bitmap()` signatureTaylor Blau, Dec 14, 2023
  85. 10/26 pack-bitmap: return multiple packs via `reuse_partial_packfile_from_bitmap()`Taylor Blau, Dec 14, 2023
  86. 11/26 pack-objects: parameterize pack-reuse routines over a single packTaylor Blau, Dec 14, 2023
  87. 12/26 pack-objects: keep track of `pack_start` for each reuse packTaylor Blau, Dec 14, 2023
  88. 13/26 pack-objects: pass `bitmapped_pack`'s to pack-reuse functionsTaylor Blau, Dec 14, 2023
  89. 14/26 pack-objects: prepare `write_reused_pack()` for multi-pack reuseTaylor Blau, Dec 14, 2023
  90. 15/26 pack-objects: prepare `write_reused_pack_verbatim()` for multi-pack reuseTaylor Blau, Dec 14, 2023
  91. 16/26 pack-objects: include number of packs reused in outputTaylor Blau, Dec 14, 2023
  92. 17/26 git-compat-util.h: implement checked size_t to uint32_t conversionTaylor Blau, Dec 14, 2023
  93. 18/26 midx: implement `midx_preferred_pack()`Taylor Blau, Dec 14, 2023
  94. 19/26 pack-revindex: factor out `midx_key_to_pack_pos()` helperTaylor Blau, Dec 14, 2023
  95. 20/26 pack-revindex: implement `midx_pair_to_pack_pos()`Taylor Blau, Dec 14, 2023
  96. 21/26 pack-bitmap: prepare to mark objects from multiple packs for reuseTaylor Blau, Dec 14, 2023
  97. 22/26 pack-objects: add tracing for various packfile metricsTaylor Blau, Dec 14, 2023
  98. 23/26 t/test-lib-functions.sh: implement `test_trace2_data` helperTaylor Blau, Dec 14, 2023
  99. 24/26 pack-objects: allow setting `pack.allowPackReuse` to "single"Taylor Blau, Dec 14, 2023
  100. 25/26 pack-bitmap: enable reuse from all bitmapped packsTaylor Blau, Dec 14, 2023
  101. 26/26 t/perf: add performance tests for multi-pack reuseTaylor Blau, Dec 14, 2023
  102. Junio C HamanoDec 15, 2023
  103. Taylor BlauDec 15, 2023
  104. Junio C HamanoDec 15, 2023
  105. Taylor BlauDec 15, 2023
  106. Jeff KingDec 21, 2023
  107. Taylor BlauJan 4, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.