git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 00/11] pack-bitmap: convert offset to ref deltas where possible

From
Taylor Blau <me@ttaylorr.com>
Date
Nov 19, 2024, 23:34 UTC
Message-ID
<Zz0gdHrwC4CTAtZn@nand.local>
In-Reply-To
<Zz0aaPdHiFyoRkKg@nand.local>
On Tue, Nov 19, 2024 at 06:08:24PM -0500, Taylor Blau wrote:
Show 6 quoted lines
> Then when running the same command, we get results that are quite
> encouraging. The runtime jumps to 24.213 seconds, which is ~9.73 seconds
> slower than the average of the baseline measurements. But it takes
> ~10.418 seconds on my machine to compute the forward index. So it's
> really around 688ms *faster* than the baseline, despite doing a little
> more work.

Sorry, there is a much quicker way to generate the forward index at runtime, which is the following:

--- 8< ---
diff --git a/midx.c b/midx.c
index 67e0d64004..1023bfa51a 100644
--- a/midx.c
+++ b/midx.c
@@ -989,3 +989,17 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag

 	return verify_midx_error;
 }
+
+void midx_populate_forward_index(struct multi_pack_index *m)
+{
+	uint32_t i;
+
+	ALLOC_ARRAY(m->forward_idx, m->num_objects);
+
+	trace2_region_enter("midx", "populate_forward_index", the_repository);
+
+	for (i = 0; i < m->num_objects; i++)
+		m->forward_idx[pack_pos_to_midx(m, i)] = i;
+
+	trace2_region_leave("midx", "populate_forward_index", the_repository);
+}
--- >8 ---

I don't know if this makes as much of a difference for regular
multi-pack reuse, since there we're only making calls to
midx_pair_to_pack_pos(), which already uses midx_pack_order_cmp(), so
there isn't any additional translation to do.

Thanks,
Taylor
Previous: Taylor BlauNext: Jeff King
Message 30 of 31 in “pack-bitmap: convert offset to ref deltas where possible”
  1. 00/11 pack-bitmap: convert offset to ref deltas where possibleTaylor Blau, Oct 9, 2024
  2. 01/11 pack-bitmap.c: do not pass `pack_pos` to `try_partial_reuse()`Taylor Blau, Oct 9, 2024
  3. 02/11 pack-bitmap.c: avoid unnecessary `offset_to_pack_pos()`Taylor Blau, Oct 9, 2024
  4. 03/11 pack-bitmap.c: delay calling 'offset_to_pack_pos()'Taylor Blau, Oct 9, 2024
  5. 04/11 pack-bitmap.c: compare `base_offset` to `delta_obj_offset`Taylor Blau, Oct 9, 2024
  6. 05/11 pack-bitmap.c: extract `find_base_bitmap_pos()`Taylor Blau, Oct 9, 2024
  7. 06/11 pack-bitmap: drop `from_midx` field from `bitmapped_pack`Taylor Blau, Oct 9, 2024
  8. 07/11 write_reused_pack_one(): translate bit positions directlyTaylor Blau, Oct 9, 2024
  9. Jeff KingOct 11, 2024
  10. Taylor BlauNov 4, 2024
  11. 08/11 t5332: enable OFS_DELTAs via test_pack_objects_reusedTaylor Blau, Oct 9, 2024
  12. Jeff KingOct 11, 2024
  13. Taylor BlauNov 4, 2024
  14. 09/11 pack-bitmap: enable cross-pack delta reuseTaylor Blau, Oct 9, 2024
  15. Jeff KingOct 11, 2024
  16. Taylor BlauNov 4, 2024
  17. 10/11 pack-bitmap.c: record whether the result was filteredTaylor Blau, Oct 9, 2024
  18. Jeff KingOct 11, 2024
  19. Taylor BlauNov 4, 2024
  20. 11/11 pack-bitmap: enable reusing deltas with base objects in 'haves' bitmapTaylor Blau, Oct 9, 2024
  21. Junio C HamanoOct 10, 2024
  22. Taylor BlauOct 10, 2024
  23. Junio C HamanoOct 10, 2024
  24. Taylor BlauOct 10, 2024
  25. Jeff KingOct 11, 2024
  26. Jeff KingOct 11, 2024
  27. Junio C HamanoOct 11, 2024
  28. Jeff KingOct 11, 2024
  29. Taylor BlauNov 19, 2024
  30. Taylor BlauNov 19, 2024
  31. Jeff KingDec 18, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.