git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 09/11] pack-bitmap: enable cross-pack delta reuse

From
Jeff King <peff@peff.net>
Date
Oct 11, 2024, 08:31 UTC
Message-ID
<20241011083134.GG18010@coredump.intra.peff.net>
In-Reply-To
<ca3a916cd6e93aaa09ccff9f77fc127254222068.1728505840.git.me@ttaylorr.com>
On Wed, Oct 09, 2024 at 04:31:28PM -0400, Taylor Blau wrote:
Show 29 quoted lines
> In order to do this, the pack-reuse code within pack-bitmap.c marks
> bits in a separate bitmap called 'reuse_as_ref_delta'. Objects whose
> bits are marked in that bitmap must be converted from OFS_DELTAs to
> REF_DELTAs.
> 
> To mark bits in that bitmap, we adjust find_base_bitmap_pos() to
> return the bitmap position of any delta object's base regardless of
> whether or not it came from the same pack. This is done by:
> 
>   1. First converting the base object's into a pack position (via
>      `offset_to_pack_pos()`).
> 
>   2. Then converting from pack position into into lexical order (via
>      `pack_pos_to_index()`).
> 
>   3. Then into an object ID (via `nth_packed_object_id()`).
> 
>   4. Then into a position in the MIDX's lexical order of object IDs
>      (via `bsearch_midx()`).
> 
>   5. And finally into a position in the MIDX's pseudo-pack order (via
>      `midx_pair_to_pack_pos()`).
> 
> If we can find that base object in the MIDX, then we use its position
> in the MIDX's pseudo-pack order to determine whether or not it was
> selected from the same pack. If it is, no adjustment is necessary.
> Otherwise, we mark the delta object's position in the new
> `reuse_as_ref_delta` bitmap, and convert accordingly from within
> `write_reused_pack_one()`.

OK, that makes sense. It does feel like a non-trivial amount of work to do for each delta we're going to (potentially) reuse from a midx'd pack. Can we recognize the common case that the base is in the same pack and also being sent/reused without doing the full conversion to oid and the resulting bsearch?

Show 25 quoted lines
> @@ -1182,10 +1188,24 @@ static size_t write_reused_pack_verbatim(struct bitmapped_pack *reuse_packfile,
>  	if (pos >= end)
>  		return reuse_packfile->bitmap_pos / BITS_IN_EWORD;
>  
> -	while (pos < end &&
> -	       reuse_packfile_bitmap->words[pos / BITS_IN_EWORD] == (eword_t)~0)
> +	while (pos < end) {
> +		size_t wpos = pos / BITS_IN_EWORD;
> +		eword_t reuse;
> +
> +		reuse = reuse_packfile_bitmap->words[wpos];
> +		if (reuse_as_ref_delta_packfile_bitmap) {
> +			/*
> +			 * Can't reuse verbatim any objects which need
> +			 * to be first rewritten as REF_DELTAs.
> +			 */
> +			reuse &= ~reuse_as_ref_delta_packfile_bitmap->words[wpos];
> +		}
> +
> +		if (reuse != (eword_t)~0)
> +			break;
> +
>  		pos += BITS_IN_EWORD;
> -
> +	}

This is accessing reuse_as_ref_delta_packfile_bitmap->words directly using pos/end as limits. But those come from reuse_packfile_bitmap. Are we guaranteed to have zero-extended the reuse_as_ref_delta bitmap as far as the original went?

Could we just be calling bitmap_get() here, which would do the length check for us? Though I guess we would miss out on some whole-word magic it is doing. So maybe we need to just do that length check ourselves.

-Peff
Previous: Taylor BlauNext: Taylor Blau
Message 15 of 31 in “pack-bitmap: convert offset to ref deltas where possible”
  1. 00/11 pack-bitmap: convert offset to ref deltas where possibleTaylor Blau, Oct 9, 2024
  2. 01/11 pack-bitmap.c: do not pass `pack_pos` to `try_partial_reuse()`Taylor Blau, Oct 9, 2024
  3. 02/11 pack-bitmap.c: avoid unnecessary `offset_to_pack_pos()`Taylor Blau, Oct 9, 2024
  4. 03/11 pack-bitmap.c: delay calling 'offset_to_pack_pos()'Taylor Blau, Oct 9, 2024
  5. 04/11 pack-bitmap.c: compare `base_offset` to `delta_obj_offset`Taylor Blau, Oct 9, 2024
  6. 05/11 pack-bitmap.c: extract `find_base_bitmap_pos()`Taylor Blau, Oct 9, 2024
  7. 06/11 pack-bitmap: drop `from_midx` field from `bitmapped_pack`Taylor Blau, Oct 9, 2024
  8. 07/11 write_reused_pack_one(): translate bit positions directlyTaylor Blau, Oct 9, 2024
  9. Jeff KingOct 11, 2024
  10. Taylor BlauNov 4, 2024
  11. 08/11 t5332: enable OFS_DELTAs via test_pack_objects_reusedTaylor Blau, Oct 9, 2024
  12. Jeff KingOct 11, 2024
  13. Taylor BlauNov 4, 2024
  14. 09/11 pack-bitmap: enable cross-pack delta reuseTaylor Blau, Oct 9, 2024
  15. Jeff KingOct 11, 2024
  16. Taylor BlauNov 4, 2024
  17. 10/11 pack-bitmap.c: record whether the result was filteredTaylor Blau, Oct 9, 2024
  18. Jeff KingOct 11, 2024
  19. Taylor BlauNov 4, 2024
  20. 11/11 pack-bitmap: enable reusing deltas with base objects in 'haves' bitmapTaylor Blau, Oct 9, 2024
  21. Junio C HamanoOct 10, 2024
  22. Taylor BlauOct 10, 2024
  23. Junio C HamanoOct 10, 2024
  24. Taylor BlauOct 10, 2024
  25. Jeff KingOct 11, 2024
  26. Jeff KingOct 11, 2024
  27. Junio C HamanoOct 11, 2024
  28. Jeff KingOct 11, 2024
  29. Taylor BlauNov 19, 2024
  30. Taylor BlauNov 19, 2024
  31. Jeff KingDec 18, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.