git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 4/4] midx-write.c: use `--stdin-packs` when repacking

From
Junio C Hamano <gitster@pobox.com>
Date
Apr 1, 2024, 21:45 UTC
Message-ID
<xmqqbk6s92p4.fsf@gitster.g>
In-Reply-To
<b5d6ba5802aef6ddf1542f1b0efffe43c22436ba.1712006190.git.me@ttaylorr.com>
Taylor Blau <me@ttaylorr.com> writes:
Show 36 quoted lines
> When constructing a new pack `git multi-pack-index repack` provides a
> list of objects which is the union of objects in all MIDX'd packs which
> were "included" in the repack.
>
> Though correct, this typically yields a poorly structured pack, since
> providing the objects list over stdin does not give pack-objects a
> chance to discover the namehash values for each object, leading to
> sub-optimal delta selection.
>
> We can use `--stdin-packs` instead, which has a couple of benefits:
>
>   - it does a supplemental walk over objects in the supplied list of
>     packs to discover their namehash, leading to higher-quality delta
>     selection
>
>   - it requires us to list far less data over stdin; instead of listing
>     each object in the resulting pack, we need only list the
>     constituent packs from which those objects were selected in the MIDX
>
> Of course, this comes at a slight cost: though we save time on listing
> packs versus objects over stdin[^1] (around ~650 milliseconds), we add a
> non-trivial amount of time walking over the given objects in order to
> find better deltas.
>
> In general, this is likely to more closely match the user's expectations
> (i.e. that packs generated via `git multi-pack-index repack` are written
> with high-quality deltas). But if not, we can always introduce a new
> option in pack-objects to disable the supplemental object walk, which
> would yield a pure CPU-time savings, at the cost of the on-disk size of
> the resulting pack.
>
> [^1]: In a patched version of Git that doesn't perform the supplemental
>   object walk in `pack-objects --stdin-packs`, we save around ~650ms
>   (from 5.968 to 5.325 seconds) when running `git multi-pack-index
>   repack --batch-size=0` on git.git with all objects packed, and all
>   packs in a MIDX.

There are some measures in the mind of readers' who have read the explanation so far.

 - So, this gives us a resulting pack with better delta selection.
   How much better would it get in a sample repository?  10%?  40%?
 - Of course, the better delta selection comes with cost.  How much
   more time do we spend?  20%?  150%?
 - As we do not enumerate all the object names, we save some time.
   Around 0.65 seconds in a sample repository.

I think among the three, the first two are more interesting numbers, no?

I wonder if we can leverage the trick that reuses existing packdata when we stream packs to feed the "git fetch" clients---we rely on the fact that existing packs are tightly packed with good delta selection, and using bitmap stream contiguous section(s) as much as possible without disturbing the existing delta chain. Wouldn't the "we have many packs, let's repack them into one" workload benefit the same way?

Show 29 quoted lines
> -	strvec_push(&cmd.args, "pack-objects");
> +	strvec_pushl(&cmd.args, "pack-objects", "--stdin-packs", "--non-empty",
> +		     NULL);
>  
>  	strvec_pushf(&cmd.args, "%s/pack/pack", object_dir);
>  
> @@ -1498,16 +1499,15 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,
>  	}
>  
>  	cmd_in = xfdopen(cmd.in, "w");
> -
> -	for (i = 0; i < m->num_objects; i++) {
> -		struct object_id oid;
> -		uint32_t pack_int_id = nth_midxed_pack_int_id(m, i);
> -
> -		if (!include_pack[pack_int_id])
> +	for (i = 0; i < m->num_packs; i++) {
> +		struct packed_git *p = m->packs[i];
> +		if (!p)
>  			continue;
>  
> -		nth_midxed_object_oid(&oid, m, i);
> -		fprintf(cmd_in, "%s\n", oid_to_hex(&oid));
> +		if (include_pack[i])
> +			fprintf(cmd_in, "%s\n", pack_basename(p));
> +		else
> +			fprintf(cmd_in, "^%s\n", pack_basename(p));
>  	}
>  	fclose(cmd_in);
Looking very straight-forward.  Will queue.
Thanks.
Previous: Taylor BlauNext: Patrick Steinhardt
Message 29 of 30 in “midx: split MIDX writing routines into midx-write.c, cleanup”
  1. 00/11 midx: split MIDX writing routines into midx-write.c, cleanupTaylor Blau, Mar 25, 2024
  2. 01/11 midx-write: initial commitTaylor Blau, Mar 25, 2024
  3. Junio C HamanoMar 25, 2024
  4. Taylor BlauMar 25, 2024
  5. 02/11 midx: extern a pair of shared functionsTaylor Blau, Mar 25, 2024
  6. 03/11 midx: move `midx_repack` (and related functions) to midx-write.cTaylor Blau, Mar 25, 2024
  7. 04/11 midx: move `expire_midx_packs` to midx-write.cTaylor Blau, Mar 25, 2024
  8. 05/11 midx: move `write_midx_file_only` to midx-write.cTaylor Blau, Mar 25, 2024
  9. 06/11 midx: move `write_midx_file` to midx-write.cTaylor Blau, Mar 25, 2024
  10. 07/11 midx: move `write_midx_internal` (and related functions) to midx-write.cTaylor Blau, Mar 25, 2024
  11. 08/11 midx-write.c: avoid directly managed temporary strbufTaylor Blau, Mar 25, 2024
  12. Junio C HamanoMar 25, 2024
  13. Taylor BlauMar 25, 2024
  14. 09/11 midx-write.c: factor out common want_included_pack() routineTaylor Blau, Mar 25, 2024
  15. Junio C HamanoMar 25, 2024
  16. Jeff KingMar 27, 2024
  17. 10/11 midx-write.c: check count of packs to repack after groupingTaylor Blau, Mar 25, 2024
  18. Junio C HamanoMar 25, 2024
  19. Taylor BlauMar 25, 2024
  20. 11/11 midx-write.c: use `--stdin-packs` when repackingTaylor Blau, Mar 25, 2024
  21. Jeff KingMar 27, 2024
  22. Jeff KingMar 27, 2024
  23. 0/4 midx: split MIDX writing routines into midx-write.c, cleanupTaylor Blau, Apr 1, 2024
  24. 1/4 midx-write: move writing-related functions from midx.cTaylor Blau, Apr 1, 2024
  25. 2/4 midx-write.c: factor out common want_included_pack() routineTaylor Blau, Apr 1, 2024
  26. Patrick SteinhardtApr 2, 2024
  27. 3/4 midx-write.c: check count of packs to repack after groupingTaylor Blau, Apr 1, 2024
  28. 4/4 midx-write.c: use `--stdin-packs` when repackingTaylor Blau, Apr 1, 2024
  29. Junio C HamanoApr 1, 2024
  30. Patrick SteinhardtApr 2, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.