git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 6/9] midx: load multi-pack indices via their source

From
Taylor Blau <me@ttaylorr.com>
Date
Aug 7, 2025, 22:25 UTC
Message-ID
<aJUn0qeliNQ/nnWr@nand.local>
In-Reply-To
<20250807-b4-pks-midx-deduplicate-source-info-v2-6-bcffb8fc119c@pks.im>
On Thu, Aug 07, 2025 at 10:09:56AM +0200, Patrick Steinhardt wrote:
Show 56 quoted lines
> To load a multi-pack index the caller is expected to pass both the
> repository and the object directory where the multi-pack index is
> located. While this works, this layout has a couple of downsides:
>
>   - We need to pass in information reduntant with the owning source,
>     namely its object directory and whether the source is local or not.
>
>   - We don't have access to the source when loading the multi-pack
>     index. If we had that access, we could store a pointer to the owning
>     source in the MIDX and thus deduplicate some information.
>
>   - Multi-pack indices are inherently specific to the object source and
>     its format. With the goal of pluggable object backends in mind we
>     will eventually want the backends to own the logic of reading and
>     writing multi-pack indices. Making the logic work on top of object
>     sources is a step into that direction.
>
> Refactor loading of multi-pack indices accordingly.
>
> This surfaces one small problem though: git-multi-pack-index(1) and our
> MIDX test helper both know to read and write multi-pack-indices located
> in a different object directory. This issue is addressed by adding the
> user-provided object directory as an in-memory alternate.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  builtin/multi-pack-index.c  | 18 ++++++++++++--
>  midx.c                      | 57 ++++++++++++++++++++-------------------------
>  midx.h                      |  6 ++---
>  t/helper/test-read-midx.c   | 25 ++++++++++++--------
>  t/t5319-multi-pack-index.sh |  8 +++----
>  5 files changed, 62 insertions(+), 52 deletions(-)
>
> diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c
> index aa25b06f9d..e4a9305af3 100644
> --- a/builtin/multi-pack-index.c
> +++ b/builtin/multi-pack-index.c
> @@ -64,12 +64,20 @@ static int parse_object_dir(const struct option *opt, const char *arg,
>  	char **value = opt->value;
>  	free(*value);
>  	if (unset)
> -		*value = xstrdup(repo_get_object_directory(the_repository));
> +		*value = xstrdup(the_repository->objects->sources->path);
>  	else
>  		*value = real_pathdup(arg, 1);
>  	return 0;
>  }
>
> +static struct odb_source *handle_object_dir_option(struct repository *repo)
> +{
> +	struct odb_source *source = odb_find_source(repo->objects, opts.object_dir);
> +	if (!source)
> +		source = odb_add_to_alternates_memory(repo->objects, opts.object_dir);
> +	return source;
> +}
> +

Hmm... I admit that I am pretty unfamiliar with the --object-dir option here and only have a vague recollection of why it was added in the first place.

I am not sure one way or another if adding the user-provided directory as an in-memory alternate is the right thing to do here.

Stolee (CC'd) would you mind sharing your thoughts on this?

Thanks, Taylor

Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 34 of 56 in “midx: stop deduplicating info redundant with their sources”
  1. 0/8 midx: stop deduplicating info redundant with their sourcesPatrick Steinhardt, Jul 29, 2025
  2. 1/8 odb: store locality in object database sourcesPatrick Steinhardt, Jul 29, 2025
  3. Toon ClaesAug 6, 2025
  4. Patrick SteinhardtAug 7, 2025
  5. Karthik NayakAug 7, 2025
  6. 2/8 odb: allow `odb_find_source()` to failPatrick Steinhardt, Jul 29, 2025
  7. 3/8 odb: return newly created in-memory sourcesPatrick Steinhardt, Jul 29, 2025
  8. Toon ClaesAug 6, 2025
  9. Patrick SteinhardtAug 7, 2025
  10. Karthik NayakAug 7, 2025
  11. 4/8 midx: drop redundant `struct repository` parameterPatrick Steinhardt, Jul 29, 2025
  12. 5/8 midx: load multi-pack indices via their sourcePatrick Steinhardt, Jul 29, 2025
  13. Karthik NayakAug 7, 2025
  14. Karthik NayakAug 7, 2025
  15. 6/8 midx: write multi-pack indices via their sourcePatrick Steinhardt, Jul 29, 2025
  16. Karthik NayakAug 7, 2025
  17. 7/8 midx: stop duplicating info redundant with its owning sourcePatrick Steinhardt, Jul 29, 2025
  18. 8/8 midx: compute paths via their sourcePatrick Steinhardt, Jul 29, 2025
  19. Junio C HamanoJul 29, 2025
  20. Patrick SteinhardtJul 30, 2025
  21. 0/9 midx: stop duplicating info redundant with their sourcesPatrick Steinhardt, Aug 7, 2025
  22. 1/9 odb: store locality in object database sourcesPatrick Steinhardt, Aug 7, 2025
  23. Taylor BlauAug 7, 2025
  24. 2/9 odb: allow `odb_find_source()` to failPatrick Steinhardt, Aug 7, 2025
  25. Taylor BlauAug 7, 2025
  26. Patrick SteinhardtAug 11, 2025
  27. 3/9 odb: return newly created in-memory sourcesPatrick Steinhardt, Aug 7, 2025
  28. Taylor BlauAug 7, 2025
  29. Patrick SteinhardtAug 11, 2025
  30. 4/9 odb: simplify calling `link_alt_odb_entry()`Patrick Steinhardt, Aug 7, 2025
  31. Taylor BlauAug 7, 2025
  32. 5/9 midx: drop redundant `struct repository` parameterPatrick Steinhardt, Aug 7, 2025
  33. 6/9 midx: load multi-pack indices via their sourcePatrick Steinhardt, Aug 7, 2025
  34. Taylor BlauAug 7, 2025
  35. 7/9 midx: write multi-pack indices via their sourcePatrick Steinhardt, Aug 7, 2025
  36. Taylor BlauAug 7, 2025
  37. 8/9 midx: stop duplicating info redundant with its owning sourcePatrick Steinhardt, Aug 7, 2025
  38. 9/9 midx: compute paths via their sourcePatrick Steinhardt, Aug 7, 2025
  39. Taylor BlauAug 7, 2025
  40. Patrick SteinhardtAug 11, 2025
  41. Karthik NayakAug 7, 2025
  42. 00/10 midx: stop duplicating info redundant with their sourcesPatrick Steinhardt, Aug 11, 2025
  43. 01/10 odb: store locality in object database sourcesPatrick Steinhardt, Aug 11, 2025
  44. 02/10 odb: allow `odb_find_source()` to failPatrick Steinhardt, Aug 11, 2025
  45. 03/10 odb: consistently use "dir" to refer to alternate's directoryPatrick Steinhardt, Aug 11, 2025
  46. 04/10 odb: return newly created in-memory sourcesPatrick Steinhardt, Aug 11, 2025
  47. 05/10 odb: simplify calling `link_alt_odb_entry()`Patrick Steinhardt, Aug 11, 2025
  48. 06/10 midx: drop redundant `struct repository` parameterPatrick Steinhardt, Aug 11, 2025
  49. 07/10 midx: load multi-pack indices via their sourcePatrick Steinhardt, Aug 11, 2025
  50. 08/10 midx: write multi-pack indices via their sourcePatrick Steinhardt, Aug 11, 2025
  51. 09/10 midx: stop duplicating info redundant with its owning sourcePatrick Steinhardt, Aug 11, 2025
  52. 10/10 midx: compute paths via their sourcePatrick Steinhardt, Aug 11, 2025
  53. Junio C HamanoAug 28, 2025
  54. Taylor BlauAug 29, 2025
  55. Derrick StoleeAug 30, 2025
  56. Patrick SteinhardtSep 2, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.