git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 5/8] packfile: refactor `get_multi_pack_index()` to work on sources

From
Patrick Steinhardt <ps@pks.im>
Date
Jul 15, 2025, 08:27 UTC
Message-ID
<aHYQ2m2_uv8L8ZS9@pks.im>
In-Reply-To
<aHBTLFvL+eQcw6J0@nand.local>
On Thu, Jul 10, 2025 at 07:56:28PM -0400, Taylor Blau wrote:
Show 22 quoted lines
> On Wed, Jul 09, 2025 at 09:54:53AM +0200, Patrick Steinhardt wrote:
> > diff --git a/builtin/repack.c b/builtin/repack.c
> > index 9bbf032b6dd..5956df5d927 100644
> > --- a/builtin/repack.c
> > +++ b/builtin/repack.c
> > @@ -218,9 +218,9 @@ static void mark_packs_for_deletion(struct existing_packs *existing,
> >  static void remove_redundant_pack(const char *dir_name, const char *base_name)
> >  {
> >  	struct strbuf buf = STRBUF_INIT;
> > -	struct multi_pack_index *m = get_local_multi_pack_index(the_repository);
> > +	struct multi_pack_index *m = get_multi_pack_index(the_repository->objects->sources);
> 
> Is the first source always guaranteed to be the local one? I assume that the
> answer here is "yes", and there are certainly other places in the code
> where we make a similar assumption. But just wanted to make sure as
> this popped into my head while reading.
> 
> >  	strbuf_addf(&buf, "%s.pack", base_name);
> > -	if (m && midx_contains_pack(m, buf.buf))
> > +	if (m && m->local && midx_contains_pack(m, buf.buf))
> 
> ...hmm, maybe not?

The answer is "almost always". The only exception is in case there is a temporary object source added via `odb_set_temporary_primary_source()`, for example for quarantine directories.

Generally speaking this patch here doesn't really change anything, and we could just as well drop the `m->local` part above. But I think that overall we're not doing a good job to track the local object source, so it felt safer to me to add the above guard.

Show 38 quoted lines
> > diff --git a/midx-write.c b/midx-write.c
> > index f2cfb85476e..c1ae62d3549 100644
> > --- a/midx-write.c
> > +++ b/midx-write.c
> > @@ -916,26 +916,8 @@ static int write_midx_bitmap(struct write_midx_context *ctx,
> >  static struct multi_pack_index *lookup_multi_pack_index(struct repository *r,
> >  							const char *object_dir)
> >  {
> > -	struct multi_pack_index *result = NULL;
> > -	struct multi_pack_index *cur;
> > -	char *obj_dir_real = real_pathdup(object_dir, 1);
> > -	struct strbuf cur_path_real = STRBUF_INIT;
> > -
> > -	/* Ensure the given object_dir is local, or a known alternate. */
> > -	odb_find_source(r->objects, obj_dir_real);
> > -
> > -	for (cur = get_multi_pack_index(r); cur; cur = cur->next) {
> > -		strbuf_realpath(&cur_path_real, cur->object_dir, 1);
> > -		if (!strcmp(obj_dir_real, cur_path_real.buf)) {
> > -			result = cur;
> > -			goto cleanup;
> > -		}
> > -	}
> > -
> > -cleanup:
> > -	free(obj_dir_real);
> > -	strbuf_release(&cur_path_real);
> > -	return result;
> > +	struct odb_source *source = odb_find_source(r->objects, object_dir);
> > +	return get_multi_pack_index(source);
> 
> When I first read this I wondered what would happen if we passed in an
> unknown object_dir such that odb_find_source() returned NULL. But that
> function will never return NULL, and instead will die() if given a bogus
> object_dir.
> 
> So this is fine, though I would have imagined that we'd return NULL
> within odb_find_source() and let the caller die() (or not).

Fully agreed. I've got a follow-up patch series that does this refactoring.

Show 35 quoted lines
> > diff --git a/object-name.c b/object-name.c
> > index ddafe7f9b13..1e7fdcb90a8 100644
> > --- a/object-name.c
> > +++ b/object-name.c
> > @@ -198,16 +198,19 @@ static void unique_in_pack(struct packed_git *p,
> >
> >  static void find_short_packed_object(struct disambiguate_state *ds)
> >  {
> > -	struct multi_pack_index *m;
> >  	struct packed_git *p;
> >
> >  	/* Skip, unless oids from the storage hash algorithm are wanted */
> >  	if (ds->bin_pfx.algo && (&hash_algos[ds->bin_pfx.algo] != ds->repo->hash_algo))
> >  		return;
> >
> > -	for (m = get_multi_pack_index(ds->repo); m && !ds->ambiguous;
> > -	     m = m->next)
> > -		unique_in_midx(m, ds);
> > +	odb_prepare_alternates(ds->repo->objects);
> > +	for (struct odb_source *source = ds->repo->objects->sources; source && !ds->ambiguous; source = source->next) {
> > +		struct multi_pack_index *m = get_multi_pack_index(source);
> > +		if (m)
> > +			unique_in_midx(m, ds);
> > +	}
> > +
> 
> Makes sense, though now having seen this pattern a few times, I am
> wondering if it would be worth it to add a utility function that takes a
> callback and iterates over the various MIDXs. But perhaps that is taking
> DRY-ing things up a little too far ;-).
> 
> For what it's worth, I do think that what you wrote here makes more
> logical sense: MIDXs are tied to individual alternate object DBs, which
> here means that there is one MIDX per "struct odb_source". It just is a
> little more verbose to type out.

I was wondering whether it would make sense to add a looping macro, but I ultimately decided against it as it doesn't make too much of a difference to really matter. I wouldn't mind adding it though if you or others feel strongly about it.

Patrick
Previous: Taylor BlauNext: Patrick Steinhardt
Message 24 of 43 in “odb: track multi-pack-indices via their object sources”
  1. 0/8 odb: track multi-pack-indices via their object sourcesPatrick Steinhardt, Jul 9, 2025
  2. 1/8 midx: start tracking per object database sourcePatrick Steinhardt, Jul 9, 2025
  3. Justin ToblerJul 10, 2025
  4. Taylor BlauJul 10, 2025
  5. Taylor BlauJul 10, 2025
  6. Patrick SteinhardtJul 15, 2025
  7. Patrick SteinhardtJul 15, 2025
  8. 2/8 packfile: refactor `prepare_packed_git_one()` to work on sourcesPatrick Steinhardt, Jul 9, 2025
  9. Justin ToblerJul 10, 2025
  10. Taylor BlauJul 10, 2025
  11. Patrick SteinhardtJul 15, 2025
  12. 3/8 midx: stop using linked list when closing MIDXPatrick Steinhardt, Jul 9, 2025
  13. Justin ToblerJul 10, 2025
  14. Patrick SteinhardtJul 15, 2025
  15. Taylor BlauJul 10, 2025
  16. Patrick SteinhardtJul 15, 2025
  17. 4/8 midx: track whether we have loaded the MIDXPatrick Steinhardt, Jul 9, 2025
  18. Justin ToblerJul 10, 2025
  19. Taylor BlauJul 10, 2025
  20. Patrick SteinhardtJul 15, 2025
  21. 5/8 packfile: refactor `get_multi_pack_index()` to work on sourcesPatrick Steinhardt, Jul 9, 2025
  22. Justin ToblerJul 10, 2025
  23. Taylor BlauJul 10, 2025
  24. Patrick SteinhardtJul 15, 2025
  25. 6/8 packfile: stop using linked MIDX list in `find_pack_entry()`Patrick Steinhardt, Jul 9, 2025
  26. 7/8 packfile: stop using linked MIDX list in `get_all_packs()`Patrick Steinhardt, Jul 9, 2025
  27. 8/8 midx: remove now-unused linked list of multi-pack indicesPatrick Steinhardt, Jul 9, 2025
  28. Justin ToblerJul 10, 2025
  29. Junio C HamanoJul 9, 2025
  30. Taylor BlauJul 10, 2025
  31. Patrick SteinhardtJul 15, 2025
  32. 0/7 odb: track multi-pack-indices via their object sourcesPatrick Steinhardt, Jul 15, 2025
  33. 1/7 midx: start tracking per object database sourcePatrick Steinhardt, Jul 15, 2025
  34. 2/7 packfile: refactor `prepare_packed_git_one()` to work on sourcesPatrick Steinhardt, Jul 15, 2025
  35. 3/7 midx: stop using linked list when closing MIDXPatrick Steinhardt, Jul 15, 2025
  36. 4/7 packfile: refactor `get_multi_pack_index()` to work on sourcesPatrick Steinhardt, Jul 15, 2025
  37. 5/7 packfile: stop using linked MIDX list in `find_pack_entry()`Patrick Steinhardt, Jul 15, 2025
  38. 6/7 packfile: stop using linked MIDX list in `get_all_packs()`Patrick Steinhardt, Jul 15, 2025
  39. 7/7 midx: remove now-unused linked list of multi-pack indicesPatrick Steinhardt, Jul 15, 2025
  40. Justin ToblerJul 15, 2025
  41. Junio C HamanoJul 23, 2025
  42. Patrick SteinhardtJul 24, 2025
  43. Taylor BlauAug 12, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.