git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] packfile: avoid access(3p) calls for missing packs

From
Junio C Hamano <gitster@pobox.com>
Date
May 19, 2025, 15:46 UTC
Message-ID
<xmqqsel0ob72.fsf@gitster.g>
In-Reply-To
<20250519065221.GC102701@coredump.intra.peff.net>
Jeff King <peff@peff.net> writes:
Show 5 quoted lines
> The code in prepare_midx_pack() converts a numeric pack id (referenced
> inside the midx) into a "struct packed_git" pointer, caching the results
> in multi_pack_index->packs. That field holds NULL for "we have not
> looked it up yet" or a valid pointer to a packed_git. It probably needs
> to hold a third state: "we tried and failed".

Yeah, this was exactly why I asked "are we having repeated failures in the same process", and I am happy with this direction.

Show 36 quoted lines
> Something like this (large untested) patch:
>
> diff --git a/midx.c b/midx.c
> index 3d0015f782..354b1f886c 100644
> --- a/midx.c
> +++ b/midx.c
> @@ -405,7 +405,7 @@ void close_midx(struct multi_pack_index *m)
>  	munmap((unsigned char *)m->data, m->data_len);
>  
>  	for (i = 0; i < m->num_packs; i++) {
> -		if (m->packs[i])
> +		if (m->packs[i] && m->packs[i] != (void *)(intptr_t)-1)
>  			m->packs[i]->multi_pack_index = 0;
>  	}
>  	FREE_AND_NULL(m->packs);
> @@ -458,6 +458,8 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,
>  
>  	pack_int_id = midx_for_pack(&m, pack_int_id);
>  
> +	if (m->packs[pack_int_id] == (void *)(intptr_t)-1)
> +		return 1;
>  	if (m->packs[pack_int_id])
>  		return 0;
>  
> @@ -482,8 +484,10 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,
>  	strbuf_release(&pack_name);
>  	strbuf_release(&key);
>  
> -	if (!p)
> +	if (!p) {
> +		m->packs[pack_int_id] = (void *)(intptr_t)-1;
>  		return 1;
> +	}
>  
>  	p->multi_pack_index = 1;
>  	m->packs[pack_int_id] = p;
Previous: Jeff KingNext: Patrick Steinhardt
Message 4 of 53 in “packfile: avoid access(3p) calls for missing packs”
  1. packfile: avoid access(3p) calls for missing packsPatrick Steinhardt, May 16, 2025
  2. Junio C HamanoMay 16, 2025
  3. Jeff KingMay 19, 2025
  4. Junio C HamanoMay 19, 2025
  5. Patrick SteinhardtMay 20, 2025
  6. Jeff KingMay 22, 2025
  7. Taylor BlauMay 23, 2025
  8. Jeff KingMay 23, 2025
  9. 0/2 packfile: avoid access(3p) calls for missing packsPatrick Steinhardt, May 20, 2025
  10. 1/2 packfile: explain ordering of how we look up auxiliary pack filesPatrick Steinhardt, May 20, 2025
  11. Taylor BlauMay 23, 2025
  12. 2/2 midx: stop repeatedly looking up nonexistent packfilesPatrick Steinhardt, May 20, 2025
  13. Jeff KingMay 22, 2025
  14. Junio C HamanoMay 22, 2025
  15. Jeff KingMay 22, 2025
  16. Junio C HamanoMay 22, 2025
  17. Taylor BlauMay 23, 2025
  18. Jeff KingMay 23, 2025
  19. Taylor BlauMay 23, 2025
  20. 0/5 midx: improve prepare_midx_pack() ergonomicsTaylor Blau, May 25, 2025
  21. 1/5 pack-bitmap.c: fix broken warning() when missing MIDX'd packTaylor Blau, May 25, 2025
  22. Patrick SteinhardtMay 26, 2025
  23. Taylor BlauMay 28, 2025
  24. 2/5 midx-write.c: guard against incremental MIDXs in want_included_pack()Taylor Blau, May 25, 2025
  25. Patrick SteinhardtMay 26, 2025
  26. Taylor BlauMay 28, 2025
  27. 3/5 midx-write.c: simplify fill_packs_from_midx()Taylor Blau, May 25, 2025
  28. Patrick SteinhardtMay 26, 2025
  29. Taylor BlauMay 28, 2025
  30. 4/5 midx-write.c: extract inner loop from fill_packs_from_midx()Taylor Blau, May 25, 2025
  31. 5/5 midx: return a `packed_git` pointer from `prepare_midx_pack()`Taylor Blau, May 25, 2025
  32. Patrick SteinhardtMay 26, 2025
  33. Taylor BlauMay 28, 2025
  34. Patrick SteinhardtMay 28, 2025
  35. 0/4 midx: improve prepare_midx_pack() ergonomicsTaylor Blau, May 28, 2025
  36. 1/4 midx: access pack names through `nth_midxed_pack_name()`Taylor Blau, May 28, 2025
  37. Junio C HamanoMay 29, 2025
  38. Taylor BlauJun 3, 2025
  39. Junio C HamanoMay 29, 2025
  40. Taylor BlauJun 3, 2025
  41. 2/4 midx-write.c: guard against incremental MIDXs in want_included_pack()Taylor Blau, May 28, 2025
  42. 3/4 midx-write.c: extract inner loop from fill_packs_from_midx()Taylor Blau, May 28, 2025
  43. 4/4 midx: return a `packed_git` pointer from `prepare_midx_pack()`Taylor Blau, May 28, 2025
  44. Jeff KingMay 30, 2025
  45. Taylor BlauJun 3, 2025
  46. Junio C HamanoAug 28, 2025
  47. Taylor BlauMay 23, 2025
  48. Jeff KingMay 23, 2025
  49. Junio C HamanoMay 21, 2025
  50. 0/2 packfile: avoid access(3p) calls for missing packsPatrick Steinhardt, May 28, 2025
  51. 2/2 midx: stop repeatedly looking up nonexistent packfilesPatrick Steinhardt, May 28, 2025
  52. 1/2 packfile: explain ordering of how we look up auxiliary pack filesPatrick Steinhardt, May 28, 2025
  53. Jeff KingMay 30, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.