git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Bug report: v2.47.0 cannot fetch version 1 pack indexes

From
Taylor Blau <me@ttaylorr.com>
Date
Oct 21, 2024, 20:33 UTC
Message-ID
<Zxa6ixwP2aOJdfmL@nand.local>
In-Reply-To
<20241020024022.GA615104@coredump.intra.peff.net>
On Sat, Oct 19, 2024 at 10:40:22PM -0400, Jeff King wrote:
Show 36 quoted lines
> diff --git a/http.c b/http.c
> index d59e59f66b..6df032e40f 100644
> --- a/http.c
> +++ b/http.c
> @@ -19,6 +19,7 @@
>  #include "string-list.h"
>  #include "object-file.h"
>  #include "object-store-ll.h"
> +#include "tempfile.h"
>
>  static struct trace_key trace_curl = TRACE_KEY_INIT(CURL);
>  static int trace_curl_data = 1;
> @@ -2388,8 +2389,24 @@ static char *fetch_pack_index(unsigned char *hash, const char *base_url)
>  	strbuf_addf(&buf, "objects/pack/pack-%s.idx", hash_to_hex(hash));
>  	url = strbuf_detach(&buf, NULL);
>
> -	strbuf_addf(&buf, "%s.temp", sha1_pack_index_name(hash));
> -	tmp = strbuf_detach(&buf, NULL);
> +	/*
> +	 * Don't put this into packs/, since it's just temporary and we don't
> +	 * want to confuse it with our local .idx files.  We'll generate our
> +	 * own index if we choose to download the matching packfile.
> +	 *
> +	 * It's tempting to use xmks_tempfile() here, but it's important that
> +	 * the file not exist, otherwise http_get_file() complains. So we
> +	 * create a filename that should be unique, and then just register it
> +	 * as a tempfile so that it will get cleaned up on exit.
> +	 *
> +	 * Arguably it would be better to hold on to the tempfile handle so
> +	 * that we can remove it as soon as we download the pack and generate
> +	 * the real index, but that might need more surgery.
> +	 */
> +	tmp = xstrfmt("%s/tmp_pack_%s.idx",
> +		      repo_get_object_directory(the_repository),
> +		      hash_to_hex(hash));
> +	register_tempfile(tmp);
Makes perfect sense, and the comment above here is much appreciated.

I thought about trying to use some intermediate state of the strbuf here to avoid an extra xstrfmt() call, but couldn't come up with anything I didn't think was awkward.

Show 10 quoted lines
>  	if (http_get_file(url, tmp, NULL) != HTTP_OK) {
>  		error("Unable to get pack index %s", url);
> @@ -2427,10 +2444,8 @@ static int fetch_and_setup_pack_index(struct packed_git **packs_head,
>  	}
>
>  	ret = verify_pack_index(new_pack);
> -	if (!ret) {
> +	if (!ret)
>  		close_pack_index(new_pack);
> -		ret = finalize_object_file(tmp_idx, sha1_pack_index_name(sha1));
And here's where we stop calling finalize_object_file(). Good.
Show 24 quoted lines
> -	}
>  	free(tmp_idx);
>  	if (ret)
>  		return -1;
> diff --git a/packfile.c b/packfile.c
> index df4ba67719..16d3bcf7f7 100644
> --- a/packfile.c
> +++ b/packfile.c
> @@ -237,13 +237,22 @@ static struct packed_git *alloc_packed_git(int extra)
>  	return p;
>  }
>
> +static char *pack_path_from_idx(const char *idx_path)
> +{
> +	size_t len;
> +	if (!strip_suffix(idx_path, ".idx", &len))
> +		BUG("idx path does not end in .idx: %s", idx_path);
> +	return xstrfmt("%.*s.pack", (int)len, idx_path);
> +}
> +
>  struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path)
>  {
> -	const char *path = sha1_pack_name(sha1);
> +	char *path = pack_path_from_idx(idx_path);

Huh. I would have thought we have such a helper function already. I guess we probably do, but that it's also defined statically because it's so easy to write.

In any case, this looks like the right thing to do here. It would be nice to have a corresponding test here, since unlike the other finalize_object_file() changes, this one can be provoked deterministically.

Would you mind submitting this as a bona-fide patch, which I can then pick up and start merging down?

In any event, I appreciate you figuring out what was going on here and outlining a couple of paths forward.

Thanks, Taylor

Previous: Jeff KingNext: Jeff King
Message 6 of 31 in “Bug report: v2.47.0 cannot fetch version 1 pack indexes”
  1. foxOct 19, 2024
  2. Eric SunshineOct 20, 2024
  3. Taylor BlauOct 21, 2024
  4. Jeff KingOct 20, 2024
  5. Jeff KingOct 20, 2024
  6. Taylor BlauOct 21, 2024
  7. Jeff KingOct 22, 2024
  8. Taylor BlauOct 22, 2024
  9. 0/11 dumb-http pack index v1 regression + cleanupsJeff King, Oct 25, 2024
  10. 01/11 midx: avoid duplicate packed_git entriesJeff King, Oct 25, 2024
  11. Taylor BlauOct 25, 2024
  12. 02/11 t5550: count fetches in "previously-fetched .idx" testJeff King, Oct 25, 2024
  13. 03/11 dumb-http: store downloaded pack idx as tempfileJeff King, Oct 25, 2024
  14. Taylor BlauOct 25, 2024
  15. Jeff KingOct 26, 2024
  16. Taylor BlauOct 28, 2024
  17. 04/11 packfile: drop has_pack_index()Jeff King, Oct 25, 2024
  18. Taylor BlauOct 25, 2024
  19. 05/11 packfile: drop sha1_pack_name()Jeff King, Oct 25, 2024
  20. 06/11 packfile: drop sha1_pack_index_name()Jeff King, Oct 25, 2024
  21. 07/11 packfile: warn people away from parse_packed_git()Jeff King, Oct 25, 2024
  22. Taylor BlauOct 25, 2024
  23. 08/11 http-walker: use object_id instead of bare hashJeff King, Oct 25, 2024
  24. 09/11 packfile: convert find_sha1_pack() to use object_idJeff King, Oct 25, 2024
  25. 10/11 packfile: use object_id in find_pack_entry_one()Jeff King, Oct 25, 2024
  26. Taylor BlauOct 25, 2024
  27. 11/11 packfile: use oidread() instead of hashcpy() to fill object_idJeff King, Oct 25, 2024
  28. Taylor BlauOct 25, 2024
  29. Taylor BlauOct 21, 2024
  30. Jeff KingOct 22, 2024
  31. Taylor BlauOct 22, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.