git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Bug report: v2.47.0 cannot fetch version 1 pack indexes

From
Jeff King <peff@peff.net>
Date
Oct 22, 2024, 05:14 UTC
Message-ID
<20241022051402.GB1247135@coredump.intra.peff.net>
In-Reply-To
<Zxa6ixwP2aOJdfmL@nand.local>
On Mon, Oct 21, 2024 at 04:33:15PM -0400, Taylor Blau wrote:
Show 30 quoted lines
> > @@ -2388,8 +2389,24 @@ static char *fetch_pack_index(unsigned char *hash, const char *base_url)
> >  	strbuf_addf(&buf, "objects/pack/pack-%s.idx", hash_to_hex(hash));
> >  	url = strbuf_detach(&buf, NULL);
> >
> > -	strbuf_addf(&buf, "%s.temp", sha1_pack_index_name(hash));
> > -	tmp = strbuf_detach(&buf, NULL);
> > +	/*
> > +	 * Don't put this into packs/, since it's just temporary and we don't
> > +	 * want to confuse it with our local .idx files.  We'll generate our
> > +	 * own index if we choose to download the matching packfile.
> > +	 *
> > +	 * It's tempting to use xmks_tempfile() here, but it's important that
> > +	 * the file not exist, otherwise http_get_file() complains. So we
> > +	 * create a filename that should be unique, and then just register it
> > +	 * as a tempfile so that it will get cleaned up on exit.
> > +	 *
> > +	 * Arguably it would be better to hold on to the tempfile handle so
> > +	 * that we can remove it as soon as we download the pack and generate
> > +	 * the real index, but that might need more surgery.
> > +	 */
> > +	tmp = xstrfmt("%s/tmp_pack_%s.idx",
> > +		      repo_get_object_directory(the_repository),
> > +		      hash_to_hex(hash));
> > +	register_tempfile(tmp);
> 
> Makes perfect sense, and the comment above here is much appreciated.
> 
> I thought about trying to use some intermediate state of the strbuf here
> to avoid an extra xstrfmt() call, but couldn't come up with anything I
> didn't think was awkward.

I don't think there's any useful intermediate state. The earlier %s is the base url, but here it's our local directory.

We could continue to re-use the scratch strbuf as the existing code did (and which xstrfmt() is doing under the hood). It wasn't really intentional for me to change that, but I went through a lot of attempts to get here (using mks_tempfile(), and so on).

Show 16 quoted lines
> > +static char *pack_path_from_idx(const char *idx_path)
> > +{
> > +	size_t len;
> > +	if (!strip_suffix(idx_path, ".idx", &len))
> > +		BUG("idx path does not end in .idx: %s", idx_path);
> > +	return xstrfmt("%.*s.pack", (int)len, idx_path);
> > +}
> > +
> >  struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path)
> >  {
> > -	const char *path = sha1_pack_name(sha1);
> > +	char *path = pack_path_from_idx(idx_path);
> 
> Huh. I would have thought we have such a helper function already. I
> guess we probably do, but that it's also defined statically because it's
> so easy to write.

I thought so, too, but couldn't find one. We have pack_bitmap_filename() (and so on for .rev and .midx files) that goes from .pack to those extensions. But here we want to go from .idx to .pack. I think most stuff goes from ".pack" because that's what we store in the packed_git struct.

There's also sha1_pack_index_name(), but that goes from a csum-file hash to a filename.

I grepped around and strip_suffix() seems to be par for the course in similar situations within pack/repack code, so I think it's OK here.

Show 7 quoted lines
> In any case, this looks like the right thing to do here. It would be
> nice to have a corresponding test here, since unlike the other
> finalize_object_file() changes, this one can be provoked
> deterministically.
> 
> Would you mind submitting this as a bona-fide patch, which I can then
> pick up and start merging down?
Yeah, the test is easy:
diff --git a/t/t5550-http-fetch-dumb.sh b/t/t5550-http-fetch-dumb.sh
index 58189c9f7d..50a7b98813 100755
--- a/t/t5550-http-fetch-dumb.sh
+++ b/t/t5550-http-fetch-dumb.sh
@@ -507,4 +507,14 @@ test_expect_success 'fetching via http alternates works' '
 	git -c http.followredirects=true clone "$HTTPD_URL/dumb/alt-child.git"
 '
 
+test_expect_success 'dumb http can fetch index v1' '
+	server=$HTTPD_DOCUMENT_ROOT_PATH/idx-v1.git &&
+	git init --bare "$server" &&
+	git -C "$server" --work-tree=. commit --allow-empty -m foo &&
+	git -C "$server" -c pack.indexVersion=1 gc &&
+
+	git clone "$HTTPD_URL/dumb/idx-v1.git" &&
+	git -C idx-v1 fsck
+'
+
 test_done

I raised some other more philosophical issues in the other part of the
thread, but assuming the answer is "no, let's do the simplest thing",
then I think this approach is OK.

I'd also like to see if I can clean things up around parse_pack_index(),
whose semantics I'm changing here (and which violates all manner of
assumptions that we usually have about packed_git structs). It's used
only by the dumb-http code, and I think we want to refactor it a bit so
that nobody else is tempted to use it.

I'll try to send something out tonight or tomorrow.

-Peff
Previous: Taylor BlauNext: Taylor Blau
Message 7 of 31 in “Bug report: v2.47.0 cannot fetch version 1 pack indexes”
  1. foxOct 19, 2024
  2. Eric SunshineOct 20, 2024
  3. Taylor BlauOct 21, 2024
  4. Jeff KingOct 20, 2024
  5. Jeff KingOct 20, 2024
  6. Taylor BlauOct 21, 2024
  7. Jeff KingOct 22, 2024
  8. Taylor BlauOct 22, 2024
  9. 0/11 dumb-http pack index v1 regression + cleanupsJeff King, Oct 25, 2024
  10. 01/11 midx: avoid duplicate packed_git entriesJeff King, Oct 25, 2024
  11. Taylor BlauOct 25, 2024
  12. 02/11 t5550: count fetches in "previously-fetched .idx" testJeff King, Oct 25, 2024
  13. 03/11 dumb-http: store downloaded pack idx as tempfileJeff King, Oct 25, 2024
  14. Taylor BlauOct 25, 2024
  15. Jeff KingOct 26, 2024
  16. Taylor BlauOct 28, 2024
  17. 04/11 packfile: drop has_pack_index()Jeff King, Oct 25, 2024
  18. Taylor BlauOct 25, 2024
  19. 05/11 packfile: drop sha1_pack_name()Jeff King, Oct 25, 2024
  20. 06/11 packfile: drop sha1_pack_index_name()Jeff King, Oct 25, 2024
  21. 07/11 packfile: warn people away from parse_packed_git()Jeff King, Oct 25, 2024
  22. Taylor BlauOct 25, 2024
  23. 08/11 http-walker: use object_id instead of bare hashJeff King, Oct 25, 2024
  24. 09/11 packfile: convert find_sha1_pack() to use object_idJeff King, Oct 25, 2024
  25. 10/11 packfile: use object_id in find_pack_entry_one()Jeff King, Oct 25, 2024
  26. Taylor BlauOct 25, 2024
  27. 11/11 packfile: use oidread() instead of hashcpy() to fill object_idJeff King, Oct 25, 2024
  28. Taylor BlauOct 25, 2024
  29. Taylor BlauOct 21, 2024
  30. Jeff KingOct 22, 2024
  31. Taylor BlauOct 22, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.