git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 2/3] upload-pack: skip parse-object re-hashing of "want" objects

From
Jeff King <peff@peff.net>
Date
Sep 7, 2022, 21:02 UTC
Message-ID
<YxkG3A30vNfu55Sx@coredump.intra.peff.net>
In-Reply-To
<YxkAxutS+B8//0WF@coredump.intra.peff.net>
On Wed, Sep 07, 2022 at 04:36:22PM -0400, Jeff King wrote:
> Is that worth having? I dunno. It's kind of brittle in that a later
> change could mean we're finding the corruption elsewhere, and not
> checking exactly what we think we are. OTOH, it probably doesn't hurt to
> cover more cases here, and it's not a very expensive test.
So here's what it would look like as a patch on top.

It could also be squashed in, of course, which saves the vague reference to "a recent commit...". OTOH, it's sufficiently complicated to discuss the testing that it's nice not to overwhelm the already-long commit message of the original. ;)

-- >8 --
Subject: [PATCH] t1060: check partial clone of misnamed blob

A recent commit (upload-pack: skip parse-object re-hashing of "want" objects, 2022-09-06) loosened the behavior of upload-pack so that it does not verify the sha1 of objects it receives directly via "want" requests. The existing corruption tests in t1060 aren't affected by this: the corruptions are blobs reachable from commits, and the client requests the commits.

The more interesting case here is a partial clone, where the client will directly ask for the corrupted blob when it does an on-demand fetch of the filtered object. And that is not covered at all, so let's add a test.

It's important here that we use the "misnamed" corruption and not "bit-error". The latter is sufficiently corrupted that upload-pack cannot even figure out the type of the object, so it bails identically both before and after the recent change. But with "misnamed", with the hash-checks enabled it sees the problem (though the error messages are a bit confusing because of the inability to create a "struct object" to store the flags):

  error: hash mismatch d95f3ad14dee633a758d2e331151e950dd13e4ed
  fatal: git upload-pack: not our ref d95f3ad14dee633a758d2e331151e950dd13e4ed
  fatal: remote error: upload-pack: not our ref d95f3ad14dee633a758d2e331151e950dd13e4ed

After the change to skip the hash check, the server side happily sends the bogus object, but the client correctly realizes that it did not get the necessary data:

  remote: Enumerating objects: 1, done.
  remote: Counting objects: 100% (1/1), done.
  remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
  Receiving objects: 100% (1/1), 49 bytes | 49.00 KiB/s, done.
  fatal: bad revision 'd95f3ad14dee633a758d2e331151e950dd13e4ed'
  error: [...]/misnamed did not send all necessary objects
which is exactly what we expect to happen.
Signed-off-by: Jeff King <peff@peff.net>
---
 t/t1060-object-corruption.sh | 7 +++++++
 1 file changed, 7 insertions(+)
diff --git a/t/t1060-object-corruption.sh b/t/t1060-object-corruption.sh
index 5b8e47e346..35261afc9d 100755
--- a/t/t1060-object-corruption.sh
+++ b/t/t1060-object-corruption.sh
@@ -139,4 +139,11 @@ test_expect_success 'internal tree objects are not "missing"' '
 	)
 '
 
+test_expect_success 'partial clone of corrupted repository' '
+	test_config -C misnamed uploadpack.allowFilter true &&
+	git clone --no-local --no-checkout --filter=blob:none \
+		misnamed corrupt-partial && \
+	test_must_fail git -C corrupt-partial checkout --force
+'
+
 test_done
-- 
2.37.3.1138.gedcf976b26
Previous: rsbecker@nexbridge.comNext: Junio C Hamano
Message 31 of 40 in “Partial-clone cause big performance impact on server”
  1. 程洋Aug 11, 2022
  2. Jonathan TanAug 11, 2022
  3. 回复: [External Mail]Re: Partial-clone cause big performance impact on server程洋, Aug 13, 2022
  4. 回复: [External Mail]Re: Partial-clone cause big performance impact on server程洋, Aug 13, 2022
  5. ZheNing HuAug 15, 2022
  6. 程洋Aug 15, 2022
  7. Derrick StoleeAug 12, 2022
  8. Jeff KingAug 14, 2022
  9. Derrick StoleeAug 15, 2022
  10. 程洋Aug 15, 2022
  11. 程洋Aug 17, 2022
  12. Derrick StoleeAug 17, 2022
  13. Jeff KingAug 18, 2022
  14. 程洋Sep 1, 2022
  15. Jeff KingSep 1, 2022
  16. 程洋Sep 5, 2022
  17. Jeff KingSep 6, 2022
  18. 0/3 speeding up on-demand fetch for blobs in partial cloneJeff King, Sep 6, 2022
  19. 1/3 parse_object(): allow skipping hash checkJeff King, Sep 6, 2022
  20. Derrick StoleeSep 7, 2022
  21. Jeff KingSep 7, 2022
  22. 2/3 upload-pack: skip parse-object re-hashing of "want" objectsJeff King, Sep 6, 2022
  23. Derrick StoleeSep 7, 2022
  24. Derrick StoleeSep 7, 2022
  25. Jeff KingSep 7, 2022
  26. Junio C HamanoSep 7, 2022
  27. Jeff KingSep 7, 2022
  28. [BUG] t1800: Fails for error text comparisonrsbecker@nexbridge.com, Sep 7, 2022
  29. Junio C HamanoSep 7, 2022
  30. rsbecker@nexbridge.comSep 7, 2022
  31. Jeff KingSep 7, 2022
  32. Junio C HamanoSep 7, 2022
  33. Jeff KingSep 8, 2022
  34. Junio C HamanoSep 8, 2022
  35. 3/3 parse_object(): check commit-graph when skip_hash setJeff King, Sep 6, 2022
  36. Derrick StoleeSep 7, 2022
  37. Junio C HamanoSep 7, 2022
  38. 程洋Sep 8, 2022
  39. Jeff KingSep 8, 2022
  40. Derrick StoleeSep 7, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.