git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 0/5] handling 4GB .idx files

From
Thomas Braun <thomas.braun@virtuell-zuhause.de>
Date
Nov 30, 2020, 22:57 UTC
Message-ID
<1403797985.37893.1606777048311@ox.hosteurope.de>
In-Reply-To
<20201116041051.GA883199@coredump.intra.peff.net>
> Jeff King <peff@peff.net> hat am 16.11.2020 05:10 geschrieben:
[...]
> So I dunno. I wouldn't be opposed to codifying some of that in 
> a script, but I can't imagine anybody ever running it unless they 
> were working on this specific problem.
Thanks for the pointers.
Below is what I came up with. It passes here. I've replaced awk with cut from the original draft, and also moved the perl script out of the test as I think the quoting is getting way too messy otherwise. And I've added --no-dangling to git fsck as otherwise it takes forever to output the obvious dangling blobs. The unpack limit is mostly for testing the test itself with a smaller amount of blobs. But I still think it is worthwile to force everything into a pack.
--- a/t/t1600-index.sh
+++ b/t/t1600-index.sh
@@ -97,4 +97,34 @@ test_expect_success 'index version config precedence' '
 	test_index_version 0 true 2 2
 '
 
+{
+	echo "#!$SHELL_PATH"
+	cat <<'EOF'
+	   "$PERL_PATH" -e '
+		for (0..154_000_000) {
+			print "blob\n";
+			print "data <<EOF\n";
+			print "$_\n";
+			print "EOF\n";
+		} '
+EOF
+
+} >dump
+chmod +x dump
+
+test_expect_success EXPENSIVE,PERL 'Test 4GB boundary for the index' '
+	test_config fastimport.unpacklimit 0 &&
+	./dump | git fast-import &&
+	blob=$(echo 0 | git hash-object --stdin) &&
+	git cat-file blob $blob >actual &&
+	echo 0 >expect &&
+	test_cmp expect actual &&
+	idx_pack=$(ls .git/objects/pack/*.idx) &&
+	test_file_not_empty $idx_pack &&
+	final=$(git show-index <$idx_pack | tail -1 | cut -d " " -f2) &&
+	git cat-file blob $final &&
+	git cat-file blob fffffff &&
+	git fsck --strict --no-dangling
+'
+
 test_done
--
Previous: Jeff KingNext: Jeff King
Message 12 of 19 in “handling 4GB .idx files”
  1. 0/5 handling 4GB .idx filesJeff King, Nov 13, 2020
  2. 1/5 compute pack .idx byte offsets using size_tJeff King, Nov 13, 2020
  3. 2/5 use size_t to store pack .idx byte offsetsJeff King, Nov 13, 2020
  4. 3/5 fsck: correctly compute checksums on idx files larger than 4GBJeff King, Nov 13, 2020
  5. 4/5 block-sha1: take a size_t length parameterJeff King, Nov 13, 2020
  6. 5/5 packfile: detect overflow in .idx file size checksJeff King, Nov 13, 2020
  7. Johannes SchindelinNov 13, 2020
  8. Thomas BraunNov 15, 2020
  9. Jeff KingNov 16, 2020
  10. Derrick StoleeNov 16, 2020
  11. Jeff KingNov 16, 2020
  12. Thomas BraunNov 30, 2020
  13. Jeff KingDec 1, 2020
  14. t7900's new expensive testJeff King, Dec 1, 2020
  15. Taylor BlauDec 1, 2020
  16. Derrick StoleeDec 1, 2020
  17. t7900: speed up expensive testJeff King, Dec 2, 2020
  18. Jeff KingDec 2, 2020
  19. Derrick StoleeDec 3, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.