git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 1/5] archive-zip: add tests for big ZIP archives

From
René Scharfe <l.s.r@web.de>
Date
Apr 24, 2017, 17:29 UTC
Message-ID
<7361b84d-41c4-8224-ff83-36703837eb28@web.de>
In-Reply-To
<85f2b6d1-107b-0624-af82-92446f28269e@web.de>

Test the creation of ZIP archives bigger than 4GB and containing files bigger than 4GB. They are marked as EXPENSIVE because they take quite a while and because the first one needs a bit more than 4GB of disk space to store the resulting archive.

The big archive in the first test is made up of a tree containing thousands of copies of a small file. Yet the test has to write out the full archive because unzip doesn't offer a way to read from stdin.

The big file in the second test is provided as a zipped pack file to avoid writing another 4GB file to disk and then adding it.

Signed-off-by: Rene Scharfe <l.s.r@web.de>
---
 t/t5004-archive-corner-cases.sh |  45 ++++++++++++++++++++++++++++++++++++++++
 t/t5004/big-pack.zip            | Bin 0 -> 7373 bytes
 2 files changed, 45 insertions(+)
 create mode 100644 t/t5004/big-pack.zip
diff --git a/t/t5004-archive-corner-cases.sh b/t/t5004-archive-corner-cases.sh
index cca23383c5..bc052c803a 100755
--- a/t/t5004-archive-corner-cases.sh
+++ b/t/t5004-archive-corner-cases.sh
@@ -155,4 +155,49 @@ test_expect_success ZIPINFO 'zip archive with many entries' '
 	test_cmp expect actual
 '
 
+test_expect_failure EXPENSIVE,UNZIP 'zip archive bigger than 4GB' '
+	# build string containing 65536 characters
+	s=0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef &&
+	s=$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s &&
+	s=$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s$s &&
+
+	# create blob with a length of 65536 + 1 bytes
+	blob=$(echo $s | git hash-object -w --stdin) &&
+
+	# create tree containing 65500 entries of that blob
+	for i in $(test_seq 1 65500)
+	do
+		echo "100644 blob $blob	$i"
+	done >tree &&
+	tree=$(git mktree <tree) &&
+
+	# zip it, creating an archive a bit bigger than 4GB
+	git archive -0 -o many-big.zip $tree &&
+
+	"$GIT_UNZIP" -t many-big.zip 9999 65500 &&
+	"$GIT_UNZIP" -t many-big.zip
+'
+
+test_expect_failure EXPENSIVE,UNZIP,ZIPINFO 'zip archive with files bigger than 4GB' '
+	# Pack created with:
+	#   dd if=/dev/zero of=file bs=1M count=4100 && git hash-object -w file
+	mkdir -p .git/objects/pack &&
+	(
+		cd .git/objects/pack &&
+		"$GIT_UNZIP" "$TEST_DIRECTORY"/t5004/big-pack.zip
+	) &&
+	blob=754a93d6fada4c6873360e6cb4b209132271ab0e &&
+	size=$(expr 4100 "*" 1024 "*" 1024) &&
+
+	# create a tree containing the file
+	tree=$(echo "100644 blob $blob	big-file" | git mktree) &&
+
+	# zip it, creating an archive with a file bigger than 4GB
+	git archive -o big.zip $tree &&
+
+	"$GIT_UNZIP" -t big.zip &&
+	"$ZIPINFO" big.zip >big.lst &&
+	grep $size big.lst
+'
+
 test_done
diff --git a/t/t5004/big-pack.zip b/t/t5004/big-pack.zip
new file mode 100644
index 0000000000000000000000000000000000000000..caaf614eeece6f818c525e433561e37560a75b05
GIT binary patch
literal 7373
zcmWIWW@Zs#U}E54xLY#A%SmVLl4u471|Jp%215oJhJwW8Y+ZvC^HlQ`3(KT5gS6zd
z#6$y=#3VyYlN6KGlw?bTG()50MB`LTb5p&{l#0+0P6p<OAOA*xaA^fM10%}|W(Ec@
z@jL#}{4)m*95`aY)ux;vb9C_xiI`WWKX0v%wv#-XR%01$R>-9-y6#!_6|EcGQyxED
zyZY^&^YwM2r?Wic_gf3wz0#?R{8hB^<Js4()@9Ms>$gt{iHiUEJN{iALjZ~|R(VzA
zOp{_@IDE*S!H8sEfc%Wl8*lHd?-DJPX@5BLD~n)>se}uQ{sHa{9CBhhi*hb3*-*jg
zc5o4&4%_XW4Ba<OG7P)VrhVaOTdmN<Ho4-C?R3NUpAYE!{5{5hc=zr*JFR#QUppur
z?|(T<>b?Enk9jiu`<7qbA8y|-Ck>2+hMBYN-pgCcFt>ev-5*|-&jb`JE>Hh`B*~cT
z&t2=?C44}E8GDaU*PD0ekK%{l7w@(f14RzJnfvea+fQlS75nRoU$s?(g=&R(fOLb%
zK_JPHAqeJ(fjJ$>oKcz4&>2l3kc=^!7e@2KXkHl23k(gTVK5p7z>;7z9gKznQp0()
zeK6WS7;PVn){Ud}!f4$%I)=h9I*!CJ8V10UU^E?!h5@KqG@1@Z!(cQWK$^#7<%OS{
z_HQoT!K!n;di%NDON?i(ygj-((dOTWgOj)4mXFhko42#<@9S6BTc6*5|Cc?$n~_P5
z8P`mn1SleafRRC^5k!+Qug40R*F&4rL$?-n>J8c2V<cM(nTW$>FDo0!BTPW}9!Q@6
I&6hC%0B76lcmMzZ

literal 0
HcmV?d00001
-- 
2.12.2
Previous: René ScharfeNext: René Scharfe
Message 11 of 44 in “Git archive doesn't fully support zip64”
  1. Keith GoldfarbApr 21, 2017
  2. archive-zip: Add zip64 headers when file size is too large for 32 bitsPeter Krefting, Apr 22, 2017
  3. Johannes SixtApr 22, 2017
  4. archive-zip: Add zip64 headers when file size is too large for 32 bitsPeter Krefting, Apr 22, 2017
  5. Johannes SixtApr 23, 2017
  6. Peter KreftingApr 23, 2017
  7. Johannes SixtApr 23, 2017
  8. Peter KreftingApr 24, 2017
  9. René ScharfeApr 24, 2017
  10. 0/5 archive-zip: support files and archives bigger than 4GBRené Scharfe, Apr 24, 2017
  11. 1/5 archive-zip: add tests for big ZIP archivesRené Scharfe, Apr 24, 2017
  12. 2/5 archive-zip: use strbuf for ZIP directoryRené Scharfe, Apr 24, 2017
  13. Junio C HamanoApr 25, 2017
  14. René ScharfeApr 25, 2017
  15. 3/5 archive-zip: write ZIP dir entry directly to strbufRené Scharfe, Apr 24, 2017
  16. 4/5 archive-zip: support archives bigger than 4GBRené Scharfe, Apr 24, 2017
  17. Peter KreftingApr 24, 2017
  18. René ScharfeApr 24, 2017
  19. René ScharfeApr 24, 2017
  20. Johannes SixtApr 24, 2017
  21. René ScharfeApr 24, 2017
  22. Peter KreftingApr 25, 2017
  23. René ScharfeApr 25, 2017
  24. Peter KreftingApr 26, 2017
  25. René ScharfeApr 26, 2017
  26. Peter KreftingApr 27, 2017
  27. René ScharfeApr 27, 2017
  28. Peter KreftingApr 28, 2017
  29. 5/5 archive-zip: support files bigger than 4GBRené Scharfe, Apr 24, 2017
  30. Keith GoldfarbApr 24, 2017
  31. Junio C HamanoApr 25, 2017
  32. René ScharfeApr 25, 2017
  33. Torsten BögershausenApr 29, 2017
  34. René ScharfeApr 29, 2017
  35. Torsten BögershausenApr 30, 2017
  36. René ScharfeApr 30, 2017
  37. Torsten BögershausenApr 30, 2017
  38. Johannes SixtApr 30, 2017
  39. René ScharfeApr 30, 2017
  40. Junio C HamanoApr 30, 2017
  41. René ScharfeMay 1, 2017
  42. René ScharfeApr 23, 2017
  43. Peter KreftingApr 23, 2017
  44. Johannes SixtApr 23, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.