git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 4/8] t1051: introduce a smudge filter test for extremely large files

From
Johannes Schindelin <johannes.schindelin@gmx.de>
Date
Nov 2, 2021, 14:56 UTC
Message-ID
<nycvar.QRO.7.76.6.2111021542110.56@tvgsbejvaqbjf.bet>
In-Reply-To
<xmqqlf2bfmdb.fsf@gitster.g>
Hi Junio,
On Fri, 29 Oct 2021, Junio C Hamano wrote:
Show 21 quoted lines
> Junio C Hamano <gitster@pobox.com> writes:
>
> >> +# This smudge filter prepends 5GB of zeros to the file it checks out. This
> >> +# ensures that smudging doesn't mangle large files on 64-bit Windows.
> >> +test_expect_failure EXPENSIVE,SIZE_T_IS_64BIT,!LONG_IS_64BIT \
> >> +		'files over 4GB convert on output' '
> >> +	test_commit test small "a small file" &&
> >> +	test_config filter.makelarge.smudge \
> >> +		"test-tool genzeros $((5*1024*1024*1024)) && cat" &&
> >> +	echo "small filter=makelarge" >.gitattributes &&
> >> +	rm small &&
> >> +	git checkout -- small &&
> >> +	size=$(test_file_size small) &&
> >> +	test "$size" -ge $((5 * 1024 * 1024 * 1024))
> >> +'
> >
> > Why not exactly 5G, but anything that is at least 5G is OK?
>
> I know it is more than 5G, thanks to the "&& cat".  THe question was
> why aren't we measuring the size of "a small file" so that we can
> check against an exact size to be expected.

The main problem solved by this patch series is the fact that by virtue of using `unsigned long`, Git truncated the contents to less than 4GB. So, really, what we care about here is that that does not happen anymore.

I will change it to take the extra time to determine `small`'s file size and add that to 5GB, to avoid confusing readers, though.

Ciao, Dscho

Previous: Junio C HamanoNext: Johannes Schindelin
Message 34 of 78 in “Allow clean/smudge filters to handle huge files in the LLP64 data model”
  1. 0/5 Allow clean/smudge filters to handle huge files in the LLP64 data modelJohannes Schindelin via GitGitGadget, Oct 27, 2021
  2. 1/5 t1051: introduce a smudge filter test for extremely large filesMatt Cooper via GitGitGadget, Oct 27, 2021
  3. Carlo ArenasOct 28, 2021
  4. helper/test-genzeros: allow more than 2G zeros in WindowsCarlo Marcelo Arenas Belón, Oct 28, 2021
  5. Johannes SchindelinOct 28, 2021
  6. 2/5 odb: teach read_blob_entry to use size_tMatt Cooper via GitGitGadget, Oct 27, 2021
  7. 3/5 git-compat-util: introduce more size_t helpersJohannes Schindelin via GitGitGadget, Oct 27, 2021
  8. 5/5 clean/smudge: allow clean filters to process extremely large filesMatt Cooper via GitGitGadget, Oct 27, 2021
  9. 4/5 odb: guard against data loss checking out a huge fileMatt Cooper via GitGitGadget, Oct 27, 2021
  10. 0/7 Allow clean/smudge filters to handle huge files in the LLP64 data modelJohannes Schindelin via GitGitGadget, Oct 28, 2021
  11. 1/7 test-genzeros: allow more than 2G zeros in WindowsCarlo Marcelo Arenas Belón via GitGitGadget, Oct 28, 2021
  12. 2/7 test-tool genzeros: generate large amounts of data more efficientlyJohannes Schindelin via GitGitGadget, Oct 28, 2021
  13. Junio C HamanoOct 28, 2021
  14. 3/7 t1051: introduce a smudge filter test for extremely large filesMatt Cooper via GitGitGadget, Oct 28, 2021
  15. 5/7 git-compat-util: introduce more size_t helpersJohannes Schindelin via GitGitGadget, Oct 28, 2021
  16. Junio C HamanoOct 28, 2021
  17. 4/7 odb: teach read_blob_entry to use size_tMatt Cooper via GitGitGadget, Oct 28, 2021
  18. Carlo ArenasOct 28, 2021
  19. Johannes SchindelinOct 28, 2021
  20. 6/7 odb: guard against data loss checking out a huge fileMatt Cooper via GitGitGadget, Oct 28, 2021
  21. 7/7 clean/smudge: allow clean filters to process extremely large filesMatt Cooper via GitGitGadget, Oct 28, 2021
  22. brian m. carlsonOct 28, 2021
  23. Junio C HamanoOct 28, 2021
  24. 0/8 Allow clean/smudge filters to handle huge files in the LLP64 data modelJohannes Schindelin via GitGitGadget, Oct 29, 2021
  25. 1/8 test-genzeros: allow more than 2G zeros in WindowsCarlo Marcelo Arenas Belón via GitGitGadget, Oct 29, 2021
  26. 2/8 test-tool genzeros: generate large amounts of data more efficientlyJohannes Schindelin via GitGitGadget, Oct 29, 2021
  27. Junio C HamanoOct 29, 2021
  28. 3/8 test-lib: add prerequisite for 64-bit platformsCarlo Marcelo Arenas Belón via GitGitGadget, Oct 29, 2021
  29. Junio C HamanoOct 29, 2021
  30. Johannes SchindelinNov 2, 2021
  31. 4/8 t1051: introduce a smudge filter test for extremely large filesMatt Cooper via GitGitGadget, Oct 29, 2021
  32. Junio C HamanoOct 29, 2021
  33. Junio C HamanoOct 29, 2021
  34. Johannes SchindelinNov 2, 2021
  35. Johannes SchindelinNov 2, 2021
  36. 5/8 odb: teach read_blob_entry to use size_tMatt Cooper via GitGitGadget, Oct 29, 2021
  37. Junio C HamanoOct 29, 2021
  38. Johannes SchindelinNov 2, 2021
  39. 6/8 git-compat-util: introduce more size_t helpersJohannes Schindelin via GitGitGadget, Oct 29, 2021
  40. Junio C HamanoOct 29, 2021
  41. 7/8 odb: guard against data loss checking out a huge fileMatt Cooper via GitGitGadget, Oct 29, 2021
  42. Junio C HamanoOct 29, 2021
  43. 8/8 clean/smudge: allow clean filters to process extremely large filesMatt Cooper via GitGitGadget, Oct 29, 2021
  44. Junio C HamanoOct 29, 2021
  45. Johannes SchindelinNov 2, 2021
  46. Junio C HamanoOct 29, 2021
  47. Johannes SchindelinNov 2, 2021
  48. 0/8 Allow clean/smudge filters to handle huge files in the LLP64 data modelJohannes Schindelin via GitGitGadget, Nov 2, 2021
  49. 1/8 test-genzeros: allow more than 2G zeros in WindowsCarlo Marcelo Arenas Belón via GitGitGadget, Nov 2, 2021
  50. 2/8 test-tool genzeros: generate large amounts of data more efficientlyJohannes Schindelin via GitGitGadget, Nov 2, 2021
  51. 3/8 test-lib: add prerequisite for 64-bit platformsCarlo Marcelo Arenas Belón via GitGitGadget, Nov 2, 2021
  52. 4/8 t1051: introduce a smudge filter test for extremely large filesMatt Cooper via GitGitGadget, Nov 2, 2021
  53. 5/8 odb: teach read_blob_entry to use size_tMatt Cooper via GitGitGadget, Nov 2, 2021
  54. Torsten BögershausenNov 2, 2021
  55. Johannes SchindelinNov 4, 2021
  56. Philip OakleyNov 4, 2021
  57. 6/8 git-compat-util: introduce more size_t helpersJohannes Schindelin via GitGitGadget, Nov 2, 2021
  58. 7/8 odb: guard against data loss checking out a huge fileMatt Cooper via GitGitGadget, Nov 2, 2021
  59. 8/8 clean/smudge: allow clean filters to process extremely large filesMatt Cooper via GitGitGadget, Nov 2, 2021
  60. Torsten BögershausenNov 2, 2021
  61. Johannes SchindelinNov 4, 2021
  62. Torsten BögershausenNov 4, 2021
  63. Junio C HamanoNov 4, 2021
  64. Torsten BögershausenNov 2, 2021
  65. Johannes SixtNov 3, 2021
  66. 0/3 Allow clean/smudge filters to handle huge files in the LLP64 data modelCarlo Marcelo Arenas Belón, Oct 28, 2021
  67. 1/3 test-lib: add prerequisite for 64-bit platformsCarlo Marcelo Arenas Belón, Oct 28, 2021
  68. Johannes SchindelinOct 28, 2021
  69. Carlo ArenasOct 28, 2021
  70. Junio C HamanoOct 28, 2021
  71. Johannes SchindelinNov 2, 2021
  72. 2/3 fixup! t1051: introduce a smudge filter test for extremely large filesCarlo Marcelo Arenas Belón, Oct 28, 2021
  73. 3/3 fixup! clean/smudge: allow clean filters to process extremely large filesCarlo Marcelo Arenas Belón, Oct 28, 2021
  74. Johannes SchindelinOct 29, 2021
  75. Junio C HamanoOct 29, 2021
  76. Philip OakleyOct 30, 2021
  77. Torsten BögershausenOct 30, 2021
  78. Philip OakleyOct 30, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.