git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v4 5/8] odb: teach read_blob_entry to use size_t

From
Johannes Schindelin <johannes.schindelin@gmx.de>
Date
Nov 4, 2021, 00:09 UTC
Message-ID
<nycvar.QRO.7.76.6.2111040102560.56@tvgsbejvaqbjf.bet>
In-Reply-To
<20211102204040.nnndsqrzu7ozpva7@tb-raspi4>
Hi Torsten,
On Tue, 2 Nov 2021, Torsten Bögershausen wrote:
Show 10 quoted lines
> On Tue, Nov 02, 2021 at 03:46:08PM +0000, Matt Cooper via GitGitGadget wrote:
> > From: Matt Cooper <vtbassmatt@gmail.com>
> >
> > There is mixed use of size_t and unsigned long to deal with sizes in the
> > codebase. Recall that Windows defines unsigned long as 32 bits even on
> > 64-bit platforms, meaning that converting size_t to unsigned long narrows
> > the range. This mostly doesn't cause a problem since Git rarely deals
> > with files larger than 2^32 bytes.
>
> What does this mean ?

I found the explanation to be quite clear (otherwise I would have changed it before submitting, obviously).

Git is rarely used with large files, as it was meant to track source code files, and it is pretty rare that a source code file is larger than 4GB. Therefore, the described issues are edge cases rather than common ones.

> > ... This mostly doesn't cause a problem since Git rarely deals
> > with files larger than 2^32 bytes.
>
> Is "mostly" is a good wording here ?
I do think so.
> May be
s/May be/Maybe/ since you're nitpicking wording ;-)
> This doesn't cause a problem when files smaller than 2^32 bytes are handled by Git.

That would lose the rather important fact that it is common to encounter only tracked files that are much smaller than 4GB.

Show 11 quoted lines
> > But adjunct systems such as Git LFS, which use smudge/clean filters to
> > keep huge files out of the repository, may have huge file contents passed
> > through some of the functions in entry.c and convert.c. On Windows, this
> > results in a truncated file being written to the workdir. I traced this to
> > one specific use of unsigned long in write_entry (and a similar instance
> > in write_pc_item_to_fd for parallel checkout). That appeared to be for
> > the call to read_blob_entry, which expects a pointer to unsigned long.
> >
> > By altering the signature of read_blob_entry to expect a size_t,
>
> "expect" -> "use"

I would say that in the context of talking about a signature, "expect" is a better verb than "use".

But then, just like you I am not a native speaker, so I think we should maybe stop telling a native speaker like Matt how to use his native tongue...

> (I am not a native English speaker, would "changing" be better than "altering" ?)
>  By changing the signature of read_blob_entry to use size_t,

As I said, I am not a native English speaker, either. So I believe that Matt knows better than the two of us together how to phrase things in English.

Since you had nothing to say about the patch itself, may I assume that you're fine with it?

Ciao, Dscho

Previous: Torsten BögershausenNext: Philip Oakley
Message 55 of 78 in “Allow clean/smudge filters to handle huge files in the LLP64 data model”
  1. 0/5 Allow clean/smudge filters to handle huge files in the LLP64 data modelJohannes Schindelin via GitGitGadget, Oct 27, 2021
  2. 1/5 t1051: introduce a smudge filter test for extremely large filesMatt Cooper via GitGitGadget, Oct 27, 2021
  3. Carlo ArenasOct 28, 2021
  4. helper/test-genzeros: allow more than 2G zeros in WindowsCarlo Marcelo Arenas Belón, Oct 28, 2021
  5. Johannes SchindelinOct 28, 2021
  6. 2/5 odb: teach read_blob_entry to use size_tMatt Cooper via GitGitGadget, Oct 27, 2021
  7. 3/5 git-compat-util: introduce more size_t helpersJohannes Schindelin via GitGitGadget, Oct 27, 2021
  8. 5/5 clean/smudge: allow clean filters to process extremely large filesMatt Cooper via GitGitGadget, Oct 27, 2021
  9. 4/5 odb: guard against data loss checking out a huge fileMatt Cooper via GitGitGadget, Oct 27, 2021
  10. 0/7 Allow clean/smudge filters to handle huge files in the LLP64 data modelJohannes Schindelin via GitGitGadget, Oct 28, 2021
  11. 1/7 test-genzeros: allow more than 2G zeros in WindowsCarlo Marcelo Arenas Belón via GitGitGadget, Oct 28, 2021
  12. 2/7 test-tool genzeros: generate large amounts of data more efficientlyJohannes Schindelin via GitGitGadget, Oct 28, 2021
  13. Junio C HamanoOct 28, 2021
  14. 3/7 t1051: introduce a smudge filter test for extremely large filesMatt Cooper via GitGitGadget, Oct 28, 2021
  15. 5/7 git-compat-util: introduce more size_t helpersJohannes Schindelin via GitGitGadget, Oct 28, 2021
  16. Junio C HamanoOct 28, 2021
  17. 4/7 odb: teach read_blob_entry to use size_tMatt Cooper via GitGitGadget, Oct 28, 2021
  18. Carlo ArenasOct 28, 2021
  19. Johannes SchindelinOct 28, 2021
  20. 6/7 odb: guard against data loss checking out a huge fileMatt Cooper via GitGitGadget, Oct 28, 2021
  21. 7/7 clean/smudge: allow clean filters to process extremely large filesMatt Cooper via GitGitGadget, Oct 28, 2021
  22. brian m. carlsonOct 28, 2021
  23. Junio C HamanoOct 28, 2021
  24. 0/8 Allow clean/smudge filters to handle huge files in the LLP64 data modelJohannes Schindelin via GitGitGadget, Oct 29, 2021
  25. 1/8 test-genzeros: allow more than 2G zeros in WindowsCarlo Marcelo Arenas Belón via GitGitGadget, Oct 29, 2021
  26. 2/8 test-tool genzeros: generate large amounts of data more efficientlyJohannes Schindelin via GitGitGadget, Oct 29, 2021
  27. Junio C HamanoOct 29, 2021
  28. 3/8 test-lib: add prerequisite for 64-bit platformsCarlo Marcelo Arenas Belón via GitGitGadget, Oct 29, 2021
  29. Junio C HamanoOct 29, 2021
  30. Johannes SchindelinNov 2, 2021
  31. 4/8 t1051: introduce a smudge filter test for extremely large filesMatt Cooper via GitGitGadget, Oct 29, 2021
  32. Junio C HamanoOct 29, 2021
  33. Junio C HamanoOct 29, 2021
  34. Johannes SchindelinNov 2, 2021
  35. Johannes SchindelinNov 2, 2021
  36. 5/8 odb: teach read_blob_entry to use size_tMatt Cooper via GitGitGadget, Oct 29, 2021
  37. Junio C HamanoOct 29, 2021
  38. Johannes SchindelinNov 2, 2021
  39. 6/8 git-compat-util: introduce more size_t helpersJohannes Schindelin via GitGitGadget, Oct 29, 2021
  40. Junio C HamanoOct 29, 2021
  41. 7/8 odb: guard against data loss checking out a huge fileMatt Cooper via GitGitGadget, Oct 29, 2021
  42. Junio C HamanoOct 29, 2021
  43. 8/8 clean/smudge: allow clean filters to process extremely large filesMatt Cooper via GitGitGadget, Oct 29, 2021
  44. Junio C HamanoOct 29, 2021
  45. Johannes SchindelinNov 2, 2021
  46. Junio C HamanoOct 29, 2021
  47. Johannes SchindelinNov 2, 2021
  48. 0/8 Allow clean/smudge filters to handle huge files in the LLP64 data modelJohannes Schindelin via GitGitGadget, Nov 2, 2021
  49. 1/8 test-genzeros: allow more than 2G zeros in WindowsCarlo Marcelo Arenas Belón via GitGitGadget, Nov 2, 2021
  50. 2/8 test-tool genzeros: generate large amounts of data more efficientlyJohannes Schindelin via GitGitGadget, Nov 2, 2021
  51. 3/8 test-lib: add prerequisite for 64-bit platformsCarlo Marcelo Arenas Belón via GitGitGadget, Nov 2, 2021
  52. 4/8 t1051: introduce a smudge filter test for extremely large filesMatt Cooper via GitGitGadget, Nov 2, 2021
  53. 5/8 odb: teach read_blob_entry to use size_tMatt Cooper via GitGitGadget, Nov 2, 2021
  54. Torsten BögershausenNov 2, 2021
  55. Johannes SchindelinNov 4, 2021
  56. Philip OakleyNov 4, 2021
  57. 6/8 git-compat-util: introduce more size_t helpersJohannes Schindelin via GitGitGadget, Nov 2, 2021
  58. 7/8 odb: guard against data loss checking out a huge fileMatt Cooper via GitGitGadget, Nov 2, 2021
  59. 8/8 clean/smudge: allow clean filters to process extremely large filesMatt Cooper via GitGitGadget, Nov 2, 2021
  60. Torsten BögershausenNov 2, 2021
  61. Johannes SchindelinNov 4, 2021
  62. Torsten BögershausenNov 4, 2021
  63. Junio C HamanoNov 4, 2021
  64. Torsten BögershausenNov 2, 2021
  65. Johannes SixtNov 3, 2021
  66. 0/3 Allow clean/smudge filters to handle huge files in the LLP64 data modelCarlo Marcelo Arenas Belón, Oct 28, 2021
  67. 1/3 test-lib: add prerequisite for 64-bit platformsCarlo Marcelo Arenas Belón, Oct 28, 2021
  68. Johannes SchindelinOct 28, 2021
  69. Carlo ArenasOct 28, 2021
  70. Junio C HamanoOct 28, 2021
  71. Johannes SchindelinNov 2, 2021
  72. 2/3 fixup! t1051: introduce a smudge filter test for extremely large filesCarlo Marcelo Arenas Belón, Oct 28, 2021
  73. 3/3 fixup! clean/smudge: allow clean filters to process extremely large filesCarlo Marcelo Arenas Belón, Oct 28, 2021
  74. Johannes SchindelinOct 29, 2021
  75. Junio C HamanoOct 29, 2021
  76. Philip OakleyOct 30, 2021
  77. Torsten BögershausenOct 30, 2021
  78. Philip OakleyOct 30, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.