git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 5/8] odb: teach read_blob_entry to use size_t

From
Junio C Hamano <gitster@pobox.com>
Date
Oct 29, 2021, 23:17 UTC
Message-ID
<xmqqwnlvfmjr.fsf@gitster.g>
In-Reply-To
<dbef8168bc7044dbc8471c5ecc7146cf3e979263.1635515959.git.gitgitgadget@gmail.com>
"Matt Cooper via GitGitGadget" <gitgitgadget@gmail.com> writes:
Show 9 quoted lines
> -void *read_blob_entry(const struct cache_entry *ce, unsigned long *size)
> +void *read_blob_entry(const struct cache_entry *ce, size_t *size)
>  {
>  	enum object_type type;
> -	void *blob_data = read_object_file(&ce->oid, &type, size);
> +	unsigned long ul;
> +	void *blob_data = read_object_file(&ce->oid, &type, &ul);
>  
> +	*size = ul;

It is a bit curious place to draw the line; we want to make sure that blob_entry can hold huge data, but in this step we do not mind read_object_file() is not capable of going full 64-bit?

I guess I'll see soon enough why by reading later steps. I can see that for the purpose of making write_entry() aware of the size_t, this is necessary at the minimum.

Looking good.
Show 51 quoted lines
>  	if (blob_data) {
>  		if (type == OBJ_BLOB)
>  			return blob_data;
> @@ -270,7 +272,7 @@ static int write_entry(struct cache_entry *ce, char *path, struct conv_attrs *ca
>  	int fd, ret, fstat_done = 0;
>  	char *new_blob;
>  	struct strbuf buf = STRBUF_INIT;
> -	unsigned long size;
> +	size_t size;
>  	ssize_t wrote;
>  	size_t newsize = 0;
>  	struct stat st;
> diff --git a/entry.h b/entry.h
> index b8c0e170dc7..61ee8c17604 100644
> --- a/entry.h
> +++ b/entry.h
> @@ -51,7 +51,7 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts);
>   */
>  void unlink_entry(const struct cache_entry *ce);
>  
> -void *read_blob_entry(const struct cache_entry *ce, unsigned long *size);
> +void *read_blob_entry(const struct cache_entry *ce, size_t *size);
>  int fstat_checkout_output(int fd, const struct checkout *state, struct stat *st);
>  void update_ce_after_write(const struct checkout *state, struct cache_entry *ce,
>  			   struct stat *st);
> diff --git a/parallel-checkout.c b/parallel-checkout.c
> index 6b1af32bb3d..b6f4a25642e 100644
> --- a/parallel-checkout.c
> +++ b/parallel-checkout.c
> @@ -261,7 +261,7 @@ static int write_pc_item_to_fd(struct parallel_checkout_item *pc_item, int fd,
>  	struct stream_filter *filter;
>  	struct strbuf buf = STRBUF_INIT;
>  	char *blob;
> -	unsigned long size;
> +	size_t size;
>  	ssize_t wrote;
>  
>  	/* Sanity check */
> diff --git a/t/t1051-large-conversion.sh b/t/t1051-large-conversion.sh
> index bff86c13208..8b23d862600 100755
> --- a/t/t1051-large-conversion.sh
> +++ b/t/t1051-large-conversion.sh
> @@ -85,7 +85,7 @@ test_expect_success 'ident converts on output' '
>  
>  # This smudge filter prepends 5GB of zeros to the file it checks out. This
>  # ensures that smudging doesn't mangle large files on 64-bit Windows.
> -test_expect_failure EXPENSIVE,SIZE_T_IS_64BIT,!LONG_IS_64BIT \
> +test_expect_success EXPENSIVE,SIZE_T_IS_64BIT,!LONG_IS_64BIT \
>  		'files over 4GB convert on output' '
>  	test_commit test small "a small file" &&
>  	test_config filter.makelarge.smudge \
Previous: Matt Cooper via GitGitGadgetNext: Johannes Schindelin
Message 37 of 78 in “Allow clean/smudge filters to handle huge files in the LLP64 data model”
  1. 0/5 Allow clean/smudge filters to handle huge files in the LLP64 data modelJohannes Schindelin via GitGitGadget, Oct 27, 2021
  2. 1/5 t1051: introduce a smudge filter test for extremely large filesMatt Cooper via GitGitGadget, Oct 27, 2021
  3. Carlo ArenasOct 28, 2021
  4. helper/test-genzeros: allow more than 2G zeros in WindowsCarlo Marcelo Arenas Belón, Oct 28, 2021
  5. Johannes SchindelinOct 28, 2021
  6. 2/5 odb: teach read_blob_entry to use size_tMatt Cooper via GitGitGadget, Oct 27, 2021
  7. 3/5 git-compat-util: introduce more size_t helpersJohannes Schindelin via GitGitGadget, Oct 27, 2021
  8. 5/5 clean/smudge: allow clean filters to process extremely large filesMatt Cooper via GitGitGadget, Oct 27, 2021
  9. 4/5 odb: guard against data loss checking out a huge fileMatt Cooper via GitGitGadget, Oct 27, 2021
  10. 0/7 Allow clean/smudge filters to handle huge files in the LLP64 data modelJohannes Schindelin via GitGitGadget, Oct 28, 2021
  11. 1/7 test-genzeros: allow more than 2G zeros in WindowsCarlo Marcelo Arenas Belón via GitGitGadget, Oct 28, 2021
  12. 2/7 test-tool genzeros: generate large amounts of data more efficientlyJohannes Schindelin via GitGitGadget, Oct 28, 2021
  13. Junio C HamanoOct 28, 2021
  14. 3/7 t1051: introduce a smudge filter test for extremely large filesMatt Cooper via GitGitGadget, Oct 28, 2021
  15. 5/7 git-compat-util: introduce more size_t helpersJohannes Schindelin via GitGitGadget, Oct 28, 2021
  16. Junio C HamanoOct 28, 2021
  17. 4/7 odb: teach read_blob_entry to use size_tMatt Cooper via GitGitGadget, Oct 28, 2021
  18. Carlo ArenasOct 28, 2021
  19. Johannes SchindelinOct 28, 2021
  20. 6/7 odb: guard against data loss checking out a huge fileMatt Cooper via GitGitGadget, Oct 28, 2021
  21. 7/7 clean/smudge: allow clean filters to process extremely large filesMatt Cooper via GitGitGadget, Oct 28, 2021
  22. brian m. carlsonOct 28, 2021
  23. Junio C HamanoOct 28, 2021
  24. 0/8 Allow clean/smudge filters to handle huge files in the LLP64 data modelJohannes Schindelin via GitGitGadget, Oct 29, 2021
  25. 1/8 test-genzeros: allow more than 2G zeros in WindowsCarlo Marcelo Arenas Belón via GitGitGadget, Oct 29, 2021
  26. 2/8 test-tool genzeros: generate large amounts of data more efficientlyJohannes Schindelin via GitGitGadget, Oct 29, 2021
  27. Junio C HamanoOct 29, 2021
  28. 3/8 test-lib: add prerequisite for 64-bit platformsCarlo Marcelo Arenas Belón via GitGitGadget, Oct 29, 2021
  29. Junio C HamanoOct 29, 2021
  30. Johannes SchindelinNov 2, 2021
  31. 4/8 t1051: introduce a smudge filter test for extremely large filesMatt Cooper via GitGitGadget, Oct 29, 2021
  32. Junio C HamanoOct 29, 2021
  33. Junio C HamanoOct 29, 2021
  34. Johannes SchindelinNov 2, 2021
  35. Johannes SchindelinNov 2, 2021
  36. 5/8 odb: teach read_blob_entry to use size_tMatt Cooper via GitGitGadget, Oct 29, 2021
  37. Junio C HamanoOct 29, 2021
  38. Johannes SchindelinNov 2, 2021
  39. 6/8 git-compat-util: introduce more size_t helpersJohannes Schindelin via GitGitGadget, Oct 29, 2021
  40. Junio C HamanoOct 29, 2021
  41. 7/8 odb: guard against data loss checking out a huge fileMatt Cooper via GitGitGadget, Oct 29, 2021
  42. Junio C HamanoOct 29, 2021
  43. 8/8 clean/smudge: allow clean filters to process extremely large filesMatt Cooper via GitGitGadget, Oct 29, 2021
  44. Junio C HamanoOct 29, 2021
  45. Johannes SchindelinNov 2, 2021
  46. Junio C HamanoOct 29, 2021
  47. Johannes SchindelinNov 2, 2021
  48. 0/8 Allow clean/smudge filters to handle huge files in the LLP64 data modelJohannes Schindelin via GitGitGadget, Nov 2, 2021
  49. 1/8 test-genzeros: allow more than 2G zeros in WindowsCarlo Marcelo Arenas Belón via GitGitGadget, Nov 2, 2021
  50. 2/8 test-tool genzeros: generate large amounts of data more efficientlyJohannes Schindelin via GitGitGadget, Nov 2, 2021
  51. 3/8 test-lib: add prerequisite for 64-bit platformsCarlo Marcelo Arenas Belón via GitGitGadget, Nov 2, 2021
  52. 4/8 t1051: introduce a smudge filter test for extremely large filesMatt Cooper via GitGitGadget, Nov 2, 2021
  53. 5/8 odb: teach read_blob_entry to use size_tMatt Cooper via GitGitGadget, Nov 2, 2021
  54. Torsten BögershausenNov 2, 2021
  55. Johannes SchindelinNov 4, 2021
  56. Philip OakleyNov 4, 2021
  57. 6/8 git-compat-util: introduce more size_t helpersJohannes Schindelin via GitGitGadget, Nov 2, 2021
  58. 7/8 odb: guard against data loss checking out a huge fileMatt Cooper via GitGitGadget, Nov 2, 2021
  59. 8/8 clean/smudge: allow clean filters to process extremely large filesMatt Cooper via GitGitGadget, Nov 2, 2021
  60. Torsten BögershausenNov 2, 2021
  61. Johannes SchindelinNov 4, 2021
  62. Torsten BögershausenNov 4, 2021
  63. Junio C HamanoNov 4, 2021
  64. Torsten BögershausenNov 2, 2021
  65. Johannes SixtNov 3, 2021
  66. 0/3 Allow clean/smudge filters to handle huge files in the LLP64 data modelCarlo Marcelo Arenas Belón, Oct 28, 2021
  67. 1/3 test-lib: add prerequisite for 64-bit platformsCarlo Marcelo Arenas Belón, Oct 28, 2021
  68. Johannes SchindelinOct 28, 2021
  69. Carlo ArenasOct 28, 2021
  70. Junio C HamanoOct 28, 2021
  71. Johannes SchindelinNov 2, 2021
  72. 2/3 fixup! t1051: introduce a smudge filter test for extremely large filesCarlo Marcelo Arenas Belón, Oct 28, 2021
  73. 3/3 fixup! clean/smudge: allow clean filters to process extremely large filesCarlo Marcelo Arenas Belón, Oct 28, 2021
  74. Johannes SchindelinOct 29, 2021
  75. Junio C HamanoOct 29, 2021
  76. Philip OakleyOct 30, 2021
  77. Torsten BögershausenOct 30, 2021
  78. Philip OakleyOct 30, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.