# [PATCH] sha1_file: don't convert off_t to size_t too early to avoid potential die() 3 messages from 2014-09-21 to 2014-09-22. Participants: Steffen Prohaska, Junio C Hamano. Thread: https://gitlist.dev/t/37609 ## Steffen Prohaska, 2014-09-21 10:03 Subject: [PATCH] sha1_file: don't convert off_t to size_t too early to avoid potential die() Message-ID: <1411293806-3087-1-git-send-email-prohaska@zib.de> URL: https://gitlist.dev/e/1411293806-3087-1-git-send-email-prohaska%40zib.de ``` xsize_t() checks if an off_t argument can be safely converted to a size_t return value. If the check is executed too early, it could fail for large files on 32-bit architectures even if the size_t code path is not taken. Other paths might be able to handle the large file. Specifically, index_stream_convert_blob() is able to handle a large file if a filter is configured that returns a small result. Signed-off-by: Steffen Prohaska --- This patch should be applied on top of sp/stream-clean-filter. index_stream() might internally also be able to handle large files to some extent. But it uses size_t for its third argument, and we must already die() when calling it. It might be a good idea to convert its interface to use off_t and push the size checks further down the stack. In general, it might be good idea to carefully consider whether to use off_t or size_t when passing file-related sizes around. To me it looks like a separate issue for a separate patch series (I have no specific plans to prepare one). sha1_file.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/sha1_file.c b/sha1_file.c index 5b0e67a..6f18c22 100644 --- a/sha1_file.c +++ b/sha1_file.c @@ -3180,17 +3180,22 @@ int index_fd(unsigned char *sha1, int fd, struct stat *st, enum object_type type, const char *path, unsigned flags) { int ret; - size_t size = xsize_t(st->st_size); + /* + * Call xsize_t() only when needed to avoid potentially unnecessary + * die() for large files. + */ if (type == OBJ_BLOB && path && would_convert_to_git_filter_fd(path)) ret = index_stream_convert_blob(sha1, fd, path, flags); else if (!S_ISREG(st->st_mode)) ret = index_pipe(sha1, fd, type, path, flags); - else if (size <= big_file_threshold || type != OBJ_BLOB || + else if (st->st_size <= big_file_threshold || type != OBJ_BLOB || (path && would_convert_to_git(path))) - ret = index_core(sha1, fd, size, type, path, flags); + ret = index_core(sha1, fd, xsize_t(st->st_size), type, path, + flags); else - ret = index_stream(sha1, fd, size, type, path, flags); + ret = index_stream(sha1, fd, xsize_t(st->st_size), type, path, + flags); close(fd); return ret; } -- 2.1.0.139.g351b19f ``` ## Junio C Hamano, 2014-09-22 17:49 Subject: Re: [PATCH] sha1_file: don't convert off_t to size_t too early to avoid potential die() Message-ID: URL: https://gitlist.dev/e/xmqqfvfjlf4k.fsf%40gitster.dls.corp.google.com In-Reply-To: <1411293806-3087-1-git-send-email-prohaska@zib.de> ``` Steffen Prohaska writes: > xsize_t() checks if an off_t argument can be safely converted to > a size_t return value. If the check is executed too early, it could > fail for large files on 32-bit architectures even if the size_t code > path is not taken. Other paths might be able to handle the large file. > Specifically, index_stream_convert_blob() is able to handle a large file > if a filter is configured that returns a small result. > > Signed-off-by: Steffen Prohaska > --- > > This patch should be applied on top of sp/stream-clean-filter. > > index_stream() might internally also be able to handle large files to > some extent. But it uses size_t for its third argument, and we must > already die() when calling it. It might be a good idea to convert its > interface to use off_t and push the size checks further down the stack. Yes, if we want to futz in this area, I think that would be the right approach. ``` ## Junio C Hamano, 2014-09-22 19:41 Subject: Re: [PATCH] sha1_file: don't convert off_t to size_t too early to avoid potential die() Message-ID: URL: https://gitlist.dev/e/xmqqvbofjvdf.fsf%40gitster.dls.corp.google.com In-Reply-To: <1411293806-3087-1-git-send-email-prohaska@zib.de> ``` Steffen Prohaska writes: > This patch should be applied on top of sp/stream-clean-filter. ... or it can be squashed in as a fix, as the topic is not yet in 'next'. > index_stream() might internally also be able to handle large files to > some extent. But it uses size_t for its third argument, and we must > already die() when calling it. It might be a good idea to convert its > interface to use off_t and push the size checks further down the stack. > In general, it might be good idea to carefully consider whether to use > off_t or size_t when passing file-related sizes around. To me it looks > like a separate issue for a separate patch series (I have no specific > plans to prepare one). > > sha1_file.c | 13 +++++++++---- > 1 file changed, 9 insertions(+), 4 deletions(-) > > diff --git a/sha1_file.c b/sha1_file.c > index 5b0e67a..6f18c22 100644 > --- a/sha1_file.c > +++ b/sha1_file.c > @@ -3180,17 +3180,22 @@ int index_fd(unsigned char *sha1, int fd, struct stat *st, > enum object_type type, const char *path, unsigned flags) > { > int ret; > - size_t size = xsize_t(st->st_size); > > + /* > + * Call xsize_t() only when needed to avoid potentially unnecessary > + * die() for large files. > + */ > if (type == OBJ_BLOB && path && would_convert_to_git_filter_fd(path)) > ret = index_stream_convert_blob(sha1, fd, path, flags); > else if (!S_ISREG(st->st_mode)) > ret = index_pipe(sha1, fd, type, path, flags); > - else if (size <= big_file_threshold || type != OBJ_BLOB || > + else if (st->st_size <= big_file_threshold || type != OBJ_BLOB || > (path && would_convert_to_git(path))) > - ret = index_core(sha1, fd, size, type, path, flags); > + ret = index_core(sha1, fd, xsize_t(st->st_size), type, path, > + flags); > else > - ret = index_stream(sha1, fd, size, type, path, flags); > + ret = index_stream(sha1, fd, xsize_t(st->st_size), type, path, > + flags); > close(fd); > return ret; > } ```