{"thread":{"id":"34731","subject":"[PATCH] stream_to_pack: xread does not guarantee to read all requested bytes","startedAt":"2013-08-20T09:15:26Z","lastAt":"2013-08-20T19:37:14Z","messageCount":8,"participants":["Johannes Sixt","Junio C Hamano","Antoine Pelisse"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"225503","messageId":"521333AE.1090506@kdbg.org","threadId":"34731","inReplyTo":null,"subject":"[PATCH] stream_to_pack: xread does not guarantee to read all requested bytes","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2013-08-20T09:15:26Z","receivedAt":"2013-08-20T09:15:26Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"The deflate loop in bulk-checkin::stream_to_pack expects to get all bytes\nfrom a file that it requests to read in a single function call. But it\nused xread(), which does not give that guarantee. Replace it by\nread_in_full().\n\nSigned-off-by: Johannes Sixt \n---\n The size is limited to sizeof(ibuf) == 16384 bytes, so that there\n should not be a problem with the unpatched code on any OS in practice.\n Nevertheless, this change seems reasonable from a code hygiene POV.\n\n bulk-checkin.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex 6b0b6d4..118c625 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -114,7 +114,7 @@ static int stream_to_pack(struct bulk_checkin_state *state,\n \n \t\tif (size && !s.avail_in) {\n \t\t\tssize_t rsize = size < sizeof(ibuf) ? size : sizeof(ibuf);\n-\t\t\tif (xread(fd, ibuf, rsize) != rsize)\n+\t\t\tif (read_in_full(fd, ibuf, rsize) != rsize)\n \t\t\t\tdie(\"failed to read %d bytes from '%s'\",\n \t\t\t\t (int)rsize, path);\n \t\t\toffset += rsize;\n-- \n1.8.4.rc3.1241.gcbfd92d.dirty\n"},{"id":"225520","messageId":"xmqqsiy476h4.fsf@gitster.dls.corp.google.com","threadId":"34731","inReplyTo":"521333AE.1090506@kdbg.org","subject":"Re: [PATCH] stream_to_pack: xread does not guarantee to read all requested bytes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-08-20T15:00:07Z","receivedAt":"2013-08-20T15:00:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Sixt writes:\n\n> The deflate loop in bulk-checkin::stream_to_pack expects to get all bytes\n> from a file that it requests to read in a single function call. But it\n> used xread(), which does not give that guarantee. Replace it by\n> read_in_full().\n>\n> Signed-off-by: Johannes Sixt \n> ---\n> The size is limited to sizeof(ibuf) == 16384 bytes, so that there\n> should not be a problem with the unpatched code on any OS in practice.\n> Nevertheless, this change seems reasonable from a code hygiene POV.\n>\n> bulk-checkin.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/bulk-checkin.c b/bulk-checkin.c\n> index 6b0b6d4..118c625 100644\n> --- a/bulk-checkin.c\n> +++ b/bulk-checkin.c\n> @@ -114,7 +114,7 @@ static int stream_to_pack(struct bulk_checkin_state *state,\n> \n> \t\tif (size && !s.avail_in) {\n> \t\t\tssize_t rsize = size < sizeof(ibuf) ? size : sizeof(ibuf);\n> -\t\t\tif (xread(fd, ibuf, rsize) != rsize)\n> +\t\t\tif (read_in_full(fd, ibuf, rsize) != rsize)\n\nThis is the kind of thing i was wondering and worried about with the\nother \"clipped xread/xwrite\" patch. The original of this caller is\nobviously wrong. Thanks for spotting and fixing.\n\nI wonder if there are more like this broken caller or xread and/or\nxwrite.\n\n> \t\t\t\tdie(\"failed to read %d bytes from '%s'\",\n> \t\t\t\t (int)rsize, path);\n> \t\t\toffset += rsize;\n"},{"id":"225524","messageId":"CALWbr2yaQaCte6+Y3GEa8Hxyhq6GPS64aTNzkGo8pdpcd9ZVUg@mail.gmail.com","threadId":"34731","inReplyTo":"xmqqsiy476h4.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH] stream_to_pack: xread does not guarantee to read all requested bytes","fromName":"Antoine Pelisse","fromEmail":"apelisse@gmail.com","sentAt":"2013-08-20T15:16:22Z","receivedAt":"2013-08-20T15:16:22Z","isPatch":true,"sender":{"key":"apelisse@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1929644?v=4"},"body":"On Tue, Aug 20, 2013 at 5:00 PM, Junio C Hamano wrote:\n> Johannes Sixt writes:\n>\n>> The deflate loop in bulk-checkin::stream_to_pack expects to get all bytes\n>> from a file that it requests to read in a single function call. But it\n>> used xread(), which does not give that guarantee. Replace it by\n>> read_in_full().\n>>\n>> Signed-off-by: Johannes Sixt \n>> ---\n>> The size is limited to sizeof(ibuf) == 16384 bytes, so that there\n>> should not be a problem with the unpatched code on any OS in practice.\n>> Nevertheless, this change seems reasonable from a code hygiene POV.\n>>\n>> bulk-checkin.c | 2 +-\n>> 1 file changed, 1 insertion(+), 1 deletion(-)\n>>\n>> diff --git a/bulk-checkin.c b/bulk-checkin.c\n>> index 6b0b6d4..118c625 100644\n>> --- a/bulk-checkin.c\n>> +++ b/bulk-checkin.c\n>> @@ -114,7 +114,7 @@ static int stream_to_pack(struct bulk_checkin_state *state,\n>>\n>> if (size && !s.avail_in) {\n>> ssize_t rsize = size < sizeof(ibuf) ? size : sizeof(ibuf);\n>> - if (xread(fd, ibuf, rsize) != rsize)\n>> + if (read_in_full(fd, ibuf, rsize) != rsize)\n>\n> This is the kind of thing i was wondering and worried about with the\n> other \"clipped xread/xwrite\" patch. The original of this caller is\n> obviously wrong. Thanks for spotting and fixing.\n>\n> I wonder if there are more like this broken caller or xread and/or\n> xwrite.\n\nI was actually wondering when it's better to use xread() over\nread_in_full() ? Considering that we don't know if xread() will read\nthe whole buffer or not, would it not be better to always use\nread_in_full() ? I guess there is a drawback to this, but I'm not\nexactly sure what it is.\n"},{"id":"225535","messageId":"5213B41B.1080106@kdbg.org","threadId":"34731","inReplyTo":"xmqqsiy476h4.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH] stream_to_pack: xread does not guarantee to read all requested bytes","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2013-08-20T18:23:23Z","receivedAt":"2013-08-20T18:23:23Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 20.08.2013 17:00, schrieb Junio C Hamano:\n> I wonder if there are more like this broken caller or xread and/or\n> xwrite.\n\nLooking at the grep -C1 output, there are no others.\n\nThe only one that looked suspicious was xread in remote-curl.c, but it is \nfine (it just eats all data from the input).\n\n-- Hannes\n"},{"id":"225536","messageId":"xmqq8uzw6wuw.fsf@gitster.dls.corp.google.com","threadId":"34731","inReplyTo":"CALWbr2yaQaCte6+Y3GEa8Hxyhq6GPS64aTNzkGo8pdpcd9ZVUg@mail.gmail.com","subject":"Re: [PATCH] stream_to_pack: xread does not guarantee to read all requested bytes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-08-20T18:27:51Z","receivedAt":"2013-08-20T18:27:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Antoine Pelisse writes:\n\n> I was actually wondering when it's better to use xread() over\n> read_in_full()?\n\nWhen the caller wants to do more control over a read that may have\nto loop. For example, this loop in builtin/index-pack.c::fill()\n\n\tdo {\n\t\tssize_t ret = xread(input_fd, input_buffer + input_len,\n\t\t\t\tsizeof(input_buffer) - input_len);\n\t\tif (ret <= 0) {\n\t\t\tif (!ret)\n\t\t\t\tdie(_(\"early EOF\"));\n\t\t\tdie_errno(_(\"read error on input\"));\n\t\t}\n\t\tinput_len += ret;\n\t\tif (from_stdin)\n\t\t\tdisplay_throughput(progress, consumed_bytes + input_len);\n\t} while (input_len < min);\n\ncannot be replaced blindly with read_in_full() because (1) the\ncaller wants to do the \"display_throughput()\" part in the loop, and\n(2) the caller wants to fill at least \"min\" bytes but can happily\naccept to read more up to the size of the input_buffer.\n"},{"id":"225537","messageId":"5213B52F.7030205@kdbg.org","threadId":"34731","inReplyTo":"CALWbr2yaQaCte6+Y3GEa8Hxyhq6GPS64aTNzkGo8pdpcd9ZVUg@mail.gmail.com","subject":"Re: [PATCH] stream_to_pack: xread does not guarantee to read all requested bytes","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2013-08-20T18:27:59Z","receivedAt":"2013-08-20T18:27:59Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 20.08.2013 17:16, schrieb Antoine Pelisse:\n> I was actually wondering when it's better to use xread() over\n> read_in_full() ? Considering that we don't know if xread() will read\n> the whole buffer or not, would it not be better to always use\n> read_in_full() ?\n\nOf course, you know whether the whole buffer was filled: xread() returns \nthe number of bytes read. Same for xwrite().\n\n> I guess there is a drawback to this, but I'm not\n> exactly sure what it is.\n\nThe disadvantage of read_in_full() is that it can happen that it reads \ndata from the stream successfully, but then reports an error. In this \ncase, the data read is lost.\n\nAnalogously, when write_in_full() writes (some) data successfully, but \nultimately fails, you don't know how much was written successfully.\n\n-- Hannes\n"},{"id":"225542","messageId":"xmqq4nak6vpv.fsf@gitster.dls.corp.google.com","threadId":"34731","inReplyTo":"xmqqsiy476h4.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH] stream_to_pack: xread does not guarantee to read all requested bytes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-08-20T18:52:28Z","receivedAt":"2013-08-20T18:52:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> I wonder if there are more like this broken caller or xread and/or\n> xwrite.\n\nHere is a result of a quick audit (of 1.8.0.x codebase).\n\nAs xwrite() will not be splitting a single-byte request, the patch\nto cat-file is more or less a theoretical fix, but if writing the\ndate string can fail in I/O error, writing a terminating LF after it\ncan fail the same way, so we should be consistent.\n\nEverybody supports the side-band tranfer these days, so the patches\nto receive-pack and upload-pack are also theoretical fixes, I\nthink. Note that in the more recent codebase, safe_write() is gone\nand we use write_or_die() instead in upload-pack.\n\n builtin/cat-file.c | 2 +-\n builtin/receive-pack.c | 2 +-\n upload-pack.c | 5 -----\n 3 files changed, 2 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 00528dd..4beb4d8 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -63,7 +63,7 @@ static void pprint_tag(const unsigned char *sha1, const char *buf, unsigned long\n \t\t\t\t\ttz = strtol(ep, NULL, 10);\n \t\t\t\t\tsp = show_date(date, tz, 0);\n \t\t\t\t\twrite_or_die(1, sp, strlen(sp));\n-\t\t\t\t\txwrite(1, \"\\n\", 1);\n+\t\t\t\t\twrite_or_die(1, \"\\n\", 1);\n \t\t\t\t\tbreak;\n \t\t\t\t}\n \t\t\t}\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex ff781fe..a41740d 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -202,7 +202,7 @@ static void report_message(const char *prefix, const char *err, va_list params)\n \tif (use_sideband)\n \t\tsend_sideband(1, 2, msg, sz, use_sideband);\n \telse\n-\t\txwrite(2, msg, sz);\n+\t\twrite_in_full(2, msg, sz);\n }\n \n static void rp_warning(const char *err, ...)\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 2e90ccb..7a3e4fd 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -64,11 +64,6 @@ static ssize_t send_client_data(int fd, const char *data, ssize_t sz)\n \tif (fd == 3)\n \t\t/* emergency quit */\n \t\tfd = 2;\n-\tif (fd == 2) {\n-\t\t/* XXX: are we happy to lose stuff here? */\n-\t\txwrite(fd, data, sz);\n-\t\treturn sz;\n-\t}\n \treturn safe_write(fd, data, sz);\n }\n \n"},{"id":"225550","messageId":"5213C56A.6010305@kdbg.org","threadId":"34731","inReplyTo":"xmqq4nak6vpv.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH] stream_to_pack: xread does not guarantee to read all requested bytes","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2013-08-20T19:37:14Z","receivedAt":"2013-08-20T19:37:14Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 20.08.2013 20:52, schrieb Junio C Hamano:\n> Junio C Hamano writes:\n>\n>> I wonder if there are more like this broken caller or xread and/or\n>> xwrite.\n>\n> Here is a result of a quick audit (of 1.8.0.x codebase).\n>\n> As xwrite() will not be splitting a single-byte request, the patch\n> to cat-file is more or less a theoretical fix, but if writing the\n> date string can fail in I/O error, writing a terminating LF after it\n> can fail the same way, so we should be consistent.\n>\n> Everybody supports the side-band tranfer these days, so the patches\n> to receive-pack and upload-pack are also theoretical fixes, I\n> think. Note that in the more recent codebase, safe_write() is gone\n> and we use write_or_die() instead in upload-pack.\n\nThese changes look reasonable. Thank you!\n\n>\n> builtin/cat-file.c | 2 +-\n> builtin/receive-pack.c | 2 +-\n> upload-pack.c | 5 -----\n> 3 files changed, 2 insertions(+), 7 deletions(-)\n>\n> diff --git a/builtin/cat-file.c b/builtin/cat-file.c\n> index 00528dd..4beb4d8 100644\n> --- a/builtin/cat-file.c\n> +++ b/builtin/cat-file.c\n> @@ -63,7 +63,7 @@ static void pprint_tag(const unsigned char *sha1, const char *buf, unsigned long\n> \t\t\t\t\ttz = strtol(ep, NULL, 10);\n> \t\t\t\t\tsp = show_date(date, tz, 0);\n> \t\t\t\t\twrite_or_die(1, sp, strlen(sp));\n> -\t\t\t\t\txwrite(1, \"\\n\", 1);\n> +\t\t\t\t\twrite_or_die(1, \"\\n\", 1);\n> \t\t\t\t\tbreak;\n> \t\t\t\t}\n> \t\t\t}\n> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\n> index ff781fe..a41740d 100644\n> --- a/builtin/receive-pack.c\n> +++ b/builtin/receive-pack.c\n> @@ -202,7 +202,7 @@ static void report_message(const char *prefix, const char *err, va_list params)\n> \tif (use_sideband)\n> \t\tsend_sideband(1, 2, msg, sz, use_sideband);\n> \telse\n> -\t\txwrite(2, msg, sz);\n> +\t\twrite_in_full(2, msg, sz);\n> }\n>\n> static void rp_warning(const char *err, ...)\n> diff --git a/upload-pack.c b/upload-pack.c\n> index 2e90ccb..7a3e4fd 100644\n> --- a/upload-pack.c\n> +++ b/upload-pack.c\n> @@ -64,11 +64,6 @@ static ssize_t send_client_data(int fd, const char *data, ssize_t sz)\n> \tif (fd == 3)\n> \t\t/* emergency quit */\n> \t\tfd = 2;\n> -\tif (fd == 2) {\n> -\t\t/* XXX: are we happy to lose stuff here? */\n> -\t\txwrite(fd, data, sz);\n> -\t\treturn sz;\n> -\t}\n> \treturn safe_write(fd, data, sz);\n> }\n>\n>\n"}]}