{"thread":{"id":"48574","subject":"[PATCH] packfile: Correct zlib buffer handling","startedAt":"2018-05-25T23:17:25Z","lastAt":"2018-05-26T01:06:20Z","messageCount":3,"participants":["Jeremy Linton","Eric Sunshine","Todd Zullinger"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"348570","messageId":"20180525231713.23047-1-lintonrjeremy@gmail.com","threadId":"48574","inReplyTo":null,"subject":"[PATCH] packfile: Correct zlib buffer handling","fromName":"Jeremy Linton","fromEmail":"lintonrjeremy@gmail.com","sentAt":"2018-05-25T23:17:13Z","receivedAt":"2018-05-25T23:17:25Z","isPatch":true,"sender":{"key":"lintonrjeremy@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1294586?v=4"},"body":"The buffer being passed to zlib includes a null terminator that\ngit needs to keep in place. unpack_compressed_entry() attempts to\ndetect the case that the source buffer hasn't been fully consumed\nby checking to see if the destination buffer has been over consumed.\n\nThis yields two problems, first a single byte overrun won't be detected\nproperly because the Z_STREAM_END will then be set, but the null\nterminator will have been overwritten. The other problem is that\nmore recent zlib patches have been poisoning the unconsumed portions\nof the buffers which also overwrites the null, while correctly\nreturning length and status.\n\nLets rely on the fact that the source buffer will only be fully\nconsumed when the when the destination buffer is inflated to the\ncorrect size. We can do this by passing zlib the correct buffer size\nand properly checking the return status. The latter check actually\nalready exists if the buffer size is correct.\n\nSigned-off-by: Jeremy Linton \n---\n packfile.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 7c1a2519f..245eb3204 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1416,7 +1416,7 @@ static void *unpack_compressed_entry(struct packed_git *p,\n \t\treturn NULL;\n \tmemset(&stream, 0, sizeof(stream));\n \tstream.next_out = buffer;\n-\tstream.avail_out = size + 1;\n+\tstream.avail_out = size;\n \n \tgit_inflate_init(&stream);\n \tdo {\n@@ -1424,7 +1424,7 @@ static void *unpack_compressed_entry(struct packed_git *p,\n \t\tstream.next_in = in;\n \t\tst = git_inflate(&stream, Z_FINISH);\n \t\tif (!stream.avail_out)\n-\t\t\tbreak; /* the payload is larger than it should be */\n+\t\t\tbreak; /* done, st indicates if source fully consumed */\n \t\tcurpos += stream.next_in - in;\n \t} while (st == Z_OK || st == Z_BUF_ERROR);\n \tgit_inflate_end(&stream);\n-- \n2.13.6\n\n"},{"id":"348571","messageId":"CAPig+cS=7EbGviDYoNFNF-CrMfCY4mySQoWX3JtJBhbNf105sw@mail.gmail.com","threadId":"48574","inReplyTo":"20180525231713.23047-1-lintonrjeremy@gmail.com","subject":"Re: [PATCH] packfile: Correct zlib buffer handling","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-05-25T23:36:42Z","receivedAt":"2018-05-25T23:36:46Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, May 25, 2018 at 7:17 PM, Jeremy Linton wrote:\n> The buffer being passed to zlib includes a null terminator that\n> git needs to keep in place. unpack_compressed_entry() attempts to\n> detect the case that the source buffer hasn't been fully consumed\n> by checking to see if the destination buffer has been over consumed.\n>\n> This yields two problems, first a single byte overrun won't be detected\n> properly because the Z_STREAM_END will then be set, but the null\n> terminator will have been overwritten. The other problem is that\n> more recent zlib patches have been poisoning the unconsumed portions\n> of the buffers which also overwrites the null, while correctly\n> returning length and status.\n>\n> Lets rely on the fact that the source buffer will only be fully\n\ns/Lets/Let's/\n\n> consumed when the when the destination buffer is inflated to the\n\ns/when the when the/when the/\n\n> correct size. We can do this by passing zlib the correct buffer size\n> and properly checking the return status. The latter check actually\n> already exists if the buffer size is correct.\n>\n> Signed-off-by: Jeremy Linton \n"},{"id":"348573","messageId":"20180526010611.GI26695@zaya.teonanacatl.net","threadId":"48574","inReplyTo":"20180525231713.23047-1-lintonrjeremy@gmail.com","subject":"Re: [PATCH] packfile: Correct zlib buffer handling","fromName":"Todd Zullinger","fromEmail":"tmz@pobox.com","sentAt":"2018-05-26T01:06:11Z","receivedAt":"2018-05-26T01:06:20Z","isPatch":true,"sender":{"key":"tmz@pobox.com","avatar":"https://avatars.githubusercontent.com/u/806319?v=4"},"body":"Jeremy Linton wrote:\n> The buffer being passed to zlib includes a null terminator that\n> git needs to keep in place. unpack_compressed_entry() attempts to\n> detect the case that the source buffer hasn't been fully consumed\n> by checking to see if the destination buffer has been over consumed.\n> \n> This yields two problems, first a single byte overrun won't be detected\n> properly because the Z_STREAM_END will then be set, but the null\n> terminator will have been overwritten. The other problem is that\n> more recent zlib patches have been poisoning the unconsumed portions\n> of the buffers which also overwrites the null, while correctly\n> returning length and status.\n> \n> Lets rely on the fact that the source buffer will only be fully\n> consumed when the when the destination buffer is inflated to the\n> correct size. We can do this by passing zlib the correct buffer size\n> and properly checking the return status. The latter check actually\n> already exists if the buffer size is correct.\n> \n> Signed-off-by: Jeremy Linton \n> ---\n\nAs a little background, earlier today Pavel Cahyna filed a\nticket about a regression in a recent zlib update on aarch64\nin Fedora[1].\n\nWhile he was doing that I was just beginning to look at why\nthe git test suite failed fairly badly a build last\nnight[2]. The aarch64 build on Fedora 28 failed, while it\nsucceeded on all other architectures (armv7hl, i686, ppc64,\nppc64le, s390x, x86_64). It also passed on newer and older\nFedora releases.\n\nThe difference was that the Fedora 28 zlib build has some\naarch64 optimization patches applied[3]. (Those patches are\nin rawhide/f29 as well, but due to an unrelated issue have\nnot yet made it into the buildroot used for the git build.)\n\nI'm woefully unqualified to comment on the patch, but if\nthere are any questions about how this was found, I hope the\nabove background will be helpful.\n\nA big thanks to Jeremy for dropping whatever he had planned\nto do today and dig into this issue. I can only hope it was\neither more fun or less work than what he hoped to do with\nhis Friday. :)\n\nThanks also to Pavel for finding the source of the failures\nand filing a detailed bug report to get things moving.\n\n[1] https://bugzilla.redhat.com/1582555\n[2] https://fedorapeople.org/~tmz/git-aarch64-make-test\n[3] https://src.fedoraproject.org/rpms/zlib/c/25e9802\n\n> packfile.c | 4 ++--\n> 1 file changed, 2 insertions(+), 2 deletions(-)\n> \n> diff --git a/packfile.c b/packfile.c\n> index 7c1a2519f..245eb3204 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -1416,7 +1416,7 @@ static void *unpack_compressed_entry(struct packed_git *p,\n> \t\treturn NULL;\n> \tmemset(&stream, 0, sizeof(stream));\n> \tstream.next_out = buffer;\n> -\tstream.avail_out = size + 1;\n> +\tstream.avail_out = size;\n> \n> \tgit_inflate_init(&stream);\n> \tdo {\n> @@ -1424,7 +1424,7 @@ static void *unpack_compressed_entry(struct packed_git *p,\n> \t\tstream.next_in = in;\n> \t\tst = git_inflate(&stream, Z_FINISH);\n> \t\tif (!stream.avail_out)\n> -\t\t\tbreak; /* the payload is larger than it should be */\n> +\t\t\tbreak; /* done, st indicates if source fully consumed */\n> \t\tcurpos += stream.next_in - in;\n> \t} while (st == Z_OK || st == Z_BUF_ERROR);\n> \tgit_inflate_end(&stream);\n\n-- \nTodd\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nAn election is coming. Universal peace is declared and the foxes have\na sincere interest in prolonging the lives of the poultry.\n -- T.S. Eliot\n\n"}]}