{"thread":{"id":"51306","subject":"[PATCH] cleanup: fix possible overflow errors in binary search, part 2","startedAt":"2019-06-13T17:52:16Z","lastAt":"2019-06-14T04:19:08Z","messageCount":5,"participants":["René Scharfe","Derrick Stolee","Martin Ågren"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"377163","messageId":"f0059613-d3e4-a1e2-2c01-727579e069d2@web.de","threadId":"51306","inReplyTo":null,"subject":"[PATCH] cleanup: fix possible overflow errors in binary search, part 2","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2019-06-13T17:51:56Z","receivedAt":"2019-06-13T17:52:16Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Calculating the sum of two array indexes to find the midpoint between\nthem can overflow, i.e. code like this is unsafe for big arrays:\n\n\tmid = (first + last) >> 1;\n\nMake sure the intermediate value stays within the boundaries instead,\nlike this:\n\n\tmid = first + ((last - first) >> 1);\n\nThe loop condition of the binary search makes sure that 'last' is\nalways greater than 'first', so this is safe as long as 'first' is\nnot negative.  And that can be verified easily using the pre-context\nof each change, except for name-hash.c, so add an assertion to that\neffect there.\n\nThe unsafe calculations were found with:\n\n\tgit grep '(.*+.*) *>> *1'\n\nThis is a continuation of 19716b21a4 (cleanup: fix possible overflow\nerrors in binary search, 2017-10-08).\n\nSigned-off-by: Rene Scharfe <l.s.r@web.de>\n---\n builtin/ls-files.c  | 2 +-\n diffcore-rename.c   | 4 ++--\n dir.c               | 2 +-\n name-hash.c         | 3 ++-\n read-cache.c        | 2 +-\n sh-i18n--envsubst.c | 2 +-\n 6 files changed, 8 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex 7f83c9a6f2..670e8fb93c 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -373,7 +373,7 @@ static void prune_index(struct index_state *istate,\n \tfirst = pos;\n \tlast = istate->cache_nr;\n \twhile (last > first) {\n-\t\tint next = (last + first) >> 1;\n+\t\tint next = first + ((last - first) >> 1);\n \t\tconst struct cache_entry *ce = istate->cache[next];\n \t\tif (!strncmp(ce->name, prefix, prefixlen)) {\n \t\t\tfirst = next+1;\ndiff --git a/diffcore-rename.c b/diffcore-rename.c\nindex 07bd34b631..6af92d5eba 100644\n--- a/diffcore-rename.c\n+++ b/diffcore-rename.c\n@@ -23,7 +23,7 @@ static int find_rename_dst(struct diff_filespec *two)\n \tfirst = 0;\n \tlast = rename_dst_nr;\n \twhile (last > first) {\n-\t\tint next = (last + first) >> 1;\n+\t\tint next = first + ((last - first) >> 1);\n \t\tstruct diff_rename_dst *dst = &(rename_dst[next]);\n \t\tint cmp = strcmp(two->path, dst->two->path);\n \t\tif (!cmp)\n@@ -83,7 +83,7 @@ static struct diff_rename_src *register_rename_src(struct diff_filepair *p)\n \tfirst = 0;\n \tlast = rename_src_nr;\n \twhile (last > first) {\n-\t\tint next = (last + first) >> 1;\n+\t\tint next = first + ((last - first) >> 1);\n \t\tstruct diff_rename_src *src = &(rename_src[next]);\n \t\tint cmp = strcmp(one->path, src->p->one->path);\n \t\tif (!cmp)\ndiff --git a/dir.c b/dir.c\nindex ba4a51c296..d021c908e5 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -701,7 +701,7 @@ static struct untracked_cache_dir *lookup_untracked(struct untracked_cache *uc,\n \tfirst = 0;\n \tlast = dir->dirs_nr;\n \twhile (last > first) {\n-\t\tint cmp, next = (last + first) >> 1;\n+\t\tint cmp, next = first + ((last - first) >> 1);\n \t\td = dir->dirs[next];\n \t\tcmp = strncmp(name, d->name, len);\n \t\tif (!cmp && strlen(d->name) > len)\ndiff --git a/name-hash.c b/name-hash.c\nindex b4861bc7b0..695908609f 100644\n--- a/name-hash.c\n+++ b/name-hash.c\n@@ -345,8 +345,9 @@ static int handle_range_dir(\n \telse {\n \t\tint begin = k_start;\n \t\tint end = k_end;\n+\t\tassert(begin >= 0);\n \t\twhile (begin < end) {\n-\t\t\tint mid = (begin + end) >> 1;\n+\t\t\tint mid = begin + ((end - begin) >> 1);\n \t\t\tint cmp = strncmp(istate->cache[mid]->name, prefix->buf, prefix->len);\n \t\t\tif (cmp == 0) /* mid has same prefix; look in second part */\n \t\t\t\tbegin = mid + 1;\ndiff --git a/read-cache.c b/read-cache.c\nindex 22e7b9944e..4f81fca320 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -549,7 +549,7 @@ static int index_name_stage_pos(const struct index_state *istate, const char *na\n \tfirst = 0;\n \tlast = istate->cache_nr;\n \twhile (last > first) {\n-\t\tint next = (last + first) >> 1;\n+\t\tint next = first + ((last - first) >> 1);\n \t\tstruct cache_entry *ce = istate->cache[next];\n \t\tint cmp = cache_name_stage_compare(name, namelen, stage, ce->name, ce_namelen(ce), ce_stage(ce));\n \t\tif (!cmp)\ndiff --git a/sh-i18n--envsubst.c b/sh-i18n--envsubst.c\nindex cecfdd36c7..e7430b9aa8 100644\n--- a/sh-i18n--envsubst.c\n+++ b/sh-i18n--envsubst.c\n@@ -249,7 +249,7 @@ sorted_string_list_member (const string_list_ty *slp, const char *s)\n \t{\n \t  /* Here we know that if s is in the list, it is at an index j\n \t     with j1 <= j < j2.  */\n-\t  size_t j = (j1 + j2) >> 1;\n+\t  size_t j = j1 + ((j2 - j1) >> 1);\n \t  int result = strcmp (slp->item[j], s);\n\n \t  if (result > 0)\n--\n2.22.0\n"},{"id":"377167","messageId":"ac09e504-369d-d6d5-a356-07a75386d3fb@gmail.com","threadId":"51306","inReplyTo":"f0059613-d3e4-a1e2-2c01-727579e069d2@web.de","subject":"Re: [PATCH] cleanup: fix possible overflow errors in binary search, part 2","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-06-13T18:06:00Z","receivedAt":"2019-06-13T18:06:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/13/2019 1:51 PM, René Scharfe wrote:\n> Calculating the sum of two array indexes to find the midpoint between\n> them can overflow, i.e. code like this is unsafe for big arrays:\n> \n> \tmid = (first + last) >> 1;\n> \n> Make sure the intermediate value stays within the boundaries instead,\n> like this:\n> \n> \tmid = first + ((last - first) >> 1);\n> \n> The loop condition of the binary search makes sure that 'last' is\n> always greater than 'first', so this is safe as long as 'first' is\n> not negative.  And that can be verified easily using the pre-context\n> of each change, except for name-hash.c, so add an assertion to that\n> effect there.\n> \n> The unsafe calculations were found with:\n> \n> \tgit grep '(.*+.*) *>> *1'\n> \n> This is a continuation of 19716b21a4 (cleanup: fix possible overflow\n> errors in binary search, 2017-10-08).\n\nThank you for finding these additional examples!\n\nLGTM. Pretty easy to see this is correct.\n\n-Stolee\n\n"},{"id":"377183","messageId":"CAN0heSriaQV7zd4h8gfazi0dY_Dc5qLBWfJBtZGxOtm1+cgZqw@mail.gmail.com","threadId":"51306","inReplyTo":"f0059613-d3e4-a1e2-2c01-727579e069d2@web.de","subject":"Re: [PATCH] cleanup: fix possible overflow errors in binary search, part 2","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2019-06-13T19:42:53Z","receivedAt":"2019-06-13T19:43:06Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Thu, 13 Jun 2019 at 19:54, René Scharfe <l.s.r@web.de> wrote:\n>\n> Calculating the sum of two array indexes to find the midpoint between\n> them can overflow, i.e. code like this is unsafe for big arrays:\n>\n>         mid = (first + last) >> 1;\n>\n> Make sure the intermediate value stays within the boundaries instead,\n> like this:\n>\n>         mid = first + ((last - first) >> 1);\n>\n> The loop condition of the binary search makes sure that 'last' is\n> always greater than 'first', so this is safe as long as 'first' is\n> not negative.  And that can be verified easily using the pre-context\n> of each change, except for name-hash.c, so add an assertion to that\n> effect there.\n\nRight, with \"safe\", one might mean something like \"no undefined behavior\ndue to shifting a signed value with the high bit set\". Especially since\nwe're worrying about overflows, we're obviously having large values in\nmind, so we're right to consider the sign bit. But, we're fine as you\nnote.  Because we subtract, and `last` doesn't have its sign bit set,\nand `first` is non-negative and not greater than `last`, the sign bit of\n`(last - first)` is always zero.\n\nSo all is well. But maybe we should write `(last - first) / 2` anyway.\nWe could then drop the extra parenthesis, and we would keep future\nreaders (and static analysis?) from wondering whether we might ever be\nshifting a signed value with the sign bit set. A few spots fewer to\naudit in the future...\n\nMartin\n"},{"id":"377189","messageId":"1473675e-4d27-8221-0e7a-f88eb31ee11f@web.de","threadId":"51306","inReplyTo":"CAN0heSriaQV7zd4h8gfazi0dY_Dc5qLBWfJBtZGxOtm1+cgZqw@mail.gmail.com","subject":"Re: [PATCH] cleanup: fix possible overflow errors in binary search, part 2","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2019-06-13T21:33:00Z","receivedAt":"2019-06-13T21:33:10Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 13.06.19 um 21:42 schrieb Martin Ågren:\n> On Thu, 13 Jun 2019 at 19:54, René Scharfe <l.s.r@web.de> wrote:\n>>\n>> Calculating the sum of two array indexes to find the midpoint between\n>> them can overflow, i.e. code like this is unsafe for big arrays:\n>>\n>>         mid = (first + last) >> 1;\n>>\n>> Make sure the intermediate value stays within the boundaries instead,\n>> like this:\n>>\n>>         mid = first + ((last - first) >> 1);\n>>\n>> The loop condition of the binary search makes sure that 'last' is\n>> always greater than 'first', so this is safe as long as 'first' is\n>> not negative.  And that can be verified easily using the pre-context\n>> of each change, except for name-hash.c, so add an assertion to that\n>> effect there.\n>\n> Right, with \"safe\", one might mean something like \"no undefined behavior\n> due to shifting a signed value with the high bit set\". Especially since\n> we're worrying about overflows, we're obviously having large values in\n> mind, so we're right to consider the sign bit. But, we're fine as you\n> note.  Because we subtract, and `last` doesn't have its sign bit set,\n> and `first` is non-negative and not greater than `last`, the sign bit of\n> `(last - first)` is always zero.\n>\n> So all is well. But maybe we should write `(last - first) / 2` anyway.\n> We could then drop the extra parenthesis, and we would keep future\n> readers (and static analysis?) from wondering whether we might ever be\n> shifting a signed value with the sign bit set. A few spots fewer to\n> audit in the future...\n\nYes, thought about that.  When I saw Clang 8 emitting extra opcodes for\nhandling the sign for the version with division I decided to restrict\nthe patch to just do overflow prevention and leave the right shifts in\nplace..\n\nRené\n"},{"id":"377213","messageId":"CAN0heSrEK80XHFYYuXhb4rg76Xv6n7-p-oUdyLH39vpAfjkxZg@mail.gmail.com","threadId":"51306","inReplyTo":"1473675e-4d27-8221-0e7a-f88eb31ee11f@web.de","subject":"Re: [PATCH] cleanup: fix possible overflow errors in binary search, part 2","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2019-06-14T04:18:55Z","receivedAt":"2019-06-14T04:19:08Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Thu, 13 Jun 2019 at 23:33, René Scharfe <l.s.r@web.de> wrote:\n>\n> Am 13.06.19 um 21:42 schrieb Martin Ågren:\n> > On Thu, 13 Jun 2019 at 19:54, René Scharfe <l.s.r@web.de> wrote:\n\n> >> Make sure the intermediate value stays within the boundaries instead,\n> >> like this:\n> >>\n> >>         mid = first + ((last - first) >> 1);\n> >>\n> >> The loop condition of the binary search makes sure that 'last' is\n> >> always greater than 'first', so this is safe as long as 'first' is\n> >> not negative.  And that can be verified easily using the pre-context\n> >> of each change, except for name-hash.c, so add an assertion to that\n> >> effect there.\n\n> > So all is well. But maybe we should write `(last - first) / 2` anyway.\n> > We could then drop the extra parenthesis, and we would keep future\n> > readers (and static analysis?) from wondering whether we might ever be\n> > shifting a signed value with the sign bit set. A few spots fewer to\n> > audit in the future...\n>\n> Yes, thought about that.  When I saw Clang 8 emitting extra opcodes for\n> handling the sign for the version with division I decided to restrict\n> the patch to just do overflow prevention and leave the right shifts in\n> place..\n\nAh, ok, I should have known you were on top of it. I guess that means\nthat clang isn't able to figure out we're guaranteed to be working with\nnon-negative numbers. Which I guess means that it can't be sure the\nshift is safe, but with undefined behavior it has the leeway it needs,\nso...\n\nMartin\n"}]}