{"thread":{"id":"34309","subject":"[PATCH 2/2] commit: reject overlong UTF-8 sequences","startedAt":"2013-06-29T17:41:41Z","lastAt":"2013-07-01T07:14:02Z","messageCount":2,"participants":["brian m. carlson","Peter Krefting"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"222232","messageId":"20130629174140.GC226907@vauxhall.crustytoothpaste.net","threadId":"34309","inReplyTo":null,"subject":"[PATCH 2/2] commit: reject overlong UTF-8 sequences","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2013-06-29T17:41:41Z","receivedAt":"2013-06-29T17:41:41Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"The commit code accepts pseudo-UTF-8 sequences that encode a character with more\nbytes than necessary.  Reject such sequences, since they are not valid UTF-8.\n\nSigned-off-by: brian m. carlson <sandals@crustytoothpaste.net>\n---\n commit.c                   | 14 ++++++++++++--\n t/t3900-i18n-commit.sh     |  8 ++++++++\n t/t3900/UTF-8-overlong.txt |  3 +++\n 3 files changed, 23 insertions(+), 2 deletions(-)\n create mode 100644 t/t3900/UTF-8-overlong.txt\n\ndiff --git a/commit.c b/commit.c\nindex 2280413..e57fec6 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -1240,11 +1240,15 @@ int commit_tree(const struct strbuf *msg, unsigned char *tree,\n static int find_invalid_utf8(const char *buf, int len)\n {\n \tint offset = 0;\n+\tstatic const unsigned int max_codepoint[] = {\n+\t\t0x7f, 0x7ff, 0xffff, 0x1fffff\n+\t};\n \n \twhile (len) {\n \t\tunsigned char c = *buf++;\n \t\tint bytes, bad_offset;\n \t\tunsigned int codepoint;\n+\t\tunsigned int min_val, max_val;\n \n \t\tlen--;\n \t\toffset++;\n@@ -1273,8 +1277,12 @@ static int find_invalid_utf8(const char *buf, int len)\n \t\tif (len < bytes)\n \t\t\treturn bad_offset;\n \n-\t\t/* Place the encoded bits at the bottom of the value. */\n+\t\t/* Place the encoded bits at the bottom of the value and compute the\n+\t\t * valid range.\n+\t\t */\n \t\tcodepoint = (c & 0x7f) >> bytes;\n+\t\tmin_val = max_codepoint[bytes-1] + 1;\n+\t\tmax_val = max_codepoint[bytes];\n \n \t\toffset += bytes;\n \t\tlen -= bytes;\n@@ -1287,13 +1295,15 @@ static int find_invalid_utf8(const char *buf, int len)\n \t\t\t\treturn bad_offset;\n \t\t} while (--bytes);\n \n-\t\t/* Check the value here */\n+\t\t/* Check the value and length here */\n \t\tif (codepoint >= 0xd800 && codepoint <= 0xdfff)\n \t\t\treturn bad_offset;\n \t\tif (codepoint > 0x10ffff)\n \t\t\treturn bad_offset;\n \t\tif ((codepoint & 0x1ffffe) == 0xfffe)\n \t\t\treturn bad_offset;\n+\t\tif (codepoint < min_val || codepoint > max_val)\n+\t\t\treturn bad_offset;\n \t}\n \treturn -1;\n }\ndiff --git a/t/t3900-i18n-commit.sh b/t/t3900-i18n-commit.sh\nindex 16ed707..de0fcef 100755\n--- a/t/t3900-i18n-commit.sh\n+++ b/t/t3900-i18n-commit.sh\n@@ -47,6 +47,14 @@ test_expect_success 'UTF-8 invalid characters refused' '\n \tgrep \"did not conform\" \"$HOME\"/stderr\n '\n \n+test_expect_success 'UTF-8 overlong sequences rejected' '\n+\trm -f \"$HOME/stderr\" &&\n+\techo \"UTF-8 overlong\" >F &&\n+\tgit commit -a -F \"$TEST_DIRECTORY\"/t3900/UTF-8-overlong.txt \\\n+\t\t2>\"$HOME\"/stderr &&\n+\tgrep \"did not conform\" \"$HOME\"/stderr\n+'\n+\n rm -f \"$HOME/stderr\"\n \n for H in ISO8859-1 eucJP ISO-2022-JP\ndiff --git a/t/t3900/UTF-8-overlong.txt b/t/t3900/UTF-8-overlong.txt\nnew file mode 100644\nindex 0000000..fa157e9\n--- /dev/null\n+++ b/t/t3900/UTF-8-overlong.txt\n@@ -0,0 +1,3 @@\n+???ommit message\n+\n+This is not a space:??\n-- \n1.8.3.1\n"},{"id":"222270","messageId":"alpine.DEB.2.00.1307010812470.21709@ds9.cixit.se","threadId":"34309","inReplyTo":"20130629174140.GC226907@vauxhall.crustytoothpaste.net","subject":"Re: [PATCH 2/2] commit: reject overlong UTF-8 sequences","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2013-07-01T07:14:02Z","receivedAt":"2013-07-01T07:14:02Z","isPatch":true,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"brian m. carlson:\n\n> \tint offset = 0;\n> +\tstatic const unsigned int max_codepoint[] = {\n> +\t\t0x7f, 0x7ff, 0xffff, 0x1fffff\n> +\t};\n\nSince Unicode is not defined beyond U+10FFFF, you can easily make the \nlast range end at U+10FFF. Doing that, ...\n\n> \t\tif (codepoint > 0x10ffff)\n> \t\t\treturn bad_offset;\n\n... you can also drop this test, as it will have been tested by the \nmax test already.\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"}]}