{"thread":{"id":"47511","subject":"[PATCH v2 0/5] convert: add support for different encodings","startedAt":"2017-12-29T15:22:51Z","lastAt":"2018-01-06T00:55:24Z","messageCount":26,"participants":["lars.schneider@autodesk.com","Torsten Bögershausen","Lars Schneider","tboegi@web.de","Johannes Schindelin","Junio C Hamano"],"isPatch":true,"patchVersion":2,"patchTotal":5},"messages":[{"id":"335532","messageId":"20171229152222.39680-1-lars.schneider@autodesk.com","threadId":"47511","inReplyTo":null,"subject":"[PATCH v2 0/5] convert: add support for different encodings","fromName":"","fromEmail":"lars.schneider@autodesk.com","sentAt":"2017-12-29T15:22:17Z","receivedAt":"2017-12-29T15:22:51Z","isPatch":true,"sender":{"key":"lars.schneider@autodesk.com","avatar":"https://gravatar.com/avatar/524188df3a38b02692619777bb90d9cc735db1ad75f5304e0bf09f8efd299775?d=mp&s=160"},"body":"From: Lars Schneider \n\nHi,\n\nnotable changes since v1:\n\n* In [1] Peff described a situation where you \"couldn't checkout _away_\n from\" problems because of \"die() in convert_to_git()\". I fixed this and\n tried to replicate the situation with the test \"error if encoding\n garbage is already in Git\". To achieve that I had to pass the\n 'write_obj' variable through all the functions to let the encoding\n code know if Git actually writes content. If Git actually writes\n content then we die. Otherwise we just print an error. This ensures\n no garbage is added to Git.\n\n* I renamed the attribute to \"checkout-encoding\" to avoid trouble with\n the existing \"encoding\" attribute. I also considered\n \"working-tree-encoding\". We also discussed \"worktree-encoding\" as\n attribute name but I am was worried that people could (wrongly)\n associated that with the 'git worktree' feature. I am happy to hear\n arguments for/against any of the different options.\n\n* I read a bit on the Internet and as far as I understand it to/from\n UTF-8 reencodings are mostly round trip safe. Notable exceptions\n are some characters in the SHIFT-JIS character set:\n https://support.microsoft.com/en-us/help/170559/prb-conversion-problem-between-shift-jis-and-unicode\n\n However, I was not able to replicate the issue. Consider this:\n X=\"\\x87\\x90\"\n printf \"$X\" | od -h\n printf \"$X\" | iconv -f SHIFT-JIS -t UTF-8 | iconv -f UTF-8 -t SHIFT-JIS | od -h\n\n ... the reencoding from SHIFT-JIS to UTF-8 would always fail.\n Therefore, I was not able to generate a test case that produces a\n different result after the round trip. Anyone an idea how to do that?\n\n* Patch 1, 2, 3, and 5 are helper functions. Patch 4 is the real change.\n\nThanks,\nLars\n\n RFC: ttps://public-inbox.org/git/BDB9B884-6D17-4BE3-A83C-F67E2AFA2B46@gmail.comh\n v1: https://public-inbox.org/git/20171211155023.1405-1-lars.schneider@autodesk.com/\n\nBase Ref: master\nWeb-Diff: https://github.com/larsxschneider/git/commit/e11e375fdb\nCheckout: git fetch https://github.com/larsxschneider/git encoding-v2 && git checkout e11e375fdb\n\n\n### Interdiff (v1..v2):\n\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex 84de2fa49c..0039bd38c3 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -146,33 +146,6 @@ Unspecified::\n Any other value causes Git to act as if `text` has been left\n unspecified.\n\n-`encoding`\n-^^^^^^^^^^\n-\n-By default Git assumes UTF-8 encoding for text files. The `encoding`\n-attribute sets the encoding to be used in the working directory.\n-If the path is added to the index, then Git encodes the content to\n-UTF-8. On checkout the content is encoded back to the original\n-encoding. Consequently, you can use all built-in Git text processing\n-tools (e.g. git diff, line ending conversions, etc.) with your\n-non-UTF-8 encoded file.\n-\n-Please note that re-encoding content can cause errors and requires\n-resources. Use the `encoding` attribute only if you cannot store\n-a file in UTF-8 encoding and if you want Git to be able to process\n-the text.\n-\n-------------------------\n-*.txt\t\ttext encoding=UTF-16\n-------------------------\n-\n-All `iconv` encodings with a stable round-trip conversion to and from\n-UTF-8 are supported. You can see a full list with the following command:\n-\n-------------------------\n-iconv --list\n-------------------------\n-\n `eol`\n ^^^^^\n\n@@ -299,6 +272,65 @@ few exceptions. Even though...\n catch potential problems early, safety triggers.\n\n\n+`checkout-encoding`\n+^^^^^^^^^^^^^^^^^^^\n+\n+Git recognizes files encoded with ASCII or one of its supersets (e.g.\n+UTF-8 or ISO-8859-1) as text files. All other encodings are usually\n+interpreted as binary and consequently built-in Git text processing\n+tools (e.g. 'git diff') as well as most Git web front ends do not\n+visualize the content.\n+\n+In these cases you can teach Git the encoding of a file in the working\n+directory with the `checkout-encoding` attribute. If a file with this\n+attributes is added to Git, then Git reencodes the content from the\n+specified encoding to UTF-8 and stores the result in its internal data\n+structure. On checkout the content is encoded back to the specified\n+encoding.\n+\n+Please note that using the `checkout-encoding` attribute has a number\n+of drawbacks:\n+\n+- Reencoding content to non-UTF encodings (e.g. SHIFT-JIS) can cause\n+ errors as the conversion might not be round trip safe.\n+\n+- Reencoding content requires resources that might slow down certain\n+ Git operations (e.g 'git checkout' or 'git add').\n+\n+- Git clients that do not support the `checkout-encoding` attribute or\n+ the used encoding will checkout the respective files as UTF-8 encoded.\n+ That means the content appears to be different which could cause\n+ trouble. Affected clients are older Git versions and alternative Git\n+ implementations such as JGit or libgit2 (as of January 2018).\n+\n+Use the `checkout-encoding` attribute only if you cannot store a file in\n+UTF-8 encoding and if you want Git to be able to process the content as\n+text.\n+\n+Use the following attributes if your '*.txt' files are UTF-16 encoded\n+with byte order mark (BOM) and you want Git to perform automatic line\n+ending conversion based on your platform.\n+\n+------------------------\n+*.txt\t\ttext checkout-encoding=UTF-16\n+------------------------\n+\n+Use the following attributes if your '*.txt' files are UTF-16 little\n+endian encoded without BOM and you want Git to use Windows line endings\n+in the working directory.\n+\n+------------------------\n+*.txt \t\tcheckout-encoding=UTF-16LE text eol=CRLF\n+------------------------\n+\n+You can get a list of all available encodings on your platform with the\n+following command:\n+\n+------------------------\n+iconv --list\n+------------------------\n+\n+\n `ident`\n ^^^^^^^\n\ndiff --git a/apply.c b/apply.c\nindex 321a9fa68d..c4bd5cf1f2 100644\n--- a/apply.c\n+++ b/apply.c\n@@ -2281,7 +2281,7 @@ static int read_old_data(struct stat *st, struct patch *patch,\n \t\t * should never look at the index when explicit crlf option\n \t\t * is given.\n \t\t */\n-\t\tconvert_to_git(NULL, path, buf->buf, buf->len, buf, safe_crlf);\n+\t\tconvert_to_git(NULL, path, buf->buf, buf->len, buf, safe_crlf, 0);\n \t\treturn 0;\n \tdefault:\n \t\treturn -1;\ndiff --git a/blame.c b/blame.c\nindex 2893f3c103..388b66897b 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -229,7 +229,7 @@ static struct commit *fake_working_tree_commit(struct diff_options *opt,\n \t\tif (strbuf_read(&buf, 0, 0) < 0)\n \t\t\tdie_errno(\"failed to read from stdin\");\n \t}\n-\tconvert_to_git(&the_index, path, buf.buf, buf.len, &buf, 0);\n+\tconvert_to_git(&the_index, path, buf.buf, buf.len, &buf, 0, 0);\n \torigin->file.ptr = buf.buf;\n \torigin->file.size = buf.len;\n \tpretend_sha1_file(buf.buf, buf.len, OBJ_BLOB, origin->blob_oid.hash);\ndiff --git a/combine-diff.c b/combine-diff.c\nindex 2505de119a..4555e49b5f 100644\n--- a/combine-diff.c\n+++ b/combine-diff.c\n@@ -1053,7 +1053,7 @@ static void show_patch_diff(struct combine_diff_path *elem, int num_parent,\n \t\t\tif (is_file) {\n \t\t\t\tstruct strbuf buf = STRBUF_INIT;\n\n-\t\t\t\tif (convert_to_git(&the_index, elem->path, result, len, &buf, safe_crlf)) {\n+\t\t\t\tif (convert_to_git(&the_index, elem->path, result, len, &buf, safe_crlf, 0)) {\n \t\t\t\t\tfree(result);\n \t\t\t\t\tresult = strbuf_detach(&buf, &len);\n \t\t\t\t\tresult_size = len;\ndiff --git a/convert.c b/convert.c\nindex ee19c17104..ca7b2f3e5c 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -257,28 +257,40 @@ static int will_convert_lf_to_crlf(size_t len, struct text_stat *stats,\n\n }\n\n-#ifdef NO_ICONV\n-#ifndef _ICONV_T\n-/* The type is just a placeholder and not actually used. */\n-typedef void* iconv_t;\n-#endif\n-#endif\n+static void trace_encoding(const char *context, const char *path,\n+\t\t\t const char *encoding, const char *buf, size_t len)\n+{\n+\tstatic struct trace_key coe = TRACE_KEY_INIT(CHECKOUT_ENCODING);\n+\tstruct strbuf trace = STRBUF_INIT;\n+\tint i;\n+\n+\tstrbuf_addf(&trace, \"%s (%s, considered %s):\\n\", context, path, encoding);\n+\tfor (i = 0; i < len && buf; ++i) {\n+\t\tstrbuf_addf(\n+\t\t\t&trace,\"| \\e[2m%2i:\\e[0m %2x \\e[2m%c\\e[0m%c\",\n+\t\t\ti,\n+\t\t\t(unsigned char) buf[i],\n+\t\t\t(buf[i] > 32 && buf[i] < 127 ? buf[i] : ' '),\n+\t\t\t((i+1) % 8 && (i+1) < len ? ' ' : '\\n')\n+\t\t);\n+\t}\n+\tstrbuf_addchars(&trace, '\\n', 1);\n+\n+\ttrace_strbuf(&coe, &trace);\n+\tstrbuf_release(&trace);\n+}\n\n static struct encoding {\n \tconst char *name;\n-\ticonv_t to_git; /* conversion to Git's canonical encoding (UTF-8) */\n-\ticonv_t to_worktree; /* conversion to user-defined encoding */\n \tstruct encoding *next;\n } *encoding, **encoding_tail;\n-static const char *default_encoding = \"utf-8\";\n-static iconv_t invalid_conversion = (iconv_t)-1;\n+static const char *default_encoding = \"UTF-8\";\n\n static int encode_to_git(const char *path, const char *src, size_t src_len,\n-\t\t\t struct strbuf *buf, struct encoding *enc)\n+\t\t\t struct strbuf *buf, struct encoding *enc, int write_obj)\n {\n-#ifndef NO_ICONV\n-\tchar *dst, *re_src;\n-\tint dst_len, re_src_len;\n+\tchar *dst;\n+\tint dst_len;\n\n \t/*\n \t * No encoding is specified or there is nothing to encode.\n@@ -296,70 +308,102 @@ static int encode_to_git(const char *path, const char *src, size_t src_len,\n \tif (!buf && !src)\n \t\treturn 1;\n\n-\tif (enc->to_git == invalid_conversion) {\n-\t\tenc->to_git = iconv_open(default_encoding, encoding->name);\n-\t\tif (enc->to_git == invalid_conversion)\n-\t\t\twarning(_(\"unsupported encoding %s\"), encoding->name);\n+\tif (has_prohibited_utf_bom(enc->name, src, src_len)) {\n+\t\tconst char *error_msg = _(\n+\t\t\t\"BOM is prohibited for '%s' if encoded as %s\");\n+\t\tconst char *advise_msg = _(\n+\t\t\t\"You told Git to treat '%s' as %s. A byte order mark \"\n+\t\t\t\"(BOM) is prohibited with this encoding. Either use \"\n+\t\t\t\"%.6s as checkout encoding or remove the BOM from the \"\n+\t\t\t\"file.\");\n+\n+\t\tadvise(advise_msg, path, enc->name, enc->name, enc->name);\n+\t\tif (write_obj)\n+\t\t\tdie(error_msg, path, enc->name);\n+\t\telse\n+\t\t\terror(error_msg, path, enc->name);\n+\n+\n+\t} else if (has_missing_utf_bom(enc->name, src, src_len)) {\n+\t\tconst char *error_msg = _(\n+\t\t\t\"BOM is required for '%s' if encoded as %s\");\n+\t\tconst char *advise_msg = _(\n+\t\t\t\"You told Git to treat '%s' as %s. A byte order mark \"\n+\t\t\t\"(BOM) is required with this encoding. Either use \"\n+\t\t\t\"%sBE/%sLE as checkout encoding or add a BOM to the \"\n+\t\t\t\"file.\");\n+\t\tadvise(advise_msg, path, enc->name, enc->name, enc->name);\n+\t\tif (write_obj)\n+\t\t\tdie(error_msg, path, enc->name);\n+\t\telse\n+\t\t\terror(error_msg, path, enc->name);\n \t}\n\n-\tif (enc->to_worktree == invalid_conversion)\n-\t\tenc->to_worktree = iconv_open(encoding->name, default_encoding);\n-\n-\t/*\n-\t * Do nothing if the encoding is not supported. This could happen in\n-\t * two cases:\n-\t * (1) The encoding is garbage. That is no problem as we would not\n-\t * encode the content to UTF-8 on \"add\" and we would not encode\n-\t * it back on \"checkout\".\n-\t * (2) Git users use different iconv versions that support different\n-\t * encodings. This could lead to problems, as the content might\n-\t * not be encoded on \"add\" but encoded back on \"checkout\" (or\n-\t * the other way around).\n-\t * We print a one-time warning to the user in both cases above.\n-\t */\n-\tif (enc->to_git == invalid_conversion || enc->to_worktree == invalid_conversion)\n-\t\treturn 0;\n-\n-\tdst = reencode_string_iconv(src, src_len, enc->to_git, &dst_len);\n-\tif (!dst)\n+\ttrace_encoding(\"source\", path, enc->name, src, src_len);\n+\tdst = reencode_string_len(src, src_len, default_encoding, enc->name,\n+\t\t\t\t &dst_len);\n+\tif (!dst) {\n \t\t/*\n \t\t * We could add the blob \"as-is\" to Git. However, on checkout\n \t\t * we would try to reencode to the original encoding. This\n \t\t * would fail and we would leave the user with a messed-up\n \t\t * working tree. Let's try to avoid this by screaming loud.\n \t\t */\n-\t\tdie(_(\"failed to encode '%s' from %s to %s\"),\n-\t\t\tpath, enc->name, default_encoding);\n+\t\tconst char* msg = _(\"failed to encode '%s' from %s to %s\");\n+\t\tif (write_obj)\n+\t\t\tdie(msg, path, enc->name, default_encoding);\n+\t\telse\n+\t\t\terror(msg, path, enc->name, default_encoding);\n+\t}\n+\ttrace_encoding(\"destination\", path, default_encoding, dst, dst_len);\n\n \t/*\n-\t * Encode dst back to ensure no information is lost. This wastes\n-\t * a few cycles as most conversions are round trip conversion\n-\t * safe. However, content that has an invalid encoding might not\n-\t * match its original byte sequence after the UTF-8 conversion\n-\t * round trip. Let's play safe here and check the round trip\n-\t * conversion.\n+\t * UTF supports lossless round tripping [1]. UTF to other encoding are\n+\t * mostly round trip safe as Unicode aims to be a superset of all other\n+\t * character encodings. However, the SHIFT-JIS (Japanese character set)\n+\t * is an exception as some codes are not round trip safe [2].\n+\t *\n+\t * Reverse the transformation of 'dst' and check the result with 'src'\n+\t * if content is written to Git. This ensures no information is lost\n+\t * during conversion to/from UTF-8.\n+\t *\n+\t * Please note, the code below is not tested because I was not able to\n+\t * generate a faulty round trip without iconv error.\n+\t *\n+\t * [1] http://unicode.org/faq/utf_bom.html#gen2\n+\t * [2] https://support.microsoft.com/en-us/help/170559/prb-conversion-problem-between-shift-jis-and-unicode\n \t */\n-\tre_src = reencode_string_iconv(dst, dst_len, enc->to_worktree, &re_src_len);\n-\tif (!re_src || strcmp(src, re_src)) {\n-\t\tdie(_(\"encoding '%s' from %s to %s and back is not the same\"),\n-\t\t\tpath, enc->name, default_encoding);\n+\tif (write_obj && !strcmp(enc->name, \"SHIFT-JIS\")) {\n+\t\tchar *re_src;\n+\t\tint re_src_len;\n+\n+\t\tre_src = reencode_string_len(dst, dst_len,\n+\t\t\t\t\t enc->name, default_encoding,\n+\t\t\t\t\t &re_src_len);\n+\n+\t\ttrace_encoding(\"reencoded source\", path, enc->name,\n+\t\t\t re_src, re_src_len);\n+\n+\t\tif (!re_src || src_len != re_src_len ||\n+\t\t memcmp(src, re_src, src_len)) {\n+\t\t\tconst char* msg = _(\"encoding '%s' from %s to %s and \"\n+\t\t\t\t\t \"back is not the same\");\n+\t\t\tif (write_obj)\n+\t\t\t\tdie(msg, path, enc->name, default_encoding);\n+\t\t\telse\n+\t\t\t\terror(msg, path, enc->name, default_encoding);\n \t\t}\n+\n \t\tfree(re_src);\n+\t}\n\n \tstrbuf_attach(buf, dst, dst_len, dst_len + 1);\n \treturn 1;\n-#else\n-\twarning(_(\"cannot encode '%s' from %s to %s because \"\n-\t\t\"your Git was not compiled with encoding support\"),\n-\t\tpath, enc->name, default_encoding);\n-\treturn 0;\n-#endif\n }\n\n static int encode_to_worktree(const char *path, const char *src, size_t src_len,\n \t\t\t struct strbuf *buf, struct encoding *enc)\n {\n-#ifndef NO_ICONV\n \tchar *dst;\n \tint dst_len;\n\n@@ -370,34 +414,16 @@ static int encode_to_worktree(const char *path, const char *src, size_t src_len,\n \tif (!enc || (src && !src_len))\n \t\treturn 0;\n\n-\tif (enc->to_worktree == invalid_conversion) {\n-\t\tenc->to_worktree = iconv_open(encoding->name, default_encoding);\n-\t\tif (enc->to_worktree == invalid_conversion)\n-\t\t\twarning(\"unsupported encoding %s\", encoding->name);\n-\t}\n-\n-\t/*\n-\t * Do nothing if the encoding is not supported.\n-\t * See detailed explanation in encode_to_git().\n-\t */\n-\tif (enc->to_worktree == invalid_conversion)\n-\t\treturn 0;\n-\n-\tdst = reencode_string_iconv(src, src_len, enc->to_worktree, &dst_len);\n+\tdst = reencode_string_len(src, src_len, enc->name, default_encoding,\n+\t\t\t\t &dst_len);\n \tif (!dst) {\n-\t\twarning(\"failed to encode '%s' from %s to %s\",\n-\t\t\tpath, default_encoding, encoding->name);\n+\t\terror(\"failed to encode '%s' from %s to %s\",\n+\t\t\tpath, enc->name, default_encoding);\n \t\treturn 0;\n \t}\n\n \tstrbuf_attach(buf, dst, dst_len, dst_len + 1);\n \treturn 1;\n-#else\n-\twarning(_(\"cannot encode '%s' from %s to %s because \"\n-\t\t\"your Git was not compiled with encoding support\"),\n-\t\tpath, default_encoding, encoding->name);\n-\treturn 0;\n-#endif\n }\n\n static int crlf_to_git(const struct index_state *istate,\n@@ -1118,11 +1144,12 @@ static struct encoding *git_path_check_encoding(struct attr_check_item *check)\n \tconst char *value = check->value;\n \tstruct encoding *enc;\n\n-\tif (ATTR_UNSET(value))\n+\tif (ATTR_TRUE(value) || ATTR_FALSE(value) || ATTR_UNSET(value) ||\n+\t !strlen(value))\n \t\treturn NULL;\n\n \tfor (enc = encoding; enc; enc = enc->next)\n-\t\tif (!strcmp(value, enc->name))\n+\t\tif (!strcasecmp(value, enc->name))\n \t\t\treturn enc;\n\n \t/* Don't encode to the default encoding */\n@@ -1130,9 +1157,7 @@ static struct encoding *git_path_check_encoding(struct attr_check_item *check)\n \t\treturn NULL;\n\n \tenc = xcalloc(1, sizeof(struct convert_driver));\n-\tenc->name = xstrdup(value);\n-\tenc->to_git = invalid_conversion;\n-\tenc->to_worktree = invalid_conversion;\n+\tenc->name = xstrdup_toupper(value); /* aways use upper case names! */\n \t*encoding_tail = enc;\n \tencoding_tail = &(enc->next);\n\n@@ -1194,7 +1219,7 @@ struct conv_attrs {\n \tenum crlf_action attr_action; /* What attr says */\n \tenum crlf_action crlf_action; /* When no attr is set, use core.autocrlf */\n \tint ident;\n-\tstruct encoding *encoding; /* Supported encoding or default encoding if NULL */\n+\tstruct encoding *checkout_encoding; /* Supported encoding or default encoding if NULL */\n };\n\n static void convert_attrs(struct conv_attrs *ca, const char *path)\n@@ -1203,7 +1228,8 @@ static void convert_attrs(struct conv_attrs *ca, const char *path)\n\n \tif (!check) {\n \t\tcheck = attr_check_initl(\"crlf\", \"ident\", \"filter\",\n-\t\t\t\t\t \"eol\", \"text\", \"encoding\", NULL);\n+\t\t\t\t\t \"eol\", \"text\", \"checkout-encoding\",\n+\t\t\t\t\t NULL);\n \t\tuser_convert_tail = &user_convert;\n \t\tencoding_tail = &encoding;\n \t\tgit_config(read_convert_config, NULL);\n@@ -1227,7 +1253,7 @@ static void convert_attrs(struct conv_attrs *ca, const char *path)\n \t\t\telse if (eol_attr == EOL_CRLF)\n \t\t\t\tca->crlf_action = CRLF_TEXT_CRLF;\n \t\t}\n-\t\tca->encoding = git_path_check_encoding(ccheck + 5);\n+\t\tca->checkout_encoding = git_path_check_encoding(ccheck + 5);\n \t} else {\n \t\tca->drv = NULL;\n \t\tca->crlf_action = CRLF_UNDEFINED;\n@@ -1293,7 +1319,7 @@ const char *get_convert_attr_ascii(const char *path)\n\n int convert_to_git(const struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n- struct strbuf *dst, enum safe_crlf checksafe)\n+\t\t struct strbuf *dst, enum safe_crlf checksafe, int write_obj)\n {\n \tint ret = 0;\n \tstruct conv_attrs ca;\n@@ -1309,7 +1335,7 @@ int convert_to_git(const struct index_state *istate,\n \t\tlen = dst->len;\n \t}\n\n-\tret |= encode_to_git(path, src, len, dst, ca.encoding);\n+\tret |= encode_to_git(path, src, len, dst, ca.checkout_encoding, write_obj);\n \tif (ret && dst) {\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\n@@ -1327,7 +1353,7 @@ int convert_to_git(const struct index_state *istate,\n\n void convert_to_git_filter_fd(const struct index_state *istate,\n \t\t\t const char *path, int fd, struct strbuf *dst,\n-\t\t\t enum safe_crlf checksafe)\n+\t\t\t enum safe_crlf checksafe, int write_obj)\n {\n \tstruct conv_attrs ca;\n \tconvert_attrs(&ca, path);\n@@ -1338,7 +1364,7 @@ void convert_to_git_filter_fd(const struct index_state *istate,\n \tif (!apply_filter(path, NULL, 0, fd, dst, ca.drv, CAP_CLEAN, NULL))\n \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n\n-\tencode_to_git(path, dst->buf, dst->len, dst, ca.encoding);\n+\tencode_to_git(path, dst->buf, dst->len, dst, ca.checkout_encoding, write_obj);\n \tcrlf_to_git(istate, path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);\n \tident_to_git(path, dst->buf, dst->len, dst, ca.ident);\n }\n@@ -1370,7 +1396,7 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n \t\t}\n \t}\n\n-\tret |= encode_to_worktree(path, src, len, dst, ca.encoding);\n+\tret |= encode_to_worktree(path, src, len, dst, ca.checkout_encoding);\n \tif (ret) {\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\n@@ -1404,7 +1430,7 @@ int renormalize_buffer(const struct index_state *istate, const char *path,\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\n \t}\n-\treturn ret | convert_to_git(istate, path, src, len, dst, SAFE_CRLF_RENORMALIZE);\n+\treturn ret | convert_to_git(istate, path, src, len, dst, SAFE_CRLF_RENORMALIZE, 0);\n }\n\n /*****************************************************************\n@@ -1842,7 +1868,7 @@ struct stream_filter *get_stream_filter(const char *path, const unsigned char *s\n \tif (ca.drv && (ca.drv->process || ca.drv->smudge || ca.drv->clean))\n \t\treturn NULL;\n\n-\tif (ca.encoding)\n+\tif (ca.checkout_encoding)\n \t\treturn NULL;\n\n \tif (ca.crlf_action == CRLF_AUTO || ca.crlf_action == CRLF_AUTO_CRLF)\ndiff --git a/convert.h b/convert.h\nindex 4f2da225a8..9e4e884ec1 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -66,7 +66,8 @@ extern const char *get_convert_attr_ascii(const char *path);\n /* returns 1 if *dst was used */\n extern int convert_to_git(const struct index_state *istate,\n \t\t\t const char *path, const char *src, size_t len,\n-\t\t\t struct strbuf *dst, enum safe_crlf checksafe);\n+\t\t\t struct strbuf *dst, enum safe_crlf checksafe,\n+\t\t\t int write_obj);\n extern int convert_to_working_tree(const char *path, const char *src,\n \t\t\t\t size_t len, struct strbuf *dst);\n extern int async_convert_to_working_tree(const char *path, const char *src,\n@@ -79,13 +80,14 @@ extern int renormalize_buffer(const struct index_state *istate,\n static inline int would_convert_to_git(const struct index_state *istate,\n \t\t\t\t const char *path)\n {\n-\treturn convert_to_git(istate, path, NULL, 0, NULL, 0);\n+\treturn convert_to_git(istate, path, NULL, 0, NULL, 0, 0);\n }\n /* Precondition: would_convert_to_git_filter_fd(path) == true */\n extern void convert_to_git_filter_fd(const struct index_state *istate,\n \t\t\t\t const char *path, int fd,\n \t\t\t\t struct strbuf *dst,\n-\t\t\t\t enum safe_crlf checksafe);\n+\t\t\t\t enum safe_crlf checksafe,\n+\t\t\t\t int write_obj);\n extern int would_convert_to_git_filter_fd(const char *path);\n\n /*****************************************************************\ndiff --git a/diff.c b/diff.c\nindex 2ebe2227b4..16ca0bf0df 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -3599,7 +3599,7 @@ int diff_populate_filespec(struct diff_filespec *s, unsigned int flags)\n \t\t/*\n \t\t * Convert from working tree format to canonical git format\n \t\t */\n-\t\tif (convert_to_git(&the_index, s->path, s->data, s->size, &buf, crlf_warn)) {\n+\t\tif (convert_to_git(&the_index, s->path, s->data, s->size, &buf, crlf_warn, 0)) {\n \t\t\tsize_t size = 0;\n \t\t\tmunmap(s->data, s->size);\n \t\t\ts->should_munmap = 0;\ndiff --git a/sha1_file.c b/sha1_file.c\nindex afe4b90f6e..75800248d2 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1694,7 +1694,8 @@ static int index_mem(struct object_id *oid, void *buf, size_t size,\n \tif ((type == OBJ_BLOB) && path) {\n \t\tstruct strbuf nbuf = STRBUF_INIT;\n \t\tif (convert_to_git(&the_index, path, buf, size, &nbuf,\n-\t\t\t\t get_safe_crlf(flags))) {\n+\t\t\t\t get_safe_crlf(flags),\n+\t\t\t\t write_object)) {\n \t\t\tbuf = strbuf_detach(&nbuf, &size);\n \t\t\tre_allocated = 1;\n \t\t}\n@@ -1728,7 +1729,7 @@ static int index_stream_convert_blob(struct object_id *oid, int fd,\n \tassert(would_convert_to_git_filter_fd(path));\n\n \tconvert_to_git_filter_fd(&the_index, path, fd, &sbuf,\n-\t\t\t\t get_safe_crlf(flags));\n+\t\t\t\t get_safe_crlf(flags), write_object);\n\n \tif (write_object)\n \t\tret = write_sha1_file(sbuf.buf, sbuf.len, typename(OBJ_BLOB),\ndiff --git a/strbuf.c b/strbuf.c\nindex 323c49ceb3..54276e96e7 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -760,6 +760,19 @@ char *xstrdup_tolower(const char *string)\n \treturn result;\n }\n\n+char *xstrdup_toupper(const char *string)\n+{\n+\tchar *result;\n+\tsize_t len, i;\n+\n+\tlen = strlen(string);\n+\tresult = xmallocz(len);\n+\tfor (i = 0; i < len; i++)\n+\t\tresult[i] = toupper(string[i]);\n+\tresult[i] = '\\0';\n+\treturn result;\n+}\n+\n char *xstrvfmt(const char *fmt, va_list ap)\n {\n \tstruct strbuf buf = STRBUF_INIT;\ndiff --git a/strbuf.h b/strbuf.h\nindex 0a74acb236..2bc148526f 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -616,6 +616,7 @@ __attribute__((format (printf,2,3)))\n extern int fprintf_ln(FILE *fp, const char *fmt, ...);\n\n char *xstrdup_tolower(const char *);\n+char *xstrdup_toupper(const char *);\n\n /**\n * Create a newly allocated string using printf format. You can do this easily\ndiff --git a/t/t0028-checkout-encoding.sh b/t/t0028-checkout-encoding.sh\nnew file mode 100755\nindex 0000000000..df3cc91269\n--- /dev/null\n+++ b/t/t0028-checkout-encoding.sh\n@@ -0,0 +1,199 @@\n+#!/bin/sh\n+\n+test_description='checkout-encoding conversion via gitattributes'\n+\n+. ./test-lib.sh\n+\n+GIT_TRACE_CHECKOUT_ENCODING=1 && export GIT_TRACE_CHECKOUT_ENCODING\n+\n+test_expect_success 'setup test repo' '\n+\n+\ttext=\"hallo there!\\ncan you read me?\" &&\n+\n+\techo \"*.utf16 text checkout-encoding=utf-16\" >.gitattributes &&\n+\n+\tprintf \"$text\" >test.utf8.raw &&\n+\tprintf \"$text\" | iconv -f UTF-8 -t UTF-16 >test.utf16.raw &&\n+\tcp test.utf16.raw test.utf16 &&\n+\n+\tgit add .gitattributes test.utf16 &&\n+\tgit commit -m initial\n+'\n+\n+test_expect_success 'ensure UTF-8 is stored in Git' '\n+\tgit cat-file -p :test.utf16 >test.utf16.git &&\n+\ttest_cmp_bin test.utf8.raw test.utf16.git &&\n+\trm test.utf8.raw test.utf16.git\n+'\n+\n+test_expect_success 're-encode to UTF-16 on checkout' '\n+\trm test.utf16 &&\n+\tgit checkout test.utf16 &&\n+\ttest_cmp_bin test.utf16.raw test.utf16 &&\n+\n+\t# cleanup\n+\trm test.utf16.raw\n+'\n+\n+test_expect_success 'check prohibited UTF BOM' '\n+\tprintf \"\\0a\\0b\\0c\" >nobom.utf16be.raw &&\n+\tprintf \"a\\0b\\0c\\0\" >nobom.utf16le.raw &&\n+\tprintf \"\\376\\777\\0a\\0b\\0c\" >bebom.utf16be.raw &&\n+\tprintf \"\\777\\376a\\0b\\0c\\0\" >lebom.utf16le.raw &&\n+\n+\tprintf \"\\0\\0\\0a\\0\\0\\0b\\0\\0\\0c\" >nobom.utf32be.raw &&\n+\tprintf \"a\\0\\0\\0b\\0\\0\\0c\\0\\0\\0\" >nobom.utf32le.raw &&\n+\tprintf \"\\0\\0\\376\\777\\0\\0\\0a\\0\\0\\0b\\0\\0\\0c\" >bebom.utf32be.raw &&\n+\tprintf \"\\777\\376\\0\\0a\\0\\0\\0b\\0\\0\\0c\\0\\0\\0\" >lebom.utf32le.raw &&\n+\n+\techo \"*.utf16be text checkout-encoding=utf-16be\" >>.gitattributes &&\n+\techo \"*.utf16le text checkout-encoding=utf-16le\" >>.gitattributes &&\n+\techo \"*.utf32be text checkout-encoding=utf-32be\" >>.gitattributes &&\n+\techo \"*.utf32le text checkout-encoding=utf-32le\" >>.gitattributes &&\n+\n+\t# Here we add a UTF-16 files with BOM (big-endian and little-endian)\n+\t# but we tell Git to treat it as UTF-16BE/UTF-16LE. In these cases\n+\t# the BOM is prohibited.\n+\tcp bebom.utf16be.raw bebom.utf16be &&\n+\ttest_must_fail git add bebom.utf16be 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-16BE\" err.out &&\n+\n+\tcp lebom.utf16le.raw lebom.utf16be &&\n+\ttest_must_fail git add lebom.utf16be 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-16BE\" err.out &&\n+\n+\tcp bebom.utf16be.raw bebom.utf16le &&\n+\ttest_must_fail git add bebom.utf16le 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-16LE\" err.out &&\n+\n+\tcp lebom.utf16le.raw lebom.utf16le &&\n+\ttest_must_fail git add lebom.utf16le 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-16LE\" err.out &&\n+\n+\t# ... and the same for UTF-32\n+\tcp bebom.utf32be.raw bebom.utf32be &&\n+\ttest_must_fail git add bebom.utf32be 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-32BE\" err.out &&\n+\n+\tcp lebom.utf32le.raw lebom.utf32be &&\n+\ttest_must_fail git add lebom.utf32be 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-32BE\" err.out &&\n+\n+\tcp bebom.utf32be.raw bebom.utf32le &&\n+\ttest_must_fail git add bebom.utf32le 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-32LE\" err.out &&\n+\n+\tcp lebom.utf32le.raw lebom.utf32le &&\n+\ttest_must_fail git add lebom.utf32le 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-32LE\" err.out &&\n+\n+\t# cleanup\n+\tgit reset --hard HEAD\n+'\n+\n+test_expect_success 'check required UTF BOM' '\n+\techo \"*.utf32 text checkout-encoding=utf-32\" >>.gitattributes &&\n+\n+\tcp nobom.utf16be.raw nobom.utf16 &&\n+\ttest_must_fail git add nobom.utf16 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is required .* UTF-16\" err.out &&\n+\n+\tcp nobom.utf16le.raw nobom.utf16 &&\n+\ttest_must_fail git add nobom.utf16 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is required .* UTF-16\" err.out &&\n+\n+\tcp nobom.utf32be.raw nobom.utf32 &&\n+\ttest_must_fail git add nobom.utf32 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is required .* UTF-32\" err.out &&\n+\n+\tcp nobom.utf32le.raw nobom.utf32 &&\n+\ttest_must_fail git add nobom.utf32 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is required .* UTF-32\" err.out &&\n+\n+\t# cleanup\n+\trm nobom.utf16 nobom.utf32 &&\n+\tgit reset --hard HEAD\n+'\n+\n+test_expect_success 'eol conversion for UTF-16 encoded files on checkout' '\n+\tprintf \"one\\ntwo\\nthree\\n\" >lf.utf8.raw &&\n+\tprintf \"one\\r\\ntwo\\r\\nthree\\r\\n\" >crlf.utf8.raw &&\n+\n+\tcat lf.utf8.raw | iconv -f UTF-8 -t UTF-16 >lf.utf16.raw &&\n+\tcat crlf.utf8.raw | iconv -f UTF-8 -t UTF-16 >crlf.utf16.raw &&\n+\tcp crlf.utf16.raw eol.utf16 &&\n+\n+\tgit add eol.utf16 &&\n+\tgit commit -m eol &&\n+\n+\t# UTF-16 with CRLF (Windows line endings)\n+\trm eol.utf16 &&\n+\tgit -c core.eol=crlf checkout eol.utf16 &&\n+\ttest_cmp_bin crlf.utf16.raw eol.utf16 &&\n+\n+\t# UTF-16 with LF (Unix line endings)\n+\trm eol.utf16 &&\n+\tgit -c core.eol=lf checkout eol.utf16 &&\n+\ttest_cmp_bin lf.utf16.raw eol.utf16 &&\n+\n+\trm crlf.utf16.raw crlf.utf8.raw lf.utf16.raw lf.utf8.raw &&\n+\n+\t# cleanup\n+\tgit reset --hard HEAD^\n+'\n+\n+test_expect_success 'check unsupported encodings' '\n+\n+\techo \"*.nothing text checkout-encoding=\" >>.gitattributes &&\n+\tprintf \"nothing\" >t.nothing &&\n+\tgit add t.nothing &&\n+\n+\techo \"*.garbage text checkout-encoding=garbage\" >>.gitattributes &&\n+\tprintf \"garbage\" >t.garbage &&\n+\ttest_must_fail git add t.garbage 2>err.out &&\n+\ttest_i18ngrep \"fatal: failed to encode\" err.out &&\n+\n+\t# cleanup\n+\trm err.out &&\n+\tgit reset --hard HEAD\n+'\n+\n+test_expect_success 'error if encoding round trip is not the same during refresh' '\n+\tBEFORE_STATE=$(git rev-parse HEAD) &&\n+\n+\t# Skip the UTF-16 filter for the added file\n+\t# This simulates a Git version that has no checkoutEncoding support\n+\techo \"hallo\" >nonsense.utf16 &&\n+\tTEST_HASH=$(git hash-object --no-filters -w nonsense.utf16) &&\n+\tgit update-index --add --cacheinfo 100644 $TEST_HASH nonsense.utf16 &&\n+\tCOMMIT=$(git commit-tree -p $(git rev-parse HEAD) -m \"plain commit\" $(git write-tree)) &&\n+\tgit update-ref refs/heads/master $COMMIT &&\n+\n+\ttest_must_fail git checkout HEAD^ 2>err.out &&\n+\ttest_i18ngrep \"error: .* overwritten by checkout:\" err.out &&\n+\n+\t# cleanup\n+\trm err.out &&\n+\tgit reset --hard $BEFORE_STATE\n+'\n+\n+test_expect_success 'error if encoding garbage is already in Git' '\n+\tBEFORE_STATE=$(git rev-parse HEAD) &&\n+\n+\t# Skip the UTF-16 filter for the added file\n+\t# This simulates a Git version that has no checkoutEncoding support\n+\tcp nobom.utf16be.raw nonsense.utf16 &&\n+\tTEST_HASH=$(git hash-object --no-filters -w nonsense.utf16) &&\n+\tgit update-index --add --cacheinfo 100644 $TEST_HASH nonsense.utf16 &&\n+\tCOMMIT=$(git commit-tree -p $(git rev-parse HEAD) -m \"plain commit\" $(git write-tree)) &&\n+\tgit update-ref refs/heads/master $COMMIT &&\n+\n+\tgit diff 2>err.out &&\n+\ttest_i18ngrep \"error: BOM is required\" err.out &&\n+\n+\t# cleanup\n+\trm err.out &&\n+\tgit reset --hard $BEFORE_STATE\n+'\n+\n+test_done\ndiff --git a/t/t0028-encoding.sh b/t/t0028-encoding.sh\ndeleted file mode 100755\nindex 24f24f9e0d..0000000000\n--- a/t/t0028-encoding.sh\n+++ /dev/null\n@@ -1,60 +0,0 @@\n-#!/bin/sh\n-\n-test_description='encoding conversion via gitattributes'\n-\n-. ./test-lib.sh\n-\n-test_expect_success 'setup test repo' '\n-\n-\ttext=\"hallo there!\\ncan you read me?\" &&\n-\n-\techo \"*.utf16 text encoding=utf-16\" >.gitattributes &&\n-\tprintf \"$text\" >t.utf8.raw &&\n-\tprintf \"$text\" | iconv -f UTF-8 -t UTF-16 >t.utf16.raw &&\n-\tcp t.utf16.raw t.utf16 &&\n-\n-\tgit add .gitattributes t.utf16.raw t.utf16 &&\n-\tgit commit -m initial\n-'\n-\n-test_expect_success 'ensure UTF-8 is stored in Git' '\n-\tgit cat-file -p :t.utf16 >t.utf16.git &&\n-\ttest_cmp_bin t.utf8.raw t.utf16.git\n-'\n-\n-test_expect_success 're-encode to UTF-16 on checkout' '\n-\trm t.utf16 &&\n-\tgit checkout t.utf16 &&\n-\ttest_cmp_bin t.utf16.raw t.utf16\n-'\n-\n-test_expect_success 'warn if an unsupported encoding is used' '\n-\techo \"*.garbage text encoding=garbage\" >>.gitattributes &&\n-\tprintf \"garbage\" >t.garbage &&\n-\tgit add t.garbage 2>error.out &&\n-\ttest_i18ngrep \"warning: unsupported encoding\" error.out &&\n-\n-\t# cleanup\n-\tgit reset --hard HEAD\n-'\n-\n-test_expect_success 'fail if files with invalid encoding are added' '\n-\tprintf \"\\0\\0h\\0a\" >error.utf16 &&\n-\t# The test string encoding would fail\n-\t# test_must_fail iconv -f utf-16 -t utf-8 error.utf16 &&\n-\ttest_must_fail git add error.utf16\n-'\n-\n-# Some sequences might trigger errno == E2BIG in reencode_string_iconv, utf.8.\n-# This would cause no error on \"git add\" and, consequently, the Git internal\n-# UTF-8 encoded blob would contain garbage. Hence, the worktree file after a\n-# checkout would contain garbage, too. This garbage would not match the file\n-# that was initially added.\n-test_expect_success 'fail if encoding from X to UTF-8 and back to X is not the same' '\n-\tprintf \"\\xc3\\x28\" >error.utf16 &&\n-\t# The test string re-encoding would fail\n-\t# iconv -f utf-16 -t utf-8 error.utf16 | iconv -f utf-8 -t utf-16 &&\n-\ttest_must_fail git add error.utf16\n-'\n-\n-test_done\ndiff --git a/utf8.c b/utf8.c\nindex 2c27ce0137..1978d6c42a 100644\n--- a/utf8.c\n+++ b/utf8.c\n@@ -538,6 +538,43 @@ char *reencode_string_len(const char *in, int insz,\n }\n #endif\n\n+static int has_bom_prefix(const char *data, size_t len,\n+\t\t\t const char *bom, size_t bom_len)\n+{\n+\treturn (len >= bom_len) && !memcmp(data, bom, bom_len);\n+}\n+\n+const char utf16_be_bom[] = {0xFE, 0xFF};\n+const char utf16_le_bom[] = {0xFF, 0xFE};\n+const char utf32_be_bom[] = {0x00, 0x00, 0xFE, 0xFF};\n+const char utf32_le_bom[] = {0xFF, 0xFE, 0x00, 0x00};\n+\n+int has_prohibited_utf_bom(const char *enc, const char *data, size_t len)\n+{\n+\treturn (\n+\t (!strcmp(enc, \"UTF-16BE\") || !strcmp(enc, \"UTF-16LE\")) &&\n+\t (has_bom_prefix(data, len, utf16_be_bom, sizeof(utf16_be_bom)) ||\n+\t has_bom_prefix(data, len, utf16_le_bom, sizeof(utf16_le_bom)))\n+\t) || (\n+\t (!strcmp(enc, \"UTF-32BE\") || !strcmp(enc, \"UTF-32LE\")) &&\n+\t (has_bom_prefix(data, len, utf32_be_bom, sizeof(utf32_be_bom)) ||\n+\t has_bom_prefix(data, len, utf32_le_bom, sizeof(utf32_le_bom)))\n+\t);\n+}\n+\n+int has_missing_utf_bom(const char *enc, const char *data, size_t len)\n+{\n+\treturn (\n+\t !strcmp(enc, \"UTF-16\") &&\n+\t !(has_bom_prefix(data, len, utf16_be_bom, sizeof(utf16_be_bom)) ||\n+\t has_bom_prefix(data, len, utf16_le_bom, sizeof(utf16_le_bom)))\n+\t) || (\n+\t !strcmp(enc, \"UTF-32\") &&\n+\t !(has_bom_prefix(data, len, utf32_be_bom, sizeof(utf32_be_bom)) ||\n+\t has_bom_prefix(data, len, utf32_le_bom, sizeof(utf32_le_bom)))\n+\t);\n+}\n+\n /*\n * Returns first character length in bytes for multi-byte `text` according to\n * `encoding`.\ndiff --git a/utf8.h b/utf8.h\nindex 6bbcf31a83..26b5e91852 100644\n--- a/utf8.h\n+++ b/utf8.h\n@@ -70,4 +70,29 @@ typedef enum {\n void strbuf_utf8_align(struct strbuf *buf, align_type position, unsigned int width,\n \t\t const char *s);\n\n+/*\n+ * Whenever a data stream is declared to be UTF-16BE, UTF-16LE, UTF-32BE\n+ * or UTF-32LE a BOM must not be used [1]. The function returns true if\n+ * this is the case.\n+ *\n+ * [1] http://unicode.org/faq/utf_bom.html#bom10\n+ */\n+int has_prohibited_utf_bom(const char *enc, const char *data, size_t len);\n+\n+/*\n+ * If the endianness is not defined in the encoding name, then we\n+ * require a BOM. The function returns true if a required BOM is missing.\n+ *\n+ * The Unicode standard instructs to assume big-endian if there\n+ * in no BOM for UTF-16/32 [1][2]. However, the W3C/WHATWG\n+ * encoding standard used in HTML5 recommends to assume\n+ * little-endian to \"deal with deployed content\" [3].\n+ *\n+ * [1] http://unicode.org/faq/utf_bom.html#gen6\n+ * [2] http://www.unicode.org/versions/Unicode10.0.0/ch03.pdf\n+ * Section 3.10, D98, page 132\n+ * [3] https://encoding.spec.whatwg.org/#utf-16le\n+ */\n+int has_missing_utf_bom(const char *enc, const char *data, size_t len);\n+\n #endif\n\n\n### Patches\n\nLars Schneider (5):\n strbuf: add xstrdup_toupper()\n utf8: add function to detect prohibited UTF-16/32 BOM\n utf8: add function to detect a missing UTF-16/32 BOM\n convert: add support for 'checkout-encoding' attribute\n convert: add tracing for checkout-encoding\n\n Documentation/gitattributes.txt | 59 +++++++++++\n apply.c | 2 +-\n blame.c | 2 +-\n combine-diff.c | 2 +-\n convert.c | 224 +++++++++++++++++++++++++++++++++++++++-\n convert.h | 8 +-\n diff.c | 2 +-\n sha1_file.c | 5 +-\n strbuf.c | 13 +++\n strbuf.h | 1 +\n t/t0028-checkout-encoding.sh | 199 +++++++++++++++++++++++++++++++++++\n utf8.c | 37 +++++++\n utf8.h | 25 +++++\n 13 files changed, 566 insertions(+), 13 deletions(-)\n create mode 100755 t/t0028-checkout-encoding.sh\n\n\nbase-commit: 95ec6b1b3393eb6e26da40c565520a8db9796e9f\n--\n2.15.1\n\n"},{"id":"335533","messageId":"20171229152222.39680-2-lars.schneider@autodesk.com","threadId":"47511","inReplyTo":"20171229152222.39680-1-lars.schneider@autodesk.com","subject":"[PATCH v2 1/5] strbuf: add xstrdup_toupper()","fromName":"","fromEmail":"lars.schneider@autodesk.com","sentAt":"2017-12-29T15:22:18Z","receivedAt":"2017-12-29T15:22:54Z","isPatch":true,"sender":{"key":"lars.schneider@autodesk.com","avatar":"https://gravatar.com/avatar/524188df3a38b02692619777bb90d9cc735db1ad75f5304e0bf09f8efd299775?d=mp&s=160"},"body":"From: Lars Schneider \n\nCreate a copy of an existing string and make all characters upper case.\nSimilar xstrdup_tolower().\n\nThis function is used in a subsequent commit.\n\nSigned-off-by: Lars Schneider \n---\n strbuf.c | 13 +++++++++++++\n strbuf.h | 1 +\n 2 files changed, 14 insertions(+)\n\ndiff --git a/strbuf.c b/strbuf.c\nindex 323c49ceb3..54276e96e7 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -760,6 +760,19 @@ char *xstrdup_tolower(const char *string)\n \treturn result;\n }\n \n+char *xstrdup_toupper(const char *string)\n+{\n+\tchar *result;\n+\tsize_t len, i;\n+\n+\tlen = strlen(string);\n+\tresult = xmallocz(len);\n+\tfor (i = 0; i < len; i++)\n+\t\tresult[i] = toupper(string[i]);\n+\tresult[i] = '\\0';\n+\treturn result;\n+}\n+\n char *xstrvfmt(const char *fmt, va_list ap)\n {\n \tstruct strbuf buf = STRBUF_INIT;\ndiff --git a/strbuf.h b/strbuf.h\nindex 0a74acb236..2bc148526f 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -616,6 +616,7 @@ __attribute__((format (printf,2,3)))\n extern int fprintf_ln(FILE *fp, const char *fmt, ...);\n \n char *xstrdup_tolower(const char *);\n+char *xstrdup_toupper(const char *);\n \n /**\n * Create a newly allocated string using printf format. You can do this easily\n-- \n2.15.1\n\n"},{"id":"335534","messageId":"20171229152222.39680-3-lars.schneider@autodesk.com","threadId":"47511","inReplyTo":"20171229152222.39680-1-lars.schneider@autodesk.com","subject":"[PATCH v2 2/5] utf8: add function to detect prohibited UTF-16/32 BOM","fromName":"","fromEmail":"lars.schneider@autodesk.com","sentAt":"2017-12-29T15:22:19Z","receivedAt":"2017-12-29T15:22:55Z","isPatch":true,"sender":{"key":"lars.schneider@autodesk.com","avatar":"https://gravatar.com/avatar/524188df3a38b02692619777bb90d9cc735db1ad75f5304e0bf09f8efd299775?d=mp&s=160"},"body":"From: Lars Schneider \n\nWhenever a data stream is declared to be UTF-16BE, UTF-16LE, UTF-32BE\nor UTF-32LE a BOM must not be used [1]. The function returns true if\nthis is the case.\n\nThis function is used in a subsequent commit.\n\n[1] http://unicode.org/faq/utf_bom.html#bom10\n\nSigned-off-by: Lars Schneider \n---\n utf8.c | 24 ++++++++++++++++++++++++\n utf8.h | 9 +++++++++\n 2 files changed, 33 insertions(+)\n\ndiff --git a/utf8.c b/utf8.c\nindex 2c27ce0137..776660ee12 100644\n--- a/utf8.c\n+++ b/utf8.c\n@@ -538,6 +538,30 @@ char *reencode_string_len(const char *in, int insz,\n }\n #endif\n \n+static int has_bom_prefix(const char *data, size_t len,\n+\t\t\t const char *bom, size_t bom_len)\n+{\n+\treturn (len >= bom_len) && !memcmp(data, bom, bom_len);\n+}\n+\n+const char utf16_be_bom[] = {0xFE, 0xFF};\n+const char utf16_le_bom[] = {0xFF, 0xFE};\n+const char utf32_be_bom[] = {0x00, 0x00, 0xFE, 0xFF};\n+const char utf32_le_bom[] = {0xFF, 0xFE, 0x00, 0x00};\n+\n+int has_prohibited_utf_bom(const char *enc, const char *data, size_t len)\n+{\n+\treturn (\n+\t (!strcmp(enc, \"UTF-16BE\") || !strcmp(enc, \"UTF-16LE\")) &&\n+\t (has_bom_prefix(data, len, utf16_be_bom, sizeof(utf16_be_bom)) ||\n+\t has_bom_prefix(data, len, utf16_le_bom, sizeof(utf16_le_bom)))\n+\t) || (\n+\t (!strcmp(enc, \"UTF-32BE\") || !strcmp(enc, \"UTF-32LE\")) &&\n+\t (has_bom_prefix(data, len, utf32_be_bom, sizeof(utf32_be_bom)) ||\n+\t has_bom_prefix(data, len, utf32_le_bom, sizeof(utf32_le_bom)))\n+\t);\n+}\n+\n /*\n * Returns first character length in bytes for multi-byte `text` according to\n * `encoding`.\ndiff --git a/utf8.h b/utf8.h\nindex 6bbcf31a83..4711429af9 100644\n--- a/utf8.h\n+++ b/utf8.h\n@@ -70,4 +70,13 @@ typedef enum {\n void strbuf_utf8_align(struct strbuf *buf, align_type position, unsigned int width,\n \t\t const char *s);\n \n+/*\n+ * Whenever a data stream is declared to be UTF-16BE, UTF-16LE, UTF-32BE\n+ * or UTF-32LE a BOM must not be used [1]. The function returns true if\n+ * this is the case.\n+ *\n+ * [1] http://unicode.org/faq/utf_bom.html#bom10\n+ */\n+int has_prohibited_utf_bom(const char *enc, const char *data, size_t len);\n+\n #endif\n-- \n2.15.1\n\n"},{"id":"335535","messageId":"20171229152222.39680-6-lars.schneider@autodesk.com","threadId":"47511","inReplyTo":"20171229152222.39680-1-lars.schneider@autodesk.com","subject":"[PATCH v2 5/5] convert: add tracing for checkout-encoding","fromName":"","fromEmail":"lars.schneider@autodesk.com","sentAt":"2017-12-29T15:22:22Z","receivedAt":"2017-12-29T15:22:59Z","isPatch":true,"sender":{"key":"lars.schneider@autodesk.com","avatar":"https://gravatar.com/avatar/524188df3a38b02692619777bb90d9cc735db1ad75f5304e0bf09f8efd299775?d=mp&s=160"},"body":"From: Lars Schneider \n\nAdd the GIT_TRACE_CHECKOUT_ENCODING environment variable to enable\ntracing for content that is reencoded with the checkout-encoding\nattribute.\n\nSigned-off-by: Lars Schneider \n---\n convert.c | 28 ++++++++++++++++++++++++++++\n t/t0028-checkout-encoding.sh | 2 ++\n 2 files changed, 30 insertions(+)\n\ndiff --git a/convert.c b/convert.c\nindex fc8c96b670..ca7b2f3e5c 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -257,6 +257,29 @@ static int will_convert_lf_to_crlf(size_t len, struct text_stat *stats,\n \n }\n \n+static void trace_encoding(const char *context, const char *path,\n+\t\t\t const char *encoding, const char *buf, size_t len)\n+{\n+\tstatic struct trace_key coe = TRACE_KEY_INIT(CHECKOUT_ENCODING);\n+\tstruct strbuf trace = STRBUF_INIT;\n+\tint i;\n+\n+\tstrbuf_addf(&trace, \"%s (%s, considered %s):\\n\", context, path, encoding);\n+\tfor (i = 0; i < len && buf; ++i) {\n+\t\tstrbuf_addf(\n+\t\t\t&trace,\"| \\e[2m%2i:\\e[0m %2x \\e[2m%c\\e[0m%c\",\n+\t\t\ti,\n+\t\t\t(unsigned char) buf[i],\n+\t\t\t(buf[i] > 32 && buf[i] < 127 ? buf[i] : ' '),\n+\t\t\t((i+1) % 8 && (i+1) < len ? ' ' : '\\n')\n+\t\t);\n+\t}\n+\tstrbuf_addchars(&trace, '\\n', 1);\n+\n+\ttrace_strbuf(&coe, &trace);\n+\tstrbuf_release(&trace);\n+}\n+\n static struct encoding {\n \tconst char *name;\n \tstruct encoding *next;\n@@ -316,6 +339,7 @@ static int encode_to_git(const char *path, const char *src, size_t src_len,\n \t\t\terror(error_msg, path, enc->name);\n \t}\n \n+\ttrace_encoding(\"source\", path, enc->name, src, src_len);\n \tdst = reencode_string_len(src, src_len, default_encoding, enc->name,\n \t\t\t\t &dst_len);\n \tif (!dst) {\n@@ -331,6 +355,7 @@ static int encode_to_git(const char *path, const char *src, size_t src_len,\n \t\telse\n \t\t\terror(msg, path, enc->name, default_encoding);\n \t}\n+\ttrace_encoding(\"destination\", path, default_encoding, dst, dst_len);\n \n \t/*\n \t * UTF supports lossless round tripping [1]. UTF to other encoding are\n@@ -356,6 +381,9 @@ static int encode_to_git(const char *path, const char *src, size_t src_len,\n \t\t\t\t\t enc->name, default_encoding,\n \t\t\t\t\t &re_src_len);\n \n+\t\ttrace_encoding(\"reencoded source\", path, enc->name,\n+\t\t\t re_src, re_src_len);\n+\n \t\tif (!re_src || src_len != re_src_len ||\n \t\t memcmp(src, re_src, src_len)) {\n \t\t\tconst char* msg = _(\"encoding '%s' from %s to %s and \"\ndiff --git a/t/t0028-checkout-encoding.sh b/t/t0028-checkout-encoding.sh\nindex 1a329ab933..df3cc91269 100755\n--- a/t/t0028-checkout-encoding.sh\n+++ b/t/t0028-checkout-encoding.sh\n@@ -4,6 +4,8 @@ test_description='checkout-encoding conversion via gitattributes'\n \n . ./test-lib.sh\n \n+GIT_TRACE_CHECKOUT_ENCODING=1 && export GIT_TRACE_CHECKOUT_ENCODING\n+\n test_expect_success 'setup test repo' '\n \n \ttext=\"hallo there!\\ncan you read me?\" &&\n-- \n2.15.1\n\n"},{"id":"335536","messageId":"20171229152222.39680-5-lars.schneider@autodesk.com","threadId":"47511","inReplyTo":"20171229152222.39680-1-lars.schneider@autodesk.com","subject":"[PATCH v2 4/5] convert: add support for 'checkout-encoding' attribute","fromName":"","fromEmail":"lars.schneider@autodesk.com","sentAt":"2017-12-29T15:22:21Z","receivedAt":"2017-12-29T15:23:03Z","isPatch":true,"sender":{"key":"lars.schneider@autodesk.com","avatar":"https://gravatar.com/avatar/524188df3a38b02692619777bb90d9cc735db1ad75f5304e0bf09f8efd299775?d=mp&s=160"},"body":"From: Lars Schneider \n\nGit and its tools (e.g. git diff) expect all text files in UTF-8\nencoding. Git will happily accept content in all other encodings, too,\nbut it might not be able to process the text (e.g. viewing diffs or\nchanging line endings).\n\nAdd an attribute to tell Git what encoding the user has defined for a\ngiven file. If the content is added to the index, then Git converts the\ncontent to a canonical UTF-8 representation. On checkout Git will\nreverse the conversion.\n\nSigned-off-by: Lars Schneider \n---\n Documentation/gitattributes.txt | 59 ++++++++++++\n apply.c | 2 +-\n blame.c | 2 +-\n combine-diff.c | 2 +-\n convert.c | 196 ++++++++++++++++++++++++++++++++++++++-\n convert.h | 8 +-\n diff.c | 2 +-\n sha1_file.c | 5 +-\n t/t0028-checkout-encoding.sh | 197 ++++++++++++++++++++++++++++++++++++++++\n 9 files changed, 460 insertions(+), 13 deletions(-)\n create mode 100755 t/t0028-checkout-encoding.sh\n\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex 30687de81a..0039bd38c3 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -272,6 +272,65 @@ few exceptions. Even though...\n catch potential problems early, safety triggers.\n \n \n+`checkout-encoding`\n+^^^^^^^^^^^^^^^^^^^\n+\n+Git recognizes files encoded with ASCII or one of its supersets (e.g.\n+UTF-8 or ISO-8859-1) as text files. All other encodings are usually\n+interpreted as binary and consequently built-in Git text processing\n+tools (e.g. 'git diff') as well as most Git web front ends do not\n+visualize the content.\n+\n+In these cases you can teach Git the encoding of a file in the working\n+directory with the `checkout-encoding` attribute. If a file with this\n+attributes is added to Git, then Git reencodes the content from the\n+specified encoding to UTF-8 and stores the result in its internal data\n+structure. On checkout the content is encoded back to the specified\n+encoding.\n+\n+Please note that using the `checkout-encoding` attribute has a number\n+of drawbacks:\n+\n+- Reencoding content to non-UTF encodings (e.g. SHIFT-JIS) can cause\n+ errors as the conversion might not be round trip safe.\n+\n+- Reencoding content requires resources that might slow down certain\n+ Git operations (e.g 'git checkout' or 'git add').\n+\n+- Git clients that do not support the `checkout-encoding` attribute or\n+ the used encoding will checkout the respective files as UTF-8 encoded.\n+ That means the content appears to be different which could cause\n+ trouble. Affected clients are older Git versions and alternative Git\n+ implementations such as JGit or libgit2 (as of January 2018).\n+\n+Use the `checkout-encoding` attribute only if you cannot store a file in\n+UTF-8 encoding and if you want Git to be able to process the content as\n+text.\n+\n+Use the following attributes if your '*.txt' files are UTF-16 encoded\n+with byte order mark (BOM) and you want Git to perform automatic line\n+ending conversion based on your platform.\n+\n+------------------------\n+*.txt\t\ttext checkout-encoding=UTF-16\n+------------------------\n+\n+Use the following attributes if your '*.txt' files are UTF-16 little\n+endian encoded without BOM and you want Git to use Windows line endings\n+in the working directory.\n+\n+------------------------\n+*.txt \t\tcheckout-encoding=UTF-16LE text eol=CRLF\n+------------------------\n+\n+You can get a list of all available encodings on your platform with the\n+following command:\n+\n+------------------------\n+iconv --list\n+------------------------\n+\n+\n `ident`\n ^^^^^^^\n \ndiff --git a/apply.c b/apply.c\nindex 321a9fa68d..c4bd5cf1f2 100644\n--- a/apply.c\n+++ b/apply.c\n@@ -2281,7 +2281,7 @@ static int read_old_data(struct stat *st, struct patch *patch,\n \t\t * should never look at the index when explicit crlf option\n \t\t * is given.\n \t\t */\n-\t\tconvert_to_git(NULL, path, buf->buf, buf->len, buf, safe_crlf);\n+\t\tconvert_to_git(NULL, path, buf->buf, buf->len, buf, safe_crlf, 0);\n \t\treturn 0;\n \tdefault:\n \t\treturn -1;\ndiff --git a/blame.c b/blame.c\nindex 2893f3c103..388b66897b 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -229,7 +229,7 @@ static struct commit *fake_working_tree_commit(struct diff_options *opt,\n \t\tif (strbuf_read(&buf, 0, 0) < 0)\n \t\t\tdie_errno(\"failed to read from stdin\");\n \t}\n-\tconvert_to_git(&the_index, path, buf.buf, buf.len, &buf, 0);\n+\tconvert_to_git(&the_index, path, buf.buf, buf.len, &buf, 0, 0);\n \torigin->file.ptr = buf.buf;\n \torigin->file.size = buf.len;\n \tpretend_sha1_file(buf.buf, buf.len, OBJ_BLOB, origin->blob_oid.hash);\ndiff --git a/combine-diff.c b/combine-diff.c\nindex 2505de119a..4555e49b5f 100644\n--- a/combine-diff.c\n+++ b/combine-diff.c\n@@ -1053,7 +1053,7 @@ static void show_patch_diff(struct combine_diff_path *elem, int num_parent,\n \t\t\tif (is_file) {\n \t\t\t\tstruct strbuf buf = STRBUF_INIT;\n \n-\t\t\t\tif (convert_to_git(&the_index, elem->path, result, len, &buf, safe_crlf)) {\n+\t\t\t\tif (convert_to_git(&the_index, elem->path, result, len, &buf, safe_crlf, 0)) {\n \t\t\t\t\tfree(result);\n \t\t\t\t\tresult = strbuf_detach(&buf, &len);\n \t\t\t\t\tresult_size = len;\ndiff --git a/convert.c b/convert.c\nindex 20d7ab67bd..fc8c96b670 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -7,6 +7,7 @@\n #include \"sigchain.h\"\n #include \"pkt-line.h\"\n #include \"sub-process.h\"\n+#include \"utf8.h\"\n \n /*\n * convert.c - convert a file when checking it out and checking it in.\n@@ -256,6 +257,147 @@ static int will_convert_lf_to_crlf(size_t len, struct text_stat *stats,\n \n }\n \n+static struct encoding {\n+\tconst char *name;\n+\tstruct encoding *next;\n+} *encoding, **encoding_tail;\n+static const char *default_encoding = \"UTF-8\";\n+\n+static int encode_to_git(const char *path, const char *src, size_t src_len,\n+\t\t\t struct strbuf *buf, struct encoding *enc, int write_obj)\n+{\n+\tchar *dst;\n+\tint dst_len;\n+\n+\t/*\n+\t * No encoding is specified or there is nothing to encode.\n+\t * Tell the caller that the content was not modified.\n+\t */\n+\tif (!enc || (src && !src_len))\n+\t\treturn 0;\n+\n+\t/*\n+\t * Looks like we got called from \"would_convert_to_git()\".\n+\t * This means Git wants to know if it would encode (= modify!)\n+\t * the content. Let's answer with \"yes\", since an encoding was\n+\t * specified.\n+\t */\n+\tif (!buf && !src)\n+\t\treturn 1;\n+\n+\tif (has_prohibited_utf_bom(enc->name, src, src_len)) {\n+\t\tconst char *error_msg = _(\n+\t\t\t\"BOM is prohibited for '%s' if encoded as %s\");\n+\t\tconst char *advise_msg = _(\n+\t\t\t\"You told Git to treat '%s' as %s. A byte order mark \"\n+\t\t\t\"(BOM) is prohibited with this encoding. Either use \"\n+\t\t\t\"%.6s as checkout encoding or remove the BOM from the \"\n+\t\t\t\"file.\");\n+\n+\t\tadvise(advise_msg, path, enc->name, enc->name, enc->name);\n+\t\tif (write_obj)\n+\t\t\tdie(error_msg, path, enc->name);\n+\t\telse\n+\t\t\terror(error_msg, path, enc->name);\n+\n+\n+\t} else if (has_missing_utf_bom(enc->name, src, src_len)) {\n+\t\tconst char *error_msg = _(\n+\t\t\t\"BOM is required for '%s' if encoded as %s\");\n+\t\tconst char *advise_msg = _(\n+\t\t\t\"You told Git to treat '%s' as %s. A byte order mark \"\n+\t\t\t\"(BOM) is required with this encoding. Either use \"\n+\t\t\t\"%sBE/%sLE as checkout encoding or add a BOM to the \"\n+\t\t\t\"file.\");\n+\t\tadvise(advise_msg, path, enc->name, enc->name, enc->name);\n+\t\tif (write_obj)\n+\t\t\tdie(error_msg, path, enc->name);\n+\t\telse\n+\t\t\terror(error_msg, path, enc->name);\n+\t}\n+\n+\tdst = reencode_string_len(src, src_len, default_encoding, enc->name,\n+\t\t\t\t &dst_len);\n+\tif (!dst) {\n+\t\t/*\n+\t\t * We could add the blob \"as-is\" to Git. However, on checkout\n+\t\t * we would try to reencode to the original encoding. This\n+\t\t * would fail and we would leave the user with a messed-up\n+\t\t * working tree. Let's try to avoid this by screaming loud.\n+\t\t */\n+\t\tconst char* msg = _(\"failed to encode '%s' from %s to %s\");\n+\t\tif (write_obj)\n+\t\t\tdie(msg, path, enc->name, default_encoding);\n+\t\telse\n+\t\t\terror(msg, path, enc->name, default_encoding);\n+\t}\n+\n+\t/*\n+\t * UTF supports lossless round tripping [1]. UTF to other encoding are\n+\t * mostly round trip safe as Unicode aims to be a superset of all other\n+\t * character encodings. However, the SHIFT-JIS (Japanese character set)\n+\t * is an exception as some codes are not round trip safe [2].\n+\t *\n+\t * Reverse the transformation of 'dst' and check the result with 'src'\n+\t * if content is written to Git. This ensures no information is lost\n+\t * during conversion to/from UTF-8.\n+\t *\n+\t * Please note, the code below is not tested because I was not able to\n+\t * generate a faulty round trip without iconv error.\n+\t *\n+\t * [1] http://unicode.org/faq/utf_bom.html#gen2\n+\t * [2] https://support.microsoft.com/en-us/help/170559/prb-conversion-problem-between-shift-jis-and-unicode\n+\t */\n+\tif (write_obj && !strcmp(enc->name, \"SHIFT-JIS\")) {\n+\t\tchar *re_src;\n+\t\tint re_src_len;\n+\n+\t\tre_src = reencode_string_len(dst, dst_len,\n+\t\t\t\t\t enc->name, default_encoding,\n+\t\t\t\t\t &re_src_len);\n+\n+\t\tif (!re_src || src_len != re_src_len ||\n+\t\t memcmp(src, re_src, src_len)) {\n+\t\t\tconst char* msg = _(\"encoding '%s' from %s to %s and \"\n+\t\t\t\t\t \"back is not the same\");\n+\t\t\tif (write_obj)\n+\t\t\t\tdie(msg, path, enc->name, default_encoding);\n+\t\t\telse\n+\t\t\t\terror(msg, path, enc->name, default_encoding);\n+\t\t}\n+\n+\t\tfree(re_src);\n+\t}\n+\n+\tstrbuf_attach(buf, dst, dst_len, dst_len + 1);\n+\treturn 1;\n+}\n+\n+static int encode_to_worktree(const char *path, const char *src, size_t src_len,\n+\t\t\t struct strbuf *buf, struct encoding *enc)\n+{\n+\tchar *dst;\n+\tint dst_len;\n+\n+\t/*\n+\t * No encoding is specified or there is nothing to encode.\n+\t * Tell the caller that the content was not modified.\n+\t */\n+\tif (!enc || (src && !src_len))\n+\t\treturn 0;\n+\n+\tdst = reencode_string_len(src, src_len, enc->name, default_encoding,\n+\t\t\t\t &dst_len);\n+\tif (!dst) {\n+\t\terror(\"failed to encode '%s' from %s to %s\",\n+\t\t\tpath, enc->name, default_encoding);\n+\t\treturn 0;\n+\t}\n+\n+\tstrbuf_attach(buf, dst, dst_len, dst_len + 1);\n+\treturn 1;\n+}\n+\n static int crlf_to_git(const struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n \t\t struct strbuf *buf,\n@@ -969,6 +1111,31 @@ static int ident_to_worktree(const char *path, const char *src, size_t len,\n \treturn 1;\n }\n \n+static struct encoding *git_path_check_encoding(struct attr_check_item *check)\n+{\n+\tconst char *value = check->value;\n+\tstruct encoding *enc;\n+\n+\tif (ATTR_TRUE(value) || ATTR_FALSE(value) || ATTR_UNSET(value) ||\n+\t !strlen(value))\n+\t\treturn NULL;\n+\n+\tfor (enc = encoding; enc; enc = enc->next)\n+\t\tif (!strcasecmp(value, enc->name))\n+\t\t\treturn enc;\n+\n+\t/* Don't encode to the default encoding */\n+\tif (!strcasecmp(value, default_encoding))\n+\t\treturn NULL;\n+\n+\tenc = xcalloc(1, sizeof(struct convert_driver));\n+\tenc->name = xstrdup_toupper(value); /* aways use upper case names! */\n+\t*encoding_tail = enc;\n+\tencoding_tail = &(enc->next);\n+\n+\treturn enc;\n+}\n+\n static enum crlf_action git_path_check_crlf(struct attr_check_item *check)\n {\n \tconst char *value = check->value;\n@@ -1024,6 +1191,7 @@ struct conv_attrs {\n \tenum crlf_action attr_action; /* What attr says */\n \tenum crlf_action crlf_action; /* When no attr is set, use core.autocrlf */\n \tint ident;\n+\tstruct encoding *checkout_encoding; /* Supported encoding or default encoding if NULL */\n };\n \n static void convert_attrs(struct conv_attrs *ca, const char *path)\n@@ -1032,8 +1200,10 @@ static void convert_attrs(struct conv_attrs *ca, const char *path)\n \n \tif (!check) {\n \t\tcheck = attr_check_initl(\"crlf\", \"ident\", \"filter\",\n-\t\t\t\t\t \"eol\", \"text\", NULL);\n+\t\t\t\t\t \"eol\", \"text\", \"checkout-encoding\",\n+\t\t\t\t\t NULL);\n \t\tuser_convert_tail = &user_convert;\n+\t\tencoding_tail = &encoding;\n \t\tgit_config(read_convert_config, NULL);\n \t}\n \n@@ -1055,6 +1225,7 @@ static void convert_attrs(struct conv_attrs *ca, const char *path)\n \t\t\telse if (eol_attr == EOL_CRLF)\n \t\t\t\tca->crlf_action = CRLF_TEXT_CRLF;\n \t\t}\n+\t\tca->checkout_encoding = git_path_check_encoding(ccheck + 5);\n \t} else {\n \t\tca->drv = NULL;\n \t\tca->crlf_action = CRLF_UNDEFINED;\n@@ -1120,7 +1291,7 @@ const char *get_convert_attr_ascii(const char *path)\n \n int convert_to_git(const struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n- struct strbuf *dst, enum safe_crlf checksafe)\n+\t\t struct strbuf *dst, enum safe_crlf checksafe, int write_obj)\n {\n \tint ret = 0;\n \tstruct conv_attrs ca;\n@@ -1135,6 +1306,13 @@ int convert_to_git(const struct index_state *istate,\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\n \t}\n+\n+\tret |= encode_to_git(path, src, len, dst, ca.checkout_encoding, write_obj);\n+\tif (ret && dst) {\n+\t\tsrc = dst->buf;\n+\t\tlen = dst->len;\n+\t}\n+\n \tif (checksafe != SAFE_CRLF_KEEP_CRLF) {\n \t\tret |= crlf_to_git(istate, path, src, len, dst, ca.crlf_action, checksafe);\n \t\tif (ret && dst) {\n@@ -1147,7 +1325,7 @@ int convert_to_git(const struct index_state *istate,\n \n void convert_to_git_filter_fd(const struct index_state *istate,\n \t\t\t const char *path, int fd, struct strbuf *dst,\n-\t\t\t enum safe_crlf checksafe)\n+\t\t\t enum safe_crlf checksafe, int write_obj)\n {\n \tstruct conv_attrs ca;\n \tconvert_attrs(&ca, path);\n@@ -1158,6 +1336,7 @@ void convert_to_git_filter_fd(const struct index_state *istate,\n \tif (!apply_filter(path, NULL, 0, fd, dst, ca.drv, CAP_CLEAN, NULL))\n \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n \n+\tencode_to_git(path, dst->buf, dst->len, dst, ca.checkout_encoding, write_obj);\n \tcrlf_to_git(istate, path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);\n \tident_to_git(path, dst->buf, dst->len, dst, ca.ident);\n }\n@@ -1189,6 +1368,12 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n \t\t}\n \t}\n \n+\tret |= encode_to_worktree(path, src, len, dst, ca.checkout_encoding);\n+\tif (ret) {\n+\t\tsrc = dst->buf;\n+\t\tlen = dst->len;\n+\t}\n+\n \tret_filter = apply_filter(\n \t\tpath, src, len, -1, dst, ca.drv, CAP_SMUDGE, dco);\n \tif (!ret_filter && ca.drv && ca.drv->required)\n@@ -1217,7 +1402,7 @@ int renormalize_buffer(const struct index_state *istate, const char *path,\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\n \t}\n-\treturn ret | convert_to_git(istate, path, src, len, dst, SAFE_CRLF_RENORMALIZE);\n+\treturn ret | convert_to_git(istate, path, src, len, dst, SAFE_CRLF_RENORMALIZE, 0);\n }\n \n /*****************************************************************\n@@ -1655,6 +1840,9 @@ struct stream_filter *get_stream_filter(const char *path, const unsigned char *s\n \tif (ca.drv && (ca.drv->process || ca.drv->smudge || ca.drv->clean))\n \t\treturn NULL;\n \n+\tif (ca.checkout_encoding)\n+\t\treturn NULL;\n+\n \tif (ca.crlf_action == CRLF_AUTO || ca.crlf_action == CRLF_AUTO_CRLF)\n \t\treturn NULL;\n \ndiff --git a/convert.h b/convert.h\nindex 4f2da225a8..9e4e884ec1 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -66,7 +66,8 @@ extern const char *get_convert_attr_ascii(const char *path);\n /* returns 1 if *dst was used */\n extern int convert_to_git(const struct index_state *istate,\n \t\t\t const char *path, const char *src, size_t len,\n-\t\t\t struct strbuf *dst, enum safe_crlf checksafe);\n+\t\t\t struct strbuf *dst, enum safe_crlf checksafe,\n+\t\t\t int write_obj);\n extern int convert_to_working_tree(const char *path, const char *src,\n \t\t\t\t size_t len, struct strbuf *dst);\n extern int async_convert_to_working_tree(const char *path, const char *src,\n@@ -79,13 +80,14 @@ extern int renormalize_buffer(const struct index_state *istate,\n static inline int would_convert_to_git(const struct index_state *istate,\n \t\t\t\t const char *path)\n {\n-\treturn convert_to_git(istate, path, NULL, 0, NULL, 0);\n+\treturn convert_to_git(istate, path, NULL, 0, NULL, 0, 0);\n }\n /* Precondition: would_convert_to_git_filter_fd(path) == true */\n extern void convert_to_git_filter_fd(const struct index_state *istate,\n \t\t\t\t const char *path, int fd,\n \t\t\t\t struct strbuf *dst,\n-\t\t\t\t enum safe_crlf checksafe);\n+\t\t\t\t enum safe_crlf checksafe,\n+\t\t\t\t int write_obj);\n extern int would_convert_to_git_filter_fd(const char *path);\n \n /*****************************************************************\ndiff --git a/diff.c b/diff.c\nindex 2ebe2227b4..16ca0bf0df 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -3599,7 +3599,7 @@ int diff_populate_filespec(struct diff_filespec *s, unsigned int flags)\n \t\t/*\n \t\t * Convert from working tree format to canonical git format\n \t\t */\n-\t\tif (convert_to_git(&the_index, s->path, s->data, s->size, &buf, crlf_warn)) {\n+\t\tif (convert_to_git(&the_index, s->path, s->data, s->size, &buf, crlf_warn, 0)) {\n \t\t\tsize_t size = 0;\n \t\t\tmunmap(s->data, s->size);\n \t\t\ts->should_munmap = 0;\ndiff --git a/sha1_file.c b/sha1_file.c\nindex afe4b90f6e..75800248d2 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1694,7 +1694,8 @@ static int index_mem(struct object_id *oid, void *buf, size_t size,\n \tif ((type == OBJ_BLOB) && path) {\n \t\tstruct strbuf nbuf = STRBUF_INIT;\n \t\tif (convert_to_git(&the_index, path, buf, size, &nbuf,\n-\t\t\t\t get_safe_crlf(flags))) {\n+\t\t\t\t get_safe_crlf(flags),\n+\t\t\t\t write_object)) {\n \t\t\tbuf = strbuf_detach(&nbuf, &size);\n \t\t\tre_allocated = 1;\n \t\t}\n@@ -1728,7 +1729,7 @@ static int index_stream_convert_blob(struct object_id *oid, int fd,\n \tassert(would_convert_to_git_filter_fd(path));\n \n \tconvert_to_git_filter_fd(&the_index, path, fd, &sbuf,\n-\t\t\t\t get_safe_crlf(flags));\n+\t\t\t\t get_safe_crlf(flags), write_object);\n \n \tif (write_object)\n \t\tret = write_sha1_file(sbuf.buf, sbuf.len, typename(OBJ_BLOB),\ndiff --git a/t/t0028-checkout-encoding.sh b/t/t0028-checkout-encoding.sh\nnew file mode 100755\nindex 0000000000..1a329ab933\n--- /dev/null\n+++ b/t/t0028-checkout-encoding.sh\n@@ -0,0 +1,197 @@\n+#!/bin/sh\n+\n+test_description='checkout-encoding conversion via gitattributes'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup test repo' '\n+\n+\ttext=\"hallo there!\\ncan you read me?\" &&\n+\n+\techo \"*.utf16 text checkout-encoding=utf-16\" >.gitattributes &&\n+\n+\tprintf \"$text\" >test.utf8.raw &&\n+\tprintf \"$text\" | iconv -f UTF-8 -t UTF-16 >test.utf16.raw &&\n+\tcp test.utf16.raw test.utf16 &&\n+\n+\tgit add .gitattributes test.utf16 &&\n+\tgit commit -m initial\n+'\n+\n+test_expect_success 'ensure UTF-8 is stored in Git' '\n+\tgit cat-file -p :test.utf16 >test.utf16.git &&\n+\ttest_cmp_bin test.utf8.raw test.utf16.git &&\n+\trm test.utf8.raw test.utf16.git\n+'\n+\n+test_expect_success 're-encode to UTF-16 on checkout' '\n+\trm test.utf16 &&\n+\tgit checkout test.utf16 &&\n+\ttest_cmp_bin test.utf16.raw test.utf16 &&\n+\n+\t# cleanup\n+\trm test.utf16.raw\n+'\n+\n+test_expect_success 'check prohibited UTF BOM' '\n+\tprintf \"\\0a\\0b\\0c\" >nobom.utf16be.raw &&\n+\tprintf \"a\\0b\\0c\\0\" >nobom.utf16le.raw &&\n+\tprintf \"\\376\\777\\0a\\0b\\0c\" >bebom.utf16be.raw &&\n+\tprintf \"\\777\\376a\\0b\\0c\\0\" >lebom.utf16le.raw &&\n+\n+\tprintf \"\\0\\0\\0a\\0\\0\\0b\\0\\0\\0c\" >nobom.utf32be.raw &&\n+\tprintf \"a\\0\\0\\0b\\0\\0\\0c\\0\\0\\0\" >nobom.utf32le.raw &&\n+\tprintf \"\\0\\0\\376\\777\\0\\0\\0a\\0\\0\\0b\\0\\0\\0c\" >bebom.utf32be.raw &&\n+\tprintf \"\\777\\376\\0\\0a\\0\\0\\0b\\0\\0\\0c\\0\\0\\0\" >lebom.utf32le.raw &&\n+\n+\techo \"*.utf16be text checkout-encoding=utf-16be\" >>.gitattributes &&\n+\techo \"*.utf16le text checkout-encoding=utf-16le\" >>.gitattributes &&\n+\techo \"*.utf32be text checkout-encoding=utf-32be\" >>.gitattributes &&\n+\techo \"*.utf32le text checkout-encoding=utf-32le\" >>.gitattributes &&\n+\n+\t# Here we add a UTF-16 files with BOM (big-endian and little-endian)\n+\t# but we tell Git to treat it as UTF-16BE/UTF-16LE. In these cases\n+\t# the BOM is prohibited.\n+\tcp bebom.utf16be.raw bebom.utf16be &&\n+\ttest_must_fail git add bebom.utf16be 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-16BE\" err.out &&\n+\n+\tcp lebom.utf16le.raw lebom.utf16be &&\n+\ttest_must_fail git add lebom.utf16be 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-16BE\" err.out &&\n+\n+\tcp bebom.utf16be.raw bebom.utf16le &&\n+\ttest_must_fail git add bebom.utf16le 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-16LE\" err.out &&\n+\n+\tcp lebom.utf16le.raw lebom.utf16le &&\n+\ttest_must_fail git add lebom.utf16le 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-16LE\" err.out &&\n+\n+\t# ... and the same for UTF-32\n+\tcp bebom.utf32be.raw bebom.utf32be &&\n+\ttest_must_fail git add bebom.utf32be 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-32BE\" err.out &&\n+\n+\tcp lebom.utf32le.raw lebom.utf32be &&\n+\ttest_must_fail git add lebom.utf32be 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-32BE\" err.out &&\n+\n+\tcp bebom.utf32be.raw bebom.utf32le &&\n+\ttest_must_fail git add bebom.utf32le 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-32LE\" err.out &&\n+\n+\tcp lebom.utf32le.raw lebom.utf32le &&\n+\ttest_must_fail git add lebom.utf32le 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-32LE\" err.out &&\n+\n+\t# cleanup\n+\tgit reset --hard HEAD\n+'\n+\n+test_expect_success 'check required UTF BOM' '\n+\techo \"*.utf32 text checkout-encoding=utf-32\" >>.gitattributes &&\n+\n+\tcp nobom.utf16be.raw nobom.utf16 &&\n+\ttest_must_fail git add nobom.utf16 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is required .* UTF-16\" err.out &&\n+\n+\tcp nobom.utf16le.raw nobom.utf16 &&\n+\ttest_must_fail git add nobom.utf16 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is required .* UTF-16\" err.out &&\n+\n+\tcp nobom.utf32be.raw nobom.utf32 &&\n+\ttest_must_fail git add nobom.utf32 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is required .* UTF-32\" err.out &&\n+\n+\tcp nobom.utf32le.raw nobom.utf32 &&\n+\ttest_must_fail git add nobom.utf32 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is required .* UTF-32\" err.out &&\n+\n+\t# cleanup\n+\trm nobom.utf16 nobom.utf32 &&\n+\tgit reset --hard HEAD\n+'\n+\n+test_expect_success 'eol conversion for UTF-16 encoded files on checkout' '\n+\tprintf \"one\\ntwo\\nthree\\n\" >lf.utf8.raw &&\n+\tprintf \"one\\r\\ntwo\\r\\nthree\\r\\n\" >crlf.utf8.raw &&\n+\n+\tcat lf.utf8.raw | iconv -f UTF-8 -t UTF-16 >lf.utf16.raw &&\n+\tcat crlf.utf8.raw | iconv -f UTF-8 -t UTF-16 >crlf.utf16.raw &&\n+\tcp crlf.utf16.raw eol.utf16 &&\n+\n+\tgit add eol.utf16 &&\n+\tgit commit -m eol &&\n+\n+\t# UTF-16 with CRLF (Windows line endings)\n+\trm eol.utf16 &&\n+\tgit -c core.eol=crlf checkout eol.utf16 &&\n+\ttest_cmp_bin crlf.utf16.raw eol.utf16 &&\n+\n+\t# UTF-16 with LF (Unix line endings)\n+\trm eol.utf16 &&\n+\tgit -c core.eol=lf checkout eol.utf16 &&\n+\ttest_cmp_bin lf.utf16.raw eol.utf16 &&\n+\n+\trm crlf.utf16.raw crlf.utf8.raw lf.utf16.raw lf.utf8.raw &&\n+\n+\t# cleanup\n+\tgit reset --hard HEAD^\n+'\n+\n+test_expect_success 'check unsupported encodings' '\n+\n+\techo \"*.nothing text checkout-encoding=\" >>.gitattributes &&\n+\tprintf \"nothing\" >t.nothing &&\n+\tgit add t.nothing &&\n+\n+\techo \"*.garbage text checkout-encoding=garbage\" >>.gitattributes &&\n+\tprintf \"garbage\" >t.garbage &&\n+\ttest_must_fail git add t.garbage 2>err.out &&\n+\ttest_i18ngrep \"fatal: failed to encode\" err.out &&\n+\n+\t# cleanup\n+\trm err.out &&\n+\tgit reset --hard HEAD\n+'\n+\n+test_expect_success 'error if encoding round trip is not the same during refresh' '\n+\tBEFORE_STATE=$(git rev-parse HEAD) &&\n+\n+\t# Skip the UTF-16 filter for the added file\n+\t# This simulates a Git version that has no checkoutEncoding support\n+\techo \"hallo\" >nonsense.utf16 &&\n+\tTEST_HASH=$(git hash-object --no-filters -w nonsense.utf16) &&\n+\tgit update-index --add --cacheinfo 100644 $TEST_HASH nonsense.utf16 &&\n+\tCOMMIT=$(git commit-tree -p $(git rev-parse HEAD) -m \"plain commit\" $(git write-tree)) &&\n+\tgit update-ref refs/heads/master $COMMIT &&\n+\n+\ttest_must_fail git checkout HEAD^ 2>err.out &&\n+\ttest_i18ngrep \"error: .* overwritten by checkout:\" err.out &&\n+\n+\t# cleanup\n+\trm err.out &&\n+\tgit reset --hard $BEFORE_STATE\n+'\n+\n+test_expect_success 'error if encoding garbage is already in Git' '\n+\tBEFORE_STATE=$(git rev-parse HEAD) &&\n+\n+\t# Skip the UTF-16 filter for the added file\n+\t# This simulates a Git version that has no checkoutEncoding support\n+\tcp nobom.utf16be.raw nonsense.utf16 &&\n+\tTEST_HASH=$(git hash-object --no-filters -w nonsense.utf16) &&\n+\tgit update-index --add --cacheinfo 100644 $TEST_HASH nonsense.utf16 &&\n+\tCOMMIT=$(git commit-tree -p $(git rev-parse HEAD) -m \"plain commit\" $(git write-tree)) &&\n+\tgit update-ref refs/heads/master $COMMIT &&\n+\n+\tgit diff 2>err.out &&\n+\ttest_i18ngrep \"error: BOM is required\" err.out &&\n+\n+\t# cleanup\n+\trm err.out &&\n+\tgit reset --hard $BEFORE_STATE\n+'\n+\n+test_done\n-- \n2.15.1\n\n"},{"id":"335537","messageId":"20171229152222.39680-4-lars.schneider@autodesk.com","threadId":"47511","inReplyTo":"20171229152222.39680-1-lars.schneider@autodesk.com","subject":"[PATCH v2 3/5] utf8: add function to detect a missing UTF-16/32 BOM","fromName":"","fromEmail":"lars.schneider@autodesk.com","sentAt":"2017-12-29T15:22:20Z","receivedAt":"2017-12-29T15:23:06Z","isPatch":true,"sender":{"key":"lars.schneider@autodesk.com","avatar":"https://gravatar.com/avatar/524188df3a38b02692619777bb90d9cc735db1ad75f5304e0bf09f8efd299775?d=mp&s=160"},"body":"From: Lars Schneider \n\nIf the endianness is not defined in the encoding name, then let's\nbe strict and require a BOM to avoid any encoding confusion. The\nhas_missing_utf_bom() function returns true if a required BOM is\nmissing.\n\nThe Unicode standard instructs to assume big-endian if there in no BOM\nfor UTF-16/32 [1][2]. However, the W3C/WHATWG encoding standard used\nin HTML5 recommends to assume little-endian to \"deal with deployed\ncontent\" [3]. Strictly requiring a BOM seems to be the safest option\nfor content in Git.\n\nThis function is used in a subsequent commit.\n\n[1] http://unicode.org/faq/utf_bom.html#gen6\n[2] http://www.unicode.org/versions/Unicode10.0.0/ch03.pdf\n Section 3.10, D98, page 132\n[3] https://encoding.spec.whatwg.org/#utf-16le\n\nSigned-off-by: Lars Schneider \n---\n utf8.c | 13 +++++++++++++\n utf8.h | 16 ++++++++++++++++\n 2 files changed, 29 insertions(+)\n\ndiff --git a/utf8.c b/utf8.c\nindex 776660ee12..1978d6c42a 100644\n--- a/utf8.c\n+++ b/utf8.c\n@@ -562,6 +562,19 @@ int has_prohibited_utf_bom(const char *enc, const char *data, size_t len)\n \t);\n }\n \n+int has_missing_utf_bom(const char *enc, const char *data, size_t len)\n+{\n+\treturn (\n+\t !strcmp(enc, \"UTF-16\") &&\n+\t !(has_bom_prefix(data, len, utf16_be_bom, sizeof(utf16_be_bom)) ||\n+\t has_bom_prefix(data, len, utf16_le_bom, sizeof(utf16_le_bom)))\n+\t) || (\n+\t !strcmp(enc, \"UTF-32\") &&\n+\t !(has_bom_prefix(data, len, utf32_be_bom, sizeof(utf32_be_bom)) ||\n+\t has_bom_prefix(data, len, utf32_le_bom, sizeof(utf32_le_bom)))\n+\t);\n+}\n+\n /*\n * Returns first character length in bytes for multi-byte `text` according to\n * `encoding`.\ndiff --git a/utf8.h b/utf8.h\nindex 4711429af9..26b5e91852 100644\n--- a/utf8.h\n+++ b/utf8.h\n@@ -79,4 +79,20 @@ void strbuf_utf8_align(struct strbuf *buf, align_type position, unsigned int wid\n */\n int has_prohibited_utf_bom(const char *enc, const char *data, size_t len);\n \n+/*\n+ * If the endianness is not defined in the encoding name, then we\n+ * require a BOM. The function returns true if a required BOM is missing.\n+ *\n+ * The Unicode standard instructs to assume big-endian if there\n+ * in no BOM for UTF-16/32 [1][2]. However, the W3C/WHATWG\n+ * encoding standard used in HTML5 recommends to assume\n+ * little-endian to \"deal with deployed content\" [3].\n+ *\n+ * [1] http://unicode.org/faq/utf_bom.html#gen6\n+ * [2] http://www.unicode.org/versions/Unicode10.0.0/ch03.pdf\n+ * Section 3.10, D98, page 132\n+ * [3] https://encoding.spec.whatwg.org/#utf-16le\n+ */\n+int has_missing_utf_bom(const char *enc, const char *data, size_t len);\n+\n #endif\n-- \n2.15.1\n\n"},{"id":"335538","messageId":"20171229155642.GA23935@tor.lan","threadId":"47511","inReplyTo":"20171229152222.39680-2-lars.schneider@autodesk.com","subject":"Re: [PATCH v2 1/5] strbuf: add xstrdup_toupper()","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2017-12-29T15:56:42Z","receivedAt":"2017-12-29T15:57:03Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Fri, Dec 29, 2017 at 04:22:18PM +0100, lars.schneider@autodesk.com wrote:\n> From: Lars Schneider \n> \n> Create a copy of an existing string and make all characters upper case.\n> Similar xstrdup_tolower().\n> \n> This function is used in a subsequent commit.\n> \n> Signed-off-by: Lars Schneider \n> ---\n> strbuf.c | 13 +++++++++++++\n> strbuf.h | 1 +\n> 2 files changed, 14 insertions(+)\n> \n> diff --git a/strbuf.c b/strbuf.c\n> index 323c49ceb3..54276e96e7 100644\n> --- a/strbuf.c\n> +++ b/strbuf.c\n> @@ -760,6 +760,19 @@ char *xstrdup_tolower(const char *string)\n> \treturn result;\n> }\n> \n> +char *xstrdup_toupper(const char *string)\n> +{\n> +\tchar *result;\n> +\tsize_t len, i;\n> +\n> +\tlen = strlen(string);\n> +\tresult = xmallocz(len);\n> +\tfor (i = 0; i < len; i++)\n> +\t\tresult[i] = toupper(string[i]);\n> +\tresult[i] = '\\0';\n ^^^^^^^^^^^^^^^^\n\tIsn't this already done by xmallocz()\n\n> +\treturn result;\n> +}\n> +\n> char *xstrvfmt(const char *fmt, va_list ap)\n> {\n> \tstruct strbuf buf = STRBUF_INIT;\n> diff --git a/strbuf.h b/strbuf.h\n> index 0a74acb236..2bc148526f 100644\n> --- a/strbuf.h\n> +++ b/strbuf.h\n> @@ -616,6 +616,7 @@ __attribute__((format (printf,2,3)))\n> extern int fprintf_ln(FILE *fp, const char *fmt, ...);\n> \n> char *xstrdup_tolower(const char *);\n> +char *xstrdup_toupper(const char *);\n> \n> /**\n> * Create a newly allocated string using printf format. You can do this easily\n> -- \n> 2.15.1\n> \n"},{"id":"335540","messageId":"0D8114C4-BC45-43D3-AA33-8954AEC9EA10@gmail.com","threadId":"47511","inReplyTo":"20171229155642.GA23935@tor.lan","subject":"Re: [PATCH v2 1/5] strbuf: add xstrdup_toupper()","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-12-29T16:55:57Z","receivedAt":"2017-12-29T16:56:05Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 29 Dec 2017, at 16:56, Torsten Bögershausen wrote:\n> \n> On Fri, Dec 29, 2017 at 04:22:18PM +0100, lars.schneider@autodesk.com wrote:\n>> From: Lars Schneider \n>> \n>> Create a copy of an existing string and make all characters upper case.\n>> Similar xstrdup_tolower().\n>> \n>> This function is used in a subsequent commit.\n>> \n>> Signed-off-by: Lars Schneider \n>> ---\n>> strbuf.c | 13 +++++++++++++\n>> strbuf.h | 1 +\n>> 2 files changed, 14 insertions(+)\n>> \n>> diff --git a/strbuf.c b/strbuf.c\n>> index 323c49ceb3..54276e96e7 100644\n>> --- a/strbuf.c\n>> +++ b/strbuf.c\n>> @@ -760,6 +760,19 @@ char *xstrdup_tolower(const char *string)\n>> \treturn result;\n>> }\n>> \n>> +char *xstrdup_toupper(const char *string)\n>> +{\n>> +\tchar *result;\n>> +\tsize_t len, i;\n>> +\n>> +\tlen = strlen(string);\n>> +\tresult = xmallocz(len);\n>> +\tfor (i = 0; i < len; i++)\n>> +\t\tresult[i] = toupper(string[i]);\n>> +\tresult[i] = '\\0';\n> ^^^^^^^^^^^^^^^^\n> \tIsn't this already done by xmallocz()\n\nI copied that code from xstrdup_tolower().\n\nThe original implementation [1] and its refactored version [2]\nused xmalloc(). Later on xmallocz [3] was introduced.\n\n[3] states \"we can stop manually placing NUL at the end of the\nallocated buffer. But that's only safe if it's clear that\nthe contents will always fill the buffer.\"\n\nAs far as I understand it, the content should always fill the\nbuffer in the upper/lower case conversion. Therefore, I agree \nwith you that the assignment is not necessary.\n\n- Lars\n\n\n[1] d4770964d5 (config: \"git config --get-urlmatch\" parses section..key, 2013-07-31)\n[2] 88d5a6f6cd (daemon/config: factor out duplicate xstrdup_tolower, 2014-05-22)\n[3] 3733e69464 (use xmallocz to avoid size arithmetic, 2016-02-22)\n\n"},{"id":"335542","messageId":"20171229172858.GB23935@tor.lan","threadId":"47511","inReplyTo":"20171229152222.39680-5-lars.schneider@autodesk.com","subject":"Re: [PATCH v2 4/5] convert: add support for 'checkout-encoding' attribute","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2017-12-29T17:28:58Z","receivedAt":"2017-12-29T17:29:23Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"I probably need to look at convert.c more closer, some other comments inline.\n\n\nOn Fri, Dec 29, 2017 at 04:22:21PM +0100, lars.schneider@autodesk.com wrote:\n> From: Lars Schneider \n> \n> Git and its tools (e.g. git diff) expect all text files in UTF-8\n> encoding. Git will happily accept content in all other encodings, too,\n> but it might not be able to process the text (e.g. viewing diffs or\n> changing line endings).\n\nUTF-8 is too strict, the text from below is more correct:\n +Git recognizes files encoded with ASCII or one of its supersets (e.g.\n +UTF-8 or ISO-8859-1) as text files. All other encodings are usually\n +interpreted as binary and consequently built-in Git text processing\n +tools (e.g. 'git diff') as well as most Git web front ends do not\n +visualize the content.\n\n\n> \n> Add an attribute to tell Git what encoding the user has defined for a\n> given file. If the content is added to the index, then Git converts the\n> content to a canonical UTF-8 representation. On checkout Git will\n\nMinor question about \"canonical\":\nWould this mean the same ?\n ...then Git converts the content into UTF-8.\n\n\n> reverse the conversion.\n> \n> Signed-off-by: Lars Schneider \n> ---\n> Documentation/gitattributes.txt | 59 ++++++++++++\n> apply.c | 2 +-\n> blame.c | 2 +-\n> combine-diff.c | 2 +-\n> convert.c | 196 ++++++++++++++++++++++++++++++++++++++-\n> convert.h | 8 +-\n> diff.c | 2 +-\n> sha1_file.c | 5 +-\n> t/t0028-checkout-encoding.sh | 197 ++++++++++++++++++++++++++++++++++++++++\n> 9 files changed, 460 insertions(+), 13 deletions(-)\n> create mode 100755 t/t0028-checkout-encoding.sh\n> \n> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\n> index 30687de81a..0039bd38c3 100644\n> --- a/Documentation/gitattributes.txt\n> +++ b/Documentation/gitattributes.txt\n> @@ -272,6 +272,65 @@ few exceptions. Even though...\n> catch potential problems early, safety triggers.\n> \n> \n> +`checkout-encoding`\n> +^^^^^^^^^^^^^^^^^^^\n> +\n> +Git recognizes files encoded with ASCII or one of its supersets (e.g.\n> +UTF-8 or ISO-8859-1) as text files. All other encodings are usually\n> +interpreted as binary and consequently built-in Git text processing\n> +tools (e.g. 'git diff') as well as most Git web front ends do not\n> +visualize the content.\n> +\n\n> +In these cases you can teach Git the encoding of a file in the working\n teach ? tell ? \n> +directory with the `checkout-encoding` attribute. If a file with this\n> +attributes is added to Git, then Git reencodes the content from the\n> +specified encoding to UTF-8 and stores the result in its internal data\n> +structure.\n\nMinor Q:\n> its internal data structure.\nShould we simply write \"stores the result in the index\" ?\n\n> On checkout the content is encoded back to the specified\n> +encoding.\n\n> +\n> +Please note that using the `checkout-encoding` attribute has a number\n> +of drawbacks:\n> +\n> +- Reencoding content to non-UTF encodings (e.g. SHIFT-JIS) can cause\n> + errors as the conversion might not be round trip safe.\n> +\n> +- Reencoding content requires resources that might slow down certain\n> + Git operations (e.g 'git checkout' or 'git add').\n> +\n> +- Git clients that do not support the `checkout-encoding` attribute or\n> + the used encoding will checkout the respective files as UTF-8 encoded.\n> + That means the content appears to be different which could cause\n> + trouble. Affected clients are older Git versions and alternative Git\n> + implementations such as JGit or libgit2 (as of January 2018).\n> +\n> +Use the `checkout-encoding` attribute only if you cannot store a file in\n> +UTF-8 encoding and if you want Git to be able to process the content as\n> +text.\n> +\n\nI would maybe rephrase a little bit (first things first):\n\nPlease note that using the `checkout-encoding` attribute may have a number\nof pitfalls:\n\n- Git clients that do not support the `checkout-encoding` attribute\n will checkout the respective files as UTF-8 encoded, which typically\n causes trouble.\n Escpecialy when older Git versions are used or alternative Git\n implementations such as JGit or libgit2 (as of January 2018).\n\n- Reencoding content to non-UTF encodings (e.g. SHIFT-JIS) can cause\n errors as the conversion might not be round trip safe.\n\n- Reencoding content requires resources that might slow down certain\n Git operations (e.g 'git checkout' or 'git add').\n\nUse the `checkout-encoding` attribute only if you cannot store a file in\nUTF-8 encoding and if you want Git to be able to process the content as\ntext.\n\n-----\nSide question: What happens if \"the used encoding\" is not supported by\nthe underlying iconv lib ?\nWill Git fail, delete the file from the working tree ?\nThat may be worth to mention. (And I need to do the code-review)\n\n> +Use the following attributes if your '*.txt' files are UTF-16 encoded\n> +with byte order mark (BOM) and you want Git to perform automatic line\n> +ending conversion based on your platform.\n> +\n> +------------------------\n> +*.txt\t\ttext checkout-encoding=UTF-16\n> +------------------------\n> +\n> +Use the following attributes if your '*.txt' files are UTF-16 little\n> +endian encoded without BOM and you want Git to use Windows line endings\n> +in the working directory.\n> +\n> +------------------------\n> +*.txt \t\tcheckout-encoding=UTF-16LE text eol=CRLF\n> +------------------------\n> +\n> +You can get a list of all available encodings on your platform with the\n> +following command:\n> +\n> +------------------------\n> +iconv --list\n> +------------------------\n> +\n> +\n> `ident`\n> ^^^^^^^\n> \n> diff --git a/apply.c b/apply.c\n> index 321a9fa68d..c4bd5cf1f2 100644\n> --- a/apply.c\n> +++ b/apply.c\n> @@ -2281,7 +2281,7 @@ static int read_old_data(struct stat *st, struct patch *patch,\n> \t\t * should never look at the index when explicit crlf option\n> \t\t * is given.\n> \t\t */\n> -\t\tconvert_to_git(NULL, path, buf->buf, buf->len, buf, safe_crlf);\n> +\t\tconvert_to_git(NULL, path, buf->buf, buf->len, buf, safe_crlf, 0);\n\nHm. Do we really need another parameter here?\nThe only caller that needs it (and sets it to 1) is sha1_file.c.\nWhen an invalid encoding/content is used, it should be safe to die() always?\nJust loud thinking..\n\nIf really needed, it may need less changes in the code base, if a new function\nconvert_to_git_or_die() is defined, which is called by convert_to_git()\n\n(and the other alternative would be to convert safe_crlf into a bitmap,\n see https://public-inbox.org/git/20171229132828.17594-1-tboegi@web.de/T/#u\n what do you think ?)\n\n\n> \t\treturn 0;\n> \tdefault:\n> \t\treturn -1;\n> diff --git a/blame.c b/blame.c\n> index 2893f3c103..388b66897b 100644\n> --- a/blame.c\n> +++ b/blame.c\n> @@ -229,7 +229,7 @@ static struct commit *fake_working_tree_commit(struct diff_options *opt,\n> \t\tif (strbuf_read(&buf, 0, 0) < 0)\n> \t\t\tdie_errno(\"failed to read from stdin\");\n> \t}\n> -\tconvert_to_git(&the_index, path, buf.buf, buf.len, &buf, 0);\n> +\tconvert_to_git(&the_index, path, buf.buf, buf.len, &buf, 0, 0);\n> \torigin->file.ptr = buf.buf;\n> \torigin->file.size = buf.len;\n> \tpretend_sha1_file(buf.buf, buf.len, OBJ_BLOB, origin->blob_oid.hash);\n> diff --git a/combine-diff.c b/combine-diff.c\n> index 2505de119a..4555e49b5f 100644\n> --- a/combine-diff.c\n> +++ b/combine-diff.c\n> @@ -1053,7 +1053,7 @@ static void show_patch_diff(struct combine_diff_path *elem, int num_parent,\n> \t\t\tif (is_file) {\n> \t\t\t\tstruct strbuf buf = STRBUF_INIT;\n> \n> -\t\t\t\tif (convert_to_git(&the_index, elem->path, result, len, &buf, safe_crlf)) {\n> +\t\t\t\tif (convert_to_git(&the_index, elem->path, result, len, &buf, safe_crlf, 0)) {\n> \t\t\t\t\tfree(result);\n> \t\t\t\t\tresult = strbuf_detach(&buf, &len);\n> \t\t\t\t\tresult_size = len;\n> diff --git a/convert.c b/convert.c\n> index 20d7ab67bd..fc8c96b670 100644\n> --- a/convert.c\n> +++ b/convert.c\n> @@ -7,6 +7,7 @@\n> #include \"sigchain.h\"\n> #include \"pkt-line.h\"\n> #include \"sub-process.h\"\n> +#include \"utf8.h\"\n> \n> /*\n> * convert.c - convert a file when checking it out and checking it in.\n> @@ -256,6 +257,147 @@ static int will_convert_lf_to_crlf(size_t len, struct text_stat *stats,\n> \n> }\n> \n> +static struct encoding {\n> +\tconst char *name;\n> +\tstruct encoding *next;\n> +} *encoding, **encoding_tail;\n> +static const char *default_encoding = \"UTF-8\";\n> +\n> +static int encode_to_git(const char *path, const char *src, size_t src_len,\n> +\t\t\t struct strbuf *buf, struct encoding *enc, int write_obj)\n> +{\n> +\tchar *dst;\n> +\tint dst_len;\n> +\n> +\t/*\n> +\t * No encoding is specified or there is nothing to encode.\n> +\t * Tell the caller that the content was not modified.\n> +\t */\n> +\tif (!enc || (src && !src_len))\n> +\t\treturn 0;\n> +\n> +\t/*\n> +\t * Looks like we got called from \"would_convert_to_git()\".\n> +\t * This means Git wants to know if it would encode (= modify!)\n> +\t * the content. Let's answer with \"yes\", since an encoding was\n> +\t * specified.\n> +\t */\n> +\tif (!buf && !src)\n> +\t\treturn 1;\n> +\n> +\tif (has_prohibited_utf_bom(enc->name, src, src_len)) {\n> +\t\tconst char *error_msg = _(\n> +\t\t\t\"BOM is prohibited for '%s' if encoded as %s\");\n> +\t\tconst char *advise_msg = _(\n> +\t\t\t\"You told Git to treat '%s' as %s. A byte order mark \"\n> +\t\t\t\"(BOM) is prohibited with this encoding. Either use \"\n> +\t\t\t\"%.6s as checkout encoding or remove the BOM from the \"\n> +\t\t\t\"file.\");\n> +\n> +\t\tadvise(advise_msg, path, enc->name, enc->name, enc->name);\n> +\t\tif (write_obj)\n> +\t\t\tdie(error_msg, path, enc->name);\n> +\t\telse\n> +\t\t\terror(error_msg, path, enc->name);\n\nAs said before, I would just die(). Or do I miss something ?\nBeing strict with BOMs seams like a good idea.\n\n> +\n> +\n> +\t} else if (has_missing_utf_bom(enc->name, src, src_len)) {\n> +\t\tconst char *error_msg = _(\n> +\t\t\t\"BOM is required for '%s' if encoded as %s\");\n> +\t\tconst char *advise_msg = _(\n> +\t\t\t\"You told Git to treat '%s' as %s. A byte order mark \"\n> +\t\t\t\"(BOM) is required with this encoding. Either use \"\n> +\t\t\t\"%sBE/%sLE as checkout encoding or add a BOM to the \"\n> +\t\t\t\"file.\");\n> +\t\tadvise(advise_msg, path, enc->name, enc->name, enc->name);\n> +\t\tif (write_obj)\n> +\t\t\tdie(error_msg, path, enc->name);\n> +\t\telse\n> +\t\t\terror(error_msg, path, enc->name);\n> +\t}\n> +\n> +\tdst = reencode_string_len(src, src_len, default_encoding, enc->name,\n> +\t\t\t\t &dst_len);\n> +\tif (!dst) {\n> +\t\t/*\n> +\t\t * We could add the blob \"as-is\" to Git. However, on checkout\n> +\t\t * we would try to reencode to the original encoding. This\n> +\t\t * would fail and we would leave the user with a messed-up\n> +\t\t * working tree. Let's try to avoid this by screaming loud.\n> +\t\t */\n> +\t\tconst char* msg = _(\"failed to encode '%s' from %s to %s\");\n> +\t\tif (write_obj)\n> +\t\t\tdie(msg, path, enc->name, default_encoding);\n> +\t\telse\n> +\t\t\terror(msg, path, enc->name, default_encoding);\n> +\t}\n> +\n> +\t/*\n> +\t * UTF supports lossless round tripping [1]. UTF to other encoding are\n> +\t * mostly round trip safe as Unicode aims to be a superset of all other\n> +\t * character encodings. However, the SHIFT-JIS (Japanese character set)\n> +\t * is an exception as some codes are not round trip safe [2].\n> +\t *\n> +\t * Reverse the transformation of 'dst' and check the result with 'src'\n> +\t * if content is written to Git. This ensures no information is lost\n> +\t * during conversion to/from UTF-8.\n> +\t *\n> +\t * Please note, the code below is not tested because I was not able to\n> +\t * generate a faulty round trip without iconv error.\n> +\t *\n> +\t * [1] http://unicode.org/faq/utf_bom.html#gen2\n> +\t * [2] https://support.microsoft.com/en-us/help/170559/prb-conversion-problem-between-shift-jis-and-unicode\n> +\t */\n> +\tif (write_obj && !strcmp(enc->name, \"SHIFT-JIS\")) {\n> +\t\tchar *re_src;\n> +\t\tint re_src_len;\n> +\n> +\t\tre_src = reencode_string_len(dst, dst_len,\n> +\t\t\t\t\t enc->name, default_encoding,\n> +\t\t\t\t\t &re_src_len);\n> +\n> +\t\tif (!re_src || src_len != re_src_len ||\n> +\t\t memcmp(src, re_src, src_len)) {\n> +\t\t\tconst char* msg = _(\"encoding '%s' from %s to %s and \"\n> +\t\t\t\t\t \"back is not the same\");\n> +\t\t\tif (write_obj)\n> +\t\t\t\tdie(msg, path, enc->name, default_encoding);\n> +\t\t\telse\n> +\t\t\t\terror(msg, path, enc->name, default_encoding);\n> +\t\t}\n> +\n> +\t\tfree(re_src);\n> +\t}\n> +\n> +\tstrbuf_attach(buf, dst, dst_len, dst_len + 1);\n> +\treturn 1;\n> +}\n> +\n> +static int encode_to_worktree(const char *path, const char *src, size_t src_len,\n> +\t\t\t struct strbuf *buf, struct encoding *enc)\n> +{\n> +\tchar *dst;\n> +\tint dst_len;\n> +\n> +\t/*\n> +\t * No encoding is specified or there is nothing to encode.\n> +\t * Tell the caller that the content was not modified.\n> +\t */\n> +\tif (!enc || (src && !src_len))\n> +\t\treturn 0;\n> +\n> +\tdst = reencode_string_len(src, src_len, enc->name, default_encoding,\n> +\t\t\t\t &dst_len);\n> +\tif (!dst) {\n> +\t\terror(\"failed to encode '%s' from %s to %s\",\n> +\t\t\tpath, enc->name, default_encoding);\n> +\t\treturn 0;\n> +\t}\n> +\n> +\tstrbuf_attach(buf, dst, dst_len, dst_len + 1);\n> +\treturn 1;\n> +}\n> +\n> static int crlf_to_git(const struct index_state *istate,\n> \t\t const char *path, const char *src, size_t len,\n> \t\t struct strbuf *buf,\n> @@ -969,6 +1111,31 @@ static int ident_to_worktree(const char *path, const char *src, size_t len,\n> \treturn 1;\n> }\n> \n> +static struct encoding *git_path_check_encoding(struct attr_check_item *check)\n> +{\n> +\tconst char *value = check->value;\n> +\tstruct encoding *enc;\n> +\n> +\tif (ATTR_TRUE(value) || ATTR_FALSE(value) || ATTR_UNSET(value) ||\n> +\t !strlen(value))\n> +\t\treturn NULL;\n> +\n> +\tfor (enc = encoding; enc; enc = enc->next)\n> +\t\tif (!strcasecmp(value, enc->name))\n> +\t\t\treturn enc;\n> +\n> +\t/* Don't encode to the default encoding */\n> +\tif (!strcasecmp(value, default_encoding))\n> +\t\treturn NULL;\n> +\n> +\tenc = xcalloc(1, sizeof(struct convert_driver));\n> +\tenc->name = xstrdup_toupper(value); /* aways use upper case names! */\n> +\t*encoding_tail = enc;\n> +\tencoding_tail = &(enc->next);\n> +\n> +\treturn enc;\n> +}\n> +\n> static enum crlf_action git_path_check_crlf(struct attr_check_item *check)\n> {\n> \tconst char *value = check->value;\n> @@ -1024,6 +1191,7 @@ struct conv_attrs {\n> \tenum crlf_action attr_action; /* What attr says */\n> \tenum crlf_action crlf_action; /* When no attr is set, use core.autocrlf */\n> \tint ident;\n> +\tstruct encoding *checkout_encoding; /* Supported encoding or default encoding if NULL */\n> };\n> \n> static void convert_attrs(struct conv_attrs *ca, const char *path)\n> @@ -1032,8 +1200,10 @@ static void convert_attrs(struct conv_attrs *ca, const char *path)\n> \n> \tif (!check) {\n> \t\tcheck = attr_check_initl(\"crlf\", \"ident\", \"filter\",\n> -\t\t\t\t\t \"eol\", \"text\", NULL);\n> +\t\t\t\t\t \"eol\", \"text\", \"checkout-encoding\",\n> +\t\t\t\t\t NULL);\n> \t\tuser_convert_tail = &user_convert;\n> +\t\tencoding_tail = &encoding;\n> \t\tgit_config(read_convert_config, NULL);\n> \t}\n> \n> @@ -1055,6 +1225,7 @@ static void convert_attrs(struct conv_attrs *ca, const char *path)\n> \t\t\telse if (eol_attr == EOL_CRLF)\n> \t\t\t\tca->crlf_action = CRLF_TEXT_CRLF;\n> \t\t}\n> +\t\tca->checkout_encoding = git_path_check_encoding(ccheck + 5);\n> \t} else {\n> \t\tca->drv = NULL;\n> \t\tca->crlf_action = CRLF_UNDEFINED;\n> @@ -1120,7 +1291,7 @@ const char *get_convert_attr_ascii(const char *path)\n> \n> int convert_to_git(const struct index_state *istate,\n> \t\t const char *path, const char *src, size_t len,\n> - struct strbuf *dst, enum safe_crlf checksafe)\n> +\t\t struct strbuf *dst, enum safe_crlf checksafe, int write_obj)\n> {\n> \tint ret = 0;\n> \tstruct conv_attrs ca;\n> @@ -1135,6 +1306,13 @@ int convert_to_git(const struct index_state *istate,\n> \t\tsrc = dst->buf;\n> \t\tlen = dst->len;\n> \t}\n> +\n> +\tret |= encode_to_git(path, src, len, dst, ca.checkout_encoding, write_obj);\n> +\tif (ret && dst) {\n> +\t\tsrc = dst->buf;\n> +\t\tlen = dst->len;\n> +\t}\n> +\n> \tif (checksafe != SAFE_CRLF_KEEP_CRLF) {\n> \t\tret |= crlf_to_git(istate, path, src, len, dst, ca.crlf_action, checksafe);\n> \t\tif (ret && dst) {\n> @@ -1147,7 +1325,7 @@ int convert_to_git(const struct index_state *istate,\n> \n> void convert_to_git_filter_fd(const struct index_state *istate,\n> \t\t\t const char *path, int fd, struct strbuf *dst,\n> -\t\t\t enum safe_crlf checksafe)\n> +\t\t\t enum safe_crlf checksafe, int write_obj)\n> {\n> \tstruct conv_attrs ca;\n> \tconvert_attrs(&ca, path);\n> @@ -1158,6 +1336,7 @@ void convert_to_git_filter_fd(const struct index_state *istate,\n> \tif (!apply_filter(path, NULL, 0, fd, dst, ca.drv, CAP_CLEAN, NULL))\n> \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n> \n> +\tencode_to_git(path, dst->buf, dst->len, dst, ca.checkout_encoding, write_obj);\n> \tcrlf_to_git(istate, path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);\n> \tident_to_git(path, dst->buf, dst->len, dst, ca.ident);\n> }\n> @@ -1189,6 +1368,12 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n> \t\t}\n> \t}\n> \n> +\tret |= encode_to_worktree(path, src, len, dst, ca.checkout_encoding);\n> +\tif (ret) {\n> +\t\tsrc = dst->buf;\n> +\t\tlen = dst->len;\n> +\t}\n> +\n> \tret_filter = apply_filter(\n> \t\tpath, src, len, -1, dst, ca.drv, CAP_SMUDGE, dco);\n> \tif (!ret_filter && ca.drv && ca.drv->required)\n> @@ -1217,7 +1402,7 @@ int renormalize_buffer(const struct index_state *istate, const char *path,\n> \t\tsrc = dst->buf;\n> \t\tlen = dst->len;\n> \t}\n> -\treturn ret | convert_to_git(istate, path, src, len, dst, SAFE_CRLF_RENORMALIZE);\n> +\treturn ret | convert_to_git(istate, path, src, len, dst, SAFE_CRLF_RENORMALIZE, 0);\n> }\n> \n> /*****************************************************************\n> @@ -1655,6 +1840,9 @@ struct stream_filter *get_stream_filter(const char *path, const unsigned char *s\n> \tif (ca.drv && (ca.drv->process || ca.drv->smudge || ca.drv->clean))\n> \t\treturn NULL;\n> \n> +\tif (ca.checkout_encoding)\n> +\t\treturn NULL;\n> +\n> \tif (ca.crlf_action == CRLF_AUTO || ca.crlf_action == CRLF_AUTO_CRLF)\n> \t\treturn NULL;\n> \n> diff --git a/convert.h b/convert.h\n> index 4f2da225a8..9e4e884ec1 100644\n> --- a/convert.h\n> +++ b/convert.h\n> @@ -66,7 +66,8 @@ extern const char *get_convert_attr_ascii(const char *path);\n> /* returns 1 if *dst was used */\n> extern int convert_to_git(const struct index_state *istate,\n> \t\t\t const char *path, const char *src, size_t len,\n> -\t\t\t struct strbuf *dst, enum safe_crlf checksafe);\n> +\t\t\t struct strbuf *dst, enum safe_crlf checksafe,\n> +\t\t\t int write_obj);\n> extern int convert_to_working_tree(const char *path, const char *src,\n> \t\t\t\t size_t len, struct strbuf *dst);\n> extern int async_convert_to_working_tree(const char *path, const char *src,\n> @@ -79,13 +80,14 @@ extern int renormalize_buffer(const struct index_state *istate,\n> static inline int would_convert_to_git(const struct index_state *istate,\n> \t\t\t\t const char *path)\n> {\n> -\treturn convert_to_git(istate, path, NULL, 0, NULL, 0);\n> +\treturn convert_to_git(istate, path, NULL, 0, NULL, 0, 0);\n> }\n> /* Precondition: would_convert_to_git_filter_fd(path) == true */\n> extern void convert_to_git_filter_fd(const struct index_state *istate,\n> \t\t\t\t const char *path, int fd,\n> \t\t\t\t struct strbuf *dst,\n> -\t\t\t\t enum safe_crlf checksafe);\n> +\t\t\t\t enum safe_crlf checksafe,\n> +\t\t\t\t int write_obj);\n> extern int would_convert_to_git_filter_fd(const char *path);\n> \n> /*****************************************************************\n> diff --git a/diff.c b/diff.c\n> index 2ebe2227b4..16ca0bf0df 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -3599,7 +3599,7 @@ int diff_populate_filespec(struct diff_filespec *s, unsigned int flags)\n> \t\t/*\n> \t\t * Convert from working tree format to canonical git format\n> \t\t */\n> -\t\tif (convert_to_git(&the_index, s->path, s->data, s->size, &buf, crlf_warn)) {\n> +\t\tif (convert_to_git(&the_index, s->path, s->data, s->size, &buf, crlf_warn, 0)) {\n> \t\t\tsize_t size = 0;\n> \t\t\tmunmap(s->data, s->size);\n> \t\t\ts->should_munmap = 0;\n> diff --git a/sha1_file.c b/sha1_file.c\n> index afe4b90f6e..75800248d2 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> @@ -1694,7 +1694,8 @@ static int index_mem(struct object_id *oid, void *buf, size_t size,\n> \tif ((type == OBJ_BLOB) && path) {\n> \t\tstruct strbuf nbuf = STRBUF_INIT;\n> \t\tif (convert_to_git(&the_index, path, buf, size, &nbuf,\n> -\t\t\t\t get_safe_crlf(flags))) {\n> +\t\t\t\t get_safe_crlf(flags),\n> +\t\t\t\t write_object)) {\n> \t\t\tbuf = strbuf_detach(&nbuf, &size);\n> \t\t\tre_allocated = 1;\n> \t\t}\n> @@ -1728,7 +1729,7 @@ static int index_stream_convert_blob(struct object_id *oid, int fd,\n> \tassert(would_convert_to_git_filter_fd(path));\n> \n> \tconvert_to_git_filter_fd(&the_index, path, fd, &sbuf,\n> -\t\t\t\t get_safe_crlf(flags));\n> +\t\t\t\t get_safe_crlf(flags), write_object);\n> \n> \tif (write_object)\n> \t\tret = write_sha1_file(sbuf.buf, sbuf.len, typename(OBJ_BLOB),\n> diff --git a/t/t0028-checkout-encoding.sh b/t/t0028-checkout-encoding.sh\n> new file mode 100755\n> index 0000000000..1a329ab933\n> --- /dev/null\n> +++ b/t/t0028-checkout-encoding.sh\n> @@ -0,0 +1,197 @@\n> +#!/bin/sh\n> +\n> +test_description='checkout-encoding conversion via gitattributes'\n> +\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'setup test repo' '\n> +\n> +\ttext=\"hallo there!\\ncan you read me?\" &&\n\nIs this portable ? (the \"\\n\")\n\n\n> +\n> +\techo \"*.utf16 text checkout-encoding=utf-16\" >.gitattributes &&\n> +\n> +\tprintf \"$text\" >test.utf8.raw &&\n> +\tprintf \"$text\" | iconv -f UTF-8 -t UTF-16 >test.utf16.raw &&\n> +\tcp test.utf16.raw test.utf16 &&\n> +\n> +\tgit add .gitattributes test.utf16 &&\n> +\tgit commit -m initial\n> +'\n> +\n> +test_expect_success 'ensure UTF-8 is stored in Git' '\n> +\tgit cat-file -p :test.utf16 >test.utf16.git &&\n> +\ttest_cmp_bin test.utf8.raw test.utf16.git &&\n> +\trm test.utf8.raw test.utf16.git\n> +'\n> +\n> +test_expect_success 're-encode to UTF-16 on checkout' '\n> +\trm test.utf16 &&\n> +\tgit checkout test.utf16 &&\n> +\ttest_cmp_bin test.utf16.raw test.utf16 &&\n> +\n> +\t# cleanup\n> +\trm test.utf16.raw\n> +'\n> +\n> +test_expect_success 'check prohibited UTF BOM' '\n> +\tprintf \"\\0a\\0b\\0c\" >nobom.utf16be.raw &&\n> +\tprintf \"a\\0b\\0c\\0\" >nobom.utf16le.raw &&\n> +\tprintf \"\\376\\777\\0a\\0b\\0c\" >bebom.utf16be.raw &&\n> +\tprintf \"\\777\\376a\\0b\\0c\\0\" >lebom.utf16le.raw &&\n> +\n> +\tprintf \"\\0\\0\\0a\\0\\0\\0b\\0\\0\\0c\" >nobom.utf32be.raw &&\n> +\tprintf \"a\\0\\0\\0b\\0\\0\\0c\\0\\0\\0\" >nobom.utf32le.raw &&\n> +\tprintf \"\\0\\0\\376\\777\\0\\0\\0a\\0\\0\\0b\\0\\0\\0c\" >bebom.utf32be.raw &&\n> +\tprintf \"\\777\\376\\0\\0a\\0\\0\\0b\\0\\0\\0c\\0\\0\\0\" >lebom.utf32le.raw &&\n> +\n> +\techo \"*.utf16be text checkout-encoding=utf-16be\" >>.gitattributes &&\n> +\techo \"*.utf16le text checkout-encoding=utf-16le\" >>.gitattributes &&\n> +\techo \"*.utf32be text checkout-encoding=utf-32be\" >>.gitattributes &&\n> +\techo \"*.utf32le text checkout-encoding=utf-32le\" >>.gitattributes &&\n> +\n> +\t# Here we add a UTF-16 files with BOM (big-endian and little-endian)\n> +\t# but we tell Git to treat it as UTF-16BE/UTF-16LE. In these cases\n> +\t# the BOM is prohibited.\n> +\tcp bebom.utf16be.raw bebom.utf16be &&\n> +\ttest_must_fail git add bebom.utf16be 2>err.out &&\n> +\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-16BE\" err.out &&\n> +\n> +\tcp lebom.utf16le.raw lebom.utf16be &&\n> +\ttest_must_fail git add lebom.utf16be 2>err.out &&\n> +\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-16BE\" err.out &&\n> +\n> +\tcp bebom.utf16be.raw bebom.utf16le &&\n> +\ttest_must_fail git add bebom.utf16le 2>err.out &&\n> +\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-16LE\" err.out &&\n> +\n> +\tcp lebom.utf16le.raw lebom.utf16le &&\n> +\ttest_must_fail git add lebom.utf16le 2>err.out &&\n> +\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-16LE\" err.out &&\n> +\n> +\t# ... and the same for UTF-32\n> +\tcp bebom.utf32be.raw bebom.utf32be &&\n> +\ttest_must_fail git add bebom.utf32be 2>err.out &&\n> +\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-32BE\" err.out &&\n> +\n> +\tcp lebom.utf32le.raw lebom.utf32be &&\n> +\ttest_must_fail git add lebom.utf32be 2>err.out &&\n> +\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-32BE\" err.out &&\n> +\n> +\tcp bebom.utf32be.raw bebom.utf32le &&\n> +\ttest_must_fail git add bebom.utf32le 2>err.out &&\n> +\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-32LE\" err.out &&\n> +\n> +\tcp lebom.utf32le.raw lebom.utf32le &&\n> +\ttest_must_fail git add lebom.utf32le 2>err.out &&\n> +\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-32LE\" err.out &&\n> +\n> +\t# cleanup\n> +\tgit reset --hard HEAD\n> +'\n> +\n> +test_expect_success 'check required UTF BOM' '\n> +\techo \"*.utf32 text checkout-encoding=utf-32\" >>.gitattributes &&\n> +\n> +\tcp nobom.utf16be.raw nobom.utf16 &&\n> +\ttest_must_fail git add nobom.utf16 2>err.out &&\n> +\ttest_i18ngrep \"fatal: BOM is required .* UTF-16\" err.out &&\n> +\n> +\tcp nobom.utf16le.raw nobom.utf16 &&\n> +\ttest_must_fail git add nobom.utf16 2>err.out &&\n> +\ttest_i18ngrep \"fatal: BOM is required .* UTF-16\" err.out &&\n> +\n> +\tcp nobom.utf32be.raw nobom.utf32 &&\n> +\ttest_must_fail git add nobom.utf32 2>err.out &&\n> +\ttest_i18ngrep \"fatal: BOM is required .* UTF-32\" err.out &&\n> +\n> +\tcp nobom.utf32le.raw nobom.utf32 &&\n> +\ttest_must_fail git add nobom.utf32 2>err.out &&\n> +\ttest_i18ngrep \"fatal: BOM is required .* UTF-32\" err.out &&\n> +\n> +\t# cleanup\n> +\trm nobom.utf16 nobom.utf32 &&\n> +\tgit reset --hard HEAD\n> +'\n> +\n> +test_expect_success 'eol conversion for UTF-16 encoded files on checkout' '\n> +\tprintf \"one\\ntwo\\nthree\\n\" >lf.utf8.raw &&\n> +\tprintf \"one\\r\\ntwo\\r\\nthree\\r\\n\" >crlf.utf8.raw &&\n> +\n> +\tcat lf.utf8.raw | iconv -f UTF-8 -t UTF-16 >lf.utf16.raw &&\n> +\tcat crlf.utf8.raw | iconv -f UTF-8 -t UTF-16 >crlf.utf16.raw &&\n> +\tcp crlf.utf16.raw eol.utf16 &&\n> +\n> +\tgit add eol.utf16 &&\n> +\tgit commit -m eol &&\n> +\n> +\t# UTF-16 with CRLF (Windows line endings)\n> +\trm eol.utf16 &&\n> +\tgit -c core.eol=crlf checkout eol.utf16 &&\n> +\ttest_cmp_bin crlf.utf16.raw eol.utf16 &&\n> +\n> +\t# UTF-16 with LF (Unix line endings)\n> +\trm eol.utf16 &&\n> +\tgit -c core.eol=lf checkout eol.utf16 &&\n> +\ttest_cmp_bin lf.utf16.raw eol.utf16 &&\n> +\n> +\trm crlf.utf16.raw crlf.utf8.raw lf.utf16.raw lf.utf8.raw &&\n> +\n> +\t# cleanup\n> +\tgit reset --hard HEAD^\n> +'\n> +\n> +test_expect_success 'check unsupported encodings' '\n> +\n> +\techo \"*.nothing text checkout-encoding=\" >>.gitattributes &&\n> +\tprintf \"nothing\" >t.nothing &&\n> +\tgit add t.nothing &&\n> +\n> +\techo \"*.garbage text checkout-encoding=garbage\" >>.gitattributes &&\n> +\tprintf \"garbage\" >t.garbage &&\n> +\ttest_must_fail git add t.garbage 2>err.out &&\n> +\ttest_i18ngrep \"fatal: failed to encode\" err.out &&\n> +\n> +\t# cleanup\n> +\trm err.out &&\n> +\tgit reset --hard HEAD\n> +'\n> +\n> +test_expect_success 'error if encoding round trip is not the same during refresh' '\n> +\tBEFORE_STATE=$(git rev-parse HEAD) &&\n> +\n> +\t# Skip the UTF-16 filter for the added file\n> +\t# This simulates a Git version that has no checkoutEncoding support\n> +\techo \"hallo\" >nonsense.utf16 &&\n> +\tTEST_HASH=$(git hash-object --no-filters -w nonsense.utf16) &&\n> +\tgit update-index --add --cacheinfo 100644 $TEST_HASH nonsense.utf16 &&\n> +\tCOMMIT=$(git commit-tree -p $(git rev-parse HEAD) -m \"plain commit\" $(git write-tree)) &&\n> +\tgit update-ref refs/heads/master $COMMIT &&\n> +\n> +\ttest_must_fail git checkout HEAD^ 2>err.out &&\n> +\ttest_i18ngrep \"error: .* overwritten by checkout:\" err.out &&\n> +\n> +\t# cleanup\n> +\trm err.out &&\n> +\tgit reset --hard $BEFORE_STATE\n> +'\n> +\n> +test_expect_success 'error if encoding garbage is already in Git' '\n> +\tBEFORE_STATE=$(git rev-parse HEAD) &&\n> +\n> +\t# Skip the UTF-16 filter for the added file\n> +\t# This simulates a Git version that has no checkoutEncoding support\n> +\tcp nobom.utf16be.raw nonsense.utf16 &&\n> +\tTEST_HASH=$(git hash-object --no-filters -w nonsense.utf16) &&\n> +\tgit update-index --add --cacheinfo 100644 $TEST_HASH nonsense.utf16 &&\n> +\tCOMMIT=$(git commit-tree -p $(git rev-parse HEAD) -m \"plain commit\" $(git write-tree)) &&\n> +\tgit update-ref refs/heads/master $COMMIT &&\n> +\n> +\tgit diff 2>err.out &&\n> +\ttest_i18ngrep \"error: BOM is required\" err.out &&\n> +\n> +\t# cleanup\n> +\trm err.out &&\n> +\tgit reset --hard $BEFORE_STATE\n> +'\n> +\n> +test_done\n> -- \n> 2.15.1\n> \n"},{"id":"335566","messageId":"99B51DE1-0825-41FC-8DD3-677BCF7B52A5@gmail.com","threadId":"47511","inReplyTo":"20171229172858.GB23935@tor.lan","subject":"Re: [PATCH v2 4/5] convert: add support for 'checkout-encoding' attribute","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-12-30T19:58:02Z","receivedAt":"2017-12-30T19:58:15Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 29 Dec 2017, at 18:28, Torsten Bögershausen wrote:\n> \n> I probably need to look at convert.c more closer, some other comments inline.\n> \n> \n> On Fri, Dec 29, 2017 at 04:22:21PM +0100, lars.schneider@autodesk.com wrote:\n>> From: Lars Schneider \n>> \n>> Git and its tools (e.g. git diff) expect all text files in UTF-8\n>> encoding. Git will happily accept content in all other encodings, too,\n>> but it might not be able to process the text (e.g. viewing diffs or\n>> changing line endings).\n> \n> UTF-8 is too strict, the text from below is more correct:\n> +Git recognizes files encoded with ASCII or one of its supersets (e.g.\n> +UTF-8 or ISO-8859-1) as text files. All other encodings are usually\n> +interpreted as binary and consequently built-in Git text processing\n> +tools (e.g. 'git diff') as well as most Git web front ends do not\n> +visualize the content.\n\nAgreed. I'll replace it.\n\n\n>> Add an attribute to tell Git what encoding the user has defined for a\n>> given file. If the content is added to the index, then Git converts the\n>> content to a canonical UTF-8 representation. On checkout Git will\n> \n> Minor question about \"canonical\":\n> Would this mean the same ?\n> ...then Git converts the content into UTF-8.\n\nI feel the word canonical makes sense in this context to express\nthat UTF-8 is not some randomly chosen encoding. AFAIK UTF-8 and \nLF are the canonical form for text in Git, no?\n\n\n>> +In these cases you can teach Git the encoding of a file in the working\n> teach ? tell ? \n\n\"tell\", agreed!\n\n\n>> +directory with the `checkout-encoding` attribute. If a file with this\n>> +attributes is added to Git, then Git reencodes the content from the\n>> +specified encoding to UTF-8 and stores the result in its internal data\n>> +structure.\n> \n> Minor Q:\n>> its internal data structure.\n> Should we simply write \"stores the result in the index\" ?\n\nThis text is targeted at the end user and I know a lot of end users\nthat are confused by the word \"index\". How about:\n\n ...stores the result in its internal data structure (\"the index\").\n\nWould that work?\n\n> \n>> On checkout the content is encoded back to the specified\n>> +encoding.\n> \n>> +\n>> +Please note that using the `checkout-encoding` attribute has a number\n>> +of drawbacks:\n>> +\n>> +- Reencoding content to non-UTF encodings (e.g. SHIFT-JIS) can cause\n>> + errors as the conversion might not be round trip safe.\n>> +\n>> +- Reencoding content requires resources that might slow down certain\n>> + Git operations (e.g 'git checkout' or 'git add').\n>> +\n>> +- Git clients that do not support the `checkout-encoding` attribute or\n>> + the used encoding will checkout the respective files as UTF-8 encoded.\n>> + That means the content appears to be different which could cause\n>> + trouble. Affected clients are older Git versions and alternative Git\n>> + implementations such as JGit or libgit2 (as of January 2018).\n>> +\n>> +Use the `checkout-encoding` attribute only if you cannot store a file in\n>> +UTF-8 encoding and if you want Git to be able to process the content as\n>> +text.\n>> +\n> \n> I would maybe rephrase a little bit (first things first):\n> \n> Please note that using the `checkout-encoding` attribute may have a number\n> of pitfalls:\n> \n> - Git clients that do not support the `checkout-encoding` attribute\n> will checkout the respective files as UTF-8 encoded, which typically\n> causes trouble.\n> Escpecialy when older Git versions are used or alternative Git\n> implementations such as JGit or libgit2 (as of January 2018).\n> \n> - Reencoding content to non-UTF encodings (e.g. SHIFT-JIS) can cause\n> errors as the conversion might not be round trip safe.\n> \n> - Reencoding content requires resources that might slow down certain\n> Git operations (e.g 'git checkout' or 'git add').\n> \n> Use the `checkout-encoding` attribute only if you cannot store a file in\n> UTF-8 encoding and if you want Git to be able to process the content as\n> text.\n\nAgreed!\n\n\n> -----\n> Side question: What happens if \"the used encoding\" is not supported by\n> the underlying iconv lib ?\n> Will Git fail, delete the file from the working tree ?\n> That may be worth to mention. (And I need to do the code-review)\n\nIt should checkout the file in UTF-8 and print an error.\nIf you would try to add a file with an unsupported encoding,\nthen Git should die() and refuse the operation.\n(see t0028: check unsupported encoding)\n\n\n> \n>> +Use the following attributes if your '*.txt' files are UTF-16 encoded\n>> +with byte order mark (BOM) and you want Git to perform automatic line\n>> +ending conversion based on your platform.\n>> +\n>> +------------------------\n>> +*.txt\t\ttext checkout-encoding=UTF-16\n>> +------------------------\n>> +\n>> +Use the following attributes if your '*.txt' files are UTF-16 little\n>> +endian encoded without BOM and you want Git to use Windows line endings\n>> +in the working directory.\n>> +\n>> +------------------------\n>> +*.txt \t\tcheckout-encoding=UTF-16LE text eol=CRLF\n>> +------------------------\n>> +\n>> +You can get a list of all available encodings on your platform with the\n>> +following command:\n>> +\n>> +------------------------\n>> +iconv --list\n>> +------------------------\n>> +\n>> +\n>> `ident`\n>> ^^^^^^^\n>> \n>> diff --git a/apply.c b/apply.c\n>> index 321a9fa68d..c4bd5cf1f2 100644\n>> --- a/apply.c\n>> +++ b/apply.c\n>> @@ -2281,7 +2281,7 @@ static int read_old_data(struct stat *st, struct patch *patch,\n>> \t\t * should never look at the index when explicit crlf option\n>> \t\t * is given.\n>> \t\t */\n>> -\t\tconvert_to_git(NULL, path, buf->buf, buf->len, buf, safe_crlf);\n>> +\t\tconvert_to_git(NULL, path, buf->buf, buf->len, buf, safe_crlf, 0);\n> \n> Hm. Do we really need another parameter here?\n> The only caller that needs it (and sets it to 1) is sha1_file.c.\n> When an invalid encoding/content is used, it should be safe to die() always?\n> Just loud thinking..\n\nUnfortunately it is not safe to die() always. Peff described a situation here\nwhere you could not \"checkout away\" from an error state:\nhttps://public-inbox.org/git/20171215095838.GA3567@sigill.intra.peff.net/\n\n\n> If really needed, it may need less changes in the code base, if a new function\n> convert_to_git_or_die() is defined, which is called by convert_to_git()\n> \n> (and the other alternative would be to convert safe_crlf into a bitmap,\n> see https://public-inbox.org/git/20171229132828.17594-1-tboegi@web.de/T/#u\n> what do you think ?)\n\nI still need to review your patch, but a bitmap sounds like a good idea.\nBut why do we need a special bitmap? Can't we just pass \"flags\" and\ncalculate get_safe_crlf() in convert.c?\n\n\n>> \n>> +\n>> +\tif (has_prohibited_utf_bom(enc->name, src, src_len)) {\n>> +\t\tconst char *error_msg = _(\n>> +\t\t\t\"BOM is prohibited for '%s' if encoded as %s\");\n>> +\t\tconst char *advise_msg = _(\n>> +\t\t\t\"You told Git to treat '%s' as %s. A byte order mark \"\n>> +\t\t\t\"(BOM) is prohibited with this encoding. Either use \"\n>> +\t\t\t\"%.6s as checkout encoding or remove the BOM from the \"\n>> +\t\t\t\"file.\");\n>> +\n>> +\t\tadvise(advise_msg, path, enc->name, enc->name, enc->name);\n>> +\t\tif (write_obj)\n>> +\t\t\tdie(error_msg, path, enc->name);\n>> +\t\telse\n>> +\t\t\terror(error_msg, path, enc->name);\n> \n> As said before, I would just die(). Or do I miss something ?\n> Being strict with BOMs seams like a good idea.\n\nSee above, Peffs test case. We run convert in a lot of places\n(e.g. 'git diff' or if checkout away from a bad state).\n\n\n>> \n>> diff --git a/t/t0028-checkout-encoding.sh b/t/t0028-checkout-encoding.sh\n>> new file mode 100755\n>> index 0000000000..1a329ab933\n>> --- /dev/null\n>> +++ b/t/t0028-checkout-encoding.sh\n>> @@ -0,0 +1,197 @@\n>> +#!/bin/sh\n>> +\n>> +test_description='checkout-encoding conversion via gitattributes'\n>> +\n>> +. ./test-lib.sh\n>> +\n>> +test_expect_success 'setup test repo' '\n>> +\n>> +\ttext=\"hallo there!\\ncan you read me?\" &&\n> \n> Is this portable ? (the \"\\n\")\n\nI assume it as it is used in t0001, t0008, t0024 and others.\n\n\nThanks for the review,\nLars\n\n"},{"id":"335569","messageId":"20171231080548.22215-1-tboegi@web.de","threadId":"47511","inReplyTo":"20171229152222.39680-1-lars.schneider@autodesk.com","subject":"[PATCH 0/5] V2B: simplify convert.c/h","fromName":"","fromEmail":"tboegi@web.de","sentAt":"2017-12-31T08:05:48Z","receivedAt":"2017-12-31T08:06:16Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"From: Torsten Bögershausen \n\nSimplify the convert.h/convert.c logic amd don't touch convert_to_git()\nThe rest is v2 from Lars\n\nLars Schneider (4):\n strbuf: add xstrdup_toupper()\n utf8: add function to detect prohibited UTF-16/32 BOM\n utf8: add function to detect a missing UTF-16/32 BOM\n convert: add support for 'checkout-encoding' attribute\n\nTorsten Bögershausen (1):\n convert_to_git(): checksafe becomes an integer\n\n Documentation/gitattributes.txt | 59 +++++++++++\n apply.c | 4 +-\n convert.c | 210 +++++++++++++++++++++++++++++++++++++---\n convert.h | 19 ++--\n diff.c | 4 +-\n environment.c | 2 +-\n sha1_file.c | 8 +-\n strbuf.c | 13 +++\n strbuf.h | 1 +\n t/t0028-checkout-encoding.sh | 197 +++++++++++++++++++++++++++++++++++++\n utf8.c | 37 +++++++\n utf8.h | 25 +++++\n 12 files changed, 549 insertions(+), 30 deletions(-)\n create mode 100755 t/t0028-checkout-encoding.sh\n\n-- \n2.16.0.rc0.4.ga4e00d4fa4\n\n"},{"id":"335570","messageId":"20171231080601.22444-1-tboegi@web.de","threadId":"47511","inReplyTo":"20171229152222.39680-1-lars.schneider@autodesk.com","subject":"[PATCH 5/5] convert: add support for 'checkout-encoding' attribute","fromName":"","fromEmail":"tboegi@web.de","sentAt":"2017-12-31T08:06:01Z","receivedAt":"2017-12-31T08:06:22Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"From: Lars Schneider \n\nGit and its tools (e.g. git diff) expect all text files in UTF-8\nencoding. Git will happily accept content in all other encodings, too,\nbut it might not be able to process the text (e.g. viewing diffs or\nchanging line endings).\n\nAdd an attribute to tell Git what encoding the user has defined for a\ngiven file. If the content is added to the index, then Git converts the\ncontent to a canonical UTF-8 representation. On checkout Git will\nreverse the conversion.\n\nSigned-off-by: Lars Schneider \nSigned-off-by: Torsten Bögershausen \n---\n Documentation/gitattributes.txt | 59 ++++++++++++\n convert.c | 190 +++++++++++++++++++++++++++++++++++++-\n convert.h | 11 ++-\n sha1_file.c | 2 +-\n t/t0028-checkout-encoding.sh | 197 ++++++++++++++++++++++++++++++++++++++++\n 5 files changed, 452 insertions(+), 7 deletions(-)\n create mode 100755 t/t0028-checkout-encoding.sh\n\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex 30687de81a..0039bd38c3 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -272,6 +272,65 @@ few exceptions. Even though...\n catch potential problems early, safety triggers.\n \n \n+`checkout-encoding`\n+^^^^^^^^^^^^^^^^^^^\n+\n+Git recognizes files encoded with ASCII or one of its supersets (e.g.\n+UTF-8 or ISO-8859-1) as text files. All other encodings are usually\n+interpreted as binary and consequently built-in Git text processing\n+tools (e.g. 'git diff') as well as most Git web front ends do not\n+visualize the content.\n+\n+In these cases you can teach Git the encoding of a file in the working\n+directory with the `checkout-encoding` attribute. If a file with this\n+attributes is added to Git, then Git reencodes the content from the\n+specified encoding to UTF-8 and stores the result in its internal data\n+structure. On checkout the content is encoded back to the specified\n+encoding.\n+\n+Please note that using the `checkout-encoding` attribute has a number\n+of drawbacks:\n+\n+- Reencoding content to non-UTF encodings (e.g. SHIFT-JIS) can cause\n+ errors as the conversion might not be round trip safe.\n+\n+- Reencoding content requires resources that might slow down certain\n+ Git operations (e.g 'git checkout' or 'git add').\n+\n+- Git clients that do not support the `checkout-encoding` attribute or\n+ the used encoding will checkout the respective files as UTF-8 encoded.\n+ That means the content appears to be different which could cause\n+ trouble. Affected clients are older Git versions and alternative Git\n+ implementations such as JGit or libgit2 (as of January 2018).\n+\n+Use the `checkout-encoding` attribute only if you cannot store a file in\n+UTF-8 encoding and if you want Git to be able to process the content as\n+text.\n+\n+Use the following attributes if your '*.txt' files are UTF-16 encoded\n+with byte order mark (BOM) and you want Git to perform automatic line\n+ending conversion based on your platform.\n+\n+------------------------\n+*.txt\t\ttext checkout-encoding=UTF-16\n+------------------------\n+\n+Use the following attributes if your '*.txt' files are UTF-16 little\n+endian encoded without BOM and you want Git to use Windows line endings\n+in the working directory.\n+\n+------------------------\n+*.txt \t\tcheckout-encoding=UTF-16LE text eol=CRLF\n+------------------------\n+\n+You can get a list of all available encodings on your platform with the\n+following command:\n+\n+------------------------\n+iconv --list\n+------------------------\n+\n+\n `ident`\n ^^^^^^^\n \ndiff --git a/convert.c b/convert.c\nindex 5efcc3b73b..22c70d87e5 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -7,6 +7,7 @@\n #include \"sigchain.h\"\n #include \"pkt-line.h\"\n #include \"sub-process.h\"\n+#include \"utf8.h\"\n \n /*\n * convert.c - convert a file when checking it out and checking it in.\n@@ -265,6 +266,147 @@ static int will_convert_lf_to_crlf(size_t len, struct text_stat *stats,\n \n }\n \n+static struct encoding {\n+\tconst char *name;\n+\tstruct encoding *next;\n+} *encoding, **encoding_tail;\n+static const char *default_encoding = \"UTF-8\";\n+\n+static int encode_to_git(const char *path, const char *src, size_t src_len,\n+\t\t\t struct strbuf *buf, struct encoding *enc, int die_on_failure)\n+{\n+\tchar *dst;\n+\tint dst_len;\n+\n+\t/*\n+\t * No encoding is specified or there is nothing to encode.\n+\t * Tell the caller that the content was not modified.\n+\t */\n+\tif (!enc || (src && !src_len))\n+\t\treturn 0;\n+\n+\t/*\n+\t * Looks like we got called from \"would_convert_to_git()\".\n+\t * This means Git wants to know if it would encode (= modify!)\n+\t * the content. Let's answer with \"yes\", since an encoding was\n+\t * specified.\n+\t */\n+\tif (!buf && !src)\n+\t\treturn 1;\n+\n+\tif (has_prohibited_utf_bom(enc->name, src, src_len)) {\n+\t\tconst char *error_msg = _(\n+\t\t\t\"BOM is prohibited for '%s' if encoded as %s\");\n+\t\tconst char *advise_msg = _(\n+\t\t\t\"You told Git to treat '%s' as %s. A byte order mark \"\n+\t\t\t\"(BOM) is prohibited with this encoding. Either use \"\n+\t\t\t\"%.6s as checkout encoding or remove the BOM from the \"\n+\t\t\t\"file.\");\n+\n+\t\tadvise(advise_msg, path, enc->name, enc->name, enc->name);\n+\t\tif (die_on_failure)\n+\t\t\tdie(error_msg, path, enc->name);\n+\t\telse\n+\t\t\terror(error_msg, path, enc->name);\n+\n+\n+\t} else if (has_missing_utf_bom(enc->name, src, src_len)) {\n+\t\tconst char *error_msg = _(\n+\t\t\t\"BOM is required for '%s' if encoded as %s\");\n+\t\tconst char *advise_msg = _(\n+\t\t\t\"You told Git to treat '%s' as %s. A byte order mark \"\n+\t\t\t\"(BOM) is required with this encoding. Either use \"\n+\t\t\t\"%sBE/%sLE as checkout encoding or add a BOM to the \"\n+\t\t\t\"file.\");\n+\t\tadvise(advise_msg, path, enc->name, enc->name, enc->name);\n+\t\tif (die_on_failure)\n+\t\t\tdie(error_msg, path, enc->name);\n+\t\telse\n+\t\t\terror(error_msg, path, enc->name);\n+\t}\n+\n+\tdst = reencode_string_len(src, src_len, default_encoding, enc->name,\n+\t\t\t\t &dst_len);\n+\tif (!dst) {\n+\t\t/*\n+\t\t * We could add the blob \"as-is\" to Git. However, on checkout\n+\t\t * we would try to reencode to the original encoding. This\n+\t\t * would fail and we would leave the user with a messed-up\n+\t\t * working tree. Let's try to avoid this by screaming loud.\n+\t\t */\n+\t\tconst char* msg = _(\"failed to encode '%s' from %s to %s\");\n+\t\tif (die_on_failure)\n+\t\t\tdie(msg, path, enc->name, default_encoding);\n+\t\telse\n+\t\t\terror(msg, path, enc->name, default_encoding);\n+\t}\n+\n+\t/*\n+\t * UTF supports lossless round tripping [1]. UTF to other encoding are\n+\t * mostly round trip safe as Unicode aims to be a superset of all other\n+\t * character encodings. However, the SHIFT-JIS (Japanese character set)\n+\t * is an exception as some codes are not round trip safe [2].\n+\t *\n+\t * Reverse the transformation of 'dst' and check the result with 'src'\n+\t * if content is written to Git. This ensures no information is lost\n+\t * during conversion to/from UTF-8.\n+\t *\n+\t * Please note, the code below is not tested because I was not able to\n+\t * generate a faulty round trip without iconv error.\n+\t *\n+\t * [1] http://unicode.org/faq/utf_bom.html#gen2\n+\t * [2] https://support.microsoft.com/en-us/help/170559/prb-conversion-problem-between-shift-jis-and-unicode\n+\t */\n+\tif (die_on_failure && !strcmp(enc->name, \"SHIFT-JIS\")) {\n+\t\tchar *re_src;\n+\t\tint re_src_len;\n+\n+\t\tre_src = reencode_string_len(dst, dst_len,\n+\t\t\t\t\t enc->name, default_encoding,\n+\t\t\t\t\t &re_src_len);\n+\n+\t\tif (!re_src || src_len != re_src_len ||\n+\t\t memcmp(src, re_src, src_len)) {\n+\t\t\tconst char* msg = _(\"encoding '%s' from %s to %s and \"\n+\t\t\t\t\t \"back is not the same\");\n+\t\t\tif (die_on_failure)\n+\t\t\t\tdie(msg, path, enc->name, default_encoding);\n+\t\t\telse\n+\t\t\t\terror(msg, path, enc->name, default_encoding);\n+\t\t}\n+\n+\t\tfree(re_src);\n+\t}\n+\n+\tstrbuf_attach(buf, dst, dst_len, dst_len + 1);\n+\treturn 1;\n+}\n+\n+static int encode_to_worktree(const char *path, const char *src, size_t src_len,\n+\t\t\t struct strbuf *buf, struct encoding *enc)\n+{\n+\tchar *dst;\n+\tint dst_len;\n+\n+\t/*\n+\t * No encoding is specified or there is nothing to encode.\n+\t * Tell the caller that the content was not modified.\n+\t */\n+\tif (!enc || (src && !src_len))\n+\t\treturn 0;\n+\n+\tdst = reencode_string_len(src, src_len, enc->name, default_encoding,\n+\t\t\t\t &dst_len);\n+\tif (!dst) {\n+\t\terror(\"failed to encode '%s' from %s to %s\",\n+\t\t\tpath, enc->name, default_encoding);\n+\t\treturn 0;\n+\t}\n+\n+\tstrbuf_attach(buf, dst, dst_len, dst_len + 1);\n+\treturn 1;\n+}\n+\n static int crlf_to_git(const struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n \t\t struct strbuf *buf,\n@@ -978,6 +1120,31 @@ static int ident_to_worktree(const char *path, const char *src, size_t len,\n \treturn 1;\n }\n \n+static struct encoding *git_path_check_encoding(struct attr_check_item *check)\n+{\n+\tconst char *value = check->value;\n+\tstruct encoding *enc;\n+\n+\tif (ATTR_TRUE(value) || ATTR_FALSE(value) || ATTR_UNSET(value) ||\n+\t !strlen(value))\n+\t\treturn NULL;\n+\n+\tfor (enc = encoding; enc; enc = enc->next)\n+\t\tif (!strcasecmp(value, enc->name))\n+\t\t\treturn enc;\n+\n+\t/* Don't encode to the default encoding */\n+\tif (!strcasecmp(value, default_encoding))\n+\t\treturn NULL;\n+\n+\tenc = xcalloc(1, sizeof(struct convert_driver));\n+\tenc->name = xstrdup_toupper(value); /* aways use upper case names! */\n+\t*encoding_tail = enc;\n+\tencoding_tail = &(enc->next);\n+\n+\treturn enc;\n+}\n+\n static enum crlf_action git_path_check_crlf(struct attr_check_item *check)\n {\n \tconst char *value = check->value;\n@@ -1033,6 +1200,7 @@ struct conv_attrs {\n \tenum crlf_action attr_action; /* What attr says */\n \tenum crlf_action crlf_action; /* When no attr is set, use core.autocrlf */\n \tint ident;\n+\tstruct encoding *checkout_encoding; /* Supported encoding or default encoding if NULL */\n };\n \n static void convert_attrs(struct conv_attrs *ca, const char *path)\n@@ -1041,8 +1209,10 @@ static void convert_attrs(struct conv_attrs *ca, const char *path)\n \n \tif (!check) {\n \t\tcheck = attr_check_initl(\"crlf\", \"ident\", \"filter\",\n-\t\t\t\t\t \"eol\", \"text\", NULL);\n+\t\t\t\t\t \"eol\", \"text\", \"checkout-encoding\",\n+\t\t\t\t\t NULL);\n \t\tuser_convert_tail = &user_convert;\n+\t\tencoding_tail = &encoding;\n \t\tgit_config(read_convert_config, NULL);\n \t}\n \n@@ -1064,6 +1234,7 @@ static void convert_attrs(struct conv_attrs *ca, const char *path)\n \t\t\telse if (eol_attr == EOL_CRLF)\n \t\t\t\tca->crlf_action = CRLF_TEXT_CRLF;\n \t\t}\n+\t\tca->checkout_encoding = git_path_check_encoding(ccheck + 5);\n \t} else {\n \t\tca->drv = NULL;\n \t\tca->crlf_action = CRLF_UNDEFINED;\n@@ -1132,6 +1303,7 @@ int convert_to_git(const struct index_state *istate,\n \t\t struct strbuf *dst, int checksafe)\n {\n \tint ret = 0;\n+\tint die_on_failure = checksafe & SAFE_CRLF_DIE_ON_ERROR;\n \tstruct conv_attrs ca;\n \n \tconvert_attrs(&ca, path);\n@@ -1140,6 +1312,11 @@ int convert_to_git(const struct index_state *istate,\n \tif (!ret && ca.drv && ca.drv->required)\n \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n \n+\tif (ret && dst) {\n+\t\tsrc = dst->buf;\n+\t\tlen = dst->len;\n+\t}\n+\tret |= encode_to_git(path, src, len, dst, ca.checkout_encoding, die_on_failure);\n \tif (ret && dst) {\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\n@@ -1159,6 +1336,7 @@ void convert_to_git_filter_fd(const struct index_state *istate,\n \t\t\t int checksafe)\n {\n \tstruct conv_attrs ca;\n+\tint die_on_failure = checksafe & SAFE_CRLF_DIE_ON_ERROR;\n \tconvert_attrs(&ca, path);\n \n \tassert(ca.drv);\n@@ -1167,6 +1345,7 @@ void convert_to_git_filter_fd(const struct index_state *istate,\n \tif (!apply_filter(path, NULL, 0, fd, dst, ca.drv, CAP_CLEAN, NULL))\n \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n \n+\tencode_to_git(path, dst->buf, dst->len, dst, ca.checkout_encoding, die_on_failure);\n \tcrlf_to_git(istate, path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);\n \tident_to_git(path, dst->buf, dst->len, dst, ca.ident);\n }\n@@ -1198,6 +1377,12 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n \t\t}\n \t}\n \n+\tret |= encode_to_worktree(path, src, len, dst, ca.checkout_encoding);\n+\tif (ret) {\n+\t\tsrc = dst->buf;\n+\t\tlen = dst->len;\n+\t}\n+\n \tret_filter = apply_filter(\n \t\tpath, src, len, -1, dst, ca.drv, CAP_SMUDGE, dco);\n \tif (!ret_filter && ca.drv && ca.drv->required)\n@@ -1664,6 +1849,9 @@ struct stream_filter *get_stream_filter(const char *path, const unsigned char *s\n \tif (ca.drv && (ca.drv->process || ca.drv->smudge || ca.drv->clean))\n \t\treturn NULL;\n \n+\tif (ca.checkout_encoding)\n+\t\treturn NULL;\n+\n \tif (ca.crlf_action == CRLF_AUTO || ca.crlf_action == CRLF_AUTO_CRLF)\n \t\treturn NULL;\n \ndiff --git a/convert.h b/convert.h\nindex 532af00423..3a85c2f9a5 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -8,11 +8,12 @@\n \n struct index_state;\n \n-#define SAFE_CRLF_FALSE 0\n-#define SAFE_CRLF_FAIL (1<<0)\n-#define SAFE_CRLF_WARN (1<<1)\n-#define SAFE_CRLF_RENORMALIZE (1<<2)\n-#define SAFE_CRLF_KEEP_CRLF (1<<3)\n+#define SAFE_CRLF_FALSE 0\n+#define SAFE_CRLF_FAIL (1<<0)\n+#define SAFE_CRLF_WARN (1<<1)\n+#define SAFE_CRLF_RENORMALIZE (1<<2)\n+#define SAFE_CRLF_KEEP_CRLF (1<<3)\n+#define SAFE_CRLF_DIE_ON_ERROR (1<<4)\n \n extern int safe_crlf;\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex 78e002392e..c39b298f83 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -138,7 +138,7 @@ static int get_checksafe(unsigned flags)\n \tif (flags & HASH_RENORMALIZE)\n \t\treturn SAFE_CRLF_RENORMALIZE;\n \telse if (flags & HASH_WRITE_OBJECT)\n-\t\treturn safe_crlf;\n+\t\treturn safe_crlf | SAFE_CRLF_DIE_ON_ERROR;\n \telse\n \t\treturn SAFE_CRLF_FALSE;\n }\ndiff --git a/t/t0028-checkout-encoding.sh b/t/t0028-checkout-encoding.sh\nnew file mode 100755\nindex 0000000000..1a329ab933\n--- /dev/null\n+++ b/t/t0028-checkout-encoding.sh\n@@ -0,0 +1,197 @@\n+#!/bin/sh\n+\n+test_description='checkout-encoding conversion via gitattributes'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup test repo' '\n+\n+\ttext=\"hallo there!\\ncan you read me?\" &&\n+\n+\techo \"*.utf16 text checkout-encoding=utf-16\" >.gitattributes &&\n+\n+\tprintf \"$text\" >test.utf8.raw &&\n+\tprintf \"$text\" | iconv -f UTF-8 -t UTF-16 >test.utf16.raw &&\n+\tcp test.utf16.raw test.utf16 &&\n+\n+\tgit add .gitattributes test.utf16 &&\n+\tgit commit -m initial\n+'\n+\n+test_expect_success 'ensure UTF-8 is stored in Git' '\n+\tgit cat-file -p :test.utf16 >test.utf16.git &&\n+\ttest_cmp_bin test.utf8.raw test.utf16.git &&\n+\trm test.utf8.raw test.utf16.git\n+'\n+\n+test_expect_success 're-encode to UTF-16 on checkout' '\n+\trm test.utf16 &&\n+\tgit checkout test.utf16 &&\n+\ttest_cmp_bin test.utf16.raw test.utf16 &&\n+\n+\t# cleanup\n+\trm test.utf16.raw\n+'\n+\n+test_expect_success 'check prohibited UTF BOM' '\n+\tprintf \"\\0a\\0b\\0c\" >nobom.utf16be.raw &&\n+\tprintf \"a\\0b\\0c\\0\" >nobom.utf16le.raw &&\n+\tprintf \"\\376\\777\\0a\\0b\\0c\" >bebom.utf16be.raw &&\n+\tprintf \"\\777\\376a\\0b\\0c\\0\" >lebom.utf16le.raw &&\n+\n+\tprintf \"\\0\\0\\0a\\0\\0\\0b\\0\\0\\0c\" >nobom.utf32be.raw &&\n+\tprintf \"a\\0\\0\\0b\\0\\0\\0c\\0\\0\\0\" >nobom.utf32le.raw &&\n+\tprintf \"\\0\\0\\376\\777\\0\\0\\0a\\0\\0\\0b\\0\\0\\0c\" >bebom.utf32be.raw &&\n+\tprintf \"\\777\\376\\0\\0a\\0\\0\\0b\\0\\0\\0c\\0\\0\\0\" >lebom.utf32le.raw &&\n+\n+\techo \"*.utf16be text checkout-encoding=utf-16be\" >>.gitattributes &&\n+\techo \"*.utf16le text checkout-encoding=utf-16le\" >>.gitattributes &&\n+\techo \"*.utf32be text checkout-encoding=utf-32be\" >>.gitattributes &&\n+\techo \"*.utf32le text checkout-encoding=utf-32le\" >>.gitattributes &&\n+\n+\t# Here we add a UTF-16 files with BOM (big-endian and little-endian)\n+\t# but we tell Git to treat it as UTF-16BE/UTF-16LE. In these cases\n+\t# the BOM is prohibited.\n+\tcp bebom.utf16be.raw bebom.utf16be &&\n+\ttest_must_fail git add bebom.utf16be 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-16BE\" err.out &&\n+\n+\tcp lebom.utf16le.raw lebom.utf16be &&\n+\ttest_must_fail git add lebom.utf16be 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-16BE\" err.out &&\n+\n+\tcp bebom.utf16be.raw bebom.utf16le &&\n+\ttest_must_fail git add bebom.utf16le 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-16LE\" err.out &&\n+\n+\tcp lebom.utf16le.raw lebom.utf16le &&\n+\ttest_must_fail git add lebom.utf16le 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-16LE\" err.out &&\n+\n+\t# ... and the same for UTF-32\n+\tcp bebom.utf32be.raw bebom.utf32be &&\n+\ttest_must_fail git add bebom.utf32be 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-32BE\" err.out &&\n+\n+\tcp lebom.utf32le.raw lebom.utf32be &&\n+\ttest_must_fail git add lebom.utf32be 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-32BE\" err.out &&\n+\n+\tcp bebom.utf32be.raw bebom.utf32le &&\n+\ttest_must_fail git add bebom.utf32le 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-32LE\" err.out &&\n+\n+\tcp lebom.utf32le.raw lebom.utf32le &&\n+\ttest_must_fail git add lebom.utf32le 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-32LE\" err.out &&\n+\n+\t# cleanup\n+\tgit reset --hard HEAD\n+'\n+\n+test_expect_success 'check required UTF BOM' '\n+\techo \"*.utf32 text checkout-encoding=utf-32\" >>.gitattributes &&\n+\n+\tcp nobom.utf16be.raw nobom.utf16 &&\n+\ttest_must_fail git add nobom.utf16 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is required .* UTF-16\" err.out &&\n+\n+\tcp nobom.utf16le.raw nobom.utf16 &&\n+\ttest_must_fail git add nobom.utf16 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is required .* UTF-16\" err.out &&\n+\n+\tcp nobom.utf32be.raw nobom.utf32 &&\n+\ttest_must_fail git add nobom.utf32 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is required .* UTF-32\" err.out &&\n+\n+\tcp nobom.utf32le.raw nobom.utf32 &&\n+\ttest_must_fail git add nobom.utf32 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is required .* UTF-32\" err.out &&\n+\n+\t# cleanup\n+\trm nobom.utf16 nobom.utf32 &&\n+\tgit reset --hard HEAD\n+'\n+\n+test_expect_success 'eol conversion for UTF-16 encoded files on checkout' '\n+\tprintf \"one\\ntwo\\nthree\\n\" >lf.utf8.raw &&\n+\tprintf \"one\\r\\ntwo\\r\\nthree\\r\\n\" >crlf.utf8.raw &&\n+\n+\tcat lf.utf8.raw | iconv -f UTF-8 -t UTF-16 >lf.utf16.raw &&\n+\tcat crlf.utf8.raw | iconv -f UTF-8 -t UTF-16 >crlf.utf16.raw &&\n+\tcp crlf.utf16.raw eol.utf16 &&\n+\n+\tgit add eol.utf16 &&\n+\tgit commit -m eol &&\n+\n+\t# UTF-16 with CRLF (Windows line endings)\n+\trm eol.utf16 &&\n+\tgit -c core.eol=crlf checkout eol.utf16 &&\n+\ttest_cmp_bin crlf.utf16.raw eol.utf16 &&\n+\n+\t# UTF-16 with LF (Unix line endings)\n+\trm eol.utf16 &&\n+\tgit -c core.eol=lf checkout eol.utf16 &&\n+\ttest_cmp_bin lf.utf16.raw eol.utf16 &&\n+\n+\trm crlf.utf16.raw crlf.utf8.raw lf.utf16.raw lf.utf8.raw &&\n+\n+\t# cleanup\n+\tgit reset --hard HEAD^\n+'\n+\n+test_expect_success 'check unsupported encodings' '\n+\n+\techo \"*.nothing text checkout-encoding=\" >>.gitattributes &&\n+\tprintf \"nothing\" >t.nothing &&\n+\tgit add t.nothing &&\n+\n+\techo \"*.garbage text checkout-encoding=garbage\" >>.gitattributes &&\n+\tprintf \"garbage\" >t.garbage &&\n+\ttest_must_fail git add t.garbage 2>err.out &&\n+\ttest_i18ngrep \"fatal: failed to encode\" err.out &&\n+\n+\t# cleanup\n+\trm err.out &&\n+\tgit reset --hard HEAD\n+'\n+\n+test_expect_success 'error if encoding round trip is not the same during refresh' '\n+\tBEFORE_STATE=$(git rev-parse HEAD) &&\n+\n+\t# Skip the UTF-16 filter for the added file\n+\t# This simulates a Git version that has no checkoutEncoding support\n+\techo \"hallo\" >nonsense.utf16 &&\n+\tTEST_HASH=$(git hash-object --no-filters -w nonsense.utf16) &&\n+\tgit update-index --add --cacheinfo 100644 $TEST_HASH nonsense.utf16 &&\n+\tCOMMIT=$(git commit-tree -p $(git rev-parse HEAD) -m \"plain commit\" $(git write-tree)) &&\n+\tgit update-ref refs/heads/master $COMMIT &&\n+\n+\ttest_must_fail git checkout HEAD^ 2>err.out &&\n+\ttest_i18ngrep \"error: .* overwritten by checkout:\" err.out &&\n+\n+\t# cleanup\n+\trm err.out &&\n+\tgit reset --hard $BEFORE_STATE\n+'\n+\n+test_expect_success 'error if encoding garbage is already in Git' '\n+\tBEFORE_STATE=$(git rev-parse HEAD) &&\n+\n+\t# Skip the UTF-16 filter for the added file\n+\t# This simulates a Git version that has no checkoutEncoding support\n+\tcp nobom.utf16be.raw nonsense.utf16 &&\n+\tTEST_HASH=$(git hash-object --no-filters -w nonsense.utf16) &&\n+\tgit update-index --add --cacheinfo 100644 $TEST_HASH nonsense.utf16 &&\n+\tCOMMIT=$(git commit-tree -p $(git rev-parse HEAD) -m \"plain commit\" $(git write-tree)) &&\n+\tgit update-ref refs/heads/master $COMMIT &&\n+\n+\tgit diff 2>err.out &&\n+\ttest_i18ngrep \"error: BOM is required\" err.out &&\n+\n+\t# cleanup\n+\trm err.out &&\n+\tgit reset --hard $BEFORE_STATE\n+'\n+\n+test_done\n-- \n2.16.0.rc0.4.ga4e00d4fa4\n\n"},{"id":"335571","messageId":"20171231080556.22344-1-tboegi@web.de","threadId":"47511","inReplyTo":"20171229152222.39680-1-lars.schneider@autodesk.com","subject":"[PATCH 3/5] utf8: add function to detect prohibited UTF-16/32 BOM","fromName":"","fromEmail":"tboegi@web.de","sentAt":"2017-12-31T08:05:56Z","receivedAt":"2017-12-31T08:06:25Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"From: Lars Schneider \n\nWhenever a data stream is declared to be UTF-16BE, UTF-16LE, UTF-32BE\nor UTF-32LE a BOM must not be used [1]. The function returns true if\nthis is the case.\n\nThis function is used in a subsequent commit.\n\n[1] http://unicode.org/faq/utf_bom.html#bom10\n\nSigned-off-by: Lars Schneider \nSigned-off-by: Torsten Bögershausen \n---\n utf8.c | 24 ++++++++++++++++++++++++\n utf8.h | 9 +++++++++\n 2 files changed, 33 insertions(+)\n\ndiff --git a/utf8.c b/utf8.c\nindex 2c27ce0137..776660ee12 100644\n--- a/utf8.c\n+++ b/utf8.c\n@@ -538,6 +538,30 @@ char *reencode_string_len(const char *in, int insz,\n }\n #endif\n \n+static int has_bom_prefix(const char *data, size_t len,\n+\t\t\t const char *bom, size_t bom_len)\n+{\n+\treturn (len >= bom_len) && !memcmp(data, bom, bom_len);\n+}\n+\n+const char utf16_be_bom[] = {0xFE, 0xFF};\n+const char utf16_le_bom[] = {0xFF, 0xFE};\n+const char utf32_be_bom[] = {0x00, 0x00, 0xFE, 0xFF};\n+const char utf32_le_bom[] = {0xFF, 0xFE, 0x00, 0x00};\n+\n+int has_prohibited_utf_bom(const char *enc, const char *data, size_t len)\n+{\n+\treturn (\n+\t (!strcmp(enc, \"UTF-16BE\") || !strcmp(enc, \"UTF-16LE\")) &&\n+\t (has_bom_prefix(data, len, utf16_be_bom, sizeof(utf16_be_bom)) ||\n+\t has_bom_prefix(data, len, utf16_le_bom, sizeof(utf16_le_bom)))\n+\t) || (\n+\t (!strcmp(enc, \"UTF-32BE\") || !strcmp(enc, \"UTF-32LE\")) &&\n+\t (has_bom_prefix(data, len, utf32_be_bom, sizeof(utf32_be_bom)) ||\n+\t has_bom_prefix(data, len, utf32_le_bom, sizeof(utf32_le_bom)))\n+\t);\n+}\n+\n /*\n * Returns first character length in bytes for multi-byte `text` according to\n * `encoding`.\ndiff --git a/utf8.h b/utf8.h\nindex 6bbcf31a83..4711429af9 100644\n--- a/utf8.h\n+++ b/utf8.h\n@@ -70,4 +70,13 @@ typedef enum {\n void strbuf_utf8_align(struct strbuf *buf, align_type position, unsigned int width,\n \t\t const char *s);\n \n+/*\n+ * Whenever a data stream is declared to be UTF-16BE, UTF-16LE, UTF-32BE\n+ * or UTF-32LE a BOM must not be used [1]. The function returns true if\n+ * this is the case.\n+ *\n+ * [1] http://unicode.org/faq/utf_bom.html#bom10\n+ */\n+int has_prohibited_utf_bom(const char *enc, const char *data, size_t len);\n+\n #endif\n-- \n2.16.0.rc0.4.ga4e00d4fa4\n\n"},{"id":"335572","messageId":"20171231080552.22258-1-tboegi@web.de","threadId":"47511","inReplyTo":"20171229152222.39680-1-lars.schneider@autodesk.com","subject":"[PATCH 1/5] convert_to_git(): checksafe becomes an integer","fromName":"","fromEmail":"tboegi@web.de","sentAt":"2017-12-31T08:05:52Z","receivedAt":"2017-12-31T08:06:27Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"From: Torsten Bögershausen \n\nWhen calling convert_to_git(), the checksafe parameter has been used to\ncheck if commit would give a non-roundtrip conversion of EOL.\n\nWhen checksafe was introduced, 3 values had been in use:\nSAFE_CRLF_FALSE: no warning\nSAFE_CRLF_FAIL: reject the commit if EOL do not roundtrip\nSAFE_CRLF_WARN: warn the user if EOL do not roundtrip\n\nToday a small flaw is found in the code base:\nAn integer with the value 0 is passed as the parameter checksafe\ninstead of the correct enum value SAFE_CRLF_FALSE.\n\nIn the next commit there is a need to turn checksafe into a bitmap, which\nallows to tell convert_to_git() to obey the encoding attribute or not.\n\nSigned-off-by: Torsten Bögershausen \n---\n apply.c | 4 ++--\n convert.c | 20 ++++++++++----------\n convert.h | 18 ++++++++----------\n diff.c | 4 ++--\n environment.c | 2 +-\n sha1_file.c | 6 +++---\n 6 files changed, 26 insertions(+), 28 deletions(-)\n\ndiff --git a/apply.c b/apply.c\nindex 321a9fa68d..a422516062 100644\n--- a/apply.c\n+++ b/apply.c\n@@ -2263,7 +2263,7 @@ static void show_stats(struct apply_state *state, struct patch *patch)\n static int read_old_data(struct stat *st, struct patch *patch,\n \t\t\t const char *path, struct strbuf *buf)\n {\n-\tenum safe_crlf safe_crlf = patch->crlf_in_old ?\n+\tint checksafe = patch->crlf_in_old ?\n \t\tSAFE_CRLF_KEEP_CRLF : SAFE_CRLF_RENORMALIZE;\n \tswitch (st->st_mode & S_IFMT) {\n \tcase S_IFLNK:\n@@ -2281,7 +2281,7 @@ static int read_old_data(struct stat *st, struct patch *patch,\n \t\t * should never look at the index when explicit crlf option\n \t\t * is given.\n \t\t */\n-\t\tconvert_to_git(NULL, path, buf->buf, buf->len, buf, safe_crlf);\n+\t\tconvert_to_git(NULL, path, buf->buf, buf->len, buf, checksafe);\n \t\treturn 0;\n \tdefault:\n \t\treturn -1;\ndiff --git a/convert.c b/convert.c\nindex 1a41a48e15..5efcc3b73b 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -195,13 +195,13 @@ static enum eol output_eol(enum crlf_action crlf_action)\n \n static void check_safe_crlf(const char *path, enum crlf_action crlf_action,\n \t\t\t struct text_stat *old_stats, struct text_stat *new_stats,\n-\t\t\t enum safe_crlf checksafe)\n+\t\t\t int checksafe)\n {\n \tif (old_stats->crlf && !new_stats->crlf ) {\n \t\t/*\n \t\t * CRLFs would not be restored by checkout\n \t\t */\n-\t\tif (checksafe == SAFE_CRLF_WARN)\n+\t\tif (checksafe & SAFE_CRLF_WARN)\n \t\t\twarning(_(\"CRLF will be replaced by LF in %s.\\n\"\n \t\t\t\t \"The file will have its original line\"\n \t\t\t\t \" endings in your working directory.\"), path);\n@@ -211,7 +211,7 @@ static void check_safe_crlf(const char *path, enum crlf_action crlf_action,\n \t\t/*\n \t\t * CRLFs would be added by checkout\n \t\t */\n-\t\tif (checksafe == SAFE_CRLF_WARN)\n+\t\tif (checksafe & SAFE_CRLF_WARN)\n \t\t\twarning(_(\"LF will be replaced by CRLF in %s.\\n\"\n \t\t\t\t \"The file will have its original line\"\n \t\t\t\t \" endings in your working directory.\"), path);\n@@ -268,7 +268,7 @@ static int will_convert_lf_to_crlf(size_t len, struct text_stat *stats,\n static int crlf_to_git(const struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n \t\t struct strbuf *buf,\n-\t\t enum crlf_action crlf_action, enum safe_crlf checksafe)\n+\t\t enum crlf_action crlf_action, int checksafe)\n {\n \tstruct text_stat stats;\n \tchar *dst;\n@@ -298,12 +298,12 @@ static int crlf_to_git(const struct index_state *istate,\n \t\t * unless we want to renormalize in a merge or\n \t\t * cherry-pick.\n \t\t */\n-\t\tif ((checksafe != SAFE_CRLF_RENORMALIZE) &&\n+\t\tif ((!(checksafe & SAFE_CRLF_RENORMALIZE)) &&\n \t\t has_crlf_in_index(istate, path))\n \t\t\tconvert_crlf_into_lf = 0;\n \t}\n-\tif ((checksafe == SAFE_CRLF_WARN ||\n-\t (checksafe == SAFE_CRLF_FAIL)) && len) {\n+\tif (((checksafe & SAFE_CRLF_WARN) ||\n+\t ((checksafe & SAFE_CRLF_FAIL) && len))) {\n \t\tstruct text_stat new_stats;\n \t\tmemcpy(&new_stats, &stats, sizeof(new_stats));\n \t\t/* simulate \"git add\" */\n@@ -1129,7 +1129,7 @@ const char *get_convert_attr_ascii(const char *path)\n \n int convert_to_git(const struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n- struct strbuf *dst, enum safe_crlf checksafe)\n+\t\t struct strbuf *dst, int checksafe)\n {\n \tint ret = 0;\n \tstruct conv_attrs ca;\n@@ -1144,7 +1144,7 @@ int convert_to_git(const struct index_state *istate,\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\n \t}\n-\tif (checksafe != SAFE_CRLF_KEEP_CRLF) {\n+\tif (!(checksafe & SAFE_CRLF_KEEP_CRLF)) {\n \t\tret |= crlf_to_git(istate, path, src, len, dst, ca.crlf_action, checksafe);\n \t\tif (ret && dst) {\n \t\t\tsrc = dst->buf;\n@@ -1156,7 +1156,7 @@ int convert_to_git(const struct index_state *istate,\n \n void convert_to_git_filter_fd(const struct index_state *istate,\n \t\t\t const char *path, int fd, struct strbuf *dst,\n-\t\t\t enum safe_crlf checksafe)\n+\t\t\t int checksafe)\n {\n \tstruct conv_attrs ca;\n \tconvert_attrs(&ca, path);\ndiff --git a/convert.h b/convert.h\nindex 4f2da225a8..532af00423 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -8,15 +8,13 @@\n \n struct index_state;\n \n-enum safe_crlf {\n-\tSAFE_CRLF_FALSE = 0,\n-\tSAFE_CRLF_FAIL = 1,\n-\tSAFE_CRLF_WARN = 2,\n-\tSAFE_CRLF_RENORMALIZE = 3,\n-\tSAFE_CRLF_KEEP_CRLF = 4\n-};\n+#define SAFE_CRLF_FALSE 0\n+#define SAFE_CRLF_FAIL (1<<0)\n+#define SAFE_CRLF_WARN (1<<1)\n+#define SAFE_CRLF_RENORMALIZE (1<<2)\n+#define SAFE_CRLF_KEEP_CRLF (1<<3)\n \n-extern enum safe_crlf safe_crlf;\n+extern int safe_crlf;\n \n enum auto_crlf {\n \tAUTO_CRLF_FALSE = 0,\n@@ -66,7 +64,7 @@ extern const char *get_convert_attr_ascii(const char *path);\n /* returns 1 if *dst was used */\n extern int convert_to_git(const struct index_state *istate,\n \t\t\t const char *path, const char *src, size_t len,\n-\t\t\t struct strbuf *dst, enum safe_crlf checksafe);\n+\t\t\t struct strbuf *dst, int checksafe);\n extern int convert_to_working_tree(const char *path, const char *src,\n \t\t\t\t size_t len, struct strbuf *dst);\n extern int async_convert_to_working_tree(const char *path, const char *src,\n@@ -85,7 +83,7 @@ static inline int would_convert_to_git(const struct index_state *istate,\n extern void convert_to_git_filter_fd(const struct index_state *istate,\n \t\t\t\t const char *path, int fd,\n \t\t\t\t struct strbuf *dst,\n-\t\t\t\t enum safe_crlf checksafe);\n+\t\t\t\t int checksafe);\n extern int would_convert_to_git_filter_fd(const char *path);\n \n /*****************************************************************\ndiff --git a/diff.c b/diff.c\nindex fb22b19f09..5e3aaea6e0 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -3524,7 +3524,7 @@ int diff_populate_filespec(struct diff_filespec *s, unsigned int flags)\n \t * demote FAIL to WARN to allow inspecting the situation\n \t * instead of refusing.\n \t */\n-\tenum safe_crlf crlf_warn = (safe_crlf == SAFE_CRLF_FAIL\n+\tint checksafe = (safe_crlf == SAFE_CRLF_FAIL\n \t\t\t\t ? SAFE_CRLF_WARN\n \t\t\t\t : safe_crlf);\n \n@@ -3603,7 +3603,7 @@ int diff_populate_filespec(struct diff_filespec *s, unsigned int flags)\n \t\t/*\n \t\t * Convert from working tree format to canonical git format\n \t\t */\n-\t\tif (convert_to_git(&the_index, s->path, s->data, s->size, &buf, crlf_warn)) {\n+\t\tif (convert_to_git(&the_index, s->path, s->data, s->size, &buf, checksafe)) {\n \t\t\tsize_t size = 0;\n \t\t\tmunmap(s->data, s->size);\n \t\t\ts->should_munmap = 0;\ndiff --git a/environment.c b/environment.c\nindex 63ac38a46f..10ee89a28d 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -49,7 +49,7 @@ enum auto_crlf auto_crlf = AUTO_CRLF_FALSE;\n int check_replace_refs = 1;\n char *git_replace_ref_base;\n enum eol core_eol = EOL_UNSET;\n-enum safe_crlf safe_crlf = SAFE_CRLF_WARN;\n+int safe_crlf = SAFE_CRLF_WARN;\n unsigned whitespace_rule_cfg = WS_DEFAULT_RULE;\n enum branch_track git_branch_track = BRANCH_TRACK_REMOTE;\n enum rebase_setup_type autorebase = AUTOREBASE_NEVER;\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 3da70ac650..78e002392e 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -133,7 +133,7 @@ static struct cached_object *find_cached_object(const unsigned char *sha1)\n }\n \n \n-static enum safe_crlf get_safe_crlf(unsigned flags)\n+static int get_checksafe(unsigned flags)\n {\n \tif (flags & HASH_RENORMALIZE)\n \t\treturn SAFE_CRLF_RENORMALIZE;\n@@ -1752,7 +1752,7 @@ static int index_mem(struct object_id *oid, void *buf, size_t size,\n \tif ((type == OBJ_BLOB) && path) {\n \t\tstruct strbuf nbuf = STRBUF_INIT;\n \t\tif (convert_to_git(&the_index, path, buf, size, &nbuf,\n-\t\t\t\t get_safe_crlf(flags))) {\n+\t\t\t\t get_checksafe(flags))) {\n \t\t\tbuf = strbuf_detach(&nbuf, &size);\n \t\t\tre_allocated = 1;\n \t\t}\n@@ -1786,7 +1786,7 @@ static int index_stream_convert_blob(struct object_id *oid, int fd,\n \tassert(would_convert_to_git_filter_fd(path));\n \n \tconvert_to_git_filter_fd(&the_index, path, fd, &sbuf,\n-\t\t\t\t get_safe_crlf(flags));\n+\t\t\t\t get_checksafe(flags));\n \n \tif (write_object)\n \t\tret = write_sha1_file(sbuf.buf, sbuf.len, typename(OBJ_BLOB),\n-- \n2.16.0.rc0.4.ga4e00d4fa4\n\n"},{"id":"335573","messageId":"20171231080554.22301-1-tboegi@web.de","threadId":"47511","inReplyTo":"20171229152222.39680-1-lars.schneider@autodesk.com","subject":"[PATCH 2/5] strbuf: add xstrdup_toupper()","fromName":"","fromEmail":"tboegi@web.de","sentAt":"2017-12-31T08:05:54Z","receivedAt":"2017-12-31T08:06:29Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"From: Lars Schneider \n\nCreate a copy of an existing string and make all characters upper case.\nSimilar xstrdup_tolower().\n\nThis function is used in a subsequent commit.\n\nSigned-off-by: Lars Schneider \nSigned-off-by: Torsten Bögershausen \n---\n strbuf.c | 13 +++++++++++++\n strbuf.h | 1 +\n 2 files changed, 14 insertions(+)\n\ndiff --git a/strbuf.c b/strbuf.c\nindex 8007be8fba..ee05626dc1 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -785,6 +785,19 @@ char *xstrdup_tolower(const char *string)\n \treturn result;\n }\n \n+char *xstrdup_toupper(const char *string)\n+{\n+\tchar *result;\n+\tsize_t len, i;\n+\n+\tlen = strlen(string);\n+\tresult = xmallocz(len);\n+\tfor (i = 0; i < len; i++)\n+\t\tresult[i] = toupper(string[i]);\n+\tresult[i] = '\\0';\n+\treturn result;\n+}\n+\n char *xstrvfmt(const char *fmt, va_list ap)\n {\n \tstruct strbuf buf = STRBUF_INIT;\ndiff --git a/strbuf.h b/strbuf.h\nindex 14c8c10d66..df7ced53ed 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -607,6 +607,7 @@ __attribute__((format (printf,2,3)))\n extern int fprintf_ln(FILE *fp, const char *fmt, ...);\n \n char *xstrdup_tolower(const char *);\n+char *xstrdup_toupper(const char *);\n \n /**\n * Create a newly allocated string using printf format. You can do this easily\n-- \n2.16.0.rc0.4.ga4e00d4fa4\n\n"},{"id":"335574","messageId":"20171231080558.22401-1-tboegi@web.de","threadId":"47511","inReplyTo":"20171229152222.39680-1-lars.schneider@autodesk.com","subject":"[PATCH 4/5] utf8: add function to detect a missing UTF-16/32 BOM","fromName":"","fromEmail":"tboegi@web.de","sentAt":"2017-12-31T08:05:58Z","receivedAt":"2017-12-31T08:06:31Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"From: Lars Schneider \n\nIf the endianness is not defined in the encoding name, then let's\nbe strict and require a BOM to avoid any encoding confusion. The\nhas_missing_utf_bom() function returns true if a required BOM is\nmissing.\n\nThe Unicode standard instructs to assume big-endian if there in no BOM\nfor UTF-16/32 [1][2]. However, the W3C/WHATWG encoding standard used\nin HTML5 recommends to assume little-endian to \"deal with deployed\ncontent\" [3]. Strictly requiring a BOM seems to be the safest option\nfor content in Git.\n\nThis function is used in a subsequent commit.\n\n[1] http://unicode.org/faq/utf_bom.html#gen6\n[2] http://www.unicode.org/versions/Unicode10.0.0/ch03.pdf\n Section 3.10, D98, page 132\n[3] https://encoding.spec.whatwg.org/#utf-16le\n\nSigned-off-by: Lars Schneider \nSigned-off-by: Torsten Bögershausen \n---\n utf8.c | 13 +++++++++++++\n utf8.h | 16 ++++++++++++++++\n 2 files changed, 29 insertions(+)\n\ndiff --git a/utf8.c b/utf8.c\nindex 776660ee12..1978d6c42a 100644\n--- a/utf8.c\n+++ b/utf8.c\n@@ -562,6 +562,19 @@ int has_prohibited_utf_bom(const char *enc, const char *data, size_t len)\n \t);\n }\n \n+int has_missing_utf_bom(const char *enc, const char *data, size_t len)\n+{\n+\treturn (\n+\t !strcmp(enc, \"UTF-16\") &&\n+\t !(has_bom_prefix(data, len, utf16_be_bom, sizeof(utf16_be_bom)) ||\n+\t has_bom_prefix(data, len, utf16_le_bom, sizeof(utf16_le_bom)))\n+\t) || (\n+\t !strcmp(enc, \"UTF-32\") &&\n+\t !(has_bom_prefix(data, len, utf32_be_bom, sizeof(utf32_be_bom)) ||\n+\t has_bom_prefix(data, len, utf32_le_bom, sizeof(utf32_le_bom)))\n+\t);\n+}\n+\n /*\n * Returns first character length in bytes for multi-byte `text` according to\n * `encoding`.\ndiff --git a/utf8.h b/utf8.h\nindex 4711429af9..26b5e91852 100644\n--- a/utf8.h\n+++ b/utf8.h\n@@ -79,4 +79,20 @@ void strbuf_utf8_align(struct strbuf *buf, align_type position, unsigned int wid\n */\n int has_prohibited_utf_bom(const char *enc, const char *data, size_t len);\n \n+/*\n+ * If the endianness is not defined in the encoding name, then we\n+ * require a BOM. The function returns true if a required BOM is missing.\n+ *\n+ * The Unicode standard instructs to assume big-endian if there\n+ * in no BOM for UTF-16/32 [1][2]. However, the W3C/WHATWG\n+ * encoding standard used in HTML5 recommends to assume\n+ * little-endian to \"deal with deployed content\" [3].\n+ *\n+ * [1] http://unicode.org/faq/utf_bom.html#gen6\n+ * [2] http://www.unicode.org/versions/Unicode10.0.0/ch03.pdf\n+ * Section 3.10, D98, page 132\n+ * [3] https://encoding.spec.whatwg.org/#utf-16le\n+ */\n+int has_missing_utf_bom(const char *enc, const char *data, size_t len);\n+\n #endif\n-- \n2.16.0.rc0.4.ga4e00d4fa4\n\n"},{"id":"335581","messageId":"96B6CD4C-0A0C-47F5-922D-B8BAFB832FD1@gmail.com","threadId":"47511","inReplyTo":"20171231080552.22258-1-tboegi@web.de","subject":"Re: [PATCH 1/5] convert_to_git(): checksafe becomes an integer","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-12-31T12:23:25Z","receivedAt":"2017-12-31T12:23:37Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 31 Dec 2017, at 09:05, tboegi@web.de wrote:\n> \n> From: Torsten Bögershausen \n> \n> When calling convert_to_git(), the checksafe parameter has been used to\n> check if commit would give a non-roundtrip conversion of EOL.\n> \n> When checksafe was introduced, 3 values had been in use:\n> SAFE_CRLF_FALSE: no warning\n> SAFE_CRLF_FAIL: reject the commit if EOL do not roundtrip\n> SAFE_CRLF_WARN: warn the user if EOL do not roundtrip\n\nIn general, I really like the direction as this simplifies\nmy patch later on in 5/5. However, I see a few problems:\n\n(1) The prefix \"SAFE_CRLF\" confuses me because the main theme\n is EOL and CRLF just happens to be a EOL type.\n\n What do you think about something like this:\n CONVERT_ERROR_IGNORE 0\n CONVERT_EOL_ERROR_DIE (1<<0)\n CONVERT_EOL_ERROR_WARN (1<<1)\n CONVERT_EOL_TO_LF (1<<2)\n CONVERT_EOL_KEEP_CRLF (1<<3)\n CONVERT_ENCODE_ERROR_DIE (1<<4)\n\n(2) We mix error reporting switches (FALSE/FAIL/WARN) with\n switches that change the content (RENORMALIZE/KEEP_CRLF).\n Plus, these the switches should be mutually exclusive\n (e.g. we don't want to enable the FAIL and WARN bit at\n the same time).\n\n(3) We kind of replicate some flags defined in cache.h:\n #define HASH_WRITE_OBJECT 1\n #define HASH_RENORMALIZE 4\n\n\n- Lars\n\n\n"},{"id":"335595","messageId":"20180101215959.1460-1-tboegi@web.de","threadId":"47511","inReplyTo":"96B6CD4C-0A0C-47F5-922D-B8BAFB832FD1@gmail.com","subject":"[PATCH v3 1/1] convert_to_git(): checksafe becomes int conv_flags","fromName":"","fromEmail":"tboegi@web.de","sentAt":"2018-01-01T21:59:59Z","receivedAt":"2018-01-01T22:00:28Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"From: Torsten Bögershausen \n\nWhen calling convert_to_git(), the checksafe parameter has been used to\ncheck if commit would give a non-roundtrip conversion of EOL.\n\nWhen checksafe was introduced, 3 values had been in use:\nSAFE_CRLF_FALSE: no warning\nSAFE_CRLF_FAIL: reject the commit if EOL do not roundtrip\nSAFE_CRLF_WARN: warn the user if EOL do not roundtrip\n\nAlready today the integer value 0 is passed as the parameter checksafe\ninstead of the correct enum value SAFE_CRLF_FALSE.\n\nTurn the whole call chain to use an integer with single bits, which\ncan be extended in the next commits:\n- The global configuration variable safe_crlf is now conv_flags_eol.\n- The parameter checksafe is renamed into conv_flags.\n\nHelped-By: Lars Schneider \nSigned-off-by: Torsten Bögershausen \n---\nThis is my suggestion.\n(1) The flag bits had been renamed.\n(2) The (theoretical ?) mix of WARN/FAIL is still there,\n I am not sure if this is a real problem.\n\n(3) There are 2 reasons that CONV_EOL_RENORMALIZE is set.\n Either in a renormalizing merge, or by running\n git add --renormalize .\n Therefor HASH_RENORMALIZE is not the same as CONV_EOL_RENORMALIZE.\n\napply.c | 6 +++---\n combine-diff.c | 2 +-\n config.c | 7 +++++--\n convert.c | 38 +++++++++++++++++++-------------------\n convert.h | 17 +++++++----------\n diff.c | 8 ++++----\n environment.c | 2 +-\n sha1_file.c | 12 ++++++------\n 8 files changed, 46 insertions(+), 46 deletions(-)\n\ndiff --git a/apply.c b/apply.c\nindex 321a9fa68d..f8b67bfee2 100644\n--- a/apply.c\n+++ b/apply.c\n@@ -2263,8 +2263,8 @@ static void show_stats(struct apply_state *state, struct patch *patch)\n static int read_old_data(struct stat *st, struct patch *patch,\n \t\t\t const char *path, struct strbuf *buf)\n {\n-\tenum safe_crlf safe_crlf = patch->crlf_in_old ?\n-\t\tSAFE_CRLF_KEEP_CRLF : SAFE_CRLF_RENORMALIZE;\n+\tint conv_flags = patch->crlf_in_old ?\n+\t\tCONV_EOL_KEEP_CRLF : CONV_EOL_RENORMALIZE;\n \tswitch (st->st_mode & S_IFMT) {\n \tcase S_IFLNK:\n \t\tif (strbuf_readlink(buf, path, st->st_size) < 0)\n@@ -2281,7 +2281,7 @@ static int read_old_data(struct stat *st, struct patch *patch,\n \t\t * should never look at the index when explicit crlf option\n \t\t * is given.\n \t\t */\n-\t\tconvert_to_git(NULL, path, buf->buf, buf->len, buf, safe_crlf);\n+\t\tconvert_to_git(NULL, path, buf->buf, buf->len, buf, conv_flags);\n \t\treturn 0;\n \tdefault:\n \t\treturn -1;\ndiff --git a/combine-diff.c b/combine-diff.c\nindex 2505de119a..dbc877d0fe 100644\n--- a/combine-diff.c\n+++ b/combine-diff.c\n@@ -1053,7 +1053,7 @@ static void show_patch_diff(struct combine_diff_path *elem, int num_parent,\n \t\t\tif (is_file) {\n \t\t\t\tstruct strbuf buf = STRBUF_INIT;\n \n-\t\t\t\tif (convert_to_git(&the_index, elem->path, result, len, &buf, safe_crlf)) {\n+\t\t\t\tif (convert_to_git(&the_index, elem->path, result, len, &buf, conv_flags_eol)) {\n \t\t\t\t\tfree(result);\n \t\t\t\t\tresult = strbuf_detach(&buf, &len);\n \t\t\t\t\tresult_size = len;\ndiff --git a/config.c b/config.c\nindex e617c2018d..bdc7ce2a7e 100644\n--- a/config.c\n+++ b/config.c\n@@ -1149,11 +1149,14 @@ static int git_default_core_config(const char *var, const char *value)\n \t}\n \n \tif (!strcmp(var, \"core.safecrlf\")) {\n+\t\tint eol_rndtrp_die;\n \t\tif (value && !strcasecmp(value, \"warn\")) {\n-\t\t\tsafe_crlf = SAFE_CRLF_WARN;\n+\t\t\tconv_flags_eol = CONV_EOL_RNDTRP_WARN;\n \t\t\treturn 0;\n \t\t}\n-\t\tsafe_crlf = git_config_bool(var, value);\n+\t\teol_rndtrp_die = git_config_bool(var, value);\n+\t\tconv_flags_eol = eol_rndtrp_die ?\n+\t\t\tCONV_EOL_RNDTRP_DIE : CONV_EOL_RNDTRP_WARN;\n \t\treturn 0;\n \t}\n \ndiff --git a/convert.c b/convert.c\nindex 1a41a48e15..0207ddab24 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -193,30 +193,30 @@ static enum eol output_eol(enum crlf_action crlf_action)\n \treturn core_eol;\n }\n \n-static void check_safe_crlf(const char *path, enum crlf_action crlf_action,\n+static void check_conv_flags_eol(const char *path, enum crlf_action crlf_action,\n \t\t\t struct text_stat *old_stats, struct text_stat *new_stats,\n-\t\t\t enum safe_crlf checksafe)\n+\t\t\t int conv_flags)\n {\n \tif (old_stats->crlf && !new_stats->crlf ) {\n \t\t/*\n \t\t * CRLFs would not be restored by checkout\n \t\t */\n-\t\tif (checksafe == SAFE_CRLF_WARN)\n+\t\tif (conv_flags & CONV_EOL_RNDTRP_DIE)\n+\t\t\tdie(_(\"CRLF would be replaced by LF in %s.\"), path);\n+\t\telse if (conv_flags & CONV_EOL_RNDTRP_WARN)\n \t\t\twarning(_(\"CRLF will be replaced by LF in %s.\\n\"\n \t\t\t\t \"The file will have its original line\"\n \t\t\t\t \" endings in your working directory.\"), path);\n-\t\telse /* i.e. SAFE_CRLF_FAIL */\n-\t\t\tdie(_(\"CRLF would be replaced by LF in %s.\"), path);\n \t} else if (old_stats->lonelf && !new_stats->lonelf ) {\n \t\t/*\n \t\t * CRLFs would be added by checkout\n \t\t */\n-\t\tif (checksafe == SAFE_CRLF_WARN)\n+\t\tif (conv_flags & CONV_EOL_RNDTRP_DIE)\n+\t\t\tdie(_(\"LF would be replaced by CRLF in %s\"), path);\n+\t\telse if (conv_flags & CONV_EOL_RNDTRP_WARN)\n \t\t\twarning(_(\"LF will be replaced by CRLF in %s.\\n\"\n \t\t\t\t \"The file will have its original line\"\n \t\t\t\t \" endings in your working directory.\"), path);\n-\t\telse /* i.e. SAFE_CRLF_FAIL */\n-\t\t\tdie(_(\"LF would be replaced by CRLF in %s\"), path);\n \t}\n }\n \n@@ -268,7 +268,7 @@ static int will_convert_lf_to_crlf(size_t len, struct text_stat *stats,\n static int crlf_to_git(const struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n \t\t struct strbuf *buf,\n-\t\t enum crlf_action crlf_action, enum safe_crlf checksafe)\n+\t\t enum crlf_action crlf_action, int conv_flags)\n {\n \tstruct text_stat stats;\n \tchar *dst;\n@@ -298,12 +298,12 @@ static int crlf_to_git(const struct index_state *istate,\n \t\t * unless we want to renormalize in a merge or\n \t\t * cherry-pick.\n \t\t */\n-\t\tif ((checksafe != SAFE_CRLF_RENORMALIZE) &&\n+\t\tif ((!(conv_flags & CONV_EOL_RENORMALIZE)) &&\n \t\t has_crlf_in_index(istate, path))\n \t\t\tconvert_crlf_into_lf = 0;\n \t}\n-\tif ((checksafe == SAFE_CRLF_WARN ||\n-\t (checksafe == SAFE_CRLF_FAIL)) && len) {\n+\tif (((conv_flags & CONV_EOL_RNDTRP_WARN) ||\n+\t ((conv_flags & CONV_EOL_RNDTRP_DIE) && len))) {\n \t\tstruct text_stat new_stats;\n \t\tmemcpy(&new_stats, &stats, sizeof(new_stats));\n \t\t/* simulate \"git add\" */\n@@ -316,7 +316,7 @@ static int crlf_to_git(const struct index_state *istate,\n \t\t\tnew_stats.crlf += new_stats.lonelf;\n \t\t\tnew_stats.lonelf = 0;\n \t\t}\n-\t\tcheck_safe_crlf(path, crlf_action, &stats, &new_stats, checksafe);\n+\t\tcheck_conv_flags_eol(path, crlf_action, &stats, &new_stats, conv_flags);\n \t}\n \tif (!convert_crlf_into_lf)\n \t\treturn 0;\n@@ -1129,7 +1129,7 @@ const char *get_convert_attr_ascii(const char *path)\n \n int convert_to_git(const struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n- struct strbuf *dst, enum safe_crlf checksafe)\n+\t\t struct strbuf *dst, int conv_flags)\n {\n \tint ret = 0;\n \tstruct conv_attrs ca;\n@@ -1144,8 +1144,8 @@ int convert_to_git(const struct index_state *istate,\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\n \t}\n-\tif (checksafe != SAFE_CRLF_KEEP_CRLF) {\n-\t\tret |= crlf_to_git(istate, path, src, len, dst, ca.crlf_action, checksafe);\n+\tif (!(conv_flags & CONV_EOL_KEEP_CRLF)) {\n+\t\tret |= crlf_to_git(istate, path, src, len, dst, ca.crlf_action, conv_flags);\n \t\tif (ret && dst) {\n \t\t\tsrc = dst->buf;\n \t\t\tlen = dst->len;\n@@ -1156,7 +1156,7 @@ int convert_to_git(const struct index_state *istate,\n \n void convert_to_git_filter_fd(const struct index_state *istate,\n \t\t\t const char *path, int fd, struct strbuf *dst,\n-\t\t\t enum safe_crlf checksafe)\n+\t\t\t int conv_flags)\n {\n \tstruct conv_attrs ca;\n \tconvert_attrs(&ca, path);\n@@ -1167,7 +1167,7 @@ void convert_to_git_filter_fd(const struct index_state *istate,\n \tif (!apply_filter(path, NULL, 0, fd, dst, ca.drv, CAP_CLEAN, NULL))\n \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n \n-\tcrlf_to_git(istate, path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);\n+\tcrlf_to_git(istate, path, dst->buf, dst->len, dst, ca.crlf_action, conv_flags);\n \tident_to_git(path, dst->buf, dst->len, dst, ca.ident);\n }\n \n@@ -1226,7 +1226,7 @@ int renormalize_buffer(const struct index_state *istate, const char *path,\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\n \t}\n-\treturn ret | convert_to_git(istate, path, src, len, dst, SAFE_CRLF_RENORMALIZE);\n+\treturn ret | convert_to_git(istate, path, src, len, dst, CONV_EOL_RENORMALIZE);\n }\n \n /*****************************************************************\ndiff --git a/convert.h b/convert.h\nindex 4f2da225a8..21a176f16c 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -8,15 +8,12 @@\n \n struct index_state;\n \n-enum safe_crlf {\n-\tSAFE_CRLF_FALSE = 0,\n-\tSAFE_CRLF_FAIL = 1,\n-\tSAFE_CRLF_WARN = 2,\n-\tSAFE_CRLF_RENORMALIZE = 3,\n-\tSAFE_CRLF_KEEP_CRLF = 4\n-};\n+#define CONV_EOL_RNDTRP_DIE (1<<0)\n+#define CONV_EOL_RNDTRP_WARN (1<<1)\n+#define CONV_EOL_RENORMALIZE (1<<2)\n+#define CONV_EOL_KEEP_CRLF (1<<3)\n \n-extern enum safe_crlf safe_crlf;\n+extern int conv_flags_eol;\n \n enum auto_crlf {\n \tAUTO_CRLF_FALSE = 0,\n@@ -66,7 +63,7 @@ extern const char *get_convert_attr_ascii(const char *path);\n /* returns 1 if *dst was used */\n extern int convert_to_git(const struct index_state *istate,\n \t\t\t const char *path, const char *src, size_t len,\n-\t\t\t struct strbuf *dst, enum safe_crlf checksafe);\n+\t\t\t struct strbuf *dst, int conv_flags);\n extern int convert_to_working_tree(const char *path, const char *src,\n \t\t\t\t size_t len, struct strbuf *dst);\n extern int async_convert_to_working_tree(const char *path, const char *src,\n@@ -85,7 +82,7 @@ static inline int would_convert_to_git(const struct index_state *istate,\n extern void convert_to_git_filter_fd(const struct index_state *istate,\n \t\t\t\t const char *path, int fd,\n \t\t\t\t struct strbuf *dst,\n-\t\t\t\t enum safe_crlf checksafe);\n+\t\t\t\t int conv_flags);\n extern int would_convert_to_git_filter_fd(const char *path);\n \n /*****************************************************************\ndiff --git a/diff.c b/diff.c\nindex fb22b19f09..2470af52b2 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -3524,9 +3524,9 @@ int diff_populate_filespec(struct diff_filespec *s, unsigned int flags)\n \t * demote FAIL to WARN to allow inspecting the situation\n \t * instead of refusing.\n \t */\n-\tenum safe_crlf crlf_warn = (safe_crlf == SAFE_CRLF_FAIL\n-\t\t\t\t ? SAFE_CRLF_WARN\n-\t\t\t\t : safe_crlf);\n+\tint conv_flags = (conv_flags_eol == CONV_EOL_RNDTRP_DIE\n+\t\t\t\t ? CONV_EOL_RNDTRP_WARN\n+\t\t\t\t : conv_flags_eol);\n \n \tif (!DIFF_FILE_VALID(s))\n \t\tdie(\"internal error: asking to populate invalid file.\");\n@@ -3603,7 +3603,7 @@ int diff_populate_filespec(struct diff_filespec *s, unsigned int flags)\n \t\t/*\n \t\t * Convert from working tree format to canonical git format\n \t\t */\n-\t\tif (convert_to_git(&the_index, s->path, s->data, s->size, &buf, crlf_warn)) {\n+\t\tif (convert_to_git(&the_index, s->path, s->data, s->size, &buf, conv_flags)) {\n \t\t\tsize_t size = 0;\n \t\t\tmunmap(s->data, s->size);\n \t\t\ts->should_munmap = 0;\ndiff --git a/environment.c b/environment.c\nindex 63ac38a46f..0df21eb633 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -49,7 +49,7 @@ enum auto_crlf auto_crlf = AUTO_CRLF_FALSE;\n int check_replace_refs = 1;\n char *git_replace_ref_base;\n enum eol core_eol = EOL_UNSET;\n-enum safe_crlf safe_crlf = SAFE_CRLF_WARN;\n+int conv_flags_eol = CONV_EOL_RNDTRP_WARN;\n unsigned whitespace_rule_cfg = WS_DEFAULT_RULE;\n enum branch_track git_branch_track = BRANCH_TRACK_REMOTE;\n enum rebase_setup_type autorebase = AUTOREBASE_NEVER;\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 3da70ac650..c0714c1107 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -133,14 +133,14 @@ static struct cached_object *find_cached_object(const unsigned char *sha1)\n }\n \n \n-static enum safe_crlf get_safe_crlf(unsigned flags)\n+static int get_conv_flags(unsigned flags)\n {\n \tif (flags & HASH_RENORMALIZE)\n-\t\treturn SAFE_CRLF_RENORMALIZE;\n+\t\treturn CONV_EOL_RENORMALIZE;\n \telse if (flags & HASH_WRITE_OBJECT)\n-\t\treturn safe_crlf;\n+\t\treturn conv_flags_eol;\n \telse\n-\t\treturn SAFE_CRLF_FALSE;\n+\t\treturn 0;\n }\n \n \n@@ -1752,7 +1752,7 @@ static int index_mem(struct object_id *oid, void *buf, size_t size,\n \tif ((type == OBJ_BLOB) && path) {\n \t\tstruct strbuf nbuf = STRBUF_INIT;\n \t\tif (convert_to_git(&the_index, path, buf, size, &nbuf,\n-\t\t\t\t get_safe_crlf(flags))) {\n+\t\t\t\t get_conv_flags(flags))) {\n \t\t\tbuf = strbuf_detach(&nbuf, &size);\n \t\t\tre_allocated = 1;\n \t\t}\n@@ -1786,7 +1786,7 @@ static int index_stream_convert_blob(struct object_id *oid, int fd,\n \tassert(would_convert_to_git_filter_fd(path));\n \n \tconvert_to_git_filter_fd(&the_index, path, fd, &sbuf,\n-\t\t\t\t get_safe_crlf(flags));\n+\t\t\t\t get_conv_flags(flags));\n \n \tif (write_object)\n \t\tret = write_sha1_file(sbuf.buf, sbuf.len, typename(OBJ_BLOB),\n-- \n2.16.0.rc0.4.ga4e00d4fa4\n\n"},{"id":"335618","messageId":"37D4996F-CC42-4D01-B15E-6287F1F5BF83@gmail.com","threadId":"47511","inReplyTo":"20180101215959.1460-1-tboegi@web.de","subject":"Re: [PATCH v3 1/1] convert_to_git(): checksafe becomes int conv_flags","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2018-01-02T19:11:51Z","receivedAt":"2018-01-02T19:12:07Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 01 Jan 2018, at 22:59, tboegi@web.de wrote:\n> \n> From: Torsten Bögershausen \n> \n> When calling convert_to_git(), the checksafe parameter has been used to\n> check if commit would give a non-roundtrip conversion of EOL.\n> \n> When checksafe was introduced, 3 values had been in use:\n> SAFE_CRLF_FALSE: no warning\n> SAFE_CRLF_FAIL: reject the commit if EOL do not roundtrip\n> SAFE_CRLF_WARN: warn the user if EOL do not roundtrip\n> \n> Already today the integer value 0 is passed as the parameter checksafe\n> instead of the correct enum value SAFE_CRLF_FALSE.\n> \n> Turn the whole call chain to use an integer with single bits, which\n> can be extended in the next commits:\n> - The global configuration variable safe_crlf is now conv_flags_eol.\n> - The parameter checksafe is renamed into conv_flags.\n> \n> Helped-By: Lars Schneider \n> Signed-off-by: Torsten Bögershausen \n> ---\n> This is my suggestion.\n> (1) The flag bits had been renamed.\n> (2) The (theoretical ?) mix of WARN/FAIL is still there,\n> I am not sure if this is a real problem.\n> \n> (3) There are 2 reasons that CONV_EOL_RENORMALIZE is set.\n> Either in a renormalizing merge, or by running\n> git add --renormalize .\n> Therefor HASH_RENORMALIZE is not the same as CONV_EOL_RENORMALIZE.\n> \n> apply.c | 6 +++---\n> combine-diff.c | 2 +-\n> config.c | 7 +++++--\n> convert.c | 38 +++++++++++++++++++-------------------\n> convert.h | 17 +++++++----------\n> diff.c | 8 ++++----\n> environment.c | 2 +-\n> sha1_file.c | 12 ++++++------\n> 8 files changed, 46 insertions(+), 46 deletions(-)\n> ...\n> -static void check_safe_crlf(const char *path, enum crlf_action crlf_action,\n> +static void check_conv_flags_eol(const char *path, enum crlf_action crlf_action,\n> \t\t\t struct text_stat *old_stats, struct text_stat *new_stats,\n> -\t\t\t enum safe_crlf checksafe)\n> +\t\t\t int conv_flags)\n> {\n> \tif (old_stats->crlf && !new_stats->crlf ) {\n> \t\t/*\n> \t\t * CRLFs would not be restored by checkout\n> \t\t */\n> -\t\tif (checksafe == SAFE_CRLF_WARN)\n> +\t\tif (conv_flags & CONV_EOL_RNDTRP_DIE)\n> +\t\t\tdie(_(\"CRLF would be replaced by LF in %s.\"), path);\n> +\t\telse if (conv_flags & CONV_EOL_RNDTRP_WARN)\n\nHere we go with CONV_EOL_RNDTRP_DIE if there is garbage in conv_flags.\nGarbage example: CONV_EOL_RNDTRP_DIE *and* CONV_EOL_RNDTRP_WARN are set.\n\nGood!\n\n> \t...\n> /*****************************************************************\n> diff --git a/diff.c b/diff.c\n> index fb22b19f09..2470af52b2 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -3524,9 +3524,9 @@ int diff_populate_filespec(struct diff_filespec *s, unsigned int flags)\n> \t * demote FAIL to WARN to allow inspecting the situation\n> \t * instead of refusing.\n> \t */\n> -\tenum safe_crlf crlf_warn = (safe_crlf == SAFE_CRLF_FAIL\n> -\t\t\t\t ? SAFE_CRLF_WARN\n> -\t\t\t\t : safe_crlf);\n> +\tint conv_flags = (conv_flags_eol == CONV_EOL_RNDTRP_DIE\n> +\t\t\t\t ? CONV_EOL_RNDTRP_WARN\n> +\t\t\t\t : conv_flags_eol);\n\nIf there is garbage in conv_flags_eol then we would not demote the DIE\nflag here.\n\nHow about something like that:\nint conv_flags = conv_flags_eol & ~CONV_EOL_RNDTRP_DIE;\n\n---\n\nIn general I like the patch as I think the variables are a bit easier to understand. \nOne thing I stumbled over while reading the patch:\n\nThe global variable \"conv_flags_eol\". I think the Git coding guidelines\nhave no recommendation for naming global variables. I think a \"global_conv_flags_eol\"\nor something would have helped me. I can also see how others might frown upon such \na naming scheme.\n\nIf this patch gets accepted then I will rebase my encoding series on top of it:\nhttps://public-inbox.org/git/20171229152222.39680-1-lars.schneider@autodesk.com/\n\n\nThanks,\nLars\n\n"},{"id":"335701","messageId":"20180103053654.GA428@tor.lan","threadId":"47511","inReplyTo":"37D4996F-CC42-4D01-B15E-6287F1F5BF83@gmail.com","subject":"Re: [PATCH v3 1/1] convert_to_git(): checksafe becomes int conv_flags","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2018-01-03T05:36:54Z","receivedAt":"2018-01-03T05:37:14Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Tue, Jan 02, 2018 at 08:11:51PM +0100, Lars Schneider wrote:\n \n[snip]\n\n> > /*****************************************************************\n> > diff --git a/diff.c b/diff.c\n> > index fb22b19f09..2470af52b2 100644\n> > --- a/diff.c\n> > +++ b/diff.c\n> > @@ -3524,9 +3524,9 @@ int diff_populate_filespec(struct diff_filespec *s, unsigned int flags)\n> > \t * demote FAIL to WARN to allow inspecting the situation\n> > \t * instead of refusing.\n> > \t */\n> > -\tenum safe_crlf crlf_warn = (safe_crlf == SAFE_CRLF_FAIL\n> > -\t\t\t\t ? SAFE_CRLF_WARN\n> > -\t\t\t\t : safe_crlf);\n> > +\tint conv_flags = (conv_flags_eol == CONV_EOL_RNDTRP_DIE\n> > +\t\t\t\t ? CONV_EOL_RNDTRP_WARN\n> > +\t\t\t\t : conv_flags_eol);\n> \n> If there is garbage in conv_flags_eol then we would not demote the DIE\n> flag here.\n> \n> How about something like that:\n> int conv_flags = conv_flags_eol & ~CONV_EOL_RNDTRP_DIE;\n\nThe next version will probably look like this:\n\nint diff_populate_filespec(struct diff_filespec *s, unsigned int flags)\n{\n\tint size_only = flags & CHECK_SIZE_ONLY;\n\tint err = 0;\n\tint conv_flags = conv_flags_eol;\n\t/*\n\t * demote FAIL to WARN to allow inspecting the situation\n\t * instead of refusing.\n\t */\n\tif (conv_flags & CONV_EOL_RNDTRP_DIE)\n\t\tconv_flags =CONV_EOL_RNDTRP_WARN;\n\n> \n> ---\n> \n> In general I like the patch as I think the variables are a bit easier to understand. \n> One thing I stumbled over while reading the patch:\n> \n> The global variable \"conv_flags_eol\". I think the Git coding guidelines\n> have no recommendation for naming global variables. I think a \"global_conv_flags_eol\"\n> or something would have helped me. I can also see how others might frown upon such \n> a naming scheme.\n\nI don't have a problem with \"global_conv_flags_eol\".\nThank for the comments, let's wait for more comments before I send out V4.\n"},{"id":"335708","messageId":"8C03FD92-FB9A-4065-9D88-0FBD268F8737@gmail.com","threadId":"47511","inReplyTo":"20180103053654.GA428@tor.lan","subject":"Re: [PATCH v3 1/1] convert_to_git(): checksafe becomes int conv_flags","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2018-01-03T09:37:02Z","receivedAt":"2018-01-03T09:37:31Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 03 Jan 2018, at 06:36, Torsten Bögershausen wrote:\n> \n> On Tue, Jan 02, 2018 at 08:11:51PM +0100, Lars Schneider wrote:\n> \n> [snip]\n> \n>>> /*****************************************************************\n>>> diff --git a/diff.c b/diff.c\n>>> index fb22b19f09..2470af52b2 100644\n>>> --- a/diff.c\n>>> +++ b/diff.c\n>>> @@ -3524,9 +3524,9 @@ int diff_populate_filespec(struct diff_filespec *s, unsigned int flags)\n>>> \t * demote FAIL to WARN to allow inspecting the situation\n>>> \t * instead of refusing.\n>>> \t */\n>>> -\tenum safe_crlf crlf_warn = (safe_crlf == SAFE_CRLF_FAIL\n>>> -\t\t\t\t ? SAFE_CRLF_WARN\n>>> -\t\t\t\t : safe_crlf);\n>>> +\tint conv_flags = (conv_flags_eol == CONV_EOL_RNDTRP_DIE\n>>> +\t\t\t\t ? CONV_EOL_RNDTRP_WARN\n>>> +\t\t\t\t : conv_flags_eol);\n>> \n>> If there is garbage in conv_flags_eol then we would not demote the DIE\n>> flag here.\n>> \n>> How about something like that:\n>> int conv_flags = conv_flags_eol & ~CONV_EOL_RNDTRP_DIE;\n> \n> The next version will probably look like this:\n> \n> int diff_populate_filespec(struct diff_filespec *s, unsigned int flags)\n> {\n> \tint size_only = flags & CHECK_SIZE_ONLY;\n> \tint err = 0;\n> \tint conv_flags = conv_flags_eol;\n> \t/*\n> \t * demote FAIL to WARN to allow inspecting the situation\n> \t * instead of refusing.\n> \t */\n> \tif (conv_flags & CONV_EOL_RNDTRP_DIE)\n> \t\tconv_flags =CONV_EOL_RNDTRP_WARN;\n\nThis looks good. Sorry, I just realized that my suggestion \nwas garbage as it only disabled the bit. It did not demote \nDIE to WARN.\n\nOne final nit:\nCan we be sure that 'conv_flags_eol' is only ever used for CONV_EOL_RNDTRP_DIE\nand CONV_EOL_RNDTRP_WARN? Maybe a solution like that would be more future\nproof?\n\n if (conv_flags & CONV_EOL_RNDTRP_DIE) {\n conv_flags &= ~CONV_EOL_RNDTRP_DIE;\n conv_flags |= CONV_EOL_RNDTRP_WARN;\n }\n\n\n>> \n>> ---\n>> \n>> In general I like the patch as I think the variables are a bit easier to understand. \n>> One thing I stumbled over while reading the patch:\n>> \n>> The global variable \"conv_flags_eol\". I think the Git coding guidelines\n>> have no recommendation for naming global variables. I think a \"global_conv_flags_eol\"\n>> or something would have helped me. I can also see how others might frown upon such \n>> a naming scheme.\n> \n> I don't have a problem with \"global_conv_flags_eol\".\n> Thank for the comments, let's wait for more comments before I send out V4.\n\nSounds good. A \"global_\" like prefix is not used anywhere else in Git...\n\n- Lars"},{"id":"335930","messageId":"alpine.DEB.2.21.1.1801051542550.31@MININT-6BKU6QN.europe.corp.microsoft.com","threadId":"47511","inReplyTo":"20171231080601.22444-1-tboegi@web.de","subject":"Re: [PATCH 5/5] convert: add support for 'checkout-encoding' attribute","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-01-05T14:45:00Z","receivedAt":"2018-01-05T14:45:18Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sun, 31 Dec 2017, tboegi@web.de wrote:\n\n> diff --git a/t/t0028-checkout-encoding.sh b/t/t0028-checkout-encoding.sh\n> new file mode 100755\n> index 0000000000..1a329ab933\n> --- /dev/null\n> +++ b/t/t0028-checkout-encoding.sh\n> @@ -0,0 +1,197 @@\n> +#!/bin/sh\n> +\n> +test_description='checkout-encoding conversion via gitattributes'\n> +\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'setup test repo' '\n> +\n> +\ttext=\"hallo there!\\ncan you read me?\" &&\n> +\n> +\techo \"*.utf16 text checkout-encoding=utf-16\" >.gitattributes &&\n> +\n> +\tprintf \"$text\" >test.utf8.raw &&\n> +\tprintf \"$text\" | iconv -f UTF-8 -t UTF-16 >test.utf16.raw &&\n> +\tcp test.utf16.raw test.utf16 &&\n> +\n> +\tgit add .gitattributes test.utf16 &&\n> +\tgit commit -m initial\n> +'\n> +\n> +test_expect_success 'ensure UTF-8 is stored in Git' '\n> +\tgit cat-file -p :test.utf16 >test.utf16.git &&\n> +\ttest_cmp_bin test.utf8.raw test.utf16.git &&\n> +\trm test.utf8.raw test.utf16.git\n> +'\n> +\n> +test_expect_success 're-encode to UTF-16 on checkout' '\n> +\trm test.utf16 &&\n> +\tgit checkout test.utf16 &&\n> +\ttest_cmp_bin test.utf16.raw test.utf16 &&\n\nThis fails here, due to an extra CR in test.utf16. This diff fixes it\n(please include it in the next iteration of this patch series):\n\n-- snipsnap --\ndiff --git a/t/t0028-checkout-encoding.sh b/t/t0028-checkout-encoding.sh\nindex 1a329ab933b..aaf61bc9d7c 100755\n--- a/t/t0028-checkout-encoding.sh\n+++ b/t/t0028-checkout-encoding.sh\n@@ -6,6 +6,7 @@ test_description='checkout-encoding conversion via\ngitattributes'\n \n test_expect_success 'setup test repo' '\n \n+\tgit config core.eol lf &&\n \ttext=\"hallo there!\\ncan you read me?\" &&\n \n \techo \"*.utf16 text checkout-encoding=utf-16\" >.gitattributes &&\n\n"},{"id":"335942","messageId":"B45F57FC-E798-4C8F-9C40-200944D86029@gmail.com","threadId":"47511","inReplyTo":"20180101215959.1460-1-tboegi@web.de","subject":"Re: [PATCH v3 1/1] convert_to_git(): checksafe becomes int conv_flags","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2018-01-05T19:00:57Z","receivedAt":"2018-01-05T19:01:18Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 01 Jan 2018, at 22:59, tboegi@web.de wrote:\n> \n> From: Torsten Bögershausen \n> \n> When calling convert_to_git(), the checksafe parameter has been used to\n> check if commit would give a non-roundtrip conversion of EOL.\n> \n> When checksafe was introduced, 3 values had been in use:\n> SAFE_CRLF_FALSE: no warning\n> SAFE_CRLF_FAIL: reject the commit if EOL do not roundtrip\n> SAFE_CRLF_WARN: warn the user if EOL do not roundtrip\n> \n> Already today the integer value 0 is passed as the parameter checksafe\n> instead of the correct enum value SAFE_CRLF_FALSE.\n> \n> Turn the whole call chain to use an integer with single bits, which\n> can be extended in the next commits:\n> - The global configuration variable safe_crlf is now conv_flags_eol.\n> - The parameter checksafe is renamed into conv_flags.\n> \n> Helped-By: Lars Schneider \n> Signed-off-by: Torsten Bögershausen \n> ---\n> This is my suggestion.\n> (1) The flag bits had been renamed.\n> (2) The (theoretical ?) mix of WARN/FAIL is still there,\n> I am not sure if this is a real problem.\n> \n> (3) There are 2 reasons that CONV_EOL_RENORMALIZE is set.\n> Either in a renormalizing merge, or by running\n> git add --renormalize .\n> Therefor HASH_RENORMALIZE is not the same as CONV_EOL_RENORMALIZE.\n\nCan you elaborate a bit? I am diving into the code but I am still confused.\n\nI also noticed that the \"flags\" integer is potentially double booked with\nthe following values (see read-cache.c:add_to_index()):\n\n#define ADD_CACHE_VERBOSE 1\n#define ADD_CACHE_PRETEND 2\n#define ADD_CACHE_IGNORE_ERRORS\t4\n\n#define HASH_WRITE_OBJECT 1\n#define HASH_FORMAT_CHECK 2\n#define HASH_RENORMALIZE 4\n\nIs this intentional? \n\nThanks,\nLars\n\n\nMore context:\n https://public-inbox.org/git/96B6CD4C-0A0C-47F5-922D-B8BAFB832FD1@gmail.com/\n (3) We kind of replicate some flags defined in cache.h:\n #define HASH_WRITE_OBJECT 1\n #define HASH_RENORMALIZE 4\n\n\n"},{"id":"335946","messageId":"08dbcad8-5827-b56b-6586-55592e41003f@web.de","threadId":"47511","inReplyTo":"B45F57FC-E798-4C8F-9C40-200944D86029@gmail.com","subject":"Re: [PATCH v3 1/1] convert_to_git(): checksafe becomes int conv_flags","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2018-01-05T19:33:46Z","receivedAt":"2018-01-05T19:34:07Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 2018-01-05 20:00, Lars Schneider wrote:\n> \n>> On 01 Jan 2018, at 22:59, tboegi@web.de wrote:\n>>\n>> From: Torsten Bögershausen \n>>\n>> When calling convert_to_git(), the checksafe parameter has been used to\n>> check if commit would give a non-roundtrip conversion of EOL.\n>>\n>> When checksafe was introduced, 3 values had been in use:\n>> SAFE_CRLF_FALSE: no warning\n>> SAFE_CRLF_FAIL: reject the commit if EOL do not roundtrip\n>> SAFE_CRLF_WARN: warn the user if EOL do not roundtrip\n>>\n>> Already today the integer value 0 is passed as the parameter checksafe\n>> instead of the correct enum value SAFE_CRLF_FALSE.\n>>\n>> Turn the whole call chain to use an integer with single bits, which\n>> can be extended in the next commits:\n>> - The global configuration variable safe_crlf is now conv_flags_eol.\n>> - The parameter checksafe is renamed into conv_flags.\n>>\n>> Helped-By: Lars Schneider \n>> Signed-off-by: Torsten Bögershausen \n>> ---\n>> This is my suggestion.\n>> (1) The flag bits had been renamed.\n>> (2) The (theoretical ?) mix of WARN/FAIL is still there,\n>> I am not sure if this is a real problem.\n>>\n>> (3) There are 2 reasons that CONV_EOL_RENORMALIZE is set.\n>> Either in a renormalizing merge, or by running\n>> git add --renormalize .\n>> Therefor HASH_RENORMALIZE is not the same as CONV_EOL_RENORMALIZE.\n> \n> Can you elaborate a bit? I am diving into the code but I am still confused.\n> \n> I also noticed that the \"flags\" integer is potentially double booked with\n> the following values (see read-cache.c:add_to_index()):\n> \n> #define ADD_CACHE_VERBOSE 1\n> #define ADD_CACHE_PRETEND 2\n> #define ADD_CACHE_IGNORE_ERRORS\t4\n> \n> #define HASH_WRITE_OBJECT 1\n> #define HASH_FORMAT_CHECK 2\n> #define HASH_RENORMALIZE 4\n> \n> Is this intentional? \n\nAll these flags have a different context,\nand the right #define must be used in the right context/function call.\nSo there is no intention.\nYou start with 1, then use 2 and so on.\n\nThe same way as family Schmidt and family Meier both call\ntheir child \"Hans\".\nThere is no connection.\n\n\n> \n> Thanks,\n> Lars\n> \n> \n> More context:\n> https://public-inbox.org/git/96B6CD4C-0A0C-47F5-922D-B8BAFB832FD1@gmail.com/\n> (3) We kind of replicate some flags defined in cache.h:\n> #define HASH_WRITE_OBJECT 1\n> #define HASH_RENORMALIZE 4\n> \n> \n\n"},{"id":"336027","messageId":"xmqqincfao1b.fsf@gitster.mtv.corp.google.com","threadId":"47511","inReplyTo":"96B6CD4C-0A0C-47F5-922D-B8BAFB832FD1@gmail.com","subject":"Re: [PATCH 1/5] convert_to_git(): checksafe becomes an integer","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-01-05T23:22:56Z","receivedAt":"2018-01-05T23:23:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lars Schneider writes:\n\n>> On 31 Dec 2017, at 09:05, tboegi@web.de wrote:\n>> \n>> From: Torsten Bögershausen \n>> \n>> When calling convert_to_git(), the checksafe parameter has been used to\n>> check if commit would give a non-roundtrip conversion of EOL.\n>> \n>> When checksafe was introduced, 3 values had been in use:\n>> SAFE_CRLF_FALSE: no warning\n>> SAFE_CRLF_FAIL: reject the commit if EOL do not roundtrip\n>> SAFE_CRLF_WARN: warn the user if EOL do not roundtrip\n>\n> In general, I really like the direction as this simplifies\n> my patch later on in 5/5. However, I see a few problems:\n> ...\n\nYes, this looks like a sensible way to go. I saw Torsten's v3 for\n1/5 but will end up queuing v2b, as I suspect 5/5 would need to be\nadjusted for the change between the two versions.\n\nThanks.\n"},{"id":"336043","messageId":"02B54B51-5347-49D3-A932-0641611A3CE1@gmail.com","threadId":"47511","inReplyTo":"xmqqincfao1b.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 1/5] convert_to_git(): checksafe becomes an integer","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2018-01-06T00:55:09Z","receivedAt":"2018-01-06T00:55:24Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 06 Jan 2018, at 00:22, Junio C Hamano wrote:\n> \n> Lars Schneider writes:\n> \n>>> On 31 Dec 2017, at 09:05, tboegi@web.de wrote:\n>>> \n>>> From: Torsten Bögershausen \n>>> \n>>> When calling convert_to_git(), the checksafe parameter has been used to\n>>> check if commit would give a non-roundtrip conversion of EOL.\n>>> \n>>> When checksafe was introduced, 3 values had been in use:\n>>> SAFE_CRLF_FALSE: no warning\n>>> SAFE_CRLF_FAIL: reject the commit if EOL do not roundtrip\n>>> SAFE_CRLF_WARN: warn the user if EOL do not roundtrip\n>> \n>> In general, I really like the direction as this simplifies\n>> my patch later on in 5/5. However, I see a few problems:\n>> ...\n> \n> Yes, this looks like a sensible way to go. I saw Torsten's v3 for\n> 1/5 but will end up queuing v2b, as I suspect 5/5 would need to be\n> adjusted for the change between the two versions.\n\nI just send out my v3 which integrates the latest version of Torsten's\npatch into my series:\n\nhttps://public-inbox.org/git/20180106004808.77513-1-lars.schneider@autodesk.com/\n\nPlease note: I did not rebase my series. Therefore, there is a small\nconflict with 86ff70a0f0 (convert: tighten the safe autocrlf handling, 2017-11-26)\nbecause has_cr_in_index() was renamed to has_crlf_in_index().\n\n@Junio: What do you prefer in these cases? A rebased series or the conflict?\n\nThanks,\nLars"}]}