{"thread":{"id":"47551","subject":"[PATCH v3 0/7] convert: add support for different encodings","startedAt":"2018-01-06T00:48:47Z","lastAt":"2018-01-09T06:20:53Z","messageCount":15,"participants":["lars.schneider@autodesk.com","Torsten Bögershausen","Lars Schneider","Junio C Hamano"],"isPatch":true,"patchVersion":3,"patchTotal":7},"messages":[{"id":"336035","messageId":"20180106004808.77513-1-lars.schneider@autodesk.com","threadId":"47551","inReplyTo":null,"subject":"[PATCH v3 0/7] convert: add support for different encodings","fromName":"","fromEmail":"lars.schneider@autodesk.com","sentAt":"2018-01-06T00:48:01Z","receivedAt":"2018-01-06T00:48:47Z","isPatch":true,"sender":{"key":"lars.schneider@autodesk.com","avatar":"https://gravatar.com/avatar/524188df3a38b02692619777bb90d9cc735db1ad75f5304e0bf09f8efd299775?d=mp&s=160"},"body":"From: Lars Schneider \n\nHi,\n\nPatches 1-5 and 6 are helper functions and preparation.\nPatch 6 is the actual change.\n\nI am still torn between \"checkout-encoding\" and \"working-tree-encoding\"\nas attribute name. I am happy to hear arguments for/against one or the\nother.\n\nChanges since v2:\n\n* Added Torsten's crlfsave refactoring patch (patch 5)\n @Torsten: I tried to make the commit message more clean, added\n some comments to and renamed conv_flags_eol to\n global_conv_flags_eol.\n\n* Improved documentation and commit message (Torsten)\n\n* Removed unnecessary NUL assignment in xstrdup_tolower() (Torsten)\n\n* Set \"git config core.eol lf\" to made the test run on Windows (Dscho)\n\n* Made BOM arrays static (Ramsay)\n\n\nThanks,\nLars\n\n\n RFC: https://public-inbox.org/git/BDB9B884-6D17-4BE3-A83C-F67E2AFA2B46@gmail.com/\n v1: https://public-inbox.org/git/20171211155023.1405-1-lars.schneider@autodesk.com/\n v2: https://public-inbox.org/git/20171229152222.39680-1-lars.schneider@autodesk.com/\n\n\n\nBase Ref: master\nWeb-Diff: https://github.com/larsxschneider/git/commit/f21a1841a4\nCheckout: git fetch https://github.com/larsxschneider/git encoding-v3 && git checkout f21a1841a4\n\n\n### Interdiff (v2..v3):\n\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex 0039bd38c3..1bc03e69cb 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -285,11 +285,18 @@ In these cases you can teach Git the encoding of a file in the working\n directory with the `checkout-encoding` attribute. If a file with this\n attributes is added to Git, then Git reencodes the content from the\n specified encoding to UTF-8 and stores the result in its internal data\n-structure. On checkout the content is encoded back to the specified\n-encoding.\n+structure (called \"the index\"). On checkout the content is encoded\n+back to the specified encoding.\n\n-Please note that using the `checkout-encoding` attribute has a number\n-of drawbacks:\n+Please note that using the `checkout-encoding` attribute may have a\n+number of pitfalls:\n+\n+- Git clients that do not support the `checkout-encoding` attribute\n+ will checkout the respective files UTF-8 encoded and not in the\n+ expected encoding. Consequently, these files will appear different\n+ which typically causes trouble. This is in particular the case for\n+ older Git versions and alternative Git implementations such as JGit\n+ or libgit2 (as of January 2018).\n\n - Reencoding content to non-UTF encodings (e.g. SHIFT-JIS) can cause\n errors as the conversion might not be round trip safe.\n@@ -297,12 +304,6 @@ of drawbacks:\n - Reencoding content requires resources that might slow down certain\n Git operations (e.g 'git checkout' or 'git add').\n\n-- Git clients that do not support the `checkout-encoding` attribute or\n- the used encoding will checkout the respective files as UTF-8 encoded.\n- That means the content appears to be different which could cause\n- trouble. Affected clients are older Git versions and alternative Git\n- implementations such as JGit or libgit2 (as of January 2018).\n-\n Use the `checkout-encoding` attribute only if you cannot store a file in\n UTF-8 encoding and if you want Git to be able to process the content as\n text.\ndiff --git a/apply.c b/apply.c\nindex c4bd5cf1f2..f8b67bfee2 100644\n--- a/apply.c\n+++ b/apply.c\n@@ -2263,8 +2263,8 @@ static void show_stats(struct apply_state *state, struct patch *patch)\n static int read_old_data(struct stat *st, struct patch *patch,\n \t\t\t const char *path, struct strbuf *buf)\n {\n-\tenum safe_crlf safe_crlf = patch->crlf_in_old ?\n-\t\tSAFE_CRLF_KEEP_CRLF : SAFE_CRLF_RENORMALIZE;\n+\tint conv_flags = patch->crlf_in_old ?\n+\t\tCONV_EOL_KEEP_CRLF : CONV_EOL_RENORMALIZE;\n \tswitch (st->st_mode & S_IFMT) {\n \tcase S_IFLNK:\n \t\tif (strbuf_readlink(buf, path, st->st_size) < 0)\n@@ -2281,7 +2281,7 @@ static int read_old_data(struct stat *st, struct patch *patch,\n \t\t * should never look at the index when explicit crlf option\n \t\t * is given.\n \t\t */\n-\t\tconvert_to_git(NULL, path, buf->buf, buf->len, buf, safe_crlf, 0);\n+\t\tconvert_to_git(NULL, path, buf->buf, buf->len, buf, conv_flags);\n \t\treturn 0;\n \tdefault:\n \t\treturn -1;\ndiff --git a/blame.c b/blame.c\nindex 388b66897b..2893f3c103 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -229,7 +229,7 @@ static struct commit *fake_working_tree_commit(struct diff_options *opt,\n \t\tif (strbuf_read(&buf, 0, 0) < 0)\n \t\t\tdie_errno(\"failed to read from stdin\");\n \t}\n-\tconvert_to_git(&the_index, path, buf.buf, buf.len, &buf, 0, 0);\n+\tconvert_to_git(&the_index, path, buf.buf, buf.len, &buf, 0);\n \torigin->file.ptr = buf.buf;\n \torigin->file.size = buf.len;\n \tpretend_sha1_file(buf.buf, buf.len, OBJ_BLOB, origin->blob_oid.hash);\ndiff --git a/combine-diff.c b/combine-diff.c\nindex 4555e49b5f..19f30c3353 100644\n--- a/combine-diff.c\n+++ b/combine-diff.c\n@@ -1053,7 +1053,7 @@ static void show_patch_diff(struct combine_diff_path *elem, int num_parent,\n \t\t\tif (is_file) {\n \t\t\t\tstruct strbuf buf = STRBUF_INIT;\n\n-\t\t\t\tif (convert_to_git(&the_index, elem->path, result, len, &buf, safe_crlf, 0)) {\n+\t\t\t\tif (convert_to_git(&the_index, elem->path, result, len, &buf, global_conv_flags_eol)) {\n \t\t\t\t\tfree(result);\n \t\t\t\t\tresult = strbuf_detach(&buf, &len);\n \t\t\t\t\tresult_size = len;\ndiff --git a/config.c b/config.c\nindex e617c2018d..1f003fbb90 100644\n--- a/config.c\n+++ b/config.c\n@@ -1149,11 +1149,14 @@ static int git_default_core_config(const char *var, const char *value)\n \t}\n\n \tif (!strcmp(var, \"core.safecrlf\")) {\n+\t\tint eol_rndtrp_die;\n \t\tif (value && !strcasecmp(value, \"warn\")) {\n-\t\t\tsafe_crlf = SAFE_CRLF_WARN;\n+\t\t\tglobal_conv_flags_eol = CONV_EOL_RNDTRP_WARN;\n \t\t\treturn 0;\n \t\t}\n-\t\tsafe_crlf = git_config_bool(var, value);\n+\t\teol_rndtrp_die = git_config_bool(var, value);\n+\t\tglobal_conv_flags_eol = eol_rndtrp_die ?\n+\t\t\tCONV_EOL_RNDTRP_DIE : CONV_EOL_RNDTRP_WARN;\n \t\treturn 0;\n \t}\n\ndiff --git a/convert.c b/convert.c\nindex ca7b2f3e5c..525958bb56 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -194,30 +194,30 @@ static enum eol output_eol(enum crlf_action crlf_action)\n \treturn core_eol;\n }\n\n-static void check_safe_crlf(const char *path, enum crlf_action crlf_action,\n+static void check_global_conv_flags_eol(const char *path, enum crlf_action crlf_action,\n \t\t\t struct text_stat *old_stats, struct text_stat *new_stats,\n-\t\t\t enum safe_crlf checksafe)\n+\t\t\t int conv_flags)\n {\n \tif (old_stats->crlf && !new_stats->crlf ) {\n \t\t/*\n \t\t * CRLFs would not be restored by checkout\n \t\t */\n-\t\tif (checksafe == SAFE_CRLF_WARN)\n+\t\tif (conv_flags & CONV_EOL_RNDTRP_DIE)\n+\t\t\tdie(_(\"CRLF would be replaced by LF in %s.\"), path);\n+\t\telse if (conv_flags & CONV_EOL_RNDTRP_WARN)\n \t\t\twarning(_(\"CRLF will be replaced by LF in %s.\\n\"\n \t\t\t\t \"The file will have its original line\"\n \t\t\t\t \" endings in your working directory.\"), path);\n-\t\telse /* i.e. SAFE_CRLF_FAIL */\n-\t\t\tdie(_(\"CRLF would be replaced by LF in %s.\"), path);\n \t} else if (old_stats->lonelf && !new_stats->lonelf ) {\n \t\t/*\n \t\t * CRLFs would be added by checkout\n \t\t */\n-\t\tif (checksafe == SAFE_CRLF_WARN)\n+\t\tif (conv_flags & CONV_EOL_RNDTRP_DIE)\n+\t\t\tdie(_(\"LF would be replaced by CRLF in %s\"), path);\n+\t\telse if (conv_flags & CONV_EOL_RNDTRP_WARN)\n \t\t\twarning(_(\"LF will be replaced by CRLF in %s.\\n\"\n \t\t\t\t \"The file will have its original line\"\n \t\t\t\t \" endings in your working directory.\"), path);\n-\t\telse /* i.e. SAFE_CRLF_FAIL */\n-\t\t\tdie(_(\"LF would be replaced by CRLF in %s\"), path);\n \t}\n }\n\n@@ -287,7 +287,7 @@ static struct encoding {\n static const char *default_encoding = \"UTF-8\";\n\n static int encode_to_git(const char *path, const char *src, size_t src_len,\n-\t\t\t struct strbuf *buf, struct encoding *enc, int write_obj)\n+\t\t\t struct strbuf *buf, struct encoding *enc, int conv_flags)\n {\n \tchar *dst;\n \tint dst_len;\n@@ -318,7 +318,7 @@ static int encode_to_git(const char *path, const char *src, size_t src_len,\n \t\t\t\"file.\");\n\n \t\tadvise(advise_msg, path, enc->name, enc->name, enc->name);\n-\t\tif (write_obj)\n+\t\tif (conv_flags & CONV_WRITE_OBJECT)\n \t\t\tdie(error_msg, path, enc->name);\n \t\telse\n \t\t\terror(error_msg, path, enc->name);\n@@ -333,7 +333,7 @@ static int encode_to_git(const char *path, const char *src, size_t src_len,\n \t\t\t\"%sBE/%sLE as checkout encoding or add a BOM to the \"\n \t\t\t\"file.\");\n \t\tadvise(advise_msg, path, enc->name, enc->name, enc->name);\n-\t\tif (write_obj)\n+\t\tif (conv_flags & CONV_WRITE_OBJECT)\n \t\t\tdie(error_msg, path, enc->name);\n \t\telse\n \t\t\terror(error_msg, path, enc->name);\n@@ -350,7 +350,7 @@ static int encode_to_git(const char *path, const char *src, size_t src_len,\n \t\t * working tree. Let's try to avoid this by screaming loud.\n \t\t */\n \t\tconst char* msg = _(\"failed to encode '%s' from %s to %s\");\n-\t\tif (write_obj)\n+\t\tif (conv_flags & CONV_WRITE_OBJECT)\n \t\t\tdie(msg, path, enc->name, default_encoding);\n \t\telse\n \t\t\terror(msg, path, enc->name, default_encoding);\n@@ -373,7 +373,7 @@ static int encode_to_git(const char *path, const char *src, size_t src_len,\n \t * [1] http://unicode.org/faq/utf_bom.html#gen2\n \t * [2] https://support.microsoft.com/en-us/help/170559/prb-conversion-problem-between-shift-jis-and-unicode\n \t */\n-\tif (write_obj && !strcmp(enc->name, \"SHIFT-JIS\")) {\n+\tif ((conv_flags & CONV_WRITE_OBJECT) && !strcmp(enc->name, \"SHIFT-JIS\")) {\n \t\tchar *re_src;\n \t\tint re_src_len;\n\n@@ -388,7 +388,7 @@ static int encode_to_git(const char *path, const char *src, size_t src_len,\n \t\t memcmp(src, re_src, src_len)) {\n \t\t\tconst char* msg = _(\"encoding '%s' from %s to %s and \"\n \t\t\t\t\t \"back is not the same\");\n-\t\t\tif (write_obj)\n+\t\t\tif (conv_flags & CONV_WRITE_OBJECT)\n \t\t\t\tdie(msg, path, enc->name, default_encoding);\n \t\t\telse\n \t\t\t\terror(msg, path, enc->name, default_encoding);\n@@ -429,7 +429,7 @@ static int encode_to_worktree(const char *path, const char *src, size_t src_len,\n static int crlf_to_git(const struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n \t\t struct strbuf *buf,\n-\t\t enum crlf_action crlf_action, enum safe_crlf checksafe)\n+\t\t enum crlf_action crlf_action, int conv_flags)\n {\n \tstruct text_stat stats;\n \tchar *dst;\n@@ -459,12 +459,12 @@ static int crlf_to_git(const struct index_state *istate,\n \t\t * unless we want to renormalize in a merge or\n \t\t * cherry-pick.\n \t\t */\n-\t\tif ((checksafe != SAFE_CRLF_RENORMALIZE) &&\n+\t\tif ((!(conv_flags & CONV_EOL_RENORMALIZE)) &&\n \t\t has_cr_in_index(istate, path))\n \t\t\tconvert_crlf_into_lf = 0;\n \t}\n-\tif ((checksafe == SAFE_CRLF_WARN ||\n-\t (checksafe == SAFE_CRLF_FAIL)) && len) {\n+\tif (((conv_flags & CONV_EOL_RNDTRP_WARN) ||\n+\t ((conv_flags & CONV_EOL_RNDTRP_DIE) && len))) {\n \t\tstruct text_stat new_stats;\n \t\tmemcpy(&new_stats, &stats, sizeof(new_stats));\n \t\t/* simulate \"git add\" */\n@@ -477,7 +477,7 @@ static int crlf_to_git(const struct index_state *istate,\n \t\t\tnew_stats.crlf += new_stats.lonelf;\n \t\t\tnew_stats.lonelf = 0;\n \t\t}\n-\t\tcheck_safe_crlf(path, crlf_action, &stats, &new_stats, checksafe);\n+\t\tcheck_global_conv_flags_eol(path, crlf_action, &stats, &new_stats, conv_flags);\n \t}\n \tif (!convert_crlf_into_lf)\n \t\treturn 0;\n@@ -1319,7 +1319,7 @@ const char *get_convert_attr_ascii(const char *path)\n\n int convert_to_git(const struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n-\t\t struct strbuf *dst, enum safe_crlf checksafe, int write_obj)\n+\t\t struct strbuf *dst, int conv_flags)\n {\n \tint ret = 0;\n \tstruct conv_attrs ca;\n@@ -1335,14 +1335,14 @@ int convert_to_git(const struct index_state *istate,\n \t\tlen = dst->len;\n \t}\n\n-\tret |= encode_to_git(path, src, len, dst, ca.checkout_encoding, write_obj);\n+\tret |= encode_to_git(path, src, len, dst, ca.checkout_encoding, conv_flags);\n \tif (ret && dst) {\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\n \t}\n\n-\tif (checksafe != SAFE_CRLF_KEEP_CRLF) {\n-\t\tret |= crlf_to_git(istate, path, src, len, dst, ca.crlf_action, checksafe);\n+\tif (!(conv_flags & CONV_EOL_KEEP_CRLF)) {\n+\t\tret |= crlf_to_git(istate, path, src, len, dst, ca.crlf_action, conv_flags);\n \t\tif (ret && dst) {\n \t\t\tsrc = dst->buf;\n \t\t\tlen = dst->len;\n@@ -1353,7 +1353,7 @@ int convert_to_git(const struct index_state *istate,\n\n void convert_to_git_filter_fd(const struct index_state *istate,\n \t\t\t const char *path, int fd, struct strbuf *dst,\n-\t\t\t enum safe_crlf checksafe, int write_obj)\n+\t\t\t int conv_flags)\n {\n \tstruct conv_attrs ca;\n \tconvert_attrs(&ca, path);\n@@ -1364,8 +1364,8 @@ void convert_to_git_filter_fd(const struct index_state *istate,\n \tif (!apply_filter(path, NULL, 0, fd, dst, ca.drv, CAP_CLEAN, NULL))\n \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n\n-\tencode_to_git(path, dst->buf, dst->len, dst, ca.checkout_encoding, write_obj);\n-\tcrlf_to_git(istate, path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);\n+\tencode_to_git(path, dst->buf, dst->len, dst, ca.checkout_encoding, conv_flags);\n+\tcrlf_to_git(istate, path, dst->buf, dst->len, dst, ca.crlf_action, conv_flags);\n \tident_to_git(path, dst->buf, dst->len, dst, ca.ident);\n }\n\n@@ -1430,7 +1430,7 @@ int renormalize_buffer(const struct index_state *istate, const char *path,\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\n \t}\n-\treturn ret | convert_to_git(istate, path, src, len, dst, SAFE_CRLF_RENORMALIZE, 0);\n+\treturn ret | convert_to_git(istate, path, src, len, dst, CONV_EOL_RENORMALIZE);\n }\n\n /*****************************************************************\ndiff --git a/convert.h b/convert.h\nindex 9e4e884ec1..1d9539ed0b 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -8,15 +8,13 @@\n\n struct index_state;\n\n-enum safe_crlf {\n-\tSAFE_CRLF_FALSE = 0,\n-\tSAFE_CRLF_FAIL = 1,\n-\tSAFE_CRLF_WARN = 2,\n-\tSAFE_CRLF_RENORMALIZE = 3,\n-\tSAFE_CRLF_KEEP_CRLF = 4\n-};\n+#define CONV_EOL_RNDTRP_DIE (1<<0) /* Die if CRLF to LF to CRLF is different */\n+#define CONV_EOL_RNDTRP_WARN (1<<1) /* Warn if CRLF to LF to CRLF is different */\n+#define CONV_EOL_RENORMALIZE (1<<2) /* Convert CRLF to LF */\n+#define CONV_EOL_KEEP_CRLF (1<<3) /* Keep CRLF line endings as is */\n+#define CONV_WRITE_OBJECT (1<<4) /* Content is written to the index */\n\n-extern enum safe_crlf safe_crlf;\n+extern int global_conv_flags_eol;\n\n enum auto_crlf {\n \tAUTO_CRLF_FALSE = 0,\n@@ -66,8 +64,7 @@ extern const char *get_convert_attr_ascii(const char *path);\n /* returns 1 if *dst was used */\n extern int convert_to_git(const struct index_state *istate,\n \t\t\t const char *path, const char *src, size_t len,\n-\t\t\t struct strbuf *dst, enum safe_crlf checksafe,\n-\t\t\t int write_obj);\n+\t\t\t struct strbuf *dst, int conv_flags);\n extern int convert_to_working_tree(const char *path, const char *src,\n \t\t\t\t size_t len, struct strbuf *dst);\n extern int async_convert_to_working_tree(const char *path, const char *src,\n@@ -80,14 +77,13 @@ extern int renormalize_buffer(const struct index_state *istate,\n static inline int would_convert_to_git(const struct index_state *istate,\n \t\t\t\t const char *path)\n {\n-\treturn convert_to_git(istate, path, NULL, 0, NULL, 0, 0);\n+\treturn convert_to_git(istate, path, NULL, 0, NULL, 0);\n }\n /* Precondition: would_convert_to_git_filter_fd(path) == true */\n extern void convert_to_git_filter_fd(const struct index_state *istate,\n \t\t\t\t const char *path, int fd,\n \t\t\t\t struct strbuf *dst,\n-\t\t\t\t enum safe_crlf checksafe,\n-\t\t\t\t int write_obj);\n+\t\t\t\t int conv_flags);\n extern int would_convert_to_git_filter_fd(const char *path);\n\n /*****************************************************************\ndiff --git a/diff.c b/diff.c\nindex 16ca0bf0df..fe3ff6e876 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -3516,13 +3516,13 @@ int diff_populate_filespec(struct diff_filespec *s, unsigned int flags)\n {\n \tint size_only = flags & CHECK_SIZE_ONLY;\n \tint err = 0;\n+\tint conv_flags = global_conv_flags_eol;\n \t/*\n \t * demote FAIL to WARN to allow inspecting the situation\n \t * instead of refusing.\n \t */\n-\tenum safe_crlf crlf_warn = (safe_crlf == SAFE_CRLF_FAIL\n-\t\t\t\t ? SAFE_CRLF_WARN\n-\t\t\t\t : safe_crlf);\n+\tif (conv_flags & CONV_EOL_RNDTRP_DIE)\n+\t\tconv_flags = CONV_EOL_RNDTRP_WARN;\n\n \tif (!DIFF_FILE_VALID(s))\n \t\tdie(\"internal error: asking to populate invalid file.\");\n@@ -3599,7 +3599,7 @@ int diff_populate_filespec(struct diff_filespec *s, unsigned int flags)\n \t\t/*\n \t\t * Convert from working tree format to canonical git format\n \t\t */\n-\t\tif (convert_to_git(&the_index, s->path, s->data, s->size, &buf, crlf_warn, 0)) {\n+\t\tif (convert_to_git(&the_index, s->path, s->data, s->size, &buf, conv_flags)) {\n \t\t\tsize_t size = 0;\n \t\t\tmunmap(s->data, s->size);\n \t\t\ts->should_munmap = 0;\ndiff --git a/environment.c b/environment.c\nindex 8fa032f307..888b873f90 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -49,7 +49,7 @@ enum auto_crlf auto_crlf = AUTO_CRLF_FALSE;\n int check_replace_refs = 1;\n char *git_replace_ref_base;\n enum eol core_eol = EOL_UNSET;\n-enum safe_crlf safe_crlf = SAFE_CRLF_WARN;\n+int global_conv_flags_eol = CONV_EOL_RNDTRP_WARN;\n unsigned whitespace_rule_cfg = WS_DEFAULT_RULE;\n enum branch_track git_branch_track = BRANCH_TRACK_REMOTE;\n enum rebase_setup_type autorebase = AUTOREBASE_NEVER;\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 75800248d2..dcb02e9ffd 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -75,14 +75,14 @@ static struct cached_object *find_cached_object(const unsigned char *sha1)\n }\n\n\n-static enum safe_crlf get_safe_crlf(unsigned flags)\n+static int get_conv_flags(unsigned flags)\n {\n \tif (flags & HASH_RENORMALIZE)\n-\t\treturn SAFE_CRLF_RENORMALIZE;\n+\t\treturn CONV_EOL_RENORMALIZE;\n \telse if (flags & HASH_WRITE_OBJECT)\n-\t\treturn safe_crlf;\n+\t\treturn global_conv_flags_eol | CONV_WRITE_OBJECT;\n \telse\n-\t\treturn SAFE_CRLF_FALSE;\n+\t\treturn 0;\n }\n\n\n@@ -1694,8 +1694,7 @@ static int index_mem(struct object_id *oid, void *buf, size_t size,\n \tif ((type == OBJ_BLOB) && path) {\n \t\tstruct strbuf nbuf = STRBUF_INIT;\n \t\tif (convert_to_git(&the_index, path, buf, size, &nbuf,\n-\t\t\t\t get_safe_crlf(flags),\n-\t\t\t\t write_object)) {\n+\t\t\t\t get_conv_flags(flags))) {\n \t\t\tbuf = strbuf_detach(&nbuf, &size);\n \t\t\tre_allocated = 1;\n \t\t}\n@@ -1729,7 +1728,7 @@ static int index_stream_convert_blob(struct object_id *oid, int fd,\n \tassert(would_convert_to_git_filter_fd(path));\n\n \tconvert_to_git_filter_fd(&the_index, path, fd, &sbuf,\n-\t\t\t\t get_safe_crlf(flags), write_object);\n+\t\t\t\t get_conv_flags(flags));\n\n \tif (write_object)\n \t\tret = write_sha1_file(sbuf.buf, sbuf.len, typename(OBJ_BLOB),\ndiff --git a/strbuf.c b/strbuf.c\nindex 54276e96e7..703a1556cb 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -756,7 +756,6 @@ char *xstrdup_tolower(const char *string)\n \tresult = xmallocz(len);\n \tfor (i = 0; i < len; i++)\n \t\tresult[i] = tolower(string[i]);\n-\tresult[i] = '\\0';\n \treturn result;\n }\n\n@@ -769,7 +768,6 @@ char *xstrdup_toupper(const char *string)\n \tresult = xmallocz(len);\n \tfor (i = 0; i < len; i++)\n \t\tresult[i] = toupper(string[i]);\n-\tresult[i] = '\\0';\n \treturn result;\n }\n\ndiff --git a/t/t0028-checkout-encoding.sh b/t/t0028-checkout-encoding.sh\nindex df3cc91269..5f1c911c07 100755\n--- a/t/t0028-checkout-encoding.sh\n+++ b/t/t0028-checkout-encoding.sh\n@@ -7,11 +7,10 @@ test_description='checkout-encoding conversion via gitattributes'\n GIT_TRACE_CHECKOUT_ENCODING=1 && export GIT_TRACE_CHECKOUT_ENCODING\n\n test_expect_success 'setup test repo' '\n+\tgit config core.eol lf &&\n\n \ttext=\"hallo there!\\ncan you read me?\" &&\n-\n \techo \"*.utf16 text checkout-encoding=utf-16\" >.gitattributes &&\n-\n \tprintf \"$text\" >test.utf8.raw &&\n \tprintf \"$text\" | iconv -f UTF-8 -t UTF-16 >test.utf16.raw &&\n \tcp test.utf16.raw test.utf16 &&\ndiff --git a/utf8.c b/utf8.c\nindex 1978d6c42a..f033fec1c2 100644\n--- a/utf8.c\n+++ b/utf8.c\n@@ -544,10 +544,10 @@ static int has_bom_prefix(const char *data, size_t len,\n \treturn (len >= bom_len) && !memcmp(data, bom, bom_len);\n }\n\n-const char utf16_be_bom[] = {0xFE, 0xFF};\n-const char utf16_le_bom[] = {0xFF, 0xFE};\n-const char utf32_be_bom[] = {0x00, 0x00, 0xFE, 0xFF};\n-const char utf32_le_bom[] = {0xFF, 0xFE, 0x00, 0x00};\n+static const char utf16_be_bom[] = {0xFE, 0xFF};\n+static const char utf16_le_bom[] = {0xFF, 0xFE};\n+static const char utf32_be_bom[] = {0x00, 0x00, 0xFE, 0xFF};\n+static const char utf32_le_bom[] = {0xFF, 0xFE, 0x00, 0x00};\n\n int has_prohibited_utf_bom(const char *enc, const char *data, size_t len)\n {\n\n\n### Patches\n\nLars Schneider (6):\n strbuf: remove unnecessary NUL assignment in xstrdup_tolower()\n strbuf: add xstrdup_toupper()\n utf8: add function to detect prohibited UTF-16/32 BOM\n utf8: add function to detect a missing UTF-16/32 BOM\n convert: add support for 'checkout-encoding' attribute\n convert: add tracing for checkout-encoding\n\nTorsten Bögershausen (1):\n convert_to_git(): safe_crlf/checksafe becomes int conv_flags\n\n Documentation/gitattributes.txt | 60 ++++++++++\n apply.c | 6 +-\n combine-diff.c | 2 +-\n config.c | 7 +-\n convert.c | 256 ++++++++++++++++++++++++++++++++++++----\n convert.h | 18 ++-\n diff.c | 8 +-\n environment.c | 2 +-\n sha1_file.c | 12 +-\n strbuf.c | 13 +-\n strbuf.h | 1 +\n t/t0028-checkout-encoding.sh | 198 +++++++++++++++++++++++++++++++\n utf8.c | 37 ++++++\n utf8.h | 25 ++++\n 14 files changed, 597 insertions(+), 48 deletions(-)\n create mode 100755 t/t0028-checkout-encoding.sh\n\n\nbase-commit: 95ec6b1b3393eb6e26da40c565520a8db9796e9f\n--\n2.15.1\n\n"},{"id":"336036","messageId":"20180106004808.77513-2-lars.schneider@autodesk.com","threadId":"47551","inReplyTo":"20180106004808.77513-1-lars.schneider@autodesk.com","subject":"[PATCH v3 1/7] strbuf: remove unnecessary NUL assignment in xstrdup_tolower()","fromName":"","fromEmail":"lars.schneider@autodesk.com","sentAt":"2018-01-06T00:48:02Z","receivedAt":"2018-01-06T00:48:52Z","isPatch":true,"sender":{"key":"lars.schneider@autodesk.com","avatar":"https://gravatar.com/avatar/524188df3a38b02692619777bb90d9cc735db1ad75f5304e0bf09f8efd299775?d=mp&s=160"},"body":"From: Lars Schneider \n\nSince 3733e69464 (use xmallocz to avoid size arithmetic, 2016-02-22) we\nallocate the buffer for the lower case string with xmallocz(). This\nalready ensures a NUL at the end of the allocated buffer.\n\nRemove the unnecessary assignment.\n\nSigned-off-by: Lars Schneider \n---\n strbuf.c | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/strbuf.c b/strbuf.c\nindex 323c49ceb3..b5d03a5029 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -756,7 +756,6 @@ char *xstrdup_tolower(const char *string)\n \tresult = xmallocz(len);\n \tfor (i = 0; i < len; i++)\n \t\tresult[i] = tolower(string[i]);\n-\tresult[i] = '\\0';\n \treturn result;\n }\n \n-- \n2.15.1\n\n"},{"id":"336037","messageId":"20180106004808.77513-4-lars.schneider@autodesk.com","threadId":"47551","inReplyTo":"20180106004808.77513-1-lars.schneider@autodesk.com","subject":"[PATCH v3 3/7] utf8: add function to detect prohibited UTF-16/32 BOM","fromName":"","fromEmail":"lars.schneider@autodesk.com","sentAt":"2018-01-06T00:48:04Z","receivedAt":"2018-01-06T00:48:54Z","isPatch":true,"sender":{"key":"lars.schneider@autodesk.com","avatar":"https://gravatar.com/avatar/524188df3a38b02692619777bb90d9cc735db1ad75f5304e0bf09f8efd299775?d=mp&s=160"},"body":"From: Lars Schneider \n\nWhenever a data stream is declared to be UTF-16BE, UTF-16LE, UTF-32BE\nor UTF-32LE a BOM must not be used [1]. The function returns true if\nthis is the case.\n\nThis function is used in a subsequent commit.\n\n[1] http://unicode.org/faq/utf_bom.html#bom10\n\nSigned-off-by: Lars Schneider \n---\n utf8.c | 24 ++++++++++++++++++++++++\n utf8.h | 9 +++++++++\n 2 files changed, 33 insertions(+)\n\ndiff --git a/utf8.c b/utf8.c\nindex 2c27ce0137..914881cd1f 100644\n--- a/utf8.c\n+++ b/utf8.c\n@@ -538,6 +538,30 @@ char *reencode_string_len(const char *in, int insz,\n }\n #endif\n \n+static int has_bom_prefix(const char *data, size_t len,\n+\t\t\t const char *bom, size_t bom_len)\n+{\n+\treturn (len >= bom_len) && !memcmp(data, bom, bom_len);\n+}\n+\n+static const char utf16_be_bom[] = {0xFE, 0xFF};\n+static const char utf16_le_bom[] = {0xFF, 0xFE};\n+static const char utf32_be_bom[] = {0x00, 0x00, 0xFE, 0xFF};\n+static const char utf32_le_bom[] = {0xFF, 0xFE, 0x00, 0x00};\n+\n+int has_prohibited_utf_bom(const char *enc, const char *data, size_t len)\n+{\n+\treturn (\n+\t (!strcmp(enc, \"UTF-16BE\") || !strcmp(enc, \"UTF-16LE\")) &&\n+\t (has_bom_prefix(data, len, utf16_be_bom, sizeof(utf16_be_bom)) ||\n+\t has_bom_prefix(data, len, utf16_le_bom, sizeof(utf16_le_bom)))\n+\t) || (\n+\t (!strcmp(enc, \"UTF-32BE\") || !strcmp(enc, \"UTF-32LE\")) &&\n+\t (has_bom_prefix(data, len, utf32_be_bom, sizeof(utf32_be_bom)) ||\n+\t has_bom_prefix(data, len, utf32_le_bom, sizeof(utf32_le_bom)))\n+\t);\n+}\n+\n /*\n * Returns first character length in bytes for multi-byte `text` according to\n * `encoding`.\ndiff --git a/utf8.h b/utf8.h\nindex 6bbcf31a83..4711429af9 100644\n--- a/utf8.h\n+++ b/utf8.h\n@@ -70,4 +70,13 @@ typedef enum {\n void strbuf_utf8_align(struct strbuf *buf, align_type position, unsigned int width,\n \t\t const char *s);\n \n+/*\n+ * Whenever a data stream is declared to be UTF-16BE, UTF-16LE, UTF-32BE\n+ * or UTF-32LE a BOM must not be used [1]. The function returns true if\n+ * this is the case.\n+ *\n+ * [1] http://unicode.org/faq/utf_bom.html#bom10\n+ */\n+int has_prohibited_utf_bom(const char *enc, const char *data, size_t len);\n+\n #endif\n-- \n2.15.1\n\n"},{"id":"336038","messageId":"20180106004808.77513-3-lars.schneider@autodesk.com","threadId":"47551","inReplyTo":"20180106004808.77513-1-lars.schneider@autodesk.com","subject":"[PATCH v3 2/7] strbuf: add xstrdup_toupper()","fromName":"","fromEmail":"lars.schneider@autodesk.com","sentAt":"2018-01-06T00:48:03Z","receivedAt":"2018-01-06T00:48:56Z","isPatch":true,"sender":{"key":"lars.schneider@autodesk.com","avatar":"https://gravatar.com/avatar/524188df3a38b02692619777bb90d9cc735db1ad75f5304e0bf09f8efd299775?d=mp&s=160"},"body":"From: Lars Schneider \n\nCreate a copy of an existing string and make all characters upper case.\nSimilar xstrdup_tolower().\n\nThis function is used in a subsequent commit.\n\nSigned-off-by: Lars Schneider \n---\n strbuf.c | 12 ++++++++++++\n strbuf.h | 1 +\n 2 files changed, 13 insertions(+)\n\ndiff --git a/strbuf.c b/strbuf.c\nindex b5d03a5029..703a1556cb 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -759,6 +759,18 @@ char *xstrdup_tolower(const char *string)\n \treturn result;\n }\n \n+char *xstrdup_toupper(const char *string)\n+{\n+\tchar *result;\n+\tsize_t len, i;\n+\n+\tlen = strlen(string);\n+\tresult = xmallocz(len);\n+\tfor (i = 0; i < len; i++)\n+\t\tresult[i] = toupper(string[i]);\n+\treturn result;\n+}\n+\n char *xstrvfmt(const char *fmt, va_list ap)\n {\n \tstruct strbuf buf = STRBUF_INIT;\ndiff --git a/strbuf.h b/strbuf.h\nindex 0a74acb236..2bc148526f 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -616,6 +616,7 @@ __attribute__((format (printf,2,3)))\n extern int fprintf_ln(FILE *fp, const char *fmt, ...);\n \n char *xstrdup_tolower(const char *);\n+char *xstrdup_toupper(const char *);\n \n /**\n * Create a newly allocated string using printf format. You can do this easily\n-- \n2.15.1\n\n"},{"id":"336039","messageId":"20180106004808.77513-5-lars.schneider@autodesk.com","threadId":"47551","inReplyTo":"20180106004808.77513-1-lars.schneider@autodesk.com","subject":"[PATCH v3 4/7] utf8: add function to detect a missing UTF-16/32 BOM","fromName":"","fromEmail":"lars.schneider@autodesk.com","sentAt":"2018-01-06T00:48:05Z","receivedAt":"2018-01-06T00:48:58Z","isPatch":true,"sender":{"key":"lars.schneider@autodesk.com","avatar":"https://gravatar.com/avatar/524188df3a38b02692619777bb90d9cc735db1ad75f5304e0bf09f8efd299775?d=mp&s=160"},"body":"From: Lars Schneider \n\nIf the endianness is not defined in the encoding name, then let's\nbe strict and require a BOM to avoid any encoding confusion. The\nhas_missing_utf_bom() function returns true if a required BOM is\nmissing.\n\nThe Unicode standard instructs to assume big-endian if there in no BOM\nfor UTF-16/32 [1][2]. However, the W3C/WHATWG encoding standard used\nin HTML5 recommends to assume little-endian to \"deal with deployed\ncontent\" [3]. Strictly requiring a BOM seems to be the safest option\nfor content in Git.\n\nThis function is used in a subsequent commit.\n\n[1] http://unicode.org/faq/utf_bom.html#gen6\n[2] http://www.unicode.org/versions/Unicode10.0.0/ch03.pdf\n Section 3.10, D98, page 132\n[3] https://encoding.spec.whatwg.org/#utf-16le\n\nSigned-off-by: Lars Schneider \n---\n utf8.c | 13 +++++++++++++\n utf8.h | 16 ++++++++++++++++\n 2 files changed, 29 insertions(+)\n\ndiff --git a/utf8.c b/utf8.c\nindex 914881cd1f..f033fec1c2 100644\n--- a/utf8.c\n+++ b/utf8.c\n@@ -562,6 +562,19 @@ int has_prohibited_utf_bom(const char *enc, const char *data, size_t len)\n \t);\n }\n \n+int has_missing_utf_bom(const char *enc, const char *data, size_t len)\n+{\n+\treturn (\n+\t !strcmp(enc, \"UTF-16\") &&\n+\t !(has_bom_prefix(data, len, utf16_be_bom, sizeof(utf16_be_bom)) ||\n+\t has_bom_prefix(data, len, utf16_le_bom, sizeof(utf16_le_bom)))\n+\t) || (\n+\t !strcmp(enc, \"UTF-32\") &&\n+\t !(has_bom_prefix(data, len, utf32_be_bom, sizeof(utf32_be_bom)) ||\n+\t has_bom_prefix(data, len, utf32_le_bom, sizeof(utf32_le_bom)))\n+\t);\n+}\n+\n /*\n * Returns first character length in bytes for multi-byte `text` according to\n * `encoding`.\ndiff --git a/utf8.h b/utf8.h\nindex 4711429af9..26b5e91852 100644\n--- a/utf8.h\n+++ b/utf8.h\n@@ -79,4 +79,20 @@ void strbuf_utf8_align(struct strbuf *buf, align_type position, unsigned int wid\n */\n int has_prohibited_utf_bom(const char *enc, const char *data, size_t len);\n \n+/*\n+ * If the endianness is not defined in the encoding name, then we\n+ * require a BOM. The function returns true if a required BOM is missing.\n+ *\n+ * The Unicode standard instructs to assume big-endian if there\n+ * in no BOM for UTF-16/32 [1][2]. However, the W3C/WHATWG\n+ * encoding standard used in HTML5 recommends to assume\n+ * little-endian to \"deal with deployed content\" [3].\n+ *\n+ * [1] http://unicode.org/faq/utf_bom.html#gen6\n+ * [2] http://www.unicode.org/versions/Unicode10.0.0/ch03.pdf\n+ * Section 3.10, D98, page 132\n+ * [3] https://encoding.spec.whatwg.org/#utf-16le\n+ */\n+int has_missing_utf_bom(const char *enc, const char *data, size_t len);\n+\n #endif\n-- \n2.15.1\n\n"},{"id":"336040","messageId":"20180106004808.77513-6-lars.schneider@autodesk.com","threadId":"47551","inReplyTo":"20180106004808.77513-1-lars.schneider@autodesk.com","subject":"[PATCH v3 5/7] convert_to_git(): safe_crlf/checksafe becomes int conv_flags","fromName":"","fromEmail":"lars.schneider@autodesk.com","sentAt":"2018-01-06T00:48:06Z","receivedAt":"2018-01-06T00:49:02Z","isPatch":true,"sender":{"key":"lars.schneider@autodesk.com","avatar":"https://gravatar.com/avatar/524188df3a38b02692619777bb90d9cc735db1ad75f5304e0bf09f8efd299775?d=mp&s=160"},"body":"From: Torsten Bögershausen \n\nWhen calling convert_to_git(), the checksafe parameter defined what\nshould happen if the EOL conversion (CRLF --> LF --> CRLF) does not\nroundtrip cleanly. In addition, it also defined if line endings should\nbe renormalized (CRLF --> LF) or kept as they are.\n\nchecksafe was an safe_crlf enum with these values:\nSAFE_CRLF_FALSE: do nothing in case of EOL roundtrip errors\nSAFE_CRLF_FAIL: die in case of EOL roundtrip errors\nSAFE_CRLF_WARN: print a warning in case of EOL roundtrip errors\nSAFE_CRLF_RENORMALIZE: change CRLF to LF\nSAFE_CRLF_KEEP_CRLF: keep all line endings as they are\n\nIn some cases the integer value 0 was passed as checksafe parameter\ninstead of the correct enum value SAFE_CRLF_FALSE. That was no problem\nbecause SAFE_CRLF_FALSE is defined as 0.\n\nFALSE/FAIL/WARN are different from RENORMALIZE and KEEP_CRLF. Therefore,\nan enum is not ideal. Let's use a integer bit pattern instead and rename\nthe parameter to conv_flags to make it more generically usable. This\nallows us to extend the bit pattern in a subsequent commit.\n\nHelped-By: Lars Schneider \nSigned-off-by: Torsten Bögershausen \nSigned-off-by: Lars Schneider \n---\n apply.c | 6 +++---\n combine-diff.c | 2 +-\n config.c | 7 +++++--\n convert.c | 38 +++++++++++++++++++-------------------\n convert.h | 17 +++++++----------\n diff.c | 8 ++++----\n environment.c | 2 +-\n sha1_file.c | 12 ++++++------\n 8 files changed, 46 insertions(+), 46 deletions(-)\n\ndiff --git a/apply.c b/apply.c\nindex 321a9fa68d..f8b67bfee2 100644\n--- a/apply.c\n+++ b/apply.c\n@@ -2263,8 +2263,8 @@ static void show_stats(struct apply_state *state, struct patch *patch)\n static int read_old_data(struct stat *st, struct patch *patch,\n \t\t\t const char *path, struct strbuf *buf)\n {\n-\tenum safe_crlf safe_crlf = patch->crlf_in_old ?\n-\t\tSAFE_CRLF_KEEP_CRLF : SAFE_CRLF_RENORMALIZE;\n+\tint conv_flags = patch->crlf_in_old ?\n+\t\tCONV_EOL_KEEP_CRLF : CONV_EOL_RENORMALIZE;\n \tswitch (st->st_mode & S_IFMT) {\n \tcase S_IFLNK:\n \t\tif (strbuf_readlink(buf, path, st->st_size) < 0)\n@@ -2281,7 +2281,7 @@ static int read_old_data(struct stat *st, struct patch *patch,\n \t\t * should never look at the index when explicit crlf option\n \t\t * is given.\n \t\t */\n-\t\tconvert_to_git(NULL, path, buf->buf, buf->len, buf, safe_crlf);\n+\t\tconvert_to_git(NULL, path, buf->buf, buf->len, buf, conv_flags);\n \t\treturn 0;\n \tdefault:\n \t\treturn -1;\ndiff --git a/combine-diff.c b/combine-diff.c\nindex 2505de119a..19f30c3353 100644\n--- a/combine-diff.c\n+++ b/combine-diff.c\n@@ -1053,7 +1053,7 @@ static void show_patch_diff(struct combine_diff_path *elem, int num_parent,\n \t\t\tif (is_file) {\n \t\t\t\tstruct strbuf buf = STRBUF_INIT;\n \n-\t\t\t\tif (convert_to_git(&the_index, elem->path, result, len, &buf, safe_crlf)) {\n+\t\t\t\tif (convert_to_git(&the_index, elem->path, result, len, &buf, global_conv_flags_eol)) {\n \t\t\t\t\tfree(result);\n \t\t\t\t\tresult = strbuf_detach(&buf, &len);\n \t\t\t\t\tresult_size = len;\ndiff --git a/config.c b/config.c\nindex e617c2018d..1f003fbb90 100644\n--- a/config.c\n+++ b/config.c\n@@ -1149,11 +1149,14 @@ static int git_default_core_config(const char *var, const char *value)\n \t}\n \n \tif (!strcmp(var, \"core.safecrlf\")) {\n+\t\tint eol_rndtrp_die;\n \t\tif (value && !strcasecmp(value, \"warn\")) {\n-\t\t\tsafe_crlf = SAFE_CRLF_WARN;\n+\t\t\tglobal_conv_flags_eol = CONV_EOL_RNDTRP_WARN;\n \t\t\treturn 0;\n \t\t}\n-\t\tsafe_crlf = git_config_bool(var, value);\n+\t\teol_rndtrp_die = git_config_bool(var, value);\n+\t\tglobal_conv_flags_eol = eol_rndtrp_die ?\n+\t\t\tCONV_EOL_RNDTRP_DIE : CONV_EOL_RNDTRP_WARN;\n \t\treturn 0;\n \t}\n \ndiff --git a/convert.c b/convert.c\nindex 20d7ab67bd..f39150cde9 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -193,30 +193,30 @@ static enum eol output_eol(enum crlf_action crlf_action)\n \treturn core_eol;\n }\n \n-static void check_safe_crlf(const char *path, enum crlf_action crlf_action,\n+static void check_global_conv_flags_eol(const char *path, enum crlf_action crlf_action,\n \t\t\t struct text_stat *old_stats, struct text_stat *new_stats,\n-\t\t\t enum safe_crlf checksafe)\n+\t\t\t int conv_flags)\n {\n \tif (old_stats->crlf && !new_stats->crlf ) {\n \t\t/*\n \t\t * CRLFs would not be restored by checkout\n \t\t */\n-\t\tif (checksafe == SAFE_CRLF_WARN)\n+\t\tif (conv_flags & CONV_EOL_RNDTRP_DIE)\n+\t\t\tdie(_(\"CRLF would be replaced by LF in %s.\"), path);\n+\t\telse if (conv_flags & CONV_EOL_RNDTRP_WARN)\n \t\t\twarning(_(\"CRLF will be replaced by LF in %s.\\n\"\n \t\t\t\t \"The file will have its original line\"\n \t\t\t\t \" endings in your working directory.\"), path);\n-\t\telse /* i.e. SAFE_CRLF_FAIL */\n-\t\t\tdie(_(\"CRLF would be replaced by LF in %s.\"), path);\n \t} else if (old_stats->lonelf && !new_stats->lonelf ) {\n \t\t/*\n \t\t * CRLFs would be added by checkout\n \t\t */\n-\t\tif (checksafe == SAFE_CRLF_WARN)\n+\t\tif (conv_flags & CONV_EOL_RNDTRP_DIE)\n+\t\t\tdie(_(\"LF would be replaced by CRLF in %s\"), path);\n+\t\telse if (conv_flags & CONV_EOL_RNDTRP_WARN)\n \t\t\twarning(_(\"LF will be replaced by CRLF in %s.\\n\"\n \t\t\t\t \"The file will have its original line\"\n \t\t\t\t \" endings in your working directory.\"), path);\n-\t\telse /* i.e. SAFE_CRLF_FAIL */\n-\t\t\tdie(_(\"LF would be replaced by CRLF in %s\"), path);\n \t}\n }\n \n@@ -259,7 +259,7 @@ static int will_convert_lf_to_crlf(size_t len, struct text_stat *stats,\n static int crlf_to_git(const struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n \t\t struct strbuf *buf,\n-\t\t enum crlf_action crlf_action, enum safe_crlf checksafe)\n+\t\t enum crlf_action crlf_action, int conv_flags)\n {\n \tstruct text_stat stats;\n \tchar *dst;\n@@ -289,12 +289,12 @@ static int crlf_to_git(const struct index_state *istate,\n \t\t * unless we want to renormalize in a merge or\n \t\t * cherry-pick.\n \t\t */\n-\t\tif ((checksafe != SAFE_CRLF_RENORMALIZE) &&\n+\t\tif ((!(conv_flags & CONV_EOL_RENORMALIZE)) &&\n \t\t has_cr_in_index(istate, path))\n \t\t\tconvert_crlf_into_lf = 0;\n \t}\n-\tif ((checksafe == SAFE_CRLF_WARN ||\n-\t (checksafe == SAFE_CRLF_FAIL)) && len) {\n+\tif (((conv_flags & CONV_EOL_RNDTRP_WARN) ||\n+\t ((conv_flags & CONV_EOL_RNDTRP_DIE) && len))) {\n \t\tstruct text_stat new_stats;\n \t\tmemcpy(&new_stats, &stats, sizeof(new_stats));\n \t\t/* simulate \"git add\" */\n@@ -307,7 +307,7 @@ static int crlf_to_git(const struct index_state *istate,\n \t\t\tnew_stats.crlf += new_stats.lonelf;\n \t\t\tnew_stats.lonelf = 0;\n \t\t}\n-\t\tcheck_safe_crlf(path, crlf_action, &stats, &new_stats, checksafe);\n+\t\tcheck_global_conv_flags_eol(path, crlf_action, &stats, &new_stats, conv_flags);\n \t}\n \tif (!convert_crlf_into_lf)\n \t\treturn 0;\n@@ -1120,7 +1120,7 @@ const char *get_convert_attr_ascii(const char *path)\n \n int convert_to_git(const struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n- struct strbuf *dst, enum safe_crlf checksafe)\n+\t\t struct strbuf *dst, int conv_flags)\n {\n \tint ret = 0;\n \tstruct conv_attrs ca;\n@@ -1135,8 +1135,8 @@ int convert_to_git(const struct index_state *istate,\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\n \t}\n-\tif (checksafe != SAFE_CRLF_KEEP_CRLF) {\n-\t\tret |= crlf_to_git(istate, path, src, len, dst, ca.crlf_action, checksafe);\n+\tif (!(conv_flags & CONV_EOL_KEEP_CRLF)) {\n+\t\tret |= crlf_to_git(istate, path, src, len, dst, ca.crlf_action, conv_flags);\n \t\tif (ret && dst) {\n \t\t\tsrc = dst->buf;\n \t\t\tlen = dst->len;\n@@ -1147,7 +1147,7 @@ int convert_to_git(const struct index_state *istate,\n \n void convert_to_git_filter_fd(const struct index_state *istate,\n \t\t\t const char *path, int fd, struct strbuf *dst,\n-\t\t\t enum safe_crlf checksafe)\n+\t\t\t int conv_flags)\n {\n \tstruct conv_attrs ca;\n \tconvert_attrs(&ca, path);\n@@ -1158,7 +1158,7 @@ void convert_to_git_filter_fd(const struct index_state *istate,\n \tif (!apply_filter(path, NULL, 0, fd, dst, ca.drv, CAP_CLEAN, NULL))\n \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n \n-\tcrlf_to_git(istate, path, dst->buf, dst->len, dst, ca.crlf_action, checksafe);\n+\tcrlf_to_git(istate, path, dst->buf, dst->len, dst, ca.crlf_action, conv_flags);\n \tident_to_git(path, dst->buf, dst->len, dst, ca.ident);\n }\n \n@@ -1217,7 +1217,7 @@ int renormalize_buffer(const struct index_state *istate, const char *path,\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\n \t}\n-\treturn ret | convert_to_git(istate, path, src, len, dst, SAFE_CRLF_RENORMALIZE);\n+\treturn ret | convert_to_git(istate, path, src, len, dst, CONV_EOL_RENORMALIZE);\n }\n \n /*****************************************************************\ndiff --git a/convert.h b/convert.h\nindex 4f2da225a8..65ab3e5167 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -8,15 +8,12 @@\n \n struct index_state;\n \n-enum safe_crlf {\n-\tSAFE_CRLF_FALSE = 0,\n-\tSAFE_CRLF_FAIL = 1,\n-\tSAFE_CRLF_WARN = 2,\n-\tSAFE_CRLF_RENORMALIZE = 3,\n-\tSAFE_CRLF_KEEP_CRLF = 4\n-};\n+#define CONV_EOL_RNDTRP_DIE (1<<0) /* Die if CRLF to LF to CRLF is different */\n+#define CONV_EOL_RNDTRP_WARN (1<<1) /* Warn if CRLF to LF to CRLF is different */\n+#define CONV_EOL_RENORMALIZE (1<<2) /* Convert CRLF to LF */\n+#define CONV_EOL_KEEP_CRLF (1<<3) /* Keep CRLF line endings as is */\n \n-extern enum safe_crlf safe_crlf;\n+extern int global_conv_flags_eol;\n \n enum auto_crlf {\n \tAUTO_CRLF_FALSE = 0,\n@@ -66,7 +63,7 @@ extern const char *get_convert_attr_ascii(const char *path);\n /* returns 1 if *dst was used */\n extern int convert_to_git(const struct index_state *istate,\n \t\t\t const char *path, const char *src, size_t len,\n-\t\t\t struct strbuf *dst, enum safe_crlf checksafe);\n+\t\t\t struct strbuf *dst, int conv_flags);\n extern int convert_to_working_tree(const char *path, const char *src,\n \t\t\t\t size_t len, struct strbuf *dst);\n extern int async_convert_to_working_tree(const char *path, const char *src,\n@@ -85,7 +82,7 @@ static inline int would_convert_to_git(const struct index_state *istate,\n extern void convert_to_git_filter_fd(const struct index_state *istate,\n \t\t\t\t const char *path, int fd,\n \t\t\t\t struct strbuf *dst,\n-\t\t\t\t enum safe_crlf checksafe);\n+\t\t\t\t int conv_flags);\n extern int would_convert_to_git_filter_fd(const char *path);\n \n /*****************************************************************\ndiff --git a/diff.c b/diff.c\nindex 2ebe2227b4..fe3ff6e876 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -3516,13 +3516,13 @@ int diff_populate_filespec(struct diff_filespec *s, unsigned int flags)\n {\n \tint size_only = flags & CHECK_SIZE_ONLY;\n \tint err = 0;\n+\tint conv_flags = global_conv_flags_eol;\n \t/*\n \t * demote FAIL to WARN to allow inspecting the situation\n \t * instead of refusing.\n \t */\n-\tenum safe_crlf crlf_warn = (safe_crlf == SAFE_CRLF_FAIL\n-\t\t\t\t ? SAFE_CRLF_WARN\n-\t\t\t\t : safe_crlf);\n+\tif (conv_flags & CONV_EOL_RNDTRP_DIE)\n+\t\tconv_flags = CONV_EOL_RNDTRP_WARN;\n \n \tif (!DIFF_FILE_VALID(s))\n \t\tdie(\"internal error: asking to populate invalid file.\");\n@@ -3599,7 +3599,7 @@ int diff_populate_filespec(struct diff_filespec *s, unsigned int flags)\n \t\t/*\n \t\t * Convert from working tree format to canonical git format\n \t\t */\n-\t\tif (convert_to_git(&the_index, s->path, s->data, s->size, &buf, crlf_warn)) {\n+\t\tif (convert_to_git(&the_index, s->path, s->data, s->size, &buf, conv_flags)) {\n \t\t\tsize_t size = 0;\n \t\t\tmunmap(s->data, s->size);\n \t\t\ts->should_munmap = 0;\ndiff --git a/environment.c b/environment.c\nindex 8fa032f307..888b873f90 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -49,7 +49,7 @@ enum auto_crlf auto_crlf = AUTO_CRLF_FALSE;\n int check_replace_refs = 1;\n char *git_replace_ref_base;\n enum eol core_eol = EOL_UNSET;\n-enum safe_crlf safe_crlf = SAFE_CRLF_WARN;\n+int global_conv_flags_eol = CONV_EOL_RNDTRP_WARN;\n unsigned whitespace_rule_cfg = WS_DEFAULT_RULE;\n enum branch_track git_branch_track = BRANCH_TRACK_REMOTE;\n enum rebase_setup_type autorebase = AUTOREBASE_NEVER;\ndiff --git a/sha1_file.c b/sha1_file.c\nindex afe4b90f6e..dcb02e9ffd 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -75,14 +75,14 @@ static struct cached_object *find_cached_object(const unsigned char *sha1)\n }\n \n \n-static enum safe_crlf get_safe_crlf(unsigned flags)\n+static int get_conv_flags(unsigned flags)\n {\n \tif (flags & HASH_RENORMALIZE)\n-\t\treturn SAFE_CRLF_RENORMALIZE;\n+\t\treturn CONV_EOL_RENORMALIZE;\n \telse if (flags & HASH_WRITE_OBJECT)\n-\t\treturn safe_crlf;\n+\t\treturn global_conv_flags_eol | CONV_WRITE_OBJECT;\n \telse\n-\t\treturn SAFE_CRLF_FALSE;\n+\t\treturn 0;\n }\n \n \n@@ -1694,7 +1694,7 @@ static int index_mem(struct object_id *oid, void *buf, size_t size,\n \tif ((type == OBJ_BLOB) && path) {\n \t\tstruct strbuf nbuf = STRBUF_INIT;\n \t\tif (convert_to_git(&the_index, path, buf, size, &nbuf,\n-\t\t\t\t get_safe_crlf(flags))) {\n+\t\t\t\t get_conv_flags(flags))) {\n \t\t\tbuf = strbuf_detach(&nbuf, &size);\n \t\t\tre_allocated = 1;\n \t\t}\n@@ -1728,7 +1728,7 @@ static int index_stream_convert_blob(struct object_id *oid, int fd,\n \tassert(would_convert_to_git_filter_fd(path));\n \n \tconvert_to_git_filter_fd(&the_index, path, fd, &sbuf,\n-\t\t\t\t get_safe_crlf(flags));\n+\t\t\t\t get_conv_flags(flags));\n \n \tif (write_object)\n \t\tret = write_sha1_file(sbuf.buf, sbuf.len, typename(OBJ_BLOB),\n-- \n2.15.1\n\n"},{"id":"336041","messageId":"20180106004808.77513-7-lars.schneider@autodesk.com","threadId":"47551","inReplyTo":"20180106004808.77513-1-lars.schneider@autodesk.com","subject":"[PATCH v3 6/7] convert: add support for 'checkout-encoding' attribute","fromName":"","fromEmail":"lars.schneider@autodesk.com","sentAt":"2018-01-06T00:48:07Z","receivedAt":"2018-01-06T00:49:05Z","isPatch":true,"sender":{"key":"lars.schneider@autodesk.com","avatar":"https://gravatar.com/avatar/524188df3a38b02692619777bb90d9cc735db1ad75f5304e0bf09f8efd299775?d=mp&s=160"},"body":"From: Lars Schneider \n\nGit recognizes files encoded with ASCII or one of its supersets (e.g.\nUTF-8 or ISO-8859-1) as text files. All other encodings are usually\ninterpreted as binary and consequently built-in Git text processing\ntools (e.g. 'git diff') as well as most Git web front ends do not\nvisualize the content.\n\nAdd an attribute to teach Git what encoding the user has defined for a\ngiven file. If the content is added to the index, then Git converts the\ncontent to a canonical UTF-8 representation. On checkout Git will\nreverse the conversion.\n\nSigned-off-by: Lars Schneider \n---\n Documentation/gitattributes.txt | 60 ++++++++++++\n convert.c | 190 +++++++++++++++++++++++++++++++++++++-\n convert.h | 1 +\n t/t0028-checkout-encoding.sh | 196 ++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 446 insertions(+), 1 deletion(-)\n create mode 100755 t/t0028-checkout-encoding.sh\n\ndiff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\nindex 30687de81a..1bc03e69cb 100644\n--- a/Documentation/gitattributes.txt\n+++ b/Documentation/gitattributes.txt\n@@ -272,6 +272,66 @@ few exceptions. Even though...\n catch potential problems early, safety triggers.\n \n \n+`checkout-encoding`\n+^^^^^^^^^^^^^^^^^^^\n+\n+Git recognizes files encoded with ASCII or one of its supersets (e.g.\n+UTF-8 or ISO-8859-1) as text files. All other encodings are usually\n+interpreted as binary and consequently built-in Git text processing\n+tools (e.g. 'git diff') as well as most Git web front ends do not\n+visualize the content.\n+\n+In these cases you can teach Git the encoding of a file in the working\n+directory with the `checkout-encoding` attribute. If a file with this\n+attributes is added to Git, then Git reencodes the content from the\n+specified encoding to UTF-8 and stores the result in its internal data\n+structure (called \"the index\"). On checkout the content is encoded\n+back to the specified encoding.\n+\n+Please note that using the `checkout-encoding` attribute may have a\n+number of pitfalls:\n+\n+- Git clients that do not support the `checkout-encoding` attribute\n+ will checkout the respective files UTF-8 encoded and not in the\n+ expected encoding. Consequently, these files will appear different\n+ which typically causes trouble. This is in particular the case for\n+ older Git versions and alternative Git implementations such as JGit\n+ or libgit2 (as of January 2018).\n+\n+- Reencoding content to non-UTF encodings (e.g. SHIFT-JIS) can cause\n+ errors as the conversion might not be round trip safe.\n+\n+- Reencoding content requires resources that might slow down certain\n+ Git operations (e.g 'git checkout' or 'git add').\n+\n+Use the `checkout-encoding` attribute only if you cannot store a file in\n+UTF-8 encoding and if you want Git to be able to process the content as\n+text.\n+\n+Use the following attributes if your '*.txt' files are UTF-16 encoded\n+with byte order mark (BOM) and you want Git to perform automatic line\n+ending conversion based on your platform.\n+\n+------------------------\n+*.txt\t\ttext checkout-encoding=UTF-16\n+------------------------\n+\n+Use the following attributes if your '*.txt' files are UTF-16 little\n+endian encoded without BOM and you want Git to use Windows line endings\n+in the working directory.\n+\n+------------------------\n+*.txt \t\tcheckout-encoding=UTF-16LE text eol=CRLF\n+------------------------\n+\n+You can get a list of all available encodings on your platform with the\n+following command:\n+\n+------------------------\n+iconv --list\n+------------------------\n+\n+\n `ident`\n ^^^^^^^\n \ndiff --git a/convert.c b/convert.c\nindex f39150cde9..13f766d2a2 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -7,6 +7,7 @@\n #include \"sigchain.h\"\n #include \"pkt-line.h\"\n #include \"sub-process.h\"\n+#include \"utf8.h\"\n \n /*\n * convert.c - convert a file when checking it out and checking it in.\n@@ -256,6 +257,147 @@ static int will_convert_lf_to_crlf(size_t len, struct text_stat *stats,\n \n }\n \n+static struct encoding {\n+\tconst char *name;\n+\tstruct encoding *next;\n+} *encoding, **encoding_tail;\n+static const char *default_encoding = \"UTF-8\";\n+\n+static int encode_to_git(const char *path, const char *src, size_t src_len,\n+\t\t\t struct strbuf *buf, struct encoding *enc, int conv_flags)\n+{\n+\tchar *dst;\n+\tint dst_len;\n+\n+\t/*\n+\t * No encoding is specified or there is nothing to encode.\n+\t * Tell the caller that the content was not modified.\n+\t */\n+\tif (!enc || (src && !src_len))\n+\t\treturn 0;\n+\n+\t/*\n+\t * Looks like we got called from \"would_convert_to_git()\".\n+\t * This means Git wants to know if it would encode (= modify!)\n+\t * the content. Let's answer with \"yes\", since an encoding was\n+\t * specified.\n+\t */\n+\tif (!buf && !src)\n+\t\treturn 1;\n+\n+\tif (has_prohibited_utf_bom(enc->name, src, src_len)) {\n+\t\tconst char *error_msg = _(\n+\t\t\t\"BOM is prohibited for '%s' if encoded as %s\");\n+\t\tconst char *advise_msg = _(\n+\t\t\t\"You told Git to treat '%s' as %s. A byte order mark \"\n+\t\t\t\"(BOM) is prohibited with this encoding. Either use \"\n+\t\t\t\"%.6s as checkout encoding or remove the BOM from the \"\n+\t\t\t\"file.\");\n+\n+\t\tadvise(advise_msg, path, enc->name, enc->name, enc->name);\n+\t\tif (conv_flags & CONV_WRITE_OBJECT)\n+\t\t\tdie(error_msg, path, enc->name);\n+\t\telse\n+\t\t\terror(error_msg, path, enc->name);\n+\n+\n+\t} else if (has_missing_utf_bom(enc->name, src, src_len)) {\n+\t\tconst char *error_msg = _(\n+\t\t\t\"BOM is required for '%s' if encoded as %s\");\n+\t\tconst char *advise_msg = _(\n+\t\t\t\"You told Git to treat '%s' as %s. A byte order mark \"\n+\t\t\t\"(BOM) is required with this encoding. Either use \"\n+\t\t\t\"%sBE/%sLE as checkout encoding or add a BOM to the \"\n+\t\t\t\"file.\");\n+\t\tadvise(advise_msg, path, enc->name, enc->name, enc->name);\n+\t\tif (conv_flags & CONV_WRITE_OBJECT)\n+\t\t\tdie(error_msg, path, enc->name);\n+\t\telse\n+\t\t\terror(error_msg, path, enc->name);\n+\t}\n+\n+\tdst = reencode_string_len(src, src_len, default_encoding, enc->name,\n+\t\t\t\t &dst_len);\n+\tif (!dst) {\n+\t\t/*\n+\t\t * We could add the blob \"as-is\" to Git. However, on checkout\n+\t\t * we would try to reencode to the original encoding. This\n+\t\t * would fail and we would leave the user with a messed-up\n+\t\t * working tree. Let's try to avoid this by screaming loud.\n+\t\t */\n+\t\tconst char* msg = _(\"failed to encode '%s' from %s to %s\");\n+\t\tif (conv_flags & CONV_WRITE_OBJECT)\n+\t\t\tdie(msg, path, enc->name, default_encoding);\n+\t\telse\n+\t\t\terror(msg, path, enc->name, default_encoding);\n+\t}\n+\n+\t/*\n+\t * UTF supports lossless round tripping [1]. UTF to other encoding are\n+\t * mostly round trip safe as Unicode aims to be a superset of all other\n+\t * character encodings. However, the SHIFT-JIS (Japanese character set)\n+\t * is an exception as some codes are not round trip safe [2].\n+\t *\n+\t * Reverse the transformation of 'dst' and check the result with 'src'\n+\t * if content is written to Git. This ensures no information is lost\n+\t * during conversion to/from UTF-8.\n+\t *\n+\t * Please note, the code below is not tested because I was not able to\n+\t * generate a faulty round trip without iconv error.\n+\t *\n+\t * [1] http://unicode.org/faq/utf_bom.html#gen2\n+\t * [2] https://support.microsoft.com/en-us/help/170559/prb-conversion-problem-between-shift-jis-and-unicode\n+\t */\n+\tif ((conv_flags & CONV_WRITE_OBJECT) && !strcmp(enc->name, \"SHIFT-JIS\")) {\n+\t\tchar *re_src;\n+\t\tint re_src_len;\n+\n+\t\tre_src = reencode_string_len(dst, dst_len,\n+\t\t\t\t\t enc->name, default_encoding,\n+\t\t\t\t\t &re_src_len);\n+\n+\t\tif (!re_src || src_len != re_src_len ||\n+\t\t memcmp(src, re_src, src_len)) {\n+\t\t\tconst char* msg = _(\"encoding '%s' from %s to %s and \"\n+\t\t\t\t\t \"back is not the same\");\n+\t\t\tif (conv_flags & CONV_WRITE_OBJECT)\n+\t\t\t\tdie(msg, path, enc->name, default_encoding);\n+\t\t\telse\n+\t\t\t\terror(msg, path, enc->name, default_encoding);\n+\t\t}\n+\n+\t\tfree(re_src);\n+\t}\n+\n+\tstrbuf_attach(buf, dst, dst_len, dst_len + 1);\n+\treturn 1;\n+}\n+\n+static int encode_to_worktree(const char *path, const char *src, size_t src_len,\n+\t\t\t struct strbuf *buf, struct encoding *enc)\n+{\n+\tchar *dst;\n+\tint dst_len;\n+\n+\t/*\n+\t * No encoding is specified or there is nothing to encode.\n+\t * Tell the caller that the content was not modified.\n+\t */\n+\tif (!enc || (src && !src_len))\n+\t\treturn 0;\n+\n+\tdst = reencode_string_len(src, src_len, enc->name, default_encoding,\n+\t\t\t\t &dst_len);\n+\tif (!dst) {\n+\t\terror(\"failed to encode '%s' from %s to %s\",\n+\t\t\tpath, enc->name, default_encoding);\n+\t\treturn 0;\n+\t}\n+\n+\tstrbuf_attach(buf, dst, dst_len, dst_len + 1);\n+\treturn 1;\n+}\n+\n static int crlf_to_git(const struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n \t\t struct strbuf *buf,\n@@ -969,6 +1111,31 @@ static int ident_to_worktree(const char *path, const char *src, size_t len,\n \treturn 1;\n }\n \n+static struct encoding *git_path_check_encoding(struct attr_check_item *check)\n+{\n+\tconst char *value = check->value;\n+\tstruct encoding *enc;\n+\n+\tif (ATTR_TRUE(value) || ATTR_FALSE(value) || ATTR_UNSET(value) ||\n+\t !strlen(value))\n+\t\treturn NULL;\n+\n+\tfor (enc = encoding; enc; enc = enc->next)\n+\t\tif (!strcasecmp(value, enc->name))\n+\t\t\treturn enc;\n+\n+\t/* Don't encode to the default encoding */\n+\tif (!strcasecmp(value, default_encoding))\n+\t\treturn NULL;\n+\n+\tenc = xcalloc(1, sizeof(struct convert_driver));\n+\tenc->name = xstrdup_toupper(value); /* aways use upper case names! */\n+\t*encoding_tail = enc;\n+\tencoding_tail = &(enc->next);\n+\n+\treturn enc;\n+}\n+\n static enum crlf_action git_path_check_crlf(struct attr_check_item *check)\n {\n \tconst char *value = check->value;\n@@ -1024,6 +1191,7 @@ struct conv_attrs {\n \tenum crlf_action attr_action; /* What attr says */\n \tenum crlf_action crlf_action; /* When no attr is set, use core.autocrlf */\n \tint ident;\n+\tstruct encoding *checkout_encoding; /* Supported encoding or default encoding if NULL */\n };\n \n static void convert_attrs(struct conv_attrs *ca, const char *path)\n@@ -1032,8 +1200,10 @@ static void convert_attrs(struct conv_attrs *ca, const char *path)\n \n \tif (!check) {\n \t\tcheck = attr_check_initl(\"crlf\", \"ident\", \"filter\",\n-\t\t\t\t\t \"eol\", \"text\", NULL);\n+\t\t\t\t\t \"eol\", \"text\", \"checkout-encoding\",\n+\t\t\t\t\t NULL);\n \t\tuser_convert_tail = &user_convert;\n+\t\tencoding_tail = &encoding;\n \t\tgit_config(read_convert_config, NULL);\n \t}\n \n@@ -1055,6 +1225,7 @@ static void convert_attrs(struct conv_attrs *ca, const char *path)\n \t\t\telse if (eol_attr == EOL_CRLF)\n \t\t\t\tca->crlf_action = CRLF_TEXT_CRLF;\n \t\t}\n+\t\tca->checkout_encoding = git_path_check_encoding(ccheck + 5);\n \t} else {\n \t\tca->drv = NULL;\n \t\tca->crlf_action = CRLF_UNDEFINED;\n@@ -1135,6 +1306,13 @@ int convert_to_git(const struct index_state *istate,\n \t\tsrc = dst->buf;\n \t\tlen = dst->len;\n \t}\n+\n+\tret |= encode_to_git(path, src, len, dst, ca.checkout_encoding, conv_flags);\n+\tif (ret && dst) {\n+\t\tsrc = dst->buf;\n+\t\tlen = dst->len;\n+\t}\n+\n \tif (!(conv_flags & CONV_EOL_KEEP_CRLF)) {\n \t\tret |= crlf_to_git(istate, path, src, len, dst, ca.crlf_action, conv_flags);\n \t\tif (ret && dst) {\n@@ -1158,6 +1336,7 @@ void convert_to_git_filter_fd(const struct index_state *istate,\n \tif (!apply_filter(path, NULL, 0, fd, dst, ca.drv, CAP_CLEAN, NULL))\n \t\tdie(\"%s: clean filter '%s' failed\", path, ca.drv->name);\n \n+\tencode_to_git(path, dst->buf, dst->len, dst, ca.checkout_encoding, conv_flags);\n \tcrlf_to_git(istate, path, dst->buf, dst->len, dst, ca.crlf_action, conv_flags);\n \tident_to_git(path, dst->buf, dst->len, dst, ca.ident);\n }\n@@ -1189,6 +1368,12 @@ static int convert_to_working_tree_internal(const char *path, const char *src,\n \t\t}\n \t}\n \n+\tret |= encode_to_worktree(path, src, len, dst, ca.checkout_encoding);\n+\tif (ret) {\n+\t\tsrc = dst->buf;\n+\t\tlen = dst->len;\n+\t}\n+\n \tret_filter = apply_filter(\n \t\tpath, src, len, -1, dst, ca.drv, CAP_SMUDGE, dco);\n \tif (!ret_filter && ca.drv && ca.drv->required)\n@@ -1655,6 +1840,9 @@ struct stream_filter *get_stream_filter(const char *path, const unsigned char *s\n \tif (ca.drv && (ca.drv->process || ca.drv->smudge || ca.drv->clean))\n \t\treturn NULL;\n \n+\tif (ca.checkout_encoding)\n+\t\treturn NULL;\n+\n \tif (ca.crlf_action == CRLF_AUTO || ca.crlf_action == CRLF_AUTO_CRLF)\n \t\treturn NULL;\n \ndiff --git a/convert.h b/convert.h\nindex 65ab3e5167..1d9539ed0b 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -12,6 +12,7 @@ struct index_state;\n #define CONV_EOL_RNDTRP_WARN (1<<1) /* Warn if CRLF to LF to CRLF is different */\n #define CONV_EOL_RENORMALIZE (1<<2) /* Convert CRLF to LF */\n #define CONV_EOL_KEEP_CRLF (1<<3) /* Keep CRLF line endings as is */\n+#define CONV_WRITE_OBJECT (1<<4) /* Content is written to the index */\n \n extern int global_conv_flags_eol;\n \ndiff --git a/t/t0028-checkout-encoding.sh b/t/t0028-checkout-encoding.sh\nnew file mode 100755\nindex 0000000000..3a9951fdf3\n--- /dev/null\n+++ b/t/t0028-checkout-encoding.sh\n@@ -0,0 +1,196 @@\n+#!/bin/sh\n+\n+test_description='checkout-encoding conversion via gitattributes'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup test repo' '\n+\tgit config core.eol lf &&\n+\n+\ttext=\"hallo there!\\ncan you read me?\" &&\n+\techo \"*.utf16 text checkout-encoding=utf-16\" >.gitattributes &&\n+\tprintf \"$text\" >test.utf8.raw &&\n+\tprintf \"$text\" | iconv -f UTF-8 -t UTF-16 >test.utf16.raw &&\n+\tcp test.utf16.raw test.utf16 &&\n+\n+\tgit add .gitattributes test.utf16 &&\n+\tgit commit -m initial\n+'\n+\n+test_expect_success 'ensure UTF-8 is stored in Git' '\n+\tgit cat-file -p :test.utf16 >test.utf16.git &&\n+\ttest_cmp_bin test.utf8.raw test.utf16.git &&\n+\trm test.utf8.raw test.utf16.git\n+'\n+\n+test_expect_success 're-encode to UTF-16 on checkout' '\n+\trm test.utf16 &&\n+\tgit checkout test.utf16 &&\n+\ttest_cmp_bin test.utf16.raw test.utf16 &&\n+\n+\t# cleanup\n+\trm test.utf16.raw\n+'\n+\n+test_expect_success 'check prohibited UTF BOM' '\n+\tprintf \"\\0a\\0b\\0c\" >nobom.utf16be.raw &&\n+\tprintf \"a\\0b\\0c\\0\" >nobom.utf16le.raw &&\n+\tprintf \"\\376\\777\\0a\\0b\\0c\" >bebom.utf16be.raw &&\n+\tprintf \"\\777\\376a\\0b\\0c\\0\" >lebom.utf16le.raw &&\n+\n+\tprintf \"\\0\\0\\0a\\0\\0\\0b\\0\\0\\0c\" >nobom.utf32be.raw &&\n+\tprintf \"a\\0\\0\\0b\\0\\0\\0c\\0\\0\\0\" >nobom.utf32le.raw &&\n+\tprintf \"\\0\\0\\376\\777\\0\\0\\0a\\0\\0\\0b\\0\\0\\0c\" >bebom.utf32be.raw &&\n+\tprintf \"\\777\\376\\0\\0a\\0\\0\\0b\\0\\0\\0c\\0\\0\\0\" >lebom.utf32le.raw &&\n+\n+\techo \"*.utf16be text checkout-encoding=utf-16be\" >>.gitattributes &&\n+\techo \"*.utf16le text checkout-encoding=utf-16le\" >>.gitattributes &&\n+\techo \"*.utf32be text checkout-encoding=utf-32be\" >>.gitattributes &&\n+\techo \"*.utf32le text checkout-encoding=utf-32le\" >>.gitattributes &&\n+\n+\t# Here we add a UTF-16 files with BOM (big-endian and little-endian)\n+\t# but we tell Git to treat it as UTF-16BE/UTF-16LE. In these cases\n+\t# the BOM is prohibited.\n+\tcp bebom.utf16be.raw bebom.utf16be &&\n+\ttest_must_fail git add bebom.utf16be 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-16BE\" err.out &&\n+\n+\tcp lebom.utf16le.raw lebom.utf16be &&\n+\ttest_must_fail git add lebom.utf16be 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-16BE\" err.out &&\n+\n+\tcp bebom.utf16be.raw bebom.utf16le &&\n+\ttest_must_fail git add bebom.utf16le 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-16LE\" err.out &&\n+\n+\tcp lebom.utf16le.raw lebom.utf16le &&\n+\ttest_must_fail git add lebom.utf16le 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-16LE\" err.out &&\n+\n+\t# ... and the same for UTF-32\n+\tcp bebom.utf32be.raw bebom.utf32be &&\n+\ttest_must_fail git add bebom.utf32be 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-32BE\" err.out &&\n+\n+\tcp lebom.utf32le.raw lebom.utf32be &&\n+\ttest_must_fail git add lebom.utf32be 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-32BE\" err.out &&\n+\n+\tcp bebom.utf32be.raw bebom.utf32le &&\n+\ttest_must_fail git add bebom.utf32le 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-32LE\" err.out &&\n+\n+\tcp lebom.utf32le.raw lebom.utf32le &&\n+\ttest_must_fail git add lebom.utf32le 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is prohibited .* UTF-32LE\" err.out &&\n+\n+\t# cleanup\n+\tgit reset --hard HEAD\n+'\n+\n+test_expect_success 'check required UTF BOM' '\n+\techo \"*.utf32 text checkout-encoding=utf-32\" >>.gitattributes &&\n+\n+\tcp nobom.utf16be.raw nobom.utf16 &&\n+\ttest_must_fail git add nobom.utf16 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is required .* UTF-16\" err.out &&\n+\n+\tcp nobom.utf16le.raw nobom.utf16 &&\n+\ttest_must_fail git add nobom.utf16 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is required .* UTF-16\" err.out &&\n+\n+\tcp nobom.utf32be.raw nobom.utf32 &&\n+\ttest_must_fail git add nobom.utf32 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is required .* UTF-32\" err.out &&\n+\n+\tcp nobom.utf32le.raw nobom.utf32 &&\n+\ttest_must_fail git add nobom.utf32 2>err.out &&\n+\ttest_i18ngrep \"fatal: BOM is required .* UTF-32\" err.out &&\n+\n+\t# cleanup\n+\trm nobom.utf16 nobom.utf32 &&\n+\tgit reset --hard HEAD\n+'\n+\n+test_expect_success 'eol conversion for UTF-16 encoded files on checkout' '\n+\tprintf \"one\\ntwo\\nthree\\n\" >lf.utf8.raw &&\n+\tprintf \"one\\r\\ntwo\\r\\nthree\\r\\n\" >crlf.utf8.raw &&\n+\n+\tcat lf.utf8.raw | iconv -f UTF-8 -t UTF-16 >lf.utf16.raw &&\n+\tcat crlf.utf8.raw | iconv -f UTF-8 -t UTF-16 >crlf.utf16.raw &&\n+\tcp crlf.utf16.raw eol.utf16 &&\n+\n+\tgit add eol.utf16 &&\n+\tgit commit -m eol &&\n+\n+\t# UTF-16 with CRLF (Windows line endings)\n+\trm eol.utf16 &&\n+\tgit -c core.eol=crlf checkout eol.utf16 &&\n+\ttest_cmp_bin crlf.utf16.raw eol.utf16 &&\n+\n+\t# UTF-16 with LF (Unix line endings)\n+\trm eol.utf16 &&\n+\tgit -c core.eol=lf checkout eol.utf16 &&\n+\ttest_cmp_bin lf.utf16.raw eol.utf16 &&\n+\n+\trm crlf.utf16.raw crlf.utf8.raw lf.utf16.raw lf.utf8.raw &&\n+\n+\t# cleanup\n+\tgit reset --hard HEAD^\n+'\n+\n+test_expect_success 'check unsupported encodings' '\n+\n+\techo \"*.nothing text checkout-encoding=\" >>.gitattributes &&\n+\tprintf \"nothing\" >t.nothing &&\n+\tgit add t.nothing &&\n+\n+\techo \"*.garbage text checkout-encoding=garbage\" >>.gitattributes &&\n+\tprintf \"garbage\" >t.garbage &&\n+\ttest_must_fail git add t.garbage 2>err.out &&\n+\ttest_i18ngrep \"fatal: failed to encode\" err.out &&\n+\n+\t# cleanup\n+\trm err.out &&\n+\tgit reset --hard HEAD\n+'\n+\n+test_expect_success 'error if encoding round trip is not the same during refresh' '\n+\tBEFORE_STATE=$(git rev-parse HEAD) &&\n+\n+\t# Skip the UTF-16 filter for the added file\n+\t# This simulates a Git version that has no checkoutEncoding support\n+\techo \"hallo\" >nonsense.utf16 &&\n+\tTEST_HASH=$(git hash-object --no-filters -w nonsense.utf16) &&\n+\tgit update-index --add --cacheinfo 100644 $TEST_HASH nonsense.utf16 &&\n+\tCOMMIT=$(git commit-tree -p $(git rev-parse HEAD) -m \"plain commit\" $(git write-tree)) &&\n+\tgit update-ref refs/heads/master $COMMIT &&\n+\n+\ttest_must_fail git checkout HEAD^ 2>err.out &&\n+\ttest_i18ngrep \"error: .* overwritten by checkout:\" err.out &&\n+\n+\t# cleanup\n+\trm err.out &&\n+\tgit reset --hard $BEFORE_STATE\n+'\n+\n+test_expect_success 'error if encoding garbage is already in Git' '\n+\tBEFORE_STATE=$(git rev-parse HEAD) &&\n+\n+\t# Skip the UTF-16 filter for the added file\n+\t# This simulates a Git version that has no checkoutEncoding support\n+\tcp nobom.utf16be.raw nonsense.utf16 &&\n+\tTEST_HASH=$(git hash-object --no-filters -w nonsense.utf16) &&\n+\tgit update-index --add --cacheinfo 100644 $TEST_HASH nonsense.utf16 &&\n+\tCOMMIT=$(git commit-tree -p $(git rev-parse HEAD) -m \"plain commit\" $(git write-tree)) &&\n+\tgit update-ref refs/heads/master $COMMIT &&\n+\n+\tgit diff 2>err.out &&\n+\ttest_i18ngrep \"error: BOM is required\" err.out &&\n+\n+\t# cleanup\n+\trm err.out &&\n+\tgit reset --hard $BEFORE_STATE\n+'\n+\n+test_done\n-- \n2.15.1\n\n"},{"id":"336042","messageId":"20180106004808.77513-8-lars.schneider@autodesk.com","threadId":"47551","inReplyTo":"20180106004808.77513-1-lars.schneider@autodesk.com","subject":"[PATCH v3 7/7] convert: add tracing for checkout-encoding","fromName":"","fromEmail":"lars.schneider@autodesk.com","sentAt":"2018-01-06T00:48:08Z","receivedAt":"2018-01-06T00:49:07Z","isPatch":true,"sender":{"key":"lars.schneider@autodesk.com","avatar":"https://gravatar.com/avatar/524188df3a38b02692619777bb90d9cc735db1ad75f5304e0bf09f8efd299775?d=mp&s=160"},"body":"From: Lars Schneider \n\nAdd the GIT_TRACE_CHECKOUT_ENCODING environment variable to enable\ntracing for content that is reencoded with the checkout-encoding\nattribute.\n\nSigned-off-by: Lars Schneider \n---\n convert.c | 28 ++++++++++++++++++++++++++++\n t/t0028-checkout-encoding.sh | 2 ++\n 2 files changed, 30 insertions(+)\n\ndiff --git a/convert.c b/convert.c\nindex 13f766d2a2..525958bb56 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -257,6 +257,29 @@ static int will_convert_lf_to_crlf(size_t len, struct text_stat *stats,\n \n }\n \n+static void trace_encoding(const char *context, const char *path,\n+\t\t\t const char *encoding, const char *buf, size_t len)\n+{\n+\tstatic struct trace_key coe = TRACE_KEY_INIT(CHECKOUT_ENCODING);\n+\tstruct strbuf trace = STRBUF_INIT;\n+\tint i;\n+\n+\tstrbuf_addf(&trace, \"%s (%s, considered %s):\\n\", context, path, encoding);\n+\tfor (i = 0; i < len && buf; ++i) {\n+\t\tstrbuf_addf(\n+\t\t\t&trace,\"| \\e[2m%2i:\\e[0m %2x \\e[2m%c\\e[0m%c\",\n+\t\t\ti,\n+\t\t\t(unsigned char) buf[i],\n+\t\t\t(buf[i] > 32 && buf[i] < 127 ? buf[i] : ' '),\n+\t\t\t((i+1) % 8 && (i+1) < len ? ' ' : '\\n')\n+\t\t);\n+\t}\n+\tstrbuf_addchars(&trace, '\\n', 1);\n+\n+\ttrace_strbuf(&coe, &trace);\n+\tstrbuf_release(&trace);\n+}\n+\n static struct encoding {\n \tconst char *name;\n \tstruct encoding *next;\n@@ -316,6 +339,7 @@ static int encode_to_git(const char *path, const char *src, size_t src_len,\n \t\t\terror(error_msg, path, enc->name);\n \t}\n \n+\ttrace_encoding(\"source\", path, enc->name, src, src_len);\n \tdst = reencode_string_len(src, src_len, default_encoding, enc->name,\n \t\t\t\t &dst_len);\n \tif (!dst) {\n@@ -331,6 +355,7 @@ static int encode_to_git(const char *path, const char *src, size_t src_len,\n \t\telse\n \t\t\terror(msg, path, enc->name, default_encoding);\n \t}\n+\ttrace_encoding(\"destination\", path, default_encoding, dst, dst_len);\n \n \t/*\n \t * UTF supports lossless round tripping [1]. UTF to other encoding are\n@@ -356,6 +381,9 @@ static int encode_to_git(const char *path, const char *src, size_t src_len,\n \t\t\t\t\t enc->name, default_encoding,\n \t\t\t\t\t &re_src_len);\n \n+\t\ttrace_encoding(\"reencoded source\", path, enc->name,\n+\t\t\t re_src, re_src_len);\n+\n \t\tif (!re_src || src_len != re_src_len ||\n \t\t memcmp(src, re_src, src_len)) {\n \t\t\tconst char* msg = _(\"encoding '%s' from %s to %s and \"\ndiff --git a/t/t0028-checkout-encoding.sh b/t/t0028-checkout-encoding.sh\nindex 3a9951fdf3..5f1c911c07 100755\n--- a/t/t0028-checkout-encoding.sh\n+++ b/t/t0028-checkout-encoding.sh\n@@ -4,6 +4,8 @@ test_description='checkout-encoding conversion via gitattributes'\n \n . ./test-lib.sh\n \n+GIT_TRACE_CHECKOUT_ENCODING=1 && export GIT_TRACE_CHECKOUT_ENCODING\n+\n test_expect_success 'setup test repo' '\n \tgit config core.eol lf &&\n \n-- \n2.15.1\n\n"},{"id":"336081","messageId":"20180107093815.GA7442@tor.lan","threadId":"47551","inReplyTo":"20180106004808.77513-1-lars.schneider@autodesk.com","subject":"Re: [PATCH v3 0/7] convert: add support for different encodings","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2018-01-07T09:38:15Z","receivedAt":"2018-01-07T09:38:44Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Sat, Jan 06, 2018 at 01:48:01AM +0100, lars.schneider@autodesk.com wrote:\n> From: Lars Schneider \n> \n> Hi,\n> \n> Patches 1-5 and 6 are helper functions and preparation.\n> Patch 6 is the actual change.\n> \n> I am still torn between \"checkout-encoding\" and \"working-tree-encoding\"\n> as attribute name. I am happy to hear arguments for/against one or the\n> other.\n\ncheckout-encoding is probably misleading, as it is even the checkin-encoding.\n\nWhat is wrong with working-tree-encoding ?\nI think the 2 \"-\".\n\nWhat was wrong with workingtree-encoding ?\nOr\nworkdir-encoding ?\n\n\n\n> \n> Changes since v2:\n> \n> * Added Torsten's crlfsave refactoring patch (patch 5)\n> @Torsten: I tried to make the commit message more clean, added\n> some comments to and renamed conv_flags_eol to\n> global_conv_flags_eol.\n> \n> * Improved documentation and commit message (Torsten)\n\nGood, thanks.\n> \n> * Removed unnecessary NUL assignment in xstrdup_tolower() (Torsten)\n> \n> * Set \"git config core.eol lf\" to made the test run on Windows (Dscho)\n> \n> * Made BOM arrays static (Ramsay)\n\n\nSome comments:\n\nI would like to have the CRLF conversion a little bit more strict -\nmany users tend to set core.autocrlf=true or write \"* text=auto\"\nin the .gitattributes.\nReading all the effort about BOM markers and UTF-16LE, I think there\nshould ne some effort to make the line endings round trip.\nTherefore I changed convert.c to demand that the \"text\" attribute\nis set to enable CRLF conversions.\n(If I had submitted the patch, I would have demanded\n\"text eol=lf\" or \"text eol=crlf\", but the test case t0028 indicates\nthat there is a demand to produce line endings as configured in core.eol)\n\nAnyway, I rebased it onto git.git/master, changed the docu, and pushed it to\nhttps://github.com/tboegi/git/tree/180107-0935-For-lars-schneider-encode-V3B\n\nHere is a inter-diff against your version:\n\n diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\n index 1bc03e69c..b8d9f91c8 100644\n --- a/Documentation/gitattributes.txt\n +++ b/Documentation/gitattributes.txt\n @@ -281,7 +281,7 @@ interpreted as binary and consequently built-in Git text processing\n tools (e.g. 'git diff') as well as most Git web front ends do not\n visualize the content.\n \n -In these cases you can teach Git the encoding of a file in the working\n +In these cases you can tell Git the encoding of a file in the working\n directory with the `checkout-encoding` attribute. If a file with this\n attributes is added to Git, then Git reencodes the content from the\n specified encoding to UTF-8 and stores the result in its internal data\n @@ -308,17 +308,20 @@ Use the `checkout-encoding` attribute only if you cannot store a file in\n UTF-8 encoding and if you want Git to be able to process the content as\n text.\n \n +Note that when `checkout-encoding` is defined, by default the line\n +endings are not converted. `text=auto` and core.autocrlf are ignored.\n +Set the `text` attribute to enable CRLF conversions.\n +\n Use the following attributes if your '*.txt' files are UTF-16 encoded\n -with byte order mark (BOM) and you want Git to perform automatic line\n -ending conversion based on your platform.\n +with byte order mark (BOM).\n \n ------------------------\n -*.txt\t\ttext checkout-encoding=UTF-16\n +*.txt\t\tcheckout-encoding=UTF-16\n ------------------------\n \n Use the following attributes if your '*.txt' files are UTF-16 little\n -endian encoded without BOM and you want Git to use Windows line endings\n -in the working directory.\n +endian encoded without BOM and you want Git to use LF in the repo and\n +CRLF in the working directory.\n \n ------------------------\n *.txt \t\tcheckout-encoding=UTF-16LE text eol=CRLF\n diff --git a/convert.c b/convert.c\n index 13f766d2a..1e29f515e 100644\n --- a/convert.c\n +++ b/convert.c\n @@ -221,18 +221,27 @@ static void check_global_conv_flags_eol(const char *path, enum crlf_action crlf_\n \t}\n }\n \n \n static int will_convert_lf_to_crlf(size_t len, struct text_stat *stats,\n @@ -432,7 +441,7 @@ static int crlf_to_git(const struct index_state *istate,\n \t\t * cherry-pick.\n \t\t */\n \t\tif ((!(conv_flags & CONV_EOL_RENORMALIZE)) &&\n -\t\t has_cr_in_index(istate, path))\n +\t\t has_crlf_in_index(istate, path))\n \t\t\tconvert_crlf_into_lf = 0;\n \t}\n \tif (((conv_flags & CONV_EOL_RNDTRP_WARN) ||\n @@ -1214,9 +1223,28 @@ static void convert_attrs(struct conv_attrs *ca, const char *path)\n \t\t\tca->crlf_action = git_path_check_crlf(ccheck + 0);\n \t\tca->ident = git_path_check_ident(ccheck + 1);\n \t\tca->drv = git_path_check_convert(ccheck + 2);\n +\t\tca->checkout_encoding = git_path_check_encoding(ccheck + 5);\n \t\tif (ca->crlf_action != CRLF_BINARY) {\n \t\t\tenum eol eol_attr = git_path_check_eol(ccheck + 3);\n -\t\t\tif (ca->crlf_action == CRLF_AUTO && eol_attr == EOL_LF)\n +\t\t\tif (ca->checkout_encoding) {\n +\t\t\t\tenum crlf_action crlf_action = CRLF_BINARY;\n +\t\t\t\t/*\n +\t\t\t\t * encoded files don't use auto.\n +\t\t\t\t * 'text' must be specified to\n +\t\t\t\t * do crlf conversions\n +\t\t\t\t */\n +\t\t\t\tif (ca->crlf_action == CRLF_TEXT) {\n +\t\t\t\t\tif (eol_attr == EOL_LF)\n +\t\t\t\t\t\tcrlf_action = CRLF_TEXT_INPUT;\n +\t\t\t\t\telse if (eol_attr == EOL_CRLF)\n +\t\t\t\t\t\tcrlf_action = CRLF_TEXT_CRLF;\n +\t\t\t\t\telse if (text_eol_is_crlf())\n +\t\t\t\t\t\tcrlf_action = CRLF_TEXT_CRLF;\n +\t\t\t\t\telse\n +\t\t\t\t\t\tcrlf_action = CRLF_TEXT_INPUT;\n +\t\t\t\t}\n +\t\t\t\tca->crlf_action = crlf_action;\n +\t\t\t} else if (ca->crlf_action == CRLF_AUTO && eol_attr == EOL_LF)\n \t\t\t\tca->crlf_action = CRLF_AUTO_INPUT;\n \t\t\telse if (ca->crlf_action == CRLF_AUTO && eol_attr == EOL_CRLF)\n \t\t\t\tca->crlf_action = CRLF_AUTO_CRLF;\n @@ -1225,11 +1253,11 @@ static void convert_attrs(struct conv_attrs *ca, const char *path)\n \t\t\telse if (eol_attr == EOL_CRLF)\n \t\t\t\tca->crlf_action = CRLF_TEXT_CRLF;\n \t\t}\n -\t\tca->checkout_encoding = git_path_check_encoding(ccheck + 5);\n \t} else {\n \t\tca->drv = NULL;\n \t\tca->crlf_action = CRLF_UNDEFINED;\n \t\tca->ident = 0;\n +\t\tca->checkout_encoding = NULL;\n \t}\n \n \t/* Save attr and make a decision for action */\n\n\n"},{"id":"336144","messageId":"B51940E6-95AB-424A-AF62-0018E9934279@gmail.com","threadId":"47551","inReplyTo":"20180107093815.GA7442@tor.lan","subject":"Re: [PATCH v3 0/7] convert: add support for different encodings","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2018-01-08T14:38:48Z","receivedAt":"2018-01-08T14:39:00Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 07 Jan 2018, at 10:38, Torsten Bögershausen wrote:\n> \n> On Sat, Jan 06, 2018 at 01:48:01AM +0100, lars.schneider@autodesk.com wrote:\n>> From: Lars Schneider \n>> \n>> Hi,\n>> \n>> Patches 1-5 and 6 are helper functions and preparation.\n>> Patch 6 is the actual change.\n>> \n>> I am still torn between \"checkout-encoding\" and \"working-tree-encoding\"\n>> as attribute name. I am happy to hear arguments for/against one or the\n>> other.\n> \n> checkout-encoding is probably misleading, as it is even the checkin-encoding.\n\nYeah, I start to think the same.\n\n\n> What is wrong with working-tree-encoding ?\n> I think the 2 \"-\".\n> \n> What was wrong with workingtree-encoding ?\n\nYeah, the two dashes are a minor annoyance.\n\nHowever, consider this:\n\n$ git grep 'working tree' -- '*.txt' | wc -l\n 570\n\n$ git grep 'working-tree' -- '*.txt' | wc -l\n 6\n\n$ git grep 'workingtree' -- '*.txt' | wc -l\n 0\n\n\n$ git grep 'working tree' -- po | wc -l\n 704\n\n$ git grep 'working-tree' -- po | wc -l\n 0\n\n$ git grep 'workingtree' -- po | wc -l\n 0\n\nI think \"working tree\" is a pretty established term that\nendusers might be able to understand. Therefore, I would\nlike to go with \"working-tree-encoding\" as it was written\nthat way at least 6 times in the Git tree before.\n\nWould that work for you?\n\n\n> Or\n> workdir-encoding ?\n\nAlthough I like the shortness, the term \"workdir\" might already \nbe occupied [1]. Could that cause confusion?\n\n[1] 4f01748d51 (contrib/workdir: add a simple script to create a working directory, 2007-03-27)\n\n\n>> \n>> * Removed unnecessary NUL assignment in xstrdup_tolower() (Torsten)\n>> \n>> * Set \"git config core.eol lf\" to made the test run on Windows (Dscho)\n>> \n>> * Made BOM arrays static (Ramsay)\n> \n> \n> Some comments:\n> \n> I would like to have the CRLF conversion a little bit more strict -\n> many users tend to set core.autocrlf=true or write \"* text=auto\"\n> in the .gitattributes.\n> Reading all the effort about BOM markers and UTF-16LE, I think there\n> should ne some effort to make the line endings round trip.\n> Therefore I changed convert.c to demand that the \"text\" attribute\n> is set to enable CRLF conversions.\n> (If I had submitted the patch, I would have demanded\n> \"text eol=lf\" or \"text eol=crlf\", but the test case t0028 indicates\n> that there is a demand to produce line endings as configured in core.eol)\n\nBut wouldn't that be inconvenient for the users? E.g. if I add a UTF-16\nfile on Windows with CRLF then it would be nice if Git would automatically\nconvert the line endings to LF on Linux, no?\n\nIOW: Why should we handle text files that have a defined checkout-encoding\ndifferently compared to UTF-8 encoded text files? Wouldn't that be unexpected\nto the user?\n\nThanks,\nLars\n\n\n\n> \n> Anyway, I rebased it onto git.git/master, changed the docu, and pushed it to\n> https://github.com/tboegi/git/tree/180107-0935-For-lars-schneider-encode-V3B\n> \n> Here is a inter-diff against your version:\n> \n> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt\n> index 1bc03e69c..b8d9f91c8 100644\n> --- a/Documentation/gitattributes.txt\n> +++ b/Documentation/gitattributes.txt\n> @@ -281,7 +281,7 @@ interpreted as binary and consequently built-in Git text processing\n> tools (e.g. 'git diff') as well as most Git web front ends do not\n> visualize the content.\n> \n> -In these cases you can teach Git the encoding of a file in the working\n> +In these cases you can tell Git the encoding of a file in the working\n\nOops. I meant to change that already. Thanks!\n\n> directory with the `checkout-encoding` attribute. If a file with this\n> attributes is added to Git, then Git reencodes the content from the\n> specified encoding to UTF-8 and stores the result in its internal data\n> @@ -308,17 +308,20 @@ Use the `checkout-encoding` attribute only if you cannot store a file in\n> UTF-8 encoding and if you want Git to be able to process the content as\n> text.\n> \n> +Note that when `checkout-encoding` is defined, by default the line\n> +endings are not converted. `text=auto` and core.autocrlf are ignored.\n> +Set the `text` attribute to enable CRLF conversions.\n> +\n> Use the following attributes if your '*.txt' files are UTF-16 encoded\n> -with byte order mark (BOM) and you want Git to perform automatic line\n> -ending conversion based on your platform.\n> +with byte order mark (BOM).\n> \n> ------------------------\n> -*.txt\t\ttext checkout-encoding=UTF-16\n> +*.txt\t\tcheckout-encoding=UTF-16\n> ------------------------\n> \n> Use the following attributes if your '*.txt' files are UTF-16 little\n> -endian encoded without BOM and you want Git to use Windows line endings\n> -in the working directory.\n> +endian encoded without BOM and you want Git to use LF in the repo and\n> +CRLF in the working directory.\n> \n> ------------------------\n> *.txt \t\tcheckout-encoding=UTF-16LE text eol=CRLF\n> diff --git a/convert.c b/convert.c\n> index 13f766d2a..1e29f515e 100644\n> --- a/convert.c\n> +++ b/convert.c\n> @@ -221,18 +221,27 @@ static void check_global_conv_flags_eol(const char *path, enum crlf_action crlf_\n> \t}\n> }\n> \n> \n> static int will_convert_lf_to_crlf(size_t len, struct text_stat *stats,\n> @@ -432,7 +441,7 @@ static int crlf_to_git(const struct index_state *istate,\n> \t\t * cherry-pick.\n> \t\t */\n> \t\tif ((!(conv_flags & CONV_EOL_RENORMALIZE)) &&\n> -\t\t has_cr_in_index(istate, path))\n> +\t\t has_crlf_in_index(istate, path))\n> \t\t\tconvert_crlf_into_lf = 0;\n> \t}\n> \tif (((conv_flags & CONV_EOL_RNDTRP_WARN) ||\n> @@ -1214,9 +1223,28 @@ static void convert_attrs(struct conv_attrs *ca, const char *path)\n> \t\t\tca->crlf_action = git_path_check_crlf(ccheck + 0);\n> \t\tca->ident = git_path_check_ident(ccheck + 1);\n> \t\tca->drv = git_path_check_convert(ccheck + 2);\n> +\t\tca->checkout_encoding = git_path_check_encoding(ccheck + 5);\n> \t\tif (ca->crlf_action != CRLF_BINARY) {\n> \t\t\tenum eol eol_attr = git_path_check_eol(ccheck + 3);\n> -\t\t\tif (ca->crlf_action == CRLF_AUTO && eol_attr == EOL_LF)\n> +\t\t\tif (ca->checkout_encoding) {\n> +\t\t\t\tenum crlf_action crlf_action = CRLF_BINARY;\n> +\t\t\t\t/*\n> +\t\t\t\t * encoded files don't use auto.\n> +\t\t\t\t * 'text' must be specified to\n> +\t\t\t\t * do crlf conversions\n> +\t\t\t\t */\n> +\t\t\t\tif (ca->crlf_action == CRLF_TEXT) {\n> +\t\t\t\t\tif (eol_attr == EOL_LF)\n> +\t\t\t\t\t\tcrlf_action = CRLF_TEXT_INPUT;\n> +\t\t\t\t\telse if (eol_attr == EOL_CRLF)\n> +\t\t\t\t\t\tcrlf_action = CRLF_TEXT_CRLF;\n> +\t\t\t\t\telse if (text_eol_is_crlf())\n> +\t\t\t\t\t\tcrlf_action = CRLF_TEXT_CRLF;\n> +\t\t\t\t\telse\n> +\t\t\t\t\t\tcrlf_action = CRLF_TEXT_INPUT;\n> +\t\t\t\t}\n> +\t\t\t\tca->crlf_action = crlf_action;\n> +\t\t\t} else if (ca->crlf_action == CRLF_AUTO && eol_attr == EOL_LF)\n> \t\t\t\tca->crlf_action = CRLF_AUTO_INPUT;\n> \t\t\telse if (ca->crlf_action == CRLF_AUTO && eol_attr == EOL_CRLF)\n> \t\t\t\tca->crlf_action = CRLF_AUTO_CRLF;\n> @@ -1225,11 +1253,11 @@ static void convert_attrs(struct conv_attrs *ca, const char *path)\n> \t\t\telse if (eol_attr == EOL_CRLF)\n> \t\t\t\tca->crlf_action = CRLF_TEXT_CRLF;\n> \t\t}\n> -\t\tca->checkout_encoding = git_path_check_encoding(ccheck + 5);\n> \t} else {\n> \t\tca->drv = NULL;\n> \t\tca->crlf_action = CRLF_UNDEFINED;\n> \t\tca->ident = 0;\n> +\t\tca->checkout_encoding = NULL;\n> \t}\n> \n> \t/* Save attr and make a decision for action */\n> \n> \n\n"},{"id":"336160","messageId":"20180108180826.GA16999@tor.lan","threadId":"47551","inReplyTo":"B51940E6-95AB-424A-AF62-0018E9934279@gmail.com","subject":"Re: [PATCH v3 0/7] convert: add support for different encodings","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2018-01-08T18:08:26Z","receivedAt":"2018-01-08T18:08:53Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Mon, Jan 08, 2018 at 03:38:48PM +0100, Lars Schneider wrote:\n[]\n> > Some comments:\n> > \n> > I would like to have the CRLF conversion a little bit more strict -\n> > many users tend to set core.autocrlf=true or write \"* text=auto\"\n> > in the .gitattributes.\n> > Reading all the effort about BOM markers and UTF-16LE, I think there\n> > should ne some effort to make the line endings round trip.\n> > Therefore I changed convert.c to demand that the \"text\" attribute\n> > is set to enable CRLF conversions.\n> > (If I had submitted the patch, I would have demanded\n> > \"text eol=lf\" or \"text eol=crlf\", but the test case t0028 indicates\n> > that there is a demand to produce line endings as configured in core.eol)\n> \n> But wouldn't that be inconvenient for the users? E.g. if I add a UTF-16\n> file on Windows with CRLF then it would be nice if Git would automatically\n> convert the line endings to LF on Linux, no?\n> \n> IOW: Why should we handle text files that have a defined checkout-encoding\n> differently compared to UTF-8 encoded text files? Wouldn't that be unexpected\n> to the user?\n> \n> Thanks,\n> Lars\n\nThe problem is, if user A has core.autocrlf=true and user B\ncore.autocrlf=false.\n(The line endings don't show up as expected at user B)\n\nHaving said that in all shortness, you convinced me:\nIf text=auto, we care about line endings.\nIf text, we care about line endings.\n\nIf the .gitattributes don't say anything about text,\nwe don't convert eol.\n(In other words: we don't look at core.autocrlf, when checkout-encoding\nis defined)\n\nA new branch is push to github/tboegi\n\n\n"},{"id":"336201","messageId":"xmqq4lnw82hi.fsf@gitster.mtv.corp.google.com","threadId":"47551","inReplyTo":"20180106004808.77513-6-lars.schneider@autodesk.com","subject":"Re: [PATCH v3 5/7] convert_to_git(): safe_crlf/checksafe becomes int conv_flags","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-01-08T21:28:09Z","receivedAt":"2018-01-08T21:28:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"lars.schneider@autodesk.com writes:\n\n> diff --git a/sha1_file.c b/sha1_file.c\n> index afe4b90f6e..dcb02e9ffd 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> @@ -75,14 +75,14 @@ static struct cached_object *find_cached_object(const unsigned char *sha1)\n> }\n> \n> \n> -static enum safe_crlf get_safe_crlf(unsigned flags)\n> +static int get_conv_flags(unsigned flags)\n> {\n> \tif (flags & HASH_RENORMALIZE)\n> -\t\treturn SAFE_CRLF_RENORMALIZE;\n> +\t\treturn CONV_EOL_RENORMALIZE;\n> \telse if (flags & HASH_WRITE_OBJECT)\n> -\t\treturn safe_crlf;\n> +\t\treturn global_conv_flags_eol | CONV_WRITE_OBJECT;\n\nThis macro has not yet introduced at this point (it appears in 6/7\nif I am not mistaken).\n"},{"id":"336214","messageId":"FFFF054A-6049-49A6-9B31-3E2D6585B21A@gmail.com","threadId":"47551","inReplyTo":"xmqq4lnw82hi.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3 5/7] convert_to_git(): safe_crlf/checksafe becomes int conv_flags","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2018-01-08T22:47:20Z","receivedAt":"2018-01-08T22:47:29Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 08 Jan 2018, at 22:28, Junio C Hamano wrote:\n> \n> lars.schneider@autodesk.com writes:\n> \n>> diff --git a/sha1_file.c b/sha1_file.c\n>> index afe4b90f6e..dcb02e9ffd 100644\n>> --- a/sha1_file.c\n>> +++ b/sha1_file.c\n>> @@ -75,14 +75,14 @@ static struct cached_object *find_cached_object(const unsigned char *sha1)\n>> }\n>> \n>> \n>> -static enum safe_crlf get_safe_crlf(unsigned flags)\n>> +static int get_conv_flags(unsigned flags)\n>> {\n>> \tif (flags & HASH_RENORMALIZE)\n>> -\t\treturn SAFE_CRLF_RENORMALIZE;\n>> +\t\treturn CONV_EOL_RENORMALIZE;\n>> \telse if (flags & HASH_WRITE_OBJECT)\n>> -\t\treturn safe_crlf;\n>> +\t\treturn global_conv_flags_eol | CONV_WRITE_OBJECT;\n> \n> This macro has not yet introduced at this point (it appears in 6/7\n> if I am not mistaken).\n\nNice catch. I'll fix that in the next iteration.\n\nIs it OK if I send the next iteration soon or would you prefer\nit if I wait until after 2.16 release?\n\nPlus, is it ok to keep the base of the series or would you prefer\nit if I rebase it to the latest master (because of a minor conflict)?\n\nThanks,\nLars"},{"id":"336215","messageId":"xmqqmv1o6iud.fsf@gitster.mtv.corp.google.com","threadId":"47551","inReplyTo":"FFFF054A-6049-49A6-9B31-3E2D6585B21A@gmail.com","subject":"Re: [PATCH v3 5/7] convert_to_git(): safe_crlf/checksafe becomes int conv_flags","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-01-08T23:17:46Z","receivedAt":"2018-01-08T23:17:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lars Schneider writes:\n\n> Nice catch. I'll fix that in the next iteration.\n>\n> Is it OK if I send the next iteration soon or would you prefer\n> it if I wait until after 2.16 release?\n>\n> Plus, is it ok to keep the base of the series or would you prefer\n> it if I rebase it to the latest master (because of a minor conflict)?\n\nI do not see this topic as a fix for grave bug that needs to go to\nolder maintenance track---it is rather a new feature, isn't it? So\na rebased series that cleanly applies on top of 2.16 final would be\na reasonable way to go forward.\n\nThanks.\n"},{"id":"336220","messageId":"20180109062029.GA18135@tor.lan","threadId":"47551","inReplyTo":"FFFF054A-6049-49A6-9B31-3E2D6585B21A@gmail.com","subject":"Re: [PATCH v3 5/7] convert_to_git(): safe_crlf/checksafe becomes int conv_flags","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2018-01-09T06:20:29Z","receivedAt":"2018-01-09T06:20:53Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Mon, Jan 08, 2018 at 11:47:20PM +0100, Lars Schneider wrote:\n> \n> > On 08 Jan 2018, at 22:28, Junio C Hamano wrote:\n> > \n> > lars.schneider@autodesk.com writes:\n> > \n> >> diff --git a/sha1_file.c b/sha1_file.c\n> >> index afe4b90f6e..dcb02e9ffd 100644\n> >> --- a/sha1_file.c\n> >> +++ b/sha1_file.c\n> >> @@ -75,14 +75,14 @@ static struct cached_object *find_cached_object(const unsigned char *sha1)\n> >> }\n> >> \n> >> \n> >> -static enum safe_crlf get_safe_crlf(unsigned flags)\n> >> +static int get_conv_flags(unsigned flags)\n> >> {\n> >> \tif (flags & HASH_RENORMALIZE)\n> >> -\t\treturn SAFE_CRLF_RENORMALIZE;\n> >> +\t\treturn CONV_EOL_RENORMALIZE;\n> >> \telse if (flags & HASH_WRITE_OBJECT)\n> >> -\t\treturn safe_crlf;\n> >> +\t\treturn global_conv_flags_eol | CONV_WRITE_OBJECT;\n> > \n> > This macro has not yet introduced at this point (it appears in 6/7\n> > if I am not mistaken).\n> \n> Nice catch. I'll fix that in the next iteration.\n> \n> Is it OK if I send the next iteration soon or would you prefer\n> it if I wait until after 2.16 release?\n> \n> Plus, is it ok to keep the base of the series or would you prefer\n> it if I rebase it to the latest master (because of a minor conflict)?\n> \n> Thanks,\n> Lars\n\nI noticed the missing macro as well, while doing the rebase\nto git.git/master, but forget to mention it here on the list\n\nLars, if you want, please have a look here:\nhttps://github.com/tboegi/git/tree/180108-1858-For-lars-schneider-encode-V3C\n"}]}