{"thread":{"id":"41059","subject":"[PATCH v7] ls-files: Add eol diagnostics","startedAt":"2015-12-22T11:45:33Z","lastAt":"2015-12-22T23:35:31Z","messageCount":2,"participants":["Torsten Bögershausen","Junio C Hamano"],"isPatch":true,"patchVersion":7,"patchTotal":null},"messages":[{"id":"274851","messageId":"B82763FE-0B35-4474-9956-B4B52DF6602C@web.de","threadId":"41059","inReplyTo":null,"subject":"[PATCH v7] ls-files: Add eol diagnostics","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2015-12-22T11:45:33Z","receivedAt":"2015-12-22T11:45:33Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"When working in a cross-platform environment, a user wants to\ncheck if text files are stored normalized in the repository and if\n.gitattributes are set appropriately.\n\nMake it possible to let Git show the line endings in the index and\nin the working tree and the effective text/eol attributes.\n\nThe end of line (\"eolinfo\") are shown like this:\n\"binary\"       binary file\n\"text-no-eol\"  text file without any EOL\n\"text-lf\"      text file with LF\n\"text-crlf\"    text file with CRLF\n\"text-crlf-lf\" text file with mixed line endings.\n\nThe effective text/eol attribute is one of these:\n\"\", \"-text\", \"text\", \"text=auto\", \"eol=lf\", \"eol=crlf\"\n\ngit ls-files --eol gives an output like this:\n\ni/text-no-eol   w/text-no-eol   attr/text=auto t/t5100/empty\ni/binary        w/binary        attr/-text     t/test-binary-2.png\ni/text-lf       w/text-lf       attr/eol=lf    t/t5100/rfc2047-info-0007\ni/text-lf       w/text-crlf     attr/eol=crlf  doit.bat\ni/text-crlf-lf  w/text-crlf-lf  attr/          locale/XX.po\n\nNote that the output is meant to be human-readable and may change.\n\nAdd test cases in t0027, thanks to Junio C Hamano for the optimized\ngrep-less sed expression.\n\nHelped-By: Eric Sunshine <sunshine@sunshineco.com>\nSigned-off-by: Torsten Bögershausen <tboegi@web.de>\n---\nChanges against v6 on pu:\n- convert.c: fixed possible mem leak in get_wt_convert_stats_ascii()\n- builtin/ls-files.c: removed not needed ()\n- t0027: Re-added empty line\n        Don't use grep/sed, but sed with an address /(Thanks Junio)\n        More test_when_finished, other small things from the review\n        Add comment on the last TC, which used ls-files -d\n        The last TC may still be classified as fragile, but I couln't\n        motivate myself to use a separated creation/commit for this\n        TC and decided to safe some CPU cycles and re-use what we have\n        in the repo.\nDocumentation/git-ls-files.txt |  22 ++++++++\nbuiltin/ls-files.c             |  19 +++++++\nconvert.c                      |  85 +++++++++++++++++++++++++++++++\nconvert.h                      |   3 ++\nt/t0027-auto-crlf.sh           | 112 ++++++++++++++++++++++++++++++++++++-----\n5 files changed, 229 insertions(+), 12 deletions(-)\n\ndiff --git a/Documentation/git-ls-files.txt b/Documentation/git-ls-files.txt\nindex e26f01f..8f29c99 100644\n--- a/Documentation/git-ls-files.txt\n+++ b/Documentation/git-ls-files.txt\n@@ -12,6 +12,7 @@ SYNOPSIS\n'git ls-files' [-z] [-t] [-v]\n\t\t(--[cached|deleted|others|ignored|stage|unmerged|killed|modified])*\n\t\t(-[c|d|o|i|s|u|k|m])*\n+\t\t[--eol]\n\t\t[-x <pattern>|--exclude=<pattern>]\n\t\t[-X <file>|--exclude-from=<file>]\n\t\t[--exclude-per-directory=<file>]\n@@ -147,6 +148,18 @@ a space) at the start of each line:\n\tpossible for manual inspection; the exact format may change at\n\tany time.\n\n+--eol::\n+\tShow line endings (\"eolinfo\") and the text/eol attributes (\"texteolattr\") of files.\n+\t\"eolinfo\" is the file content identification used by Git when\n+\tthe \"text\" attribute is \"auto\", or core.autocrlf != false.\n+\n+\t\"eolinfo\" is either \"\" (when the the info is not available\"), or one of \"binary\",\n+\t\"text-no-eol\", \"text-lf\", \"text-crlf\" or \"text-crlf-lf\".\n+\tThe \"texteolattr\" can be \"\", \"-text\", \"text\", \"text=auto\", \"eol=lf\", \"eol=crlf\".\n+\n+\tBoth the content in the index (\"i/\") and the content in the working tree (\"w/\")\n+\tare shown for regular files, followed by the \"texteolattr (\"attr/\").\n+\n\\--::\n\tDo not interpret any more arguments as options.\n\n@@ -161,6 +174,15 @@ which case it outputs:\n\n       [<tag> ]<mode> <object> <stage> <file>\n\n+'git ls-files --eol' will show\n+\ti/<eolinfo> w/<eolinfo> attr/<eolattr> <file>\n+\n+'git ls-files --eol -o' will show\n+\ti/          w/<eolinfo> attr/<eolattr> <file>\n+\n+'git ls-files --eol -s' will show\n+[<tag> ]<mode> <object> <stage> i/<eolinfo> w/<eolinfo> attr/<eolattr> <file>\n+\n'git ls-files --unmerged' and 'git ls-files --stage' can be used to examine\ndetailed information on unmerged paths.\n\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex b6a7cb0..9eacc64 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -27,6 +27,7 @@ static int show_killed;\nstatic int show_valid_bit;\nstatic int line_terminator = '\\n';\nstatic int debug_mode;\n+static int show_eol;\n\nstatic const char *prefix;\nstatic int max_prefix_len;\n@@ -47,6 +48,21 @@ static const char *tag_modified = \"\";\nstatic const char *tag_skip_worktree = \"\";\nstatic const char *tag_resolve_undo = \"\";\n\n+static void write_eolinfo(const struct cache_entry *ce, const char *path)\n+{\n+\tstruct stat st;\n+\tconst char *i_txt = \"\";\n+\tconst char *w_txt = \"\";\n+\tif (!show_eol)\n+\t\treturn;\n+\tif (ce && S_ISREG(ce->ce_mode))\n+\t\ti_txt = get_cached_convert_stats_ascii(ce->name);\n+\tif (!lstat(path, &st) && S_ISREG(st.st_mode))\n+\t\tw_txt = get_wt_convert_stats_ascii(path);\n+\tprintf(\"i/%-13s w/%-13s attr/%-9s \", i_txt, w_txt,\n+\t\t\t\t get_convert_attr_ascii(path));\n+}\n+\nstatic void write_name(const char *name)\n{\n\t/*\n@@ -68,6 +84,7 @@ static void show_dir_entry(const char *tag, struct dir_entry *ent)\n\t\treturn;\n\n\tfputs(tag, stdout);\n+\twrite_eolinfo(NULL, ent->name);\n\twrite_name(ent->name);\n}\n\n@@ -170,6 +187,7 @@ static void show_ce_entry(const char *tag, const struct cache_entry *ce)\n\t\t       find_unique_abbrev(ce->sha1,abbrev),\n\t\t       ce_stage(ce));\n\t}\n+\twrite_eolinfo(ce, ce->name);\n\twrite_name(ce->name);\n\tif (debug_mode) {\n\t\tconst struct stat_data *sd = &ce->ce_stat_data;\n@@ -433,6 +451,7 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n\t\tOPT_BIT(0, \"directory\", &dir.flags,\n\t\t\tN_(\"show 'other' directories' names only\"),\n\t\t\tDIR_SHOW_OTHER_DIRECTORIES),\n+\t\tOPT_BOOL(0, \"eol\", &show_eol, N_(\"show line endings of files\")),\n\t\tOPT_NEGBIT(0, \"empty-directory\", &dir.flags,\n\t\t\tN_(\"don't show empty directories\"),\n\t\t\tDIR_HIDE_EMPTY_DIRECTORIES),\ndiff --git a/convert.c b/convert.c\nindex 814e814..69a5588 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -13,6 +13,11 @@\n* translation when the \"text\" attribute or \"auto_crlf\" option is set.\n*/\n\n+/* Stat bits: When BIN is set, the txt bits are unset */\n+#define CONVERT_STAT_BITS_TXT_LF   (1)\n+#define CONVERT_STAT_BITS_TXT_CRLF (2)\n+#define CONVERT_STAT_BITS_BIN      (4)\n+\nenum crlf_action {\n\tCRLF_GUESS = -1,\n\tCRLF_BINARY = 0,\n@@ -95,6 +100,62 @@ static int is_binary(unsigned long size, struct text_stat *stats)\n\treturn 0;\n}\n\n+static unsigned int gather_convert_stats(const char *data, unsigned long size)\n+{\n+\tstruct text_stat stats;\n+\tif (!data || !size)\n+\t\treturn 0;\n+\tgather_stats(data, size, &stats);\n+\tif (is_binary(size, &stats) || stats.cr != stats.crlf)\n+\t\treturn CONVERT_STAT_BITS_BIN;\n+\telse if (stats.crlf && stats.crlf == stats.lf)\n+\t\treturn CONVERT_STAT_BITS_TXT_CRLF;\n+\telse if (stats.crlf && stats.lf)\n+\t\treturn CONVERT_STAT_BITS_TXT_CRLF | CONVERT_STAT_BITS_TXT_LF;\n+\telse if (stats.lf)\n+\t\treturn CONVERT_STAT_BITS_TXT_LF;\n+\telse\n+\t\treturn 0;\n+}\n+\n+static const char *gather_convert_stats_ascii(const char *data, unsigned long size)\n+{\n+\tunsigned int convert_stats = gather_convert_stats(data, size);\n+\n+\tif (convert_stats & CONVERT_STAT_BITS_BIN)\n+\t\treturn \"binary\";\n+\tswitch (convert_stats) {\n+\t\tcase CONVERT_STAT_BITS_TXT_LF:\n+\t\t\treturn \"text-lf\";\n+\t\tcase CONVERT_STAT_BITS_TXT_CRLF:\n+\t\t\treturn \"text-crlf\";\n+\t\tcase CONVERT_STAT_BITS_TXT_LF | CONVERT_STAT_BITS_TXT_CRLF:\n+\t\t\treturn \"text-crlf-lf\";\n+\t\tdefault:\n+\t\t\treturn \"text-no-eol\";\n+\t}\n+}\n+\n+const char *get_cached_convert_stats_ascii(const char *path)\n+{\n+\tconst char *ret;\n+\tunsigned long sz;\n+\tvoid *data = read_blob_data_from_cache(path, &sz);\n+\tret = gather_convert_stats_ascii(data, sz);\n+\tfree(data);\n+\treturn ret;\n+}\n+\n+const char *get_wt_convert_stats_ascii(const char *path)\n+{\n+\tconst char *ret = \"\";\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tif (strbuf_read_file(&sb, path, 0) >= 0)\n+\t\tret = gather_convert_stats_ascii(sb.buf, sb.len);\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\nstatic enum eol output_eol(enum crlf_action crlf_action)\n{\n\tswitch (crlf_action) {\n@@ -777,6 +838,30 @@ int would_convert_to_git_filter_fd(const char *path)\n\treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv->clean);\n}\n\n+const char *get_convert_attr_ascii(const char *path)\n+{\n+\tstruct conv_attrs ca;\n+\tenum crlf_action crlf_action;\n+\n+\tconvert_attrs(&ca, path);\n+\tcrlf_action = input_crlf_action(ca.crlf_action, ca.eol_attr);\n+\tswitch (crlf_action) {\n+\t\tcase CRLF_GUESS:\n+\t\t\treturn \"\";\n+\t\tcase CRLF_BINARY:\n+\t\t\treturn \"-text\";\n+\t\tcase CRLF_TEXT:\n+\t\t\treturn \"text\";\n+\t\tcase CRLF_INPUT:\n+\t\t\treturn \"eol=lf\";\n+\t\tcase CRLF_CRLF:\n+\t\t\treturn \"eol=crlf\";\n+\t\tcase CRLF_AUTO:\n+\t\t\treturn \"text=auto\";\n+\t}\n+\treturn \"\";\n+}\n+\nint convert_to_git(const char *path, const char *src, size_t len,\n                  struct strbuf *dst, enum safe_crlf checksafe)\n{\ndiff --git a/convert.h b/convert.h\nindex d9d853c..ccf436b 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -32,6 +32,9 @@ enum eol {\n};\n\nextern enum eol core_eol;\n+extern const char *get_cached_convert_stats_ascii(const char *path);\n+extern const char *get_wt_convert_stats_ascii(const char *path);\n+extern const char *get_convert_attr_ascii(const char *path);\n\n/* returns 1 if *dst was used */\nextern int convert_to_git(const char *path, const char *src, size_t len,\ndiff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh\nindex b343651..8580dec 100755\n--- a/t/t0027-auto-crlf.sh\n+++ b/t/t0027-auto-crlf.sh\n@@ -56,21 +56,16 @@ create_gitattributes () {\n}\n\ncreate_NNO_files () {\n-\tlfname=$1\n-\tcrlfname=$2\n-\tlfmixcrlf=$3\n-\tlfmixcr=$4\n-\tcrlfnul=$5\n\tfor crlf in false true input\n\tdo\n\t\tfor attr in \"\" auto text -text lf crlf\n\t\tdo\n\t\t\tpfx=NNO_${crlf}_attr_${attr} &&\n-\t\t\tcp $lfname    ${pfx}_LF.txt &&\n-\t\t\tcp $crlfname  ${pfx}_CRLF.txt &&\n-\t\t\tcp $lfmixcrlf ${pfx}_CRLF_mix_LF.txt &&\n-\t\t\tcp $lfmixcr   ${pfx}_LF_mix_CR.txt &&\n-\t\t\tcp $crlfnul   ${pfx}_CRLF_nul.txt\n+\t\t\tcp CRLF_mix_LF ${pfx}_LF.txt &&\n+\t\t\tcp CRLF_mix_LF ${pfx}_CRLF.txt &&\n+\t\t\tcp CRLF_mix_LF ${pfx}_CRLF_mix_LF.txt &&\n+\t\t\tcp CRLF_mix_LF ${pfx}_LF_mix_CR.txt &&\n+\t\t\tcp CRLF_mix_LF ${pfx}_CRLF_nul.txt\n\t\tdone\n\tdone\n}\n@@ -96,7 +91,7 @@ commit_check_warn () {\n\tcrlfnul=$7\n\tpfx=crlf_${crlf}_attr_${attr}\n\tcreate_gitattributes \"$attr\" &&\n-\tfor f in LF CRLF repoMIX LF_mix_CR CRLF_mix_LF LF_nul CRLF_nul\n+\tfor f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul CRLF_nul\n\tdo\n\t\tfname=${pfx}_$f.txt &&\n\t\tcp $f $fname &&\n@@ -149,6 +144,36 @@ commit_chk_wrnNNO () {\n\t'\n}\n\n+stats_ascii () {\n+\tcase \"$1\" in\n+\t\tLF)\n+\t\techo text-lf\n+\t\t;;\n+\t\tCRLF)\n+\t\techo text-crlf\n+\t\t;;\n+\t\tCRLF_mix_LF)\n+\t\techo text-crlf-lf\n+\t\t;;\n+\t\tLF_mix_CR)\n+\t\techo binary\n+\t\t;;\n+\t\tCRLF_nul)\n+\t\techo binary\n+\t\t;;\n+\t\tLF_nul)\n+\t\techo binary\n+\t\t;;\n+\t\tCRLF_mix_CR)\n+\t\techo binary\n+\t\t;;\n+\t\t*)\n+\t\techo error_invalid $1\n+\t\t;;\n+\tesac\n+\n+}\n+\ncheck_files_in_repo () {\n\tcrlf=$1\n\tattr=$2\n@@ -214,6 +239,20 @@ checkout_files () {\n\t\tfi\n\tdone\n\n+\ttest_expect_success \"ls-files --eol $lfname ${pfx}LF.txt\" \"\n+\t\ttest_when_finished 'rm e expect actual' &&\n+\t\tcat >e <<-EOF &&\n+\t\ti/text-crlf w/$(stats_ascii $crlfname) ${src}CRLF.txt\n+\t\ti/text-crlf-lf w/$(stats_ascii $lfmixcrlf) ${src}CRLF_mix_LF.txt\n+\t\ti/text-lf w/$(stats_ascii $lfname) ${src}LF.txt\n+\t\ti/binary w/$(stats_ascii $lfmixcr) ${src}LF_mix_CR.txt\n+\t\ti/binary w/$(stats_ascii $crlfnul) ${src}CRLF_nul.txt\n+\t\ti/binary w/$(stats_ascii $crlfnul) ${src}LF_nul.txt\n+\t\tEOF\n+\t\tsort <e >expect &&\n+\t\tgit ls-files --eol $src* | sed -e 's!attr/[=a-z-]*!!g' -e 's/  */ /g' | sort >actual &&\n+\t\ttest_cmp expect actual\n+\t\"\n\ttest_expect_success \"checkout core.eol=$eol core.autocrlf=$crlf gitattributes=$attr file=LF\" \"\n\t\tcompare_ws_file $pfx $lfname    ${src}LF.txt\n\t\"\n@@ -231,7 +270,41 @@ checkout_files () {\n\t\"\n}\n\n-#######\n+# Test control characters\n+# NUL SOH CR EOF==^Z\n+test_expect_success 'ls-files --eol -o Text/Binary' '\n+\ttest_when_finished \"rm e expect actual TeBi_*\" &&\n+\tSTRT=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA &&\n+\tSTR=$STRT$STRT$STRT$STRT &&\n+\tprintf \"${STR}BBB\\001\" >TeBi_127_S &&\n+\tprintf \"${STR}BBBB\\001\">TeBi_128_S &&\n+\tprintf \"${STR}BBB\\032\" >TeBi_127_E &&\n+\tprintf \"\\032${STR}BBB\" >TeBi_E_127 &&\n+\tprintf \"${STR}BBBB\\000\">TeBi_128_N &&\n+\tprintf \"${STR}BBB\\012\">TeBi_128_L &&\n+\tprintf \"${STR}BBB\\015\">TeBi_127_C &&\n+\tprintf \"${STR}BB\\015\\012\" >TeBi_126_CL &&\n+\tprintf \"${STR}BB\\015\\012\\015\" >TeBi_126_CLC &&\n+\tcat >e <<-\\EOF &&\n+\ti/ w/binary TeBi_127_S\n+\ti/ w/text-no-eol TeBi_128_S\n+\ti/ w/text-no-eol TeBi_127_E\n+\ti/ w/binary TeBi_E_127\n+\ti/ w/binary TeBi_128_N\n+\ti/ w/text-lf TeBi_128_L\n+\ti/ w/binary TeBi_127_C\n+\ti/ w/text-crlf TeBi_126_CL\n+\ti/ w/binary TeBi_126_CLC\n+\tEOF\n+\tsort <e >expect &&\n+\tgit ls-files --eol -o |\n+\tsed -n -e \"/TeBi_/{s!attr/[=a-z-]*!!g\n+\ts!  *! !g\n+\tp\n+\t}\" | sort >actual &&\n+\ttest_cmp expect actual\n+'\n+\ntest_expect_success 'setup master' '\n\techo >.gitattributes &&\n\tgit checkout -b master &&\n@@ -480,4 +553,19 @@ checkout_files    native  true  \"lf\"      LF    CRLF  CRLF_mix_LF  LF_mix_CR\ncheckout_files    native  false \"crlf\"    CRLF  CRLF  CRLF         CRLF_mix_CR  CRLF_nul\ncheckout_files    native  true  \"crlf\"    CRLF  CRLF  CRLF         CRLF_mix_CR  CRLF_nul\n\n+# Should be the last test case: remove some files from the worktree\n+# This test assumes that \"rm\" can remove staged files\n+test_expect_success 'ls-files --eol -d' \"\n+\trm crlf_false_attr__CRLF.txt crlf_false_attr__CRLF_mix_LF.txt crlf_false_attr__LF.txt .gitattributes &&\n+\tcat >expect <<-\\EOF &&\n+\ti/text-crlf w/ crlf_false_attr__CRLF.txt\n+\ti/text-crlf-lf w/ crlf_false_attr__CRLF_mix_LF.txt\n+\ti/text-lf w/ .gitattributes\n+\ti/text-lf w/ crlf_false_attr__LF.txt\n+\tEOF\n+\tgit ls-files --eol -d | sed -e 's!attr/[=a-z-]*!!g' -e 's/  */ /g' | sort >actual &&\n+\ttest_cmp expect actual &&\n+\trm expect actual\n+\"\n+\ntest_done\n-- \n2.5.0\n"},{"id":"274924","messageId":"xmqq37uucb0c.fsf@gitster.mtv.corp.google.com","threadId":"41059","inReplyTo":"B82763FE-0B35-4474-9956-B4B52DF6602C@web.de","subject":"Re: [PATCH v7] ls-files: Add eol diagnostics","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-12-22T23:35:31Z","receivedAt":"2015-12-22T23:35:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen <tboegi@web.de> writes:\n\n> When working in a cross-platform environment, a user wants to\n> check if text files are stored normalized in the repository and if\n> .gitattributes are set appropriately.\n>\n> Make it possible to let Git show the line endings in the index and\n> in the working tree and the effective text/eol attributes.\n>\n> The end of line (\"eolinfo\") are shown like this:\n> \"binary\"       binary file\n> \"text-no-eol\"  text file without any EOL\n> \"text-lf\"      text file with LF\n> \"text-crlf\"    text file with CRLF\n> \"text-crlf-lf\" text file with mixed line endings.\n>\n> The effective text/eol attribute is one of these:\n> \"\", \"-text\", \"text\", \"text=auto\", \"eol=lf\", \"eol=crlf\"\n>\n> git ls-files --eol gives an output like this:\n>\n> i/text-no-eol   w/text-no-eol   attr/text=auto t/t5100/empty\n> i/binary        w/binary        attr/-text     t/test-binary-2.png\n> i/text-lf       w/text-lf       attr/eol=lf    t/t5100/rfc2047-info-0007\n> i/text-lf       w/text-crlf     attr/eol=crlf  doit.bat\n> i/text-crlf-lf  w/text-crlf-lf  attr/          locale/XX.po\n>\n> Note that the output is meant to be human-readable and may change.\n\nWait, what?  \n\nI've been assuming that these output being designed was to be read\nby machine, because this new feature is implemented as a part of the\ncommand \"ls-files\", which is plumbing whose output is meant for\nscript consumption.\n\n> +--eol::\n> +\tShow line endings (\"eolinfo\") and the text/eol attributes (\"texteolattr\") of files.\n> +\t\"eolinfo\" is the file content identification used by Git when\n> +\tthe \"text\" attribute is \"auto\", or core.autocrlf != false.\n> +\n> +\t\"eolinfo\" is either \"\" (when the the info is not available\"), or one of \"binary\",\n> +\t\"text-no-eol\", \"text-lf\", \"text-crlf\" or \"text-crlf-lf\".\n> +\tThe \"texteolattr\" can be \"\", \"-text\", \"text\", \"text=auto\", \"eol=lf\", \"eol=crlf\".\n> +\n> +\tBoth the content in the index (\"i/\") and the content in the working tree (\"w/\")\n> +\tare shown for regular files, followed by the \"texteolattr (\"attr/\").\n> +\n\nDoes this format well, or would the second and third paragraph be\nformat in a funny way?\n"}]}