{"thread":{"id":"45551","subject":"[BUG?] iconv used as textconv, and spurious ^M on added lines on Windows","startedAt":"2017-03-30T19:35:48Z","lastAt":"2017-04-02T11:41:18Z","messageCount":10,"participants":["Jakub Narębski","Jeff King","Torsten Bögershausen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"315811","messageId":"feaeade7-aeb5-fa67-ab29-9106aeadb2a6@gmail.com","threadId":"45551","inReplyTo":null,"subject":"[BUG?] iconv used as textconv, and spurious ^M on added lines on Windows","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2017-03-30T19:35:27Z","receivedAt":"2017-03-30T19:35:48Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Hello,\n\nRecently I had to work on a project which uses legacy 8-bit encoding\n(namely cp1250 encoding) instead of utf-8 for text files (LaTeX\ndocuments).  My terminal, that is Git Bash from Git for Windows is set\nup for utf-8.\n\nI wanted for \"git diff\" and friends to return something sane on said\nutf-8 terminal, instead of mojibake.  There is 'encoding'\ngitattribute... but it works only for GUI ('git gui', that is).\n\nTherefore I have (ab)used textconv facility to convert from cp1250 of\nfile encoding to utf-8 encoding of console.\n\nI have set the following in .gitattributes file:\n\n  ## LaTeX documents in cp1250 encoding\n  *.tex text diff=mylatex\n\nThe 'mylatex' driver is defined as:\n\n  [diff \"mylatex\"]\n        xfuncname = \"^(\\\\\\\\((sub)*section|chapter|part)\\\\*{0,1}\\\\{.*)$\"\n        wordRegex = \"\\\\\\\\[a-zA-Z]+|[{}]|\\\\\\\\.|[^\\\\{}[:space:]]+\"\n        textconv  = \\\"C:/Program Files/Git/usr/bin/iconv.exe\\\" -f cp1250 -t utf-8\n        cachetextconv = true\n\nAnd everything would be all right... if not the fact that Git appends\nspurious ^M to added lines in the `git diff` output.  Files use CRLF\nend-of-line convention (the native MS Windows one).\n\n  $ git diff test.tex\n  diff --git a/test.tex b/test.tex\n  index 029646e..250ab16 100644\n  --- a/test.tex\n  +++ b/test.tex\n  @@ -1,4 +1,4 @@\n  -\\documentclass{article}\n  +\\documentclass{mwart}^M\n  \n   \\usepackage[cp1250]{inputenc}\n   \\usepackage{polski}\n\nWhat gives?  Why there is this ^M tacked on the end of added lines,\nwhile it is not present in deleted lines, nor in content lines?\n\nPuzzled.\n\nP.S. Git has `i18n.commitEncoding` and `i18n.logOutputEncoding`; pity\nthat it doesn't supports in core `encoding` attribute together with\nhaving `i18n.outputEncoding`.\n--\nJakub Narębski\n\n\n"},{"id":"315816","messageId":"20170330200021.c2l5jak3xb5aoxyc@sigill.intra.peff.net","threadId":"45551","inReplyTo":"feaeade7-aeb5-fa67-ab29-9106aeadb2a6@gmail.com","subject":"Re: [BUG?] iconv used as textconv, and spurious ^M on added lines on Windows","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-03-30T20:00:22Z","receivedAt":"2017-03-30T20:00:28Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 30, 2017 at 09:35:27PM +0200, Jakub Narębski wrote:\n\n> And everything would be all right... if not the fact that Git appends\n> spurious ^M to added lines in the `git diff` output.  Files use CRLF\n> end-of-line convention (the native MS Windows one).\n> \n>   $ git diff test.tex\n>   diff --git a/test.tex b/test.tex\n>   index 029646e..250ab16 100644\n>   --- a/test.tex\n>   +++ b/test.tex\n>   @@ -1,4 +1,4 @@\n>   -\\documentclass{article}\n>   +\\documentclass{mwart}^M\n>   \n>    \\usepackage[cp1250]{inputenc}\n>    \\usepackage{polski}\n> \n> What gives?  Why there is this ^M tacked on the end of added lines,\n> while it is not present in deleted lines, nor in content lines?\n\nPerhaps it's trailing whitespace highlighting for added lines? You can\nadd \"cr-at-eol\" to core.whitespace to suppress it.\n\nI suspect in the normal case that git is doing line-ending conversion,\nbut it's suppressed when textconv is in use.\n\n-Peff\n"},{"id":"315881","messageId":"264c72d0-9558-fa0d-e5ee-eaca894538be@web.de","threadId":"45551","inReplyTo":"feaeade7-aeb5-fa67-ab29-9106aeadb2a6@gmail.com","subject":"Re: [BUG?] iconv used as textconv, and spurious ^M on added lines on Windows","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2017-03-31T12:38:05Z","receivedAt":"2017-03-31T12:38:15Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 30.03.17 21:35, Jakub Narębski wrote:\n> Hello,\n> \n> Recently I had to work on a project which uses legacy 8-bit encoding\n> (namely cp1250 encoding) instead of utf-8 for text files (LaTeX\n> documents).  My terminal, that is Git Bash from Git for Windows is set\n> up for utf-8.\n> \n> I wanted for \"git diff\" and friends to return something sane on said\n> utf-8 terminal, instead of mojibake.  There is 'encoding'\n> gitattribute... but it works only for GUI ('git gui', that is).\n> \n> Therefore I have (ab)used textconv facility to convert from cp1250 of\n> file encoding to utf-8 encoding of console.\n> \n> I have set the following in .gitattributes file:\n> \n>   ## LaTeX documents in cp1250 encoding\n>   *.tex text diff=mylatex\n> \n> The 'mylatex' driver is defined as:\n> \n>   [diff \"mylatex\"]\n>         xfuncname = \"^(\\\\\\\\((sub)*section|chapter|part)\\\\*{0,1}\\\\{.*)$\"\n>         wordRegex = \"\\\\\\\\[a-zA-Z]+|[{}]|\\\\\\\\.|[^\\\\{}[:space:]]+\"\n>         textconv  = \\\"C:/Program Files/Git/usr/bin/iconv.exe\\\" -f cp1250 -t utf-8\n>         cachetextconv = true\n> \n> And everything would be all right... if not the fact that Git appends\n> spurious ^M to added lines in the `git diff` output.  Files use CRLF\n> end-of-line convention (the native MS Windows one).\n> \n>   $ git diff test.tex\n>   diff --git a/test.tex b/test.tex\n>   index 029646e..250ab16 100644\n>   --- a/test.tex\n>   +++ b/test.tex\n>   @@ -1,4 +1,4 @@\n>   -\\documentclass{article}\n>   +\\documentclass{mwart}^M\n>   \n>    \\usepackage[cp1250]{inputenc}\n>    \\usepackage{polski}\n> \n> What gives?  Why there is this ^M tacked on the end of added lines,\n> while it is not present in deleted lines, nor in content lines?\n> \n> Puzzled.\n> \n> P.S. Git has `i18n.commitEncoding` and `i18n.logOutputEncoding`; pity\n> that it doesn't supports in core `encoding` attribute together with\n> having `i18n.outputEncoding`.\n> --\n> Jakub Narębski\n> \n> \nIs there a chance to give us a receipt how to reproduce it?\nA complete test script or ?\n(I don't want to speculate, if the invocation of iconv is the problem,\n where stdout is not in \"binary mode\", or however this is called under Windows)\n\n\n\n\n"},{"id":"315887","messageId":"e1a6d44c-b01b-993c-6a22-e6ac0abca03c@gmail.com","threadId":"45551","inReplyTo":"20170330200021.c2l5jak3xb5aoxyc@sigill.intra.peff.net","subject":"Re: [BUG?] iconv used as textconv, and spurious ^M on added lines on Windows","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2017-03-31T13:24:48Z","receivedAt":"2017-03-31T13:26:25Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 30.03.2017 o 22:00, Jeff King pisze:\n> On Thu, Mar 30, 2017 at 09:35:27PM +0200, Jakub Narębski wrote:\n> \n>> And everything would be all right... if not the fact that Git appends\n>> spurious ^M to added lines in the `git diff` output.  Files use CRLF\n>> end-of-line convention (the native MS Windows one).\n>>\n>>   $ git diff test.tex\n>>   diff --git a/test.tex b/test.tex\n>>   index 029646e..250ab16 100644\n>>   --- a/test.tex\n>>   +++ b/test.tex\n>>   @@ -1,4 +1,4 @@\n>>   -\\documentclass{article}\n>>   +\\documentclass{mwart}^M\n>>   \n>>    \\usepackage[cp1250]{inputenc}\n>>    \\usepackage{polski}\n>>\n>> What gives?  Why there is this ^M tacked on the end of added lines,\n>> while it is not present in deleted lines, nor in content lines?\n\nGah, I forgot that Git for Windows installed with default options uses\n`core.autocrlf=true`, so file contents is stored in repository and\nin the index using LF end-of-line convention -- that is why there is\nno ^M in pre-image (in removed lines).\n \n> Perhaps it's trailing whitespace highlighting for added lines? You can\n> add \"cr-at-eol\" to core.whitespace to suppress it.\n\nThanks! That solves the problem (or rather workarounds it).\n\n> \n> I suspect in the normal case that git is doing line-ending conversion,\n> but it's suppressed when textconv is in use.\n\nI would not consider this a bug if not for the fact that there is no ^M\nwithout using iconv as textconv.\n\nCompare (without textconv => no ^M, but mojibake):\n\n  $ git diff test.txt\n  diff --git a/test.txt b/test.txt\n  index 029646e..38cd657 100644\n  --- a/test.txt\n  +++ b/test.txt\n  @@ -1,9 +1,10 @@\n  -\\documentclass{article}\n  +\\documentclass{mwart}\n  \n   \\usepackage[cp1250]{inputenc}\n   \\usepackage{polski}\n  \n   \\begin{document}\n  +Za<BF><F3><B3><E6> g<EA><9C>l<B9> ja<9F><F1>!\n  \n   \\end{document}\n\nwith the following (with textconv => no gibberish, but ^M):\n\n  $ git diff test.tex\n  diff --git a/test.tex b/test.tex\n  index 029646e..38cd657 100644\n  --- a/test.tex\n  +++ b/test.tex\n  @@ -1,9 +1,10 @@\n  -\\documentclass{article}\n  +\\documentclass{mwart}^M\n  \n   \\usepackage[cp1250]{inputenc}\n   \\usepackage{polski}\n  \n   \\begin{document}\n  +Zażółć gęślą jaźń!^M\n  \n   \\end{document}\n\n-- \nJakub Narębski\n\n"},{"id":"315951","messageId":"bbd60ab1-1309-6b1e-9b7f-09764bab5ccd@gmail.com","threadId":"45551","inReplyTo":"264c72d0-9558-fa0d-e5ee-eaca894538be@web.de","subject":"Re: [BUG?] iconv used as textconv, and spurious ^M on added lines on Windows","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2017-03-31T19:44:15Z","receivedAt":"2017-03-31T19:44:39Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 31.03.2017 o 14:38, Torsten Bögershausen pisze:\n> On 30.03.17 21:35, Jakub Narębski wrote:\n>> Hello,\n>>\n>> Recently I had to work on a project which uses legacy 8-bit encoding\n>> (namely cp1250 encoding) instead of utf-8 for text files (LaTeX\n>> documents).  My terminal, that is Git Bash from Git for Windows is set\n>> up for utf-8.\n>>\n>> I wanted for \"git diff\" and friends to return something sane on said\n>> utf-8 terminal, instead of mojibake.  There is 'encoding'\n>> gitattribute... but it works only for GUI ('git gui', that is).\n>>\n>> Therefore I have (ab)used textconv facility to convert from cp1250 of\n>> file encoding to utf-8 encoding of console.\n>>\n>> I have set the following in .gitattributes file:\n>>\n>>   ## LaTeX documents in cp1250 encoding\n>>   *.tex text diff=mylatex\n>>\n>> The 'mylatex' driver is defined as:\n>>\n>>   [diff \"mylatex\"]\n>>         xfuncname = \"^(\\\\\\\\((sub)*section|chapter|part)\\\\*{0,1}\\\\{.*)$\"\n>>         wordRegex = \"\\\\\\\\[a-zA-Z]+|[{}]|\\\\\\\\.|[^\\\\{}[:space:]]+\"\n>>         textconv  = \\\"C:/Program Files/Git/usr/bin/iconv.exe\\\" -f cp1250 -t utf-8\n>>         cachetextconv = true\n>>\n>> And everything would be all right... if not the fact that Git appends\n>> spurious ^M to added lines in the `git diff` output.  Files use CRLF\n>> end-of-line convention (the native MS Windows one).\n>>\n>>   $ git diff test.tex\n>>   diff --git a/test.tex b/test.tex\n>>   index 029646e..250ab16 100644\n>>   --- a/test.tex\n>>   +++ b/test.tex\n>>   @@ -1,4 +1,4 @@\n>>   -\\documentclass{article}\n>>   +\\documentclass{mwart}^M\n>>   \n>>    \\usepackage[cp1250]{inputenc}\n>>    \\usepackage{polski}\n>>\n>> What gives?  Why there is this ^M tacked on the end of added lines,\n>> while it is not present in deleted lines, nor in content lines?\n>>\n>> Puzzled.\n>>\n>> P.S. Git has `i18n.commitEncoding` and `i18n.logOutputEncoding`; pity\n>> that it doesn't supports in core `encoding` attribute together with\n>> having `i18n.outputEncoding`.\n>\n> Is there a chance to give us a receipt how to reproduce it?\n> A complete test script or ?\n> (I don't want to speculate, if the invocation of iconv is the problem,\n>  where stdout is not in \"binary mode\", or however this is called under Windows)\n\nI'm sorry, I though I posted whole recipe, but I missed some details\nin the above description of the case.\n\nFirst, files are stored on filesystem using CRLF eol (DOS end-of-line\nconvention).  Due to `core.autocrlf` they are converted to LF in blobs,\nthat is in the index and in the repository.\n\nSecond, a textconv with filter preserving end-of-line needs to be\nconfigured.  I have used `iconv`, but I suspect that the problem would\nhappen also for `cat`.\n\nIn the .gitattributes file, or .git/info/attributes add, for example:\n\n  *.tex text diff=myconv\n\nIn the .git/config configure the textconv filter, for example:\n\n  [diff \"myconv\"]\n         textconv  = iconv.exe -f cp1250 -t utf-8\n\nCreate a file which filename matches the attribute line, and which\nuses CRLF end of line convention, and add it to Git (adding it to\nthe index):\n\n  $ printf \"foo\\r\\n\" >foo.tex\n  $ git add foo.tex\n\nModify file (also with CRLF):\n\n  $ printf \"bar\\r\\n\" >foo.tex\n\nCheck the difference\n\n  $ git diff foo.tex\n\nHTH\n-- \nJakub Narębski\n\n"},{"id":"315992","messageId":"20170401060800.hrpqqgdx6t262c7f@sigill.intra.peff.net","threadId":"45551","inReplyTo":"e1a6d44c-b01b-993c-6a22-e6ac0abca03c@gmail.com","subject":"Re: [BUG?] iconv used as textconv, and spurious ^M on added lines on Windows","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-04-01T06:08:00Z","receivedAt":"2017-04-01T06:08:08Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 31, 2017 at 03:24:48PM +0200, Jakub Narębski wrote:\n\n> > I suspect in the normal case that git is doing line-ending conversion,\n> > but it's suppressed when textconv is in use.\n> \n> I would not consider this a bug if not for the fact that there is no ^M\n> without using iconv as textconv.\n\nI don't think it's a bug, though. You have told Git that you will\nconvert the contents (whatever their format) into the canonical format,\nbut your program to do so includes a CR.\n\nWe _could_ further process with other canonicalizations, but I'm not\nsure that is a good idea (line-endings sound reasonably harmless, but\nalmost certainly we should not be doing clean/smudge filtering). And I'm\nnot sure if there would be any compatibility fallouts.\n\nSo I think the behavior is perhaps not what you want, but it's not an\nunreasonable one. And the solution is to define your textconv such that\nit produces clean LF-only output. Perhaps:\n\n  [diff.whatever]\n  textconv = \"iconv ... | tr -d '\\r'\"\n\n?\n\n-Peff\n"},{"id":"316019","messageId":"63eb5546-0dce-2f69-c2f8-1e777b97c532@gmail.com","threadId":"45551","inReplyTo":"20170401060800.hrpqqgdx6t262c7f@sigill.intra.peff.net","subject":"Re: [BUG?] iconv used as textconv, and spurious ^M on added lines on Windows","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2017-04-01T18:31:27Z","receivedAt":"2017-04-01T18:31:46Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 01.04.2017 o 08:08, Jeff King pisze:\n> On Fri, Mar 31, 2017 at 03:24:48PM +0200, Jakub Narębski wrote:\n> \n>>> I suspect in the normal case that git is doing line-ending conversion,\n>>> but it's suppressed when textconv is in use.\n>>\n>> I would not consider this a bug if not for the fact that there is no ^M\n>> without using iconv as textconv.\n> \n> I don't think it's a bug, though. You have told Git that you will\n> convert the contents (whatever their format) into the canonical format,\n> but your program to do so includes a CR.\n\nWell, I have not declared file binary with \"binary = true\" in diff driver\ndefinition, isn't it?\n\n> \n> We _could_ further process with other canonicalizations, but I'm not\n> sure that is a good idea (line-endings sound reasonably harmless, but\n> almost certainly we should not be doing clean/smudge filtering). And I'm\n> not sure if there would be any compatibility fallouts.\n\nYes, gitattributes(5) defines interaction between 'text'/'eol', 'ident'\nand 'filter' attributes, but nothing about 'diff' and 'text'/'eol'.\n\n> \n> So I think the behavior is perhaps not what you want, but it's not an\n> unreasonable one. And the solution is to define your textconv such that\n> it produces clean LF-only output. Perhaps:\n> \n>   [diff.whatever]\n>   textconv = \"iconv ... | tr -d '\\r'\"\n\nWell, either this (or equivalent using dos2unix), or using 'whitespace'\nattribute (or 'core.whitespace') with cr-at-eol.\n\n\nP.S. What do you think about Git supporting 'encoding' attribute (or\n'core.encoding' config) plus 'core.outputEncoding' in-core?\n\nBest,\n-- \nJakub Narębski\n"},{"id":"316038","messageId":"fde70c33-72ed-4035-2a55-2184fa9ac553@web.de","threadId":"45551","inReplyTo":"bbd60ab1-1309-6b1e-9b7f-09764bab5ccd@gmail.com","subject":"Re: [BUG?] iconv used as textconv, and spurious ^M on added lines on Windows","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2017-04-02T04:34:26Z","receivedAt":"2017-04-02T04:34:31Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 2017-03-31 21:44, Jakub Narębski wrote:\n> W dniu 31.03.2017 o 14:38, Torsten Bögershausen pisze:\n>> On 30.03.17 21:35, Jakub Narębski wrote:\n>>> Hello,\n>>>\n>>> Recently I had to work on a project which uses legacy 8-bit encoding\n>>> (namely cp1250 encoding) instead of utf-8 for text files (LaTeX\n>>> documents).  My terminal, that is Git Bash from Git for Windows is set\n>>> up for utf-8.\n>>>\n>>> I wanted for \"git diff\" and friends to return something sane on said\n>>> utf-8 terminal, instead of mojibake.  There is 'encoding'\n>>> gitattribute... but it works only for GUI ('git gui', that is).\n>>>\n>>> Therefore I have (ab)used textconv facility to convert from cp1250 of\n>>> file encoding to utf-8 encoding of console.\n>>>\n>>> I have set the following in .gitattributes file:\n>>>\n>>>   ## LaTeX documents in cp1250 encoding\n>>>   *.tex text diff=mylatex\n>>>\n>>> The 'mylatex' driver is defined as:\n>>>\n>>>   [diff \"mylatex\"]\n>>>         xfuncname = \"^(\\\\\\\\((sub)*section|chapter|part)\\\\*{0,1}\\\\{.*)$\"\n>>>         wordRegex = \"\\\\\\\\[a-zA-Z]+|[{}]|\\\\\\\\.|[^\\\\{}[:space:]]+\"\n>>>         textconv  = \\\"C:/Program Files/Git/usr/bin/iconv.exe\\\" -f cp1250 -t utf-8\n>>>         cachetextconv = true\n>>>\n>>> And everything would be all right... if not the fact that Git appends\n>>> spurious ^M to added lines in the `git diff` output.  Files use CRLF\n>>> end-of-line convention (the native MS Windows one).\n>>>\n>>>   $ git diff test.tex\n>>>   diff --git a/test.tex b/test.tex\n>>>   index 029646e..250ab16 100644\n>>>   --- a/test.tex\n>>>   +++ b/test.tex\n>>>   @@ -1,4 +1,4 @@\n>>>   -\\documentclass{article}\n>>>   +\\documentclass{mwart}^M\n>>>   \n>>>    \\usepackage[cp1250]{inputenc}\n>>>    \\usepackage{polski}\n>>>\n>>> What gives?  Why there is this ^M tacked on the end of added lines,\n>>> while it is not present in deleted lines, nor in content lines?\n>>>\n>>> Puzzled.\n>>>\n>>> P.S. Git has `i18n.commitEncoding` and `i18n.logOutputEncoding`; pity\n>>> that it doesn't supports in core `encoding` attribute together with\n>>> having `i18n.outputEncoding`.\n>>\n>> Is there a chance to give us a receipt how to reproduce it?\n>> A complete test script or ?\n>> (I don't want to speculate, if the invocation of iconv is the problem,\n>>  where stdout is not in \"binary mode\", or however this is called under Windows)\n> \n> I'm sorry, I though I posted whole recipe, but I missed some details\n> in the above description of the case.\n> \n> First, files are stored on filesystem using CRLF eol (DOS end-of-line\n> convention).  Due to `core.autocrlf` they are converted to LF in blobs,\n> that is in the index and in the repository.\n> \n> Second, a textconv with filter preserving end-of-line needs to be\n> configured.  I have used `iconv`, but I suspect that the problem would\n> happen also for `cat`.\n> \n> In the .gitattributes file, or .git/info/attributes add, for example:\n> \n>   *.tex text diff=myconv\n> \n> In the .git/config configure the textconv filter, for example:\n> \n>   [diff \"myconv\"]\n>          textconv  = iconv.exe -f cp1250 -t utf-8\n> \n> Create a file which filename matches the attribute line, and which\n> uses CRLF end of line convention, and add it to Git (adding it to\n> the index):\n> \n>   $ printf \"foo\\r\\n\" >foo.tex\n>   $ git add foo.tex\n> \n> Modify file (also with CRLF):\n> \n>   $ printf \"bar\\r\\n\" >foo.tex\n> \n> Check the difference\n> \n>   $ git diff foo.tex\n> \n> HTH\n> \n\nThere seems to be a bug in Git, when it comes to \"git diff\".\nBefore we feed the content of the working tree into the\ndiff machinery, a call to convert_to_git() should be made.\nBut it seems as there is something missing, the expected\n\"+fox\" becomes a \"+foxQ\"\n\n#!/bin/sh\n\ntest_description='CRLF with diff filter'\n\n. ./test-lib.sh\n\ntest_expect_success 'setup' '\n\tgit config core.autocrlf input &&\n\tprintf \"foo\\r\\n\" >foo.tex &&\n  git add foo.tex &&\n\techo >.gitattributes &&\n\tgit checkout -b master &&\n\tgit add .gitattributes &&\n\tgit commit -m \"Add foo.txt\" &&\n\tcat >.git/config <<-\\EOF\n\t[diff \"myconv\"]\n\t       textconv  = sed -e \"s/f/g\"\nEOF\n'\n\ntest_expect_success 'check EOL in diff' '\n\tprintf \"fox\\r\\n\" >foo.tex &&\n\tcat >expect <<-\\EOF &&\ndiff --git a/foo.tex b/foo.tex\nindex 257cc56..88c2893 100644\n--- a/foo.tex\n+++ b/foo.tex\n@@ -1 +1 @@\n-foo\n+fox\nEOF\n\tgit diff foo.tex | tr \"\\015\" Q >actual &&\n\ttest_cmp expect actual\n'\n\ntest_done\n\n"},{"id":"316049","messageId":"20170402074522.4qhannjus4ynwx4i@sigill.intra.peff.net","threadId":"45551","inReplyTo":"63eb5546-0dce-2f69-c2f8-1e777b97c532@gmail.com","subject":"Re: [BUG?] iconv used as textconv, and spurious ^M on added lines on Windows","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-04-02T07:45:22Z","receivedAt":"2017-04-02T07:45:29Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Apr 01, 2017 at 08:31:27PM +0200, Jakub Narębski wrote:\n\n> W dniu 01.04.2017 o 08:08, Jeff King pisze:\n> > On Fri, Mar 31, 2017 at 03:24:48PM +0200, Jakub Narębski wrote:\n> > \n> >>> I suspect in the normal case that git is doing line-ending conversion,\n> >>> but it's suppressed when textconv is in use.\n> >>\n> >> I would not consider this a bug if not for the fact that there is no ^M\n> >> without using iconv as textconv.\n> > \n> > I don't think it's a bug, though. You have told Git that you will\n> > convert the contents (whatever their format) into the canonical format,\n> > but your program to do so includes a CR.\n> \n> Well, I have not declared file binary with \"binary = true\" in diff driver\n> definition, isn't it?\n\nI don't think binary has anything to do with it. A textconv filter takes\ninput (binary or not) and delivers a normalized representation to feed\nto the diff algorithm. There's no further post-processing, and it's the\nresponsibility of the filter to deliver the bytes it wants diffed.\n\nLike I said, I could see an argument for treating the filter output as\ntext to be pre-processed, but that's not how it works (and I don't think\nit is a good idea to change it now, unless by adding an option to the\ndiff filter).\n\n> P.S. What do you think about Git supporting 'encoding' attribute (or\n> 'core.encoding' config) plus 'core.outputEncoding' in-core?\n\nSupporting an \"encoding\" attribute to normalize file encodings in diffs\nseems reasonable to me. But it would have to be enabled only for\nhuman-readable diffs, as the result could not be applied (so the same as\ntextconv).\n\nI don't think core.outputEncoding is necessarily a good idea. We are not\nreally equipped anything that isn't an ascii superset, as we intermingle\nthe bytes with ascii diff headers (though I think that is true of the\ncommitEncoding stuff; I assume everything breaks horribly if you tried\nto set that to UTF-16, but I've never tried it).\n\n-Peff\n"},{"id":"316054","messageId":"3e35eda2-0240-8504-b56e-f66092cc1775@gmail.com","threadId":"45551","inReplyTo":"20170402074522.4qhannjus4ynwx4i@sigill.intra.peff.net","subject":"Re: [BUG?] iconv used as textconv, and spurious ^M on added lines on Windows","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2017-04-02T11:40:58Z","receivedAt":"2017-04-02T11:41:18Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 02.04.2017 o 09:45, Jeff King pisze:\n> On Sat, Apr 01, 2017 at 08:31:27PM +0200, Jakub Narębski wrote:\n> \n>> W dniu 01.04.2017 o 08:08, Jeff King pisze:\n>>> On Fri, Mar 31, 2017 at 03:24:48PM +0200, Jakub Narębski wrote:\n>>>\n>>>>> I suspect in the normal case that git is doing line-ending conversion,\n>>>>> but it's suppressed when textconv is in use.\n>>>>\n>>>> I would not consider this a bug if not for the fact that there is no ^M\n>>>> without using iconv as textconv.\n>>>\n>>> I don't think it's a bug, though. You have told Git that you will\n>>> convert the contents (whatever their format) into the canonical format,\n>>> but your program to do so includes a CR.\n>>\n>> Well, I have not declared file binary with \"binary = true\" in diff driver\n>> definition, isn't it?\n> \n> I don't think binary has anything to do with it. A textconv filter takes\n> input (binary or not) and delivers a normalized representation to feed\n> to the diff algorithm. There's no further post-processing, and it's the\n> responsibility of the filter to deliver the bytes it wants diffed.\n> \n> Like I said, I could see an argument for treating the filter output as\n> text to be pre-processed, but that's not how it works (and I don't think\n> it is a good idea to change it now, unless by adding an option to the\n> diff filter).\n\nI think that actually there is something wrong.\n\nIf textconv really gets normalized representation of pre-image (the index\nversion) and post-image (the filesystem version), as it should I think,\nboth pre-image lines ('-') and post-image lines ('+') should use CRLF,\nso there should be no warning, i.e. ^M\n\nOr textconv filter gets normalized representation (it looks this way\nwhen examining diff result saved to file with `git diff test.tex >test.diff`;\nI were unable to use `tr '\\r' 'Q', either I got \"fatal: bad config line\"\nfrom Git, or \"tr: extra operand\" from tr), and somehow Git mistakes\nwhat is happening and writes those ^M.\n\nIf I understand it correctly, if pre-image, post-image and context\nall use the same eol, there should be no warning, isn't it?\n\n> \n>> P.S. What do you think about Git supporting 'encoding' attribute (or\n>> 'core.encoding' config) plus 'core.outputEncoding' in-core?\n> \n> Supporting an \"encoding\" attribute to normalize file encodings in diffs\n> seems reasonable to me. But it would have to be enabled only for\n> human-readable diffs, as the result could not be applied (so the same as\n> textconv).\n\nI was thinking about human readable diffs, and 'git show <blob>', same\nas with textconv.\n\n> \n> I don't think core.outputEncoding is necessarily a good idea. We are not\n> really equipped anything that isn't an ascii superset, as we intermingle\n> the bytes with ascii diff headers (though I think that is true of the\n> commitEncoding stuff; I assume everything breaks horribly if you tried\n> to set that to UTF-16, but I've never tried it).\n\nWell, the understanding would be that the same limitation as for \ncore.logOutputEncoding (documented if it isn't) that only encodings that\nare ASCII compatibile are supported.\n \n-- \nJakub Narębski\n\n"}]}