git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[BUG?] iconv used as textconv, and spurious ^M on added lines on Windows

From
Jakub Narębski <jnareb@gmail.com>
Date
Mar 30, 2017, 19:35 UTC
Message-ID
<feaeade7-aeb5-fa67-ab29-9106aeadb2a6@gmail.com>
Hello,

Recently I had to work on a project which uses legacy 8-bit encoding (namely cp1250 encoding) instead of utf-8 for text files (LaTeX documents). My terminal, that is Git Bash from Git for Windows is set up for utf-8.

I wanted for "git diff" and friends to return something sane on said utf-8 terminal, instead of mojibake. There is 'encoding' gitattribute... but it works only for GUI ('git gui', that is).

Therefore I have (ab)used textconv facility to convert from cp1250 of file encoding to utf-8 encoding of console.

I have set the following in .gitattributes file:
  ## LaTeX documents in cp1250 encoding
  *.tex text diff=mylatex
The 'mylatex' driver is defined as:
  [diff "mylatex"]
        xfuncname = "^(\\\\((sub)*section|chapter|part)\\*{0,1}\\{.*)$"
        wordRegex = "\\\\[a-zA-Z]+|[{}]|\\\\.|[^\\{}[:space:]]+"
        textconv  = \"C:/Program Files/Git/usr/bin/iconv.exe\" -f cp1250 -t utf-8
        cachetextconv = true

And everything would be all right... if not the fact that Git appends spurious ^M to added lines in the `git diff` output. Files use CRLF end-of-line convention (the native MS Windows one).

  $ git diff test.tex
  diff --git a/test.tex b/test.tex
  index 029646e..250ab16 100644
  --- a/test.tex
  +++ b/test.tex
  @@ -1,4 +1,4 @@
  -\documentclass{article}
  +\documentclass{mwart}^M
  
   \usepackage[cp1250]{inputenc}
   \usepackage{polski}

What gives? Why there is this ^M tacked on the end of added lines, while it is not present in deleted lines, nor in content lines?

Puzzled.

P.S. Git has `i18n.commitEncoding` and `i18n.logOutputEncoding`; pity that it doesn't supports in core `encoding` attribute together with having `i18n.outputEncoding`. -- Jakub Narębski

Next: Jeff King
Message 1 of 10 in “[BUG?] iconv used as textconv, and spurious ^M on added lines on Windows”
  1. Jakub NarębskiMar 30, 2017
  2. Jeff KingMar 30, 2017
  3. Jakub NarębskiMar 31, 2017
  4. Jeff KingApr 1, 2017
  5. Jakub NarębskiApr 1, 2017
  6. Jeff KingApr 2, 2017
  7. Jakub NarębskiApr 2, 2017
  8. Torsten BögershausenMar 31, 2017
  9. Jakub NarębskiMar 31, 2017
  10. Torsten BögershausenApr 2, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.