git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH/RFC v5 7/7] Careful with CRLF when using e.g. UTF-16 for working-tree-encoding

From
Lars Schneider <larsxschneider@gmail.com>
Date
Jan 30, 2018, 15:14 UTC
Message-ID
<10091BA4-1069-4A65-9057-CAAD87F9B55F@gmail.com>
In-Reply-To
<20180130144002.GA30211@tor.lan>
Show 58 quoted lines
> On 30 Jan 2018, at 15:40, Torsten Bögershausen <tboegi@web.de> wrote:
> 
> On Tue, Jan 30, 2018 at 12:23:47PM +0100, Lars Schneider wrote:
>> 
>>> On 29 Jan 2018, at 21:19, tboegi@web.de wrote:
>>> 
>>> From: Torsten Bögershausen <tboegi@web.de>
>>> 
>>> UTF-16 encoded files are treated as "binary" by Git, and no CRLF
>>> conversion is done.
>>> When the UTF-16 encoded files are converted into UF-8 using the new
>> s/UF-8/UTF-8/
>> 
>> 
>>> "working-tree-encoding", the CRLF are converted if core.autocrlf is true.
>>> 
>>> This may lead to confusion:
>>> A tool writes an UTF-16 encoded file with CRLF.
>>> The file is commited with core.autocrlf=true, the CLRF are converted into LF.
>>> The repo is pushed somewhere and cloned by a different user, who has
>>> decided to use core.autocrlf=false.
>>> He uses the same tool, and now the CRLF are not there as expected, but LF,
>>> make the file useless for the tool.
>>> 
>>> Avoid this (possible) confusion by ignoring core.autocrlf for all files
>>> which have "working-tree-encoding" defined.
>> 
>> Maybe I don't understand your use case but I think this will generate even 
>> more confusion because that's not what I would expect as a user. I think Git 
>> should behave consistently independent of the used encoding. Here are my arguments:
> 
> To start with: I have probably seen too many repos with CRLF messed up.
> 
>> 
>>  (1) Legacy users are *not* affected. If you don't use the "working-tree-encoding"
>>      attribute then nothing changes for you.
> 
> People who don't use "working-tree-encoding" are not affected,
> I never ment to state that.
> 
> I am thinking about people who use "working-tree-encoding" without thinking
> about line endings.
> Or the ones that have in mind that core.autocrlf=true will leave the
> line endings for UTF-16 encoded files as is, but that changes as soon as they
> are converted into UTF-8 and the "auto" check is now done
> -after- the conversion. I would find that confusing.
> 
>> 
>>  (2) If you use the "working-tree-encoding" attribute *and* you want to ensure 
>>      your file keeps CRLF then you can define that in the attributes too. E.g.:
>> 
>>      *.proj textworking-tree-encoding=UTF-16 eol=crlf
> 
> That is a good one.
> If you ever plan a re-roll (I don't at the moment) the *.proj extemsion
> make much more sense in Documentation/gitattributes that *.tx
> There no text files encoded in UTF-16 wich are called xxx.txt, but those
> are non-ideal examples. *.proj makes good sense as an example.

OK, I'll do that. Would that fix the problem which this patch tries to address for you? (I would also explicitly add a paragraph to discuss line endings)

- Lars
Previous: Torsten BögershausenNext: Torsten Bögershausen
Message 37 of 43 in “convert: add support for different encodings”
  1. 0/6 convert: add support for different encodingslars.schneider@autodesk.com, Jan 20, 2018
  2. 1/6 strbuf: remove unnecessary NUL assignment in xstrdup_tolower()lars.schneider@autodesk.com, Jan 20, 2018
  3. 2/6 strbuf: add xstrdup_toupper()lars.schneider@autodesk.com, Jan 20, 2018
  4. 3/6 utf8: add function to detect prohibited UTF-16/32 BOMlars.schneider@autodesk.com, Jan 20, 2018
  5. 4/6 utf8: add function to detect a missing UTF-16/32 BOMlars.schneider@autodesk.com, Jan 20, 2018
  6. 5/6 convert: add 'working-tree-encoding' attributelars.schneider@autodesk.com, Jan 20, 2018
  7. Simon RuderichJan 21, 2018
  8. Lars SchneiderJan 22, 2018
  9. Jeff KingJan 23, 2018
  10. Simon RuderichJan 23, 2018
  11. Jeff KingJan 23, 2018
  12. Junio C HamanoJan 23, 2018
  13. Simon RuderichJan 23, 2018
  14. SQUASH convert: add tracing for 'working-tree-encoding' attributelars.schneider@autodesk.com, Jan 22, 2018
  15. Eric SunshineJan 22, 2018
  16. SQUASH convert: add tracing for 'working-tree-encoding' attributelars.schneider@autodesk.com, Jan 23, 2018
  17. 6/6 convert: add tracing for 'working-tree-encoding' attributelars.schneider@autodesk.com, Jan 20, 2018
  18. Torsten BögershausenJan 23, 2018
  19. Junio C HamanoJan 23, 2018
  20. 0/7 convert: add support for different encodingstboegi@web.de, Jan 29, 2018
  21. 2/7 strbuf: add xstrdup_toupper()tboegi@web.de, Jan 29, 2018
  22. 3/7 utf8: add function to detect prohibited UTF-16/32 BOMtboegi@web.de, Jan 29, 2018
  23. 6/7 convert: add tracing for 'working-tree-encoding' attributetboegi@web.de, Jan 29, 2018
  24. 4/7 utf8: add function to detect a missing UTF-16/32 BOMtboegi@web.de, Jan 29, 2018
  25. Junio C HamanoJan 30, 2018
  26. Lars SchneiderJan 30, 2018
  27. Junio C HamanoJan 30, 2018
  28. 5/7 convert: add 'working-tree-encoding' attributetboegi@web.de, Jan 29, 2018
  29. Junio C HamanoJan 30, 2018
  30. Lars SchneiderJan 30, 2018
  31. Junio C HamanoJan 30, 2018
  32. Lars SchneiderJan 31, 2018
  33. Junio C HamanoJan 31, 2018
  34. 7/7 Careful with CRLF when using e.g. UTF-16 for working-tree-encodingtboegi@web.de, Jan 29, 2018
  35. Lars SchneiderJan 30, 2018
  36. Torsten BögershausenJan 30, 2018
  37. Lars SchneiderJan 30, 2018
  38. Torsten BögershausenJan 31, 2018
  39. Lars SchneiderJan 31, 2018
  40. Junio C HamanoFeb 2, 2018
  41. Torsten BögershausenFeb 7, 2018
  42. Junio C HamanoFeb 7, 2018
  43. 1/7 strbuf: remove unnecessary NUL assignment in xstrdup_tolower()tboegi@web.de, Jan 29, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.