git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH/RFC v5 7/7] Careful with CRLF when using e.g. UTF-16 for working-tree-encoding

From
Torsten Bögershausen <tboegi@web.de>
Date
Feb 7, 2018, 06:31 UTC
Message-ID
<20180207063147.GA22714@tor.lan>
In-Reply-To
<xmqqtvuzcibz.fsf@gitster-ct.c.googlers.com>
On Fri, Feb 02, 2018 at 11:17:04AM -0800, Junio C Hamano wrote:
Show 23 quoted lines
> Torsten Bögershausen <tboegi@web.de> writes:
> 
> > There are 2 opposite opionions/user expectations here:
> >
> > a) They are binary in the working tree, so git should leave the line endings
> >    as is. (Unless specified otherwise in the .attributes file)
> > ...
> > b) They are text files in the index. Git will convert line endings
> >    if core.autocrlf is true (or the .gitattributes file specifies "-text")
> 
> I sense that you seem to be focusing on the distinction between "in
> the working tree" vs "in the index" while contrasting.  The "binary
> vs text" in your "binary in wt, text in index" is based on the
> default heuristics without any input from end-users or the project
> that uses Git that happens to contain such files.  If the users and
> the project that uses Git want to treat contents in a path as text,
> it is text even when it is (re-)encoded to UTF-16, no?
> 
> Such files may be (mis)classified as binary with the default
> heuristics when there is no help from what is written in the
> .gitattributes file, but here we are talking about the case where
> the user explicitly tells us it is in UTF-16, right?  Is there such a
> thing as UTF-16 binary?

I don't think so, by definiton UTF-16 is ment to be text. (this means that git ls-files --eol needs some update, I can have a look)

Do we agree that UTF-16 is text ? If yes, could Git assume that the "text" attribute is set when working-tree-encoding is set ?

I would even go a step further and demand that the user -must- make a decision about the line endings for working-tree-encoded files: working-tree-encoding=UTF-16 # illegal, die() working-tree-encoding=UTF-16 text=auto # illegal, die() working-tree-encoding=UTF-16 -text # no eol conversion working-tree-encoding=UTF-16 text # eol according to core.eol working-tree-encoding=UTF-16 text eol=lf # LF working-tree-encoding=UTF-16 text eol=crlf # CRLF

What do you think ?
Previous: Junio C HamanoNext: Junio C Hamano
Message 41 of 43 in “convert: add support for different encodings”
  1. 0/6 convert: add support for different encodingslars.schneider@autodesk.com, Jan 20, 2018
  2. 1/6 strbuf: remove unnecessary NUL assignment in xstrdup_tolower()lars.schneider@autodesk.com, Jan 20, 2018
  3. 2/6 strbuf: add xstrdup_toupper()lars.schneider@autodesk.com, Jan 20, 2018
  4. 3/6 utf8: add function to detect prohibited UTF-16/32 BOMlars.schneider@autodesk.com, Jan 20, 2018
  5. 4/6 utf8: add function to detect a missing UTF-16/32 BOMlars.schneider@autodesk.com, Jan 20, 2018
  6. 5/6 convert: add 'working-tree-encoding' attributelars.schneider@autodesk.com, Jan 20, 2018
  7. Simon RuderichJan 21, 2018
  8. Lars SchneiderJan 22, 2018
  9. Jeff KingJan 23, 2018
  10. Simon RuderichJan 23, 2018
  11. Jeff KingJan 23, 2018
  12. Junio C HamanoJan 23, 2018
  13. Simon RuderichJan 23, 2018
  14. SQUASH convert: add tracing for 'working-tree-encoding' attributelars.schneider@autodesk.com, Jan 22, 2018
  15. Eric SunshineJan 22, 2018
  16. SQUASH convert: add tracing for 'working-tree-encoding' attributelars.schneider@autodesk.com, Jan 23, 2018
  17. 6/6 convert: add tracing for 'working-tree-encoding' attributelars.schneider@autodesk.com, Jan 20, 2018
  18. Torsten BögershausenJan 23, 2018
  19. Junio C HamanoJan 23, 2018
  20. 0/7 convert: add support for different encodingstboegi@web.de, Jan 29, 2018
  21. 2/7 strbuf: add xstrdup_toupper()tboegi@web.de, Jan 29, 2018
  22. 3/7 utf8: add function to detect prohibited UTF-16/32 BOMtboegi@web.de, Jan 29, 2018
  23. 6/7 convert: add tracing for 'working-tree-encoding' attributetboegi@web.de, Jan 29, 2018
  24. 4/7 utf8: add function to detect a missing UTF-16/32 BOMtboegi@web.de, Jan 29, 2018
  25. Junio C HamanoJan 30, 2018
  26. Lars SchneiderJan 30, 2018
  27. Junio C HamanoJan 30, 2018
  28. 5/7 convert: add 'working-tree-encoding' attributetboegi@web.de, Jan 29, 2018
  29. Junio C HamanoJan 30, 2018
  30. Lars SchneiderJan 30, 2018
  31. Junio C HamanoJan 30, 2018
  32. Lars SchneiderJan 31, 2018
  33. Junio C HamanoJan 31, 2018
  34. 7/7 Careful with CRLF when using e.g. UTF-16 for working-tree-encodingtboegi@web.de, Jan 29, 2018
  35. Lars SchneiderJan 30, 2018
  36. Torsten BögershausenJan 30, 2018
  37. Lars SchneiderJan 30, 2018
  38. Torsten BögershausenJan 31, 2018
  39. Lars SchneiderJan 31, 2018
  40. Junio C HamanoFeb 2, 2018
  41. Torsten BögershausenFeb 7, 2018
  42. Junio C HamanoFeb 7, 2018
  43. 1/7 strbuf: remove unnecessary NUL assignment in xstrdup_tolower()tboegi@web.de, Jan 29, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.