git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Suggested clarification for .gitattributes reference documentation

From
MLMichael Litwak <michael.litwak@nuix.com>
Date
Jan 12, 2024, 21:25 UTC
Message-ID
<SJ0PR10MB569379A093B83BE01A04C789FA6F2@SJ0PR10MB5693.namprd10.prod.outlook.com>

The .gitattributes documentation should be clarified to ensure files encoded as UTF-16 are properly accounted for, In particular for Windows users.

Specifically, within the working-tree encoding topic https://git-scm.com/docs/gitattributes#_working_tree_encoding, I suggest the following edits:

NEW BULLETED PARAGRAPH UNDER THE HEADING "Please note that using the working-tree-encoding attribute may have a number of pitfalls:"

    * Git for Windows is not able to access the iconv.exe text conversion program from an ordinary
      Command Prompt.  Be sure to run 'git clone' or 'git add' from a git bash console or a Git
      GUI.
OLD TEXT
    As an example, use the following attributes if your *.ps1 files are UTF-16 encoded with byte order mark (BOM) 
    and you want Git to perform automatic line ending conversion based on your platform.
    
    *.ps1       text working-tree-encoding=UTF-16
    
    Use the following attributes if your *.ps1 files are UTF-16 little endian encoded without BOM
    and you want Git to use Windows line endings in the working directory (use UTF-16LE-BOM instead 
    of UTF-16LE if you want UTF-16 little endian with BOM). Please note, it is highly recommended 
    to explicitly define the line endings with eol if the working-tree-encoding attribute is used 
    to avoid ambiguity.
    
    *.ps1      text working-tree-encoding=UTF-16LE eol=CRLF
    
NEW TEXT (SPECIFYING UTF-16BE EXPLICITLY IN THE FIRST EXAMPLE, AND WITH A NEW SEPARATE EXAMPLE FOR UTF-16LE WITH BOM)
    As an example, use the following attributes if your *.ps1 files are UTF-16 big endian encoded
    with byte order mark (BOM) and you want Git to perform automatic line ending conversion
    based on your platform.
    
    *.ps1       text working-tree-encoding=UTF-16
    
    Use the following attributes if your *.ps1 files are UTF-16 little endian encoded without BOM
    and you want Git to use Windows line endings in the working directory.
    
    *.ps1      text working-tree-encoding=UTF-16LE eol=CRLF
    
    Use the following attributes if your *.ps1 files are UTF-16 little endian encoded with BOM
    and you want Git to use Windows line endings in the working directory.
    
    *.ps1      text working-tree-encoding=UTF-16LE-BOM eol=CRLF
    Please note, it is highly recommended to explicitly define the line endings with eol 
    if the working-tree-encoding attribute is used to avoid ambiguity.
    
    Please note, Git for Windows does not support UTF-16LE encoding when running git
    commands from an ordinary Command Prompt.  Use a git bash console instead.
OLD TEXT:
    
    You can get a list of all available encodings on your platform with the following command:
    
    iconv --list
NEW TEXT:
    
    You can get a list of all available encodings on your platform with the following command:
    
    iconv --list
    
    For Git for Windows users the command, above, is only supported when running in a 'git bash' console.
In the thread "help request: unable to merge UTF-16-LE "text" file" at  https://lore.kernel.org/git/Yl8uiflurfjuLIvD@camp.crustytoothpaste.net/, Brian m. Carlson,  Chris Torek and others describe tips for dealing with improper encoding, such as the following:
    if you have already checked the file in without an appropriate
    working-tree-encoding, you should run `git add --renormalize .` and then
    commit.  You'll need to do that (or merge in a commit that does that) on
    every branch you want to work with.
    > For that to work, it is likely that you'd need to convert not just
    > the tips of two branches getting merged, but also the merge base
    > commit, so that all three trees involved in the 3-way merge are in
    > the same text encoding.
    The old merge-recursive has `-X renormalize` that I believe would
    do this for you. (I see code in merge-ort for this as well, but have no
    handy means to test it myself.)

So a NEW SECTION describing ways to deal with improper text file encoding could be added under the working-tree-encoding topic, specifically a description of what the following two commands can do to remedy improper encoding:

    git add --renormalize
    git merge-recursive -X renormalize
CONCLUSION:
Text files encoded with UTF-16LE with BOM are common in the Windows world, as some versions of Visual Studio will use this as the default encoding for .rc or .mc files.  Solution files, project files and other Visual Studio files can also be in this format.  Other encodings are common, too, e.g. some older versions of PowerShell defaulted to UTF-16BE with BOM for new .ps1 files. Yet users continue to experience encoding errors even when they are using the proper working-tree-encoding in their .gitattributes file.  Part of this is due to the complexity of Git and the number of different platforms it supports.
Ideally Git would automatically detect the most common UTF encodings and treat these files as diffable text files on all platforms -- without the need for entries in .gitattributes.  And it would be great if Git for Windows could handle common UTF text encodings when executed in an ordinary Command Prompt.  Until then, clarifying and enhancing the documentation for .gitattributes could go a long way in making text encoding easier for Git users.  Thanks for considering these revisions.
- Michael
Next: brian m. carlson
Message 1 of 10 in “Suggested clarification for .gitattributes reference documentation”
  1. Michael LitwakJan 12, 2024
  2. brian m. carlsonJan 12, 2024
  3. Michael LitwakJan 12, 2024
  4. Michael LitwakJan 13, 2024
  5. Torsten BögershausenJan 13, 2024
  6. Matthias AßhauerJan 13, 2024
  7. Johannes SchindelinFeb 18, 2024
  8. Michael LitwakJan 16, 2024
  9. brian m. carlsonJan 16, 2024
  10. Torsten BögershausenJan 16, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.