The .gitattributes documentation should be clarified to ensure files encoded as UTF-16 are properly accounted for, In particular for Windows users.
Specifically, within the working-tree encoding topic https://git-scm.com/docs/gitattributes#_working_tree_encoding, I suggest the following edits:
NEW BULLETED PARAGRAPH UNDER THE HEADING "Please note that using the working-tree-encoding attribute may have a number of pitfalls:"
* Git for Windows is not able to access the iconv.exe text conversion program from an ordinary
Command Prompt. Be sure to run 'git clone' or 'git add' from a git bash console or a Git
GUI.OLD TEXT
As an example, use the following attributes if your *.ps1 files are UTF-16 encoded with byte order mark (BOM)
and you want Git to perform automatic line ending conversion based on your platform.
*.ps1 text working-tree-encoding=UTF-16
Use the following attributes if your *.ps1 files are UTF-16 little endian encoded without BOM
and you want Git to use Windows line endings in the working directory (use UTF-16LE-BOM instead
of UTF-16LE if you want UTF-16 little endian with BOM). Please note, it is highly recommended
to explicitly define the line endings with eol if the working-tree-encoding attribute is used
to avoid ambiguity.
*.ps1 text working-tree-encoding=UTF-16LE eol=CRLF
NEW TEXT (SPECIFYING UTF-16BE EXPLICITLY IN THE FIRST EXAMPLE, AND WITH A NEW SEPARATE EXAMPLE FOR UTF-16LE WITH BOM)
As an example, use the following attributes if your *.ps1 files are UTF-16 big endian encoded
with byte order mark (BOM) and you want Git to perform automatic line ending conversion
based on your platform.
*.ps1 text working-tree-encoding=UTF-16
Use the following attributes if your *.ps1 files are UTF-16 little endian encoded without BOM
and you want Git to use Windows line endings in the working directory.
*.ps1 text working-tree-encoding=UTF-16LE eol=CRLF
Use the following attributes if your *.ps1 files are UTF-16 little endian encoded with BOM
and you want Git to use Windows line endings in the working directory.
*.ps1 text working-tree-encoding=UTF-16LE-BOM eol=CRLF Please note, it is highly recommended to explicitly define the line endings with eol
if the working-tree-encoding attribute is used to avoid ambiguity.
Please note, Git for Windows does not support UTF-16LE encoding when running git
commands from an ordinary Command Prompt. Use a git bash console instead.OLD TEXT:
You can get a list of all available encodings on your platform with the following command:
iconv --listNEW TEXT:
You can get a list of all available encodings on your platform with the following command:
iconv --list
For Git for Windows users the command, above, is only supported when running in a 'git bash' console.In the thread "help request: unable to merge UTF-16-LE "text" file" at https://lore.kernel.org/git/Yl8uiflurfjuLIvD@camp.crustytoothpaste.net/, Brian m. Carlson, Chris Torek and others describe tips for dealing with improper encoding, such as the following:
if you have already checked the file in without an appropriate
working-tree-encoding, you should run `git add --renormalize .` and then
commit. You'll need to do that (or merge in a commit that does that) on
every branch you want to work with. > For that to work, it is likely that you'd need to convert not just
> the tips of two branches getting merged, but also the merge base
> commit, so that all three trees involved in the 3-way merge are in
> the same text encoding. The old merge-recursive has `-X renormalize` that I believe would
do this for you. (I see code in merge-ort for this as well, but have no
handy means to test it myself.)So a NEW SECTION describing ways to deal with improper text file encoding could be added under the working-tree-encoding topic, specifically a description of what the following two commands can do to remedy improper encoding:
git add --renormalize
git merge-recursive -X renormalizeCONCLUSION:
Text files encoded with UTF-16LE with BOM are common in the Windows world, as some versions of Visual Studio will use this as the default encoding for .rc or .mc files. Solution files, project files and other Visual Studio files can also be in this format. Other encodings are common, too, e.g. some older versions of PowerShell defaulted to UTF-16BE with BOM for new .ps1 files. Yet users continue to experience encoding errors even when they are using the proper working-tree-encoding in their .gitattributes file. Part of this is due to the complexity of Git and the number of different platforms it supports.
Ideally Git would automatically detect the most common UTF encodings and treat these files as diffable text files on all platforms -- without the need for entries in .gitattributes. And it would be great if Git for Windows could handle common UTF text encodings when executed in an ordinary Command Prompt. Until then, clarifying and enhancing the documentation for .gitattributes could go a long way in making text encoding easier for Git users. Thanks for considering these revisions.
- Michael
