# Handling text files encoded in little-endian UTF-16 with BOM 3 messages from 2019-07-05 to 2019-07-05. Participants: Mateusz Loskot, Torsten Bögershausen. Thread: https://gitlist.dev/t/51434 ## Mateusz Loskot, 2019-07-05 11:35 Subject: Handling text files encoded in little-endian UTF-16 with BOM Message-ID: URL: https://gitlist.dev/e/CABUeae_90xG7avh1yt2_X78pmpYUvLtXAZ2Kd9A%3DFpcEayvi-Q%40mail.gmail.com ``` Hi, Using git version 2.22.0.windows.1 I have a repository with number of .txt files encoded in little-endian UTF-16 with BOM. What are the best practice and recommended configuration to manage such files with Git to avoid unexpected re-encoding to UTF-8 or others? Currently, there is .gitattriuts with entries like resource/*.txt working-tree-encoding=UTF-16LE-BOM -text Despite that some of team members have noticed that the files occacionally get re-encoded to UTF-8. It is unknow what are actual steps leading to that. BTW, there a few Git clients in use: git in Git Bash, VSCode, Fork. What bothers me in the .gitattributes is this `-text` attribute. Is the use of `working-tree-encoding` and `-text` together a valid combination at all? The documentation at https://git-scm.com/docs/gitattributes does not seem to touch on that. I'll appreciate any suggestions on those UTF-16LE-BOM files. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net ``` ## Torsten Bögershausen, 2019-07-05 16:25 Subject: Re: Handling text files encoded in little-endian UTF-16 with BOM Message-ID: <20190705162544.gckigv4cbr4k6zpz@tb-raspi4> URL: https://gitlist.dev/e/20190705162544.gckigv4cbr4k6zpz%40tb-raspi4 In-Reply-To: ``` On Fri, Jul 05, 2019 at 01:35:13PM +0200, Mateusz Loskot wrote: > Hi, > > Using git version 2.22.0.windows.1 > > I have a repository with number of .txt files encoded in > little-endian UTF-16 with BOM. > > What are the best practice and recommended configuration to > manage such files with Git to avoid unexpected re-encoding to > UTF-8 or others? > > Currently, there is .gitattriuts with entries like > > resource/*.txt working-tree-encoding=UTF-16LE-BOM -text > > Despite that some of team members have noticed that the files > occacionally get re-encoded to UTF-8. It is unknow what are > actual steps leading to that. BTW, there a few Git clients > in use: git in Git Bash, VSCode, Fork. If possible, I don't want to comment on this kind of "sometimes something happens something on someones computer" thing. A little bit more information could be helpful. > > What bothers me in the .gitattributes is this `-text` attribute. > > Is the use of `working-tree-encoding` and `-text` together a > valid combination at all? Yes, it means that the content re-encoded between the repo and the working tree, (that is what you want) And the "-text" means "leave the line endings" (LF or CRLF) as is, don't convert them. In that sense you can call that a legal combination, but may be not a recommended one. > > The documentation at https://git-scm.com/docs/gitattributes > does not seem to touch on that. > > I'll appreciate any suggestions on those UTF-16LE-BOM files. > My suggestion would be to use the "text" attribute: resource/*.txt working-tree-encoding=UTF-16LE-BOM text And depending on your application: Do the resource files need a special line ending ? The use either resource/*.txt working-tree-encoding=UTF-16LE-BOM text eol=LF or resource/*.txt working-tree-encoding=UTF-16LE-BOM text eol=CRLF I hope that helps a little bit. > Best regards, > -- > Mateusz Loskot, http://mateusz.loskot.net ``` ## Mateusz Loskot, 2019-07-05 18:52 Subject: Re: Handling text files encoded in little-endian UTF-16 with BOM Message-ID: URL: https://gitlist.dev/e/CABUeae-kmc7WV1iv5e1wY%2BBPMrpXpAoJY0H3B-iQ9D2yyg_hMg%40mail.gmail.com In-Reply-To: <20190705162544.gckigv4cbr4k6zpz@tb-raspi4> ``` On Fri, 5 Jul 2019 at 18:25, Torsten Bögershausen wrote: > On Fri, Jul 05, 2019 at 01:35:13PM +0200, Mateusz Loskot wrote: > > > > Using git version 2.22.0.windows.1 > > > > I have a repository with number of .txt files encoded in > > little-endian UTF-16 with BOM. > > > > What are the best practice and recommended configuration to > > manage such files with Git to avoid unexpected re-encoding to > > UTF-8 or others? > > > > Currently, there is .gitattriuts with entries like > > > > resource/*.txt working-tree-encoding=UTF-16LE-BOM -text > > > > Despite that some of team members have noticed that the files > > occacionally get re-encoded to UTF-8. It is unknow what are > > actual steps leading to that. BTW, there a few Git clients > > in use: git in Git Bash, VSCode, Fork. > > If possible, I don't want to comment on this kind of > "sometimes something happens something on someones computer" thing. Perfectly understood. > A little bit more information could be helpful. If there was more, I'd have provided. > > What bothers me in the .gitattributes is this `-text` attribute. > > > > Is the use of `working-tree-encoding` and `-text` together a > > valid combination at all? > > Yes, it means that the content re-encoded between the repo and the working tree, > (that is what you want) > And the "-text" means "leave the line endings" (LF or CRLF) as is, don't convert them. That's quite a useful insight. I understood "-text" means content is not a text, but binary. > In that sense you can call that a legal combination, but may be not a recommended one. Right. > > The documentation at https://git-scm.com/docs/gitattributes > > does not seem to touch on that. > > > > I'll appreciate any suggestions on those UTF-16LE-BOM files. > > > > My suggestion would be to use the "text" attribute: > resource/*.txt working-tree-encoding=UTF-16LE-BOM text > > And depending on your application: Do the resource files need a special line ending ? I need CRLF. > The use either > resource/*.txt working-tree-encoding=UTF-16LE-BOM text eol=LF > or > resource/*.txt working-tree-encoding=UTF-16LE-BOM text eol=CRLF This is very helpful. Thanks! -- Mateusz Loskot, http://mateusz.loskot.net ```