{"thread":{"id":"51434","subject":"Handling text files encoded in little-endian UTF-16 with BOM","startedAt":"2019-07-05T11:35:42Z","lastAt":"2019-07-05T18:52:39Z","messageCount":3,"participants":["Mateusz Loskot","Torsten Bögershausen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"378623","messageId":"CABUeae_90xG7avh1yt2_X78pmpYUvLtXAZ2Kd9A=FpcEayvi-Q@mail.gmail.com","threadId":"51434","inReplyTo":null,"subject":"Handling text files encoded in little-endian UTF-16 with BOM","fromName":"Mateusz Loskot","fromEmail":"mateusz@loskot.net","sentAt":"2019-07-05T11:35:13Z","receivedAt":"2019-07-05T11:35:42Z","isPatch":false,"sender":{"key":"mateusz@loskot.net","avatar":"https://gravatar.com/avatar/fe5a0ce8995d4986f5c46aad6353e21717808da58e4bc710da1a100e247be239?d=mp&s=160"},"body":"Hi,\n\nUsing git version 2.22.0.windows.1\n\nI have a repository with number of .txt files encoded in\nlittle-endian UTF-16 with BOM.\n\nWhat are the best practice and recommended configuration to\nmanage such files with Git to avoid unexpected re-encoding to\nUTF-8 or others?\n\nCurrently, there is .gitattriuts with entries like\n\n   resource/*.txt   working-tree-encoding=UTF-16LE-BOM -text\n\nDespite that some of team members have noticed that the files\noccacionally get re-encoded to UTF-8. It is unknow what are\nactual steps leading to that. BTW, there a few Git clients\nin use: git in Git Bash, VSCode, Fork.\n\nWhat bothers me in the .gitattributes is this `-text` attribute.\n\nIs the use of `working-tree-encoding` and `-text` together a\nvalid combination at all?\n\nThe documentation at https://git-scm.com/docs/gitattributes\ndoes not seem to touch on that.\n\nI'll appreciate any suggestions on those UTF-16LE-BOM files.\n\nBest regards,\n-- \nMateusz Loskot, http://mateusz.loskot.net\n"},{"id":"378628","messageId":"20190705162544.gckigv4cbr4k6zpz@tb-raspi4","threadId":"51434","inReplyTo":"CABUeae_90xG7avh1yt2_X78pmpYUvLtXAZ2Kd9A=FpcEayvi-Q@mail.gmail.com","subject":"Re: Handling text files encoded in little-endian UTF-16 with BOM","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2019-07-05T16:25:44Z","receivedAt":"2019-07-05T16:25:48Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Fri, Jul 05, 2019 at 01:35:13PM +0200, Mateusz Loskot wrote:\n> Hi,\n>\n> Using git version 2.22.0.windows.1\n>\n> I have a repository with number of .txt files encoded in\n> little-endian UTF-16 with BOM.\n>\n> What are the best practice and recommended configuration to\n> manage such files with Git to avoid unexpected re-encoding to\n> UTF-8 or others?\n>\n> Currently, there is .gitattriuts with entries like\n>\n>    resource/*.txt   working-tree-encoding=UTF-16LE-BOM -text\n>\n> Despite that some of team members have noticed that the files\n> occacionally get re-encoded to UTF-8. It is unknow what are\n> actual steps leading to that. BTW, there a few Git clients\n> in use: git in Git Bash, VSCode, Fork.\n\nIf possible, I don't want to comment on this kind of\n\"sometimes something happens something on someones computer\" thing.\nA little bit more information could be helpful.\n\n>\n> What bothers me in the .gitattributes is this `-text` attribute.\n>\n> Is the use of `working-tree-encoding` and `-text` together a\n> valid combination at all?\n\nYes, it means that the content re-encoded between the repo and the working tree,\n(that is what you want)\nAnd the \"-text\" means \"leave the line endings\" (LF or CRLF) as is, don't convert them.\n\nIn that sense you can call that a legal combination, but may be not a recommended one.\n\n>\n> The documentation at https://git-scm.com/docs/gitattributes\n> does not seem to touch on that.\n>\n> I'll appreciate any suggestions on those UTF-16LE-BOM files.\n>\n\nMy suggestion would be to use the \"text\" attribute:\n  resource/*.txt   working-tree-encoding=UTF-16LE-BOM text\n\nAnd depending on your application: Do the resource files need a special line ending ?\nThe use either\n  resource/*.txt   working-tree-encoding=UTF-16LE-BOM text eol=LF\nor\n  resource/*.txt   working-tree-encoding=UTF-16LE-BOM text eol=CRLF\n\nI hope that helps a little bit.\n\n> Best regards,\n> --\n> Mateusz Loskot, http://mateusz.loskot.net\n"},{"id":"378649","messageId":"CABUeae-kmc7WV1iv5e1wY+BPMrpXpAoJY0H3B-iQ9D2yyg_hMg@mail.gmail.com","threadId":"51434","inReplyTo":"20190705162544.gckigv4cbr4k6zpz@tb-raspi4","subject":"Re: Handling text files encoded in little-endian UTF-16 with BOM","fromName":"Mateusz Loskot","fromEmail":"mateusz@loskot.net","sentAt":"2019-07-05T18:52:09Z","receivedAt":"2019-07-05T18:52:39Z","isPatch":false,"sender":{"key":"mateusz@loskot.net","avatar":"https://gravatar.com/avatar/fe5a0ce8995d4986f5c46aad6353e21717808da58e4bc710da1a100e247be239?d=mp&s=160"},"body":"On Fri, 5 Jul 2019 at 18:25, Torsten Bögershausen <tboegi@web.de> wrote:\n> On Fri, Jul 05, 2019 at 01:35:13PM +0200, Mateusz Loskot wrote:\n> >\n> > Using git version 2.22.0.windows.1\n> >\n> > I have a repository with number of .txt files encoded in\n> > little-endian UTF-16 with BOM.\n> >\n> > What are the best practice and recommended configuration to\n> > manage such files with Git to avoid unexpected re-encoding to\n> > UTF-8 or others?\n> >\n> > Currently, there is .gitattriuts with entries like\n> >\n> >    resource/*.txt   working-tree-encoding=UTF-16LE-BOM -text\n> >\n> > Despite that some of team members have noticed that the files\n> > occacionally get re-encoded to UTF-8. It is unknow what are\n> > actual steps leading to that. BTW, there a few Git clients\n> > in use: git in Git Bash, VSCode, Fork.\n>\n> If possible, I don't want to comment on this kind of\n> \"sometimes something happens something on someones computer\" thing.\n\nPerfectly understood.\n\n> A little bit more information could be helpful.\n\nIf there was more, I'd have provided.\n\n> > What bothers me in the .gitattributes is this `-text` attribute.\n> >\n> > Is the use of `working-tree-encoding` and `-text` together a\n> > valid combination at all?\n>\n> Yes, it means that the content re-encoded between the repo and the working tree,\n> (that is what you want)\n> And the \"-text\" means \"leave the line endings\" (LF or CRLF) as is, don't convert them.\n\nThat's quite a useful insight. I understood \"-text\" means content is\nnot a text, but binary.\n\n> In that sense you can call that a legal combination, but may be not a recommended one.\n\nRight.\n\n> > The documentation at https://git-scm.com/docs/gitattributes\n> > does not seem to touch on that.\n> >\n> > I'll appreciate any suggestions on those UTF-16LE-BOM files.\n> >\n>\n> My suggestion would be to use the \"text\" attribute:\n>   resource/*.txt   working-tree-encoding=UTF-16LE-BOM text\n>\n> And depending on your application: Do the resource files need a special line ending ?\n\nI need CRLF.\n\n> The use either\n>   resource/*.txt   working-tree-encoding=UTF-16LE-BOM text eol=LF\n> or\n>   resource/*.txt   working-tree-encoding=UTF-16LE-BOM text eol=CRLF\n\nThis is very helpful. Thanks!\n\n-- \nMateusz Loskot, http://mateusz.loskot.net\n"}]}