{"thread":{"id":"57765","subject":"help request: unable to merge UTF-16-LE \"text\" file","startedAt":"2022-04-19T19:36:34Z","lastAt":"2022-04-24T01:57:55Z","messageCount":9,"participants":["Kevin Long","brian m. carlson","Erik Cervin Edin","Junio C Hamano","Plato Kiorpelidis","Chris Torek"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"453938","messageId":"CAJCZ4U-9KYGK0jwnL1GyOYW-CMjV_Eo--g85CbexsV=aQnn7aw@mail.gmail.com","threadId":"57765","inReplyTo":null,"subject":"help request: unable to merge UTF-16-LE \"text\" file","fromName":"Kevin Long","fromEmail":"kevinlong206@gmail.com","sentAt":"2022-04-19T19:36:19Z","receivedAt":"2022-04-19T19:36:34Z","isPatch":false,"sender":{"key":"kevinlong206@gmail.com","avatar":null},"body":"Greetings,\n\nI've been struggling to merge branches because of a UTF-16-LE (with BOM?) file.\n\nWindows 11 / git version 2.35.3.windows.1\n\nThe problem file is a .sln file (Visual Studio \"solution\"). Edited in\nboth branches. It is a \"text\" file, but is encoded as such:\n\nFacilityMaster.sln: Unicode text, UTF-16, little-endian text, with\nCRLF line terminators\n\nI have tried several \"working-tree-encoding\" settings in\n.gitattributes in my local working directory, to no avail yet:\n\n*.sln working-tree-encoding=UTF-16-LE eol=CRLF, results in:\nerror: failed to encode 'FacilityMaster.sln' from UTF-16-LE to UTF-8\nwarning: Cannot merge binary files: FacilityMaster.sln (HEAD vs. master)\n\n*.sln working-tree-encoding=UTF-16 eol=CRLF, results in:\nwarning: Cannot merge binary files: FacilityMaster.sln (HEAD vs. master)\n\n*.sln working-tree-encoding=UTF-16-LE-BOM eol=CRLF\nerror: failed to encode 'FacilityMaster.sln' from UTF-16-LE-BOM to UTF-8\nwarning: Cannot merge binary files: FacilityMaster.sln (HEAD vs. master)\n\n\nHoping for some suggestions. I've also tried to save the file as UTF-8\nin both branches, commit, then merge, but still that did not work. I\njust want to merge it like a normal source code file.\n\n\nThank you,\n\nKevin Long\n"},{"id":"453943","messageId":"Yl8uiflurfjuLIvD@camp.crustytoothpaste.net","threadId":"57765","inReplyTo":"CAJCZ4U-9KYGK0jwnL1GyOYW-CMjV_Eo--g85CbexsV=aQnn7aw@mail.gmail.com","subject":"Re: help request: unable to merge UTF-16-LE \"text\" file","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2022-04-19T21:50:01Z","receivedAt":"2022-04-19T21:50:10Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2022-04-19 at 19:36:19, Kevin Long wrote:\n> Greetings,\n> \n> I've been struggling to merge branches because of a UTF-16-LE (with BOM?) file.\n> \n> Windows 11 / git version 2.35.3.windows.1\n> \n> The problem file is a .sln file (Visual Studio \"solution\"). Edited in\n> both branches. It is a \"text\" file, but is encoded as such:\n> \n> FacilityMaster.sln: Unicode text, UTF-16, little-endian text, with\n> CRLF line terminators\n\nGit does not consider files using UTF-16 to be text because they contain\nNUL bytes. In some sense they do represent textual content, but Git\nconsiders them to be binary.\n\n> I have tried several \"working-tree-encoding\" settings in\n> .gitattributes in my local working directory, to no avail yet:\n> \n> *.sln working-tree-encoding=UTF-16-LE eol=CRLF, results in:\n> error: failed to encode 'FacilityMaster.sln' from UTF-16-LE to UTF-8\n> warning: Cannot merge binary files: FacilityMaster.sln (HEAD vs. master)\n> \n> *.sln working-tree-encoding=UTF-16 eol=CRLF, results in:\n> warning: Cannot merge binary files: FacilityMaster.sln (HEAD vs. master)\n> \n> *.sln working-tree-encoding=UTF-16-LE-BOM eol=CRLF\n> error: failed to encode 'FacilityMaster.sln' from UTF-16-LE-BOM to UTF-8\n> warning: Cannot merge binary files: FacilityMaster.sln (HEAD vs. master)\n\nThe proper encoding you want here is \"UTF-16LE-BOM\". Many Windows\nprograms use a non-standard encoding where everything _must_ be both\nlittle-endian and have a BOM. (The standard encoding UTF-16LE must\nalways be little endian but omits the BOM, and UTF-16 could be of either\nendianness, and must only contain a BOM if little endian, but could in\neither case.)\n\nThat will result in the file being stored as UTF-8 in the repository and\nconverted to this non-standard Windows encoding on checkout. However,\nif you have already checked the file in without an appropriate\nworking-tree-encoding, you should run `git add --renormalize .` and then\ncommit. You'll need to do that (or merge in a commit that does that) on\nevery branch you want to work with.\n\n> Hoping for some suggestions. I've also tried to save the file as UTF-8\n> in both branches, commit, then merge, but still that did not work. I\n> just want to merge it like a normal source code file.\n\nHowever, in order for the merge to work, both branches must have the\nfile checked in correctly. That is, both master and the branch from\nwhich you're merging need to have the file as UTF-8 in the repository.\nIf you make the working-tree-encoding changes above (or the switch to\nUTF-8) on only one of those branches, then the other one will still have\nthe binary blob, and merging won't be possible.\n\nIf you can keep it as UTF-8, that's ideal. It should definitely work if\nboth sides have UTF-8 files. If you still see a message about binary\nfiles, then it could well be that something didn't get saved properly as\nUTF-8, or that these really aren't text files and that they contain\nbinary contents.\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"453987","messageId":"CA+JQ7M8+FR+_t+Fafoggz47bT0joAZFEkjhjf72teZgdNEoNZQ@mail.gmail.com","threadId":"57765","inReplyTo":"CAJCZ4U-9KYGK0jwnL1GyOYW-CMjV_Eo--g85CbexsV=aQnn7aw@mail.gmail.com","subject":"Re: help request: unable to merge UTF-16-LE \"text\" file","fromName":"Erik Cervin Edin","fromEmail":"erik@cervined.in","sentAt":"2022-04-20T17:25:42Z","receivedAt":"2022-04-20T17:26:24Z","isPatch":false,"sender":{"key":"erik@cervined.in","avatar":null},"body":"On Wed, Apr 20, 2022 at 6:29 PM Kevin Long wrote:\n>\n> The problem file is a .sln file (Visual Studio \"solution\"). Edited in\n> both branches. It is a \"text\" file, but is encoded as such:\n\nCan you convert it to utf-8 in both branches and then merge?\n iconv.exe -f utf-16le -t utf-8 foo.sln > tmp.sln && mv -f tmp.sln foo.sln\n"},{"id":"453988","messageId":"CAJCZ4U-2mmkEGL98-Q-fizxfz0UEOv6GZJKJvNvzei=9xMja2Q@mail.gmail.com","threadId":"57765","inReplyTo":"CA+JQ7M8+FR+_t+Fafoggz47bT0joAZFEkjhjf72teZgdNEoNZQ@mail.gmail.com","subject":"Re: help request: unable to merge UTF-16-LE \"text\" file","fromName":"Kevin Long","fromEmail":"kevinlong206@gmail.com","sentAt":"2022-04-20T17:38:35Z","receivedAt":"2022-04-20T17:38:49Z","isPatch":false,"sender":{"key":"kevinlong206@gmail.com","avatar":null},"body":"Thanks, I actually did try that without success. I'm wondering if\nthere is an \"ancestor\" object that is still binary or something, deep\nrooted in the tree.\n\nConverting it once and for all to utf-8 would be great, as that will\nwork fine with Visual Studio. Not sure why the original committer\nchose UTF-16.\n\nOn Wed, Apr 20, 2022 at 10:26 AM Erik Cervin Edin wrote:\n>\n> On Wed, Apr 20, 2022 at 6:29 PM Kevin Long wrote:\n> >\n> > The problem file is a .sln file (Visual Studio \"solution\"). Edited in\n> > both branches. It is a \"text\" file, but is encoded as such:\n>\n> Can you convert it to utf-8 in both branches and then merge?\n> iconv.exe -f utf-16le -t utf-8 foo.sln > tmp.sln && mv -f tmp.sln foo.sln\n"},{"id":"453991","messageId":"xmqq8rrzr6dp.fsf@gitster.g","threadId":"57765","inReplyTo":"CA+JQ7M8+FR+_t+Fafoggz47bT0joAZFEkjhjf72teZgdNEoNZQ@mail.gmail.com","subject":"Re: help request: unable to merge UTF-16-LE \"text\" file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-04-20T17:50:42Z","receivedAt":"2022-04-20T17:50:47Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Erik Cervin Edin writes:\n\n> On Wed, Apr 20, 2022 at 6:29 PM Kevin Long wrote:\n>>\n>> The problem file is a .sln file (Visual Studio \"solution\"). Edited in\n>> both branches. It is a \"text\" file, but is encoded as such:\n>\n> Can you convert it to utf-8 in both branches and then merge?\n> iconv.exe -f utf-16le -t utf-8 foo.sln > tmp.sln && mv -f tmp.sln foo.sln\n\nFor that to work, it is likely that you'd need to convert not just\nthe tips of two branches getting merged, but also the merge base\ncommit, so that all three trees involved in the 3-way merge are in\nthe same text encoding.\n"},{"id":"453992","messageId":"CA+JQ7M9o+kZUgLkCq7aauCUwzZy6OzwKpyGQ1DxQyaL2jkJBSg@mail.gmail.com","threadId":"57765","inReplyTo":"xmqq8rrzr6dp.fsf@gitster.g","subject":"Re: help request: unable to merge UTF-16-LE \"text\" file","fromName":"Erik Cervin Edin","fromEmail":"erik@cervined.in","sentAt":"2022-04-20T17:55:39Z","receivedAt":"2022-04-20T17:56:21Z","isPatch":false,"sender":{"key":"erik@cervined.in","avatar":null},"body":"On Wed, Apr 20, 2022 at 7:50 PM Junio C Hamano wrote:\n>\n> For that to work, it is likely that you'd need to convert not just\n> the tips of two branches getting merged, but also the merge base\n> commit, so that all three trees involved in the 3-way merge are in\n> the same text encoding.\n\nHmm.. Right, the common ancestor would still be the old encoding :/\n\nOn Wed, Apr 20, 2022 at 7:38 PM Kevin Long wrote:\n>\n> Thanks, I actually did try that without success. I'm wondering if\n> there is an \"ancestor\" object that is still binary or something, deep\n> rooted in the tree.\n\nYes, as Junio correctly pointed out.\n\n> Converting it once and for all to utf-8 would be great, as that will\n> work fine with Visual Studio. Not sure why the original committer\n> chose UTF-16.\n\nVisual Studio probably saved it as utf-16le with BOM :(\n\n> I have tried several \"working-tree-encoding\" settings in\n> .gitattributes in my local working directory, to no avail yet:\n>\n> *.sln working-tree-encoding=UTF-16-LE eol=CRLF, results in:\n> error: failed to encode 'FacilityMaster.sln' from UTF-16-LE to UTF-8\n> warning: Cannot merge binary files: FacilityMaster.sln (HEAD vs. master)\n\nCould the issue be that it should be UTF-16LE (without hyphen)?\n"},{"id":"453993","messageId":"CA+JQ7M_Q_Qkh=+qziKg379e_xywQn2RBeAPNx29Dw6a5RXkY1A@mail.gmail.com","threadId":"57765","inReplyTo":"CA+JQ7M9o+kZUgLkCq7aauCUwzZy6OzwKpyGQ1DxQyaL2jkJBSg@mail.gmail.com","subject":"Re: help request: unable to merge UTF-16-LE \"text\" file","fromName":"Erik Cervin Edin","fromEmail":"erik@cervined.in","sentAt":"2022-04-20T18:03:15Z","receivedAt":"2022-04-20T18:03:57Z","isPatch":false,"sender":{"key":"erik@cervined.in","avatar":null},"body":"On Wed, Apr 20, 2022 at 6:29 PM Kevin Long wrote:\n>\n> Hoping for some suggestions. I've also tried to save the file as UTF-8\n> in both branches, commit, then merge, but still that did not work. I\n> just want to merge it like a normal source code file.\n\nSolution files tend to be of the \"for machines only\" variety.\nIn my experience, it often ends up being a bunch of csproj GUIDs\nchanging back and forth.\nConsider just taking one of the two and manually fixing the solution file?\n"},{"id":"454269","messageId":"20220422184211.5z67sxrgq2zm3tvd@compass","threadId":"57765","inReplyTo":"CAJCZ4U-9KYGK0jwnL1GyOYW-CMjV_Eo--g85CbexsV=aQnn7aw@mail.gmail.com","subject":"Re: help request: unable to merge UTF-16-LE \"text\" file","fromName":"Plato Kiorpelidis","fromEmail":"kioplato@gmail.com","sentAt":"2022-04-22T18:42:11Z","receivedAt":"2022-04-22T18:48:04Z","isPatch":false,"sender":{"key":"kioplato@gmail.com","avatar":null},"body":"On 22/04/19 12:36PM, Kevin Long wrote:\n> Greetings,\n\nHey Kevin,\n\n> [..]\n\nIt would also help if you create a minimal working example[1] and upload it to a\npopular Git hosting service like Github/Gitlab, so we can try to perform the\nmerge. This way, you will probably get more precise directions.\n\n[1]: https://en.wikipedia.org/wiki/Minimal_reproducible_example\n\nThanks,\nPlato\n"},{"id":"454351","messageId":"CAPx1GvdhWR2ZxcN1xsJSoX0BOqs0yp_+QXJ8+Y4asFY+2jrakw@mail.gmail.com","threadId":"57765","inReplyTo":"xmqq8rrzr6dp.fsf@gitster.g","subject":"Re: help request: unable to merge UTF-16-LE \"text\" file","fromName":"Chris Torek","fromEmail":"chris.torek@gmail.com","sentAt":"2022-04-24T01:57:37Z","receivedAt":"2022-04-24T01:57:55Z","isPatch":false,"sender":{"key":"chris.torek@gmail.com","avatar":"https://avatars.githubusercontent.com/u/16826774?v=4"},"body":"On Fri, Apr 22, 2022 at 10:27 AM Junio C Hamano wrote:\n> For that to work, it is likely that you'd need to convert not just\n> the tips of two branches getting merged, but also the merge base\n> commit, so that all three trees involved in the 3-way merge are in\n> the same text encoding.\n\nThe old merge-recursive has `-X renormalize` that I believe would\ndo this for you. (I see code in merge-ort for this as well, but have no\nhandy means to test it myself.)\n\nChris\n"}]}