{"thread":{"id":"33463","subject":"Re: Script for handling UTF-16 files","startedAt":"2013-04-10T18:59:56Z","lastAt":"2013-04-11T19:11:25Z","messageCount":2,"participants":["Karsten Blees","Ken Ismert"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"213804","messageId":"5165B6AC.5090403@gmail.com","threadId":"33463","inReplyTo":"608a349b-cc71-4cba-9197-3783049e9f47@googlegroups.com","subject":"Re: Script for handling UTF-16 files","fromName":"Karsten Blees","fromEmail":"karsten.blees@gmail.com","sentAt":"2013-04-10T18:59:56Z","receivedAt":"2013-04-10T18:59:56Z","isPatch":false,"sender":{"key":"karsten.blees@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1111200?v=4"},"body":"Am 10.04.2013 01:47, schrieb Ken Ismert:\n> \n> I bumped into the UTF-16 display problem with Git Extensions running on top of msysGit. After lots of searching and experimenting, I came up with a solution that works for me.\n> \n> Note: Please see questions below.\n> \n> This method is for MSysGit 1.8.1, and is tested on Windows XP. I use Git Extensions 2.44, but since the changes are at the Git level, they should work for Git Gui as well. Steps:\n\nThere has been a discussion about handling UTF-16 on the git ML a while back, see http://thread.gmane.org/gmane.comp.version-control.git/159708\n\nAs suggested there, I would try to use a clean/smudge filter (i.e. store UTF-16 files as UTF-8 in the repository and convert back to UTF-16 on checkout). That way git can treat your UTF-16 files as text in most cases (i.e. you can merge them, git-grep works, gitattributes work (eol-conversion, ident-replacement, built-in diff patterns...)).\n\nIf you use a textconv filter, UTF-16 content will be treated as binary by most git operations.\n\nThere's also an 'encoding' attribute and a 'gui.encoding' setting which in theory should solve your issue (i.e. specify encoding of files for display by GUI tools). I don't know if Git Extensions supports that, or whether its supposed to work for binary files at all.\n\n> 3) Modify the global ~/Git/etc/gitconfig or your local ~/.git/config file, and add these lines:\n> \n>     [diff \"astextutf16\"]\n>         textconv = astextutf16\n\nWhy not simply \"textconv = iconv -f utf-16 -t utf-8\", without the extra script?\n\n> c) I had success with iconv, but is there any built-in UTF-16 to UTF-8 converter that ships with msysGit?\n\nThere are ready-to-use UTF-conversion functions in the codebase, but these are not accessible as a git command or built-in filter.\n\n> As a quick fix, how hard would it be to add a 'utf16' diff filter, similar to cpp or |csharp? Or is this simply the wrong place to put in a work-around?\n\nAs described above, I think a diff filter is not the right tool for the job. The only universal format for text content that works reasonably well with established text-based technologies (merge algorithms, regex etc.) is UTF-8. If we want to benefit from these technologies, git should store text files as UTF-8 and convert from / to platform-specific formats on checkin / checkout or for display.\n\nBye,\nKarsten\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/groups/opt_out.\n"},{"id":"213993","messageId":"62acca4c-9328-4c38-bf35-6db55d45c895@googlegroups.com","threadId":"33463","inReplyTo":"5165B6AC.5090403@gmail.com","subject":"Re: Script for handling UTF-16 files","fromName":"Ken Ismert","fromEmail":"kismert@gmail.com","sentAt":"2013-04-11T19:11:25Z","receivedAt":"2013-04-11T19:11:25Z","isPatch":false,"sender":{"key":"kismert@gmail.com","avatar":null},"body":"\nKarsten, \n\nThere's also an 'encoding' attribute and a 'gui.encoding' setting which in \n> theory should solve your issue (i.e. specify encoding of files for display \n> by GUI tools). I don't know if Git Extensions supports that, or whether its \n> supposed to work for binary files at all. \n>\n\nI played around with this, and neither 'encoding' or 'gui.encoding' solves \nmy problem in Get Extensions or Git Gui.\n\nAccording to the man page, I would put this entry in my .gitattributes:\n*.txt encoding=utf-8\n\nThis sounds rather tautological -- \"Why yes, I do want to display my UTF-16 \nfile as text!\"\n\nMore to the point, since Git *doesn't know* the original encoding of a \n'non-standard' Unicode file, how can it possibly know how to convert it to \nUTF-8?\n\nSo, 'encoding' and 'gui.encoding' seem to be poorly conceived, and \ncertainly aren't helpful for the Windows UTF-16 problem.\n\n-Ken\n\n-- \n-- \n*** Please reply-to-all at all times ***\n*** (do not pretend to know who is subscribed and who is not) ***\n*** Please avoid top-posting. ***\nThe msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.\n\nYou received this message because you are subscribed to the Google\nGroups \"msysGit\" group.\nTo post to this group, send email to msysgit@googlegroups.com\nTo unsubscribe from this group, send email to\nmsysgit+unsubscribe@googlegroups.com\nFor more options, and view previous threads, visit this group at\nhttp://groups.google.com/group/msysgit?hl=en_US?hl=en\n\n--- \nYou received this message because you are subscribed to the Google Groups \"msysGit\" group.\nTo unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.\nFor more options, visit https://groups.google.com/groups/opt_out.\n\n\n"}]}