{"thread":{"id":"25524","subject":"[RFC] Print diffs of UTF-16 to console / patches to email as UTF-8...?","startedAt":"2010-10-22T16:06:48Z","lastAt":"2010-10-22T21:49:20Z","messageCount":15,"participants":["Drew Northup","Jonathan Nieder","Jakub Narebski","Joshua Juran","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"154165","messageId":"1287763608.31218.63.camel@drew-northup.unet.maine.edu","threadId":"25524","inReplyTo":null,"subject":"[RFC] Print diffs of UTF-16 to console / patches to email as UTF-8...?","fromName":"Drew Northup","fromEmail":"drew.northup@maine.edu","sentAt":"2010-10-22T16:06:48Z","receivedAt":"2010-10-22T16:06:48Z","isPatch":false,"sender":{"key":"drew.northup@maine.edu","avatar":"https://avatars.githubusercontent.com/u/18331571?v=4"},"body":"I am currently thinking about what the best way to preset readable (and\nsafely email-able) patches to the user may be when the content is\nUTF-16. This is part of my ongoing work to treat UTF-16 as text (in\nother words, the crlf options will work and .gitattributes hacks won't\nbe required to display diffs, etc).\nI was also concerned that the result be re-importable to valid UTF-16 in\nthe end. This has led me to consider printing diffs as UTF-8 (no data\nloss, at least 16->8) when the source text is UTF-16. This should also\nbe git-gui / gitk friendly (in theory). I would favorably consider this\nas a configurable option (export_unicode_diff_as_utf8 ?) leaving plain\nUTF-16 output as the standard output from \"git diff\" (once I convince it\nthat UTF-16 is indeed text).\nAlso, there is the issue of being able to recognize UTF-16 as UTF-16 in\ndiffs/patches. Is there a precedent/standard I should be aware of with\nrespect to BOMs and patches? I would think that adhering to the UTF-16\nstandard with respect to whole text files would make sense here (no BOM\n== Big Endian, BOM used to match LE/BE otherwise).\n\nComments welcome!\n\n-- \n-Drew Northup N1XIM\n AKA RvnPhnx on OPN\n________________________________________________\n\"As opposed to vegetable or mineral error?\"\n-John Pescatore, SANS NewsBites Vol. 12 Num. 59\n"},{"id":"154166","messageId":"20101022161851.GH9224@burratino","threadId":"25524","inReplyTo":"1287763608.31218.63.camel@drew-northup.unet.maine.edu","subject":"Re: [RFC] Print diffs of UTF-16 to console / patches to email as UTF-8...?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-10-22T16:18:51Z","receivedAt":"2010-10-22T16:18:51Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nDrew Northup wrote:\n\n> This is part of my ongoing work to treat UTF-16 as text (in\n> other words, the crlf options will work and .gitattributes hacks won't\n> be required to display diffs, etc).\n\nWhat's wrong with .gitattributes for this use case? I would think a\nclean/smudge filter would produce very good behavior from most git\ncommands.\n\nIf speed is the issue, maybe a built-in clean/smudge filter would\naddress that?\n"},{"id":"154170","messageId":"1287766916.31218.71.camel@drew-northup.unet.maine.edu","threadId":"25524","inReplyTo":"20101022161851.GH9224@burratino","subject":"Re: [RFC] Print diffs of UTF-16 to console / patches to email as UTF-8...?","fromName":"Drew Northup","fromEmail":"drew.northup@maine.edu","sentAt":"2010-10-22T17:01:56Z","receivedAt":"2010-10-22T17:01:56Z","isPatch":false,"sender":{"key":"drew.northup@maine.edu","avatar":"https://avatars.githubusercontent.com/u/18331571?v=4"},"body":"\nOn Fri, 2010-10-22 at 11:18 -0500, Jonathan Nieder wrote:\n> Hi,\n> \n> Drew Northup wrote:\n> \n> > This is part of my ongoing work to treat UTF-16 as text (in\n> > other words, the crlf options will work and .gitattributes hacks won't\n> > be required to display diffs, etc).\n> \n> What's wrong with .gitattributes for this use case? I would think a\n> clean/smudge filter would produce very good behavior from most git\n> commands.\n> \n> If speed is the issue, maybe a built-in clean/smudge filter would\n> address that?\n\nThat still doesn't fix the crlf issue, for starters. Also, I would like\nto be able to email patches for files that are in UTF-16 and properly\nre-import them. Unless I'm missing something really big there's not much\nthat a display filter is going to do for me there. I want this to be\ntwo-way.\nTo be honest, the not needing of a display filter goes along with\ntreating UTF-16 patches as text. The same sort of code will be required\nfor both, so why not get predictable behavior for another text encoding\nat the same time?\n\n-- \n-Drew Northup N1XIM\n AKA RvnPhnx on OPN\n________________________________________________\n\"As opposed to vegetable or mineral error?\"\n-John Pescatore, SANS NewsBites Vol. 12 Num. 59\n"},{"id":"154171","messageId":"20101022171248.GA11794@burratino","threadId":"25524","inReplyTo":"1287766916.31218.71.camel@drew-northup.unet.maine.edu","subject":"Re: [RFC] Print diffs of UTF-16 to console / patches to email as UTF-8...?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-10-22T17:12:48Z","receivedAt":"2010-10-22T17:12:48Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi again,\n\nDrew Northup wrote:\n\n> That still doesn't fix the crlf issue, for starters. Also, I would like\n> to be able to email patches for files that are in UTF-16 and properly\n> re-import them. Unless I'm missing something really big there's not much\n> that a display filter is going to do for me there.\n\nRight, I think you're missing something big. textconv is a display\nfilter. clean/smudge convert between internal and external\nrepresentation (and your clean/smudge scripts could take care of CRLF\nthemselves if desired).\n\nThat said, I wouldn't be surprised if clean/smudge filters don't do\neverything you want. If you do go that way, please keep the list\nposted so the mechanism can be improved.\n\nAnd longer term, maybe people will want something tailor-made after\nall? I just imagine it would be more productive to try out the\ngeneric mechanisms first.\n"},{"id":"154172","messageId":"1287768426.31218.75.camel@drew-northup.unet.maine.edu","threadId":"25524","inReplyTo":"20101022171248.GA11794@burratino","subject":"Re: [RFC] Print diffs of UTF-16 to console / patches to email as UTF-8...?","fromName":"Drew Northup","fromEmail":"drew.northup@maine.edu","sentAt":"2010-10-22T17:27:06Z","receivedAt":"2010-10-22T17:27:06Z","isPatch":false,"sender":{"key":"drew.northup@maine.edu","avatar":"https://avatars.githubusercontent.com/u/18331571?v=4"},"body":"\nOn Fri, 2010-10-22 at 12:12 -0500, Jonathan Nieder wrote:\n> Hi again,\n> \n> Drew Northup wrote:\n> \n> > That still doesn't fix the crlf issue, for starters. Also, I would like\n> > to be able to email patches for files that are in UTF-16 and properly\n> > re-import them. Unless I'm missing something really big there's not much\n> > that a display filter is going to do for me there.\n> \n> Right, I think you're missing something big. textconv is a display\n> filter. clean/smudge convert between internal and external\n> representation (and your clean/smudge scripts could take care of CRLF\n> themselves if desired).\n> \n> That said, I wouldn't be surprised if clean/smudge filters don't do\n> everything you want. If you do go that way, please keep the list\n> posted so the mechanism can be improved.\n\nWell I shall plumb the documentation again.... just in case. I'm not\nholding my breath that it will do what I (and frankly a fair number of\nother people) want. We just want version control that treats text like\ntext. FULL STOP. Why isn't UTF-16 text???????\n\n> And longer term, maybe people will want something tailor-made after\n> all? I just imagine it would be more productive to try out the\n> generic mechanisms first.\n\nPlease forgive me for being offended that UTF-16 text is not \"generic\"\nenough.\n\n-- \n-Drew Northup N1XIM\n AKA RvnPhnx on OPN\n________________________________________________\n\"As opposed to vegetable or mineral error?\"\n-John Pescatore, SANS NewsBites Vol. 12 Num. 59\n"},{"id":"154174","messageId":"20101022173055.GA11923@burratino","threadId":"25524","inReplyTo":"1287768426.31218.75.camel@drew-northup.unet.maine.edu","subject":"Re: [RFC] Print diffs of UTF-16 to console / patches to email as UTF-8...?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-10-22T17:30:55Z","receivedAt":"2010-10-22T17:30:55Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Drew Northup wrote:\n\n> Please forgive me for being offended that UTF-16 text is not \"generic\"\n> enough.\n\nFirst some words of explanation.\n\nBy \"generic\" I did not mean ubiquitous, unbranded, popular, or some\nother almost-synonym. What I actually meant is that it is not obvious\nwhat to do with UTF-16. Should it be converted to UTF-8 for output?\nShould it always be normalized when added to the index, so that\nswitching between canonically equivalent sequences does not result\nin spurious diffs? Should the byte-for-byte representation be\nfaithfully preserved, even when it is not valid UTF-16?\n\nWhen in such a situation, often a good approach is the following:\ntake care of mechanism first, then policy. So the first thing to do\nis to make sure that the code is _capable_ of what people are trying\nto do; then one can try various configurations and see what is most\nconvenient; and finally, one can make sure the program behaves in an\nintuitive way by setting a reasonable default.\n\nSo by \"generic\" I meant those mechanisms that can be used in the\ncontext of multiple policies.\n\nApologies; I never meant to offend; please carry on and I will leave\nyou in peace.\n\nJonathan\n"},{"id":"154175","messageId":"m31v7iktkn.fsf@localhost.localdomain","threadId":"25524","inReplyTo":"1287768426.31218.75.camel@drew-northup.unet.maine.edu","subject":"Re: [RFC] Print diffs of UTF-16 to console / patches to email as UTF-8...?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-10-22T17:48:49Z","receivedAt":"2010-10-22T17:48:49Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Drew Northup writes:\n\n> On Fri, 2010-10-22 at 12:12 -0500, Jonathan Nieder wrote:\n> > \n> > Drew Northup wrote:\n> > \n> > > That still doesn't fix the crlf issue, for starters. Also, I would like\n> > > to be able to email patches for files that are in UTF-16 and properly\n> > > re-import them. Unless I'm missing something really big there's not much\n> > > that a display filter is going to do for me there.\n> > \n> > Right, I think you're missing something big. textconv is a display\n> > filter. clean/smudge convert between internal and external\n> > representation (and your clean/smudge scripts could take care of CRLF\n> > themselves if desired).\n> > \n> > That said, I wouldn't be surprised if clean/smudge filters don't do\n> > everything you want. If you do go that way, please keep the list\n> > posted so the mechanism can be improved.\n> \n> Well I shall plumb the documentation again.... just in case. I'm not\n> holding my breath that it will do what I (and frankly a fair number of\n> other people) want. We just want version control that treats text like\n> text. FULL STOP. Why isn't UTF-16 text???????\n\nIf you are asking why Git detects files with text in UTF-16 / USC-2 as\nbinary, it is because Git (re)uses the same heuristic that e.g. GNU\ndiff (and probably also -T file test in Perl), and one of heuristics\nis that if file contains NUL (\"\\0\") character, then it is most\nporbably binary (because legacy C programs for text would have\ntroubles with NUL characters).\n\nThat probably doesn't help you any...\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"154176","messageId":"m3wrpajek6.fsf@localhost.localdomain","threadId":"25524","inReplyTo":"20101022173055.GA11923@burratino","subject":"Re: [RFC] Print diffs of UTF-16 to console / patches to email as UTF-8...?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-10-22T17:58:35Z","receivedAt":"2010-10-22T17:58:35Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Jonathan Nieder writes:\n\n> Drew Northup wrote:\n> \n> > Please forgive me for being offended that UTF-16 text is not \"generic\"\n> > enough.\n> \n> First some words of explanation.\n> \n> By \"generic\" I did not mean ubiquitous, unbranded, popular, or some\n> other almost-synonym. What I actually meant is that it is not obvious\n> what to do with UTF-16. Should it be converted to UTF-8 for output?\n> Should it always be normalized when added to the index, so that\n> switching between canonically equivalent sequences does not result\n> in spurious diffs? Should the byte-for-byte representation be\n> faithfully preserved, even when it is not valid UTF-16?\n> \n> When in such a situation, often a good approach is the following:\n> take care of mechanism first, then policy. So the first thing to do\n> is to make sure that the code is _capable_ of what people are trying\n> to do; then one can try various configurations and see what is most\n> convenient; and finally, one can make sure the program behaves in an\n> intuitive way by setting a reasonable default.\n> \n> So by \"generic\" I meant those mechanisms that can be used in the\n> context of multiple policies.\n\nIt would be nice if there was a way (perhaps stearable via\ngitattributes) to change whether Git is to treat file as sequence of\nbytes (as it is now), or as sequence of characters (probably like \nPerl 6, i.e. as sequence of graphemes), though this would require\nto specify encoding (and normalization) used.\n\nWishful thinking\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"154177","messageId":"1287770805.819.7.camel@drew-northup.unet.maine.edu","threadId":"25524","inReplyTo":"m31v7iktkn.fsf@localhost.localdomain","subject":"Re: [RFC] Print diffs of UTF-16 to console / patches to email as UTF-8...?","fromName":"Drew Northup","fromEmail":"drew.northup@maine.edu","sentAt":"2010-10-22T18:06:45Z","receivedAt":"2010-10-22T18:06:45Z","isPatch":false,"sender":{"key":"drew.northup@maine.edu","avatar":"https://avatars.githubusercontent.com/u/18331571?v=4"},"body":"\nOn Fri, 2010-10-22 at 10:48 -0700, Jakub Narebski wrote:\n> Drew Northup writes:\n\n> > Well I shall plumb the documentation again.... just in case. I'm not\n> > holding my breath that it will do what I (and frankly a fair number of\n> > other people) want. We just want version control that treats text like\n> > text. FULL STOP. Why isn't UTF-16 text???????\n> \n> If you are asking why Git detects files with text in UTF-16 / USC-2 as\n> binary, it is because Git (re)uses the same heuristic that e.g. GNU\n> diff (and probably also -T file test in Perl), and one of heuristics\n> is that if file contains NUL (\"\\0\") character, then it is most\n> porbably binary (because legacy C programs for text would have\n> troubles with NUL characters).\n> \n> That probably doesn't help you any...\n\nI did find that already. I still have not decided that correct place to\nshoehorn in Unicode detection, but I'll be sure to do that before I\nbother anybody else with it. I already wrote code to detect (reasonably)\nvalid UTF-16 (if it isn't obviously valid then I'll just as soon deal\nwith it as binary data, so as to avoid a foot-shooting exercise).\nMy main motivation here has been to get some feedback as I write stuff\nso as to not waste a lot of time during writing something that could be\ndone better. \n(As opposed to not done at all, which is the feeling I'm getting from a\nfew people around here...)\n-- \n-Drew Northup N1XIM\n AKA RvnPhnx on OPN\n________________________________________________\n\"As opposed to vegetable or mineral error?\"\n-John Pescatore, SANS NewsBites Vol. 12 Num. 59\n"},{"id":"154178","messageId":"E7645863-A3AD-4EE1-AF6B-71C50A859619@gmail.com","threadId":"25524","inReplyTo":"20101022161851.GH9224@burratino","subject":"Re: [RFC] Print diffs of UTF-16 to console / patches to email as UTF-8...?","fromName":"Joshua Juran","fromEmail":"jjuran@gmail.com","sentAt":"2010-10-22T18:28:44Z","receivedAt":"2010-10-22T18:28:44Z","isPatch":false,"sender":{"key":"jjuran@gmail.com","avatar":null},"body":"On Oct 22, 2010, at 9:18 AM, Jonathan Nieder wrote:\n\n> Drew Northup wrote:\n>\n>> This is part of my ongoing work to treat UTF-16 as text (in\n>> other words, the crlf options will work and .gitattributes hacks \n>> won't\n>> be required to display diffs, etc).\n\nI would like to see the same thing for MacRoman-encoded text.[1] This \nis the encoding used by classic Mac development tools such as \nMetrowerks C/C++ (packaged as CodeWarrior) and Apple's Rez resource \ncompiler (even the version in OS X). Clearly, UTF-8 checkouts are not \nan option here.\n\n> What's wrong with .gitattributes for this use case? I would think a\n> clean/smudge filter would produce very good behavior from most git\n> commands.\n\nI wrote a Mac<->UTF-8 converter in C++ and set it as the clean/smudge \nfilter for .r (Rez) files. Checkouts were noticeably slower (on a \nreal machine, not one of my antiques). This would be much worse if I \nalso applied it to C and C++ source files (most, but not all, of which \nare ASCII anyway).\n\n> If speed is the issue, maybe a built-in clean/smudge filter would\n> address that?\n\nWhile the performance cost could be overlooked, a worse problem \noccurred when I checked out a branch into which the conversion of \nfiles from MacRoman to UTF-8 hadn't occurred. It automatically \ndirtied my working tree, requiring me to temporarily disable the \nfilter attribute and reset --hard. I also resorted to checkout -f a \nnumber of times -- a bad habit, I'm sure.\n\nIn the end I concluded that (a) these files are definitely text, and \n(b) they are natively MacRoman and should be stored that way. There \nis no advantage to using UTF-8 since the tools can't handle it, and \neven were one to write a UTF-8-capable Rez compiler, the resources it \noutputs are still MacRoman-encoded, so no Unicode support is possible.\n\nFinally, (c) the end-to-end principle applies. Don't convert data en \nroute, but wait until it's necessary. Premature conversion was the \ncurse of FTP; let's not repeat it. But Git should definitely convert \ndata to match the encoding of the display device; writing anything but \nvalid UTF-8 to a UTF-8 terminal is in error. The same applies in gitk.\n\nAnother issue (on which my thoughts are less clear) is the use of CR \nnewlines. CR is also native to classic Mac OS, but in contrast to \nUTF-8, Mac developer tools are generally newline-agnostic, whereas \ntypical Unix programs are less forgiving in expecting LF -- so I've \nbeen using linefeeds in my source code. However, it might be useful \nto retain platform-customary newlines for the purpose of guessing non- \nUTF character encodings: Carriage returns would almost certainly \nindicate MacRoman rather than ISO-8859-1. But a more complete and \nrobust solution would be to store the encoding somewhere, possibly in \nthe blob itself, or in the tree storing the filename.\n\nJosh\n\n[1] MacRoman is an extended ASCII character set native to classic Mac \nOS. \n"},{"id":"154190","messageId":"20101022191339.GA13581@sigill.intra.peff.net","threadId":"25524","inReplyTo":"E7645863-A3AD-4EE1-AF6B-71C50A859619@gmail.com","subject":"Re: [RFC] Print diffs of UTF-16 to console / patches to email as UTF-8...?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-10-22T19:13:39Z","receivedAt":"2010-10-22T19:13:39Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Oct 22, 2010 at 11:28:44AM -0700, Joshua Juran wrote:\n\n> >What's wrong with .gitattributes for this use case? I would think a\n> >clean/smudge filter would produce very good behavior from most git\n> >commands.\n> \n> I wrote a Mac<->UTF-8 converter in C++ and set it as the clean/smudge\n> filter for .r (Rez) files. Checkouts were noticeably slower (on a\n> real machine, not one of my antiques). This would be much worse if I\n> also applied it to C and C++ source files (most, but not all, of\n> which are ASCII anyway).\n\nNot surprising, as you were probably running your filter a lot. Clean\nand smudge could perhaps benefit from the same notes-caching layer that\ntextconv uses (caching the \"smudged\" version of each clean file).\n\nBut that would only impact checkout. Most other operations use the\n\"clean\" representation already, so they should be full-speed.\n\nYou could also cache the other way (mapping smudged sha1's into clean\nsha1's). But I doubt that would do you any good. We generally see those\nwhen updating the index with \"git add\", which means either the stat\ninformation is clean (and we don't have the clean the file) or it isn't\n(in which case you probably have new content that has not been seen\nbefore, which means a cache miss).\n\nAnd of course it doesn't help the other clean/smudge inconveniences you\nran into.\n\n-Peff\n"},{"id":"154192","messageId":"201010222118.09212.jnareb@gmail.com","threadId":"25524","inReplyTo":"1287770805.819.7.camel@drew-northup.unet.maine.edu","subject":"Re: [RFC] Print diffs of UTF-16 to console / patches to email as UTF-8...?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-10-22T19:18:06Z","receivedAt":"2010-10-22T19:18:06Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Fri, 22 Oct 2010, Drew Northup wrote:\n> On Fri, 2010-10-22 at 10:48 -0700, Jakub Narebski wrote:\n> > Drew Northup writes:\n> \n> > > Well I shall plumb the documentation again.... just in case. I'm not\n> > > holding my breath that it will do what I (and frankly a fair number of\n> > > other people) want. We just want version control that treats text like\n> > > text. FULL STOP. Why isn't UTF-16 text???????\n> > \n> > If you are asking why Git detects files with text in UTF-16 / USC-2 as\n> > binary, it is because Git (re)uses the same heuristic that e.g. GNU\n> > diff (and probably also -T file test in Perl), and one of heuristics\n> > is that if file contains NUL (\"\\0\") character, then it is most\n> > porbably binary (because legacy C programs for text would have\n> > troubles with NUL characters).\n> > \n> > That probably doesn't help you any...\n> \n> I did find that already. I still have not decided that correct place to\n> shoehorn in Unicode detection, but I'll be sure to do that before I\n> bother anybody else with it. I already wrote code to detect (reasonably)\n> valid UTF-16 (if it isn't obviously valid then I'll just as soon deal\n> with it as binary data, so as to avoid a foot-shooting exercise).\n> My main motivation here has been to get some feedback as I write stuff\n> so as to not waste a lot of time during writing something that could be\n> done better. \n>\n> (As opposed to not done at all, which is the feeling I'm getting from a\n> few people around here...)\n\nGit supports well different encoding used in commit message (which is\nalways text, as opposed to file contents which might be binary or text).\n\nYou specify what encoding you use to format commit messages with\ni18n.commitEncoding (defaults to 'utf-8'); if it is different than utf-8\nit gets saved in 'encoding' header. You can even specify that encoding\nthat your terminal uses is different from i18n.commitEncoding with\ni18n.logOutputEncoding\n\nThe only support for different encoding of file contents is used by\ngit-gui. You provide encoding that a file uses via .gitattributes\n(the `encoding` attribute). You specify what output encoding git-gui\n(Tcl/Tk) uses with `gui.encoding` config variable.\n\nI guess that what you need to support for diffs and 'git show '\netc. is respecting `encoding` .gitattribute, and providing encoding\nthat console uses with e.g. i18n.blobOutputEncoding (or something like\nthat).\n\nHTH\n-- \nJakub Narebski\nPoland\n"},{"id":"154198","messageId":"20101022195331.GA12014@burratino","threadId":"25524","inReplyTo":"E7645863-A3AD-4EE1-AF6B-71C50A859619@gmail.com","subject":"Re: [RFC] Print diffs of UTF-16 to console / patches to email as UTF-8...?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-10-22T19:53:31Z","receivedAt":"2010-10-22T19:53:31Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Joshua Juran wrote:\n\n> I would like to see the same thing for MacRoman-encoded text.[1]\n> This is the encoding used by classic Mac development tools such as\n> Metrowerks C/C++ (packaged as CodeWarrior) and Apple's Rez resource\n> compiler (even the version in OS X). Clearly, UTF-8 checkouts are\n> not an option here.\n\nYes, makes sense.\n\nThere are (at least) two approaches you could use here: treat the\ncontent as precious and use e.g. textconv for readable diffs, or\ntreat the content as UTF-8 text and use clean/smudge to ensure\nthe checkout has the right encoding.\n\nSo let's see what happens with the latter:\n\n> I wrote a Mac<->UTF-8 converter in C++ and set it as the\n> clean/smudge filter for .r (Rez) files. Checkouts were noticeably\n> slower (on a real machine, not one of my antiques).\n\nVague ideas to mitigate that:\n\n a) allow a single clean/smudge filter invocation for a batch of\n files\n b) cache, as Jeff hinted\n c) allow custom \"native\" clean/smudge filters, executed using dlopen()\n\n> While the performance cost could be overlooked, a worse problem\n> occurred when I checked out a branch into which the conversion of\n> files from MacRoman to UTF-8 hadn't occurred. It automatically\n> dirtied my working tree, requiring me to temporarily disable the\n> filter attribute and reset --hard. I also resorted to checkout -f a\n> number of times -- a bad habit, I'm sure.\n\nThe jn/merge-renormalize topic from pu might help somewhat (or might\nnot). In any event, if you have a test case, I would be happy to look\nat it.\n\n> In the end I concluded that (a) these files are definitely text, and\n> (b) they are natively MacRoman and should be stored that way. There\n> is no advantage to using UTF-8 since the tools can't handle it, and\n> even were one to write a UTF-8-capable Rez compiler, the resources\n> it outputs are still MacRoman-encoded, so no Unicode support is\n> possible.\n> \n> Finally, (c) the end-to-end principle applies.\n\nYep.\n\nAlthough \"definitely text\" seems somewhat abstract to me. Is the\nproblem that \"git diff\" fails to default to --text in some situation?\n\n> But Git should definitely\n> convert data to match the encoding of the display device; writing\n> anything but valid UTF-8 to a UTF-8 terminal is in error.\n\nOh, this is what you mean. Except for log encoding, git is not paying\nattention to the display encoding at all.\n\n[...]\n> But a more\n> complete and robust solution would be to store the encoding\n> somewhere, possibly in the blob itself, or in the tree storing the\n> filename.\n\nHow about Jakub's idea of keeping it in .gitattributes (or some\nsimilarly visible key/value store)? Two reasons:\n\n 1. When asked to declare encoding, half the time people will be\n wrong. So it seems worthwhile to make the declared encoding\n visible enough to fix.\n\n 2. Two ASCII files identical except that one is declared as\n latin1 and the other utf8 should be considered identical.\n\nThanks for some food for thought.\n"},{"id":"154201","messageId":"1287778695.819.17.camel@drew-northup.unet.maine.edu","threadId":"25524","inReplyTo":"20101022195331.GA12014@burratino","subject":"Git Attribute: File Text Encoding {WAS: Re: [RFC] Print diffs of UTF-16 to console / patches to email as UTF-8...?}","fromName":"Drew Northup","fromEmail":"drew.northup@maine.edu","sentAt":"2010-10-22T20:18:15Z","receivedAt":"2010-10-22T20:18:15Z","isPatch":false,"sender":{"key":"drew.northup@maine.edu","avatar":"https://avatars.githubusercontent.com/u/18331571?v=4"},"body":"\nOn Fri, 2010-10-22 at 14:53 -0500, Jonathan Nieder wrote:\n> Joshua Juran wrote:\n\n> > But a more\n> > complete and robust solution would be to store the encoding\n> > somewhere, possibly in the blob itself, or in the tree storing the\n> > filename.\n> \n> How about Jakub's idea of keeping it in .gitattributes (or some\n> similarly visible key/value store)? Two reasons:\n> \n> 1. When asked to declare encoding, half the time people will be\n> wrong. So it seems worthwhile to make the declared encoding\n> visible enough to fix.\n> \n> 2. Two ASCII files identical except that one is declared as\n> latin1 and the other utf8 should be considered identical.\n> \n> Thanks for some food for thought.\n\nI think that's a fine place to start. Shall I start a branch for it when\nI get home (where the code I'm working on is located)? It would be good\npractice if nothing else...\n\n-- \n-Drew Northup N1XIM\n AKA RvnPhnx on OPN\n________________________________________________\n\"As opposed to vegetable or mineral error?\"\n-John Pescatore, SANS NewsBites Vol. 12 Num. 59\n"},{"id":"154213","messageId":"m3sjzxkifz.fsf@localhost.localdomain","threadId":"25524","inReplyTo":"1287778695.819.17.camel@drew-northup.unet.maine.edu","subject":"Re: Git Attribute: File Text Encoding {WAS: Re: [RFC] Print diffs of UTF-16 to console / patches to email as UTF-8...?}","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-10-22T21:49:20Z","receivedAt":"2010-10-22T21:49:20Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Drew Northup writes:\n> On Fri, 2010-10-22 at 14:53 -0500, Jonathan Nieder wrote:\n> > Joshua Juran wrote:\n> \n> > > But a more\n> > > complete and robust solution would be to store the encoding\n> > > somewhere, possibly in the blob itself, or in the tree storing the\n> > > filename.\n> > \n> > How about Jakub's idea of keeping it in .gitattributes (or some\n> > similarly visible key/value store)? Two reasons:\n> > \n> > 1. When asked to declare encoding, half the time people will be\n> > wrong. So it seems worthwhile to make the declared encoding\n> > visible enough to fix.\n> > \n> > 2. Two ASCII files identical except that one is declared as\n> > latin1 and the other utf8 should be considered identical.\n> > \n> > Thanks for some food for thought.\n> \n> I think that's a fine place to start. Shall I start a branch for it when\n> I get home (where the code I'm working on is located)? It would be good\n> practice if nothing else...\n\nJust for clarification: Git supports and uses `encoding` gitattribute,\nbut it is used currently _only_ bu git-gui (by converting from `encoding`\ngitattribute to `git.encoding` given by config).\n\nThe places that are missing:\n\n1. A way to check attribute of a file in given tree. Currently \n git-check-attr checks for .gitattributes _only_ from working\n area (in addition to unversioned .git/info/attributes, and perhaps\n in the future core.attributesFile a la core.excludesFile).\n\n2. A consensus where conversion should take place: at which level\n in stack, for what output destination, etc.\n\n3. Support for i18n.blobOutputEncoding, which would convert (unless\n overriden) e.g. in 'git show ', and perhaps also in diff.\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"}]}