{"thread":{"id":"2470","subject":"GIT character codecs","startedAt":"2005-11-12T12:37:12Z","lastAt":"2005-11-12T22:00:38Z","messageCount":2,"participants":["Marco Costalba","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"11670","messageId":"20051112123712.75182.qmail@web26301.mail.ukl.yahoo.com","threadId":"2470","inReplyTo":null,"subject":"GIT character codecs","fromName":"Marco Costalba","fromEmail":"mcostalba@yahoo.it","sentAt":"2005-11-12T12:37:12Z","receivedAt":"2005-11-12T12:37:12Z","isPatch":false,"sender":{"key":"mcostalba@gmail.com","avatar":null},"body":"Hi all,\n\n   this morning I pulled from git repo and tried to look at Junio japanese name with:\n\n      git-diff-tree -p e6bd23911efd0a2bd756c77d9e7ba6576eb739a1\n\nWith my big sorry the name was erroneously shown on my box.\n\nI modified qgit to force the use of utf-8 codec instead of the local one, \nin my case ISO-8859-15.\n\nNow I can see Junio name in beautiful japanese characters, and also the \ncorrect 'Lukas Sandstrom' name.\n\nThe problem is now qgit on my local git archives no more correctly \nshows special italian characters.\n\nSo here we arrive at the points:\n\n1) If utf8 is the 'only good one' for GIT repos why git-diff-tree uses \nthe local codec, in my case ISO-8859-15.\n\n2) If utf8 is not the 'only good one' but the commits are always saved using the\nlocal codec, how is possible to set the correct codec to use after pulling \nfrom a remote repo with a different codec (in this case public git repo that \nis in utf8).\n\nThanks\nMarco\n\n\n\n\t\t\n__________________________________ \nYahoo! FareChase: Search multiple travel sites in one click.\nhttp://farechase.yahoo.com\n"},{"id":"11701","messageId":"7vzmo9edbt.fsf@assigned-by-dhcp.cox.net","threadId":"2470","inReplyTo":"20051112123712.75182.qmail@web26301.mail.ukl.yahoo.com","subject":"Re: GIT character codecs","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-11-12T22:00:38Z","receivedAt":"2005-11-12T22:00:38Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Marco Costalba <mcostalba@yahoo.it> writes:\n\n> I modified qgit to force the use of utf-8 codec instead of the local one, \n> in my case ISO-8859-15.\n\nI suspect that trying to have a globa single encoding is a wrong\napproach overall.  There are a handful issues to think about.\n\nFirst the easiest one -- the commit log messages.  We encourage\nuse of UTF-8, because they hold people's names which are the\nsignificant source of i18n needs.  Even if your project starts\nout with just a group of people whose names can be spelled in\nASCII just fine, or all western european in which case you might\nbe tempted to do latin-1 or 8859-15, or all Japanese with\nEUC-JP, your project _might_ someday have a member whose name\ncannot be spelled if you pick one of these encodings [*2*].  So\nwe encourage use of UTF-8 and help people by having -u flag in\ngit-mailinfo, for example.\n\nBut we do not _enforce_ UTF-8.  I've never worked actively with\nnon-English free software projects, and I do not know if people\nin, say, Japan write their commit log messages in Japanese.  I\nhowever think it is a reasonable thing to do if the project is\nlocal and exclusive (e.g. a company internal project).  And if\nlocal encoding is more convenient to handle for users\n(e.g. their \"less\" is configured to expect something non UTF-8),\nit is perfectly reasonable to do all their commit log messages\nin their local encoding, as long as the members understand that\nis the project's policy.  And it is probably reasonable to\nassume that within one project, a single encoding is used for\ncommit log messages, although it may not be UTF-8.\n\nWhat this means for qgit is that it is often sufficient for its\nlog browser to support one encoding at a time, provided if it\nallows the user to switch which encoding to use depending on\nwhat project is being viewed.\n\nThe trouble you had is however not with commit messages, but\nwith the blob contents (i.e. user data).  First of all, the user\ndata is just any binary goo for git, and the interpretation of\nit is left to the Porcelains and the users.  There are two\nthings to talk about here.\n\nI imagine qgit has, in addition to diff-between-revisions\nviewer, a blob viewer that lets your users browse a whole file\nin a given revision.  The files in the Documentation directory\nin git.git/ right now happens to be encoded in UTF-8 because\nthey are asciidoc sources.  But other projects may use different\nencodings, and even a single project can have its i18n message\nfiles in different encodings and charsets in different files.\nUsers probably want to be able to view all of them, even if they\nonly understand a couple of languages and not others.\n\nWhat this means for qgit is that at least you should be able to\nshow a whole file in a single encoding, but if you show more\nthan two files at the same time, one in each window, these\nwindows may be showing its contents in different encodings and\ncharsets.  So you would need to give a way to your users to tell\nyou what encoding each file is in.  Using global locale as the\ndefault and having a way to override that per file basis would\nbe sufficient.\n\nA more interesting thing about user data is the diff between\nrevisions.  How should you show the diff between\ngit-pack-redundant documentation that changes Lukas' name\nencoded from latin-1 to UTF-8?\n\nFirst, git-diff (\"diff\" it calls) is encoding agnostic.  The\nlines that come out from it are taken from its input files and\nif two revisions you compared both store your words in Italian\nencoded in iso-8859-15, the output would be iso-8859-15.  IOW,\ndiff would end up doing the right thing but without knowing it.\n\nThe Lukas' name encoding patch would be handled the same way.\nDiff output would just say \"these lines have different bytes\",\nand spit out '-' line encoded in latin-1 and '+' in UTF-8.\n\nThis is a mess in one sense but not necessarily so.  If we could\nsay \"Mr diff, the first file is in latin-1 and the second one is\nin UTF-8, so compare them accordingly, and by the way we would\nlike the output in UTF-8 please\", then it would be interesting\nand may be sometimes useful, but in this particular case it is\nuseless -- there would not be any diff.  And the reason _I_ ran\ndiff before committing that change was to make sure I did the\nencoding conversion correctly, so noticing the byte-level\ndifferences was the right thing to do.\n\nWhat should qgit do about this?  This \"encoding gotcha fix\npatch\" is a special case, so I think it is perfectly reasonable\nif qgit did not do anything special about it.  So the \"per-file\nencoding override\" I suggested for whole-file viewer would be\nadequate.  For the special \"encoding fix\" case, as I discussed\nabove, the most useful output is \"raw bytes\", so maybe throw in\na special encoding \"raw\" as one of the choices and I think you\nare done.\n\n\n[Footnote]\n\n*1* Some people may argue unicode cannot spell all names, but\nlet's pretend it can -- or at least it can cover wider character\nsets than any single local encodings.  Some people further may\nargue that iso-2022 does better, but let's not go there.\n\n*2* We do not forbid you to store arbitrary binary blob in your\ncommit objects.\n"}]}