{"thread":{"id":"6051","subject":"specify charset for commits (Was: [PATCH] Fix documentation copy&paste typo)","startedAt":"2006-12-08T11:44:31Z","lastAt":"2006-12-25T04:03:54Z","messageCount":39,"participants":["Uwe Kleine-König","Johannes Schindelin","Santi Béjar","Alexander Litvinov","Jakub Narebski","Junio C Hamano","Nicolas Pitre","Uwe Kleine-Koenig"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"294281","messageId":"11655782712452-git-send-email-zeisberg@informatik.uni-freiburg.de","threadId":"6051","inReplyTo":null,"subject":"[PATCH] Fix documentation copy&paste typo","fromName":"Uwe Kleine-Koenig","fromEmail":"zeisberg@informatik.uni-freiburg.de","sentAt":"2006-12-08T11:44:31Z","receivedAt":"2006-12-08T11:44:31Z","isPatch":true,"sender":{"key":"u.kleine-koenig@pengutronix.de","avatar":"https://gravatar.com/avatar/354b5e3ceb2806a2f1e1e382ac29ddbdad18288654da62b61eb13583a857eee7?d=mp&s=160"},"body":"From: Uwe Zeisberger <zeisberg@informatik.uni-freiburg.de>\n\nThis was introduced in 45a3b12cfd3eaa05bbb0954790d5be5b8240a7b5\n\nSigned-off-by: Uwe Kleine-König <zeisberg@informatik.uni-freiburg.de>\n---\n gitweb/gitweb.perl |    2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 093bd72..ed40810 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -120,7 +120,7 @@ our %feature = (\n \t# To disable system wide have in $GITWEB_CONFIG\n \t# $feature{'snapshot'}{'default'} = [undef];\n \t# To have project specific config enable override in $GITWEB_CONFIG\n-\t# $feature{'blame'}{'override'} = 1;\n+\t# $feature{'snapshot'}{'override'} = 1;\n \t# and in project config gitweb.snapshot = none|gzip|bzip2;\n \t'snapshot' => {\n \t\t'sub' => \\&feature_snapshot,\n-- \n1.4.4.2.gb772\n"},{"id":"296502","messageId":"20061219141618.GA2539@cepheus","threadId":"6051","inReplyTo":"11655782712452-git-send-email-zeisberg@informatik.uni-freiburg.de","subject":"Re: [PATCH] Fix documentation copy&paste typo","fromName":"Uwe Kleine-König","fromEmail":"zeisberg@informatik.uni-freiburg.de","sentAt":"2006-12-19T14:16:18Z","receivedAt":"2006-12-19T14:16:18Z","isPatch":true,"sender":{"key":"u.kleine-koenig@pengutronix.de","avatar":"https://gravatar.com/avatar/354b5e3ceb2806a2f1e1e382ac29ddbdad18288654da62b61eb13583a857eee7?d=mp&s=160"},"body":"Hello Junio,\n\nUwe Kleine-König wrote:\n> From: Uwe Zeisberger <zeisberg@informatik.uni-freiburg.de>\n> \n> This was introduced in 45a3b12cfd3eaa05bbb0954790d5be5b8240a7b5\n> \n> Signed-off-by: Uwe Kleine-König <zeisberg@informatik.uni-freiburg.de>\n> ---\n> [...]\nyou took that patch as bbee1d971dc07c29f840b439aa2a2c890a12cf9f, thanks\nfor that.\n\nSomehow the 'ö' (o-umlaut) in my name was messed up.  If I do\n\n\tgit cat-file -p bbee1d971dc07c29 | xxd | grep eine\n\nI get:\n\n\t0000160: 6569 6e65 2d4b 1b2c 4143 361b 2842 6e69 eine-K.,AC6.(Bni\n\nThat is, the 'ö' became 8 byte long.  Can you tell me what went wrong\nthere?\n\nThe commits by Karl Hasselström <kha@treskal.com> (e.g. e67c66251a4165)\nuse UTF-8.\n\nDoes there exist a (maybe project specific) convention for the encoding\nof commit logs?\n\nBest regards\nUwe\n\n-- \nUwe Kleine-Koenig\n\n"},{"id":"298500","messageId":"7vac1jre1u.fsf@assigned-by-dhcp.cox.net","threadId":"6051","inReplyTo":"20061219141618.GA2539@cepheus","subject":"Re: [PATCH] Fix documentation copy&paste typo","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-12-19T17:27:57Z","receivedAt":"2006-12-19T17:27:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Uwe Kleine-König <zeisberg@informatik.uni-freiburg.de> writes:\n\n> I get:\n>\n> \t0000160: 6569 6e65 2d4b 1b2c 4143 361b 2842 6e69 eine-K.,AC6.(Bni\n>\n> That is, the 'ö' became 8 byte long.  Can you tell me what went wrong\n> there?\n\nMe, keyboard and Emacs screwed up and stored it in ISO-2022\ninstead of UTF-8.  Sorry.\n\n"},{"id":"29916","messageId":"20061221085907.GA2244@cepheus","threadId":"6051","inReplyTo":"7vac1jre1u.fsf@assigned-by-dhcp.cox.net","subject":"specify charset for commits (Was: [PATCH] Fix documentation copy&paste typo)","fromName":"Uwe Kleine-König","fromEmail":"zeisberg@informatik.uni-freiburg.de","sentAt":"2006-12-21T08:59:07Z","receivedAt":"2006-12-21T08:59:07Z","isPatch":true,"sender":{"key":"u.kleine-koenig@pengutronix.de","avatar":"https://gravatar.com/avatar/354b5e3ceb2806a2f1e1e382ac29ddbdad18288654da62b61eb13583a857eee7?d=mp&s=160"},"body":"Hello Junio,\n\nJunio C Hamano wrote:\n> Me, keyboard and Emacs screwed up and stored it in ISO-2022\n> instead of UTF-8.  Sorry.\nIt's a pity, but too late to change.[1]\n\nWhat do you think about a patch that makes git-commit-tree call iconv on\nits input to get it to UTF-8 (or any other charset).  Maybe it\nmakes sense to add another header to commit objects (e.g.\n\"charset UTF-8\") if something in the commit object is non-ASCII?\n\nIn my eyes it would make sense to even force UTF-8 for commit logs (and\nauthor, committer).  The downside is that it becomes impossible to store\narbitrary byte sequences in commit objects.  (IMHO not a real\nlimitation.)\n\nBest regards\nUwe\n\n[1] actually I think it's worse, because my iconv (from Debian's libc6\n    version 2.3.6.ds1-8) was unable to convert it correctly to utf-8 for\n    any encoding that starts with ISO-2022.\n\n-- \nUwe Kleine-König\n\nhttp://www.google.com/search?q=sin%28pi%2F2%29\n"},{"id":"29926","messageId":"Pine.LNX.4.63.0612211050450.19693@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6051","inReplyTo":"20061221085907.GA2244@cepheus","subject":"Re: specify charset for commits (Was: [PATCH] Fix documentation copy&paste typo)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-21T09:51:44Z","receivedAt":"2006-12-21T09:51:44Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 21 Dec 2006, Uwe Kleine-König wrote:\n\n> Junio C Hamano wrote:\n> > Me, keyboard and Emacs screwed up and stored it in ISO-2022\n> > instead of UTF-8.  Sorry.\n> It's a pity, but too late to change.[1]\n> \n> What do you think about a patch that makes git-commit-tree call iconv on\n> its input to get it to UTF-8 (or any other charset).\n\nWe had this discussion over and over again. Last time (I think) was here:\n\nhttp://article.gmane.org/gmane.comp.version-control.git/11710\n\nSummary: we do not want to force the use of utf8.\n\nHth,\nDscho"},{"id":"29928","messageId":"8aa486160612210211n64dc2973jcc4e7670fec2cabc@mail.gmail.com","threadId":"6051","inReplyTo":"Pine.LNX.4.63.0612211050450.19693@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: specify charset for commits (Was: [PATCH] Fix documentation copy&paste typo)","fromName":"Santi Béjar","fromEmail":"sbejar@gmail.com","sentAt":"2006-12-21T10:11:15Z","receivedAt":"2006-12-21T10:11:15Z","isPatch":true,"sender":{"key":"santi@agolina.net","avatar":null},"body":"On 12/21/06, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:\n> Hi,\n>\n> On Thu, 21 Dec 2006, Uwe Kleine-König wrote:\n>\n> > Junio C Hamano wrote:\n> > > Me, keyboard and Emacs screwed up and stored it in ISO-2022\n> > > instead of UTF-8.  Sorry.\n> > It's a pity, but too late to change.[1]\n> >\n> > What do you think about a patch that makes git-commit-tree call iconv on\n> > its input to get it to UTF-8 (or any other charset).\n>\n> We had this discussion over and over again. Last time (I think) was here:\n>\n> http://article.gmane.org/gmane.comp.version-control.git/11710\n>\n> Summary: we do not want to force the use of utf8.\n>\n\nMaybe git could have an example of the hook \"commit-msg\" that checks\nthat the commit message is in certain charset.\n\nSanti\n"},{"id":"29929","messageId":"200612211623.14236.litvinov2004@gmail.com","threadId":"6051","inReplyTo":"Pine.LNX.4.63.0612211050450.19693@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: specify charset for commits (Was: [PATCH] Fix documentation copy&paste typo)","fromName":"Alexander Litvinov","fromEmail":"litvinov2004@gmail.com","sentAt":"2006-12-21T10:23:13Z","receivedAt":"2006-12-21T10:23:13Z","isPatch":true,"sender":{"key":"litvinov2004@gmail.com","avatar":null},"body":"> > What do you think about a patch that makes git-commit-tree call iconv on\n> > its input to get it to UTF-8 (or any other charset).\n>\n> We had this discussion over and over again. Last time (I think) was here:\n> http://article.gmane.org/gmane.comp.version-control.git/11710\n> Summary: we do not want to force the use of utf8.\n\nMay we can add new header into commit with commit text encoding ?\n"},{"id":"29932","messageId":"emdorr$kt$1@sea.gmane.org","threadId":"6051","inReplyTo":"200612211623.14236.litvinov2004@gmail.com","subject":"Re: specify charset for commits (Was: [PATCH] Fix documentation copy&paste typo)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-12-21T10:52:12Z","receivedAt":"2006-12-21T10:52:12Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Alexander Litvinov wrote:\n\n>>> What do you think about a patch that makes git-commit-tree call iconv on\n>>> its input to get it to UTF-8 (or any other charset).\n>>\n>> We had this discussion over and over again. Last time (I think) was here:\n>> http://article.gmane.org/gmane.comp.version-control.git/11710\n>> Summary: we do not want to force the use of utf8.\n> \n> May we can add new header into commit with commit text encoding ?\n\nI think it should be repository-wide decision. And we have\ni18n.commitEncoding configuration variable (perhaps it should be propagated\non clone?).\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"29944","messageId":"200612211905.27555.litvinov2004@gmail.com","threadId":"6051","inReplyTo":"emdorr$kt$1@sea.gmane.org","subject":"Re: specify charset for commits (Was: [PATCH] Fix documentation copy&paste typo)","fromName":"Alexander Litvinov","fromEmail":"litvinov2004@gmail.com","sentAt":"2006-12-21T13:05:27Z","receivedAt":"2006-12-21T13:05:27Z","isPatch":true,"sender":{"key":"litvinov2004@gmail.com","avatar":null},"body":"В сообщении от Thursday 21 December 2006 16:52 Jakub Narebski написал(a):\n> > May we can add new header into commit with commit text encoding ?\n>\n> I think it should be repository-wide decision. And we have\n> i18n.commitEncoding configuration variable (perhaps it should be propagated\n> on clone?).\n\nI would disagree with you. Is is not hard to imagine international project \nmanaged by git. We [developers] can start to use utf-8 or similar universal \nencoding but it is not easy sometimes. Fir example, not long ago all russian \nlinux machines has LANG set to ru_RU.KOI8-R, now it tend to be ru_RU.UTF-8. \nIt will not big surprise to me if developr from China or Japan use something \nvery unusual. And just imagine one developer using Windows and Cygwin - ha ha \nha, try to ask him to change the encoding :-)\n\nThe easiest way for git is just to store commit encoding and let tool for \nhistory browsing deal with encoding. Or as it does now - simply ignore all \nencodings at all and work with bytes not chars. But at this case history \nbrowsing tool must have some magic knowlage about encoding taken from \nair :-). Or from config file that cover most used encodings.\n\nAlexander Litvinov.\n"},{"id":"29945","messageId":"eme17b$vff$1@sea.gmane.org","threadId":"6051","inReplyTo":"200612211905.27555.litvinov2004@gmail.com","subject":"Re: specify charset for commits (Was: [PATCH] Fix documentation copy&paste typo)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-12-21T13:14:52Z","receivedAt":"2006-12-21T13:14:52Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Alexander Litvinov wrote:\n\n> ? ????????? ?? Thursday 21 December 2006 16:52 Jakub Narebski ???????(a):\n>>> May we can add new header into commit with commit text encoding ?\n>>\n>> I think it should be repository-wide decision. And we have\n>> i18n.commitEncoding configuration variable (perhaps it should be propagated\n>> on clone?).\n> \n> I would disagree with you. Is is not hard to imagine international project \n> managed by git. We [developers] can start to use utf-8 or similar universal \n> encoding but it is not easy sometimes. Fir example, not long ago all russian \n> linux machines has LANG set to ru_RU.KOI8-R, now it tend to be ru_RU.UTF-8. \n> It will not big surprise to me if developr from China or Japan use something \n> very unusual. And just imagine one developer using Windows and Cygwin - ha ha \n> ha, try to ask him to change the encoding :-)\n> \n> The easiest way for git is just to store commit encoding and let tool for \n> history browsing deal with encoding. Or as it does now - simply ignore all \n> encodings at all and work with bytes not chars. But at this case history \n> browsing tool must have some magic knowlage about encoding taken from \n> air :-). Or from config file that cover most used encodings.\n\nPerhaps it is time to resurrect idea about \"note\" header in commit object?\nIt could be used to store charset for those commits which doesn't use\ndefault charset...\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"29948","messageId":"20061221134338.GA2246@cepheus","threadId":"6051","inReplyTo":"emdorr$kt$1@sea.gmane.org","subject":"Re: specify charset for commits (Was: [PATCH] Fix documentation copy&paste typo)","fromName":"Uwe Kleine-König","fromEmail":"zeisberg@informatik.uni-freiburg.de","sentAt":"2006-12-21T13:43:38Z","receivedAt":"2006-12-21T13:43:38Z","isPatch":true,"sender":{"key":"u.kleine-koenig@pengutronix.de","avatar":"https://gravatar.com/avatar/354b5e3ceb2806a2f1e1e382ac29ddbdad18288654da62b61eb13583a857eee7?d=mp&s=160"},"body":"Hello,\n\nJakub Narebski wrote:\n> Alexander Litvinov wrote:\n> \n> >>> What do you think about a patch that makes git-commit-tree call iconv on\n> >>> its input to get it to UTF-8 (or any other charset).\n> >>\n> >> We had this discussion over and over again. Last time (I think) was here:\n> >> http://article.gmane.org/gmane.comp.version-control.git/11710\n> >> Summary: we do not want to force the use of utf8.\n> > \n> > May we can add new header into commit with commit text encoding ?\n> \n> I think it should be repository-wide decision. And we have\n> i18n.commitEncoding configuration variable\nThe disadvantage from a repository-wide decision is that you cannot\nchange it after a while.\n\nI didn't know that variable, but I think as it exists,\ngit-commit-tree should iconv the commit message from local to\ni18n.commitEncoding before writing it.\n\nMoreover I like the idea of a new header for commits specifing the\nencoding.  Git could default to the handling as it is now (i.e. just\nbytes) if the header is missing.\n\nBest regards\nUwe\n\n-- \nUwe Kleine-König\n\nhttp://www.google.com/search?q=1+newton+in+kg*m+%2F+s%5E2\n"},{"id":"29959","messageId":"7vejqtaz7q.fsf@assigned-by-dhcp.cox.net","threadId":"6051","inReplyTo":"200612211623.14236.litvinov2004@gmail.com","subject":"Re: specify charset for commits","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-12-21T18:19:37Z","receivedAt":"2006-12-21T18:19:37Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Alexander Litvinov <litvinov2004@gmail.com> writes:\n\n> May we can add new header into commit with commit text encoding ?\n\nI do not think we want to change the commit header, nor we would\nwant to re-encode, but I can see two possible improvements:\n\n (1) git-am should default to -u; this was suggested on the list\n     long time ago, but is an incompatible change.  v1.5.0 we\n     can afford to be incompatible to make it more usable and\n     safer.\n\n (2) update commit-tree to reject non utf-8 log messages and\n     author/committer names when i18n.commitEncoding is _NOT_\n     set, or set to utf-8.\n\n     Maybe later we can use encoding validation routines for\n     other encodings by checking i18n.commitEncoding, but at the\n     minimum the above would be safe enough for recommended UTF-8\n     only cases.\n"},{"id":"29962","messageId":"Pine.LNX.4.64.0612211336240.18171@xanadu.home","threadId":"6051","inReplyTo":"7vejqtaz7q.fsf@assigned-by-dhcp.cox.net","subject":"Re: specify charset for commits","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-12-21T18:48:35Z","receivedAt":"2006-12-21T18:48:35Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 21 Dec 2006, Junio C Hamano wrote:\n\n> Alexander Litvinov <litvinov2004@gmail.com> writes:\n> \n> > May we can add new header into commit with commit text encoding ?\n> \n> I do not think we want to change the commit header, nor we would\n> want to re-encode, but I can see two possible improvements:\n> \n>  (1) git-am should default to -u; this was suggested on the list\n>      long time ago, but is an incompatible change.  v1.5.0 we\n>      can afford to be incompatible to make it more usable and\n>      safer.\n> \n>  (2) update commit-tree to reject non utf-8 log messages and\n>      author/committer names when i18n.commitEncoding is _NOT_\n>      set, or set to utf-8.\n\nThis would be a good thing, both of them actually.\n\nThe Linux kernel already contains different charsets for author full \nname and/or commit messages making git-log output a mix of encodings \nalready.  And sometimes this inconsistency comes from the _same_ author.\n\n\nNicolas\n"},{"id":"29964","messageId":"20061221191141.GB27072@informatik.uni-freiburg.de","threadId":"6051","inReplyTo":"7vejqtaz7q.fsf@assigned-by-dhcp.cox.net","subject":"Re: specify charset for commits","fromName":"Uwe Kleine-König","fromEmail":"zeisberg@informatik.uni-freiburg.de","sentAt":"2006-12-21T19:11:41Z","receivedAt":"2006-12-21T19:11:41Z","isPatch":false,"sender":{"key":"u.kleine-koenig@pengutronix.de","avatar":"https://gravatar.com/avatar/354b5e3ceb2806a2f1e1e382ac29ddbdad18288654da62b61eb13583a857eee7?d=mp&s=160"},"body":"Junio C Hamano wrote:\n> Alexander Litvinov <litvinov2004@gmail.com> writes:\n> \n> > May we can add new header into commit with commit text encoding ?\n> \n> I do not think we want to change the commit header, nor we would\n> want to re-encode, but I can see two possible improvements:\n> \n>  (1) git-am should default to -u; this was suggested on the list\n>      long time ago, but is an incompatible change.  v1.5.0 we\n>      can afford to be incompatible to make it more usable and\n>      safer.\n> \n>  (2) update commit-tree to reject non utf-8 log messages and\n>      author/committer names when i18n.commitEncoding is _NOT_\n>      set, or set to utf-8.\n> \n>      Maybe later we can use encoding validation routines for\n>      other encodings by checking i18n.commitEncoding, but at the\n>      minimum the above would be safe enough for recommended UTF-8\n>      only cases.\nAs I only want to use UTF-8 both suggestions are fine for me.\n\nIs there a generic way to check an encoding?  (I don't know if there is\nan encoding that can encode everything.  If so, we could use iconv -f\n$enc -t superencoding.  Until now I thought UTF-8 can do, but in the\npost Johannes pointed out, you (Junio) implyed that it cannot.)\n\nBest regards\nUwe\n\n-- \nUwe Kleine-König\n\ndd if=/proc/self/exe bs=1 skip=1 count=3 2>/dev/null\n"},{"id":"29965","messageId":"200612220136.39905.litvinov2004@gmail.com","threadId":"6051","inReplyTo":"7vejqtaz7q.fsf@assigned-by-dhcp.cox.net","subject":"Re: specify charset for commits","fromName":"Alexander Litvinov","fromEmail":"litvinov2004@gmail.com","sentAt":"2006-12-21T19:36:39Z","receivedAt":"2006-12-21T19:36:39Z","isPatch":false,"sender":{"key":"litvinov2004@gmail.com","avatar":null},"body":"> I do not think we want to change the commit header\nCan you please explain why not ?\n\n>  (1) git-am should default to -u; this was suggested on the list\n>      long time ago, but is an incompatible change.  v1.5.0 we\n>      can afford to be incompatible to make it more usable and\n>      safer.\nI use git-am rarely so can't comment on this\n\n>  (2) update commit-tree to reject non utf-8 log messages and\n>      author/committer names when i18n.commitEncoding is _NOT_\n>      set, or set to utf-8.\n>\n>      Maybe later we can use encoding validation routines for\n>      other encodings by checking i18n.commitEncoding, but at the\n>      minimum the above would be safe enough for recommended UTF-8\n>      only cases.\nSee the situation:\n1. I have utf-8 encoded repo.\n2. Somebody clone my repo, try to commit using non-utf-8 encoding, fail and \nchange i18n.commitEncoding. When it commits something and ask me to pull.\n3. I pull and got non-utf-8 commit message :-)\n"},{"id":"30046","messageId":"Pine.LNX.4.63.0612220351520.19693@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6051","inReplyTo":"7vejqtaz7q.fsf@assigned-by-dhcp.cox.net","subject":"Re: specify charset for commits","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-22T12:07:23Z","receivedAt":"2006-12-22T12:07:23Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 21 Dec 2006, Junio C Hamano wrote:\n\n>  (2) update commit-tree to reject non utf-8 log messages and\n>      author/committer names when i18n.commitEncoding is _NOT_\n>      set, or set to utf-8.\n\nThe problem is: you cannot easily recognize if it is UTF8 or not, \nprogramatically. There is a good indicator _against_ UTF8, namely the \nfirst byte can _only_ be 0xxxxxxx, 110xxxxx, 1110xxxx, 11110xxx. But there \nis no _positive_ sign that it is UTF8. For example, many umlauts and other \nspecial modifications to letters, stay in the range 0x7f-0xff.\n\nCiao,\nDscho\n"},{"id":"30060","messageId":"20061222150948.GA6005@cepheus","threadId":"6051","inReplyTo":"Pine.LNX.4.63.0612220351520.19693@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: specify charset for commits","fromName":"Uwe Kleine-König","fromEmail":"zeisberg@informatik.uni-freiburg.de","sentAt":"2006-12-22T15:09:49Z","receivedAt":"2006-12-22T15:09:49Z","isPatch":false,"sender":{"key":"u.kleine-koenig@pengutronix.de","avatar":"https://gravatar.com/avatar/354b5e3ceb2806a2f1e1e382ac29ddbdad18288654da62b61eb13583a857eee7?d=mp&s=160"},"body":"Hello Johannes,\n\nJohannes Schindelin wrote:\n> The problem is: you cannot easily recognize if it is UTF8 or not, \n> programatically. There is a good indicator _against_ UTF8, namely the \n> first byte can _only_ be 0xxxxxxx, 110xxxxx, 1110xxxx, 11110xxx. But there \n> is no _positive_ sign that it is UTF8. For example, many umlauts and other \n> special modifications to letters, stay in the range 0x7f-0xff.\nThat's not the only indication.  Here comes a (Python) function that\nchecks is string s is correctly UTF-8 encoded:\n\n\tdef is_utf8_str(s):\n\t  cnt_furtherbytes = 0\n\t  for c in s:\n\t    if cnt_furtherbytes > 0:\n\t      if ord(c) & 0xc0 == 0x80:\n\t\tcnt_furtherbytes -= 1\n\t      else:\n\t\treturn False\n\t    else:\n\t      if ord(c) < 0x80:\n\t\tcontinue\n\t      elif ord(c) < 0xc0:\n\t        return False\n\t      elif ord(c) < 0xe0:\n\t\tcnt_furtherbytes = 1\n\t      elif ord(c) < 0xf0:\n\t\tcnt_furtherbytes = 2\n\t      elif ord(c) < 0xf8:\n\t\tcnt_furtherbytes = 3\n\t      elif ord(c) < 0xfc:\n\t\tcnt_furtherbytes = 4\n\t      elif ord(c) < 0xfe:\n\t\tcnt_furtherbytes = 5\n\t      else:\n\t\treturn False\n\t  return True\n\nAn UTF-8 character is either one byte long with the msb 0 or a sequence\nstarting with a value between 0xc0 and 0xfd (inclusive) and depending on\nthat first value up to six further bytes in the range 0x80 to 0xbf.\n\nYou could even be more strict by checking for Unicode 3.1 conformance\n(i.e. a character has to be encoded in it's shortest form).\n\nLook at utf8(7) for further details.  (This manpage is included in the\nDebian manpages package.)\n\nBest regards\nUwe\n\n-- \nUwe Kleine-König\n\nhttp://www.google.com/search?q=5+choose+3\n"},{"id":"30064","messageId":"Pine.LNX.4.64.0612221030440.18171@xanadu.home","threadId":"6051","inReplyTo":"Pine.LNX.4.63.0612220351520.19693@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: specify charset for commits","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-12-22T15:31:53Z","receivedAt":"2006-12-22T15:31:53Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 22 Dec 2006, Johannes Schindelin wrote:\n\n> Hi,\n> \n> On Thu, 21 Dec 2006, Junio C Hamano wrote:\n> \n> >  (2) update commit-tree to reject non utf-8 log messages and\n> >      author/committer names when i18n.commitEncoding is _NOT_\n> >      set, or set to utf-8.\n> \n> The problem is: you cannot easily recognize if it is UTF8 or not, \n> programatically. There is a good indicator _against_ UTF8, namely the \n> first byte can _only_ be 0xxxxxxx, 110xxxxx, 1110xxxx, 11110xxx. But there \n> is no _positive_ sign that it is UTF8. For example, many umlauts and other \n> special modifications to letters, stay in the range 0x7f-0xff.\n\nStill... that would be a good enough thing to have in the majority of \ncases, wouldn't it?\n\n\nNicolas\n"},{"id":"30078","messageId":"7vslf7zrdp.fsf@assigned-by-dhcp.cox.net","threadId":"6051","inReplyTo":"Pine.LNX.4.64.0612221030440.18171@xanadu.home","subject":"Re: specify charset for commits","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-12-22T19:01:54Z","receivedAt":"2006-12-22T19:01:54Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre <nico@cam.org> writes:\n\n> On Fri, 22 Dec 2006, Johannes Schindelin wrote:\n>\n>> Hi,\n>> \n>> On Thu, 21 Dec 2006, Junio C Hamano wrote:\n>> \n>> >  (2) update commit-tree to reject non utf-8 log messages and\n>> >      author/committer names when i18n.commitEncoding is _NOT_\n>> >      set, or set to utf-8.\n>> \n>> The problem is: you cannot easily recognize if it is UTF8 or not, \n>> programatically. There is a good indicator _against_ UTF8, namely the \n>> first byte can _only_ be 0xxxxxxx, 110xxxxx, 1110xxxx, 11110xxx. But there \n>> is no _positive_ sign that it is UTF8. For example, many umlauts and other \n>> special modifications to letters, stay in the range 0x7f-0xff.\n>\n> Still... that would be a good enough thing to have in the majority of \n> cases, wouldn't it?\n\nI think that would be very sane thing to do.\n"},{"id":"30100","messageId":"Pine.LNX.4.63.0612222201200.19693@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6051","inReplyTo":"7vslf7zrdp.fsf@assigned-by-dhcp.cox.net","subject":"[PATCH 1/2] libgit.a: add some UTF-8 handling functions","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-22T21:03:53Z","receivedAt":"2006-12-22T21:03:53Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"\nThis adds utf8_byte_count(), utf8_strlen() and print_wrapped_text().\n\nThe most important is probably utf8_strlen(), which returns the length\nof the text, if it is in UTF-8, otherwise -1.\n\nNote that we do not go the full nine yards: we could also check that\nthe character is encoded with the minimum amount of bytes, as pointed\nout by Uwe Kleine-Koenig.\n\nThe function print_wrapped_text() can be used to wrap text to a certain\nline length.\n\nSigned-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>\n---\n\n\tOn Fri, 22 Dec 2006, Junio C Hamano wrote:\n\n\t> Nicolas Pitre <nico@cam.org> writes:\n\t> \n\t> > On Fri, 22 Dec 2006, Johannes Schindelin wrote:\n\t> >> \n\t> >> On Thu, 21 Dec 2006, Junio C Hamano wrote:\n\t> >> \n\t> >> >  (2) update commit-tree to reject non utf-8 log messages and\n\t> >> >      author/committer names when i18n.commitEncoding is _NOT_\n\t> >> >      set, or set to utf-8.\n\t> >> \n\t> >> The problem is: you cannot easily recognize if it is UTF8 or \n\t> >> not, programatically. There is a good indicator _against_ \n\t> >> UTF8, namely the first byte can _only_ be 0xxxxxxx, 110xxxxx, \n\t> >> 1110xxxx, 11110xxx. But there is no _positive_ sign that it \n\t> >> is UTF8. For example, many umlauts and other special \n\t> >> modifications to letters, stay in the range 0x7f-0xff.\n\t> >\n\t> > Still... that would be a good enough thing to have in the \n\t> > majority of cases, wouldn't it?\n\t> \n\t> I think that would be very sane thing to do.\n\n\tWell, this patch together with the next one implements that.\n\n Makefile |    6 ++-\n utf8.c   |   93 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n utf8.h   |    8 +++++\n 3 files changed, 105 insertions(+), 2 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 29c4662..b4ca48b 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -237,7 +237,8 @@ LIB_H = \\\n \tarchive.h blob.h cache.h commit.h csum-file.h delta.h grep.h \\\n \tdiff.h object.h pack.h pkt-line.h quote.h refs.h list-objects.h sideband.h \\\n \trun-command.h strbuf.h tag.h tree.h git-compat-util.h revision.h \\\n-\ttree-walk.h log-tree.h dir.h path-list.h unpack-trees.h builtin.h\n+\ttree-walk.h log-tree.h dir.h path-list.h unpack-trees.h builtin.h \\\n+\tutf8.h\n \n DIFF_OBJS = \\\n \tdiff.o diff-lib.o diffcore-break.o diffcore-order.o \\\n@@ -256,7 +257,8 @@ LIB_OBJS = \\\n \trevision.o pager.o tree-walk.o xdiff-interface.o \\\n \twrite_or_die.o trace.o list-objects.o grep.o \\\n \talloc.o merge-file.o path-list.o help.o unpack-trees.o $(DIFF_OBJS) \\\n-\tcolor.o wt-status.o archive-zip.o archive-tar.o shallow.o\n+\tcolor.o wt-status.o archive-zip.o archive-tar.o shallow.o \\\n+\tutf8.o\n \n BUILTIN_OBJS = \\\n \tbuiltin-add.o \\\ndiff --git a/utf8.c b/utf8.c\nnew file mode 100644\nindex 0000000..06a66c7\n--- /dev/null\n+++ b/utf8.c\n@@ -0,0 +1,93 @@\n+#include \"git-compat-util.h\"\n+#include \"utf8.h\"\n+\n+/*\n+ * This function returns the number of bytes occupied by the character\n+ * pointed to by the variable start. If it is not valid UTF-8, it\n+ * returns -1.\n+ */\n+int utf8_byte_count(const char *start)\n+{\n+\tunsigned char c = *(unsigned char *)start;\n+\tint i, count = 0;\n+\n+\tif (!(c & 0x80))\n+\t\tcount = 1;\n+\telse if ((c & 0xe0) == 0xc0)\n+\t\tcount = 2;\n+\telse if ((c & 0xf0) == 0xe0)\n+\t\tcount = 3;\n+\telse if ((c & 0xf8) == 0xf0)\n+\t\tcount = 4;\n+\telse\n+\t\treturn -1;\n+\n+\tfor (i = 1; i < count; i++)\n+\t\tif ((start[i] & 0xc0) != 0x80)\n+\t\t\treturn -1;\n+\treturn count;\n+}\n+\n+int utf8_strlen(const char *text)\n+{\n+\tint len = 0;\n+\twhile (*text) {\n+\t\tint count = utf8_byte_count(text);\n+\t\tif (count < 0)\n+\t\t\treturn -1;\n+\t\tlen += count;\n+\t\ttext += count;\n+\t}\n+\treturn len;\n+}\n+\n+static void print_spaces(int count)\n+{\n+\tstatic const char s[] = \"                    \";\n+\twhile (count >= sizeof(s)) {\n+\t\tfwrite(s, sizeof(s) - 1, 1, stdout);\n+\t\tcount -= sizeof(s) - 1;\n+\t}\n+\tfwrite(s, count, 1, stdout);\n+}\n+\n+/*\n+ * Wrap the text, if necessary. The variable indent is the indent for the\n+ * first line, indent2 is the indent for all other lines.\n+ */\n+void print_wrapped_text(const char *text, int indent, int indent2, int len)\n+{\n+\tint count = 0, space = -1;\n+\tint l = utf8_strlen(text), assume_utf8 = (l >= 0);\n+\n+\tl = indent;\n+\n+\tfor (;;) {\n+\t\tchar c = text[count];\n+\t\tif (!c || isspace(c)) {\n+\t\t\tif (l < len || space < 0) {\n+\t\t\t\tconst char *start = text;\n+\t\t\t\tif (space >= 0)\n+\t\t\t\t\tstart += space;\n+\t\t\t\telse\n+\t\t\t\t\tprint_spaces(indent);\n+\t\t\t\tfwrite(start, text + count - start, 1, stdout);\n+\t\t\t\tif (!c) {\n+\t\t\t\t\tputchar('\\n');\n+\t\t\t\t\treturn;\n+\t\t\t\t} else if (c == '\\t')\n+\t\t\t\t\tl |= 0x07;\n+\t\t\t\tspace = count;\n+\t\t\t} else {\n+\t\t\t\tputchar('\\n');\n+\t\t\t\ttext += space + 1;\n+\t\t\t\tindent = indent2;\n+\t\t\t\tspace = -1;\n+\t\t\t\tcount = l = 0;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\t\tcount += assume_utf8 ? utf8_byte_count(text + count) : 1;\n+\t\tl++;\n+\t}\n+}\ndiff --git a/utf8.h b/utf8.h\nnew file mode 100644\nindex 0000000..96dded9\n--- /dev/null\n+++ b/utf8.h\n@@ -0,0 +1,8 @@\n+#ifndef GIT_UTF8_H\n+#define GIT_UTF8_H\n+\n+int utf8_byte_count(const char *start);\n+int utf8_strlen(const char *text);\n+void print_wrapped_text(const char *text, int indent, int indent2, int len);\n+\n+#endif\n-- \n1.4.4.3.ge5f98-dirty\n"},{"id":"30101","messageId":"Pine.LNX.4.63.0612222204010.19693@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6051","inReplyTo":"7vslf7zrdp.fsf@assigned-by-dhcp.cox.net","subject":"[PATCH 2/2] git-commit-tree: if i18n.commitencoding is utf-8 (default), check it","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-22T21:06:08Z","receivedAt":"2006-12-22T21:06:08Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"\nNow, git-commit-tree refuses to commit when i18n.commitencoding is\neither unset, or set to \"utf-8\", and the commit message does not\nminimally conform to the UTF-8 encoding.\n\nSigned-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>\n---\n\n\tUnfortunately, I could not think of a shorter oneline description. \n\tBut my next patch fixes at least the output in shortlog.\n\n builtin-commit-tree.c |   14 ++++++++++++--\n 1 files changed, 12 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin-commit-tree.c b/builtin-commit-tree.c\nindex 856f3cd..810b440 100644\n--- a/builtin-commit-tree.c\n+++ b/builtin-commit-tree.c\n@@ -7,6 +7,7 @@\n #include \"commit.h\"\n #include \"tree.h\"\n #include \"builtin.h\"\n+#include \"utf8.h\"\n \n #define BLOCKING (1ul << 14)\n \n@@ -32,7 +33,7 @@ static void add_buffer(char **bufp, unsigned int *sizep, const char *fmt, ...)\n \tlen = vsnprintf(one_line, sizeof(one_line), fmt, args);\n \tva_end(args);\n \tsize = *sizep;\n-\tnewsize = size + len;\n+\tnewsize = size + len + 1;\n \talloc = (size + 32767) & ~32767;\n \tbuf = *bufp;\n \tif (newsize > alloc) {\n@@ -40,7 +41,7 @@ static void add_buffer(char **bufp, unsigned int *sizep, const char *fmt, ...)\n \t\tbuf = xrealloc(buf, alloc);\n \t\t*bufp = buf;\n \t}\n-\t*sizep = newsize;\n+\t*sizep = newsize - 1;\n \tmemcpy(buf + size, one_line, len);\n }\n \n@@ -127,6 +128,15 @@ int cmd_commit_tree(int argc, const char **argv, const char *prefix)\n \twhile (fgets(comment, sizeof(comment), stdin) != NULL)\n \t\tadd_buffer(&buffer, &size, \"%s\", comment);\n \n+\t/* And check the encoding */\n+\tbuffer[size] = '\\0';\n+\tif (!strcmp(git_commit_encoding, \"utf-8\") && utf8_strlen(buffer) < 0) {\n+\t\tfprintf(stderr, \"Commit message does not conform to UTF-8.\\n\"\n+\t\t\t\"Please fix the message,\"\n+\t\t\t\" or set the config variable i18n.commitencoding.\\n\");\n+\t\treturn 1;\n+\t}\n+\n \tif (!write_sha1_file(buffer, size, commit_type, commit_sha1)) {\n \t\tprintf(\"%s\\n\", sha1_to_hex(commit_sha1));\n \t\treturn 0;\n-- \n1.4.4.3.ge5f98-dirty\n"},{"id":"30103","messageId":"Pine.LNX.4.63.0612222208310.19693@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6051","inReplyTo":"7vslf7zrdp.fsf@assigned-by-dhcp.cox.net","subject":"[RFC/PATCH 3/2] Wrap lines in shortlog","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-22T21:15:59Z","receivedAt":"2006-12-22T21:15:59Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nIt is nicer to wrap the lines of too long oneline descriptions. This patch \neven works in UTF-8.\n\nThe patch is attached, since I cannot find the setting in pine to make it \na UTF-8 one. Besides, I deliberately fscked up one test case.\n\nCiao,\nDscho\n\n\n[PATCH] Use print_wrapped_text() in shortlog\n\nSome oneline descriptions are just too long. In shortlog, it looks much\nnicer when they are wrapped. Since print_wrapped_text() is UTF-8 aware,\nit also works with those descriptions.\n\nSigned-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>\n\n---\n\n\tProbably this should check i18n.commitencoding, too...\n\n builtin-shortlog.c  |    4 +++-\n t/t4201-shortlog.sh |   50 ++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 53 insertions(+), 1 deletions(-)\n\ndiff --git a/builtin-shortlog.c b/builtin-shortlog.c\nindex edb4042..30e7cb5 100644\n--- a/builtin-shortlog.c\n+++ b/builtin-shortlog.c\n@@ -4,6 +4,7 @@\n #include \"diff.h\"\n #include \"path-list.h\"\n #include \"revision.h\"\n+#include \"utf8.h\"\n \n static const char shortlog_usage[] =\n \"git-shortlog [-n] [-s] [<commit-id>... ]\";\n@@ -321,7 +322,8 @@ int cmd_shortlog(int argc, const char **argv, const char *prefix)\n \t\t} else {\n \t\t\tprintf(\"%s (%d):\\n\", list.items[i].path, onelines->nr);\n \t\t\tfor (j = onelines->nr - 1; j >= 0; j--)\n-\t\t\t\tprintf(\"      %s\\n\", onelines->items[j].path);\n+\t\t\t\tprint_wrapped_text(onelines->items[j].path,\n+\t\t\t\t\t6, 9, 76);\n \t\t\tprintf(\"\\n\");\n \t\t}\n \ndiff --git a/t/t4201-shortlog.sh b/t/t4201-shortlog.sh\nnew file mode 100644\nindex 0000000..e4085f9\n--- /dev/null\n+++ b/t/t4201-shortlog.sh\n@@ -0,0 +1,50 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2006 Johannes E. Schindelin\n+#\n+\n+test_description='git-shortlog\n+'\n+\n+. ./test-lib.sh\n+\n+echo 1 > a1\n+git add a1\n+tree=$(git write-tree)\n+commit=$((echo \"Test\"; echo) | git commit-tree $tree)\n+git update-ref HEAD $commit \n+\n+echo 2 > a1\n+git commit -m \"This is a very, very long first line for the commit message to see if it is wrapped correctly\" a1\n+\n+# test if the wrapping is still valid when replacing all i's by treble clefs.\n+echo 3 > a1\n+git commit -m \"$(echo \"This is a very, very long first line for the commit message to see if it is wrapped correctly\" | sed \"s/i/1234/g\" | tr 1234 '\\360\\235\\204\\236')\" a1\n+\n+# now fsck up the utf8\n+git repo-config i18n.commitencoding non-utf-8\n+echo 4 > a1\n+git commit -m \"$(echo \"This is a very, very long first line for the commit message to see if it is wrapped correctly\" | sed \"s/i/1234/g\" | tr 1234 '\\370\\235\\204\\236')\" a1\n+\n+echo 5 > a1\n+git commit -m \"a\t\t\t\t\t\t\t\t12\t34\t56\t78\" a1\n+\n+git shortlog HEAD > out\n+\n+cat > expect << EOF\n+A U Thor (5):\n+      Test\n+      This is a very, very long first line for the commit message to see if\n+         it is wrapped correctly\n+      Th𝄞s 𝄞s a very, very long f𝄞rst l𝄞ne for the comm𝄞t message to see 𝄞f\n+         𝄞t 𝄞s wrapped correctly\n+      Thø„žs ø„žs a very, very long fø„žrst lø„žne for the commø„žt\n+         message to see ø„žf ø„žt ø„žs wrapped correctly\n+      a\t\t\t\t\t\t\t\t12\t34\n+         56\t78\n+\n+EOF\n+\n+test_expect_success 'shortlog wrapping' 'diff -u expect out'\n+\n+test_done\n"},{"id":"30105","messageId":"7virg3y624.fsf@assigned-by-dhcp.cox.net","threadId":"6051","inReplyTo":"Pine.LNX.4.63.0612222201200.19693@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH 1/2] libgit.a: add some UTF-8 handling functions","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-12-22T21:27:47Z","receivedAt":"2006-12-22T21:27:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> This adds utf8_byte_count(), utf8_strlen() and print_wrapped_text().\n>\n> The most important is probably utf8_strlen(), which returns the length\n> of the text, if it is in UTF-8, otherwise -1.\n>\n> Note that we do not go the full nine yards: we could also check that\n> the character is encoded with the minimum amount of bytes, as pointed\n> out by Uwe Kleine-Koenig.\n>\n> The function print_wrapped_text() can be used to wrap text to a certain\n> line length.\n\nIf you do wrapped_text, I think you do not _want_ strlen (the\ndefinition to me of strlen is \"number of characters in the\nstring\").  What you want is a function that returns the number\nof columns consumed when displayed on monospace terminal.\n"},{"id":"30106","messageId":"Pine.LNX.4.63.0612222233150.19693@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6051","inReplyTo":"7virg3y624.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH 1/2] libgit.a: add some UTF-8 handling functions","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-22T21:36:05Z","receivedAt":"2006-12-22T21:36:05Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 22 Dec 2006, Junio C Hamano wrote:\n\n> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n> \n> > This adds utf8_byte_count(), utf8_strlen() and print_wrapped_text().\n> >\n> > The most important is probably utf8_strlen(), which returns the length\n> > of the text, if it is in UTF-8, otherwise -1.\n> >\n> > Note that we do not go the full nine yards: we could also check that\n> > the character is encoded with the minimum amount of bytes, as pointed\n> > out by Uwe Kleine-Koenig.\n> >\n> > The function print_wrapped_text() can be used to wrap text to a certain\n> > line length.\n> \n> If you do wrapped_text, I think you do not _want_ strlen (the\n> definition to me of strlen is \"number of characters in the\n> string\").  What you want is a function that returns the number\n> of columns consumed when displayed on monospace terminal.\n\nTo me, characters are the symbols occupying one \"column\" each. Bytes are \nthe 8-bit thingies that you usually use to encode the characters.\n\nCiao,\nDscho\n"},{"id":"30109","messageId":"7vy7ozwqf5.fsf@assigned-by-dhcp.cox.net","threadId":"6051","inReplyTo":"Pine.LNX.4.63.0612222204010.19693@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH 2/2] git-commit-tree: if i18n.commitencoding is utf-8 (default), check it","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-12-22T21:50:54Z","receivedAt":"2006-12-22T21:50:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> Now, git-commit-tree refuses to commit when i18n.commitencoding is\n> either unset, or set to \"utf-8\", and the commit message does not\n> minimally conform to the UTF-8 encoding.\n>\n> Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>\n> ---\n>\n> \tUnfortunately, I could not think of a shorter oneline description. \n> \tBut my next patch fixes at least the output in shortlog.\n\nI think the rule that you described on the one-line description\nmakes more sense than \"either unset of set to utf-8\".  In other\nwords, I'd prefer doing this in a repository that explicitly\nasks for it.\n\nI do not want to get burned by too many incompatible changes X-<.\n"},{"id":"30111","messageId":"7vpsabwq34.fsf@assigned-by-dhcp.cox.net","threadId":"6051","inReplyTo":"Pine.LNX.4.63.0612222233150.19693@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH 1/2] libgit.a: add some UTF-8 handling functions","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-12-22T21:58:07Z","receivedAt":"2006-12-22T21:58:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n>> If you do wrapped_text, I think you do not _want_ strlen (the\n>> definition to me of strlen is \"number of characters in the\n>> string\").  What you want is a function that returns the number\n>> of columns consumed when displayed on monospace terminal.\n>\n> To me, characters are the symbols occupying one \"column\" each. Bytes are \n> the 8-bit thingies that you usually use to encode the characters.\n\nI cannot tell from your reponse if you are very well aware of\nAsian \"double-width\" characters and your version of strlen()\ncounts one such character as two, or if you are totally unaware\nabout the issue and your function returns 1 for a string that\nconsists of a single such character.\n\nIf the former, then the function is not strlen() anymore, and if\nthe latter, then it is unusable for wrapping purposes.\n"},{"id":"30113","messageId":"20061222220204.GA2407@cepheus","threadId":"6051","inReplyTo":"20061222150948.GA6005@cepheus","subject":"Re: specify charset for commits","fromName":"Uwe Kleine-König","fromEmail":"zeisberg@informatik.uni-freiburg.de","sentAt":"2006-12-22T22:02:04Z","receivedAt":"2006-12-22T22:02:04Z","isPatch":false,"sender":{"key":"u.kleine-koenig@pengutronix.de","avatar":"https://gravatar.com/avatar/354b5e3ceb2806a2f1e1e382ac29ddbdad18288654da62b61eb13583a857eee7?d=mp&s=160"},"body":"Hello,\n\nUwe Kleine-König wrote:\n> \tdef is_utf8_str(s):\n> \t  cnt_furtherbytes = 0\n> \t  for c in s:\n> \t    if cnt_furtherbytes > 0:\n> \t      if ord(c) & 0xc0 == 0x80:\n> \t        cnt_furtherbytes -= 1\n> \t      else:\n> \t        return False\n> \t    else:\n> \t      if ord(c) < 0x80:\n> \t        continue\n> \t      elif ord(c) < 0xc0:\n> \t        return False\n> \t      elif ord(c) < 0xe0:\n> \t        cnt_furtherbytes = 1\n> \t      elif ord(c) < 0xf0:\n> \t        cnt_furtherbytes = 2\n> \t      elif ord(c) < 0xf8:\n> \t        cnt_furtherbytes = 3\n> \t      elif ord(c) < 0xfc:\n> \t        cnt_furtherbytes = 4\n> \t      elif ord(c) < 0xfe:\n> \t        cnt_furtherbytes = 5\n> \t      else:\n> \t        return False\n> \t  return True\nWhile I washed the dishes I noticed that the last \"return True\" should\nbe \"return cnt_furtherbytes == 0\".  Just before someone else corrects me\n... :-)\n\nBest regards\nUwe\n\n-- \nUwe Kleine-König\n\nhttp://www.google.com/search?q=parsec%5E2*Joule%2FNewton+in+tablespoon\n"},{"id":"30114","messageId":"20061222221448.GB2407@cepheus","threadId":"6051","inReplyTo":"Pine.LNX.4.63.0612222233150.19693@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH 1/2] libgit.a: add some UTF-8 handling functions","fromName":"Uwe Kleine-König","fromEmail":"zeisberg@informatik.uni-freiburg.de","sentAt":"2006-12-22T22:14:48Z","receivedAt":"2006-12-22T22:14:48Z","isPatch":true,"sender":{"key":"u.kleine-koenig@pengutronix.de","avatar":"https://gravatar.com/avatar/354b5e3ceb2806a2f1e1e382ac29ddbdad18288654da62b61eb13583a857eee7?d=mp&s=160"},"body":"Hello Johannes,\n\nJohannes Schindelin wrote:\n> On Fri, 22 Dec 2006, Junio C Hamano wrote:\n> \n> > Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n> > \n> > > This adds utf8_byte_count(), utf8_strlen() and print_wrapped_text().\n> > >\n> > > The most important is probably utf8_strlen(), which returns the length\n> > > of the text, if it is in UTF-8, otherwise -1.\n> > >\n> > > Note that we do not go the full nine yards: we could also check that\n> > > the character is encoded with the minimum amount of bytes, as pointed\n> > > out by Uwe Kleine-Koenig.\n> > >\n> > > The function print_wrapped_text() can be used to wrap text to a certain\n> > > line length.\n> > \n> > If you do wrapped_text, I think you do not _want_ strlen (the\n> > definition to me of strlen is \"number of characters in the\n> > string\").  What you want is a function that returns the number\n> > of columns consumed when displayed on monospace terminal.\n> \n> To me, characters are the symbols occupying one \"column\" each. Bytes are \n> the 8-bit thingies that you usually use to encode the characters.\nQuoting utf-8(7):\n\n\tare no longer valid in UTF-8 locales.  Firstly, a single byte\n\tdoes not necessarily correspond any more to a single character.\n\tSecondly, since modern terminal emulators in UTF-8 mode also\n\tsupport Chinese, Japanese, and Korean double-width characters as\n\twell as non-spacing combining characters, outputting a single\n\tcharacter does not necessarily advance the cursor by one\n\tposition as it did in ASCII.  Library functions such as\n\tmbsrtowcs(3) and wcswidth(3) should be used today to count\n\tcharacters and cursor positions.\n\nI'd prefer using a similar naming scheme.  To acknowledge Junio,\nwcslen(3) (the wide-character equivalent of the strlen() function)\ncounts the number of (wide-)characters in a string.\n\nBest regards,\nUwe\n\n-- \nUwe Kleine-König\n\nhttp://www.google.com/search?q=e+%5E+%28i+pi%29\n"},{"id":"30115","messageId":"20061222221913.GA3071@cepheus","threadId":"6051","inReplyTo":"Pine.LNX.4.63.0612222201200.19693@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH 1/2] libgit.a: add some UTF-8 handling functions","fromName":"Uwe Kleine-König","fromEmail":"zeisberg@informatik.uni-freiburg.de","sentAt":"2006-12-22T22:19:13Z","receivedAt":"2006-12-22T22:19:13Z","isPatch":true,"sender":{"key":"u.kleine-koenig@pengutronix.de","avatar":"https://gravatar.com/avatar/354b5e3ceb2806a2f1e1e382ac29ddbdad18288654da62b61eb13583a857eee7?d=mp&s=160"},"body":"Hello,\n\nJohannes Schindelin wrote:\n> Note that we do not go the full nine yards: we could also check that\n> the character is encoded with the minimum amount of bytes, as pointed\n> out by Uwe Kleine-Koenig.\nWhile we're talking about UTF-8 in commit-logs:  I'd prefer to have my\nname properly written with o-umlaut.\n\nBest regards\nUwe\n\n-- \nUwe Kleine-König\n\nhttp://www.google.com/search?q=0+degree+Celsius+in+kelvin\n"},{"id":"30116","messageId":"Pine.LNX.4.63.0612222319230.19693@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6051","inReplyTo":"7vpsabwq34.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH 1/2] libgit.a: add some UTF-8 handling functions","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-22T22:20:12Z","receivedAt":"2006-12-22T22:20:12Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 22 Dec 2006, Junio C Hamano wrote:\n\n> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n> \n> >> If you do wrapped_text, I think you do not _want_ strlen (the\n> >> definition to me of strlen is \"number of characters in the\n> >> string\").  What you want is a function that returns the number\n> >> of columns consumed when displayed on monospace terminal.\n> >\n> > To me, characters are the symbols occupying one \"column\" each. Bytes are \n> > the 8-bit thingies that you usually use to encode the characters.\n> \n> I cannot tell from your reponse if you are very well aware of\n> Asian \"double-width\" characters and your version of strlen()\n> counts one such character as two, or if you are totally unaware\n> about the issue and your function returns 1 for a string that\n> consists of a single such character.\n> \n> If the former, then the function is not strlen() anymore, and if\n> the latter, then it is unusable for wrapping purposes.\n\nThe latter. Oh, well. Call me a Western idiot.\n\nAnd scrap that patch.\n\nCiao,\nDscho\n"},{"id":"30117","messageId":"Pine.LNX.4.63.0612222320580.19693@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6051","inReplyTo":"7vy7ozwqf5.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH 2/2] git-commit-tree: if i18n.commitencoding is utf-8 (default), check it","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-22T22:21:50Z","receivedAt":"2006-12-22T22:21:50Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 22 Dec 2006, Junio C Hamano wrote:\n\n> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n> \n> > Now, git-commit-tree refuses to commit when i18n.commitencoding is\n> > either unset, or set to \"utf-8\", and the commit message does not\n> > minimally conform to the UTF-8 encoding.\n> >\n> > Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>\n> > ---\n> >\n> > \tUnfortunately, I could not think of a shorter oneline description. \n> > \tBut my next patch fixes at least the output in shortlog.\n> \n> I think the rule that you described on the one-line description\n> makes more sense than \"either unset of set to utf-8\".  In other\n> words, I'd prefer doing this in a repository that explicitly\n> asks for it.\n\nWell, the problem is this line:\n\nenvironment.c:21:char git_commit_encoding[MAX_ENCODING_LENGTH] = \"utf-8\";\n\n> I do not want to get burned by too many incompatible changes X-<.\n\nUnderstandable.\n\nCiao,\nDscho\n"},{"id":"30119","messageId":"7vlkkzwogy.fsf@assigned-by-dhcp.cox.net","threadId":"6051","inReplyTo":"Pine.LNX.4.63.0612222319230.19693@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH 1/2] libgit.a: add some UTF-8 handling functions","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-12-22T22:33:01Z","receivedAt":"2006-12-22T22:33:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n>> > To me, characters are the symbols occupying one \"column\" each. Bytes are \n>> > the 8-bit thingies that you usually use to encode the characters.\n>> \n>> I cannot tell from your reponse if you are very well aware of\n>> Asian \"double-width\" characters and your version of strlen()\n>> counts one such character as two, or if you are totally unaware\n>> about the issue and your function returns 1 for a string that\n>> consists of a single such character.\n>> \n>> If the former, then the function is not strlen() anymore, and if\n>> the latter, then it is unusable for wrapping purposes.\n>\n> The latter. Oh, well. Call me a Western idiot.\n>\n> And scrap that patch.\n\nHey, don't give up too quickly.  Although the execution of the\ninitial revision might have been less than desirable, the series\nreally meant well and was in the right direction.  We do that\nall the time ;-).\n"},{"id":"30121","messageId":"Pine.LNX.4.63.0612222331581.19693@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6051","inReplyTo":"20061222221913.GA3071@cepheus","subject":"Re: [PATCH 1/2] libgit.a: add some UTF-8 handling functions","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-22T22:34:45Z","receivedAt":"2006-12-22T22:34:45Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Dear Mr Zeisberg,\n\nOn Fri, 22 Dec 2006, Uwe Kleine-König wrote:\n\n> Johannes Schindelin wrote:\n> > Note that we do not go the full nine yards: we could also check that\n> > the character is encoded with the minimum amount of bytes, as pointed\n> > out by Uwe Kleine-Koenig.\n> While we're talking about UTF-8 in commit-logs:  I'd prefer to have my\n> name properly written with o-umlaut.\n\nI did this because I have no easy way to input UTF-8, and because I am \nlazy, and because I did not know how many times this patch has to be \nrevised.\n\nApart from that, it seems that the checking of UTF-8 is actually quite \nsimple, and we could even copy it from \nhttp://www.cl.cam.ac.uk/~mgk25/ucs/utf8_check.c, where the check you \nproposed is included.\n\nBut I had enough of UTF-8 for a day.\n\nCiao,\nDscho\n"},{"id":"30130","messageId":"Pine.LNX.4.63.0612230048350.19693@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6051","inReplyTo":"Pine.LNX.4.63.0612222331581.19693@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH 1/2] libgit.a: add some UTF-8 handling functions","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-22T23:50:21Z","receivedAt":"2006-12-22T23:50:21Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 22 Dec 2006, Johannes Schindelin wrote:\n\n> Dear Mr Zeisberg,\n> \n> On Fri, 22 Dec 2006, Uwe Kleine-König wrote:\n> \n> > Johannes Schindelin wrote:\n> > > Note that we do not go the full nine yards: we could also check that\n> > > the character is encoded with the minimum amount of bytes, as pointed\n> > > out by Uwe Kleine-Koenig.\n> > While we're talking about UTF-8 in commit-logs:  I'd prefer to have my\n> > name properly written with o-umlaut.\n> \n> I did this because I have no easy way to input UTF-8, and because I am \n> lazy, and because I did not know how many times this patch has to be \n> revised.\n> \n> Apart from that, it seems that the checking of UTF-8 is actually quite \n> simple, and we could even copy it from \n> http://www.cl.cam.ac.uk/~mgk25/ucs/utf8_check.c, where the check you \n> proposed is included.\n> \n> But I had enough of UTF-8 for a day.\n\nOkay, so I lied (this are both patches revised and combined):\n\n--\n Makefile              |    6 +\n builtin-commit-tree.c |   14 ++\n utf8.c                |  277 +++++++++++++++++++++++++++++++++++++++++++++++++\n utf8.h                |    8 +\n 4 files changed, 301 insertions(+), 4 deletions(-)\n\ndiff --git a/Makefile b/Makefile\n--- a/Makefile\n+++ b/Makefile\n@@ -234,7 +237,8 @@ LIB_H = \\\n \tarchive.h blob.h cache.h commit.h csum-file.h delta.h grep.h \\\n \tdiff.h object.h pack.h pkt-line.h quote.h refs.h list-objects.h sideband.h \\\n \trun-command.h strbuf.h tag.h tree.h git-compat-util.h revision.h \\\n-\ttree-walk.h log-tree.h dir.h path-list.h unpack-trees.h builtin.h\n+\ttree-walk.h log-tree.h dir.h path-list.h unpack-trees.h builtin.h \\\n+\tutf8.h\n \n DIFF_OBJS = \\\n \tdiff.o diff-lib.o diffcore-break.o diffcore-order.o \\\n@@ -253,7 +257,8 @@ LIB_OBJS = \\\n \trevision.o pager.o tree-walk.o xdiff-interface.o \\\n \twrite_or_die.o trace.o list-objects.o grep.o \\\n \talloc.o merge-file.o path-list.o help.o unpack-trees.o $(DIFF_OBJS) \\\n-\tcolor.o wt-status.o archive-zip.o archive-tar.o shallow.o\n+\tcolor.o wt-status.o archive-zip.o archive-tar.o shallow.o \\\n+\tutf8.o\n \n BUILTIN_OBJS = \\\n \tbuiltin-add.o \\\ndiff --git a/builtin-commit-tree.c b/builtin-commit-tree.c\nindex 856f3cd..ef7cc91 100644\n--- a/builtin-commit-tree.c\n+++ b/builtin-commit-tree.c\n@@ -7,6 +7,7 @@\n #include \"commit.h\"\n #include \"tree.h\"\n #include \"builtin.h\"\n+#include \"utf8.h\"\n \n #define BLOCKING (1ul << 14)\n \n@@ -32,7 +33,7 @@ static void add_buffer(char **bufp, unsigned int *sizep, const char *fmt, ...)\n \tlen = vsnprintf(one_line, sizeof(one_line), fmt, args);\n \tva_end(args);\n \tsize = *sizep;\n-\tnewsize = size + len;\n+\tnewsize = size + len + 1;\n \talloc = (size + 32767) & ~32767;\n \tbuf = *bufp;\n \tif (newsize > alloc) {\n@@ -40,7 +41,7 @@ static void add_buffer(char **bufp, unsigned int *sizep, const char *fmt, ...)\n \t\tbuf = xrealloc(buf, alloc);\n \t\t*bufp = buf;\n \t}\n-\t*sizep = newsize;\n+\t*sizep = newsize - 1;\n \tmemcpy(buf + size, one_line, len);\n }\n \n@@ -127,6 +128,15 @@ int cmd_commit_tree(int argc, const char **argv, const char *prefix)\n \twhile (fgets(comment, sizeof(comment), stdin) != NULL)\n \t\tadd_buffer(&buffer, &size, \"%s\", comment);\n \n+\t/* And check the encoding */\n+\tbuffer[size] = '\\0';\n+\tif (!strcmp(git_commit_encoding, \"utf-8\") && !is_utf8(buffer)) {\n+\t\tfprintf(stderr, \"Commit message does not conform to UTF-8.\\n\"\n+\t\t\t\"Please fix the message,\"\n+\t\t\t\" or set the config variable i18n.commitencoding.\\n\");\n+\t\treturn 1;\n+\t}\n+\n \tif (!write_sha1_file(buffer, size, commit_type, commit_sha1)) {\n \t\tprintf(\"%s\\n\", sha1_to_hex(commit_sha1));\n \t\treturn 0;\ndiff --git a/utf8.c b/utf8.c\nnew file mode 100644\nindex 0000000..8daec78\n--- /dev/null\n+++ b/utf8.c\n@@ -0,0 +1,277 @@\n+#include \"git-compat-util.h\"\n+#include \"utf8.h\"\n+\n+/* This code is originally from http://www.cl.cam.ac.uk/~mgk25/ucs/ */\n+\n+struct interval {\n+  int first;\n+  int last;\n+};\n+\n+/* auxiliary function for binary search in interval table */\n+static int bisearch(wchar_t ucs, const struct interval *table, int max) {\n+\tint min = 0;\n+\tint mid;\n+\n+\tif (ucs < table[0].first || ucs > table[max].last)\n+\t\treturn 0;\n+\twhile (max >= min) {\n+\t\tmid = (min + max) / 2;\n+\t\tif (ucs > table[mid].last)\n+\t\t\tmin = mid + 1;\n+\t\telse if (ucs < table[mid].first)\n+\t\t\tmax = mid - 1;\n+\t\telse\n+\t\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/* The following two functions define the column width of an ISO 10646\n+ * character as follows:\n+ *\n+ *    - The null character (U+0000) has a column width of 0.\n+ *\n+ *    - Other C0/C1 control characters and DEL will lead to a return\n+ *      value of -1.\n+ *\n+ *    - Non-spacing and enclosing combining characters (general\n+ *      category code Mn or Me in the Unicode database) have a\n+ *      column width of 0.\n+ *\n+ *    - SOFT HYPHEN (U+00AD) has a column width of 1.\n+ *\n+ *    - Other format characters (general category code Cf in the Unicode\n+ *      database) and ZERO WIDTH SPACE (U+200B) have a column width of 0.\n+ *\n+ *    - Hangul Jamo medial vowels and final consonants (U+1160-U+11FF)\n+ *      have a column width of 0.\n+ *\n+ *    - Spacing characters in the East Asian Wide (W) or East Asian\n+ *      Full-width (F) category as defined in Unicode Technical\n+ *      Report #11 have a column width of 2.\n+ *\n+ *    - All remaining characters (including all printable\n+ *      ISO 8859-1 and WGL4 characters, Unicode control characters,\n+ *      etc.) have a column width of 1.\n+ *\n+ * This implementation assumes that wchar_t characters are encoded\n+ * in ISO 10646.\n+ */\n+\n+static int wcwidth(wchar_t ch)\n+{\n+\t/*\n+\t * Sorted list of non-overlapping intervals of non-spacing characters,\n+\t * generated by\n+\t *   \"uniset +cat=Me +cat=Mn +cat=Cf -00AD +1160-11FF +200B c\".\n+\t */\n+\tstatic const struct interval combining[] = {\n+\t\t{ 0x0300, 0x0357 }, { 0x035D, 0x036F }, { 0x0483, 0x0486 },\n+\t\t{ 0x0488, 0x0489 }, { 0x0591, 0x05A1 }, { 0x05A3, 0x05B9 },\n+\t\t{ 0x05BB, 0x05BD }, { 0x05BF, 0x05BF }, { 0x05C1, 0x05C2 },\n+\t\t{ 0x05C4, 0x05C4 }, { 0x0600, 0x0603 }, { 0x0610, 0x0615 },\n+\t\t{ 0x064B, 0x0658 }, { 0x0670, 0x0670 }, { 0x06D6, 0x06E4 },\n+\t\t{ 0x06E7, 0x06E8 }, { 0x06EA, 0x06ED }, { 0x070F, 0x070F },\n+\t\t{ 0x0711, 0x0711 }, { 0x0730, 0x074A }, { 0x07A6, 0x07B0 },\n+\t\t{ 0x0901, 0x0902 }, { 0x093C, 0x093C }, { 0x0941, 0x0948 },\n+\t\t{ 0x094D, 0x094D }, { 0x0951, 0x0954 }, { 0x0962, 0x0963 },\n+\t\t{ 0x0981, 0x0981 }, { 0x09BC, 0x09BC }, { 0x09C1, 0x09C4 },\n+\t\t{ 0x09CD, 0x09CD }, { 0x09E2, 0x09E3 }, { 0x0A01, 0x0A02 },\n+\t\t{ 0x0A3C, 0x0A3C }, { 0x0A41, 0x0A42 }, { 0x0A47, 0x0A48 },\n+\t\t{ 0x0A4B, 0x0A4D }, { 0x0A70, 0x0A71 }, { 0x0A81, 0x0A82 },\n+\t\t{ 0x0ABC, 0x0ABC }, { 0x0AC1, 0x0AC5 }, { 0x0AC7, 0x0AC8 },\n+\t\t{ 0x0ACD, 0x0ACD }, { 0x0AE2, 0x0AE3 }, { 0x0B01, 0x0B01 },\n+\t\t{ 0x0B3C, 0x0B3C }, { 0x0B3F, 0x0B3F }, { 0x0B41, 0x0B43 },\n+\t\t{ 0x0B4D, 0x0B4D }, { 0x0B56, 0x0B56 }, { 0x0B82, 0x0B82 },\n+\t\t{ 0x0BC0, 0x0BC0 }, { 0x0BCD, 0x0BCD }, { 0x0C3E, 0x0C40 },\n+\t\t{ 0x0C46, 0x0C48 }, { 0x0C4A, 0x0C4D }, { 0x0C55, 0x0C56 },\n+\t\t{ 0x0CBC, 0x0CBC }, { 0x0CBF, 0x0CBF }, { 0x0CC6, 0x0CC6 },\n+\t\t{ 0x0CCC, 0x0CCD }, { 0x0D41, 0x0D43 }, { 0x0D4D, 0x0D4D },\n+\t\t{ 0x0DCA, 0x0DCA }, { 0x0DD2, 0x0DD4 }, { 0x0DD6, 0x0DD6 },\n+\t\t{ 0x0E31, 0x0E31 }, { 0x0E34, 0x0E3A }, { 0x0E47, 0x0E4E },\n+\t\t{ 0x0EB1, 0x0EB1 }, { 0x0EB4, 0x0EB9 }, { 0x0EBB, 0x0EBC },\n+\t\t{ 0x0EC8, 0x0ECD }, { 0x0F18, 0x0F19 }, { 0x0F35, 0x0F35 },\n+\t\t{ 0x0F37, 0x0F37 }, { 0x0F39, 0x0F39 }, { 0x0F71, 0x0F7E },\n+\t\t{ 0x0F80, 0x0F84 }, { 0x0F86, 0x0F87 }, { 0x0F90, 0x0F97 },\n+\t\t{ 0x0F99, 0x0FBC }, { 0x0FC6, 0x0FC6 }, { 0x102D, 0x1030 },\n+\t\t{ 0x1032, 0x1032 }, { 0x1036, 0x1037 }, { 0x1039, 0x1039 },\n+\t\t{ 0x1058, 0x1059 }, { 0x1160, 0x11FF }, { 0x1712, 0x1714 },\n+\t\t{ 0x1732, 0x1734 }, { 0x1752, 0x1753 }, { 0x1772, 0x1773 },\n+\t\t{ 0x17B4, 0x17B5 }, { 0x17B7, 0x17BD }, { 0x17C6, 0x17C6 },\n+\t\t{ 0x17C9, 0x17D3 }, { 0x17DD, 0x17DD }, { 0x180B, 0x180D },\n+\t\t{ 0x18A9, 0x18A9 }, { 0x1920, 0x1922 }, { 0x1927, 0x1928 },\n+\t\t{ 0x1932, 0x1932 }, { 0x1939, 0x193B }, { 0x200B, 0x200F },\n+\t\t{ 0x202A, 0x202E }, { 0x2060, 0x2063 }, { 0x206A, 0x206F },\n+\t\t{ 0x20D0, 0x20EA }, { 0x302A, 0x302F }, { 0x3099, 0x309A },\n+\t\t{ 0xFB1E, 0xFB1E }, { 0xFE00, 0xFE0F }, { 0xFE20, 0xFE23 },\n+\t\t{ 0xFEFF, 0xFEFF }, { 0xFFF9, 0xFFFB }, { 0x1D167, 0x1D169 },\n+\t\t{ 0x1D173, 0x1D182 }, { 0x1D185, 0x1D18B },\n+\t\t{ 0x1D1AA, 0x1D1AD }, { 0xE0001, 0xE0001 },\n+\t\t{ 0xE0020, 0xE007F }, { 0xE0100, 0xE01EF }\n+\t};\n+\n+\t/* test for 8-bit control characters */\n+\tif (ch == 0)\n+\t\treturn 0;\n+\tif (ch < 32 || (ch >= 0x7f && ch < 0xa0))\n+\t\treturn -1;\n+\n+\t/* binary search in table of non-spacing characters */\n+\tif (bisearch(ch, combining, sizeof(combining)\n+\t\t\t\t/ sizeof(struct interval) - 1))\n+\t\treturn 0;\n+\n+\t/*\n+\t * If we arrive here, ch is neither a combining nor a C0/C1\n+\t * control character.\n+\t */\n+\n+\treturn 1 +\n+\t\t(ch >= 0x1100 &&\n+                    /* Hangul Jamo init. consonants */\n+\t\t (ch <= 0x115f ||\n+\t\t  ch == 0x2329 || ch == 0x232a ||\n+                  /* CJK ... Yi */\n+\t\t  (ch >= 0x2e80 && ch <= 0xa4cf &&\n+\t\t   ch != 0x303f) ||\n+\t\t  /* Hangul Syllables */\n+\t\t  (ch >= 0xac00 && ch <= 0xd7a3) ||\n+\t\t  /* CJK Compatibility Ideographs */\n+\t\t  (ch >= 0xf900 && ch <= 0xfaff) ||\n+\t\t  /* CJK Compatibility Forms */\n+\t\t  (ch >= 0xfe30 && ch <= 0xfe6f) ||\n+\t\t  /* Fullwidth Forms */\n+\t\t  (ch >= 0xff00 && ch <= 0xff60) ||\n+\t\t  (ch >= 0xffe0 && ch <= 0xffe6) ||\n+\t\t  (ch >= 0x20000 && ch <= 0x2fffd) ||\n+\t\t  (ch >= 0x30000 && ch <= 0x3fffd)));\n+}\n+\n+/*\n+ * This function returns the number of columns occupied by the character\n+ * pointed to by the variable start. The pointer is updated to point at\n+ * the next character. If it was not valid UTF-8, the pointer is set to NULL.\n+ */\n+int utf8_width(const char **start)\n+{\n+\tunsigned char *s = (unsigned char *)*start;\n+\twchar_t ch;\n+\n+\tif (*s < 0x80) {\n+\t\t/* 0xxxxxxx */\n+\t\tch = *s;\n+\t\t*start += 1;\n+\t} else if ((s[0] & 0xe0) == 0xc0) {\n+\t\t/* 110XXXXx 10xxxxxx */\n+\t\tif ((s[1] & 0xc0) != 0x80 ||\n+\t\t\t\t/* overlong? */\n+\t\t\t\t(s[0] & 0xfe) == 0xc0)\n+\t\t\tgoto invalid;\n+\t\tch = ((s[0] & 0x1f) << 6) | (s[1] & 0x3f);\n+\t\t*start += 2;\n+\t} else if ((s[0] & 0xf0) == 0xe0) {\n+\t\t/* 1110XXXX 10Xxxxxx 10xxxxxx */\n+\t\tif ((s[1] & 0xc0) != 0x80 ||\n+\t\t\t\t(s[2] & 0xc0) != 0x80 ||\n+\t\t\t\t/* overlong? */\n+\t\t\t\t(s[0] == 0xe0 && (s[1] & 0xe0) == 0x80) ||\n+\t\t\t\t/* surrogate? */\n+\t\t\t\t(s[0] == 0xed && (s[1] & 0xe0) == 0xa0) ||\n+\t\t\t\t/* U+FFFE or U+FFFF? */\n+\t\t\t\t(s[0] == 0xef && s[1] == 0xbf &&\n+\t\t\t\t (s[2] & 0xfe) == 0xbe))\n+\t\t\tgoto invalid;\n+\t\tch = ((s[0] & 0x0f) << 12) |\n+\t\t\t((s[1] & 0x3f) << 6) | (s[2] & 0x3f);\n+\t\t*start += 3;\n+\t} else if ((s[0] & 0xf8) == 0xf0) {\n+\t\t/* 11110XXX 10XXxxxx 10xxxxxx 10xxxxxx */\n+\t\tif ((s[1] & 0xc0) != 0x80 ||\n+\t\t\t\t(s[2] & 0xc0) != 0x80 ||\n+\t\t\t\t(s[3] & 0xc0) != 0x80 ||\n+\t\t\t\t/* overlong? */\n+\t\t\t\t(s[0] == 0xf0 && (s[1] & 0xf0) == 0x80) ||\n+\t\t\t\t/* > U+10FFFF? */\n+\t\t\t\t(s[0] == 0xf4 && s[1] > 0x8f) || s[0] > 0xf4)\n+\t\t\tgoto invalid;\n+\t\tch = ((s[0] & 0x07) << 18) | ((s[1] & 0x3f) << 12) |\n+\t\t\t((s[2] & 0x3f) << 6) | (s[3] & 0x3f);\n+\t\t*start += 4;\n+\t} else {\n+invalid:\n+\t\t*start = NULL;\n+\t\treturn 0;\n+\t}\n+\n+\treturn wcwidth(ch);\n+}\n+\n+int is_utf8(const char *text)\n+{\n+\twhile (*text) {\n+\t\tif (*text == '\\n' || *text == '\\t' || *text == '\\r') {\n+\t\t\ttext++;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tutf8_width(&text);\n+\t\tif (!text)\n+\t\t\treturn 0;\n+\t}\n+\treturn 1;\n+}\n+\n+static void print_spaces(int count)\n+{\n+\tstatic const char s[] = \"                    \";\n+\twhile (count >= sizeof(s)) {\n+\t\tfwrite(s, sizeof(s) - 1, 1, stdout);\n+\t\tcount -= sizeof(s) - 1;\n+\t}\n+\tfwrite(s, count, 1, stdout);\n+}\n+\n+/*\n+ * Wrap the text, if necessary. The variable indent is the indent for the\n+ * first line, indent2 is the indent for all other lines.\n+ */\n+void print_wrapped_text(const char *text, int indent, int indent2, int width)\n+{\n+\tint w = indent, assume_utf8 = is_utf8(text);\n+\tconst char *bol = text, *space = NULL;\n+\n+\tfor (;;) {\n+\t\tchar c = *text;\n+\t\tif (!c || isspace(c)) {\n+\t\t\tif (w < width || space < 0) {\n+\t\t\t\tconst char *start = bol;\n+\t\t\t\tif (space)\n+\t\t\t\t\tstart = space;\n+\t\t\t\telse\n+\t\t\t\t\tprint_spaces(indent);\n+\t\t\t\tfwrite(start, text - start, 1, stdout);\n+\t\t\t\tif (!c) {\n+\t\t\t\t\tputchar('\\n');\n+\t\t\t\t\treturn;\n+\t\t\t\t} else if (c == '\\t')\n+\t\t\t\t\tw |= 0x07;\n+\t\t\t\tspace = text;\n+\t\t\t\tw++;\n+\t\t\t\ttext++;\n+\t\t\t} else {\n+\t\t\t\tputchar('\\n');\n+\t\t\t\ttext = bol = space + 1;\n+\t\t\t\tspace = NULL;\n+\t\t\t\tw = indent = indent2;\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (assume_utf8)\n+\t\t\tw += utf8_width(&text);\n+\t\telse {\n+\t\t\tw++;\n+\t\t\ttext++;\n+\t\t}\n+\t}\n+}\ndiff --git a/utf8.h b/utf8.h\nnew file mode 100644\nindex 0000000..a0d7f59\n--- /dev/null\n+++ b/utf8.h\n@@ -0,0 +1,8 @@\n+#ifndef GIT_UTF8_H\n+#define GIT_UTF8_H\n+\n+int utf8_width(const char **start);\n+int is_utf8(const char *text);\n+void print_wrapped_text(const char *text, int indent, int indent2, int len);\n+\n+#endif"},{"id":"30186","messageId":"20061223085206.GA2189@cepheus","threadId":"6051","inReplyTo":"Pine.LNX.4.63.0612230048350.19693@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH 1/2] libgit.a: add some UTF-8 handling functions","fromName":"Uwe Kleine-König","fromEmail":"zeisberg@informatik.uni-freiburg.de","sentAt":"2006-12-23T08:52:06Z","receivedAt":"2006-12-23T08:52:06Z","isPatch":true,"sender":{"key":"u.kleine-koenig@pengutronix.de","avatar":"https://gravatar.com/avatar/354b5e3ceb2806a2f1e1e382ac29ddbdad18288654da62b61eb13583a857eee7?d=mp&s=160"},"body":"Hallo Johannes,\n\nJohannes Schindelin wrote:\n> @@ -127,6 +128,15 @@ int cmd_commit_tree(int argc, const char **argv, const char *prefix)\n>  \twhile (fgets(comment, sizeof(comment), stdin) != NULL)\n>  \t\tadd_buffer(&buffer, &size, \"%s\", comment);\n>  \n> +\t/* And check the encoding */\n> +\tbuffer[size] = '\\0';\n> +\tif (!strcmp(git_commit_encoding, \"utf-8\") && !is_utf8(buffer)) {\nMaybe you could be more generous here.  E.g.\n\n\tif ((!strcasecmp(git_commit_encoding, \"utf-8\") ||\n\t!strcasecmp(git_commit_encoding, \"utf8\")) && !is_utf8(buffer))\n\nJunio suggested to make this check if i18n.commitEncoding is empty.  I\ndidn't check the code to see if this case is included.\n\nGruessle\nUwe\n\n-- \nUwe Kleine-König\n\nhttp://www.google.com/search?q=2+to+the+power+of+12\n"},{"id":"30197","messageId":"Pine.LNX.4.63.0612231510130.19693@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6051","inReplyTo":"20061223085206.GA2189@cepheus","subject":"Re: [PATCH 1/2] libgit.a: add some UTF-8 handling functions","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-23T14:12:13Z","receivedAt":"2006-12-23T14:12:13Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 23 Dec 2006, Uwe Kleine-König wrote:\n\n> Johannes Schindelin wrote:\n> > @@ -127,6 +128,15 @@ int cmd_commit_tree(int argc, const char **argv, const char *prefix)\n> >  \twhile (fgets(comment, sizeof(comment), stdin) != NULL)\n> >  \t\tadd_buffer(&buffer, &size, \"%s\", comment);\n> >  \n> > +\t/* And check the encoding */\n> > +\tbuffer[size] = '\\0';\n> > +\tif (!strcmp(git_commit_encoding, \"utf-8\") && !is_utf8(buffer)) {\n> Maybe you could be more generous here.  E.g.\n> \n> \tif ((!strcasecmp(git_commit_encoding, \"utf-8\") ||\n> \t!strcasecmp(git_commit_encoding, \"utf8\")) && !is_utf8(buffer))\n> \n> Junio suggested to make this check if i18n.commitEncoding is empty.  I\n> didn't check the code to see if this case is included.\n\nThe problem is, as I pointed out in another mail, that environment.c sets \nthe default git_commit_encoding to \"utf-8\". This is hardwired, and I have \nno way to check if that was set by the config or not, other than reparsing \nthe config myself.\n\n> Gruessle\n\nHah! You don't use umlauts and ssharp yourself!\n\nCiao,\nDscho"},{"id":"30211","messageId":"7vfyb6qth6.fsf_-_@assigned-by-dhcp.cox.net","threadId":"6051","inReplyTo":"Pine.LNX.4.63.0612230048350.19693@wbgn013.biozentrum.uni-wuerzburg.de","subject":"warn non utf-8 commit log messages.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-12-23T19:53:41Z","receivedAt":"2006-12-23T19:53:41Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n>> But I had enough of UTF-8 for a day.\n>\n> Okay, so I lied (this are both patches revised and combined):\n\nI am thinking of putting this in 'next', with the following\nchanges on top of your combined patch.\n\ngit-commit-tree warns if the commit message does not minimally\nconform to the UTF-8 encoding when i18n.commitencoding is either\nunset, or set to \"utf-8\".  It does not die as in your version.\n\n builtin-commit-tree.c |   13 +++++++------\n utf8.c                |    2 +-\n 2 files changed, 8 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin-commit-tree.c b/builtin-commit-tree.c\nindex f274721..f641787 100644\n--- a/builtin-commit-tree.c\n+++ b/builtin-commit-tree.c\n@@ -78,6 +78,11 @@ static int new_parent(int idx)\n \treturn 1;\n }\n \n+static const char commit_utf8_warn[] =\n+\"Warning: commit message does not conform to UTF-8.\\n\"\n+\"You may want to amend it after fixing the message, or set the config\\n\"\n+\"variable i18n.commitencoding to the encoding your project uses.\\n\";\n+\n int cmd_commit_tree(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n@@ -133,12 +138,8 @@ int cmd_commit_tree(int argc, const char **argv, const char *prefix)\n \n \t/* And check the encoding */\n \tbuffer[size] = '\\0';\n-\tif (!strcmp(git_commit_encoding, \"utf-8\") && !is_utf8(buffer)) {\n-\t\tfprintf(stderr, \"Commit message does not conform to UTF-8.\\n\"\n-\t\t\t\"Please fix the message,\"\n-\t\t\t\" or set the config variable i18n.commitencoding.\\n\");\n-\t\treturn 1;\n-\t}\n+\tif (!strcmp(git_commit_encoding, \"utf-8\") && !is_utf8(buffer))\n+\t\tfprintf(stderr, commit_utf8_warn);\n \n \tif (!write_sha1_file(buffer, size, commit_type, commit_sha1)) {\n \t\tprintf(\"%s\\n\", sha1_to_hex(commit_sha1));\ndiff --git a/utf8.c b/utf8.c\nindex aed60ad..8fa6257 100644\n--- a/utf8.c\n+++ b/utf8.c\n@@ -244,7 +244,7 @@ void print_wrapped_text(const char *text, int indent, int indent2, int width)\n \tfor (;;) {\n \t\tchar c = *text;\n \t\tif (!c || isspace(c)) {\n-\t\t\tif (w < width || space < 0) {\n+\t\t\tif (w < width || !space) {\n \t\t\t\tconst char *start = bol;\n \t\t\t\tif (space)\n \t\t\t\t\tstart = space;\n"},{"id":"30215","messageId":"Pine.LNX.4.63.0612240043510.19693@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6051","inReplyTo":"7vfyb6qth6.fsf_-_@assigned-by-dhcp.cox.net","subject":"Re: warn non utf-8 commit log messages.","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-23T23:46:57Z","receivedAt":"2006-12-23T23:46:57Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 23 Dec 2006, Junio C Hamano wrote:\n\n> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n> \n> >> But I had enough of UTF-8 for a day.\n> >\n> > Okay, so I lied (this are both patches revised and combined):\n> \n> I am thinking of putting this in 'next', with the following\n> changes on top of your combined patch.\n> \n> git-commit-tree warns if the commit message does not minimally\n> conform to the UTF-8 encoding when i18n.commitencoding is either\n> unset, or set to \"utf-8\".  It does not die as in your version.\n\nYeah, this is nicer.\n\n> -\t\t\tif (w < width || space < 0) {\n> +\t\t\tif (w < width || !space) {\n\nThis is a real bug fix. Thank you. I changed quite a bit between offset \nand char*, and eventually forgot this part.\n\nCiao,\nDscho\n"},{"id":"30261","messageId":"200612251003.55018.litvinov2004@gmail.com","threadId":"6051","inReplyTo":"Pine.LNX.4.63.0612222319230.19693@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH 1/2] libgit.a: add some UTF-8 handling functions","fromName":"Alexander Litvinov","fromEmail":"litvinov2004@gmail.com","sentAt":"2006-12-25T04:03:54Z","receivedAt":"2006-12-25T04:03:54Z","isPatch":true,"sender":{"key":"litvinov2004@gmail.com","avatar":null},"body":"> > > To me, characters are the symbols occupying one \"column\" each. Bytes\n> > > are the 8-bit thingies that you usually use to encode the characters.\n\nYou can check man 3 wcwidth:\nwcwidth - determine columns needed for a wide character\n\nWe possible could convert utf-8 encoded string into wchar_t[] and use that \nfunction.\n"}]}