{"thread":{"id":"11709","subject":"[RFC/PATCH] git-gui: Use gitattribute \"encoding\" for file content display","startedAt":"2008-01-23T05:47:09Z","lastAt":"2008-01-24T03:39:01Z","messageCount":7,"participants":["Shawn O. Pearce","Junio C Hamano","Pedro Melo","Steffen Prohaska","Jakub Narebski"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"66369","messageId":"20080123054709.GA13166@spearce.org","threadId":"11709","inReplyTo":null,"subject":"[RFC/PATCH] git-gui: Use gitattribute \"encoding\" for file content display","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-01-23T05:47:09Z","receivedAt":"2008-01-23T05:47:09Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"I've got the following change in my \"pu\" right now and am considering\nadding it to git-gui 0.9.2, which would be in git 1.5.4.\n\nI've CC'd a number of people who have emailed me in the past\nabout git-gui's diff or blame failing to display a non US-ASCII\nfile content correctly and I am interested to hear if this would\nresolve the issue for you.  Its configurable on a per-path basis\nby an \"encoding\" attribute in .gitattributes (see git-gui's own\nexample below).\n\nIf we go this route we'll also want to have core Git document in\nits gitattributes manpage what this \"encoding\" attribute is for...\n\n\n--8>--\ngit-gui: Use gitattribute \"encoding\" for file content display\n\nMost folks using git-gui on internationalized files have complained\nthat it doesn't recognize UTF-8 correctly.  In the past we have just\nignored the problem and showed the file contents as binary/US-ASCII,\nwhich is wrong no matter how you look at it.\n\nThis really should be a per-file attribute, managed by .gitattributes,\nso we now pull the \"encoding\" attribute data for the given path from\nthe .gitattributes (if available) and use that, falling back to UTF-8\nif the attributes are unavailable, git-check-attr is broken, or an\nencoding for this path not specified.\n\nWe apply the encoding anytime we show file content, which currently\nis limited to only the diff viewer and the blame viewer.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n .gitattributes |    3 +++\n git-gui.sh     |   13 +++++++++++++\n lib/blame.tcl  |    5 ++++-\n lib/diff.tcl   |    9 ++++++---\n 4 files changed, 26 insertions(+), 4 deletions(-)\n create mode 100644 .gitattributes\n\ndiff --git a/.gitattributes b/.gitattributes\nnew file mode 100644\nindex 0000000..f96112d\n--- /dev/null\n+++ b/.gitattributes\n@@ -0,0 +1,3 @@\n+*           encoding=US-ASCII\n+git-gui.sh  encoding=UTF-8\n+/po/*.po    encoding=UTF-8\ndiff --git a/git-gui.sh b/git-gui.sh\nindex f42e461..adc25d0 100755\n--- a/git-gui.sh\n+++ b/git-gui.sh\n@@ -466,6 +466,19 @@ proc githook_read {hook_name args} {\n \treturn {}\n }\n \n+proc gitattr {path attr default} {\n+\tif {[catch {set r [git check-attr $attr -- $path]}]} {\n+\t\tset r unspecified\n+\t} else {\n+\t\tset r [join [lrange [split $r :] 2 end] :]\n+\t\tregsub {^ } $r {} r\n+\t}\n+\tif {$r eq {unspecified}} {\n+\t\treturn $default\n+\t}\n+\treturn $r\n+}\n+\n proc sq {value} {\n \tregsub -all ' $value \"'\\\\''\" value\n \treturn \"'$value'\"\ndiff --git a/lib/blame.tcl b/lib/blame.tcl\nindex 00ecf21..f33d48f 100644\n--- a/lib/blame.tcl\n+++ b/lib/blame.tcl\n@@ -374,7 +374,10 @@ method _load {jump} {\n \t} else {\n \t\tset fd [git_read cat-file blob \"$commit:$path\"]\n \t}\n-\tfconfigure $fd -blocking 0 -translation lf -encoding binary\n+\tfconfigure $fd \\\n+\t\t-blocking 0 \\\n+\t\t-translation lf \\\n+\t\t-encoding [tcl_encoding [gitattr $path encoding UTF-8]]\n \tfileevent $fd readable [cb _read_file $fd $jump]\n \tset current_fd $fd\n }\ndiff --git a/lib/diff.tcl b/lib/diff.tcl\nindex d04f6db..0f030e3 100644\n--- a/lib/diff.tcl\n+++ b/lib/diff.tcl\n@@ -98,8 +98,11 @@ proc show_diff {path w {lno {}}} {\n \t\t\t\t\tset sz [string length $content]\n \t\t\t\t}\n \t\t\t\tfile {\n+\t\t\t\t\tset enc [gitattr $path encoding UTF-8]\n \t\t\t\t\tset fd [open $path r]\n-\t\t\t\t\tfconfigure $fd -eofchar {}\n+\t\t\t\t\tfconfigure $fd \\\n+\t\t\t\t\t\t-eofchar {} \\\n+\t\t\t\t\t\t-encoding [tcl_encoding $enc]\n \t\t\t\t\tset content [read $fd $max_sz]\n \t\t\t\t\tclose $fd\n \t\t\t\t\tset sz [file size $path]\n@@ -188,8 +191,8 @@ proc show_diff {path w {lno {}}} {\n \n \tfconfigure $fd \\\n \t\t-blocking 0 \\\n-\t\t-encoding binary \\\n-\t\t-translation binary\n+\t\t-encoding [tcl_encoding [gitattr $path encoding UTF-8]] \\\n+\t\t-translation lf\n \tfileevent $fd readable [list read_diff $fd]\n }\n \n-- \n1.5.4.rc4.1130.g9ad85\n"},{"id":"66370","messageId":"7v7ii1ozly.fsf@gitster.siamese.dyndns.org","threadId":"11709","inReplyTo":"20080123054709.GA13166@spearce.org","subject":"Re: [RFC/PATCH] git-gui: Use gitattribute \"encoding\" for file content display","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-01-23T05:55:05Z","receivedAt":"2008-01-23T05:55:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" <spearce@spearce.org> writes:\n\n> git-gui: Use gitattribute \"encoding\" for file content display\n>\n> Most folks using git-gui on internationalized files have complained\n> that it doesn't recognize UTF-8 correctly.  In the past we have just\n> ignored the problem and showed the file contents as binary/US-ASCII,\n> which is wrong no matter how you look at it.\n\nHmmm.\n\nAt least for now in 1.5.4, I'd prefer the way gitk shows UTF-8\n(if I recall correctly latin-1 or other legacy encoding, as long\nas LANG/LC_* is given appropriately, as well) contents without\nper-path configuration without introducing new attributes.\n"},{"id":"66373","messageId":"7E519595-ED6F-454C-9F2A-3011F1E94A50@simplicidade.org","threadId":"11709","inReplyTo":"20080123054709.GA13166@spearce.org","subject":"Re: [RFC/PATCH] git-gui: Use gitattribute \"encoding\" for file content display","fromName":"Pedro Melo","fromEmail":"melo@simplicidade.org","sentAt":"2008-01-23T07:02:27Z","receivedAt":"2008-01-23T07:02:27Z","isPatch":true,"sender":{"key":"melo@simplicidade.org","avatar":"https://gravatar.com/avatar/13ddbb01e300285a93aa1e3739653a81f9b1d3438bd03a4ac36b88e4ffeeafc3?d=mp&s=160"},"body":"Hi,\n\nOn Jan 23, 2008, at 5:47 AM, Shawn O. Pearce wrote:\n\n> I've got the following change in my \"pu\" right now and am considering\n> adding it to git-gui 0.9.2, which would be in git 1.5.4.\n>\n> I've CC'd a number of people who have emailed me in the past\n> about git-gui's diff or blame failing to display a non US-ASCII\n> file content correctly and I am interested to hear if this would\n> resolve the issue for you.  Its configurable on a per-path basis\n> by an \"encoding\" attribute in .gitattributes (see git-gui's own\n> example below).\n\nThis solves the problem for me.\n\nThe diff display correctly display utf-8 characters.\n\nBest regards,\n-- \nPedro Melo\nBlog: http://www.simplicidade.org/notes/\nXMPP ID: melo@simplicidade.org\nUse XMPP!\n"},{"id":"66379","messageId":"EFF72DA9-A717-44A1-9C5C-649D08BB7E96@zib.de","threadId":"11709","inReplyTo":"7v7ii1ozly.fsf@gitster.siamese.dyndns.org","subject":"Re: [RFC/PATCH] git-gui: Use gitattribute \"encoding\" for file content display","fromName":"Steffen Prohaska","fromEmail":"prohaska@zib.de","sentAt":"2008-01-23T08:41:23Z","receivedAt":"2008-01-23T08:41:23Z","isPatch":true,"sender":{"key":"prohaska@zib.de","avatar":"https://avatars.githubusercontent.com/u/217580?v=4"},"body":"\nOn Jan 23, 2008, at 6:55 AM, Junio C Hamano wrote:\n\n> \"Shawn O. Pearce\" <spearce@spearce.org> writes:\n>\n>> git-gui: Use gitattribute \"encoding\" for file content display\n>>\n>> Most folks using git-gui on internationalized files have complained\n>> that it doesn't recognize UTF-8 correctly.  In the past we have just\n>> ignored the problem and showed the file contents as binary/US-ASCII,\n>> which is wrong no matter how you look at it.\n>\n> Hmmm.\n>\n> At least for now in 1.5.4, I'd prefer the way gitk shows UTF-8\n> (if I recall correctly latin-1 or other legacy encoding, as long\n> as LANG/LC_* is given appropriately, as well) contents without\n> per-path configuration without introducing new attributes.\n\nShouldn't we first try harder to get things right without adding\nan attribute?  Maybe we could continue a good tradition and look\nat the content of the first: we could first look for hints in the\nfile about the encoding.  XML and many text files contain such\nhints already to help editors.  For example,  Python source can\nexplicitly contain the encoding [1]; and I guess there are many\nother examples.  If we don't find a direct hint, we could have\nsome magic auto-detection similar to what we do for autocrlf.  As\na fallback the user could specify a default encoding.  But only\nas a last resort, I'd use explicit attributes.\n\n[1] http://www.python.org/dev/peps/pep-0263/\n\n\tSteffen\n"},{"id":"66388","messageId":"200801231128.36504.jnareb@gmail.com","threadId":"11709","inReplyTo":"EFF72DA9-A717-44A1-9C5C-649D08BB7E96@zib.de","subject":"Re: [RFC/PATCH] git-gui: Use gitattribute \"encoding\" for file content display","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-01-23T10:28:35Z","receivedAt":"2008-01-23T10:28:35Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Wed. 23 Jan 2008, Steffen Prohaska wrote:\n> On Jan 23, 2008, at 6:55 AM, Junio C Hamano wrote:\n>> \"Shawn O. Pearce\" <spearce@spearce.org> writes:\n>>\n>>> git-gui: Use gitattribute \"encoding\" for file content display\n>>>\n>>> Most folks using git-gui on internationalized files have complained\n>>> that it doesn't recognize UTF-8 correctly.  In the past we have just\n>>> ignored the problem and showed the file contents as binary/US-ASCII,\n>>> which is wrong no matter how you look at it.\n>>\n>> Hmmm.\n>>\n>> At least for now in 1.5.4, I'd prefer the way gitk shows UTF-8\n>> (if I recall correctly latin-1 or other legacy encoding, as long\n>> as LANG/LC_* is given appropriately, as well) contents without\n>> per-path configuration without introducing new attributes.\n> \n> Shouldn't we first try harder to get things right without adding\n> an attribute?  Maybe we could continue a good tradition and look\n> at the content of the first: we could first look for hints in the\n> file about the encoding.  XML and many text files contain such\n> hints already to help editors.  For example,  Python source can\n> explicitly contain the encoding [1]; and I guess there are many\n> other examples.\n\nFor example LaTeX files either use inputenc package to set encoding\n(e.g. \\usepackage[latin2]{inputenc}) or use magic first line to\nspecify TCX (TeX character translation) file \n(e.g. %& -translate-file=il2-t1).\n\nEmacs encourages to use file variables, either in the form of magic\nfirst line, or file variables at the end of file; I think the same\nis true for Vim.\n\n\nI'd like then for it to be at least as configurable as diff.*.funcname \nis for diff.\n\n> If we don't find a direct hint, we could have \n> some magic auto-detection similar to what we do for autocrlf.\n\nWe can at least try to and check for UTF-16 magic first two bytes, and \ndetect if we have character which is invalid in UTF-8 (for performance \nI guess checking only beginning of file)... \n\n> As a fallback the user could specify a default encoding.  But only\n> as a last resort, I'd use explicit attributes.\n\n...and then falling back to fallback encoding, like gitweb does.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"66445","messageId":"20080124033640.GM24004@spearce.org","threadId":"11709","inReplyTo":"7v7ii1ozly.fsf@gitster.siamese.dyndns.org","subject":"Re: [RFC/PATCH] git-gui: Use gitattribute \"encoding\" for file content display","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-01-24T03:36:40Z","receivedAt":"2008-01-24T03:36:40Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano <gitster@pobox.com> wrote:\n> \"Shawn O. Pearce\" <spearce@spearce.org> writes:\n> \n> > git-gui: Use gitattribute \"encoding\" for file content display\n> >\n> > Most folks using git-gui on internationalized files have complained\n> > that it doesn't recognize UTF-8 correctly.  In the past we have just\n> > ignored the problem and showed the file contents as binary/US-ASCII,\n> > which is wrong no matter how you look at it.\n> \n> Hmmm.\n> \n> At least for now in 1.5.4, I'd prefer the way gitk shows UTF-8\n> (if I recall correctly latin-1 or other legacy encoding, as long\n> as LANG/LC_* is given appropriately, as well) contents without\n> per-path configuration without introducing new attributes.\n\nHmm.  I'll try to rework something along those lines for 1.5.4 then.\n\n-- \nShawn.\n"},{"id":"66446","messageId":"20080124033901.GN24004@spearce.org","threadId":"11709","inReplyTo":"4FF40048-FCF4-4BAD-AD08-6ADAD30E7B6A@simplicidade.org","subject":"Re: [RFC/PATCH] git-gui: Use gitattribute \"encoding\" for file content display","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-01-24T03:39:01Z","receivedAt":"2008-01-24T03:39:01Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Pedro Melo <melo@simplicidade.org> wrote:\n> On Jan 23, 2008, at 5:47 AM, Shawn O. Pearce wrote:\n> \n> >I've got the following change in my \"pu\" right now and am considering\n> >adding it to git-gui 0.9.2, which would be in git 1.5.4.\n> \n> Spoke too soon.\n> \n> I can see the UTF-8 chars in the diffs, but if I use Stage Hunk for  \n> Commit in a hunk with accentuated utf-8 chars, git-gui will throw an  \n> error like \"patch does not apply\".\n\nYea, I know why.  It actually kept me awake for a little bit last\nnight after I sent the patch out for comment, but wasn't big enough\nto get me out of bed to fix it.\n\nThe issue is git-gui doesn't convert back from character data to\nbinary correctly, so the patch won't apply, as the context will\nnot match.  Its simple enough to fix, we just have to remember\nthe encoding we used to go from git-diff --> Tcl and use that\nsame encoding in the other direction (Tcl --> git-apply).\n\n-- \nShawn.\n"}]}