{"thread":{"id":"15875","subject":"[PATCH (GITK) v3 1/4] gitk: Port new encoding logic from git-gui.","startedAt":"2008-10-13T08:12:30Z","lastAt":"2008-10-15T13:09:12Z","messageCount":8,"participants":["Alexander Gavrilov","Paul Mackerras"],"isPatch":true,"patchVersion":3,"patchTotal":4},"messages":[{"id":"92909","messageId":"1223885554-27718-1-git-send-email-angavrilov@gmail.com","threadId":"15875","inReplyTo":null,"subject":"[PATCH (GITK) v3 0/4] Enhance encoding support.","fromName":"Alexander Gavrilov","fromEmail":"angavrilov@gmail.com","sentAt":"2008-10-13T08:12:30Z","receivedAt":"2008-10-13T08:12:30Z","isPatch":true,"sender":{"key":"angavrilov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42666?v=4"},"body":"Currently GUI tools don't provide enough support for\nviewing files that contain non-ASCII characters. This set of\npatches addresses this issue. The git-gui part of the series\nis based on patches that were in the 'pu' branch of the\ngit-gui repository.\n\nUPDATE: the git-gui part of the series is in master.\n\nFile encoding can be specified in the following ways:\n\n1) It defaults to the current locale encoding.\n2) It can be overridden by setting the gui.encoding option.\n3) It can be further set on per-file basis by specifying\n   the 'encoding' attribute in gitattributes.\n4) Finally, git-gui allows directly selecting encodings\n   through a popup menu.\n\nSince git apparently cannot work with filenames in non-locale\nencodings anyway, I did not try to do anything about it apart\nfrom fixing some obvious bugs.\n\n\nCHANGES v2:\n\n- Wrote better comments for the first three patches.\n- Added a patch to speed up loading of very large diffs.\n\nCHANGES v3:\n\n- Killed get_cached_encoding, replacing it with get_path_encoding.\n- Squashed file name and content patches into one.\n- Added the tcl_encoding reimplementation patch. I should\n  say that without the second cache this really makes a\n  difference with thousand-file diffs.\n\nGITK:\n\tgitk: Port new encoding logic from git-gui.\n\t---\n\t gitk |   39 ++++++++++++++++++++++++++++++++++++---\n\t 1 files changed, 36 insertions(+), 3 deletions(-)\n\n\tgitk: Enhance file encoding support.\n\t---\n\t gitk |   35 +++++++++++++++++++++++++++--------\n\t 1 files changed, 27 insertions(+), 8 deletions(-)\n\n\tgitk: Implement batch lookup and caching of encoding attrs.\n\t---\n\t gitk |   35 ++++++++++++++++++++++++++++++++++-\n\t 1 files changed, 34 insertions(+), 1 deletions(-)\n\n\tgitk: Optimize encoding name resolution using a lookup table.\n\t---\n\t gitk |   84 ++++++++++++++++++++++++++++++++++++++++++-----------------------\n\t 1 files changed, 54 insertions(+), 30 deletions(-)\n"},{"id":"92908","messageId":"1223885554-27718-2-git-send-email-angavrilov@gmail.com","threadId":"15875","inReplyTo":"1223885554-27718-1-git-send-email-angavrilov@gmail.com","subject":"[PATCH (GITK) v3 1/4] gitk: Port new encoding logic from git-gui.","fromName":"Alexander Gavrilov","fromEmail":"angavrilov@gmail.com","sentAt":"2008-10-13T08:12:31Z","receivedAt":"2008-10-13T08:12:31Z","isPatch":true,"sender":{"key":"angavrilov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42666?v=4"},"body":"Recently handling of file encodings in GIT-GUI has\nbeen greatly enhanced. It now follows these rules:\n\n- File encoding defaults to the system encoding.\n- It can be overridden by setting the gui.encoding option.\n- Finally, the 'encoding' attribute is checked on\n  per-file basis; it has the last word.\n\nThis patch inserts copies of git-gui functions necessary\nfor supporting this into gitk. The following patch will\nconnect them to the rest of the code.\n\nNote: Since git-check-attr does not provide support for\nreading attributes from trees, attribute lookup is done\nusing files from the working directory.\n\nIt also extends the range of supported encoding\nnames, to match changes in git-gui.\n\nSigned-off-by: Alexander Gavrilov <angavrilov@gmail.com>\nTested-by: Johannes Sixt <johannes.sixt@telecom.at>\n---\n gitk |   39 ++++++++++++++++++++++++++++++++++++---\n 1 files changed, 36 insertions(+), 3 deletions(-)\n\ndiff --git a/gitk b/gitk\nindex dce17e9..8682838 100755\n--- a/gitk\n+++ b/gitk\n@@ -9727,7 +9727,7 @@ set encoding_aliases {\n     { ISO-8859-16 iso-ir-226 ISO_8859-16:2001 ISO_8859-16 latin10 l10 }\n     { GBK CP936 MS936 windows-936 }\n     { JIS_Encoding csJISEncoding }\n-    { Shift_JIS MS_Kanji csShiftJIS }\n+    { Shift_JIS MS_Kanji csShiftJIS ShiftJIS Shift-JIS }\n     { Extended_UNIX_Code_Packed_Format_for_Japanese csEUCPkdFmtJapanese\n       EUC-JP }\n     { Extended_UNIX_Code_Fixed_Width_for_Japanese csEUCFixWidJapanese }\n@@ -9769,7 +9769,7 @@ proc tcl_encoding {enc} {\n     set i [lsearch -exact $lcnames $enc]\n     if {$i < 0} {\n \t# look for \"isonnn\" instead of \"iso-nnn\" or \"iso_nnn\"\n-\tif {[regsub {^iso[-_]} $enc iso encx]} {\n+\tif {[regsub {^(iso|cp|ibm|jis)[-_]} $enc {\\1} encx]} {\n \t    set i [lsearch -exact $lcnames $encx]\n \t}\n     }\n@@ -9781,7 +9781,7 @@ proc tcl_encoding {enc} {\n \t    foreach e $ll {\n \t\tset i [lsearch -exact $lcnames $e]\n \t\tif {$i < 0} {\n-\t\t    if {[regsub {^iso[-_]} $e iso ex]} {\n+\t\t    if {[regsub {^(iso|cp|ibm|jis)[-_]} $e {\\1} ex]} {\n \t\t\tset i [lsearch -exact $lcnames $ex]\n \t\t    }\n \t\t}\n@@ -9796,6 +9796,34 @@ proc tcl_encoding {enc} {\n     return {}\n }\n \n+proc gitattr {path attr default} {\n+\tif {[catch {set r [exec git check-attr $attr -- $path]}]} {\n+\t\tset r unspecified\n+\t} else {\n+\t\tset r [join [lrange [split $r :] 2 end] :]\n+\t\tregsub {^ } $r {} r\n+\t}\n+\tif {$r eq {unspecified}} {\n+\t\treturn $default\n+\t}\n+\treturn $r\n+}\n+\n+proc get_path_encoding {path} {\n+\tglobal gui_encoding\n+\tset tcl_enc [tcl_encoding $gui_encoding]\n+\tif {$tcl_enc eq {}} {\n+\t\tset tcl_enc [encoding system]\n+\t}\n+\tif {$path ne {}} {\n+\t\tset enc2 [tcl_encoding [gitattr $path encoding $tcl_enc]]\n+\t\tif {$enc2 ne {}} {\n+\t\t\tset tcl_enc $enc2\n+\t\t}\n+\t}\n+\treturn $tcl_enc\n+}\n+\n # First check that Tcl/Tk is recent enough\n if {[catch {package require Tk 8.4} err]} {\n     show_error {} . [mc \"Sorry, gitk cannot run with this version of Tcl/Tk.\\n\\\n@@ -9818,6 +9846,11 @@ if {$tclencoding == {}} {\n     puts stderr \"Warning: encoding $gitencoding is not supported by Tcl/Tk\"\n }\n \n+set gui_encoding [encoding system]\n+catch {\n+\tset gui_encoding [exec git config --get gui.encoding]\n+}\n+\n set mainfont {Helvetica 9}\n set textfont {Courier 9}\n set uifont {Helvetica 9 bold}\n-- \n1.6.0.20.g6148bc\n"},{"id":"92912","messageId":"1223885554-27718-3-git-send-email-angavrilov@gmail.com","threadId":"15875","inReplyTo":"1223885554-27718-2-git-send-email-angavrilov@gmail.com","subject":"[PATCH (GITK) v3 2/4] gitk: Enhance file encoding support.","fromName":"Alexander Gavrilov","fromEmail":"angavrilov@gmail.com","sentAt":"2008-10-13T08:12:32Z","receivedAt":"2008-10-13T08:12:32Z","isPatch":true,"sender":{"key":"angavrilov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42666?v=4"},"body":"The previous patch has added functions that implement\nper-file contents encoding selection logic. This one\nmodifies core gitk code to use them.\n\nSince diffs usually contain multiple files, which may\nhave different values of the encoding attribute, they\nare initially read as binary, and encoding conversion\nis applied to individual lines, based on the active\nset of diff headers.\n\nAs a side job, this patch also fixes some bugs in\nhandling of non-ASCII filenames. Core git apparently\nsupports only locale-encoded filenames, so processing\nis done using the system encoding.\n\nSigned-off-by: Alexander Gavrilov <angavrilov@gmail.com>\nTested-by: Johannes Sixt <johannes.sixt@telecom.at>\n---\n gitk |   35 +++++++++++++++++++++++++++--------\n 1 files changed, 27 insertions(+), 8 deletions(-)\n\ndiff --git a/gitk b/gitk\nindex 8682838..5f35f61 100755\n--- a/gitk\n+++ b/gitk\n@@ -6229,7 +6229,7 @@ proc gettree {id} {\n \t    set treepending $id\n \t    set treefilelist($id) {}\n \t    set treeidlist($id) {}\n-\t    fconfigure $gtf -blocking 0\n+\t    fconfigure $gtf -blocking 0 -encoding binary\n \t    filerun $gtf [list gettreeline $gtf $id]\n \t}\n     } else {\n@@ -6251,11 +6251,12 @@ proc gettreeline {gtf id} {\n \t    set line [string range $line 0 [expr {$i-1}]]\n \t    if {$diffids ne $nullid2 && [lindex $line 1] ne \"blob\"} continue\n \t    set sha1 [lindex $line 2]\n-\t    if {[string index $fname 0] eq \"\\\"\"} {\n-\t\tset fname [lindex $fname 0]\n-\t    }\n \t    lappend treeidlist($id) $sha1\n \t}\n+\tif {[string index $fname 0] eq \"\\\"\"} {\n+\t    set fname [lindex $fname 0]\n+\t}\n+\tset fname [encoding convertfrom $fname]\n \tlappend treefilelist($id) $fname\n     }\n     if {![eof $gtf]} {\n@@ -6296,7 +6297,7 @@ proc showfile {f} {\n \t    return\n \t}\n     }\n-    fconfigure $bf -blocking 0\n+    fconfigure $bf -blocking 0 -encoding [get_path_encoding $f]\n     filerun $bf [list getblobline $bf $diffids]\n     $ctext config -state normal\n     clear_ctext $commentend\n@@ -6334,6 +6335,7 @@ proc mergediff {id} {\n     global diffids\n     global parents\n     global diffcontext\n+    global diffencoding\n     global limitdiffs vfilelimit curview\n \n     set diffmergeid $id\n@@ -6347,9 +6349,10 @@ proc mergediff {id} {\n \terror_popup \"[mc \"Error getting merge diffs:\"] $err\"\n \treturn\n     }\n-    fconfigure $mdf -blocking 0\n+    fconfigure $mdf -blocking 0 -encoding binary\n     set mdifffd($id) $mdf\n     set np [llength $parents($curview,$id)]\n+    set diffencoding [get_path_encoding {}]\n     settabs $np\n     filerun $mdf [list getmergediffline $mdf $id $np]\n }\n@@ -6357,6 +6360,7 @@ proc mergediff {id} {\n proc getmergediffline {mdf id np} {\n     global diffmergeid ctext cflist mergemax\n     global difffilestart mdifffd\n+    global diffencoding\n \n     $ctext conf -state normal\n     set nr 0\n@@ -6368,18 +6372,22 @@ proc getmergediffline {mdf id np} {\n \t}\n \tif {[regexp {^diff --cc (.*)} $line match fname]} {\n \t    # start of a new file\n+\t    set fname [encoding convertfrom $fname]\n \t    $ctext insert end \"\\n\"\n \t    set here [$ctext index \"end - 1c\"]\n \t    lappend difffilestart $here\n \t    add_flist [list $fname]\n+\t    set diffencoding [get_path_encoding $fname]\n \t    set l [expr {(78 - [string length $fname]) / 2}]\n \t    set pad [string range \"----------------------------------------\" 1 $l]\n \t    $ctext insert end \"$pad $fname $pad\\n\" filesep\n \t} elseif {[regexp {^@@} $line]} {\n+\t    set line [encoding convertfrom $diffencoding $line]\n \t    $ctext insert end \"$line\\n\" hunksep\n \t} elseif {[regexp {^[0-9a-f]{40}$} $line] || [regexp {^index} $line]} {\n \t    # do nothing\n \t} else {\n+\t    set line [encoding convertfrom $diffencoding $line]\n \t    # parse the prefix - one ' ', '-' or '+' for each parent\n \t    set spaces {}\n \t    set minuses {}\n@@ -6514,7 +6522,7 @@ proc gettreediffs {ids} {\n \n     set treepending $ids\n     set treediff {}\n-    fconfigure $gdtf -blocking 0\n+    fconfigure $gdtf -blocking 0 -encoding binary\n     filerun $gdtf [list gettreediffline $gdtf $ids]\n }\n \n@@ -6530,6 +6538,7 @@ proc gettreediffline {gdtf ids} {\n \t    if {[string index $file 0] eq \"\\\"\"} {\n \t\tset file [lindex $file 0]\n \t    }\n+\t    set file [encoding convertfrom $file]\n \t    lappend treediff $file\n \t}\n     }\n@@ -6587,6 +6596,7 @@ proc getblobdiffs {ids} {\n     global diffcontext\n     global ignorespace\n     global limitdiffs vfilelimit curview\n+    global diffencoding\n \n     set cmd [diffcmd $ids \"-p -C --no-commit-id -U$diffcontext\"]\n     if {$ignorespace} {\n@@ -6600,7 +6610,8 @@ proc getblobdiffs {ids} {\n \treturn\n     }\n     set diffinhdr 0\n-    fconfigure $bdf -blocking 0\n+    set diffencoding [get_path_encoding {}]\n+    fconfigure $bdf -blocking 0 -encoding binary\n     set blobdifffd($ids) $bdf\n     filerun $bdf [list getblobdiffline $bdf $diffids]\n }\n@@ -6634,6 +6645,7 @@ proc getblobdiffline {bdf ids} {\n     global diffids blobdifffd ctext curdiffstart\n     global diffnexthead diffnextnote difffilestart\n     global diffinhdr treediffs\n+    global diffencoding\n \n     set nr 0\n     $ctext conf -state normal\n@@ -6671,10 +6683,13 @@ proc getblobdiffline {bdf ids} {\n \t    } else {\n \t\tset fname [string range $line 2 [expr {$i - 1}]]\n \t    }\n+\t    set fname [encoding convertfrom $fname]\n+\t    set diffencoding [get_path_encoding $fname]\n \t    makediffhdr $fname $ids\n \n \t} elseif {[regexp {^@@ -([0-9]+)(,[0-9]+)? \\+([0-9]+)(,[0-9]+)? @@(.*)} \\\n \t\t       $line match f1l f1c f2l f2c rest]} {\n+\t    set line [encoding convertfrom $diffencoding $line]\n \t    $ctext insert end \"$line\\n\" hunksep\n \t    set diffinhdr 0\n \n@@ -6684,6 +6699,7 @@ proc getblobdiffline {bdf ids} {\n \t\tif {[string index $fname 0] eq \"\\\"\"} {\n \t\t    set fname [lindex $fname 0]\n \t\t}\n+\t\tset fname [encoding convertfrom $fname]\n \t\tset i [lsearch -exact $treediffs($ids) $fname]\n \t\tif {$i >= 0} {\n \t\t    setinlist difffilestart $i $curdiffstart\n@@ -6694,6 +6710,8 @@ proc getblobdiffline {bdf ids} {\n \t\tif {[string index $fname 0] eq \"\\\"\"} {\n \t\t    set fname [lindex $fname 0]\n \t\t}\n+\t\tset fname [encoding convertfrom $fname]\n+\t\tset diffencoding [get_path_encoding $fname]\n \t\tmakediffhdr $fname $ids\n \t    } elseif {[string compare -length 3 $line \"---\"] == 0} {\n \t\t# do nothing\n@@ -6705,6 +6723,7 @@ proc getblobdiffline {bdf ids} {\n \t    $ctext insert end \"$line\\n\" filesep\n \n \t} else {\n+\t    set line [encoding convertfrom $diffencoding $line]\n \t    set x [string range $line 0 0]\n \t    if {$x == \"-\" || $x == \"+\"} {\n \t\tset tag [expr {$x == \"+\"}]\n-- \n1.6.0.20.g6148bc\n"},{"id":"92911","messageId":"1223885554-27718-4-git-send-email-angavrilov@gmail.com","threadId":"15875","inReplyTo":"1223885554-27718-3-git-send-email-angavrilov@gmail.com","subject":"[PATCH (GITK) v3 3/4] gitk: Implement batch lookup and caching of encoding attrs.","fromName":"Alexander Gavrilov","fromEmail":"angavrilov@gmail.com","sentAt":"2008-10-13T08:12:33Z","receivedAt":"2008-10-13T08:12:33Z","isPatch":true,"sender":{"key":"angavrilov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42666?v=4"},"body":"When the diff contains thousands of files, calling git-check-attr\nonce per file is very slow. With this patch gitk does attribute\nlookup in batches of 30 files while reading the diff file list,\nwhich leads to a very noticeable speedup.\n\nIt may be possible to reimplement this even more efficiently,\nif git-check-attr is modified to support a --stdin-paths option.\nAdditionally, it should quote the ':' character in file paths,\nor provide a more robust way of column separation.\n\nSigned-off-by: Alexander Gavrilov <angavrilov@gmail.com>\nTested-by: Johannes Sixt <johannes.sixt@telecom.at>\n---\n gitk |   35 ++++++++++++++++++++++++++++++++++-\n 1 files changed, 34 insertions(+), 1 deletions(-)\n\ndiff --git a/gitk b/gitk\nindex 5f35f61..8cd3171 100755\n--- a/gitk\n+++ b/gitk\n@@ -6531,6 +6531,7 @@ proc gettreediffline {gdtf ids} {\n     global cmitmode vfilelimit curview limitdiffs\n \n     set nr 0\n+    set sublist {}\n     while {[incr nr] <= 1000 && [gets $gdtf line] >= 0} {\n \tset i [string first \"\\t\" $line]\n \tif {$i >= 0} {\n@@ -6540,8 +6541,10 @@ proc gettreediffline {gdtf ids} {\n \t    }\n \t    set file [encoding convertfrom $file]\n \t    lappend treediff $file\n+\t    lappend sublist $file\n \t}\n     }\n+    cache_gitattr encoding $sublist\n     if {![eof $gdtf]} {\n \treturn [expr {$nr >= 1000? 2: 1}]\n     }\n@@ -9816,18 +9819,48 @@ proc tcl_encoding {enc} {\n }\n \n proc gitattr {path attr default} {\n-\tif {[catch {set r [exec git check-attr $attr -- $path]}]} {\n+\tglobal path_attr_cache\n+\tif {[info exists path_attr_cache($attr,$path)]} {\n+\t\tset r $path_attr_cache($attr,$path)\n+\t} elseif {[catch {set r [exec git check-attr $attr -- $path]}]} {\n \t\tset r unspecified\n \t} else {\n \t\tset r [join [lrange [split $r :] 2 end] :]\n \t\tregsub {^ } $r {} r\n \t}\n+\tset path_attr_cache($attr,$path) $r\n \tif {$r eq {unspecified}} {\n \t\treturn $default\n \t}\n \treturn $r\n }\n \n+proc cache_gitattr {attr pathlist} {\n+\tglobal path_attr_cache\n+\tset newlist {}\n+\tforeach path $pathlist {\n+\t\tif {[info exists path_attr_cache($attr,$path)]} continue\n+\t\tlappend newlist $path\n+\t}\n+\twhile {$newlist ne {}} {\n+\t\tset head [lrange $newlist 0 29]\n+\t\tset newlist [lrange $newlist 30 end]\n+\t\tif {![catch {set rlist [eval exec git check-attr $attr -- $head]}]} {\n+\t\t\tforeach row [split $rlist \"\\n\"] {\n+\t\t\t\tset cols [split $row :]\n+\t\t\t\tset path [lindex $cols 0]\n+\t\t\t\tset value [join [lrange $cols 2 end] :]\n+\t\t\t\tif {[string index $path 0] eq \"\\\"\"} {\n+\t\t\t\t\tset path [encoding convertfrom [lindex $path 0]]\n+\t\t\t\t}\n+\t\t\t\tregsub {^ } $value {} value\n+\t\t\t\tset path_attr_cache($attr,$path) $value\n+\t\t\t}\n+\t\t}\n+\t\tupdate\n+\t}\n+}\n+\n proc get_path_encoding {path} {\n \tglobal gui_encoding\n \tset tcl_enc [tcl_encoding $gui_encoding]\n-- \n1.6.0.20.g6148bc\n"},{"id":"92910","messageId":"1223885554-27718-5-git-send-email-angavrilov@gmail.com","threadId":"15875","inReplyTo":"1223885554-27718-4-git-send-email-angavrilov@gmail.com","subject":"[PATCH (GITK) v3 4/4] gitk: Optimize encoding name resolution using a lookup table.","fromName":"Alexander Gavrilov","fromEmail":"angavrilov@gmail.com","sentAt":"2008-10-13T08:12:34Z","receivedAt":"2008-10-13T08:12:34Z","isPatch":true,"sender":{"key":"angavrilov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42666?v=4"},"body":"Current implementation of tcl_encoding uses linear search,\nand is rather slow when called hundreds of times. This commit\nreimplements it using a lookup table, which is efficiently\ncalculated on the first run. After that, resolution costs two\ncalls to [info exists], and one actual fetch from the table.\n\nThis is a port of git-gui commit a1c3feb7fac7.\n\nSigned-off-by: Alexander Gavrilov <angavrilov@gmail.com>\n---\n gitk |   84 ++++++++++++++++++++++++++++++++++++++++++-----------------------\n 1 files changed, 54 insertions(+), 30 deletions(-)\n\ndiff --git a/gitk b/gitk\nindex 8cd3171..134df00 100755\n--- a/gitk\n+++ b/gitk\n@@ -9783,39 +9783,63 @@ set encoding_aliases {\n     { Big5 csBig5 }\n }\n \n-proc tcl_encoding {enc} {\n-    global encoding_aliases\n-    set names [encoding names]\n-    set lcnames [string tolower $names]\n-    set enc [string tolower $enc]\n-    set i [lsearch -exact $lcnames $enc]\n-    if {$i < 0} {\n-\t# look for \"isonnn\" instead of \"iso-nnn\" or \"iso_nnn\"\n-\tif {[regsub {^(iso|cp|ibm|jis)[-_]} $enc {\\1} encx]} {\n-\t    set i [lsearch -exact $lcnames $encx]\n+proc build_encoding_table {} {\n+\tglobal encoding_aliases encoding_lookup_table\n+\n+\t# Prepare the lookup list; cannot use lsort -nocase because\n+\t# of compatibility issues with older Tcl (e.g. in msysgit)\n+\tset names [list]\n+\tforeach item [encoding names] {\n+\t\tlappend names [list [string tolower $item] $item]\n+\t}\n+\tset names [lsort -ascii -index 0 $names]\n+\t# neither can we use lsearch -index\n+\tset lnames [list]\n+\tforeach item $names {\n+\t\tlappend lnames [lindex $item 0]\n+\t}\n+\n+\tforeach grp $encoding_aliases {\n+\t\tset target {}\n+\t\tforeach item $grp {\n+\t\t\tset i [lsearch -sorted -ascii $lnames \\\n+\t\t\t\t\t[string tolower $item]]\n+\t\t\tif {$i >= 0} {\n+\t\t\t\tset target [lindex $names $i 1]\n+\t\t\t\tbreak\n+\t\t\t}\n+\t\t}\n+\t\tif {$target eq {}} continue\n+\t\tforeach item $grp {\n+\t\t\tset encoding_lookup_table([string tolower $item]) $target\n+\t\t}\n \t}\n-    }\n-    if {$i < 0} {\n-\tforeach l $encoding_aliases {\n-\t    set ll [string tolower $l]\n-\t    if {[lsearch -exact $ll $enc] < 0} continue\n-\t    # look through the aliases for one that tcl knows about\n-\t    foreach e $ll {\n-\t\tset i [lsearch -exact $lcnames $e]\n-\t\tif {$i < 0} {\n-\t\t    if {[regsub {^(iso|cp|ibm|jis)[-_]} $e {\\1} ex]} {\n-\t\t\tset i [lsearch -exact $lcnames $ex]\n-\t\t    }\n+\n+\tforeach item $names {\n+\t\tset encoding_lookup_table([lindex $item 0]) [lindex $item 1]\n+\t}\n+}\n+\n+proc tcl_encoding {enc} {\n+\tglobal encoding_lookup_table\n+\tif {$enc eq {}} {\n+\t\treturn {}\n+\t}\n+\tif {![info exists encoding_lookup_table]} {\n+\t\tbuild_encoding_table\n+\t}\n+\tset enc [string tolower $enc]\n+\tif {![info exists encoding_lookup_table($enc)]} {\n+\t\t# look for \"isonnn\" instead of \"iso-nnn\" or \"iso_nnn\"\n+\t\tif {[regsub {^(iso|cp|ibm|jis)[-_]} $enc {\\1} encx]} {\n+\t\t\tset enc $encx\n \t\t}\n-\t\tif {$i >= 0} break\n-\t    }\n-\t    break\n \t}\n-    }\n-    if {$i >= 0} {\n-\treturn [lindex $names $i]\n-    }\n-    return {}\n+\tif {[info exists encoding_lookup_table($enc)]} {\n+\t\treturn $encoding_lookup_table($enc)\n+\t} else {\n+\t\treturn {}\n+\t}\n }\n \n proc gitattr {path attr default} {\n-- \n1.6.0.20.g6148bc\n"},{"id":"93087","messageId":"18677.58054.209169.479165@cargo.ozlabs.ibm.com","threadId":"15875","inReplyTo":"1223885554-27718-1-git-send-email-angavrilov@gmail.com","subject":"Re: [PATCH (GITK) v3 0/4] Enhance encoding support.","fromName":"Paul Mackerras","fromEmail":"paulus@samba.org","sentAt":"2008-10-15T12:32:06Z","receivedAt":"2008-10-15T12:32:06Z","isPatch":true,"sender":{"key":"paulus@samba.org","avatar":"https://avatars.githubusercontent.com/u/1606439?v=4"},"body":"Alexander Gavrilov writes:\n\n> Currently GUI tools don't provide enough support for\n> viewing files that contain non-ASCII characters. This set of\n> patches addresses this issue. The git-gui part of the series\n> is based on patches that were in the 'pu' branch of the\n> git-gui repository.\n\nThanks.  I applied 1/4 and 2/4 as one commit and 3/4 as another.\nHowever, it seems invoking git check-attr is quite slow, or at least,\nthere is a large startup cost which is not really amortized by\nbatching up only 30 paths per call.\n\nAs a test I tried displaying the diff in the kernel tree from 2.6.27\nto Linus' master as of a day or so ago (3fa8749e) which has 5277 files\ndifferent.  With the 30-way batching this takes over 6 seconds to get\nthe file encodings (all of which are \"unspecified\" since I don't have\nany .gitattributes files), which we need before we display the list of\nchanged files, which before only took 0.12 seconds -- so we're 50x\nslower at showing the list of changed files as a result of adding the\nper-file encoding support.  Also, each call to cache_gitattr was\ntaking 1.2 seconds, which is too long since file handlers are supposed\nto limit themselves to less than 100ms (preferably less than 50ms) of\nprocessing on any one call, and the cache_gitattr call was making\ngettreediffline take way too long.\n\nSo I have added another commit which makes the per-file encoding\nsupport optional and default to off for now.  It also increases the\namount of batching in cache_gitattr (except under windows) and reduces\nthe number of lines that gettreediffline does when per-file encoding\nsupport is on.  Along the way, it also converts the new code to the\ngitk coding style, cleans up a few things and adds a simple cache to\ntclencoding.\n\nIf git check-attr gets the --stdin-paths flag added, that will be\ngood, except then getblobdiffline will need to do something clever if\nit needs the encoding for a file but the result hasn't come back from\ngit-check-attr yet.  That could get quite tricky in fact...\n\nPaul.\n"},{"id":"93088","messageId":"18677.58426.326371.788610@cargo.ozlabs.ibm.com","threadId":"15875","inReplyTo":"1223885554-27718-1-git-send-email-angavrilov@gmail.com","subject":"Re: [PATCH (GITK) v3 0/4] Enhance encoding support.","fromName":"Paul Mackerras","fromEmail":"paulus@samba.org","sentAt":"2008-10-15T12:38:18Z","receivedAt":"2008-10-15T12:38:18Z","isPatch":true,"sender":{"key":"paulus@samba.org","avatar":"https://avatars.githubusercontent.com/u/1606439?v=4"},"body":"Alexander Gavrilov writes:\n\n> Since git apparently cannot work with filenames in non-locale\n> encodings anyway, I did not try to do anything about it apart\n> from fixing some obvious bugs.\n\nWhat we did before was read filenames and convert them from the system\nencoding (done implicitly by gets) before unquoting filenames that\nwere quoted.  What we do now with your patch 1/2 is that we read the\nfilenames in binary and unquote any quoted filenames before converting\nfrom the system encoding.  So I don't think your patch would have made\nas much difference as it might appear.  If there is a reason for\nunquoting before converting from the system encoding rather than\nafter, it seems pretty subtle to me and wasn't explained in the patch\ndescription.  An explanation, preferably with examples, would be\nuseful.\n\nAlso, you didn't say whether you found the \"obvious bugs\" by\ninspection or by encountering their effects in actual running (and if\nso, what those effects were).  That information is also good to have\nin the patch description.\n\nPaul.\n"},{"id":"93091","messageId":"bb6f213e0810150609y3e74935bq7cf35432fa219f55@mail.gmail.com","threadId":"15875","inReplyTo":"18677.58426.326371.788610@cargo.ozlabs.ibm.com","subject":"Re: [PATCH (GITK) v3 0/4] Enhance encoding support.","fromName":"Alexander Gavrilov","fromEmail":"angavrilov@gmail.com","sentAt":"2008-10-15T13:09:12Z","receivedAt":"2008-10-15T13:09:12Z","isPatch":true,"sender":{"key":"angavrilov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42666?v=4"},"body":"On Wed, Oct 15, 2008 at 4:38 PM, Paul Mackerras <paulus@samba.org> wrote:\n> Alexander Gavrilov writes:\n>\n>> Since git apparently cannot work with filenames in non-locale\n>> encodings anyway, I did not try to do anything about it apart\n>> from fixing some obvious bugs.\n>\n> What we did before was read filenames and convert them from the system\n> encoding (done implicitly by gets) before unquoting filenames that\n> were quoted.  What we do now with your patch 1/2 is that we read the\n> filenames in binary and unquote any quoted filenames before converting\n> from the system encoding.  So I don't think your patch would have made\n> as much difference as it might appear.  If there is a reason for\n> unquoting before converting from the system encoding rather than\n> after, it seems pretty subtle to me and wasn't explained in the patch\n> description.  An explanation, preferably with examples, would be\n> useful.\n\nThe reason is that non-ASCII characters may be quoted too, so the\nstring that we read looks like\n\"\\204\\206\\204y\\204s\\204\\200\\204r\\204y\\204~\\204p.txt\". There is no\npoint decoding it before unquoting.\n\n> Also, you didn't say whether you found the \"obvious bugs\" by\n> inspection or by encountering their effects in actual running (and if\n> so, what those effects were).  That information is also good to have\n> in the patch description.\n\nI actually created a test repository with non-ASCII filenames. If I\nremember it correctly, the bugs manifested as strings in the tree view\nappearing as if they were decoded using ISO-8859-1 (the result of\ndecoding before unquoting), or unstaged files being listed quoted as\nabove.\n\n\nNow the remaining encoding issues are:\n\n1) Commit messages that are loaded through readcommit are decoded\nusing the system encoding. It is rare, but it happens. This is a bug.\n\nproc readcommit {id} {\n    if {[catch {set contents [exec git cat-file commit $id]}]} return\n    parsecommit $id $contents 0\n}\n\n2) Gitk cannot process commits stored in multiple different encodings:\nthey all are decoded using the current value of i18n.commitencoding.\nThis seems to be low priority, because most GUI users are better off\nusing utf-8 for their commits anyway.\n\nAlexander\n"}]}