{"thread":{"id":"4542","subject":"[PATCH] gitweb: safely output binary files for 'blob_plain' action","startedAt":"2006-06-17T11:32:15Z","lastAt":"2006-06-17T22:30:59Z","messageCount":6,"participants":["Jakub Narebski","Petr Baudis","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"21950","messageId":"200606171332.15591.jnareb@gmail.com","threadId":"4542","inReplyTo":null,"subject":"[PATCH] gitweb: safely output binary files for 'blob_plain' action","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-06-17T11:32:15Z","receivedAt":"2006-06-17T11:32:15Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"gitweb tries now to output correct Content-Type header for \n'blob_plain' action; for now text/plain for text files, \nappropriate image MIME type for *.png, *.gif and *.jpg/*.jpeg files, \nand application/octet-stream for other binary files. \n\nIntroduced new configuration variables: $default_blob_plain_mimetype \nand $default_text_plain_charset (only 'utf-8' is guaranteed to work\nfor the latter).\n\nbinmode changed to ':raw' in git_blob_plain for output of non-text files.\n\n---\n gitweb/gitweb.cgi |   43 +++++++++++++++++++++++++++++++++++++++----\n 1 files changed, 39 insertions(+), 4 deletions(-)\n\ndiff --git a/gitweb/gitweb.cgi b/gitweb/gitweb.cgi\nindex 9d902b7..b37ec50 100755\n--- a/gitweb/gitweb.cgi\n+++ b/gitweb/gitweb.cgi\n@@ -39,12 +39,16 @@ # html text to include at home page\n my $home_text =\t\t\"indextext.html\";\n \n # URI of default stylesheet\n-my $stylesheet = \t\"gitweb.css\";\n+my $stylesheet =\t\"gitweb.css\";\n \n # source of projects list\n #my $projects_list =\t$projectroot;\n my $projects_list =\t\"index/index.aux\";\n \n+# default blob_plain mimetype and default charset for text/plain blob\n+my $default_blob_plain_mimetype = 'text/plain';\n+my $default_text_plain_charset  = 'utf-8';\t# can be undefined\n+\n # input validation and dispatch\n my $action = $cgi->param('a');\n if (defined $action) {\n@@ -1354,15 +1358,46 @@ sub git_blob {\n \tgit_footer_html();\n }\n \n+sub git_blob_plain_mimetype {\n+\tmy $fd = shift;\n+\tmy $filename = shift;\n+\n+\t# just in case\n+\treturn $default_blob_plain_mimetype unless $fd;\n+\n+\tif (-T $fd) {\n+\t\treturn 'text/plain' .\n+\t\t       ($default_text_plain_charset ? '; charset='.$default_text_plain_charset : '');\n+\t} elsif (! $filename) {\n+\t\treturn 'application/octet-stream';\n+\t} elsif ($filename =~ m/\\.png$/i) {\n+\t\treturn 'image/png';\n+\t} elsif ($filename =~ m/\\.gif$/i) {\n+\t\treturn 'image/gif';\n+\t} elsif ($filename =~ m/\\.jpe?g$/i) {\n+\t\treturn 'image/jpeg';\n+\t} else {\n+\t\treturn 'application/octet-stream';\n+\t}\n+}\n+\n sub git_blob_plain {\n-\tmy $save_as = \"$hash.txt\";\n+\topen my $fd, \"-|\", \"$gitbin/git-cat-file blob $hash\" or return;\n+\tmy $type = git_blob_plain_mimetype($fd, $file_name);\n+\t\n+\t# save as filename, even when no $file_name is given\n+\tmy $save_as = \"$hash\";\n \tif (defined $file_name) {\n \t\t$save_as = $file_name;\n+\t} elsif ($type =~ m/^text\\//) {\n+\t\t$save_as .= '.txt';\n \t}\n-\tprint $cgi->header(-type => \"text/plain\", -charset => 'utf-8', '-content-disposition' => \"inline; filename=\\\"$save_as\\\"\");\n-\topen my $fd, \"-|\", \"$gitbin/git-cat-file blob $hash\" or return;\n+\t\n+\tprint $cgi->header(-type => \"$type\", '-content-disposition' => \"inline; filename=\\\"$save_as\\\"\");\n \tundef $/;\n+\tbinmode STDOUT, ':raw' unless $type =~ m/^text\\//;\n \tprint <$fd>;\n+\tbinmode STDOUT, ':utf8' unless $type =~ m/^text\\//;\n \t$/ = \"\\n\";\n \tclose $fd;\n }\n-- \n1.3.0\n"},{"id":"21958","messageId":"20060617153540.GI2609@pasky.or.cz","threadId":"4542","inReplyTo":"200606171332.15591.jnareb@gmail.com","subject":"Re: [PATCH] gitweb: safely output binary files for 'blob_plain' action","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2006-06-17T15:35:40Z","receivedAt":"2006-06-17T15:35:40Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Sat, Jun 17, 2006 at 01:32:15PM CEST, I got a letter\nwhere Jakub Narebski <jnareb@gmail.com> said that...\n> Introduced new configuration variables: $default_blob_plain_mimetype \n> and $default_text_plain_charset (only 'utf-8' is guaranteed to work\n> for the latter).\n\nNah, defaulting to 'utf-8' is horrible - usually, you just don't have a\nclue and should refrain from sending any charset information at all, so\nI think undef is a much saner default.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nA person is just about as big as the things that make them angry.\n"},{"id":"21960","messageId":"e719bi$vfd$1@sea.gmane.org","threadId":"4542","inReplyTo":"20060617153540.GI2609@pasky.or.cz","subject":"[PATCH] gitweb: text files for 'blob_plain' action without charset by default","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-06-17T16:07:24Z","receivedAt":"2006-06-17T16:07:24Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"$default_text_plain_charset is undefined (no specified charset) by\ndefault. Additionally ':raw' layer for binmode is used for outputting file\ncontent.\n\n---\nThis patch depends on the previous patch in the thread:\n  \"gitweb: safely output binary files for 'blob_plain' action\"\nContrary to the previous patch it is not based on other unrelated gitweb.cgi\npatches (this changes only line numbers in patch).\n\n gitweb/gitweb.cgi |    8 ++++----\n 1 files changed, 4 insertions(+), 4 deletions(-)\n\n53209981db06a5dde7c59caada279bf63d329da8\ndiff --git a/gitweb/gitweb.cgi b/gitweb/gitweb.cgi\nindex acac1f4..f082e5d 100755\n--- a/gitweb/gitweb.cgi\n+++ b/gitweb/gitweb.cgi\n@@ -44,7 +44,7 @@ my $projects_list =   \"index/index.aux\";\n \n # default blob_plain mimetype and default charset for text/plain blob\n my $default_blob_plain_mimetype = 'text/plain';\n-my $default_text_plain_charset  = 'utf-8';     # can be undefined\n+my $default_text_plain_charset  = undef;       # was: 'utf-8'\n \n # input validation and dispatch\n my $action = $cgi->param('a');\n@@ -1451,9 +1451,9 @@ sub git_blob_plain {\n \n        print $cgi->header(-type => \"$type\", '-content-disposition' => \"inline; filename=\\\"$save_as\\\"\");\n        undef $/;\n-       binmode STDOUT, ':raw' unless $type =~ m/^text\\//;\n+       binmode STDOUT, ':raw';\n        print <$fd>;\n-       binmode STDOUT, ':utf8' unless $type =~ m/^text\\//;\n+       binmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi\n        $/ = \"\\n\";\n        close $fd;\n }\n-- \n1.3.0\n"},{"id":"21968","messageId":"7v8xnv8ozp.fsf@assigned-by-dhcp.cox.net","threadId":"4542","inReplyTo":"20060617153540.GI2609@pasky.or.cz","subject":"Re: [PATCH] gitweb: safely output binary files for 'blob_plain' action","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-06-17T21:13:30Z","receivedAt":"2006-06-17T21:13:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Petr Baudis <pasky@suse.cz> writes:\n\n> Dear diary, on Sat, Jun 17, 2006 at 01:32:15PM CEST, I got a letter\n> where Jakub Narebski <jnareb@gmail.com> said that...\n>> Introduced new configuration variables: $default_blob_plain_mimetype \n>> and $default_text_plain_charset (only 'utf-8' is guaranteed to work\n>> for the latter).\n>\n> Nah, defaulting to 'utf-8' is horrible - usually, you just don't have a\n> clue and should refrain from sending any charset information at all, so\n> I think undef is a much saner default.\n\nConcurred.  I see Jakub's second patch to make this\nconfigurable, but I wonder about a few things:\n\n - we might want to have a configuration mechanism in place\n   before enhancing gitweb.  My gut feeling is that we can use\n   [gitweb] section in project.git/config (and probably\n   duplicate first and deprecate later existing \"description\" as\n   well).\n\n - the blob charset should be per path -- otherwise the feature\n   would be not useful for projects that maintains bunch of po\n   files.\n\nIn other words, something like this:\n\n  (in torvalds/linux-2.6.git/config)\n\n\t[gitweb]\n        description = \"Linus's kernel tree\"\n        ; defaultblobcharset = \"latin1\"\n        blobmimemapfile = \"mime-map\"\n\n  (in torvalds/linux-2.6.git/mime-map, first match decides)\n\n\tfs/nls/nls_euc-jp.c\ttext/plain; charset=euc_jp\n        *.c\ttext/plain; charset=utf-8\n        *.h     text/plain; charset=utf-8\n\nI do not think defaultblobcharset above is a good idea though.\nYou could just have the last entry in mime-map file to be:\n\n        *\ttext/plain; charset=latin1\n"},{"id":"21974","messageId":"20060617220106.GJ2609@pasky.or.cz","threadId":"4542","inReplyTo":"7v8xnv8ozp.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] gitweb: safely output binary files for 'blob_plain' action","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2006-06-17T22:01:06Z","receivedAt":"2006-06-17T22:01:06Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Sat, Jun 17, 2006 at 11:13:30PM CEST, I got a letter\nwhere Junio C Hamano <junkio@cox.net> said that...\n>  - we might want to have a configuration mechanism in place\n>    before enhancing gitweb.  My gut feeling is that we can use\n>    [gitweb] section in project.git/config (and probably\n>    duplicate first and deprecate later existing \"description\" as\n>    well).\n\nAgreed. (I planned to back this up with a patch, then looked at the\nclock.)\n\nHmm, after I'm over my exam period, since there's now another .pl thing\nin the git tree I might start working on some kind of universal Git.pm\ninterface. I'm gonna need it for Cogito in the longer term anyway. ;-)\n\n>  - the blob charset should be per path -- otherwise the feature\n>    would be not useful for projects that maintains bunch of po\n>    files.\n> \n> In other words, something like this:\n> \n>   (in torvalds/linux-2.6.git/config)\n> \n> \t[gitweb]\n>         description = \"Linus's kernel tree\"\n>         ; defaultblobcharset = \"latin1\"\n>         blobmimemapfile = \"mime-map\"\n> \n>   (in torvalds/linux-2.6.git/mime-map, first match decides)\n> \n> \tfs/nls/nls_euc-jp.c\ttext/plain; charset=euc_jp\n>         *.c\ttext/plain; charset=utf-8\n>         *.h     text/plain; charset=utf-8\n\nYou could as well just support the mime.types format and load\n/etc/mime.types for this kind of mapping (see below for a patch). The\nadvantage is that this pretty much covers all the MIME types you will\nneed, the disadvantage is that it's less flexible and the charset part\nwouldn't probably fit in nicely.\n\nWe could obviously do both. :-)\n\n---\n[PATCH] Support for the standard mime.types map in gitweb\n\ngitweb will try to look up the filename mimetype in /etc/mime.types\nand optionally a user-configured mime.types map as well.\n\nSigned-off-by: Petr Baudis <pasky@suse.cz>\n---\n\n Depends on Jakub's mime patches.\n\n gitweb/gitweb.cgi |   44 ++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 44 insertions(+), 0 deletions(-)\n\ndiff --git a/gitweb/gitweb.cgi b/gitweb/gitweb.cgi\nindex 9250548..0116531 100755\n--- a/gitweb/gitweb.cgi\n+++ b/gitweb/gitweb.cgi\n@@ -46,6 +46,11 @@ # default blob_plain mimetype and defaul\n my $default_blob_plain_mimetype = 'text/plain';\n my $default_text_plain_charset  = undef;       # was: 'utf-8'\n \n+# file to use for guessing MIME types before trying /etc/mime.types\n+# (relative to the current git repository)\n+my $mimetypes_file              = undef;\n+\n+\n # input validation and dispatch\n my $action = $cgi->param('a');\n if (defined $action) {\n@@ -1414,6 +1419,40 @@ sub git_blob {\n \tgit_footer_html();\n }\n \n+sub mimetype_guess_file {\n+\tmy $filename = shift;\n+\tmy $mimemap = shift;\n+\t-r $mimemap or return undef;\n+\n+\tmy %mimemap;\n+\topen(MIME, $mimemap) or return undef;\n+\twhile (<MIME>) {\n+\t\tmy ($mime, $exts) = split(/\\t+/);\n+\t\tmy @exts = split(/\\s+/, $exts);\n+\t\tforeach my $ext (@exts) {\n+\t\t\t$mimemap{$ext} = $mime;\n+\t\t}\n+\t}\n+\tclose(MIME);\n+\n+\t$filename =~ /\\.(.*?)$/;\n+\treturn $mimemap{$1};\n+}\n+\n+sub mimetype_guess {\n+\tmy $filename = shift;\n+\tmy $mime;\n+\t$filename =~ /\\./ or return undef;\n+\n+\tif ($mimetypes_file) {\n+\t\tmy $file = $mimetypes_file;\n+\t\t$file =~ m#^/# or $file = \"$projectroot/$path/$file\";\n+\t\t$mime = mimetype_guess_file($filename, $file);\n+\t}\n+\t$mime ||= mimetype_guess_file($filename, '/etc/mime.types');\n+\treturn $mime;\n+}\n+\n sub git_blob_plain_mimetype {\n \tmy $fd = shift;\n \tmy $filename = shift;\n@@ -1421,6 +1460,11 @@ sub git_blob_plain_mimetype {\n \t# just in case\n \treturn $default_blob_plain_mimetype unless $fd;\n \n+\tif ($filename) {\n+\t\tmy $mime = mimetype_guess($filename);\n+\t\t$mime and return $mime;\n+\t}\n+\n \tif (-T $fd) {\n \t\treturn 'text/plain' .\n \t\t       ($default_text_plain_charset ? '; charset='.$default_text_plain_charset : '');\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nA person is just about as big as the things that make them angry.\n"},{"id":"21976","messageId":"7vejxn76u4.fsf@assigned-by-dhcp.cox.net","threadId":"4542","inReplyTo":"20060617220106.GJ2609@pasky.or.cz","subject":"Re: [PATCH] gitweb: safely output binary files for 'blob_plain' action","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-06-17T22:30:59Z","receivedAt":"2006-06-17T22:30:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Petr Baudis <pasky@suse.cz> writes:\n\n>> In other words, something like this:\n>> \n>>   (in torvalds/linux-2.6.git/config)\n>> \n>> \t[gitweb]\n>>         description = \"Linus's kernel tree\"\n>>         ; defaultblobcharset = \"latin1\"\n>>         blobmimemapfile = \"mime-map\"\n>> \n>>   (in torvalds/linux-2.6.git/mime-map, first match decides)\n>> \n>> \tfs/nls/nls_euc-jp.c\ttext/plain; charset=euc_jp\n>>         *.c\ttext/plain; charset=utf-8\n>>         *.h     text/plain; charset=utf-8\n>\n> You could as well just support the mime.types format and load\n> /etc/mime.types for this kind of mapping (see below for a patch). The\n> advantage is that this pretty much covers all the MIME types you will\n> need, the disadvantage is that it's less flexible and the charset part\n> wouldn't probably fit in nicely.\n\nAh, I thought Jakub's patch was already taking care of\nmime.types but apparently that was not the case.  As you say,\nusing /etc/mime.types for this is obviously a good point to\nstart.\n\n> We could obviously do both. :-)\n\nThe point of my example was about charset part; comparing the\nsuffix part only is not good enough for .po files, so we should\nobviously do both.\n"}]}