{"thread":{"id":"4801","subject":"[PATCH] gitweb.cgi: Use File::MMagic; \"a=blob\" action knows the blob/file type","startedAt":"2006-07-08T04:10:21Z","lastAt":"2006-07-09T01:17:56Z","messageCount":3,"participants":["Luben Tuikov","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"23405","messageId":"20060708041021.24704.qmail@web31804.mail.mud.yahoo.com","threadId":"4801","inReplyTo":null,"subject":"[PATCH] gitweb.cgi: Use File::MMagic; \"a=blob\" action knows the blob/file type","fromName":"Luben Tuikov","fromEmail":"ltuikov@yahoo.com","sentAt":"2006-07-08T04:10:21Z","receivedAt":"2006-07-08T04:10:21Z","isPatch":true,"sender":{"key":"ltuikov@yahoo.com","avatar":null},"body":"Use File::MMagic to determine the MIME type of a blob/file.\nThe variable magic_mime_file holds the location of the\n\"magic.mime\" file, usually \"/usr/share/file/magic.mime\".\nIf not defined, the magic numbers internally stored in the\nFile::MMagic module are used.\n\nAction \"blob\" knows the file type: if the file type is\nnot \"text/*\" then action \"blob\" defaults to \"blob_plain\",\ni.e. the file is downloaded raw for the browser to interpret.\nIf the file type is \"text/*\", then \"blob\" defaults to the\ncurrent \"cat -n\"-like output, from which you can click\n\"plain\", to get the \"blob_plain\" output.\n\nSigned-off-by: Luben Tuikov <ltuikov@yahoo.com>\n---\n gitweb/gitweb.cgi |  140 +++++++++++++++++++++--------------------------------\n 1 files changed, 56 insertions(+), 84 deletions(-)\n\ndiff --git a/gitweb/gitweb.cgi b/gitweb/gitweb.cgi\nindex cce0753..6798990 100755\n--- a/gitweb/gitweb.cgi\n+++ b/gitweb/gitweb.cgi\n@@ -14,6 +14,8 @@ use CGI::Util qw(unescape);\n use CGI::Carp qw(fatalsToBrowser);\n use Encode;\n use Fcntl ':mode';\n+use File::MMagic;\n+use FileHandle;\n binmode STDOUT, ':utf8';\n \n our $cgi = new CGI;\n@@ -54,9 +56,15 @@ #our $projects_list = $projectroot;\n our $projects_list = \"index/index.aux\";\n \n # default blob_plain mimetype and default charset for text/plain blob\n-our $default_blob_plain_mimetype = 'text/plain';\n+our $default_blob_mimetype = 'text/plain';\n our $default_text_plain_charset  = undef;\n \n+# magic_mime_file: if defined this file will be used by File::MMagic\n+# to guess the file type, else the magic numbers stored internally\n+# in File::MMagic will be used.  Either relative or absolute name\n+# can be given.  E.g. \"/usr/share/file/magic.mime\".\n+our $magic_mime_file = undef;\n+\n # file to use for guessing MIME types before trying /etc/mime.types\n # (relative to the current git repository)\n our $mimetypes_file = undef;\n@@ -1455,11 +1463,58 @@ sub git_get_hash_by_path {\n \t}\n }\n \n+#\n+# Strangely enough the File::MMagic package, version 1.27, has a bug\n+# whereby reading from a piped filehandle (e.g. STDIN, or \"-|\") always\n+# returns 'text/plain', but reading from a file on a file system (as it\n+# would be the case for the checktype_filename() method) properly\n+# determines the file type.\n+#\n+sub get_blob_mimetype {\n+\tmy $blob_file = \"$git_temp/blob-$hash\";\n+\tif (! -r $blob_file) {\n+\t    open my $fd_in, \"-|\", \"$gitbin/git-cat-file blob $hash\" or return $default_blob_mimetype;\n+\t    open my $fd_out, \"> $blob_file\";\n+\t    my @file = <$fd_in>;\n+\t    print $fd_out @file;\n+\t    close $fd_out;\n+\t    close $fd_in;\n+\t}\n+\tmy $mm = $magic_mime_file ? File::MMagic->new($magic_mime_file) : new File::MMagic;\n+\tmy $mime = $mm->checktype_filename($blob_file);\n+\treturn $mime;\n+}\n+\n+sub git_blob_plain {\n+\topen my $fd, \"-|\", \"$gitbin/git-cat-file blob $hash\" or return;\n+\tmy $mimetype = get_blob_mimetype();\n+\n+\t# save as filename, even when no $file_name is given\n+\tmy $save_as = \"$hash\";\n+\tif (defined $file_name) {\n+\t\t$save_as = $file_name;\n+\t} elsif ($mimetype =~ m/^text\\//) {\n+\t\t$save_as .= '.txt';\n+\t}\n+\n+\tprint $cgi->header(-type => \"$mimetype\", '-content-disposition' => \"inline; filename=\\\"$save_as\\\"\");\n+\tundef $/;\n+\tbinmode STDOUT, ':raw';\n+\tprint <$fd>;\n+\tbinmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi\n+\t$/ = \"\\n\";\n+\tclose $fd;\n+}\n+\n sub git_blob {\n \tif (!defined $hash && defined $file_name) {\n \t\tmy $base = $hash_base || git_read_head($project);\n \t\t$hash = git_get_hash_by_path($base, $file_name, \"blob\") || die_error(undef, \"Error lookup file.\");\n \t}\n+\tmy $mimetype = get_blob_mimetype();\n+\tif ($mimetype !~ m/^text\\//) {\n+\t\treturn git_blob_plain();\n+\t}\n \tmy $have_blame = git_get_project_config_bool ('blame');\n \topen my $fd, \"-|\", \"$gitbin/git-cat-file blob $hash\" or die_error(undef, \"Open failed.\");\n \tgit_header_html();\n@@ -1510,89 +1565,6 @@ sub git_blob {\n \tgit_footer_html();\n }\n \n-sub mimetype_guess_file {\n-\tmy $filename = shift;\n-\tmy $mimemap = shift;\n-\t-r $mimemap or return undef;\n-\n-\tmy %mimemap;\n-\topen(MIME, $mimemap) or return undef;\n-\twhile (<MIME>) {\n-\t\tmy ($mime, $exts) = split(/\\t+/);\n-\t\tmy @exts = split(/\\s+/, $exts);\n-\t\tforeach my $ext (@exts) {\n-\t\t\t$mimemap{$ext} = $mime;\n-\t\t}\n-\t}\n-\tclose(MIME);\n-\n-\t$filename =~ /\\.(.*?)$/;\n-\treturn $mimemap{$1};\n-}\n-\n-sub mimetype_guess {\n-\tmy $filename = shift;\n-\tmy $mime;\n-\t$filename =~ /\\./ or return undef;\n-\n-\tif ($mimetypes_file) {\n-\t\tmy $file = $mimetypes_file;\n-\t\t#$file =~ m#^/# or $file = \"$projectroot/$path/$file\";\n-\t\t$mime = mimetype_guess_file($filename, $file);\n-\t}\n-\t$mime ||= mimetype_guess_file($filename, '/etc/mime.types');\n-\treturn $mime;\n-}\n-\n-sub git_blob_plain_mimetype {\n-\tmy $fd = shift;\n-\tmy $filename = shift;\n-\n-\t# just in case\n-\treturn $default_blob_plain_mimetype unless $fd;\n-\n-\tif ($filename) {\n-\t\tmy $mime = mimetype_guess($filename);\n-\t\t$mime and return $mime;\n-\t}\n-\n-\tif (-T $fd) {\n-\t\treturn 'text/plain' .\n-\t\t       ($default_text_plain_charset ? '; charset='.$default_text_plain_charset : '');\n-\t} elsif (! $filename) {\n-\t\treturn 'application/octet-stream';\n-\t} elsif ($filename =~ m/\\.png$/i) {\n-\t\treturn 'image/png';\n-\t} elsif ($filename =~ m/\\.gif$/i) {\n-\t\treturn 'image/gif';\n-\t} elsif ($filename =~ m/\\.jpe?g$/i) {\n-\t\treturn 'image/jpeg';\n-\t} else {\n-\t\treturn 'application/octet-stream';\n-\t}\n-}\n-\n-sub git_blob_plain {\n-\topen my $fd, \"-|\", \"$gitbin/git-cat-file blob $hash\" or return;\n-\tmy $type = git_blob_plain_mimetype($fd, $file_name);\n-\n-\t# save as filename, even when no $file_name is given\n-\tmy $save_as = \"$hash\";\n-\tif (defined $file_name) {\n-\t\t$save_as = $file_name;\n-\t} elsif ($type =~ m/^text\\//) {\n-\t\t$save_as .= '.txt';\n-\t}\n-\n-\tprint $cgi->header(-type => \"$type\", '-content-disposition' => \"inline; filename=\\\"$save_as\\\"\");\n-\tundef $/;\n-\tbinmode STDOUT, ':raw';\n-\tprint <$fd>;\n-\tbinmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi\n-\t$/ = \"\\n\";\n-\tclose $fd;\n-}\n-\n sub git_tree {\n \tif (!defined $hash) {\n \t\t$hash = git_read_head($project);\n-- \n1.4.1.g2f3c\n\n"},{"id":"23406","messageId":"7vzmfksjpe.fsf@assigned-by-dhcp.cox.net","threadId":"4801","inReplyTo":"20060708041021.24704.qmail@web31804.mail.mud.yahoo.com","subject":"Re: [PATCH] gitweb.cgi: Use File::MMagic; \"a=blob\" action knows the blob/file type","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-07-08T06:18:37Z","receivedAt":"2006-07-08T06:18:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Luben Tuikov <ltuikov@yahoo.com> writes:\n\n> Use File::MMagic to determine the MIME type of a blob/file.\n> The variable magic_mime_file holds the location of the\n> \"magic.mime\" file, usually \"/usr/share/file/magic.mime\".\n> If not defined, the magic numbers internally stored in the\n> File::MMagic module are used.\n\nI am sorry to ask you this, but would you mind redoing this\npatch without File::MMagic bits?  I think giving \"a=blob\" an\nability to automatically switch to git_blob_plain is a good\naddition (as is your earlier patch to give a direct link to\nreach blob_plain from the list), so let's have that part in\nfirst.  I haven't applied your earlier one but it will appear in\n\"next\" shortly.\n\nExisting filename based mimetypes_guess should be a lot cheaper\nthan exploding a blob and feeding it to File::MMagic.  I was\nhoping File::MMagic to be used when we cannot guess the content\ntype that way (i.e. when mimetypes_guess returns undef or\napplication/octet-stream).\n\nSince the repository owner can correct misidentification by the\nstandard /etc/mime.types by supplying a custom per-repository\n$mimetypes_file (modulo that the current implementation of\nmimetype_guess_file does not allow it if the file does not have\nan extension that is specific enough), File::MMagic might be an\noverkill, especially if used in the way this patch does.  To\nallow finer grained differentiation that cannot be done with\nfile extensions alone (e.g. some files may have .dat extension\nbut one can be VCD mpeg wrapped in RIFF, and another can be a\nZ-machine story file), it might be simpler to allow the\nrepository owner to specify full $file_name for such an ambiguous\nfile in their custom $mimetypes_file, and try to match it in\nmimetype_guess_file sub.  That way we may not even need to use\nFile::MMagic.\n\nAre there cases where only $hash is given without $file_name?\nIf so we may need to fall back on File::MMagic in such a case\nafter all, but get_blob_mimetype sub copies the whole blob to a\ntemporary file to work around a problem with version 1.27 you\nstate in the comment -- this is way too much (and nobody seems\nto clean up the tempfile).  Looking at magic.mime, I suspect we\nmight be able to get away with the first 4k bytes or so at most\n(the largest offset except iso9660 image is \"Biff5\" appearing at\n2114 to signal an Excel spreadsheet).\n"},{"id":"23446","messageId":"20060709011756.92415.qmail@web31815.mail.mud.yahoo.com","threadId":"4801","inReplyTo":"7vzmfksjpe.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] gitweb.cgi: Use File::MMagic; \"a=blob\" action knows the blob/file type","fromName":"Luben Tuikov","fromEmail":"ltuikov@yahoo.com","sentAt":"2006-07-09T01:17:56Z","receivedAt":"2006-07-09T01:17:56Z","isPatch":true,"sender":{"key":"ltuikov@yahoo.com","avatar":null},"body":"--- Junio C Hamano <junkio@cox.net> wrote:\n> Luben Tuikov <ltuikov@yahoo.com> writes:\n> \n> > Use File::MMagic to determine the MIME type of a blob/file.\n> > The variable magic_mime_file holds the location of the\n> > \"magic.mime\" file, usually \"/usr/share/file/magic.mime\".\n> > If not defined, the magic numbers internally stored in the\n> > File::MMagic module are used.\n> \n> I am sorry to ask you this, but would you mind redoing this\n> patch without File::MMagic bits?\n\nYeah, no problem, will do.\n\n> I think giving \"a=blob\" an\n> ability to automatically switch to git_blob_plain is a good\n> addition (as is your earlier patch to give a direct link to\n> reach blob_plain from the list), so let's have that part in\n> first.\n\nOk.\n\n> I haven't applied your earlier one but it will appear in\n> \"next\" shortly.\n\nOk.\n\n> Existing filename based mimetypes_guess should be a lot cheaper\n> than exploding a blob and feeding it to File::MMagic.  I was\n> hoping File::MMagic to be used when we cannot guess the content\n> type that way (i.e. when mimetypes_guess returns undef or\n> application/octet-stream).\n\nThe MIME guessing is used in git_blob_plain(), where the blob\nis already exploded, since a file descriptor is passed to the MIME\nguessing routines.\n\nInitially I tried using the already opened file descriptor to\nthe blob, but File::MMagic v1.27 always returns \"text/plain\"\non that.  I was astounded by this and a 5 line perl script\nusing that same module confirmed my suspicion that the module\nor \"seek\" is broken in perl or in FileHandle.  I.e.\nchecktype_filename() succeeds, but checktype_filehandle()\nalways returns \"text/plain\".\n\nHad simply reading the file descriptor succeeded, then we'd not\nneed to explode blobs in $git_temp.\n\nThe other reason I decided to completely go with File::MMagic is\nthat if I'm going to use it to decide the mime type after\nthe default method didn't succeed, why not forget the default\nmethod and just use File::MMagic in general -- seemed like a logical\nsimplification.\n\n> Since the repository owner can correct misidentification by the\n> standard /etc/mime.types by supplying a custom per-repository\n> $mimetypes_file (modulo that the current implementation of\n> mimetype_guess_file does not allow it if the file does not have\n> an extension that is specific enough), File::MMagic might be an\n> overkill, especially if used in the way this patch does.  To\n> allow finer grained differentiation that cannot be done with\n> file extensions alone (e.g. some files may have .dat extension\n> but one can be VCD mpeg wrapped in RIFF, and another can be a\n> Z-machine story file), it might be simpler to allow the\n> repository owner to specify full $file_name for such an ambiguous\n> file in their custom $mimetypes_file, and try to match it in\n> mimetype_guess_file sub.  That way we may not even need to use\n> File::MMagic.\n\nThis is true, but I wonder how many people are going to go and\ncreate their own mime.type files.  Most of the time people would\ndo minimal gitweb.cgi changes not even specifying $mimetypes_file\nnor even $magic_mime_file.\n\n> Are there cases where only $hash is given without $file_name?\n> If so we may need to fall back on File::MMagic in such a case\n> after all, but get_blob_mimetype sub copies the whole blob to a\n> temporary file to work around a problem with version 1.27 you\n> state in the comment -- this is way too much (and nobody seems\n> to clean up the tempfile).  Looking at magic.mime, I suspect we\n> might be able to get away with the first 4k bytes or so at most\n> (the largest offset except iso9660 image is \"Biff5\" appearing at\n> 2114 to signal an Excel spreadsheet).\n\nI was hoping that tmpwatch(8) would clean up the blobs in Linux/UNIX.\nBut we can also delete them after the lookup.  Anothing thing is that\ngetting a blob isn't that often and when it happens, after the MIME\nlookup it is already in the pagecache.\n\n    Luben\n"}]}