{"thread":{"id":"18117","subject":"Re: [Orinoco-users] linux-firmware binary corruption with gitweb","startedAt":"2009-03-03T18:59:55Z","lastAt":"2009-03-06T00:03:30Z","messageCount":5,"participants":["Dave","Jakub Narebski","Pavel Roskin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"106911","messageId":"49AD7E2B.3010101@gmail.com","threadId":"18117","inReplyTo":"1235886467.3195.15.camel@mj","subject":"Re: [Orinoco-users] linux-firmware binary corruption with gitweb","fromName":"Dave","fromEmail":"kilroyd@googlemail.com","sentAt":"2009-03-03T18:59:55Z","receivedAt":"2009-03-03T18:59:55Z","isPatch":false,"sender":{"key":"kilroyd@googlemail.com","avatar":null},"body":"Adding the git mailing list.\n\nPavel Roskin wrote:\n> On Sat, 2009-02-28 at 19:24 +0000, Dave wrote:\n>> I'm aware of at least a couple users of orinoco who have picked up\n>> corrupt firmware# from the linux-firmware tree*.\n>>\n>> I've verified that the firmware in the repository itself is correct.\n>>\n>> It appears that downloading the file using the blob/raw links from\n>> gitweb causes the corruption (0xc3 everywhere). At least it does with\n>> firefox.\n> \n> I can confirm the problem with Firefox 3.0.6. But it's not \"0xc3\n> everywhere\". The corrupted file is a result of recoding from iso-8859-1\n> to utf-8. The correct agere_sta_fw.bin is 65046 bytes long. The\n> corrupted agere_sta_fw.bin is 89729 bytes long.\n> \n> There is a way to recode the original binary with GNU recode:\n> recode utf8..iso8859-1 agere_sta_fw.bin\n> \n> wget 1.11.4 also gets a corrupted file 89729 bytes long.\n> \n> curl 7.18.2 also get the corrupted file:\n> \n> My strong impression is that the recoding takes place on the server. I\n> think the bug should be reported to the gitweb maintainers unless it a\n> local breakage on the kernel.org site.\n\nThanks Pavel.\n\nI just did a quick scan of the gitweb README - is this an issue with the\n$mimetypes_file or $fallback_encoding configurations variables?\n\n\nRegards,\n\nDave.\n\n#\n*\n"},{"id":"106939","messageId":"m3iqmqt9ox.fsf@localhost.localdomain","threadId":"18117","inReplyTo":"49AD7E2B.3010101@gmail.com","subject":"Re: [Orinoco-users] linux-firmware binary corruption with gitweb","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-03-04T00:26:28Z","receivedAt":"2009-03-04T00:26:28Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Dave writes:\n\n> Adding the git mailing list.\n> \n> Pavel Roskin wrote:\n> > On Sat, 2009-02-28 at 19:24 +0000, Dave wrote:\n\n>>> I'm aware of at least a couple users of orinoco who have picked up\n>>> corrupt firmware# from the linux-firmware tree*.\n>>>\n>>> I've verified that the firmware in the repository itself is correct.\n>>>\n>>> It appears that downloading the file using the blob/raw links from\n>>> gitweb causes the corruption (0xc3 everywhere). At least it does with\n>>> firefox.\n>> \n>> I can confirm the problem with Firefox 3.0.6. But it's not \"0xc3\n>> everywhere\". The corrupted file is a result of recoding from iso-8859-1\n>> to utf-8. The correct agere_sta_fw.bin is 65046 bytes long. The\n>> corrupted agere_sta_fw.bin is 89729 bytes long.\n\n[...]\n>> My strong impression is that the recoding takes place on the server. I\n>> think the bug should be reported to the gitweb maintainers unless it a\n>> local breakage on the kernel.org site.\n> \n> Thanks Pavel.\n> \n> I just did a quick scan of the gitweb README - is this an issue with the\n> $mimetypes_file or $fallback_encoding configurations variables?\n\nFirst, what version of gitweb do you use? It should be in 'Generator'\nmeta header, or (in older gitweb) in comments in HTML source at the\ntop of the page.\n\nSecond, the file is actually sent to browser 'as is', using binmode :raw\n(or at least should be according to my understanding of Perl). And *.bin\nbinary file gets application/octet-stream mimetype, and doesn't send any\ncharset info. git.kernel.org should have modern enough gitweb to use this.\nStrange...\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"107009","messageId":"49AF1429.9080009@gmail.com","threadId":"18117","inReplyTo":"m3iqmqt9ox.fsf@localhost.localdomain","subject":"Re: [Orinoco-users] linux-firmware binary corruption with gitweb","fromName":"Dave","fromEmail":"kilroyd@googlemail.com","sentAt":"2009-03-04T23:52:09Z","receivedAt":"2009-03-04T23:52:09Z","isPatch":false,"sender":{"key":"kilroyd@googlemail.com","avatar":null},"body":"Jakub Narebski wrote:\n> Dave writes:\n>>> My strong impression is that the recoding takes place on the server. I\n>>> think the bug should be reported to the gitweb maintainers unless it a\n>>> local breakage on the kernel.org site.\n>> Thanks Pavel.\n>>\n>> I just did a quick scan of the gitweb README - is this an issue with the\n>> $mimetypes_file or $fallback_encoding configurations variables?\n> \n> First, what version of gitweb do you use? It should be in 'Generator'\n> meta header, or (in older gitweb) in comments in HTML source at the\n> top of the page.\n\nNot sure where I'd find the meta header, but at the top of the HTML:\n\n\n\n\n> Second, the file is actually sent to browser 'as is', using binmode :raw\n> (or at least should be according to my understanding of Perl). And *.bin\n> binary file gets application/octet-stream mimetype, and doesn't send any\n> charset info. git.kernel.org should have modern enough gitweb to use this.\n> Strange...\n\nDug around gitweb.perl in the main git repo. Then looked at the\ngit/warthog9/gitweb.git repo (after noting the Git Wiki says kernel.org\nis running John Hawley's branch).\n\nOne notable change to git_blob_plain:\n\n undef $/;\n binmode STDOUT, ':raw';\n- print <$fd>;\n+ #print <$fd>;\n+ $output .= <$fd>;\n binmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi\n $/ = \"\\n\";\n\n close $fd;\n+\n+ return $output;\n\nIf that's the code that's running, doesn't that mean the output mode\nchange doesn't impact the concatenation to $output? So the blob gets utf\nencoding when actually printed.\n\n\nRegards,\n\nDave.\n"},{"id":"107083","messageId":"1236273981.24072.16.camel@mj","threadId":"18117","inReplyTo":"49AF1429.9080009-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org","subject":"Re: linux-firmware binary corruption with gitweb","fromName":"Pavel Roskin","fromEmail":"proski-mxxj517/zsq@public.gmane.org","sentAt":"2009-03-05T17:26:21Z","receivedAt":"2009-03-05T17:26:21Z","isPatch":false,"sender":{"key":"proski-mxxj517/zsq@public.gmane.org","avatar":null},"body":"On Wed, 2009-03-04 at 23:52 +0000, Dave wrote:\n> binmode STDOUT, ':raw';\n> - print <$fd>;\n> + #print <$fd>;\n> + $output .= <$fd>;\n> binmode STDOUT, ':utf8'; # as set at the beginning of\n> gitweb.cgi\n\nNice catch!\n\nLooking at the gitweb repository from kernel.org, two instances of\ncircumventing binmode were introduced by this commit:\n\ncommit c79ae555fb3c89d91b4cafbfce306e695720507b\nAuthor: John Hawley \nDate: Thu Dec 28 21:59:43 2006 -0800\n\n Last of the changes to deal with channeling the text through the caching\n engine. Wow is this a total hack.\n\nThe original behavior was restored in git_snapshot() by the recent\ncommit c15229acd9bedf165f1eb05d99fa989d3b9f3e32, but git_blob_plain()\nremains broken.\n\nI don't see an easy fix. We cannot manipulate the blob to counteract\nthe encoding, as it may not be valid utf-8, and therefore won't be\noutput in the utf-8 mode.\n\nMaybe binmode should be raw everywhere, and adding to $output should\nrecode data to utf-8 from other encodings where needed, but it would be\na massive patch, I'm afraid. Or it would be a small patch requiring\nmassive testing.\n\nAdding John Hawley to cc:\n\n-- \nRegards,\nPavel Roskin\n\n------------------------------------------------------------------------------\nOpen Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA\n-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise\n-Strategies to boost innovation and cut costs with open source participation\n-Receive a $600 discount off the registration fee with the source code: SFAD\nhttp://p.sf.net/sfu/XcvMzF8H\n"},{"id":"107121","messageId":"200903060103.32653.jnareb@gmail.com","threadId":"18117","inReplyTo":"49AF1429.9080009@gmail.com","subject":"Re: [Orinoco-users] linux-firmware binary corruption with gitweb","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-03-06T00:03:30Z","receivedAt":"2009-03-06T00:03:30Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Thu, 5 March 2009, Dave wrote:\n> Jakub Narebski wrote:\n>> Dave writes:\n\n>>>> My strong impression is that the recoding takes place on the server. I\n>>>> think the bug should be reported to the gitweb maintainers unless it a\n>>>> local breakage on the kernel.org site.\n\nIt is on server, but kernel.org runs modified version of gitweb, and\nthe bug is in the modifications. See below.\n\nCC-ed John 'Warthog9' Hawley, maintainer of gitweb on kernel.org\n\n>>>>\n>>> Thanks Pavel.\n>>>\n>>> I just did a quick scan of the gitweb README - is this an issue with the\n>>> $mimetypes_file or $fallback_encoding configurations variables?\n>> \n>> First, what version of gitweb do you use? It should be in 'Generator'\n>> meta header, or (in older gitweb) in comments in HTML source at the\n>> top of the page.\n> \n> Not sure where I'd find the meta header,\n\n\n\n> but at the top of the HTML: \n> \n> \n> \n\nThe question was if it is extremely old version of gitweb, without fix\nof raw blob ('blob_plain') output for non-utf8, non-text files. But the\nanswer is that it is _modified_ version of gitweb, see below.\n\n> \n>> Second, the file is actually sent to browser 'as is', using binmode :raw\n>> (or at least should be according to my understanding of Perl). And *.bin\n>> binary file gets application/octet-stream mimetype, and doesn't send any\n>> charset info. git.kernel.org should have modern enough gitweb to use this.\n>> Strange...\n> \n> Dug around gitweb.perl in the main git repo. Then looked at the\n> git/warthog9/gitweb.git repo (after noting the Git Wiki says kernel.org\n> is running John Hawley's branch).\n> \n> One notable change to git_blob_plain:\n> \n> undef $/;\n> binmode STDOUT, ':raw';\n> - print <$fd>;\n> + #print <$fd>;\n> + $output .= <$fd>;\n> binmode STDOUT, ':utf8'; # as set at the beginning of gitweb.cgi\n> $/ = \"\\n\";\n> \n> close $fd;\n> +\n> + return $output;\n> \n> If that's the code that's running, doesn't that mean the output mode\n> change doesn't impact the concatenation to $output? So the blob gets utf\n> encoding when actually printed.\n\nThat is the culprit. kernel.org runs modified version of gitweb, with\nadded caching. I guess that the above change was to have 'blob_plain'\noutput cached... but it loses \"rawness\", and I guess it also loses\nmimetype info (unless \"print $cgi->header(...)\" is also changed to\nappending to $output).\n\nOne possible solution would be to redirect STDOUT to scalar, and return\nthat scalar; do that always when caching _output_, and print :raw all\ncached _output_ data.\n close STDOUT;\n open STDOUT, '>', \\$output or die \"Can't open STDOUT: $!\";\n\n\nBTW. f5aa79d (gitweb: safely output binary files for 'blob_plain' action)\nwas my third patch for git...\n\n-- \nJakub Narebski\nPoland\n"}]}