{"thread":{"id":"28023","subject":"(unknown)","startedAt":"2011-08-04T17:27:23Z","lastAt":"2011-08-10T19:55:28Z","messageCount":6,"participants":["Hin-Tak Leung","René Scharfe"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"172936","messageId":"1312478843.69439.YahooMailClassic@web29510.mail.ird.yahoo.com","threadId":"28023","inReplyTo":null,"subject":"(unknown)","fromName":"Hin-Tak Leung","fromEmail":"htl10@users.sourceforge.net","sentAt":"2011-08-04T17:27:23Z","receivedAt":"2011-08-04T17:27:23Z","isPatch":false,"sender":{"key":"htl10@users.sourceforge.net","avatar":null},"body":"--- On Thu, 4/8/11, Jeff King wrote:\n\n> On Wed, Aug 03, 2011 at 11:17:59PM\n> +0100, Hin-Tak Leung wrote:\n> \n> > The summary of the problem is that, git-archive's\n> \"--format=tar\"\n> > option really write the pax format most of the time,\n> and some\n> > cross-platform archive extraction library (rather than\n> the\n> > all-powerful GNU tar) really does not think that's the\n> tar format and\n> > bail out.\n> \n> Out of curiosity, what is the library? Putting pax headers\n> into ustar\n> format has been standardized in POSIX since 2001.\n\nThe R people wrote their own in the R language (and the person who responded to my post in that R-devel thread was possibly the one who wrote that part, I believe) - The main reason is probably cross-platform-ness and independence from other programming languages - they are gradually moving away from invoking system() (because they need to run on windows and they used to require many cygwin/mingw tools to build add-on packages).\n\nThe R code which does that, scan for certain patterns after seek() into the gunzip'ed stream after 512(?) bytes. It is in a file \nR/src/library/utils/R/tar.R under the source repository.\n\n> That header contains useful information (the commit id from\n> which the\n> archive was generated). And there is a way to turn it off:\n> give a tree\n> id instead of a commit id. There is an example in the\n> git-archive\n> manpage that does exactly this already. Look for the\n> example mentioning\n> \"pax header\" here:\n> \n>   http://www.kernel.org/pub/software/scm/git/docs/git-archive.html\n> \n> It might be a bit more obvious to find if we actually had\n> a\n> --no-pax-header option, though.\n> \n> -Peff\n> \n\n\nI think I tried the tree example and the R code also didn't like it much... may be I'll give it another try.\n"},{"id":"172940","messageId":"4E3ADCCB.6080206@lsrfire.ath.cx","threadId":"28023","inReplyTo":"1312478843.69439.YahooMailClassic@web29510.mail.ird.yahoo.com","subject":"Re: git-archive's wrong documentation: really write pax rather than tar","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2011-08-04T17:54:19Z","receivedAt":"2011-08-04T17:54:19Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 04.08.2011 19:27, schrieb Hin-Tak Leung:\n> --- On Thu, 4/8/11, Jeff King wrote:\n>> Out of curiosity, what is the library? Putting pax headers into\n>> ustar format has been standardized in POSIX since 2001.\n> \n> The R people wrote their own in the R language (and the person who\n> responded to my post in that R-devel thread was possibly the one who\n> wrote that part, I believe) - The main reason is probably\n> cross-platform-ness and independence from other programming languages\n> - they are gradually moving away from invoking system() (because they\n> need to run on windows and they used to require many cygwin/mingw\n> tools to build add-on packages).\n> \n> The R code which does that, scan for certain patterns after seek()\n> into the gunzip'ed stream after 512(?) bytes. It is in a file \n> R/src/library/utils/R/tar.R under the source repository.\n\nAh, here it is:\n\n https://svn.r-project.org/R/trunk/src/library/utils/R/tar.R\n\nIt's the ctype handling in function untar2 that rejects unknown entry types.\n\nFor reference, the documentation of the pax format including a\nsuggestion to treat unknown types like regular files can be found here\n(search for \"typename\"):\n\n http://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html\n\n> I think I tried the tree example and the R code also didn't like it\n> much... may be I'll give it another try.\n\nDid you try adding a \":\" to the tree argument, e.g. this:\n\n\t$ git archive HEAD:\n\ninstead of this?\n\n\t$ git archive HEAD\n\nRené\n"},{"id":"172942","messageId":"1312482802.68635.YahooMailClassic@web29518.mail.ird.yahoo.com","threadId":"28023","inReplyTo":"4E3ADCCB.6080206@lsrfire.ath.cx","subject":"Re: git-archive's wrong documentation: really write pax rather than tar","fromName":"Hin-Tak Leung","fromEmail":"htl10@users.sourceforge.net","sentAt":"2011-08-04T18:33:22Z","receivedAt":"2011-08-04T18:33:22Z","isPatch":false,"sender":{"key":"htl10@users.sourceforge.net","avatar":null},"body":"--- On Thu, 4/8/11, René Scharfe wrote:\n\n> From: René Scharfe \n\n> Ah, here it is:\n> \n>   https://svn.r-project.org/R/trunk/src/library/utils/R/tar.R\n> \n> It's the ctype handling in function untar2 that rejects\n> unknown entry types.\n> \n> For reference, the documentation of the pax format\n> including a\n> suggestion to treat unknown types like regular files can be\n> found here\n> (search for \"typename\"):\n> \n>   http://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html\n> \n> > I think I tried the tree example and the R code also\n> didn't like it\n> > much... may be I'll give it another try.\n> \n> Did you try adding a \":\" to the tree argument, e.g. this:\n> \n>     $ git archive HEAD:\n> \n> instead of this?\n> \n>     $ git archive HEAD\n> \n> René\n\nThat's better! With a HEAD: , that code does a lot of: \n\nWarning in untar2(tarfile, files, list, exdir) :\n checksum error for entry 'file...'\n\nfor each file it tries to extract, but at least it is extracting the files. I wasn't entirely sure about the notation used in the man page - is\n\"v1.4.0^{tree}\" same as \"v1.4.0:\" ? \"HEAD:\" is clearer, as most people has a HEAD... \n"},{"id":"173067","messageId":"4E3D8EA9.3000609@lsrfire.ath.cx","threadId":"28023","inReplyTo":"1312482802.68635.YahooMailClassic@web29518.mail.ird.yahoo.com","subject":"Re: git-archive's wrong documentation: really write pax rather than tar","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2011-08-06T18:57:45Z","receivedAt":"2011-08-06T18:57:45Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 04.08.2011 20:33, schrieb Hin-Tak Leung:\n> --- On Thu, 4/8/11, René Scharfe \n> wrote:\n> \n>> From: René Scharfe \n> \n>> Ah, here it is:\n>> \n>> https://svn.r-project.org/R/trunk/src/library/utils/R/tar.R\n>> \n>> It's the ctype handling in function untar2 that rejects unknown\n>> entry types.\n>> \n>> For reference, the documentation of the pax format including a \n>> suggestion to treat unknown types like regular files can be found\n>> here (search for \"typename\"):\n>> \n>> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html\n>> \n>>> I think I tried the tree example and the R code also\n>> didn't like it\n>>> much... may be I'll give it another try.\n>> \n>> Did you try adding a \":\" to the tree argument, e.g. this:\n>> \n>> $ git archive HEAD:\n>> \n>> instead of this?\n>> \n>> $ git archive HEAD\n>> \n>> René\n> \n> That's better! With a HEAD: , that code does a lot of:\n> \n> Warning in untar2(tarfile, files, list, exdir) : checksum error for\n> entry 'file...'\n> \n> for each file it tries to extract, but at least it is extracting the\n> files. \n\nThat doesn't sound good. Looking at the R source, however, I can see\nthat they use a two different algorithms to compute the checksum than\nthe one specified by POSIX (even though I don't fully understand what it\nactually is their doing, since I don't know R). So worry too much about\nthe warning; as long e.g. \"tar tf \" doesn't complain your archive\nshould be intact.\n\n> I wasn't entirely sure about the notation used in the man page\n> - is \"v1.4.0^{tree}\" same as \"v1.4.0:\" ? \"HEAD:\" is clearer, as most\n> people has a HEAD...\n\nThey're a bit different in principle, but point to the same target in\nthis particular case. \":\" gets you an object (blob or tree)\nof a commit; with an empty you get the root tree.\n\"^{}\" gives you the requested object of (tag, commit\nor tree) behind the ; with \"tree\" you get the root tree of the commit.\n\nRené\n"},{"id":"173230","messageId":"1312945714.193.YahooMailClassic@web29510.mail.ird.yahoo.com","threadId":"28023","inReplyTo":"4E3D8EA9.3000609@lsrfire.ath.cx","subject":"Re: git-archive's wrong documentation: really write pax rather than tar","fromName":"Hin-Tak Leung","fromEmail":"htl10@users.sourceforge.net","sentAt":"2011-08-10T03:08:34Z","receivedAt":"2011-08-10T03:08:34Z","isPatch":false,"sender":{"key":"htl10@users.sourceforge.net","avatar":null},"body":"--- On Sat, 6/8/11, René Scharfe wrote:\n\n> That doesn't sound good. Looking at the R source,\n> however, I can see\n> that they use a two different algorithms to compute the\n> checksum than\n> the one specified by POSIX (even though I don't fully\n> understand what it\n> actually is their doing, since I don't know R). So\n> worry too much about\n> the warning; as long e.g. \"tar tf \" doesn't\n> complain your archive\n> should be intact.\n\nI filed the bug,\nhttps://bugs.r-project.org/bugzilla3/show_bug.cgi?id=14654\nand they have fixed it bug has a few comments to make:\n\n---------------\nFixed in R-devel and patched (your checksum field has more than 6 digits which\nis highly unusual [since it can't be larger than 6 digits] but technically\nallowable).\n\nI should add that the original tar format mandated that checksums are\nterminated by NUL SPACE, so in that sense your tar file is invalid (there can't\nbe more than 6 digits since the checksum field consists of 8 bytes). untar2\nwill now be more forgiving, but whatever program created that tar file should\nbe fixed.\n-----------------\n\nPlease feel free to respond directly at the R bug tracking system, or I can cut-and-paste bits of e-mails also...\n"},{"id":"173249","messageId":"4E42E230.5060303@lsrfire.ath.cx","threadId":"28023","inReplyTo":"1312945714.193.YahooMailClassic@web29510.mail.ird.yahoo.com","subject":"Re: git-archive's wrong documentation: really write pax rather than tar","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2011-08-10T19:55:28Z","receivedAt":"2011-08-10T19:55:28Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 10.08.2011 05:08, schrieb Hin-Tak Leung:\n> --- On Sat, 6/8/11, René Scharfe wrote:\n> \n>> That doesn't sound good. Looking at the R source,\n>> however, I can see\n>> that they use a two different algorithms to compute the\n>> checksum than\n>> the one specified by POSIX (even though I don't fully\n>> understand what it\n>> actually is their doing, since I don't know R). So\n>> worry too much about\n>> the warning; as long e.g. \"tar tf \" doesn't\n>> complain your archive\n>> should be intact.\n> \n> I filed the bug,\n> https://bugs.r-project.org/bugzilla3/show_bug.cgi?id=14654\n> and they have fixed it bug has a few comments to make:\n> \n> ---------------\n> Fixed in R-devel and patched (your checksum field has more than 6 digits which\n> is highly unusual [since it can't be larger than 6 digits] but technically\n> allowable).\n> \n> I should add that the original tar format mandated that checksums are\n> terminated by NUL SPACE, so in that sense your tar file is invalid (there can't\n> be more than 6 digits since the checksum field consists of 8 bytes). untar2\n> will now be more forgiving, but whatever program created that tar file should\n> be fixed.\n> -----------------\n\nThat's good to read, thanks.\n\n> Please feel free to respond directly at the R bug tracking system, or\n> I can cut-and-paste bits of e-mails also...\n\nI don't think any further action is needed; there's nothing to fix.\n\nThere is not such thing as \"the\" tar archive format -- there are\nmultiple dialects. While compatibility is important, ancient versions\nof tar haven't been a target for git archive so far.\n\nIt creates files in the same format as POSIX pax. I called it \"tar\"\ninstead of \"pax\" because the result can be extracted using GNU tar,\nbsdtar, 7-Zip etc., supports long file names as well as symlinks -- and\nI had never heard of anyone actually using the pax command, while tar is\nquite common. (And pax doesn't restrict the checksum to six digits.)\n\nIt would be interesting to know which other tar extractor insists on the\nchecksum being stored as six zero-padded octal digits followed by NUL\nand SPACE instead of seven zero-padded octal digits followed by NUL,\nthough. (Side note: I guess the reason for using only six digits is\nthat this enough for even the biggest possible header checksum.)\n\nRené\n"}]}