# (unknown) 6 messages from 2011-08-04 to 2011-08-10. Participants: Hin-Tak Leung, René Scharfe. Thread: https://gitlist.dev/t/28023 ## Hin-Tak Leung, 2011-08-04 17:27 Subject: (unknown) Message-ID: <1312478843.69439.YahooMailClassic@web29510.mail.ird.yahoo.com> URL: https://gitlist.dev/e/1312478843.69439.YahooMailClassic%40web29510.mail.ird.yahoo.com ``` --- On Thu, 4/8/11, Jeff King wrote: > On Wed, Aug 03, 2011 at 11:17:59PM > +0100, Hin-Tak Leung wrote: > > > The summary of the problem is that, git-archive's > "--format=tar" > > option really write the pax format most of the time, > and some > > cross-platform archive extraction library (rather than > the > > all-powerful GNU tar) really does not think that's the > tar format and > > bail out. > > Out of curiosity, what is the library? Putting pax headers > into ustar > format has been standardized in POSIX since 2001. The R people wrote their own in the R language (and the person who responded to my post in that R-devel thread was possibly the one who wrote that part, I believe) - The main reason is probably cross-platform-ness and independence from other programming languages - they are gradually moving away from invoking system() (because they need to run on windows and they used to require many cygwin/mingw tools to build add-on packages). The R code which does that, scan for certain patterns after seek() into the gunzip'ed stream after 512(?) bytes. It is in a file R/src/library/utils/R/tar.R under the source repository. > That header contains useful information (the commit id from > which the > archive was generated). And there is a way to turn it off: > give a tree > id instead of a commit id. There is an example in the > git-archive > manpage that does exactly this already. Look for the > example mentioning > "pax header" here: > >   http://www.kernel.org/pub/software/scm/git/docs/git-archive.html > > It might be a bit more obvious to find if we actually had > a > --no-pax-header option, though. > > -Peff > I think I tried the tree example and the R code also didn't like it much... may be I'll give it another try. ``` ## René Scharfe, 2011-08-04 17:54 Subject: Re: git-archive's wrong documentation: really write pax rather than tar Message-ID: <4E3ADCCB.6080206@lsrfire.ath.cx> URL: https://gitlist.dev/e/4E3ADCCB.6080206%40lsrfire.ath.cx In-Reply-To: <1312478843.69439.YahooMailClassic@web29510.mail.ird.yahoo.com> ``` Am 04.08.2011 19:27, schrieb Hin-Tak Leung: > --- On Thu, 4/8/11, Jeff King wrote: >> Out of curiosity, what is the library? Putting pax headers into >> ustar format has been standardized in POSIX since 2001. > > The R people wrote their own in the R language (and the person who > responded to my post in that R-devel thread was possibly the one who > wrote that part, I believe) - The main reason is probably > cross-platform-ness and independence from other programming languages > - they are gradually moving away from invoking system() (because they > need to run on windows and they used to require many cygwin/mingw > tools to build add-on packages). > > The R code which does that, scan for certain patterns after seek() > into the gunzip'ed stream after 512(?) bytes. It is in a file > R/src/library/utils/R/tar.R under the source repository. Ah, here it is: https://svn.r-project.org/R/trunk/src/library/utils/R/tar.R It's the ctype handling in function untar2 that rejects unknown entry types. For reference, the documentation of the pax format including a suggestion to treat unknown types like regular files can be found here (search for "typename"): http://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html > I think I tried the tree example and the R code also didn't like it > much... may be I'll give it another try. Did you try adding a ":" to the tree argument, e.g. this: $ git archive HEAD: instead of this? $ git archive HEAD René ``` ## Hin-Tak Leung, 2011-08-04 18:33 Subject: Re: git-archive's wrong documentation: really write pax rather than tar Message-ID: <1312482802.68635.YahooMailClassic@web29518.mail.ird.yahoo.com> URL: https://gitlist.dev/e/1312482802.68635.YahooMailClassic%40web29518.mail.ird.yahoo.com In-Reply-To: <4E3ADCCB.6080206@lsrfire.ath.cx> ``` --- On Thu, 4/8/11, René Scharfe wrote: > From: René Scharfe > Ah, here it is: > >   https://svn.r-project.org/R/trunk/src/library/utils/R/tar.R > > It's the ctype handling in function untar2 that rejects > unknown entry types. > > For reference, the documentation of the pax format > including a > suggestion to treat unknown types like regular files can be > found here > (search for "typename"): > >   http://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html > > > I think I tried the tree example and the R code also > didn't like it > > much... may be I'll give it another try. > > Did you try adding a ":" to the tree argument, e.g. this: > >     $ git archive HEAD: > > instead of this? > >     $ git archive HEAD > > René That's better! With a HEAD: , that code does a lot of: Warning in untar2(tarfile, files, list, exdir) : checksum error for entry 'file...' for each file it tries to extract, but at least it is extracting the files. I wasn't entirely sure about the notation used in the man page - is "v1.4.0^{tree}" same as "v1.4.0:" ? "HEAD:" is clearer, as most people has a HEAD... ``` ## René Scharfe, 2011-08-06 18:57 Subject: Re: git-archive's wrong documentation: really write pax rather than tar Message-ID: <4E3D8EA9.3000609@lsrfire.ath.cx> URL: https://gitlist.dev/e/4E3D8EA9.3000609%40lsrfire.ath.cx In-Reply-To: <1312482802.68635.YahooMailClassic@web29518.mail.ird.yahoo.com> ``` Am 04.08.2011 20:33, schrieb Hin-Tak Leung: > --- On Thu, 4/8/11, René Scharfe > wrote: > >> From: René Scharfe > >> Ah, here it is: >> >> https://svn.r-project.org/R/trunk/src/library/utils/R/tar.R >> >> It's the ctype handling in function untar2 that rejects unknown >> entry types. >> >> For reference, the documentation of the pax format including a >> suggestion to treat unknown types like regular files can be found >> here (search for "typename"): >> >> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html >> >>> I think I tried the tree example and the R code also >> didn't like it >>> much... may be I'll give it another try. >> >> Did you try adding a ":" to the tree argument, e.g. this: >> >> $ git archive HEAD: >> >> instead of this? >> >> $ git archive HEAD >> >> René > > That's better! With a HEAD: , that code does a lot of: > > Warning in untar2(tarfile, files, list, exdir) : checksum error for > entry 'file...' > > for each file it tries to extract, but at least it is extracting the > files. That doesn't sound good. Looking at the R source, however, I can see that they use a two different algorithms to compute the checksum than the one specified by POSIX (even though I don't fully understand what it actually is their doing, since I don't know R). So worry too much about the warning; as long e.g. "tar tf " doesn't complain your archive should be intact. > I wasn't entirely sure about the notation used in the man page > - is "v1.4.0^{tree}" same as "v1.4.0:" ? "HEAD:" is clearer, as most > people has a HEAD... They're a bit different in principle, but point to the same target in this particular case. ":" gets you an object (blob or tree) of a commit; with an empty you get the root tree. "^{}" gives you the requested object of (tag, commit or tree) behind the ; with "tree" you get the root tree of the commit. René ``` ## Hin-Tak Leung, 2011-08-10 03:08 Subject: Re: git-archive's wrong documentation: really write pax rather than tar Message-ID: <1312945714.193.YahooMailClassic@web29510.mail.ird.yahoo.com> URL: https://gitlist.dev/e/1312945714.193.YahooMailClassic%40web29510.mail.ird.yahoo.com In-Reply-To: <4E3D8EA9.3000609@lsrfire.ath.cx> ``` --- On Sat, 6/8/11, René Scharfe wrote: > That doesn't sound good. Looking at the R source, > however, I can see > that they use a two different algorithms to compute the > checksum than > the one specified by POSIX (even though I don't fully > understand what it > actually is their doing, since I don't know R). So > worry too much about > the warning; as long e.g. "tar tf " doesn't > complain your archive > should be intact. I filed the bug, https://bugs.r-project.org/bugzilla3/show_bug.cgi?id=14654 and they have fixed it bug has a few comments to make: --------------- Fixed in R-devel and patched (your checksum field has more than 6 digits which is highly unusual [since it can't be larger than 6 digits] but technically allowable). I should add that the original tar format mandated that checksums are terminated by NUL SPACE, so in that sense your tar file is invalid (there can't be more than 6 digits since the checksum field consists of 8 bytes). untar2 will now be more forgiving, but whatever program created that tar file should be fixed. ----------------- Please feel free to respond directly at the R bug tracking system, or I can cut-and-paste bits of e-mails also... ``` ## René Scharfe, 2011-08-10 19:55 Subject: Re: git-archive's wrong documentation: really write pax rather than tar Message-ID: <4E42E230.5060303@lsrfire.ath.cx> URL: https://gitlist.dev/e/4E42E230.5060303%40lsrfire.ath.cx In-Reply-To: <1312945714.193.YahooMailClassic@web29510.mail.ird.yahoo.com> ``` Am 10.08.2011 05:08, schrieb Hin-Tak Leung: > --- On Sat, 6/8/11, René Scharfe wrote: > >> That doesn't sound good. Looking at the R source, >> however, I can see >> that they use a two different algorithms to compute the >> checksum than >> the one specified by POSIX (even though I don't fully >> understand what it >> actually is their doing, since I don't know R). So >> worry too much about >> the warning; as long e.g. "tar tf " doesn't >> complain your archive >> should be intact. > > I filed the bug, > https://bugs.r-project.org/bugzilla3/show_bug.cgi?id=14654 > and they have fixed it bug has a few comments to make: > > --------------- > Fixed in R-devel and patched (your checksum field has more than 6 digits which > is highly unusual [since it can't be larger than 6 digits] but technically > allowable). > > I should add that the original tar format mandated that checksums are > terminated by NUL SPACE, so in that sense your tar file is invalid (there can't > be more than 6 digits since the checksum field consists of 8 bytes). untar2 > will now be more forgiving, but whatever program created that tar file should > be fixed. > ----------------- That's good to read, thanks. > Please feel free to respond directly at the R bug tracking system, or > I can cut-and-paste bits of e-mails also... I don't think any further action is needed; there's nothing to fix. There is not such thing as "the" tar archive format -- there are multiple dialects. While compatibility is important, ancient versions of tar haven't been a target for git archive so far. It creates files in the same format as POSIX pax. I called it "tar" instead of "pax" because the result can be extracted using GNU tar, bsdtar, 7-Zip etc., supports long file names as well as symlinks -- and I had never heard of anyone actually using the pax command, while tar is quite common. (And pax doesn't restrict the checksum to six digits.) It would be interesting to know which other tar extractor insists on the checksum being stored as six zero-padded octal digits followed by NUL and SPACE instead of seven zero-padded octal digits followed by NUL, though. (Side note: I guess the reason for using only six digits is that this enough for even the biggest possible header checksum.) René ```