{"thread":{"id":"6781","subject":"[PATCH] import-tars: use Archive::Tar instead of unpack()","startedAt":"2007-02-12T14:17:11Z","lastAt":"2007-04-24T10:55:54Z","messageCount":7,"participants":["Michael Loeffler","Johannes Schindelin","Shawn O. Pearce","Karl Hasselström","Sam Vilain"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"34284","messageId":"1171289831.629.6.camel@ibook.zvpunry.de","threadId":"6781","inReplyTo":null,"subject":"[PATCH] import-tars: use Archive::Tar instead of unpack()","fromName":"Michael Loeffler","fromEmail":"zvpunry@zvpunry.de","sentAt":"2007-02-12T14:17:11Z","receivedAt":"2007-02-12T14:17:11Z","isPatch":true,"sender":{"key":"zvpunry@zvpunry.de","avatar":null},"body":"this is less obscure, does not use gzcat (which is often installed as zcat)\nand it is shorter.\n\nSigned-off-by: Michael Loeffler <zvpunry@zvpunry.de>\n---\n\nThis version does no longer support bzip2 or compress which will be fixed in\nan amend. I did this patch to solve 2 problems. Maybe I do another patch with\nGetoptLong and bzip2/compress support.\n\nThe first is a bug with this $git_mode variable which should be 0644 of\n0755, but nothing else I think.\n\nThe second problem was the usage of gzcat, I don't have this link\n(Debian sid).\n\ndiff --git a/contrib/fast-import/import-tars.perl b/contrib/fast-import/import-tars.perl\nindex 26c42c9..c084573 100755\n--- a/contrib/fast-import/import-tars.perl\n+++ b/contrib/fast-import/import-tars.perl\n@@ -10,6 +10,10 @@\n ##\n \n use strict;\n+use Archive::Tar;\n+use Archive::Tar::File;\n+use Archive::Tar::Constant;\n+\n die \"usage: import-tars *.tar.{gz,bz2,Z}\\n\" unless @ARGV;\n \n my $branch_name = 'import-tars';\n@@ -23,48 +27,25 @@ foreach my $tar_file (@ARGV)\n {\n \t$tar_file =~ m,([^/]+)$,;\n \tmy $tar_name = $1;\n-\n-\tif ($tar_name =~ s/\\.(tar\\.gz|tgz)$//) {\n-\t\topen(I, '-|', 'gzcat', $tar_file) or die \"Unable to gzcat $tar_file: $!\\n\";\n-\t} elsif ($tar_name =~ s/\\.(tar\\.bz2|tbz2)$//) {\n-\t\topen(I, '-|', 'bzcat', $tar_file) or die \"Unable to bzcat $tar_file: $!\\n\";\n-\t} elsif ($tar_name =~ s/\\.tar\\.Z$//) {\n-\t\topen(I, '-|', 'zcat', $tar_file) or die \"Unable to zcat $tar_file: $!\\n\";\n-\t} elsif ($tar_name =~ s/\\.tar$//) {\n-\t\topen(I, $tar_file) or die \"Unable to open $tar_file: $!\\n\";\n-\t} else {\n-\t\tdie \"Unrecognized compression format: $tar_file\\n\";\n-\t}\n+\t$tar_name =~ s/\\.(tar|tgz|tar\\.gz)$//;\n+\tmy $tar = new Archive::Tar($tar_file) or die \"Unable to open $tar_file: $!\\n\";\n \n \tmy $commit_time = 0;\n \tmy $next_mark = 1;\n \tmy $have_top_dir = 1;\n \tmy ($top_dir, %files);\n \n-\twhile (read(I, $_, 512) == 512) {\n-\t\tmy ($name, $mode, $uid, $gid, $size, $mtime,\n-\t\t\t$chksum, $typeflag, $linkname, $magic,\n-\t\t\t$version, $uname, $gname, $devmajor, $devminor,\n-\t\t\t$prefix) = unpack 'Z100 Z8 Z8 Z8 Z12 Z12\n-\t\t\tZ8 Z1 Z100 Z6\n-\t\t\tZ2 Z32 Z32 Z8 Z8 Z*', $_;\n-\t\tlast unless $name;\n-\t\t$mode = oct $mode;\n-\t\t$size = oct $size;\n-\t\t$mtime = oct $mtime;\n-\t\tnext if $mode & 0040000;\n-\n-\t\tprint FI \"blob\\n\", \"mark :$next_mark\\n\", \"data $size\\n\";\n-\t\twhile ($size > 0 && read(I, $_, 512) == 512) {\n-\t\t\tprint FI substr($_, 0, $size);\n-\t\t\t$size -= 512;\n-\t\t}\n-\t\tprint FI \"\\n\";\n-\n-\t\tmy $path = \"$prefix$name\";\n-\t\t$files{$path} = [$next_mark++, $mode];\n-\n-\t\t$commit_time = $mtime if $mtime > $commit_time;\n+\tforeach my $entry ($tar->get_files()) {\n+\t\tnext if $entry->type != FILE;\n+\n+\t\tprintf FI \"blob\\nmark :%s\\ndata %s\\n%s\\n\", $next_mark,\n+\t\t\t$entry->size, $entry->get_content();\n+\n+\t\tmy $path = $entry->prefix . $entry->name;\n+\t\t$files{$path} = [$next_mark++, $entry->mode];\n+\n+\t\t$commit_time = $entry->mtime if $entry->mtime > $commit_time;\n+\n \t\t$path =~ m,^([^/]+)/,;\n \t\t$top_dir = $1 unless $top_dir;\n \t\t$have_top_dir = 0 if $top_dir ne $1;\n@@ -83,10 +64,8 @@ EOF\n \tforeach my $path (keys %files)\n \t{\n \t\tmy ($mark, $mode) = @{$files{$path}};\n-\t\tmy $git_mode = 0644;\n-\t\t$git_mode |= 0700 if $mode & 0111;\n \t\t$path =~ s,^([^/]+)/,, if $have_top_dir;\n-\t\tprintf FI \"M %o :%i %s\\n\", $git_mode, $mark, $path;\n+\t\tprintf FI \"M %o :%i %s\\n\", $mode & 0111 ? 0755 : 0644, $mark, $path;\n \t}\n \tprint FI \"\\n\";\n \n-- \n1.5.0.rc4\n"},{"id":"34285","messageId":"Pine.LNX.4.63.0702121524172.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6781","inReplyTo":"1171289831.629.6.camel@ibook.zvpunry.de","subject":"Re: [PATCH] import-tars: use Archive::Tar instead of unpack()","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-02-12T14:25:19Z","receivedAt":"2007-02-12T14:25:19Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 12 Feb 2007, Michael Loeffler wrote:\n\n> this is less obscure, does not use gzcat (which is often installed as \n> zcat) and it is shorter.\n\n... and it relies on a package which is usually not installed. Whereas you \ncan symbolically link zcat to gzcat.\n\nCiao,\nDscho\n"},{"id":"34300","messageId":"20070212172848.GC29621@spearce.org","threadId":"6781","inReplyTo":"1171289831.629.6.camel@ibook.zvpunry.de","subject":"Re: [PATCH] import-tars: use Archive::Tar instead of unpack()","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-02-12T17:28:48Z","receivedAt":"2007-02-12T17:28:48Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Michael Loeffler <zvpunry@zvpunry.de> wrote:\n> This version does no longer support bzip2 or compress which will be fixed in\n> an amend. I did this patch to solve 2 problems. Maybe I do another patch with\n> GetoptLong and bzip2/compress support.\n\nbzip2 and compress are popular formats applied to tars.\n \n> @@ -10,6 +10,10 @@\n>  ##\n>  \n>  use strict;\n> +use Archive::Tar;\n> +use Archive::Tar::File;\n> +use Archive::Tar::Constant;\n> +\n\nI did not apply this hunk.  Not everyone has Archive::Tar installed.\nBut then again, not everyone will use this example program either.\nI'm debating it.  Archive::Tar's parser will certainly be much\nmore robust than the one I hand-crafted.  It might also let us deal\nwith symlinks.  ;)\n\nI would considering applying something like this if it would also\nsupport at least bz2.  This is an example program meant to teach\npeople how to use fast-import, and maybe also to help someone who\nwants to quickly import one or more .tar.gz for use with git-grep.\nRequiring Archive::Tar here is not the end of Git as we know it. :)\n\n> @@ -83,10 +64,8 @@ EOF\n>  \tforeach my $path (keys %files)\n>  \t{\n>  \t\tmy ($mark, $mode) = @{$files{$path}};\n> -\t\tmy $git_mode = 0644;\n> -\t\t$git_mode |= 0700 if $mode & 0111;\n>  \t\t$path =~ s,^([^/]+)/,, if $have_top_dir;\n> -\t\tprintf FI \"M %o :%i %s\\n\", $git_mode, $mark, $path;\n> +\t\tprintf FI \"M %o :%i %s\\n\", $mode & 0111 ? 0755 : 0644, $mark, $path;\n>  \t}\n>  \tprint FI \"\\n\";\n\nThis hunk is completely unrelated to the Archive::Tar rewrite.\nIt also fixes a rather embarrassing bug on my part; I should\nhave been able to get the mode right!  :-)\n\nI've applied this hunk (and only this hunk) to my fastimport tree\nand pushed it out.\n\n-- \nShawn.\n"},{"id":"34543","messageId":"1171468992.629.68.camel@ibook.zvpunry.de","threadId":"6781","inReplyTo":"20070212172848.GC29621@spearce.org","subject":"Re: [PATCH] import-tars: use Archive::Tar instead of unpack()","fromName":"Michael Loeffler","fromEmail":"zvpunry@zvpunry.de","sentAt":"2007-02-14T16:03:12Z","receivedAt":"2007-02-14T16:03:12Z","isPatch":true,"sender":{"key":"zvpunry@zvpunry.de","avatar":null},"body":"Am Montag, den 12.02.2007, 12:28 -0500 schrieb Shawn O. Pearce:\n... \n> bzip2 and compress are popular formats applied to tars.\nyes, and this should be supported. There is an simple example in the\nArchive::Tar manpage.\n\nSo something like \"$tar = new Archive::Tar(*I)\" is possible to use the\noriginal filehandle but I removed this code before reading the whole\nmanpage.\n\n>  \n> > @@ -10,6 +10,10 @@\n> >  ##\n> >  \n> >  use strict;\n> > +use Archive::Tar;\n> > +use Archive::Tar::File;\n> > +use Archive::Tar::Constant;\n> > +\n> \n> I did not apply this hunk.  Not everyone has Archive::Tar installed.\nAnd Archive::Tar is a little bit slower then your unpack() and it reads\nthe whole file into memory which is not so good.\n\n> But then again, not everyone will use this example program either.\nI have used it to play a bit with this nice fast-import stuff.\n\n> I'm debating it.  Archive::Tar's parser will certainly be much\n> more robust than the one I hand-crafted.  It might also let us deal\n> with symlinks.  ;)\nMaybe, but I haven't tested it with some bigger tar files.\n\n> I would considering applying something like this if it would also\n> support at least bz2.  This is an example program meant to teach\n> people how to use fast-import, and maybe also to help someone who\n> wants to quickly import one or more .tar.gz for use with git-grep.\n> Requiring Archive::Tar here is not the end of Git as we know it. :)\nI think we should stay with your unpack() version, it works, is faster\nand doesn't read the whole file into memory.\n\nIn the Archive::Tar manpage they use gunzip and uncompress instead of\nzcat or gzcat, so what do you think about the following patch?\n\ndiff --git a/contrib/fast-import/import-tars.perl b/contrib/fast-import/import-tars.perl\nindex 990c9e7..5585a8b 100755\n--- a/contrib/fast-import/import-tars.perl\n+++ b/contrib/fast-import/import-tars.perl\n@@ -25,11 +25,14 @@ foreach my $tar_file (@ARGV)\n \tmy $tar_name = $1;\n \n \tif ($tar_name =~ s/\\.(tar\\.gz|tgz)$//) {\n-\t\topen(I, '-|', 'gzcat', $tar_file) or die \"Unable to gzcat $tar_file: $!\\n\";\n+\t\topen(I, '-|', 'gunzip', '-c', $tar_file)\n+\t\t\tor die \"Unable to gunzip -c $tar_file: $!\\n\";\n \t} elsif ($tar_name =~ s/\\.(tar\\.bz2|tbz2)$//) {\n-\t\topen(I, '-|', 'bzcat', $tar_file) or die \"Unable to bzcat $tar_file: $!\\n\";\n+\t\topen(I, '-|', 'bunzip2', '-c', $tar_file)\n+\t\t\tor die \"Unable to bunzip2 -c $tar_file: $!\\n\";\n \t} elsif ($tar_name =~ s/\\.tar\\.Z$//) {\n-\t\topen(I, '-|', 'zcat', $tar_file) or die \"Unable to zcat $tar_file: $!\\n\";\n+\t\topen(I, '-|', 'uncompress', '-c', $tar_file)\n+\t\t\tor die \"Unable to uncompress -c $tar_file: $!\\n\";\n \t} elsif ($tar_name =~ s/\\.tar$//) {\n \t\topen(I, $tar_file) or die \"Unable to open $tar_file: $!\\n\";\n \t} else {\n\n\n... the $git_mode patch ...\n> \n> This hunk is completely unrelated to the Archive::Tar rewrite.\nYes, I had this change first in another patch but i forgot to mail it to\nthe list.\n\n...\n> I've applied this hunk (and only this hunk) to my fastimport tree\n> and pushed it out.\nGood, I no longer think this Archive::Tar stuff is such a good idea.\n"},{"id":"34674","messageId":"20070215025159.GA29944@spearce.org","threadId":"6781","inReplyTo":"1171468992.629.68.camel@ibook.zvpunry.de","subject":"Re: [PATCH] import-tars: use Archive::Tar instead of unpack()","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-02-15T02:51:59Z","receivedAt":"2007-02-15T02:51:59Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Michael Loeffler <zvpunry@zvpunry.de> wrote:\n> In the Archive::Tar manpage they use gunzip and uncompress instead of\n> zcat or gzcat, so what do you think about the following patch?\n> \n> diff --git a/contrib/fast-import/import-tars.perl b/contrib/fast-import/import-tars.perl\n> index 990c9e7..5585a8b 100755\n> --- a/contrib/fast-import/import-tars.perl\n> +++ b/contrib/fast-import/import-tars.perl\n> @@ -25,11 +25,14 @@ foreach my $tar_file (@ARGV)\n>  \tmy $tar_name = $1;\n>  \n>  \tif ($tar_name =~ s/\\.(tar\\.gz|tgz)$//) {\n> -\t\topen(I, '-|', 'gzcat', $tar_file) or die \"Unable to gzcat $tar_file: $!\\n\";\n> +\t\topen(I, '-|', 'gunzip', '-c', $tar_file)\n> +\t\t\tor die \"Unable to gunzip -c $tar_file: $!\\n\";\n>  \t} elsif ($tar_name =~ s/\\.(tar\\.bz2|tbz2)$//) {\n> -\t\topen(I, '-|', 'bzcat', $tar_file) or die \"Unable to bzcat $tar_file: $!\\n\";\n> +\t\topen(I, '-|', 'bunzip2', '-c', $tar_file)\n> +\t\t\tor die \"Unable to bunzip2 -c $tar_file: $!\\n\";\n>  \t} elsif ($tar_name =~ s/\\.tar\\.Z$//) {\n> -\t\topen(I, '-|', 'zcat', $tar_file) or die \"Unable to zcat $tar_file: $!\\n\";\n> +\t\topen(I, '-|', 'uncompress', '-c', $tar_file)\n> +\t\t\tor die \"Unable to uncompress -c $tar_file: $!\\n\";\n>  \t} elsif ($tar_name =~ s/\\.tar$//) {\n>  \t\topen(I, $tar_file) or die \"Unable to open $tar_file: $!\\n\";\n>  \t} else {\n\nYes, that's what I should have done initially.  Thanks.  How about\nthe following message and sbo?\n\ncommit 908387056949c0fb8153fbb84f4dbeb6695611e6\nAuthor: Michael Loeffler <zvpunry@zvpunry.de>\nDate:   Wed Feb 14 17:03:12 2007 +0100\n\n    Use gunzip -c over gzcat in import-tars example.\n    \n    Not everyone has gzcat or bzcat installed on their system, but\n    gunzip -c and bunzip2 -c perform the same task and are available\n    if the user has installed gzip support or bzip2 support.\n    \n    Signed-off-by: Michael Loeffler <zvpunry@zvpunry.de>\n    Signed-off-by: Shawn O. Pearce <spearce@spearce.org>\n\n-- \nShawn.\n"},{"id":"40316","messageId":"20070424101323.GC23811@diana.vm.bytemark.co.uk","threadId":"6781","inReplyTo":"127B27FE-1F9A-4328-A87A-77B907FFEBA7@zvpunry.de","subject":"Re: [PATCH] import-tars: use Archive::Tar instead of unpack()","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2007-04-24T10:13:23Z","receivedAt":"2007-04-24T10:13:23Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"On 2007-04-24 10:39:54 +0200, Michael Loeffler wrote:\n\n> Search for this mail and try the patch.\n>\n> > Von: Michael Loeffler <zvpunry@zvpunry.de>\n> > Datum: 12. Februar 2007 15:17:11 MEZ\n> > An: git@vger.kernel.org\n> > Betreff: [PATCH] import-tars: use Archive::Tar instead of unpack()\n\nSorry, the same objection as Shawn raised in the original thread\napplies here too: I don't have Archive::Tar installed! :-(\n\nI might try it later, but I don't really have the time to chase after\nthat dependency right now. Sorry.\n\n-- \nKarl Hasselström, kha@treskal.com\n      www.treskal.com/kalle\n"},{"id":"40318","messageId":"462DE23A.8000501@vilain.net","threadId":"6781","inReplyTo":"20070424101323.GC23811@diana.vm.bytemark.co.uk","subject":"Re: [PATCH] import-tars: use Archive::Tar instead of unpack()","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2007-04-24T10:55:54Z","receivedAt":"2007-04-24T10:55:54Z","isPatch":true,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Karl Hasselström wrote:\n> Sorry, the same objection as Shawn raised in the original thread\n> applies here too: I don't have Archive::Tar installed! :-(\n>\n> I might try it later, but I don't really have the time to chase after\n> that dependency right now. Sorry.\n>   \n\nHow about allowing dependencies, so long as they are included as submodules?\n\nSam.\n"}]}