{"thread":{"id":"2466","subject":"[PATCH] archimport improvements","startedAt":"2005-11-12T09:23:37Z","lastAt":"2005-12-05T18:53:19Z","messageCount":39,"participants":["Eric Wong","Martin Langhoff","Andreas Ericsson","Junio C Hamano","Linus Torvalds"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"11653","messageId":"20051112092336.GA16218@Muzzle","threadId":"2466","inReplyTo":null,"subject":"[PATCH] archimport improvements","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-12T09:23:37Z","receivedAt":"2005-11-12T09:23:37Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Hello,\n\nI'm another Arch-user trying out git. Unfortunately, I encountered\nseveral problems with git-archimport that I needed fixed before my\ndevelopment trees could be imported into git.\n\nHere's a summary of the changes:\n\nBug Fixes:\n\n* Support for '--branch'-less Arch version names.\n Encoding '/' to '--' (as was previously done) is not 100% reversable\n because the \"--branch\" portion of an fully-qualified Arch version name\n is optional (though not many people or Arch-related tools know this).\n\n* I'm encoding the '/' in the fully-qualified name as ',' to not confuse\n other porcelains, but leaving '/' in branch names may be alright\n provided porcelains can support them.\n\n* Identify git branches as an Arch \"archive,category<--branch>--version\"\n Anything less than that is ambiguous as far as history and patch\n relationships go.\n\n* Renamed directories containing renamed/moved files inside didn't get\n tracked properly. The original code was inadequate for this, and\n making it support all rename cases that Arch supports is too much\n work. Instead, I maintain full-blown Arch trees in the temp dir and\n replay patches + rsync based on that. Performance is slightly slower\n than before, but accuracy is more important to me.\n\n* Permission (execute bit only because of git) tracking as a side effect\n of the above.\n\n* Tracking changes from branches that are only cherry-picked now works\n\n* Pika-escaped filenames unhandled. This seems fixed in the latest\n git, but I fixed it more generally and removed the ShellQuote module\n dependency along the way.\n\n* Don't die() when a merge-base can't be found. Arch supports\n merging between unrelated trees.\n\n\nUsability enhancements:\n\n* Optionally detect merged branches and attempt to import their history,\n too. Use the -D option for this. Specifying a \n greater than 1 is usually not needed unless the tree you're tracking\n has had history pruned.\n \n* Optionally attempt to auto-register unknown Arch archives from\n mirrors.sourcecontrol.net to pull their history with the -a (boolean)\n switch. Not sure how useful users will find this.\n\n* Removed -A usage (unnecessary in all cases) and made all\n Arch calls and output parsing to be compatible with both tla (tested\n 1.3.3) and baz (1.4.2). Default is still tla, but the ARCH_CLIENT\n environment variable may be changed to baz.\n\n\nCurrent weaknesses:\n\n* (Present in the original code as well).\n The code still assumes that dates in commit logs can be trusted, which is\n fine in most cases, but a wayward branch can screw up git-archimport and\n cause parents to be missed.\n\n-- \nEric Wong\n"},{"id":"11654","messageId":"20051112092533.GB16218@Muzzle","threadId":"2466","inReplyTo":"20051112092336.GA16218@Muzzle","subject":"[PATCH 1/5] remove shellquote usage for tags","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-12T09:25:34Z","receivedAt":"2005-11-12T09:25:34Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"use ',' to encode '/' in \"archivename/foo--bar--0.0\" so we can allow\n\"--branch\"-less trees which are valid in Arch (\"archivename/foo--0.0\")\n\nSigned-off-by: Eric Wong \n\n---\n\n git-archimport.perl | 55 ++++++++++++++++++++++++++-------------------------\n 1 files changed, 28 insertions(+), 27 deletions(-)\n\napplies-to: 76d3d1c302c20b82fd976e958aabd19f7f01e7b5\n28d4f9ee8ba83b35eea66d4dd19b8ec26a0218c7\ndiff --git a/git-archimport.perl b/git-archimport.perl\nindex e22c816..7c15184 100755\n--- a/git-archimport.perl\n+++ b/git-archimport.perl\n@@ -52,6 +52,7 @@ $ENV{'TZ'}=\"UTC\";\n \n my $git_dir = $ENV{\"GIT_DIR\"} || \".git\";\n $ENV{\"GIT_DIR\"} = $git_dir;\n+my $ptag_dir = \"$git_dir/archimport/tags\";\n \n our($opt_h,$opt_v, $opt_T,\n $opt_C,$opt_t);\n@@ -195,16 +196,19 @@ unless (-d $git_dir) { # initial import\n opendir(DIR, \"$git_dir/archimport/tags\")\n \t|| die \"can't opendir: $!\";\n while (my $file = readdir(DIR)) {\n-\t# skip non-interesting-files\n-\tnext unless -f \"$git_dir/archimport/tags/$file\";\n-\tnext if $file =~ m/--base-0$/; # don't care for base-0\n+ # skip non-interesting-files\n+ next unless -f \"$ptag_dir/$file\";\n+ \n+ # convert first '--' to '/' from old git-archimport to use\n+ # as an archivename/c--b--v private tag\n+ if ($file !~ m!,!) {\n+ my $oldfile = $file;\n+ $file =~ s!--!,!;\n+ print STDERR \"converting old tag $oldfile to $file\\n\";\n+ rename(\"$ptag_dir/$oldfile\", \"$ptag_dir/$file\") or die $!;\n+ }\n \tmy $sha = ptag($file);\n \tchomp $sha;\n-\t# reconvert the 3rd '--' sequence from the end\n-\t# into a slash\n-\t# $file = reverse $file;\n-\t# $file =~ s!^(.+?--.+?--.+?--.+?)--(.+)$!$1/$2!;\n-\t# $file = reverse $file;\n \t$rptags{$sha} = $file;\n }\n closedir DIR;\n@@ -582,19 +586,20 @@ sub parselog {\n # write/read a tag\n sub tag {\n my ($tag, $commit) = @_;\n- $tag =~ s|/|--|g; \n- $tag = shell_quote($tag);\n+ \n+ # don't use subdirs for tags yet, it could screw up other porcelains\n+ $tag =~ s|/|,|;\n \n if ($commit) {\n- open(C,\">$git_dir/refs/tags/$tag\")\n+ open(C,\">\",\"$git_dir/refs/tags/$tag\")\n or die \"Cannot create tag $tag: $!\\n\";\n print C \"$commit\\n\"\n or die \"Cannot write tag $tag: $!\\n\";\n close(C)\n or die \"Cannot write tag $tag: $!\\n\";\n- print \" * Created tag ' $tag' on '$commit'\\n\" if $opt_v;\n+ print \" * Created tag '$tag' on '$commit'\\n\" if $opt_v;\n } else { # read\n- open(C,\"<$git_dir/refs/tags/$tag\")\n+ open(C,\"<\",\"$git_dir/refs/tags/$tag\")\n or die \"Cannot read tag $tag: $!\\n\";\n $commit = ;\n chomp $commit;\n@@ -609,15 +614,16 @@ sub tag {\n # reads fail softly if the tag isn't there\n sub ptag {\n my ($tag, $commit) = @_;\n- $tag =~ s|/|--|g; \n- $tag = shell_quote($tag);\n+\n+ # don't use subdirs for tags yet, it could screw up other porcelains\n+ $tag =~ s|/|,|g; \n \n- unless (-d \"$git_dir/archimport/tags\") {\n- mkpath(\"$git_dir/archimport/tags\");\n- }\n+ my $tag_file = \"$ptag_dir/$tag\";\n+ my $tag_branch_dir = dirname($tag_file);\n+ mkpath($tag_branch_dir) unless (-d $tag_branch_dir);\n \n if ($commit) { # write\n- open(C,\">$git_dir/archimport/tags/$tag\")\n+ open(C,\">\",$tag_file)\n or die \"Cannot create tag $tag: $!\\n\";\n print C \"$commit\\n\"\n or die \"Cannot write tag $tag: $!\\n\";\n@@ -627,10 +633,10 @@ sub ptag {\n \t unless $tag =~ m/--base-0$/;\n } else { # read\n # if the tag isn't there, return 0\n- unless ( -s \"$git_dir/archimport/tags/$tag\") {\n+ unless ( -s $tag_file) {\n return 0;\n }\n- open(C,\"<$git_dir/archimport/tags/$tag\")\n+ open(C,\"<\",$tag_file)\n or die \"Cannot read tag $tag: $!\\n\";\n $commit = ;\n chomp $commit;\n@@ -780,12 +786,7 @@ sub commitid2pset {\n chomp $commitid;\n my $name = $rptags{$commitid} \n \t|| die \"Cannot find reverse tag mapping for $commitid\";\n- # the keys in %rptag are slightly munged; unmunge\n- # reconvert the 3rd '--' sequence from the end\n- # into a slash\n- $name = reverse $name;\n- $name =~ s!^(.+?--.+?--.+?--.+?)--(.+)$!$1/$2!;\n- $name = reverse $name;\n+ $name =~ s|,|/|;\n my $ps = $psets{$name} \n \t|| (print Dumper(sort keys %psets)) && die \"Cannot find patchset for $name\";\n return $ps;\n---\n0.99.9.GIT\n"},{"id":"11655","messageId":"20051112092721.GC16218@Muzzle","threadId":"2466","inReplyTo":"20051112092533.GB16218@Muzzle","subject":"[PATCH 2/5] archimport: don't die on merge-base failure","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-12T09:27:22Z","receivedAt":"2005-11-12T09:27:22Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Don't die if we can't find a merge base, Arch allows arbitrary\ncherry-picks between unrelated branches and we should not\ndie when that happens\n\nSigned-off-by: Eric Wong \n\n---\n\n git-archimport.perl | 8 +++++++-\n 1 files changed, 7 insertions(+), 1 deletions(-)\n\napplies-to: 07dfd96ba53890d6a20fa0b028cf96e0e49bc027\n7d099adadc041d74a0defc107656f273b35f57cb\ndiff --git a/git-archimport.perl b/git-archimport.perl\nindex 7c15184..699d5f6 100755\n--- a/git-archimport.perl\n+++ b/git-archimport.perl\n@@ -693,7 +693,13 @@ sub find_parents {\n \tnext unless -e \"$git_dir/refs/heads/$branch\";\n \n \tmy $mergebase = `git-merge-base $branch $ps->{branch}`;\n-\tdie \"Cannot find merge base for $branch and $ps->{branch}\" if $?;\n+ \tif ($?) { \n+ \t # Don't die here, Arch supports one-way cherry-picking\n+ \t # between branches with no common base (or any relationship\n+ \t # at all beforehand)\n+ \t warn \"Cannot find merge base for $branch and $ps->{branch}\";\n+ \t next;\n+ \t}\n \tchomp $mergebase;\n \n \t# now walk up to the mergepoint collecting what patches we have\n---\n0.99.9.GIT\n"},{"id":"11656","messageId":"20051112092920.GD16218@Muzzle","threadId":"2466","inReplyTo":"20051112092721.GC16218@Muzzle","subject":"[PATCH 3/5] Disambiguate the term 'branch' in Arch vs git","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-12T09:29:20Z","receivedAt":"2005-11-12T09:29:20Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Disambiguate the term 'branch' in Arch vs git,\nand start using fully-qualified names.\n\nSigned-off-by: Eric Wong \n\n---\n\n git-archimport.perl | 65 ++++++++++++++++++++++++++++++++++++++++++---------\n 1 files changed, 54 insertions(+), 11 deletions(-)\n\napplies-to: bbfe032e4900efc45bb94fb687af0140ccb0a858\nede672b4cd544b5e5418cc5088e92f2e0d2f7394\ndiff --git a/git-archimport.perl b/git-archimport.perl\nindex 699d5f6..f2bcbb4 100755\n--- a/git-archimport.perl\n+++ b/git-archimport.perl\n@@ -30,6 +30,24 @@ See man (1) git-archimport for more deta\n \n Add print in front of the shell commands invoked via backticks. \n \n+=head1 Devel Notes\n+\n+There are several places where Arch and git terminology are intermixed\n+and potentially confused.\n+\n+The notion of a \"branch\" in git is approximately equivalent to\n+a \"archive/category--branch--version\" in Arch. Also, it should be noted\n+that the \"--branch\" portion of \"archive/category--branch--version\" is really\n+optional in Arch although not many people (nor tools!) seem to know this.\n+This means that \"archive/category--version\" is also a valid \"branch\"\n+in git terms.\n+\n+We always refer to Arch names by their fully qualified variant (which\n+means the \"archive\" name is prefixed.\n+\n+For people unfamiliar with Arch, an \"archive\" is the term for \"repository\",\n+and can contain multiple, unrelated branches.\n+\n =cut\n \n use strict;\n@@ -215,9 +233,41 @@ unless (-d $git_dir) { # initial import\n }\n \n # process patchsets\n-foreach my $ps (@psets) {\n+# extract the Arch repository name (Arch \"archive\" in Arch-speak)\n+sub extract_reponame {\n+ my $fq_cvbr = shift; # archivename/[[[[category]branch]version]revision]\n+ return (split(/\\//, $fq_cvbr))[0];\n+}\n+ \n+sub extract_versionname {\n+ my $name = shift;\n+ $name =~ s/--(?:patch|version(?:fix)?|base)-\\d+$//;\n+ return $name;\n+}\n \n- $ps->{branch} = branchname($ps->{id});\n+# convert a fully-qualified revision or version to a unique dirname:\n+# normalperson@yhbt.net-05/mpd--uclinux--1--patch-2 \n+# becomes: normalperson@yhbt.net-05,mpd--uclinux--1\n+#\n+# the git notion of a branch is closer to\n+# archive/category--branch--version than archive/category--branch, so we\n+# use this to convert to git branch names.\n+# Also, keep archive names but replace '/' with ',' since it won't require\n+# subdirectories, and is safer than swapping '--' which could confuse\n+# reverse-mapping when dealing with bastard branches that\n+# are just archive/category--version (no --branch)\n+sub tree_dirname {\n+ my $revision = shift;\n+ my $name = extract_versionname($revision);\n+ $name =~ s#/#,#;\n+ return $name;\n+}\n+\n+*git_branchname = *tree_dirname;\n+\n+# process patchsets\n+foreach my $ps (@psets) {\n+ $ps->{branch} = git_branchname($ps->{id});\n \n #\n # ensure we have a clean state \n@@ -429,16 +479,9 @@ foreach my $ps (@psets) {\n $opt_v && print \" + parents: $par \\n\";\n }\n \n-sub branchname {\n- my $id = shift;\n- $id =~ s#^.+?/##;\n- my @parts = split(m/--/, $id);\n- return join('--', @parts[0..1]);\n-}\n-\n sub apply_import {\n my $ps = shift;\n- my $bname = branchname($ps->{id});\n+ my $bname = git_branchname($ps->{id});\n \n `mkdir -p $tmp`;\n \n@@ -669,7 +712,7 @@ sub find_parents {\n # simple loop to split the merges\n # per branch\n foreach my $merge (@{$ps->{merges}}) {\n-\tmy $branch = branchname($merge);\n+\tmy $branch = git_branchname($merge);\n \tunless (defined $branches{$branch} ){\n \t $branches{$branch} = [];\n \t}\n---\n0.99.9.GIT\n-- \nEric Wong\n"},{"id":"11657","messageId":"20051112093045.GE16218@Muzzle","threadId":"2466","inReplyTo":"20051112092920.GD16218@Muzzle","subject":"[PATCH 4/5] Overhaul of changeset application","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-12T09:30:46Z","receivedAt":"2005-11-12T09:30:46Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Overhaul of changeset application to use native Arch tree operations.\nThis results in:\n - reliable rename handling (esp. when dealing with renamed with files\n that already got renamed)\n - permissions tracking (execute only for git).\n - no need to shell-escape or pika-unescape anything. All arguments to\n external programs are always passed as an array. File modifications\n are automatically tracked using git (no need to parse Arch patch-log\n to look for modified files).\n - Correctly parse multi-line summary text in patch-logs\n\nSigned-off-by: Eric Wong \n\n---\n\n git-archimport.perl | 381 ++++++++++++++++++++-------------------------------\n 1 files changed, 146 insertions(+), 235 deletions(-)\n\napplies-to: 12cd9f2d764e50ae4fe2c6cd8b64fc72c668e0dd\nd3cbba7b8e8e3db61dac685ab55055d360e6138d\ndiff --git a/git-archimport.perl b/git-archimport.perl\nindex f2bcbb4..5616d42 100755\n--- a/git-archimport.perl\n+++ b/git-archimport.perl\n@@ -55,7 +55,7 @@ use warnings;\n use Getopt::Std;\n use File::Spec;\n use File::Temp qw(tempfile tempdir);\n-use File::Path qw(mkpath);\n+use File::Path qw(mkpath rmtree);\n use File::Basename qw(basename dirname);\n use String::ShellQuote;\n use Time::Local;\n@@ -90,16 +90,17 @@ usage if $opt_h;\n @ARGV >= 1 or usage();\n my @arch_roots = @ARGV;\n \n-my ($tmpdir, $tmpdirname) = tempdir('git-archimport-XXXXXX', TMPDIR => 1, CLEANUP => 1);\n-my $tmp = $opt_t || 1;\n-$tmp = tempdir('git-archimport-XXXXXX', TMPDIR => 1, CLEANUP => 1);\n-$opt_v && print \"+ Using $tmp as temporary directory\\n\";\n+my $tmptree;\n+$ENV{'TMPDIR'} = $opt_t if $opt_t;\n+$tmptree = tempdir('git-archimport-XXXXXX', TMPDIR => 1, CLEANUP => 1);\n+$opt_v && print \"+ Using $tmptree to store temporary trees\\n\";\n \n my @psets = (); # the collection\n my %psets = (); # the collection, by name\n \n my %rptags = (); # my reverse private tags\n # to map a SHA1 to a commitid\n+my $TLA = $ENV{'ARCH_CLIENT'} || 'tla';\n \n foreach my $root (@arch_roots) {\n my ($arepo, $abranch) = split(m!/!, $root);\n@@ -211,7 +212,7 @@ unless (-d $git_dir) { # initial import\n }\n } else { # progressing an import\n # load the rptags\n- opendir(DIR, \"$git_dir/archimport/tags\")\n+ opendir(DIR, $ptag_dir)\n \t|| die \"can't opendir: $!\";\n while (my $file = readdir(DIR)) {\n # skip non-interesting-files\n@@ -288,26 +289,37 @@ foreach my $ps (@psets) {\n \n print \" * Starting to work on $ps->{id}\\n\";\n \n- # \n- # create the branch if needed\n- #\n- if ($ps->{type} eq 'i' && !$import) {\n- die \"Should not have more than one 'Initial import' per GIT import: $ps->{id}\";\n+ # switch to that branch if we're not already in that branch:\n+ if (-e \"$git_dir/refs/heads/$ps->{branch}\") {\n+ system('git-checkout','-f',$ps->{branch}) == 0 or die \"$! $?\\n\";\n+\n+ # remove any old stuff that got leftover:\n+ chomp(my @rm = safe_pipe_capture('git-ls-files','--others'));\n+ rmtree(\\@rm) if @rm;\n }\n-\n- unless ($import) { # skip for import\n- if ( -e \"$git_dir/refs/heads/$ps->{branch}\") {\n- # we know about this branch\n- `git checkout $ps->{branch}`;\n- } else {\n- # new branch! we need to verify a few things\n- die \"Branch on a non-tag!\" unless $ps->{type} eq 't';\n- my $branchpoint = ptag($ps->{tag});\n- die \"Tagging from unknown id unsupported: $ps->{tag}\" \n- unless $branchpoint;\n+ \n+ # Apply the import/changeset/merge into the working tree\n+ my $dir = sync_to_ps($ps);\n+ # read the new log entry:\n+ my @commitlog = safe_pipe_capture($TLA,'cat-log','-d',$dir,$ps->{id});\n+ die \"Error in cat-log: $!\" if $?;\n+ chomp @commitlog;\n+\n+ # grab variables we want from the log, new fields get added to $ps:\n+ # (author, date, email, summary, message body ...)\n+ parselog($ps, \\@commitlog);\n+\n+ if ($ps->{id} =~ /--base-0$/ && $ps->{id} ne $psets[0]{id}) {\n+ # this should work when importing continuations \n+ if ($ps->{tag} && (my $branchpoint = eval { ptag($ps->{tag}) })) {\n \n # find where we are supposed to branch from\n- `git checkout -b $ps->{branch} $branchpoint`;\n+ system('git-checkout','-f','-b',$ps->{branch},\n+ $branchpoint) == 0 or die \"$! $?\\n\";\n+ \n+ # remove any old stuff that got leftover:\n+ chomp(my @rm = safe_pipe_capture('git-ls-files','--others'));\n+ rmtree(\\@rm) if @rm;\n \n # If we trust Arch with the fact that this is just \n # a tag, and it does not affect the state of the tree\n@@ -316,95 +328,26 @@ foreach my $ps (@psets) {\n ptag($ps->{id}, $branchpoint);\n print \" * Tagged $ps->{id} at $branchpoint\\n\";\n next;\n- } \n- die $! if $?;\n+ } else {\n+ warn \"Tagging from unknown id unsupported\\n\" if $ps->{tag};\n+ }\n+ # allow multiple bases/imports here since Arch supports cherry-picks\n+ # from unrelated trees\n } \n-\n- #\n- # Apply the import/changeset/merge into the working tree\n- # \n- if ($ps->{type} eq 'i' || $ps->{type} eq 't') {\n- apply_import($ps) or die $!;\n- $import=0;\n- } elsif ($ps->{type} eq 's') {\n- apply_cset($ps);\n- }\n-\n- #\n- # prepare update git's index, based on what arch knows\n- # about the pset, resolve parents, etc\n- #\n- my $tree;\n \n- my $commitlog = `tla cat-archive-log -A $ps->{repo} $ps->{id}`; \n- die \"Error in cat-archive-log: $!\" if $?;\n- \n- # parselog will git-add/rm files\n- # and generally prepare things for the commit\n- # NOTE: parselog will shell-quote filenames! \n- my ($sum, $msg, $add, $del, $mod, $ren) = parselog($commitlog);\n- my $logmessage = \"$sum\\n$msg\";\n-\n-\n- # imports don't give us good info\n- # on added files. Shame on them\n- if ($ps->{type} eq 'i' || $ps->{type} eq 't') { \n- `find . -type f -print0 | grep -zv '^./$git_dir' | xargs -0 -l100 git-update-index --add`;\n- `git-ls-files --deleted -z | xargs --no-run-if-empty -0 -l100 git-update-index --remove`;\n- }\n-\n- if (@$add) {\n- while (@$add) {\n- my @slice = splice(@$add, 0, 100);\n- my $slice = join(' ', @slice); \n- `git-update-index --add $slice`;\n- die \"Error in git-update-index --add: $!\" if $?;\n- }\n- }\n- if (@$del) {\n- foreach my $file (@$del) {\n- unlink $file or die \"Problems deleting $file : $!\";\n- }\n- while (@$del) {\n- my @slice = splice(@$del, 0, 100);\n- my $slice = join(' ', @slice);\n- `git-update-index --remove $slice`;\n- die \"Error in git-update-index --remove: $!\" if $?;\n- }\n- }\n- if (@$ren) { # renamed\n- if (@$ren % 2) {\n- die \"Odd number of entries in rename!?\";\n- }\n- ;\n- while (@$ren) {\n- my $from = pop @$ren;\n- my $to = pop @$ren; \n-\n- unless (-d dirname($to)) {\n- mkpath(dirname($to)); # will die on err\n- }\n- #print \"moving $from $to\";\n- `mv $from $to`;\n- die \"Error renaming $from $to : $!\" if $?;\n- `git-update-index --remove $from`;\n- die \"Error in git-update-index --remove: $!\" if $?;\n- `git-update-index --add $to`;\n- die \"Error in git-update-index --add: $!\" if $?;\n- }\n-\n- }\n- if (@$mod) { # must be _after_ renames\n- while (@$mod) {\n- my @slice = splice(@$mod, 0, 100);\n- my $slice = join(' ', @slice);\n- `git-update-index $slice`;\n- die \"Error in git-update-index: $!\" if $?;\n- }\n- }\n-\n- # warn \"errors when running git-update-index! $!\";\n- $tree = `git-write-tree`;\n+ # update the index with all the changes we got\n+ system('git-ls-files --others -z | '.\n+ 'git-update-index --add -z --stdin') == 0 or die \"$! $?\\n\";\n+ system('git-ls-files --deleted -z | '.\n+ 'git-update-index --remove -z --stdin') == 0 or die \"$! $?\\n\";\n+\n+ # just brute force this and update everything, it's faster than\n+ # parsing the Modified-files header and then having to pika-unescape\n+ # each one in case it has weird characters\n+ system('git-ls-files -z | '.\n+ 'git-update-index -z --stdin') == 0 or die \"$! $?\\n\";\n+ \n+ my $tree = `git-write-tree`;\n die \"cannot write tree $!\" if $?;\n chomp $tree;\n \n@@ -414,7 +357,7 @@ foreach my $ps (@psets) {\n #\n my @par;\n if ( -e \"$git_dir/refs/heads/$ps->{branch}\") {\n- if (open HEAD, \"<$git_dir/refs/heads/$ps->{branch}\") {\n+ if (open HEAD, \"<\",\"$git_dir/refs/heads/$ps->{branch}\") {\n my $p = ;\n close HEAD;\n chomp $p;\n@@ -429,7 +372,6 @@ foreach my $ps (@psets) {\n if ($ps->{merges}) {\n push @par, find_parents($ps);\n }\n- my $par = join (' ', @par);\n \n # \n # Commit, tag and clean state\n@@ -442,13 +384,14 @@ foreach my $ps (@psets) {\n $ENV{GIT_COMMITTER_EMAIL} = $ps->{email};\n $ENV{GIT_COMMITTER_DATE} = $ps->{date};\n \n- my ($pid, $commit_rh, $commit_wh);\n- $commit_rh = 'commit_rh';\n- $commit_wh = 'commit_wh';\n- \n- $pid = open2(*READER, *WRITER, \"git-commit-tree $tree $par\") \n+ my $pid = open2(*READER, *WRITER, 'git-commit-tree',$tree,@par) \n or die $!;\n- print WRITER $logmessage; # write\n+ print WRITER $ps->{summary},\"\\n\";\n+ print WRITER $ps->{message},\"\\n\";\n+\n+ # make it easy to backtrack and figure out which Arch revision this was:\n+ print WRITER 'git-archimport-id: ',$ps->{id},\"\\n\";\n+ \n close WRITER;\n my $commitid = ; # read\n chomp $commitid;\n@@ -461,7 +404,7 @@ foreach my $ps (@psets) {\n #\n # Update the branch\n # \n- open HEAD, \">$git_dir/refs/heads/$ps->{branch}\";\n+ open HEAD, \">\",\"$git_dir/refs/heads/$ps->{branch}\";\n print HEAD $commitid;\n close HEAD;\n unlink (\"$git_dir/HEAD\");\n@@ -476,71 +419,41 @@ foreach my $ps (@psets) {\n print \" + tree $tree\\n\";\n print \" + commit $commitid\\n\";\n $opt_v && print \" + commit date is $ps->{date} \\n\";\n- $opt_v && print \" + parents: $par \\n\";\n+ $opt_v && print \" + parents: \".join(' ',@par).\"\\n\";\n }\n \n-sub apply_import {\n+sub sync_to_ps {\n my $ps = shift;\n- my $bname = git_branchname($ps->{id});\n+ my $tree_dir = $tmptree.'/'.tree_dirname($ps->{id});\n \n- `mkdir -p $tmp`;\n-\n- `tla get -s --no-pristine -A $ps->{repo} $ps->{id} $tmp/import`;\n- die \"Cannot get import: $!\" if $?; \n- `rsync -v --archive --delete --exclude '$git_dir' --exclude '.arch-ids' --exclude '{arch}' $tmp/import/* ./`;\n- die \"Cannot rsync import:$!\" if $?;\n- \n- `rm -fr $tmp/import`;\n- die \"Cannot remove tempdir: $!\" if $?;\n- \n-\n- return 1;\n-}\n-\n-sub apply_cset {\n- my $ps = shift;\n-\n- `mkdir -p $tmp`;\n-\n- # get the changeset\n- `tla get-changeset -A $ps->{repo} $ps->{id} $tmp/changeset`;\n- die \"Cannot get changeset: $!\" if $?;\n- \n- # apply patches\n- if (`find $tmp/changeset/patches -type f -name '*.patch'`) {\n- # this can be sped up considerably by doing\n- # (find | xargs cat) | patch\n- # but that cna get mucked up by patches\n- # with missing trailing newlines or the standard \n- # 'missing newline' flag in the patch - possibly\n- # produced with an old/buggy diff.\n- # slow and safe, we invoke patch once per patchfile\n- `find $tmp/changeset/patches -type f -name '*.patch' -print0 | grep -zv '{arch}' | xargs -iFILE -0 --no-run-if-empty patch -p1 --forward -iFILE`;\n- die \"Problem applying patches! $!\" if $?;\n- }\n-\n- # apply changed binary files\n- if (my @modified = `find $tmp/changeset/patches -type f -name '*.modified'`) {\n- foreach my $mod (@modified) {\n- chomp $mod;\n- my $orig = $mod;\n- $orig =~ s/\\.modified$//; # lazy\n- $orig =~ s!^\\Q$tmp\\E/changeset/patches/!!;\n- #print \"rsync -p '$mod' '$orig'\";\n- `rsync -p $mod ./$orig`;\n- die \"Problem applying binary changes! $!\" if $?;\n+ if (-d $tree_dir) {\n+ if ($ps->{type} eq 't' && defined $ps->{tag}) {\n+ # looks like a tag-only or (worse,) a mixed tags/changeset branch,\n+ # can't rely on replay to work correctly on these\n+ rmtree($tree_dir);\n+ safe_pipe_capture($TLA,'get','--no-pristine',$ps->{id},$tree_dir);\n+ } else {\n+ my $tree_id = arch_tree_id($tree_dir);\n+ if ($ps->{parent_id} eq $tree_id) {\n+ safe_pipe_capture($TLA,'replay','-d',$tree_dir,$ps->{id});\n+ } else {\n+ safe_pipe_capture($TLA,'apply-delta','-d',$tree_dir,\n+ $tree_id, $ps->{id});\n+ }\n }\n+ } else {\n+ safe_pipe_capture($TLA,'get','--no-pristine',$ps->{id},$tree_dir);\n }\n-\n- # bring in new files\n- `rsync --archive --exclude '$git_dir' --exclude '.arch-ids' --exclude '{arch}' $tmp/changeset/new-files-archive/* ./`;\n-\n- # deleted files are hinted from the commitlog processing\n-\n- `rm -fr $tmp/changeset`;\n+ \n+ # added -I flag to rsync since we're going to fast! AIEEEEE!!!!\n+ system('rsync','-aI','--delete','--exclude',$git_dir,\n+# '--exclude','.arch-inventory',\n+ '--exclude','.arch-ids','--exclude','{arch}',\n+ '--exclude','+*','--exclude',',*',\n+ \"$tree_dir/\",'./') == 0 or die \"Cannot rsync $tree_dir: $! $?\";\n+ return $tree_dir;\n }\n \n-\n # =for reference\n # A log entry looks like \n # Revision: moodle-org--moodle--1.3.3--patch-15\n@@ -560,70 +473,42 @@ sub apply_cset {\n # admin/editor.html backup/lib.php backup/restore.php\n # New-patches: arch-eduforge@catalyst.net.nz--2004/moodle-org--moodle--1.3.3--patch-15\n # Summary: Updating to latest from MOODLE_14_STABLE (1.4.5+)\n+# summary can be multiline with a leading space just like the above fields\n # Keywords:\n #\n # Updating yadda tadda tadda madda\n sub parselog {\n- my $log = shift;\n- #print $log;\n-\n- my (@add, @del, @mod, @ren, @kw, $sum, $msg );\n-\n- if ($log =~ m/(?:\\n|^)New-files:(.*?)(?=\\n\\w)/s ) {\n- my $files = $1;\n- @add = split(m/\\s+/s, $files);\n- }\n- \n- if ($log =~ m/(?:\\n|^)Removed-files:(.*?)(?=\\n\\w)/s ) {\n- my $files = $1;\n- @del = split(m/\\s+/s, $files);\n- }\n- \n- if ($log =~ m/(?:\\n|^)Modified-files:(.*?)(?=\\n\\w)/s ) {\n- my $files = $1;\n- @mod = split(m/\\s+/s, $files);\n- }\n- \n- if ($log =~ m/(?:\\n|^)Renamed-files:(.*?)(?=\\n\\w)/s ) {\n- my $files = $1;\n- @ren = split(m/\\s+/s, $files);\n- }\n-\n- $sum ='';\n- if ($log =~ m/^Summary:(.+?)$/m ) {\n- $sum = $1;\n- $sum =~ s/^\\s+//;\n- $sum =~ s/\\s+$//;\n- }\n-\n- $msg = '';\n- if ($log =~ m/\\n\\n(.+)$/s) {\n- $msg = $1;\n- $msg =~ s/^\\s+//;\n- $msg =~ s/\\s+$//;\n- }\n-\n-\n- # cleanup the arrays\n- foreach my $ref ( (\\@add, \\@del, \\@mod, \\@ren) ) {\n- my @tmp = ();\n- while (my $t = pop @$ref) {\n- next unless length ($t);\n- next if $t =~ m!\\{arch\\}/!;\n- next if $t =~ m!\\.arch-ids/!;\n- next if $t =~ m!\\.arch-inventory$!;\n- # tla cat-archive-log will give us filenames with spaces as file\\(sp)name - why?\n- # we can assume that any filename with \\ indicates some pika escaping that we want to get rid of.\n- if ($t =~ /\\\\/ ){\n- $t = `tla escape --unescaped '$t'`;\n- }\n- push (@tmp, shell_quote($t));\n+ my ($ps, $log) = @_;\n+ my $key = undef;\n+ while ($_ = shift @$log) {\n+ if (/^Continuation-of:\\s*(.*)/) {\n+ $ps->{tag} = $1;\n+ $key = undef;\n+ } elsif (/^Summary:\\s*(.*)$/ ) {\n+ # summary can be multiline as long as it has a leading space\n+ $ps->{summary} = [ $1 ];\n+ $key = 'summary';\n+ } elsif (/^Creator: (.*)\\s*<([^\\>]+)>/) {\n+ $ps->{author} = $1;\n+ $ps->{email} = $2;\n+ $key = undef;\n+ } elsif (/^$/) {\n+ last; # remainder of @$log that didn't get shifted off is message\n+ } elsif ($key) {\n+ if (/^\\s+(.*)$/) {\n+ if ($key eq 'summary') {\n+ push @{$ps->{$key}}, $1;\n+ } else {\n+ push @{$ps->{$key}}, split(/\\s+/, $1);\n+ }\n+ } else {\n+ $key = undef;\n+ }\n }\n- @$ref = @tmp;\n }\n \n- #print Dumper [$sum, $msg, \\@add, \\@del, \\@mod, \\@ren]; \n- return ($sum, $msg, \\@add, \\@del, \\@mod, \\@ren); \n+ $ps->{summary} = join(\"\\n\",@{$ps->{summary}}).\"\\n\";\n+ $ps->{message} = join(\"\\n\",@$log);\n }\n \n # write/read a tag\n@@ -816,8 +701,11 @@ sub find_parents {\n \t }\n \t}\n }\n- @parents = keys %parents;\n- @parents = map { \" -p \" . ptag($_) } @parents;\n+ \n+ @parents = ();\n+ foreach (keys %parents) {\n+ push @parents, '-p', ptag($_);\n+ }\n return @parents;\n }\n \n@@ -840,3 +728,26 @@ sub commitid2pset {\n \t|| (print Dumper(sort keys %psets)) && die \"Cannot find patchset for $name\";\n return $ps;\n }\n+\n+\n+# an alterative to `command` that allows input to be passed as an array\n+# to work around shell problems with weird characters in arguments\n+sub safe_pipe_capture {\n+ my @output;\n+ if (my $pid = open my $child, '-|') {\n+ @output = (<$child>);\n+ close $child or die join(' ',@_).\": $! $?\";\n+ } else {\n+\texec(@_) or die $?; # exec() can fail the executable can't be found\n+ }\n+ return wantarray ? @output : join('',@output);\n+}\n+\n+# `tla logs -rf -d | head -n1` or `baz tree-id `\n+sub arch_tree_id {\n+ my $dir = shift;\n+ chomp( my $ret = (safe_pipe_capture($TLA,'logs','-rf','-d',$dir))[0] );\n+ return $ret;\n+}\n+\n+\n---\n0.99.9.GIT\n"},{"id":"11658","messageId":"20051112093208.GF16218@Muzzle","threadId":"2466","inReplyTo":"20051112093045.GE16218@Muzzle","subject":"[PATCH 5/5] -D option to recurse into merged branches","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-12T09:32:08Z","receivedAt":"2005-11-12T09:32:08Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"-D option to recurse into merged branches\n-a auto-register Arch archive if it's on mirrors.sourcecontrol.net\n\nfix for dealing with tag revisions\nremove unused module loading (no more String::ShellQuote dep)\n\nSigned-off-by: Eric Wong \n\n---\n\n git-archimport.perl | 257 ++++++++++++++++++++++++++++-----------------------\n 1 files changed, 141 insertions(+), 116 deletions(-)\n\napplies-to: d6d3e5272bc39ea086e5c1b0b39ceb5b51ade1ff\n2fe160b44c5e5da1a139668767ba184b6b63f605\ndiff --git a/git-archimport.perl b/git-archimport.perl\nindex 5616d42..a0ea016 100755\n--- a/git-archimport.perl\n+++ b/git-archimport.perl\n@@ -22,9 +22,10 @@ See man (1) git-archimport for more deta\n =head1 TODO\n \n - create tag objects instead of ref tags\n- - audit shell-escaping of filenames\n - hide our private tags somewhere smarter\n- - find a way to make \"cat *patches | patch\" safe even when patchfiles are missing newlines \n+ - sort and apply patches by graphing ancestry relations instead of just\n+ relying in dates supplied in the changeset itself.\n+ tla ancestry-graph -m could be helpful here...\n \n =head1 Devel tricks\n \n@@ -53,15 +54,9 @@ and can contain multiple, unrelated bran\n use strict;\n use warnings;\n use Getopt::Std;\n-use File::Spec;\n-use File::Temp qw(tempfile tempdir);\n+use File::Temp qw(tempdir);\n use File::Path qw(mkpath rmtree);\n use File::Basename qw(basename dirname);\n-use String::ShellQuote;\n-use Time::Local;\n-use IO::Socket;\n-use IO::Pipe;\n-use POSIX qw(strftime dup2);\n use Data::Dumper qw/ Dumper /;\n use IPC::Open2;\n \n@@ -72,29 +67,35 @@ my $git_dir = $ENV{\"GIT_DIR\"} || \".git\";\n $ENV{\"GIT_DIR\"} = $git_dir;\n my $ptag_dir = \"$git_dir/archimport/tags\";\n \n-our($opt_h,$opt_v, $opt_T,\n- $opt_C,$opt_t);\n+our($opt_h,$opt_v,$opt_T,$opt_t,$opt_D,$opt_a);\n \n sub usage() {\n print STDERR <= 1 or usage();\n-my @arch_roots = @ARGV;\n \n+# $arch_branches:\n+# values associated with keys:\n+# =1 - Arch version / git 'branch' detected via abrowse on a limit\n+# >1 - Arch version / git 'branch' of an auxilliary branch we've merged\n+my %arch_branches = map { $_ => 1 } @ARGV;\n+ \n my $tmptree;\n $ENV{'TMPDIR'} = $opt_t if $opt_t;\n $tmptree = tempdir('git-archimport-XXXXXX', TMPDIR => 1, CLEANUP => 1);\n $opt_v && print \"+ Using $tmptree to store temporary trees\\n\";\n \n+my %reachable = (); # Arch repositories we can access\n+my %unreachable = (); # Arch repositories we can't access :<\n my @psets = (); # the collection\n my %psets = (); # the collection, by name\n \n@@ -102,114 +103,117 @@ my %rptags = (); # my rev\n # to map a SHA1 to a commitid\n my $TLA = $ENV{'ARCH_CLIENT'} || 'tla';\n \n-foreach my $root (@arch_roots) {\n- my ($arepo, $abranch) = split(m!/!, $root);\n- open ABROWSE, \"tla abrowse -f -A $arepo --desc --merges $abranch |\" \n- or die \"Problems with tla abrowse: $!\";\n- \n- my %ps = (); # the current one\n- my $mode = '';\n- my $lastseen = '';\n- \n- while () {\n- chomp;\n- \n- # first record padded w 8 spaces\n- if (s/^\\s{8}\\b//) {\n- \n- # store the record we just captured\n- if (%ps) {\n- my %temp = %ps; # break references\n- push (@psets, \\%temp);\n-\t\t$psets{$temp{id}} = \\%temp;\n- %ps = ();\n- }\n- \n- my ($id, $type) = split(m/\\s{3}/, $_);\n- $ps{id} = $id;\n- $ps{repo} = $arepo;\n-\n- # deal with types\n- if ($type =~ m/^\\(simple changeset\\)/) {\n- $ps{type} = 's';\n- } elsif ($type eq '(initial import)') {\n- $ps{type} = 'i';\n- } elsif ($type =~ m/^\\(tag revision of (.+)\\)/) {\n- $ps{type} = 't';\n- $ps{tag} = $1;\n- } else { \n- warn \"Unknown type $type\";\n- }\n- $lastseen = 'id';\n- }\n- \n- if (s/^\\s{10}//) { \n- # 10 leading spaces or more \n- # indicate commit metadata\n- \n- # date & author \n- if ($lastseen eq 'id' && m/^\\d{4}-\\d{2}-\\d{2}/) {\n+sub do_abrowse {\n+ my $stage = shift;\n+ while (my ($limit, $level) = each %arch_branches) {\n+ next unless $level == $stage;\n+ \n+ open ABROWSE, \"$TLA abrowse -fkD --merges $limit |\" \n+ or die \"Problems with tla abrowse: $!\";\n+ \n+ my %ps = (); # the current one\n+ my $lastseen = '';\n+ \n+ while () {\n+ chomp;\n+ \n+ # first record padded w 8 spaces\n+ if (s/^\\s{8}\\b//) {\n+ my ($id, $type) = split(m/\\s+/, $_, 2);\n+\n+ my %last_ps;\n+ # store the record we just captured\n+ if (%ps && !exists $psets{ $ps{id} }) {\n+ %last_ps = %ps; # break references\n+ push (@psets, \\%last_ps);\n+ $psets{ $last_ps{id} } = \\%last_ps;\n+ }\n \n- my ($date, $authoremail) = split(m/\\s{2,}/, $_);\n- $ps{date} = $date;\n- $ps{date} =~ s/\\bGMT$//; # strip off trailign GMT\n- if ($ps{date} =~ m/\\b\\w+$/) {\n- warn 'Arch dates not in GMT?! - imported dates will be wrong';\n+ my $branch = extract_versionname($id);\n+ %ps = ( id => $id, branch => $branch );\n+ if (%last_ps && ($last_ps{branch} eq $branch)) {\n+ $ps{parent_id} = $last_ps{id};\n+ }\n+ \n+ $arch_branches{$branch} = 1;\n+ $lastseen = 'id';\n+\n+ # deal with types (should work with baz or tla):\n+ if ($type =~ m/\\(.*changeset\\)/) {\n+ $ps{type} = 's';\n+ } elsif ($type =~ /\\(.*import\\)/) {\n+ $ps{type} = 'i';\n+ } elsif ($type =~ m/\\(tag.*\\)/) {\n+ $ps{type} = 't';\n+ # read which revision we've tagged when we parse the log\n+ #$ps{tag} = $1;\n+ } else { \n+ warn \"Unknown type $type\";\n+ }\n+\n+ $arch_branches{$branch} = 1;\n+ $lastseen = 'id';\n+ } elsif (s/^\\s{10}//) { \n+ # 10 leading spaces or more \n+ # indicate commit metadata\n+ \n+ # date\n+ if ($lastseen eq 'id' && m/^(\\d{4}-\\d\\d-\\d\\d \\d\\d:\\d\\d:\\d\\d)/){\n+ $ps{date} = $1;\n+ $lastseen = 'date';\n+ } elsif ($_ eq 'merges in:') {\n+ $ps{merges} = [];\n+ $lastseen = 'merges';\n+ } elsif ($lastseen eq 'merges' && s/^\\s{2}//) {\n+ my $id = $_;\n+ push (@{$ps{merges}}, $id);\n+ \n+ # aggressive branch finding:\n+ if ($opt_D) {\n+ my $branch = extract_versionname($id);\n+ my $repo = extract_reponame($branch);\n+ \n+ if (archive_reachable($repo) &&\n+ !defined $arch_branches{$branch}) {\n+ $arch_branches{$branch} = $stage + 1;\n+ }\n+ }\n+ } else {\n+ warn \"more metadata after merges!?: $_\\n\" unless /^\\s*$/;\n }\n- \n- $authoremail =~ m/^(.+)\\s(\\S+)$/;\n- $ps{author} = $1;\n- $ps{email} = $2;\n- \n- $lastseen = 'date';\n- \n- } elsif ($lastseen eq 'date') {\n- # the only hint is position\n- # subject is after date\n- $ps{subj} = $_;\n- $lastseen = 'subj';\n- \n- } elsif ($lastseen eq 'subj' && $_ eq 'merges in:') {\n- $ps{merges} = [];\n- $lastseen = 'merges';\n- \n- } elsif ($lastseen eq 'merges' && s/^\\s{2}//) {\n- push (@{$ps{merges}}, $_);\n- } else {\n- warn 'more metadata after merges!?';\n }\n- \n }\n- }\n \n- if (%ps) {\n- my %temp = %ps; # break references\n- push (@psets, \\%temp); \n-\t$psets{ $temp{id} } = \\%temp;\n- %ps = ();\n- } \n- close ABROWSE;\n+ if (%ps && !exists $psets{ $ps{id} }) {\n+ my %temp = %ps; # break references\n+ if ($psets[$#psets]{branch} eq $ps{branch}) {\n+ $temp{parent_id} = $psets[$#psets]{id};\n+ }\n+ push (@psets, \\%temp); \n+ $psets{ $temp{id} } = \\%temp;\n+ } \n+ \n+ close ABROWSE or die \"$TLA abrowse failed on $limit\\n\";\n+ }\n } # end foreach $root\n \n+do_abrowse(1);\n+my $depth = 2;\n+$opt_D ||= 0;\n+while ($depth <= $opt_D) {\n+ do_abrowse($depth);\n+ $depth++;\n+}\n+ \n ## Order patches by time\n+# FIXME see if we can find a more optimal way to do this by graphing\n+# the ancestry data and walking it, that way we won't have to rely on\n+# client-supplied dates\n @psets = sort {$a->{date}.$b->{id} cmp $b->{date}.$b->{id}} @psets;\n \n-#print Dumper \\@psets;\n-\n-##\n-## TODO cleanup irrelevant patches\n-## and put an initial import\n-## or a full tag\n-my $import = 0;\n unless (-d $git_dir) { # initial import\n- if ($psets[0]{type} eq 'i' || $psets[0]{type} eq 't') {\n- print \"Starting import from $psets[0]{id}\\n\";\n-\t`git-init-db`;\n-\tdie $! if $?;\n-\t$import = 1;\n- } else {\n- die \"Need to start from an import or a tag -- cannot use $psets[0]{id}\";\n- }\n+ print \"Starting import from $psets[0]{id}\\n\";\n+ system('git-init-db') == 0 or die \"$! $?\\n\";\n } else { # progressing an import\n # load the rptags\n opendir(DIR, $ptag_dir)\n@@ -233,7 +237,6 @@ unless (-d $git_dir) { # initial import\n closedir DIR;\n }\n \n-# process patchsets\n # extract the Arch repository name (Arch \"archive\" in Arch-speak)\n sub extract_reponame {\n my $fq_cvbr = shift; # archivename/[[[[category]branch]version]revision]\n@@ -266,21 +269,21 @@ sub tree_dirname {\n \n *git_branchname = *tree_dirname;\n \n-# process patchsets\n+# process patchsets in ancestry order\n foreach my $ps (@psets) {\n $ps->{branch} = git_branchname($ps->{id});\n \n #\n # ensure we have a clean state \n # \n- if (`git diff-files`) {\n+ if (`git-diff-files`) {\n die \"Unclean tree when about to process $ps->{id} \" .\n \" - did we fail to commit cleanly before?\";\n }\n die $! if $?;\n \n #\n- # skip commits already in repo\n+ # skip commits already in git repo\n #\n if (ptag($ps->{id})) {\n $opt_v && print \" * Skipping already imported: $ps->{id}\\n\";\n@@ -427,7 +430,7 @@ sub sync_to_ps {\n my $tree_dir = $tmptree.'/'.tree_dirname($ps->{id});\n \n if (-d $tree_dir) {\n- if ($ps->{type} eq 't' && defined $ps->{tag}) {\n+ if ($ps->{type} eq 't') {\n # looks like a tag-only or (worse,) a mixed tags/changeset branch,\n # can't rely on replay to work correctly on these\n rmtree($tree_dir);\n@@ -435,13 +438,16 @@ sub sync_to_ps {\n } else {\n my $tree_id = arch_tree_id($tree_dir);\n if ($ps->{parent_id} eq $tree_id) {\n+ # the common case (hopefully)\n safe_pipe_capture($TLA,'replay','-d',$tree_dir,$ps->{id});\n } else {\n+ # this can happen if branches cherry-pick\n safe_pipe_capture($TLA,'apply-delta','-d',$tree_dir,\n $tree_id, $ps->{id});\n }\n }\n } else {\n+ # new branch work\n safe_pipe_capture($TLA,'get','--no-pristine',$ps->{id},$tree_dir);\n }\n \n@@ -750,4 +756,23 @@ sub arch_tree_id {\n return $ret;\n }\n \n+sub archive_reachable {\n+ my $archive = shift;\n+ return 1 if $reachable{$archive};\n+ return 0 if $unreachable{$archive};\n+ \n+ if (system \"$TLA whereis-archive $archive >/dev/null\") {\n+ if ($opt_a && (system($TLA,'register-archive',\n+ \"http://mirrors.sourcecontrol.net/$archive\") == 0)) {\n+ $reachable{$archive} = 1;\n+ return 1;\n+ }\n+ print STDERR \"Archive is unreachable: $archive\\n\";\n+ $unreachable{$archive} = 1;\n+ return 0;\n+ } else {\n+ $reachable{$archive} = 1;\n+ return 1;\n+ }\n+}\n \n---\n0.99.9.GIT\n"},{"id":"11663","messageId":"46a038f90511120354n4584aedfhb1f2928ac41478ab@mail.gmail.com","threadId":"2466","inReplyTo":"20051112092336.GA16218@Muzzle","subject":"Re: [PATCH] archimport improvements","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2005-11-12T11:54:36Z","receivedAt":"2005-11-12T11:54:36Z","isPatch":true,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"Eric,\n\n\nOn 11/12/05, Eric Wong wrote:\n> I'm another Arch-user trying out git. Unfortunately, I encountered\n> several problems with git-archimport that I needed fixed before my\n> development trees could be imported into git.\n\nWelcome and good stuff! I'll give your patches a try when I sober up.\nIn the meantime, some notes after having read the patches a bit...\n\n> Bug Fixes:\n>\n> * Support for '--branch'-less Arch version names.\n> Encoding '/' to '--' (as was previously done) is not 100% reversable\n> because the \"--branch\" portion of an fully-qualified Arch version name\n> is optional (though not many people or Arch-related tools know this).\n>\n> * I'm encoding the '/' in the fully-qualified name as ',' to not confuse\n> other porcelains, but leaving '/' in branch names may be alright\n> provided porcelains can support them.\n>\n> * Identify git branches as an Arch \"archive,category<--branch>--version\"\n> Anything less than that is ambiguous as far as history and patch\n> relationships go.\n\nThese bug/sanity fixes are _good_. As you mention, I wasn't aware that\npatchnames could show up not having a --branch part. Tricky...\n\n> * Renamed directories containing renamed/moved files inside didn't get\n> tracked properly. The original code was inadequate for this, and\n> making it support all rename cases that Arch supports is too much\n> work. Instead, I maintain full-blown Arch trees in the temp dir and\n> replay patches + rsync based on that. Performance is slightly slower\n> than before, but accuracy is more important to me.\n>\n> * Permission (execute bit only because of git) tracking as a side effect\n> of the above.\n\nHmmm. I understand what you are doing, but I'm not sure we'd want to\nreplace the current code with this strategy. Importing large trees\nwith hundreds (thousands) of commits is so slow it is just a no go.\nRenames are described quite well in the 'commit log', and the current\ncode does handle file renames...\n\n> * Tracking changes from branches that are only cherry-picked now works\n\nCan you elaborate a bit more on this?\n\n> * Pika-escaped filenames unhandled. This seems fixed in the latest\n> git, but I fixed it more generally and removed the ShellQuote module\n> dependency along the way.\n\nYes, this got fixed recently. Your change here goes together with the\n'tla get' + rsync strategy which I'm not sure about.\n\n> * Don't die() when a merge-base can't be found. Arch supports\n> merging between unrelated trees.\n\nFair enough. Does it result on a good graft in git?\n\n> Usability enhancements:\n>\n> * Optionally detect merged branches and attempt to import their history,\n> too. Use the -D option for this. Specifying a \n> greater than 1 is usually not needed unless the tree you're tracking\n> has had history pruned.\n>\n> * Optionally attempt to auto-register unknown Arch archives from\n> mirrors.sourcecontrol.net to pull their history with the -a (boolean)\n> switch. Not sure how useful users will find this.\n\nThose two are interesting!\n\n> * Removed -A usage (unnecessary in all cases) and made all\n> Arch calls and output parsing to be compatible with both tla (tested\n> 1.3.3) and baz (1.4.2). Default is still tla, but the ARCH_CLIENT\n> environment variable may be changed to baz.\n\nThat's excellent -- thanks!\n\n> Current weaknesses:\n>\n> * (Present in the original code as well).\n> The code still assumes that dates in commit logs can be trusted, which is\n> fine in most cases, but a wayward branch can screw up git-archimport and\n> cause parents to be missed.\n\nFair enough. You mention an alternative strategy (tla ancestry) --\nhave you tried it at all?\n\ncheers,\n\n\nmartin\n"},{"id":"11664","messageId":"46a038f90511120407y68031071v15244175f6458c54@mail.gmail.com","threadId":"2466","inReplyTo":"20051112093045.GE16218@Muzzle","subject":"Re: [PATCH 4/5] Overhaul of changeset application","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2005-11-12T12:07:12Z","receivedAt":"2005-11-12T12:07:12Z","isPatch":true,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"Eric,\n\n I'd actually like to improve the script to handle directory renames\nand file modes correctly so we don't need to ever call the glacially\nslow `tla get` -- I don't think it's that much work, all I need is a\nsample repo. OTOH, if you think (or can convince me) that there are\nmore serious problems ahead, perhaps we can have this as an\nalternative import mechanism?\n\nOn 11/12/05, Eric Wong wrote:\n> - Correctly parse multi-line summary text in patch-logs\n\nWas this broken!? I'm sure I've imported multiline summaries!\n\ncheers,\n\n\nmartin\n"},{"id":"11694","messageId":"20051112202150.GA2037@Muzzle","threadId":"2466","inReplyTo":"46a038f90511120354n4584aedfhb1f2928ac41478ab@mail.gmail.com","subject":"Re: [PATCH] archimport improvements","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-12T20:21:51Z","receivedAt":"2005-11-12T20:21:51Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Martin Langhoff wrote:\n> Eric,\n> \n> \n> On 11/12/05, Eric Wong wrote:\n> > I'm another Arch-user trying out git. Unfortunately, I encountered\n> > several problems with git-archimport that I needed fixed before my\n> > development trees could be imported into git.\n> \n> Welcome and good stuff! I'll give your patches a try when I sober up.\n> In the meantime, some notes after having read the patches a bit...\n> \n> > Bug Fixes:\n> >\n> > * Support for '--branch'-less Arch version names.\n> > Encoding '/' to '--' (as was previously done) is not 100% reversable\n> > because the \"--branch\" portion of an fully-qualified Arch version name\n> > is optional (though not many people or Arch-related tools know this).\n> >\n> > * I'm encoding the '/' in the fully-qualified name as ',' to not confuse\n> > other porcelains, but leaving '/' in branch names may be alright\n> > provided porcelains can support them.\n> >\n> > * Identify git branches as an Arch \"archive,category<--branch>--version\"\n> > Anything less than that is ambiguous as far as history and patch\n> > relationships go.\n> \n> These bug/sanity fixes are _good_. As you mention, I wasn't aware that\n> patchnames could show up not having a --branch part. Tricky...\n\nThanks. I got lazy one day and started ignoring --branch on some of my\npersonal projects to save my fingers :)\n\n> > * Renamed directories containing renamed/moved files inside didn't get\n> > tracked properly. The original code was inadequate for this, and\n> > making it support all rename cases that Arch supports is too much\n> > work. Instead, I maintain full-blown Arch trees in the temp dir and\n> > replay patches + rsync based on that. Performance is slightly slower\n> > than before, but accuracy is more important to me.\n> >\n> > * Permission (execute bit only because of git) tracking as a side effect\n> > of the above.\n> \n> Hmmm. I understand what you are doing, but I'm not sure we'd want to\n> replace the current code with this strategy. Importing large trees\n> with hundreds (thousands) of commits is so slow it is just a no go.\n> Renames are described quite well in the 'commit log', and the current\n> code does handle file renames...\n\nUntouched files inside renamed directories aren't explicitly tracked.\nRenamed directories are especially a pain when a renamed one contains\nsub-directories that are also renamed.\n\n> > * Tracking changes from branches that are only cherry-picked now works\n> \n> Can you elaborate a bit more on this?\n\nBasically, don't die when merge-base fails, look a few lines down.\n\n> > * Pika-escaped filenames unhandled. This seems fixed in the latest\n> > git, but I fixed it more generally and removed the ShellQuote module\n> > dependency along the way.\n> \n> Yes, this got fixed recently. Your change here goes together with the\n> 'tla get' + rsync strategy which I'm not sure about.\n> \n> > * Don't die() when a merge-base can't be found. Arch supports\n> > merging between unrelated trees.\n> \n> Fair enough. Does it result on a good graft in git?\n\nRight now I end up with separate branches that are imported (according\nto git-branch) but the git-log and gitk don't seem to to show\nrelationships between the unrelated trees. I think find_parents()\nmay need to use an alternate strategy instead of warning and skipping\nif a merge-base can't be found.\n\n> > Usability enhancements:\n> >\n> > * Optionally detect merged branches and attempt to import their history,\n> > too. Use the -D option for this. Specifying a \n> > greater than 1 is usually not needed unless the tree you're tracking\n> > has had history pruned.\n> >\n> > * Optionally attempt to auto-register unknown Arch archives from\n> > mirrors.sourcecontrol.net to pull their history with the -a (boolean)\n> > switch. Not sure how useful users will find this.\n> \n> Those two are interesting!\n> \n> > * Removed -A usage (unnecessary in all cases) and made all\n> > Arch calls and output parsing to be compatible with both tla (tested\n> > 1.3.3) and baz (1.4.2). Default is still tla, but the ARCH_CLIENT\n> > environment variable may be changed to baz.\n> \n> That's excellent -- thanks!\n> \n> > Current weaknesses:\n> >\n> > * (Present in the original code as well).\n> > The code still assumes that dates in commit logs can be trusted, which is\n> > fine in most cases, but a wayward branch can screw up git-archimport and\n> > cause parents to be missed.\n> \n> Fair enough. You mention an alternative strategy (tla ancestry) --\n> have you tried it at all?\n\nNo, not yet.\n\n-- \nEric Wong\n"},{"id":"11697","messageId":"20051112204928.GA31205@mail.yhbt.net","threadId":"2466","inReplyTo":"46a038f90511120407y68031071v15244175f6458c54@mail.gmail.com","subject":"Re: [PATCH 4/5] Overhaul of changeset application","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-12T20:49:28Z","receivedAt":"2005-11-12T20:49:28Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Martin Langhoff wrote:\n> Eric,\n> \n> I'd actually like to improve the script to handle directory renames\n> and file modes correctly so we don't need to ever call the glacially\n> slow `tla get` -- I don't think it's that much work, all I need is a\n> sample repo. OTOH, if you think (or can convince me) that there are\n> more serious problems ahead, perhaps we can have this as an\n> alternative import mechanism?\n\ntla get is hardly ever called, I don't think it's called any more than\nbefore, even. tla replay by far the most common case and still\nreasonably fast. I had to add the -I flag to rsync because it was going\nat > 1 patch per second, plenty fast enough for me.\n\nI also had an alternate implementation for using the revision library,\nbut that was slower than the current strategy because it had to do\ntwo full Arch tree integrity checks for each patch applied.\n\nEven on a hot (fully filled) revlib and all I had to do was tla\nlibrary-find + rsync, it ran more slowly, probably because rsync\ncouldn't take advantage of kernel/fs-level caching when it had to\nwork on a different directory each time.\n\nTracking renamed directories (especially when nested subdirectories are\nalso renamed) is very, very far from pleasant.\n\n> On 11/12/05, Eric Wong wrote:\n> > - Correctly parse multi-line summary text in patch-logs\n> \n> Was this broken!? I'm sure I've imported multiline summaries!\n\nIt only got the first summary line when I tried it. Also, it's possible\nfor hand-made message bodies to fool archimport if it has \"headers\"\nafter the first \\n\\n. IIRC, some old tools copied entire logs of merged\nchangesets into the message body.\n\n-- \nEric Wong\n"},{"id":"11753","messageId":"20051114020134.GC10998@mail.yhbt.net","threadId":"2466","inReplyTo":"20051112093208.GF16218@Muzzle","subject":"Re: [PATCH 5/5] -D option to recurse into merged branches","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-14T02:01:34Z","receivedAt":"2005-11-14T02:01:34Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"One small fix on top of this one:\n\nDon't check for parents if the only revision we have is a base-0\nand @psets is empty.\n\nSigned-off-by: Eric Wong \n\n---\n\n git-archimport.perl | 2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\napplies-to: 8a7e18ff0884cae74a1127d5c96577a85acca3f4\n5f2896558284724bcc87eb64daa0933b544ec20d\ndiff --git a/git-archimport.perl b/git-archimport.perl\nindex a0ea016..b624ba6 100755\n--- a/git-archimport.perl\n+++ b/git-archimport.perl\n@@ -186,7 +186,7 @@ sub do_abrowse {\n \n if (%ps && !exists $psets{ $ps{id} }) {\n my %temp = %ps; # break references\n- if ($psets[$#psets]{branch} eq $ps{branch}) {\n+ if (@psets && $psets[$#psets]{branch} eq $ps{branch}) {\n $temp{parent_id} = $psets[$#psets]{id};\n }\n push (@psets, \\%temp); \n---\n0.99.9.GIT\n"},{"id":"11849","messageId":"46a038f90511141438q1d85d429vedcf2a3b54d761e1@mail.gmail.com","threadId":"2466","inReplyTo":"20051112202150.GA2037@Muzzle","subject":"Re: [PATCH] archimport improvements","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2005-11-14T22:38:20Z","receivedAt":"2005-11-14T22:38:20Z","isPatch":true,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"Eric,\n\nthanks for resending those so quickly. I think I'm going to sit on the\n'overhaul of changeset application' patch a bit -- I'll test & ack\nyour other patches for merge soonish but I want to review and test\nthis one carefully.\n\nMy main concern is that it seems to be calling tla get for each\nrevision that it imports. For large trees, this is slow. I would be\nmuch happier with a fast Perl-based approach. Have you got a public\nrepo with directory renames?\n\nAdditional comments follow...\n\nOn 11/13/05, Eric Wong wrote:\n> > > * Identify git branches as an Arch \"archive,category<--branch>--version\"\n> > > Anything less than that is ambiguous as far as history and patch\n> > > relationships go.\n> >\n> > These bug/sanity fixes are _good_. As you mention, I wasn't aware that\n> > patchnames could show up not having a --branch part. Tricky...\n>\n> Thanks. I got lazy one day and started ignoring --branch on some of my\n> personal projects to save my fingers :)\n\nYup, makes sense. My concern now is that existing imports will change\nthe name of branches and tags going forward. Can I ask you to resend\nthat patch with the new branchname mangling as default, and the old\none as optional?\n\nI know it'll force us to go back to using shellquote, but I am not too\nworried by that dependency at the moment.\n\n> > > Current weaknesses:\n> > >\n> > > * (Present in the original code as well).\n> > > The code still assumes that dates in commit logs can be trusted, which is\n> > > fine in most cases, but a wayward branch can screw up git-archimport and\n> > > cause parents to be missed.\n> >\n> > Fair enough. You mention an alternative strategy (tla ancestry) --\n> > have you tried it at all?\n>\n> No, not yet.\n\nAlso interested in this if you get around to it.\n\ncheers,\n\n\nmartin\n"},{"id":"11873","messageId":"20051115080352.GG7484@mail.yhbt.net","threadId":"2466","inReplyTo":"46a038f90511141438q1d85d429vedcf2a3b54d761e1@mail.gmail.com","subject":"Re: [PATCH] archimport improvements","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-15T08:03:52Z","receivedAt":"2005-11-15T08:03:52Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Martin Langhoff wrote:\n> Eric,\n> \n> thanks for resending those so quickly. I think I'm going to sit on the\n> 'overhaul of changeset application' patch a bit -- I'll test & ack\n> your other patches for merge soonish but I want to review and test\n> this one carefully.\n> \n> My main concern is that it seems to be calling tla get for each\n> revision that it imports. For large trees, this is slow. I would be\n> much happier with a fast Perl-based approach. Have you got a public\n> repo with directory renames?\n\nPlease read my sync_to_ps() function very carefully. Next is a patch\nthat helps you track which Arch command (get/replay/apply-delta) is used\nfor each changeset.\n\ntla replay is the most common for any halfway normal (changeset-based)\ntree by far.\n\ntla get is not called any more often than before.\n\napply-delta is hardly, if ever called. It may not even be reachable\nunless somebody commits revisions to the same tree with clocks out of\norder from patchlevel order. Heck, if it's ever called, it's most\nlikely faster just to rmtree and tla get again.\n\nUnfortunately, my heavily used and abused trees are private.\n\n> Additional comments follow...\n> \n> On 11/13/05, Eric Wong wrote:\n> > > > * Identify git branches as an Arch \"archive,category<--branch>--version\"\n> > > > Anything less than that is ambiguous as far as history and patch\n> > > > relationships go.\n> > >\n> > > These bug/sanity fixes are _good_. As you mention, I wasn't aware that\n> > > patchnames could show up not having a --branch part. Tricky...\n> >\n> > Thanks. I got lazy one day and started ignoring --branch on some of my\n> > personal projects to save my fingers :)\n> \n> Yup, makes sense. My concern now is that existing imports will change\n> the name of branches and tags going forward. Can I ask you to resend\n> that patch with the new branchname mangling as default, and the old\n> one as optional?\n\nOk, good idea. My previous patch already automatically converted the\nprivate tags, which we actually need to parse, and I see no reason to\nchange that, but branch names and public tags which affect\nnon-gitarchimport users can be preserved with the -o flag.\n\n> I know it'll force us to go back to using shellquote, but I am not too\n> worried by that dependency at the moment.\n \nActually, usage of shell_quote() in git-archimport was always\nunnecessary. Passing arguments to external programs as an array,\nusing the 3-argument version of open() for files, and using -z in\ngit-commands with pipes are better ways to go.\n\n> > > > Current weaknesses:\n> > > >\n> > > > * (Present in the original code as well).\n> > > > The code still assumes that dates in commit logs can be trusted, which is\n> > > > fine in most cases, but a wayward branch can screw up git-archimport and\n> > > > cause parents to be missed.\n> > >\n> > > Fair enough. You mention an alternative strategy (tla ancestry) --\n> > > have you tried it at all?\n> >\n> > No, not yet.\n> \n> Also interested in this if you get around to it.\n\nIt's not a high priority for me and I probably don't have time to do\nthis.\n\n-- \nEric Wong\n"},{"id":"11874","messageId":"20051115080546.GA30506@Muzzle","threadId":"2466","inReplyTo":"20051115080352.GG7484@mail.yhbt.net","subject":"[PATCH 1/2] archimport: allow for old style branch and public tag names","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-15T08:05:46Z","receivedAt":"2005-11-15T08:05:46Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"This patch adds the -o switch, which lets old trees tracked by\ngit-archmirror continue working with their old branch and tag names\nto make life easier for people tracking your tree.\n\nPrivate tags that are only used internally by git-archimport continue to be\nnew-style, and automatically converted upon first run.\n\nSigned-off-by:: Eric Wong \n\n---\n\n git-archimport.perl | 22 +++++++++++++++++-----\n 1 files changed, 17 insertions(+), 5 deletions(-)\n\napplies-to: 44d831812786f4dfbf54a67b51e5f48c7d5afd66\n4b341dd903883db0a89fe2f04e93dab053beb045\ndiff --git a/git-archimport.perl b/git-archimport.perl\nindex 1f721f6..304d462 100755\n--- a/git-archimport.perl\n+++ b/git-archimport.perl\n@@ -67,12 +67,12 @@ my $git_dir = $ENV{\"GIT_DIR\"} || \".git\";\n $ENV{\"GIT_DIR\"} = $git_dir;\n my $ptag_dir = \"$git_dir/archimport/tags\";\n \n-our($opt_h,$opt_v,$opt_T,$opt_t,$opt_D,$opt_a);\n+our($opt_h,$opt_v,$opt_T,$opt_t,$opt_D,$opt_a,$opt_o);\n \n sub usage() {\n print STDERR < part:\n+sub old_style_branchname {\n+ my $id = shift;\n+ my $ret = safe_pipe_capture($TLA,'parse-package-name','-p',$id);\n+ chomp $ret;\n+ return $ret;\n+}\n+\n+*git_branchname = $opt_o ? *old_style_branchname : *tree_dirname;\n \n # process patchsets in ancestry order\n foreach my $ps (@psets) {\n@@ -527,8 +535,12 @@ sub parselog {\n sub tag {\n my ($tag, $commit) = @_;\n \n- # don't use subdirs for tags yet, it could screw up other porcelains\n- $tag =~ s|/|,|;\n+ if ($opt_o) {\n+ $tag =~ s|/|--|g;\n+ } else {\n+ # don't use subdirs for tags yet, it could screw up other porcelains\n+ $tag =~ s|/|,|g;\n+ }\n \n if ($commit) {\n open(C,\">\",\"$git_dir/refs/tags/$tag\")\n---\n0.99.9.GIT\n"},{"id":"11875","messageId":"20051115080652.GB30506@Muzzle","threadId":"2466","inReplyTo":"20051115080546.GA30506@Muzzle","subject":"[PATCH 2/2] archimport: sync_to_ps() messages for tracking tla methods","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-15T08:06:52Z","receivedAt":"2005-11-15T08:06:52Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"This patch adds debug messages (enabled with the usual -v switch) for\ntracking how often each tla command is called.\n\nSigned-off-by: Eric Wong \n\n---\n\n git-archimport.perl | 6 ++++++\n 1 files changed, 6 insertions(+), 0 deletions(-)\n\napplies-to: 8a7cc429d0fd935805851ac5ac10941d0bd86e94\nb4de7920e0116afb35016435131a404658818ced\ndiff --git a/git-archimport.perl b/git-archimport.perl\nindex b624ba6..1f721f6 100755\n--- a/git-archimport.perl\n+++ b/git-archimport.perl\n@@ -429,18 +429,23 @@ sub sync_to_ps {\n my $ps = shift;\n my $tree_dir = $tmptree.'/'.tree_dirname($ps->{id});\n \n+ $opt_v && print \"sync_to_ps($ps->{id}) method: \";\n+\n if (-d $tree_dir) {\n if ($ps->{type} eq 't') {\n # looks like a tag-only or (worse,) a mixed tags/changeset branch,\n # can't rely on replay to work correctly on these\n rmtree($tree_dir);\n+\t $opt_v && print \"get (tag)\\n\";\n safe_pipe_capture($TLA,'get','--no-pristine',$ps->{id},$tree_dir);\n } else {\n my $tree_id = arch_tree_id($tree_dir);\n if ($ps->{parent_id} eq $tree_id) {\n # the common case (hopefully)\n+\t\t $opt_v && print \"replay\\n\";\n safe_pipe_capture($TLA,'replay','-d',$tree_dir,$ps->{id});\n } else {\n+\t\t $opt_v && print \"apply-delta\\n\";\n # this can happen if branches cherry-pick\n safe_pipe_capture($TLA,'apply-delta','-d',$tree_dir,\n $tree_id, $ps->{id});\n@@ -448,6 +453,7 @@ sub sync_to_ps {\n }\n } else {\n # new branch work\n+ $opt_v && print \"get (new tree)\\n\";\n safe_pipe_capture($TLA,'get','--no-pristine',$ps->{id},$tree_dir);\n }\n \n---\n0.99.9.GIT\n"},{"id":"11876","messageId":"20051115080742.GC30506@Muzzle","threadId":"2466","inReplyTo":"20051115080546.GA30506@Muzzle","subject":"Re: [PATCH 1/2] archimport: allow for old style branch and public tag names","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-15T08:07:42Z","receivedAt":"2005-11-15T08:07:42Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Oops, I sent these two patches out of order. They should apply\nfine without conflicts either way.\n\n-- \nEric Wong\n"},{"id":"12081","messageId":"46a038f90511170126l5f04f4e0ta578fc2cb375db68@mail.gmail.com","threadId":"2466","inReplyTo":"20051112092336.GA16218@Muzzle","subject":"Re: [PATCH] archimport improvements","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2005-11-17T09:26:39Z","receivedAt":"2005-11-17T09:26:39Z","isPatch":true,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"Eric,\n\nI've merged and pushed out to\nhttp://locke.catalyst.net.nz/git/git-martinlanghoff.git/#tojunio\n\n [PATCH 1/5] remove shellquote usage for tags\n [PATCH 2/5] archimport: don't die on merge-base failure\n [PATCH 3/5] Disambiguate the term 'branch' in Arch vs git\n [PATCH 1/2] archimport: allow for old style branch and public tag names\n\nThat last one had a small edit to rebase it to the top of the head --\nwill probably have a small conflict for you on the usage line and\ngetopts() line.\n\nWhat is pending is...\n\n* [PATCH 4/5] Overhaul of changeset application\n\nI am testing it right now. Finding it rather slow on an idle linux\nworkstation with fast IDE disks, no X.org loaded and 1GB or RAM.\niowait is pegged at 90%. Wonder what will happen on a system with slow\ndisk access. tla/baz are unusable under any OS where the fs stack is\nnot _that_ polished (OSX and friends).\n\nThe early versions of the import also used $TLA for all ops, and I was\nforced to change it to get my repos transformed in a reasonable time.\n\nCan you send me a patch that makes it optional, so users can choose\nfast or correct? I don't want to force glacial imports on anyone,\nspecially me. Testing an import of a reasonably sized repo must be a\nquick operation or I won't do it ;-) And I do work on OSX too.\n\nOn the other hand, I might just implement renamed directories tracking\nseparately, specially if someone can point me to a public repo with\nsome interesting cases of renamed directories.\n\nThese patches seem to hang from 4/5 so will need rebasing after a\nreworked. The first one seems to be 3 or 4 patches in one. It'd be\ngood to break it up.\n\n* [ PATCH 5/5] -D option to recurse into merged branches\n* Re: [PATCH 5/5] -D option to recurse into merged branches\n* [PATCH 2/2] archimport: sync_to_ps() messages for tracking tla methods\n\nIf you want to see the repos I'm testing with, register\narch-eduforge@catalyst.net.nz--2004\nhttp://nzvle.eduforge.org/arch-mirror/ and try:\n\n~/local/git/git-archimport.perl -v \\\n arch-eduforge@catalyst.net.nz--2004/moodle-org--moodle \\\n arch-eduforge@catalyst.net.nz--2004/moodle--local \\\n arch-eduforge@catalyst.net.nz--2004/moodle--local-forum-types \\\n arch-eduforge@catalyst.net.nz--2004/moodle--local-lock-content \\\n arch-eduforge@catalyst.net.nz--2004/moodle--nmit \\\n arch-eduforge@catalyst.net.nz--2004/moodle--topnz\n\ncheers,\n\n\nmartin\n"},{"id":"12671","messageId":"20051124074605.GA4789@mail.yhbt.net","threadId":"2466","inReplyTo":"46a038f90511170126l5f04f4e0ta578fc2cb375db68@mail.gmail.com","subject":"Re: [PATCH] archimport improvements","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-24T07:46:06Z","receivedAt":"2005-11-24T07:46:06Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Martin Langhoff wrote:\n> Eric,\n> \n> I've merged and pushed out to\n> http://locke.catalyst.net.nz/git/git-martinlanghoff.git/#tojunio\n> \n> [PATCH 1/5] remove shellquote usage for tags\n> [PATCH 2/5] archimport: don't die on merge-base failure\n> [PATCH 3/5] Disambiguate the term 'branch' in Arch vs git\n> [PATCH 1/2] archimport: allow for old style branch and public tag names\n> \n> That last one had a small edit to rebase it to the top of the head --\n> will probably have a small conflict for you on the usage line and\n> getopts() line.\n> \n> What is pending is...\n> \n> * [PATCH 4/5] Overhaul of changeset application\n> \n> I am testing it right now. Finding it rather slow on an idle linux\n> workstation with fast IDE disks, no X.org loaded and 1GB or RAM.\n> iowait is pegged at 90%. Wonder what will happen on a system with slow\n> disk access. tla/baz are unusable under any OS where the fs stack is\n> not _that_ polished (OSX and friends).\n\nOk, I didn't expect you guys to have 12k of files in your trees. None\nof your source trees are remotely close to that size (but I have many\nmore changesets). I'm surprised you guys were able to put up\nwith Arch in the first place!\n\n125m58.431s with my method.\n 8m24.504s with yours :)\n\nAll of my usual source trees imported 1k changesets in 10-15 minutes\n\n> The early versions of the import also used $TLA for all ops, and I was\n> forced to change it to get my repos transformed in a reasonable time.\n> \n> Can you send me a patch that makes it optional, so users can choose\n> fast or correct? I don't want to force glacial imports on anyone,\n> specially me. Testing an import of a reasonably sized repo must be a\n> quick operation or I won't do it ;-) And I do work on OSX too.\n\nPatches on the way.\n\nOTOH, the time spent importing the bulk of the history is a one-time\noperation for most people and I'd much rather it get things as right as\npossible and move on.\n\n> On the other hand, I might just implement renamed directories tracking\n> separately, specially if someone can point me to a public repo with\n> some interesting cases of renamed directories.\n\nIIRC, there are several nasty cases all of which are ordering-related,\nespecially with regard to nested directories or file renames inside\ndirectories that are also renamed. It should be noted that not even tla\ngets all the possible directory rename cases right (baz seems better\nfrom my observations). \n\n> These patches seem to hang from 4/5 so will need rebasing after a\n> reworked. The first one seems to be 3 or 4 patches in one. It'd be\n> good to break it up.\n\nSorry, I rushed through the initial overhaul and didn't generate neat\npatches because I wanted to get some of my work moved to git ASAP.\n\n> * [ PATCH 5/5] -D option to recurse into merged branches\n> * Re: [PATCH 5/5] -D option to recurse into merged branches\n> * [PATCH 2/2] archimport: sync_to_ps() messages for tracking tla methods\n\n-- \nEric Wong\n"},{"id":"12672","messageId":"20051124074739.GB4789@mail.yhbt.net","threadId":"2466","inReplyTo":"20051124074605.GA4789@mail.yhbt.net","subject":"[PATCH 1/9] archimport: first, make sure it still compiles","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-24T07:47:39Z","receivedAt":"2005-11-24T07:47:39Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Signed-off-by: Eric Wong \n\n---\n\n git-archimport.perl | 16 ++++++++++++++++\n 1 files changed, 16 insertions(+), 0 deletions(-)\n\napplies-to: a17c1f442587b9c7d68b4f7e08c5f6786599c61e\n119b07aa2bdb23d5f4977c4d696dd5e7eea56ca6\ndiff --git a/git-archimport.perl b/git-archimport.perl\nindex c3bed08..b5f8a2c 100755\n--- a/git-archimport.perl\n+++ b/git-archimport.perl\n@@ -99,6 +99,7 @@ my %psets = (); # the co\n \n my %rptags = (); # my reverse private tags\n # to map a SHA1 to a commitid\n+my $TLA = $ENV{'ARCH_CLIENT'} || 'tla';\n \n foreach my $root (@arch_roots) {\n my ($arepo, $abranch) = split(m!/!, $root);\n@@ -850,3 +851,18 @@ sub commitid2pset {\n \t|| (print Dumper(sort keys %psets)) && die \"Cannot find patchset for $name\";\n return $ps;\n }\n+\n+# an alterative to `command` that allows input to be passed as an array\n+# to work around shell problems with weird characters in arguments\n+sub safe_pipe_capture {\n+ my @output;\n+ if (my $pid = open my $child, '-|') {\n+ @output = (<$child>);\n+ close $child or die join(' ',@_).\": $! $?\";\n+ } else {\n+\texec(@_) or die $?; # exec() can fail the executable can't be found\n+ }\n+ return wantarray ? @output : join('',@output);\n+}\n+\n+\n---\n0.99.9.GIT\n\n-- \nEric Wong\n"},{"id":"12673","messageId":"20051124074857.GC4789@mail.yhbt.net","threadId":"2466","inReplyTo":"20051124074739.GB4789@mail.yhbt.net","subject":"[PATCH 2/9] remove String::ShellQuote dependency.","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-24T07:48:57Z","receivedAt":"2005-11-24T07:48:57Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"use safe_pipe_capture() or system() over backticks where\nshellquoting may have been necessary.\nMore changes planned, so I'm not touching the parts I'm\nplanning on replacing entirely.\n\nSigned-off-by: Eric Wong \n\n---\n\n git-archimport.perl | 51 ++++++++++++++++++++++++++++-----------------------\n 1 files changed, 28 insertions(+), 23 deletions(-)\n\napplies-to: 83307766d30e928179b9aa85a3d7bb906cc08846\n80494a7d496ab9f6e0a76a60b1f0b4215fdff442\ndiff --git a/git-archimport.perl b/git-archimport.perl\nindex b5f8a2c..b7e2480 100755\n--- a/git-archimport.perl\n+++ b/git-archimport.perl\n@@ -55,9 +55,8 @@ use warnings;\n use Getopt::Std;\n use File::Spec;\n use File::Temp qw(tempfile tempdir);\n-use File::Path qw(mkpath);\n+use File::Path qw(mkpath rmtree);\n use File::Basename qw(basename dirname);\n-use String::ShellQuote;\n use Time::Local;\n use IO::Socket;\n use IO::Pipe;\n@@ -306,7 +305,7 @@ foreach my $ps (@psets) {\n unless ($import) { # skip for import\n if ( -e \"$git_dir/refs/heads/$ps->{branch}\") {\n # we know about this branch\n- `git checkout $ps->{branch}`;\n+ system('git-checkout',$ps->{branch});\n } else {\n # new branch! we need to verify a few things\n die \"Branch on a non-tag!\" unless $ps->{type} eq 't';\n@@ -315,7 +314,7 @@ foreach my $ps (@psets) {\n unless $branchpoint;\n \n # find where we are supposed to branch from\n- `git checkout -b $ps->{branch} $branchpoint`;\n+ system('git-checkout','-b',$ps->{branch},$branchpoint);\n \n # If we trust Arch with the fact that this is just \n # a tag, and it does not affect the state of the tree\n@@ -344,7 +343,7 @@ foreach my $ps (@psets) {\n #\n my $tree;\n \n- my $commitlog = `tla cat-archive-log -A $ps->{repo} $ps->{id}`; \n+ my $commitlog = safe_pipe_capture($TLA,'cat-archive-log',$ps->{id}); \n die \"Error in cat-archive-log: $!\" if $?;\n \n # parselog will git-add/rm files\n@@ -422,7 +421,7 @@ foreach my $ps (@psets) {\n #\n my @par;\n if ( -e \"$git_dir/refs/heads/$ps->{branch}\") {\n- if (open HEAD, \"<$git_dir/refs/heads/$ps->{branch}\") {\n+ if (open HEAD, \"<\",\"$git_dir/refs/heads/$ps->{branch}\") {\n my $p = ;\n close HEAD;\n chomp $p;\n@@ -437,7 +436,6 @@ foreach my $ps (@psets) {\n if ($ps->{merges}) {\n push @par, find_parents($ps);\n }\n- my $par = join (' ', @par);\n \n # \n # Commit, tag and clean state\n@@ -454,7 +452,7 @@ foreach my $ps (@psets) {\n $commit_rh = 'commit_rh';\n $commit_wh = 'commit_wh';\n \n- $pid = open2(*READER, *WRITER, \"git-commit-tree $tree $par\") \n+ $pid = open2(*READER, *WRITER,'git-commit-tree',$tree,@par) \n or die $!;\n print WRITER $logmessage; # write\n close WRITER;\n@@ -469,7 +467,7 @@ foreach my $ps (@psets) {\n #\n # Update the branch\n # \n- open HEAD, \">$git_dir/refs/heads/$ps->{branch}\";\n+ open HEAD, \">\",\"$git_dir/refs/heads/$ps->{branch}\";\n print HEAD $commitid;\n close HEAD;\n system('git-update-ref', 'HEAD', \"$ps->{branch}\");\n@@ -483,21 +481,23 @@ foreach my $ps (@psets) {\n print \" + tree $tree\\n\";\n print \" + commit $commitid\\n\";\n $opt_v && print \" + commit date is $ps->{date} \\n\";\n- $opt_v && print \" + parents: $par \\n\";\n+ $opt_v && print \" + parents: \",join(' ',@par),\"\\n\";\n }\n \n sub apply_import {\n my $ps = shift;\n my $bname = git_branchname($ps->{id});\n \n- `mkdir -p $tmp`;\n+ mkpath($tmp);\n \n- `tla get -s --no-pristine -A $ps->{repo} $ps->{id} $tmp/import`;\n+ safe_pipe_capture($TLA,'get','-s','--no-pristine',$ps->{id},\"$tmp/import\");\n die \"Cannot get import: $!\" if $?; \n- `rsync -v --archive --delete --exclude '$git_dir' --exclude '.arch-ids' --exclude '{arch}' $tmp/import/* ./`;\n+ system('rsync','-aI','--delete', '--exclude',$git_dir,\n+\t\t'--exclude','.arch-ids','--exclude','{arch}',\n+\t\t\"$tmp/import/\", './');\n die \"Cannot rsync import:$!\" if $?;\n \n- `rm -fr $tmp/import`;\n+ rmtree(\"$tmp/import\");\n die \"Cannot remove tempdir: $!\" if $?;\n \n \n@@ -507,10 +507,10 @@ sub apply_import {\n sub apply_cset {\n my $ps = shift;\n \n- `mkdir -p $tmp`;\n+ mkpath($tmp);\n \n # get the changeset\n- `tla get-changeset -A $ps->{repo} $ps->{id} $tmp/changeset`;\n+ safe_pipe_capture($TLA,'get-changeset',$ps->{id},\"$tmp/changeset\");\n die \"Cannot get changeset: $!\" if $?;\n \n # apply patches\n@@ -534,17 +534,20 @@ sub apply_cset {\n $orig =~ s/\\.modified$//; # lazy\n $orig =~ s!^\\Q$tmp\\E/changeset/patches/!!;\n #print \"rsync -p '$mod' '$orig'\";\n- `rsync -p $mod ./$orig`;\n+ system('rsync','-p',$mod,\"./$orig\");\n die \"Problem applying binary changes! $!\" if $?;\n }\n }\n \n # bring in new files\n- `rsync --archive --exclude '$git_dir' --exclude '.arch-ids' --exclude '{arch}' $tmp/changeset/new-files-archive/* ./`;\n+ system('rsync','-aI','--exclude',$git_dir,\n+ \t\t'--exclude','.arch-ids',\n+\t\t'--exclude', '{arch}',\n+\t\t\"$tmp/changeset/new-files-archive/\",'./');\n \n # deleted files are hinted from the commitlog processing\n \n- `rm -fr $tmp/changeset`;\n+ rmtree(\"$tmp/changeset\");\n }\n \n \n@@ -622,9 +625,9 @@ sub parselog {\n # tla cat-archive-log will give us filenames with spaces as file\\(sp)name - why?\n # we can assume that any filename with \\ indicates some pika escaping that we want to get rid of.\n if ($t =~ /\\\\/ ){\n- $t = `tla escape --unescaped '$t'`;\n+ $t = (safe_pipe_capture($TLA,'escape','--unescaped',$t))[0];\n }\n- push (@tmp, shell_quote($t));\n+ push (@tmp, $t);\n }\n @$ref = @tmp;\n }\n@@ -827,8 +830,10 @@ sub find_parents {\n \t }\n \t}\n }\n- @parents = keys %parents;\n- @parents = map { \" -p \" . ptag($_) } @parents;\n+ @parents = ();\n+ foreach (keys %parents) {\n+ push @parents, '-p', ptag($_);\n+ }\n return @parents;\n }\n \n---\n0.99.9.GIT\n"},{"id":"12674","messageId":"20051124075027.GD4789@mail.yhbt.net","threadId":"2466","inReplyTo":"20051124074857.GC4789@mail.yhbt.net","subject":"[PATCH 3/9] fix -t tmpdir switch","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-24T07:50:27Z","receivedAt":"2005-11-24T07:50:27Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"set TMPDIR env correctly if -t is passed from the command-line.\nsetting TMPDIR => 1 as an argument to tempdir() has no effect otherwise\n\nSigned-off-by: Eric Wong \n\n---\n\n git-archimport.perl | 5 ++---\n 1 files changed, 2 insertions(+), 3 deletions(-)\n\napplies-to: 015fcfae8cdd564f0993940c5bac303c41913b1a\n25aadaf3ebc18fcc3c7948dc831d3f93447b03b6\ndiff --git a/git-archimport.perl b/git-archimport.perl\nindex b7e2480..2ed2e3c 100755\n--- a/git-archimport.perl\n+++ b/git-archimport.perl\n@@ -88,9 +88,8 @@ usage if $opt_h;\n @ARGV >= 1 or usage();\n my @arch_roots = @ARGV;\n \n-my ($tmpdir, $tmpdirname) = tempdir('git-archimport-XXXXXX', TMPDIR => 1, CLEANUP => 1);\n-my $tmp = $opt_t || 1;\n-$tmp = tempdir('git-archimport-XXXXXX', TMPDIR => 1, CLEANUP => 1);\n+$ENV{'TMPDIR'} = $opt_t if $opt_t; # $ENV{TMPDIR} will affect tempdir() calls:\n+my $tmp = tempdir('git-archimport-XXXXXX', TMPDIR => 1, CLEANUP => 1);\n $opt_v && print \"+ Using $tmp as temporary directory\\n\";\n \n my @psets = (); # the collection\n---\n0.99.9.GIT\n"},{"id":"12675","messageId":"20051124075133.GE4789@mail.yhbt.net","threadId":"2466","inReplyTo":"20051124075027.GD4789@mail.yhbt.net","subject":"[PATCH 4/9] remove git wrapper dependency","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-24T07:51:33Z","receivedAt":"2005-11-24T07:51:33Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"use git-diff-files instead of git diff-files so we don't rely on the\nwrapper being installed (some people may have git as GNU interactive\ntools :)\n\nSigned-off-by: Eric Wong \n\n---\n\n git-archimport.perl | 2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\napplies-to: 36c4ad6c2442e7700e5f1435d834f86d2680dd7f\n40b5378433aa6b09dd358d482d7713c2db6a5d92\ndiff --git a/git-archimport.perl b/git-archimport.perl\nindex 2ed2e3c..938fa2b 100755\n--- a/git-archimport.perl\n+++ b/git-archimport.perl\n@@ -278,7 +278,7 @@ foreach my $ps (@psets) {\n #\n # ensure we have a clean state \n # \n- if (`git diff-files`) {\n+ if (`git-diff-files`) {\n die \"Unclean tree when about to process $ps->{id} \" .\n \" - did we fail to commit cleanly before?\";\n }\n---\n0.99.9.GIT\n"},{"id":"12676","messageId":"20051124075243.GF4789@mail.yhbt.net","threadId":"2466","inReplyTo":"20051124075133.GE4789@mail.yhbt.net","subject":"[PATCH 5/9] add -D and -a switch","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-24T07:52:43Z","receivedAt":"2005-11-24T07:52:43Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"\nadd -D option to abrowse add -a switch to attempt to\nauto-register archives at mirrors.sourcecontrol.net\n\nSigned-off-by: Eric Wong \n\n---\n\n git-archimport.perl | 227 +++++++++++++++++++++++++++++++--------------------\n 1 files changed, 136 insertions(+), 91 deletions(-)\n\napplies-to: 4fd19922030196431179453910cb2b850b54c17d\nc52baf62ec29772f7a4e5b0aa2b605bf0f6aa8f7\ndiff --git a/git-archimport.perl b/git-archimport.perl\nindex 938fa2b..3968740 100755\n--- a/git-archimport.perl\n+++ b/git-archimport.perl\n@@ -53,14 +53,9 @@ and can contain multiple, unrelated bran\n use strict;\n use warnings;\n use Getopt::Std;\n-use File::Spec;\n-use File::Temp qw(tempfile tempdir);\n+use File::Temp qw(tempdir);\n use File::Path qw(mkpath rmtree);\n use File::Basename qw(basename dirname);\n-use Time::Local;\n-use IO::Socket;\n-use IO::Pipe;\n-use POSIX qw(strftime dup2);\n use Data::Dumper qw/ Dumper /;\n use IPC::Open2;\n \n@@ -71,27 +66,33 @@ my $git_dir = $ENV{\"GIT_DIR\"} || \".git\";\n $ENV{\"GIT_DIR\"} = $git_dir;\n my $ptag_dir = \"$git_dir/archimport/tags\";\n \n-our($opt_h,$opt_v, $opt_T,$opt_t,$opt_o);\n+our($opt_h,$opt_v,$opt_T,$opt_t,$opt_D,$opt_a,$opt_o);\n \n sub usage() {\n print STDERR <= 1 or usage();\n-my @arch_roots = @ARGV;\n+# $arch_branches:\n+# values associated with keys:\n+# =1 - Arch version / git 'branch' detected via abrowse on a limit\n+# >1 - Arch version / git 'branch' of an auxilliary branch we've merged\n+my %arch_branches = map { $_ => 1 } @ARGV;\n \n $ENV{'TMPDIR'} = $opt_t if $opt_t; # $ENV{TMPDIR} will affect tempdir() calls:\n my $tmp = tempdir('git-archimport-XXXXXX', TMPDIR => 1, CLEANUP => 1);\n $opt_v && print \"+ Using $tmp as temporary directory\\n\";\n \n+my %reachable = (); # Arch repositories we can access\n+my %unreachable = (); # Arch repositories we can't access :<\n my @psets = (); # the collection\n my %psets = (); # the collection, by name\n \n@@ -99,96 +100,112 @@ my %rptags = (); # my rev\n # to map a SHA1 to a commitid\n my $TLA = $ENV{'ARCH_CLIENT'} || 'tla';\n \n-foreach my $root (@arch_roots) {\n- my ($arepo, $abranch) = split(m!/!, $root);\n- open ABROWSE, \"tla abrowse -f -A $arepo --desc --merges $abranch |\" \n- or die \"Problems with tla abrowse: $!\";\n- \n- my %ps = (); # the current one\n- my $mode = '';\n- my $lastseen = '';\n- \n- while () {\n- chomp;\n+sub do_abrowse {\n+ my $stage = shift;\n+ while (my ($limit, $level) = each %arch_branches) {\n+ next unless $level == $stage;\n \n- # first record padded w 8 spaces\n- if (s/^\\s{8}\\b//) {\n- \n- # store the record we just captured\n- if (%ps) {\n- my %temp = %ps; # break references\n- push (@psets, \\%temp);\n-\t\t$psets{$temp{id}} = \\%temp;\n- %ps = ();\n- }\n- \n- my ($id, $type) = split(m/\\s{3}/, $_);\n- $ps{id} = $id;\n- $ps{repo} = $arepo;\n-\n- # deal with types\n- if ($type =~ m/^\\(simple changeset\\)/) {\n- $ps{type} = 's';\n- } elsif ($type eq '(initial import)') {\n- $ps{type} = 'i';\n- } elsif ($type =~ m/^\\(tag revision of (.+)\\)/) {\n- $ps{type} = 't';\n- $ps{tag} = $1;\n- } else { \n- warn \"Unknown type $type\";\n- }\n- $lastseen = 'id';\n- }\n- \n- if (s/^\\s{10}//) { \n- # 10 leading spaces or more \n- # indicate commit metadata\n+\topen ABROWSE, \"$TLA abrowse -fkD --merges $limit |\" \n+ or die \"Problems with tla abrowse: $!\";\n+ \n+ my %ps = (); # the current one\n+ my $lastseen = '';\n+ \n+ while () {\n+ chomp;\n \n- # date & author \n- if ($lastseen eq 'id' && m/^\\d{4}-\\d{2}-\\d{2}/) {\n+ # first record padded w 8 spaces\n+ if (s/^\\s{8}\\b//) {\n+ my ($id, $type) = split(m/\\s+/, $_, 2);\n+\n+ my %last_ps;\n+ # store the record we just captured\n+ if (%ps && !exists $psets{ $ps{id} }) {\n+ %last_ps = %ps; # break references\n+ push (@psets, \\%last_ps);\n+ $psets{ $last_ps{id} } = \\%last_ps;\n+ }\n \n- my ($date, $authoremail) = split(m/\\s{2,}/, $_);\n- $ps{date} = $date;\n- $ps{date} =~ s/\\bGMT$//; # strip off trailign GMT\n- if ($ps{date} =~ m/\\b\\w+$/) {\n- warn 'Arch dates not in GMT?! - imported dates will be wrong';\n+ my $branch = extract_versionname($id);\n+ %ps = ( id => $id, branch => $branch );\n+ if (%last_ps && ($last_ps{branch} eq $branch)) {\n+ $ps{parent_id} = $last_ps{id};\n+ }\n+ \n+ $arch_branches{$branch} = 1;\n+ $lastseen = 'id';\n+\n+ # deal with types (should work with baz or tla):\n+ if ($type =~ m/\\(.*changeset\\)/) {\n+ $ps{type} = 's';\n+ } elsif ($type =~ /\\(.*import\\)/) {\n+ $ps{type} = 'i';\n+ } elsif ($type =~ m/\\(tag.*\\)/) {\n+ $ps{type} = 't';\n+ # read which revision we've tagged when we parse the log\n+ #$ps{tag} = $1;\n+ } else { \n+ warn \"Unknown type $type\";\n+ }\n+\n+ $arch_branches{$branch} = 1;\n+ $lastseen = 'id';\n+ } elsif (s/^\\s{10}//) { \n+ # 10 leading spaces or more \n+ # indicate commit metadata\n+ \n+ # date\n+ if ($lastseen eq 'id' && m/^(\\d{4}-\\d\\d-\\d\\d \\d\\d:\\d\\d:\\d\\d)/){\n+ $ps{date} = $1;\n+ $lastseen = 'date';\n+ } elsif ($_ eq 'merges in:') {\n+ $ps{merges} = [];\n+ $lastseen = 'merges';\n+ } elsif ($lastseen eq 'merges' && s/^\\s{2}//) {\n+ my $id = $_;\n+ push (@{$ps{merges}}, $id);\n+ \n+ # aggressive branch finding:\n+ if ($opt_D) {\n+ my $branch = extract_versionname($id);\n+ my $repo = extract_reponame($branch);\n+ \n+ if (archive_reachable($repo) &&\n+ !defined $arch_branches{$branch}) {\n+ $arch_branches{$branch} = $stage + 1;\n+ }\n+ }\n+ } else {\n+ warn \"more metadata after merges!?: $_\\n\" unless /^\\s*$/;\n }\n- \n- $authoremail =~ m/^(.+)\\s(\\S+)$/;\n- $ps{author} = $1;\n- $ps{email} = $2;\n- \n- $lastseen = 'date';\n- \n- } elsif ($lastseen eq 'date') {\n- # the only hint is position\n- # subject is after date\n- $ps{subj} = $_;\n- $lastseen = 'subj';\n- \n- } elsif ($lastseen eq 'subj' && $_ eq 'merges in:') {\n- $ps{merges} = [];\n- $lastseen = 'merges';\n- \n- } elsif ($lastseen eq 'merges' && s/^\\s{2}//) {\n- push (@{$ps{merges}}, $_);\n- } else {\n- warn 'more metadata after merges!?';\n }\n- \n }\n- }\n \n- if (%ps) {\n- my %temp = %ps; # break references\n- push (@psets, \\%temp); \n-\t$psets{ $temp{id} } = \\%temp;\n- %ps = ();\n- } \n- close ABROWSE;\n+ if (%ps && !exists $psets{ $ps{id} }) {\n+ my %temp = %ps; # break references\n+ if (@psets && $psets[$#psets]{branch} eq $ps{branch}) {\n+ $temp{parent_id} = $psets[$#psets]{id};\n+ }\n+ push (@psets, \\%temp); \n+ $psets{ $temp{id} } = \\%temp;\n+ } \n+ \n+ close ABROWSE or die \"$TLA abrowse failed on $limit\\n\";\n+ }\n } # end foreach $root\n \n+do_abrowse(1);\n+my $depth = 2;\n+$opt_D ||= 0;\n+while ($depth <= $opt_D) {\n+ do_abrowse($depth);\n+ $depth++;\n+}\n+\n ## Order patches by time\n+# FIXME see if we can find a more optimal way to do this by graphing\n+# the ancestry data and walking it, that way we won't have to rely on\n+# client-supplied dates\n @psets = sort {$a->{date}.$b->{id} cmp $b->{date}.$b->{id}} @psets;\n \n #print Dumper \\@psets;\n@@ -209,7 +226,7 @@ unless (-d $git_dir) { # initial import\n }\n } else { # progressing an import\n # load the rptags\n- opendir(DIR, \"$git_dir/archimport/tags\")\n+ opendir(DIR, $ptag_dir)\n \t|| die \"can't opendir: $!\";\n while (my $file = readdir(DIR)) {\n # skip non-interesting-files\n@@ -829,6 +846,7 @@ sub find_parents {\n \t }\n \t}\n }\n+\n @parents = ();\n foreach (keys %parents) {\n push @parents, '-p', ptag($_);\n@@ -856,6 +874,7 @@ sub commitid2pset {\n return $ps;\n }\n \n+\n # an alterative to `command` that allows input to be passed as an array\n # to work around shell problems with weird characters in arguments\n sub safe_pipe_capture {\n@@ -869,4 +888,30 @@ sub safe_pipe_capture {\n return wantarray ? @output : join('',@output);\n }\n \n+# `tla logs -rf -d | head -n1` or `baz tree-id `\n+sub arch_tree_id {\n+ my $dir = shift;\n+ chomp( my $ret = (safe_pipe_capture($TLA,'logs','-rf','-d',$dir))[0] );\n+ return $ret;\n+}\n+\n+sub archive_reachable {\n+ my $archive = shift;\n+ return 1 if $reachable{$archive};\n+ return 0 if $unreachable{$archive};\n+ \n+ if (system \"$TLA whereis-archive $archive >/dev/null\") {\n+ if ($opt_a && (system($TLA,'register-archive',\n+ \"http://mirrors.sourcecontrol.net/$archive\") == 0)) {\n+ $reachable{$archive} = 1;\n+ return 1;\n+ }\n+ print STDERR \"Archive is unreachable: $archive\\n\";\n+ $unreachable{$archive} = 1;\n+ return 0;\n+ } else {\n+ $reachable{$archive} = 1;\n+ return 1;\n+ }\n+}\n \n---\n0.99.9.GIT\n"},{"id":"12678","messageId":"20051124075355.GG4789@mail.yhbt.net","threadId":"2466","inReplyTo":"20051124075243.GF4789@mail.yhbt.net","subject":"[PATCH 6/9] safer log file parsing","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-24T07:53:55Z","receivedAt":"2005-11-24T07:53:55Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Better logfile parsing, no longer confused by 'headers' after the first\nblank line.\n\nRe-enabled tag-reading with abrowse (baz and tla compatible)\n\nRemove need to quote args to external processes\n\nSigned-off-by: Eric Wong \n\n---\n\n git-archimport.perl | 211 +++++++++++++++++++++++++++------------------------\n 1 files changed, 112 insertions(+), 99 deletions(-)\n\napplies-to: 1633bcf09400e93aca2eb335181db298a5f49350\n3e12af1d958e2d631e27a2f696ca71f83094c7c3\ndiff --git a/git-archimport.perl b/git-archimport.perl\nindex 3968740..8676f35 100755\n--- a/git-archimport.perl\n+++ b/git-archimport.perl\n@@ -140,10 +140,10 @@ sub do_abrowse {\n $ps{type} = 's';\n } elsif ($type =~ /\\(.*import\\)/) {\n $ps{type} = 'i';\n- } elsif ($type =~ m/\\(tag.*\\)/) {\n+ } elsif ($type =~ m/\\(tag.*?(\\S+\\@\\S+).*?\\)/) {\n $ps{type} = 't';\n # read which revision we've tagged when we parse the log\n- #$ps{tag} = $1;\n+ $ps{tag} = $1;\n } else { \n warn \"Unknown type $type\";\n }\n@@ -359,78 +359,73 @@ foreach my $ps (@psets) {\n #\n my $tree;\n \n- my $commitlog = safe_pipe_capture($TLA,'cat-archive-log',$ps->{id}); \n+ my @commitlog = safe_pipe_capture($TLA,'cat-archive-log',$ps->{id}); \n die \"Error in cat-archive-log: $!\" if $?;\n \n- # parselog will git-add/rm files\n- # and generally prepare things for the commit\n- # NOTE: parselog will shell-quote filenames! \n- my ($sum, $msg, $add, $del, $mod, $ren) = parselog($commitlog);\n- my $logmessage = \"$sum\\n$msg\";\n-\n+ parselog($ps,\\@commitlog);\n \n # imports don't give us good info\n # on added files. Shame on them\n- if ($ps->{type} eq 'i' || $ps->{type} eq 't') { \n- `find . -type f -print0 | grep -zv '^./$git_dir' | xargs -0 -l100 git-update-index --add`;\n- `git-ls-files --deleted -z | xargs --no-run-if-empty -0 -l100 git-update-index --remove`;\n+ if ($ps->{type} eq 'i' || $ps->{type} eq 't') {\n+ system('git-ls-files --others -z | '.\n+ 'git-update-index --add -z --stdin') == 0 or die \"$! $?\\n\";\n+ system('git-ls-files --deleted -z | '.\n+ 'git-update-index --remove -z --stdin') == 0 or die \"$! $?\\n\";\n }\n \n- if (@$add) {\n+ # TODO: handle removed_directories and renamed_directories:\n+ \n+ if (my $add = $ps->{new_files}) {\n while (@$add) {\n my @slice = splice(@$add, 0, 100);\n- my $slice = join(' ', @slice); \n- `git-update-index --add $slice`;\n- die \"Error in git-update-index --add: $!\" if $?;\n+ system('git-update-index','--add','--',@slice) == 0 or\n+ die \"Error in git-update-index --add: $! $?\\n\";\n }\n }\n- if (@$del) {\n- foreach my $file (@$del) {\n- unlink $file or die \"Problems deleting $file : $!\";\n- }\n+ \n+ if (my $del = $ps->{removed_files}) {\n+ unlink @$del;\n while (@$del) {\n my @slice = splice(@$del, 0, 100);\n- my $slice = join(' ', @slice);\n- `git-update-index --remove $slice`;\n- die \"Error in git-update-index --remove: $!\" if $?;\n+ system('git-update-index','--remove','--',@slice) == 0 or\n+ die \"Error in git-update-index --remove: $! $?\\n\";\n }\n }\n- if (@$ren) { # renamed\n+\n+ if (my $ren = $ps->{renamed_files}) { # renamed\n if (@$ren % 2) {\n die \"Odd number of entries in rename!?\";\n }\n- ;\n+ \n while (@$ren) {\n- my $from = pop @$ren;\n- my $to = pop @$ren; \n+ my $from = shift @$ren;\n+ my $to = shift @$ren; \n \n unless (-d dirname($to)) {\n mkpath(dirname($to)); # will die on err\n }\n- #print \"moving $from $to\";\n- `mv $from $to`;\n- die \"Error renaming $from $to : $!\" if $?;\n- `git-update-index --remove $from`;\n- die \"Error in git-update-index --remove: $!\" if $?;\n- `git-update-index --add $to`;\n- die \"Error in git-update-index --add: $!\" if $?;\n+ print \"moving $from $to\";\n+ rename($from, $to) or die \"Error renaming '$from' '$to': $!\\n\";\n+ system('git-update-index','--remove','--',$from) == 0 or\n+ die \"Error in git-update-index --remove: $! $?\\n\";\n+ system('git-update-index','--add','--',$to) == 0 or\n+ die \"Error in git-update-index --add: $! $?\\n\";\n }\n \n }\n- if (@$mod) { # must be _after_ renames\n+\n+ if (my $mod = $ps->{modified_files}) {\n while (@$mod) {\n my @slice = splice(@$mod, 0, 100);\n- my $slice = join(' ', @slice);\n- `git-update-index $slice`;\n- die \"Error in git-update-index: $!\" if $?;\n+ system('git-update-index','--',@slice) == 0 or\n+ die \"Error in git-update-index: $! $?\\n\";\n }\n }\n-\n+ \n # warn \"errors when running git-update-index! $!\";\n $tree = `git-write-tree`;\n die \"cannot write tree $!\" if $?;\n chomp $tree;\n- \n \n #\n # Who's your daddy?\n@@ -464,13 +459,14 @@ foreach my $ps (@psets) {\n $ENV{GIT_COMMITTER_EMAIL} = $ps->{email};\n $ENV{GIT_COMMITTER_DATE} = $ps->{date};\n \n- my ($pid, $commit_rh, $commit_wh);\n- $commit_rh = 'commit_rh';\n- $commit_wh = 'commit_wh';\n- \n- $pid = open2(*READER, *WRITER,'git-commit-tree',$tree,@par) \n+ my $pid = open2(*READER, *WRITER,'git-commit-tree',$tree,@par) \n or die $!;\n- print WRITER $logmessage; # write\n+ print WRITER $ps->{summary},\"\\n\";\n+ print WRITER $ps->{message},\"\\n\";\n+ \n+ # make it easy to backtrack and figure out which Arch revision this was:\n+ print WRITER 'git-archimport-id: ',$ps->{id},\"\\n\";\n+ \n close WRITER;\n my $commitid = ; # read\n chomp $commitid;\n@@ -568,7 +564,9 @@ sub apply_cset {\n \n \n # =for reference\n-# A log entry looks like \n+# notes: *-files/-directories keys cannot have spaces, they're always\n+# pika-escaped. Everything after the first newline\n+# A log entry looks like:\n # Revision: moodle-org--moodle--1.3.3--patch-15\n # Archive: arch-eduforge@catalyst.net.nz--2004\n # Creator: Penny Leach \n@@ -586,70 +584,85 @@ sub apply_cset {\n # admin/editor.html backup/lib.php backup/restore.php\n # New-patches: arch-eduforge@catalyst.net.nz--2004/moodle-org--moodle--1.3.3--patch-15\n # Summary: Updating to latest from MOODLE_14_STABLE (1.4.5+)\n+# summary can be multiline with a leading space just like the above fields\n # Keywords:\n #\n # Updating yadda tadda tadda madda\n sub parselog {\n- my $log = shift;\n- #print $log;\n-\n- my (@add, @del, @mod, @ren, @kw, $sum, $msg );\n-\n- if ($log =~ m/(?:\\n|^)New-files:(.*?)(?=\\n\\w)/s ) {\n- my $files = $1;\n- @add = split(m/\\s+/s, $files);\n- }\n- \n- if ($log =~ m/(?:\\n|^)Removed-files:(.*?)(?=\\n\\w)/s ) {\n- my $files = $1;\n- @del = split(m/\\s+/s, $files);\n- }\n- \n- if ($log =~ m/(?:\\n|^)Modified-files:(.*?)(?=\\n\\w)/s ) {\n- my $files = $1;\n- @mod = split(m/\\s+/s, $files);\n- }\n- \n- if ($log =~ m/(?:\\n|^)Renamed-files:(.*?)(?=\\n\\w)/s ) {\n- my $files = $1;\n- @ren = split(m/\\s+/s, $files);\n- }\n-\n- $sum ='';\n- if ($log =~ m/^Summary:(.+?)$/m ) {\n- $sum = $1;\n- $sum =~ s/^\\s+//;\n- $sum =~ s/\\s+$//;\n- }\n+ my ($ps, $log) = @_;\n+ my $key = undef;\n \n- $msg = '';\n- if ($log =~ m/\\n\\n(.+)$/s) {\n- $msg = $1;\n- $msg =~ s/^\\s+//;\n- $msg =~ s/\\s+$//;\n+ # headers we want that contain filenames:\n+ my %want_headers = (\n+ new_files => 1,\n+ modified_files => 1,\n+ renamed_files => 1,\n+ renamed_directories => 1,\n+ removed_files => 1,\n+ removed_directories => 1,\n+ );\n+ \n+ chomp (@$log);\n+ while ($_ = shift @$log) {\n+ if (/^Continuation-of:\\s*(.*)/) {\n+ $ps->{tag} = $1;\n+ $key = undef;\n+ } elsif (/^Summary:\\s*(.*)$/ ) {\n+ # summary can be multiline as long as it has a leading space\n+ $ps->{summary} = [ $1 ];\n+ $key = 'summary';\n+ } elsif (/^Creator: (.*)\\s*<([^\\>]+)>/) {\n+ $ps->{author} = $1;\n+ $ps->{email} = $2;\n+ $key = undef;\n+ # any *-files or *-directories can be read here:\n+ } elsif (/^([A-Z][a-z\\-]+):\\s*(.*)$/) {\n+ my $val = $2;\n+ $key = lc $1;\n+ $key =~ tr/-/_/; # too lazy to quote :P\n+ if ($want_headers{$key}) {\n+ push @{$ps->{$key}}, split(/\\s+/, $val);\n+ } else {\n+ $key = undef;\n+ }\n+ } elsif (/^$/) {\n+ last; # remainder of @$log that didn't get shifted off is message\n+ } elsif ($key) {\n+ if (/^\\s+(.*)$/) {\n+ if ($key eq 'summary') {\n+ push @{$ps->{$key}}, $1;\n+ } else { # files/directories:\n+ push @{$ps->{$key}}, split(/\\s+/, $1);\n+ }\n+ } else {\n+ $key = undef;\n+ }\n+ }\n }\n-\n-\n- # cleanup the arrays\n- foreach my $ref ( (\\@add, \\@del, \\@mod, \\@ren) ) {\n- my @tmp = ();\n- while (my $t = pop @$ref) {\n- next unless length ($t);\n- next if $t =~ m!\\{arch\\}/!;\n- next if $t =~ m!\\.arch-ids/!;\n- next if $t =~ m!\\.arch-inventory$!;\n+ \n+ # post-processing:\n+ $ps->{summary} = join(\"\\n\",@{$ps->{summary}}).\"\\n\";\n+ $ps->{message} = join(\"\\n\",@$log);\n+ \n+ # skip Arch control files, unescape pika-escaped files\n+ foreach my $k (keys %want_headers) {\n+ next unless (defined $ps->{$k});\n+ my @tmp;\n+ foreach my $t (@{$ps->{$k}}) {\n+ next unless length ($t);\n+ next if $t =~ m!\\{arch\\}/!;\n+ next if $t =~ m!\\.arch-ids/!;\n+ # should we skip this?\n+ next if $t =~ m!\\.arch-inventory$!;\n # tla cat-archive-log will give us filenames with spaces as file\\(sp)name - why?\n # we can assume that any filename with \\ indicates some pika escaping that we want to get rid of.\n- if ($t =~ /\\\\/ ){\n+ if ($t =~ /\\\\/ ){\n $t = (safe_pipe_capture($TLA,'escape','--unescaped',$t))[0];\n }\n- push (@tmp, $t);\n+ push @tmp, $t;\n }\n- @$ref = @tmp;\n+ $ps->{$k} = \\@tmp if scalar @tmp;\n }\n- \n- #print Dumper [$sum, $msg, \\@add, \\@del, \\@mod, \\@ren]; \n- return ($sum, $msg, \\@add, \\@del, \\@mod, \\@ren); \n }\n \n # write/read a tag\n---\n0.99.9.GIT\n"},{"id":"12677","messageId":"20051124075504.GH4789@mail.yhbt.net","threadId":"2466","inReplyTo":"20051124075355.GG4789@mail.yhbt.net","subject":"[PATCH 7/9] Add the accurate changeset applyer","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-24T07:55:04Z","receivedAt":"2005-11-24T07:55:04Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"And make it the default.\nThis includes stats tracking to verbose mode\n\nSigned-off-by: Eric Wong \n\n---\n\n git-archimport.perl | 201 ++++++++++++++++++++++++++++++++++++++++++++-------\n 1 files changed, 172 insertions(+), 29 deletions(-)\n\napplies-to: aa9140057c95e59f65de0794f9054796fbfc96e5\n32e5887eedb01ac4c398a06b0a1433ff6f4599fe\ndiff --git a/git-archimport.perl b/git-archimport.perl\nindex 8676f35..1cf1261 100755\n--- a/git-archimport.perl\n+++ b/git-archimport.perl\n@@ -25,6 +25,9 @@ See man (1) git-archimport for more deta\n - audit shell-escaping of filenames\n - hide our private tags somewhere smarter\n - find a way to make \"cat *patches | patch\" safe even when patchfiles are missing newlines \n+ - sort and apply patches by graphing ancestry relations instead of just\n+ relying in dates supplied in the changeset itself.\n+ tla ancestry-graph -m could be helpful here...\n \n =head1 Devel tricks\n \n@@ -66,18 +69,18 @@ my $git_dir = $ENV{\"GIT_DIR\"} || \".git\";\n $ENV{\"GIT_DIR\"} = $git_dir;\n my $ptag_dir = \"$git_dir/archimport/tags\";\n \n-our($opt_h,$opt_v,$opt_T,$opt_t,$opt_D,$opt_a,$opt_o);\n+our($opt_h,$opt_f,$opt_v,$opt_T,$opt_t,$opt_D,$opt_a,$opt_o);\n \n sub usage() {\n print STDERR <= 1 or usage();\n@@ -95,6 +98,10 @@ my %reachable = (); # Arch r\n my %unreachable = (); # Arch repositories we can't access :<\n my @psets = (); # the collection\n my %psets = (); # the collection, by name\n+my %stats = (\t\t\t# Track which strategy we used to import:\n+\tget_tag => 0, replay => 0, get_new => 0, get_delta => 0,\n+ simple_changeset => 0, import_or_tag => 0\n+);\n \n my %rptags = (); # my reverse private tags\n # to map a SHA1 to a commitid\n@@ -288,29 +295,69 @@ sub old_style_branchname {\n \n *git_branchname = $opt_o ? *old_style_branchname : *tree_dirname;\n \n-# process patchsets\n-foreach my $ps (@psets) {\n- $ps->{branch} = git_branchname($ps->{id});\n-\n- #\n- # ensure we have a clean state \n- # \n- if (`git-diff-files`) {\n- die \"Unclean tree when about to process $ps->{id} \" .\n- \" - did we fail to commit cleanly before?\";\n- }\n- die $! if $?;\n-\n- #\n- # skip commits already in repo\n- #\n- if (ptag($ps->{id})) {\n- $opt_v && print \" * Skipping already imported: $ps->{id}\\n\";\n- next;\n+sub process_patchset_accurate {\n+ my $ps = shift;\n+ \n+ # switch to that branch if we're not already in that branch:\n+ if (-e \"$git_dir/refs/heads/$ps->{branch}\") {\n+ system('git-checkout','-f',$ps->{branch}) == 0 or die \"$! $?\\n\";\n+\n+ # remove any old stuff that got leftover:\n+ my $rm = safe_pipe_capture('git-ls-files','--others','-z');\n+ rmtree(split(/\\0/,$rm)) if $rm;\n }\n+ \n+ # Apply the import/changeset/merge into the working tree\n+ my $dir = sync_to_ps($ps);\n+ # read the new log entry:\n+ my @commitlog = safe_pipe_capture($TLA,'cat-log','-d',$dir,$ps->{id});\n+ die \"Error in cat-log: $!\" if $?;\n+ chomp @commitlog;\n+\n+ # grab variables we want from the log, new fields get added to $ps:\n+ # (author, date, email, summary, message body ...)\n+ parselog($ps, \\@commitlog);\n+\n+ if ($ps->{id} =~ /--base-0$/ && $ps->{id} ne $psets[0]{id}) {\n+ # this should work when importing continuations \n+ if ($ps->{tag} && (my $branchpoint = eval { ptag($ps->{tag}) })) {\n+ \n+ # find where we are supposed to branch from\n+ system('git-checkout','-f','-b',$ps->{branch},\n+ $branchpoint) == 0 or die \"$! $?\\n\";\n+ \n+ # remove any old stuff that got leftover:\n+ my $rm = safe_pipe_capture('git-ls-files','--others','-z');\n+ rmtree(split(/\\0/,$rm)) if $rm;\n \n- print \" * Starting to work on $ps->{id}\\n\";\n+ # If we trust Arch with the fact that this is just \n+ # a tag, and it does not affect the state of the tree\n+ # then we just tag and move on\n+ tag($ps->{id}, $branchpoint);\n+ ptag($ps->{id}, $branchpoint);\n+ print \" * Tagged $ps->{id} at $branchpoint\\n\";\n+ return 0;\n+ } else {\n+ warn \"Tagging from unknown id unsupported\\n\" if $ps->{tag};\n+ }\n+ # allow multiple bases/imports here since Arch supports cherry-picks\n+ # from unrelated trees\n+ } \n+ \n+ # update the index with all the changes we got\n+ system('git-ls-files --others -z | '.\n+ 'git-update-index --add -z --stdin') == 0 or die \"$! $?\\n\";\n+ system('git-ls-files --deleted -z | '.\n+ 'git-update-index --remove -z --stdin') == 0 or die \"$! $?\\n\";\n+ system('git-ls-files -z | '.\n+ 'git-update-index -z --stdin') == 0 or die \"$! $?\\n\";\n+ return 1;\n+}\n \n+# the native changeset processing strategy. This is very fast, but\n+# does not handle permissions or any renames involving directories\n+sub process_patchset_fast {\n+ my $ps = shift;\n # \n # create the branch if needed\n #\n@@ -338,7 +385,7 @@ foreach my $ps (@psets) {\n tag($ps->{id}, $branchpoint);\n ptag($ps->{id}, $branchpoint);\n print \" * Tagged $ps->{id} at $branchpoint\\n\";\n- next;\n+ return 0;\n } \n die $! if $?;\n } \n@@ -348,16 +395,17 @@ foreach my $ps (@psets) {\n # \n if ($ps->{type} eq 'i' || $ps->{type} eq 't') {\n apply_import($ps) or die $!;\n+ $stats{import_or_tag}++;\n $import=0;\n } elsif ($ps->{type} eq 's') {\n apply_cset($ps);\n+ $stats{simple_changeset}++;\n }\n \n #\n # prepare update git's index, based on what arch knows\n # about the pset, resolve parents, etc\n #\n- my $tree;\n \n my @commitlog = safe_pipe_capture($TLA,'cat-archive-log',$ps->{id}); \n die \"Error in cat-archive-log: $!\" if $?;\n@@ -404,14 +452,13 @@ foreach my $ps (@psets) {\n unless (-d dirname($to)) {\n mkpath(dirname($to)); # will die on err\n }\n- print \"moving $from $to\";\n+ # print \"moving $from $to\";\n rename($from, $to) or die \"Error renaming '$from' '$to': $!\\n\";\n system('git-update-index','--remove','--',$from) == 0 or\n die \"Error in git-update-index --remove: $! $?\\n\";\n system('git-update-index','--add','--',$to) == 0 or\n die \"Error in git-update-index --add: $! $?\\n\";\n }\n-\n }\n \n if (my $mod = $ps->{modified_files}) {\n@@ -421,9 +468,46 @@ foreach my $ps (@psets) {\n die \"Error in git-update-index: $! $?\\n\";\n }\n }\n+ return 1; # we successfully applied the changeset\n+}\n+\n+if ($opt_f) {\n+ print \"Will import patchsets using the fast strategy\\n\",\n+ \"Renamed directories and permission changes will be missed\\n\";\n+ *process_patchset = *process_patchset_fast;\n+} else {\n+ print \"Using the default (accurate) import strategy.\\n\",\n+ \"Things may be a bit slow\\n\";\n+ *process_patchset = *process_patchset_accurate;\n+}\n+ \n+foreach my $ps (@psets) {\n+ # process patchsets\n+ $ps->{branch} = git_branchname($ps->{id});\n+\n+ #\n+ # ensure we have a clean state \n+ # \n+ if (my $dirty = `git-diff-files`) {\n+ die \"Unclean tree when about to process $ps->{id} \" .\n+ \" - did we fail to commit cleanly before?\\n$dirty\";\n+ }\n+ die $! if $?;\n \n+ #\n+ # skip commits already in repo\n+ #\n+ if (ptag($ps->{id})) {\n+ $opt_v && print \" * Skipping already imported: $ps->{id}\\n\";\n+ return 0;\n+ }\n+\n+ print \" * Starting to work on $ps->{id}\\n\";\n+\n+ process_patchset($ps) or next;\n+\n # warn \"errors when running git-update-index! $!\";\n- $tree = `git-write-tree`;\n+ my $tree = `git-write-tree`;\n die \"cannot write tree $!\" if $?;\n chomp $tree;\n \n@@ -494,6 +578,65 @@ foreach my $ps (@psets) {\n print \" + commit $commitid\\n\";\n $opt_v && print \" + commit date is $ps->{date} \\n\";\n $opt_v && print \" + parents: \",join(' ',@par),\"\\n\";\n+ if (my $dirty = `git-diff-files`) {\n+ die \"22 Unclean tree when about to process $ps->{id} \" .\n+ \" - did we fail to commit cleanly before?\\n$dirty\";\n+ }\n+}\n+\n+if ($opt_v) {\n+ foreach (sort keys %stats) {\n+ print\" $_: $stats{$_}\\n\";\n+ }\n+}\n+exit 0;\n+\n+# used by the accurate strategy:\n+sub sync_to_ps {\n+ my $ps = shift;\n+ my $tree_dir = $tmp.'/'.tree_dirname($ps->{id});\n+ \n+ $opt_v && print \"sync_to_ps($ps->{id}) method: \";\n+\n+ if (-d $tree_dir) {\n+ if ($ps->{type} eq 't') {\n+\t $opt_v && print \"get (tag)\\n\";\n+ # looks like a tag-only or (worse,) a mixed tags/changeset branch,\n+ # can't rely on replay to work correctly on these\n+ rmtree($tree_dir);\n+ safe_pipe_capture($TLA,'get','--no-pristine',$ps->{id},$tree_dir);\n+ $stats{get_tag}++;\n+ } else {\n+ my $tree_id = arch_tree_id($tree_dir);\n+ if ($ps->{parent_id} && ($ps->{parent_id} eq $tree_id)) {\n+ # the common case (hopefully)\n+\t\t $opt_v && print \"replay\\n\";\n+ safe_pipe_capture($TLA,'replay','-d',$tree_dir,$ps->{id});\n+ $stats{replay}++;\n+ } else {\n+ # getting one tree is usually faster than getting two trees\n+ # and applying the delta ...\n+ rmtree($tree_dir);\n+\t\t $opt_v && print \"apply-delta\\n\";\n+ safe_pipe_capture($TLA,'get','--no-pristine',\n+ $ps->{id},$tree_dir);\n+ $stats{get_delta}++;\n+ }\n+ }\n+ } else {\n+ # new branch work\n+ $opt_v && print \"get (new tree)\\n\";\n+ safe_pipe_capture($TLA,'get','--no-pristine',$ps->{id},$tree_dir);\n+ $stats{get_new}++;\n+ }\n+ \n+ # added -I flag to rsync since we're going to fast! AIEEEEE!!!!\n+ system('rsync','-aI','--delete','--exclude',$git_dir,\n+# '--exclude','.arch-inventory',\n+ '--exclude','.arch-ids','--exclude','{arch}',\n+ '--exclude','+*','--exclude',',*',\n+ \"$tree_dir/\",'./') == 0 or die \"Cannot rsync $tree_dir: $! $?\";\n+ return $tree_dir;\n }\n \n sub apply_import {\n@@ -896,7 +1039,7 @@ sub safe_pipe_capture {\n @output = (<$child>);\n close $child or die join(' ',@_).\": $! $?\";\n } else {\n-\texec(@_) or die $?; # exec() can fail the executable can't be found\n+\texec(@_) or die \"$! $?\"; # exec() can fail the executable can't be found\n }\n return wantarray ? @output : join('',@output);\n }\n---\n0.99.9.GIT\n"},{"id":"12679","messageId":"20051124075631.GI4789@mail.yhbt.net","threadId":"2466","inReplyTo":"20051124075504.GH4789@mail.yhbt.net","subject":"[PATCH 8/9] Fix a bug I introduced in the new log parser","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-24T07:56:31Z","receivedAt":"2005-11-24T07:56:31Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"This fixes the case (that worked originally in Martin's version)\nwhere the only new/modified files are Arch control files.\n\nSigned-off-by: Eric Wong \n\n---\n\n git-archimport.perl | 4 ++--\n 1 files changed, 2 insertions(+), 2 deletions(-)\n\napplies-to: db1362fe6567e349ff9dd9d70ce23c88a42a0ff2\nebe0689722f6c1440e680ec9a235b3dd571c7de0\ndiff --git a/git-archimport.perl b/git-archimport.perl\nindex 1cf1261..0080850 100755\n--- a/git-archimport.perl\n+++ b/git-archimport.perl\n@@ -790,7 +790,7 @@ sub parselog {\n # skip Arch control files, unescape pika-escaped files\n foreach my $k (keys %want_headers) {\n next unless (defined $ps->{$k});\n- my @tmp;\n+ my @tmp = ();\n foreach my $t (@{$ps->{$k}}) {\n next unless length ($t);\n next if $t =~ m!\\{arch\\}/!;\n@@ -804,7 +804,7 @@ sub parselog {\n }\n push @tmp, $t;\n }\n- $ps->{$k} = \\@tmp if scalar @tmp;\n+ $ps->{$k} = \\@tmp;\n }\n }\n \n---\n0.99.9.GIT\n"},{"id":"12680","messageId":"20051124075816.GJ4789@mail.yhbt.net","threadId":"2466","inReplyTo":"20051124075631.GI4789@mail.yhbt.net","subject":"[PATCH 9/9] fix a in new changeset applyer addition","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-24T07:58:16Z","receivedAt":"2005-11-24T07:58:16Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Fix a stupid bug I introduced when splitting the \naccurate and fast changeset appliers.\n\nAlso, remove an old debugging statement I added\n\nSigned-off-by: Eric Wong \n\n---\n\n git-archimport.perl | 6 +-----\n 1 files changed, 1 insertions(+), 5 deletions(-)\n\napplies-to: 6dfed0cb7c209cf47902d6dfcd02a974d252041b\nb081cb1e0f79f1a290bcf1f2161d63415ec5e2a9\ndiff --git a/git-archimport.perl b/git-archimport.perl\nindex 0080850..aab4e38 100755\n--- a/git-archimport.perl\n+++ b/git-archimport.perl\n@@ -499,7 +499,7 @@ foreach my $ps (@psets) {\n #\n if (ptag($ps->{id})) {\n $opt_v && print \" * Skipping already imported: $ps->{id}\\n\";\n- return 0;\n+ next;\n }\n \n print \" * Starting to work on $ps->{id}\\n\";\n@@ -578,10 +578,6 @@ foreach my $ps (@psets) {\n print \" + commit $commitid\\n\";\n $opt_v && print \" + commit date is $ps->{date} \\n\";\n $opt_v && print \" + parents: \",join(' ',@par),\"\\n\";\n- if (my $dirty = `git-diff-files`) {\n- die \"22 Unclean tree when about to process $ps->{id} \" .\n- \" - did we fail to commit cleanly before?\\n$dirty\";\n- }\n }\n \n if ($opt_v) {\n---\n0.99.9.GIT\n"},{"id":"12683","messageId":"438577E0.30109@op5.se","threadId":"2466","inReplyTo":"20051124075133.GE4789@mail.yhbt.net","subject":"Re: [PATCH 4/9] remove git wrapper dependency","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2005-11-24T08:20:48Z","receivedAt":"2005-11-24T08:20:48Z","isPatch":true,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Eric Wong wrote:\n> use git-diff-files instead of git diff-files so we don't rely on the\n> wrapper being installed (some people may have git as GNU interactive\n> tools :)\n> \n\nThis one should do\n\tgit --exec-path\n\nfirst to get the proper path to git-diff-files. Fall back to it being in \nthe path if finding out fails.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n"},{"id":"12685","messageId":"7voe4awii5.fsf@assigned-by-dhcp.cox.net","threadId":"2466","inReplyTo":"438577E0.30109@op5.se","subject":"Re: [PATCH 4/9] remove git wrapper dependency","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-11-24T08:35:30Z","receivedAt":"2005-11-24T08:35:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Andreas Ericsson writes:\n\n> Eric Wong wrote:\n>> use git-diff-files instead of git diff-files so we don't rely on the\n>> wrapper being installed (some people may have git as GNU interactive\n>> tools :)\n>>\n>\n> This one should do\n> \tgit --exec-path\n>\n> first to get the proper path to git-diff-files. Fall back to it being in \n> the path if finding out fails.\n\nEric is worried about the case where git on your PATH is GNU\ninteractive tools, so \"git --exec-path\" would not give you what\nyou want ;-).\n"},{"id":"12687","messageId":"20051124085020.GK4789@mail.yhbt.net","threadId":"2466","inReplyTo":"7voe4awii5.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH 4/9] remove git wrapper dependency","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-24T08:50:21Z","receivedAt":"2005-11-24T08:50:21Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Junio C Hamano wrote:\n> Andreas Ericsson writes:\n> \n> > Eric Wong wrote:\n> >> use git-diff-files instead of git diff-files so we don't rely on the\n> >> wrapper being installed (some people may have git as GNU interactive\n> >> tools :)\n> >>\n> >\n> > This one should do\n> > \tgit --exec-path\n> >\n> > first to get the proper path to git-diff-files. Fall back to it being in \n> > the path if finding out fails.\n> \n> Eric is worried about the case where git on your PATH is GNU\n> interactive tools, so \"git --exec-path\" would not give you what\n> you want ;-).\n\nRight on. I'm actually not a GNU interactive tools user, but I do have\nempathy for them being a cg (cgvg) user myself for many, many years.\n\n-- \nEric Wong\n"},{"id":"12689","messageId":"46a038f90511240125s1617cd44n868d77204ea87f78@mail.gmail.com","threadId":"2466","inReplyTo":"20051124074605.GA4789@mail.yhbt.net","subject":"Re: [PATCH] archimport improvements","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2005-11-24T09:25:31Z","receivedAt":"2005-11-24T09:25:31Z","isPatch":true,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 11/24/05, Eric Wong wrote:\n> Ok, I didn't expect you guys to have 12k of files in your trees. None\n> of your source trees are remotely close to that size (but I have many\n> more changesets). I'm surprised you guys were able to put up\n> with Arch in the first place!\n>\n> 125m58.431s with my method.\n> 8m24.504s with yours :)\n>\n> All of my usual source trees imported 1k changesets in 10-15 minutes\n\n:-) I'm happy that you managed to wait patiently for it to complete --\nall my attempts to run your import code were ended by a sleepy ctrl-c.\n\n> Patches on the way.\n\nCool -- will review, but may take a couple days, as I'm away from home\nthis week.\n\n> OTOH, the time spent importing the bulk of the history is a one-time\n> operation for most people and I'd much rather it get things as right as\n> possible and move on.\n\nHmmm. Some teams -- such as mine -- just run it every couple hours to\nmaintain an Arch2cvs gateway.\n\nMore later,\n\n\nmartin\n"},{"id":"12701","messageId":"Pine.LNX.4.64.0511241051170.13959@g5.osdl.org","threadId":"2466","inReplyTo":"20051124074739.GB4789@mail.yhbt.net","subject":"Re: [PATCH 1/9] archimport: first, make sure it still compiles","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-11-24T18:54:10Z","receivedAt":"2005-11-24T18:54:10Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nEric,\n I don't know about Junio, but if I were him, I'd have preferred that all \nyour patches had a\n\n\tarchimport: ..\n\nprefix in the subject line, not just the first one.\n\nFor example, if you just merge the patches as-is now, and then look at the \nend result with gitk (or any of the tools that show the shortlog format: \njust the first line of the commit), you get explanations like\n\n\tfix -t tmpdir switch\n\nwhich is clearly _correct_, but it's much nicer if they show which area \nwas implied, ie\n\n\tarchimport: fix -t tmpdir switch\n\nso that you can tell from the shortlog whether it was a \"global\" change, \nor somethign that affected a specific program.\n\nJust a suggestion,\n\n\t\tLinus\n"},{"id":"12768","messageId":"46a038f90511260251x7c470c2esc6e87048cf8eda6b@mail.gmail.com","threadId":"2466","inReplyTo":"Pine.LNX.4.64.0511241051170.13959@g5.osdl.org","subject":"Re: [PATCH 1/9] archimport: first, make sure it still compiles","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2005-11-26T10:51:14Z","receivedAt":"2005-11-26T10:51:14Z","isPatch":true,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 11/24/05, Linus Torvalds wrote:\n> I don't know about Junio, but if I were him, I'd have preferred that all\n> your patches had a\n>\n> archimport: ..\n>\n> prefix in the subject line, not just the first one.\n\nGood catch -- I'll prefix them all as I merge them. If Junio pulls\nfrom my tree, he'll get them prefixed.\n\ncheers,\n\n\n\nmartin\n"},{"id":"12777","messageId":"20051126204338.GB3899@mail.yhbt.net","threadId":"2466","inReplyTo":"Pine.LNX.4.64.0511241051170.13959@g5.osdl.org","subject":"Re: [PATCH 1/9] archimport: first, make sure it still compiles","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-26T20:43:38Z","receivedAt":"2005-11-26T20:43:38Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Linus Torvalds wrote:\n> \n> \n> Eric,\n> I don't know about Junio, but if I were him, I'd have preferred that all \n> your patches had a\n> \n> \tarchimport: ..\n> \n> prefix in the subject line, not just the first one.\n\nGood idea, will do for future patches.\n\n-- \nEric Wong\n"},{"id":"12787","messageId":"46a038f90511262024g4eca7dc2oa42d1f72febf9e27@mail.gmail.com","threadId":"2466","inReplyTo":"20051124075504.GH4789@mail.yhbt.net","subject":"Re: [PATCH 7/9] Add the accurate changeset applyer","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2005-11-27T04:24:55Z","receivedAt":"2005-11-27T04:24:55Z","isPatch":true,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 11/24/05, Eric Wong wrote:\n> And make it the default.\n\nCheeky, but right ;-)\n\nWould it be a good idea to read the log entry and decide what kind of\nsmarts do we need to apply the changeset? If the log entry looks\nplain, use process_patchset_fast(), else invoke $TLA?\n\ncheers,\n\n\nmartin\n"},{"id":"12788","messageId":"20051127054333.GA4707@mail.yhbt.net","threadId":"2466","inReplyTo":"46a038f90511262024g4eca7dc2oa42d1f72febf9e27@mail.gmail.com","subject":"Re: [PATCH 7/9] Add the accurate changeset applyer","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-11-27T05:43:33Z","receivedAt":"2005-11-27T05:43:33Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Martin Langhoff wrote:\n> On 11/24/05, Eric Wong wrote:\n> > And make it the default.\n> \n> Cheeky, but right ;-)\n> \n> Would it be a good idea to read the log entry and decide what kind of\n> smarts do we need to apply the changeset? If the log entry looks\n> plain, use process_patchset_fast(), else invoke $TLA?\n\nThis could work. For it to work efficiently, process_patchset_fast()\nshould probably be modified to work on real Arch trees and rsync with\nthe git one. Basically, we can replace the bulk of the tla replay calls\nwith your fast changeset applier. Once the fast mode hits a changeset\nit can't handle, it can do a tla replay on a single changeset instead of\nhaving to do a slow get/apply-delta on an out-of-date tree.\n\nprocess_patchset_fast() must understand how to handle permissions\nchanges, though, as Arch log entries are completely useless for that.\n\nUnfortunately, doing this right and fast probably still requires more\ntime than it's worth. Let's face it, trees with 12k files are extremely\nrare in the Arch world (as are trees constantly reorganized by\nobsessive-compulsives :), but many trees do get a small handful of\ndirectory renames in their lifetime.\n\n-- \nEric Wong\n"},{"id":"13053","messageId":"46a038f90512010902lb11c326p99af9ff99dacf9b4@mail.gmail.com","threadId":"2466","inReplyTo":"20051124075504.GH4789@mail.yhbt.net","subject":"Re: [PATCH 7/9] Add the accurate changeset applyer","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2005-12-01T17:02:53Z","receivedAt":"2005-12-01T17:02:53Z","isPatch":true,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"Eric,\n\nMy test results are a bit of a mixed bag. On one hand, I'm satisfied\nthat both fast and correct imports reach the same tree (minus file\nmodes) for the same commit with the arch repos I imported.\n\nOn the other hand, with my \"moodle\" repo, the 'correct' import seems\nto have stop importing a lot earlier than it should have. I am\nre-running it now to try to continue from where it left off, but it's\nunclear why it abandoned -- I didn't see any error. How widely have\nyou tested this method?\n\ncheers,\n\n\nmartin\n"},{"id":"13132","messageId":"20051203025150.GA13179@mail.yhbt.net","threadId":"2466","inReplyTo":"46a038f90512010902lb11c326p99af9ff99dacf9b4@mail.gmail.com","subject":"Re: [PATCH 7/9] Add the accurate changeset applyer","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2005-12-03T02:51:50Z","receivedAt":"2005-12-03T02:51:50Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Martin Langhoff wrote:\n> Eric,\n> \n> My test results are a bit of a mixed bag. On one hand, I'm satisfied\n> that both fast and correct imports reach the same tree (minus file\n> modes) for the same commit with the arch repos I imported.\n> \n> On the other hand, with my \"moodle\" repo, the 'correct' import seems\n> to have stop importing a lot earlier than it should have. I am\n> re-running it now to try to continue from where it left off, but it's\n> unclear why it abandoned -- I didn't see any error. How widely have\n> you tested this method?\n\nThis was from the moodle repo I archive-mirrored locally a few weeks ago\nfor testing:\n\nget_new: 6\nget_tag: 0\nimport_or_tag: 0\nreplay: 356\n\nRerunning it doesn't seem to pull anymore. IIRC, My previous times\nonly imported around ~150 patchsets. The time it took to run this\nwas certainly longer than the last run (~4 hours here, vs ~2 hours\nI mentioned in <20051124074605.GA4789@mail.yhbt.net>, so there may\nbe a bug somewhere... Unfortunately, I no longer have those old\ntrees around.\n\nI've imported several trees with >1000 revisions without problems,\nmpd-uclinux is among them:\n\nhttp://mpd.bogomips.org/mpd-uclinux.git/\n\n-- \nEric Wong\n"},{"id":"13211","messageId":"46a038f90512051053p66923112l237e4ab80ecb0cca@mail.gmail.com","threadId":"2466","inReplyTo":"20051203025150.GA13179@mail.yhbt.net","subject":"Re: [PATCH 7/9] Add the accurate changeset applyer","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2005-12-05T18:53:19Z","receivedAt":"2005-12-05T18:53:19Z","isPatch":true,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 12/3/05, Eric Wong wrote:\n> Rerunning it doesn't seem to pull anymore. IIRC, My previous times\n> only imported around ~150 patchsets. The time it took to run this\n> was certainly longer than the last run (~4 hours here, vs ~2 hours\n> I mentioned in <20051124074605.GA4789@mail.yhbt.net>, so there may\n> be a bug somewhere... Unfortunately, I no longer have those old\n> trees around.\n>\n> I've imported several trees with >1000 revisions without problems,\n> mpd-uclinux is among them:\n>\n> http://mpd.bogomips.org/mpd-uclinux.git/\n\nHaven't been able to retest an import and have it finished without my\nssh session dropping (should have used gnu screen). I'll be able to\ntest it more thoroughly in a couple of days. Very sorry about the\ndelay.\n\ncheers,\n\n\n\nmartin\n"}]}