{"thread":{"id":"4483","subject":"[PATCH] cvsimport: keep one index per branch during import","startedAt":"2006-06-12T11:50:49Z","lastAt":"2006-06-12T11:50:49Z","messageCount":1,"participants":["Martin Langhoff"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"21644","messageId":"11501130491187-git-send-email-martin@catalyst.net.nz","threadId":"4483","inReplyTo":null,"subject":"[PATCH] cvsimport: keep one index per branch during import","fromName":"Martin Langhoff","fromEmail":"martin@catalyst.net.nz","sentAt":"2006-06-12T11:50:49Z","receivedAt":"2006-06-12T11:50:49Z","isPatch":true,"sender":{"key":"martin@laptop.org","avatar":null},"body":"With this patch we have a speedup and much lower IO when\nimporting trees with many branches. Instead of forcing\nindex re-population for each branch switch, we keep\nmany index files around, one per branch.\n\nSigned-off-by: Martin Langhoff <martin@catalyst.net.nz>\n\n---\n\nThis patch should get some review. It is trivial, but not fully tested.\nI am testing it on the moz repo (which will take a while) to check that I get\nthe same result with and without it. \n\nPerformance-wise, it seems to be doing ~15K commits per hour, with\nthe mozilla repo, up from ~6Kcph on the same hardware. Of course, \nthis is only noticeable in projects with lots of concurrent branches.\nLinear projects don't get much from this patch.\n\nWith this change, we are now truly waiting on cvs to hand over the\nfiles pronto! Running locally, it is apparent that it isn't IO wait\nbut the latency of the chatty cvs protocol that is making this slow.\n\nProbably forking 2 or 3 processes to prefetch filerevs from cvs\nand put them in a queue directory for the main process to pick\nup would work wonders. Actually, they could call git-hash-object\nand just put some file metadata in the queue directory. \n---\n git-cvsimport.perl |   37 ++++++++++++++++++++++++++++++-------\n 1 files changed, 30 insertions(+), 7 deletions(-)\n\ndiff --git a/git-cvsimport.perl b/git-cvsimport.perl\nold mode 100755\nnew mode 100644\nindex 76f6246..9c4588f\n--- a/git-cvsimport.perl\n+++ b/git-cvsimport.perl\n@@ -465,10 +465,15 @@ my $git_dir = $ENV{\"GIT_DIR\"} || \".git\";\n $ENV{\"GIT_DIR\"} = $git_dir;\n my $orig_git_index;\n $orig_git_index = $ENV{GIT_INDEX_FILE} if exists $ENV{GIT_INDEX_FILE};\n-my ($git_ih, $git_index) = tempfile('gitXXXXXX', SUFFIX => '.idx',\n-\t\t\t\t    DIR => File::Spec->tmpdir());\n-close ($git_ih);\n-$ENV{GIT_INDEX_FILE} = $git_index;\n+\n+my %index; # holds filenames of one index per branch\n+{   # init with an index for origin\n+    my ($fh, $fn) = tempfile('gitXXXXXX', SUFFIX => '.idx',\n+\t\t\t     DIR => File::Spec->tmpdir());\n+    close ($fh);\n+    $index{$opt_o} = $fn;\n+}\n+$ENV{GIT_INDEX_FILE} = $index{$opt_o};\n unless(-d $git_dir) {\n \tsystem(\"git-init-db\");\n \tdie \"Cannot init the GIT db at $git_tree: $?\\n\" if $?;\n@@ -496,6 +501,13 @@ unless(-d $git_dir) {\n \t$tip_at_start = `git-rev-parse --verify HEAD`;\n \n \t# populate index\n+\tunless ($index{$last_branch}) {\n+\t    my ($fh, $fn) = tempfile('gitXXXXXX', SUFFIX => '.idx',\n+\t\t\t\t     DIR => File::Spec->tmpdir());\n+\t    close ($fh);\n+\t    $index{$last_branch} = $fn;\n+\t}\n+\t$ENV{GIT_INDEX_FILE} = $index{$last_branch};\n \tsystem('git-read-tree', $last_branch);\n \tdie \"read-tree failed: $?\\n\" if $?;\n \n@@ -776,8 +788,17 @@ while(<CVS>) {\n \t\t}\n \t\tif(($ancestor || $branch) ne $last_branch) {\n \t\t\tprint \"Switching from $last_branch to $branch\\n\" if $opt_v;\n-\t\t\tsystem(\"git-read-tree\", $branch);\n-\t\t\tdie \"read-tree failed: $?\\n\" if $?;\n+\t\t\tunless ($index{$branch}) {\n+\t\t\t    my ($fh, $fn) = tempfile('gitXXXXXX', SUFFIX => '.idx',\n+\t\t\t\t\t\t     DIR => File::Spec->tmpdir());\n+\t\t\t    close ($fh);\n+\t\t\t    $index{$branch} = $fn;\n+\t\t\t    $ENV{GIT_INDEX_FILE} = $index{$branch};\n+\t\t\t    system(\"git-read-tree\", $branch);\n+\t\t\t    die \"read-tree failed: $?\\n\" if $?;\n+\t\t\t} else {\n+\t\t\t    $ENV{GIT_INDEX_FILE} = $index{$branch};\n+\t\t        }\n \t\t}\n \t\t$last_branch = $branch if $branch ne $last_branch;\n \t\t$state = 9;\n@@ -841,7 +862,9 @@ #\tVERSION:1.96->1.96.2.1\n }\n commit() if $branch and $state != 11;\n \n-unlink($git_index);\n+foreach my $git_index (values %index) {\n+    unlink($git_index);\n+}\n \n if (defined $orig_git_index) {\n \t$ENV{GIT_INDEX_FILE} = $orig_git_index;\n-- \n1.4.0.g5fba\n"}]}