{"thread":{"id":"20707","subject":"[PATCH] Add script for importing bits-and-pieces to Git.","startedAt":"2009-08-23T20:11:50Z","lastAt":"2009-08-24T18:37:57Z","messageCount":4,"participants":["Peter Krefting","Thomas Adam","Randal L. Schwartz"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"121575","messageId":"20090823201624.206F52FC20@perkele","threadId":"20707","inReplyTo":null,"subject":"[PATCH] Add script for importing bits-and-pieces to Git.","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-08-23T20:11:50Z","receivedAt":"2009-08-23T20:11:50Z","isPatch":true,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Allows the user to import version history that is stored in bits and\npieces in the file system, for instance snapshots of old development\ntrees, or day-by-day backups. A configuration file is used to\ndescribe the relationship between the different files and allow\ndescribing branches and merges, as well as authorship and commit\nmessages.\n\nOutput is created in a format compatible with git-fast-import.\n\nFull documentation is provided inline in perldoc format.\n\nSigned-off-by: Peter Krefting <peter@softwolves.pp.se>\n---\nI used this to import a couple of projects which had history that\nwas too complex for import-tars or import-zips. The configuration\nfile language for this script allow creation of both branches and\nmerges.\n\n contrib/fast-import/import-directories.perl |  333 +++++++++++++++++++++++++++\n 1 files changed, 333 insertions(+), 0 deletions(-)\n create mode 100755 contrib/fast-import/import-directories.perl\n\ndiff --git a/contrib/fast-import/import-directories.perl b/contrib/fast-import/import-directories.perl\nnew file mode 100755\nindex 0000000..1f5ea07\n--- /dev/null\n+++ b/contrib/fast-import/import-directories.perl\n@@ -0,0 +1,333 @@\n+#!/usr/bin/perl -w\n+#\n+# Copyright 2008-2009 Peter Krefting <peter@softwolves.pp.se>\n+#\n+# ------------------------------------------------------------------------\n+#\n+# This program is free software; you can redistribute it and/or modify\n+# it under the terms of the GNU General Public License as published by\n+# the Free Software Foundation.\n+#\n+# This program is distributed in the hope that it will be useful,\n+# but WITHOUT ANY WARRANTY; without even the implied warranty of\n+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\n+# GNU General Public License for more details.\n+#\n+# You should have received a copy of the GNU General Public License\n+# along with this program; if not, write to the Free Software\n+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.\n+#\n+# ------------------------------------------------------------------------\n+\n+=pod\n+\n+=head1 NAME\n+\n+import-directories - Import bits and pieces to Git.\n+\n+=head1 SYNOPSIS\n+\n+B<import-directories.perl> F<configfile>\n+\n+=head1 DESCRIPTION\n+\n+Script to import arbitrary projects version controlled by the \"copy the\n+source directory to a new location and edit it there\"-version controlled\n+projects into version control. Handles projects with arbitrary branching\n+and version trees, taking a file describing the inputs and generating a\n+file compatible with the L<git-fast-import(1)> format.\n+\n+=head1 CONFIGURATION FILE\n+\n+=head2 Format\n+\n+The configuration file is using a standard I<.ini> format.\n+\n+ ; Comments start with semi-colons\n+ [section]\n+ key=value\n+\n+=head2 Global configuration\n+\n+Global configuration is done in the B<[config]> section, which should be\n+the first section in the file. Configuration can be changed by\n+repeating configuration sections later on.\n+\n+ [config]\n+ ; configure conversion of CRLFs. \"convert\" means that all CRLFs\n+ ; should be converted into LFs (suitable for the core.autocrlf\n+ ; setting set to true in Git). \"none\" means that all data is\n+ ; treated as binary.\n+ crlf=convert\n+\n+=head2 Revision configuration\n+\n+Each revision that is to be imported is described in three\n+sections. Sections should be defined chronologically, so that a\n+revision's parent has always been defined when a new revision\n+is introduced. All sections for one revision should be defined\n+before defining the next revision.\n+\n+Revisions are specified numerically, but they numbers need not be\n+consecutive, only unique.\n+\n+=pod\n+\n+=head3 Revision description section\n+\n+A section whose section name is just an integer gives meta-data\n+about the revision.\n+\n+ [3]\n+ ; author sets the author of the revisions\n+ author=Peter Krefting <peter@softwolves.pp.se>\n+ ; branch sets the branch that the revision should be committed to\n+ branch=master\n+ ; parent describes the revision that is the parent of this commit\n+ ; (optional)\n+ parent=1\n+ ; merges describes a revision that is merged into this commit\n+ ; (optional; can be repeated)\n+ merges=2\n+ ; selects one file to take the timestamp from\n+ ; (optional; if unspecified, the most recent file from the .files\n+ ;  section is used)\n+ timestamp=3/source.c\n+\n+=head3 Revision contents section\n+\n+A section whose section name is an integer followed by B<.files>\n+describes the files included in this revision.\n+\n+ [3.files]\n+ ; the key is the path inside the repository, the value is the path\n+ ; as seen from the importer script.\n+ source.c=3/source.c\n+ source.h=3/source.h\n+\n+=head3 Revision commit message section\n+\n+A section whose section name is an integer followed by B<.message>\n+gives the commit message. This section is read verbatim.\n+\n+ [3.message]\n+ Implement foobar.\n+ ; trailing blank lines are ignored.\n+\n+=cut\n+\n+# Globals\n+use strict;\n+use integer;\n+my $crlfmode = 0;\n+my @revs;\n+my (%revmap, %message, %files, %author, %branch, %parent, %merges, %time, %timesource);\n+my $sectiontype = 0;\n+my $rev = 0;\n+my $mark = 1;\n+\n+# Check command line\n+if ($#ARGV == -1 || $ARGV[0] =~ /^--?h/)\n+{\n+    exec('perldoc', $0);\n+    exit 1;\n+}\n+\n+# Open configuration\n+my $config = $ARGV[0];\n+open CFG, '<', $config or die \"Cannot open configuration file \\\"$config\\\": \";\n+\n+# Open output\n+my $output = $ARGV[1];\n+open OUT, '>', $output or die \"Cannot create output file \\\"$output\\\": \";\n+binmode OUT;\n+\n+LINE: while (my $line = <CFG>)\n+{\n+\t$line =~ s/\\r?\\n$//;\n+\tnext LINE if $sectiontype != 4 && $line eq '';\n+\tnext LINE if $line =~ /^;/;\n+\tmy $oldsectiontype = $sectiontype;\n+\tmy $oldrev = $rev;\n+\n+\t# Sections\n+\tif ($line =~ m\"^\\[(config|(\\d+)(|\\.files|\\.message))\\]$\")\n+\t{\n+\t\tif ($1 eq 'config')\n+\t\t{\n+\t\t\t$sectiontype = 1;\n+\t\t}\n+\t\telsif ($3 eq '')\n+\t\t{\n+\t\t\t$sectiontype = 2;\n+\t\t\t$rev = $2;\n+\t\t\t# Create a new revision\n+\t\t\tdie \"Duplicate rev: $line\\n \" if defined $revmap{$rev};\n+\t\t\tprint \"Reading revision $rev\\n\";\n+\t\t\tpush @revs, $rev;\n+\t\t\t$revmap{$rev} = $mark ++;\n+\t\t\t$time{$revmap{$rev}} = 0;\n+\t\t}\n+\t\telsif ($3 eq '.files')\n+\t\t{\n+\t\t\t$sectiontype = 3;\n+\t\t\t$rev = $2;\n+\t\t\tdie \"Revision mismatch: $line\\n \" unless $rev == $oldrev;\n+\t\t}\n+\t\telsif ($3 eq '.message')\n+\t\t{\n+\t\t\t$sectiontype = 4;\n+\t\t\t$rev = $2;\n+\t\t\tdie \"Revision mismatch: $line\\n \" unless $rev == $oldrev;\n+\t\t}\n+\t\telse\n+\t\t{\n+\t\t\tdie \"Internal parse error: $line\\n \";\n+\t\t}\n+\t\tnext LINE;\n+\t}\n+\n+\t# Parse data\n+\tif ($sectiontype != 4)\n+\t{\n+\t\t# Key and value\n+\t\tif ($line =~ m\"^(.*)=(.*)$\")\n+\t\t{\n+\t\t\tmy ($key, $value) = ($1, $2);\n+\t\t\t# Global configuration\n+\t\t\tif (1 == $sectiontype)\n+\t\t\t{\n+\t\t\t\tif ($key eq 'crlf')\n+\t\t\t\t{\n+\t\t\t\t\t$crlfmode = 1, next LINE if $value eq 'convert';\n+\t\t\t\t\t$crlfmode = 0, next LINE if $value eq 'none';\n+\t\t\t\t}\n+\t\t\t\tdie \"Unknown configuration option: $line\\n \";\n+\t\t\t}\n+\t\t\t# Revision specification\n+\t\t\tif (2 == $sectiontype)\n+\t\t\t{\n+\t\t\t\tmy $current = $revmap{$rev};\n+\t\t\t\t$author{$current} = $value, next LINE if $key eq 'author';\n+\t\t\t\t$branch{$current} = $value, next LINE if $key eq 'branch';\n+\t\t\t\t$parent{$current} = $value, next LINE if $key eq 'parent';\n+\t\t\t\t$timesource{$current} = $value, next LINE if $key eq 'timestamp';\n+\t\t\t\tpush(@{$merges{$current}}, $value), next LINE if $key eq 'merges';\n+\t\t\t\tdie \"Unknown revision option: $line\\n \";\n+\t\t\t}\n+\t\t\t# Filespecs\n+\t\t\tif (3 == $sectiontype)\n+\t\t\t{\n+\t\t\t\t# Add the file and create a marker\n+\t\t\t\tdie \"File not found: $line\\n \" unless -f $value;\n+\t\t\t\tmy $current = $revmap{$rev};\n+\t\t\t\t${$files{$current}}{$key} = $mark;\n+\t\t\t\tmy $time = &fileblob($value, $crlfmode, $mark ++);\n+\n+\t\t\t\t# Update revision timestamp if more recent than other\n+\t\t\t\t# files seen, or if this is the file we have selected\n+\t\t\t\t# to take the time stamp from using the \"timestamp\"\n+\t\t\t\t# directive.\n+\t\t\t\tif ((defined $timesource{$current} && $timesource{$current} eq $value)\n+\t\t\t\t    || $time > $time{$current})\n+\t\t\t\t{\n+\t\t\t\t\t$time{$current} = $time;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t}\n+\t\telse\n+\t\t{\n+\t\t\tdie \"Parse error: $line\\n \";\n+\t\t}\n+\t}\n+\telse\n+\t{\n+\t\t# Commit message\n+\t\tmy $current = $revmap{$rev};\n+\t\tif (defined $message{$current})\n+\t\t{\n+\t\t\t$message{$current} .= \"\\n\";\n+\t\t}\n+\t\t$message{$current} .= $line;\n+\t}\n+}\n+close CFG;\n+\n+# Start spewing out data for git-fast-import\n+foreach my $commit (@revs)\n+{\n+\t# Progress\n+\tprint OUT \"progress Creating revision $commit\\n\";\n+\n+\t# Create commit header\n+\tmy $mark = $revmap{$commit};\n+\n+\t# Branch and commit id\n+\tprint OUT \"commit refs/heads/\", $branch{$mark}, \"\\nmark :\", $mark, \"\\n\";\n+\n+\t# Author and timestamp\n+\tdie \"No timestamp defined for $commit (no files?)\\n\" unless defined $time{$mark};\n+\tprint OUT \"committer \", $author{$mark}, \" \", $time{$mark}, \" +0100\\n\";\n+\n+\t# Commit message\n+\tdie \"No message defined for $commit\\n\" unless defined $message{$mark};\n+\tmy $message = $message{$mark};\n+\t$message =~ s/\\n$//; # Kill trailing empty line\n+\tprint OUT \"data \", length($message), \"\\n\", $message, \"\\n\";\n+\n+\t# Parent and any merges\n+\tprint OUT \"from :\", $revmap{$parent{$mark}}, \"\\n\" if defined $parent{$mark};\n+\tif (defined $merges{$mark})\n+\t{\n+\t\tforeach my $merge (@{$merges{$mark}})\n+\t\t{\n+\t\t\tprint OUT \"merge :\", $revmap{$merge}, \"\\n\";\n+\t\t}\n+\t}\n+\n+\t# Output file marks\n+\tprint OUT \"deleteall\\n\"; # start from scratch\n+\tforeach my $file (sort keys %{$files{$mark}})\n+\t{\n+\t\tprint OUT \"M 644 :\", ${$files{$mark}}{$file}, \" $file\\n\";\n+\t}\n+\tprint OUT \"\\n\";\n+}\n+\n+# Create one file blob\n+sub fileblob\n+{\n+\tmy ($filename, $crlfmode, $mark) = @_;\n+\n+\t# Import the file\n+\tprint OUT \"progress Importing $filename\\nblob\\nmark :$mark\\n\";\n+\topen FILE, '<', $filename or die \"Cannot read $filename\\n \";\n+\tbinmode FILE;\n+\tmy (undef, undef, undef, undef, undef, undef,\n+\t\tundef, $size, undef, $time, undef, undef, undef) = stat FILE;\n+\tmy $file;\n+\tread FILE, $file, $size;\n+\tclose FILE;\n+\t$file =~ s/\\r\\n/\\n/g if $crlfmode;\n+\tprint OUT \"data \", length($file), \"\\n\", $file, \"\\n\";\n+\n+\treturn $time;\n+}\n+\n+__END__\n+\n+=pod\n+\n+=head1 EXAMPLES\n+\n+B<import-directories.perl> F<project.import>\n+\n+=head1 AUTHOR\n+\n+Copyright 2008-2009 Peter Krefting E<lt>peter@softwolves.pp.se>\n+\n+This program is free software; you can redistribute it and/or modify\n+it under the terms of the GNU General Public License as published by\n+the Free Software Foundation.\n+\n+=cut\n-- \n1.6.3.3\n"},{"id":"121578","messageId":"18071eea0908231328r58b0e22ap20e664d2904420a0@mail.gmail.com","threadId":"20707","inReplyTo":"20090823201624.206F52FC20@perkele","subject":"Re: [PATCH] Add script for importing bits-and-pieces to Git.","fromName":"Thomas Adam","fromEmail":"thomas.adam22@gmail.com","sentAt":"2009-08-23T20:28:23Z","receivedAt":"2009-08-23T20:28:23Z","isPatch":true,"sender":{"key":"thomas.adam22@gmail.com","avatar":"https://gravatar.com/avatar/137f9858bc6bfd5b2f743aefd988c81ce0cbd306248889df80e269519cfc8741?d=mp&s=160"},"body":"2009/8/23 Peter Krefting <peter@softwolves.pp.se>:\n> +       my (undef, undef, undef, undef, undef, undef,\n> +               undef, $size, undef, $time, undef, undef, undef) = stat FILE;\n\nMinor nit.  Better written as:\n\nmy ($size, $mtime) = (stat(FILE))[7,9];\n\n-- Thomas Adam\n"},{"id":"121651","messageId":"alpine.DEB.2.00.0908241909420.16576@perkele.intern.softwolves.pp.se","threadId":"20707","inReplyTo":"18071eea0908231328r58b0e22ap20e664d2904420a0@mail.gmail.com","subject":"Re: [PATCH] Add script for importing bits-and-pieces to Git.","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-08-24T17:10:16Z","receivedAt":"2009-08-24T17:10:16Z","isPatch":true,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Thomas Adam:\n\n> Minor nit.  Better written as:\n> my ($size, $mtime) = (stat(FILE))[7,9];\n\nAh. Another obscure Perl syntax to consider :-) I'll send an updated patch. \nThanks.\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"121659","messageId":"86k50tt5ve.fsf@blue.stonehenge.com","threadId":"20707","inReplyTo":"alpine.DEB.2.00.0908241909420.16576@perkele.intern.softwolves.pp.se","subject":"Re: [PATCH] Add script for importing bits-and-pieces to Git.","fromName":"Randal L. Schwartz","fromEmail":"merlyn@stonehenge.com","sentAt":"2009-08-24T18:37:57Z","receivedAt":"2009-08-24T18:37:57Z","isPatch":true,"sender":{"key":"merlyn@stonehenge.com","avatar":"https://gravatar.com/avatar/dc528d210743ff0333e6213f9ee7b33b23f1b7bc1f3c5a8c2d819074ecd7ab19?d=mp&s=160"},"body":">>>>> \"Peter\" == Peter Krefting <peter@softwolves.pp.se> writes:\n\nPeter> Ah. Another obscure Perl syntax to consider :-) I'll send an updated\nPeter> patch. Thanks.\n\nIt's fair game to say it's not \"obscure\" if it's in Learning Perl,\nand that was in Learning Perl.\n\nMinor trivia note: that's one of the three Perl syntax features\nthat I can take credit for having created, back in the day.\n\n-- \nRandal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095\n<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>\nSmalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.\nSee http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion\n"}]}