Re: gitpacker progress report and a question

From

Felipe Contreras <felipe.contreras@gmail.com>

Date

Nov 26, 2012, 20:07 UTC

Message-ID

<CAMP44s30px2FgieC9VcGji7T+yWFo7gMSwQhtuztHrqto8B7Aw@mail.gmail.com>

In-Reply-To

<20121115212818.GA21558@thyrsus.com>

On Thu, Nov 15, 2012 at 10:28 PM, Eric S. Raymond <esr@thyrsus.com> wrote:

> Some days ago I reported that I was attempting to write a tool that could
> (a) take a git repo and unpack it into a tarball sequence plus a metadata log,
> (b) reverse that operation, packing a tarball and log sequence into a repo.
>
> Thanks in part to advice by Andreas Schwab and in part to looking at the
> text of the p4 import script, this effort has succeeded.  A proof of
> concept is enclosed.  It isn't documented yet, and has not been tested
> on a repository with branches or merges in the history, but I am confident
> that the distance from here to a finished and tested tool is short.
>
> The immediate intended use is for importing older projects that are
> available only as sequences of release tarballs, but there are other
> sorts of repository surgery that would become easier using it.
>
> I'm still looking for a better name for it and would welcome suggestions.
>
> Before I do much further work, I need to determine how this will be shipped.
> I see two possibilities: either I ship it as a small standalone project,
> or it becomes a git subcommand shipped with the git suite. How I document
> it and set up its tests would differ between these two cases.

1) I tried it, and it doesn't seem to import (pack?) are repository
with sub-directories in it

2) Using 'git fast-import' is probably simpler, and more efficient

Cheers.

>From eb3c34699d7f5d4eec4f088344659b8d9b6a07ea Mon Sep 17 00:00:00 2001

From: Felipe Contreras <felipe.contreras@gmail.com>
Date: Mon, 26 Nov 2012 20:48:38 +0100
Subject: [PATCH] Add new git-weave tool

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---
 contrib/weave/git-weave | 166 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 166 insertions(+)
 create mode 100755 contrib/weave/git-weave

diff --git a/contrib/weave/git-weave b/contrib/weave/git-weave
new file mode 100755
index 0000000..3106121
--- /dev/null
+++ b/contrib/weave/git-weave
@@ -0,0 +1,166 @@
+#!/usr/bin/env ruby
+
+require 'optparse'
+require 'find'
+require 'fileutils'
+
+def export(indir = '.', out = STDOUT)
+  open(File.join(indir, 'log')).each("\n.\n") do |data|
+
+    @msg = nil
+    @parents = []
+
+    data.chomp(".\n").each_line do |l|
+      if not @msg
+        case l
+        when /^commit (.+)$/
+          @id = $1
+        when /^author (.+)$/
+          @author = $1
+        when /^committer (.+)$/
+          @committer = $1
+        when /^parent (.+)$/
+          @parents << $1
+        when /^$/
+          @msg = ""
+        end
+      else
+        @msg << l
+      end
+    end
+
+    out.puts "commit refs/heads/master"
+    out.puts "mark :#{@id}"
+    out.puts "author #{@author}"
+    out.puts "committer #{@committer}"
+    out.puts "data #{@msg.bytesize}"
+    out.puts @msg
+
+    @parents.each_with_index do |p, i|
+      if i == 0
+        out.puts "from :%u" % p
+      else
+        out.puts "merge :%u" % p
+      end
+    end
+
+    # files
+    out.puts 'deleteall'
+    FileUtils.cd(File.join(indir, @id)) do
+      Find.find('.') do |e|
+        next unless File.file?(e)
+        content = File.read(e)
+        filename = e.split(File::SEPARATOR).slice(1..-1).join(File::SEPARATOR)
+        mode = File.executable?(e) ? '100755' : '100644'
+        if File.symlink?(e)
+          mode = '120000'
+          content = File.readlink(e)
+        end
+        out.puts 'M %s inline %s' % [mode, filename]
+        out.puts "data #{content.bytesize}"
+        out.puts content
+      end
+    end
+
+  end
+end
+
+def import(outdir, out)
+  format = 'format:commit %H%nauthor %an <%ae> %ad%ncommitter %cn
<%ce> %cd%nparents %P%n%n%B'
+  cmd = ['git', 'log', '-z', '-s', '--date=raw', '--format=%s' %
format, '--all', '--reverse']
+  commits = {}
+
+  IO.popen(cmd).each_with_index("\0") do |data, i|
+    @msg = nil
+    @parents = []
+    data.chomp("\0").each_line do |l|
+      if not @msg
+        case l
+        when /^commit (.+)$/
+          @id = $1
+        when /^author (.+)$/
+          @author = $1
+        when /^committer (.+)$/
+          @committer = $1
+        when /^parents (.+)$/
+          @parents = $1.split(" ")
+        when /^$/
+          @msg = ""
+        end
+      else
+        @msg << l
+      end
+    end
+
+    num = i + 1
+    commits[@id] = num
+
+    out.puts "commit #{num}"
+    @parents.each do |p|
+      out.puts "parent #{commits[p]}"
+    end
+    out.puts "author #{@author}"
+    out.puts "committer #{@committer}"
+    out.puts
+    out.puts @msg.gsub(/\n\n+/, "\n") # why?
+    out.puts "."
+
+    wd = File.join(outdir, num.to_s)
+    FileUtils.mkdir_p(wd)
+    system('git', '--work-tree', wd, 'checkout', '-f', '-q', @id)
+  end
+end
+
+def git_pack(indir, outdir)
+  indir = File.absolute_path(indir)
+  system('git', 'init', outdir)
+  FileUtils.cd(outdir) do
+    IO.popen(['git', 'fast-import'], 'w') do |io|
+      export(indir, io)
+    end
+    system('git', 'reset', '--hard')
+  end
+end
+
+def git_unpack(indir, outdir)
+  begin
+    FileUtils.mkdir_p(outdir)
+    log = File.open(File.join(outdir, 'log'), 'w')
+    ENV['GIT_DIR'] = File.join(indir, '.git')
+    import(outdir, log)
+  ensure
+    system('git', 'symbolic-ref', 'HEAD', 'refs/heads/master')
+    ENV.delete('GIT_DIR')
+    log.close if log
+  end
+end
+
+$indir = '.'
+
+begin
+  OptionParser.new do |opts|
+    opts.on('-x') do
+      $mode = 'unpack'
+    end
+    opts.on('-c') do
+      $mode = 'pack'
+    end
+    opts.on('-o', '--outdir DIR') do |v|
+      $outdir = v
+    end
+    opts.on('-i', '--indir DIR') do |v|
+      $indir = v
+    end
+  end.parse!
+rescue OptionParser::InvalidOption
+end
+
+$mode = File.exists?(File.join($indir, '.git')) ? 'unpack' : 'pack'
unless $mode
+$outdir = File.join($indir, $mode == 'pack' ? 'packed' : 'unpacked2')
unless $outdir
+
+case $mode
+when 'pack'
+  git_pack($indir, $outdir)
+when 'unpack'
+  git_unpack($indir, $outdir)
+end

-- 
1.8.0

-- 
Felipe Contreras