{"thread":{"id":"32124","subject":"gitpacker progress report and a question","startedAt":"2012-11-15T21:28:18Z","lastAt":"2012-11-27T08:51:01Z","messageCount":16,"participants":["Eric S. Raymond","Max Horn","Andreas Schwab","Felipe Contreras"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"203314","messageId":"20121115212818.GA21558@thyrsus.com","threadId":"32124","inReplyTo":null,"subject":"gitpacker progress report and a question","fromName":"Eric S. Raymond","fromEmail":"esr@thyrsus.com","sentAt":"2012-11-15T21:28:18Z","receivedAt":"2012-11-15T21:28:18Z","isPatch":false,"sender":{"key":"esr@thyrsus.com","avatar":"https://avatars.githubusercontent.com/u/727961?v=4"},"body":"Some days ago I reported that I was attempting to write a tool that could\n(a) take a git repo and unpack it into a tarball sequence plus a metadata log,\n(b) reverse that operation, packing a tarball and log sequence into a repo.\n\nThanks in part to advice by Andreas Schwab and in part to looking at the\ntext of the p4 import script, this effort has succeeded.  A proof of\nconcept is enclosed.  It isn't documented yet, and has not been tested\non a repository with branches or merges in the history, but I am confident\nthat the distance from here to a finished and tested tool is short. \n\nThe immediate intended use is for importing older projects that are\navailable only as sequences of release tarballs, but there are other\nsorts of repository surgery that would become easier using it.\n\nI'm still looking for a better name for it and would welcome suggestions.\n\nBefore I do much further work, I need to determine how this will be shipped.\nI see two possibilities: either I ship it as a small standalone project,\nor it becomes a git subcommand shipped with the git suite. How I document \nit and set up its tests would differ between these two cases.\n\nIs there a process for submitting new subcommands?  What are the \ntest-suite and documentation requirements?\n-- \n\t\t<a href=\"http://www.catb.org/~esr/\">Eric S. Raymond</a>\n\n\n#!/usr/bin/env python\n\"\"\"\ngitpacker - assemble tree sequences into repository histories\n\nRequires git and cpio.\n\"\"\"\nimport sys, os, getopt, subprocess, time, tempfile\n\nDEBUG_GENERAL  = 1\nDEBUG_PROGRESS = 2\nDEBUG_COMMANDS = 3\n\nclass Fatal(Exception):\n    \"Unrecoverable error.\"\n    def __init__(self, msg):\n        Exception.__init__(self)\n        self.msg = msg\n\nclass Baton:\n    \"Ship progress indications to stdout.\"\n    def __init__(self, prompt, endmsg='done', enable=False):\n        self.prompt = prompt\n        self.endmsg = endmsg\n        self.countfmt = None\n        self.counter = 0\n        if enable:\n            self.stream = sys.stdout\n        else:\n            self.stream = None\n        self.count = 0\n        self.time = 0\n    def __enter__(self):\n        if self.stream:\n            self.stream.write(self.prompt + \"...\")\n            if os.isatty(self.stream.fileno()):\n                self.stream.write(\" \\010\")\n            self.stream.flush()\n        self.count = 0\n        self.time = time.time()\n        return self\n    def startcounter(self, countfmt, initial=1):\n        self.countfmt = countfmt\n        self.counter = initial\n    def bumpcounter(self):\n        if self.stream is None:\n            return\n        if os.isatty(self.stream.fileno()):\n            if self.countfmt:\n                update = self.countfmt % self.counter\n                self.stream.write(update + (\"\\010\" * len(update)))\n                self.stream.flush()\n            else:\n                self.twirl()\n        self.counter = self.counter + 1\n    def endcounter(self):\n        if self.stream:\n            w = len(self.countfmt % self.count)\n            self.stream.write((\" \" * w) + (\"\\010\" * w))\n            self.stream.flush()\n        self.countfmt = None\n    def twirl(self, ch=None):\n        \"One twirl of the baton.\"\n        if self.stream is None:\n            return\n        if os.isatty(self.stream.fileno()):\n            if ch:\n                self.stream.write(ch)\n                self.stream.flush()\n                return\n            else:\n                update = \"-/|\\\\\"[self.count % 4]\n                self.stream.write(update + (\"\\010\" * len(update)))\n                self.stream.flush()\n        self.count = self.count + 1\n    def __exit__(self, extype, value_unused, traceback_unused):\n        if extype == KeyboardInterrupt:\n            self.endmsg = \"interrupted\"\n        if extype == Fatal:\n            self.endmsg = \"aborted by error\"\n        if self.stream:\n            self.stream.write(\"...(%2.2f sec) %s.\\n\" \\\n                              % (time.time() - self.time, self.endmsg))\n        return False\n\ndef do_or_die(dcmd, legend=\"\"):\n    \"Either execute a command or raise a fatal exception.\"\n    if legend:\n        legend = \" \"  + legend\n    if verbose >= DEBUG_COMMANDS:\n        sys.stdout.write(\"executing '%s'%s\\n\" % (dcmd, legend))\n    try:\n        retcode = subprocess.call(dcmd, shell=True)\n        if retcode < 0:\n            raise Fatal(\"child was terminated by signal %d.\" % -retcode)\n        elif retcode != 0:\n            raise Fatal(\"child returned %d.\" % retcode)\n    except (OSError, IOError) as e:\n        raise Fatal(\"execution of %s%s failed: %s\" % (dcmd, legend, e))\n\ndef capture_or_die(dcmd, legend=\"\"):\n    \"Either execute a command and capture its output or die.\"\n    if legend:\n        legend = \" \"  + legend\n    if verbose >= DEBUG_COMMANDS:\n        sys.stdout.write(\"executing '%s'%s\\n\" % (dcmd, legend))\n    try:\n        return subprocess.check_output(dcmd, shell=True)\n    except subprocess.CalledProcessError as e:\n        if e.returncode < 0:\n            raise Fatal(\"child was terminated by signal %d.\" % -e.returncode)\n        elif e.returncode != 0:\n            sys.stderr.write(\"gitpacker: child returned %d.\" % e.returncode)\n        sys.exit(1)\n    \ndef git_pack(indir, outdir, quiet=False):\n    \"Pack a tree sequence and associated logfile into a repository\"\n    do_or_die(\"mkdir %s; git init -q %s\" % (outdir, outdir))\n    logfile = os.path.join(indir, \"log\")\n    commit_id = [None]\n    state = 0\n    parents = []\n    comment = committername = authorname = \"\"\n    commitdate = authordate = commitemail = authoremail = \"\"\n    commitcount = 1;\n    linecount = 0\n    with Baton(\"Packing\", enable=not quiet) as baton:\n        for line in open(logfile):\n            if verbose > DEBUG_PROGRESS:\n                print \"Looking at: '%s'\" % repr(line)\n            if state == 0:\n                if line == '\\n':\n                    state = 1\n                else:\n                    try:\n                        space = line.index(' ')\n                        leader = line[:space]\n                        follower = line[space:].strip()\n                        if leader == \"commit\":\n                            commit = follower\n                        elif leader == \"parent\":\n                            parents.append(follower)\n                        elif leader not in (\"author\", \"committer\"):\n                            raise Fatal(\"unexpected log attribute at %s\" \\\n                                        % repr(line))\n                        elif leader == \"committer\":\n                            (committername, committeremail, committerdate) = [x.strip() for x in follower.replace('>','<').split('<')]\n                        elif leader == \"author\":\n                            (authorname, authoremail, authordate) = [x.strip() for x in follower.replace('>','<').split('<')]\n                    except ValueError:\n                        raise Fatal('\"%s\", line %d: ill-formed log entry' % (logfile, linecount))\n            elif state == 1:\n                if line == \".\\n\":\n                    if verbose > DEBUG_PROGRESS:\n                        print \"Interpretation begins\"\n                    os.chdir(outdir)\n                    if commitcount > 1:\n                        do_or_die(\"rm `git ls-tree --name-only HEAD`\")\n                    if verbose > DEBUG_PROGRESS:\n                        print \"Copying\"\n                    os.chdir(\"%s/%d\" % (indir, commitcount))\n                    do_or_die(\"find . -print | cpio -pd --quiet %s\" % (outdir,))\n                    os.chdir(outdir)\n                    do_or_die(\"git add -A\") \n                    tree_id = capture_or_die(\"git write-tree\").strip()\n                    if verbose > DEBUG_PROGRESS:\n                        print \"Tree ID is\", tree_id\n                    (_, commentfile) = tempfile.mkstemp()\n                    with open(commentfile, \"w\") as cfp:\n                        cfp.write(comment)\n                    command = \"git commit-tree %s \" % tree_id\n                    command += \" \".join(map(lambda p: \"-p \" + commit_id[int(p)],parents))\n                    command += \"<'%s'\" % commentfile\n                    environment = \"\"\n                    environment += \" GIT_AUTHOR_NAME='%s' \" % authorname\n                    environment += \" GIT_AUTHOR_EMAIL='%s' \" % authoremail \n                    environment += \" GIT_AUTHOR_DATE='%s' \" % authordate \n                    environment += \" GIT_COMMITTER_NAME='%s' \" % committername\n                    environment += \" GIT_COMMITTER_EMAIL='%s' \" % committeremail \n                    environment += \" GIT_COMMITTER_DATE='%s' \" % committerdate \n                    commit_id.append(capture_or_die(environment + command).strip())\n                    do_or_die(\"git update-ref HEAD %s\" % commit_id[-1])\n                    os.remove(commentfile)\n                    state = 0\n                    parents = []\n                    comment = committername = authorname = \"\"\n                    committerdate = authordate = committeremail = authoremail = \"\"\n                    commitcount += 1\n                    baton.twirl()\n                    if maxcommit != 0 and commitcount >= maxcommit:\n                        break\n                else:\n                    if line.startswith(\".\"):\n                        line = line[1:]\n                    comment += line\n\ndef git_unpack(indir, outdir, quiet=False):\n    \"Unpack a repository into a tree sequence and associated logfile.\"\n    rawlogfile = os.path.join(outdir, \"rawlog\")\n    with Baton(\"Unpacking\", enable=not quiet) as baton:\n        do_or_die(\"rm -fr %s; mkdir %s\" % (outdir, outdir))\n        baton.twirl()\n        do_or_die(\"cd %s; git log --all --reverse --format=raw >%s\" % (indir, rawlogfile))\n        baton.twirl()\n        commitcount = 1\n        commit_map = {}\n        os.chdir(indir)\n        try:\n            for line in open(rawlogfile):\n                baton.twirl()\n                if line.startswith(\"commit \"):\n                    commit = line.split()[1]\n                    commit_map[commit] = commitcount\n                    do_or_die(\"git checkout %s 2>/dev/null; mkdir %s/%d\" \\\n                              % (commit, outdir, commitcount))\n                    do_or_die(\"git ls-tree -r --name-only --full-tree %s | cpio -pd --quiet %s/%d\"\n                              % (commit, outdir, commitcount))\n                    commitcount += 1\n\n        finally:\n            do_or_die(\"git reset --hard >/dev/null; git checkout master >/dev/null 2>&1\")\n        cooked = os.path.join(outdir, \"log\")\n        body_latch = False\n        try:\n            with open(cooked, \"w\") as wfp:\n                linecount = 0\n                for line in open(rawlogfile):\n                    linecount += 1\n                    if line[0].isspace():\n                        if line.startswith(\" \" * 4):\n                            line = line[4:]\n                            # Old-school byte stuffing.\n                            if line.startswith(\".\"):\n                                line = \".\" + line\n                    else:\n                        space = line.index(' ')\n                        leader = line[:space]\n                        follower = line[space:].strip()\n                        if leader == \"tree\":\n                            continue\n                        if leader == \"commit\" and linecount > 1:\n                            wfp.write(\".\\n\")\n                        # FIXME: Check that log raw emits one parent per line\n                        if leader in (\"commit\", \"parent\"):\n                            line = \"%s %s\\n\" % (leader, commit_map[follower])\n                            body_latch = False\n                        elif leader not in (\"author\", \"committer\"):\n                            raise Fatal(\"unexpected log attribute at %s\" \\\n                                        % repr(line))\n                    if line == '\\n':\n                        if not body_latch:\n                            body_latch = True\n                        else:\n                            continue\n                    wfp.write(line)\n                wfp.write(\".\\n\")\n        except (ValueError, IndexError, KeyError):\n            raise Fatal(\"log rewrite failed on %s\" % repr(line))\n    os.remove(rawlogfile)\n\nif __name__ == '__main__':\n    (options, arguments) = getopt.getopt(sys.argv[1:], \"ci:m:o:qxv\")\n    mode = 'auto'\n    indir = '.'\n    outdir = None\n    quiet = False\n    maxcommit = 0\n    verbose = 0\n    for (opt, val) in options:\n        if opt == '-x':\n            mode = 'unpack'\n        elif opt == '-c':\n            mode = 'pack'\n        elif opt == '-m':\n            indir = int(val)\n        elif opt == '-i':\n            indir = val\n        elif opt == '-o':\n            outdir = val\n        elif opt == '-q':\n            quiet = True\n        elif opt == '-v':\n            verbose += 1\n    if not os.path.exists(indir):\n        sys.stderr.write(\"gitpacker: input directory %s must exist.\\n\" % indir)\n        sys.exit(1)\n    if mode == 'auto':\n        if os.path.exists(os.path.join(indir, \".git\")):\n            mode = 'unpack'\n        else:\n            mode = 'pack'\n    assert mode == 'pack' or mode == 'unpack'\n    if outdir is None:\n        if mode == 'pack':\n            outdir = indir + \"/packed\"\n        elif mode == 'unpack':\n            outdir = indir + \"/unpacked\"\n    if os.path.exists(outdir):\n        sys.stderr.write(\"gitpacker: output directory %s must not exist.\\n\" % outdir)\n        sys.exit(1)\n    indir = os.path.abspath(indir)\n    outdir = os.path.abspath(outdir)\n    if verbose >= DEBUG_PROGRESS:\n        sys.stderr.write(\"gitpacker: %s from %s to %s.\\n\" % (mode, indir, outdir))\n    try:\n        try:\n            here = os.getcwd()\n            if mode == 'pack':\n                git_pack(indir, outdir, quiet=quiet)\n            elif mode == 'unpack':\n                git_unpack(indir, outdir, quiet=quiet)\n        finally:\n            os.chdir(here)\n    except Fatal, e:\n        sys.stderr.write(e.msg + \"\\n\")\n        sys.exit(1)\n    except KeyboardInterrupt:\n        pass\n\n# end\n\n"},{"id":"203318","messageId":"877E14DE-313D-4863-980B-13D077230F7F@quendi.de","threadId":"32124","inReplyTo":"20121115212818.GA21558@thyrsus.com","subject":"Re: gitpacker progress report and a question","fromName":"Max Horn","fromEmail":"postbox@quendi.de","sentAt":"2012-11-15T22:35:10Z","receivedAt":"2012-11-15T22:35:10Z","isPatch":false,"sender":{"key":"postbox@quendi.de","avatar":null},"body":"\nOn 15.11.2012, at 22:28, Eric S. Raymond wrote:\n\n> Some days ago I reported that I was attempting to write a tool that could\n> (a) take a git repo and unpack it into a tarball sequence plus a metadata log,\n> (b) reverse that operation, packing a tarball and log sequence into a repo.\n\nAh, I could have used such a tool a year or so ago. Sounds useful to me, anyway :)\n\n> \n> Thanks in part to advice by Andreas Schwab and in part to looking at the\n> text of the p4 import script, this effort has succeeded.  A proof of\n> concept is enclosed.  It isn't documented yet, and has not been tested\n> on a repository with branches or merges in the history, but I am confident\n> that the distance from here to a finished and tested tool is short. \n> \n> The immediate intended use is for importing older projects that are\n> available only as sequences of release tarballs, but there are other\n> sorts of repository surgery that would become easier using it.\n> \n> I'm still looking for a better name for it and would welcome suggestions.\n\nIsn't \"gitar\" the kind of natural choice? ;) At least for a stand-alone tool, not for a git subcommand.\n\n\nCheers,\nMax\n\n> \n> Before I do much further work, I need to determine how this will be shipped.\n> I see two possibilities: either I ship it as a small standalone project,\n> or it becomes a git subcommand shipped with the git suite. How I document \n> it and set up its tests would differ between these two cases.\n> \n> Is there a process for submitting new subcommands?  What are the \n> test-suite and documentation requirements?\n> -- \n> \t\t<a href=\"http://www.catb.org/~esr/\">Eric S. Raymond</a>\n> <gitpacker.txt>\n"},{"id":"203319","messageId":"20121115230535.GB25000@thyrsus.com","threadId":"32124","inReplyTo":"877E14DE-313D-4863-980B-13D077230F7F@quendi.de","subject":"Re: gitpacker progress report and a question","fromName":"Eric S. Raymond","fromEmail":"esr@thyrsus.com","sentAt":"2012-11-15T23:05:35Z","receivedAt":"2012-11-15T23:05:35Z","isPatch":false,"sender":{"key":"esr@thyrsus.com","avatar":"https://avatars.githubusercontent.com/u/727961?v=4"},"body":"Max Horn <postbox@quendi.de>:\n> > I'm still looking for a better name for it and would welcome suggestions.\n> \n> Isn't \"gitar\" the kind of natural choice? ;) At least for a stand-alone tool, not for a git subcommand.\n\nI just renamed it git-weave.  I keep talking about tarballs because I keep\nthinking about using it archeologically on projects that only exist as\ntarball sequences, but the tool actually oacks and unpacks *file tree*\nsequences.\n-- \n\t\t<a href=\"http://www.catb.org/~esr/\">Eric S. Raymond</a>\n"},{"id":"203338","messageId":"m2d2zdvgfk.fsf@igel.home","threadId":"32124","inReplyTo":"20121115212818.GA21558@thyrsus.com","subject":"Re: gitpacker progress report and a question","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2012-11-16T13:13:51Z","receivedAt":"2012-11-16T13:13:51Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"\"Eric S. Raymond\" <esr@thyrsus.com> writes:\n\n>                     if commitcount > 1:\n>                         do_or_die(\"rm `git ls-tree --name-only HEAD`\")\n\nThis will fail on file names containing whitespace or glob meta\ncharacters.  Better use \"git rm -qr .\" here.  You don't have to care\nabout the index since you are doing \"git add -A\" later anyway.\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5\n\"And now for something completely different.\"\n"},{"id":"203913","messageId":"CAMP44s30px2FgieC9VcGji7T+yWFo7gMSwQhtuztHrqto8B7Aw@mail.gmail.com","threadId":"32124","inReplyTo":"20121115212818.GA21558@thyrsus.com","subject":"Re: gitpacker progress report and a question","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2012-11-26T20:07:07Z","receivedAt":"2012-11-26T20:07:07Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Thu, Nov 15, 2012 at 10:28 PM, Eric S. Raymond <esr@thyrsus.com> wrote:\n> Some days ago I reported that I was attempting to write a tool that could\n> (a) take a git repo and unpack it into a tarball sequence plus a metadata log,\n> (b) reverse that operation, packing a tarball and log sequence into a repo.\n>\n> Thanks in part to advice by Andreas Schwab and in part to looking at the\n> text of the p4 import script, this effort has succeeded.  A proof of\n> concept is enclosed.  It isn't documented yet, and has not been tested\n> on a repository with branches or merges in the history, but I am confident\n> that the distance from here to a finished and tested tool is short.\n>\n> The immediate intended use is for importing older projects that are\n> available only as sequences of release tarballs, but there are other\n> sorts of repository surgery that would become easier using it.\n>\n> I'm still looking for a better name for it and would welcome suggestions.\n>\n> Before I do much further work, I need to determine how this will be shipped.\n> I see two possibilities: either I ship it as a small standalone project,\n> or it becomes a git subcommand shipped with the git suite. How I document\n> it and set up its tests would differ between these two cases.\n\nPlease look at Documentation/SubmittingPatches, you should send\npatches in inline format, preferably with 'git format-patch -M', and\npreferably with 'git send-email' (in which case you don't need\nformat-patch), otherwise people will have trouble reviewing, or miss\nit completely (as it was the case for me).\n\nI have many comments, but I'll wait until you send the patch inlined,\nI'll just address these:\n\n1) I tried it, and it doesn't seem to import (pack?) are repository\nwith sub-directories in it\n\n2) Using 'git fast-import' is probably simpler, and more efficient\n\nHere is a proof of concept I wrote in ruby that is half the size, and\nseems to implement the same functionality. The format is exactly the\nsame, but I think it should be modified to be more efficient.\n\nCheers.\n\n>From eb3c34699d7f5d4eec4f088344659b8d9b6a07ea Mon Sep 17 00:00:00 2001\nFrom: Felipe Contreras <felipe.contreras@gmail.com>\nDate: Mon, 26 Nov 2012 20:48:38 +0100\nSubject: [PATCH] Add new git-weave tool\n\nSigned-off-by: Felipe Contreras <felipe.contreras@gmail.com>\n---\n contrib/weave/git-weave | 166 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 166 insertions(+)\n create mode 100755 contrib/weave/git-weave\n\ndiff --git a/contrib/weave/git-weave b/contrib/weave/git-weave\nnew file mode 100755\nindex 0000000..3106121\n--- /dev/null\n+++ b/contrib/weave/git-weave\n@@ -0,0 +1,166 @@\n+#!/usr/bin/env ruby\n+\n+require 'optparse'\n+require 'find'\n+require 'fileutils'\n+\n+def export(indir = '.', out = STDOUT)\n+  open(File.join(indir, 'log')).each(\"\\n.\\n\") do |data|\n+\n+    @msg = nil\n+    @parents = []\n+\n+    data.chomp(\".\\n\").each_line do |l|\n+      if not @msg\n+        case l\n+        when /^commit (.+)$/\n+          @id = $1\n+        when /^author (.+)$/\n+          @author = $1\n+        when /^committer (.+)$/\n+          @committer = $1\n+        when /^parent (.+)$/\n+          @parents << $1\n+        when /^$/\n+          @msg = \"\"\n+        end\n+      else\n+        @msg << l\n+      end\n+    end\n+\n+    out.puts \"commit refs/heads/master\"\n+    out.puts \"mark :#{@id}\"\n+    out.puts \"author #{@author}\"\n+    out.puts \"committer #{@committer}\"\n+    out.puts \"data #{@msg.bytesize}\"\n+    out.puts @msg\n+\n+    @parents.each_with_index do |p, i|\n+      if i == 0\n+        out.puts \"from :%u\" % p\n+      else\n+        out.puts \"merge :%u\" % p\n+      end\n+    end\n+\n+    # files\n+    out.puts 'deleteall'\n+    FileUtils.cd(File.join(indir, @id)) do\n+      Find.find('.') do |e|\n+        next unless File.file?(e)\n+        content = File.read(e)\n+        filename = e.split(File::SEPARATOR).slice(1..-1).join(File::SEPARATOR)\n+        mode = File.executable?(e) ? '100755' : '100644'\n+        if File.symlink?(e)\n+          mode = '120000'\n+          content = File.readlink(e)\n+        end\n+        out.puts 'M %s inline %s' % [mode, filename]\n+        out.puts \"data #{content.bytesize}\"\n+        out.puts content\n+      end\n+    end\n+\n+  end\n+end\n+\n+def import(outdir, out)\n+  format = 'format:commit %H%nauthor %an <%ae> %ad%ncommitter %cn\n<%ce> %cd%nparents %P%n%n%B'\n+  cmd = ['git', 'log', '-z', '-s', '--date=raw', '--format=%s' %\nformat, '--all', '--reverse']\n+  commits = {}\n+\n+  IO.popen(cmd).each_with_index(\"\\0\") do |data, i|\n+    @msg = nil\n+    @parents = []\n+    data.chomp(\"\\0\").each_line do |l|\n+      if not @msg\n+        case l\n+        when /^commit (.+)$/\n+          @id = $1\n+        when /^author (.+)$/\n+          @author = $1\n+        when /^committer (.+)$/\n+          @committer = $1\n+        when /^parents (.+)$/\n+          @parents = $1.split(\" \")\n+        when /^$/\n+          @msg = \"\"\n+        end\n+      else\n+        @msg << l\n+      end\n+    end\n+\n+    num = i + 1\n+    commits[@id] = num\n+\n+    out.puts \"commit #{num}\"\n+    @parents.each do |p|\n+      out.puts \"parent #{commits[p]}\"\n+    end\n+    out.puts \"author #{@author}\"\n+    out.puts \"committer #{@committer}\"\n+    out.puts\n+    out.puts @msg.gsub(/\\n\\n+/, \"\\n\") # why?\n+    out.puts \".\"\n+\n+    wd = File.join(outdir, num.to_s)\n+    FileUtils.mkdir_p(wd)\n+    system('git', '--work-tree', wd, 'checkout', '-f', '-q', @id)\n+  end\n+end\n+\n+def git_pack(indir, outdir)\n+  indir = File.absolute_path(indir)\n+  system('git', 'init', outdir)\n+  FileUtils.cd(outdir) do\n+    IO.popen(['git', 'fast-import'], 'w') do |io|\n+      export(indir, io)\n+    end\n+    system('git', 'reset', '--hard')\n+  end\n+end\n+\n+def git_unpack(indir, outdir)\n+  begin\n+    FileUtils.mkdir_p(outdir)\n+    log = File.open(File.join(outdir, 'log'), 'w')\n+    ENV['GIT_DIR'] = File.join(indir, '.git')\n+    import(outdir, log)\n+  ensure\n+    system('git', 'symbolic-ref', 'HEAD', 'refs/heads/master')\n+    ENV.delete('GIT_DIR')\n+    log.close if log\n+  end\n+end\n+\n+$indir = '.'\n+\n+begin\n+  OptionParser.new do |opts|\n+    opts.on('-x') do\n+      $mode = 'unpack'\n+    end\n+    opts.on('-c') do\n+      $mode = 'pack'\n+    end\n+    opts.on('-o', '--outdir DIR') do |v|\n+      $outdir = v\n+    end\n+    opts.on('-i', '--indir DIR') do |v|\n+      $indir = v\n+    end\n+  end.parse!\n+rescue OptionParser::InvalidOption\n+end\n+\n+$mode = File.exists?(File.join($indir, '.git')) ? 'unpack' : 'pack'\nunless $mode\n+$outdir = File.join($indir, $mode == 'pack' ? 'packed' : 'unpacked2')\nunless $outdir\n+\n+case $mode\n+when 'pack'\n+  git_pack($indir, $outdir)\n+when 'unpack'\n+  git_unpack($indir, $outdir)\n+end\n-- \n1.8.0\n\n-- \nFelipe Contreras\n"},{"id":"203940","messageId":"20121126220108.GB1713@thyrsus.com","threadId":"32124","inReplyTo":"CAMP44s30px2FgieC9VcGji7T+yWFo7gMSwQhtuztHrqto8B7Aw@mail.gmail.com","subject":"Re: gitpacker progress report and a question","fromName":"Eric S. Raymond","fromEmail":"esr@thyrsus.com","sentAt":"2012-11-26T22:01:08Z","receivedAt":"2012-11-26T22:01:08Z","isPatch":false,"sender":{"key":"esr@thyrsus.com","avatar":"https://avatars.githubusercontent.com/u/727961?v=4"},"body":"Felipe Contreras <felipe.contreras@gmail.com>:\n> 1) I tried it, and it doesn't seem to import (pack?) are repository\n> with sub-directories in it\n\nI'll make sure my regression test checks this case.  The options to git \nls-files are a bit confusing and it's possible my invocation of it\nneeds to change.\n \n> 2) Using 'git fast-import' is probably simpler, and more efficient\n\nThat might well be.  I'm not worried about \"efficiency\" in this context \nbut reducing the code size is significant and I'm willing to re-code\nto do that.\n \n> Here is a proof of concept I wrote in ruby that is half the size, and\n> seems to implement the same functionality.\n\nNot anywhere near the same.  It only handles commits, not tags. It\ndoesn't issue delete ops.  And it doesn't rebuild branch heads.\nIf I were willing to omit those features, I'm sure I could halve\nthe size of my implementation, too.  Of course, it would then be\nalmost completely useless...\n\n>                           The format is exactly the\n> same, but I think it should be modified to be more efficient.\n\nI'm not wedded to the log format as it is, so I'll cheerfully\ntake suggestions about it.\n\nBe aware, however, that I consider easy editability by human beings\nmuch more important than squeezing the last microsecond out of the\nprocessing time.  So, for example, I won't use data byte counts rather\nthan end delimiters, the way import streams do.\n-- \n\t\t<a href=\"http://www.catb.org/~esr/\">Eric S. Raymond</a>\n"},{"id":"203948","messageId":"CAMP44s2+NDyL2Vf=iKR09f-YUnp=G2BCiYj-+qKuP7JA-+YkmQ@mail.gmail.com","threadId":"32124","inReplyTo":"20121126220108.GB1713@thyrsus.com","subject":"Re: gitpacker progress report and a question","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2012-11-26T23:14:06Z","receivedAt":"2012-11-26T23:14:06Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Mon, Nov 26, 2012 at 11:01 PM, Eric S. Raymond <esr@thyrsus.com> wrote:\n> Felipe Contreras <felipe.contreras@gmail.com>:\n>> 1) I tried it, and it doesn't seem to import (pack?) are repository\n>> with sub-directories in it\n>\n> I'll make sure my regression test checks this case.  The options to git\n> ls-files are a bit confusing and it's possible my invocation of it\n> needs to change.\n\nMight be easier to just call 'git ls-files --with-three foo', but I\ndon't see the point of those calls:\n\n% git --work-tree=unpacked/1 checkout master\n% git --work-tree=unpacked/1 add -A\n\nShould work just fine.\n\n>> 2) Using 'git fast-import' is probably simpler, and more efficient\n>\n> That might well be.  I'm not worried about \"efficiency\" in this context\n> but reducing the code size is significant and I'm willing to re-code\n> to do that.\n\nI don't see how the code-size would increase dramatically.\n\n>> Here is a proof of concept I wrote in ruby that is half the size, and\n>> seems to implement the same functionality.\n>\n> Not anywhere near the same.  It only handles commits, not tags.\n\nThe attached code doesn't handle tags either.\n\n> It doesn't issue delete ops.\n\nWhat do you mean?\n\n    out.puts 'deleteall' <- All current files are removed\n\nAnd then added.\n\n> And it doesn't rebuild branch heads.\n\nWhat do you mean? Your code only exports a single branch, the branch\nthat is currently checked out. And then:\n\ngit reset --hard >/dev/null; git checkout master >/dev/null 2>&1\n\nIt's resuming to 'master', which might not be the branch the user had\ncheckout out, and might not even exist.\n\n> If I were willing to omit those features, I'm sure I could halve\n> the size of my implementation, too.  Of course, it would then be\n> almost completely useless...\n\nThat's what the code currently does.\n\nDo you want me to show you step by step how they do *exactly the\nsame*? Of course, I would need to fix your version first so that it\ndoesn't crash with sub-directories.\n\n>>                           The format is exactly the\n>> same, but I think it should be modified to be more efficient.\n>\n> I'm not wedded to the log format as it is, so I'll cheerfully\n> take suggestions about it.\n>\n> Be aware, however, that I consider easy editability by human beings\n> much more important than squeezing the last microsecond out of the\n> processing time.  So, for example, I won't use data byte counts rather\n> than end delimiters, the way import streams do.\n\nWell, if there's a line with a single dot in the commit message ('.'),\nthings would go very bad.\n\nPersonally I would prefer something like this:\n\ntag v0.1 gst-av-0.1.tar \"Release 0.1\"\ntag v0.2 gst-av-0.2.tar \"Release 0.2\"\ntag v0.3 gst-av-0.3.tar \"Release 0.3\"\n\nAnd the script in bash would be very simple:\n\n#!/bin/sh\n\ntag() {\n\td=`mktemp -d` &&\n\t(\n\tcd $d &&\n\ttar -xf \"$orig/$2\" &&\n\tcd * &&\n\tgit add --all &&\n\tgit commit -q -m \"$3\" &&\n\tgit tag $1) || error=1\n\trm -rf $d\n\ttest -n \"$error\" && exit -1\n}\n\norig=\"$PWD\"\nrepo=\"$1\"\n\ngit init -q $repo\nexport GIT_DIR=\"$orig/$repo/.git\"\n\nsource \"$orig/$2\"\n\ncd \"$orig/$repo\" && git reset -q --hard\n\n-- \nFelipe Contreras\n"},{"id":"203953","messageId":"20121126234359.GA8042@thyrsus.com","threadId":"32124","inReplyTo":"CAMP44s2+NDyL2Vf=iKR09f-YUnp=G2BCiYj-+qKuP7JA-+YkmQ@mail.gmail.com","subject":"Re: gitpacker progress report and a question","fromName":"Eric S. Raymond","fromEmail":"esr@thyrsus.com","sentAt":"2012-11-26T23:43:59Z","receivedAt":"2012-11-26T23:43:59Z","isPatch":false,"sender":{"key":"esr@thyrsus.com","avatar":"https://avatars.githubusercontent.com/u/727961?v=4"},"body":"Felipe Contreras <felipe.contreras@gmail.com>:\n> Might be easier to just call 'git ls-files --with-three foo', but I\n> don't see the point of those calls:\n\nAh, much is now explained.  You were looking at an old version.  I had\nin fact already fixed the subdirectories bug (I've updated my\nregression test to check) and have full support for branchy repos,\npreserving tags and branch heads.\n\n> > It doesn't issue delete ops.\n> \n> What do you mean?\n> \n>     out.puts 'deleteall' <- All current files are removed\n\nYours emits no D ops for files removed after a particular snapshot.\n\n> > Be aware, however, that I consider easy editability by human beings\n> > much more important than squeezing the last microsecond out of the\n> > processing time.  So, for example, I won't use data byte counts rather\n> > than end delimiters, the way import streams do.\n> \n> Well, if there's a line with a single dot in the commit message ('.'),\n> things would go very bad.\n\nApparently you missed the part where I byte-stuffed the message content.\nIt's a technique used in a lot of old-school Internet protocols, notably\nin SMTP.\n \n> Personally I would prefer something like this:\n\nThere's a certain elegance to that, but it would be hard to generate by hand.\n\nRemember that a major use case for this tool is making repositories \nfrom projects whose back history exists only as tarballs.  So, let's\nsay you have the following:\n\nfoo-1.1.tar.gz\nfoo-1.2.tar.gz\nfoo-1.3.tar.gz\n\nWhat you're going to do before weaving is drop the untarred file trees\nin a 'foo' scratch directory, then hand-craft a log file that might\nlook a bit like this:\n\n-----------------------------------\ncommit 1\ndirectory foo-1.1\n\nRelease 1.1 of project foo\n.\ncommit 2\ndirectory foo-1.2\n\n..This is an example of a byte-stuffed line.\n\nRelease 1.2 of project foo\n.\ncommit 3\ndirectory foo-1.3\n\nRelease 1.3 of project foo\n.\n-----------------------------------\n\nThe main objective of the logfile design is to make hand-crafting \nthese easy.  \n-- \n\t\t<a href=\"http://www.catb.org/~esr/\">Eric S. Raymond</a>\n"},{"id":"203957","messageId":"CAMP44s3HAzSPsrGwcpQpx_3n2aHK5wm++_7_Cbk3qRWMkxDh6g@mail.gmail.com","threadId":"32124","inReplyTo":"20121126234359.GA8042@thyrsus.com","subject":"Re: gitpacker progress report and a question","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2012-11-27T01:29:22Z","receivedAt":"2012-11-27T01:29:22Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Tue, Nov 27, 2012 at 12:43 AM, Eric S. Raymond <esr@thyrsus.com> wrote:\n> Felipe Contreras <felipe.contreras@gmail.com>:\n>> Might be easier to just call 'git ls-files --with-three foo', but I\n>> don't see the point of those calls:\n>\n> Ah, much is now explained.  You were looking at an old version.  I had\n> in fact already fixed the subdirectories bug (I've updated my\n> regression test to check) and have full support for branchy repos,\n> preserving tags and branch heads.\n\nSo you are criticizing my code saying \"it would then be almost\ncompletely useless...\", when this is in fact what you sent to the\nlist.\n\nFor the record, here is the output of a test with your script vs.\nmine: the output is *exactly the same*:\n\n---\n== log ==\n* afcbedc (tag: v0.2, master) bump\n| * cbd2dce (devel) dev\n|/\n* 46f1813 (HEAD, test) remove\n* df95e41 dot .\n* ede0876 with\n* d6f10fc extra\n* e6362b1 (tag: v0.1) one\n== files ==\nfile\n== spaces ==\nwith\n\nspaces\n\n== dot ==\ndot\n.\n\n== orig ref ==\nrefs/heads/test\n\n== script ==\nbc9a7d99132f97adeb5d2ca266bd3d8bc64ccb21  /home/felipec/Downloads/gitpacker.txt\nUnpacking......(0.13 sec) done.\nPacking......(0.28 sec) done.\n== log ==\n* 5d0b634 (HEAD, master) bump\n* 2fe4a6d remove\n* 0c27d3b dot .\n* 5e36d3f with spaces\n* d6f10fc extra\n* e6362b1 one\n== files ==\nfile\n== spaces ==\nwith\nspaces\n\n== dot ==\ndot\n.\n\n== orig ref ==\nrefs/heads/master\n\n== script ==\n33edcb28667b683fbb5f8782383f782f73c5e9e1  /home/felipec/bin/git-weave\n== log ==\n* afcbedc (HEAD, master) bump\n* 46f1813 remove\n* df95e41 dot .\n* ede0876 with\n* d6f10fc extra\n* e6362b1 one\n== files ==\nfile\n== spaces ==\nwith\n\nspaces\n\n== dot ==\ndot\n.\n\n== orig ref ==\nrefs/heads/test\n---\n\nUnfortunately, when I enable some testing stuff, this is what your\nscript throws:\n---\n== script ==\nbc9a7d99132f97adeb5d2ca266bd3d8bc64ccb21  /home/felipec/Downloads/gitpacker.txt\nUnpacking......(0.17 sec) done.\nPacking......(0.02 sec) done.\nTraceback (most recent call last):\n  File \"/home/felipec/Downloads/gitpacker.txt\", line 308, in <module>\n    git_pack(indir, outdir, quiet=quiet)\n  File \"/home/felipec/Downloads/gitpacker.txt\", line 171, in git_pack\n    command += \" \".join(map(lambda p: \"-p \" + commit_id[int(p)],parents))\n  File \"/home/felipec/Downloads/gitpacker.txt\", line 171, in <lambda>\n    command += \" \".join(map(lambda p: \"-p \" + commit_id[int(p)],parents))\nIndexError: list index out of range\n== log ==\nfatal: bad default revision 'HEAD'\n== files ==\nfatal: tree-ish master not found.\n== spaces ==\nfatal: ambiguous argument ':/with': unknown revision or path not in\nthe working tree.\nUse '--' to separate paths from revisions, like this:\n'git <command> [<revision>...] -- [<file>...]'\n== dot ==\nfatal: ambiguous argument ':/dot': unknown revision or path not in the\nworking tree.\nUse '--' to separate paths from revisions, like this:\n'git <command> [<revision>...] -- [<file>...]'\n== orig ref ==\nrefs/heads/master\n---\n\nI'm attaching it in case you are interested.\n\nAnyway, I can add support for branches and tags in no time, but I\nwonder what's the point. Who will take so much time and effort to\ngenerate all the branches and tags, and the log file?\n\nIf the goal is as you say \"importing older projects that are available\nonly as sequences of release tarballs\", then that code is overkill,\nand it's not even making it easier to import the tarballs.\n\nFor that case my proposed format:\n\ntag v0.1 gst-av-0.1.tar \"Release 0.1\"\ntag v0.2 gst-av-0.2.tar \"Release 0.2\"\ntag v0.3 gst-av-0.3.tar \"Release 0.3\"\n\nWould be much more suitable.\n\n>> > It doesn't issue delete ops.\n>>\n>> What do you mean?\n>>\n>>     out.puts 'deleteall' <- All current files are removed\n>\n> Yours emits no D ops for files removed after a particular snapshot.\n\nman git fast-import\n\n---\nThis command is extremely useful if the frontend does not know (or\ndoes not care to know) what files are currently on the branch, and\ntherefore cannot generate the proper filedelete commands to update the\ncontent.\n---\n\nWhy would I want to emit D operations, again, deleteall takes care of that.\n\n>> > Be aware, however, that I consider easy editability by human beings\n>> > much more important than squeezing the last microsecond out of the\n>> > processing time.  So, for example, I won't use data byte counts rather\n>> > than end delimiters, the way import streams do.\n>>\n>> Well, if there's a line with a single dot in the commit message ('.'),\n>> things would go very bad.\n>\n> Apparently you missed the part where I byte-stuffed the message content.\n> It's a technique used in a lot of old-school Internet protocols, notably\n> in SMTP.\n\nYou might have done that, but the user that generated the log file\nmight have not.\n\n>> Personally I would prefer something like this:\n>\n> There's a certain elegance to that, but it would be hard to generate by hand.\n\nYou think this is hard to generate by hand:\n\n---\ntag v0.1 gst-av-0.1.tar \"Release 0.1\"\ntag v0.2 gst-av-0.2.tar \"Release 0.2\"\ntag v0.3 gst-av-0.3.tar \"Release 0.3\"\n---\n\nThan this?\n\n---\ncommit 1\ndirectory gst-av-0.1\n\nRelease 0.1\n.\ncommit 2\ndirectory gst-av-0.2\n\nRelease 0.2\n.\ncommit 3\ndirectory gst-av-0.3\n\nRelease 0.3\n.\n---\n\nAfter of course, extracting the tarballs, which my script already does\nautomatically.\n\n> Remember that a major use case for this tool is making repositories\n> from projects whose back history exists only as tarballs.\n\nWhich is exactly what my script does, except even easier, because it\nextracts the tarballs automatically.\n\n> The main objective of the logfile design is to make hand-crafting\n> these easy.\n\nWhat does the above log file achieve, that my log file doesn't?\n\n-- \nFelipe Contreras\n"},{"id":"203959","messageId":"CAMP44s29Ns2ea9f8ozr+W-eKAv=Q8f8d30t9zrfXfV7Jf5ZpFw@mail.gmail.com","threadId":"32124","inReplyTo":"CAMP44s3HAzSPsrGwcpQpx_3n2aHK5wm++_7_Cbk3qRWMkxDh6g@mail.gmail.com","subject":"Re: gitpacker progress report and a question","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2012-11-27T01:38:24Z","receivedAt":"2012-11-27T01:38:24Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Tue, Nov 27, 2012 at 2:29 AM, Felipe Contreras\n<felipe.contreras@gmail.com> wrote:\n\nActually no, they are not exactly the same, your version has a bug\nwhen dealing with spaces in a commit message (which pretty much all\nproper multi-line commit messages have).\n\n> == spaces ==\n> with\n> spaces\n>\n\nMine doesn't:\n\n> == spaces ==\n> with\n>\n> spaces\n>\n\n-- \nFelipe Contreras\n"},{"id":"203968","messageId":"CAMP44s3=VpMv-S2eV9rXRaH9U3SvaR8B6Dto=vAmVQ_XB1uBXg@mail.gmail.com","threadId":"32124","inReplyTo":"20121126234359.GA8042@thyrsus.com","subject":"Re: gitpacker progress report and a question","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2012-11-27T06:29:05Z","receivedAt":"2012-11-27T06:29:05Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Tue, Nov 27, 2012 at 12:43 AM, Eric S. Raymond <esr@thyrsus.com> wrote:\n\n> -----------------------------------\n> commit 1\n> directory foo-1.1\n>\n> Release 1.1 of project foo\n> .\n> commit 2\n> directory foo-1.2\n>\n> ..This is an example of a byte-stuffed line.\n>\n> Release 1.2 of project foo\n> .\n> commit 3\n> directory foo-1.3\n>\n> Release 1.3 of project foo\n> .\n> -----------------------------------\n>\n> The main objective of the logfile design is to make hand-crafting\n> these easy.\n\nHere's another version with YAML:\n---\n-\n  author: &me Felipe Contreras <felipe.contreras@gmail.com>\n  date: 2011-1-1\n  msg: one\n- tag v0.1\n-\n  author: *me\n  date: 2011-1-2\n  msg: extra\n-\n  author: *me\n  date: 2011-1-3\n  msg: |\n    with\n\n    spaces\n-\n  author: *me\n  date: 2011-1-4\n  msg: |\n    dot\n    .\n-\n  author: *me\n  date: 2011-1-5\n  msg: remove\n  ref: remove\n- checkout devel\n-\n  author: *me\n  date: 2011-1-6\n  msg: dev\n- checkout master\n-\n  author: *me\n  date: 2011-1-7\n  msg: bump\n- tag v0.2\n- checkout test remove\n---\n\nI believe that log file is much more human readable. Yet I still fail\nto see why would anybody want so much detail only to import tarballs.\n\ndiff --git a/contrib/weave/git-weave b/contrib/weave/git-weave\nnew file mode 100755\nindex 0000000..646aeaa\n--- /dev/null\n+++ b/contrib/weave/git-weave\n@@ -0,0 +1,234 @@\n+#!/usr/bin/env ruby\n+\n+require 'optparse'\n+require 'find'\n+require 'fileutils'\n+require 'yaml'\n+\n+$last = nil\n+$branches = {}\n+$branch = 'master'\n+$refs = {}\n+\n+class Commit\n+\n+  attr_reader :id, :parents, :author, :committer, :date, :msg, :ref\n+\n+  @@num = 0\n+\n+  def initialize(args)\n+    @id = @@num += 1\n+    @parents = []\n+    args.each do |key, value|\n+      instance_variable_set(\"@#{key}\", value)\n+    end\n+    if @author =~ /(.+ <.+>) (.+)/\n+      @author = $1\n+    end\n+    if @committer =~ /(.+ <.+>) (.+)/\n+      @committer = $1\n+      @date = DateTime.strptime($2, '%s %z')\n+    end\n+    $refs[@ref] = @id if @ref\n+  end\n+\n+end\n+\n+def export_commit(cmd, indir, out)\n+\n+  c = Commit.new(cmd)\n+  $last = c.id\n+\n+  # commit\n+  out.puts 'commit refs/heads/%s' % $branch\n+  out.puts 'mark :%u' % c.id\n+  if c.author and c.committer\n+    out.puts 'author %s %s' % [c.author, c.date.strftime('%s %z')]\n+    out.puts 'committer %s %s' % [c.committer, c.date.strftime('%s %z')]\n+  else\n+    out.puts 'committer %s %s' % [c.author, c.date.strftime('%s %z')]\n+  end\n+  out.puts 'data %u' % c.msg.bytesize\n+  out.puts c.msg\n+\n+  # parents\n+  c.parents.each_with_index do |p, i|\n+    ref = $refs[p]\n+    if i == 0\n+      out.puts 'from :%u' % ref\n+    else\n+      out.puts 'merge :%u' % ref\n+    end\n+  end\n+\n+  # files\n+  out.puts 'deleteall'\n+  FileUtils.cd(File.join(indir, c.id.to_s)) do\n+    Find.find('.') do |e|\n+      next unless File.file?(e)\n+      content = File.read(e)\n+      filename = e.split(File::SEPARATOR).slice(1..-1).join(File::SEPARATOR)\n+      if File.symlink?(e)\n+        mode = '120000'\n+        content = File.readlink(e)\n+      else\n+        mode = File.executable?(e) ? '100755' : '100644'\n+      end\n+      out.puts 'M %s inline %s' % [mode, filename]\n+      out.puts 'data %u' % content.bytesize\n+      out.puts content\n+    end\n+  end\n+  out.puts\n+\n+end\n+\n+def do_reset(out, ref, from)\n+  out.puts \"reset %s\" % ref\n+  out.puts \"from :%u\" % from\n+  out.puts\n+end\n+\n+def export_reset(cmd, indir, out)\n+  _, ref, from = cmd.split\n+  do_reset(out, ref, from)\n+end\n+\n+def export_checkout(cmd, indir, out)\n+  _, $branch, from = cmd.split\n+  from = ':%u' % $last if not $branches[$branch]\n+  do_reset(out, 'refs/heads/%s' % $branch, from) if from\n+  $branches[$branch] = true\n+end\n+\n+def export_tag(cmd, indir, out)\n+  _, tag = cmd.split\n+  do_reset(out, 'refs/tags/%s' % tag, $last)\n+end\n+\n+def export(indir = '.', out = STDOUT)\n+\n+  $branches['master'] = true\n+\n+  YAML.load_file(File.join(indir, 'log')).each do |e|\n+    case e\n+    when Hash\n+      export_commit(e, indir, out)\n+    when /^checkout /\n+      export_checkout(e, indir, out)\n+    when /^tag /\n+      export_tag(e, indir, out)\n+    when /^reset /\n+      export_reset(e, indir, out)\n+    end\n+  end\n+\n+end\n+\n+def import(outdir, out)\n+  format = 'format:commit %H%nauthor %an <%ae> %ad%ncommitter %cn\n<%ce> %cd%nparents %P%n%n%B'\n+  cmd = ['git', 'log', '-z', '-s', '--date=raw', '--format=%s' %\nformat, '--reverse', '--all']\n+  commits = {}\n+\n+  cmds = []\n+\n+  IO.popen(cmd).each_with_index(\"\\0\") do |data, i|\n+    @msg = nil\n+    @parents = []\n+    data.chomp(\"\\0\").each_line do |l|\n+      if not @msg\n+        case l\n+        when /^commit (.+)$/\n+          @id = $1\n+        when /^author (.+)$/\n+          @author = $1\n+        when /^committer (.+)$/\n+          @committer = $1\n+        when /^parents (.+)$/\n+          @parents = $1.split(\" \")\n+        when /^$/\n+          @msg = \"\"\n+        end\n+      else\n+        @msg << l\n+      end\n+    end\n+\n+    num = i + 1\n+    commits[@id] = num\n+\n+    cmds << {\n+      :author => @author,\n+      :committer => @committer,\n+      :msg => @msg,\n+      :ref => num,\n+      :parents => @parents.map { |e| commits[e] },\n+    }\n+\n+    wd = File.join(outdir, num.to_s)\n+    FileUtils.mkdir_p(wd)\n+    system('git', '--work-tree', wd, 'checkout', '-f', '-q', @id)\n+  end\n+\n+  IO.popen(['git', 'show-ref', '--tags', '--heads']).each do |e|\n+    id, ref = e.chomp.split\n+    cmds << 'reset %s %s' % [ref, commits[id]]\n+  end\n+\n+  out.write(cmds.to_yaml)\n+end\n+\n+def git_pack(indir, outdir)\n+  indir = File.absolute_path(indir)\n+  system('git', 'init', '--quiet', outdir)\n+  FileUtils.cd(outdir) do\n+    IO.popen(['git', 'fast-import', '--quiet'], 'w') do |io|\n+      export(indir, io)\n+    end\n+    system('git', 'reset', '--quiet', '--hard')\n+  end\n+end\n+\n+def git_unpack(indir, outdir)\n+  begin\n+    FileUtils.mkdir_p(outdir)\n+    log = File.open(File.join(outdir, 'log'), 'w')\n+    ENV['GIT_DIR'] = File.join(indir, '.git')\n+    oldref = %x[git symbolic-ref HEAD]\n+    import(outdir, log)\n+  ensure\n+    system('git', 'symbolic-ref', 'HEAD', oldref) if oldref\n+    ENV.delete('GIT_DIR')\n+    log.close if log\n+  end\n+end\n+\n+$indir = '.'\n+\n+begin\n+  OptionParser.new do |opts|\n+    opts.on('-x') do\n+      $mode = 'unpack'\n+    end\n+    opts.on('-c') do\n+      $mode = 'pack'\n+    end\n+    opts.on('-o', '--outdir DIR') do |v|\n+      $outdir = v\n+    end\n+    opts.on('-i', '--indir DIR') do |v|\n+      $indir = v\n+    end\n+  end.parse!\n+rescue OptionParser::InvalidOption\n+end\n+\n+$mode = File.exists?(File.join($indir, '.git')) ? 'unpack' : 'pack'\nunless $mode\n+$outdir = File.join($indir, $mode == 'pack' ? 'packed' : 'unpacked2')\nunless $outdir\n+\n+case $mode\n+when 'pack'\n+  git_pack($indir, $outdir)\n+when 'unpack'\n+  git_unpack($indir, $outdir)\n+end\n\n-- \nFelipe Contreras\n"},{"id":"203969","messageId":"20121127072700.GA23169@thyrsus.com","threadId":"32124","inReplyTo":"CAMP44s3=VpMv-S2eV9rXRaH9U3SvaR8B6Dto=vAmVQ_XB1uBXg@mail.gmail.com","subject":"Re: gitpacker progress report and a question","fromName":"Eric S. Raymond","fromEmail":"esr@thyrsus.com","sentAt":"2012-11-27T07:27:00Z","receivedAt":"2012-11-27T07:27:00Z","isPatch":false,"sender":{"key":"esr@thyrsus.com","avatar":"https://avatars.githubusercontent.com/u/727961?v=4"},"body":"Felipe Contreras <felipe.contreras@gmail.com>:\n> I believe that log file is much more human readable. Yet I still fail\n> to see why would anybody want so much detail only to import tarballs.\n\nThe first time I needed such a tool (and I really should have built it then) \nwas during the events I wrote up in 2010 the INTERCAL Reconstruction Massacree;\nfull story at <http://esr.ibiblio.org/?p=2491>  Note in particular the\nfollowing paragraphs:\n\n    Reconstructing the history of C-INTERCAL turned out to be something of\n    an epic in itself. 1990 was back in the Dark Ages as far as version\n    control and release-management practices go; our tools were\n    paleolithic and our procedures likewise. The earliest versions of\n    C-INTERCAL were so old that even CVS wasn’t generally available yet\n    (CVS 1.0 didn’t even ship until six months after C-INTERCAL 0.3, my\n    first public release). SCCS had existed since the early 1980s but was\n    proprietary; the only game in town was RCS. Primitive, file-oriented\n    RCS.\n\n    I was a very early adopter of version control; when I wrote\n    Emacs’s VC mode in 1992 the idea of integrating version control\n    into normal workflow that closely was way out in front of current\n    practice. Today’s routine use of such tools wasn’t even a gleam in\n    anyone’s eye then, if only because disks were orders of magnitude\n    smaller and there was a lot of implied pressure to actually throw\n    away old versions of stuff. So I only RCSed some of the files in\n    the project at the time, and didn’t think much about that.\n\n    As a result, reconstructing C-INTERCAL’s history turned into about two\n    weeks of work. A good deal of it was painstaking digital archeology,\n    digging into obscure corners of the net for ancient release tarballs\n    Alex and I didn’t have on hand any more. I ended up stitching together\n    material from 18 different release tarballs, 11 unreleased snapshot\n    tarballs, one release tarball I could reconstruct, one release tarball\n    mined out of an obsolete Red Hat source RPM, two shar archives, a pax\n    archive, five published patches, two zip files, a darcs archive, and\n    my partial RCS history, and that’s before we got to the aerial\n    photography. To perform the surgery needed to integrate this, I wrote\n    a custom Python program assisted by two shellscripts, topping out at a\n    hair over 1200 lines of code.\n\nThe second time was much more recent and concerned a project called\n(seriously) robotfindskitten.  This code existed as a partial CVS \nrepository created by someone other than the original author,\nand some disconnected tarballs from before the repo.  The author\nhas requested that I knit the tarballs and the CVS history (which\nis now in git) into one repository.\n\nIn both cases the object was to assemble a coherent history \nfrom all the available metadata as if the projects had been using\nversion control all along.\n\nI know of at least one other group of disconnected tarballs, of a\nprogram called xlife, that is likely to need similar treatment. It's\nnot an uncommon situation for projects over a certain age, and there is\nlots of code like xlife dating from before the mid-1990s waiting for\nsomeone to pick up the pieces.\n-- \n\t\t<a href=\"http://www.catb.org/~esr/\">Eric S. Raymond</a>\n"},{"id":"203970","messageId":"20121127073033.GB23169@thyrsus.com","threadId":"32124","inReplyTo":"CAMP44s3=VpMv-S2eV9rXRaH9U3SvaR8B6Dto=vAmVQ_XB1uBXg@mail.gmail.com","subject":"Re: gitpacker progress report and a question","fromName":"Eric S. Raymond","fromEmail":"esr@thyrsus.com","sentAt":"2012-11-27T07:30:33Z","receivedAt":"2012-11-27T07:30:33Z","isPatch":false,"sender":{"key":"esr@thyrsus.com","avatar":"https://avatars.githubusercontent.com/u/727961?v=4"},"body":"Felipe Contreras <felipe.contreras@gmail.com>:\n> > The main objective of the logfile design is to make hand-crafting\n> > these easy.\n> \n> Here's another version with YAML:\n\nClever.  \n\nNow I have to decide if I should allow my aesthetic dislike of YAML to\nprevail despite the fact that it's pretty well suited to this job.  There\nis definitely a case for applying a standard metaprotocol like YAML (ugh)\nor XML (double ugh).\n-- \n\t\t<a href=\"http://www.catb.org/~esr/\">Eric S. Raymond</a>\n"},{"id":"203972","messageId":"CAMP44s1=sjWypEN3zzC=aPqZYnBFU0C+Dhrh3zj=jY57PHJ6wg@mail.gmail.com","threadId":"32124","inReplyTo":"20121127072700.GA23169@thyrsus.com","subject":"Re: gitpacker progress report and a question","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2012-11-27T08:20:52Z","receivedAt":"2012-11-27T08:20:52Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Tue, Nov 27, 2012 at 8:27 AM, Eric S. Raymond <esr@thyrsus.com> wrote:\n> Felipe Contreras <felipe.contreras@gmail.com>:\n>> I believe that log file is much more human readable. Yet I still fail\n>> to see why would anybody want so much detail only to import tarballs.\n\n> In both cases the object was to assemble a coherent history\n> from all the available metadata as if the projects had been using\n> version control all along.\n\nI didn't say I couldn't see why somebody would need such a tool, I\nsaid I couldn't see why somebody would need such a tool _with so much\ndetail_.\n\nMost of those old projects have a linear history, so a log file like\nthis would suffice:\n\ntag v0.1 gst-av-0.1.tar \"Release 0.1\"\ntag v0.2 gst-av-0.2.tar \"Release 0.2\"\ntag v0.3 gst-av-0.3.tar \"Release 0.3\"\n\nAnd if they really had release branches, it shouldn't be difficult to\nmodify it for:\n\ntag v0.1 gst-av-0.1.tar \"Release 0.1\"\ntag v0.2 gst-av-0.2.tar \"Release 0.2\"\ntag v0.2.1 gst-av-0.2.tar \"Release 0.2.1\"\ncheckout v0.2\ntag v0.3 gst-av-0.3.tar \"Release 0.3\"\n\nBut different commit/author and respective dates, and merges? Sounds\nlike overkill.\n\nCheers.\n\n-- \nFelipe Contreras\n"},{"id":"203973","messageId":"20121127083639.GA26935@thyrsus.com","threadId":"32124","inReplyTo":"CAMP44s1=sjWypEN3zzC=aPqZYnBFU0C+Dhrh3zj=jY57PHJ6wg@mail.gmail.com","subject":"Re: gitpacker progress report and a question","fromName":"Eric S. Raymond","fromEmail":"esr@thyrsus.com","sentAt":"2012-11-27T08:36:39Z","receivedAt":"2012-11-27T08:36:39Z","isPatch":false,"sender":{"key":"esr@thyrsus.com","avatar":"https://avatars.githubusercontent.com/u/727961?v=4"},"body":"Felipe Contreras <felipe.contreras@gmail.com>:\n> Most of those old projects have a linear history,\n\nINTERCAL didn't.  There were two branches for platform ports.\n\n> But different commit/author and respective dates, and merges? Sounds\n> like overkill.\n\nI felt it was important that the metadata format be able to specify\ngit's entire metadata and DAG semantics.  Otherwise, as sure as the\nsun rises, *somebody* would run into a corner case not covered, and\n(quite rightly) curse me for a shortsighted fool who had done a\nhalf-assed job.\n\nI don't do half-assed jobs.  Not ever, no way, nohow.\n-- \n\t\t<a href=\"http://www.catb.org/~esr/\">Eric S. Raymond</a>\n"},{"id":"203980","messageId":"CAMP44s0ctVh3cbJF3rykGKn-ZH5qQfwaX-=G+MT1jhU6vK=WYw@mail.gmail.com","threadId":"32124","inReplyTo":"20121127083639.GA26935@thyrsus.com","subject":"Re: gitpacker progress report and a question","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2012-11-27T08:51:01Z","receivedAt":"2012-11-27T08:51:01Z","isPatch":false,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Tue, Nov 27, 2012 at 9:36 AM, Eric S. Raymond <esr@thyrsus.com> wrote:\n> Felipe Contreras <felipe.contreras@gmail.com>:\n>> Most of those old projects have a linear history,\n>\n> INTERCAL didn't.  There were two branches for platform ports.\n\nFine:\n\ntag v0.1 gst-av-0.1.tar \"Release 0.1\"\ntag v0.2 gst-av-0.2.tar \"Release 0.2\"\ncheckout port1\ntag v0.2-p1 gst-av-0.2-p1.tar \"Release 0.2 p1\"\ncheckout port2 v0.2\ntag v0.2-p2 gst-av-0.2-p2.tar \"Release 0.2 p2\"\ncheckout master\ntag v0.3 gst-av-0.3.tar \"Release 0.3\"\n\nProblem solved.\n\n>> But different commit/author and respective dates, and merges? Sounds\n>> like overkill.\n>\n> I felt it was important that the metadata format be able to specify\n> git's entire metadata and DAG semantics.  Otherwise, as sure as the\n> sun rises, *somebody* would run into a corner case not covered, and\n> (quite rightly) curse me for a shortsighted fool who had done a\n> half-assed job.\n\nI'm willing to bet that won't happen.\n\n> I don't do half-assed jobs.  Not ever, no way, nohow.\n\nSo you prefer code that is way more complicated that it needs to be,\nand with a higher likelihood of introducing bugs? There's a point of\ndiminishing returns where the code that nobody uses causes bugs for\nreal use-cases. That's not good.\n\nI prefer code that does one thing, and does it well. And when the need\narises, evolve.\n\nCheers.\n\n-- \nFelipe Contreras\n"}]}