[RFC] Applying a graft to a tree and "rippling" the changes through the history

I've written a tool that will take a single commit, add it as a parent of another commit, and recreate the history above that second commit in a fully compatible manner.

This is mostly useful for creating a fully merged-up repository of the Linux Historical tree, and the current working tree.

I run this with /graft-ripple.pl linux-history.tmp/ linus origin

Where "origin" is the branch the historical repository is on, and "linus" is the branch the current repository is on.

Note: This does not end up fixing up HEAD or any branches, it just pulls
all the objects together and recreates the full history.

GPLv2, but I'll redo with a proper patch, signed-off-by, command line options and help and docs if anyone else feels this is useful as a general tool.

========= cut here =============

#!/usr/bin/perl

use strict; use warnings;

use Data::Dumper; use IPC::Open2;

sub git_commit_tree {
        my ($tree,$comments,@parents) = @_;

        my @cparents;
        foreach my $p (@parents) {
                push @cparents,"-p",$p;
        }

        my $pid = open2(*Reader, *Writer,
		"git-commit-tree",$tree,@cparents);
        print Writer $comments;
        close(Writer);
        my $commit = <Reader>;

        waitpid $pid, 0;
        close(Reader);

        chomp $commit;

        return $commit;
}

chdir($ARGV[0]);
open(GRL,"-|","git-rev-list","--parents",$ARGV[1])
	or die "Failed to run git-rev-list: " . $!;

my %csets;
my @revs;
while(<GRL>) {
	chomp;
	my ($commit,@parents) = split /\s+/;
	$csets{$commit}{parents} = \@parents;
	push @revs, $commit;

	open(GCF,"-|","git-cat-file","commit",$commit)
		or die "Failed to open git-cat-file: " . $!;

	my $in_comments = 0;
	while(<GCF>) {
		chomp;
		if ($in_comments) {
			$csets{$commit}{comments} .= $_ . "\n";

		} elsif (m/^tree (.+)$/) {
			$csets{$commit}{tree} = $1;
			#printf("tree = %s\n",$1);

		} elsif (m/^parent (.+)$/) {
			# Do nothing, we already got
			# the parents from rev-list.

		} elsif (m/^(author|committer) (.*) <(.*)> (.*)$/) {
			#printf("%s = %s <%s> at %s\n",$1, $2,$3,$4);
			@{$csets{$commit}{$1}}{qw(name email datetime)}
				= ($2,$3,$4);

		} elsif (length == 0) {
			$in_comments = 1;
			$csets{$commit}{comments} = "";
			next;
		}
	}
	close(GCF);

} close(GRL);

@revs = reverse @revs; push @{$csets{$revs[0]}{parents}},$ARGV[2];

my %newcsets;
foreach my $old (@revs) {
	printf("Processing commit %s\n",$old);
        $ENV{GIT_AUTHOR_EMAIL} = $csets{$old}{author}{email};
        $ENV{GIT_AUTHOR_NAME} = $csets{$old}{author}{name};
        $ENV{GIT_AUTHOR_DATE} = $csets{$old}{author}{datetime};
        $ENV{GIT_COMMITTER_DATE} = $csets{$old}{committer}{datetime};
        $ENV{GIT_COMMITTER_EMAIL} = $csets{$old}{committer}{email};
        $ENV{GIT_COMMITTER_NAME} = $csets{$old}{committer}{name};

	my @parents = @{$csets{$old}{parents}};
	foreach my $p (@{$csets{$old}{parents}}) {
		if (exists $newcsets{$p}) {
			push @parents, $newcsets{$p}
				if exists $newcsets{$p};
			printf("Found new csetid %s for %s\n",
				$newcsets{$p},$p);
		}
	}
        my $commit = git_commit_tree($csets{$old}{tree},
		$csets{$old}{comments},@parents);
	$newcsets{$old} = $commit;
        printf("Commit for version %s is %s\n",$old,$newcsets{$old});
}

Ryan> chdir($ARGV[0]);

That's dangerous without an "or-die". Being in the wrong directory before you do a lot of edits is a good way to bust your disk. :)

Ryan> 	my ($commit,@parents) = split /\s+/;

split with no args splits $_ on whitespace, tossing leading whitespace, just in case they ever put whitespace indentation ahead.

Ryan Anderson <ryan@michonline.com> writes:

I think the procedure is reproducible, which is a very nice property to have for a tool like this, but I am not sure what you mean by "in a fully compatible manner". What are you compatible with?

Also another rhetorical, tongue-in-cheek question. What is your plan to ripple the graft through to update signed tags? ;-)

Junio C Hamano wrote:

Well, what I meant was, "It creates a history that is purely a superset of the old history, so merges should work cleanly from the pre-graft subhistory to the fully merged history."

But clearly I was too ... terse.

IOW, this should work perfectly, assuming neither tree has been pulled into since the history was merged into historical-graft tree:

$ cd linux-head $ git branch -b ryan-hacking HEAD $ quilt push -a $ git commit -a -m "Apply quilt tree"

$ cd ../linux-historical-graft/ $ git pull ../linux-historical-graft/

:) Well, since I can't resist answering your rhetorical question:

They signed a specific DAG. I'm providing a richer, more complete DAG that is a pure-superset of the one they signed. It is not, however, equivalent, so their signature is not related to the superset DAG I have created. In practice, however, I don't expect that any tag-signers would state that there is a meaningful difference between the two DAGs, from the perspective of their signature.

FYI - I don't think merging the trees like this is a good idea, from the perspective of something like gitk - gitk took long enough to startup and display something on my merged tree here that I gave up and killed it off.

Hi, Junio C Hamano wrote:

You're not the only one. My tool is different, however, in that it accepts a list of old=>new commits. I have used it successfully to re-graft my changes from a CVS->GIT tree to the corresponding CVS->SVN->GIT tree (which cannot be identical, because (a) SVN's timestamps are not accurate enough and (b) SVN's CVS import is more accurate in reproducing CVS archives than cvsps can be).

I'd suggest adding the capability of grafting something onto a tag...

#!/usr/bin/python

# This is a simple script which clones a git subtree to another.

import sys,re,optparse,os,subprocess

parser = optparse.OptionParser("corresponding_file [ old_commit ]", conflict_handler="resolve", description="""\ Transfer a list of commits from one repository to another.

The first argument is a list of SHA1 entries of the form
	old new
It lists which commits are "the same".

old_commit and those of its parents which are not listed in the corresponding_file are copied, i.e. new commit objects are created. Their SHA1s are added to the file so that you may repeat the process with other commits, or run it incrementally.

""")
parser.add_option("-h","--help","-?", action="help",
                    help="Print this help message and exit")
parser.add_option("-v", "--verbose", dest="verbose", action="store_true",
                    help="Report progress")

(options, args) = parser.parse_args()

if len(args) < 1 or len(args) > 3:
	parser.error("requires one to three arguments")

def end(p):
	try:
		retcode = p.wait()
	except OSError,e:
		print >>sys.stderr, "git-rev-list failed:", e
	else:
		if retcode < 0:
			print >>sys.stderr, "git-rev-list was terminated by signal", -retcode
			sys.exit(1)
		elif retcode > 0:
			print >>sys.stderr, "git-rev-list exited with non-zero exit code", retcode
			sys.exit(1)

re_cmt = re.compile(r'\s*#.*')
corr = {}
for l in open(args[0]):
	l = re_cmt.sub("",l).strip()
	if l == "": continue
	try:
		a,b = l.split()
	except ValueError:
		continue
	if len(a) != 40: continue
	if len(b) != 40: continue
	corr[a]=b

corrf=open(args[0],"a")

if len(args) >= 2:
	srctag = args[1]
else:
	srctag = "HEAD"

srcrepo = os.path.curdir

commits=[]
cmd = ["git-rev-list",srctag]
for k in corr.iterkeys():
	cmd.append("^"+k)
if options.verbose:
	print cmd

p=subprocess.Popen(cmd, stdout=subprocess.PIPE)
for l in p.stdout:
	l = l.strip()
	commits.append(l)
end(p)

while len(commits):
	c = commits.pop()
	if c in corr: continue
	if options.verbose:
		print "Processing:",c

	p=subprocess.Popen(["git-cat-file","commit",c], stdout=subprocess.PIPE)
	qf=os.tempnam()
	try:
		q=open(qf,"w")
		nx=False
		for l in p.stdout:
			if nx:
				q.write(l)
				continue
			l = l.strip()
			if l == "":
				nx=True
				q.write("\n")
				continue
			a,b = l.split(" ",1)
			if a == "parent":
				b = corr[b]
			print >>q,a,b
		q.close()
		q=subprocess.Popen(["git-hash-object","-w","-t","commit",qf], stdout=subprocess.PIPE)
		d = q.stdout.read().strip()
		end(q)
	finally:
		os.unlink(qf)
	end(p)
	corr[c] = d
	print >>corrf, c,d
	if options.verbose:
		print c,d,l

# OK, everything is done. corrf.close() print d