Re: Merging submodules (was Re: Feature suggestion: git-hist)

This message got eaten by a syntax error somewhere.  This is a re-send, sorry for any duplicate messages.

On Jul 30, 2008, at 12:26 PM, H.Merijn Brand wrote:

Hm. I did mention this was completely untested, yes? The problem comes from the fact that '-s ours' wants to use HEAD, not the index. But you can't amend a normal commit into a merge, apparently. And I don't think you want a commit that adds the files and a commit that "does the merge" as two separate steps.

Well, I don't know how to make the porcelain do this then. But the plumbing can definitely do it. Hopefully someone more used to doing strange things like this can give a simpler recipe, but this should work.

# First reset to the commit you made with all the modules added.
vim commit-message # Create a merge message
commit=$(git commit-tree HEAD: -p HEAD^ -p module_a/master -p
		module_b/master -p module_c/master < commit-message)
git update-ref HEAD $commit  # Update your current ref

~~ Brian

On Wed, 30 Jul 2008 19:03:37 -0400, Brian Gernhardt <benji@silverinsanity.com> wrote:

                                 ^^^^^^^^
had to remove that part

Some history --- I'm aware I started at the wrong end of being a git user. I had to move from SCCS to `something better', and at that point only git, svn, and hq seemed to be likely candidates.

hq being python, and our company not using python, but perl, made that an easy drop. I gave up compiling svn on HP-UX in 64bit mode after about a week, mainly because it depended on way too many things, and the new VCS has to run on this platform, as it is our main development system. I got git up and running in two days (compile in less than two hours, but then I got to chase HP-UX and 64bit oddities).

By the I knew a lot about the git source code, make files, and test scripts, but still had no idea about the whole plumbing/porcelain approach. The plan was to make that someone else's job.

Once it was up and running, I had to create a way to convert all our SCCS repo's to git, so we could get started and test if it met our needs. That part went smooth, and with a little help from Sam Villain to get some speed into the conversions using git-fast-import, it is now available to the public on CPAN as VCS::SCCS, with git2sccs in the examples folder.

Using git-gui and gitk my users were enthousiastic, and they saw all the advantages of using git over SCCS. Of course, with every change there are a few (serious) drawbacks, but we have to live with those.

Being a perl5 porter/maintainer, I was used to p4v (perforce) and still wonder why there are two GUI's instead of just one, and why they don't offer the functionality I love in p4v. Not that I think perforce is better than git, but their GUI certainly is. ---

So, back to this merging issue. Now you might understand why I have all those `silly' questions and have (still) no good idea of what all these commands do. (The person that were to do all that never came into the picture). I'm learning.

I'm VERY happy and thankful for the help I get from you here, and I get the impression that my feedback on getting git running in our somewhat different environment to you is also appreciated.

I had to cut down my number of modules to merge, as I got an error that the maximum number of merges was 16. I had 18.

I will now be playing with the results a bit. I have attached the script, in case you might want to use it in documentation or examples. For now, all the mods are hardcoded. No arguments and so on.

Again, Thanks!

$ bash git-merge-mods.sh
Re-initialize GIT repo
Initialized empty Git repository in /work/lep/4gl/.git/
Recovering original module repo's
Fetching for i00f000
remote: Counting objects: 24, done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 24 (delta 9), reused 24 (delta 9)
Unpacking objects: 100% (24/24), done.
From i00f000
 * [new branch]      master     -> i00f000/master
Fetching for i99f000
:
:
Receiving objects: 100% (356/356), 139.05 KiB, done.
Resolving deltas: 100% (180/180), done.
From rpt
 * [new branch]      master     -> rpt/master
Removing module repo's
Adding modules
Commit
Merge
Automatic merge went well; stopped before committing as requested
Commit
=========
53229f046c5d85d11bbd500cf04b468fd3f62c08
=========
Update
$

On Thu, 31 Jul 2008 09:21:04 +0200, "H.Merijn Brand" <h.m.brand@xs4all.nl> wrote:

There is a slight problem with this merging approach. The path names are as they are/were in the submodules. In module_a, foo.pl was without a leading module_a/ path, and now after integration, it still is. Is it possible to rethink this whole process that integrates/merges the several git repo's in subfolders into the current folder, as-if they would have been in this folder in the first place?

Short summary: History made me have the current situation

	4gl
	4gl/fnc
	4gl/fnc/.git
	4gl/foo
	4gl/foo/.git
	:
	:
	4gl/zlork
	4gl/zlork/.git

Where I want

	4gl
	4gl/.git
	4gl/fnc
	4gl/foo
	:
	4gl/zlork

On Thu, Jul 31, 2008 at 02:39:55PM +0200, H.Merijn Brand wrote:

I would suggest re-reading Santi's suggestions:

I think the latter is specifically what you want.

On Thu, 31 Jul 2008 15:06:26 +0200, Petr Baudis <pasky@suse.cz> wrote:

I got stuck in that process, as the new repo thinks all the files from the sub-folders belong to the top-folder: After this process, there are a lot of files from subfolder include in the "new" top-level folder.

--8<--- git-join.sh #!/bin/sh

export MODS MODS="include fnc" MODS="$MODS i00f000 i99f000" MODS="$MODS l00m000 l01f000 l02f000 l03f000 l06f000 l90z000" MODS="$MODS rpt" MODS="$MODS leerpl mutbev prtabel tabellen zoomen"

echo "Creating merge environment" rm -rf new mkdir new cd new

echo "Recovering original repo's" tbz -s -x ../lep4gl-git

echo "Initializing new repo" git init

for mod in $MODS ; do
    echo "Merging $mod ..."
    git remote add -f B$mod $mod
    git merge -s ours --no-commit B$mod/master
    git read-tree --prefix=$mod/ -u B$mod/master
    git commit -m "Merge $mod as our subdirectory"
    git pull -s subtree B$mod master
    done

echo Done -->8---

On Thu, Jul 31, 2008 at 17:01, H.Merijn Brand <h.m.brand@xs4all.nl> wrote:

I see all OK. Can you provide a self consistent simple testcase that shows what is wrong?

# prepare the project and the modules mkdir project cd project git init ... mkdir module_a cd module_a git init ... # your script # show what you get and what you expect, for example:

$ git show HEAD: # or something equivalent $ ls $ git status ...

This is only needed when updating the subtree if the module has been updated (but it should do nothing in you case).

Santi

On Thu, 31 Jul 2008 17:24:40 +0200, "Santi Béjar" <sbejar@gmail.com> wrote:

Yes. was rather easy. http://www.xs4all.nl/~hmbrand/testcase.tgz

test 133 > ll * 23875989 -rwxr-xr-x 1 merijn users 486 2008-07-31 20:08 git-join.sh 23875990 -rw-rw-rw- 1 merijn users 14385 2008-07-31 20:07 test-git.tgz

4gl:
total 16
23871493 drwxrwxrwx 4 merijn users 4096 2008-07-31 20:01 .
23871492 drwxrwxrwx 3 merijn users 4096 2008-07-31 20:08 ..
23871494 drwxrwxrwx 3 merijn users 4096 2008-07-31 19:56 fnc
23875966 drwxrwxrwx 3 merijn users 4096 2008-07-31 20:04 include
test 134 > bash git-join.sh
Creating merge environment
Recovering original repo's
Initializing new repo
Initialized empty Git repository in /work/lep/test/new/.git/
Merging include ...
Updating Binclude
remote: Counting objects: 9, done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 9 (delta 1), reused 6 (delta 1)
Unpacking objects: 100% (9/9), done.
From include
 * [new branch]      master     -> Binclude/master
Created commit c2c491e: Merge include as our subdirectory
 1 files changed, 4 insertions(+), 0 deletions(-)
 create mode 100644 include/ini0.h
From include
 * branch            master     -> FETCH_HEAD
Already up-to-date.
Merging fnc ...
Updating Bfnc
warning: no common commits
remote: Counting objects: 12, done.
remote: Compressing objects: 100% (4/4), done.
Unpacking objects: 100% (12/12), done.
remote: Total 12 (delta 0), reused 9 (delta 0)
From fnc
 * [new branch]      master     -> Bfnc/master
Automatic merge went well; stopped before committing as requested
Created commit d7da4d6: Merge fnc as our subdirectory
From fnc
 * branch            master     -> FETCH_HEAD
Already up-to-date.
Done
test 135 > ll new
total 24
23887892 drwxrwxrwx 5 merijn users 4096 2008-07-31 20:11 .
23871492 drwxrwxrwx 4 merijn users 4096 2008-07-31 20:11 ..
23887973 drwxrwxrwx 8 merijn users 4096 2008-07-31 20:11 .git
23887893 drwxrwxrwx 3 merijn users 4096 2008-07-31 20:11 fnc
23887933 drwxrwxrwx 3 merijn users 4096 2008-07-31 20:11 include
23888029 -rw-rw-rw- 1 merijn users   56 2008-07-31 20:11 ini0.h
test 136 > ll new/include/
total 16
23887933 drwxrwxrwx 3 merijn users 4096 2008-07-31 20:11 .
23887892 drwxrwxrwx 5 merijn users 4096 2008-07-31 20:11 ..
23887934 drwxrwxrwx 8 merijn users 4096 2008-07-31 20:04 .git
23888031 -rw-rw-rw- 1 merijn users   56 2008-07-31 20:11 ini0.h
test 137 >

What is ini0.h doing in new/ ?

test 137 > cd new test/new 138 > git-ls-files fnc/foo.txt include/ini0.h ini0.h test/new 139 >

On Thu, Jul 31, 2008 at 20:15, H.Merijn Brand <h.m.brand@xs4all.nl> wrote:

It is because you cannot merge a branch with an empty branch. So, or you create an initial commit in the "superproject" or you create a commit just moving the files of the first module as in:

http://article.gmane.org/gmane.comp.version-control.git/79887

Santi

On Thu, 31 Jul 2008 21:03:26 +0200, "Santi Béjar" <sbejar@gmail.com> wrote:

Super. So I start with a .gitignore and continue this process. It worked!

Thanks again!

On Thu, 31 Jul 2008 21:03:26 +0200, "Santi Béjar" <sbejar@gmail.com> wrote:

Almost perfect now.

4gl/new 121 > git-ls-files | grep fnc_declare
include/fnc_declare.h
4gl/new 122 > git-show d971ec91a3dbc9f1f27a96e4f9b95366babd036c
commit d971ec91a3dbc9f1f27a96e4f9b95366babd036c
Author: H.M. Brand <merijn@a5.(none)>
Date:   Tue Jul 29 16:45:43 2008 +0200

    Backward comp functie voor PV indicatie

On Fri, Aug 1, 2008 at 09:04, H.Merijn Brand <h.m.brand@xs4all.nl> wrote:

It is correct for old commits, when you look at an old commit it doesn't know that later it is merge in a subfolder. For new commits it depends on how you made them, in the modules and doing a subtree merge or directly in the "superproject".

Santi

On Fri, 1 Aug 2008 11:52:08 +0200, "Santi Béjar" <sbejar@gmail.com> wrote:

After we join/merge these into the super-project, we're going to remove the sup-repos, so all new commits will be made in the super-repo.

What I'm a bit worried about is that if people are looking for a change in "include/inih001.h", they will not be able to find the commits on that path, as the old commits have stored it without the path element.

Will that be a problem, or is git/gitk clever enough to still find commits to both?

On Fri, Aug 1, 2008 at 12:35, H.Merijn Brand <h.m.brand@xs4all.nl> wrote:

If you'll remove the subrepos, the best thing would be to rewrite the history on those subrepos just moving all the path to the corresponding subfolder (with git-filter-branch, and you have exactly what you need at the end of the example section in the manpage). And then just do a normal merge. Or, even, you could try to create a project with everything there in the correct order and location, I don't know if git-filter-branch or git-fast-import/export (you have some examples in git.git in contrib/fast-import) can do it, but if you get it, please, post it here because it can be useful for others).

Santi

On Fri, 1 Aug 2008 13:34:42 +0200, "Santi Béjar" <sbejar@gmail.com> wrote:

I gave up, even thought I'm not satisfied with the result: Merging with subtrees works, but completely ruins my view over the history, cause gitk shows the branches as a whole, and not `merged' by date. I did change the merging process to merge the repo with the least recent change date first, so that it would show up at the bottom and the newest on top. This is workable, but far from perfect.

The fast-export/fast import with renaming started out fine, but fast-import does not merge. For the latter I (tried to) use the following approach:

--8<--- #!/pro/bin/perl

use strict; use warnings;

sub usage
{
    my $err = shift and select STDERR;
    print "usage: $0 dir|repo.tar\n";
    @_ and print join "\n", @_, "";
    exit $err;
    } # usage

-d "new" and die "Dir new already exists\n";

use Cwd;
use File::Find;
use Getopt::Long qw(:config bundling nopermute);
my $opt_v = 1;
GetOptions (
    "help|?"    => sub { usage (0); },

    "v:2"       => \$opt_v,
    ) or usage (1);

@ARGV == 1 or usage (1);

my $tmp_archive = "/tmp/git-join-$$.tgz"; END { unlink $tmp_archive };

my $archive = shift;
if (-d $archive) {
    my $cwd = getcwd;
    my @dir;
    chdir $archive or die "$archive: $!\n";
    find (sub {
        $_ eq ".git" && -d $_ and push @dir, $File::Find::name;
        }, ".");
    qx{ tar czf $tmp_archive @dir };
    $archive = $tmp_archive;
    chdir $cwd;
    }

sub pr_time
{
    my @d = @_;
    sprintf "%4d-%02d-%02d %02d:%02d:%02d", 1900 + $d[5], ++$d[4], @d[3,2,1,0];
    } # pr_time

-f $archive && -s _ or usage (1, "Archive is not a file");

my @cmd =
    $archive =~ m/\.tar$/               ? qw( tar xf  )         :
    $archive =~ m/\.t(ar\.)?gz$/        ? qw( tar xzf )         :
    $archive =~ m/\.t(ar\.)?bz2?$/      ? qw( tar xhf )         :
    usage (1, "$archive is not a recognized archive type");

print STDERR "Creating merge environment\n"; mkdir "new", 0777; chdir "new" or die "Canot use folder new\n";

print STDERR "Recovering original repo's\n"; system @cmd, $archive;

my %modules;
find (sub {
    (my $f = $File::Find::name) =~ s{^\./}{};
    $f =~ s{/\.git$}{};
    $_ eq ".git" && -d $_ && !$modules{$f} or return;
    print "Found $f\n";
    system "git-log '--pretty=format:%ct' | head -1";
    chomp ($modules{$f} = `git-log '--pretty=format:%ct' | head -1`);
    }, ".");
my @modules = sort { $modules{$a} <=> $modules{$b} } keys %modules;

sub git
{
    system "git", @_;
    } # git

my $top = getcwd;

print STDERR "Initializing new repo\n"; git "init";

foreach my $mod (@modules) {
    print STDERR "Merging ", pr_time (localtime $modules{$mod}), " $mod ...\n";
    chdir $mod;

    git "checkout", "-f";
    git "filter-branch", "--index-filter",
        qq{git ls-files -s | sed "s-\t-&$mod/-" | }.
         q{GIT_INDEX_FILE=$GIT_INDEX_FILE.new }.
         q{git update-index --index-info && }.
         q{mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE}, "HEAD";
    git "fast-export", "--all", ">/tmp/git-export-$$.bin";

    chdir $top;
    git "fast-import", "</tmp/git-export-$$.bin";
    }

print STDERR "Done\n"; -->8---

Which looks perfect after the first import, but fails on every next

Using --force for fast-import removes every previous import, so it is useless for this process.

I do have to work with the repo, and that is more important than having a perfect repo.

On Mon, Aug 04, 2008 at 03:24:43PM +0200, H.Merijn Brand wrote:

You might consider adopting a (relatively?) common strategy when importing historical projects: Actually start the history from scratch (git init && git add . && git commit -m"Initial commit") and fine-tune your historical import in a separate repository. Then, provide a script that people interested in the old history can run and it will graft the imported history to your pure-git history.

Conceptually, it should be pretty simple:

	git fetch git://perl-company.nl/sccs-import.git
	echo initial_git_commit_sha1 last_imported_commit_sha1 \
		>>$(git rev-parse --git-dir)/info/grafts

Example of a fine-tuned script:

	http://repo.or.cz/w/elinks.git?a=blob;f=contrib/grafthistory.sh

If you find out that the import is not perfect later on, you can just redo it, refetch and rewrite the info/grafts line.

On Mon, 4 Aug 2008 15:40:53 +0200, Petr Baudis <pasky@suse.cz> wrote:

We already changed the approach for converting projects from SCCS to create a recursive repo from all SCCS repo's. And that *is* perfect.

Problem with the current join/merge is that there already have been commits to the git repo after the SCCS convert that I do not want to loose.

I'll have a look at this anyway.

On Mon, Aug 04, 2008 at 03:57:33PM +0200, H.Merijn Brand wrote:

Yes, but that's orthogonal to whether to separate the imported history from the pure history? (Not that I'd want to force it upon you. If you have a lot of nested repositories, the grafting can get a bit obnoxious if you don't automate it well, I suppose.)

You could rebase these to the new initial commit. But if you already use Git intensively and rewriting history would create big headache for you at this point already, it's too late, I guess.

On Thu, Jul 31, 2008 at 14:39, H.Merijn Brand <h.m.brand@xs4all.nl> wrote:

If you want to have module_a/foo.pl in the old commits you will need to rewrite the history, either with "git filter-branch" or fast-export/import, or importing again.

For the new commits it depends on how you would do them. If you modify and commit without these module subfolders (without the module_a/.git as your git-merge-mods does) then the new commits will have module_a/foo.pl. So at the end you will have a merge of all the mods (using the subtree strategy), and after that you will have a single project.

All depends in a few factors, if you have published the history (for the rewriting), if you want to have the module_# as independent projects (this way you can merge them in other projects),...

Santi