{"thread":{"id":"19071","subject":"[PATCH/RFC 1/2] Add 'git subtree' command for tracking history of subtrees separately.","startedAt":"2009-04-26T22:29:42Z","lastAt":"2009-07-17T17:46:07Z","messageCount":13,"participants":["Avery Pennarun","Ping Yin","Finn Arne Gangstad","Andrey Smirnov"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"112358","messageId":"1240784983-1477-1-git-send-email-apenwarr@gmail.com","threadId":"19071","inReplyTo":null,"subject":"[PATCH/RFC 1/2] Add 'git subtree' command for tracking history of subtrees separately.","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2009-04-26T22:29:42Z","receivedAt":"2009-04-26T22:29:42Z","isPatch":true,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"Many projects are made of a combination of several subprojects/libraries and\nsome application-specific code.  In some cases, particularly when the\nsubprojects are all maintained independently, 'git submodule' is the best\nway to deal with this situation.  But if you frequently change the\nsubprojects as part of developing your application, use multiple branches,\nand sometimes want to push your subproject changes upstream, the overhead of\nmanually managing submodules can be excessive.\n\n'git subtree' provides an alternative mechanism, based around the\n'git merge -s subtree' merge strategy.  Instead of tracking a submodule\nseparately, you merge its history into your main project, and occasionally\nextract a new \"virtual history\" from your mainline that can be easily merged\nback into the upstream project.  The virtual history can be incrementally\nexpanded as you make more changes to the superproject.\n\nYou would normally then merge the virtual history back into your mainline\n(the --rejoin option). This results in extra commits in your application\nthat appear to change the same files, but these extra commits will tend to\nbe ignored by git's merge simplification algorithm anyway.\n\nFor example, gitweb (commit 1130ef3) was merged into git as of commit\n0a8f4f0, after which it was no longer maintained separately.  But imagine it\nhad been maintained separately, and we wanted to extract git's changes to\ngitweb since that time, to share with the upstream.  You could do this:\n\ngit subtree split --prefix=gitweb --annotate='(split) ' \\\n\t0a8f4f0^.. --onto=1130ef3 --rejoin\n\nIf gitweb had originally been merged using 'git subtree add' (or a previous\nsplit had been done with --rejoin specified), then you could incrementally\nproduce the list of new changes without needing to remember any commit ids:\n\ngit subtree split --prefix=gitweb --annotate='(split) ' --rejoin\n---\n Makefile         |    1 +\n command-list.txt |    1 +\n git-subtree.sh   |  435 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 437 insertions(+), 0 deletions(-)\n create mode 100755 git-subtree.sh\n\ndiff --git a/Makefile b/Makefile\nindex 5c8e83a..f14e11c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -305,6 +305,7 @@ SCRIPT_SH += git-request-pull.sh\n SCRIPT_SH += git-sh-setup.sh\n SCRIPT_SH += git-stash.sh\n SCRIPT_SH += git-submodule.sh\n+SCRIPT_SH += git-subtree.sh\n SCRIPT_SH += git-web--browse.sh\n \n SCRIPT_PERL += git-add--interactive.perl\ndiff --git a/command-list.txt b/command-list.txt\nindex fb03a2e..9be4774 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -113,6 +113,7 @@ git-stash                               mainporcelain\n git-status                              mainporcelain common\n git-stripspace                          purehelpers\n git-submodule                           mainporcelain\n+git-subtree                             mainporcelain\n git-svn                                 foreignscminterface\n git-symbolic-ref                        plumbingmanipulators\n git-tag                                 mainporcelain common\ndiff --git a/git-subtree.sh b/git-subtree.sh\nnew file mode 100755\nindex 0000000..39c377c\n--- /dev/null\n+++ b/git-subtree.sh\n@@ -0,0 +1,435 @@\n+#!/bin/bash\n+#\n+# git-subtree.sh: split/join git repositories in subdirectories of this one\n+#\n+# Copyright (C) 2009 Avery Pennarun <apenwarr@gmail.com>\n+#\n+if [ $# -eq 0 ]; then\n+    set -- -h\n+fi\n+OPTS_SPEC=\"\\\n+git subtree add --prefix=<prefix> <commit>\n+git subtree split [options...] --prefix=<prefix> <commit...>\n+git subtree merge --prefix=<prefix> <commit>\n+git subtree pull  --prefix=<prefix> <repository> <refspec...>\n+--\n+h,help        show the help\n+q             quiet\n+d             show debug messages\n+prefix=       the name of the subdir to split out\n+ options for 'split'\n+annotate=     add a prefix to commit message of new commits\n+onto=         try connecting new tree to an existing one\n+rejoin        merge the new branch back into HEAD\n+ignore-joins  ignore prior --rejoin commits\n+\"\n+eval $(echo \"$OPTS_SPEC\" | git rev-parse --parseopt -- \"$@\" || echo exit $?)\n+. git-sh-setup\n+require_work_tree\n+\n+quiet=\n+debug=\n+command=\n+onto=\n+rejoin=\n+ignore_joins=\n+annotate=\n+\n+debug()\n+{\n+\tif [ -n \"$debug\" ]; then\n+\t\techo \"$@\" >&2\n+\tfi\n+}\n+\n+say()\n+{\n+\tif [ -z \"$quiet\" ]; then\n+\t\techo \"$@\" >&2\n+\tfi\n+}\n+\n+assert()\n+{\n+\tif \"$@\"; then\n+\t\t:\n+\telse\n+\t\tdie \"assertion failed: \" \"$@\"\n+\tfi\n+}\n+\n+\n+#echo \"Options: $*\"\n+\n+while [ $# -gt 0 ]; do\n+\topt=\"$1\"\n+\tshift\n+\tcase \"$opt\" in\n+\t\t-q) quiet=1 ;;\n+\t\t-d) debug=1 ;;\n+\t\t--annotate) annotate=\"$1\"; shift ;;\n+\t\t--no-annotate) annotate= ;;\n+\t\t--prefix) prefix=\"$1\"; shift ;;\n+\t\t--no-prefix) prefix= ;;\n+\t\t--onto) onto=\"$1\"; shift ;;\n+\t\t--no-onto) onto= ;;\n+\t\t--rejoin) rejoin=1 ;;\n+\t\t--no-rejoin) rejoin= ;;\n+\t\t--ignore-joins) ignore_joins=1 ;;\n+\t\t--no-ignore-joins) ignore_joins= ;;\n+\t\t--) break ;;\n+\tesac\n+done\n+\n+command=\"$1\"\n+shift\n+case \"$command\" in\n+\tadd|merge|pull) default= ;;\n+\tsplit) default=\"--default HEAD\" ;;\n+\t*) die \"Unknown command '$command'\" ;;\n+esac\n+\n+if [ -z \"$prefix\" ]; then\n+\tdie \"You must provide the --prefix option.\"\n+fi\n+dir=\"$prefix\"\n+\n+if [ \"$command\" != \"pull\" ]; then\n+\trevs=$(git rev-parse $default --revs-only \"$@\") || exit $?\n+\tdirs=\"$(git rev-parse --no-revs --no-flags \"$@\")\" || exit $?\n+\tif [ -n \"$dirs\" ]; then\n+\t\tdie \"Error: Use --prefix instead of bare filenames.\"\n+\tfi\n+fi\n+\n+debug \"command: {$command}\"\n+debug \"quiet: {$quiet}\"\n+debug \"revs: {$revs}\"\n+debug \"dir: {$dir}\"\n+debug \"opts: {$*}\"\n+debug\n+\n+cache_setup()\n+{\n+\tcachedir=\"$GIT_DIR/subtree-cache/$$\"\n+\trm -rf \"$cachedir\" || die \"Can't delete old cachedir: $cachedir\"\n+\tmkdir -p \"$cachedir\" || die \"Can't create new cachedir: $cachedir\"\n+\tdebug \"Using cachedir: $cachedir\" >&2\n+}\n+\n+cache_get()\n+{\n+\tfor oldrev in $*; do\n+\t\tif [ -r \"$cachedir/$oldrev\" ]; then\n+\t\t\tread newrev <\"$cachedir/$oldrev\"\n+\t\t\techo $newrev\n+\t\tfi\n+\tdone\n+}\n+\n+cache_set()\n+{\n+\toldrev=\"$1\"\n+\tnewrev=\"$2\"\n+\tif [ \"$oldrev\" != \"latest_old\" \\\n+\t     -a \"$oldrev\" != \"latest_new\" \\\n+\t     -a -e \"$cachedir/$oldrev\" ]; then\n+\t\tdie \"cache for $oldrev already exists!\"\n+\tfi\n+\techo \"$newrev\" >\"$cachedir/$oldrev\"\n+}\n+\n+# if a commit doesn't have a parent, this might not work.  But we only want\n+# to remove the parent from the rev-list, and since it doesn't exist, it won't\n+# be there anyway, so do nothing in that case.\n+try_remove_previous()\n+{\n+\tif git rev-parse \"$1^\" >/dev/null 2>&1; then\n+\t\techo \"^$1^\"\n+\tfi\n+}\n+\n+find_existing_splits()\n+{\n+\tdebug \"Looking for prior splits...\"\n+\tdir=\"$1\"\n+\trevs=\"$2\"\n+\tgit log --grep=\"^git-subtree-dir: $dir\\$\" \\\n+\t\t--pretty=format:'%s%n%n%b%nEND' $revs |\n+\twhile read a b junk; do\n+\t\tcase \"$a\" in\n+\t\t\tgit-subtree-mainline:) main=\"$b\" ;;\n+\t\t\tgit-subtree-split:) sub=\"$b\" ;;\n+\t\t\t*)\n+\t\t\t\tif [ -n \"$main\" -a -n \"$sub\" ]; then\n+\t\t\t\t\tdebug \"  Prior: $main -> $sub\"\n+\t\t\t\t\tcache_set $main $sub\n+\t\t\t\t\ttry_remove_previous \"$main\"\n+\t\t\t\t\ttry_remove_previous \"$sub\"\n+\t\t\t\t\tmain=\n+\t\t\t\t\tsub=\n+\t\t\t\tfi\n+\t\t\t\t;;\n+\t\tesac\n+\tdone\n+}\n+\n+copy_commit()\n+{\n+\t# We're doing to set some environment vars here, so\n+\t# do it in a subshell to get rid of them safely later\n+\tdebug copy_commit \"{$1}\" \"{$2}\" \"{$3}\"\n+\tgit log -1 --pretty=format:'%an%n%ae%n%ad%n%cn%n%ce%n%cd%n%s%n%n%b' \"$1\" |\n+\t(\n+\t\tread GIT_AUTHOR_NAME\n+\t\tread GIT_AUTHOR_EMAIL\n+\t\tread GIT_AUTHOR_DATE\n+\t\tread GIT_COMMITTER_NAME\n+\t\tread GIT_COMMITTER_EMAIL\n+\t\tread GIT_COMMITTER_DATE\n+\t\texport  GIT_AUTHOR_NAME \\\n+\t\t\tGIT_AUTHOR_EMAIL \\\n+\t\t\tGIT_AUTHOR_DATE \\\n+\t\t\tGIT_COMMITTER_NAME \\\n+\t\t\tGIT_COMMITTER_EMAIL \\\n+\t\t\tGIT_COMMITTER_DATE\n+\t\t(echo -n \"$annotate\"; cat ) |\n+\t\tgit commit-tree \"$2\" $3  # reads the rest of stdin\n+\t) || die \"Can't copy commit $1\"\n+}\n+\n+add_msg()\n+{\n+\tdir=\"$1\"\n+\tlatest_old=\"$2\"\n+\tlatest_new=\"$3\"\n+\tcat <<-EOF\n+\t\tAdd '$dir/' from commit '$latest_new'\n+\t\t\n+\t\tgit-subtree-dir: $dir\n+\t\tgit-subtree-mainline: $latest_old\n+\t\tgit-subtree-split: $latest_new\n+\tEOF\n+}\n+\n+merge_msg()\n+{\n+\tdir=\"$1\"\n+\tlatest_old=\"$2\"\n+\tlatest_new=\"$3\"\n+\tcat <<-EOF\n+\t\tSplit '$dir/' into commit '$latest_new'\n+\t\t\n+\t\tgit-subtree-dir: $dir\n+\t\tgit-subtree-mainline: $latest_old\n+\t\tgit-subtree-split: $latest_new\n+\tEOF\n+}\n+\n+toptree_for_commit()\n+{\n+\tcommit=\"$1\"\n+\tgit log -1 --pretty=format:'%T' \"$commit\" -- || exit $?\n+}\n+\n+subtree_for_commit()\n+{\n+\tcommit=\"$1\"\n+\tdir=\"$2\"\n+\tgit ls-tree \"$commit\" -- \"$dir\" |\n+\twhile read mode type tree name; do\n+\t\tassert [ \"$name\" = \"$dir\" ]\n+\t\techo $tree\n+\t\tbreak\n+\tdone\n+}\n+\n+tree_changed()\n+{\n+\ttree=$1\n+\tshift\n+\tif [ $# -ne 1 ]; then\n+\t\treturn 0   # weird parents, consider it changed\n+\telse\n+\t\tptree=$(toptree_for_commit $1)\n+\t\tif [ \"$ptree\" != \"$tree\" ]; then\n+\t\t\treturn 0   # changed\n+\t\telse\n+\t\t\treturn 1   # not changed\n+\t\tfi\n+\tfi\n+}\n+\n+copy_or_skip()\n+{\n+\trev=\"$1\"\n+\ttree=\"$2\"\n+\tnewparents=\"$3\"\n+\tassert [ -n \"$tree\" ]\n+\n+\tidentical=\n+\tnonidentical=\n+\tp=\n+\tgotparents=\n+\tfor parent in $newparents; do\n+\t\tptree=$(toptree_for_commit $parent) || exit $?\n+\t\t[ -z \"$ptree\" ] && continue\n+\t\tif [ \"$ptree\" = \"$tree\" ]; then\n+\t\t\t# an identical parent could be used in place of this rev.\n+\t\t\tidentical=\"$parent\"\n+\t\telse\n+\t\t\tnonidentical=\"$parent\"\n+\t\tfi\n+\t\t\n+\t\t# sometimes both old parents map to the same newparent;\n+\t\t# eliminate duplicates\n+\t\tis_new=1\n+\t\tfor gp in $gotparents; do\n+\t\t\tif [ \"$gp\" = \"$parent\" ]; then\n+\t\t\t\tis_new=\n+\t\t\t\tbreak\n+\t\t\tfi\n+\t\tdone\n+\t\tif [ -n \"$is_new\" ]; then\n+\t\t\tgotparents=\"$gotparents $parent\"\n+\t\t\tp=\"$p -p $parent\"\n+\t\tfi\n+\tdone\n+\t\n+\tif [ -n \"$identical\" ]; then\n+\t\techo $identical\n+\telse\n+\t\tcopy_commit $rev $tree \"$p\" || exit $?\n+\tfi\n+}\n+\n+ensure_clean()\n+{\n+\tif ! git diff-index HEAD --exit-code --quiet; then\n+\t\tdie \"Working tree has modifications.  Cannot add.\"\n+\tfi\n+\tif ! git diff-index --cached HEAD --exit-code --quiet; then\n+\t\tdie \"Index has modifications.  Cannot add.\"\n+\tfi\n+}\n+\n+cmd_add()\n+{\n+\tif [ -e \"$dir\" ]; then\n+\t\tdie \"'$dir' already exists.  Cannot add.\"\n+\tfi\n+\tensure_clean\n+\t\n+\tset -- $revs\n+\tif [ $# -ne 1 ]; then\n+\t\tdie \"You must provide exactly one revision.  Got: '$revs'\"\n+\tfi\n+\trev=\"$1\"\n+\t\n+\tdebug \"Adding $dir as '$rev'...\"\n+\tgit read-tree --prefix=\"$dir\" $rev || exit $?\n+\tgit checkout \"$dir\" || exit $?\n+\ttree=$(git write-tree) || exit $?\n+\t\n+\theadrev=$(git rev-parse HEAD) || exit $?\n+\tif [ -n \"$headrev\" -a \"$headrev\" != \"$rev\" ]; then\n+\t\theadp=\"-p $headrev\"\n+\telse\n+\t\theadp=\n+\tfi\n+\tcommit=$(add_msg \"$dir\" \"$headrev\" \"$rev\" |\n+\t\t git commit-tree $tree $headp -p \"$rev\") || exit $?\n+\tgit reset \"$commit\" || exit $?\n+}\n+\n+cmd_split()\n+{\n+\tdebug \"Splitting $dir...\"\n+\tcache_setup || exit $?\n+\t\n+\tif [ -n \"$onto\" ]; then\n+\t\tdebug \"Reading history for --onto=$onto...\"\n+\t\tgit rev-list $onto |\n+\t\twhile read rev; do\n+\t\t\t# the 'onto' history is already just the subdir, so\n+\t\t\t# any parent we find there can be used verbatim\n+\t\t\tdebug \"  cache: $rev\"\n+\t\t\tcache_set $rev $rev\n+\t\tdone\n+\tfi\n+\t\n+\tif [ -n \"$ignore_joins\" ]; then\n+\t\tunrevs=\n+\telse\n+\t\tunrevs=\"$(find_existing_splits \"$dir\" \"$revs\")\"\n+\tfi\n+\t\n+\t# We can't restrict rev-list to only $dir here, because some of our\n+\t# parents have the $dir contents the root, and those won't match.\n+\t# (and rev-list --follow doesn't seem to solve this)\n+\tgrl='git rev-list --reverse --parents $revs $unrevs'\n+\trevmax=$(eval \"$grl\" | wc -l)\n+\trevcount=0\n+\tcreatecount=0\n+\teval \"$grl\" |\n+\twhile read rev parents; do\n+\t\trevcount=$(($revcount + 1))\n+\t\tsay -n \"$revcount/$revmax ($createcount)\n\"\n+\t\tdebug \"Processing commit: $rev\"\n+\t\texists=$(cache_get $rev)\n+\t\tif [ -n \"$exists\" ]; then\n+\t\t\tdebug \"  prior: $exists\"\n+\t\t\tcontinue\n+\t\tfi\n+\t\tcreatecount=$(($createcount + 1))\n+\t\tdebug \"  parents: $parents\"\n+\t\tnewparents=$(cache_get $parents)\n+\t\tdebug \"  newparents: $newparents\"\n+\t\t\n+\t\ttree=$(subtree_for_commit $rev \"$dir\")\n+\t\tdebug \"  tree is: $tree\"\n+\t\t[ -z $tree ] && continue\n+\n+\t\tnewrev=$(copy_or_skip \"$rev\" \"$tree\" \"$newparents\") || exit $?\n+\t\tdebug \"  newrev is: $newrev\"\n+\t\tcache_set $rev $newrev\n+\t\tcache_set latest_new $newrev\n+\t\tcache_set latest_old $rev\n+\tdone || exit $?\n+\tlatest_new=$(cache_get latest_new)\n+\tif [ -z \"$latest_new\" ]; then\n+\t\tdie \"No new revisions were found\"\n+\tfi\n+\t\n+\tif [ -n \"$rejoin\" ]; then\n+\t\tdebug \"Merging split branch into HEAD...\"\n+\t\tlatest_old=$(cache_get latest_old)\n+\t\tgit merge -s ours \\\n+\t\t\t-m \"$(merge_msg $dir $latest_old $latest_new)\" \\\n+\t\t\t$latest_new >&2\n+\tfi\n+\techo $latest_new\n+\texit 0\n+}\n+\n+cmd_merge()\n+{\n+\tensure_clean\n+\t\n+\tset -- $revs\n+\tif [ $# -ne 1 ]; then\n+\t\tdie \"You must provide exactly one revision.  Got: '$revs'\"\n+\tfi\n+\trev=\"$1\"\n+\t\n+\tgit merge -s subtree $rev\n+}\n+\n+cmd_pull()\n+{\n+\tensure_clean\n+\tset -x\n+\tgit pull -s subtree \"$@\"\n+}\n+\n+\"cmd_$command\" \"$@\"\n-- \n1.6.3.rc2.8.gbe66.dirty\n"},{"id":"112359","messageId":"1240784983-1477-2-git-send-email-apenwarr@gmail.com","threadId":"19071","inReplyTo":"1240784983-1477-1-git-send-email-apenwarr@gmail.com","subject":"[PATCH/RFC 2/2] Automated test script for 'git subtree'.","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2009-04-26T22:29:43Z","receivedAt":"2009-04-26T22:29:43Z","isPatch":true,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"TEMPORARY: this script hasn't yet been integrated into the main git unit\ntests; it runs standalone for the moment.\n---\n subtree-test.sh |  206 +++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 206 insertions(+), 0 deletions(-)\n create mode 100755 subtree-test.sh\n\ndiff --git a/subtree-test.sh b/subtree-test.sh\nnew file mode 100755\nindex 0000000..38dff7a\n--- /dev/null\n+++ b/subtree-test.sh\n@@ -0,0 +1,206 @@\n+#!/bin/bash\n+. shellopts.sh\n+set -e\n+\n+create()\n+{\n+\techo \"$1\" >\"$1\"\n+\tgit add \"$1\"\n+}\n+\n+check()\n+{\n+\techo\n+\techo \"check:\" \"$@\"\n+\tif \"$@\"; then\n+\t\techo ok\n+\t\treturn 0\n+\telse\n+\t\techo FAILED\n+\t\texit 1\n+\tfi\n+}\n+\n+check_equal()\n+{\n+\techo\n+\techo \"check a:\" \"{$1}\"\n+\techo \"      b:\" \"{$2}\"\n+\tif [ \"$1\" = \"$2\" ]; then\n+\t\treturn 0\n+\telse\n+\t\techo FAILED\n+\t\texit 1\n+\tfi\n+}\n+\n+fixnl()\n+{\t\n+\tt=\"\"\n+\twhile read x; do\n+\t\tt=\"$t$x \"\n+\tdone\n+\techo $t\n+}\n+\n+multiline()\n+{\n+\twhile read x; do\n+\t\tset -- $x\n+\t\tfor d in \"$@\"; do\n+\t\t\techo \"$d\"\n+\t\tdone\n+\tdone\n+}\n+\n+rm -rf mainline subproj\n+mkdir mainline subproj\n+\n+cd subproj\n+git init\n+\n+create sub1\n+git commit -m 'sub1'\n+git branch sub1\n+git branch -m master subproj\n+check true\n+\n+create sub2\n+git commit -m 'sub2'\n+git branch sub2\n+\n+create sub3\n+git commit -m 'sub3'\n+git branch sub3\n+\n+cd ../mainline\n+git init\n+create main4\n+git commit -m 'main4'\n+git branch -m master mainline\n+\n+git fetch ../subproj sub1\n+git branch sub1 FETCH_HEAD\n+git subtree add --prefix=subdir FETCH_HEAD\n+\n+# this shouldn't actually do anything, since FETCH_HEAD is already a parent\n+git merge -m 'merge -s -ours' -s ours FETCH_HEAD\n+\n+create subdir/main-sub5\n+git commit -m 'main-sub5'\n+\n+create main6\n+git commit -m 'main6 boring'\n+\n+create subdir/main-sub7\n+git commit -m 'main-sub7'\n+\n+git fetch ../subproj sub2\n+git branch sub2 FETCH_HEAD\n+git subtree merge --prefix=subdir FETCH_HEAD\n+git branch pre-split\n+\n+spl1=$(git subtree split --annotate='*' \\\n+\t\t--prefix subdir --onto FETCH_HEAD --rejoin)\n+echo \"spl1={$spl1}\"\n+git branch spl1 \"$spl1\"\n+\n+create subdir/main-sub8\n+git commit -m 'main-sub8'\n+\n+cd ../subproj\n+git fetch ../mainline spl1\n+git branch spl1 FETCH_HEAD\n+git merge FETCH_HEAD\n+\n+create sub9\n+git commit -m 'sub9'\n+\n+cd ../mainline\n+split2=$(git subtree split --annotate='*' --prefix subdir --rejoin)\n+git branch split2 \"$split2\"\n+\n+create subdir/main-sub10\n+git commit -m 'main-sub10'\n+\n+spl3=$(git subtree split --annotate='*' --prefix subdir --rejoin)\n+git branch spl3 \"$spl3\"\n+\n+cd ../subproj\n+git fetch ../mainline spl3\n+git branch spl3 FETCH_HEAD\n+git merge FETCH_HEAD\n+git branch subproj-merge-spl3\n+\n+chkm=\"main4 main6\"\n+chkms=\"main-sub10 main-sub5 main-sub7 main-sub8\"\n+chkms_sub=$(echo $chkms | multiline | sed 's,^,subdir/,' | fixnl)\n+chks=\"sub1 sub2 sub3 sub9\"\n+chks_sub=$(echo $chks | multiline | sed 's,^,subdir/,' | fixnl)\n+\n+# make sure exactly the right set of files ends up in the subproj\n+subfiles=$(git ls-files | fixnl)\n+check_equal \"$subfiles\" \"$chkms $chks\"\n+\n+# make sure the subproj history *only* contains commits that affect the subdir.\n+allchanges=$(git log --name-only --pretty=format:'' | sort | fixnl)\n+check_equal \"$allchanges\" \"$chkms $chks\"\n+\n+cd ../mainline\n+git fetch ../subproj subproj-merge-spl3\n+git branch subproj-merge-spl3 FETCH_HEAD\n+git subtree pull --prefix=subdir ../subproj subproj-merge-spl3\n+\n+# make sure exactly the right set of files ends up in the mainline\n+mainfiles=$(git ls-files | fixnl)\n+check_equal \"$mainfiles\" \"$chkm $chkms_sub $chks_sub\"\n+\n+# make sure each filename changed exactly once in the entire history.\n+# 'main-sub??' and '/subdir/main-sub??' both change, because those are the\n+# changes that were split into their own history.  And 'subdir/sub??' never\n+# change, since they were *only* changed in the subtree branch.\n+allchanges=$(git log --name-only --pretty=format:'' | sort | fixnl)\n+check_equal \"$allchanges\" \"$chkm $chkms $chks $chkms_sub\"\n+\n+# make sure the --rejoin commits never make it into subproj\n+check_equal \"$(git log --pretty=format:'%s' HEAD^2 | grep -i split)\" \"\"\n+\n+# make sure no 'git subtree' tagged commits make it into subproj. (They're\n+# meaningless to subproj since one side of the merge refers to the mainline)\n+check_equal \"$(git log --pretty=format:'%s%n%b' HEAD^2 | grep 'git-subtree.*:')\" \"\"\n+\n+# make sure no patch changes more than one file.  The original set of commits\n+# changed only one file each.  A multi-file change would imply that we pruned\n+# commits too aggressively.\n+joincommits()\n+{\n+\tcommit=\n+\tall=\n+\twhile read x y; do\n+\t\techo \"{$x}\" >&2\n+\t\tif [ -z \"$x\" ]; then\n+\t\t\tcontinue\n+\t\telif [ \"$x\" = \"commit:\" ]; then\n+\t\t\tif [ -n \"$commit\" ]; then\n+\t\t\t\techo \"$commit $all\"\n+\t\t\t\tall=\n+\t\t\tfi\n+\t\t\tcommit=\"$y\"\n+\t\telse\n+\t\t\tall=\"$all $y\"\n+\t\tfi\n+\tdone\n+\techo \"$commit $all\"\n+}\n+x=\n+git log --pretty=format:'commit: %H' | joincommits |\n+(\twhile read commit a b; do\n+\t\techo \"Verifying commit $commit\"\n+\t\tcheck_equal \"$b\" \"\"\n+\t\tx=1\n+\tdone\n+\tcheck_equal \"$x\" 1\n+) || exit 1\n+\n+echo\n+echo 'ok'\n-- \n1.6.3.rc2.8.gbe66.dirty\n"},{"id":"112730","messageId":"32541b130904291927m33908bacg2dbafcf64877b88f@mail.gmail.com","threadId":"19071","inReplyTo":"1240784983-1477-1-git-send-email-apenwarr@gmail.com","subject":"Re: [PATCH/RFC 1/2] Add 'git subtree' command for tracking history of subtrees separately.","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2009-04-30T02:27:44Z","receivedAt":"2009-04-30T02:27:44Z","isPatch":true,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"Many projects are made of a combination of several subprojects/libraries and\nsome application-specific code.  In some cases, particularly when the\nsubprojects are all maintained independently, 'git submodule' is the best\nway to deal with this situation.  But if you frequently change the\nsubprojects as part of developing your application, use multiple branches,\nand sometimes want to push your subproject changes upstream, the overhead of\nmanually managing submodules can be excessive.\n\n'git subtree' provides an alternative mechanism, based around the\n'git merge -s subtree' merge strategy.  Instead of tracking a submodule\nseparately, you merge its history into your main project, and occasionally\nextract a new \"virtual history\" from your mainline that can be easily merged\nback into the upstream project.  The virtual history can be incrementally\nexpanded as you make more changes to the superproject.\n\nYou would normally then merge the virtual history back into your mainline\n(the --rejoin option). This results in extra commits in your application\nthat appear to change the same files, but these extra commits will tend to\nbe ignored by git's merge simplification algorithm anyway.\n\nFor example, gitweb (commit 1130ef3) was merged into git as of commit\n0a8f4f0, after which it was no longer maintained separately.  But imagine it\nhad been maintained separately, and we wanted to extract git's changes to\ngitweb since that time, to share with the upstream.  You could do this:\n\ngit subtree split --prefix=gitweb --annotate='(split) ' \\\n       0a8f4f0^.. --onto=1130ef3 --rejoin\n\nIf gitweb had originally been merged using 'git subtree add' (or a previous\nsplit had been done with --rejoin specified), then you could incrementally\nproduce the list of new changes without needing to remember any commit ids:\n\ngit subtree split --prefix=gitweb --annotate='(split) ' --rejoin\n---\n\nResending just in case it got lost.\n\nDon't suppose anyone has any comments on this?  It's just a first\ndraft, so please let loose.  I'm in desperate need of *some* kind of\nsolution like this, however.\n\n- Avery\n\n\n Makefile         |    1 +\n command-list.txt |    1 +\n git-subtree.sh   |  435 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 437 insertions(+), 0 deletions(-)\n create mode 100755 git-subtree.sh\n\ndiff --git a/Makefile b/Makefile\nindex 5c8e83a..f14e11c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -305,6 +305,7 @@ SCRIPT_SH += git-request-pull.sh\n SCRIPT_SH += git-sh-setup.sh\n SCRIPT_SH += git-stash.sh\n SCRIPT_SH += git-submodule.sh\n+SCRIPT_SH += git-subtree.sh\n SCRIPT_SH += git-web--browse.sh\n\n SCRIPT_PERL += git-add--interactive.perl\ndiff --git a/command-list.txt b/command-list.txt\nindex fb03a2e..9be4774 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -113,6 +113,7 @@ git-stash                               mainporcelain\n git-status                              mainporcelain common\n git-stripspace                          purehelpers\n git-submodule                           mainporcelain\n+git-subtree                             mainporcelain\n git-svn                                 foreignscminterface\n git-symbolic-ref                        plumbingmanipulators\n git-tag                                 mainporcelain common\ndiff --git a/git-subtree.sh b/git-subtree.sh\nnew file mode 100755\nindex 0000000..39c377c\n--- /dev/null\n+++ b/git-subtree.sh\n@@ -0,0 +1,435 @@\n+#!/bin/bash\n+#\n+# git-subtree.sh: split/join git repositories in subdirectories of this one\n+#\n+# Copyright (C) 2009 Avery Pennarun <apenwarr@gmail.com>\n+#\n+if [ $# -eq 0 ]; then\n+    set -- -h\n+fi\n+OPTS_SPEC=\"\\\n+git subtree add --prefix=<prefix> <commit>\n+git subtree split [options...] --prefix=<prefix> <commit...>\n+git subtree merge --prefix=<prefix> <commit>\n+git subtree pull  --prefix=<prefix> <repository> <refspec...>\n+--\n+h,help        show the help\n+q             quiet\n+d             show debug messages\n+prefix=       the name of the subdir to split out\n+ options for 'split'\n+annotate=     add a prefix to commit message of new commits\n+onto=         try connecting new tree to an existing one\n+rejoin        merge the new branch back into HEAD\n+ignore-joins  ignore prior --rejoin commits\n+\"\n+eval $(echo \"$OPTS_SPEC\" | git rev-parse --parseopt -- \"$@\" || echo exit $?)\n+. git-sh-setup\n+require_work_tree\n+\n+quiet=\n+debug=\n+command=\n+onto=\n+rejoin=\n+ignore_joins=\n+annotate=\n+\n+debug()\n+{\n+       if [ -n \"$debug\" ]; then\n+               echo \"$@\" >&2\n+       fi\n+}\n+\n+say()\n+{\n+       if [ -z \"$quiet\" ]; then\n+               echo \"$@\" >&2\n+       fi\n+}\n+\n+assert()\n+{\n+       if \"$@\"; then\n+               :\n+       else\n+               die \"assertion failed: \" \"$@\"\n+       fi\n+}\n+\n+\n+#echo \"Options: $*\"\n+\n+while [ $# -gt 0 ]; do\n+       opt=\"$1\"\n+       shift\n+       case \"$opt\" in\n+               -q) quiet=1 ;;\n+               -d) debug=1 ;;\n+               --annotate) annotate=\"$1\"; shift ;;\n+               --no-annotate) annotate= ;;\n+               --prefix) prefix=\"$1\"; shift ;;\n+               --no-prefix) prefix= ;;\n+               --onto) onto=\"$1\"; shift ;;\n+               --no-onto) onto= ;;\n+               --rejoin) rejoin=1 ;;\n+               --no-rejoin) rejoin= ;;\n+               --ignore-joins) ignore_joins=1 ;;\n+               --no-ignore-joins) ignore_joins= ;;\n+               --) break ;;\n+       esac\n+done\n+\n+command=\"$1\"\n+shift\n+case \"$command\" in\n+       add|merge|pull) default= ;;\n+       split) default=\"--default HEAD\" ;;\n+       *) die \"Unknown command '$command'\" ;;\n+esac\n+\n+if [ -z \"$prefix\" ]; then\n+       die \"You must provide the --prefix option.\"\n+fi\n+dir=\"$prefix\"\n+\n+if [ \"$command\" != \"pull\" ]; then\n+       revs=$(git rev-parse $default --revs-only \"$@\") || exit $?\n+       dirs=\"$(git rev-parse --no-revs --no-flags \"$@\")\" || exit $?\n+       if [ -n \"$dirs\" ]; then\n+               die \"Error: Use --prefix instead of bare filenames.\"\n+       fi\n+fi\n+\n+debug \"command: {$command}\"\n+debug \"quiet: {$quiet}\"\n+debug \"revs: {$revs}\"\n+debug \"dir: {$dir}\"\n+debug \"opts: {$*}\"\n+debug\n+\n+cache_setup()\n+{\n+       cachedir=\"$GIT_DIR/subtree-cache/$$\"\n+       rm -rf \"$cachedir\" || die \"Can't delete old cachedir: $cachedir\"\n+       mkdir -p \"$cachedir\" || die \"Can't create new cachedir: $cachedir\"\n+       debug \"Using cachedir: $cachedir\" >&2\n+}\n+\n+cache_get()\n+{\n+       for oldrev in $*; do\n+               if [ -r \"$cachedir/$oldrev\" ]; then\n+                       read newrev <\"$cachedir/$oldrev\"\n+                       echo $newrev\n+               fi\n+       done\n+}\n+\n+cache_set()\n+{\n+       oldrev=\"$1\"\n+       newrev=\"$2\"\n+       if [ \"$oldrev\" != \"latest_old\" \\\n+            -a \"$oldrev\" != \"latest_new\" \\\n+            -a -e \"$cachedir/$oldrev\" ]; then\n+               die \"cache for $oldrev already exists!\"\n+       fi\n+       echo \"$newrev\" >\"$cachedir/$oldrev\"\n+}\n+\n+# if a commit doesn't have a parent, this might not work.  But we only want\n+# to remove the parent from the rev-list, and since it doesn't exist, it won't\n+# be there anyway, so do nothing in that case.\n+try_remove_previous()\n+{\n+       if git rev-parse \"$1^\" >/dev/null 2>&1; then\n+               echo \"^$1^\"\n+       fi\n+}\n+\n+find_existing_splits()\n+{\n+       debug \"Looking for prior splits...\"\n+       dir=\"$1\"\n+       revs=\"$2\"\n+       git log --grep=\"^git-subtree-dir: $dir\\$\" \\\n+               --pretty=format:'%s%n%n%b%nEND' $revs |\n+       while read a b junk; do\n+               case \"$a\" in\n+                       git-subtree-mainline:) main=\"$b\" ;;\n+                       git-subtree-split:) sub=\"$b\" ;;\n+                       *)\n+                               if [ -n \"$main\" -a -n \"$sub\" ]; then\n+                                       debug \"  Prior: $main -> $sub\"\n+                                       cache_set $main $sub\n+                                       try_remove_previous \"$main\"\n+                                       try_remove_previous \"$sub\"\n+                                       main=\n+                                       sub=\n+                               fi\n+                               ;;\n+               esac\n+       done\n+}\n+\n+copy_commit()\n+{\n+       # We're doing to set some environment vars here, so\n+       # do it in a subshell to get rid of them safely later\n+       debug copy_commit \"{$1}\" \"{$2}\" \"{$3}\"\n+       git log -1\n--pretty=format:'%an%n%ae%n%ad%n%cn%n%ce%n%cd%n%s%n%n%b' \"$1\" |\n+       (\n+               read GIT_AUTHOR_NAME\n+               read GIT_AUTHOR_EMAIL\n+               read GIT_AUTHOR_DATE\n+               read GIT_COMMITTER_NAME\n+               read GIT_COMMITTER_EMAIL\n+               read GIT_COMMITTER_DATE\n+               export  GIT_AUTHOR_NAME \\\n+                       GIT_AUTHOR_EMAIL \\\n+                       GIT_AUTHOR_DATE \\\n+                       GIT_COMMITTER_NAME \\\n+                       GIT_COMMITTER_EMAIL \\\n+                       GIT_COMMITTER_DATE\n+               (echo -n \"$annotate\"; cat ) |\n+               git commit-tree \"$2\" $3  # reads the rest of stdin\n+       ) || die \"Can't copy commit $1\"\n+}\n+\n+add_msg()\n+{\n+       dir=\"$1\"\n+       latest_old=\"$2\"\n+       latest_new=\"$3\"\n+       cat <<-EOF\n+               Add '$dir/' from commit '$latest_new'\n+\n+               git-subtree-dir: $dir\n+               git-subtree-mainline: $latest_old\n+               git-subtree-split: $latest_new\n+       EOF\n+}\n+\n+merge_msg()\n+{\n+       dir=\"$1\"\n+       latest_old=\"$2\"\n+       latest_new=\"$3\"\n+       cat <<-EOF\n+               Split '$dir/' into commit '$latest_new'\n+\n+               git-subtree-dir: $dir\n+               git-subtree-mainline: $latest_old\n+               git-subtree-split: $latest_new\n+       EOF\n+}\n+\n+toptree_for_commit()\n+{\n+       commit=\"$1\"\n+       git log -1 --pretty=format:'%T' \"$commit\" -- || exit $?\n+}\n+\n+subtree_for_commit()\n+{\n+       commit=\"$1\"\n+       dir=\"$2\"\n+       git ls-tree \"$commit\" -- \"$dir\" |\n+       while read mode type tree name; do\n+               assert [ \"$name\" = \"$dir\" ]\n+               echo $tree\n+               break\n+       done\n+}\n+\n+tree_changed()\n+{\n+       tree=$1\n+       shift\n+       if [ $# -ne 1 ]; then\n+               return 0   # weird parents, consider it changed\n+       else\n+               ptree=$(toptree_for_commit $1)\n+               if [ \"$ptree\" != \"$tree\" ]; then\n+                       return 0   # changed\n+               else\n+                       return 1   # not changed\n+               fi\n+       fi\n+}\n+\n+copy_or_skip()\n+{\n+       rev=\"$1\"\n+       tree=\"$2\"\n+       newparents=\"$3\"\n+       assert [ -n \"$tree\" ]\n+\n+       identical=\n+       nonidentical=\n+       p=\n+       gotparents=\n+       for parent in $newparents; do\n+               ptree=$(toptree_for_commit $parent) || exit $?\n+               [ -z \"$ptree\" ] && continue\n+               if [ \"$ptree\" = \"$tree\" ]; then\n+                       # an identical parent could be used in place\nof this rev.\n+                       identical=\"$parent\"\n+               else\n+                       nonidentical=\"$parent\"\n+               fi\n+\n+               # sometimes both old parents map to the same newparent;\n+               # eliminate duplicates\n+               is_new=1\n+               for gp in $gotparents; do\n+                       if [ \"$gp\" = \"$parent\" ]; then\n+                               is_new=\n+                               break\n+                       fi\n+               done\n+               if [ -n \"$is_new\" ]; then\n+                       gotparents=\"$gotparents $parent\"\n+                       p=\"$p -p $parent\"\n+               fi\n+       done\n+\n+       if [ -n \"$identical\" ]; then\n+               echo $identical\n+       else\n+               copy_commit $rev $tree \"$p\" || exit $?\n+       fi\n+}\n+\n+ensure_clean()\n+{\n+       if ! git diff-index HEAD --exit-code --quiet; then\n+               die \"Working tree has modifications.  Cannot add.\"\n+       fi\n+       if ! git diff-index --cached HEAD --exit-code --quiet; then\n+               die \"Index has modifications.  Cannot add.\"\n+       fi\n+}\n+\n+cmd_add()\n+{\n+       if [ -e \"$dir\" ]; then\n+               die \"'$dir' already exists.  Cannot add.\"\n+       fi\n+       ensure_clean\n+\n+       set -- $revs\n+       if [ $# -ne 1 ]; then\n+               die \"You must provide exactly one revision.  Got: '$revs'\"\n+       fi\n+       rev=\"$1\"\n+\n+       debug \"Adding $dir as '$rev'...\"\n+       git read-tree --prefix=\"$dir\" $rev || exit $?\n+       git checkout \"$dir\" || exit $?\n+       tree=$(git write-tree) || exit $?\n+\n+       headrev=$(git rev-parse HEAD) || exit $?\n+       if [ -n \"$headrev\" -a \"$headrev\" != \"$rev\" ]; then\n+               headp=\"-p $headrev\"\n+       else\n+               headp=\n+       fi\n+       commit=$(add_msg \"$dir\" \"$headrev\" \"$rev\" |\n+                git commit-tree $tree $headp -p \"$rev\") || exit $?\n+       git reset \"$commit\" || exit $?\n+}\n+\n+cmd_split()\n+{\n+       debug \"Splitting $dir...\"\n+       cache_setup || exit $?\n+\n+       if [ -n \"$onto\" ]; then\n+               debug \"Reading history for --onto=$onto...\"\n+               git rev-list $onto |\n+               while read rev; do\n+                       # the 'onto' history is already just the subdir, so\n+                       # any parent we find there can be used verbatim\n+                       debug \"  cache: $rev\"\n+                       cache_set $rev $rev\n+               done\n+       fi\n+\n+       if [ -n \"$ignore_joins\" ]; then\n+               unrevs=\n+       else\n+               unrevs=\"$(find_existing_splits \"$dir\" \"$revs\")\"\n+       fi\n+\n+       # We can't restrict rev-list to only $dir here, because some of our\n+       # parents have the $dir contents the root, and those won't match.\n+       # (and rev-list --follow doesn't seem to solve this)\n+       grl='git rev-list --reverse --parents $revs $unrevs'\n+       revmax=$(eval \"$grl\" | wc -l)\n+       revcount=0\n+       createcount=0\n+       eval \"$grl\" |\n+       while read rev parents; do\n+               revcount=$(($revcount + 1))\n+               say -n \"$revcount/$revmax ($createcount)\n\"\n+               debug \"Processing commit: $rev\"\n+               exists=$(cache_get $rev)\n+               if [ -n \"$exists\" ]; then\n+                       debug \"  prior: $exists\"\n+                       continue\n+               fi\n+               createcount=$(($createcount + 1))\n+               debug \"  parents: $parents\"\n+               newparents=$(cache_get $parents)\n+               debug \"  newparents: $newparents\"\n+\n+               tree=$(subtree_for_commit $rev \"$dir\")\n+               debug \"  tree is: $tree\"\n+               [ -z $tree ] && continue\n+\n+               newrev=$(copy_or_skip \"$rev\" \"$tree\" \"$newparents\") || exit $?\n+               debug \"  newrev is: $newrev\"\n+               cache_set $rev $newrev\n+               cache_set latest_new $newrev\n+               cache_set latest_old $rev\n+       done || exit $?\n+       latest_new=$(cache_get latest_new)\n+       if [ -z \"$latest_new\" ]; then\n+               die \"No new revisions were found\"\n+       fi\n+\n+       if [ -n \"$rejoin\" ]; then\n+               debug \"Merging split branch into HEAD...\"\n+               latest_old=$(cache_get latest_old)\n+               git merge -s ours \\\n+                       -m \"$(merge_msg $dir $latest_old $latest_new)\" \\\n+                       $latest_new >&2\n+       fi\n+       echo $latest_new\n+       exit 0\n+}\n+\n+cmd_merge()\n+{\n+       ensure_clean\n+\n+       set -- $revs\n+       if [ $# -ne 1 ]; then\n+               die \"You must provide exactly one revision.  Got: '$revs'\"\n+       fi\n+       rev=\"$1\"\n+\n+       git merge -s subtree $rev\n+}\n+\n+cmd_pull()\n+{\n+       ensure_clean\n+       set -x\n+       git pull -s subtree \"$@\"\n+}\n+\n+\"cmd_$command\" \"$@\"\n--\n1.6.3.rc2.8.gbe66.dirty\n"},{"id":"112733","messageId":"46dff0320904292044t54dfe955reed06fd8a9b6e57@mail.gmail.com","threadId":"19071","inReplyTo":"32541b130904291927m33908bacg2dbafcf64877b88f@mail.gmail.com","subject":"Re: [PATCH/RFC 1/2] Add 'git subtree' command for tracking history of subtrees separately.","fromName":"Ping Yin","fromEmail":"pkufranky@gmail.com","sentAt":"2009-04-30T03:44:19Z","receivedAt":"2009-04-30T03:44:19Z","isPatch":true,"sender":{"key":"pkufranky@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5346?v=4"},"body":"> Resending just in case it got lost.\n>\n> Don't suppose anyone has any comments on this?  It's just a first\n> draft, so please let loose.  I'm in desperate need of *some* kind of\n> solution like this, however.\n>\n\nI think it is a nice feature. Although there is a howto to introduce\nmerge as subtree, and i have done it several times, i can't still\nremember the first step. A command like this is helpful for me.\nThanks.\n"},{"id":"112750","messageId":"20090430085853.GA21880@pvv.org","threadId":"19071","inReplyTo":"32541b130904291927m33908bacg2dbafcf64877b88f@mail.gmail.com","subject":"Re: [PATCH/RFC 1/2] Add 'git subtree' command for tracking history of subtrees separately.","fromName":"Finn Arne Gangstad","fromEmail":"finnag@pvv.org","sentAt":"2009-04-30T08:58:53Z","receivedAt":"2009-04-30T08:58:53Z","isPatch":true,"sender":{"key":"finnag@pvv.org","avatar":"https://gravatar.com/avatar/b421ddd58c3f0f93aa473e17b98bb8d53c221fef741746bc8cb59fae4ec6d95e?d=mp&s=160"},"body":"On Wed, Apr 29, 2009 at 10:27:44PM -0400, Avery Pennarun wrote:\n> Many projects are made of a combination of several subprojects/libraries and\n> some application-specific code.  In some cases, particularly when the\n> subprojects are all maintained independently, 'git submodule' is the best\n> way to deal with this situation.  But if you frequently change the\n> subprojects as part of developing your application, use multiple branches,\n> and sometimes want to push your subproject changes upstream, the overhead of\n> manually managing submodules can be excessive.\n> \n> 'git subtree' provides an alternative mechanism, based around the\n> 'git merge -s subtree' merge strategy.  Instead of tracking a submodule\n> separately, you merge its history into your main project, and occasionally\n> extract a new \"virtual history\" from your mainline that can be easily merged\n> back into the upstream project.  The virtual history can be incrementally\n> expanded as you make more changes to the superproject.\n\nWe have the exact same situation. I wanted to attack this from the\nother end though, make submodules useable also in this scenario. The\nsubtree solution seems to be much easier to do in git, so maybe this\nis a better approach!\n\nLet's say you have three different projects that all use some shared\nmodules, The following operations should all be easy and fully\nsupported:\n\na) Modify project + some shared modules (in your project) with single commit\nb) Push project + shared modules (for your project)\nc) Push modifications to shared modules\nd) Merge upstream version of shared modules into your project.\n\nMy quick analysis:\nYour subtrees: a & b are easy, c & d are painful\nCurrent submodules: a & b are painful, c & d are tolerable (somewhat tedious\nwith many shared modules, easy with one)\n\nSubtrees also have the advantage that all the existing local tools\nwill be a lot more useful without any modifications (gitk, git gui,\ngit diff/patch/am/log/...)\n\nTo make subtrees realy useful, it would be good if you could improve c\n& d, syncing with the shared modules!\n\n- Finn Arne\n"},{"id":"112764","messageId":"32541b130904300732i691800f5kecc2f845584071c1@mail.gmail.com","threadId":"19071","inReplyTo":"20090430085853.GA21880@pvv.org","subject":"Re: [PATCH/RFC 1/2] Add 'git subtree' command for tracking history of subtrees separately.","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2009-04-30T14:32:48Z","receivedAt":"2009-04-30T14:32:48Z","isPatch":true,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Thu, Apr 30, 2009 at 4:58 AM, Finn Arne Gangstad <finnag@pvv.org> wrote:\n> On Wed, Apr 29, 2009 at 10:27:44PM -0400, Avery Pennarun wrote:\n>> 'git subtree' provides an alternative mechanism, based around the\n>> 'git merge -s subtree' merge strategy.  Instead of tracking a submodule\n>> separately, you merge its history into your main project, and occasionally\n>> extract a new \"virtual history\" from your mainline that can be easily merged\n>> back into the upstream project.  The virtual history can be incrementally\n>> expanded as you make more changes to the superproject.\n>\n> We have the exact same situation. I wanted to attack this from the\n> other end though, make submodules useable also in this scenario. The\n> subtree solution seems to be much easier to do in git, so maybe this\n> is a better approach!\n\nSounds like your thought process is similar to mine :)  I spent a lot\nof time trying to figure out how to convince submodules to work the\nway I wanted, until I eventually realized that subtrees were already a\nlot closer.\n\n> Let's say you have three different projects that all use some shared\n> modules, The following operations should all be easy and fully\n> supported:\n>\n> a) Modify project + some shared modules (in your project) with single commit\n> b) Push project + shared modules (for your project)\n> c) Push modifications to shared modules\n> d) Merge upstream version of shared modules into your project.\n>\n> My quick analysis:\n> Your subtrees: a & b are easy, c & d are painful\n\nMy *attempt* with git-subtree was to make all four operations easy.\nIt's up to you to decide whether I succeeded :)\n\na) Modify-and-commit: just git commit\n\nb) Push project+shared: just git push\n\nc) Push shared changes only:\n      # Should we try to make a simpler single command for this?\n      # The problem is: I suspect people will normally want to review the\n      # git subtree split output before pushing it anywhere, so combining\n      # the split/push operations may not be wise.\n      git push shared-remote $(git subtree split --prefix=shared-dir):master\n\nd) Merge upstream changes of shared module:\n      git subtree pull --prefix=shared-dir shared-remote master\n    or\n      git fetch shared-remote master\n      git subtree merge --prefix=shared-dir FETCH_HEAD\n\nHave fun,\n\nAvery\n"},{"id":"118124","messageId":"loom.20090716T160021-218@post.gmane.org","threadId":"19071","inReplyTo":"32541b130904300732i691800f5kecc2f845584071c1@mail.gmail.com","subject":"Re: [PATCH/RFC 1/2] Add 'git subtree' command for tracking history of subtrees separately.","fromName":"Andrey Smirnov","fromEmail":"allter@gmail.com","sentAt":"2009-07-16T18:04:26Z","receivedAt":"2009-07-16T18:04:26Z","isPatch":true,"sender":{"key":"allter@gmail.com","avatar":null},"body":"Hello!\n\nAvery Pennarun <apenwarr> writes:\n\n> d) Merge upstream changes of shared module:\n>       git subtree pull --prefix=shared-dir shared-remote master\n>     or\n>       git fetch shared-remote master\n>       git subtree merge --prefix=shared-dir FETCH_HEAD\n\nI found the git-subtree aproach of handling sub-repositories very interesting \nand useful to me. This is the previous-to-last feature I've awaited from DVCS\nworld since I went into it. <remark> The remaining feature I wish that's not \nalready there is the ability to automatically track the tree of repos I work\nwith and manage this tree as simple as filemanager-style clients \nlike Tortoise SVN allow. This is a feature like submodules, but with tracking\nrefs, remotes and remote URIs in a repo itself rather than in .git dirs\nand with different commands for propagation and display of changesets). \nHope someone has time to try this approach (or at least patience to discuss).\n</remark>\n\nI've just used git-subtree (latest github version) and it worked for me. \nHowever I've encountered some difficulty using it for my purpose and wish\nto shere the solution I've come to and ask if it is ok:\n\nMy goal was to rebase changes to shared library of two similar projects from one\nproject to another. The commits in the more recent project were touching both\nlib/ directory with shared library and the rest of the project.\n\nWhen I did \n   git subtree split --prefix=lib NewProj -b test-split\n and\n   git subtree split --prefix=lib OldProj -b test-split-old\nI got the following two trees without a common root:\n\n...X ----- Y ----- OldProj ----...---- Z ---- NewProj\n\nX' ----- Y'==test-split-old ----- Z'==test-split\n\nProblem:\n\nWhen I did\n   git subtree merge test-split --prefix=lib\nit printed:\n Auto-merged lib/x.cgi\n CONFLICT (add/add): Merge conflict in lib/x.cgi\n Auto-merged lib/y.cgi\n CONFLICT (add/add): Merge conflict in lib/y.cgi\n Automatic merge failed; fix conflicts and then commit the result.\n\nIt's obvious that it should be that way because logically both trees don't have\nthe same root at the time of merge. But I've expected subtree merge --prefix\nwill understand that X' is identical to changes to 'lib/*' in X, Y' to Y and Z'\nto Z.\n\nSolution: \n\n    git rebase --onto OldProj test-split-old test-split\nit printed:\n First, rewinding head to replay your work on top of it...\n Applying ZZZZZ\n error: x.cgi: does not exist in index\n error: y.cgi: does not exist in index\n Using index info to reconstruct a base tree...\n Falling back to patching base and 3-way merge...\n\nI don't know what magic it used but it did rebase right. Furthermore \"-s\nsubtree\" didn't work at all:\n    git rebase --onto OldProj test-split-old test-split -s subtree\n First, rewinding head to replay your work on top of it...\n Fast-forwarded OldProj to OldProj.\n\nAnd so I ask if this behavior is the way git-subtree was meant to work.\nIt probably has sense to add 'rebase' command to git-subtree script to let\nperform such tasks simplier.\n\nMy best regards, Andrey Smirnov.\n"},{"id":"118125","messageId":"32541b130907161134n51e070a1l93690d1b8a63bee8@mail.gmail.com","threadId":"19071","inReplyTo":"loom.20090716T160021-218@post.gmane.org","subject":"Re: [PATCH/RFC 1/2] Add 'git subtree' command for tracking history of subtrees separately.","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2009-07-16T18:34:58Z","receivedAt":"2009-07-16T18:34:58Z","isPatch":true,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Thu, Jul 16, 2009 at 2:04 PM, Andrey Smirnov<allter@gmail.com> wrote:\n> I found the git-subtree aproach of handling sub-repositories very interesting\n> and useful to me. This is the previous-to-last feature I've awaited from DVCS\n> world since I went into it.\n\nI'm glad you like it.\n\n> My goal was to rebase changes to shared library of two similar projects from one\n> project to another. The commits in the more recent project were touching both\n> lib/ directory with shared library and the rest of the project.\n>\n> When I did\n>   git subtree split --prefix=lib NewProj -b test-split\n>  and\n>   git subtree split --prefix=lib OldProj -b test-split-old\n> I got the following two trees without a common root:\n>\n> ...X ----- Y ----- OldProj ----...---- Z ---- NewProj\n>\n> X' ----- Y'==test-split-old ----- Z'==test-split\n\nSo, why don't they have a common root?  This is, of course, the\nprimary cause of your problems.\n\nHow did this shared library get merged into OldProj and NewProj in the\nfirst place?  Did you just copy the files, or did you use something\nlike 'git merge -s subtree'?  If the latter, you should be able to\nconvince git-subtree to produce two split repositories with identical\nroots, and then merge smoothly between them.\n\nIf you just copied the files (or applied patches with git-rebase, etc)\nthen a common root is impossible, and you'll have to repair the\ndamage, as I'll explain below:\n\n> Problem:\n>\n> When I did\n>   git subtree merge test-split --prefix=lib\n> it printed:\n>  Auto-merged lib/x.cgi\n>  CONFLICT (add/add): Merge conflict in lib/x.cgi\n>  Auto-merged lib/y.cgi\n>  CONFLICT (add/add): Merge conflict in lib/y.cgi\n>  Automatic merge failed; fix conflicts and then commit the result.\n>\n> It's obvious that it should be that way because logically both trees don't have\n> the same root at the time of merge. But I've expected subtree merge --prefix\n> will understand that X' is identical to changes to 'lib/*' in X, Y' to Y and Z'\n> to Z.\n\n'git subtree merge' is just like 'git merge' - if you don't have a\nshared merge-base commit, it won't know what to do, and will have to\nguess.\n\n> Solution:\n>\n>    git rebase --onto OldProj test-split-old test-split\n> it printed:\n>  First, rewinding head to replay your work on top of it...\n>  Applying ZZZZZ\n>  error: x.cgi: does not exist in index\n>  error: y.cgi: does not exist in index\n>  Using index info to reconstruct a base tree...\n>  Falling back to patching base and 3-way merge...\n>\n> I don't know what magic it used but it did rebase right. Furthermore \"-s\n> subtree\" didn't work at all:\n>    git rebase --onto OldProj test-split-old test-split -s subtree\n>  First, rewinding head to replay your work on top of it...\n>  Fast-forwarded OldProj to OldProj.\n\nNow you've just made a mess.  Instead, I might have tried something like this:\n\n  git checkout -b test-merged test-split\n  git rebase test-split-old\n\nNow you've got a split branch test-merged that should contain all the\nchanges from both test-split and test-split-old, and its parentage is\ntest-split-old.  Let's make it so that it can merge conflict-free into\neither branch:\n\n  git merge -s ours test-split\n\nNext, you merge it into one of your main projects:\n\n  git checkout OldProj\n    # you should have used split --rejoin earlier, and the next\ncommand wouldn't be necessary here\n  git subtree split --prefix=lib OldProj --rejoin\n  git subtree merge --prefix=lib test-merged\n\nAnd the other:\n\n  git checkout NewProj\n    # you should have used split --rejoin earlier, and the next\ncommand wouldn't be necessary here\n  git subtree split --prefix=lib NewProj --rejoin\n  git subtree merge --prefix=lib test-merged\n\nNow both projects are caught up with test-merged, and moreover, if you\nsplit either of them in the future, they will share a common\nmerge-base (ie. today's value of test-merged).  Future merges will\ntherefore be easier.\n\n> And so I ask if this behavior is the way git-subtree was meant to work.\n> It probably has sense to add 'rebase' command to git-subtree script to let\n> perform such tasks simplier.\n\nI don't think that's a good idea.  git-subtree is completely separate\nfrom rebasing, and doesn't deal with patches at all.  Maybe there\nshould be some kind of \"force-update\" option that does what \"git\nsubtree add\" does, but wiping out everything in the subtree before it\nstarts.  That would have simplified the above commands a bit.\n\nAvery\n"},{"id":"118141","messageId":"cdea6cd10907161509g7771c72bl608b1924785b49fc@mail.gmail.com","threadId":"19071","inReplyTo":"32541b130907161134n51e070a1l93690d1b8a63bee8@mail.gmail.com","subject":"Re: [PATCH/RFC 1/2] Add 'git subtree' command for tracking history of subtrees separately.","fromName":"Andrey Smirnov","fromEmail":"allter@gmail.com","sentAt":"2009-07-16T22:09:01Z","receivedAt":"2009-07-16T22:09:01Z","isPatch":true,"sender":{"key":"allter@gmail.com","avatar":null},"body":"Hello!\n\nOn Thu, Jul 16, 2009 at 10:34 PM, Avery Pennarun<apenwarr> wrote:\n>> When I did\n>>   git subtree split --prefix=lib NewProj -b test-split\n>>  and\n>>   git subtree split --prefix=lib OldProj -b test-split-old\n>> I got the following two trees without a common root:\n>>\n>> ...X ----- Y ----- OldProj ----...---- Z ---- NewProj\n>>\n>> X' ----- Y'==test-split-old ----- Z'==test-split\n> So, why don't they have a common root?  This is, of course, the\n> primary cause of your problems.\n\nThe line with OldProj and NewProj is story of commits for the project\nthat contains both library and other code. The line with test-split\nand test-split-old is the story of commits of the shared library alone\nwith test-split-old corresponding to OldProj and test-split\ncorresponding to NewProj.\nAnd I needed to get changes test-split-old..test-split in superproject\n(but without other\ngarbage commits that lead to NewProj).\n\n> How did this shared library get merged into OldProj and NewProj in the\n> first place?  Did you just copy the files, or did you use something\n> like 'git merge -s subtree'?  If the latter, you should be able to\n> convince git-subtree to produce two split repositories with identical\n> roots, and then merge smoothly between them.\n\nThey don't share commits because the library was never developed on its own.\nThe library evolved from the common code that was cut and pasted\ntrough about a hundred\nweb projects stored in SVN. Before I started to use git (mostly I use\nit as merge/rebase tool\nbecause our primary VCS is still Subversion) I transplanted changes in\nlibrary by manual\nsvn merge, even on individual files in some cases. While I was typing\nmy previous message, I\nfound that if I added \"--rejoin\", I would have situation that imitate\neffect of \"add test-split-old\"\ncommand followed by \"merge -s subtree test-merged\":\n\n...X ----- Y ----- OldProj ----- rejoined-merge ----...---- Z ---- NewProj\n                                     /                \\\nX' ----- Y'==test-split-old                      Z''==test-merged\n                                   \\\n                                     Z'==test-split\n\nBut Subversion and git-svn don't like git-ish merges, they need rebase. :(\n\n>  git checkout -b test-merged test-split\n>  git checkout OldProj\n>  git subtree split --prefix=lib OldProj --rejoin\n>  git subtree merge --prefix=lib test-merged\n\nYes, that's one of ways I thought of and that I pictured above. But I\nwould like\napproach that deals only with patches and not commit trees due to\ngit-svn restriction.\n\n>> And so I ask if this behavior is the way git-subtree was meant to work.\n>> It probably has sense to add 'rebase' command to git-subtree script to let\n>> perform such tasks simplier.\n> I don't think that's a good idea.  git-subtree is completely separate\n> from rebasing, and doesn't deal with patches at all.  Maybe there\n> should be some kind of \"force-update\" option that does what \"git\n> subtree add\" does, but wiping out everything in the subtree before it\n> starts.  That would have simplified the above commands a bit.\n\nThe only thing that links git-subtree with git-rebase is the fact, that\ngit-subtree \"knows\" the target commit for rebases dealing with subtrees.\nSo if one knows commit of a subtree that he wishes to see in superproject\n(in my case \"test-split\") he could issue:\n    git subtree rebase --prefix=lib OldProject test-split\n\nThough simple:\n    git rebase --onto OldProject test-split-old test-split\nworked for me, I think this was a lucky coincidence because of simplicity\nof my library commits.\n\n--\nSincerly yours, Andrey.\n"},{"id":"118143","messageId":"32541b130907161527l1955bf06pf54b5099a5988c65@mail.gmail.com","threadId":"19071","inReplyTo":"cdea6cd10907161509g7771c72bl608b1924785b49fc@mail.gmail.com","subject":"Re: [PATCH/RFC 1/2] Add 'git subtree' command for tracking history of subtrees separately.","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2009-07-16T22:27:40Z","receivedAt":"2009-07-16T22:27:40Z","isPatch":true,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Thu, Jul 16, 2009 at 6:09 PM, Andrey Smirnov<allter@gmail.com> wrote:\n> On Thu, Jul 16, 2009 at 10:34 PM, Avery Pennarun<apenwarr> wrote:\n>> I don't think that's a good idea.  git-subtree is completely separate\n>> from rebasing, and doesn't deal with patches at all.  Maybe there\n>> should be some kind of \"force-update\" option that does what \"git\n>> subtree add\" does, but wiping out everything in the subtree before it\n>> starts.  That would have simplified the above commands a bit.\n>\n> The only thing that links git-subtree with git-rebase is the fact, that\n> git-subtree \"knows\" the target commit for rebases dealing with subtrees.\n> So if one knows commit of a subtree that he wishes to see in superproject\n> (in my case \"test-split\") he could issue:\n>    git subtree rebase --prefix=lib OldProject test-split\n>\n> Though simple:\n>    git rebase --onto OldProject test-split-old test-split\n> worked for me, I think this was a lucky coincidence because of simplicity\n> of my library commits.\n\nI don't really understand what you're asking for here.  rebase doesn't\nhave any parameters called a \"target.\"  What does git-subtree know\nthat you don't know?\n\nHave fun,\n\nAvery\n"},{"id":"118156","messageId":"cdea6cd10907170016u11af7230hbbee92682604530f@mail.gmail.com","threadId":"19071","inReplyTo":"32541b130907161527l1955bf06pf54b5099a5988c65@mail.gmail.com","subject":"Re: [PATCH/RFC 1/2] Add 'git subtree' command for tracking history of subtrees separately.","fromName":"Andrey Smirnov","fromEmail":"allter@gmail.com","sentAt":"2009-07-17T07:16:00Z","receivedAt":"2009-07-17T07:16:00Z","isPatch":true,"sender":{"key":"allter@gmail.com","avatar":null},"body":"On Fri, Jul 17, 2009 at 2:27 AM, Avery Pennarun<apenwarr> wrote:\n\n>> The only thing that links git-subtree with git-rebase is the fact, that\n>> git-subtree \"knows\" the target commit for rebases dealing with subtrees.\n> rebase doesn't\n> have any parameters called a \"target.\"  What does git-subtree know\n> that you don't know?\n\nBy \"rebase target\" I mean the mutual relation of git-rebase <newbase>\nand <upstream> paramaters\nthat define where will be the rebased commits. git-subtree can infer\nthat NewProj contains library up to\ntest-split and that OldProj contains library upto test-split-old. The\nconcept of the whole git-subtee workflow\nis still blurry to me though, so I will report when I gather more\nusage statistics.\n\n> I don't really understand what you're asking for here.\n\nAt most I need generic ability to shift merged and rebased\nrepository's or ref's \"left\" (selecting some directory or file)\nand \"right\" (prepending some directory to all paths) before actual\noperation(s). I.e. the antonym of 'split'\nbut without 'add' committree-joining semantics. This can be\nimplemented with some chaining/plumbing presets.\n\n--\nSincerly yours, Andrey.\n"},{"id":"118180","messageId":"32541b130907170847g67c89d54ke9426ed8da26a9aa@mail.gmail.com","threadId":"19071","inReplyTo":"cdea6cd10907170016u11af7230hbbee92682604530f@mail.gmail.com","subject":"Re: [PATCH/RFC 1/2] Add 'git subtree' command for tracking history of subtrees separately.","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2009-07-17T15:47:31Z","receivedAt":"2009-07-17T15:47:31Z","isPatch":true,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Fri, Jul 17, 2009 at 3:16 AM, Andrey Smirnov<allter@gmail.com> wrote:\n> On Fri, Jul 17, 2009 at 2:27 AM, Avery Pennarun<apenwarr> wrote:\n>>> The only thing that links git-subtree with git-rebase is the fact, that\n>>> git-subtree \"knows\" the target commit for rebases dealing with subtrees.\n>> rebase doesn't\n>> have any parameters called a \"target.\"  What does git-subtree know\n>> that you don't know?\n>\n> By \"rebase target\" I mean the mutual relation of git-rebase <newbase>\n> and <upstream> paramaters\n> that define where will be the rebased commits. git-subtree can infer\n> that NewProj contains library up to\n> test-split and that OldProj contains library upto test-split-old. The\n> concept of the whole git-subtee workflow\n> is still blurry to me though, so I will report when I gather more\n> usage statistics.\n\nThe problem is that test-split and test-split-old are completely\nunrelated trees that have similar-looking files but no common\nancestry.  All git-subtree knows is exactly that.  It can't simplify\nanything (in your case) like you seem to think it can.\n\ngit-rebase tries to be cleverer, and starts comparing patches and file\nsimilarities so it can graft one tree onto another, and for\nconvenience, it throws away redundant commits that do exactly what\nsome other commit did (basically).  This is actually really messy.  As\nsoon as you get into that situation, you have nothing but a mess.  My\nadvice would be to clean up the mess as soon as you can (which\nappropriate use of git-subtree + git-rebase can help you do).\n\nThen you'll have actual, valid merge history, and git-subtree will be\nable to work smoothly using just that.\n\n>> I don't really understand what you're asking for here.\n>\n> At most I need generic ability to shift merged and rebased\n> repository's or ref's \"left\" (selecting some directory or file)\n> and \"right\" (prepending some directory to all paths) before actual\n> operation(s). I.e. the antonym of 'split'\n> but without 'add' committree-joining semantics. This can be\n> implemented with some chaining/plumbing presets.\n\nI think that if you're having this problem, you should look for a less\nugly solution :)\n\nWhat I think you're asking for is a way of turning all the commits in\na subdir into a patch stream (which git-subtree split can do,\nessentially), but then to add a prefix to all the paths in all the\npatches, so that you can then apply those patches on top of some other\nrepo where the files were in another location.  You can do that, I\nguess, but you're not taking advantage of git's convenience.\n\ngit-subtree encourages you to think of the files in the subtree as\ntheir own separate project, and you can then merge that separate\nproject into yours.  That's actually a more accurate model of reality,\nI think.\n\nHave fun,\n\nAvery\n"},{"id":"118190","messageId":"cdea6cd10907171046u257fa8ebjee445a90de0a282c@mail.gmail.com","threadId":"19071","inReplyTo":"32541b130907170847g67c89d54ke9426ed8da26a9aa@mail.gmail.com","subject":"Re: [PATCH/RFC 1/2] Add 'git subtree' command for tracking history of subtrees separately.","fromName":"Andrey Smirnov","fromEmail":"allter@gmail.com","sentAt":"2009-07-17T17:46:07Z","receivedAt":"2009-07-17T17:46:07Z","isPatch":true,"sender":{"key":"allter@gmail.com","avatar":null},"body":"On Fri, Jul 17, 2009 at 7:47 PM, Avery Pennarun<apenwarr@gmail.com> wrote:\n> The problem is that test-split and test-split-old are completely\n> unrelated trees that have similar-looking files but no common\n> ancestry.\n\nI understand this but if two projects share the same commit history\nfor their subdir-lib\nboth test-split and test-split-old will have same root.\n\n> git-rebase tries to be cleverer, and starts comparing patches and file\n> similarities so it can graft one tree onto another,\n\nThat is too clear for me and since I try to avoid committing\n'reversion commits' it works for my\nworkflow. I don't want to track library as separate ref (at least now).\n\nAnyway thank you for great extension - will now try using it on a\nregular basis - there're another project that is more suitable\nto track libraries as you suggest.\n\n--\nSincerly yours, Andrey.\n"}]}