{"thread":{"id":"17714","subject":"[PATCH] Rough draft: Split a git repo into superproject and submodule","startedAt":"2009-02-11T02:36:14Z","lastAt":"2009-02-11T02:36:14Z","messageCount":1,"participants":["Eric Kidd"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"104135","messageId":"1234319774-48037-1-git-send-email-git@randomhacks.net","threadId":"17714","inReplyTo":null,"subject":"[PATCH] Rough draft: Split a git repo into superproject and submodule","fromName":"Eric Kidd","fromEmail":"git@randomhacks.net","sentAt":"2009-02-11T02:36:14Z","receivedAt":"2009-02-11T02:36:14Z","isPatch":true,"sender":{"key":"git@randomhacks.net","avatar":null},"body":"I have some extremely large git repositories which I want to split into\nmodules.  This could be done using 'git filter-branch' and the following\nsteps:\n\n 1) Create a submodule using --subdirectory-filter.\n 2) Create a superproject using an index filter to delete the submodule.\n 3) Commit the submodule to the latest version of the superproject.\n\nUnfortunately, this approach loses all the historical connections\nbetween the superproject and the submodule, breaking tools like 'git\nbisect', and making it difficult to recover old releases.\n\nIdeally, each version of the newly created superproject would be linked\nto the correct version of the submodule (and all the .gitmodules entries\nwould be set up correctly, too, throughout the project's history).\n\nThe attached patch contains a _very_ rough draft of a git\nsubmodule-split command.  You can run it as follows:\n\n  git submodule-split original-repo superproject directory-to-split\n  git clone superproject.git superproject\n  cd superproject\n  git submodule update --init\n\nIt will output two repositories named superproject.git and\nsubproject.git.  Internally, it runs 'git filter-branch' on a bare\nrepository.\n\nThere's a test suite, too.\n\nI still have quite a few things left to do before this script is\nactually useful:\n\n  Rename submodule-split to something more appropriate\n  Update .gitmodules using git config\n  Merge new entries with existing .gitmodules\n  Add support for directory and repository names which differ\n  Add support for multiple possible directory names\n  Add support for directories which move around the tree\n  Add support for directories which are missing in some revisions\n\nI'm releasing this version for feedback on the general design and the\ncoding style.  I don't often write shell scripts of this magnitude, and\nthere are almost certainly some portability and style problems.  So please\nlet me know what needs improvement, and I'll try to fix it.\n\nThank you for your feedback!\n---\n .gitignore                 |    1 +\n Makefile                   |    1 +\n git-submodule-split.sh     |  113 ++++++++++++++++++++++++++++++++++++++++++++\n t/t7404-submodule-split.sh |   36 ++++++++++++++\n 4 files changed, 151 insertions(+), 0 deletions(-)\n create mode 100644 git-submodule-split.sh\n create mode 100755 t/t7404-submodule-split.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 13311f1..fa6ed07 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -119,6 +119,7 @@ git-show\n git-show-branch\n git-show-index\n git-show-ref\n+git-submodule-split\n git-stage\n git-stash\n git-status\ndiff --git a/Makefile b/Makefile\nindex 27b9569..aceac8f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -277,6 +277,7 @@ SCRIPT_SH += git-sh-setup.sh\n SCRIPT_SH += git-stash.sh\n SCRIPT_SH += git-submodule.sh\n SCRIPT_SH += git-web--browse.sh\n+SCRIPT_SH += git-submodule-split.sh\n \n SCRIPT_PERL += git-add--interactive.perl\n SCRIPT_PERL += git-archimport.perl\ndiff --git a/git-submodule-split.sh b/git-submodule-split.sh\nnew file mode 100644\nindex 0000000..d7d1080\n--- /dev/null\n+++ b/git-submodule-split.sh\n@@ -0,0 +1,113 @@\n+#!/bin/sh\n+# \n+# Split a repository into a submodule and main module, with history\n+#\n+# Copyright 2009 Eric Kidd\n+# License: GNU General Public License, version 2 or later\n+\n+USAGE=\"src-repo dst-repo submodule-repo\"\n+\n+OPTIONS_SPEC=\n+NONGIT_OK=Yes\n+. git-sh-setup\n+\n+# Keep our argument parsing simple for now.\n+test \"$#\" = 3 || usage\n+src_repo=\"$1\"\n+dst_repo=\"$2.git\"\n+sub_repo=\"$3.git\"\n+submodule_dir=\"$3\"\n+revs=\"--all\"\n+\n+# Make a bare clone of a git repo with identical branches.\n+git_mirror() {\n+\tgit clone --mirror \"$1\" \"$2\" || exit 1\n+\t# For some reason, git clone --mirror doesn't actually create our\n+\t# local branch references for us.\n+\t(cd \"$2\" && git fetch || exit 1)\n+}\n+\n+# We export these variables so that they can be used from scripts passed to\n+# git filter-branch.  Thanks to gitte for this trick, which also allows us\n+# to do the right thing when subdirectory names contain spaces and quotes.\n+export SPLIT_SUB_REPO=\"$sub_repo\"\n+export SPLIT_SUBMODULE_DIR=\"$submodule_dir\"\n+export SPLIT_MAP_DIR=\"`pwd`/$sub_repo/split-map\"\n+\n+\n+#--------------------------------------------------------------------------\n+# Create the new submodule\n+\n+# Create a copy of $src_repo to transform.\n+git_mirror \"$src_repo\" \"$sub_repo\"\n+\n+# For each commit ID, we will create a files in containing\n+# information that we'll later use to rewrite the subproject.\n+mkdir \"$SPLIT_MAP_DIR\" || exit 1\n+\n+index_filter=$(cat << \\EOF\n+map_info=\"$SPLIT_MAP_DIR/$GIT_COMMIT\"\n+if git rev-parse -q --verify $GIT_COMMIT:\"$SPLIT_SUBMODULE_DIR\"; then\n+\t# Adapted from git-filter-branch.\n+\terr=$(git read-tree -i -m $GIT_COMMIT:\"$SPLIT_SUBMODULE_DIR\" 2>&1) ||\n+\t\tdie \"$err\"\n+        echo -n \"$SPLIT_SUBMODULE_DIR\" > \"$map_info-dir\"\n+else\n+\t# We will use an empty file to indicate that the directory\n+        # doesn't exist in the tree.\n+\t# touch \"$map_info-skipped\"\n+\tdie \"Directory is missing\"\n+fi\n+EOF\n+)\n+\n+commit_filter=$(cat << \\EOF\n+map_info=\"$SPLIT_MAP_DIR/$GIT_COMMIT\"\n+new_commit=\"$(git commit-tree \"$@\")\" || exit 1\n+echo $new_commit\n+echo $new_commit > \"$map_info-submodule-commit\" ||\n+\tdie \"Can't record the commit ID of the new commit\"\n+EOF\n+)\n+\n+# Run our filters.\n+(cd \"$sub_repo\" &&\n+ git filter-branch --index-filter \"$index_filter\" \\\n+     --commit-filter \"$commit_filter\" -- \"$revs\") || exit 1\n+\n+\n+#--------------------------------------------------------------------------\n+# Create the new superproject\n+\n+# Next, create our new parent repository.\n+git_mirror \"$src_repo\" \"$dst_repo\"\n+\n+index_filter=$(cat << \\EOF\n+map_info=\"$SPLIT_MAP_DIR/$GIT_COMMIT\"\n+\n+# Splice the repo into the tree.\n+test -f \"$map_info-submodule-commit\" || die \"Can't find map for $GIT_COMMIT\"\n+git rm -q --cached -r \"$SPLIT_SUBMODULE_DIR\" || exit 1\n+echo \"160000 $(cat \"$map_info-submodule-commit\")\t$SPLIT_SUBMODULE_DIR\" |\n+\tgit update-index --index-info || exit 1\n+\n+# Construct a new .gitmodules file.\n+cat > \"$SPLIT_MAP_DIR/gitmodules\" <<EOC\n+[submodule \"$SPLIT_SUBMODULE_DIR\"]\n+\tpath = $SPLIT_SUBMODULE_DIR\n+\turl = ../$SPLIT_SUB_REPO\n+EOC\n+\n+# Write the new .gitmodules file into the tree.\n+new_obj=$(git hash-object -t blob -w \"$SPLIT_MAP_DIR/gitmodules\") ||\n+\tdie \"Error adding new .gitmodules file to tree\"\n+git update-index --add --cacheinfo 100644 \"$new_obj\" .gitmodules || exit 1\n+\n+EOF\n+)\n+\n+# Run our filter.\n+(cd \"$dst_repo\" &&\n+ git filter-branch --index-filter \"$index_filter\" -- \"$revs\") || exit 1\n+\n+exit 0\ndiff --git a/t/t7404-submodule-split.sh b/t/t7404-submodule-split.sh\nnew file mode 100755\nindex 0000000..b490c60\n--- /dev/null\n+++ b/t/t7404-submodule-split.sh\n@@ -0,0 +1,36 @@\n+#!/bin/sh\n+#\n+# Copyright 2009 Eric Kidd\n+\n+test_description='git submodule-split tests'\n+. ./test-lib.sh\n+\n+rm -rf .git\n+test_create_repo original\n+\n+test_expect_success \\\n+\t'create original repository' \\\n+\t'(cd original &&\n+\t  echo \"In main project\" > main-file &&\n+\t  mkdir sub1 &&\n+\t  echo \"In sub1\" > sub1/sub1-file &&\n+\t  git add . &&\n+\t  git commit -m \"Original project and sub1\")'\n+\n+test_expect_success \\\n+\t'split out sub1' \\\n+\t'git submodule-split original split1 sub1 &&\n+\t git clone split1.git split1 &&\n+\t test -f split1/main-file &&\n+\t ! test -f split1/sub1/sub1-file &&\n+\t git clone sub1.git sub1 &&\n+\t test -f sub1/sub1-file'\n+\n+test_expect_success \\\n+\t'compare split repositories with original' \\\n+\t'rm -rf split1 &&\n+\t git clone split1.git split1 &&\n+\t (cd split1 && git submodule init sub1 && git submodule update) &&\n+\t diff -uNr -x .git -x .gitmodules original split1'\n+\n+test_done\n-- \n1.6.0.4\n"}]}