threads / discuss / 24489

RFC: Between git-subtree and git-submodules

Subject: RFC: Between git-subtree and git-submodules

## tl;dr

5 messages between Jul 24, 2010 and Jul 26, 2010.

replies: 4people: 2as markdown or json

Santi Béjar· Jul 24, 2010, 00:13 UTC · lore
Hi *,
  now that there is a lot of traffic about git-subtree and git-submodules I
  would like to show you why *I* don't use neither of them, but something in
  between.
  First my requirements:
1) Everything[1] must be available from the same repository/branch (so I'm not
   worried about repository size)
2) The history must be as clean as possible
3) The directory content must be equal to the external module, at least when
   you add/update it[2]
4) The external module should be able to switch back and forth between
   different versions.

[1] Everything means all that you need to checkout all the commits in the superproject not in the submodule. [2] A consequence of 3) is that I lose all change I've made in the subdirectory, if they are important I have to extract them, apply them and add the module back.

git-submodule is rule out because of 1) but accomplish 2), 3) and 4). git-subtree is rule out because of 2) (even with --squash) 3) and 4) without --squash but accomplish 1) and 4) with --squash. So I need something in between or a mixture of both.

At the end I've done what I called originally git-subtree, but now I've written the prune mode of git-subtree.

The idea is that you want to add the content of a module in a subdirectory and that's all! I think that this simplicity is also very powerful as it is very clear how it behaves. You just get a commit each time you add a subtree (normal commit not a merge) without the history of the subtree.

You get something like this:
$ git log --graph
* ee225bd Subtree 'a/': 895916a Commit message 1
* aa345dg Modification to a/Makefile
* ddcd676 Subtree 'a/': 9a053f2 Commit message 2
* ea35faf Indent, whitespaces,...

Later you can extract/split all the modifications to a/ with "git-subtree split" (not yet implemented).

If you merge two branches with different content in the subtree you just merge them the normal way as with any other file.

It works quite well in my use case.

Below you can find the patch to git-subtree to add this prune mode (also attatched because of whitespace corruption). Only slightly tested as I normally use my git-subtree command (completely different) and I've just patch git-subtree but you can get the idea of what it does.

HTH, Santi

diff --git i/git-subtree.sh w/git-subtree.sh
index 781eef3..7706d72 100755
--- i/git-subtree.sh
+++ w/git-subtree.sh
@@ -27,6 +27,7 @@ onto=         try connecting new tree to an existing one
 rejoin        merge the new branch back into HEAD
  options for 'add', 'merge', 'pull' and 'push'
 squash        merge subtree changes as a single commit
+prune         prune history
 "
 eval $(echo "$OPTS_SPEC" | git rev-parse --parseopt -- "$@" || echo exit $?)

@@ -44,6 +45,7 @@ rejoin=
 ignore_joins=
 annotate=
 squash=
+prune=
 message=

 debug()
@@ -92,6 +94,8 @@ while [ $# -gt 0 ]; do
 		--no-ignore-joins) ignore_joins= ;;
 		--squash) squash=1 ;;
 		--no-squash) squash= ;;
+		--prune) prune=1;;
+		--no-prune) prune=;;
 		--) break ;;
 		*) die "Unexpected option: $opt" ;;
 	esac
@@ -110,7 +114,7 @@ if [ -z "$prefix" ]; then
 fi

 case "$command" in
-	add) [ -e "$prefix" ] &&
+	add) [ -e "$prefix" -a -z "$prune" ] &&
 		die "prefix '$prefix' already exists." ;;
 	*)   [ -e "$prefix" ] ||
 		die "'$prefix' does not exist; use 'git subtree add'" ;;
@@ -359,6 +363,17 @@ squash_msg()
 	echo "git-subtree-split: $newsub"
 }

+prune_msg()
+{
+	dir="$1"
+	newsub="$2"
+	
+	git show -s --pretty="tformat:Subtree '$dir/': %h %s" $newsub
+	echo
+	echo "git-subtree-dir: $dir"
+	echo "git-subtree-split: $newsub"
+}
+
 toptree_for_commit()
 {
 	commit="$1"
@@ -464,7 +479,7 @@ ensure_clean()

 cmd_add()
 {
-	if [ -e "$dir" ]; then
+	if [ -e "$dir" -a -z "$prune" ]; then
 		die "'$dir' already exists.  Cannot add."
 	fi

@@ -498,6 +513,10 @@ cmd_add_commit()
 	rev="$1"
 	
 	debug "Adding $dir as '$rev'..."
+	if [ -d "$dir" ]; then
+	    #TODO: write it with plumbing commands
+	    git rm -r -q $dir
+	fi
 	git read-tree --prefix="$dir" $rev || exit $?
 	git checkout -- "$dir" || exit $?
 	tree=$(git write-tree) || exit $?
@@ -513,6 +532,9 @@ cmd_add_commit()
 		rev=$(new_squash_commit "" "" "$rev") || exit $?
 		commit=$(add_squashed_msg "$rev" "$dir" |
 			 git commit-tree $tree $headp -p "$rev") || exit $?
+	elif [ -n "$prune" ]; then
+		commit=$(prune_msg "$dir" "$rev" |
+			 git commit-tree $tree -p $headrev) || exit $?
 	else
 		commit=$(add_msg "$dir" "$headrev" "$rev" |
 			 git commit-tree $tree $headp -p "$rev") || exit $?


diff --git i/git-subtree.sh w/git-subtree.sh
index 781eef3..7706d72 100755
--- i/git-subtree.sh
+++ w/git-subtree.sh
@@ -27,6 +27,7 @@ onto=         try connecting new tree to an existing one
 rejoin        merge the new branch back into HEAD
  options for 'add', 'merge', 'pull' and 'push'
 squash        merge subtree changes as a single commit
+prune         prune history
 "
 eval $(echo "$OPTS_SPEC" | git rev-parse --parseopt -- "$@" || echo exit $?)
 
@@ -44,6 +45,7 @@ rejoin=
 ignore_joins=
 annotate=
 squash=
+prune=
 message=
 
 debug()
@@ -92,6 +94,8 @@ while [ $# -gt 0 ]; do
 		--no-ignore-joins) ignore_joins= ;;
 		--squash) squash=1 ;;
 		--no-squash) squash= ;;
+		--prune) prune=1;;
+		--no-prune) prune=;;
 		--) break ;;
 		*) die "Unexpected option: $opt" ;;
 	esac
@@ -110,7 +114,7 @@ if [ -z "$prefix" ]; then
 fi
 
 case "$command" in
-	add) [ -e "$prefix" ] && 
+	add) [ -e "$prefix" -a -z "$prune" ] &&
 		die "prefix '$prefix' already exists." ;;
 	*)   [ -e "$prefix" ] || 
 		die "'$prefix' does not exist; use 'git subtree add'" ;;
@@ -359,6 +363,17 @@ squash_msg()
 	echo "git-subtree-split: $newsub"
 }
 
+prune_msg()
+{
+	dir="$1"
+	newsub="$2"
+	
+	git show -s --pretty="tformat:Subtree '$dir/': %h %s" $newsub
+	echo
+	echo "git-subtree-dir: $dir"
+	echo "git-subtree-split: $newsub"
+}
+
 toptree_for_commit()
 {
 	commit="$1"
@@ -464,7 +479,7 @@ ensure_clean()
 
 cmd_add()
 {
-	if [ -e "$dir" ]; then
+	if [ -e "$dir" -a -z "$prune" ]; then
 		die "'$dir' already exists.  Cannot add."
 	fi
 
@@ -498,6 +513,10 @@ cmd_add_commit()
 	rev="$1"
 	
 	debug "Adding $dir as '$rev'..."
+	if [ -d "$dir" ]; then
+	    #TODO: write it with plumbing commands
+	    git rm -r -q $dir
+	fi
 	git read-tree --prefix="$dir" $rev || exit $?
 	git checkout -- "$dir" || exit $?
 	tree=$(git write-tree) || exit $?
@@ -513,6 +532,9 @@ cmd_add_commit()
 		rev=$(new_squash_commit "" "" "$rev") || exit $?
 		commit=$(add_squashed_msg "$rev" "$dir" |
 			 git commit-tree $tree $headp -p "$rev") || exit $?
+	elif [ -n "$prune" ]; then
+		commit=$(prune_msg "$dir" "$rev" |
+			 git commit-tree $tree -p $headrev) || exit $?
 	else
 		commit=$(add_msg "$dir" "$headrev" "$rev" |
 			 git commit-tree $tree $headp -p "$rev") || exit $?
Avery Pennarun· Jul 24, 2010, 00:57 UTC · re: Santi Béjar · lore

Re: RFC: Between git-subtree and git-submodules

On Fri, Jul 23, 2010 at 8:13 PM, Santi Béjar <santi@agolina.net> wrote:
Show 21 quoted lines
>  First my requirements:
>
> 1) Everything[a] must be available from the same repository/branch (so I'm not
>   worried about repository size)
> 2) The history must be as clean as possible
> 3) The directory content must be equal to the external module, at least when
>   you add/update it[b]
> 4) The external module should be able to switch back and forth between
>   different versions.
>
> [a] Everything means all that you need to checkout all the commits in the
> superproject not in the submodule.
> [b] A consequence of #3 is that I lose all
> change I've made in the subdirectory, if they are important I have to extract
> them, apply them and add the module back.
>
> git-submodule is rule out because of #1 but accomplish #2, #3 and
> #4. git-subtree is rule out because of #2 (even with --squash).
> [It fails at] #3 and #4
> without --squash but accomplish #1 and #4 with --squash. So I need something
> in between or a mixture of both.

I admit to having had some trouble parsing the above, so I moved some punctuation marks around. Please let me know if I've made a mistake.

If I understand correctly, you're claiming (indirectly) that git-subtree without --squash does not accomplish #1. I don't see how this is the case. Am I misreading? I think git-subtree accomplishes #1 in both modes.

I don't understand what you mean when you say (#2) git-subtree doesn't keep your history "as clean as possible." What is "as clean as possible" and what part of git-subtree's history results don't you like? (Of course it's very different with and without --squash.)

With #3, I can see that you want something different than I do; you want to silently revert your own patches out of the submodule's history, when you upgrade the submodule to a new version. Personally, I find this concept a bit objectionable (it's like "git merge -s ours"), but okay, it's pretty easy to implement, and you've submitted a patch to git-subtree that does it. My question is: why would you want this? Isn't it clearer to 'git revert' the patches you don't want?

And for #4, it's true that git-subtree without --squash does not allow you to easily rewind to an older version of the submodule, while with --squash it does.

It sounds to me like, if we added your patch to git-subtree, then git-subtree --squash would solve #1, #3, and #4. And maybe we could fix #2 as well. Correct?

Thanks,
Avery
Santi Béjar· Jul 24, 2010, 08:33 UTC · re: Avery Pennarun · lore

Re: RFC: Between git-subtree and git-submodules

On Sat, Jul 24, 2010 at 2:57 AM, Avery Pennarun <apenwarr@gmail.com> wrote:
Show 30 quoted lines
> On Fri, Jul 23, 2010 at 8:13 PM, Santi Béjar <santi@agolina.net> wrote:
>>  First my requirements:
>>
>> 1) Everything[a] must be available from the same repository/branch (so I'm not
>>   worried about repository size)
>> 2) The history must be as clean as possible
>> 3) The directory content must be equal to the external module, at least when
>>   you add/update it[b]
>> 4) The external module should be able to switch back and forth between
>>   different versions.
>>
>> [a] Everything means all that you need to checkout all the commits in the
>> superproject not in the submodule.
>> [b] A consequence of #3 is that I lose all
>> change I've made in the subdirectory, if they are important I have to extract
>> them, apply them and add the module back.
>>
>> git-submodule is rule out because of #1 but accomplish #2, #3 and
>> #4. git-subtree is rule out because of #2 (even with --squash).
>> [It fails at] #3 and #4
>> without --squash but accomplish #1 and #4 with --squash. So I need something
>> in between or a mixture of both.
>
> I admit to having had some trouble parsing the above, so I moved some
> punctuation marks around.  Please let me know if I've made a mistake.
>
> If I understand correctly, you're claiming (indirectly) that
> git-subtree without --squash does not accomplish #1.  I don't see how
> this is the case.  Am I misreading?  I think git-subtree accomplishes
> #1 in both modes.
Yes, git-subtree accomplishes #1 in both modes. --squash only applies to #4.
Show 5 quoted lines
>
> I don't understand what you mean when you say (#2) git-subtree doesn't
> keep your history "as clean as possible."  What is "as clean as
> possible" and what part of git-subtree's history results don't you
> like?  (Of course it's very different with and without --squash.)

With git-subtree you always have the subtree history (even if it is squashed). So when you merge a second time the submodule you get the always the history of the subtree (even with --squash). So you basically always have at least two branches while examining the history. Compare this squashed history:

$ git log --graph --oneline
*   bb2dc25 (HEAD, master) Merge commit
'08b917ee90ecfd7b666364fe4ebb92aee5cdd2f7'
|\
| * 08b917e Squashed 'latex/' changes from ea35faf..895916a
* |   9de91f1 Merge commit 'b1b4c36bb8358582a6a20bb500bf98421428e2ca' as 'latex'
|\|
| * b1b4c36 Squashed 'latex/' content from commit ea35faf
* ea35faf Indent, whitespaces,...
with this pruned history:
$ git log --graph --oneline
* 8703aec (HEAD, master) Subtree 'latex/': 895916a Add files subcommand
* a942284 Subtree 'latex/': ea35faf Indent, whitespaces,...
* ea35faf Indent, whitespaces,...
But I understand that it can only be this way because of #3.
Show 9 quoted lines
>
> With #3, I can see that you want something different than I do; you
> want to silently revert your own patches out of the submodule's
> history, when you upgrade the submodule to a new version.  Personally,
> I find this concept a bit objectionable (it's like "git merge -s
> ours"), but okay, it's pretty easy to implement, and you've submitted
> a patch to git-subtree that does it.  My question is: why would you
> want this?  Isn't it clearer to 'git revert' the patches you don't
> want?

I prefer to do all the modifications in an external repository, even if at the end it is really a fork of the upstream repository. I think the proper place to modify the files in a subtree is in an external repository.

Show 8 quoted lines
>
> And for #4, it's true that git-subtree without --squash does not allow
> you to easily rewind to an older version of the submodule, while with
> --squash it does.
>
> It sounds to me like, if we added your patch to git-subtree, then
> git-subtree --squash would solve #1, #3, and #4.  And maybe we could
> fix #2 as well. Correct?
My patch does not change the behavior of --squash, it adds --prune.

Thanks, Santi

Santi Béjar· Jul 25, 2010, 00:28 UTC · re: Avery Pennarun · lore

Re: RFC: Between git-subtree and git-submodules

Show 8 quoted lines
> With #3, I can see that you want something different than I do; you
> want to silently revert your own patches out of the submodule's
> history, when you upgrade the submodule to a new version.  Personally,
> I find this concept a bit objectionable (it's like "git merge -s
> ours"), but okay, it's pretty easy to implement, and you've submitted
> a patch to git-subtree that does it.  My question is: why would you
> want this?  Isn't it clearer to 'git revert' the patches you don't
> want?

While writing the docs for the --prune option I've found this sentence in git-subtree.txt:

	'git subtree merge --squash'
	always adjusts the subtree to match the exactly
	specified commit, even if getting to that commit would
	require undoing some changes that were added earlier.

I think this is not true, and it contradicts with what you've just said. Or I don't understand it correctly.

Thanks, Santi

Avery Pennarun· Jul 26, 2010, 05:02 UTC · re: Santi Béjar · lore

Re: RFC: Between git-subtree and git-submodules

On Sat, Jul 24, 2010 at 8:28 PM, Santi Béjar <santi@agolina.net> wrote:
Show 19 quoted lines
>> With #3, I can see that you want something different than I do; you
>> want to silently revert your own patches out of the submodule's
>> history, when you upgrade the submodule to a new version.  Personally,
>> I find this concept a bit objectionable (it's like "git merge -s
>> ours"), but okay, it's pretty easy to implement, and you've submitted
>> a patch to git-subtree that does it.  My question is: why would you
>> want this?  Isn't it clearer to 'git revert' the patches you don't
>> want?
>
> While writing the docs for the --prune option I've found this sentence in
> git-subtree.txt:
>
>        'git subtree merge --squash'
>        always adjusts the subtree to match the exactly
>        specified commit, even if getting to that commit would
>        require undoing some changes that were added earlier.
>
> I think this is not true, and it contradicts with what you've just said. Or I
> don't understand it correctly.

I agree that this is a bit unclear. Basically it rewinds the *base* of the subtree to match the commit you specify. Patches you've applied locally are not removed.

What it actually does is it creates a new commit that applies the following patch:

    git diff oldbase newbase

If you've made any of your own patches that conflict with this, it'll generate the conflict and then stop. Otherwise it'll merge in your changes as you might expect.

Have fun,
Avery

← back to recent threads