{"thread":{"id":"24489","subject":"RFC: Between git-subtree and git-submodules","startedAt":"2010-07-24T00:13:19Z","lastAt":"2010-07-26T05:02:54Z","messageCount":5,"participants":["Santi Béjar","Avery Pennarun"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"146123","messageId":"AANLkTi=tDRfgzzcbp0jV=+hQ+Cs2fhNF0vK8JL1COoFu@mail.gmail.com","threadId":"24489","inReplyTo":null,"subject":"RFC: Between git-subtree and git-submodules","fromName":"Santi Béjar","fromEmail":"santi@agolina.net","sentAt":"2010-07-24T00:13:19Z","receivedAt":"2010-07-24T00:13:19Z","isPatch":false,"sender":{"key":"santi@agolina.net","avatar":null},"body":"Hi *,\n\n  now that there is a lot of traffic about git-subtree and git-submodules I\n  would like to show you why *I* don't use neither of them, but something in\n  between.\n\n  First my requirements:\n\n1) Everything[1] must be available from the same repository/branch (so I'm not\n   worried about repository size)\n2) The history must be as clean as possible\n3) The directory content must be equal to the external module, at least when\n   you add/update it[2]\n4) The external module should be able to switch back and forth between\n   different versions.\n\n[1] Everything means all that you need to checkout all the commits in the\nsuperproject not in the submodule.\n[2] A consequence of 3) is that I lose all\nchange I've made in the subdirectory, if they are important I have to extract\nthem, apply them and add the module back.\n\ngit-submodule is rule out because of 1) but accomplish 2), 3) and\n4). git-subtree is rule out because of 2) (even with --squash) 3) and 4)\nwithout --squash but accomplish 1) and 4) with --squash. So I need something\nin between or a mixture of both.\n\nAt the end I've done what I called originally git-subtree, but now I've written\nthe prune mode of git-subtree.\n\nThe idea is that you want to add the content of a module in a subdirectory and\nthat's all! I think that this simplicity is also very powerful as it\nis very clear how it\nbehaves. You just get a commit each time you add a subtree (normal commit\nnot a merge) without the history of the subtree.\n\nYou get something like this:\n\n$ git log --graph\n* ee225bd Subtree 'a/': 895916a Commit message 1\n* aa345dg Modification to a/Makefile\n* ddcd676 Subtree 'a/': 9a053f2 Commit message 2\n* ea35faf Indent, whitespaces,...\n\nLater you can extract/split all the modifications to a/ with \"git-subtree\nsplit\" (not yet implemented).\n\nIf you merge two branches with different content in the subtree you just merge\nthem the normal way as with any other file.\n\nIt works quite well in my use case.\n\nBelow you can find the patch to git-subtree to add this prune mode (also\nattatched because of whitespace corruption). Only slightly tested as I\nnormally use\nmy git-subtree command (completely different) and I've just patch git-subtree\nbut you can get the idea of what it does.\n\nHTH,\nSanti\n\ndiff --git i/git-subtree.sh w/git-subtree.sh\nindex 781eef3..7706d72 100755\n--- i/git-subtree.sh\n+++ w/git-subtree.sh\n@@ -27,6 +27,7 @@ onto=         try connecting new tree to an existing one\n rejoin        merge the new branch back into HEAD\n  options for 'add', 'merge', 'pull' and 'push'\n squash        merge subtree changes as a single commit\n+prune         prune history\n \"\n eval $(echo \"$OPTS_SPEC\" | git rev-parse --parseopt -- \"$@\" || echo exit $?)\n\n@@ -44,6 +45,7 @@ rejoin=\n ignore_joins=\n annotate=\n squash=\n+prune=\n message=\n\n debug()\n@@ -92,6 +94,8 @@ while [ $# -gt 0 ]; do\n \t\t--no-ignore-joins) ignore_joins= ;;\n \t\t--squash) squash=1 ;;\n \t\t--no-squash) squash= ;;\n+\t\t--prune) prune=1;;\n+\t\t--no-prune) prune=;;\n \t\t--) break ;;\n \t\t*) die \"Unexpected option: $opt\" ;;\n \tesac\n@@ -110,7 +114,7 @@ if [ -z \"$prefix\" ]; then\n fi\n\n case \"$command\" in\n-\tadd) [ -e \"$prefix\" ] &&\n+\tadd) [ -e \"$prefix\" -a -z \"$prune\" ] &&\n \t\tdie \"prefix '$prefix' already exists.\" ;;\n \t*)   [ -e \"$prefix\" ] ||\n \t\tdie \"'$prefix' does not exist; use 'git subtree add'\" ;;\n@@ -359,6 +363,17 @@ squash_msg()\n \techo \"git-subtree-split: $newsub\"\n }\n\n+prune_msg()\n+{\n+\tdir=\"$1\"\n+\tnewsub=\"$2\"\n+\t\n+\tgit show -s --pretty=\"tformat:Subtree '$dir/': %h %s\" $newsub\n+\techo\n+\techo \"git-subtree-dir: $dir\"\n+\techo \"git-subtree-split: $newsub\"\n+}\n+\n toptree_for_commit()\n {\n \tcommit=\"$1\"\n@@ -464,7 +479,7 @@ ensure_clean()\n\n cmd_add()\n {\n-\tif [ -e \"$dir\" ]; then\n+\tif [ -e \"$dir\" -a -z \"$prune\" ]; then\n \t\tdie \"'$dir' already exists.  Cannot add.\"\n \tfi\n\n@@ -498,6 +513,10 @@ cmd_add_commit()\n \trev=\"$1\"\n \t\n \tdebug \"Adding $dir as '$rev'...\"\n+\tif [ -d \"$dir\" ]; then\n+\t    #TODO: write it with plumbing commands\n+\t    git rm -r -q $dir\n+\tfi\n \tgit read-tree --prefix=\"$dir\" $rev || exit $?\n \tgit checkout -- \"$dir\" || exit $?\n \ttree=$(git write-tree) || exit $?\n@@ -513,6 +532,9 @@ cmd_add_commit()\n \t\trev=$(new_squash_commit \"\" \"\" \"$rev\") || exit $?\n \t\tcommit=$(add_squashed_msg \"$rev\" \"$dir\" |\n \t\t\t git commit-tree $tree $headp -p \"$rev\") || exit $?\n+\telif [ -n \"$prune\" ]; then\n+\t\tcommit=$(prune_msg \"$dir\" \"$rev\" |\n+\t\t\t git commit-tree $tree -p $headrev) || exit $?\n \telse\n \t\tcommit=$(add_msg \"$dir\" \"$headrev\" \"$rev\" |\n \t\t\t git commit-tree $tree $headp -p \"$rev\") || exit $?\n\n\ndiff --git i/git-subtree.sh w/git-subtree.sh\nindex 781eef3..7706d72 100755\n--- i/git-subtree.sh\n+++ w/git-subtree.sh\n@@ -27,6 +27,7 @@ onto=         try connecting new tree to an existing one\n rejoin        merge the new branch back into HEAD\n  options for 'add', 'merge', 'pull' and 'push'\n squash        merge subtree changes as a single commit\n+prune         prune history\n \"\n eval $(echo \"$OPTS_SPEC\" | git rev-parse --parseopt -- \"$@\" || echo exit $?)\n \n@@ -44,6 +45,7 @@ rejoin=\n ignore_joins=\n annotate=\n squash=\n+prune=\n message=\n \n debug()\n@@ -92,6 +94,8 @@ while [ $# -gt 0 ]; do\n \t\t--no-ignore-joins) ignore_joins= ;;\n \t\t--squash) squash=1 ;;\n \t\t--no-squash) squash= ;;\n+\t\t--prune) prune=1;;\n+\t\t--no-prune) prune=;;\n \t\t--) break ;;\n \t\t*) die \"Unexpected option: $opt\" ;;\n \tesac\n@@ -110,7 +114,7 @@ if [ -z \"$prefix\" ]; then\n fi\n \n case \"$command\" in\n-\tadd) [ -e \"$prefix\" ] && \n+\tadd) [ -e \"$prefix\" -a -z \"$prune\" ] &&\n \t\tdie \"prefix '$prefix' already exists.\" ;;\n \t*)   [ -e \"$prefix\" ] || \n \t\tdie \"'$prefix' does not exist; use 'git subtree add'\" ;;\n@@ -359,6 +363,17 @@ squash_msg()\n \techo \"git-subtree-split: $newsub\"\n }\n \n+prune_msg()\n+{\n+\tdir=\"$1\"\n+\tnewsub=\"$2\"\n+\t\n+\tgit show -s --pretty=\"tformat:Subtree '$dir/': %h %s\" $newsub\n+\techo\n+\techo \"git-subtree-dir: $dir\"\n+\techo \"git-subtree-split: $newsub\"\n+}\n+\n toptree_for_commit()\n {\n \tcommit=\"$1\"\n@@ -464,7 +479,7 @@ ensure_clean()\n \n cmd_add()\n {\n-\tif [ -e \"$dir\" ]; then\n+\tif [ -e \"$dir\" -a -z \"$prune\" ]; then\n \t\tdie \"'$dir' already exists.  Cannot add.\"\n \tfi\n \n@@ -498,6 +513,10 @@ cmd_add_commit()\n \trev=\"$1\"\n \t\n \tdebug \"Adding $dir as '$rev'...\"\n+\tif [ -d \"$dir\" ]; then\n+\t    #TODO: write it with plumbing commands\n+\t    git rm -r -q $dir\n+\tfi\n \tgit read-tree --prefix=\"$dir\" $rev || exit $?\n \tgit checkout -- \"$dir\" || exit $?\n \ttree=$(git write-tree) || exit $?\n@@ -513,6 +532,9 @@ cmd_add_commit()\n \t\trev=$(new_squash_commit \"\" \"\" \"$rev\") || exit $?\n \t\tcommit=$(add_squashed_msg \"$rev\" \"$dir\" |\n \t\t\t git commit-tree $tree $headp -p \"$rev\") || exit $?\n+\telif [ -n \"$prune\" ]; then\n+\t\tcommit=$(prune_msg \"$dir\" \"$rev\" |\n+\t\t\t git commit-tree $tree -p $headrev) || exit $?\n \telse\n \t\tcommit=$(add_msg \"$dir\" \"$headrev\" \"$rev\" |\n \t\t\t git commit-tree $tree $headp -p \"$rev\") || exit $?\n"},{"id":"146125","messageId":"AANLkTinyjinsGS2ddyMOW=3zDN1ThJcpF+dc4cBE6L9D@mail.gmail.com","threadId":"24489","inReplyTo":"AANLkTi=tDRfgzzcbp0jV=+hQ+Cs2fhNF0vK8JL1COoFu@mail.gmail.com","subject":"Re: RFC: Between git-subtree and git-submodules","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-07-24T00:57:06Z","receivedAt":"2010-07-24T00:57:06Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Fri, Jul 23, 2010 at 8:13 PM, Santi Béjar <santi@agolina.net> wrote:\n>  First my requirements:\n>\n> 1) Everything[a] must be available from the same repository/branch (so I'm not\n>   worried about repository size)\n> 2) The history must be as clean as possible\n> 3) The directory content must be equal to the external module, at least when\n>   you add/update it[b]\n> 4) The external module should be able to switch back and forth between\n>   different versions.\n>\n> [a] Everything means all that you need to checkout all the commits in the\n> superproject not in the submodule.\n> [b] A consequence of #3 is that I lose all\n> change I've made in the subdirectory, if they are important I have to extract\n> them, apply them and add the module back.\n>\n> git-submodule is rule out because of #1 but accomplish #2, #3 and\n> #4. git-subtree is rule out because of #2 (even with --squash).\n> [It fails at] #3 and #4\n> without --squash but accomplish #1 and #4 with --squash. So I need something\n> in between or a mixture of both.\n\nI admit to having had some trouble parsing the above, so I moved some\npunctuation marks around.  Please let me know if I've made a mistake.\n\nIf I understand correctly, you're claiming (indirectly) that\ngit-subtree without --squash does not accomplish #1.  I don't see how\nthis is the case.  Am I misreading?  I think git-subtree accomplishes\n#1 in both modes.\n\nI don't understand what you mean when you say (#2) git-subtree doesn't\nkeep your history \"as clean as possible.\"  What is \"as clean as\npossible\" and what part of git-subtree's history results don't you\nlike?  (Of course it's very different with and without --squash.)\n\nWith #3, I can see that you want something different than I do; you\nwant to silently revert your own patches out of the submodule's\nhistory, when you upgrade the submodule to a new version.  Personally,\nI find this concept a bit objectionable (it's like \"git merge -s\nours\"), but okay, it's pretty easy to implement, and you've submitted\na patch to git-subtree that does it.  My question is: why would you\nwant this?  Isn't it clearer to 'git revert' the patches you don't\nwant?\n\nAnd for #4, it's true that git-subtree without --squash does not allow\nyou to easily rewind to an older version of the submodule, while with\n--squash it does.\n\nIt sounds to me like, if we added your patch to git-subtree, then\ngit-subtree --squash would solve #1, #3, and #4.  And maybe we could\nfix #2 as well. Correct?\n\nThanks,\n\nAvery\n"},{"id":"146145","messageId":"AANLkTinYXrWSXSX8EOsb-YH_KGcmtAKB9nWZyRzBwZw5@mail.gmail.com","threadId":"24489","inReplyTo":"AANLkTinyjinsGS2ddyMOW=3zDN1ThJcpF+dc4cBE6L9D@mail.gmail.com","subject":"Re: RFC: Between git-subtree and git-submodules","fromName":"Santi Béjar","fromEmail":"santi@agolina.net","sentAt":"2010-07-24T08:33:38Z","receivedAt":"2010-07-24T08:33:38Z","isPatch":false,"sender":{"key":"santi@agolina.net","avatar":null},"body":"On Sat, Jul 24, 2010 at 2:57 AM, Avery Pennarun <apenwarr@gmail.com> wrote:\n> On Fri, Jul 23, 2010 at 8:13 PM, Santi Béjar <santi@agolina.net> wrote:\n>>  First my requirements:\n>>\n>> 1) Everything[a] must be available from the same repository/branch (so I'm not\n>>   worried about repository size)\n>> 2) The history must be as clean as possible\n>> 3) The directory content must be equal to the external module, at least when\n>>   you add/update it[b]\n>> 4) The external module should be able to switch back and forth between\n>>   different versions.\n>>\n>> [a] Everything means all that you need to checkout all the commits in the\n>> superproject not in the submodule.\n>> [b] A consequence of #3 is that I lose all\n>> change I've made in the subdirectory, if they are important I have to extract\n>> them, apply them and add the module back.\n>>\n>> git-submodule is rule out because of #1 but accomplish #2, #3 and\n>> #4. git-subtree is rule out because of #2 (even with --squash).\n>> [It fails at] #3 and #4\n>> without --squash but accomplish #1 and #4 with --squash. So I need something\n>> in between or a mixture of both.\n>\n> I admit to having had some trouble parsing the above, so I moved some\n> punctuation marks around.  Please let me know if I've made a mistake.\n>\n> If I understand correctly, you're claiming (indirectly) that\n> git-subtree without --squash does not accomplish #1.  I don't see how\n> this is the case.  Am I misreading?  I think git-subtree accomplishes\n> #1 in both modes.\n\nYes, git-subtree accomplishes #1 in both modes. --squash only applies to #4.\n\n>\n> I don't understand what you mean when you say (#2) git-subtree doesn't\n> keep your history \"as clean as possible.\"  What is \"as clean as\n> possible\" and what part of git-subtree's history results don't you\n> like?  (Of course it's very different with and without --squash.)\n\nWith git-subtree you always have the subtree history (even if it is\nsquashed). So when you merge a second time the submodule you get the always\nthe history of the subtree (even with --squash). So you basically always have\nat least two branches while examining the history. Compare this\nsquashed history:\n\n$ git log --graph --oneline\n*   bb2dc25 (HEAD, master) Merge commit\n'08b917ee90ecfd7b666364fe4ebb92aee5cdd2f7'\n|\\\n| * 08b917e Squashed 'latex/' changes from ea35faf..895916a\n* |   9de91f1 Merge commit 'b1b4c36bb8358582a6a20bb500bf98421428e2ca' as 'latex'\n|\\|\n| * b1b4c36 Squashed 'latex/' content from commit ea35faf\n* ea35faf Indent, whitespaces,...\n\nwith this pruned history:\n\n$ git log --graph --oneline\n* 8703aec (HEAD, master) Subtree 'latex/': 895916a Add files subcommand\n* a942284 Subtree 'latex/': ea35faf Indent, whitespaces,...\n* ea35faf Indent, whitespaces,...\n\nBut I understand that it can only be this way because of #3.\n\n>\n> With #3, I can see that you want something different than I do; you\n> want to silently revert your own patches out of the submodule's\n> history, when you upgrade the submodule to a new version.  Personally,\n> I find this concept a bit objectionable (it's like \"git merge -s\n> ours\"), but okay, it's pretty easy to implement, and you've submitted\n> a patch to git-subtree that does it.  My question is: why would you\n> want this?  Isn't it clearer to 'git revert' the patches you don't\n> want?\n\nI prefer to do all the modifications in an external repository, even if at the\nend it is really a fork of the upstream repository. I think the proper place\nto modify the files in a subtree is in an external repository.\n\n>\n> And for #4, it's true that git-subtree without --squash does not allow\n> you to easily rewind to an older version of the submodule, while with\n> --squash it does.\n>\n> It sounds to me like, if we added your patch to git-subtree, then\n> git-subtree --squash would solve #1, #3, and #4.  And maybe we could\n> fix #2 as well. Correct?\n\nMy patch does not change the behavior of --squash, it adds --prune.\n\nThanks,\nSanti\n"},{"id":"146228","messageId":"AANLkTikwpruLt-D0tYn+eP5JpB1h0RD55+xr_7S+RfOa@mail.gmail.com","threadId":"24489","inReplyTo":"AANLkTinyjinsGS2ddyMOW=3zDN1ThJcpF+dc4cBE6L9D@mail.gmail.com","subject":"Re: RFC: Between git-subtree and git-submodules","fromName":"Santi Béjar","fromEmail":"santi@agolina.net","sentAt":"2010-07-25T00:28:04Z","receivedAt":"2010-07-25T00:28:04Z","isPatch":false,"sender":{"key":"santi@agolina.net","avatar":null},"body":"> With #3, I can see that you want something different than I do; you\n> want to silently revert your own patches out of the submodule's\n> history, when you upgrade the submodule to a new version.  Personally,\n> I find this concept a bit objectionable (it's like \"git merge -s\n> ours\"), but okay, it's pretty easy to implement, and you've submitted\n> a patch to git-subtree that does it.  My question is: why would you\n> want this?  Isn't it clearer to 'git revert' the patches you don't\n> want?\n\nWhile writing the docs for the --prune option I've found this sentence in\ngit-subtree.txt:\n\n\t'git subtree merge --squash'\n\talways adjusts the subtree to match the exactly\n\tspecified commit, even if getting to that commit would\n\trequire undoing some changes that were added earlier.\n\nI think this is not true, and it contradicts with what you've just said. Or I\ndon't understand it correctly.\n\nThanks,\nSanti\n"},{"id":"146342","messageId":"AANLkTim_XeoEeRBa8VCLMn=rbFb3mvKm6OJ=M7D6YJBz@mail.gmail.com","threadId":"24489","inReplyTo":"AANLkTikwpruLt-D0tYn+eP5JpB1h0RD55+xr_7S+RfOa@mail.gmail.com","subject":"Re: RFC: Between git-subtree and git-submodules","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-07-26T05:02:54Z","receivedAt":"2010-07-26T05:02:54Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Sat, Jul 24, 2010 at 8:28 PM, Santi Béjar <santi@agolina.net> wrote:\n>> With #3, I can see that you want something different than I do; you\n>> want to silently revert your own patches out of the submodule's\n>> history, when you upgrade the submodule to a new version.  Personally,\n>> I find this concept a bit objectionable (it's like \"git merge -s\n>> ours\"), but okay, it's pretty easy to implement, and you've submitted\n>> a patch to git-subtree that does it.  My question is: why would you\n>> want this?  Isn't it clearer to 'git revert' the patches you don't\n>> want?\n>\n> While writing the docs for the --prune option I've found this sentence in\n> git-subtree.txt:\n>\n>        'git subtree merge --squash'\n>        always adjusts the subtree to match the exactly\n>        specified commit, even if getting to that commit would\n>        require undoing some changes that were added earlier.\n>\n> I think this is not true, and it contradicts with what you've just said. Or I\n> don't understand it correctly.\n\nI agree that this is a bit unclear.  Basically it rewinds the *base*\nof the subtree to match the commit you specify.  Patches you've\napplied locally are not removed.\n\nWhat it actually does is it creates a new commit that applies the\nfollowing patch:\n\n    git diff oldbase newbase\n\nIf you've made any of your own patches that conflict with this, it'll\ngenerate the conflict and then stop.  Otherwise it'll merge in your\nchanges as you might expect.\n\nHave fun,\n\nAvery\n"}]}