{"thread":{"id":"26168","subject":"[RFC PATCH 1/3] filter-branch: optionally dump all mappings at the end","startedAt":"2010-12-31T15:29:21Z","lastAt":"2011-01-04T19:18:32Z","messageCount":11,"participants":["Thomas Rast","Johannes Sixt","Jeffrey Phillips Freeman","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"158784","messageId":"cover.1293809100.git.trast@student.ethz.ch","threadId":"26168","inReplyTo":null,"subject":"[RFC PATCH 0/3] Submodule filtering for filter-branch","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2010-12-31T15:29:21Z","receivedAt":"2010-12-31T15:29:21Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Jeffrey Phillips Freeman aka tty1 asked on IRC about a way to fix the\nsuper-project after rewriting submodules.  This quick series provides\nsome extra tricks in the filter-branch toolbox that can be plumbed\ntogether to achieve this (see the last commit's message), and should\nhopefully be general enough to be of other use too.\n\nComes without docs and tests at this point, hence RFC.  It's also only\nlightly tested, but since Jeffrey has a use-case, I'll leave it up to\nhim to bend it a bit and see if anything breaks.\n\n\nThomas Rast (3):\n  filter-branch: optionally dump all mappings at the end\n  filter-branch: optionally load existing mappings prior to filtering\n  filter-branch: support --submodule-filter\n\n git-filter-branch.sh |   64 ++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 64 insertions(+), 0 deletions(-)\n\n-- \n1.7.4.rc0.240.g44e61\n"},{"id":"158783","messageId":"57609f1df897f19becc77d5c43c3c3608725160b.1293809100.git.trast@student.ethz.ch","threadId":"26168","inReplyTo":"cover.1293809100.git.trast@student.ethz.ch","subject":"[RFC PATCH 1/3] filter-branch: optionally dump all mappings at the end","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2010-12-31T15:29:22Z","receivedAt":"2010-12-31T15:29:22Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"In some cases, like the sub/super-project filtering attempted in this\nseries, it may be necessary to carry over the mappings from one or\nmore filter-branch run to another.\n\nAs the first part to this, make a --dump-map option that dumps a flat\ntext file with lines of the form\n\n  $sha1 $(map $sha1)\n\nafter it has established all mappings.\n\nSigned-off-by: Thomas Rast <trast@student.ethz.ch>\n---\n git-filter-branch.sh |   21 +++++++++++++++++++++\n 1 files changed, 21 insertions(+), 0 deletions(-)\n\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex 962a93b..020b076 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -126,6 +126,7 @@ orig_namespace=refs/original/\n force=\n prune_empty=\n remap_to_ancestor=\n+dump_map=\n while :\n do\n \tcase \"$1\" in\n@@ -194,6 +195,16 @@ do\n \t--original)\n \t\torig_namespace=$(expr \"$OPTARG/\" : '\\(.*[^/]\\)/*$')/\n \t\t;;\n+\t--dump-map)\n+\t\tcase \"$OPTARG\" in\n+\t\t/*)\n+\t\t\tdump_map=\"$OPTARG\"\n+\t\t\t;;\n+\t\t*)\n+\t\t\tdump_map=\"$(pwd)/$OPTARG\"\n+\t\t\t;;\n+\t\tesac\n+\t\t;;\n \t*)\n \t\tusage\n \t\t;;\n@@ -385,6 +396,16 @@ then\n \tdone < \"$tempdir\"/heads\n fi\n \n+# At this point the mappings are stable so we can dump them if\n+# requested\n+\n+if test -n \"$dump_map\"; then\n+\t( cd \"$workdir\"/../map/; ls ) |\n+\twhile read sha1; do\n+\t\techo $sha1 $(map $sha1)\n+\tdone > \"$dump_map\"\n+fi\n+\n # Finally update the refs\n \n _x40='[0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]'\n-- \n1.7.4.rc0.240.g44e61\n"},{"id":"158786","messageId":"f89203efcf1b2f49fb33097b3e7fc27e070626fb.1293809100.git.trast@student.ethz.ch","threadId":"26168","inReplyTo":"cover.1293809100.git.trast@student.ethz.ch","subject":"[RFC PATCH 2/3] filter-branch: optionally load existing mappings prior to filtering","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2010-12-31T15:29:23Z","receivedAt":"2010-12-31T15:29:23Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"In the previous commit filter-branch learned to dump its mappings.\nIntroduce a --load-map option that lets it load map files in the same\nformat.\n\nThis option does not accumulate; only the last one counts.  Letting it\naccumulate would require two argument-parsing passes since we need to\nknow the $tempdir before we can establish any maps.\n\nAlso, the maps add to all filtered maps.  This is a bit of a tradeoff.\nAs it stands, it could be useful to remap refs in another repository\nusing the map file from elsewere.  On the other hand, it does mean\nthat it may rewrite more refs than the user intended.\n\nSigned-off-by: Thomas Rast <trast@student.ethz.ch>\n---\n git-filter-branch.sh |   19 +++++++++++++++++++\n 1 files changed, 19 insertions(+), 0 deletions(-)\n\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex 020b076..9feeb26 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -127,6 +127,7 @@ force=\n prune_empty=\n remap_to_ancestor=\n dump_map=\n+load_map=\n while :\n do\n \tcase \"$1\" in\n@@ -205,6 +206,16 @@ do\n \t\t\t;;\n \t\tesac\n \t\t;;\n+\t--load-map)\n+\t\tcase \"$OPTARG\" in\n+\t\t/*)\n+\t\t\tload_map=\"$OPTARG\"\n+\t\t\t;;\n+\t\t*)\n+\t\t\tload_map=\"$(pwd)/$OPTARG\"\n+\t\t\t;;\n+\t\tesac\n+\t\t;;\n \t*)\n \t\tusage\n \t\t;;\n@@ -275,6 +286,14 @@ export GIT_INDEX_FILE\n # map old->new commit ids for rewriting parents\n mkdir ../map || die \"Could not create map/ directory\"\n \n+if test -n \"$load_map\"\n+then\n+\twhile read pre post\n+\tdo\n+\t\techo $post > ../map/$pre\n+\tdone < \"$load_map\"\n+fi\n+\n # we need \"--\" only if there are no path arguments in $@\n nonrevs=$(git rev-parse --no-revs \"$@\") || exit\n if test -z \"$nonrevs\"\n-- \n1.7.4.rc0.240.g44e61\n"},{"id":"158785","messageId":"44e6104ba28c80a6befe0f39fa4e2d6eeec56aa9.1293809100.git.trast@student.ethz.ch","threadId":"26168","inReplyTo":"cover.1293809100.git.trast@student.ethz.ch","subject":"[RFC PATCH 3/3] filter-branch: support --submodule-filter","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2010-12-31T15:29:24Z","receivedAt":"2010-12-31T15:29:24Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"This new filter gets each submodule's current sha1 and path on stdin,\nseparated by a tab.  It can then emit a new submodule sha1 and/or\npath, and filter-branch will:\n\n* if the path differs, remove the submodule at the old path;\n\n* add/replace the new sha1 at the new path.\n\nAdditionally, returning an empty new sha1 deletes the submodule.\n\nYou can tie this together with the last two commits to filter the\nsuper-project after a subproject filtering as follows:\n\n  ( cd sub1 && git filter-branch --dump-map map <filters|args> )\n  ( cd sub2 && git filter-branch --dump-map map <filters|args> )\n  cat sub1/map sub2/map > map\n  git filter-branch --load-map map \\\n  \t--submodule-filter \"map $(cut -f1)\" \\\n\t<args>\n\nOther use-cases should also be covered since we also pass through the\npath.\n\nSigned-off-by: Thomas Rast <trast@student.ethz.ch>\n---\n git-filter-branch.sh |   24 ++++++++++++++++++++++++\n 1 files changed, 24 insertions(+), 0 deletions(-)\n\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex 9feeb26..4a321c4 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -122,6 +122,7 @@ filter_msg=cat\n filter_commit=\n filter_tag_name=\n filter_subdir=\n+filter_submodule=\n orig_namespace=refs/original/\n force=\n prune_empty=\n@@ -193,6 +194,9 @@ do\n \t\tfilter_subdir=\"$OPTARG\"\n \t\tremap_to_ancestor=t\n \t\t;;\n+\t--submodule-filter)\n+\t\tfilter_submodule=\"$OPTARG\"\n+\t\t;;\n \t--original)\n \t\torig_namespace=$(expr \"$OPTARG/\" : '\\(.*[^/]\\)/*$')/\n \t\t;;\n@@ -379,6 +383,26 @@ while read commit parents; do\n \teval \"$filter_index\" < /dev/null ||\n \t\tdie \"index filter failed: $filter_index\"\n \n+\tif [ \"$filter_submodule\" ]; then\n+\t\tgit ls-files -s |\n+\t\tgrep '^160000' |\n+\t\twhile read mode sha1 stage path; do\n+\t\t\tprintf \"$sha1\\t$path\\n\" |\n+\t\t\t{ eval \"$filter_submodule\" ||\n+\t\t\t\tdie \"submodule filter failed: $filter_submodule\"; } |\n+\t\t\tread newsha1 newpath\n+\t\t\tif [ -z \"$newsha1\" ] || [ \"$path\" != \"$newpath\" ]; then\n+\t\t\t\tgit update-index --remove \"$path\" ||\n+\t\t\t\t\tdie \"failed to remove submodule $path\"\n+\t\t\tfi\n+\t\t\tif [ -n \"$newsha1\" ] && [ \"$sha1\" != \"$newsha1\" ]; then\n+\t\t\t\tgit update-index --add --replace --cacheinfo \\\n+\t\t\t\t\t160000 \"$newsha1\" \"$newpath\" ||\n+\t\t\t\t\tdie \"failed to add submodule $newpath\"\n+\t\t\tfi\n+\t\tdone\n+\tfi\n+\n \tparentstr=\n \tfor parent in $parents; do\n \t\tfor reparent in $(map \"$parent\"); do\n-- \n1.7.4.rc0.240.g44e61\n"},{"id":"158788","messageId":"201012311809.24205.j6t@kdbg.org","threadId":"26168","inReplyTo":"57609f1df897f19becc77d5c43c3c3608725160b.1293809100.git.trast@student.ethz.ch","subject":"Re: [RFC PATCH 1/3] filter-branch: optionally dump all mappings at the end","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2010-12-31T17:09:24Z","receivedAt":"2010-12-31T17:09:24Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"On Freitag, 31. Dezember 2010, Thomas Rast wrote:\n> In some cases, like the sub/super-project filtering attempted in this\n> series, it may be necessary to carry over the mappings from one or\n> more filter-branch run to another.\n>\n> As the first part to this, make a --dump-map option that dumps a flat\n> text file with lines of the form\n>\n>   $sha1 $(map $sha1)\n\nDespite the alternative implementation that I suggest below, the commit \nmessage (and also the documentation to be written) should state the output \nformat in this way.\n\n> @@ -194,6 +195,16 @@ do\n>  \t--original)\n>  \t\torig_namespace=$(expr \"$OPTARG/\" : '\\(.*[^/]\\)/*$')/\n>  \t\t;;\n> +\t--dump-map)\n> +\t\tcase \"$OPTARG\" in\n> +\t\t/*)\n\nPlease use\n\n\tif is_absolute_path \"$OPTARG\"\n\tthen\n\t...\n\n> +# At this point the mappings are stable so we can dump them if\n> +# requested\n> +\n> +if test -n \"$dump_map\"; then\n> +\t( cd \"$workdir\"/../map/; ls ) |\n> +\twhile read sha1; do\n> +\t\techo $sha1 $(map $sha1)\n\nThere must be a more efficient way than to spawn an additional process for \neach mapped SHA1. Since this excercises only one branch in the map() function \nanyway, how about this:\n\n\t( cd \"$workdir\"/../map/ && ls -1 ) |\n\twhile read sha1; do\n\t\tread mapped << $sha1 &&\n\t\techo $sha1 $mapped\n\tdone > \"$dump_map\"\n\nThough, there is some mechanism that maps one SHA1s to more than one, and my \napproach might not cover that case.\n\n> +\tdone > \"$dump_map\"\n\n-- Hannes\n"},{"id":"158789","messageId":"201012311810.53586.j6t@kdbg.org","threadId":"26168","inReplyTo":"f89203efcf1b2f49fb33097b3e7fc27e070626fb.1293809100.git.trast@student.ethz.ch","subject":"Re: [RFC PATCH 2/3] filter-branch: optionally load existing mappings prior to filtering","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2010-12-31T17:10:53Z","receivedAt":"2010-12-31T17:10:53Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"On Freitag, 31. Dezember 2010, Thomas Rast wrote:\n> +\t--load-map)\n> +\t\tcase \"$OPTARG\" in\n> +\t\t/*)\n\nPlease use is_absolute_path here as well.\n\n-- Hannes\n"},{"id":"158790","messageId":"201012311820.13863.j6t@kdbg.org","threadId":"26168","inReplyTo":"cover.1293809100.git.trast@student.ethz.ch","subject":"Re: [RFC PATCH 0/3] Submodule filtering for filter-branch","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2010-12-31T17:20:13Z","receivedAt":"2010-12-31T17:20:13Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"On Freitag, 31. Dezember 2010, Thomas Rast wrote:\n> Jeffrey Phillips Freeman aka tty1 asked on IRC about a way to fix the\n> super-project after rewriting submodules.  This quick series provides\n> some extra tricks in the filter-branch toolbox that can be plumbed\n> together to achieve this (see the last commit's message), and should\n> hopefully be general enough to be of other use too.\n\nI like the --dump-map and --load-map options. But I think that \na --submodule-filter is not necessary; it is just a \nparticular --index-filter, and IMO should be part of the documentation in the \nform of an example that illustrates how to use the new options.\n\n-- Hannes\n"},{"id":"158791","messageId":"201012311831.24057.j6t@kdbg.org","threadId":"26168","inReplyTo":"44e6104ba28c80a6befe0f39fa4e2d6eeec56aa9.1293809100.git.trast@student.ethz.ch","subject":"Re: [RFC PATCH 3/3] filter-branch: support --submodule-filter","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2010-12-31T17:31:23Z","receivedAt":"2010-12-31T17:31:23Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"On Freitag, 31. Dezember 2010, Thomas Rast wrote:\n> This new filter gets each submodule's current sha1 and path on stdin,\n> separated by a tab.  It can then emit a new submodule sha1 and/or\n> path, and filter-branch will:\n>\n> * if the path differs, remove the submodule at the old path;\n>\n> * add/replace the new sha1 at the new path.\n>\n> Additionally, returning an empty new sha1 deletes the submodule.\n>\n> You can tie this together with the last two commits to filter the\n> super-project after a subproject filtering as follows:\n>\n>   ( cd sub1 && git filter-branch --dump-map map <filters|args> )\n>   ( cd sub2 && git filter-branch --dump-map map <filters|args> )\n>   cat sub1/map sub2/map > map\n>   git filter-branch --load-map map \\\n>   \t--submodule-filter \"map $(cut -f1)\" \\\n> \t<args>\n>\n> Other use-cases should also be covered since we also pass through the\n> path.\n\nAs I said, I'm not particularly fond of a new --submodule-filter because it is \njust a special --index-filter.\n\nYour implementation is highly problematic:\n\n> +\tif [ \"$filter_submodule\" ]; then\n> +\t\tgit ls-files -s |\n> +\t\tgrep '^160000' |\n> +\t\twhile read mode sha1 stage path; do\n> +\t\t\tprintf \"$sha1\\t$path\\n\" |\n> +\t\t\t{ eval \"$filter_submodule\" ||\n> +\t\t\t\tdie \"submodule filter failed: $filter_submodule\"; } |\n\nThis 'die' will not do anything useful except to write an error message.\n\n> +\t\t\tread newsha1 newpath\n\nThis 'read' is part of a pipe, and as such many shells run it in a sub-shell; \nthe values that it reads do not survive the pipe, hence, the subsequent code \ndoes not do what you intend it.\n\nIn this case, you can put everything from 'read' to the last 'fi' below inside \na block { } because there are no process states that need to survive the \npipe.\n\n> +\t\t\tif [ -z \"$newsha1\" ] || [ \"$path\" != \"$newpath\" ]; then\n> +\t\t\t\tgit update-index --remove \"$path\" ||\n> +\t\t\t\t\tdie \"failed to remove submodule $path\"\n> +\t\t\tfi\n> +\t\t\tif [ -n \"$newsha1\" ] && [ \"$sha1\" != \"$newsha1\" ]; then\n> +\t\t\t\tgit update-index --add --replace --cacheinfo \\\n> +\t\t\t\t\t160000 \"$newsha1\" \"$newpath\" ||\n> +\t\t\t\t\tdie \"failed to add submodule $newpath\"\n> +\t\t\tfi\n> +\t\tdone\n\nThe whole if-branch is a pipe, and it's parts are run in a sub-shell (although \nshells are allowed to optimize this). This means that the 'die' calls will \nonly exit the pipe, but not terminate filter-branch. You at least need to \nhave '|| exit' behind the last 'fi' and &&-join the if-statements.\n\n> +\tfi\n> +\n>  \tparentstr=\n>  \tfor parent in $parents; do\n>  \t\tfor reparent in $(map \"$parent\"); do\n\n-- Hannes\n"},{"id":"158860","messageId":"4D225F63.1040502@syncleus.com","threadId":"26168","inReplyTo":"44e6104ba28c80a6befe0f39fa4e2d6eeec56aa9.1293809100.git.trast@student.ethz.ch","subject":"Re: [RFC PATCH 3/3] filter-branch: support --submodule-filter","fromName":"Jeffrey Phillips Freeman","fromEmail":"jeffrey.freeman@syncleus.com","sentAt":"2011-01-03T23:44:35Z","receivedAt":"2011-01-03T23:44:35Z","isPatch":true,"sender":{"key":"jeffrey.freeman@syncleus.com","avatar":null},"body":"So im kinda new with all this so bare with me guys. I wanted to figure \nout how to apply these patches, now i know i can use git to do this with \nits patch command and such. However i was curious does this exist as a \nbranch somewhere official or semi-official? I currently seem to be using \n--split-submodule which is itself in a git repo i have (which i want to \nalso find its source repo so i can keep up to date with it). So applying \nthe patch itself might be somewhat troubling due to conflicts, therefore \nid like to actually merge in a remote branch to keep things easy. So can \nyou guys point me to which repo would be best for me to keep up to date \nwith this would be?\n\nOn 12/31/2010 10:29 AM, Thomas Rast wrote:\n> This new filter gets each submodule's current sha1 and path on stdin,\n> separated by a tab.  It can then emit a new submodule sha1 and/or\n> path, and filter-branch will:\n>\n> * if the path differs, remove the submodule at the old path;\n>\n> * add/replace the new sha1 at the new path.\n>\n> Additionally, returning an empty new sha1 deletes the submodule.\n>\n> You can tie this together with the last two commits to filter the\n> super-project after a subproject filtering as follows:\n>\n>    ( cd sub1&&  git filter-branch --dump-map map<filters|args>  )\n>    ( cd sub2&&  git filter-branch --dump-map map<filters|args>  )\n>    cat sub1/map sub2/map>  map\n>    git filter-branch --load-map map \\\n>    \t--submodule-filter \"map $(cut -f1)\" \\\n> \t<args>\n>\n> Other use-cases should also be covered since we also pass through the\n> path.\n>\n> Signed-off-by: Thomas Rast<trast@student.ethz.ch>\n> ---\n>   git-filter-branch.sh |   24 ++++++++++++++++++++++++\n>   1 files changed, 24 insertions(+), 0 deletions(-)\n>\n> diff --git a/git-filter-branch.sh b/git-filter-branch.sh\n> index 9feeb26..4a321c4 100755\n> --- a/git-filter-branch.sh\n> +++ b/git-filter-branch.sh\n> @@ -122,6 +122,7 @@ filter_msg=cat\n>   filter_commit=\n>   filter_tag_name=\n>   filter_subdir=\n> +filter_submodule=\n>   orig_namespace=refs/original/\n>   force=\n>   prune_empty=\n> @@ -193,6 +194,9 @@ do\n>   \t\tfilter_subdir=\"$OPTARG\"\n>   \t\tremap_to_ancestor=t\n>   \t\t;;\n> +\t--submodule-filter)\n> +\t\tfilter_submodule=\"$OPTARG\"\n> +\t\t;;\n>   \t--original)\n>   \t\torig_namespace=$(expr \"$OPTARG/\" : '\\(.*[^/]\\)/*$')/\n>   \t\t;;\n> @@ -379,6 +383,26 @@ while read commit parents; do\n>   \teval \"$filter_index\"<  /dev/null ||\n>   \t\tdie \"index filter failed: $filter_index\"\n>\n> +\tif [ \"$filter_submodule\" ]; then\n> +\t\tgit ls-files -s |\n> +\t\tgrep '^160000' |\n> +\t\twhile read mode sha1 stage path; do\n> +\t\t\tprintf \"$sha1\\t$path\\n\" |\n> +\t\t\t{ eval \"$filter_submodule\" ||\n> +\t\t\t\tdie \"submodule filter failed: $filter_submodule\"; } |\n> +\t\t\tread newsha1 newpath\n> +\t\t\tif [ -z \"$newsha1\" ] || [ \"$path\" != \"$newpath\" ]; then\n> +\t\t\t\tgit update-index --remove \"$path\" ||\n> +\t\t\t\t\tdie \"failed to remove submodule $path\"\n> +\t\t\tfi\n> +\t\t\tif [ -n \"$newsha1\" ]&&  [ \"$sha1\" != \"$newsha1\" ]; then\n> +\t\t\t\tgit update-index --add --replace --cacheinfo \\\n> +\t\t\t\t\t160000 \"$newsha1\" \"$newpath\" ||\n> +\t\t\t\t\tdie \"failed to add submodule $newpath\"\n> +\t\t\tfi\n> +\t\tdone\n> +\tfi\n> +\n>   \tparentstr=\n>   \tfor parent in $parents; do\n>   \t\tfor reparent in $(map \"$parent\"); do\n"},{"id":"158885","messageId":"201101041414.19587.trast@student.ethz.ch","threadId":"26168","inReplyTo":"4D225F63.1040502@syncleus.com","subject":"Re: [RFC PATCH 3/3] filter-branch: support --submodule-filter","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2011-01-04T13:14:19Z","receivedAt":"2011-01-04T13:14:19Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Please don't top-post.  (There's nothing wrong with snipping the whole\nmessage if it does not really relate, as in this case.)\n\nCcs for the patch at the end; I don't really care much but I could\nroll all of it into a sort of \"submodule tools\" series for g-f-b, so\nif you like it, speak up.\n\nJeffrey Phillips Freeman wrote:\n> So im kinda new with all this so bare with me guys. I wanted to figure \n> out how to apply these patches, now i know i can use git to do this with \n> its patch command and such. However i was curious does this exist as a \n> branch somewhere official or semi-official?\n\nNot really.\n\nYou can dig through the mailing list archives and also e.g.\ngitworkflows to see how Junio handles incoming patches, but for series\nlike this you usually have to apply them yourself.  I deliberately\nflagged it RFC because I wanted to get some feedback ... and because I\nwould have had to spend more time on it for docs&tests.\n\n> I currently seem to be using\n> --split-submodule which is itself in a git repo i have (which i want to \n> also find its source repo so i can keep up to date with it).\n\nFor others reading this, I wrote --split-submodule also based on an\nIRC request from Jeffrey.  The patch is at the end.  But it's way less\nthought through than the --{dump,load}-map feature.  In particular\nI've been wondering whether it's possible to use the latter to\nimplement --split-submodule as a fairly concise index filter.\n\n> So applying \n> the patch itself might be somewhat troubling due to conflicts, therefore \n> id like to actually merge in a remote branch to keep things easy. So can \n> you guys point me to which repo would be best for me to keep up to date \n> with this would be?\n\nYou'll get the same conflicts from merging two little branches with\nthe features on them as with a 'git am -3'.\n\nMany patches are never pushed to a public repo (there are some notable\nexceptions in longer-running work).  The hassle of sending and\napplying email is far outweighed by the ease of review.  Many reviews\nhappen without applying the series at all.  That I pushed both of them\nto a public repo was an exception for your convenience.\n\n\n--- 8< ---\nSubject: TOY PATCH: filter-branch --split-submodule\n\nSometimes it makes sense to split out a path not as a subdirectory\n(that would be merged by subtree-merge), but as a submodule.  Since\ngit objects are just shaped in the right way, this is actually quite\neasy to do in a way that maintains the correct history relations:\n\nWhen splitting out DIR in commit C, the submodule commit has tree\nC:DIR and the rewritten superproject commit C' gets a submodule entry\nat C:DIR instead.\n\nParent rewriting is done in the obvious way: submodule commits have\ntheir corresponding submodule-changing ancestors as parents.  These\nare easy to fetch since we can basically use $(map C^):DIR.\n\nThis is a toy patch because of its terrible interface: you will still\nhave to put the submodule in place.  As a start, you can\n\n  git clone . DIR\n  ( cd DIR && git reset --hard $(git rev-parse :DIR) )\n\nto get a sub-repo set to the correct commit.\n\nSigned-off-by: Thomas Rast <trast@student.ethz.ch>\n\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex 962a93b..329d85c 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -191,6 +191,9 @@ do\n \t\tfilter_subdir=\"$OPTARG\"\n \t\tremap_to_ancestor=t\n \t\t;;\n+\t--split-submodule)\n+\t\tsplit_submodule=\"$OPTARG\"\n+\t\t;;\n \t--original)\n \t\torig_namespace=$(expr \"$OPTARG/\" : '\\(.*[^/]\\)/*$')/\n \t\t;;\n@@ -349,6 +352,43 @@ while read commit parents; do\n \teval \"$filter_index\" < /dev/null ||\n \t\tdie \"index filter failed: $filter_index\"\n \n+\tif test -n \"$split_submodule\"; then\n+\t\tsub_differs=\n+\t\tsub_parents=\n+\t\tsub_commit=\n+\t\tsubmodule=\"$(git rev-parse --verify $commit:$split_submodule 2>/dev/null)\"\n+\t\tif test -z \"$parents\"; then\n+\t\t\tif test -n \"$submodule\"; then\n+\t\t\t\tsub_differs=t\n+\t\t\tfi\n+\t\tfi\n+\t\tfor parent in $parents; do\n+\t\t\tif ! test \"$(git rev-parse --verify $parent:$split_submodule 2>/dev/null)\" = \"$submodule\"; then\n+\t\t\t\tsub_differs=t\n+\t\t\tfi\n+\t\t\tfor reparent in $(map \"$parent\"); do\n+\t\t\t\tp=\"$(git rev-parse --verify $reparent:$split_submodule 2>/dev/null)\"\n+\t\t\t\tif test -n \"$p\"; then\n+\t\t\t\t\tsub_parents=\"$sub_parents -p $p\"\n+\t\t\t\tfi\n+\t\t\tdone\n+\t\tdone\n+\t\tif test -n \"$sub_differs\"; then\n+\t\t\tsub_commit=\"$(sed -e '1,/^$/d' <../commit |\n+\t\t\t\t      git commit-tree $submodule $sub_parents)\" || exit\n+\t\telse\n+\t\t\tfor parent in $parents; do\n+\t\t\t\tsub_commit=\"$(git rev-parse --verify \"$(map \"$parent\")\":$split_submodule 2>/dev/null)\"\n+\t\t\t\tbreak\n+\t\t\tdone\n+\t\tfi\n+\t\tif test -n \"$sub_commit\"; then\n+\t\t\tgit update-index --add --replace --cacheinfo 160000 $sub_commit \"$split_submodule\" || exit\n+\t\telse\n+\t\t\tgit update-index --remove \"$split_submodule\"\n+\t\tfi\n+\tfi\n+\n \tparentstr=\n \tfor parent in $parents; do\n \t\tfor reparent in $(map \"$parent\"); do\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"158906","messageId":"7vfwt8fpfb.fsf@alter.siamese.dyndns.org","threadId":"26168","inReplyTo":"201101041414.19587.trast@student.ethz.ch","subject":"Re: [RFC PATCH 3/3] filter-branch: support --submodule-filter","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-01-04T19:18:32Z","receivedAt":"2011-01-04T19:18:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Rast <trast@student.ethz.ch> writes:\n\n> Subject: TOY PATCH: filter-branch --split-submodule\n>\n> Sometimes it makes sense to split out a path not as a subdirectory\n> (that would be merged by subtree-merge), but as a submodule.  Since\n> git objects are just shaped in the right way, this is actually quite\n> easy to do in a way that maintains the correct history relations:\n\nThe patch from a cursory look feels sane.\n\n> @@ -349,6 +352,43 @@ while read commit parents; do\n>  \teval \"$filter_index\" < /dev/null ||\n>  \t\tdie \"index filter failed: $filter_index\"\n>  \n> +\tif test -n \"$split_submodule\"; then\n> +\t\tsub_differs=\n> +\t\tsub_parents=\n> +\t\tsub_commit=\n\nJust a style, but I find\n\n\tif test -n \"$split_submodule\"\n        then\n        \tsub_differs= sub_parents= sub_commit=\n\neasier to read.  Not a biggie, as the neighbourhood in the script already\nis infested in the other style, but I thought I'd mention it.\n\n> +\t\tsubmodule=\"$(git rev-parse --verify $commit:$split_submodule 2>/dev/null)\"\n\nDo we need double quotes around it?\n\n> +\t\tif test -z \"$parents\"; then\n> +\t\t\tif test -n \"$submodule\"; then\n> +\t\t\t\tsub_differs=t\n> +\t\t\tfi\n> +\t\tfi\n\n\tif test -z \"$parents\" && test -n \"$submodule\"\n        then\n        \tsub_differs=t\n\tfi\n\n> +\t\tfor parent in $parents; do\n> +\t\t\tif ! test \"$(git rev-parse --verify $parent:$split_submodule 2>/dev/null)\" = \"$submodule\"; then\n> +\t\t\t\tsub_differs=t\n> +\t\t\tfi\n\nIf even one of the parents is different, we say \"differs\"...\n\n> +\t\tif test -n \"$sub_differs\"; then\n> +\t\t\tsub_commit=\"$(sed -e '1,/^$/d' <../commit |\n> +\t\t\t\t      git commit-tree $submodule $sub_parents)\" || exit\n> +\t\telse\n> +\t\t\tfor parent in $parents; do\n> +\t\t\t\tsub_commit=\"$(git rev-parse --verify \"$(map \"$parent\")\":$split_submodule 2>/dev/null)\"\n> +\t\t\t\tbreak\n\n... so we can just pick from the first parent and know all of them are the\nsame (could be empty which also is fine).  Good.\n"}]}