threads / patch / 8495

patchTeach filter-branch about subdirectory filtering

Subject: [PATCH] Teach filter-branch about subdirectory filtering

## tl;dr

7 messages between Jun 8, 2007 and Jun 8, 2007. Diffs are folded; open one to read it.

replies: 6people: 4as markdown or json

Johannes Schindelin· Jun 8, 2007, 00:30 UTC · lore

With git-filter-branch --subdirectory-filter <subdirectory> you can get at the history, as seen by a certain subdirectory. The history of the rewritten branch will only contain commits that touched that subdirectory, and the subdirectory will be rewritten to be the new project root.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 git-filter-branch.sh     |   33 ++++++++++++++++++++++++++++++---
 t/t7003-filter-branch.sh |   24 ++++++++++++++++++++++++
 2 files changed, 54 insertions(+), 3 deletions(-)
Show changes to 2 files +54 −3

git-filter-branch.sh, t/t7003-filter-branch.sh

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index e6ed7b9..4990729 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -122,6 +122,10 @@
 #	attached, the rewritten tag won't have it. Sorry. (It is by
 #	definition impossible to preserve signatures at any rate, though.)
 #
+# --subdirectory-filter DIRECTORY:: Only regard the history, as seen by
+#	the given subdirectory. The result will contain that directory as
+#	its project root.
+#
 # EXAMPLE USAGE
 # -------------
 # Suppose you want to remove a file (containing confidential information
@@ -227,7 +231,13 @@ set_ident () {
 
 # list all parent's object names for a given commit
 get_parents () {
-	git-rev-list -1 --parents "$1" | sed "s/^[0-9a-f]*//"
+	case "$filter_subdir" in
+	"")
+		git-rev-list -1 --parents "$1"
+		;;
+	*)
+		git-rev-list -1 --parents "$1" -- "$filter_subdir"
+	esac | sed "s/^[0-9a-f]*//"
 }
 
 tempdir=.git-rewrite
@@ -238,6 +248,7 @@ filter_parent=
 filter_msg=cat
 filter_commit='git-commit-tree "$@"'
 filter_tag_name=
+filter_subdir=
 while case "$#" in 0) usage;; esac
 do
 	case "$1" in
@@ -283,6 +294,9 @@ do
 	--tag-name-filter)
 		filter_tag_name="$OPTARG"
 		;;
+	--subdirectory-filter)
+		filter_subdir="$OPTARG"
+		;;
 	*)
 		usage
 		;;
@@ -316,7 +330,14 @@ ret=0
 
 mkdir ../map # map old->new commit ids for rewriting parents
 
-git-rev-list --reverse --topo-order --default HEAD "$@" >../revs
+case "$filter_subdir" in
+"")
+	git-rev-list --reverse --topo-order --default HEAD "$@"
+	;;
+*)
+	git-rev-list --reverse --topo-order --default HEAD "$@" \
+		-- "$filter_subdir"
+esac > ../revs
 commits=$(cat ../revs | wc -l | tr -d " ")
 
 test $commits -eq 0 && die "Found nothing to rewrite"
@@ -326,7 +347,13 @@ while read commit; do
 	i=$(($i+1))
 	printf "\rRewriting commits... ($i/$commits)"
 
-	git-read-tree -i -m $commit
+	case "$filter_subdir" in
+	"")
+		git-read-tree -i -m $commit
+		;;
+	*)
+		git-read-tree -i -m $commit:"$filter_subdir"
+	esac
 
 	export GIT_COMMIT=$commit
 	git-cat-file commit "$commit" >../commit
diff --git a/t/t7003-filter-branch.sh b/t/t7003-filter-branch.sh
index 3739cb1..292b837 100755
--- a/t/t7003-filter-branch.sh
+++ b/t/t7003-filter-branch.sh
@@ -54,4 +54,28 @@ test_expect_success 'common ancestor is still common (unchanged)' '
 	test "$(git-merge-base modD D)" = "$(git-rev-parse B)"
 '
 
+test_expect_success 'filter subdirectory only' '
+	mkdir subdir &&
+	touch subdir/new &&
+	git add subdir/new &&
+	test_tick &&
+	git commit -m "subdir" &&
+	echo H > a &&
+	test_tick &&
+	git commit -m "not subdir" a &&
+	echo A > subdir/new &&
+	test_tick &&
+	git commit -m "again subdir" subdir/new &&
+	git rm a &&
+	test_tick &&
+	git commit -m "again not subdir" &&
+	git-filter-branch --subdirectory-filter subdir sub
+'
+
+test_expect_success 'subdirectory filter result looks okay' '
+	test 2 = $(git-rev-list sub | wc -l) &&
+	git show sub:new &&
+	! git show sub:subdir
+'
+
 test_done
-- 
1.5.2.1.2689.gaf768-dirty
Junio C Hamano· Jun 8, 2007, 00:57 UTC · re: Johannes Schindelin · lore

Re: [PATCH] Teach filter-branch about subdirectory filtering

Good thing you folded git-split into this, which is an obviously right thing to do.

But it makes me wonder, if it is common to want to split a combined project into two (or more) sets, taking more than one directories at a time. Contents of directory A and B (without hoisting them up) are filtered into one history and remaining bits (again without hoisting them up) are filtered into another.

I do not think that is common, and you can easily do that with the index filter, I think, so it probably is pointless to make this new filter to try supporting that mode of operation.

We might however want to make sure that we detect an attempt to give more than one --subdirectory-filter parameter.

Johannes Schindelin· Jun 8, 2007, 05:04 UTC · re: Junio C Hamano · lore

Re: [PATCH] Teach filter-branch about subdirectory filtering

Hi,
On Thu, 7 Jun 2007, Junio C Hamano wrote:
> Good thing you folded git-split into this, which is an obviously right 
> thing to do.
So my secritt evvil plann did not stay secritt that long.
> But it makes me wonder, if it is common to want to split a combined 
> project into two (or more) sets, taking more than one directories at a 
> time.

It's just a guess, but now that we come really, really close to having a concise implementation of git-subproject which will probably soon propagate to master, and then maint, I gather that more and more people come and want to split their projects (which they maintained as one big project) into several subprojects (which they should have been from the start, but the tool did not easily allow for that).

> We might however want to make sure that we detect an attempt to give 
> more than one --subdirectory-filter parameter.

Yeah. But then, I think that rev-list actually does not like to be passed a file/directory which does not exist currently, so that is probably a sensible safeguard against that usage.

Unless you think -- as I fear you do -- that people could think git-filter-branch (its name not withstanding) filters _multiple_ branches _at once_.

That case I did not foresee, and you're right, I should output a big fat error for the case when somebody says "filter-branch --subdirectory-filter a/ --subdirectory-filter b/". It does not make any sense. But humans are known for their unique feeling that things should work, even if they make no sense at all, and their brain should have warned them (that it is unlikely, at best, to work), but failed to do so.

Ciao, Dscho

Johannes Sixt· Jun 8, 2007, 07:19 UTC · re: Johannes Schindelin · lore

Re: [PATCH] Teach filter-branch about subdirectory filtering

Johannes Schindelin wrote:
Show 6 quoted lines
> It's just a guess, but now that we come really, really close to having a
> concise implementation of git-subproject which will probably soon
> propagate to master, and then maint, I gather that more and more people
> come and want to split their projects (which they maintained as one big
> project) into several subprojects (which they should have been from the
> start, but the tool did not easily allow for that).

I think that --subdirectry-filter needs to become a bit smarter to be really useful for splitting a big project into sub-projects plus a super-project. The reason is that once you have extracted the sub-project(s), you have a hard time to find out which commits to gitlink into the super-project. I don't have a plan how to make it smarter, though.

-- Hannes
Johannes Schindelin· Jun 8, 2007, 13:03 UTC · re: Johannes Sixt · lore

Re: [PATCH] Teach filter-branch about subdirectory filtering

Hi,
On Fri, 8 Jun 2007, Johannes Sixt wrote:
Show 14 quoted lines
> Johannes Schindelin wrote:
> > It's just a guess, but now that we come really, really close to having 
> > a concise implementation of git-subproject which will probably soon 
> > propagate to master, and then maint, I gather that more and more 
> > people come and want to split their projects (which they maintained as 
> > one big project) into several subprojects (which they should have been 
> > from the start, but the tool did not easily allow for that).
> 
> I think that --subdirectry-filter needs to become a bit smarter to be 
> really useful for splitting a big project into sub-projects plus a 
> super-project. The reason is that once you have extracted the 
> sub-project(s), you have a hard time to find out which commits to 
> gitlink into the super-project. I don't have a plan how to make it 
> smarter, though.
Yes, it seems a good direction to follow.

I will have to think about it, but I guess that with yet another set of files in ../map/ (probably prefixed by the subdirectory name), we should even be able to smartly filter a huge project into a super-project together with its subprojects.

The command line option would be something like "--split-superproject dir1,dir2,dir3", where dir<n> are the subdirectories that are to become the subprojects.

Ciao, Dscho

Johannes Sixt· Jun 8, 2007, 13:19 UTC · re: Johannes Schindelin · lore

Re: [PATCH] Teach filter-branch about subdirectory filtering

Johannes Schindelin wrote:
Show 11 quoted lines
>  # list all parent's object names for a given commit
>  get_parents () {
> -       git-rev-list -1 --parents "$1" | sed "s/^[0-9a-f]*//"
> +       case "$filter_subdir" in
> +       "")
> +               git-rev-list -1 --parents "$1"
> +               ;;
> +       *)
> +               git-rev-list -1 --parents "$1" -- "$filter_subdir"
> +       esac | sed "s/^[0-9a-f]*//"
>  }
I think --full-history in the subdir case would make a lot of sense.
-- Hannes
Alex Riesen· Jun 8, 2007, 21:05 UTC · re: Johannes Schindelin · lore

Re: [PATCH] Teach filter-branch about subdirectory filtering

Johannes Schindelin, Fri, Jun 08, 2007 02:30:35 +0200:
Show 7 quoted lines
> 
> With git-filter-branch --subdirectory-filter <subdirectory> you can
> get at the history, as seen by a certain subdirectory. The history
> of the rewritten branch will only contain commits that touched that
> subdirectory, and the subdirectory will be rewritten to be the new
> project root.
> 
Maybe it could be made more general
    --subdirectory-filter <from> [--subdirectory-filter-to <to>]

<to> meaning root by default, and allow multiple filters to be set? You would have to deal with file name conflicts in case some filters refer to the same target directory, though.

It has already been asked how to split a project into many projects based on directory boundaries. That'd help...

← back to recent threads