threads / patch / 27109

patchgit-filter-branch: add --egrep-filter option

Subject: [PATCH] git-filter-branch: add --egrep-filter option

## tl;dr

5 messages between Apr 15, 2011 and Apr 19, 2011. Diffs are folded; open one to read it.

replies: 4people: 4as markdown or json

Michael O'Cleirigh· Apr 15, 2011, 22:50 UTC · lore

The --subdirectory-filter will look for a single directory and then rewrite history to make its content the root. This is ok except for cases where we want to retain history of those files before they were moved into that directory.

The --egrep-filter option allows specifying an egrep regex for the files in the tree of each commit to keep. For example:

Directories we want are A, B, C, D and they exist in several different lifetimes. A and B exist sometimes together then B and C and finally then D.

e.g. git-filter-branch --egrep-filter "(A|B|C|D)"

Each commit will then contain different combination's of A or B or C or D (up to A and B and C and D).
---
  git-filter-branch.sh |   12 ++++++++++++
  1 files changed, 12 insertions(+), 0 deletions(-)

Show changes to git-filter-branch.sh +12 −1

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index 962a93b..2392ad6 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -101,6 +101,7 @@ USAGE="[--env-filter<command>] [--tree-filter<command>]
              [--index-filter<command>] [--parent-filter<command>]
              [--msg-filter<command>] [--commit-filter<command>]
              [--tag-name-filter<command>] [--subdirectory-filter<directory>]
+			[--egrep-filter<filter>]
              [--original<namespace>] [-d<directory>] [-f | --force]
              [<rev-list options>...]"

@@ -122,6 +123,7 @@ filter_msg=cat
  filter_commit=
  filter_tag_name=
  filter_subdir=
+filter_egrep=
  orig_namespace=refs/original/
  force=
  prune_empty=
@@ -191,6 +193,10 @@ do
  		filter_subdir="$OPTARG"
  		remap_to_ancestor=t
  		;;
+	--egrep-filter)
+		filter_egrep="$OPTARG"
+		remap_to_ancestor=t
+		;;
  	--original)
  		orig_namespace=$(expr "$OPTARG/" : '\(.*[^/]\)/*$')/
  		;;
@@ -317,6 +323,12 @@ while read commit parents; do
  		}
  	esac || die "Could not initialize the index"

+	if [ "$filter_egrep" ]; then
+
+		git ls-tree $commit | egrep "$filter_egrep" | git mktree | xargs git read-tree -i -m
+
+	fi
+
  	GIT_COMMIT=$commit
  	export GIT_COMMIT
  	git cat-file commit "$commit">../commit ||
-- 1.7.2.3

Johannes Sixt· Apr 16, 2011, 08:16 UTC · re: Michael O'Cleirigh · lore

Re: [PATCH] git-filter-branch: add --egrep-filter option

On Samstag, 16. April 2011, Michael O'Cleirigh wrote:

Show 16 quoted lines

> The --subdirectory-filter will look for a single directory and then rewrite
> history to make its content the root.  This is ok except for cases where we
> want to retain history of those files before they were moved into that
> directory.
>
> The --egrep-filter option allows specifying an egrep regex for the files in
> the tree of each commit to keep.  For example:
>
> Directories we want are A, B, C, D and they exist in several different
> lifetimes.  A and B exist sometimes together then B and C and finally then
> D.
>
> e.g. git-filter-branch --egrep-filter "(A|B|C|D)"
>
> Each commit will then contain different combination's of A or B or C or D
> (up to A and B and C and D).

Why do you need a new --...-filter option for this? Your implementation is merely an instance of an --index-filter, and at that a very specialized one, which operates only at the top-most directory level.

> +               git ls-tree $commit | egrep "$filter_egrep" | git mktree |

xargs git read-tree -i -m

-- Hannes

Michael O'Cleirigh· Apr 17, 2011, 01:45 UTC · re: Johannes Sixt · lore

Re: [PATCH] git-filter-branch: add --egrep-filter option

Hi Johannes,

Thanks for commenting on this patch.

Show 21 quoted lines

> On Samstag, 16. April 2011, Michael O'Cleirigh wrote:
>> The --subdirectory-filter will look for a single directory and then rewrite
>> history to make its content the root.  This is ok except for cases where we
>> want to retain history of those files before they were moved into that
>> directory.
>>
>> The --egrep-filter option allows specifying an egrep regex for the files in
>> the tree of each commit to keep.  For example:
>>
>> Directories we want are A, B, C, D and they exist in several different
>> lifetimes.  A and B exist sometimes together then B and C and finally then
>> D.
>>
>> e.g. git-filter-branch --egrep-filter "(A|B|C|D)"
>>
>> Each commit will then contain different combination's of A or B or C or D
>> (up to A and B and C and D).
> Why do you need a new --...-filter option for this? Your implementation is
> merely an instance of an --index-filter, and at that a very specialized one,
> which operates only at the top-most directory level.
>

At work we needed to split out 2 more modules from a 1400 revision repository that we imported from subversion.

Each had been originally created under different names at the top level and then only recently moved into a more logical single directory per project structure. When we first ran filter-branch with the --subdirectory-filter we only had 6 commits instead of the 100 commits we ended up with after using the --egrep-filter method.

I tried a tree-filter first but it was slow and then the same method as an index filter was slower (I would search for the paths that didn't match the filter (egrep -v "pattern") and then remove each of them).

By using this egrep-filter option it only took 5 minutes per repo vs >8 hours for the tree-filter approach.

I posted to the list incase it might be useful to others; But I didn't really know if it would be useful or not.

After considering your comment I have to agree with you that it is a special case of index-filter and probably not useful/general for enough other cases to justify adding in a new command line option.

Regards,

Mike

Jonathan Nieder· Apr 19, 2011, 08:01 UTC · re: Michael O'Cleirigh · lore

Re: [PATCH] git-filter-branch: add --egrep-filter option

Hi,

Michael O'Cleirigh wrote:

> After considering your comment I have to agree with you that it is a
> special case of index-filter and probably not useful/general for
> enough other cases to justify adding in a new command line option.

Now, why do you give up so easily? ;-)

Surely what your patch is hinting at is the possibility of an --ls-tree-filter (for lack of a better name) that works with trees without the overhead of unpacking them. On the other hand I do agree with Hannes that allowing only "egrep" is a bit overspecialized.

In practice I would have used something like

	--commit-filter='
		tree=$1 &&
		new_tree=$(
			git ls-tree $commit |
			egrep "$filter_egrep" |
			git mktree
		) &&
		shift &&
		git_commit_non_empty_tree "$new_tree" "$@"
	'

so another (simpler?) solution might be an entry for the EXAMPLES section of the manual along these lines.

Ciao, Jonathan

Phil Hord· Apr 19, 2011, 16:03 UTC · re: Jonathan Nieder · lore

Re: [PATCH] git-filter-branch: add --egrep-filter option

On 04/19/2011 04:01 AM, Jonathan Nieder wrote:

Show 12 quoted lines

> Hi,
>
> Michael O'Cleirigh wrote:
>
>> After considering your comment I have to agree with you that it is a
>> special case of index-filter and probably not useful/general for
>> enough other cases to justify adding in a new command line option.
> Now, why do you give up so easily? ;-)
>
> Surely what your patch is hinting at is the possibility of an
> --ls-tree-filter (for lack of a better name) that works with trees
> without the overhead of unpacking them.

I have invented something similar[*] for git three different times in three different ways. The last one is the fastest and uses git-fast-import instead of filter-branch, but I was sure one of the filter-branch methods would have been more efficient. More examples would be very welcome.

Phil

[*] My implementations mostly focused on applying a ".gitignore" file to the repo history. I spent many hours on this. I wound up with a script that also handles file and branch renaming (the latter important so I can run different filters on the same repo and drop results into different branches). It's not patch-worthy (yet), but I would have loved to have more examples along the way such as the mktree one you just provided.