{"thread":{"id":"27109","subject":"[PATCH] git-filter-branch: add --egrep-filter option","startedAt":"2011-04-15T22:50:24Z","lastAt":"2011-04-19T16:03:13Z","messageCount":5,"participants":["Michael O'Cleirigh","Johannes Sixt","Jonathan Nieder","Phil Hord"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"165935","messageId":"4DA8CBB0.6080103@rivulet.ca","threadId":"27109","inReplyTo":null,"subject":"[PATCH] git-filter-branch: add --egrep-filter option","fromName":"Michael O'Cleirigh","fromEmail":"michael.ocleirigh@rivulet.ca","sentAt":"2011-04-15T22:50:24Z","receivedAt":"2011-04-15T22:50:24Z","isPatch":true,"sender":{"key":"michael.ocleirigh@rivulet.ca","avatar":"https://gravatar.com/avatar/80727c318d003bd815407d8f93c2d01ce86c233b431d24de2efef3409b9415cd?d=mp&s=160"},"body":"The --subdirectory-filter will look for a single directory and then rewrite\nhistory to make its content the root. This is ok except for cases where we\nwant to retain history of those files before they were moved into that\ndirectory.\n\nThe --egrep-filter option allows specifying an egrep regex for the files in the\ntree of each commit to keep. For example:\n\nDirectories we want are A, B, C, D and they exist in several different\nlifetimes. A and B exist sometimes together then B and C and finally then D.\n\ne.g. git-filter-branch --egrep-filter \"(A|B|C|D)\"\n\nEach commit will then contain different combination's of A or B or C or D (up to A and B and C and D).\n---\n git-filter-branch.sh | 12 ++++++++++++\n 1 files changed, 12 insertions(+), 0 deletions(-)\n\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex 962a93b..2392ad6 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -101,6 +101,7 @@ USAGE=\"[--env-filter] [--tree-filter]\n [--index-filter] [--parent-filter]\n [--msg-filter] [--commit-filter]\n [--tag-name-filter] [--subdirectory-filter]\n+\t\t\t[--egrep-filter]\n [--original] [-d] [-f | --force]\n [...]\"\n\n@@ -122,6 +123,7 @@ filter_msg=cat\n filter_commit=\n filter_tag_name=\n filter_subdir=\n+filter_egrep=\n orig_namespace=refs/original/\n force=\n prune_empty=\n@@ -191,6 +193,10 @@ do\n \t\tfilter_subdir=\"$OPTARG\"\n \t\tremap_to_ancestor=t\n \t\t;;\n+\t--egrep-filter)\n+\t\tfilter_egrep=\"$OPTARG\"\n+\t\tremap_to_ancestor=t\n+\t\t;;\n \t--original)\n \t\torig_namespace=$(expr \"$OPTARG/\" : '\\(.*[^/]\\)/*$')/\n \t\t;;\n@@ -317,6 +323,12 @@ while read commit parents; do\n \t\t}\n \tesac || die \"Could not initialize the index\"\n\n+\tif [ \"$filter_egrep\" ]; then\n+\n+\t\tgit ls-tree $commit | egrep \"$filter_egrep\" | git mktree | xargs git read-tree -i -m\n+\n+\tfi\n+\n \tGIT_COMMIT=$commit\n \texport GIT_COMMIT\n \tgit cat-file commit \"$commit\">../commit ||\n-- 1.7.2.3\n"},{"id":"165957","messageId":"201104161016.51690.j6t@kdbg.org","threadId":"27109","inReplyTo":"4DA8CBB0.6080103@rivulet.ca","subject":"Re: [PATCH] git-filter-branch: add --egrep-filter option","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2011-04-16T08:16:51Z","receivedAt":"2011-04-16T08:16:51Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"On Samstag, 16. April 2011, Michael O'Cleirigh wrote:\n> The --subdirectory-filter will look for a single directory and then rewrite\n> history to make its content the root. This is ok except for cases where we\n> want to retain history of those files before they were moved into that\n> directory.\n>\n> The --egrep-filter option allows specifying an egrep regex for the files in\n> the tree of each commit to keep. For example:\n>\n> Directories we want are A, B, C, D and they exist in several different\n> lifetimes. A and B exist sometimes together then B and C and finally then\n> D.\n>\n> e.g. git-filter-branch --egrep-filter \"(A|B|C|D)\"\n>\n> Each commit will then contain different combination's of A or B or C or D\n> (up to A and B and C and D).\n\nWhy do you need a new --...-filter option for this? Your implementation is \nmerely an instance of an --index-filter, and at that a very specialized one, \nwhich operates only at the top-most directory level.\n\n> +               git ls-tree $commit | egrep \"$filter_egrep\" | git mktree | \nxargs git read-tree -i -m\n\n-- Hannes\n"},{"id":"165996","messageId":"4DAA464E.7010804@rivulet.ca","threadId":"27109","inReplyTo":"201104161016.51690.j6t@kdbg.org","subject":"Re: [PATCH] git-filter-branch: add --egrep-filter option","fromName":"Michael O'Cleirigh","fromEmail":"michael.ocleirigh@rivulet.ca","sentAt":"2011-04-17T01:45:50Z","receivedAt":"2011-04-17T01:45:50Z","isPatch":true,"sender":{"key":"michael.ocleirigh@rivulet.ca","avatar":"https://gravatar.com/avatar/80727c318d003bd815407d8f93c2d01ce86c233b431d24de2efef3409b9415cd?d=mp&s=160"},"body":"Hi Johannes,\n\nThanks for commenting on this patch.\n\n> On Samstag, 16. April 2011, Michael O'Cleirigh wrote:\n>> The --subdirectory-filter will look for a single directory and then rewrite\n>> history to make its content the root. This is ok except for cases where we\n>> want to retain history of those files before they were moved into that\n>> directory.\n>>\n>> The --egrep-filter option allows specifying an egrep regex for the files in\n>> the tree of each commit to keep. For example:\n>>\n>> Directories we want are A, B, C, D and they exist in several different\n>> lifetimes. A and B exist sometimes together then B and C and finally then\n>> D.\n>>\n>> e.g. git-filter-branch --egrep-filter \"(A|B|C|D)\"\n>>\n>> Each commit will then contain different combination's of A or B or C or D\n>> (up to A and B and C and D).\n> Why do you need a new --...-filter option for this? Your implementation is\n> merely an instance of an --index-filter, and at that a very specialized one,\n> which operates only at the top-most directory level.\n>\n\nAt work we needed to split out 2 more modules from a 1400 revision \nrepository that we imported from subversion.\n\nEach had been originally created under different names at the top level \nand then only recently moved into a more logical single directory per \nproject structure. When we first ran filter-branch with the \n--subdirectory-filter we only had 6 commits instead of the 100 commits \nwe ended up with after using the --egrep-filter method.\n\nI tried a tree-filter first but it was slow and then the same method as \nan index filter was slower (I would search for the paths that didn't \nmatch the filter (egrep -v \"pattern\") and then remove each of them).\n\nBy using this egrep-filter option it only took 5 minutes per repo vs >8 \nhours for the tree-filter approach.\n\nI posted to the list incase it might be useful to others; But I didn't \nreally know if it would be useful or not.\n\nAfter considering your comment I have to agree with you that it is a \nspecial case of index-filter and probably not useful/general for enough \nother cases to justify adding in a new command line option.\n\nRegards,\n\nMike\n"},{"id":"166067","messageId":"20110419080143.GA17915@elie","threadId":"27109","inReplyTo":"4DAA464E.7010804@rivulet.ca","subject":"Re: [PATCH] git-filter-branch: add --egrep-filter option","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-04-19T08:01:43Z","receivedAt":"2011-04-19T08:01:43Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nMichael O'Cleirigh wrote:\n\n> After considering your comment I have to agree with you that it is a\n> special case of index-filter and probably not useful/general for\n> enough other cases to justify adding in a new command line option.\n\nNow, why do you give up so easily? ;-)\n\nSurely what your patch is hinting at is the possibility of an\n--ls-tree-filter (for lack of a better name) that works with trees\nwithout the overhead of unpacking them. On the other hand I do agree\nwith Hannes that allowing only \"egrep\" is a bit overspecialized.\n\nIn practice I would have used something like\n\n\t--commit-filter='\n\t\ttree=$1 &&\n\t\tnew_tree=$(\n\t\t\tgit ls-tree $commit |\n\t\t\tegrep \"$filter_egrep\" |\n\t\t\tgit mktree\n\t\t) &&\n\t\tshift &&\n\t\tgit_commit_non_empty_tree \"$new_tree\" \"$@\"\n\t'\n\nso another (simpler?) solution might be an entry for the EXAMPLES\nsection of the manual along these lines.\n\nCiao,\nJonathan\n"},{"id":"166087","messageId":"4DADB241.1060602@cisco.com","threadId":"27109","inReplyTo":"20110419080143.GA17915@elie","subject":"Re: [PATCH] git-filter-branch: add --egrep-filter option","fromName":"Phil Hord","fromEmail":"hordp@cisco.com","sentAt":"2011-04-19T16:03:13Z","receivedAt":"2011-04-19T16:03:13Z","isPatch":true,"sender":{"key":"phil.hord@gmail.com","avatar":"https://avatars.githubusercontent.com/u/123908?v=4"},"body":"\n\nOn 04/19/2011 04:01 AM, Jonathan Nieder wrote:\n> Hi,\n>\n> Michael O'Cleirigh wrote:\n>\n>> After considering your comment I have to agree with you that it is a\n>> special case of index-filter and probably not useful/general for\n>> enough other cases to justify adding in a new command line option.\n> Now, why do you give up so easily? ;-)\n>\n> Surely what your patch is hinting at is the possibility of an\n> --ls-tree-filter (for lack of a better name) that works with trees\n> without the overhead of unpacking them.\n\nI have invented something similar[*] for git three different times in\nthree different ways. The last one is the fastest and uses\ngit-fast-import instead of filter-branch, but I was sure one of the\nfilter-branch methods would have been more efficient. More examples\nwould be very welcome.\n\nPhil\n\n[*] My implementations mostly focused on applying a \".gitignore\" file to\nthe repo history. I spent many hours on this. I wound up with a script\nthat also handles file and branch renaming (the latter important so I\ncan run different filters on the same repo and drop results into\ndifferent branches). It's not patch-worthy (yet), but I would have\nloved to have more examples along the way such as the mktree one you\njust provided.\n"}]}