{"thread":{"id":"34872","subject":"[PATCH-v2] Allow git-filter-branch to process large repositories with lots of branches.","startedAt":"2013-09-07T21:03:09Z","lastAt":"2013-09-12T17:46:35Z","messageCount":7,"participants":["Lee Carver","Stefano Lattarini","Junio C Hamano"],"isPatch":true,"patchVersion":2,"patchTotal":null},"messages":[{"id":"227033","messageId":"CE50E29B.191B8%lee.carver@corp.service-now.com","threadId":"34872","inReplyTo":null,"subject":"[PATCH-v2] Allow git-filter-branch to process large repositories with lots of branches.","fromName":"Lee Carver","fromEmail":"lee.carver@servicenow.com","sentAt":"2013-09-07T21:03:09Z","receivedAt":"2013-09-07T21:03:09Z","isPatch":true,"sender":{"key":"lee.carver@servicenow.com","avatar":null},"body":"As noted in several forums, a recommended way to move trees between\nrepositories\nis to use git-filter-branch to revise the history for a single tree:\n\nhttp://gbayer.com/development/moving-files-from-one-git-repository-to-anoth\ner-preserving-history/\nhttp://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r\nepo-to-another-not-a-clone-preserving-history\n\nHowever, this can lead to argument list too long errors when the original\nrepository has many retained branches (>6k)\n\n/usr/local/git/libexec/git-core/git-filter-branch: line 270:\n/usr/local/git/libexec/git-core/git: Argument list too long\nCould not get the commits\n\nPiping the saved output from git rev-parse into git rev-list avoids this\nproblem, since the rev-parse output is not processed as a command line\nargument.\n---\n git-filter-branch.sh | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex ac2a005..60d239b 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -255,7 +255,7 @@ else\n \tremap_to_ancestor=t\n fi\n \n-rev_args=$(git rev-parse --revs-only \"$@\")\n+git rev-parse --revs-only \"$@\" > ../parse\n \n case \"$filter_subdir\" in\n \"\")\n@@ -267,8 +267,9 @@ case \"$filter_subdir\" in\n \t;;\n esac\n \n+cat ../parse | \\\n git rev-list --reverse --topo-order --default HEAD \\\n-\t--parents --simplify-merges $rev_args \"$@\" > ../revs ||\n+\t--parents --simplify-merges --stdin \"$@\" > ../revs ||\n \tdie \"Could not get the commits\"\n commits=$(wc -l <../revs | tr -d \" \")\n \n-- \n1.8.3.2\n"},{"id":"227034","messageId":"522BB170.8040507@gmail.com","threadId":"34872","inReplyTo":"CE50E29B.191B8%lee.carver@corp.service-now.com","subject":"Re: [PATCH-v2] Allow git-filter-branch to process large repositories with lots of branches.","fromName":"Stefano Lattarini","fromEmail":"stefano.lattarini@gmail.com","sentAt":"2013-09-07T23:06:24Z","receivedAt":"2013-09-07T23:06:24Z","isPatch":true,"sender":{"key":"stefano.lattarini@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1429199?v=4"},"body":"On 07/09/13 22:03, Lee Carver wrote:\n> As noted in several forums, a recommended way to move trees between\n> repositories\n> is to use git-filter-branch to revise the history for a single tree:\n>\n> http://gbayer.com/development/moving-files-from-one-git-repository-to-anoth\n> er-preserving-history/\n> http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r\n> epo-to-another-not-a-clone-preserving-history\n>\n> However, this can lead to argument list too long errors when the original\n> repository has many retained branches (>6k)\n>\n> /usr/local/git/libexec/git-core/git-filter-branch: line 270:\n> /usr/local/git/libexec/git-core/git: Argument list too long\n> Could not get the commits\n>\n> Piping the saved output from git rev-parse into git rev-list avoids this\n> problem, since the rev-parse output is not processed as a command line\n> argument.\n> ---\n>   git-filter-branch.sh | 5 +++--\n>   1 file changed, 3 insertions(+), 2 deletions(-)\n>\n> diff --git a/git-filter-branch.sh b/git-filter-branch.sh\n> index ac2a005..60d239b 100755\n> --- a/git-filter-branch.sh\n> +++ b/git-filter-branch.sh\n> @@ -255,7 +255,7 @@ else\n>   \tremap_to_ancestor=t\n>   fi\n>\n> -rev_args=$(git rev-parse --revs-only \"$@\")\n> +git rev-parse --revs-only \"$@\" > ../parse\n>\n>   case \"$filter_subdir\" in\n>   \"\")\n> @@ -267,8 +267,9 @@ case \"$filter_subdir\" in\n>   \t;;\n>   esac\n>\n> +cat ../parse | \\\n>   git rev-list --reverse --topo-order --default HEAD \\\n> -\t--parents --simplify-merges $rev_args \"$@\" > ../revs ||\n> +\t--parents --simplify-merges --stdin \"$@\" > ../revs ||\n >\nUseless use of cat IMO.  I'd suggest using a redirection instead:\n\n   git rev-list --reverse --topo-order --default HEAD \\\n-\t--parents --simplify-merges $rev_args \"$@\" > ../revs ||\n+\t--parents --simplify-merges --stdin \"$@\" > ../revs < ../parse ||\n\n>   \tdie \"Could not get the commits\"\n>   commits=$(wc -l <../revs | tr -d \" \")\n>\n>\n\nRegards,\n   Stefano\n"},{"id":"227387","messageId":"CE54F01A.19632%lee.carver@corp.service-now.com","threadId":"34872","inReplyTo":"522BB170.8040507@gmail.com","subject":"[PATCH-v3] Allow git-filter-branch to process large repositories with lots of branches.","fromName":"Lee Carver","fromEmail":"lee.carver@servicenow.com","sentAt":"2013-09-10T22:55:35Z","receivedAt":"2013-09-10T22:55:35Z","isPatch":true,"sender":{"key":"lee.carver@servicenow.com","avatar":null},"body":"As noted in several forums, a recommended way to move trees between\nrepositories\nis to use git-filter-branch to revise the history for a single tree:\n\nhttp://gbayer.com/development/moving-files-from-one-git-repository-to-anoth\ner-preserving-history/\nhttp://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r\nepo-to-another-not-a-clone-preserving-history\n\nHowever, this can lead to argument list too long errors when the original\nrepository has many retained branches (>6k)\n\n/usr/local/git/libexec/git-core/git-filter-branch: line 270:\n/usr/local/git/libexec/git-core/git: Argument list too long\nCould not get the commits\n\nPiping the saved output from git rev-parse into git rev-list avoids this\nproblem, since the rev-parse output is not processed as a command line\nargument.\n---\n git-filter-branch.sh | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex ac2a005..2091885 100755\n--- a/git-filter-branch.shgit\n+++ b/git-filter-branch.sh\n@@ -255,7 +255,7 @@ else\n \tremap_to_ancestor=t\n fi\n \n-rev_args=$(git rev-parse --revs-only \"$@\")\n+git rev-parse --revs-only \"$@\" > ../parse\n \n case \"$filter_subdir\" in\n \"\")\n@@ -268,7 +268,7 @@ case \"$filter_subdir\" in\n esac\n \n git rev-list --reverse --topo-order --default HEAD \\\n-\t--parents --simplify-merges $rev_args \"$@\" > ../revs ||\n+\t--parents --simplify-merges --stdin \"$@\" < ../parse > ../revs ||\n \tdie \"Could not get the commits\"\n commits=$(wc -l <../revs | tr -d \" \")\n \n-- \n1.8.3.2\n"},{"id":"227388","messageId":"xmqqli34uuvy.fsf@gitster.dls.corp.google.com","threadId":"34872","inReplyTo":"CE54F01A.19632%lee.carver@corp.service-now.com","subject":"Re: [PATCH-v3] Allow git-filter-branch to process large repositories with lots of branches.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-09-10T23:20:01Z","receivedAt":"2013-09-10T23:20:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lee Carver <Lee.Carver@servicenow.com> writes:\n\n> As noted in several forums, a recommended way to move trees between\n> repositories\n> is to use git-filter-branch to revise the history for a single tree:\n>\n> http://gbayer.com/development/moving-files-from-one-git-repository-to-anoth\n> er-preserving-history/\n> http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r\n> epo-to-another-not-a-clone-preserving-history\n>\n> However, this can lead to argument list too long errors when the original\n> repository has many retained branches (>6k)\n>\n> /usr/local/git/libexec/git-core/git-filter-branch: line 270:\n> /usr/local/git/libexec/git-core/git: Argument list too long\n> Could not get the commits\n>\n> Piping the saved output from git rev-parse into git rev-list avoids this\n> problem, since the rev-parse output is not processed as a command line\n> argument.\n> ---\n>  git-filter-branch.sh | 4 ++--\n>  1 file changed, 2 insertions(+), 2 deletions(-)\n>\n> diff --git a/git-filter-branch.sh b/git-filter-branch.sh\n> index ac2a005..2091885 100755\n> --- a/git-filter-branch.shgit\n> +++ b/git-filter-branch.sh\n> @@ -255,7 +255,7 @@ else\n>  \tremap_to_ancestor=t\n>  fi\n>  \n> -rev_args=$(git rev-parse --revs-only \"$@\")\n> +git rev-parse --revs-only \"$@\" > ../parse\n\nWhere is this \"rev-parse\" command running?  Is it always safe to\nclobber the file \"../parse\" like this?\n\n>  \n>  case \"$filter_subdir\" in\n>  \"\")\n> @@ -268,7 +268,7 @@ case \"$filter_subdir\" in\n>  esac\n>  \n>  git rev-list --reverse --topo-order --default HEAD \\\n> -\t--parents --simplify-merges $rev_args \"$@\" > ../revs ||\n> +\t--parents --simplify-merges --stdin \"$@\" < ../parse > ../revs ||\n>  \tdie \"Could not get the commits\"\n>  commits=$(wc -l <../revs | tr -d \" \")\n"},{"id":"227399","messageId":"CE550501.19691%lee.carver@corp.service-now.com","threadId":"34872","inReplyTo":"xmqqli34uuvy.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH-v3] Allow git-filter-branch to process large repositories with lots of branches.","fromName":"Lee Carver","fromEmail":"lee.carver@servicenow.com","sentAt":"2013-09-11T00:21:15Z","receivedAt":"2013-09-11T00:21:15Z","isPatch":true,"sender":{"key":"lee.carver@servicenow.com","avatar":null},"body":"\n\nOn 9/10/13 4:20 PM, \"Junio C Hamano\" <gitster@pobox.com> wrote:\n\n>Lee Carver <Lee.Carver@servicenow.com> writes:\n>\n>> As noted in several forums, a recommended way to move trees between\n>> repositories\n>> is to use git-filter-branch to revise the history for a single tree:\n>>\n>> \n>>http://gbayer.com/development/moving-files-from-one-git-repository-to-ano\n>>th\n>> er-preserving-history/\n>> \n>>http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git\n>>-r\n>> epo-to-another-not-a-clone-preserving-history\n>>\n>> However, this can lead to argument list too long errors when the\n>>original\n>> repository has many retained branches (>6k)\n>>\n>> /usr/local/git/libexec/git-core/git-filter-branch: line 270:\n>> /usr/local/git/libexec/git-core/git: Argument list too long\n>> Could not get the commits\n>>\n>> Piping the saved output from git rev-parse into git rev-list avoids this\n>> problem, since the rev-parse output is not processed as a command line\n>> argument.\n>> ---\n>>  git-filter-branch.sh | 4 ++--\n>>  1 file changed, 2 insertions(+), 2 deletions(-)\n>>\n>> diff --git a/git-filter-branch.sh b/git-filter-branch.sh\n>> index ac2a005..2091885 100755\n>> --- a/git-filter-branch.shgit\n>> +++ b/git-filter-branch.sh\n>> @@ -255,7 +255,7 @@ else\n>>  \tremap_to_ancestor=t\n>>  fi\n>>  \n>> -rev_args=$(git rev-parse --revs-only \"$@\")\n>> +git rev-parse --revs-only \"$@\" > ../parse\n>\n>Where is this \"rev-parse\" command running?  Is it always safe to\n>clobber the file \"../parse\" like this?\n\nIt is using the same ${tempdir} working directory that git rev-list uses\nbelow for the ../revs file\n\nIt's normally .git-rewrite/t, following the normal working directory setup\nnear line 205.\n\n>\n>>  \n>>  case \"$filter_subdir\" in\n>>  \"\")\n>> @@ -268,7 +268,7 @@ case \"$filter_subdir\" in\n>>  esac\n>>  \n>>  git rev-list --reverse --topo-order --default HEAD \\\n>> -\t--parents --simplify-merges $rev_args \"$@\" > ../revs ||\n>> +\t--parents --simplify-merges --stdin \"$@\" < ../parse > ../revs ||\n>>  \tdie \"Could not get the commits\"\n>>  commits=$(wc -l <../revs | tr -d \" \")\n"},{"id":"227455","messageId":"xmqqhadruyu4.fsf@gitster.dls.corp.google.com","threadId":"34872","inReplyTo":"CE550501.19691%lee.carver@corp.service-now.com","subject":"Re: [PATCH-v3] Allow git-filter-branch to process large repositories with lots of branches.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-09-11T16:06:59Z","receivedAt":"2013-09-11T16:06:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lee Carver <Lee.Carver@servicenow.com> writes:\n\n> It is using the same ${tempdir} working directory that git rev-list uses\n> below for the ../revs file\n\nAh, I missed that; then that should be safe.  The patch looks sane.\n\nCan we have your sign-off, too, please?\n\n>\n> It's normally .git-rewrite/t, following the normal working directory setup\n> near line 205.\n>\n>>\n>>>  \n>>>  case \"$filter_subdir\" in\n>>>  \"\")\n>>> @@ -268,7 +268,7 @@ case \"$filter_subdir\" in\n>>>  esac\n>>>  \n>>>  git rev-list --reverse --topo-order --default HEAD \\\n>>> -\t--parents --simplify-merges $rev_args \"$@\" > ../revs ||\n>>> +\t--parents --simplify-merges --stdin \"$@\" < ../parse > ../revs ||\n>>>  \tdie \"Could not get the commits\"\n>>>  commits=$(wc -l <../revs | tr -d \" \")\n"},{"id":"227544","messageId":"CE574B31.199C6%lee.carver@corp.service-now.com","threadId":"34872","inReplyTo":"xmqqhadruyu4.fsf@gitster.dls.corp.google.com","subject":"[PATCH-v4] Allow git-filter-branch to process large repositories with lots of branches.","fromName":"Lee Carver","fromEmail":"lee.carver@servicenow.com","sentAt":"2013-09-12T17:46:35Z","receivedAt":"2013-09-12T17:46:35Z","isPatch":true,"sender":{"key":"lee.carver@servicenow.com","avatar":null},"body":"As noted in several forums, a recommended way to move trees between\nrepositories\nis to use git-filter-branch to revise the history for a single tree:\n\nhttp://gbayer.com/development/moving-files-from-one-git-repository-to-anoth\ner-preserving-history/\nhttp://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r\nepo-to-another-not-a-clone-preserving-history\n\nHowever, this can lead to argument list too long errors when the original\nrepository has many retained branches (>6k)\n\n/usr/local/git/libexec/git-core/git-filter-branch: line 270:\n/usr/local/git/libexec/git-core/git: Argument list too long\nCould not get the commits\n\nPiping the saved output from git rev-parse into git rev-list avoids this\nproblem, since the rev-parse output is not processed as a command line\nargument.\n\nSigned-off-by: Lee Carver <lee.carver@servicenow.com>\n---\n git-filter-branch.sh | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex ac2a005..2091885 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -255,7 +255,7 @@ else\n \tremap_to_ancestor=t\n fi\n \n-rev_args=$(git rev-parse --revs-only \"$@\")\n+git rev-parse --revs-only \"$@\" > ../parse\n \n case \"$filter_subdir\" in\n \"\")\n@@ -268,7 +268,7 @@ case \"$filter_subdir\" in\n esac\n \n git rev-list --reverse --topo-order --default HEAD \\\n-\t--parents --simplify-merges $rev_args \"$@\" > ../revs ||\n+\t--parents --simplify-merges --stdin \"$@\" < ../parse > ../revs ||\n \tdie \"Could not get the commits\"\n commits=$(wc -l <../revs | tr -d \" \")\n \n-- \n1.8.3.2\n"}]}