# [PATCH-v2] Allow git-filter-branch to process large repositories with lots of branches. 7 messages from 2013-09-07 to 2013-09-12. Participants: Lee Carver, Stefano Lattarini, Junio C Hamano. Thread: https://gitlist.dev/t/34872 ## Lee Carver, 2013-09-07 21:03 Subject: [PATCH-v2] Allow git-filter-branch to process large repositories with lots of branches. Message-ID: URL: https://gitlist.dev/e/CE50E29B.191B8%25lee.carver%40corp.service-now.com ``` As noted in several forums, a recommended way to move trees between repositories is to use git-filter-branch to revise the history for a single tree: http://gbayer.com/development/moving-files-from-one-git-repository-to-anoth er-preserving-history/ http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r epo-to-another-not-a-clone-preserving-history However, this can lead to argument list too long errors when the original repository has many retained branches (>6k) /usr/local/git/libexec/git-core/git-filter-branch: line 270: /usr/local/git/libexec/git-core/git: Argument list too long Could not get the commits Piping the saved output from git rev-parse into git rev-list avoids this problem, since the rev-parse output is not processed as a command line argument. --- git-filter-branch.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/git-filter-branch.sh b/git-filter-branch.sh index ac2a005..60d239b 100755 --- a/git-filter-branch.sh +++ b/git-filter-branch.sh @@ -255,7 +255,7 @@ else remap_to_ancestor=t fi -rev_args=$(git rev-parse --revs-only "$@") +git rev-parse --revs-only "$@" > ../parse case "$filter_subdir" in "") @@ -267,8 +267,9 @@ case "$filter_subdir" in ;; esac +cat ../parse | \ git rev-list --reverse --topo-order --default HEAD \ - --parents --simplify-merges $rev_args "$@" > ../revs || + --parents --simplify-merges --stdin "$@" > ../revs || die "Could not get the commits" commits=$(wc -l <../revs | tr -d " ") -- 1.8.3.2 ``` ## Stefano Lattarini, 2013-09-07 23:06 Subject: Re: [PATCH-v2] Allow git-filter-branch to process large repositories with lots of branches. Message-ID: <522BB170.8040507@gmail.com> URL: https://gitlist.dev/e/522BB170.8040507%40gmail.com In-Reply-To: ``` On 07/09/13 22:03, Lee Carver wrote: > As noted in several forums, a recommended way to move trees between > repositories > is to use git-filter-branch to revise the history for a single tree: > > http://gbayer.com/development/moving-files-from-one-git-repository-to-anoth > er-preserving-history/ > http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r > epo-to-another-not-a-clone-preserving-history > > However, this can lead to argument list too long errors when the original > repository has many retained branches (>6k) > > /usr/local/git/libexec/git-core/git-filter-branch: line 270: > /usr/local/git/libexec/git-core/git: Argument list too long > Could not get the commits > > Piping the saved output from git rev-parse into git rev-list avoids this > problem, since the rev-parse output is not processed as a command line > argument. > --- > git-filter-branch.sh | 5 +++-- > 1 file changed, 3 insertions(+), 2 deletions(-) > > diff --git a/git-filter-branch.sh b/git-filter-branch.sh > index ac2a005..60d239b 100755 > --- a/git-filter-branch.sh > +++ b/git-filter-branch.sh > @@ -255,7 +255,7 @@ else > remap_to_ancestor=t > fi > > -rev_args=$(git rev-parse --revs-only "$@") > +git rev-parse --revs-only "$@" > ../parse > > case "$filter_subdir" in > "") > @@ -267,8 +267,9 @@ case "$filter_subdir" in > ;; > esac > > +cat ../parse | \ > git rev-list --reverse --topo-order --default HEAD \ > - --parents --simplify-merges $rev_args "$@" > ../revs || > + --parents --simplify-merges --stdin "$@" > ../revs || > Useless use of cat IMO. I'd suggest using a redirection instead: git rev-list --reverse --topo-order --default HEAD \ - --parents --simplify-merges $rev_args "$@" > ../revs || + --parents --simplify-merges --stdin "$@" > ../revs < ../parse || > die "Could not get the commits" > commits=$(wc -l <../revs | tr -d " ") > > Regards, Stefano ``` ## Lee Carver, 2013-09-10 22:55 Subject: [PATCH-v3] Allow git-filter-branch to process large repositories with lots of branches. Message-ID: URL: https://gitlist.dev/e/CE54F01A.19632%25lee.carver%40corp.service-now.com In-Reply-To: <522BB170.8040507@gmail.com> ``` As noted in several forums, a recommended way to move trees between repositories is to use git-filter-branch to revise the history for a single tree: http://gbayer.com/development/moving-files-from-one-git-repository-to-anoth er-preserving-history/ http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r epo-to-another-not-a-clone-preserving-history However, this can lead to argument list too long errors when the original repository has many retained branches (>6k) /usr/local/git/libexec/git-core/git-filter-branch: line 270: /usr/local/git/libexec/git-core/git: Argument list too long Could not get the commits Piping the saved output from git rev-parse into git rev-list avoids this problem, since the rev-parse output is not processed as a command line argument. --- git-filter-branch.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/git-filter-branch.sh b/git-filter-branch.sh index ac2a005..2091885 100755 --- a/git-filter-branch.shgit +++ b/git-filter-branch.sh @@ -255,7 +255,7 @@ else remap_to_ancestor=t fi -rev_args=$(git rev-parse --revs-only "$@") +git rev-parse --revs-only "$@" > ../parse case "$filter_subdir" in "") @@ -268,7 +268,7 @@ case "$filter_subdir" in esac git rev-list --reverse --topo-order --default HEAD \ - --parents --simplify-merges $rev_args "$@" > ../revs || + --parents --simplify-merges --stdin "$@" < ../parse > ../revs || die "Could not get the commits" commits=$(wc -l <../revs | tr -d " ") -- 1.8.3.2 ``` ## Junio C Hamano, 2013-09-10 23:20 Subject: Re: [PATCH-v3] Allow git-filter-branch to process large repositories with lots of branches. Message-ID: URL: https://gitlist.dev/e/xmqqli34uuvy.fsf%40gitster.dls.corp.google.com In-Reply-To: ``` Lee Carver writes: > As noted in several forums, a recommended way to move trees between > repositories > is to use git-filter-branch to revise the history for a single tree: > > http://gbayer.com/development/moving-files-from-one-git-repository-to-anoth > er-preserving-history/ > http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r > epo-to-another-not-a-clone-preserving-history > > However, this can lead to argument list too long errors when the original > repository has many retained branches (>6k) > > /usr/local/git/libexec/git-core/git-filter-branch: line 270: > /usr/local/git/libexec/git-core/git: Argument list too long > Could not get the commits > > Piping the saved output from git rev-parse into git rev-list avoids this > problem, since the rev-parse output is not processed as a command line > argument. > --- > git-filter-branch.sh | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/git-filter-branch.sh b/git-filter-branch.sh > index ac2a005..2091885 100755 > --- a/git-filter-branch.shgit > +++ b/git-filter-branch.sh > @@ -255,7 +255,7 @@ else > remap_to_ancestor=t > fi > > -rev_args=$(git rev-parse --revs-only "$@") > +git rev-parse --revs-only "$@" > ../parse Where is this "rev-parse" command running? Is it always safe to clobber the file "../parse" like this? > > case "$filter_subdir" in > "") > @@ -268,7 +268,7 @@ case "$filter_subdir" in > esac > > git rev-list --reverse --topo-order --default HEAD \ > - --parents --simplify-merges $rev_args "$@" > ../revs || > + --parents --simplify-merges --stdin "$@" < ../parse > ../revs || > die "Could not get the commits" > commits=$(wc -l <../revs | tr -d " ") ``` ## Lee Carver, 2013-09-11 00:21 Subject: Re: [PATCH-v3] Allow git-filter-branch to process large repositories with lots of branches. Message-ID: URL: https://gitlist.dev/e/CE550501.19691%25lee.carver%40corp.service-now.com In-Reply-To: ``` On 9/10/13 4:20 PM, "Junio C Hamano" wrote: >Lee Carver writes: > >> As noted in several forums, a recommended way to move trees between >> repositories >> is to use git-filter-branch to revise the history for a single tree: >> >> >>http://gbayer.com/development/moving-files-from-one-git-repository-to-ano >>th >> er-preserving-history/ >> >>http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git >>-r >> epo-to-another-not-a-clone-preserving-history >> >> However, this can lead to argument list too long errors when the >>original >> repository has many retained branches (>6k) >> >> /usr/local/git/libexec/git-core/git-filter-branch: line 270: >> /usr/local/git/libexec/git-core/git: Argument list too long >> Could not get the commits >> >> Piping the saved output from git rev-parse into git rev-list avoids this >> problem, since the rev-parse output is not processed as a command line >> argument. >> --- >> git-filter-branch.sh | 4 ++-- >> 1 file changed, 2 insertions(+), 2 deletions(-) >> >> diff --git a/git-filter-branch.sh b/git-filter-branch.sh >> index ac2a005..2091885 100755 >> --- a/git-filter-branch.shgit >> +++ b/git-filter-branch.sh >> @@ -255,7 +255,7 @@ else >> remap_to_ancestor=t >> fi >> >> -rev_args=$(git rev-parse --revs-only "$@") >> +git rev-parse --revs-only "$@" > ../parse > >Where is this "rev-parse" command running? Is it always safe to >clobber the file "../parse" like this? It is using the same ${tempdir} working directory that git rev-list uses below for the ../revs file It's normally .git-rewrite/t, following the normal working directory setup near line 205. > >> >> case "$filter_subdir" in >> "") >> @@ -268,7 +268,7 @@ case "$filter_subdir" in >> esac >> >> git rev-list --reverse --topo-order --default HEAD \ >> - --parents --simplify-merges $rev_args "$@" > ../revs || >> + --parents --simplify-merges --stdin "$@" < ../parse > ../revs || >> die "Could not get the commits" >> commits=$(wc -l <../revs | tr -d " ") ``` ## Junio C Hamano, 2013-09-11 16:06 Subject: Re: [PATCH-v3] Allow git-filter-branch to process large repositories with lots of branches. Message-ID: URL: https://gitlist.dev/e/xmqqhadruyu4.fsf%40gitster.dls.corp.google.com In-Reply-To: ``` Lee Carver writes: > It is using the same ${tempdir} working directory that git rev-list uses > below for the ../revs file Ah, I missed that; then that should be safe. The patch looks sane. Can we have your sign-off, too, please? > > It's normally .git-rewrite/t, following the normal working directory setup > near line 205. > >> >>> >>> case "$filter_subdir" in >>> "") >>> @@ -268,7 +268,7 @@ case "$filter_subdir" in >>> esac >>> >>> git rev-list --reverse --topo-order --default HEAD \ >>> - --parents --simplify-merges $rev_args "$@" > ../revs || >>> + --parents --simplify-merges --stdin "$@" < ../parse > ../revs || >>> die "Could not get the commits" >>> commits=$(wc -l <../revs | tr -d " ") ``` ## Lee Carver, 2013-09-12 17:46 Subject: [PATCH-v4] Allow git-filter-branch to process large repositories with lots of branches. Message-ID: URL: https://gitlist.dev/e/CE574B31.199C6%25lee.carver%40corp.service-now.com In-Reply-To: ``` As noted in several forums, a recommended way to move trees between repositories is to use git-filter-branch to revise the history for a single tree: http://gbayer.com/development/moving-files-from-one-git-repository-to-anoth er-preserving-history/ http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r epo-to-another-not-a-clone-preserving-history However, this can lead to argument list too long errors when the original repository has many retained branches (>6k) /usr/local/git/libexec/git-core/git-filter-branch: line 270: /usr/local/git/libexec/git-core/git: Argument list too long Could not get the commits Piping the saved output from git rev-parse into git rev-list avoids this problem, since the rev-parse output is not processed as a command line argument. Signed-off-by: Lee Carver --- git-filter-branch.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/git-filter-branch.sh b/git-filter-branch.sh index ac2a005..2091885 100755 --- a/git-filter-branch.sh +++ b/git-filter-branch.sh @@ -255,7 +255,7 @@ else remap_to_ancestor=t fi -rev_args=$(git rev-parse --revs-only "$@") +git rev-parse --revs-only "$@" > ../parse case "$filter_subdir" in "") @@ -268,7 +268,7 @@ case "$filter_subdir" in esac git rev-list --reverse --topo-order --default HEAD \ - --parents --simplify-merges $rev_args "$@" > ../revs || + --parents --simplify-merges --stdin "$@" < ../parse > ../revs || die "Could not get the commits" commits=$(wc -l <../revs | tr -d " ") -- 1.8.3.2 ```