threads / patch / 34872

v2Allow git-filter-branch to process large repositories with lots of branches.

Subject: [PATCH-v2] Allow git-filter-branch to process large repositories with lots of branches.

## tl;dr

7 messages between Sep 7, 2013 and Sep 12, 2013. Diffs are folded; open one to read it.

replies: 6people: 3as markdown or json

Lee Carver· Sep 7, 2013, 21:03 UTC · lore

As noted in several forums, a recommended way to move trees between repositories is to use git-filter-branch to revise the history for a single tree:

http://gbayer.com/development/moving-files-from-one-git-repository-to-anoth er-preserving-history/ http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r epo-to-another-not-a-clone-preserving-history

However, this can lead to argument list too long errors when the original repository has many retained branches (>6k)

/usr/local/git/libexec/git-core/git-filter-branch: line 270: /usr/local/git/libexec/git-core/git: Argument list too long Could not get the commits

Piping the saved output from git rev-parse into git rev-list avoids this
problem, since the rev-parse output is not processed as a command line
argument.
---
 git-filter-branch.sh | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

Show changes to git-filter-branch.sh +3 −2

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index ac2a005..60d239b 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -255,7 +255,7 @@ else
 	remap_to_ancestor=t
 fi
 
-rev_args=$(git rev-parse --revs-only "$@")
+git rev-parse --revs-only "$@" > ../parse
 
 case "$filter_subdir" in
 "")
@@ -267,8 +267,9 @@ case "$filter_subdir" in
 	;;
 esac
 
+cat ../parse | \
 git rev-list --reverse --topo-order --default HEAD \
-	--parents --simplify-merges $rev_args "$@" > ../revs ||
+	--parents --simplify-merges --stdin "$@" > ../revs ||
 	die "Could not get the commits"
 commits=$(wc -l <../revs | tr -d " ")

-- 
1.8.3.2

Stefano Lattarini· Sep 7, 2013, 23:06 UTC · re: Lee Carver · lore

Re: [PATCH-v2] Allow git-filter-branch to process large repositories with lots of branches.

On 07/09/13 22:03, Lee Carver wrote:

Show 44 quoted lines

> As noted in several forums, a recommended way to move trees between
> repositories
> is to use git-filter-branch to revise the history for a single tree:
>
> http://gbayer.com/development/moving-files-from-one-git-repository-to-anoth
> er-preserving-history/
> http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r
> epo-to-another-not-a-clone-preserving-history
>
> However, this can lead to argument list too long errors when the original
> repository has many retained branches (>6k)
>
> /usr/local/git/libexec/git-core/git-filter-branch: line 270:
> /usr/local/git/libexec/git-core/git: Argument list too long
> Could not get the commits
>
> Piping the saved output from git rev-parse into git rev-list avoids this
> problem, since the rev-parse output is not processed as a command line
> argument.
> ---
>   git-filter-branch.sh | 5 +++--
>   1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/git-filter-branch.sh b/git-filter-branch.sh
> index ac2a005..60d239b 100755
> --- a/git-filter-branch.sh
> +++ b/git-filter-branch.sh
> @@ -255,7 +255,7 @@ else
>   	remap_to_ancestor=t
>   fi
>
> -rev_args=$(git rev-parse --revs-only "$@")
> +git rev-parse --revs-only "$@" > ../parse
>
>   case "$filter_subdir" in
>   "")
> @@ -267,8 +267,9 @@ case "$filter_subdir" in
>   	;;
>   esac
>
> +cat ../parse | \
>   git rev-list --reverse --topo-order --default HEAD \
> -	--parents --simplify-merges $rev_args "$@" > ../revs ||
> +	--parents --simplify-merges --stdin "$@" > ../revs ||

 >
Useless use of cat IMO.  I'd suggest using a redirection instead:

   git rev-list --reverse --topo-order --default HEAD \
-	--parents --simplify-merges $rev_args "$@" > ../revs ||
+	--parents --simplify-merges --stdin "$@" > ../revs < ../parse ||

>   	die "Could not get the commits"
>   commits=$(wc -l <../revs | tr -d " ")
>
>

Regards,
   Stefano

Lee Carver· Sep 10, 2013, 22:55 UTC · re: Stefano Lattarini · lore

[PATCH-v3] Allow git-filter-branch to process large repositories with lots of branches.

As noted in several forums, a recommended way to move trees between repositories is to use git-filter-branch to revise the history for a single tree:

However, this can lead to argument list too long errors when the original repository has many retained branches (>6k)

/usr/local/git/libexec/git-core/git-filter-branch: line 270: /usr/local/git/libexec/git-core/git: Argument list too long Could not get the commits

Piping the saved output from git rev-parse into git rev-list avoids this
problem, since the rev-parse output is not processed as a command line
argument.
---
 git-filter-branch.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Show changes to git-filter-branch.sh +2 −2

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index ac2a005..2091885 100755
--- a/git-filter-branch.shgit
+++ b/git-filter-branch.sh
@@ -255,7 +255,7 @@ else
 	remap_to_ancestor=t
 fi
 
-rev_args=$(git rev-parse --revs-only "$@")
+git rev-parse --revs-only "$@" > ../parse
 
 case "$filter_subdir" in
 "")
@@ -268,7 +268,7 @@ case "$filter_subdir" in
 esac
 
 git rev-list --reverse --topo-order --default HEAD \
-	--parents --simplify-merges $rev_args "$@" > ../revs ||
+	--parents --simplify-merges --stdin "$@" < ../parse > ../revs ||
 	die "Could not get the commits"
 commits=$(wc -l <../revs | tr -d " ")

-- 
1.8.3.2

Junio C Hamano· Sep 10, 2013, 23:20 UTC · re: Lee Carver · lore

Re: [PATCH-v3] Allow git-filter-branch to process large repositories with lots of branches.

Lee Carver <Lee.Carver@servicenow.com> writes:

Show 33 quoted lines

> As noted in several forums, a recommended way to move trees between
> repositories
> is to use git-filter-branch to revise the history for a single tree:
>
> http://gbayer.com/development/moving-files-from-one-git-repository-to-anoth
> er-preserving-history/
> http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r
> epo-to-another-not-a-clone-preserving-history
>
> However, this can lead to argument list too long errors when the original
> repository has many retained branches (>6k)
>
> /usr/local/git/libexec/git-core/git-filter-branch: line 270:
> /usr/local/git/libexec/git-core/git: Argument list too long
> Could not get the commits
>
> Piping the saved output from git rev-parse into git rev-list avoids this
> problem, since the rev-parse output is not processed as a command line
> argument.
> ---
>  git-filter-branch.sh | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/git-filter-branch.sh b/git-filter-branch.sh
> index ac2a005..2091885 100755
> --- a/git-filter-branch.shgit
> +++ b/git-filter-branch.sh
> @@ -255,7 +255,7 @@ else
>  	remap_to_ancestor=t
>  fi
>  
> -rev_args=$(git rev-parse --revs-only "$@")
> +git rev-parse --revs-only "$@" > ../parse

Where is this "rev-parse" command running? Is it always safe to clobber the file "../parse" like this?

Show 11 quoted lines

>  
>  case "$filter_subdir" in
>  "")
> @@ -268,7 +268,7 @@ case "$filter_subdir" in
>  esac
>  
>  git rev-list --reverse --topo-order --default HEAD \
> -	--parents --simplify-merges $rev_args "$@" > ../revs ||
> +	--parents --simplify-merges --stdin "$@" < ../parse > ../revs ||
>  	die "Could not get the commits"
>  commits=$(wc -l <../revs | tr -d " ")

Lee Carver· Sep 11, 2013, 00:21 UTC · re: Junio C Hamano · lore

Re: [PATCH-v3] Allow git-filter-branch to process large repositories with lots of branches.

On 9/10/13 4:20 PM, "Junio C Hamano" <gitster@pobox.com> wrote:

Show 43 quoted lines

>Lee Carver <Lee.Carver@servicenow.com> writes:
>
>> As noted in several forums, a recommended way to move trees between
>> repositories
>> is to use git-filter-branch to revise the history for a single tree:
>>
>> 
>>http://gbayer.com/development/moving-files-from-one-git-repository-to-ano
>>th
>> er-preserving-history/
>> 
>>http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git
>>-r
>> epo-to-another-not-a-clone-preserving-history
>>
>> However, this can lead to argument list too long errors when the
>>original
>> repository has many retained branches (>6k)
>>
>> /usr/local/git/libexec/git-core/git-filter-branch: line 270:
>> /usr/local/git/libexec/git-core/git: Argument list too long
>> Could not get the commits
>>
>> Piping the saved output from git rev-parse into git rev-list avoids this
>> problem, since the rev-parse output is not processed as a command line
>> argument.
>> ---
>>  git-filter-branch.sh | 4 ++--
>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/git-filter-branch.sh b/git-filter-branch.sh
>> index ac2a005..2091885 100755
>> --- a/git-filter-branch.shgit
>> +++ b/git-filter-branch.sh
>> @@ -255,7 +255,7 @@ else
>>  	remap_to_ancestor=t
>>  fi
>>  
>> -rev_args=$(git rev-parse --revs-only "$@")
>> +git rev-parse --revs-only "$@" > ../parse
>
>Where is this "rev-parse" command running?  Is it always safe to
>clobber the file "../parse" like this?

It is using the same ${tempdir} working directory that git rev-list uses below for the ../revs file

It's normally .git-rewrite/t, following the normal working directory setup near line 205.

Show 12 quoted lines

>
>>  
>>  case "$filter_subdir" in
>>  "")
>> @@ -268,7 +268,7 @@ case "$filter_subdir" in
>>  esac
>>  
>>  git rev-list --reverse --topo-order --default HEAD \
>> -	--parents --simplify-merges $rev_args "$@" > ../revs ||
>> +	--parents --simplify-merges --stdin "$@" < ../parse > ../revs ||
>>  	die "Could not get the commits"
>>  commits=$(wc -l <../revs | tr -d " ")

Junio C Hamano· Sep 11, 2013, 16:06 UTC · re: Lee Carver · lore

Re: [PATCH-v3] Allow git-filter-branch to process large repositories with lots of branches.

Lee Carver <Lee.Carver@servicenow.com> writes:

> It is using the same ${tempdir} working directory that git rev-list uses
> below for the ../revs file

Ah, I missed that; then that should be safe.  The patch looks sane.

Can we have your sign-off, too, please?

Show 16 quoted lines

>
> It's normally .git-rewrite/t, following the normal working directory setup
> near line 205.
>
>>
>>>  
>>>  case "$filter_subdir" in
>>>  "")
>>> @@ -268,7 +268,7 @@ case "$filter_subdir" in
>>>  esac
>>>  
>>>  git rev-list --reverse --topo-order --default HEAD \
>>> -	--parents --simplify-merges $rev_args "$@" > ../revs ||
>>> +	--parents --simplify-merges --stdin "$@" < ../parse > ../revs ||
>>>  	die "Could not get the commits"
>>>  commits=$(wc -l <../revs | tr -d " ")

Lee Carver· Sep 12, 2013, 17:46 UTC · re: Junio C Hamano · lore

[PATCH-v4] Allow git-filter-branch to process large repositories with lots of branches.

As noted in several forums, a recommended way to move trees between repositories is to use git-filter-branch to revise the history for a single tree:

However, this can lead to argument list too long errors when the original repository has many retained branches (>6k)

/usr/local/git/libexec/git-core/git-filter-branch: line 270: /usr/local/git/libexec/git-core/git: Argument list too long Could not get the commits

Piping the saved output from git rev-parse into git rev-list avoids this problem, since the rev-parse output is not processed as a command line argument.

Signed-off-by: Lee Carver <lee.carver@servicenow.com>
---
 git-filter-branch.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Show changes to git-filter-branch.sh +2 −2

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index ac2a005..2091885 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -255,7 +255,7 @@ else
 	remap_to_ancestor=t
 fi
 
-rev_args=$(git rev-parse --revs-only "$@")
+git rev-parse --revs-only "$@" > ../parse
 
 case "$filter_subdir" in
 "")
@@ -268,7 +268,7 @@ case "$filter_subdir" in
 esac
 
 git rev-list --reverse --topo-order --default HEAD \
-	--parents --simplify-merges $rev_args "$@" > ../revs ||
+	--parents --simplify-merges --stdin "$@" < ../parse > ../revs ||
 	die "Could not get the commits"
 commits=$(wc -l <../revs | tr -d " ")

-- 
1.8.3.2