threads / patch / 25624

patch, 2 partssubmodule: preserve all arguments exactly when recursing

Subject: [PATCH 1/2] submodule: preserve all arguments exactly when recursing

## tl;dr

17 messages between Nov 3, 2010 and Nov 5, 2010. Diffs are folded; open one to read it.

replies: 16people: 3as markdown or json

Kevin Ballard· Nov 3, 2010, 04:34 UTC · lore

When performing a recursive status or update, any argments with whitespace would be split along that whitespace when passed to the recursive invocation of the update or status command.

This is caused by the special handling that sh provides to the $@ variable. Status and update stored "$@" into a separate variable, and passed that variable to the recursive invocation. Unfortunately, the special handling afforded to $@ isn't given to this new variable, and word-breaking occurs along whitespace boundaries.

We can work around this by taking advantage of an easy technique to quote any arbitrary string in the shell. Because single-quoted strings don't support backslash-escapes, any arbitrary string can be quoted by wrapping it in single-quotes and replacing any single-quotes inside the string with the sequence '\''.

This commit introduces a new shell function quote_words that uses the quote trick to produce a string containing the quoted version of all its arguments. This string can then be used with `set -` to restore the original value of $@. This shell function is used in cmd_status and in cmd_update to store the original value of $@, which is then restored before the recursive invocation takes place.

Signed-off-by: Kevin Ballard <kevin@sb.org>
---
I tried to write tests for this, but there are only two ways to get args
with spaces to be accepted and passed to the recursive invocation.
The first is via the --reference flag, but I don't think it really makes
sense to use that flag in connection with --recursive and was not comfortable
using it in a test. The second is as a pathname after the flags, but it
also doesn't make sense to pass these to recursive invocations, and in fact
the subsequent commit fixes it so the pathnames are not passed to recursive
invocations.
That said, despite the lack of tests I still believe this is a worthwhile
change, and it will certainly future-proof the command in case new flags
are added. It's also a reasonable model for how to handle this problem
in other shell commands.
 git-submodule.sh |   43 +++++++++++++++++++++++++++++++++++++++----
 1 files changed, 39 insertions(+), 4 deletions(-)
Show changes to git-submodule.sh +39 −4
diff --git a/git-submodule.sh b/git-submodule.sh
index 9ebbab7..ec7a5e4 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -64,6 +64,39 @@ module_list()
 }
 
 #
+# Emit a quoted version of the all argument suitable for passing to `eval`
+# $@ = words to quote
+#
+# This is intended to be used like the following:
+#   orig_args="$(quote_words "$@")"
+#   # do some work that includes calling shift
+#   eval "set - $orig_args"
+#   # now $@ has been restored, suitable for passing to another command
+#
+# Note that you cannot simply save off $@ into another variable because
+# the shell gives $@ and $* special handling in parameter expansion
+#
+quote_words ()
+{
+	while test $# -ne 0; do
+		# this can be done using sed like so:
+		#   printf "'%s'" "$(printf "%s" "$1" | sed -e "s/'/'\\\\''/g")"
+		# but in an attempt to avoid spawning a process for every argument, we'll
+		# just use the prefix/suffix pattern matching stuff
+		local word= suffix="$1" prefix=
+		while test -n "$suffix"
+		do
+			prefix="${suffix%%\'*}"
+			test "$prefix" != "$suffix" || break
+			suffix="${suffix#*\'}"
+			word="$word$prefix'\\''"
+		done
+		printf "'%s' " "$word$suffix"
+		shift
+	done
+}
+
+#
 # Map submodule path to submodule name
 #
 # $1 = path
@@ -374,7 +407,7 @@ cmd_init()
 cmd_update()
 {
 	# parse $args after "submodule ... update".
-	orig_args="$@"
+	orig_args="$(quote_words "$@")"
 	while test $# -ne 0
 	do
 		case "$1" in
@@ -500,7 +533,8 @@ cmd_update()
 
 		if test -n "$recursive"
 		then
-			(clear_local_git_env; cd "$path" && cmd_update $orig_args) ||
+			eval "set - $orig_args"
+			(clear_local_git_env; cd "$path" && cmd_update "$@") ||
 			die "Failed to recurse into submodule path '$path'"
 		fi
 	done
@@ -733,7 +767,7 @@ cmd_summary() {
 cmd_status()
 {
 	# parse $args after "submodule ... status".
-	orig_args="$@"
+	orig_args="$(quote_words "$@")"
 	while test $# -ne 0
 	do
 		case "$1" in
@@ -788,9 +822,10 @@ cmd_status()
 		then
 			(
 				prefix="$displaypath/"
+				eval "set - $orig_args"
 				clear_local_git_env
 				cd "$path" &&
-				cmd_status $orig_args
+				cmd_status "$@"
 			) ||
 			die "Failed to recurse into submodule path '$path'"
 		fi
-- 
1.7.3.2.200.ga1bd
Kevin Ballard· Nov 3, 2010, 04:34 UTC · re: Kevin Ballard · lore

[PATCH 2/2] submodule: only preserve flags across recursive status/update invocations

Recursive invocations of submodule update/status preserve all arguments in the recursive invocation. This tends to cause undesired behavior when those arguments include the name of a submodule, as it tries to recurse into a nested submodule with the same name rather than recursing into all child submodules of the named submodule.

This commit changes the argument preservation to only preserve flags. When specifying a submodule name on the command-line, all child submodules of that named submodule will be recursed into correctly, and it will not attempt to recurse into a phantom nested submodule with the same name as its parent.

Signed-off-by: Kevin Ballard <kevin@sb.org>
---
This commit was inspired by my coworker, who tried to run `git submodule
update --recursive molecules` (where molecules is the name of a submodule)
and received the rather unexpected error
  error: pathspec 'molecules' did not match any file(s) known to git.
  Did you forget to 'git add'?
 git-submodule.sh             |   21 +++++++++------------
 t/t7407-submodule-foreach.sh |   19 +++++++++++++++++++
 2 files changed, 28 insertions(+), 12 deletions(-)
Show changes to 2 files +28 −12

git-submodule.sh, t/t7407-submodule-foreach.sh

diff --git a/git-submodule.sh b/git-submodule.sh
index ec7a5e4..52c693c 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -407,41 +407,35 @@ cmd_init()
 cmd_update()
 {
 	# parse $args after "submodule ... update".
-	orig_args="$(quote_words "$@")"
+	orig_flags=
 	while test $# -ne 0
 	do
 		case "$1" in
 		-q|--quiet)
-			shift
 			GIT_QUIET=1
 			;;
 		-i|--init)
 			init=1
-			shift
 			;;
 		-N|--no-fetch)
-			shift
 			nofetch=1
 			;;
 		-r|--rebase)
-			shift
 			update="rebase"
 			;;
 		--reference)
 			case "$2" in '') usage ;; esac
 			reference="--reference=$2"
-			shift 2
+			orig_flags="$orig_flags $(quote_words "$1")"
+			shift
 			;;
 		--reference=*)
 			reference="$1"
-			shift
 			;;
 		-m|--merge)
-			shift
 			update="merge"
 			;;
 		--recursive)
-			shift
 			recursive=1
 			;;
 		--)
@@ -455,6 +449,8 @@ cmd_update()
 			break
 			;;
 		esac
+		orig_flags="$orig_flags $(quote_words "$1")"
+		shift
 	done
 
 	if test -n "$init"
@@ -533,7 +529,7 @@ cmd_update()
 
 		if test -n "$recursive"
 		then
-			eval "set - $orig_args"
+			eval "set - $orig_flags"
 			(clear_local_git_env; cd "$path" && cmd_update "$@") ||
 			die "Failed to recurse into submodule path '$path'"
 		fi
@@ -767,7 +763,7 @@ cmd_summary() {
 cmd_status()
 {
 	# parse $args after "submodule ... status".
-	orig_args="$(quote_words "$@")"
+	orig_flags=
 	while test $# -ne 0
 	do
 		case "$1" in
@@ -791,6 +787,7 @@ cmd_status()
 			break
 			;;
 		esac
+		orig_flags="$orig_flags $(quote_words "$1")"
 		shift
 	done
 
@@ -822,7 +819,7 @@ cmd_status()
 		then
 			(
 				prefix="$displaypath/"
-				eval "set - $orig_args"
+				eval "set - $orig_flags"
 				clear_local_git_env
 				cd "$path" &&
 				cmd_status "$@"
diff --git a/t/t7407-submodule-foreach.sh b/t/t7407-submodule-foreach.sh
index 905a8ba..2d5a855 100755
--- a/t/t7407-submodule-foreach.sh
+++ b/t/t7407-submodule-foreach.sh
@@ -238,4 +238,23 @@ test_expect_success 'use "git clone --recursive" to checkout all submodules' '
 	test -d clone4/nested1/nested2/nested3/submodule/.git
 '
 
+test_expect_success 'use "update --recursive nested1" to checkout all submodules rooted in nested1' '
+	git clone super clone5 &&
+	(
+		cd clone5 &&
+		test ! -d sub1/.git &&
+		test ! -d sub2/.git &&
+		test ! -d sub3/.git &&
+		test ! -d nested1/.git &&
+		git submodule update --init --recursive -- nested1 &&
+		test ! -d sub1/.git &&
+		test ! -d sub2/.git &&
+		test ! -d sub3/.git &&
+		test -d nested1/.git &&
+		test -d nested1/nested2/.git &&
+		test -d nested1/nested2/nested3/.git &&
+		test -d nested1/nested2/nested3/submodule/.git
+	)
+'
+
 test_done
-- 
1.7.3.2.200.ga1bd
Jonathan Nieder· Nov 3, 2010, 04:37 UTC · re: Kevin Ballard · lore

Re: [PATCH 1/2] submodule: preserve all arguments exactly when recursing

Hi Kevin,
Kevin Ballard wrote:
>            It's also a reasonable model for how to handle this problem
> in other shell commands.
This caught my attention. :)
Show 13 quoted lines
> +# Emit a quoted version of the all argument suitable for passing to `eval`
> +# $@ = words to quote
> +#
> +# This is intended to be used like the following:
> +#   orig_args="$(quote_words "$@")"
> +#   # do some work that includes calling shift
> +#   eval "set - $orig_args"
> +#   # now $@ has been restored, suitable for passing to another command
> +#
> +# Note that you cannot simply save off $@ into another variable because
> +# the shell gives $@ and $* special handling in parameter expansion
> +#
> +quote_words ()
Have you looked into "git rev-parse --sq-quote"?

Hope that helps, Jonathan

Kevin Ballard· Nov 3, 2010, 04:40 UTC · re: Jonathan Nieder · lore

Re: [PATCH 1/2] submodule: preserve all arguments exactly when recursing

On Nov 2, 2010, at 9:37 PM, Jonathan Nieder wrote:
Show 24 quoted lines
> Hi Kevin,
> 
> Kevin Ballard wrote:
> 
>>           It's also a reasonable model for how to handle this problem
>> in other shell commands.
> 
> This caught my attention. :)
> 
>> +# Emit a quoted version of the all argument suitable for passing to `eval`
>> +# $@ = words to quote
>> +#
>> +# This is intended to be used like the following:
>> +#   orig_args="$(quote_words "$@")"
>> +#   # do some work that includes calling shift
>> +#   eval "set - $orig_args"
>> +#   # now $@ has been restored, suitable for passing to another command
>> +#
>> +# Note that you cannot simply save off $@ into another variable because
>> +# the shell gives $@ and $* special handling in parameter expansion
>> +#
>> +quote_words ()
> 
> Have you looked into "git rev-parse --sq-quote"?

Well crud, I wish I'd seen that before. Looks like it does pretty much the exact same thing as my quote_words function. I'll send out another patch that uses this instead of quote_words. Thanks for the info!

-Kevin Ballard
Kevin Ballard· Nov 3, 2010, 05:05 UTC · re: Kevin Ballard · lore

[PATCHv2 1/2] submodule: preserve all arguments exactly when recursing

When performing a recursive status or update, any argments with whitespace would be split along that whitespace when passed to the recursive invocation of the update or status command.

This is caused by the special handling that sh provides to the $@ variable. Status and update stored "$@" into a separate variable, and passed that variable to the recursive invocation. Unfortunately, the special handling afforded to $@ isn't given to this new variable, and word-breaking occurs along whitespace boundaries.

We can use $(git rev-parse --sq-quote "$@") to produce a string containing a quoted version of all given args, suitable for passing to eval. We then recurse using something like `eval cmd_status "$orig_args"` instead of the former `cmd_status $orig_args`. This preserves all arguments exactly as given to the initial invocation of the command.

Helped-By: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Kevin Ballard <kevin@sb.org>
---
This patch ditches the quote_words function and uses
`git rev-parse --sq-quote "$@"` instead, as suggested by Jonathan Nieder.
 git-submodule.sh |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)
Show changes to git-submodule.sh +4 −4
diff --git a/git-submodule.sh b/git-submodule.sh
index 9ebbab7..543554b 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -374,7 +374,7 @@ cmd_init()
 cmd_update()
 {
 	# parse $args after "submodule ... update".
-	orig_args="$@"
+	orig_args="$(git rev-parse --sq-quote "$@")"
 	while test $# -ne 0
 	do
 		case "$1" in
@@ -500,7 +500,7 @@ cmd_update()
 
 		if test -n "$recursive"
 		then
-			(clear_local_git_env; cd "$path" && cmd_update $orig_args) ||
+			(clear_local_git_env; cd "$path" && eval cmd_update "$orig_args") ||
 			die "Failed to recurse into submodule path '$path'"
 		fi
 	done
@@ -733,7 +733,7 @@ cmd_summary() {
 cmd_status()
 {
 	# parse $args after "submodule ... status".
-	orig_args="$@"
+	orig_args="$(git rev-parse --sq-quote "$@")"
 	while test $# -ne 0
 	do
 		case "$1" in
@@ -790,7 +790,7 @@ cmd_status()
 				prefix="$displaypath/"
 				clear_local_git_env
 				cd "$path" &&
-				cmd_status $orig_args
+				eval cmd_status "$orig_args"
 			) ||
 			die "Failed to recurse into submodule path '$path'"
 		fi
-- 
1.7.3.2.200.g479de
Jonathan Nieder· Nov 3, 2010, 05:28 UTC · re: Kevin Ballard · lore

Re: [PATCHv2 1/2] submodule: preserve all arguments exactly when recursing

Kevin Ballard wrote:
Show 15 quoted lines
> When performing a recursive status or update, any argments with whitespace
> would be split along that whitespace when passed to the recursive invocation
> of the update or status command.
> 
> This is caused by the special handling that sh provides to the $@ variable.
> Status and update stored "$@" into a separate variable, and passed that
> variable to the recursive invocation. Unfortunately, the special handling
> afforded to $@ isn't given to this new variable, and word-breaking occurs
> along whitespace boundaries.
> 
> We can use $(git rev-parse --sq-quote "$@") to produce a string containing
> a quoted version of all given args, suitable for passing to eval. We then
> recurse using something like `eval cmd_status "$orig_args"` instead of the
> former `cmd_status $orig_args`. This preserves all arguments exactly as
> given to the initial invocation of the command.

Probably it is because it is late hear, but I find myself intimidated by the block of explanatory text. Maybe an example like

	Environment variables only hold strings, not lists of parameters,
	so $orig_args after
		orig_args="$@"
	fails to remember where each parameter starts and ends, if
	some include whitespace.  So
		git submodule update \
			--reference='/var/lib/common objects.git' \
			--recursive
	becomes
		git submodule update --reference=/var/lib/common \
			objects.git --recursive
	in the inner repositories.  Use "git rev-parse --sq-quote" to
	save parameters in quoted form ready for evaluation by the
	shell, avoiding this problem.
would be simpler?
Show 8 quoted lines
> --- a/git-submodule.sh
> +++ b/git-submodule.sh
> @@ -374,7 +374,7 @@ cmd_init()
>  cmd_update()
>  {
>  	# parse $args after "submodule ... update".
> -	orig_args="$@"
> +	orig_args="$(git rev-parse --sq-quote "$@")"
No quotes are needed around the RHS to an assignment like this.
Anyway,
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Thanks.
Kevin Ballard· Nov 3, 2010, 05:35 UTC · re: Jonathan Nieder · lore

Re: [PATCHv2 1/2] submodule: preserve all arguments exactly when recursing

On Nov 2, 2010, at 10:28 PM, Jonathan Nieder wrote:
Show 43 quoted lines
> Kevin Ballard wrote:
> 
>> When performing a recursive status or update, any argments with whitespace
>> would be split along that whitespace when passed to the recursive invocation
>> of the update or status command.
>> 
>> This is caused by the special handling that sh provides to the $@ variable.
>> Status and update stored "$@" into a separate variable, and passed that
>> variable to the recursive invocation. Unfortunately, the special handling
>> afforded to $@ isn't given to this new variable, and word-breaking occurs
>> along whitespace boundaries.
>> 
>> We can use $(git rev-parse --sq-quote "$@") to produce a string containing
>> a quoted version of all given args, suitable for passing to eval. We then
>> recurse using something like `eval cmd_status "$orig_args"` instead of the
>> former `cmd_status $orig_args`. This preserves all arguments exactly as
>> given to the initial invocation of the command.
> 
> Probably it is because it is late hear, but I find myself intimidated
> by the block of explanatory text.  Maybe an example like
> 
> 	Environment variables only hold strings, not lists of parameters,
> 	so $orig_args after
> 
> 		orig_args="$@"
> 
> 	fails to remember where each parameter starts and ends, if
> 	some include whitespace.  So
> 
> 		git submodule update \
> 			--reference='/var/lib/common objects.git' \
> 			--recursive
> 
> 	becomes
> 
> 		git submodule update --reference=/var/lib/common \
> 			objects.git --recursive
> 
> 	in the inner repositories.  Use "git rev-parse --sq-quote" to
> 	save parameters in quoted form ready for evaluation by the
> 	shell, avoiding this problem.
> 
> would be simpler?
I agree, yours is simpler. I'll send one final patch with the updated description.
Show 10 quoted lines
>> --- a/git-submodule.sh
>> +++ b/git-submodule.sh
>> @@ -374,7 +374,7 @@ cmd_init()
>> cmd_update()
>> {
>> 	# parse $args after "submodule ... update".
>> -	orig_args="$@"
>> +	orig_args="$(git rev-parse --sq-quote "$@")"
> 
> No quotes are needed around the RHS to an assignment like this.

Hrm, looks like you're right. I was under the impression it was necessary in some cases when the output of the command included newlines, but a quick test shows that belief to be wrong.

> Anyway,
> Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Thanks.
-Kevin Ballard
Kevin Ballard· Nov 3, 2010, 05:05 UTC · re: Kevin Ballard · lore

[PATCHv2 2/2] submodule: only preserve flags across recursive status/update invocations

Recursive invocations of submodule update/status preserve all arguments in the recursive invocation. This tends to cause undesired behavior when those arguments include the name of a submodule, as it tries to recurse into a nested submodule with the same name rather than recursing into all child submodules of the named submodule.

This commit changes the argument preservation to only preserve flags. When specifying a submodule name on the command-line, all child submodules of that named submodule will be recursed into correctly, and it will not attempt to recurse into a phantom nested submodule with the same name as its parent.

Signed-off-by: Kevin Ballard <kevin@sb.org>
---
I'm tempted to make this commit also omit the --reference flag in recursive
calls to cmd_update, as I don't believe it makes sense to use --reference
in conjunction with --recursive, but it may be a better idea to simply
produce an error if both flags are used together.
 git-submodule.sh             |   19 ++++++++-----------
 t/t7407-submodule-foreach.sh |   19 +++++++++++++++++++
 2 files changed, 27 insertions(+), 11 deletions(-)
Show changes to 2 files +27 −11

git-submodule.sh, t/t7407-submodule-foreach.sh

diff --git a/git-submodule.sh b/git-submodule.sh
index 543554b..4fd8982 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -374,41 +374,35 @@ cmd_init()
 cmd_update()
 {
 	# parse $args after "submodule ... update".
-	orig_args="$(git rev-parse --sq-quote "$@")"
+	orig_flags=
 	while test $# -ne 0
 	do
 		case "$1" in
 		-q|--quiet)
-			shift
 			GIT_QUIET=1
 			;;
 		-i|--init)
 			init=1
-			shift
 			;;
 		-N|--no-fetch)
-			shift
 			nofetch=1
 			;;
 		-r|--rebase)
-			shift
 			update="rebase"
 			;;
 		--reference)
 			case "$2" in '') usage ;; esac
 			reference="--reference=$2"
-			shift 2
+			orig_flags="$orig_flags $(git rev-parse --sq-quote "$1")"
+			shift
 			;;
 		--reference=*)
 			reference="$1"
-			shift
 			;;
 		-m|--merge)
-			shift
 			update="merge"
 			;;
 		--recursive)
-			shift
 			recursive=1
 			;;
 		--)
@@ -422,6 +416,8 @@ cmd_update()
 			break
 			;;
 		esac
+		orig_flags="$orig_flags $(git rev-parse --sq-quote "$1")"
+		shift
 	done
 
 	if test -n "$init"
@@ -500,7 +496,7 @@ cmd_update()
 
 		if test -n "$recursive"
 		then
-			(clear_local_git_env; cd "$path" && eval cmd_update "$orig_args") ||
+			(clear_local_git_env; cd "$path" && eval cmd_update "$orig_flags") ||
 			die "Failed to recurse into submodule path '$path'"
 		fi
 	done
@@ -733,7 +729,7 @@ cmd_summary() {
 cmd_status()
 {
 	# parse $args after "submodule ... status".
-	orig_args="$(git rev-parse --sq-quote "$@")"
+	orig_flags=
 	while test $# -ne 0
 	do
 		case "$1" in
@@ -757,6 +753,7 @@ cmd_status()
 			break
 			;;
 		esac
+		orig_flags="$orig_flags $(git rev-parse --sq-quote "$1")"
 		shift
 	done
 
diff --git a/t/t7407-submodule-foreach.sh b/t/t7407-submodule-foreach.sh
index 905a8ba..2d5a855 100755
--- a/t/t7407-submodule-foreach.sh
+++ b/t/t7407-submodule-foreach.sh
@@ -238,4 +238,23 @@ test_expect_success 'use "git clone --recursive" to checkout all submodules' '
 	test -d clone4/nested1/nested2/nested3/submodule/.git
 '
 
+test_expect_success 'use "update --recursive nested1" to checkout all submodules rooted in nested1' '
+	git clone super clone5 &&
+	(
+		cd clone5 &&
+		test ! -d sub1/.git &&
+		test ! -d sub2/.git &&
+		test ! -d sub3/.git &&
+		test ! -d nested1/.git &&
+		git submodule update --init --recursive -- nested1 &&
+		test ! -d sub1/.git &&
+		test ! -d sub2/.git &&
+		test ! -d sub3/.git &&
+		test -d nested1/.git &&
+		test -d nested1/nested2/.git &&
+		test -d nested1/nested2/nested3/.git &&
+		test -d nested1/nested2/nested3/submodule/.git
+	)
+'
+
 test_done
-- 
1.7.3.2.200.g479de
Jonathan Nieder· Nov 3, 2010, 05:38 UTC · re: Kevin Ballard · lore

Re: [PATCHv2 2/2] submodule: only preserve flags across recursive status/update invocations

Kevin Ballard wrote:
> --- a/git-submodule.sh
> +++ b/git-submodule.sh
> @@ -374,41 +374,35 @@ cmd_init()
[...]
Show 6 quoted lines
>  		--reference)
>  			case "$2" in '') usage ;; esac
>  			reference="--reference=$2"
> -			shift 2
> +			orig_flags="$orig_flags $(git rev-parse --sq-quote "$1")"
> +			shift
Hmm.  Maybe a helper would make it clearer.
	save_arg () {
		orig_flags="$orig_flags $(git rev-parse --sq-quote "$1")"
	}
making this
 +			save_arg --reference
 +			shift
Show 7 quoted lines
> --- a/t/t7407-submodule-foreach.sh
> +++ b/t/t7407-submodule-foreach.sh
> @@ -238,4 +238,23 @@ test_expect_success 'use "git clone --recursive" to checkout all submodules' '
>  	test -d clone4/nested1/nested2/nested3/submodule/.git
>  '
>  
> +test_expect_success 'use "update --recursive nested1" to checkout all submodules rooted in nested1' '
Maybe a submodule status --cached --recursive -- <files> test, too?

Sleepily, Jonathan

Kevin Ballard· Nov 3, 2010, 05:45 UTC · re: Jonathan Nieder · lore

Re: [PATCHv2 2/2] submodule: only preserve flags across recursive status/update invocations

On Nov 2, 2010, at 10:38 PM, Jonathan Nieder wrote:
Show 23 quoted lines
> Kevin Ballard wrote:
> 
>> --- a/git-submodule.sh
>> +++ b/git-submodule.sh
>> @@ -374,41 +374,35 @@ cmd_init()
> [...]
>> 		--reference)
>> 			case "$2" in '') usage ;; esac
>> 			reference="--reference=$2"
>> -			shift 2
>> +			orig_flags="$orig_flags $(git rev-parse --sq-quote "$1")"
>> +			shift
> 
> Hmm.  Maybe a helper would make it clearer.
> 
> 	save_arg () {
> 		orig_flags="$orig_flags $(git rev-parse --sq-quote "$1")"
> 	}
> 
> making this
> 
> +			save_arg --reference
> +			shift

I considered that, but I already dislike the fact that orig_flags is a global. I left it as such for the sake of not fixing what ain't broke, but the use of a helper function would imply that the global nature of the variable is intentional.

Show 9 quoted lines
>> --- a/t/t7407-submodule-foreach.sh
>> +++ b/t/t7407-submodule-foreach.sh
>> @@ -238,4 +238,23 @@ test_expect_success 'use "git clone --recursive" to checkout all submodules' '
>> 	test -d clone4/nested1/nested2/nested3/submodule/.git
>> '
>> 
>> +test_expect_success 'use "update --recursive nested1" to checkout all submodules rooted in nested1' '
> 
> Maybe a submodule status --cached --recursive -- <files> test, too?
Good idea.
-Kevin Ballard
Kevin Ballard· Nov 3, 2010, 06:26 UTC · re: Jonathan Nieder · lore

[PATCHv3 1/2] submodule: preserve all arguments exactly when recursing

Shell variables only hold strings, not lists of parameters, so $orig_args after

        orig_args="$@"

fails to remember where each parameter starts and ends, if some include whitespace. So

        git submodule update \
                --reference='/var/lib/common objects.git' \
                --recursive --init
becomes
        git submodule update --reference=/var/lib/common \
                objects.git --recursive --init

in the inner repositories. Use "git rev-parse --sq-quote" to save parameters in quoted form ready for evaluation by the shell, avoiding this problem.

Helped-By: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Kevin Ballard <kevin@sb.org>
---
This version introduces a test using --reference.
It also uses the nice description written by Jonathan
and it stops unnecessarily quoting the results of
the call to $(git rev-parse --sq-quote "$@")
 git-submodule.sh             |    8 ++++----
 t/t7407-submodule-foreach.sh |   16 ++++++++++++++++
 2 files changed, 20 insertions(+), 4 deletions(-)
Show changes to 2 files +20 −4

git-submodule.sh, t/t7407-submodule-foreach.sh

diff --git a/git-submodule.sh b/git-submodule.sh
index 9ebbab7..4d2bb37 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -374,7 +374,7 @@ cmd_init()
 cmd_update()
 {
 	# parse $args after "submodule ... update".
-	orig_args="$@"
+	orig_args=$(git rev-parse --sq-quote "$@")
 	while test $# -ne 0
 	do
 		case "$1" in
@@ -500,7 +500,7 @@ cmd_update()
 
 		if test -n "$recursive"
 		then
-			(clear_local_git_env; cd "$path" && cmd_update $orig_args) ||
+			(clear_local_git_env; cd "$path" && eval cmd_update "$orig_args") ||
 			die "Failed to recurse into submodule path '$path'"
 		fi
 	done
@@ -733,7 +733,7 @@ cmd_summary() {
 cmd_status()
 {
 	# parse $args after "submodule ... status".
-	orig_args="$@"
+	orig_args=$(git rev-parse --sq-quote "$@")
 	while test $# -ne 0
 	do
 		case "$1" in
@@ -790,7 +790,7 @@ cmd_status()
 				prefix="$displaypath/"
 				clear_local_git_env
 				cd "$path" &&
-				cmd_status $orig_args
+				eval cmd_status "$orig_args"
 			) ||
 			die "Failed to recurse into submodule path '$path'"
 		fi
diff --git a/t/t7407-submodule-foreach.sh b/t/t7407-submodule-foreach.sh
index 905a8ba..15d420f 100755
--- a/t/t7407-submodule-foreach.sh
+++ b/t/t7407-submodule-foreach.sh
@@ -238,4 +238,20 @@ test_expect_success 'use "git clone --recursive" to checkout all submodules' '
 	test -d clone4/nested1/nested2/nested3/submodule/.git
 '
 
+test_expect_success 'test "update --recursive" with a flag with spaces' '
+	git clone super "common objects" &&
+	git clone super clone5 &&
+	(
+		cd clone5 &&
+		test ! -d nested1/.git &&
+		git submodule update --init --recursive --reference="$(dirname "$PWD")/common objects" &&
+		test -d nested1/.git &&
+		test -d nested1/nested2/.git &&
+		test -d nested1/nested2/nested3/.git &&
+		test -f nested1/.git/objects/info/alternates &&
+		test -f nested1/nested2/.git/objects/info/alternates &&
+		test -f nested1/nested2/nested3/.git/objects/info/alternates
+	)
+'
+
 test_done
-- 
1.7.3.2.200.g862e8
Kevin Ballard· Nov 3, 2010, 06:26 UTC · re: Jonathan Nieder · lore

[PATCHv3 2/2] submodule: only preserve flags across recursive status/update invocations

Recursive invocations of submodule update/status preserve all arguments, so executing

        git submodule update --recursive -- foo
attempts to recursively update a submodule named "foo".

Naturally, this fails as one cannot have an infinitely-deep stack of submodules each containing a submodule named "foo". The desired behavior is instead to update foo and then recursively update all submodules inside of foo.

This commit accomplishes that by only saving the flags for use in the recursive invocation.

Signed-off-by: Kevin Ballard <kevin@sb.org>
---
As suggested by Jonathan Nieder, I've included a test for
`git submodule status --cached --recursive -- nested` to verify
that the flags are preserved but the paths aren't.
 git-submodule.sh             |   19 ++++++++-----------
 t/t7407-submodule-foreach.sh |   34 ++++++++++++++++++++++++++++++++++
 2 files changed, 42 insertions(+), 11 deletions(-)
Show changes to 2 files +42 −11

git-submodule.sh, t/t7407-submodule-foreach.sh

diff --git a/git-submodule.sh b/git-submodule.sh
index 4d2bb37..4fd8982 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -374,41 +374,35 @@ cmd_init()
 cmd_update()
 {
 	# parse $args after "submodule ... update".
-	orig_args=$(git rev-parse --sq-quote "$@")
+	orig_flags=
 	while test $# -ne 0
 	do
 		case "$1" in
 		-q|--quiet)
-			shift
 			GIT_QUIET=1
 			;;
 		-i|--init)
 			init=1
-			shift
 			;;
 		-N|--no-fetch)
-			shift
 			nofetch=1
 			;;
 		-r|--rebase)
-			shift
 			update="rebase"
 			;;
 		--reference)
 			case "$2" in '') usage ;; esac
 			reference="--reference=$2"
-			shift 2
+			orig_flags="$orig_flags $(git rev-parse --sq-quote "$1")"
+			shift
 			;;
 		--reference=*)
 			reference="$1"
-			shift
 			;;
 		-m|--merge)
-			shift
 			update="merge"
 			;;
 		--recursive)
-			shift
 			recursive=1
 			;;
 		--)
@@ -422,6 +416,8 @@ cmd_update()
 			break
 			;;
 		esac
+		orig_flags="$orig_flags $(git rev-parse --sq-quote "$1")"
+		shift
 	done
 
 	if test -n "$init"
@@ -500,7 +496,7 @@ cmd_update()
 
 		if test -n "$recursive"
 		then
-			(clear_local_git_env; cd "$path" && eval cmd_update "$orig_args") ||
+			(clear_local_git_env; cd "$path" && eval cmd_update "$orig_flags") ||
 			die "Failed to recurse into submodule path '$path'"
 		fi
 	done
@@ -733,7 +729,7 @@ cmd_summary() {
 cmd_status()
 {
 	# parse $args after "submodule ... status".
-	orig_args=$(git rev-parse --sq-quote "$@")
+	orig_flags=
 	while test $# -ne 0
 	do
 		case "$1" in
@@ -757,6 +753,7 @@ cmd_status()
 			break
 			;;
 		esac
+		orig_flags="$orig_flags $(git rev-parse --sq-quote "$1")"
 		shift
 	done
 
diff --git a/t/t7407-submodule-foreach.sh b/t/t7407-submodule-foreach.sh
index 15d420f..d8ad250 100755
--- a/t/t7407-submodule-foreach.sh
+++ b/t/t7407-submodule-foreach.sh
@@ -226,6 +226,21 @@ test_expect_success 'test "status --recursive"' '
 	test_cmp expect actual
 '
 
+sed -e "/nested1 /s/.*/+$nested1sha1 nested1 (file2~1)/;/sub[1-3]/d" < expect > expect2
+mv -f expect2 expect
+
+test_expect_success 'ensure "status --cached --recursive" preserves the --cached flag' '
+	(
+		cd clone3 &&
+		(
+			cd nested1 &&
+			test_commit file2
+		) &&
+		git submodule status --cached --recursive -- nested1 > ../actual
+	) &&
+	test_cmp expect actual
+'
+
 test_expect_success 'use "git clone --recursive" to checkout all submodules' '
 	git clone --recursive super clone4 &&
 	test -d clone4/.git &&
@@ -254,4 +269,23 @@ test_expect_success 'test "update --recursive" with a flag with spaces' '
 	)
 '
 
+test_expect_success 'use "update --recursive nested1" to checkout all submodules rooted in nested1' '
+	git clone super clone6 &&
+	(
+		cd clone6 &&
+		test ! -d sub1/.git &&
+		test ! -d sub2/.git &&
+		test ! -d sub3/.git &&
+		test ! -d nested1/.git &&
+		git submodule update --init --recursive -- nested1 &&
+		test ! -d sub1/.git &&
+		test ! -d sub2/.git &&
+		test ! -d sub3/.git &&
+		test -d nested1/.git &&
+		test -d nested1/nested2/.git &&
+		test -d nested1/nested2/nested3/.git &&
+		test -d nested1/nested2/nested3/submodule/.git
+	)
+'
+
 test_done
-- 
1.7.3.2.200.g862e8
Jonathan Nieder· Nov 3, 2010, 05:47 UTC · re: Kevin Ballard · lore

Re: [PATCH 1/2] submodule: preserve all arguments exactly when recursing

Kevin Ballard wrote:
> On Nov 2, 2010, at 9:37 PM, Jonathan Nieder wrote:
Show 5 quoted lines
>> Have you looked into "git rev-parse --sq-quote"?
>
> Well crud, I wish I'd seen that before. Looks like it does pretty much the
> exact same thing as my quote_words function. I'll send out another patch
> that uses this instead of quote_words. Thanks for the info!

Incidentally, if you still remember where it might have been useful to find this advertised (git-sh-setup?), maybe we can find a way to document it better.

Thanks for the submodule fix.  Well spotted.
Kevin Ballard· Nov 3, 2010, 05:49 UTC · re: Jonathan Nieder · lore

Re: [PATCH 1/2] submodule: preserve all arguments exactly when recursing

On Nov 2, 2010, at 10:47 PM, Jonathan Nieder wrote:
Show 12 quoted lines
> Kevin Ballard wrote:
>> On Nov 2, 2010, at 9:37 PM, Jonathan Nieder wrote:
> 
>>> Have you looked into "git rev-parse --sq-quote"?
>> 
>> Well crud, I wish I'd seen that before. Looks like it does pretty much the
>> exact same thing as my quote_words function. I'll send out another patch
>> that uses this instead of quote_words. Thanks for the info!
> 
> Incidentally, if you still remember where it might have been useful
> to find this advertised (git-sh-setup?), maybe we can find a way to document
> it better.

It might make sense to mention it in CodingGuidelines. That's where I always look when writing shell code in order to make sure I'm not using any bashisms.

-Kevin Ballard
Jonathan Nieder· Nov 3, 2010, 05:30 UTC · re: Kevin Ballard · lore

Re: [PATCH 1/2] submodule: preserve all arguments exactly when recursing

Kevin Ballard wrote:
Show 5 quoted lines
> I tried to write tests for this, but there are only two ways to get args
> with spaces to be accepted and passed to the recursive invocation.
> The first is via the --reference flag, but I don't think it really makes
> sense to use that flag in connection with --recursive and was not comfortable
> using it in a test.

Could you explain this further? I think --reference pointing to a repository with objects shared by multiple (sub)projects makes a lot of sense.

Kevin Ballard· Nov 3, 2010, 05:36 UTC · re: Jonathan Nieder · lore

Re: [PATCH 1/2] submodule: preserve all arguments exactly when recursing

On Nov 2, 2010, at 10:30 PM, Jonathan Nieder wrote:
Show 11 quoted lines
> Kevin Ballard wrote:
> 
>> I tried to write tests for this, but there are only two ways to get args
>> with spaces to be accepted and passed to the recursive invocation.
>> The first is via the --reference flag, but I don't think it really makes
>> sense to use that flag in connection with --recursive and was not comfortable
>> using it in a test.
> 
> Could you explain this further?  I think --reference pointing to a
> repository with objects shared by multiple (sub)projects makes a lot
> of sense.

I hadn't given it a lot of thought, but I suppose you're right. I'm used to using submodules to pull in disparate projects that don't share objects, but I can imagine having repos that do share objects. I will try to write a test for this using --reference.

-Kevin Ballard

← back to recent threads