threads / patch / 32563

patcht1402: work around shell quoting issue on NetBSD

Subject: [PATCH] t1402: work around shell quoting issue on NetBSD

## tl;dr

11 messages between Jan 8, 2013 and Jan 10, 2013. Diffs are folded; open one to read it.

replies: 10people: 4as markdown or json

René Scharfe· Jan 8, 2013, 20:23 UTC · lore
The test fails for me on NetBSD 6.0.1 and reports:
	ok 1 - ref name '' is invalid
	ok 2 - ref name '/' is invalid
	ok 3 - ref name '/' is invalid with options --allow-onelevel
	ok 4 - ref name '/' is invalid with options --normalize
	error: bug in the test script: not 2 or 3 parameters to test-expect-success
The alleged bug is in this line:
	invalid_ref NOT_MINGW '/' '--allow-onelevel --normalize'

invalid_ref() constructs a test case description using its last argument, but the shell seems to split it up into two pieces if it contains a space. Minimal test case:

	# on NetBSD with /bin/sh
	$ a() { echo $#-$1-$2; }
	$ t="x"; a "${t:+$t}"
	1-x-
	$ t="x y"; a "${t:+$t}"
	2-x-y
	$ t="x y"; a "${t:+x y}"
	1-x y-
	# and with bash
	$ t="x y"; a "${t:+$t}"
	1-x y-
	$ t="x y"; a "${t:+x y}"
	1-x y-

This may be a bug in the shell, but here's a simple workaround: Construct the description string first and store it in a variable, and then use that to call test_expect_success().

Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
---
 t/t1402-check-ref-format.sh | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
Show changes to t/t1402-check-ref-format.sh +4 −2
diff --git a/t/t1402-check-ref-format.sh b/t/t1402-check-ref-format.sh
index 1ae4d87..1a5a5f3 100755
--- a/t/t1402-check-ref-format.sh
+++ b/t/t1402-check-ref-format.sh
@@ -11,7 +11,8 @@ valid_ref() {
 		prereq=$1
 		shift
 	esac
-	test_expect_success $prereq "ref name '$1' is valid${2:+ with options $2}" "
+	desc="ref name '$1' is valid${2:+ with options $2}"
+	test_expect_success $prereq "$desc" "
 		git check-ref-format $2 '$1'
 	"
 }
@@ -22,7 +23,8 @@ invalid_ref() {
 		prereq=$1
 		shift
 	esac
-	test_expect_success $prereq "ref name '$1' is invalid${2:+ with options $2}" "
+	desc="ref name '$1' is invalid${2:+ with options $2}"
+	test_expect_success $prereq "$desc" "
 		test_must_fail git check-ref-format $2 '$1'
 	"
 }
-- 
1.7.12
Junio C Hamano· Jan 8, 2013, 20:39 UTC · re: René Scharfe · lore

Re: [PATCH] t1402: work around shell quoting issue on NetBSD

René Scharfe <rene.scharfe@lsrfire.ath.cx> writes:
Show 34 quoted lines
> The test fails for me on NetBSD 6.0.1 and reports:
>
> 	ok 1 - ref name '' is invalid
> 	ok 2 - ref name '/' is invalid
> 	ok 3 - ref name '/' is invalid with options --allow-onelevel
> 	ok 4 - ref name '/' is invalid with options --normalize
> 	error: bug in the test script: not 2 or 3 parameters to test-expect-success
>
> The alleged bug is in this line:
>
> 	invalid_ref NOT_MINGW '/' '--allow-onelevel --normalize'
>
> invalid_ref() constructs a test case description using its last argument,
> but the shell seems to split it up into two pieces if it contains a
> space.  Minimal test case:
>
> 	# on NetBSD with /bin/sh
> 	$ a() { echo $#-$1-$2; }
> 	$ t="x"; a "${t:+$t}"
> 	1-x-
> 	$ t="x y"; a "${t:+$t}"
> 	2-x-y
> 	$ t="x y"; a "${t:+x y}"
> 	1-x y-
>
> 	# and with bash
> 	$ t="x y"; a "${t:+$t}"
> 	1-x y-
> 	$ t="x y"; a "${t:+x y}"
> 	1-x y-
>
> This may be a bug in the shell, but here's a simple workaround: Construct
> the description string first and store it in a variable, and then use
> that to call test_expect_success().

Interesting. I notice that t0008 added recently to 'pu' has the same construct.

Show 30 quoted lines
>
> Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
> ---
>  t/t1402-check-ref-format.sh | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/t/t1402-check-ref-format.sh b/t/t1402-check-ref-format.sh
> index 1ae4d87..1a5a5f3 100755
> --- a/t/t1402-check-ref-format.sh
> +++ b/t/t1402-check-ref-format.sh
> @@ -11,7 +11,8 @@ valid_ref() {
>  		prereq=$1
>  		shift
>  	esac
> -	test_expect_success $prereq "ref name '$1' is valid${2:+ with options $2}" "
> +	desc="ref name '$1' is valid${2:+ with options $2}"
> +	test_expect_success $prereq "$desc" "
>  		git check-ref-format $2 '$1'
>  	"
>  }
> @@ -22,7 +23,8 @@ invalid_ref() {
>  		prereq=$1
>  		shift
>  	esac
> -	test_expect_success $prereq "ref name '$1' is invalid${2:+ with options $2}" "
> +	desc="ref name '$1' is invalid${2:+ with options $2}"
> +	test_expect_success $prereq "$desc" "
>  		test_must_fail git check-ref-format $2 '$1'
>  	"
>  }
René Scharfe· Jan 8, 2013, 21:13 UTC · re: Junio C Hamano · lore

Re: [PATCH] t1402: work around shell quoting issue on NetBSD

Am 08.01.2013 21:39, schrieb Junio C Hamano:
Show 22 quoted lines
> René Scharfe <rene.scharfe@lsrfire.ath.cx> writes:
>> 	# on NetBSD with /bin/sh
>> 	$ a() { echo $#-$1-$2; }
>> 	$ t="x"; a "${t:+$t}"
>> 	1-x-
>> 	$ t="x y"; a "${t:+$t}"
>> 	2-x-y
>> 	$ t="x y"; a "${t:+x y}"
>> 	1-x y-
>>
>> 	# and with bash
>> 	$ t="x y"; a "${t:+$t}"
>> 	1-x y-
>> 	$ t="x y"; a "${t:+x y}"
>> 	1-x y-
>>
>> This may be a bug in the shell, but here's a simple workaround: Construct
>> the description string first and store it in a variable, and then use
>> that to call test_expect_success().
>
> Interesting.  I notice that t0008 added recently to 'pu' has the
> same construct.

A quick check shows that subtests 64-68 and 89-93 of t0008 fail for me on Debian (10 in total) and subtests 64 and 89 fail on NetBSD (2 in total). Unlike t1402 they don't report "bug in the test script".

t0008 only uses ${:+} substitution on variables that don't contain spaces. With the test changed to store the description in a variable first I still get the same 2 failures.

There must be something else going on here. The different results are interesting, especially the higher number of failures on Debian.

René
Junio C Hamano· Jan 8, 2013, 21:37 UTC · re: René Scharfe · lore

Re: [PATCH] t1402: work around shell quoting issue on NetBSD

René Scharfe <rene.scharfe@lsrfire.ath.cx> writes:
Show 10 quoted lines
> A quick check shows that subtests 64-68 and 89-93 of t0008 fail for me
> on Debian (10 in total) and subtests 64 and 89 fail on NetBSD (2 in
> total).  Unlike t1402 they don't report "bug in the test script".
>
> t0008 only uses ${:+} substitution on variables that don't contain
> spaces.  With the test changed to store the description in a variable
> first I still get the same 2 failures.
>
> There must be something else going on here.  The different results are
> interesting, especially the higher number of failures on Debian.

I forgot to mention that some of them seem to be broken under dash but pass under bash.

René Scharfe· Jan 9, 2013, 23:49 UTC · re: Junio C Hamano · lore

[PATCH] t0008: avoid brace expansion

Brace expansion is not required by POSIX and not supported by dash nor NetBSD's sh. Explicitly list all combinations instead.

Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
---
 t/t0008-ignores.sh | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
Show changes to t/t0008-ignores.sh +3 −2
diff --git a/t/t0008-ignores.sh b/t/t0008-ignores.sh
index 9b0fcd6..0273680 100755
--- a/t/t0008-ignores.sh
+++ b/t/t0008-ignores.sh
@@ -129,8 +129,9 @@ test_expect_success 'setup' '
 		one
 		ignored-*
 	EOF
-	touch {,a/}{not-ignored,ignored-{and-untracked,but-in-index}} &&
-	git add -f {,a/}ignored-but-in-index
+	touch not-ignored ignored-and-untracked ignored-but-in-index &&
+	touch a/not-ignored a/ignored-and-untracked a/ignored-but-in-index &&
+	git add -f ignored-but-in-index a/ignored-but-in-index &&
 	cat <<-\EOF >a/.gitignore &&
 		two*
 		*three
-- 
1.8.0
Adam Spiers· Jan 9, 2013, 23:56 UTC · re: René Scharfe · lore

Re: [PATCH] t0008: avoid brace expansion

On Wed, Jan 9, 2013 at 11:49 PM, René Scharfe <rene.scharfe@lsrfire.ath.cx> wrote:

> Brace expansion is not required by POSIX and not supported by dash nor
> NetBSD's sh.  Explicitly list all combinations instead.
Good catch, thanks!
Junio C Hamano· Jan 10, 2013, 00:18 UTC · re: Adam Spiers · lore

Re: [PATCH] t0008: avoid brace expansion

Adam Spiers <git@adamspiers.org> writes:
Show 6 quoted lines
> On Wed, Jan 9, 2013 at 11:49 PM, René Scharfe
> <rene.scharfe@lsrfire.ath.cx> wrote:
>> Brace expansion is not required by POSIX and not supported by dash nor
>> NetBSD's sh.  Explicitly list all combinations instead.
>
> Good catch, thanks!
Yeah; thanks.
It would also be nice to avoid touch while we are at it, by the way.
Adam Spiers· Jan 10, 2013, 00:22 UTC · re: Junio C Hamano · lore

Re: [PATCH] t0008: avoid brace expansion

On Thu, Jan 10, 2013 at 12:18 AM, Junio C Hamano <gitster@pobox.com> wrote:
Show 12 quoted lines
> Adam Spiers <git@adamspiers.org> writes:
>
>> On Wed, Jan 9, 2013 at 11:49 PM, René Scharfe
>> <rene.scharfe@lsrfire.ath.cx> wrote:
>>> Brace expansion is not required by POSIX and not supported by dash nor
>>> NetBSD's sh.  Explicitly list all combinations instead.
>>
>> Good catch, thanks!
>
> Yeah; thanks.
>
> It would also be nice to avoid touch while we are at it, by the way.
Noted.
René Scharfe· Jan 10, 2013, 21:24 UTC · re: Junio C Hamano · lore

Re: [PATCH] t0008: avoid brace expansion

Am 10.01.2013 01:18, schrieb Junio C Hamano:
Show 12 quoted lines
> Adam Spiers <git@adamspiers.org> writes:
> 
>> On Wed, Jan 9, 2013 at 11:49 PM, René Scharfe
>> <rene.scharfe@lsrfire.ath.cx> wrote:
>>> Brace expansion is not required by POSIX and not supported by dash nor
>>> NetBSD's sh.  Explicitly list all combinations instead.
>>
>> Good catch, thanks!
> 
> Yeah; thanks.
> 
> It would also be nice to avoid touch while we are at it, by the way.
Good idea!  Replacement patch:

--- >8 --- Brace expansion is a shell feature that's not required by POSIX and not supported by dash nor NetBSD's sh. Explicitly list all combinations instead. Also avoid calling touch by creating the test files with a redirection instead, as suggested by Junio.

Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
---
 t/t0008-ignores.sh | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
Show changes to t/t0008-ignores.sh +7 −2
diff --git a/t/t0008-ignores.sh b/t/t0008-ignores.sh
index 9b0fcd6..d7df719 100755
--- a/t/t0008-ignores.sh
+++ b/t/t0008-ignores.sh
@@ -129,8 +129,13 @@ test_expect_success 'setup' '
 		one
 		ignored-*
 	EOF
-	touch {,a/}{not-ignored,ignored-{and-untracked,but-in-index}} &&
-	git add -f {,a/}ignored-but-in-index
+	for dir in . a
+	do
+		: >$dir/not-ignored &&
+		: >$dir/ignored-and-untracked &&
+		: >$dir/ignored-but-in-index
+	done &&
+	git add -f ignored-but-in-index a/ignored-but-in-index &&
 	cat <<-\EOF >a/.gitignore &&
 		two*
 		*three
-- 
1.8.0
Greg Troxel· Jan 9, 2013, 01:27 UTC · re: René Scharfe · lore

Re: [PATCH] t1402: work around shell quoting issue on NetBSD

René Scharfe <rene.scharfe@lsrfire.ath.cx> writes:
> invalid_ref() constructs a test case description using its last argument,
> but the shell seems to split it up into two pieces if it contains a
> space.  Minimal test case:

This is indeed a bug in NetBSD's shell, which I reported after finding this test case problem, and I think someone is working on a fix. But because git does not intend to be a shell torture test, if it's possible to avoid bugs in a reasonable way, I think it's nice to do so.

Greg Troxel· Jan 9, 2013, 02:07 UTC · re: René Scharfe · lore

Re: [PATCH] t1402: work around shell quoting issue on NetBSD

René Scharfe <rene.scharfe@lsrfire.ath.cx> writes:
Show 11 quoted lines
> The test fails for me on NetBSD 6.0.1 and reports:
>
> 	ok 1 - ref name '' is invalid
> 	ok 2 - ref name '/' is invalid
> 	ok 3 - ref name '/' is invalid with options --allow-onelevel
> 	ok 4 - ref name '/' is invalid with options --normalize
> 	error: bug in the test script: not 2 or 3 parameters to test-expect-success
>
> The alleged bug is in this line:
>
> 	invalid_ref NOT_MINGW '/' '--allow-onelevel --normalize'
The bug in NetBSD's sh has been fixed in -current:
  http://gnats.netbsd.org/47361

and the change will almost certainly make it to the -6 and -5 release branches.

With the fixed sh:
  414c78c (with the workaround): t1402 passes
  69637e5 (without the workaround): t1402 passes
With the buggy sh,
  414c78c (with the workaround): t1402 passes
  69637e5 (without workaround): t1402 fails
so I can confirm that the workaround is successful on NetBSD 5.
Thanks for addressing this, and sorry I didn't mention it on this list.
Greg

← back to recent threads