# [PATCH] test-lib: check Bash version for '-x' without using shell arrays 14 messages from 2019-01-01 to 2019-01-04. Participants: SZEDER Gábor, Johannes Sixt, Junio C Hamano, Eric Sunshine, Carlo Arenas, Jeff King. Thread: https://gitlist.dev/t/50142 ## SZEDER Gábor, 2019-01-01 23:19 Subject: [PATCH] test-lib: check Bash version for '-x' without using shell arrays Message-ID: <20190101231949.8184-1-szeder.dev@gmail.com> URL: https://gitlist.dev/e/20190101231949.8184-1-szeder.dev%40gmail.com ``` One of our test scripts, 't1510-repo-setup.sh' [1], still can't be reliably run with '-x' tracing enabled, unless it's executed with a Bash version supporting BASH_XTRACEFD (since v4.1). We have a lengthy condition to check the version of the shell running the test script, and disable tracing if it's not executed with a suitable Bash version [2]. This condition uses non-portable shell array accesses to easily get Bash's major and minor version number. This didn't seem to be problematic, because the simple commands expanding those array accesses are only executed when the test script is actually run with Bash. When run with Dash, the only shell I have at hand that doesn't support shell arrays, there are no issues, as it apparently skips right over the non-executed simple commands without noticing the non-supported constructs. Alas, it has been reported that NetBSD's /bin/sh does complain about them: ./test-lib.sh: 327: Syntax error: Bad substitution where line 327 contains the first ${BASH_VERSINFO[0]} array access. To my understanding both shells are right and conform to POSIX, because the standard allows both behavior by stating the following under '2.8.1 Consequences of Shell Errors': "An expansion error is one that occurs when the shell expansions define in wordexp are carried out (for example, "${x!y}", because '!' is not a valid operator); an implementation may treat these as syntax errors if it is able to detect them during tokenization, rather than during expansion." Avoid this issue with NetBSD's /bin/sh (and potentially with other, less common shells) by using a couple of parameter expansions to extract the major and minor version numbers from $BASH_VERSION. [1] 5827506928 (t1510-repo-setup: mark as untraceable with '-x', 2018-02-24) [2] 5fc98e79fc (t: add means to disable '-x' tracing for individual test scripts, 2018-02-24) Reported-by: Max Kirillov Signed-off-by: SZEDER Gábor --- This will certainly conflict with patches 3 and 6 in my stress testing patch series at https://public-inbox.org/git/20181230191629.3232-1-szeder.dev@gmail.com/T/#u t/test-lib.sh | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/t/test-lib.sh b/t/test-lib.sh index 0f1faa24b2..f47a191e3b 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -324,9 +324,12 @@ do # isn't executed with a suitable Bash version. if test -z "$test_untraceable" || { test -n "$BASH_VERSION" && { - test ${BASH_VERSINFO[0]} -gt 4 || { - test ${BASH_VERSINFO[0]} -eq 4 && - test ${BASH_VERSINFO[1]} -ge 1 + bash_major=${BASH_VERSION%%.*} + bash_minor=${BASH_VERSION#*.} + bash_minor=${bash_minor%%.*} + test $bash_major -gt 4 || { + test $bash_major -eq 4 && + test $bash_minor -ge 1 } } } -- 2.20.1.151.gec613c4b75 ``` ## Johannes Sixt, 2019-01-02 00:20 Subject: Re: [PATCH] test-lib: check Bash version for '-x' without using shell arrays Message-ID: URL: https://gitlist.dev/e/a82251fa-38e1-233e-50d3-3ed4850b4e11%40kdbg.org In-Reply-To: <20190101231949.8184-1-szeder.dev@gmail.com> ``` Am 02.01.19 um 00:19 schrieb SZEDER Gábor: > Alas, it has been reported that NetBSD's /bin/sh does complain about > them: > > ./test-lib.sh: 327: Syntax error: Bad substitution > > where line 327 contains the first ${BASH_VERSINFO[0]} array access. > diff --git a/t/test-lib.sh b/t/test-lib.sh > index 0f1faa24b2..f47a191e3b 100644 > --- a/t/test-lib.sh > +++ b/t/test-lib.sh > @@ -324,9 +324,12 @@ do > # isn't executed with a suitable Bash version. > if test -z "$test_untraceable" || { > test -n "$BASH_VERSION" && { > - test ${BASH_VERSINFO[0]} -gt 4 || { > - test ${BASH_VERSINFO[0]} -eq 4 && > - test ${BASH_VERSINFO[1]} -ge 1 > + bash_major=${BASH_VERSION%%.*} > + bash_minor=${BASH_VERSION#*.} > + bash_minor=${bash_minor%%.*} > + test $bash_major -gt 4 || { > + test $bash_major -eq 4 && > + test $bash_minor -ge 1 > } > } > } > Would it perhaps be simpler to just hide the syntax behind eval? Like if test -z "$test_untraceable" || { test -n "$BASH_VERSION" && eval ' test ${BASH_VERSINFO[0]} -gt 4 || { test ${BASH_VERSINFO[0]} -eq 4 && test ${BASH_VERSINFO[1]} -ge 1 } ' -- Hannes ``` ## Junio C Hamano, 2019-01-02 16:46 Subject: Re: [PATCH] test-lib: check Bash version for '-x' without using shell arrays Message-ID: URL: https://gitlist.dev/e/xmqqa7kjknwp.fsf%40gitster-ct.c.googlers.com In-Reply-To: ``` Johannes Sixt writes: > Would it perhaps be simpler to just hide the syntax behind eval? Like > > if test -z "$test_untraceable" || { > test -n "$BASH_VERSION" && eval ' > test ${BASH_VERSINFO[0]} -gt 4 || { > test ${BASH_VERSINFO[0]} -eq 4 && > test ${BASH_VERSINFO[1]} -ge 1 > } > ' It indeed is somewhat easier to see that your version does exactly the same as the original than the posted patch, but the pattern substitutions are not all that bad, either. ``` ## Eric Sunshine, 2019-01-02 18:05 Subject: Re: [PATCH] test-lib: check Bash version for '-x' without using shell arrays Message-ID: URL: https://gitlist.dev/e/CAPig%2BcRLqy3Zdt4rUbMXC2dd1bhOXYZ%3D8HEn%2B%2Bzsb%3DRau7zADw%40mail.gmail.com In-Reply-To: <20190101231949.8184-1-szeder.dev@gmail.com> ``` On Tue, Jan 1, 2019 at 6:20 PM SZEDER Gábor wrote: > [...] > To my understanding both shells are right and conform to POSIX, > because the standard allows both behavior by stating the following s/behavior/behaviors/ > under '2.8.1 Consequences of Shell Errors': > > Reported-by: Max Kirillov > Signed-off-by: SZEDER Gábor ``` ## Carlo Arenas, 2019-01-02 18:31 Subject: Re: [PATCH] test-lib: check Bash version for '-x' without using shell arrays Message-ID: URL: https://gitlist.dev/e/CAPUEspiQyKnUWekn5iYQTP6F8CWW1rr4jSbuFUH%2BsS%3DiY9GfSg%40mail.gmail.com In-Reply-To: ``` Tested-By: Carlo Marcelo Arenas Belón ``` ## Jeff King, 2019-01-03 04:52 Subject: Re: [PATCH] test-lib: check Bash version for '-x' without using shell arrays Message-ID: <20190103045200.GB20047@sigill.intra.peff.net> URL: https://gitlist.dev/e/20190103045200.GB20047%40sigill.intra.peff.net In-Reply-To: ``` On Wed, Jan 02, 2019 at 01:20:47AM +0100, Johannes Sixt wrote: > > diff --git a/t/test-lib.sh b/t/test-lib.sh > > index 0f1faa24b2..f47a191e3b 100644 > > --- a/t/test-lib.sh > > +++ b/t/test-lib.sh > > @@ -324,9 +324,12 @@ do > > # isn't executed with a suitable Bash version. > > if test -z "$test_untraceable" || { > > test -n "$BASH_VERSION" && { > > - test ${BASH_VERSINFO[0]} -gt 4 || { > > - test ${BASH_VERSINFO[0]} -eq 4 && > > - test ${BASH_VERSINFO[1]} -ge 1 > > + bash_major=${BASH_VERSION%%.*} > > + bash_minor=${BASH_VERSION#*.} > > + bash_minor=${bash_minor%%.*} > > + test $bash_major -gt 4 || { > > + test $bash_major -eq 4 && > > + test $bash_minor -ge 1 > > } > > } > > } > > > > Would it perhaps be simpler to just hide the syntax behind eval? Like > > if test -z "$test_untraceable" || { > test -n "$BASH_VERSION" && eval ' > test ${BASH_VERSINFO[0]} -gt 4 || { > test ${BASH_VERSINFO[0]} -eq 4 && > test ${BASH_VERSINFO[1]} -ge 1 > } > ' That was my first thought, too. :) The parsing here is simple enough that I'd be fine either with the original patch, or an eval-based version (and otherwise, the goal and description seem quite good to me). -Peff ``` ## SZEDER Gábor, 2019-01-03 11:36 Subject: Re: [PATCH] test-lib: check Bash version for '-x' without using shell arrays Message-ID: <20190103113659.GA4673@szeder.dev> URL: https://gitlist.dev/e/20190103113659.GA4673%40szeder.dev In-Reply-To: <20190101231949.8184-1-szeder.dev@gmail.com> ``` On Wed, Jan 02, 2019 at 12:19:49AM +0100, SZEDER Gábor wrote: > To my understanding both shells are right and conform to POSIX, > because the standard allows both behavior by stating the following > under '2.8.1 Consequences of Shell Errors': > > "An expansion error is one that occurs when the shell expansions > define in wordexp are carried out (for example, "${x!y}", because That "define" above always stops my (non-native) English parser for a moment or two... shouldn't it be s/define/defined/ ? It's not a copy-paste error, POSIX does indeed write "define" there, see: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_08_01 (First paragraph below the table and the Notes.) > '!' is not a valid operator); an implementation may treat these as > syntax errors if it is able to detect them during tokenization, > rather than during expansion." > ``` ## SZEDER Gábor, 2019-01-03 11:43 Subject: [PATCH v2] test-lib: check Bash version for '-x' without using shell arrays Message-ID: <20190103114317.11523-1-szeder.dev@gmail.com> URL: https://gitlist.dev/e/20190103114317.11523-1-szeder.dev%40gmail.com In-Reply-To: ``` One of our test scripts, 't1510-repo-setup.sh' [1], still can't be reliably run with '-x' tracing enabled, unless it's executed with a Bash version supporting BASH_XTRACEFD (since v4.1). We have a lengthy condition to check the version of the shell running the test script, and disable tracing if it's not executed with a suitable Bash version [2]. This condition uses non-portable shell array accesses to easily get Bash's major and minor version number. This didn't seem to be problematic, because the simple commands expanding those array accesses are only executed when the test script is actually run with Bash. When run with Dash, the only shell I have at hand that doesn't support shell arrays, there are no issues, as it apparently skips right over the non-executed simple commands without noticing the non-supported constructs. Alas, it has been reported that NetBSD's /bin/sh does complain about them: ./test-lib.sh: 327: Syntax error: Bad substitution where line 327 contains the first ${BASH_VERSINFO[0]} array access. To my understanding both shells are right and conform to POSIX, because the standard allows both behaviors by stating the following under '2.8.1 Consequences of Shell Errors' [3]: "An expansion error is one that occurs when the shell expansions define in wordexp are carried out (for example, "${x!y}", because '!' is not a valid operator); an implementation may treat these as syntax errors if it is able to detect them during tokenization, rather than during expansion." Avoid this issue with NetBSD's /bin/sh (and potentially with other, less common shells) by hiding the shell array syntax behind 'eval' that is only executed with Bash. [1] 5827506928 (t1510-repo-setup: mark as untraceable with '-x', 2018-02-24) [2] 5fc98e79fc (t: add means to disable '-x' tracing for individual test scripts, 2018-02-24) [3] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_08_01 Reported-by: Max Kirillov Helped-by: Johannes Sixt Signed-off-by: SZEDER Gábor --- Changes since v1: - Hide the shell array syntax behind 'eval'. (I'm fine with both versions, take your pick.) - Corrected typo in commit message that Eric pointed out. - Added a link to the relevant section in POSIX. t/test-lib.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/t/test-lib.sh b/t/test-lib.sh index 0f1faa24b2..c34831a4de 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -323,12 +323,12 @@ do # this test is marked as such, and ignore '-x' if it # isn't executed with a suitable Bash version. if test -z "$test_untraceable" || { - test -n "$BASH_VERSION" && { + test -n "$BASH_VERSION" && eval ' test ${BASH_VERSINFO[0]} -gt 4 || { test ${BASH_VERSINFO[0]} -eq 4 && test ${BASH_VERSINFO[1]} -ge 1 } - } + ' } then trace=t -- 2.20.1.151.gec613c4b75 ``` ## Junio C Hamano, 2019-01-03 20:29 Subject: Re: [PATCH v2] test-lib: check Bash version for '-x' without using shell arrays Message-ID: URL: https://gitlist.dev/e/xmqqa7khh4cw.fsf%40gitster-ct.c.googlers.com In-Reply-To: <20190103114317.11523-1-szeder.dev@gmail.com> ``` SZEDER Gábor writes: > One of our test scripts, 't1510-repo-setup.sh' [1], still can't be > reliably run with '-x' tracing enabled, unless it's executed with a > Bash version supporting BASH_XTRACEFD (since v4.1). We have a lengthy > condition to check the version of the shell running the test script, > and disable tracing if it's not executed with a suitable Bash version > [2]. > > This condition uses non-portable shell array accesses to easily get > Bash's major and minor version number. This didn't seem to be > problematic, because the simple commands expanding those array > accesses are only executed when the test script is actually run with > Bash. When run with Dash, the only shell I have at hand that doesn't > support shell arrays, there are no issues, as it apparently skips > right over the non-executed simple commands without noticing the > non-supported constructs. > > Alas, it has been reported that NetBSD's /bin/sh does complain about > them: > > ./test-lib.sh: 327: Syntax error: Bad substitution > > where line 327 contains the first ${BASH_VERSINFO[0]} array access. > > To my understanding both shells are right and conform to POSIX, > because the standard allows both behaviors by stating the following > under '2.8.1 Consequences of Shell Errors' [3]: > > "An expansion error is one that occurs when the shell expansions > define in wordexp are carried out (for example, "${x!y}", because > '!' is not a valid operator); an implementation may treat these as > syntax errors if it is able to detect them during tokenization, > rather than during expansion." > > Avoid this issue with NetBSD's /bin/sh (and potentially with other, > less common shells) by hiding the shell array syntax behind 'eval' > that is only executed with Bash. > > [1] 5827506928 (t1510-repo-setup: mark as untraceable with '-x', > 2018-02-24) > [2] 5fc98e79fc (t: add means to disable '-x' tracing for individual > test scripts, 2018-02-24) > [3] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_08_01 > > Reported-by: Max Kirillov > Helped-by: Johannes Sixt > Signed-off-by: SZEDER Gábor > --- > > Changes since v1: > > - Hide the shell array syntax behind 'eval'. > (I'm fine with both versions, take your pick.) > - Corrected typo in commit message that Eric pointed out. > - Added a link to the relevant section in POSIX. Let's treat this as an independent and more urgent fix-up. I think it is sufficient to apply it to 2.20.x track, even though we could go back to 2.17.x and above. And then let's tentatively kick the "stress test" series out of 'pu', and have that series rebuilt on top of 'master' and this patch. Thanks. > > t/test-lib.sh | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/t/test-lib.sh b/t/test-lib.sh > index 0f1faa24b2..c34831a4de 100644 > --- a/t/test-lib.sh > +++ b/t/test-lib.sh > @@ -323,12 +323,12 @@ do > # this test is marked as such, and ignore '-x' if it > # isn't executed with a suitable Bash version. > if test -z "$test_untraceable" || { > - test -n "$BASH_VERSION" && { > + test -n "$BASH_VERSION" && eval ' > test ${BASH_VERSINFO[0]} -gt 4 || { > test ${BASH_VERSINFO[0]} -eq 4 && > test ${BASH_VERSINFO[1]} -ge 1 > } > - } > + ' > } > then > trace=t ``` ## SZEDER Gábor, 2019-01-04 09:30 Subject: Re: [PATCH v2] test-lib: check Bash version for '-x' without using shell arrays Message-ID: <20190104093015.GC4673@szeder.dev> URL: https://gitlist.dev/e/20190104093015.GC4673%40szeder.dev In-Reply-To: ``` On Thu, Jan 03, 2019 at 12:29:35PM -0800, Junio C Hamano wrote: > Let's treat this as an independent and more urgent fix-up. I think > it is sufficient to apply it to 2.20.x track, even though we could > go back to 2.17.x and above. > > And then let's tentatively kick the "stress test" series out of > 'pu', and have that series rebuilt on top of 'master' and this > patch. I rebased my '--stress' patch series on top of 'sg/test-bash-version-fix', and the result is the same as what's at the tip of 'sg/stress-test' at 1d1416a34b. ``` ## Junio C Hamano, 2019-01-04 11:25 Subject: Re: [PATCH v2] test-lib: check Bash version for '-x' without using shell arrays Message-ID: URL: https://gitlist.dev/e/xmqq5zv4fyuy.fsf%40gitster-ct.c.googlers.com In-Reply-To: <20190104093015.GC4673@szeder.dev> ``` SZEDER Gábor writes: > On Thu, Jan 03, 2019 at 12:29:35PM -0800, Junio C Hamano wrote: >> Let's treat this as an independent and more urgent fix-up. I think >> it is sufficient to apply it to 2.20.x track, even though we could >> go back to 2.17.x and above. >> >> And then let's tentatively kick the "stress test" series out of >> 'pu', and have that series rebuilt on top of 'master' and this >> patch. > > I rebased my '--stress' patch series on top of > 'sg/test-bash-version-fix', and the result is the same as what's at > the tip of 'sg/stress-test' at 1d1416a34b. Yeah, sorry for not reporting what I did after pushing the integration result out. I ended up rebasing the stress series on top of the updated fix with eval myself. Thanks. ``` ## SZEDER Gábor, 2019-01-04 12:38 Subject: Re: [PATCH v2] test-lib: check Bash version for '-x' without using shell arrays Message-ID: <20190104123819.GD4673@szeder.dev> URL: https://gitlist.dev/e/20190104123819.GD4673%40szeder.dev In-Reply-To: ``` On Fri, Jan 04, 2019 at 03:25:57AM -0800, Junio C Hamano wrote: > SZEDER Gábor writes: > > > On Thu, Jan 03, 2019 at 12:29:35PM -0800, Junio C Hamano wrote: > >> Let's treat this as an independent and more urgent fix-up. I think > >> it is sufficient to apply it to 2.20.x track, even though we could > >> go back to 2.17.x and above. > >> > >> And then let's tentatively kick the "stress test" series out of > >> 'pu', and have that series rebuilt on top of 'master' and this > >> patch. > > > > I rebased my '--stress' patch series on top of > > 'sg/test-bash-version-fix', and the result is the same as what's at > > the tip of 'sg/stress-test' at 1d1416a34b. > > Yeah, sorry for not reporting what I did after pushing the > integration result out. I ended up rebasing the stress series on > top of the updated fix with eval myself. No problem, it still saved me from writing a cover letter :) Note that I didn't actually test either of these patches on NetBSD, though both "Should Work". Carlo sent a Tested-by for v1; I hope he'll be able to get around and test v2 as well. ``` ## Carlo Arenas, 2019-01-04 18:42 Subject: Re: [PATCH v2] test-lib: check Bash version for '-x' without using shell arrays Message-ID: URL: https://gitlist.dev/e/CAPUEspga6Kjv0pSYiD-BmZUCTf0rd%3Dy0vneY_XvAkj_cCmXC8g%40mail.gmail.com In-Reply-To: <20190104123819.GD4673@szeder.dev> ``` v2 works fine, as expected Carlo ``` ## Junio C Hamano, 2019-01-04 20:04 Subject: Re: [PATCH v2] test-lib: check Bash version for '-x' without using shell arrays Message-ID: URL: https://gitlist.dev/e/xmqqy380b35m.fsf%40gitster-ct.c.googlers.com In-Reply-To: ``` Carlo Arenas writes: > v2 works fine, as expected Thanks. ```