git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2] parallel-checkout: avoid dash local bug in tests

From
René Scharfe <l.s.r@web.de>
Date
Jun 6, 2021, 01:01 UTC
Message-ID
<5ed77a50-3b34-4f3d-8348-a11d3930872b@web.de>
In-Reply-To
<87fsxw5bav.fsf@evledraar.gmail.com>
Am 05.06.21 um 21:56 schrieb Ævar Arnfjörð Bjarmason:
Show 49 quoted lines
>
> On Sat, Jun 05 2021, René Scharfe wrote:
>
>> The parallel checkout tests fail when run with /bin/dash on MacOS 11.4,
>> reporting the following error:
>>
>>    ./t2080-parallel-checkout-basics.sh: 33: local: 0: bad variable name
>>
>> That's because wc's output contains leading spaces and this version of
>> dash erroneously expands the variable declaration as "local workers= 0",
>> i.e. it tries to set the "workers" variable to the empty string and also
>> declare a variable named "0", which not a valid name.  This is a known
>> dash bug (https://bugs.launchpad.net/ubuntu/+source/dash/+bug/139097).
>>
>> Work around it by passing the command output directly to test instead of
>> storing it in a variable first.  While at it, let grep count the number
>> of lines instead of piping its output to wc, which is a bit shorter and
>> more efficient.
>>
>> Helped-by: Matheus Tavares Bernardino <matheus.bernardino@usp.br>
>> Signed-off-by: René Scharfe <l.s.r@web.de>
>> ---
>> Changes since v1:
>> - Explain the root cause.
>> - Get rid of the local variable "workers".
>> - Adjust title accordingly.
>> - Still use grep -c, though.
>> - Remove input redirection.
>>
>>  t/lib-parallel-checkout.sh | 3 +--
>>  1 file changed, 1 insertion(+), 2 deletions(-)
>>
>> diff --git a/t/lib-parallel-checkout.sh b/t/lib-parallel-checkout.sh
>> index 21f5759732..66350d5207 100644
>> --- a/t/lib-parallel-checkout.sh
>> +++ b/t/lib-parallel-checkout.sh
>> @@ -27,8 +27,7 @@ test_checkout_workers () {
>>  	rm -f "$trace_file" &&
>>  	GIT_TRACE2="$(pwd)/$trace_file" "$@" 2>&8 &&
>>
>> -	local workers=$(grep "child_start\[..*\] git checkout--worker" "$trace_file" | wc -l) &&
>> -	test $workers -eq $expected_workers &&
>> +	test $(grep -c "child_start\[..*\] git checkout--worker" "$trace_file") -eq $expected_workers &&
>>  	rm "$trace_file"
>>  } 8>&2 2>&4
>
> I'd find this thing much clearer if the v2 just narrowly focused on
> avoiding the "local", and thus demonstrated the non-portable shell
> issue,

I was not aiming for a minimal fix and I don't think the patch above is too complex, but you're right that at this point in the release cycle a duct-tape-style patch would be better.

Show 11 quoted lines
> and perhaps with something like:
>
> 	diff --git a/t/check-non-portable-shell.pl b/t/check-non-portable-shell.pl
> 	index fd3303552be..aad6f3e2bf1 100755
> 	--- a/t/check-non-portable-shell.pl
> 	+++ b/t/check-non-portable-shell.pl
> 	@@ -48,6 +48,7 @@ sub err {
> 	 	/\bexport\s+[A-Za-z0-9_]*=/ and err '"export FOO=bar" is not portable (use FOO=bar && export FOO)';
> 	 	/^\s*([A-Z0-9_]+=(\w*|(["']).*?\3)\s+)+(\w+)/ and exists($func{$4}) and
> 	 		err '"FOO=bar shell_func" assignment extends beyond "shell_func"';
> 	+	/\blocal\b \S+=\$\(.*?\|\s*\bwc -l\)/ and err 'whitespace handling in local=$(... | wc -l) differs in some dash versions';

Any command can output whitespace, it's not limited to wc -l. So a better rule might be to always quote command substitutions in local variable declarations (local foo="$(...)"). Or to disallow assignments with local altogether, like we already do for export, but that might be a bit much.

Show 9 quoted lines
> 	 	$line = '';
> 	 	# this resets our $. for each file
> 	 	close ARGV if eof;
>
>
> The let's do grep -c while we're at it part of this IMO just adds
> confusion while skimming future portability issues with --grep=dash or
> --grep=POSIX in the future, and looking at the history in v1 it's just
> there because in v1 the root cause wasn't fully understood.

True, I was still gripping the "use grep -c instead of grep | wc -l" hammer. I better rewatch https://www.youtube.com/watch?v=Yv4tI6939q0

Show 7 quoted lines
>
> If we're doing a general cleanup of that pattern it would seem to be
> better to search-replace this with the rest of them in another commit:
>
>     $ git grep '\$\(grep.*\| wc -l' -- t | wc -l
>     27
>
Previous: Matheus Tavares BernardinoNext: Junio C Hamano
Message 11 of 14 in “parallel-checkout: use grep -c to count workers in tests”
  1. parallel-checkout: use grep -c to count workers in testsRené Scharfe, Jun 5, 2021
  2. Matheus Tavares BernardinoJun 5, 2021
  3. René ScharfeJun 5, 2021
  4. René ScharfeJun 5, 2021
  5. parallel-checkout: avoid dash local bug in testsRené Scharfe, Jun 5, 2021
  6. SZEDER GáborJun 5, 2021
  7. René ScharfeJun 6, 2021
  8. Ævar Arnfjörð BjarmasonJun 5, 2021
  9. Matheus Tavares BernardinoJun 5, 2021
  10. Matheus Tavares BernardinoJun 5, 2021
  11. René ScharfeJun 6, 2021
  12. Junio C HamanoJun 6, 2021
  13. parallel-checkout: avoid dash local bug in testsRené Scharfe, Jun 6, 2021
  14. Junio C HamanoJun 6, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.