# [PATCH 0/3] openbsd: fixes for 2.27.0-RC0 11 messages from 2020-05-20 to 2020-05-20. Participants: Carlo Marcelo Arenas Belón, Junio C Hamano, Christopher Warrington (CHRISTOPHER). Thread: https://gitlist.dev/t/53507 ## Carlo Marcelo Arenas Belón, 2020-05-20 03:44 Subject: [PATCH 0/3] openbsd: fixes for 2.27.0-RC0 Message-ID: <20200520034444.47932-1-carenas@gmail.com> URL: https://gitlist.dev/e/20200520034444.47932-1-carenas%40gmail.com ``` First one is the only one exclusive to OpenBSD (tested with 6.7) while the other 2 are likely also needed in other POSIX systems. Carlo Marcelo Arenas Belón (3): t4216: avoid unnecessary subshell in test_bloom_filters_not_used bisect: remove CR characters from revision in replay t5520: avoid alternation in grep's BRE (not POSIX) git-bisect.sh | 5 +++-- t/t4216-log-bloom.sh | 2 +- t/t5520-pull.sh | 2 +- 3 files changed, 5 insertions(+), 4 deletions(-) -- 2.27.0.rc0.187.gede8c892b8 ``` ## Carlo Marcelo Arenas Belón, 2020-05-20 03:44 Subject: [PATCH 1/3] t4216: avoid unnecessary subshell in test_bloom_filters_not_used Message-ID: <20200520034444.47932-2-carenas@gmail.com> URL: https://gitlist.dev/e/20200520034444.47932-2-carenas%40gmail.com In-Reply-To: <20200520034444.47932-1-carenas@gmail.com> ``` Seems to trigger a bug in at least OpenBSD's 6.7 sh where it is interpreted as a history lookup and therefore fails 125-126, 128, 130. Remove the subshell and get a space between ! and grep, so tests pass successfully. Signed-off-by: Carlo Marcelo Arenas Belón --- t/t4216-log-bloom.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh index 21b68dd6c8..c855bcd3e7 100755 --- a/t/t4216-log-bloom.sh +++ b/t/t4216-log-bloom.sh @@ -70,7 +70,7 @@ test_bloom_filters_used () { test_bloom_filters_not_used () { log_args=$1 setup "$log_args" && - !(grep -q "statistics:{\"filter_not_present\":" "$TRASH_DIRECTORY/trace.perf") && + ! grep -q "statistics:{\"filter_not_present\":" "$TRASH_DIRECTORY/trace.perf" && test_cmp log_wo_bloom log_w_bloom } -- 2.27.0.rc0.187.gede8c892b8 ``` ## Carlo Marcelo Arenas Belón, 2020-05-20 03:44 Subject: [PATCH 2/3] bisect: remove CR characters from revision in replay Message-ID: <20200520034444.47932-3-carenas@gmail.com> URL: https://gitlist.dev/e/20200520034444.47932-3-carenas%40gmail.com In-Reply-To: <20200520034444.47932-1-carenas@gmail.com> ``` 6c722cbe5a (bisect: allow CRLF line endings in "git bisect replay" input, 2020-05-07) includes CR as a field separator, but doesn't account for it being included in the last field, breaking when running at least under OpenBSD 6.7's sh. Read the revision into a raw variable and strip it of any possible embeded CR characters, before use. Signed-off-by: Carlo Marcelo Arenas Belón --- git-bisect.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/git-bisect.sh b/git-bisect.sh index 71b367a944..8cd6095a71 100755 --- a/git-bisect.sh +++ b/git-bisect.sh @@ -210,14 +210,15 @@ bisect_replay () { test -r "$file" || die "$(eval_gettext "cannot read \$file for replaying")" git bisect--helper --bisect-reset || exit oIFS="$IFS" IFS="$IFS$(printf '\015')" - while read git bisect command rev + while read git bisect command rawrev do test "$git $bisect" = "git bisect" || test "$git" = "git-bisect" || continue if test "$git" = "git-bisect" then - rev="$command" + rawrev="$command" command="$bisect" fi + rev=$(echo $rawrev | tr -d '\015') get_terms git bisect--helper --check-and-set-terms "$command" "$TERM_GOOD" "$TERM_BAD" || exit get_terms -- 2.27.0.rc0.187.gede8c892b8 ``` ## Carlo Marcelo Arenas Belón, 2020-05-20 03:44 Subject: [PATCH 3/3] t5520: avoid alternation in grep's BRE (not POSIX) Message-ID: <20200520034444.47932-4-carenas@gmail.com> URL: https://gitlist.dev/e/20200520034444.47932-4-carenas%40gmail.com In-Reply-To: <20200520034444.47932-1-carenas@gmail.com> ``` Instead of using a BRE, that broke tests 30-32, 37-39, 42 at least with OpenBSD 6.7; use a simpler ERE. Fixes: d9f15d37f1 (pull: pass --autostash to merge, 2020-04-07) Signed-off-by: Carlo Marcelo Arenas Belón --- t/t5520-pull.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/t5520-pull.sh b/t/t5520-pull.sh index 37535d63a9..9fae07cdfa 100755 --- a/t/t5520-pull.sh +++ b/t/t5520-pull.sh @@ -28,7 +28,7 @@ test_pull_autostash_fail () { echo dirty >new_file && git add new_file && test_must_fail git pull "$@" . copy 2>err && - test_i18ngrep "\(uncommitted changes.\)\|\(overwritten by merge:\)" err + test_i18ngrep -E "uncommitted changes.|overwritten by merge:" err } test_expect_success setup ' -- 2.27.0.rc0.187.gede8c892b8 ``` ## Junio C Hamano, 2020-05-20 14:56 Subject: Re: [PATCH 2/3] bisect: remove CR characters from revision in replay Message-ID: URL: https://gitlist.dev/e/xmqqzha2wt7t.fsf%40gitster.c.googlers.com In-Reply-To: <20200520034444.47932-3-carenas@gmail.com> ``` Carlo Marcelo Arenas Belón writes: > 6c722cbe5a (bisect: allow CRLF line endings in "git bisect replay" > input, 2020-05-07) includes CR as a field separator, but doesn't > account for it being included in the last field, breaking when > running at least under OpenBSD 6.7's sh. > > Read the revision into a raw variable and strip it of any possible > embeded CR characters, before use. That's quite unsatisfactory, as the whole point of adding CR to IFS was to avoid having to spawn extra processes for this kind of text processing. If we were to do the preprocessing, we are better off just passing the whole input thru "tr -d '\015'". > oIFS="$IFS" IFS="$IFS$(printf '\015')" > - while read git bisect command rev > + while read git bisect command rawrev > do > test "$git $bisect" = "git bisect" || test "$git" = "git-bisect" || continue > if test "$git" = "git-bisect" > then > - rev="$command" > + rawrev="$command" > command="$bisect" > fi > + rev=$(echo $rawrev | tr -d '\015') As we know that "rev" ought to consist of just hexadecimal and cannot be split into two at $IFS even if we don't tell "read" that "everything at the end of line is 'rev'", can we do while read git bisect command rev ignored so that we'll get an empty string after CR in $ignored when reading CRLF input, and an empty string because we ran out of the tokens when reading LF input? That is, ... git-bisect.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/git-bisect.sh b/git-bisect.sh index 71b367a944..2a7599b486 100755 --- a/git-bisect.sh +++ b/git-bisect.sh @@ -210,7 +210,7 @@ bisect_replay () { test -r "$file" || die "$(eval_gettext "cannot read \$file for replaying")" git bisect--helper --bisect-reset || exit oIFS="$IFS" IFS="$IFS$(printf '\015')" - while read git bisect command rev + while read git bisect command rev ignored do test "$git $bisect" = "git bisect" || test "$git" = "git-bisect" || continue if test "$git" = "git-bisect" ``` ## Junio C Hamano, 2020-05-20 14:58 Subject: Re: [PATCH 3/3] t5520: avoid alternation in grep's BRE (not POSIX) Message-ID: URL: https://gitlist.dev/e/xmqqv9kqwt4k.fsf%40gitster.c.googlers.com In-Reply-To: <20200520034444.47932-4-carenas@gmail.com> ``` Carlo Marcelo Arenas Belón writes: > Instead of using a BRE, that broke tests 30-32, 37-39, 42 at least with > OpenBSD 6.7; use a simpler ERE. Thanks. We already do use "test_i18ngrep -E" in another test, so this should be safe. > Signed-off-by: Carlo Marcelo Arenas Belón > --- > t/t5520-pull.sh | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/t/t5520-pull.sh b/t/t5520-pull.sh > index 37535d63a9..9fae07cdfa 100755 > --- a/t/t5520-pull.sh > +++ b/t/t5520-pull.sh > @@ -28,7 +28,7 @@ test_pull_autostash_fail () { > echo dirty >new_file && > git add new_file && > test_must_fail git pull "$@" . copy 2>err && > - test_i18ngrep "\(uncommitted changes.\)\|\(overwritten by merge:\)" err > + test_i18ngrep -E "uncommitted changes.|overwritten by merge:" err > } > > test_expect_success setup ' ``` ## Junio C Hamano, 2020-05-20 15:04 Subject: Re: [PATCH 1/3] t4216: avoid unnecessary subshell in test_bloom_filters_not_used Message-ID: URL: https://gitlist.dev/e/xmqqr1vewsug.fsf%40gitster.c.googlers.com In-Reply-To: <20200520034444.47932-2-carenas@gmail.com> ``` Carlo Marcelo Arenas Belón writes: > Seems to trigger a bug in at least OpenBSD's 6.7 sh where it is > interpreted as a history lookup and therefore fails 125-126, 128, > 130. > > Remove the subshell and get a space between ! and grep, so tests > pass successfully. It's strange that somebody thinks of doing history lookup in non-interactive use. But even more curious is why we have this in a subshell in the first place. I do not see a reason why we need subshell, nor use of the double-quote in the outer layer that forces us to use backslashes. > test_bloom_filters_not_used () { > log_args=$1 > setup "$log_args" && > - !(grep -q "statistics:{\"filter_not_present\":" "$TRASH_DIRECTORY/trace.perf") && > + ! grep -q "statistics:{\"filter_not_present\":" "$TRASH_DIRECTORY/trace.perf" && This is obviously the minimum fix, so I'm willing to take the change as-is, but if we were writing it today, perhaps ! grep 'statistics:{"filter_not_present":' "$TRASH_DIRECTORY/trace.perf" && is how we write it. I do not see any reason why we want to use the "-q" option either. ``` ## Carlo Marcelo Arenas Belón, 2020-05-20 17:08 Subject: Re: [PATCH 2/3] bisect: remove CR characters from revision in replay Message-ID: <20200520170843.GC20332@Carlos-MBP> URL: https://gitlist.dev/e/20200520170843.GC20332%40Carlos-MBP In-Reply-To: ``` On Wed, May 20, 2020 at 07:56:22AM -0700, Junio C Hamano wrote: > As we know that "rev" ought to consist of just hexadecimal and > cannot be split into two at $IFS even if we don't tell "read" that > "everything at the end of line is 'rev'", can we do > > while read git bisect command rev ignored that works (kind of), but will cause test t6030.66 to fail, with a git segfault nonetheless, because we can't handle anymore a line line: git bisect start '--term-new' 'term1' '--term-old' 'term2' '32a594a3fdac2d57cf6d02987e30eec68511498c' '88bcdc1839f0ad191ffdd65cae2a2a862d682151' IMHO it will be probably still cleaner to do `tr -d '\015'`, even if the patch below avoids all current issues from the testsuite. will follow up with a fix for the segfault, unless someone else beats me to it. Carlo -- >8 -- Subject: [PATCH v2] bisect: avoid tailing CR characters from revision in replay 6c722cbe5a (bisect: allow CRLF line endings in "git bisect replay" input, 2020-05-07) includes CR as a field separator, but relies on it not being included in the last field, which breaks at least when running under OpenBSD 6.7's sh. Instead of just assume the CR will get swallowed, read the rest of the line into an otherwise unused variable and ignore it everywhere except on the call for git bisect start, where it matters. Signed-off-by: Carlo Marcelo Arenas Belón --- git-bisect.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/git-bisect.sh b/git-bisect.sh index 71b367a944..08a6ed57dd 100755 --- a/git-bisect.sh +++ b/git-bisect.sh @@ -210,7 +210,7 @@ bisect_replay () { test -r "$file" || die "$(eval_gettext "cannot read \$file for replaying")" git bisect--helper --bisect-reset || exit oIFS="$IFS" IFS="$IFS$(printf '\015')" - while read git bisect command rev + while read git bisect command rev tail do test "$git $bisect" = "git bisect" || test "$git" = "git-bisect" || continue if test "$git" = "git-bisect" @@ -223,7 +223,7 @@ bisect_replay () { get_terms case "$command" in start) - cmd="bisect_start $rev" + cmd="bisect_start $rev $tail" eval "$cmd" ;; "$TERM_GOOD"|"$TERM_BAD"|skip) git bisect--helper --bisect-write "$command" "$rev" "$TERM_GOOD" "$TERM_BAD" || exit;; -- 2.27.0.rc0.187.gede8c892b8 ``` ## Junio C Hamano, 2020-05-20 17:14 Subject: Re: [PATCH 2/3] bisect: remove CR characters from revision in replay Message-ID: URL: https://gitlist.dev/e/xmqqeerewmu6.fsf%40gitster.c.googlers.com In-Reply-To: <20200520170843.GC20332@Carlos-MBP> ``` Carlo Marcelo Arenas Belón writes: > that works (kind of), but will cause test t6030.66 to fail, with a git > segfault nonetheless, because we can't handle anymore a line line: > > git bisect start '--term-new' 'term1' '--term-old' 'term2' '32a594a3fdac2d57cf6d02987e30eec68511498c' '88bcdc1839f0ad191ffdd65cae2a2a862d682151' Yuck. Thanks for noticing. > will follow up with a fix for the segfault, unless someone else beats me to > it. Thanks. ``` ## Christopher Warrington (CHRISTOPHER), 2020-05-20 20:59 Subject: RE: [EXTERNAL] Re: [PATCH 2/3] bisect: remove CR characters from revision in replay Message-ID: URL: https://gitlist.dev/e/DM5PR00MB0439A347273E56C8697A587A9BB60%40DM5PR00MB0439.namprd00.prod.outlook.com In-Reply-To: <20200520170843.GC20332@Carlos-MBP> ``` On 2020-05-20 at 10:09-07:00, Carlo Marcelo Arenas Belón wrote: > IMHO it will be probably still cleaner to do `tr -d '\015'`, even if the > patch below avoids all current issues from the testsuite. My initial attempt to handle CRLF logs was shaped like this: tr -d '\r' <"$file" | while read ... This introduces a subshell, so there were concerns about propagating variables and exits. So, Peff also suggested preprocessing to a file. Around the same time Junio tried using IFS, and that was simpler. It would be pretty easy to tr -d '\r' >"$GIT_DIR/BISECT_REPLAY_LOG" (or some other name) and then read from that. Two things I'm not sure of with such an approach: 1. Is $GIT_DIR the right place to put this? If not, is there a helper to create a temporary file? mktemp may not be available everywhere. I only see git-mergetool.sh using it and only behind the mergetool.writeToTemp variable. 2. How and when does this temp file need to be cleaned up? Looking at using something like trap "rm -f \"$temp_file\"" 0 will conflict with the traps that git-bisect.sh's bisect_start installs, and bisect_replay calls bisect_start. I could introduce a helper like maybe_unlink_temp_replay_log and invoke that from a trap installed in both bisect_replay and bisect_start. I looked at whether bisect.c's bisect_clean_state() is the right place to add such clean up: it does not look like the right place. It would result in the temp file getting deleted while it was being read. (git-bisect.sh -> bisect_replay()'s loop -> bisect_start() -> bisect--helper.c's bisect_start() -> bisect.c's bisect_clean_state()) Alas, not all file systems are OK with that happening. Any guidance? Here's a sketch that leaves cleanup on non-successful paths unaddressed. bisect_replay () { file="$1" test "$#" -eq 1 || die "$(gettext "No logfile given")" test -r "$file" || die "$(eval_gettext "cannot read \$file for replaying")" git bisect--helper --bisect-reset || exit scrubbed_file="$GIT_DIR/BISECT_CLEANED_LOG" tr -d '\r' <"$file" >"scrubbed_file" || die "badness" while read git bisect command rev do ... done <"$scrubbed_file" rm -f "$scrubbed_file" bisect_auto_next -- Christopher Warrington Microsoft Corp. ``` ## Junio C Hamano, 2020-05-20 21:14 Subject: Re: [EXTERNAL] Re: [PATCH 2/3] bisect: remove CR characters from revision in replay Message-ID: URL: https://gitlist.dev/e/xmqq1rneux55.fsf%40gitster.c.googlers.com In-Reply-To: ``` "Christopher Warrington (CHRISTOPHER)" writes: > On 2020-05-20 at 10:09-07:00, Carlo Marcelo Arenas Belón wrote: > >> IMHO it will be probably still cleaner to do `tr -d '\015'`, even if the >> patch below avoids all current issues from the testsuite. > > My initial attempt to handle CRLF logs was shaped like this: > > tr -d '\r' <"$file" | while read ... > > This introduces a subshell, so there were concerns about propagating > variables and exits. So, Peff also suggested preprocessing to a file. Around > the same time Junio tried using IFS, and that was simpler. One thing was that extra processes and temporary files are pure overhead when people don't use misbehaving editors, so we could just say "don't do it then". It may not be worth paying the cost of being pessimistic and preparing for the worst. The "$IFS will split the tokens on the line for free for us" was a no-cost solution and that was why I suggested it. In any case, I think there are folks who are rewriting bisect piece by piece to C, and at that point it would just be the matter of using strbuf_getline_lf() vs strbuf_getline() to get rid of the unwanted CR for free. ```