threads / bug / 52111

[BUG] git 2.24.0-rc1 t0500 on NonStop in Jenkins

Subject: [BUG] git 2.24.0-rc1 t0500 on NonStop in Jenkins

## tl;dr

8 messages between Oct 24, 2019 and Nov 1, 2019.

replies: 7people: 3as markdown or json

Randall S. Becker· Oct 24, 2019, 22:01 UTC · lore

Hi All,

We are running the regression suite (as we always have) in Jenkins and are finding that t0500 is basically a complete failure. I cannot recreate it using bash where we have a real terminal, but without one, the t0500 consistently fails. I wonder whether any other platform is experiencing the same situation.

We also find that we cannot run the main test suite as a background job - we get segmentation faults that I have yet to isolate, but when run in the foreground, no problem.

I wanted to get this out there because 2.24.0 goes too far as this did not happen in 2.23.0 or any release prior.

Sincerely, Randall

-- Brief whoami: NonStop developer since approximately 211288444200000000 UNIX developer since approximately 421664400 -- In my real life, I talk too much.

Johannes Schindelin· Oct 28, 2019, 13:07 UTC · re: Randall S. Becker · lore

Re: [BUG] git 2.24.0-rc1 t0500 on NonStop in Jenkins

Hi Randall,

On Thu, 24 Oct 2019, Randall S. Becker wrote:

Show 5 quoted lines

> We are running the regression suite (as we always have) in Jenkins and are
> finding that t0500 is basically a complete failure. I cannot recreate it
> using bash where we have a real terminal, but without one, the t0500
> consistently fails. I wonder whether any other platform is experiencing the
> same situation.

This says that a test failed, but it lacks the verbose log, so there is little else left to the willing reader than wild guesses that are almost certainly completely missing the culprit.

In other words: verbose log, please. That is, the output of the test script when run with `-i -v -x` (or `-i -V -x` in which case you will want to analyze the contents of `t/test-results/t0500-progress-display.out` and if that does not help, paste it into a reply).

Ciao, Johannes

Show 19 quoted lines

>
> We also find that we cannot run the main test suite as a background job - we
> get segmentation faults that I have yet to isolate, but when run in the
> foreground, no problem.
>
> I wanted to get this out there because 2.24.0 goes too far as this did not
> happen in 2.23.0 or any release prior.
>
> Sincerely,
> Randall
>
> -- Brief whoami:
> NonStop developer since approximately 211288444200000000
> UNIX developer since approximately 421664400
> -- In my real life, I talk too much.
>
>
>
>

Randall S. Becker· Oct 28, 2019, 14:15 UTC · re: Johannes Schindelin · lore

RE: [BUG] git 2.24.0-rc1 t0500 on NonStop in Jenkins

On October 28, 2019 9:07 AM, Johannes Schindelin wrote:

Show 8 quoted lines

> On Thu, 24 Oct 2019, Randall S. Becker wrote:
> > We are running the regression suite (as we always have) in Jenkins and
> > are finding that t0500 is basically a complete failure. I cannot
> > recreate it using bash where we have a real terminal, but without one,
> > the t0500 consistently fails. I wonder whether any other platform is
> > experiencing the same situation.
> 
> This says that a test failed, but it lacks the verbose log, so there is

little else

> left to the willing reader than wild guesses that are almost certainly
> completely missing the culprit.
> 
> In other words: verbose log, please. That is, the output of the test

script

> when run with `-i -v -x` (or `-i -V -x` in which case you will want to

analyze the

> contents of `t/test-results/t0500-progress-display.out` and if that does

not

> help, paste it into a reply).

Here is the verbose output from subtest 2 that failed. Sorry it took so long, we had to arrange a different Jenkins job to test this in isolation. Format issue?

Cheers, Randall

expecting success of 0500.2 'progress display with total': 
	cat >expect <<-\EOF &&
	Working hard:  33% (1/3)<CR>
	Working hard:  66% (2/3)<CR>
	Working hard: 100% (3/3)<CR>
	Working hard: 100% (3/3), done.
	EOF

	cat >in <<-\EOF &&
	progress 1
	progress 2
	progress 3
	EOF
	test-tool progress --total=3 "Working hard" <in 2>stderr &&

	show_cr <stderr >out &&
	test_i18ncmp expect out

+ cat
+ 1> expect 0< /tmp/sh1811939370.4
+ cat
+ 1> in 0< /tmp/sh1811939370.5
+ test-tool progress --total=3 Working hard
+ 0< in 2> stderr
+ show_cr
+ 0< stderr 1> out
+ test_i18ncmp expect out
--- expect	2019-10-28 14:11:40 +0000
+++ out	2019-10-28 14:11:41 +0000

@@ -1,4 +1,2 @@
-Working hard:  33% (1/3)<CR>
-Working hard:  66% (2/3)<CR>
-Working hard: 100% (3/3)<CR>
-Working hard: 100% (3/3), done.
+Working hard:   0% (1/12884901888)<CR>
+Working hard:   0% (3/12884901888), done.
error: last command exited with $?=1
not ok 2 - progress display with total
#	
#		cat >expect <<-\EOF &&
#		Working hard:  33% (1/3)<CR>
#		Working hard:  66% (2/3)<CR>
#		Working hard: 100% (3/3)<CR>
#		Working hard: 100% (3/3), done.
#		EOF
#	
#		cat >in <<-\EOF &&
#		progress 1
#		progress 2
#		progress 3
#		EOF
#		test-tool progress --total=3 "Working hard" <in 2>stderr &&
#	
#		show_cr <stderr >out &&
#		test_i18ncmp expect out
#

SZEDER Gábor· Oct 28, 2019, 14:52 UTC · re: Randall S. Becker · lore

Re: [BUG] git 2.24.0-rc1 t0500 on NonStop in Jenkins

On Mon, Oct 28, 2019 at 10:15:31AM -0400, Randall S. Becker wrote:

Show 27 quoted lines

> Here is the verbose output from subtest 2 that failed. Sorry it took so
> long, we had to arrange a different Jenkins job to test this in isolation.
> Format issue?
> 
> Cheers,
> Randall
> 
> expecting success of 0500.2 'progress display with total': 
> 	cat >expect <<-\EOF &&
> 	Working hard:  33% (1/3)<CR>
> 	Working hard:  66% (2/3)<CR>
> 	Working hard: 100% (3/3)<CR>
> 	Working hard: 100% (3/3), done.
> 	EOF
> 
> 	cat >in <<-\EOF &&
> 	progress 1
> 	progress 2
> 	progress 3
> 	EOF
> 	test-tool progress --total=3 "Working hard" <in 2>stderr &&
> 
> 	show_cr <stderr >out &&
> 	test_i18ncmp expect out
> 
> + cat
> + 1> expect 0< /tmp/sh1811939370.4

Wow :)

Show 6 quoted lines

> + cat
> + 1> in 0< /tmp/sh1811939370.5
> + test-tool progress --total=3 Working hard
> + 0< in 2> stderr
> + show_cr
> + 0< stderr 1> out

Another wow.

This is unrelated to the test failure, but makes me rather curious: you run the test script with '-x', that's why we see the trace of commands executed during the test. Interestingly, though, we don't see any trace of commands executed within any shell function invoked in the test, i.e. from show_cr above and test_i18ncmp below.

What kind of shell is this?

Show 10 quoted lines

> + test_i18ncmp expect out
> --- expect	2019-10-28 14:11:40 +0000
> +++ out	2019-10-28 14:11:41 +0000
> @@ -1,4 +1,2 @@
> -Working hard:  33% (1/3)<CR>
> -Working hard:  66% (2/3)<CR>
> -Working hard: 100% (3/3)<CR>
> -Working hard: 100% (3/3), done.
> +Working hard:   0% (1/12884901888)<CR>
> +Working hard:   0% (3/12884901888), done.

Weird, this looks exactly like the big-endian test failure that was fixed in 2b6f6ea1bd (test-progress: fix test failures on big-endian systems, 2019-10-20), but that is already in 2.24.0-rc1 (but not yet in -rc0).

https://public-inbox.org/git/f1ce445e-6954-8e7b-2dca-3a566ce689a5@physik.fu-berlin.de/

Is NonStop big or little-endian?  Does t0500 pass without 2b6f6ea1bd?

Show 21 quoted lines

> error: last command exited with $?=1
> not ok 2 - progress display with total
> #	
> #		cat >expect <<-\EOF &&
> #		Working hard:  33% (1/3)<CR>
> #		Working hard:  66% (2/3)<CR>
> #		Working hard: 100% (3/3)<CR>
> #		Working hard: 100% (3/3), done.
> #		EOF
> #	
> #		cat >in <<-\EOF &&
> #		progress 1
> #		progress 2
> #		progress 3
> #		EOF
> #		test-tool progress --total=3 "Working hard" <in 2>stderr &&
> #	
> #		show_cr <stderr >out &&
> #		test_i18ncmp expect out
> #	
>

Randall S. Becker· Oct 28, 2019, 15:20 UTC · re: SZEDER Gábor · lore

RE: [BUG] git 2.24.0-rc1 t0500 on NonStop in Jenkins

On October 28, 2019 10:52 AM, SZEDER Gábor wrote:

Show 45 quoted lines

> On Mon, Oct 28, 2019 at 10:15:31AM -0400, Randall S. Becker wrote:
> > Here is the verbose output from subtest 2 that failed. Sorry it took
> > so long, we had to arrange a different Jenkins job to test this in isolation.
> > Format issue?
> >
> > Cheers,
> > Randall
> >
> > expecting success of 0500.2 'progress display with total':
> > 	cat >expect <<-\EOF &&
> > 	Working hard:  33% (1/3)<CR>
> > 	Working hard:  66% (2/3)<CR>
> > 	Working hard: 100% (3/3)<CR>
> > 	Working hard: 100% (3/3), done.
> > 	EOF
> >
> > 	cat >in <<-\EOF &&
> > 	progress 1
> > 	progress 2
> > 	progress 3
> > 	EOF
> > 	test-tool progress --total=3 "Working hard" <in 2>stderr &&
> >
> > 	show_cr <stderr >out &&
> > 	test_i18ncmp expect out
> >
> > + cat
> > + 1> expect 0< /tmp/sh1811939370.4
> 
> Wow :)
> 
> > + cat
> > + 1> in 0< /tmp/sh1811939370.5
> > + test-tool progress --total=3 Working hard 0< in 2> stderr show_cr 0<
> > + stderr 1> out
> 
> Another wow.
> 
> This is unrelated to the test failure, but makes me rather curious:
> you run the test script with '-x', that's why we see the trace of commands
> executed during the test.  Interestingly, though, we don't see any trace of
> commands executed within any shell function invoked in the test, i.e. from
> show_cr above and test_i18ncmp below.
> 
> What kind of shell is this?

The test output requested by Johannes was to use -i -v -x. We use ksh and have for years to test git. The command used was done within a Jenkins environment and was:

cd t && sh t0500-progress-display.sh -i -v -x

Show 20 quoted lines

> 
> > + test_i18ncmp expect out
> > --- expect	2019-10-28 14:11:40 +0000
> > +++ out	2019-10-28 14:11:41 +0000
> > @@ -1,4 +1,2 @@
> > -Working hard:  33% (1/3)<CR>
> > -Working hard:  66% (2/3)<CR>
> > -Working hard: 100% (3/3)<CR>
> > -Working hard: 100% (3/3), done.
> > +Working hard:   0% (1/12884901888)<CR>
> > +Working hard:   0% (3/12884901888), done.
> 
> Weird, this looks exactly like the big-endian test failure that was fixed in
> 2b6f6ea1bd (test-progress: fix test failures on big-endian systems, 2019-10-
> 20), but that is already in 2.24.0-rc1 (but not yet in -rc0).
> 
> https://public-inbox.org/git/f1ce445e-6954-8e7b-2dca-
> 3a566ce689a5@physik.fu-berlin.de/
> 
> Is NonStop big or little-endian?  Does t0500 pass without 2b6f6ea1bd?

NonStop is big-endian. When t0500 is run from an interactive terminal, it passes. This failure seems to be a result of being in a disconnected terminal situation typical of Jenkins, not that the test result makes any sense with that being the only difference. t0500 did not exist in 2.23.0, our last build, so I can't easily get that answer. Our Jenkins is based off the master branch, so it's a hard to revert in our pipeline without a serious amount of work - that and without 2b6f6ea1bd, other things break if I remember from August.

Does the printf format use positional arguments (%digit$)? That has known issues on the platform.

FYI: int/long are 32 bits, long long is 64 bits. 12884901888 is 0x300000000, surprisingly.

Show 21 quoted lines

> > error: last command exited with $?=1
> > not ok 2 - progress display with total
> > #
> > #		cat >expect <<-\EOF &&
> > #		Working hard:  33% (1/3)<CR>
> > #		Working hard:  66% (2/3)<CR>
> > #		Working hard: 100% (3/3)<CR>
> > #		Working hard: 100% (3/3), done.
> > #		EOF
> > #
> > #		cat >in <<-\EOF &&
> > #		progress 1
> > #		progress 2
> > #		progress 3
> > #		EOF
> > #		test-tool progress --total=3 "Working hard" <in 2>stderr &&
> > #
> > #		show_cr <stderr >out &&
> > #		test_i18ncmp expect out
> > #
> >

SZEDER Gábor· Oct 29, 2019, 10:16 UTC · re: Randall S. Becker · lore

Re: [BUG] git 2.24.0-rc1 t0500 on NonStop in Jenkins

On Mon, Oct 28, 2019 at 11:20:48AM -0400, Randall S. Becker wrote:

Show 25 quoted lines

> > > + test_i18ncmp expect out
> > > --- expect	2019-10-28 14:11:40 +0000
> > > +++ out	2019-10-28 14:11:41 +0000
> > > @@ -1,4 +1,2 @@
> > > -Working hard:  33% (1/3)<CR>
> > > -Working hard:  66% (2/3)<CR>
> > > -Working hard: 100% (3/3)<CR>
> > > -Working hard: 100% (3/3), done.
> > > +Working hard:   0% (1/12884901888)<CR>
> > > +Working hard:   0% (3/12884901888), done.
> > 
> > Weird, this looks exactly like the big-endian test failure that was fixed in
> > 2b6f6ea1bd (test-progress: fix test failures on big-endian systems, 2019-10-
> > 20), but that is already in 2.24.0-rc1 (but not yet in -rc0).
> > 
> > https://public-inbox.org/git/f1ce445e-6954-8e7b-2dca-
> > 3a566ce689a5@physik.fu-berlin.de/
> > 
> > Is NonStop big or little-endian?  Does t0500 pass without 2b6f6ea1bd?
> 
> NonStop is big-endian. When t0500 is run from an interactive
> terminal, it passes. This failure seems to be a result of being in a
> disconnected terminal situation typical of Jenkins, not that the
> test result makes any sense with that being the only difference.
> t0500 did not exist in 2.23.0, our last build,

Didn't you build v2.24.0-rc0?

Show 8 quoted lines

> so I can't easily get
> that answer. Our Jenkins is based off the master branch, so it's a
> hard to revert in our pipeline without a serious amount of work -
> that and without 2b6f6ea1bd, other things break if I remember from
> August.
> 
> Does the printf format use positional arguments (%digit$)? That has
> known issues on the platform.

The output in question is formatted in display() in progress.c:124:

        strbuf_addf(counters_sb,
                    "%3u%% (%"PRIuMAX"/%"PRIuMAX")%s", percent,
                    (uintmax_t)n, (uintmax_t)progress->total,
                    tp);

where both 'n' and 'progress->total' are uint64_t, and 'PRIuMAX' is 'llu'. All of these are widely used throughout the code base.

> FYI: int/long are 32 bits, long long is 64 bits. 12884901888 is
> 0x300000000, surprisingly.

It's not surprising, basically that's what 2b6f6ea1bd is all about.

Show 22 quoted lines

> > > error: last command exited with $?=1
> > > not ok 2 - progress display with total
> > > #
> > > #		cat >expect <<-\EOF &&
> > > #		Working hard:  33% (1/3)<CR>
> > > #		Working hard:  66% (2/3)<CR>
> > > #		Working hard: 100% (3/3)<CR>
> > > #		Working hard: 100% (3/3), done.
> > > #		EOF
> > > #
> > > #		cat >in <<-\EOF &&
> > > #		progress 1
> > > #		progress 2
> > > #		progress 3
> > > #		EOF
> > > #		test-tool progress --total=3 "Working hard" <in 2>stderr &&
> > > #
> > > #		show_cr <stderr >out &&
> > > #		test_i18ncmp expect out
> > > #
> > >
>

Randall S. Becker· Oct 29, 2019, 14:56 UTC · re: SZEDER Gábor · lore

RE: [BUG] git 2.24.0-rc1 t0500 on NonStop in Jenkins

On October 29, 2019 6:16 AM, SZEDER Gábor wrote:

Show 29 quoted lines

> On Mon, Oct 28, 2019 at 11:20:48AM -0400, Randall S. Becker wrote:
> > > > + test_i18ncmp expect out
> > > > --- expect	2019-10-28 14:11:40 +0000
> > > > +++ out	2019-10-28 14:11:41 +0000
> > > > @@ -1,4 +1,2 @@
> > > > -Working hard:  33% (1/3)<CR>
> > > > -Working hard:  66% (2/3)<CR>
> > > > -Working hard: 100% (3/3)<CR>
> > > > -Working hard: 100% (3/3), done.
> > > > +Working hard:   0% (1/12884901888)<CR>
> > > > +Working hard:   0% (3/12884901888), done.
> > >
> > > Weird, this looks exactly like the big-endian test failure that was
> > > fixed in 2b6f6ea1bd (test-progress: fix test failures on big-endian
> > > systems, 2019-10- 20), but that is already in 2.24.0-rc1 (but not yet in -
> rc0).
> > >
> > > https://public-inbox.org/git/f1ce445e-6954-8e7b-2dca-
> > > 3a566ce689a5@physik.fu-berlin.de/
> > >
> > > Is NonStop big or little-endian?  Does t0500 pass without 2b6f6ea1bd?
> >
> > NonStop is big-endian. When t0500 is run from an interactive terminal,
> > it passes. This failure seems to be a result of being in a
> > disconnected terminal situation typical of Jenkins, not that the test
> > result makes any sense with that being the only difference.
> > t0500 did not exist in 2.23.0, our last build,
> 
> Didn't you build v2.24.0-rc0?

Reporter bonehead moment. Ok, the problem is in v2.24.0-rc0, *NOT* rc1. I am deeply sorry for the confusion. I ran both build test cycles and t0500 in rc1 worked. We are running the full rc1 cycle now.

Show 42 quoted lines

> > so I can't easily get
> > that answer. Our Jenkins is based off the master branch, so it's a
> > hard to revert in our pipeline without a serious amount of work - that
> > and without 2b6f6ea1bd, other things break if I remember from August.
> >
> > Does the printf format use positional arguments (%digit$)? That has
> > known issues on the platform.
> 
> The output in question is formatted in display() in progress.c:124:
> 
>         strbuf_addf(counters_sb,
>                     "%3u%% (%"PRIuMAX"/%"PRIuMAX")%s", percent,
>                     (uintmax_t)n, (uintmax_t)progress->total,
>                     tp);
> 
> where both 'n' and 'progress->total' are uint64_t, and 'PRIuMAX' is 'llu'.  All
> of these are widely used throughout the code base.
> 
> > FYI: int/long are 32 bits, long long is 64 bits. 12884901888 is
> > 0x300000000, surprisingly.
> 
> It's not surprising, basically that's what 2b6f6ea1bd is all about.
> 
> > > > error: last command exited with $?=1 not ok 2 - progress display
> > > > with total #
> > > > #		cat >expect <<-\EOF &&
> > > > #		Working hard:  33% (1/3)<CR>
> > > > #		Working hard:  66% (2/3)<CR>
> > > > #		Working hard: 100% (3/3)<CR>
> > > > #		Working hard: 100% (3/3), done.
> > > > #		EOF
> > > > #
> > > > #		cat >in <<-\EOF &&
> > > > #		progress 1
> > > > #		progress 2
> > > > #		progress 3
> > > > #		EOF
> > > > #		test-tool progress --total=3 "Working hard" <in 2>stderr &&
> > > > #
> > > > #		show_cr <stderr >out &&
> > > > #		test_i18ncmp expect out
> > > > #

Randall S. Becker· Nov 1, 2019, 19:29 UTC · re: SZEDER Gábor · lore

RE: [BUG] git 2.24.0-rc1 t0500 on NonStop in Jenkins

FYI: git 2.24.0-rc2 passes on the NonStop TNS/E platform - except for the usual t9001 and t9020 that never do and can't.

Thanks to everyone for all the hard work! Randall

-- Brief whoami:
 NonStop developer since approximately 211288444200000000
 UNIX developer since approximately 421664400
-- In my real life, I talk too much.