threads / discuss / 52338

Travis OS X gcc job is broken (again)

Subject: Travis OS X gcc job is broken (again)

## tl;dr

6 messages between Nov 26, 2019 and Nov 28, 2019.

replies: 5people: 3as markdown or json

Denton Liu· Nov 26, 2019, 01:15 UTC · lore
Hi all,

I couldn't find any threads addressing this yet (which is surprising since it's been happening for about half a week now).

It seems like gcc-8 doesn't exist anymore in our gcc OS X Travis job. As a result, that job has been failing consistently recently[1].

An old similar thread here[2]. Not sure if the failure is related.
Thanks,
Denton

[1]: https://travis-ci.org/git/git/builds/616541763 [2]: https://lore.kernel.org/git/9d80e845bf923c4543c49f9947aacb10c59ff6ce.1571789978.git.gitgitgadget@gmail.com/

Doan Tran Cong Danh· Nov 27, 2019, 16:15 UTC · re: Denton Liu · lore

[PATCH] ci: detect installed gcc in Travis CI

Travis CI has continously updated their images, including updating gcc installation.

Save us some headache by checking which version of gcc is installed in the image, and use said version to run the build.

While gcc-10 hasn't been released, yet, add it to the list to save us some headache in the future.

Signed-off-by: Doan Tran Cong Danh <congdanhqx@gmail.com>
---
 ci/lib.sh | 24 ++++++++++++++----------
 1 file changed, 14 insertions(+), 10 deletions(-)
diff --git a/ci/lib.sh b/ci/lib.sh
index c8c2c38155..4040fc1a22 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -159,12 +159,21 @@ export DEFAULT_TEST_TARGET=prove
 export GIT_TEST_CLONE_2GB=YesPlease
 
 case "$jobname" in
-linux-clang|linux-gcc)
-	if [ "$jobname" = linux-gcc ]
-	then
-		export CC=gcc-8
-	fi
+*-gcc)
+	for gitcc in gcc-10 gcc-9 gcc-8
+	do
+		if command -v $gitcc >/dev/null 2>&1
+		then
+			export CC=$gitcc
+			break;
+		fi
+	done
+	unset gitcc
+	;;
+esac
 
+case "$jobname" in
+linux-clang|linux-gcc)
 	export GIT_TEST_HTTPD=true
 
 	# The Linux build installs the defined dependency versions below.
@@ -180,11 +189,6 @@ linux-clang|linux-gcc)
 	export PATH="$GIT_LFS_PATH:$P4_PATH:$PATH"
 	;;
 osx-clang|osx-gcc)
-	if [ "$jobname" = osx-gcc ]
-	then
-		export CC=gcc-8
-	fi
-
 	# t9810 occasionally fails on Travis CI OS X
 	# t9816 occasionally fails with "TAP out of sequence errors" on
 	# Travis CI OS X
-- 
2.24.0.161.g2616cd003f.dirty
SZEDER Gábor· Nov 27, 2019, 16:48 UTC · re: Doan Tran Cong Danh · lore

Re: [PATCH] ci: detect installed gcc in Travis CI

On Wed, Nov 27, 2019 at 11:15:55PM +0700, Doan Tran Cong Danh wrote:
Show 36 quoted lines
> Travis CI has continously updated their images, including updating gcc
> installation.
> 
> Save us some headache by checking which version of gcc is installed in
> the image, and use said version to run the build.
> 
> While gcc-10 hasn't been released, yet, add it to the list to save us
> some headache in the future.
> 
> Signed-off-by: Doan Tran Cong Danh <congdanhqx@gmail.com>
> ---
>  ci/lib.sh | 24 ++++++++++++++----------
>  1 file changed, 14 insertions(+), 10 deletions(-)
> 
> diff --git a/ci/lib.sh b/ci/lib.sh
> index c8c2c38155..4040fc1a22 100755
> --- a/ci/lib.sh
> +++ b/ci/lib.sh
> @@ -159,12 +159,21 @@ export DEFAULT_TEST_TARGET=prove
>  export GIT_TEST_CLONE_2GB=YesPlease
>  
>  case "$jobname" in
> -linux-clang|linux-gcc)
> -	if [ "$jobname" = linux-gcc ]
> -	then
> -		export CC=gcc-8
> -	fi
> +*-gcc)
> +	for gitcc in gcc-10 gcc-9 gcc-8
> +	do
> +		if command -v $gitcc >/dev/null 2>&1
> +		then
> +			export CC=$gitcc
> +			break;
> +		fi
> +	done

This assummes that some of these gcc-<N> binaries can be found in PATH, but that hasn't always been the case in the past: up until end of last week Travis CI's xcode10.1 image came with GCC 8 pre-installed but not linked and without GCC 9, i.e. neither 'gcc-8' nor 'gcc-9' was available in PATH. While now that image does have GCC 9 properly installed, i.e. 'gcc-9' has been available in PATH for almost a week, I wouldn't want to rely on that...

With your changes if none of the listed gcc-<N> binaries were available in PATH, then we would silently build with the generic 'cc', which is 'clang' on OSX, and that's definitely not what we want.

Show 24 quoted lines
> +	unset gitcc
> +	;;
> +esac
>  
> +case "$jobname" in
> +linux-clang|linux-gcc)
>  	export GIT_TEST_HTTPD=true
>  
>  	# The Linux build installs the defined dependency versions below.
> @@ -180,11 +189,6 @@ linux-clang|linux-gcc)
>  	export PATH="$GIT_LFS_PATH:$P4_PATH:$PATH"
>  	;;
>  osx-clang|osx-gcc)
> -	if [ "$jobname" = osx-gcc ]
> -	then
> -		export CC=gcc-8
> -	fi
> -
>  	# t9810 occasionally fails on Travis CI OS X
>  	# t9816 occasionally fails with "TAP out of sequence errors" on
>  	# Travis CI OS X
> -- 
> 2.24.0.161.g2616cd003f.dirty
> 
Danh Doan· Nov 28, 2019, 12:03 UTC · re: SZEDER Gábor · lore

Re: [PATCH] ci: detect installed gcc in Travis CI

On 2019-11-27 17:48:20+0100, SZEDER Gábor <szeder.dev@gmail.com> wrote:
Show 45 quoted lines
> On Wed, Nov 27, 2019 at 11:15:55PM +0700, Doan Tran Cong Danh wrote:
> > Travis CI has continously updated their images, including updating gcc
> > installation.
> > 
> > Save us some headache by checking which version of gcc is installed in
> > the image, and use said version to run the build.
> > 
> > While gcc-10 hasn't been released, yet, add it to the list to save us
> > some headache in the future.
> > 
> > Signed-off-by: Doan Tran Cong Danh <congdanhqx@gmail.com>
> > ---
> >  ci/lib.sh | 24 ++++++++++++++----------
> >  1 file changed, 14 insertions(+), 10 deletions(-)
> > 
> > diff --git a/ci/lib.sh b/ci/lib.sh
> > index c8c2c38155..4040fc1a22 100755
> > --- a/ci/lib.sh
> > +++ b/ci/lib.sh
> > @@ -159,12 +159,21 @@ export DEFAULT_TEST_TARGET=prove
> >  export GIT_TEST_CLONE_2GB=YesPlease
> >  
> >  case "$jobname" in
> > -linux-clang|linux-gcc)
> > -	if [ "$jobname" = linux-gcc ]
> > -	then
> > -		export CC=gcc-8
> > -	fi
> > +*-gcc)
> > +	for gitcc in gcc-10 gcc-9 gcc-8
> > +	do
> > +		if command -v $gitcc >/dev/null 2>&1
> > +		then
> > +			export CC=$gitcc
> > +			break;
> > +		fi
> > +	done
> 
> This assummes that some of these gcc-<N> binaries can be found in
> PATH, but that hasn't always been the case in the past: up until end
> of last week Travis CI's xcode10.1 image came with GCC 8 pre-installed
> but not linked and without GCC 9, i.e. neither 'gcc-8' nor 'gcc-9' was
> available in PATH.  While now that image does have GCC 9 properly
> installed, i.e. 'gcc-9' has been available in PATH for almost a week,
> I wouldn't want to rely on that...

I thought gcc-<N> executable should be linked by `brew-install' itself, no?

This patch applied on top of the patch you replied works fine!
> With your changes if none of the listed gcc-<N> binaries were
> available in PATH, then we would silently build with the generic 'cc',
> which is 'clang' on OSX, and that's definitely not what we want.

I'm addressing this issue with a change in ci/lib.sh below. By adding /bin/false as a fallback.

If we could agree that the previous patch and this patch, together, could be good, I'll re-organise it and send again later.

Either that, or squash this into the previous patch
--------8<-------------------
From: Doan Tran Cong Danh <congdanhqx@gmail.com>
Date: Thu, 28 Nov 2019 18:24:58 +0700
Subject: [PATCH/RFC] ci: install latest gcc from brew for macOS build on travis

Until now, we ask brew install the specific version of gcc (gcc@8), but for some reason, brew decided to install the latest version instead.

With the previous change, we're assuming that gcc-<N> executable is existed in $PATH, we will find that executable instead.

While we're at it, don't ask brew to link gcc, since it should be taken by `brew-install' itself.

Signed-off-by: Doan Tran Cong Danh <congdanhqx@gmail.com>
---
 ci/install-dependencies.sh | 2 --
 ci/lib.sh                  | 4 ++--
 2 files changed, 2 insertions(+), 4 deletions(-)
diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh
index 4e64a19112..a671ee1005 100755
--- a/ci/install-dependencies.sh
+++ b/ci/install-dependencies.sh
@@ -48,8 +48,6 @@ osx-clang|osx-gcc)
 	brew install caskroom/cask/perforce
 	case "$jobname" in
 	osx-gcc)
-		brew link gcc ||
-		brew link gcc@8
 		;;
 	esac
 	;;
diff --git a/ci/lib.sh b/ci/lib.sh
index 4040fc1a22..02cdb47419 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -131,7 +131,7 @@ then
 		echo "$SYSTEM_TASKDEFINITIONSURI$SYSTEM_TEAMPROJECT/_build/results?buildId=$1"
 	}
 
-	BREW_INSTALL_PACKAGES=gcc@8
+	BREW_INSTALL_PACKAGES=gcc
 	export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save"
 	export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml"
 	MAKEFLAGS="$MAKEFLAGS --jobs=10"
@@ -160,7 +160,7 @@ export GIT_TEST_CLONE_2GB=YesPlease
 
 case "$jobname" in
 *-gcc)
-	for gitcc in gcc-10 gcc-9 gcc-8
+	for gitcc in gcc-10 gcc-9 gcc-8 false
 	do
 		if command -v $gitcc >/dev/null 2>&1
 		then
-- 
2.24.0.615.g37f5bfbdea
SZEDER Gábor· Nov 28, 2019, 12:33 UTC · re: Danh Doan · lore

Re: [PATCH] ci: detect installed gcc in Travis CI

On Thu, Nov 28, 2019 at 07:03:30PM +0700, Danh Doan wrote:
Show 49 quoted lines
> On 2019-11-27 17:48:20+0100, SZEDER Gábor <szeder.dev@gmail.com> wrote:
> > On Wed, Nov 27, 2019 at 11:15:55PM +0700, Doan Tran Cong Danh wrote:
> > > Travis CI has continously updated their images, including updating gcc
> > > installation.
> > > 
> > > Save us some headache by checking which version of gcc is installed in
> > > the image, and use said version to run the build.
> > > 
> > > While gcc-10 hasn't been released, yet, add it to the list to save us
> > > some headache in the future.
> > > 
> > > Signed-off-by: Doan Tran Cong Danh <congdanhqx@gmail.com>
> > > ---
> > >  ci/lib.sh | 24 ++++++++++++++----------
> > >  1 file changed, 14 insertions(+), 10 deletions(-)
> > > 
> > > diff --git a/ci/lib.sh b/ci/lib.sh
> > > index c8c2c38155..4040fc1a22 100755
> > > --- a/ci/lib.sh
> > > +++ b/ci/lib.sh
> > > @@ -159,12 +159,21 @@ export DEFAULT_TEST_TARGET=prove
> > >  export GIT_TEST_CLONE_2GB=YesPlease
> > >  
> > >  case "$jobname" in
> > > -linux-clang|linux-gcc)
> > > -	if [ "$jobname" = linux-gcc ]
> > > -	then
> > > -		export CC=gcc-8
> > > -	fi
> > > +*-gcc)
> > > +	for gitcc in gcc-10 gcc-9 gcc-8
> > > +	do
> > > +		if command -v $gitcc >/dev/null 2>&1
> > > +		then
> > > +			export CC=$gitcc
> > > +			break;
> > > +		fi
> > > +	done
> > 
> > This assummes that some of these gcc-<N> binaries can be found in
> > PATH, but that hasn't always been the case in the past: up until end
> > of last week Travis CI's xcode10.1 image came with GCC 8 pre-installed
> > but not linked and without GCC 9, i.e. neither 'gcc-8' nor 'gcc-9' was
> > available in PATH.  While now that image does have GCC 9 properly
> > installed, i.e. 'gcc-9' has been available in PATH for almost a week,
> > I wouldn't want to rely on that...
> 
> I thought gcc-<N> executable should be linked by `brew-install'
> itself, no?

'brew install' does not link already installed but not linked packages, but instructs the user to run 'brew link' instead; see

  https://public-inbox.org/git/20191127162416.19391-1-szeder.dev@gmail.com/
SZEDER Gábor· Nov 27, 2019, 16:24 UTC · re: Denton Liu · lore

[PATCH] ci: build Git with GCC 9 in the 'osx-gcc' build job

Our 'osx-gcc' build job on Travis CI relied on GCC 8 being installed (but not linked) in the image we use [1]. Alas, since the last update of this image a few days ago this is not the case anymore, and now it contains GCC 9 (installed and linked) instead of GCC 8. The results are failed 'osx-gcc' jobs, because they can't find the 'gcc-8' command [2].

Let's move on to use GCC 9, with hopefully better error reporting and improved -Wfoo flags and what not. On Travis CI this has the benefit that we can spare a few seconds while installing dependencies, because it already comes pre-installed, at least for now. The Azure Pipelines OSX image doesn't include GCC, so we have to install it ourselves anyway, and then we might as well install the newer version.

In a vain attempt I tried to future-proof this a bit:
  - Install 'gcc@9' specifically, so we'll still get what we want even
    after GCC 10 comes out, and the "plain" 'gcc' package starts to
    refer to 'gcc@10'.
  - Run both 'brew install gcc@9' and 'brew link gcc@9'.  If 'gcc@9'
    is already installed and linked, then both commands are noop and
    exit with success.  But as we saw in the past, sometimes the image
    contains the expected GCC package installed but not linked, so
    maybe it will happen again in the future as well.  In that case
    'brew install' is still a noop, and instructs the user to run
    'brew link' instead, so that's what we'll do.  And if 'gcc@9' is
    not installed, then 'brew install' will install it, and the
    subsequent 'brew link' becomes a noop.

An additional benefit of this patch is that from now on we won't unnecessarily install GCC and its dependencies in the 'osx-clang' jobs on Azure Pipelines.

[1] 7d4733c501 (ci: fix GCC install in the Travis CI GCC OSX job,
    2019-10-24)
[2] https://travis-ci.org/git/git/jobs/615442297#L333
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---
On Mon, Nov 25, 2019 at 05:15:36PM -0800, Denton Liu wrote:
> It seems like gcc-8 doesn't exist anymore in our gcc OS X Travis job. As
> a result, that job has been failing consistently recently[1].
> 
> An old similar thread here[2]. Not sure if the failure is related.
> [2]: https://lore.kernel.org/git/9d80e845bf923c4543c49f9947aacb10c59ff6ce.1571789978.git.gitgitgadget@gmail.com/

It's funny how merely two days after we talked about how/when Travis CI images are updated, we got yet another update, which broke our builds again... Oh, well.

 ci/install-dependencies.sh | 6 ++++--
 ci/lib.sh                  | 3 +--
 2 files changed, 5 insertions(+), 4 deletions(-)
diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh
index 4e64a19112..93e3bba434 100755
--- a/ci/install-dependencies.sh
+++ b/ci/install-dependencies.sh
@@ -48,8 +48,10 @@ osx-clang|osx-gcc)
 	brew install caskroom/cask/perforce
 	case "$jobname" in
 	osx-gcc)
-		brew link gcc ||
-		brew link gcc@8
+		brew install gcc@9
+		# Just in case the image is updated to contain gcc@9
+		# pre-installed but not linked.
+		brew link gcc@9
 		;;
 	esac
 	;;
diff --git a/ci/lib.sh b/ci/lib.sh
index c8c2c38155..10b97c05ad 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -131,7 +131,6 @@ then
 		echo "$SYSTEM_TASKDEFINITIONSURI$SYSTEM_TEAMPROJECT/_build/results?buildId=$1"
 	}
 
-	BREW_INSTALL_PACKAGES=gcc@8
 	export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save"
 	export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml"
 	MAKEFLAGS="$MAKEFLAGS --jobs=10"
@@ -182,7 +181,7 @@ linux-clang|linux-gcc)
 osx-clang|osx-gcc)
 	if [ "$jobname" = osx-gcc ]
 	then
-		export CC=gcc-8
+		export CC=gcc-9
 	fi
 
 	# t9810 occasionally fails on Travis CI OS X
-- 
2.24.0.643.gac013444ca

← back to recent threads