git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 1/4] github-actions: run gcc-8 on ubuntu-20.04 image

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
Nov 24, 2022, 16:29 UTC
Message-ID
<221124.86sfi8nvzz.gmgdl@evledraar.gmail.com>
In-Reply-To
<20221124153934.12470-2-worldhello.net@gmail.com>
On Thu, Nov 24 2022, Jiang Xin wrote:
Show 141 quoted lines
> From: Jiang Xin <zhiyou.jx@alibaba-inc.com>
>
> GitHub starts to upgrade its runner image "ubuntu-latest" from version
> "ubuntu-20.04" to version "ubuntu-22.04". It will fail to find and
> install "gcc-8" package on the new runner image.
>
> Change the runner images from "ubuntu-latest" to "ubuntu-20.04" in order
> to run with "gcc-8" as a dependency.
>
> Instead of use the environment "$runs_on_pool" as below:
>
>     case "$runs_on_pool" in
>     ubuntu-20.04 | ubuntu-latest)
> 	;;
>
> we can reuse the os field in the matrix, and use a new environment
> "$runs_on_os" as below:
>
>     case "$runs_on_os" in
>     ubuntu)
>         ;;
>
> In this way, we can change the "ubuntu-latest" runner image to any
> version such as "ubuntu-22.04" to test CI behavior in advance.
>
> Reviewed-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> Signed-off-by: Jiang Xin <zhiyou.jx@alibaba-inc.com>
> ---
>  .github/workflows/main.yml | 16 ++++++++++++----
>  ci/install-dependencies.sh |  6 +++---
>  ci/lib.sh                  |  6 +++---
>  3 files changed, 18 insertions(+), 10 deletions(-)
>
> diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
> index 592f9193a8..da0d8ab0bf 100644
> --- a/.github/workflows/main.yml
> +++ b/.github/workflows/main.yml
> @@ -224,6 +224,7 @@ jobs:
>          vector:
>            - jobname: linux-clang
>              cc: clang
> +            os: ubuntu
>              pool: ubuntu-latest
>            - jobname: linux-sha256
>              cc: clang
> @@ -232,36 +233,43 @@ jobs:
>            - jobname: linux-gcc
>              cc: gcc
>              cc_package: gcc-8
> -            pool: ubuntu-latest
> +            os: ubuntu
> +            pool: ubuntu-20.04
>            - jobname: linux-TEST-vars
>              cc: gcc
> -            os: ubuntu
>              cc_package: gcc-8
> -            pool: ubuntu-latest
> +            os: ubuntu
> +            pool: ubuntu-20.04
>            - jobname: osx-clang
>              cc: clang
> +            os: macos
>              pool: macos-latest
>            - jobname: osx-gcc
>              cc: gcc
>              cc_package: gcc-9
> +            os: macos
>              pool: macos-latest
>            - jobname: linux-gcc-default
>              cc: gcc
> +            os: ubuntu
>              pool: ubuntu-latest
>            - jobname: linux-leaks
>              cc: gcc
> +            os: ubuntu
>              pool: ubuntu-latest
>            - jobname: linux-asan
>              cc: gcc
> +            os: ubuntu
>              pool: ubuntu-latest
>            - jobname: linux-ubsan
>              cc: gcc
> +            os: ubuntu
>              pool: ubuntu-latest
>      env:
>        CC: ${{matrix.vector.cc}}
>        CC_PACKAGE: ${{matrix.vector.cc_package}}
>        jobname: ${{matrix.vector.jobname}}
> -      runs_on_pool: ${{matrix.vector.pool}}
> +      runs_on_os: ${{matrix.vector.os}}
>      runs-on: ${{matrix.vector.pool}}
>      steps:
>      - uses: actions/checkout@v2
> diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh
> index 107757a1fe..f639263a62 100755
> --- a/ci/install-dependencies.sh
> +++ b/ci/install-dependencies.sh
> @@ -11,8 +11,8 @@ UBUNTU_COMMON_PKGS="make libssl-dev libcurl4-openssl-dev libexpat-dev
>   tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl
>   libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl"
>  
> -case "$runs_on_pool" in
> -ubuntu-latest)
> +case "$runs_on_os" in
> +ubuntu)
>  	sudo apt-get -q update
>  	sudo apt-get -q -y install language-pack-is libsvn-perl apache2 \
>  		$UBUNTU_COMMON_PKGS $CC_PACKAGE
> @@ -30,7 +30,7 @@ ubuntu-latest)
>  		cp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs .
>  	popd
>  	;;
> -macos-latest)
> +macos)
>  	export HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1
>  	# Uncomment this if you want to run perf tests:
>  	# brew install gnu-time
> diff --git a/ci/lib.sh b/ci/lib.sh
> index 24d20a5d64..0c0767d354 100755
> --- a/ci/lib.sh
> +++ b/ci/lib.sh
> @@ -225,8 +225,8 @@ export DEFAULT_TEST_TARGET=prove
>  export GIT_TEST_CLONE_2GB=true
>  export SKIP_DASHED_BUILT_INS=YesPlease
>  
> -case "$runs_on_pool" in
> -ubuntu-latest)
> +case "$runs_on_os" in
> +ubuntu)
>  	if test "$jobname" = "linux-gcc-default"
>  	then
>  		break
> @@ -253,7 +253,7 @@ ubuntu-latest)
>  	GIT_LFS_PATH="$HOME/custom/git-lfs"
>  	export PATH="$GIT_LFS_PATH:$P4_PATH:$PATH"
>  	;;
> -macos-latest)
> +macos)
>  	if [ "$jobname" = osx-gcc ]
>  	then
>  		MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python3)"

At the end of [1] I suggested splitting up the refactoring part of this. I still think that would be a good thing to do, but just to comment on the *actual* change (which I was hoping a split-out would make clearer).

This is proposing to refactor "runs-on-pool={ubuntu-latest,macos-latest}" to "os={ubuntu,macos}". I think it's worth cleaning up how we match main.yml and ci/*.sh, but think it's a bit of a digression in this otherwise narrowly focused series.

Now, this seems to somewhat come back to being all my fault. AFAICT I added "os" in c08bb260105 (CI: rename the "Linux32" job to lower-case "linux32", 2021-11-23) as part of some WIP (or from an earlier version?) that ende dup doing nothing.

But looking at it again, I don't see how an "os" here makes much
sense. For "macos-latest" if we're going to refactor it let's just stop
selecting stuff based on this "pool name", and instead fold it into the
adjacent "jobname" selection:
	
	diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh
	index 107757a1fea..b5a373060c1 100755
	--- a/ci/install-dependencies.sh
	+++ b/ci/install-dependencies.sh
	@@ -30,7 +30,9 @@ ubuntu-latest)
	 		cp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs .
	 	popd
	 	;;
	-macos-latest)
	+esac
	+case "$jobname" in
	+osx-*)
	 	export HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1
	 	# Uncomment this if you want to run perf tests:
	 	# brew install gnu-time
	diff --git a/ci/lib.sh b/ci/lib.sh
	index 24d20a5d648..fef60a507e9 100755
	--- a/ci/lib.sh
	+++ b/ci/lib.sh
	@@ -253,19 +253,17 @@ ubuntu-latest)
	 	GIT_LFS_PATH="$HOME/custom/git-lfs"
	 	export PATH="$GIT_LFS_PATH:$P4_PATH:$PATH"
	 	;;
	-macos-latest)
	-	if [ "$jobname" = osx-gcc ]
	-	then
	-		MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python3)"
	-	else
	-		MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python2)"
	-		MAKEFLAGS="$MAKEFLAGS NO_APPLE_COMMON_CRYPTO=NoThanks"
	-		MAKEFLAGS="$MAKEFLAGS NO_OPENSSL=NoThanks"
	-	fi
	-	;;
	 esac
	 
	 case "$jobname" in
	+osx-gcc)
	+	MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python3)"
	+	;;
	+osx-clang)
	+	MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=$(which python2)"
	+	MAKEFLAGS="$MAKEFLAGS NO_APPLE_COMMON_CRYPTO=NoThanks"
	+	MAKEFLAGS="$MAKEFLAGS NO_OPENSSL=NoThanks"
	+	;;
	 linux32)
	 	CC=gcc
	 	;;

That nicely sidesteps the need to refactor it here, and IMO makes it more future-proof.

For the "ubuntu" case I think it's even worse, because it now *looks* like a case that should be equivalent to just:

	test "$(lsb_release -si 2>&1 | grep -x Debian)" = "Debian"

But it's not, because we're not setting this "os=ubuntu" in all the places where the OS is Ubuntu. It really just means "the stuff that used to set env.runs_on_pool=ubuntu-latest, which e.g. doesn't cover the "documentation" job, which runs on "ubuntu-latest".

So I think *if* we're going to refactor this while-at-it, let's not make it "os=ubuntu", but e.g. "extra_ubuntu_packages: 1", which is really what it means. I.e. we're intercepting it in ci/lib.sh to set the "p4" variables etc., which we only need if we're doing the optional "apt install" on the "ubuntu-latest".

1. https://lore.kernel.org/git/221124.86mt8gpqra.gmgdl@evledraar.gmail.com/
Previous: Jiang XinNext: Jiang Xin
Message 32 of 47 in “Use fixed github-actions runner image”
  1. 0/2 Use fixed github-actions runner imageJiang Xin, Nov 23, 2022
  2. 1/2 github-actions: run gcc-8 on ubuntu-20.04 imageJiang Xin, Nov 23, 2022
  3. Johannes SchindelinNov 24, 2022
  4. 2/2 ci: upgrade version of p4Jiang Xin, Nov 23, 2022
  5. Johannes SchindelinNov 24, 2022
  6. Johannes SchindelinNov 24, 2022
  7. Johannes SchindelinNov 24, 2022
  8. Jiang XinNov 24, 2022
  9. Johannes SchindelinNov 24, 2022
  10. Jiang XinNov 24, 2022
  11. Johannes SchindelinNov 24, 2022
  12. 0/3 Fix broken CI on newer github-actions runner imageJiang Xin, Nov 24, 2022
  13. Johannes SchindelinNov 24, 2022
  14. Johannes SchindelinNov 24, 2022
  15. Jiang XinNov 24, 2022
  16. python 2 EOL (was: [PATCH v2 0/3] Fix broken CI on newer github-actions runner image)Ævar Arnfjörð Bjarmason, Nov 24, 2022
  17. Junio C HamanoNov 25, 2022
  18. 0/4 Fix broken CI on newer github-actions runner imageJiang Xin, Nov 24, 2022
  19. 0/4 Fix broken CI on newer github-actions runner imageJiang Xin, Nov 25, 2022
  20. 2/4 ci: remove the pipe after "p4 -V" to cache errorsJiang Xin, Nov 25, 2022
  21. Junio C HamanoNov 27, 2022
  22. Jiang XinNov 27, 2022
  23. 1/4 github-actions: run gcc-8 on ubuntu-20.04 imageJiang Xin, Nov 25, 2022
  24. Junio C HamanoNov 27, 2022
  25. 3/4 ci: p4 on Linux has the same version as on macOSJiang Xin, Nov 25, 2022
  26. Junio C HamanoNov 27, 2022
  27. 4/4 ci: install python on ubuntuJiang Xin, Nov 25, 2022
  28. Junio C HamanoNov 27, 2022
  29. Jiang XinNov 27, 2022
  30. Junio C HamanoNov 27, 2022
  31. 1/4 github-actions: run gcc-8 on ubuntu-20.04 imageJiang Xin, Nov 24, 2022
  32. Ævar Arnfjörð BjarmasonNov 24, 2022
  33. 2/4 ci: show error message of "p4 -V"Jiang Xin, Nov 24, 2022
  34. Ævar Arnfjörð BjarmasonNov 24, 2022
  35. Junio C HamanoNov 25, 2022
  36. 3/4 ci: p4 on Linux has the same version as on macOSJiang Xin, Nov 24, 2022
  37. 4/4 ci: install python on ubuntuJiang Xin, Nov 24, 2022
  38. 1/3 github-actions: run gcc-8 on ubuntu-20.04 imageJiang Xin, Nov 24, 2022
  39. Ævar Arnfjörð BjarmasonNov 24, 2022
  40. Junio C HamanoNov 25, 2022
  41. 2/3 ci: upgrade version of p4 to 21.2Jiang Xin, Nov 24, 2022
  42. Ævar Arnfjörð BjarmasonNov 24, 2022
  43. Jiang XinNov 24, 2022
  44. 3/3 ci: install python on ubuntuJiang Xin, Nov 24, 2022
  45. Ævar Arnfjörð BjarmasonNov 24, 2022
  46. Jiang XinNov 24, 2022
  47. Ævar Arnfjörð BjarmasonNov 24, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.