git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v4 8/8] ci: add support for GitLab CI

From
Patrick Steinhardt <ps@pks.im>
Date
Oct 31, 2023, 09:05 UTC
Message-ID
<f3f2c98a0dc6042b7ed5eab9c10bee4f64858f02.1698742590.git.ps@pks.im>
In-Reply-To
<cover.1698742590.git.ps@pks.im>

We already support Azure Pipelines and GitHub Workflows in the Git project, but until now we do not have support for GitLab CI. While it is arguably not in the interest of the Git project to maintain a ton of different CI platforms, GitLab has recently ramped up its efforts and tries to contribute to the Git project more regularly.

Part of a problem we hit at GitLab rather frequently is that our own, custom CI setup we have is so different to the setup that the Git project has. More esoteric jobs like "linux-TEST-vars" that also set a couple of environment variables do not exist in GitLab's custom CI setup, and maintaining them to keep up with what Git does feels like wasted time. The result is that we regularly send patch series upstream that fail to compile or pass tests in GitHub Workflows. We would thus like to integrate the GitLab CI configuration into the Git project to help us send better patch series upstream and thus reduce overhead for the maintainer.

The integration does not necessarily have to be a first-class citizen, which would in practice only add to the fallout that pipeline failures have for the maintainer. That being said, we are happy to maintain this alternative CI setup for the Git project and will make test results available as part of our own mirror of the Git project at [1].

This commit introduces the integration into our regular CI scripts so that most of the setup continues to be shared across all of the CI solutions. Note that as the builds on GitLab CI run as unprivileged user, we need to pull in both sudo and shadow packages to our Alpine based job to set this up.

[1]: https://gitlab.com/gitlab-org/git
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml                    | 53 +++++++++++++++++++++++++++++++
 ci/install-docker-dependencies.sh | 13 +++++++-
 ci/lib.sh                         | 50 +++++++++++++++++++++++++++--
 ci/print-test-failures.sh         |  6 ++++
 4 files changed, 119 insertions(+), 3 deletions(-)
 create mode 100644 .gitlab-ci.yml
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
new file mode 100644
index 00000000000..cd98bcb18aa
--- /dev/null
+++ b/.gitlab-ci.yml
@@ -0,0 +1,53 @@
+default:
+  timeout: 2h
+
+workflow:
+  rules:
+    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
+    - if: $CI_COMMIT_TAG
+    - if: $CI_COMMIT_REF_PROTECTED == "true"
+
+test:
+  image: $image
+  before_script:
+    - ./ci/install-docker-dependencies.sh
+  script:
+    - useradd builder --create-home
+    - chown -R builder "${CI_PROJECT_DIR}"
+    - sudo --preserve-env --set-home --user=builder ./ci/run-build-and-tests.sh
+  after_script:
+    - |
+      if test "$CI_JOB_STATUS" != 'success'
+      then
+        sudo --preserve-env --set-home --user=builder ./ci/print-test-failures.sh
+      fi
+  parallel:
+    matrix:
+      - jobname: linux-sha256
+        image: ubuntu:latest
+        CC: clang
+      - jobname: linux-gcc
+        image: ubuntu:20.04
+        CC: gcc
+        CC_PACKAGE: gcc-8
+      - jobname: linux-TEST-vars
+        image: ubuntu:20.04
+        CC: gcc
+        CC_PACKAGE: gcc-8
+      - jobname: linux-gcc-default
+        image: ubuntu:latest
+        CC: gcc
+      - jobname: linux-leaks
+        image: ubuntu:latest
+        CC: gcc
+      - jobname: linux-asan-ubsan
+        image: ubuntu:latest
+        CC: clang
+      - jobname: pedantic
+        image: fedora:latest
+      - jobname: linux-musl
+        image: alpine:latest
+  artifacts:
+    paths:
+      - t/failed-test-artifacts
+    when: on_failure
diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh
index 6e845283680..48cb2e735b5 100755
--- a/ci/install-docker-dependencies.sh
+++ b/ci/install-docker-dependencies.sh
@@ -7,6 +7,9 @@
 
 begin_group "Install dependencies"
 
+# Required so that apt doesn't wait for user input on certain packages.
+export DEBIAN_FRONTEND=noninteractive
+
 case "$jobname" in
 linux32)
 	linux32 --32bit i386 sh -c '
@@ -16,11 +19,19 @@ linux32)
 	'
 	;;
 linux-musl)
-	apk add --update build-base curl-dev openssl-dev expat-dev gettext \
+	apk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \
 		pcre2-dev python3 musl-libintl perl-utils ncurses \
 		apache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \
 		bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null
 	;;
+linux-*)
+	apt update -q &&
+	apt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev \
+		libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev \
+		perl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl \
+		libdbd-sqlite3-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}} \
+		apache2 cvs cvsps gnupg libcgi-pm-perl subversion
+	;;
 pedantic)
 	dnf -yq update >/dev/null &&
 	dnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null
diff --git a/ci/lib.sh b/ci/lib.sh
index e14b1029fad..6e3d64004ec 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -14,6 +14,22 @@ then
 		need_to_end_group=
 		echo '::endgroup::' >&2
 	}
+elif test true = "$GITLAB_CI"
+then
+	begin_group () {
+		need_to_end_group=t
+		printf "\e[0Ksection_start:$(date +%s):$(echo "$1" | tr ' ' _)\r\e[0K$1\n"
+		trap "end_group '$1'" EXIT
+		set -x
+	}
+
+	end_group () {
+		test -n "$need_to_end_group" || return 0
+		set +x
+		need_to_end_group=
+		printf "\e[0Ksection_end:$(date +%s):$(echo "$1" | tr ' ' _)\r\e[0K\n"
+		trap - EXIT
+	}
 else
 	begin_group () { :; }
 	end_group () { :; }
@@ -208,6 +224,7 @@ then
 	cache_dir="$HOME/test-cache/$SYSTEM_PHASENAME"
 
 	GIT_TEST_OPTS="--write-junit-xml"
+	JOBS=10
 elif test true = "$GITHUB_ACTIONS"
 then
 	CI_TYPE=github-actions
@@ -227,14 +244,43 @@ then
 	cache_dir="$HOME/none"
 
 	GIT_TEST_OPTS="--github-workflow-markup"
+	JOBS=10
+elif test true = "$GITLAB_CI"
+then
+	CI_TYPE=gitlab-ci
+	CI_BRANCH="$CI_COMMIT_REF_NAME"
+	CI_COMMIT="$CI_COMMIT_SHA"
+	case "$CI_JOB_IMAGE" in
+	macos-*)
+		CI_OS_NAME=osx;;
+	alpine:*|fedora:*|ubuntu:*)
+		CI_OS_NAME=linux;;
+	*)
+		echo "Could not identify OS image" >&2
+		env >&2
+		exit 1
+		;;
+	esac
+	CI_REPO_SLUG="$CI_PROJECT_PATH"
+	CI_JOB_ID="$CI_JOB_ID"
+	CC="${CC_PACKAGE:-${CC:-gcc}}"
+	DONT_SKIP_TAGS=t
+	handle_failed_tests () {
+		create_failed_test_artifacts
+	}
+
+	cache_dir="$HOME/none"
+
+	runs_on_pool=$(echo "$CI_JOB_IMAGE" | tr : -)
+	JOBS=$(nproc)
 else
 	echo "Could not identify CI type" >&2
 	env >&2
 	exit 1
 fi
 
-MAKEFLAGS="$MAKEFLAGS --jobs=10"
-GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save"
+MAKEFLAGS="$MAKEFLAGS --jobs=${JOBS}"
+GIT_PROVE_OPTS="--timer --jobs ${JOBS} --state=failed,slow,save"
 
 GIT_TEST_OPTS="$GIT_TEST_OPTS --verbose-log -x"
 if test windows = "$CI_OS_NAME"
diff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh
index 57277eefcd0..c33ad4e3a22 100755
--- a/ci/print-test-failures.sh
+++ b/ci/print-test-failures.sh
@@ -51,6 +51,12 @@ do
 			tar czf failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir"
 			continue
 			;;
+		gitlab-ci)
+			mkdir -p failed-test-artifacts
+			cp "${TEST_EXIT%.exit}.out" failed-test-artifacts/
+			tar czf failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir"
+			continue
+			;;
 		*)
 			echo "Unhandled CI type: $CI_TYPE" >&2
 			exit 1

-- 
2.42.0
Previous: Patrick SteinhardtNext: Victoria Dye
Message 77 of 101 in “ci: add GitLab CI definition”
  1. 0/5 ci: add GitLab CI definitionPatrick Steinhardt, Oct 26, 2023
  2. 1/5 ci: reorder definitions for grouping functionsPatrick Steinhardt, Oct 26, 2023
  3. Oswald BuddenhagenOct 26, 2023
  4. Patrick SteinhardtOct 27, 2023
  5. 2/5 ci: make grouping setup more genericPatrick Steinhardt, Oct 26, 2023
  6. 3/5 ci: group installation of Docker dependenciesPatrick Steinhardt, Oct 26, 2023
  7. Oswald BuddenhagenOct 26, 2023
  8. Patrick SteinhardtOct 27, 2023
  9. 4/5 ci: split out logic to set up failed test artifactsPatrick Steinhardt, Oct 26, 2023
  10. Oswald BuddenhagenOct 26, 2023
  11. Christian CouderNov 3, 2023
  12. Patrick SteinhardtNov 6, 2023
  13. 5/5 ci: add support for GitLab CIPatrick Steinhardt, Oct 26, 2023
  14. Oswald BuddenhagenOct 26, 2023
  15. Patrick SteinhardtOct 27, 2023
  16. Phillip WoodOct 27, 2023
  17. Oswald BuddenhagenOct 27, 2023
  18. Phillip WoodOct 27, 2023
  19. Oswald BuddenhagenOct 27, 2023
  20. Phillip WoodOct 30, 2023
  21. Dragan SimicOct 30, 2023
  22. Oswald BuddenhagenOct 27, 2023
  23. Patrick SteinhardtOct 27, 2023
  24. 0/5 ci: add GitLab CI definitionPatrick Steinhardt, Oct 27, 2023
  25. 1/5 ci: reorder definitions for grouping functionsPatrick Steinhardt, Oct 27, 2023
  26. 2/5 ci: make grouping setup more genericPatrick Steinhardt, Oct 27, 2023
  27. 3/5 ci: group installation of Docker dependenciesPatrick Steinhardt, Oct 27, 2023
  28. 4/5 ci: split out logic to set up failed test artifactsPatrick Steinhardt, Oct 27, 2023
  29. 5/5 ci: add support for GitLab CIPatrick Steinhardt, Oct 27, 2023
  30. Phillip WoodOct 27, 2023
  31. Patrick SteinhardtOct 27, 2023
  32. Patrick SteinhardtOct 27, 2023
  33. Phillip WoodOct 27, 2023
  34. Phillip WoodOct 29, 2023
  35. Patrick SteinhardtOct 30, 2023
  36. Phillip WoodOct 29, 2023
  37. Patrick SteinhardtOct 30, 2023
  38. Junio C HamanoOct 30, 2023
  39. Oswald BuddenhagenOct 27, 2023
  40. Phillip WoodOct 27, 2023
  41. Oswald BuddenhagenOct 27, 2023
  42. Jeff KingOct 31, 2023
  43. Junio C HamanoNov 1, 2023
  44. 0/8 ci: add GitLab CI definitionPatrick Steinhardt, Oct 30, 2023
  45. 1/8 ci: reorder definitions for grouping functionsPatrick Steinhardt, Oct 30, 2023
  46. 2/8 ci: make grouping setup more genericPatrick Steinhardt, Oct 30, 2023
  47. 3/8 ci: group installation of Docker dependenciesPatrick Steinhardt, Oct 30, 2023
  48. 4/8 ci: split out logic to set up failed test artifactsPatrick Steinhardt, Oct 30, 2023
  49. 5/8 ci: unify setup of some environment variablesPatrick Steinhardt, Oct 30, 2023
  50. Phillip WoodOct 30, 2023
  51. Patrick SteinhardtOct 30, 2023
  52. Dragan SimicOct 30, 2023
  53. 6/8 ci: squelch warnings when testing with unusable Git repoPatrick Steinhardt, Oct 30, 2023
  54. 7/8 ci: install test dependencies for linux-muslPatrick Steinhardt, Oct 30, 2023
  55. Patrick SteinhardtOct 30, 2023
  56. Patrick SteinhardtOct 30, 2023
  57. Phillip WoodOct 30, 2023
  58. Patrick SteinhardtOct 30, 2023
  59. Phillip WoodOct 30, 2023
  60. 8/8 ci: add support for GitLab CIPatrick Steinhardt, Oct 30, 2023
  61. Taylor BlauOct 30, 2023
  62. Patrick SteinhardtOct 31, 2023
  63. Taylor BlauOct 31, 2023
  64. Junio C HamanoNov 1, 2023
  65. Patrick SteinhardtNov 1, 2023
  66. 0/8 ci: add GitLab CI definitionPatrick Steinhardt, Oct 31, 2023
  67. 1/8 ci: reorder definitions for grouping functionsPatrick Steinhardt, Oct 31, 2023
  68. 2/8 ci: make grouping setup more genericPatrick Steinhardt, Oct 31, 2023
  69. 3/8 ci: group installation of Docker dependenciesPatrick Steinhardt, Oct 31, 2023
  70. 4/8 ci: split out logic to set up failed test artifactsPatrick Steinhardt, Oct 31, 2023
  71. 5/8 ci: unify setup of some environment variablesPatrick Steinhardt, Oct 31, 2023
  72. Victoria DyeOct 31, 2023
  73. Junio C HamanoNov 1, 2023
  74. Patrick SteinhardtNov 1, 2023
  75. 6/8 ci: squelch warnings when testing with unusable Git repoPatrick Steinhardt, Oct 31, 2023
  76. 7/8 ci: install test dependencies for linux-muslPatrick Steinhardt, Oct 31, 2023
  77. 8/8 ci: add support for GitLab CIPatrick Steinhardt, Oct 31, 2023
  78. Victoria DyeOct 31, 2023
  79. Patrick SteinhardtNov 1, 2023
  80. Victoria DyeOct 31, 2023
  81. Junio C HamanoNov 1, 2023
  82. Patrick SteinhardtNov 1, 2023
  83. 0/8 ci: add support for GitLab CIPatrick Steinhardt, Nov 1, 2023
  84. 1/8 ci: reorder definitions for grouping functionsPatrick Steinhardt, Nov 1, 2023
  85. 2/8 ci: make grouping setup more genericPatrick Steinhardt, Nov 1, 2023
  86. 3/8 ci: group installation of Docker dependenciesPatrick Steinhardt, Nov 1, 2023
  87. 4/8 ci: split out logic to set up failed test artifactsPatrick Steinhardt, Nov 1, 2023
  88. 5/8 ci: unify setup of some environment variablesPatrick Steinhardt, Nov 1, 2023
  89. 6/8 ci: squelch warnings when testing with unusable Git repoPatrick Steinhardt, Nov 1, 2023
  90. 7/8 ci: install test dependencies for linux-muslPatrick Steinhardt, Nov 1, 2023
  91. 8/8 ci: add support for GitLab CIPatrick Steinhardt, Nov 1, 2023
  92. 0/8 ci: add GitLab CI definitionPatrick Steinhardt, Nov 9, 2023
  93. 1/8 ci: reorder definitions for grouping functionsPatrick Steinhardt, Nov 9, 2023
  94. 2/8 ci: make grouping setup more genericPatrick Steinhardt, Nov 9, 2023
  95. 3/8 ci: group installation of Docker dependenciesPatrick Steinhardt, Nov 9, 2023
  96. 4/8 ci: split out logic to set up failed test artifactsPatrick Steinhardt, Nov 9, 2023
  97. 5/8 ci: unify setup of some environment variablesPatrick Steinhardt, Nov 9, 2023
  98. 6/8 ci: squelch warnings when testing with unusable Git repoPatrick Steinhardt, Nov 9, 2023
  99. 7/8 ci: install test dependencies for linux-muslPatrick Steinhardt, Nov 9, 2023
  100. 8/8 ci: add support for GitLab CIPatrick Steinhardt, Nov 9, 2023
  101. Junio C HamanoNov 9, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.