[PATCH v2 0/5] ci: add GitLab CI definition
- From
Patrick Steinhardt <ps@pks.im>
- Date
- Oct 27, 2023, 09:25 UTC
- Message-ID
- <cover.1698398590.git.ps@pks.im>
- In-Reply-To
- <cover.1698305961.git.ps@pks.im>
Hi,
this is the second version of this patch series that adds GitLab CI definitions to the Git project. Please refer to the cover letter for v1 of this series [1] for the motivation and intent -- I won't repeat it here as it's a bit on the longer side.
Changes compared to v1:
- Improved commit messages.
- The `group ()` function is now generic across all the different CI
solutions to reduce duplication.This should hopefully address the feedback from Oswald -- thanks again!
Patrick
[1]: <cover.1698305961.git.ps@pks.im>
Patrick Steinhardt (5): ci: reorder definitions for grouping functions ci: make grouping setup more generic ci: group installation of Docker dependencies ci: split out logic to set up failed test artifacts ci: add support for GitLab CI
.gitlab-ci.yml | 51 +++++++++++ ci/install-docker-dependencies.sh | 15 +++- ci/lib.sh | 139 ++++++++++++++++++++---------- ci/print-test-failures.sh | 6 ++ 4 files changed, 166 insertions(+), 45 deletions(-) create mode 100644 .gitlab-ci.yml
Range-diff against v1:
1: 586a8d1003b ! 1: 4eb9cfc816b ci: reorder definitions for grouping functions
@@ Commit message
output in our CI, where these groups then end up as collapsible sections
in the respective pipeline platform. The way these functions are defined
is not easily extensible though as we have an up front check for the CI
- _not_ being GitLab Actions, where we define the non-stub logic in the
+ _not_ being GitHub Actions, where we define the non-stub logic in the
else branch.
- Reorder the definitions such that we explicitly handle GitHub Actions.
+ Reorder the conditional branches such that we explicitly handle GitHub
+ Actions.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
2: ec390354f15 ! 2: 85617ef8577 ci: make grouping setup more generic
@@ Commit message
or not. This ensures we can more readily add support for additional CI
platforms.
- Second, this commit changes `end_group ()` to also accept a parameter
- that indicates _which_ group should end. This will be required by a
- later commit that introduces support for GitLab CI.
+ Furthermore, the `group ()` function is made generic so that it is the
+ same for both GitHub Actions and for other platforms. There is a
+ semantic conflict here though: GitHub Actions used to call `set +x` in
+ `group ()` whereas the non-GitHub case unconditionally uses `set -x`.
+ The latter would get overriden if we kept the `set +x` in the generic
+ version of `group ()`. To resolve this conflict, we simply drop the `set
+ +x` in the generic variant of this function. As `begin_group ()` calls
+ `set -x` anyway this is not much of a change though, as the only
+ commands that aren't printed anymore now are the ones between the
+ beginning of `group ()` and the end of `begin_group ()`.
+
+ Last, this commit changes `end_group ()` to also accept a parameter that
+ indicates _which_ group should end. This will be required by a later
+ commit that introduces support for GitLab CI.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
@@ ci/lib.sh: then
echo '::endgroup::' >&2
}
- trap end_group EXIT
-
- group () {
- set +x
+-
+- group () {
+- set +x
- begin_group "$1"
-+
-+ group="$1"
- shift
-+ begin_group "$group"
-+
- # work around `dash` not supporting `set -o pipefail`
- (
- "$@" 2>&1
-@@ ci/lib.sh: then
- sed 's/^\(\([^ ]*\):\([0-9]*\):\([0-9]*:\) \)\(error\|warning\): /::\5 file=\2,line=\3::\1/'
- res=$(cat exit.status)
- rm exit.status
+- shift
+- # work around `dash` not supporting `set -o pipefail`
+- (
+- "$@" 2>&1
+- echo $? >exit.status
+- ) |
+- sed 's/^\(\([^ ]*\):\([0-9]*\):\([0-9]*:\) \)\(error\|warning\): /::\5 file=\2,line=\3::\1/'
+- res=$(cat exit.status)
+- rm exit.status
- end_group
-+
-+ end_group "$group"
- return $res
- }
+- return $res
+- }
-
- begin_group "CI setup"
else
begin_group () { :; }
end_group () { :; }
-@@ ci/lib.sh: else
+
+- group () {
+- shift
+- "$@"
+- }
set -x
fi
++group () {
++ group="$1"
++ shift
++ begin_group "$group"
++
++ # work around `dash` not supporting `set -o pipefail`
++ (
++ "$@" 2>&1
++ echo $? >exit.status
++ ) |
++ sed 's/^\(\([^ ]*\):\([0-9]*\):\([0-9]*:\) \)\(error\|warning\): /::\5 file=\2,line=\3::\1/'
++ res=$(cat exit.status)
++ rm exit.status
++
++ end_group "$group"
++ return $res
++}
++
+begin_group "CI setup"
+trap "end_group 'CI setup'" EXIT
+
3: a65d235dd3c ! 3: 57bbc50e3dc ci: group installation of Docker dependencies
@@ Metadata
## Commit message ##
ci: group installation of Docker dependencies
- Pull in "lib.sh" into "install-docker-dependencies.sh" such that we can
- set up proper groups for those dependencise. This allows the reader to
- collapse sections in the CI output on GitHub Actions (and later on on
- GitLab CI).
+ The output of CI jobs tends to be quite long-winded and hard to digest.
+ To help with this, many CI systems provide the ability to group output
+ into collapsible sections, and we're also doing this in some of our
+ scripts.
+
+ One notable omission is the script to install Docker dependencies.
+ Address it to bring more structure to the output for Docker-based jobs.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
4: 4a864a1d174 ! 4: 5ab11d5236d ci: split out logic to set up failed test artifacts
@@ Commit message
ci: split out logic to set up failed test artifacts
We have some logic in place to create a directory with the output from
- failed tests, which will then subsequently be uploaded as CI artifact.
+ failed tests, which will then subsequently be uploaded as CI artifacts.
We're about to add support for GitLab CI, which will want to reuse the
logic.
5: 35b07e5378d ! 5: 37a507e9b25 ci: add support for GitLab CI
@@ Commit message
Part of a problem we hit at GitLab rather frequently is that our own,
custom CI setup we have is so different to the setup that the Git
- project has. More esoteric jobs like "linux-TEST-vars" that also sets a
+ project has. More esoteric jobs like "linux-TEST-vars" that also set a
couple of environment variables do not exist in GitLab's custom CI
setup, and maintaining them to keep up with what Git does feels like
wasted time. The result is that we regularly send patch series upstream
- that would otherwise fail to compile or pass tests in GitHub Workflows.
- We would thus like to integrate the GitLab CI configuration into the Git
- project to help us ensure to send better patch series upstream and thus
- reduce overhead for the maintainer.
+ that fail to compile or pass tests in GitHub Workflows. We would thus
+ like to integrate the GitLab CI configuration into the Git project to
+ help us send better patch series upstream and thus reduce overhead for
+ the maintainer.
The integration does not necessarily have to be a first-class citizen,
which would in practice only add to the fallout that pipeline failures
@@ ci/install-docker-dependencies.sh: linux32)
## ci/lib.sh ##
@@ ci/lib.sh: then
+ need_to_end_group=
echo '::endgroup::' >&2
}
-
-+ group () {
-+ set +x
-+
-+ group="$1"
-+ shift
-+ begin_group "$group"
-+
-+ # work around `dash` not supporting `set -o pipefail`
-+ (
-+ "$@" 2>&1
-+ echo $? >exit.status
-+ ) |
-+ sed 's/^\(\([^ ]*\):\([0-9]*\):\([0-9]*:\) \)\(error\|warning\): /::\5 file=\2,line=\3::\1/'
-+ res=$(cat exit.status)
-+ rm exit.status
-+
-+ end_group "$group"
-+ return $res
-+ }
+elif test true = "$GITLAB_CI"
+then
+ begin_group () {
@@ ci/lib.sh: then
+ echo -e "\e[0Ksection_end:$(date +%s):$(echo "$1" | tr ' ' _)\r\e[0K"
+ trap - EXIT
+ }
-+
- group () {
- set +x
-
+ else
+ begin_group () { :; }
+ end_group () { :; }
@@ ci/lib.sh: then
MAKEFLAGS="$MAKEFLAGS --jobs=10"
test windows != "$CI_OS_NAME" ||-- 2.42.0