[PATCH v3 0/8] ci: add GitLab CI definition

Hi,

this is the third version of this patch series that adds GitLab CI definitions to the Git project. Please refer to the cover letter for v1 of this series [1] for the motivation and intent -- I won't repeat it here as it's a bit on the longer side.

Changes compared to v2:

    - Patch 5: This is a new preparatory step to unify the setup of some
      environment variables. It also fixes some smallish issues, like
      e.g. the fact that some envvars were set _after_ the export.

    - Patch 6: Another new preparatory step. It makes our infra around
      certain helper functions that have the intent to interact with the
      project repository, e.g. to cache good trees. We now detect the
      case when there is no Git or when the project is not a Git repo
      and bail out gracefully.

    - Patch 7: The last new preparatory patch. Installs a bunch of
      dependencies which are required for the test runtime in the Alpine
      based job. This increases test coverage.

    - Patch 8: Several smaller improvements:
        - Added a note why we install sudo and shadow in linux-musl now.
        - Fixed an issue where the HOME directory was part of the
          project directory, and thus Git complained about newly added
          untracked files.
        - Added the "pedantic" job that does a pednatic compilation on
          Fedora.
        - Added more test time dependencies.
        - Made the "echo -e" invocation portable by using printf
          instead.

What I didn't address yet is a suggestion by Phillip, namely to unify the logic that sets up unprivileged builds. I don't have the infra available to test any such change that would ultimately also impact GitHub Workflows and Azure Pipelines and thus do not feel comfortable to refactor this. I agree with the suggestion though, so I propose to rather handle it at a later point in time.

A test run of this patch series can be found at [2].

Thanks!

Patrick

[1]: <cover.1698305961.git.ps@pks.im> [2]: https://gitlab.com/gitlab-org/git/-/pipelines/1054750795

Patrick Steinhardt (8):
  ci: reorder definitions for grouping functions
  ci: make grouping setup more generic
  ci: group installation of Docker dependencies
  ci: split out logic to set up failed test artifacts
  ci: unify setup of some environment variables
  ci: squelch warnings when testing with unusable Git repo
  ci: install test dependencies for linux-musl
  ci: add support for GitLab CI

 .gitlab-ci.yml                    |  53 +++++++++
 ci/install-docker-dependencies.sh |  22 +++-
 ci/lib.sh                         | 187 +++++++++++++++++++++---------
 ci/print-test-failures.sh         |   6 +
 t/lib-httpd.sh                    |   3 +-
 5 files changed, 215 insertions(+), 56 deletions(-)
 create mode 100644 .gitlab-ci.yml

Range-diff against v2:
1:  4eb9cfc816b = 1:  ef44ed5c3b1 ci: reorder definitions for grouping functions
2:  85617ef8577 = 2:  77798fa7a7a ci: make grouping setup more generic
3:  57bbc50e3dc = 3:  4542bd38dc2 ci: group installation of Docker dependencies
4:  5ab11d5236d = 4:  5fdda7fd83f ci: split out logic to set up failed test artifacts
-:  ----------- > 5:  6af0075fd87 ci: unify setup of some environment variables
-:  ----------- > 6:  78d863bf24e ci: squelch warnings when testing with unusable Git repo
-:  ----------- > 7:  f150d61a1ce ci: install test dependencies for linux-musl
5:  37a507e9b25 ! 8:  5272d66d9f1 ci: add support for GitLab CI
    @@ Commit message
     
         This commit introduces the integration into our regular CI scripts so
         that most of the setup continues to be shared across all of the CI
    -    solutions.
    +    solutions. Note that as the builds on GitLab CI run as unprivileged
    +    user, we need to pull in both sudo and shadow packages to our Alpine
    +    based job to set this up.
     
         [1]: https://gitlab.com/gitlab-org/git
     
    @@ .gitlab-ci.yml (new)
     +  before_script:
     +    - ./ci/install-docker-dependencies.sh
     +  script:
    -+    - useradd builder --home-dir "${CI_PROJECT_DIR}"
    ++    - useradd builder --create-home
     +    - chown -R builder "${CI_PROJECT_DIR}"
     +    - sudo --preserve-env --set-home --user=builder ./ci/run-build-and-tests.sh
     +  after_script:
    @@ .gitlab-ci.yml (new)
     +      - jobname: linux-asan-ubsan
     +        image: ubuntu:latest
     +        CC: clang
    ++      - jobname: pedantic
    ++        image: fedora:latest
     +      - jobname: linux-musl
     +        image: alpine:latest
     +  artifacts:
    @@ ci/install-docker-dependencies.sh: linux32)
      	;;
      linux-musl)
     -	apk add --update build-base curl-dev openssl-dev expat-dev gettext \
    -+	apk add --update git shadow sudo build-base curl-dev openssl-dev expat-dev gettext \
    - 		pcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null
    ++	apk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \
    + 		pcre2-dev python3 musl-libintl perl-utils ncurses \
    + 		apache2 bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null
      	;;
     +linux-*)
     +	apt update -q &&
    -+	apt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}}
    ++	apt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev \
    ++		libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev \
    ++		perl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl \
    ++		libdbd-sqlite3-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}} \
    ++		apache2 cvs cvsps gnupg libcgi-pm-perl subversion
     +	;;
      pedantic)
      	dnf -yq update >/dev/null &&
    @@ ci/lib.sh: then
     +then
     +	begin_group () {
     +		need_to_end_group=t
    -+		echo -e "\e[0Ksection_start:$(date +%s):$(echo "$1" | tr ' ' _)\r\e[0K$1"
    ++		printf "\e[0Ksection_start:$(date +%s):$(echo "$1" | tr ' ' _)\r\e[0K$1\n"
     +		trap "end_group '$1'" EXIT
     +		set -x
     +	}
    @@ ci/lib.sh: then
     +		test -n "$need_to_end_group" || return 0
     +		set +x
     +		need_to_end_group=
    -+		echo -e "\e[0Ksection_end:$(date +%s):$(echo "$1" | tr ' ' _)\r\e[0K"
    ++		printf "\e[0Ksection_end:$(date +%s):$(echo "$1" | tr ' ' _)\r\e[0K\n"
     +		trap - EXIT
     +	}
      else
      	begin_group () { :; }
      	end_group () { :; }
     @@ ci/lib.sh: then
    - 	MAKEFLAGS="$MAKEFLAGS --jobs=10"
    - 	test windows != "$CI_OS_NAME" ||
    - 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
    + 	cache_dir="$HOME/none"
    + 
    + 	GIT_TEST_OPTS="--github-workflow-markup"
     +elif test true = "$GITLAB_CI"
     +then
     +	CI_TYPE=gitlab-ci
    @@ ci/lib.sh: then
     +	case "$CI_JOB_IMAGE" in
     +	macos-*)
     +		CI_OS_NAME=osx;;
    -+	alpine:*|ubuntu:*)
    ++	alpine:*|fedora:*|ubuntu:*)
     +		CI_OS_NAME=linux;;
     +	*)
     +		echo "Could not identify OS image" >&2
    @@ ci/lib.sh: then
     +	cache_dir="$HOME/none"
     +
     +	runs_on_pool=$(echo "$CI_JOB_IMAGE" | tr : -)
    -+
    -+	export GIT_PROVE_OPTS="--timer --jobs $(nproc)"
    -+	export GIT_TEST_OPTS="--verbose-log -x"
    -+	MAKEFLAGS="$MAKEFLAGS --jobs=$(nproc)"
    -+	test windows != "$CI_OS_NAME" ||
    -+	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
      else
      	echo "Could not identify CI type" >&2
      	env >&2