{"thread":{"id":"60432","subject":"[PATCH 1/5] ci: reorder definitions for grouping functions","startedAt":"2023-10-26T08:00:12Z","lastAt":"2023-11-09T10:06:27Z","messageCount":101,"participants":["Patrick Steinhardt","Oswald Buddenhagen","Phillip Wood","Junio C Hamano","Dragan Simic","Taylor Blau","Victoria Dye","Jeff King","Christian Couder"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"483883","messageId":"586a8d1003b6559177d238ceda2c6ef2f16cfb8d.1698305961.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698305961.git.ps@pks.im","subject":"[PATCH 1/5] ci: reorder definitions for grouping functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-26T08:00:03Z","receivedAt":"2023-10-26T08:00:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We define a set of grouping functions that are used to group together\noutput in our CI, where these groups then end up as collapsible sections\nin the respective pipeline platform. The way these functions are defined\nis not easily extensible though as we have an up front check for the CI\n_not_ being GitLab Actions, where we define the non-stub logic in the\nelse branch.\n\nReorder the definitions such that we explicitly handle GitHub Actions.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 20 ++++++++++----------\n 1 file changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 6fbb5bade12..eb384f4e952 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -1,16 +1,7 @@\n # Library of functions shared by all CI scripts\n \n-if test true != \"$GITHUB_ACTIONS\"\n+if test true = \"$GITHUB_ACTIONS\"\n then\n-\tbegin_group () { :; }\n-\tend_group () { :; }\n-\n-\tgroup () {\n-\t\tshift\n-\t\t\"$@\"\n-\t}\n-\tset -x\n-else\n \tbegin_group () {\n \t\tneed_to_end_group=t\n \t\techo \"::group::$1\" >&2\n@@ -42,6 +33,15 @@ else\n \t}\n \n \tbegin_group \"CI setup\"\n+else\n+\tbegin_group () { :; }\n+\tend_group () { :; }\n+\n+\tgroup () {\n+\t\tshift\n+\t\t\"$@\"\n+\t}\n+\tset -x\n fi\n \n # Set 'exit on error' for all CI scripts to let the caller know that\n-- \n2.42.0\n\n"},{"id":"483884","messageId":"cover.1698305961.git.ps@pks.im","threadId":"60432","inReplyTo":null,"subject":"[PATCH 0/5] ci: add GitLab CI definition","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-26T07:59:59Z","receivedAt":"2023-10-26T08:00:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series adds GitLab CI definitions to the Git project.\n\nAt GitLab, we're have already and will continue to ramp up our\ninvolvement with the Git project. I myself will be working almost\nexclusively on Git in the context of the reftable reference backend.\nThis has surfaced some issues in our own workflow, and the current CI\nintegration is one of the biggest pain points I personally feel right\nnow.\n\nIt's happened multiple times already that we sent patch series upstream\nthat passed on our own self-made pipeline at GitLab, but that didn't\npass the GitHub Actions pipeline. This is because the latter is a lot\nmore involved than what we have. There are pipelines for:\n\n - Various sanitizers in the form of the linux-leaks and\n linux-asan-ubsan jobs.\n\n - SHA256 in the form of the linux-sha256 job.\n\n - The linux-TEST-vars job, which sets several environment variables\n to non-default values.\n\n - The linux-musl job that tests on Alpine Linux.\n\nWe have none of that. And while we could of course iterate on our own\npipeline definition, the current setup results in quite a convoluted\nworkflow on our side. While I realize that this is not the problem of\nthe Git project, I really hope that we can integrate GitLab CI into the\nGit project.\n\nAnd this is exactly what this patch series does: it adds GitLab-specific\nknowledge to our CI scripts and adds a CI definition that builds on top\nof those scripts. This is rather straight forward, as the scripts\nalready know to discern Azure Pipelines and GitHub Actions, and adding\na third item to this list feels quite natural. And by building on top of\nthe preexisting infra, the actual \".gitlab-ci.yml\" is really quite\nsmall.\n\nI acknowledge that the Git project may not be willing to fully support\nGitLab CI, and that's fine with me. If we want to further stress that\npoint then I'd also be perfectly happy to move the definitions into the\n\"contrib/\" directory -- it would still be a huge win for our workflow.\nIn any case, I'm happy to keep on maintaining the intgeration with\nGitLab CI, and if things break I'll do my best to fix them fast.\n\nI hope that this sheds some light on my motivations here. I do not wish\nto replace GitHub Actions and would be okay if this was only a\nsemi-supported thing. But it would help us at GitLab and by extension\nalso the Git project because we will hopefully send higher-quality patch\nseries to the mailing list. And maybe this is even useful to somebody\noutside of GitLab.\n\nIf this is accepted I'll likely eventually iterate to also support macOS\nand/or Windows. A full pipeline run of this can be found at [1].\n\nPatrick\n\n[1]: https://gitlab.com/gitlab-org/git/-/pipelines/1045746751\n\nPatrick Steinhardt (5):\n ci: reorder definitions for grouping functions\n ci: make grouping setup more generic\n ci: group installation of Docker dependencies\n ci: split out logic to set up failed test artifacts\n ci: add support for GitLab CI\n\n .gitlab-ci.yml | 51 +++++++++++\n ci/install-docker-dependencies.sh | 15 +++-\n ci/lib.sh | 139 +++++++++++++++++++++++-------\n ci/print-test-failures.sh | 6 ++\n 4 files changed, 179 insertions(+), 32 deletions(-)\n create mode 100644 .gitlab-ci.yml\n\n-- \n2.42.0\n\n"},{"id":"483885","messageId":"ec390354f15604eb440bec0d3d0d817441e0f5b2.1698305961.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698305961.git.ps@pks.im","subject":"[PATCH 2/5] ci: make grouping setup more generic","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-26T08:00:08Z","receivedAt":"2023-10-26T08:00:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Make the grouping setup more generic by always calling `begin_group ()`\nand `end_group ()` regardless of whether we have stubbed those functions\nor not. This ensures we can more readily add support for additional CI\nplatforms.\n\nSecond, this commit changes `end_group ()` to also accept a parameter\nthat indicates _which_ group should end. This will be required by a\nlater commit that introduces support for GitLab CI.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 16 ++++++++++------\n 1 file changed, 10 insertions(+), 6 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex eb384f4e952..957fd152d9c 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -14,12 +14,14 @@ then\n \t\tneed_to_end_group=\n \t\techo '::endgroup::' >&2\n \t}\n-\ttrap end_group EXIT\n \n \tgroup () {\n \t\tset +x\n-\t\tbegin_group \"$1\"\n+\n+\t\tgroup=\"$1\"\n \t\tshift\n+\t\tbegin_group \"$group\"\n+\n \t\t# work around `dash` not supporting `set -o pipefail`\n \t\t(\n \t\t\t\"$@\" 2>&1\n@@ -28,11 +30,10 @@ then\n \t\tsed 's/^\\(\\([^ ]*\\):\\([0-9]*\\):\\([0-9]*:\\) \\)\\(error\\|warning\\): /::\\5 file=\\2,line=\\3::\\1/'\n \t\tres=$(cat exit.status)\n \t\trm exit.status\n-\t\tend_group\n+\n+\t\tend_group \"$group\"\n \t\treturn $res\n \t}\n-\n-\tbegin_group \"CI setup\"\n else\n \tbegin_group () { :; }\n \tend_group () { :; }\n@@ -44,6 +45,9 @@ else\n \tset -x\n fi\n \n+begin_group \"CI setup\"\n+trap \"end_group 'CI setup'\" EXIT\n+\n # Set 'exit on error' for all CI scripts to let the caller know that\n # something went wrong.\n #\n@@ -287,5 +291,5 @@ esac\n \n MAKEFLAGS=\"$MAKEFLAGS CC=${CC:-cc}\"\n \n-end_group\n+end_group \"CI setup\"\n set -x\n-- \n2.42.0\n\n"},{"id":"483886","messageId":"a65d235dd3c14df4945b9753507d9cdab777966c.1698305961.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698305961.git.ps@pks.im","subject":"[PATCH 3/5] ci: group installation of Docker dependencies","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-26T08:00:12Z","receivedAt":"2023-10-26T08:00:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Pull in \"lib.sh\" into \"install-docker-dependencies.sh\" such that we can\nset up proper groups for those dependencise. This allows the reader to\ncollapse sections in the CI output on GitHub Actions (and later on on\nGitLab CI).\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-docker-dependencies.sh | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\nindex 78b7e326da6..d0bc19d3bb3 100755\n--- a/ci/install-docker-dependencies.sh\n+++ b/ci/install-docker-dependencies.sh\n@@ -3,6 +3,10 @@\n # Install dependencies required to build and test Git inside container\n #\n \n+. ${0%/*}/lib.sh\n+\n+begin_group \"Install dependencies\"\n+\n case \"$jobname\" in\n linux32)\n \tlinux32 --32bit i386 sh -c '\n@@ -20,3 +24,5 @@ pedantic)\n \tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n \t;;\n esac\n+\n+end_group \"Install dependencies\"\n-- \n2.42.0\n\n"},{"id":"483887","messageId":"4a864a1d174f7d4d36202a375302d450fbe9f2a3.1698305961.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698305961.git.ps@pks.im","subject":"[PATCH 4/5] ci: split out logic to set up failed test artifacts","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-26T08:00:16Z","receivedAt":"2023-10-26T08:00:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have some logic in place to create a directory with the output from\nfailed tests, which will then subsequently be uploaded as CI artifact.\nWe're about to add support for GitLab CI, which will want to reuse the\nlogic.\n\nSplit the logic into a separate function so that it is reusable.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 40 ++++++++++++++++++++++------------------\n 1 file changed, 22 insertions(+), 18 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 957fd152d9c..33005854520 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -137,6 +137,27 @@ handle_failed_tests () {\n \treturn 1\n }\n \n+create_failed_test_artifacts () {\n+\tmkdir -p t/failed-test-artifacts\n+\n+\tfor test_exit in t/test-results/*.exit\n+\tdo\n+\t\ttest 0 != \"$(cat \"$test_exit\")\" || continue\n+\n+\t\ttest_name=\"${test_exit%.exit}\"\n+\t\ttest_name=\"${test_name##*/}\"\n+\t\tprintf \"\\\\e[33m\\\\e[1m=== Failed test: ${test_name} ===\\\\e[m\\\\n\"\n+\t\techo \"The full logs are in the 'print test failures' step below.\"\n+\t\techo \"See also the 'failed-tests-*' artifacts attached to this run.\"\n+\t\tcat \"t/test-results/$test_name.markup\"\n+\n+\t\ttrash_dir=\"t/trash directory.$test_name\"\n+\t\tcp \"t/test-results/$test_name.out\" t/failed-test-artifacts/\n+\t\ttar czf t/failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n+\tdone\n+\treturn 1\n+}\n+\n # GitHub Action doesn't set TERM, which is required by tput\n export TERM=${TERM:-dumb}\n \n@@ -177,25 +198,8 @@ then\n \tCC=\"${CC_PACKAGE:-${CC:-gcc}}\"\n \tDONT_SKIP_TAGS=t\n \thandle_failed_tests () {\n-\t\tmkdir -p t/failed-test-artifacts\n \t\techo \"FAILED_TEST_ARTIFACTS=t/failed-test-artifacts\" >>$GITHUB_ENV\n-\n-\t\tfor test_exit in t/test-results/*.exit\n-\t\tdo\n-\t\t\ttest 0 != \"$(cat \"$test_exit\")\" || continue\n-\n-\t\t\ttest_name=\"${test_exit%.exit}\"\n-\t\t\ttest_name=\"${test_name##*/}\"\n-\t\t\tprintf \"\\\\e[33m\\\\e[1m=== Failed test: ${test_name} ===\\\\e[m\\\\n\"\n-\t\t\techo \"The full logs are in the 'print test failures' step below.\"\n-\t\t\techo \"See also the 'failed-tests-*' artifacts attached to this run.\"\n-\t\t\tcat \"t/test-results/$test_name.markup\"\n-\n-\t\t\ttrash_dir=\"t/trash directory.$test_name\"\n-\t\t\tcp \"t/test-results/$test_name.out\" t/failed-test-artifacts/\n-\t\t\ttar czf t/failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n-\t\tdone\n-\t\treturn 1\n+\t\tcreate_failed_test_artifacts\n \t}\n \n \tcache_dir=\"$HOME/none\"\n-- \n2.42.0\n\n"},{"id":"483888","messageId":"35b07e5378d960b93ae8990a3abb525e1762d97d.1698305961.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698305961.git.ps@pks.im","subject":"[PATCH 5/5] ci: add support for GitLab CI","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-26T08:00:20Z","receivedAt":"2023-10-26T08:00:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We already support Azure Pipelines and GitHub Workflows in the Git\nproject, but until now we do not have support for GitLab CI. While it is\narguably not in the interest of the Git project to maintain a ton of\ndifferent CI platforms, GitLab has recently ramped up its efforts and\ntries to contribute to the Git project more regularly.\n\nPart of a problem we hit at GitLab rather frequently is that our own,\ncustom CI setup we have is so different to the setup that the Git\nproject has. More esoteric jobs like \"linux-TEST-vars\" that also sets a\ncouple of environment variables do not exist in GitLab's custom CI\nsetup, and maintaining them to keep up with what Git does feels like\nwasted time. The result is that we regularly send patch series upstream\nthat would otherwise fail to compile or pass tests in GitHub Workflows.\nWe would thus like to integrate the GitLab CI configuration into the Git\nproject to help us ensure to send better patch series upstream and thus\nreduce overhead for the maintainer.\n\nThe integration does not necessarily have to be a first-class citizen,\nwhich would in practice only add to the fallout that pipeline failures\nhave for the maintainer. That being said, we are happy to maintain this\nalternative CI setup for the Git project and will make test results\navailable as part of our own mirror of the Git project at [1].\n\nThis commit introduces the integration into our regular CI scripts so\nthat most of the setup continues to be shared across all of the CI\nsolutions.\n\n[1]: https://gitlab.com/gitlab-org/git\n\nSigned-off-by: Patrick Steinhardt \n---\n .gitlab-ci.yml | 51 +++++++++++++++++++++++\n ci/install-docker-dependencies.sh | 9 +++-\n ci/lib.sh | 69 +++++++++++++++++++++++++++++++\n ci/print-test-failures.sh | 6 +++\n 4 files changed, 134 insertions(+), 1 deletion(-)\n create mode 100644 .gitlab-ci.yml\n\ndiff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\nnew file mode 100644\nindex 00000000000..43d3a961fa0\n--- /dev/null\n+++ b/.gitlab-ci.yml\n@@ -0,0 +1,51 @@\n+default:\n+ timeout: 2h\n+\n+workflow:\n+ rules:\n+ - if: $CI_PIPELINE_SOURCE == \"merge_request_event\"\n+ - if: $CI_COMMIT_TAG\n+ - if: $CI_COMMIT_REF_PROTECTED == \"true\"\n+\n+test:\n+ image: $image\n+ before_script:\n+ - ./ci/install-docker-dependencies.sh\n+ script:\n+ - useradd builder --home-dir \"${CI_PROJECT_DIR}\"\n+ - chown -R builder \"${CI_PROJECT_DIR}\"\n+ - sudo --preserve-env --set-home --user=builder ./ci/run-build-and-tests.sh\n+ after_script:\n+ - |\n+ if test \"$CI_JOB_STATUS\" != 'success'\n+ then\n+ sudo --preserve-env --set-home --user=builder ./ci/print-test-failures.sh\n+ fi\n+ parallel:\n+ matrix:\n+ - jobname: linux-sha256\n+ image: ubuntu:latest\n+ CC: clang\n+ - jobname: linux-gcc\n+ image: ubuntu:20.04\n+ CC: gcc\n+ CC_PACKAGE: gcc-8\n+ - jobname: linux-TEST-vars\n+ image: ubuntu:20.04\n+ CC: gcc\n+ CC_PACKAGE: gcc-8\n+ - jobname: linux-gcc-default\n+ image: ubuntu:latest\n+ CC: gcc\n+ - jobname: linux-leaks\n+ image: ubuntu:latest\n+ CC: gcc\n+ - jobname: linux-asan-ubsan\n+ image: ubuntu:latest\n+ CC: clang\n+ - jobname: linux-musl\n+ image: alpine:latest\n+ artifacts:\n+ paths:\n+ - t/failed-test-artifacts\n+ when: on_failure\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\nindex d0bc19d3bb3..1cd92db1876 100755\n--- a/ci/install-docker-dependencies.sh\n+++ b/ci/install-docker-dependencies.sh\n@@ -7,6 +7,9 @@\n \n begin_group \"Install dependencies\"\n \n+# Required so that apt doesn't wait for user input on certain packages.\n+export DEBIAN_FRONTEND=noninteractive\n+\n case \"$jobname\" in\n linux32)\n \tlinux32 --32bit i386 sh -c '\n@@ -16,9 +19,13 @@ linux32)\n \t'\n \t;;\n linux-musl)\n-\tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n+\tapk add --update git shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n \t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n \t;;\n+linux-*)\n+\tapt update -q &&\n+\tapt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}}\n+\t;;\n pedantic)\n \tdnf -yq update >/dev/null &&\n \tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 33005854520..102e9d04a1f 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -15,6 +15,42 @@ then\n \t\techo '::endgroup::' >&2\n \t}\n \n+\tgroup () {\n+\t\tset +x\n+\n+\t\tgroup=\"$1\"\n+\t\tshift\n+\t\tbegin_group \"$group\"\n+\n+\t\t# work around `dash` not supporting `set -o pipefail`\n+\t\t(\n+\t\t\t\"$@\" 2>&1\n+\t\t\techo $? >exit.status\n+\t\t) |\n+\t\tsed 's/^\\(\\([^ ]*\\):\\([0-9]*\\):\\([0-9]*:\\) \\)\\(error\\|warning\\): /::\\5 file=\\2,line=\\3::\\1/'\n+\t\tres=$(cat exit.status)\n+\t\trm exit.status\n+\n+\t\tend_group \"$group\"\n+\t\treturn $res\n+\t}\n+elif test true = \"$GITLAB_CI\"\n+then\n+\tbegin_group () {\n+\t\tneed_to_end_group=t\n+\t\techo -e \"\\e[0Ksection_start:$(date +%s):$(echo \"$1\" | tr ' ' _)\\r\\e[0K$1\"\n+\t\ttrap \"end_group '$1'\" EXIT\n+\t\tset -x\n+\t}\n+\n+\tend_group () {\n+\t\ttest -n \"$need_to_end_group\" || return 0\n+\t\tset +x\n+\t\tneed_to_end_group=\n+\t\techo -e \"\\e[0Ksection_end:$(date +%s):$(echo \"$1\" | tr ' ' _)\\r\\e[0K\"\n+\t\ttrap - EXIT\n+\t}\n+\n \tgroup () {\n \t\tset +x\n \n@@ -209,6 +245,39 @@ then\n \tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n \ttest windows != \"$CI_OS_NAME\" ||\n \tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n+elif test true = \"$GITLAB_CI\"\n+then\n+\tCI_TYPE=gitlab-ci\n+\tCI_BRANCH=\"$CI_COMMIT_REF_NAME\"\n+\tCI_COMMIT=\"$CI_COMMIT_SHA\"\n+\tcase \"$CI_JOB_IMAGE\" in\n+\tmacos-*)\n+\t\tCI_OS_NAME=osx;;\n+\talpine:*|ubuntu:*)\n+\t\tCI_OS_NAME=linux;;\n+\t*)\n+\t\techo \"Could not identify OS image\" >&2\n+\t\tenv >&2\n+\t\texit 1\n+\t\t;;\n+\tesac\n+\tCI_REPO_SLUG=\"$CI_PROJECT_PATH\"\n+\tCI_JOB_ID=\"$CI_JOB_ID\"\n+\tCC=\"${CC_PACKAGE:-${CC:-gcc}}\"\n+\tDONT_SKIP_TAGS=t\n+\thandle_failed_tests () {\n+\t\tcreate_failed_test_artifacts\n+\t}\n+\n+\tcache_dir=\"$HOME/none\"\n+\n+\truns_on_pool=$(echo \"$CI_JOB_IMAGE\" | tr : -)\n+\n+\texport GIT_PROVE_OPTS=\"--timer --jobs $(nproc)\"\n+\texport GIT_TEST_OPTS=\"--verbose-log -x\"\n+\tMAKEFLAGS=\"$MAKEFLAGS --jobs=$(nproc)\"\n+\ttest windows != \"$CI_OS_NAME\" ||\n+\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n else\n \techo \"Could not identify CI type\" >&2\n \tenv >&2\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex 57277eefcd0..c33ad4e3a22 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -51,6 +51,12 @@ do\n \t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n \t\t\tcontinue\n \t\t\t;;\n+\t\tgitlab-ci)\n+\t\t\tmkdir -p failed-test-artifacts\n+\t\t\tcp \"${TEST_EXIT%.exit}.out\" failed-test-artifacts/\n+\t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n+\t\t\tcontinue\n+\t\t\t;;\n \t\t*)\n \t\t\techo \"Unhandled CI type: $CI_TYPE\" >&2\n \t\t\texit 1\n-- \n2.42.0\n\n"},{"id":"483890","messageId":"ZToin/fQiZrmUJTS@ugly","threadId":"60432","inReplyTo":"586a8d1003b6559177d238ceda2c6ef2f16cfb8d.1698305961.git.ps@pks.im","subject":"Re: [PATCH 1/5] ci: reorder definitions for grouping functions","fromName":"Oswald Buddenhagen","fromEmail":"oswald.buddenhagen@gmx.de","sentAt":"2023-10-26T08:26:07Z","receivedAt":"2023-10-26T08:26:11Z","isPatch":true,"sender":{"key":"oswald.buddenhagen@gmx.de","avatar":"https://avatars.githubusercontent.com/u/812380?v=4"},"body":"On Thu, Oct 26, 2023 at 10:00:03AM +0200, Patrick Steinhardt wrote:\n> [...]\n>_not_ being GitLab Actions, where we define the non-stub logic in the\n>\nyou meant GitHub here.\n\n>else branch.\n>\n>Reorder the definitions such that we explicitly handle GitHub Actions.\n>\ni'd say something like \"the conditional branches\". imo that makes it \nclearer that you're actually talking about code, not some markup or \nwhatever.\nfor that matter, this is my overall impression of the commit message - \nit seems way too detached from the near-trivial fact that you're just \nslightly adjusting the code structure to make it easier to implement a \ncascade (aka a switch).\n\nregards\n"},{"id":"483891","messageId":"ZTokgo1mQ7ZVH7GU@ugly","threadId":"60432","inReplyTo":"a65d235dd3c14df4945b9753507d9cdab777966c.1698305961.git.ps@pks.im","subject":"Re: [PATCH 3/5] ci: group installation of Docker dependencies","fromName":"Oswald Buddenhagen","fromEmail":"oswald.buddenhagen@gmx.de","sentAt":"2023-10-26T08:34:10Z","receivedAt":"2023-10-26T08:34:13Z","isPatch":true,"sender":{"key":"oswald.buddenhagen@gmx.de","avatar":"https://avatars.githubusercontent.com/u/812380?v=4"},"body":"On Thu, Oct 26, 2023 at 10:00:12AM +0200, Patrick Steinhardt wrote:\n>Pull in \"lib.sh\" into \"install-docker-dependencies.sh\" such that we can\n>set up proper groups for those dependencise. This allows the reader to\n\t\t\t\t\t ^^ !!!\n>collapse sections in the CI output on GitHub Actions (and later on on\n>GitLab CI).\n>\nthe structure of the text is kind of backwards - the fact that you need \nto pull in a lib is just a consequence, not the intent, which imo should \ncome first. tough it mostly doesn't matter, as it's just one short \nparagraph.\n\nregards\n"},{"id":"483892","messageId":"ZTok5cDVT7ug/GCc@ugly","threadId":"60432","inReplyTo":"4a864a1d174f7d4d36202a375302d450fbe9f2a3.1698305961.git.ps@pks.im","subject":"Re: [PATCH 4/5] ci: split out logic to set up failed test artifacts","fromName":"Oswald Buddenhagen","fromEmail":"oswald.buddenhagen@gmx.de","sentAt":"2023-10-26T08:35:49Z","receivedAt":"2023-10-26T08:35:52Z","isPatch":true,"sender":{"key":"oswald.buddenhagen@gmx.de","avatar":"https://avatars.githubusercontent.com/u/812380?v=4"},"body":"On Thu, Oct 26, 2023 at 10:00:16AM +0200, Patrick Steinhardt wrote:\n>We have some logic in place to create a directory with the output from\n>failed tests, which will then subsequently be uploaded as CI artifact.\n\t\t\t\t\t\t\t\t ^ \n\t\t\t\t\t\t\t\t +s\n\nregards\n"},{"id":"483893","messageId":"ZTosPCkpx/FMTDH5@ugly","threadId":"60432","inReplyTo":"35b07e5378d960b93ae8990a3abb525e1762d97d.1698305961.git.ps@pks.im","subject":"Re: [PATCH 5/5] ci: add support for GitLab CI","fromName":"Oswald Buddenhagen","fromEmail":"oswald.buddenhagen@gmx.de","sentAt":"2023-10-26T09:07:08Z","receivedAt":"2023-10-26T09:07:12Z","isPatch":true,"sender":{"key":"oswald.buddenhagen@gmx.de","avatar":"https://avatars.githubusercontent.com/u/812380?v=4"},"body":"On Thu, Oct 26, 2023 at 10:00:20AM +0200, Patrick Steinhardt wrote:\n>project has. More esoteric jobs like \"linux-TEST-vars\" that also sets a\n\t\t\t\t\t\t\t\t ^ \n\t\t\t\t\t\t\t\t -s\n\n>couple of environment variables do not exist in GitLab's custom CI\n>setup, and maintaining them to keep up with what Git does feels like\n>wasted time. The result is that we regularly send patch series upstream\n>that would otherwise fail to compile or pass tests in GitHub Workflows.\n ^^^^^^^^^^^^^^^\n that inverts the meaning\n\n>[...]\n>project to help us ensure to send better patch series upstream and thus\n\t\t\t ^^\n\t\t\t \"we\"\n>reduce overhead for the maintainer.\n\n>--- a/ci/install-docker-dependencies.sh\n>+++ b/ci/install-docker-dependencies.sh\n>@@ -16,9 +19,13 @@ linux32)\n> \t'\n> \t;;\n> linux-musl)\n>-\tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n>+\tapk add --update git shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n> \t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n> \t;;\n>+linux-*)\n>\nyou should probably choose a less generic name for the jobs, at least \ndebian-*.\n\n>diff --git a/ci/lib.sh b/ci/lib.sh\n>index 33005854520..102e9d04a1f 100755\n>--- a/ci/lib.sh\n>+++ b/ci/lib.sh\n>@@ -15,6 +15,42 @@ then\n> \t\techo '::endgroup::' >&2\n> \t}\n> \n>+\tgroup () {\n>[...]\n>+\t}\n>\nthis counter-intutive patch structure reveals that the function is \nduplicated between github and gitlab. you may want to factor it out and \nalias it. or change the structure entirely (circling back to patch 1/5).\n\n>+\tCI_BRANCH=\"$CI_COMMIT_REF_NAME\"\n>+\tCI_COMMIT=\"$CI_COMMIT_SHA\"\n>\nassignments need no quoting to prevent word splitting.\nrepeats below.\n\n>+\tcase \"$CI_JOB_IMAGE\" in\n>\n... as does the selector in case statements.\n\n>--- a/ci/print-test-failures.sh\n>+++ b/ci/print-test-failures.sh\n>@@ -51,6 +51,12 @@ do\n> \t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n> \t\t\tcontinue\n> \t\t\t;;\n>+\t\tgitlab-ci)\n>+\t\t\tmkdir -p failed-test-artifacts\n>+\t\t\tcp \"${TEST_EXIT%.exit}.out\" failed-test-artifacts/\n>+\t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n>\nyou're just following the precedent, but imo it's more legible to quote \nthe entire string, not just the variable expansion. the code doesn't \neven agree with itself, as the line directly above apparently agrees \nwith me.\n\nregards\n"},{"id":"483955","messageId":"ZTtyB0hgXN8gO-Tn@tanuki","threadId":"60432","inReplyTo":"ZToin/fQiZrmUJTS@ugly","subject":"Re: [PATCH 1/5] ci: reorder definitions for grouping functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-27T08:17:21Z","receivedAt":"2023-10-27T08:17:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Oct 26, 2023 at 10:26:07AM +0200, Oswald Buddenhagen wrote:\n> On Thu, Oct 26, 2023 at 10:00:03AM +0200, Patrick Steinhardt wrote:\n> > [...]\n> > _not_ being GitLab Actions, where we define the non-stub logic in the\n> > \n> you meant GitHub here.\n> \n> > else branch.\n> > \n> > Reorder the definitions such that we explicitly handle GitHub Actions.\n> > \n> i'd say something like \"the conditional branches\". imo that makes it clearer\n> that you're actually talking about code, not some markup or whatever.\n> for that matter, this is my overall impression of the commit message - it\n> seems way too detached from the near-trivial fact that you're just slightly\n> adjusting the code structure to make it easier to implement a cascade (aka a\n> switch).\n> \n> regards\n\nThe change is trivial indeed, but even a trivial change needs a reason\nwhy it should be done. Maybe it's too long, maybe it isn't... I'm happy\nto take suggestions.\n\nBut anyway, I've adopted both of the other two suggestions you made,\nthanks.\n\nPatrick\n"},{"id":"483956","messageId":"ZTtyGBLvH8Fji-C-@tanuki","threadId":"60432","inReplyTo":"ZTokgo1mQ7ZVH7GU@ugly","subject":"Re: [PATCH 3/5] ci: group installation of Docker dependencies","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-27T08:17:28Z","receivedAt":"2023-10-27T08:17:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Oct 26, 2023 at 10:34:10AM +0200, Oswald Buddenhagen wrote:\n> On Thu, Oct 26, 2023 at 10:00:12AM +0200, Patrick Steinhardt wrote:\n> > Pull in \"lib.sh\" into \"install-docker-dependencies.sh\" such that we can\n> > set up proper groups for those dependencise. This allows the reader to\n> \t\t\t\t\t ^^ !!!\n> > collapse sections in the CI output on GitHub Actions (and later on on\n> > GitLab CI).\n> > \n> the structure of the text is kind of backwards - the fact that you need to\n> pull in a lib is just a consequence, not the intent, which imo should come\n> first. tough it mostly doesn't matter, as it's just one short paragraph.\n> \n> regards\n\nGood point indeed. Will rewrite.\n\nPatrick\n"},{"id":"483957","messageId":"ZTtyHScKkMg4qnMH@tanuki","threadId":"60432","inReplyTo":"ZTosPCkpx/FMTDH5@ugly","subject":"Re: [PATCH 5/5] ci: add support for GitLab CI","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-27T08:17:33Z","receivedAt":"2023-10-27T08:17:38Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Oct 26, 2023 at 11:07:08AM +0200, Oswald Buddenhagen wrote:\n> On Thu, Oct 26, 2023 at 10:00:20AM +0200, Patrick Steinhardt wrote:\n> > project has. More esoteric jobs like \"linux-TEST-vars\" that also sets a\n> \t\t\t\t\t\t\t\t ^ \t\t\t\t\t\t\t\t -s\n> \n> > couple of environment variables do not exist in GitLab's custom CI\n> > setup, and maintaining them to keep up with what Git does feels like\n> > wasted time. The result is that we regularly send patch series upstream\n> > that would otherwise fail to compile or pass tests in GitHub Workflows.\n> ^^^^^^^^^^^^^^^\n> that inverts the meaning\n> \n> > [...]\n> > project to help us ensure to send better patch series upstream and thus\n> \t\t\t ^^\n> \t\t\t \"we\"\n> > reduce overhead for the maintainer.\n> \n> > --- a/ci/install-docker-dependencies.sh\n> > +++ b/ci/install-docker-dependencies.sh\n> > @@ -16,9 +19,13 @@ linux32)\n> > \t'\n> > \t;;\n> > linux-musl)\n> > -\tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n> > +\tapk add --update git shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n> > \t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n> > \t;;\n> > +linux-*)\n> > \n> you should probably choose a less generic name for the jobs, at least\n> debian-*.\n\nThe names are all preexisting, so I cannot change them. And the CI infra\ndoes indeed rely on the exact names to choose what to do.\n\n> > diff --git a/ci/lib.sh b/ci/lib.sh\n> > index 33005854520..102e9d04a1f 100755\n> > --- a/ci/lib.sh\n> > +++ b/ci/lib.sh\n> > @@ -15,6 +15,42 @@ then\n> > \t\techo '::endgroup::' >&2\n> > \t}\n> > \n> > +\tgroup () {\n> > [...]\n> > +\t}\n> > \n> this counter-intutive patch structure reveals that the function is\n> duplicated between github and gitlab. you may want to factor it out and\n> alias it. or change the structure entirely (circling back to patch 1/5).\n\nI don't quite know what you mean by counter-intuitive patch structure.\nBut regarding the duplication for the `group ()` function I agree, it's\na bit unfortunate. My first version did de-duplicate it indeed, but it\nresulted in some weirdness in the stubbed case.\n\nI'll revisit this.\n\n> > +\tCI_BRANCH=\"$CI_COMMIT_REF_NAME\"\n> > +\tCI_COMMIT=\"$CI_COMMIT_SHA\"\n> > \n> assignments need no quoting to prevent word splitting.\n> repeats below.\n> \n> > +\tcase \"$CI_JOB_IMAGE\" in\n> > \n> ... as does the selector in case statements.\n\nTrue, but I'm simply matching the coding style in this script.\n\n> > --- a/ci/print-test-failures.sh\n> > +++ b/ci/print-test-failures.sh\n> > @@ -51,6 +51,12 @@ do\n> > \t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n> > \t\t\tcontinue\n> > \t\t\t;;\n> > +\t\tgitlab-ci)\n> > +\t\t\tmkdir -p failed-test-artifacts\n> > +\t\t\tcp \"${TEST_EXIT%.exit}.out\" failed-test-artifacts/\n> > +\t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n> > \n> you're just following the precedent, but imo it's more legible to quote the\n> entire string, not just the variable expansion. the code doesn't even agree\n> with itself, as the line directly above apparently agrees with me.\n> \n> regards\n\nYeah, as you say, this is another case where I follow precedent. I\nhonestly don't quite care which way we go in this case.\n\nPatrick\n"},{"id":"483958","messageId":"4eb9cfc816b160a8fbce1bbec03df10f82c83dfc.1698398590.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698398590.git.ps@pks.im","subject":"[PATCH v2 1/5] ci: reorder definitions for grouping functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-27T09:25:24Z","receivedAt":"2023-10-27T09:25:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We define a set of grouping functions that are used to group together\noutput in our CI, where these groups then end up as collapsible sections\nin the respective pipeline platform. The way these functions are defined\nis not easily extensible though as we have an up front check for the CI\n_not_ being GitHub Actions, where we define the non-stub logic in the\nelse branch.\n\nReorder the conditional branches such that we explicitly handle GitHub\nActions.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 20 ++++++++++----------\n 1 file changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 6fbb5bade12..eb384f4e952 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -1,16 +1,7 @@\n # Library of functions shared by all CI scripts\n \n-if test true != \"$GITHUB_ACTIONS\"\n+if test true = \"$GITHUB_ACTIONS\"\n then\n-\tbegin_group () { :; }\n-\tend_group () { :; }\n-\n-\tgroup () {\n-\t\tshift\n-\t\t\"$@\"\n-\t}\n-\tset -x\n-else\n \tbegin_group () {\n \t\tneed_to_end_group=t\n \t\techo \"::group::$1\" >&2\n@@ -42,6 +33,15 @@ else\n \t}\n \n \tbegin_group \"CI setup\"\n+else\n+\tbegin_group () { :; }\n+\tend_group () { :; }\n+\n+\tgroup () {\n+\t\tshift\n+\t\t\"$@\"\n+\t}\n+\tset -x\n fi\n \n # Set 'exit on error' for all CI scripts to let the caller know that\n-- \n2.42.0\n\n"},{"id":"483959","messageId":"cover.1698398590.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698305961.git.ps@pks.im","subject":"[PATCH v2 0/5] ci: add GitLab CI definition","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-27T09:25:20Z","receivedAt":"2023-10-27T09:25:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the second version of this patch series that adds GitLab CI\ndefinitions to the Git project. Please refer to the cover letter for v1\nof this series [1] for the motivation and intent -- I won't repeat it\nhere as it's a bit on the longer side.\n\nChanges compared to v1:\n\n - Improved commit messages.\n\n - The `group ()` function is now generic across all the different CI\n solutions to reduce duplication.\n\nThis should hopefully address the feedback from Oswald -- thanks again!\n\nPatrick\n\n[1]: \n\n\nPatrick Steinhardt (5):\n ci: reorder definitions for grouping functions\n ci: make grouping setup more generic\n ci: group installation of Docker dependencies\n ci: split out logic to set up failed test artifacts\n ci: add support for GitLab CI\n\n .gitlab-ci.yml | 51 +++++++++++\n ci/install-docker-dependencies.sh | 15 +++-\n ci/lib.sh | 139 ++++++++++++++++++++----------\n ci/print-test-failures.sh | 6 ++\n 4 files changed, 166 insertions(+), 45 deletions(-)\n create mode 100644 .gitlab-ci.yml\n\nRange-diff against v1:\n1: 586a8d1003b ! 1: 4eb9cfc816b ci: reorder definitions for grouping functions\n @@ Commit message\n output in our CI, where these groups then end up as collapsible sections\n in the respective pipeline platform. The way these functions are defined\n is not easily extensible though as we have an up front check for the CI\n - _not_ being GitLab Actions, where we define the non-stub logic in the\n + _not_ being GitHub Actions, where we define the non-stub logic in the\n else branch.\n \n - Reorder the definitions such that we explicitly handle GitHub Actions.\n + Reorder the conditional branches such that we explicitly handle GitHub\n + Actions.\n \n Signed-off-by: Patrick Steinhardt \n \n2: ec390354f15 ! 2: 85617ef8577 ci: make grouping setup more generic\n @@ Commit message\n or not. This ensures we can more readily add support for additional CI\n platforms.\n \n - Second, this commit changes `end_group ()` to also accept a parameter\n - that indicates _which_ group should end. This will be required by a\n - later commit that introduces support for GitLab CI.\n + Furthermore, the `group ()` function is made generic so that it is the\n + same for both GitHub Actions and for other platforms. There is a\n + semantic conflict here though: GitHub Actions used to call `set +x` in\n + `group ()` whereas the non-GitHub case unconditionally uses `set -x`.\n + The latter would get overriden if we kept the `set +x` in the generic\n + version of `group ()`. To resolve this conflict, we simply drop the `set\n + +x` in the generic variant of this function. As `begin_group ()` calls\n + `set -x` anyway this is not much of a change though, as the only\n + commands that aren't printed anymore now are the ones between the\n + beginning of `group ()` and the end of `begin_group ()`.\n +\n + Last, this commit changes `end_group ()` to also accept a parameter that\n + indicates _which_ group should end. This will be required by a later\n + commit that introduces support for GitLab CI.\n \n Signed-off-by: Patrick Steinhardt \n \n @@ ci/lib.sh: then\n \t\techo '::endgroup::' >&2\n \t}\n -\ttrap end_group EXIT\n - \n - \tgroup () {\n - \t\tset +x\n +-\n +-\tgroup () {\n +-\t\tset +x\n -\t\tbegin_group \"$1\"\n -+\n -+\t\tgroup=\"$1\"\n - \t\tshift\n -+\t\tbegin_group \"$group\"\n -+\n - \t\t# work around `dash` not supporting `set -o pipefail`\n - \t\t(\n - \t\t\t\"$@\" 2>&1\n -@@ ci/lib.sh: then\n - \t\tsed 's/^\\(\\([^ ]*\\):\\([0-9]*\\):\\([0-9]*:\\) \\)\\(error\\|warning\\): /::\\5 file=\\2,line=\\3::\\1/'\n - \t\tres=$(cat exit.status)\n - \t\trm exit.status\n +-\t\tshift\n +-\t\t# work around `dash` not supporting `set -o pipefail`\n +-\t\t(\n +-\t\t\t\"$@\" 2>&1\n +-\t\t\techo $? >exit.status\n +-\t\t) |\n +-\t\tsed 's/^\\(\\([^ ]*\\):\\([0-9]*\\):\\([0-9]*:\\) \\)\\(error\\|warning\\): /::\\5 file=\\2,line=\\3::\\1/'\n +-\t\tres=$(cat exit.status)\n +-\t\trm exit.status\n -\t\tend_group\n -+\n -+\t\tend_group \"$group\"\n - \t\treturn $res\n - \t}\n +-\t\treturn $res\n +-\t}\n -\n -\tbegin_group \"CI setup\"\n else\n \tbegin_group () { :; }\n \tend_group () { :; }\n -@@ ci/lib.sh: else\n + \n +-\tgroup () {\n +-\t\tshift\n +-\t\t\"$@\"\n +-\t}\n \tset -x\n fi\n \n ++group () {\n ++\tgroup=\"$1\"\n ++\tshift\n ++\tbegin_group \"$group\"\n ++\n ++\t# work around `dash` not supporting `set -o pipefail`\n ++\t(\n ++\t\t\"$@\" 2>&1\n ++\t\techo $? >exit.status\n ++\t) |\n ++\tsed 's/^\\(\\([^ ]*\\):\\([0-9]*\\):\\([0-9]*:\\) \\)\\(error\\|warning\\): /::\\5 file=\\2,line=\\3::\\1/'\n ++\tres=$(cat exit.status)\n ++\trm exit.status\n ++\n ++\tend_group \"$group\"\n ++\treturn $res\n ++}\n ++\n +begin_group \"CI setup\"\n +trap \"end_group 'CI setup'\" EXIT\n +\n3: a65d235dd3c ! 3: 57bbc50e3dc ci: group installation of Docker dependencies\n @@ Metadata\n ## Commit message ##\n ci: group installation of Docker dependencies\n \n - Pull in \"lib.sh\" into \"install-docker-dependencies.sh\" such that we can\n - set up proper groups for those dependencise. This allows the reader to\n - collapse sections in the CI output on GitHub Actions (and later on on\n - GitLab CI).\n + The output of CI jobs tends to be quite long-winded and hard to digest.\n + To help with this, many CI systems provide the ability to group output\n + into collapsible sections, and we're also doing this in some of our\n + scripts.\n +\n + One notable omission is the script to install Docker dependencies.\n + Address it to bring more structure to the output for Docker-based jobs.\n \n Signed-off-by: Patrick Steinhardt \n \n4: 4a864a1d174 ! 4: 5ab11d5236d ci: split out logic to set up failed test artifacts\n @@ Commit message\n ci: split out logic to set up failed test artifacts\n \n We have some logic in place to create a directory with the output from\n - failed tests, which will then subsequently be uploaded as CI artifact.\n + failed tests, which will then subsequently be uploaded as CI artifacts.\n We're about to add support for GitLab CI, which will want to reuse the\n logic.\n \n5: 35b07e5378d ! 5: 37a507e9b25 ci: add support for GitLab CI\n @@ Commit message\n \n Part of a problem we hit at GitLab rather frequently is that our own,\n custom CI setup we have is so different to the setup that the Git\n - project has. More esoteric jobs like \"linux-TEST-vars\" that also sets a\n + project has. More esoteric jobs like \"linux-TEST-vars\" that also set a\n couple of environment variables do not exist in GitLab's custom CI\n setup, and maintaining them to keep up with what Git does feels like\n wasted time. The result is that we regularly send patch series upstream\n - that would otherwise fail to compile or pass tests in GitHub Workflows.\n - We would thus like to integrate the GitLab CI configuration into the Git\n - project to help us ensure to send better patch series upstream and thus\n - reduce overhead for the maintainer.\n + that fail to compile or pass tests in GitHub Workflows. We would thus\n + like to integrate the GitLab CI configuration into the Git project to\n + help us send better patch series upstream and thus reduce overhead for\n + the maintainer.\n \n The integration does not necessarily have to be a first-class citizen,\n which would in practice only add to the fallout that pipeline failures\n @@ ci/install-docker-dependencies.sh: linux32)\n \n ## ci/lib.sh ##\n @@ ci/lib.sh: then\n + \t\tneed_to_end_group=\n \t\techo '::endgroup::' >&2\n \t}\n - \n -+\tgroup () {\n -+\t\tset +x\n -+\n -+\t\tgroup=\"$1\"\n -+\t\tshift\n -+\t\tbegin_group \"$group\"\n -+\n -+\t\t# work around `dash` not supporting `set -o pipefail`\n -+\t\t(\n -+\t\t\t\"$@\" 2>&1\n -+\t\t\techo $? >exit.status\n -+\t\t) |\n -+\t\tsed 's/^\\(\\([^ ]*\\):\\([0-9]*\\):\\([0-9]*:\\) \\)\\(error\\|warning\\): /::\\5 file=\\2,line=\\3::\\1/'\n -+\t\tres=$(cat exit.status)\n -+\t\trm exit.status\n -+\n -+\t\tend_group \"$group\"\n -+\t\treturn $res\n -+\t}\n +elif test true = \"$GITLAB_CI\"\n +then\n +\tbegin_group () {\n @@ ci/lib.sh: then\n +\t\techo -e \"\\e[0Ksection_end:$(date +%s):$(echo \"$1\" | tr ' ' _)\\r\\e[0K\"\n +\t\ttrap - EXIT\n +\t}\n -+\n - \tgroup () {\n - \t\tset +x\n - \n + else\n + \tbegin_group () { :; }\n + \tend_group () { :; }\n @@ ci/lib.sh: then\n \tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n \ttest windows != \"$CI_OS_NAME\" ||\n-- \n2.42.0\n\n"},{"id":"483960","messageId":"85617ef85773af3e20701c75d227fea04ac6468a.1698398590.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698398590.git.ps@pks.im","subject":"[PATCH v2 2/5] ci: make grouping setup more generic","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-27T09:25:28Z","receivedAt":"2023-10-27T09:25:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Make the grouping setup more generic by always calling `begin_group ()`\nand `end_group ()` regardless of whether we have stubbed those functions\nor not. This ensures we can more readily add support for additional CI\nplatforms.\n\nFurthermore, the `group ()` function is made generic so that it is the\nsame for both GitHub Actions and for other platforms. There is a\nsemantic conflict here though: GitHub Actions used to call `set +x` in\n`group ()` whereas the non-GitHub case unconditionally uses `set -x`.\nThe latter would get overriden if we kept the `set +x` in the generic\nversion of `group ()`. To resolve this conflict, we simply drop the `set\n+x` in the generic variant of this function. As `begin_group ()` calls\n`set -x` anyway this is not much of a change though, as the only\ncommands that aren't printed anymore now are the ones between the\nbeginning of `group ()` and the end of `begin_group ()`.\n\nLast, this commit changes `end_group ()` to also accept a parameter that\nindicates _which_ group should end. This will be required by a later\ncommit that introduces support for GitLab CI.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 46 ++++++++++++++++++++++------------------------\n 1 file changed, 22 insertions(+), 24 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex eb384f4e952..b3411afae8e 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -14,36 +14,34 @@ then\n \t\tneed_to_end_group=\n \t\techo '::endgroup::' >&2\n \t}\n-\ttrap end_group EXIT\n-\n-\tgroup () {\n-\t\tset +x\n-\t\tbegin_group \"$1\"\n-\t\tshift\n-\t\t# work around `dash` not supporting `set -o pipefail`\n-\t\t(\n-\t\t\t\"$@\" 2>&1\n-\t\t\techo $? >exit.status\n-\t\t) |\n-\t\tsed 's/^\\(\\([^ ]*\\):\\([0-9]*\\):\\([0-9]*:\\) \\)\\(error\\|warning\\): /::\\5 file=\\2,line=\\3::\\1/'\n-\t\tres=$(cat exit.status)\n-\t\trm exit.status\n-\t\tend_group\n-\t\treturn $res\n-\t}\n-\n-\tbegin_group \"CI setup\"\n else\n \tbegin_group () { :; }\n \tend_group () { :; }\n \n-\tgroup () {\n-\t\tshift\n-\t\t\"$@\"\n-\t}\n \tset -x\n fi\n \n+group () {\n+\tgroup=\"$1\"\n+\tshift\n+\tbegin_group \"$group\"\n+\n+\t# work around `dash` not supporting `set -o pipefail`\n+\t(\n+\t\t\"$@\" 2>&1\n+\t\techo $? >exit.status\n+\t) |\n+\tsed 's/^\\(\\([^ ]*\\):\\([0-9]*\\):\\([0-9]*:\\) \\)\\(error\\|warning\\): /::\\5 file=\\2,line=\\3::\\1/'\n+\tres=$(cat exit.status)\n+\trm exit.status\n+\n+\tend_group \"$group\"\n+\treturn $res\n+}\n+\n+begin_group \"CI setup\"\n+trap \"end_group 'CI setup'\" EXIT\n+\n # Set 'exit on error' for all CI scripts to let the caller know that\n # something went wrong.\n #\n@@ -287,5 +285,5 @@ esac\n \n MAKEFLAGS=\"$MAKEFLAGS CC=${CC:-cc}\"\n \n-end_group\n+end_group \"CI setup\"\n set -x\n-- \n2.42.0\n\n"},{"id":"483961","messageId":"57bbc50e3dc33b5f9f0a0b0e70b84cc49c14c55c.1698398590.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698398590.git.ps@pks.im","subject":"[PATCH v2 3/5] ci: group installation of Docker dependencies","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-27T09:25:32Z","receivedAt":"2023-10-27T09:25:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The output of CI jobs tends to be quite long-winded and hard to digest.\nTo help with this, many CI systems provide the ability to group output\ninto collapsible sections, and we're also doing this in some of our\nscripts.\n\nOne notable omission is the script to install Docker dependencies.\nAddress it to bring more structure to the output for Docker-based jobs.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-docker-dependencies.sh | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\nindex 78b7e326da6..d0bc19d3bb3 100755\n--- a/ci/install-docker-dependencies.sh\n+++ b/ci/install-docker-dependencies.sh\n@@ -3,6 +3,10 @@\n # Install dependencies required to build and test Git inside container\n #\n \n+. ${0%/*}/lib.sh\n+\n+begin_group \"Install dependencies\"\n+\n case \"$jobname\" in\n linux32)\n \tlinux32 --32bit i386 sh -c '\n@@ -20,3 +24,5 @@ pedantic)\n \tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n \t;;\n esac\n+\n+end_group \"Install dependencies\"\n-- \n2.42.0\n\n"},{"id":"483962","messageId":"5ab11d5236d496c9100f54d7eb968b4b9f74fb25.1698398590.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698398590.git.ps@pks.im","subject":"[PATCH v2 4/5] ci: split out logic to set up failed test artifacts","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-27T09:25:36Z","receivedAt":"2023-10-27T09:25:41Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have some logic in place to create a directory with the output from\nfailed tests, which will then subsequently be uploaded as CI artifacts.\nWe're about to add support for GitLab CI, which will want to reuse the\nlogic.\n\nSplit the logic into a separate function so that it is reusable.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 40 ++++++++++++++++++++++------------------\n 1 file changed, 22 insertions(+), 18 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex b3411afae8e..9ffdf743903 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -131,6 +131,27 @@ handle_failed_tests () {\n \treturn 1\n }\n \n+create_failed_test_artifacts () {\n+\tmkdir -p t/failed-test-artifacts\n+\n+\tfor test_exit in t/test-results/*.exit\n+\tdo\n+\t\ttest 0 != \"$(cat \"$test_exit\")\" || continue\n+\n+\t\ttest_name=\"${test_exit%.exit}\"\n+\t\ttest_name=\"${test_name##*/}\"\n+\t\tprintf \"\\\\e[33m\\\\e[1m=== Failed test: ${test_name} ===\\\\e[m\\\\n\"\n+\t\techo \"The full logs are in the 'print test failures' step below.\"\n+\t\techo \"See also the 'failed-tests-*' artifacts attached to this run.\"\n+\t\tcat \"t/test-results/$test_name.markup\"\n+\n+\t\ttrash_dir=\"t/trash directory.$test_name\"\n+\t\tcp \"t/test-results/$test_name.out\" t/failed-test-artifacts/\n+\t\ttar czf t/failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n+\tdone\n+\treturn 1\n+}\n+\n # GitHub Action doesn't set TERM, which is required by tput\n export TERM=${TERM:-dumb}\n \n@@ -171,25 +192,8 @@ then\n \tCC=\"${CC_PACKAGE:-${CC:-gcc}}\"\n \tDONT_SKIP_TAGS=t\n \thandle_failed_tests () {\n-\t\tmkdir -p t/failed-test-artifacts\n \t\techo \"FAILED_TEST_ARTIFACTS=t/failed-test-artifacts\" >>$GITHUB_ENV\n-\n-\t\tfor test_exit in t/test-results/*.exit\n-\t\tdo\n-\t\t\ttest 0 != \"$(cat \"$test_exit\")\" || continue\n-\n-\t\t\ttest_name=\"${test_exit%.exit}\"\n-\t\t\ttest_name=\"${test_name##*/}\"\n-\t\t\tprintf \"\\\\e[33m\\\\e[1m=== Failed test: ${test_name} ===\\\\e[m\\\\n\"\n-\t\t\techo \"The full logs are in the 'print test failures' step below.\"\n-\t\t\techo \"See also the 'failed-tests-*' artifacts attached to this run.\"\n-\t\t\tcat \"t/test-results/$test_name.markup\"\n-\n-\t\t\ttrash_dir=\"t/trash directory.$test_name\"\n-\t\t\tcp \"t/test-results/$test_name.out\" t/failed-test-artifacts/\n-\t\t\ttar czf t/failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n-\t\tdone\n-\t\treturn 1\n+\t\tcreate_failed_test_artifacts\n \t}\n \n \tcache_dir=\"$HOME/none\"\n-- \n2.42.0\n\n"},{"id":"483963","messageId":"37a507e9b255dd40c0536438dfe3fa05b067f08f.1698398590.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698398590.git.ps@pks.im","subject":"[PATCH v2 5/5] ci: add support for GitLab CI","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-27T09:25:41Z","receivedAt":"2023-10-27T09:25:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We already support Azure Pipelines and GitHub Workflows in the Git\nproject, but until now we do not have support for GitLab CI. While it is\narguably not in the interest of the Git project to maintain a ton of\ndifferent CI platforms, GitLab has recently ramped up its efforts and\ntries to contribute to the Git project more regularly.\n\nPart of a problem we hit at GitLab rather frequently is that our own,\ncustom CI setup we have is so different to the setup that the Git\nproject has. More esoteric jobs like \"linux-TEST-vars\" that also set a\ncouple of environment variables do not exist in GitLab's custom CI\nsetup, and maintaining them to keep up with what Git does feels like\nwasted time. The result is that we regularly send patch series upstream\nthat fail to compile or pass tests in GitHub Workflows. We would thus\nlike to integrate the GitLab CI configuration into the Git project to\nhelp us send better patch series upstream and thus reduce overhead for\nthe maintainer.\n\nThe integration does not necessarily have to be a first-class citizen,\nwhich would in practice only add to the fallout that pipeline failures\nhave for the maintainer. That being said, we are happy to maintain this\nalternative CI setup for the Git project and will make test results\navailable as part of our own mirror of the Git project at [1].\n\nThis commit introduces the integration into our regular CI scripts so\nthat most of the setup continues to be shared across all of the CI\nsolutions.\n\n[1]: https://gitlab.com/gitlab-org/git\n\nSigned-off-by: Patrick Steinhardt \n---\n .gitlab-ci.yml | 51 +++++++++++++++++++++++++++++++\n ci/install-docker-dependencies.sh | 9 +++++-\n ci/lib.sh | 49 +++++++++++++++++++++++++++++\n ci/print-test-failures.sh | 6 ++++\n 4 files changed, 114 insertions(+), 1 deletion(-)\n create mode 100644 .gitlab-ci.yml\n\ndiff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\nnew file mode 100644\nindex 00000000000..43d3a961fa0\n--- /dev/null\n+++ b/.gitlab-ci.yml\n@@ -0,0 +1,51 @@\n+default:\n+ timeout: 2h\n+\n+workflow:\n+ rules:\n+ - if: $CI_PIPELINE_SOURCE == \"merge_request_event\"\n+ - if: $CI_COMMIT_TAG\n+ - if: $CI_COMMIT_REF_PROTECTED == \"true\"\n+\n+test:\n+ image: $image\n+ before_script:\n+ - ./ci/install-docker-dependencies.sh\n+ script:\n+ - useradd builder --home-dir \"${CI_PROJECT_DIR}\"\n+ - chown -R builder \"${CI_PROJECT_DIR}\"\n+ - sudo --preserve-env --set-home --user=builder ./ci/run-build-and-tests.sh\n+ after_script:\n+ - |\n+ if test \"$CI_JOB_STATUS\" != 'success'\n+ then\n+ sudo --preserve-env --set-home --user=builder ./ci/print-test-failures.sh\n+ fi\n+ parallel:\n+ matrix:\n+ - jobname: linux-sha256\n+ image: ubuntu:latest\n+ CC: clang\n+ - jobname: linux-gcc\n+ image: ubuntu:20.04\n+ CC: gcc\n+ CC_PACKAGE: gcc-8\n+ - jobname: linux-TEST-vars\n+ image: ubuntu:20.04\n+ CC: gcc\n+ CC_PACKAGE: gcc-8\n+ - jobname: linux-gcc-default\n+ image: ubuntu:latest\n+ CC: gcc\n+ - jobname: linux-leaks\n+ image: ubuntu:latest\n+ CC: gcc\n+ - jobname: linux-asan-ubsan\n+ image: ubuntu:latest\n+ CC: clang\n+ - jobname: linux-musl\n+ image: alpine:latest\n+ artifacts:\n+ paths:\n+ - t/failed-test-artifacts\n+ when: on_failure\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\nindex d0bc19d3bb3..1cd92db1876 100755\n--- a/ci/install-docker-dependencies.sh\n+++ b/ci/install-docker-dependencies.sh\n@@ -7,6 +7,9 @@\n \n begin_group \"Install dependencies\"\n \n+# Required so that apt doesn't wait for user input on certain packages.\n+export DEBIAN_FRONTEND=noninteractive\n+\n case \"$jobname\" in\n linux32)\n \tlinux32 --32bit i386 sh -c '\n@@ -16,9 +19,13 @@ linux32)\n \t'\n \t;;\n linux-musl)\n-\tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n+\tapk add --update git shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n \t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n \t;;\n+linux-*)\n+\tapt update -q &&\n+\tapt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}}\n+\t;;\n pedantic)\n \tdnf -yq update >/dev/null &&\n \tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 9ffdf743903..f518df7e2cb 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -14,6 +14,22 @@ then\n \t\tneed_to_end_group=\n \t\techo '::endgroup::' >&2\n \t}\n+elif test true = \"$GITLAB_CI\"\n+then\n+\tbegin_group () {\n+\t\tneed_to_end_group=t\n+\t\techo -e \"\\e[0Ksection_start:$(date +%s):$(echo \"$1\" | tr ' ' _)\\r\\e[0K$1\"\n+\t\ttrap \"end_group '$1'\" EXIT\n+\t\tset -x\n+\t}\n+\n+\tend_group () {\n+\t\ttest -n \"$need_to_end_group\" || return 0\n+\t\tset +x\n+\t\tneed_to_end_group=\n+\t\techo -e \"\\e[0Ksection_end:$(date +%s):$(echo \"$1\" | tr ' ' _)\\r\\e[0K\"\n+\t\ttrap - EXIT\n+\t}\n else\n \tbegin_group () { :; }\n \tend_group () { :; }\n@@ -203,6 +219,39 @@ then\n \tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n \ttest windows != \"$CI_OS_NAME\" ||\n \tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n+elif test true = \"$GITLAB_CI\"\n+then\n+\tCI_TYPE=gitlab-ci\n+\tCI_BRANCH=\"$CI_COMMIT_REF_NAME\"\n+\tCI_COMMIT=\"$CI_COMMIT_SHA\"\n+\tcase \"$CI_JOB_IMAGE\" in\n+\tmacos-*)\n+\t\tCI_OS_NAME=osx;;\n+\talpine:*|ubuntu:*)\n+\t\tCI_OS_NAME=linux;;\n+\t*)\n+\t\techo \"Could not identify OS image\" >&2\n+\t\tenv >&2\n+\t\texit 1\n+\t\t;;\n+\tesac\n+\tCI_REPO_SLUG=\"$CI_PROJECT_PATH\"\n+\tCI_JOB_ID=\"$CI_JOB_ID\"\n+\tCC=\"${CC_PACKAGE:-${CC:-gcc}}\"\n+\tDONT_SKIP_TAGS=t\n+\thandle_failed_tests () {\n+\t\tcreate_failed_test_artifacts\n+\t}\n+\n+\tcache_dir=\"$HOME/none\"\n+\n+\truns_on_pool=$(echo \"$CI_JOB_IMAGE\" | tr : -)\n+\n+\texport GIT_PROVE_OPTS=\"--timer --jobs $(nproc)\"\n+\texport GIT_TEST_OPTS=\"--verbose-log -x\"\n+\tMAKEFLAGS=\"$MAKEFLAGS --jobs=$(nproc)\"\n+\ttest windows != \"$CI_OS_NAME\" ||\n+\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n else\n \techo \"Could not identify CI type\" >&2\n \tenv >&2\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex 57277eefcd0..c33ad4e3a22 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -51,6 +51,12 @@ do\n \t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n \t\t\tcontinue\n \t\t\t;;\n+\t\tgitlab-ci)\n+\t\t\tmkdir -p failed-test-artifacts\n+\t\t\tcp \"${TEST_EXIT%.exit}.out\" failed-test-artifacts/\n+\t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n+\t\t\tcontinue\n+\t\t\t;;\n \t\t*)\n \t\t\techo \"Unhandled CI type: $CI_TYPE\" >&2\n \t\t\texit 1\n-- \n2.42.0\n\n"},{"id":"483965","messageId":"0d889da1-7fd8-4e21-965f-6222e4433ecf@gmail.com","threadId":"60432","inReplyTo":"37a507e9b255dd40c0536438dfe3fa05b067f08f.1698398590.git.ps@pks.im","subject":"Re: [PATCH v2 5/5] ci: add support for GitLab CI","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-10-27T10:19:04Z","receivedAt":"2023-10-27T10:19:09Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 27/10/2023 10:25, Patrick Steinhardt wrote:\n> We already support Azure Pipelines and GitHub Workflows in the Git\n> project, but until now we do not have support for GitLab CI. While it is\n> arguably not in the interest of the Git project to maintain a ton of\n> different CI platforms, GitLab has recently ramped up its efforts and\n> tries to contribute to the Git project more regularly.\n\nI agree we don't want to support too many CI platforms but I think \nadding support for GitLab is good as it helps to stop us being too tied \nto GitHub Actions (which should make it easier if we ever need to \ntransition to a different platform in the future) and provides an \nalternative for contributors who want to use a different platform.\n\n> Part of a problem we hit at GitLab rather frequently is that our own,\n> custom CI setup we have is so different to the setup that the Git\n> project has. More esoteric jobs like \"linux-TEST-vars\" that also set a\n> couple of environment variables do not exist in GitLab's custom CI\n> setup, and maintaining them to keep up with what Git does feels like\n> wasted time. The result is that we regularly send patch series upstream\n> that fail to compile or pass tests in GitHub Workflows. We would thus\n> like to integrate the GitLab CI configuration into the Git project to\n> help us send better patch series upstream and thus reduce overhead for\n> the maintainer.\n> \n> The integration does not necessarily have to be a first-class citizen,\n> which would in practice only add to the fallout that pipeline failures\n> have for the maintainer. That being said, we are happy to maintain this\n> alternative CI setup for the Git project and will make test results\n> available as part of our own mirror of the Git project at [1].\n\nHaving someone committed to on-going maintenance is great.\n\n> This commit introduces the integration into our regular CI scripts so\n> that most of the setup continues to be shared across all of the CI\n> solutions.\n> \n> [1]: https://gitlab.com/gitlab-org/git\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n> .gitlab-ci.yml | 51 +++++++++++++++++++++++++++++++\n> ci/install-docker-dependencies.sh | 9 +++++-\n> ci/lib.sh | 49 +++++++++++++++++++++++++++++\n> ci/print-test-failures.sh | 6 ++++\n> 4 files changed, 114 insertions(+), 1 deletion(-)\n> create mode 100644 .gitlab-ci.yml\n> \n> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\n> new file mode 100644\n> index 00000000000..43d3a961fa0\n> --- /dev/null\n> +++ b/.gitlab-ci.yml\n> @@ -0,0 +1,51 @@\n> +default:\n> + timeout: 2h\n> +\n> +workflow:\n> + rules:\n> + - if: $CI_PIPELINE_SOURCE == \"merge_request_event\"\n> + - if: $CI_COMMIT_TAG\n> + - if: $CI_COMMIT_REF_PROTECTED == \"true\"\n> +\n> +test:\n> + image: $image\n> + before_script:\n> + - ./ci/install-docker-dependencies.sh\n> + script:\n> + - useradd builder --home-dir \"${CI_PROJECT_DIR}\"\n> + - chown -R builder \"${CI_PROJECT_DIR}\"\n> + - sudo --preserve-env --set-home --user=builder ./ci/run-build-and-tests.sh\n\nIt's really good that you're running the tests as an unprivileged user. \nThis is something we used to do when we were using Travis that got lost \nin the transition to Azure Pipelines which means some tests that rely on \nhttpd are now skipped as they refuse to run as root. \nci/run-docker-build.sh is currently bit-rotting, I wonder if it is \npossible to update it so that we can run the dockerized tests in the \nsame way on all CI platforms.\n\n> + after_script:\n> + - |\n> + if test \"$CI_JOB_STATUS\" != 'success'\n> + then\n> + sudo --preserve-env --set-home --user=builder ./ci/print-test-failures.sh\n> + fi\n> + parallel:\n> + matrix:\n> + - jobname: linux-sha256\n> + image: ubuntu:latest\n> + CC: clang\n> + - jobname: linux-gcc\n> + image: ubuntu:20.04\n> + CC: gcc\n> + CC_PACKAGE: gcc-8\n> + - jobname: linux-TEST-vars\n> + image: ubuntu:20.04\n> + CC: gcc\n> + CC_PACKAGE: gcc-8\n> + - jobname: linux-gcc-default\n> + image: ubuntu:latest\n> + CC: gcc\n> + - jobname: linux-leaks\n> + image: ubuntu:latest\n> + CC: gcc\n> + - jobname: linux-asan-ubsan\n> + image: ubuntu:latest\n> + CC: clang\n> + - jobname: linux-musl\n> + image: alpine:latest\n> + artifacts:\n> + paths:\n> + - t/failed-test-artifacts\n> + when: on_failure\n\nThis file is pleasingly small.\n\n> diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\n> index d0bc19d3bb3..1cd92db1876 100755\n> --- a/ci/install-docker-dependencies.sh\n> +++ b/ci/install-docker-dependencies.sh\n> @@ -7,6 +7,9 @@\n> \n> begin_group \"Install dependencies\"\n> \n> +# Required so that apt doesn't wait for user input on certain packages.\n> +export DEBIAN_FRONTEND=noninteractive\n> +\n> case \"$jobname\" in\n> linux32)\n> \tlinux32 --32bit i386 sh -c '\n> @@ -16,9 +19,13 @@ linux32)\n> \t'\n> \t;;\n> linux-musl)\n> -\tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n> +\tapk add --update git shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n> \t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n\nIt would be helpful to explain the new dependencies in the commit \nmessage. I can see why you're adding sudo, but how were we getting away \nwithout installing the other packages for GitHub Actions?\n\n> \t;;\n> +linux-*)\n> +\tapt update -q &&\n> +\tapt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}}\n> +\t;;\n> pedantic)\n> \tdnf -yq update >/dev/null &&\n> \tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n> diff --git a/ci/lib.sh b/ci/lib.sh\n> index 9ffdf743903..f518df7e2cb 100755\n> --- a/ci/lib.sh\n> +++ b/ci/lib.sh\n> @@ -14,6 +14,22 @@ then\n> \t\tneed_to_end_group=\n> \t\techo '::endgroup::' >&2\n> \t}\n> +elif test true = \"$GITLAB_CI\"\n> +then\n> +\tbegin_group () {\n> +\t\tneed_to_end_group=t\n> +\t\techo -e \"\\e[0Ksection_start:$(date +%s):$(echo \"$1\" | tr ' ' _)\\r\\e[0K$1\"\n> +\t\ttrap \"end_group '$1'\" EXIT\n> +\t\tset -x\n> +\t}\n> +\n> +\tend_group () {\n> +\t\ttest -n \"$need_to_end_group\" || return 0\n> +\t\tset +x\n> +\t\tneed_to_end_group=\n> +\t\techo -e \"\\e[0Ksection_end:$(date +%s):$(echo \"$1\" | tr ' ' _)\\r\\e[0K\"\n> +\t\ttrap - EXIT\n> +\t}\n> else\n> \tbegin_group () { :; }\n> \tend_group () { :; }\n> @@ -203,6 +219,39 @@ then\n> \tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n> \ttest windows != \"$CI_OS_NAME\" ||\n> \tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n> +elif test true = \"$GITLAB_CI\"\n> +then\n> +\tCI_TYPE=gitlab-ci\n> +\tCI_BRANCH=\"$CI_COMMIT_REF_NAME\"\n> +\tCI_COMMIT=\"$CI_COMMIT_SHA\"\n> +\tcase \"$CI_JOB_IMAGE\" in\n> +\tmacos-*)\n> +\t\tCI_OS_NAME=osx;;\n> +\talpine:*|ubuntu:*)\n> +\t\tCI_OS_NAME=linux;;\n> +\t*)\n> +\t\techo \"Could not identify OS image\" >&2\n> +\t\tenv >&2\n> +\t\texit 1\n> +\t\t;;\n> +\tesac\n> +\tCI_REPO_SLUG=\"$CI_PROJECT_PATH\"\n> +\tCI_JOB_ID=\"$CI_JOB_ID\"\n\nI guess making this explicit is helpful, otherwise someone may wonder \nwhy CI_JOB_ID is not being set.\n\n> +\tCC=\"${CC_PACKAGE:-${CC:-gcc}}\"\n> +\tDONT_SKIP_TAGS=t\n> +\thandle_failed_tests () {\n> +\t\tcreate_failed_test_artifacts\n> +\t}\n> +\n> +\tcache_dir=\"$HOME/none\"\n> +\n> +\truns_on_pool=$(echo \"$CI_JOB_IMAGE\" | tr : -)\n> +\n> +\texport GIT_PROVE_OPTS=\"--timer --jobs $(nproc)\"\n> +\texport GIT_TEST_OPTS=\"--verbose-log -x\"\n> +\tMAKEFLAGS=\"$MAKEFLAGS --jobs=$(nproc)\"\n> +\ttest windows != \"$CI_OS_NAME\" ||\n> +\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n\nThis last paragraph feels like it should be common to all the CI \nproviders. There are some small differences but if we're going to \nsupport several providers it would be nice to set the common options \ncentrally. I'm pretty sure the --jobs=10 we use for the GitHub and Azure \nis not optimal, using $(nproc) is nice so long as it is supported on all \nthe images we use.\n\nI had a quick glance through the previous patches and they all look like \nnice cleanups that make our ci support less dependent on a single \nprovider. This series looks like a nice addition to our CI support.\n\nBest Wishes\n\nPhillip\n\n> else\n> \techo \"Could not identify CI type\" >&2\n> \tenv >&2\n> diff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\n> index 57277eefcd0..c33ad4e3a22 100755\n> --- a/ci/print-test-failures.sh\n> +++ b/ci/print-test-failures.sh\n> @@ -51,6 +51,12 @@ do\n> \t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n> \t\t\tcontinue\n> \t\t\t;;\n> +\t\tgitlab-ci)\n> +\t\t\tmkdir -p failed-test-artifacts\n> +\t\t\tcp \"${TEST_EXIT%.exit}.out\" failed-test-artifacts/\n> +\t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n> +\t\t\tcontinue\n> +\t\t\t;;\n> \t\t*)\n> \t\t\techo \"Unhandled CI type: $CI_TYPE\" >&2\n> \t\t\texit 1\n\n"},{"id":"483966","messageId":"d62b2e38-6e24-4661-b51d-2ecc59eae256@crinan.ddns.net","threadId":"60432","inReplyTo":"ZTtyHScKkMg4qnMH@tanuki","subject":"Re: [PATCH 5/5] ci: add support for GitLab CI","fromName":"Phillip Wood","fromEmail":"phil@crinan.ddns.net","sentAt":"2023-10-27T10:22:35Z","receivedAt":"2023-10-27T10:29:19Z","isPatch":true,"sender":{"key":"phil@crinan.ddns.net","avatar":null},"body":"On 27/10/2023 09:17, Patrick Steinhardt wrote:\n>>> +\tCI_BRANCH=\"$CI_COMMIT_REF_NAME\"\n>>> +\tCI_COMMIT=\"$CI_COMMIT_SHA\"\n>>>\n>> assignments need no quoting to prevent word splitting.\n>> repeats below.\n>>\n>>> +\tcase \"$CI_JOB_IMAGE\" in\n>>>\n>> ... as does the selector in case statements.\n> \n> True, but I'm simply matching the coding style in this script.\n\nI think it is quite common for us to quote variables when it isn't \nstrictly necessary as it makes it clear to anyone reading the script \nthat there is no word splitting going on and ensures that we don't start \nsplitting the variable if the contents changes in the future.\n\n>>> --- a/ci/print-test-failures.sh\n>>> +++ b/ci/print-test-failures.sh\n>>> @@ -51,6 +51,12 @@ do\n>>> \t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n>>> \t\t\tcontinue\n>>> \t\t\t;;\n>>> +\t\tgitlab-ci)\n>>> +\t\t\tmkdir -p failed-test-artifacts\n>>> +\t\t\tcp \"${TEST_EXIT%.exit}.out\" failed-test-artifacts/\n>>> +\t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n>>>\n>> you're just following the precedent, but imo it's more legible to quote the\n>> entire string, not just the variable expansion. the code doesn't even agree\n>> with itself, as the line directly above apparently agrees with me.\n>>\n>> regards\n> \n> Yeah, as you say, this is another case where I follow precedent. I\n> honestly don't quite care which way we go in this case.\n\nYes, if you're following existing practice I don't think this is worth \nworrying about.\n\nBest Wishes\n\nPhillip\n\n"},{"id":"483967","messageId":"ZTuUa5bA8dh29cyv@ugly","threadId":"60432","inReplyTo":"d62b2e38-6e24-4661-b51d-2ecc59eae256@crinan.ddns.net","subject":"Re: [PATCH 5/5] ci: add support for GitLab CI","fromName":"Oswald Buddenhagen","fromEmail":"oswald.buddenhagen@gmx.de","sentAt":"2023-10-27T10:43:55Z","receivedAt":"2023-10-27T10:44:01Z","isPatch":true,"sender":{"key":"oswald.buddenhagen@gmx.de","avatar":"https://avatars.githubusercontent.com/u/812380?v=4"},"body":"On Fri, Oct 27, 2023 at 11:22:35AM +0100, Phillip Wood wrote:\n>On 27/10/2023 09:17, Patrick Steinhardt wrote:\n>>>> +\tCI_BRANCH=\"$CI_COMMIT_REF_NAME\"\n>>>> +\tCI_COMMIT=\"$CI_COMMIT_SHA\"\n>>>>\n>>> assignments need no quoting to prevent word splitting.\n>>> repeats below.\n>>>\n>>>> +\tcase \"$CI_JOB_IMAGE\" in\n>>>>\n>>> ... as does the selector in case statements.\n>> \n>> True, but I'm simply matching the coding style in this script.\n>\n>I think it is quite common for us to quote variables when it isn't \n>strictly necessary as it makes it clear to anyone reading the script \n>that there is no word splitting going on\n\n>and ensures that we don't start splitting the variable if the contents \n>changes in the future.\n>\nthe point was that it *isn't* content-dependent; it's simply the shell \nrules. of course, many people (apparently you included) don't know these \nsubtleties, so just quoting everything isn't the worst idea (though it \nwould backfire with some *really* old buggy shells, but this doesn't \nneed to concern us).\n\nregards\n"},{"id":"483968","messageId":"ZTuVr54KxhdaIf+F@ugly","threadId":"60432","inReplyTo":"ZTtyHScKkMg4qnMH@tanuki","subject":"Re: [PATCH 5/5] ci: add support for GitLab CI","fromName":"Oswald Buddenhagen","fromEmail":"oswald.buddenhagen@gmx.de","sentAt":"2023-10-27T10:49:19Z","receivedAt":"2023-10-27T10:49:22Z","isPatch":true,"sender":{"key":"oswald.buddenhagen@gmx.de","avatar":"https://avatars.githubusercontent.com/u/812380?v=4"},"body":"On Fri, Oct 27, 2023 at 10:17:33AM +0200, Patrick Steinhardt wrote:\n>On Thu, Oct 26, 2023 at 11:07:08AM +0200, Oswald Buddenhagen wrote:\n>> you should probably choose a less generic name for the jobs, at least\n>> debian-*.\n>\n>The names are all preexisting, so I cannot change them.\n>\naren't they coming from the yml file? would adjusting them in the \ncompany setup be an unreasonable effort?\n\n>I don't quite know what you mean by counter-intuitive patch structure.\n>\nit looked like you're adding the function to the github branch, not to \nthe freshly added gitlab branch. of course that's just a diffing \nartifact.\n\nregards\n"},{"id":"483969","messageId":"ZTuYjzOfG0y27Sho@ugly","threadId":"60432","inReplyTo":"37a507e9b255dd40c0536438dfe3fa05b067f08f.1698398590.git.ps@pks.im","subject":"Re: [PATCH v2 5/5] ci: add support for GitLab CI","fromName":"Oswald Buddenhagen","fromEmail":"oswald.buddenhagen@gmx.de","sentAt":"2023-10-27T11:01:35Z","receivedAt":"2023-10-27T11:01:38Z","isPatch":true,"sender":{"key":"oswald.buddenhagen@gmx.de","avatar":"https://avatars.githubusercontent.com/u/812380?v=4"},"body":"On Fri, Oct 27, 2023 at 11:25:41AM +0200, Patrick Steinhardt wrote:\n>+\texport GIT_PROVE_OPTS=\"--timer --jobs $(nproc)\"\n>+\texport GIT_TEST_OPTS=\"--verbose-log -x\"\n>\nfwiw (as this is again only copied), export with assignment is a \nbash-ism (though (d)ash started to accept this syntax some time ago), \nand not all of the including scripts ask for bash (i didn't check \nwhether they are using these functions, but the inconsistency is an \narmed trap).\n\nregards\n"},{"id":"483970","messageId":"ZTua26fWLxvUbQ4X@tanuki","threadId":"60432","inReplyTo":"ZTuVr54KxhdaIf+F@ugly","subject":"Re: [PATCH 5/5] ci: add support for GitLab CI","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-27T11:11:23Z","receivedAt":"2023-10-27T11:11:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Oct 27, 2023 at 12:49:19PM +0200, Oswald Buddenhagen wrote:\n> On Fri, Oct 27, 2023 at 10:17:33AM +0200, Patrick Steinhardt wrote:\n> > On Thu, Oct 26, 2023 at 11:07:08AM +0200, Oswald Buddenhagen wrote:\n> > > you should probably choose a less generic name for the jobs, at least\n> > > debian-*.\n> > \n> > The names are all preexisting, so I cannot change them.\n> > \n> aren't they coming from the yml file? would adjusting them in the company\n> setup be an unreasonable effort?\n\nThey come from the \".gitlab-ci.yml\" file, but I have to reuse the exact\nnames that GitHub Actions already uses or otherwise we're not testing\nfor the same thing. The preexisting CI scripts for Git expect exactly\nthose names.\n\nI do agree that they may benefit from a redesign so that they're more\nexplicit. But I don't think this patch series here is where we should do\nthat refactoring.\n\n> > I don't quite know what you mean by counter-intuitive patch structure.\n> > \n> it looked like you're adding the function to the github branch, not to the\n> freshly added gitlab branch. of course that's just a diffing artifact.\n\nAh, thought you meant the larger \"structure of how things were layed\nout\". Agreed and fixed in v2 of the patch series.\n\nPatrick\n"},{"id":"483972","messageId":"ZTucxlBR1VrJPuSK@tanuki","threadId":"60432","inReplyTo":"0d889da1-7fd8-4e21-965f-6222e4433ecf@gmail.com","subject":"Re: [PATCH v2 5/5] ci: add support for GitLab CI","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-27T11:19:34Z","receivedAt":"2023-10-27T11:19:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Oct 27, 2023 at 11:19:04AM +0100, Phillip Wood wrote:\n> On 27/10/2023 10:25, Patrick Steinhardt wrote:\n> > We already support Azure Pipelines and GitHub Workflows in the Git\n> > project, but until now we do not have support for GitLab CI. While it is\n> > arguably not in the interest of the Git project to maintain a ton of\n> > different CI platforms, GitLab has recently ramped up its efforts and\n> > tries to contribute to the Git project more regularly.\n> \n> I agree we don't want to support too many CI platforms but I think adding\n> support for GitLab is good as it helps to stop us being too tied to GitHub\n> Actions (which should make it easier if we ever need to transition to a\n> different platform in the future) and provides an alternative for\n> contributors who want to use a different platform.\n> \n> > Part of a problem we hit at GitLab rather frequently is that our own,\n> > custom CI setup we have is so different to the setup that the Git\n> > project has. More esoteric jobs like \"linux-TEST-vars\" that also set a\n> > couple of environment variables do not exist in GitLab's custom CI\n> > setup, and maintaining them to keep up with what Git does feels like\n> > wasted time. The result is that we regularly send patch series upstream\n> > that fail to compile or pass tests in GitHub Workflows. We would thus\n> > like to integrate the GitLab CI configuration into the Git project to\n> > help us send better patch series upstream and thus reduce overhead for\n> > the maintainer.\n> > \n> > The integration does not necessarily have to be a first-class citizen,\n> > which would in practice only add to the fallout that pipeline failures\n> > have for the maintainer. That being said, we are happy to maintain this\n> > alternative CI setup for the Git project and will make test results\n> > available as part of our own mirror of the Git project at [1].\n> \n> Having someone committed to on-going maintenance is great.\n> \n> > This commit introduces the integration into our regular CI scripts so\n> > that most of the setup continues to be shared across all of the CI\n> > solutions.\n> > \n> > [1]: https://gitlab.com/gitlab-org/git\n> > \n> > Signed-off-by: Patrick Steinhardt \n> > ---\n> > .gitlab-ci.yml | 51 +++++++++++++++++++++++++++++++\n> > ci/install-docker-dependencies.sh | 9 +++++-\n> > ci/lib.sh | 49 +++++++++++++++++++++++++++++\n> > ci/print-test-failures.sh | 6 ++++\n> > 4 files changed, 114 insertions(+), 1 deletion(-)\n> > create mode 100644 .gitlab-ci.yml\n> > \n> > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\n> > new file mode 100644\n> > index 00000000000..43d3a961fa0\n> > --- /dev/null\n> > +++ b/.gitlab-ci.yml\n> > @@ -0,0 +1,51 @@\n> > +default:\n> > + timeout: 2h\n> > +\n> > +workflow:\n> > + rules:\n> > + - if: $CI_PIPELINE_SOURCE == \"merge_request_event\"\n> > + - if: $CI_COMMIT_TAG\n> > + - if: $CI_COMMIT_REF_PROTECTED == \"true\"\n> > +\n> > +test:\n> > + image: $image\n> > + before_script:\n> > + - ./ci/install-docker-dependencies.sh\n> > + script:\n> > + - useradd builder --home-dir \"${CI_PROJECT_DIR}\"\n> > + - chown -R builder \"${CI_PROJECT_DIR}\"\n> > + - sudo --preserve-env --set-home --user=builder ./ci/run-build-and-tests.sh\n> \n> It's really good that you're running the tests as an unprivileged user. This\n> is something we used to do when we were using Travis that got lost in the\n> transition to Azure Pipelines which means some tests that rely on httpd are\n> now skipped as they refuse to run as root. ci/run-docker-build.sh is\n> currently bit-rotting, I wonder if it is possible to update it so that we\n> can run the dockerized tests in the same way on all CI platforms.\n> \n> > + after_script:\n> > + - |\n> > + if test \"$CI_JOB_STATUS\" != 'success'\n> > + then\n> > + sudo --preserve-env --set-home --user=builder ./ci/print-test-failures.sh\n> > + fi\n> > + parallel:\n> > + matrix:\n> > + - jobname: linux-sha256\n> > + image: ubuntu:latest\n> > + CC: clang\n> > + - jobname: linux-gcc\n> > + image: ubuntu:20.04\n> > + CC: gcc\n> > + CC_PACKAGE: gcc-8\n> > + - jobname: linux-TEST-vars\n> > + image: ubuntu:20.04\n> > + CC: gcc\n> > + CC_PACKAGE: gcc-8\n> > + - jobname: linux-gcc-default\n> > + image: ubuntu:latest\n> > + CC: gcc\n> > + - jobname: linux-leaks\n> > + image: ubuntu:latest\n> > + CC: gcc\n> > + - jobname: linux-asan-ubsan\n> > + image: ubuntu:latest\n> > + CC: clang\n> > + - jobname: linux-musl\n> > + image: alpine:latest\n> > + artifacts:\n> > + paths:\n> > + - t/failed-test-artifacts\n> > + when: on_failure\n> \n> This file is pleasingly small.\n> \n> > diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\n> > index d0bc19d3bb3..1cd92db1876 100755\n> > --- a/ci/install-docker-dependencies.sh\n> > +++ b/ci/install-docker-dependencies.sh\n> > @@ -7,6 +7,9 @@\n> > begin_group \"Install dependencies\"\n> > +# Required so that apt doesn't wait for user input on certain packages.\n> > +export DEBIAN_FRONTEND=noninteractive\n> > +\n> > case \"$jobname\" in\n> > linux32)\n> > \tlinux32 --32bit i386 sh -c '\n> > @@ -16,9 +19,13 @@ linux32)\n> > \t'\n> > \t;;\n> > linux-musl)\n> > -\tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n> > +\tapk add --update git shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n> > \t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n> \n> It would be helpful to explain the new dependencies in the commit message. I\n> can see why you're adding sudo, but how were we getting away without\n> installing the other packages for GitHub Actions?\n\nTrue, that part is missing.\n\n- Both sudo and shadow are now required because of `useradd` that we use\n to set up the unprivileged build.\n\n- Git has been required all along, I think. `save_good_tree ()` is used\n in our CI scripts, and Toon (fellow GitLabber from my team) has\n noticed that the CI job warned about missing Git. The warning was\n mostly benign as it seems, but still, doesn't hurt to fix it while at\n it.\n\nI'll have a look at whether I can add another patch on top that adjusts\n`ci/run-docker-build.sh` to do rootless builds, which would also make it\nmore obvious why we now need to install sudo and shadow. And I'll make\nsure to document why we now need to have Git around.\n\n> > \t;;\n> > +linux-*)\n> > +\tapt update -q &&\n> > +\tapt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}}\n> > +\t;;\n> > pedantic)\n> > \tdnf -yq update >/dev/null &&\n> > \tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n> > diff --git a/ci/lib.sh b/ci/lib.sh\n> > index 9ffdf743903..f518df7e2cb 100755\n> > --- a/ci/lib.sh\n> > +++ b/ci/lib.sh\n> > @@ -14,6 +14,22 @@ then\n> > \t\tneed_to_end_group=\n> > \t\techo '::endgroup::' >&2\n> > \t}\n> > +elif test true = \"$GITLAB_CI\"\n> > +then\n> > +\tbegin_group () {\n> > +\t\tneed_to_end_group=t\n> > +\t\techo -e \"\\e[0Ksection_start:$(date +%s):$(echo \"$1\" | tr ' ' _)\\r\\e[0K$1\"\n> > +\t\ttrap \"end_group '$1'\" EXIT\n> > +\t\tset -x\n> > +\t}\n> > +\n> > +\tend_group () {\n> > +\t\ttest -n \"$need_to_end_group\" || return 0\n> > +\t\tset +x\n> > +\t\tneed_to_end_group=\n> > +\t\techo -e \"\\e[0Ksection_end:$(date +%s):$(echo \"$1\" | tr ' ' _)\\r\\e[0K\"\n> > +\t\ttrap - EXIT\n> > +\t}\n> > else\n> > \tbegin_group () { :; }\n> > \tend_group () { :; }\n> > @@ -203,6 +219,39 @@ then\n> > \tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n> > \ttest windows != \"$CI_OS_NAME\" ||\n> > \tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n> > +elif test true = \"$GITLAB_CI\"\n> > +then\n> > +\tCI_TYPE=gitlab-ci\n> > +\tCI_BRANCH=\"$CI_COMMIT_REF_NAME\"\n> > +\tCI_COMMIT=\"$CI_COMMIT_SHA\"\n> > +\tcase \"$CI_JOB_IMAGE\" in\n> > +\tmacos-*)\n> > +\t\tCI_OS_NAME=osx;;\n> > +\talpine:*|ubuntu:*)\n> > +\t\tCI_OS_NAME=linux;;\n> > +\t*)\n> > +\t\techo \"Could not identify OS image\" >&2\n> > +\t\tenv >&2\n> > +\t\texit 1\n> > +\t\t;;\n> > +\tesac\n> > +\tCI_REPO_SLUG=\"$CI_PROJECT_PATH\"\n> > +\tCI_JOB_ID=\"$CI_JOB_ID\"\n> \n> I guess making this explicit is helpful, otherwise someone may wonder why\n> CI_JOB_ID is not being set.\n> \n> > +\tCC=\"${CC_PACKAGE:-${CC:-gcc}}\"\n> > +\tDONT_SKIP_TAGS=t\n> > +\thandle_failed_tests () {\n> > +\t\tcreate_failed_test_artifacts\n> > +\t}\n> > +\n> > +\tcache_dir=\"$HOME/none\"\n> > +\n> > +\truns_on_pool=$(echo \"$CI_JOB_IMAGE\" | tr : -)\n> > +\n> > +\texport GIT_PROVE_OPTS=\"--timer --jobs $(nproc)\"\n> > +\texport GIT_TEST_OPTS=\"--verbose-log -x\"\n> > +\tMAKEFLAGS=\"$MAKEFLAGS --jobs=$(nproc)\"\n> > +\ttest windows != \"$CI_OS_NAME\" ||\n> > +\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n> \n> This last paragraph feels like it should be common to all the CI providers.\n> There are some small differences but if we're going to support several\n> providers it would be nice to set the common options centrally. I'm pretty\n> sure the --jobs=10 we use for the GitHub and Azure is not optimal, using\n> $(nproc) is nice so long as it is supported on all the images we use.\n\nTrue enough, I can have a go at it. I'm rather cautious around updating\nthe build infra as I cannot easily verify that things continue to work\nfor GitHub. But this part here might be a good candidate for deduping.\n\n> I had a quick glance through the previous patches and they all look like\n> nice cleanups that make our ci support less dependent on a single provider.\n> This series looks like a nice addition to our CI support.\n> \n> Best Wishes\n> \n> Phillip\n\nThanks!\n\nPatrick\n\n> \n> > else\n> > \techo \"Could not identify CI type\" >&2\n> > \tenv >&2\n> > diff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\n> > index 57277eefcd0..c33ad4e3a22 100755\n> > --- a/ci/print-test-failures.sh\n> > +++ b/ci/print-test-failures.sh\n> > @@ -51,6 +51,12 @@ do\n> > \t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n> > \t\t\tcontinue\n> > \t\t\t;;\n> > +\t\tgitlab-ci)\n> > +\t\t\tmkdir -p failed-test-artifacts\n> > +\t\t\tcp \"${TEST_EXIT%.exit}.out\" failed-test-artifacts/\n> > +\t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n> > +\t\t\tcontinue\n> > +\t\t\t;;\n> > \t\t*)\n> > \t\t\techo \"Unhandled CI type: $CI_TYPE\" >&2\n> > \t\t\texit 1\n> \n"},{"id":"483975","messageId":"ZTulo_rh_G-lfICQ@tanuki","threadId":"60432","inReplyTo":"ZTucxlBR1VrJPuSK@tanuki","subject":"Re: [PATCH v2 5/5] ci: add support for GitLab CI","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-27T11:57:23Z","receivedAt":"2023-10-27T11:57:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Oct 27, 2023 at 01:19:35PM +0200, Patrick Steinhardt wrote:\n> On Fri, Oct 27, 2023 at 11:19:04AM +0100, Phillip Wood wrote:\n> > On 27/10/2023 10:25, Patrick Steinhardt wrote:\n[snip]\n> > > diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\n> > > index d0bc19d3bb3..1cd92db1876 100755\n> > > --- a/ci/install-docker-dependencies.sh\n> > > +++ b/ci/install-docker-dependencies.sh\n> > > @@ -7,6 +7,9 @@\n> > > begin_group \"Install dependencies\"\n> > > +# Required so that apt doesn't wait for user input on certain packages.\n> > > +export DEBIAN_FRONTEND=noninteractive\n> > > +\n> > > case \"$jobname\" in\n> > > linux32)\n> > > \tlinux32 --32bit i386 sh -c '\n> > > @@ -16,9 +19,13 @@ linux32)\n> > > \t'\n> > > \t;;\n> > > linux-musl)\n> > > -\tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n> > > +\tapk add --update git shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n> > > \t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n> > \n> > It would be helpful to explain the new dependencies in the commit message. I\n> > can see why you're adding sudo, but how were we getting away without\n> > installing the other packages for GitHub Actions?\n> \n> True, that part is missing.\n> \n> - Both sudo and shadow are now required because of `useradd` that we use\n> to set up the unprivileged build.\n> \n> - Git has been required all along, I think. `save_good_tree ()` is used\n> in our CI scripts, and Toon (fellow GitLabber from my team) has\n> noticed that the CI job warned about missing Git. The warning was\n> mostly benign as it seems, but still, doesn't hurt to fix it while at\n> it.\n> \n> I'll have a look at whether I can add another patch on top that adjusts\n> `ci/run-docker-build.sh` to do rootless builds, which would also make it\n> more obvious why we now need to install sudo and shadow. And I'll make\n> sure to document why we now need to have Git around.\n\nHum. After having a look at `ci/run-docker-build.sh` I don't feel like\nit's sensible to update it. It's not even used anymore by our CI but\nonly by `ci/run-docker.sh`, which seems to be more of a developer-facing\nscript?\n\nAs you said, this smells like rotting bits that might rather be removed.\nBut in any case, as they don't relate to our current CI infrastructure\nexcept for being in \"ci/\" I'll leave them be for now.\n\nPatrick\n"},{"id":"483981","messageId":"6ee43015-1abc-4269-b710-0647ce80d7a2@crinan.ddns.net","threadId":"60432","inReplyTo":"ZTulo_rh_G-lfICQ@tanuki","subject":"Re: [PATCH v2 5/5] ci: add support for GitLab CI","fromName":"Phillip Wood","fromEmail":"phillip.wood31459@gmail.com","sentAt":"2023-10-27T13:02:04Z","receivedAt":"2023-10-27T13:02:09Z","isPatch":true,"sender":{"key":"phillip.wood31459@gmail.com","avatar":null},"body":"On 27/10/2023 12:57, Patrick Steinhardt wrote:\n> Hum. After having a look at `ci/run-docker-build.sh` I don't feel like\n> it's sensible to update it. It's not even used anymore by our CI but\n> only by `ci/run-docker.sh`, which seems to be more of a developer-facing\n> script?\n> \n> As you said, this smells like rotting bits that might rather be removed.\n> But in any case, as they don't relate to our current CI infrastructure\n> except for being in \"ci/\" I'll leave them be for now.\n\nI was trying to suggest that we start using these scripts again. The \nfact that we run the dockerized tests as root on the other CI platforms \nis a regression from what we used to do. I'm not an expert but I think \nfor the builds that use docker we're essentially using the some build \nenvironment regardless of the CI provider so it would make sense to \nhandle them all in the same way. I think the existing script uses \"su\" \nbut we could change it to use \"sudo\" like you're doing here.\n\nBest Wishes\n\nPhillip\n\n"},{"id":"483987","messageId":"d00b02e9-fb05-44bc-90ee-1851ef98dd26@gmail.com","threadId":"60432","inReplyTo":"ZTuYjzOfG0y27Sho@ugly","subject":"Re: [PATCH v2 5/5] ci: add support for GitLab CI","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-10-27T13:17:02Z","receivedAt":"2023-10-27T13:17:08Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 27/10/2023 12:01, Oswald Buddenhagen wrote:\n> On Fri, Oct 27, 2023 at 11:25:41AM +0200, Patrick Steinhardt wrote:\n>> +    export GIT_PROVE_OPTS=\"--timer --jobs $(nproc)\"\n>> +    export GIT_TEST_OPTS=\"--verbose-log -x\"\n>>\n> fwiw (as this is again only copied), export with assignment is a \n> bash-ism\n\nNot according to \nhttps://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#export\n\n SYNOPSIS\n\n export name[=word]...\n\n DESCRIPTION\n\n The shell shall give the export attribute to the variables\n corresponding to the specified names, which shall cause them\n to be in the environment of subsequently executed commands. If\n the name of a variable is followed by = word, then the value\n of that variable shall be set to word.\n\nIt is true that in our test suite we separate a variable assignment when \nexporting. Presumably that is because someone reported that their shell \ndid not support the \"export name=WORD\" syntax in the past. As we're \nalready using this syntax with the same docker images in Github Actions \nI think we can assume it is safe here.\n\nBest Wishes\n\nPhillip\n"},{"id":"483992","messageId":"b6ef74b7-2c77-4621-9969-d911c34561d5@crinan.ddns.net","threadId":"60432","inReplyTo":"ZTuUa5bA8dh29cyv@ugly","subject":"Re: [PATCH 5/5] ci: add support for GitLab CI","fromName":"Phillip Wood","fromEmail":"phil@crinan.ddns.net","sentAt":"2023-10-27T14:32:48Z","receivedAt":"2023-10-27T14:32:54Z","isPatch":true,"sender":{"key":"phil@crinan.ddns.net","avatar":null},"body":"On 27/10/2023 11:43, Oswald Buddenhagen wrote:\n> On Fri, Oct 27, 2023 at 11:22:35AM +0100, Phillip Wood wrote:\n>> On 27/10/2023 09:17, Patrick Steinhardt wrote:\n>>>>> +    CI_BRANCH=\"$CI_COMMIT_REF_NAME\"\n>>>>> +    CI_COMMIT=\"$CI_COMMIT_SHA\"\n>>>>>\n>>>> assignments need no quoting to prevent word splitting.\n>>>> repeats below.\n>>>>\n>>>>> +    case \"$CI_JOB_IMAGE\" in\n>>>>>\n>>>> ... as does the selector in case statements.\n>>>\n>>> True, but I'm simply matching the coding style in this script.\n>>\n>> I think it is quite common for us to quote variables when it isn't \n>> strictly necessary as it makes it clear to anyone reading the script \n>> that there is no word splitting going on\n> \n>> and ensures that we don't start splitting the variable if the contents \n>> changes in the future.\n>>\n> the point was that it *isn't* content-dependent; it's simply the shell \n> rules.\n\nOh, I'd misunderstood what you were saying which was that assignment and \ncase statements are not subject to field splitting.\n\n> of course, many people (apparently you included) don't know these \n> subtleties\n\nI find this comment to be condescending, needlessly antagonistic and \ncompletely at odds with the norms of constructive discussion on this list.\n\nBest Wishes\n\nPhillip\n"},{"id":"483996","messageId":"ZTvc/tPGxREyq17J@ugly","threadId":"60432","inReplyTo":"d00b02e9-fb05-44bc-90ee-1851ef98dd26@gmail.com","subject":"Re: [PATCH v2 5/5] ci: add support for GitLab CI","fromName":"Oswald Buddenhagen","fromEmail":"oswald.buddenhagen@gmx.de","sentAt":"2023-10-27T15:53:34Z","receivedAt":"2023-10-27T15:53:42Z","isPatch":true,"sender":{"key":"oswald.buddenhagen@gmx.de","avatar":"https://avatars.githubusercontent.com/u/812380?v=4"},"body":"On Fri, Oct 27, 2023 at 02:17:02PM +0100, Phillip Wood wrote:\n>On 27/10/2023 12:01, Oswald Buddenhagen wrote:\n>> On Fri, Oct 27, 2023 at 11:25:41AM +0200, Patrick Steinhardt wrote:\n>>> +    export GIT_PROVE_OPTS=\"--timer --jobs $(nproc)\"\n>>> +    export GIT_TEST_OPTS=\"--verbose-log -x\"\n>>>\n>> fwiw (as this is again only copied), export with assignment is a \n>> bash-ism\n>\n>Not according to \n>https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#export\n>\nhmm, it's there since at least SUSv1, aka XPG4v2, in 1994. i didn't \nbother digging deeper.\n\n>It is true that in our test suite we separate a variable assignment \n>when exporting. Presumably that is because someone reported that their \n>shell did not support the \"export name=WORD\" syntax in the past.\n>\nmost likely it's just a historical default, not the result of a specific \nbug report. the assumption that it's bash-specific is wide-spread.\n\n>As we're already using this syntax with the same docker images in \n>Github Actions I think we can assume it is safe here.\n>\ni guess so. docker seems a tad unlikely to run some ancient bourne \nshells ...\n\nregards\n"},{"id":"484000","messageId":"ZTv3to98QT8Ma+ag@ugly","threadId":"60432","inReplyTo":"b6ef74b7-2c77-4621-9969-d911c34561d5@crinan.ddns.net","subject":"Re: [PATCH 5/5] ci: add support for GitLab CI","fromName":"Oswald Buddenhagen","fromEmail":"oswald.buddenhagen@gmx.de","sentAt":"2023-10-27T17:47:34Z","receivedAt":"2023-10-27T17:47:38Z","isPatch":true,"sender":{"key":"oswald.buddenhagen@gmx.de","avatar":"https://avatars.githubusercontent.com/u/812380?v=4"},"body":"On Fri, Oct 27, 2023 at 03:32:48PM +0100, Phillip Wood wrote:\n>On 27/10/2023 11:43, Oswald Buddenhagen wrote:\n>> On Fri, Oct 27, 2023 at 11:22:35AM +0100, Phillip Wood wrote:\n>>>>>> +    CI_BRANCH=\"$CI_COMMIT_REF_NAME\"\n>>>>>> +    CI_COMMIT=\"$CI_COMMIT_SHA\"\n>>>>>>\n>>>>> assignments need no quoting to prevent word splitting.\n>>>>> repeats below.\n>>>>>\n>>> I think it is quite common for us to quote variables when it isn't \n>>> strictly necessary as it makes it clear to anyone reading the script \n>>> that there is no word splitting going on\n>> \n>>> and ensures that we don't start splitting the variable if the contents \n>>> changes in the future.\n>>>\n>> the point was that it *isn't* content-dependent; it's simply the shell \n>> rules.\n>\n>Oh, I'd misunderstood what you were saying which was that assignment and \n>case statements are not subject to field splitting.\n>\n>> of course, many people (apparently you included) don't know these \n>> subtleties\n>\n>I find this comment to be condescending, needlessly antagonistic and \n>completely at odds with the norms of constructive discussion on this list.\n>\nthe observation was necessary for the point i subsequently made (which \nwas basically agreeing with the first part of your response).\n\ni think it's a rather uncontroversial statement that many people don't \nknow the ins and outs of the shell's word splitting rules. in fact, this \nis to be expected given how arbitrary the rules seem.\n\nthat you seem to be among these people was a reasonable inference from \nwhat you actually wrote. mentioning it was meant to provide a segue and \nadd emphasis.\n\nregards\n"},{"id":"484051","messageId":"8db10489-546d-428d-9b3e-7ecea87e1313@gmail.com","threadId":"60432","inReplyTo":"6ee43015-1abc-4269-b710-0647ce80d7a2@crinan.ddns.net","subject":"Re: [PATCH v2 5/5] ci: add support for GitLab CI","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-10-29T16:13:39Z","receivedAt":"2023-10-29T16:13:46Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Patrick\n\nOn 27/10/2023 14:02, Phillip Wood wrote:\n> On 27/10/2023 12:57, Patrick Steinhardt wrote:\n>> Hum. After having a look at `ci/run-docker-build.sh` I don't feel like\n>> it's sensible to update it. It's not even used anymore by our CI but\n>> only by `ci/run-docker.sh`, which seems to be more of a developer-facing\n>> script?\n>>\n>> As you said, this smells like rotting bits that might rather be removed.\n>> But in any case, as they don't relate to our current CI infrastructure\n>> except for being in \"ci/\" I'll leave them be for now.\n> \n> I was trying to suggest that we start using these scripts again.\n\nHaving taken a closer look I think we'd be better off adding something like\n\n # Ensure the build and tests run as an unprivileged user\n if test \"$(id -u)\" -eq 0\n then\n useradd --home-dir \"$(pwd)\" builder\n chown -R builder .\n exec sudo --preserve-env --set-home --user=builder \"$0\"\n fi\n\nTo the beginning of ci/run-build-and-tests.sh.\n\nBest Wishes\n\nPhillip\n"},{"id":"484052","messageId":"a165fd76-a44b-45c7-b02c-e9b90fda3db3@gmail.com","threadId":"60432","inReplyTo":"ZTucxlBR1VrJPuSK@tanuki","subject":"Re: [PATCH v2 5/5] ci: add support for GitLab CI","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-10-29T16:27:44Z","receivedAt":"2023-10-29T16:27:51Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Patrick\n\nOn 27/10/2023 12:19, Patrick Steinhardt wrote:\n> On Fri, Oct 27, 2023 at 11:19:04AM +0100, Phillip Wood wrote:\n>> On 27/10/2023 10:25, Patrick Steinhardt wrote:\n>>> + parallel:\n>>> + matrix:\n>>> + - jobname: linux-sha256\n>>> + image: ubuntu:latest\n>>> + CC: clang\n>>> + - jobname: linux-gcc\n>>> + image: ubuntu:20.04\n>>> + CC: gcc\n>>> + CC_PACKAGE: gcc-8\n>>> + - jobname: linux-TEST-vars\n>>> + image: ubuntu:20.04\n>>> + CC: gcc\n>>> + CC_PACKAGE: gcc-8\n>>> + - jobname: linux-gcc-default\n>>> + image: ubuntu:latest\n>>> + CC: gcc\n>>> + - jobname: linux-leaks\n>>> + image: ubuntu:latest\n>>> + CC: gcc\n>>> + - jobname: linux-asan-ubsan\n>>> + image: ubuntu:latest\n>>> + CC: clang\n>>> + - jobname: linux-musl\n>>> + image: alpine:latest\n\nI assume you've chosen the configurations from the existing GitHub \nActions that give the best coverage of the various options. One thing I \nnoticed is that the is no equivalent of the \"pedantic\" job that builds \ngit with \"DEVELOPER=1 DEVOPTS=pedantic\"\n\n>>> diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\n>>> index d0bc19d3bb3..1cd92db1876 100755\n>>> --- a/ci/install-docker-dependencies.sh\n>>> +++ b/ci/install-docker-dependencies.sh\n>>> @@ -16,9 +19,13 @@ linux32)\n>>> \t'\n>>> \t;;\n>>> linux-musl)\n>>> -\tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n>>> +\tapk add --update git shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n>>> \t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n>>\n>> It would be helpful to explain the new dependencies in the commit message. I\n>> can see why you're adding sudo, but how were we getting away without\n>> installing the other packages for GitHub Actions?\n> \n> True, that part is missing.\n> \n> - Both sudo and shadow are now required because of `useradd` that we use\n> to set up the unprivileged build.\n> \n> - Git has been required all along, I think. `save_good_tree ()` is used\n> in our CI scripts, and Toon (fellow GitLabber from my team) has\n> noticed that the CI job warned about missing Git. The warning was\n> mostly benign as it seems, but still, doesn't hurt to fix it while at\n> it.\n\nOh I had a look at this and the docker based jobs on GitHub do not have \na git repository so installing git means we now get a repository not \nfound error from save_good_tree() instead. We should probably make \nsave_good_tree() and check_unignored_build_artifacts() return early if \nthere isn't a repository but that's orthogonal to this series.\n\nLooking at the test output from the link in your cover letter we should \nwe should also install apache2[1] and gnupg[2]\n\n[1] https://gitlab.com/gitlab-org/git/-/jobs/5349205374#L1444\n[2] https://gitlab.com/gitlab-org/git/-/jobs/5349205374#L1167\n\nBest Wishes\n\nPhillip\n"},{"id":"484063","messageId":"xmqqy1flez6m.fsf@gitster.g","threadId":"60432","inReplyTo":"0d889da1-7fd8-4e21-965f-6222e4433ecf@gmail.com","subject":"Re: [PATCH v2 5/5] ci: add support for GitLab CI","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-10-30T00:22:57Z","receivedAt":"2023-10-30T00:23:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phillip Wood writes:\n\n> I agree we don't want to support too many CI platforms but I think\n> adding support for GitLab is good as it helps to stop us being too\n> tied to GitHub Actions (which should make it easier if we ever need to\n> transition to a different platform in the future) and provides an\n> alternative for contributors who want to use a different platform.\n> ...\n> Having someone committed to on-going maintenance is great.\n\nYeah, it is great to see that stakeholder companies are helping Git\nin ways they can.\n\nWill queue.\n"},{"id":"484090","messageId":"68c4684b-0ab0-427a-90e5-fca602c9d993@gmail.com","threadId":"60432","inReplyTo":"ZTv3to98QT8Ma+ag@ugly","subject":"Re: [PATCH 5/5] ci: add support for GitLab CI","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-10-30T09:49:46Z","receivedAt":"2023-10-30T09:50:01Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 27/10/2023 18:47, Oswald Buddenhagen wrote:\n> On Fri, Oct 27, 2023 at 03:32:48PM +0100, Phillip Wood wrote:\n>> On 27/10/2023 11:43, Oswald Buddenhagen wrote:\n>>> On Fri, Oct 27, 2023 at 11:22:35AM +0100, Phillip Wood wrote:\n>>>>>>> +    CI_BRANCH=\"$CI_COMMIT_REF_NAME\"\n>>>>>>> +    CI_COMMIT=\"$CI_COMMIT_SHA\"\n>>>>>>>\n>>>>>> assignments need no quoting to prevent word splitting.\n>>>>>> repeats below.\n>>>>>>\n>>>> I think it is quite common for us to quote variables when it isn't \n>>>> strictly necessary as it makes it clear to anyone reading the script \n>>>> that there is no word splitting going on\n>>>\n>>>> and ensures that we don't start splitting the variable if the \n>>>> contents changes in the future.\n>>>>\n>>> the point was that it *isn't* content-dependent; it's simply the \n>>> shell rules.\n>>\n>> Oh, I'd misunderstood what you were saying which was that assignment \n>> and case statements are not subject to field splitting.\n>>\n>>> of course, many people (apparently you included) don't know these \n>>> subtleties\n>>\n>> I find this comment to be condescending, needlessly antagonistic and \n>> completely at odds with the norms of constructive discussion on this \n>> list.\n>>\n> the observation was necessary for the point i subsequently made (which \n> was basically agreeing with the first part of your response).\n\nIt was not necessary to phrase it as you did though. Before replying on \nFriday I showed your comment to someone else and their reaction was \n\"That's rude\". You could have made your point by saying something like\n\n It is hard to remember all the shell's word splitting rules so\n quoting everywhere is not a bad idea.\n\nThis is not the first time I've found your comments unnecessarily \nadversarial and at odds with the norms of constructive discussion and \nrespectful disagreement on this list. I don't think I'm the only one \neither - in [1] Junio points out an ad-hominem remark and in [2] Marc \ncomments on the unreceptive tone of you review responses.\n\nI would urge you to try and strike a more conciliatory tone in your \nmessages - it is perfectly possible to correct or disagree with someone \nwithout alienating them in the process.\n\nBest Wishes\n\nPhillip\n\n[1] https://lore.kernel.org/git/xmqqleeihok5.fsf@gitster.g/\n[2] \nhttps://lore.kernel.org/git/e33f919d-1b6a-4944-ab5d-93ad0d323b68@xiplink.com/\n"},{"id":"484096","messageId":"ZT-JWtD_bf0RLAYR@tanuki","threadId":"60432","inReplyTo":"a165fd76-a44b-45c7-b02c-e9b90fda3db3@gmail.com","subject":"Re: [PATCH v2 5/5] ci: add support for GitLab CI","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-30T10:45:46Z","receivedAt":"2023-10-30T10:46:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sun, Oct 29, 2023 at 04:27:44PM +0000, Phillip Wood wrote:\n> Hi Patrick\n> \n> On 27/10/2023 12:19, Patrick Steinhardt wrote:\n> > On Fri, Oct 27, 2023 at 11:19:04AM +0100, Phillip Wood wrote:\n> > > On 27/10/2023 10:25, Patrick Steinhardt wrote:\n> > > > + parallel:\n> > > > + matrix:\n> > > > + - jobname: linux-sha256\n> > > > + image: ubuntu:latest\n> > > > + CC: clang\n> > > > + - jobname: linux-gcc\n> > > > + image: ubuntu:20.04\n> > > > + CC: gcc\n> > > > + CC_PACKAGE: gcc-8\n> > > > + - jobname: linux-TEST-vars\n> > > > + image: ubuntu:20.04\n> > > > + CC: gcc\n> > > > + CC_PACKAGE: gcc-8\n> > > > + - jobname: linux-gcc-default\n> > > > + image: ubuntu:latest\n> > > > + CC: gcc\n> > > > + - jobname: linux-leaks\n> > > > + image: ubuntu:latest\n> > > > + CC: gcc\n> > > > + - jobname: linux-asan-ubsan\n> > > > + image: ubuntu:latest\n> > > > + CC: clang\n> > > > + - jobname: linux-musl\n> > > > + image: alpine:latest\n> \n> I assume you've chosen the configurations from the existing GitHub Actions\n> that give the best coverage of the various options. One thing I noticed is\n> that the is no equivalent of the \"pedantic\" job that builds git with\n> \"DEVELOPER=1 DEVOPTS=pedantic\"\n\nYeah, indeed. The list for sure isn't complete, and I also want to\niterate in the future to add support for macOS and Windows. But adding\nthe pedantic job is easy enough, so I'll include it.\n\n> > > > diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\n> > > > index d0bc19d3bb3..1cd92db1876 100755\n> > > > --- a/ci/install-docker-dependencies.sh\n> > > > +++ b/ci/install-docker-dependencies.sh\n> > > > @@ -16,9 +19,13 @@ linux32)\n> > > > \t'\n> > > > \t;;\n> > > > linux-musl)\n> > > > -\tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n> > > > +\tapk add --update git shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n> > > > \t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n> > > \n> > > It would be helpful to explain the new dependencies in the commit message. I\n> > > can see why you're adding sudo, but how were we getting away without\n> > > installing the other packages for GitHub Actions?\n\nI've dropped the \"git\" dependency for now. It doesn't do anything useful\nanyway for our Docker-based jobs. Instead, I've added a preparatory\ncommit that makes the relevant CI library functions more robust by\ndetecting the case where we either don't have Git available or don't use\na repository.\n\n> > True, that part is missing.\n> > \n> > - Both sudo and shadow are now required because of `useradd` that we use\n> > to set up the unprivileged build.\n> > \n> > - Git has been required all along, I think. `save_good_tree ()` is used\n> > in our CI scripts, and Toon (fellow GitLabber from my team) has\n> > noticed that the CI job warned about missing Git. The warning was\n> > mostly benign as it seems, but still, doesn't hurt to fix it while at\n> > it.\n> \n> Oh I had a look at this and the docker based jobs on GitHub do not have a\n> git repository so installing git means we now get a repository not found\n> error from save_good_tree() instead. We should probably make\n> save_good_tree() and check_unignored_build_artifacts() return early if there\n> isn't a repository but that's orthogonal to this series.\n\nYup, done now.\n\n> Looking at the test output from the link in your cover letter we should we\n> should also install apache2[1] and gnupg[2]\n\nTrue. Also missing are Bash, CVS, the Perl CGI module, Subversion and\nPerforce. I'll add them as available in the respective distro's\nrepositories.\n\nPatrick\n\n> [1] https://gitlab.com/gitlab-org/git/-/jobs/5349205374#L1444\n> [2] https://gitlab.com/gitlab-org/git/-/jobs/5349205374#L1167\n> \n> Best Wishes\n> \n> Phillip\n"},{"id":"484097","messageId":"ZT-JdyEEgN3l7T5L@tanuki","threadId":"60432","inReplyTo":"8db10489-546d-428d-9b3e-7ecea87e1313@gmail.com","subject":"Re: [PATCH v2 5/5] ci: add support for GitLab CI","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-30T10:46:15Z","receivedAt":"2023-10-30T10:46:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sun, Oct 29, 2023 at 04:13:39PM +0000, Phillip Wood wrote:\n> Hi Patrick\n> \n> On 27/10/2023 14:02, Phillip Wood wrote:\n> > On 27/10/2023 12:57, Patrick Steinhardt wrote:\n> > > Hum. After having a look at `ci/run-docker-build.sh` I don't feel like\n> > > it's sensible to update it. It's not even used anymore by our CI but\n> > > only by `ci/run-docker.sh`, which seems to be more of a developer-facing\n> > > script?\n> > > \n> > > As you said, this smells like rotting bits that might rather be removed.\n> > > But in any case, as they don't relate to our current CI infrastructure\n> > > except for being in \"ci/\" I'll leave them be for now.\n> > \n> > I was trying to suggest that we start using these scripts again.\n> \n> Having taken a closer look I think we'd be better off adding something like\n> \n> # Ensure the build and tests run as an unprivileged user\n> if test \"$(id -u)\" -eq 0\n> then\n> useradd --home-dir \"$(pwd)\" builder\n> chown -R builder .\n> exec sudo --preserve-env --set-home --user=builder \"$0\"\n> fi\n> \n> To the beginning of ci/run-build-and-tests.sh.\n\nThat indeed looks like a nice way to handle this, agreed. As mentioned\nthough, I don't really have an easy way to test this with GitHub\nWorkflows or Azure Pipelines. So I'd propose to defer this change to a\nfollow-up patch series -- and in the best case somebody who is familiar\nwith these CI solutions would pick it up rather than me.\n\nPatrick\n"},{"id":"484098","messageId":"cover.1698667545.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698305961.git.ps@pks.im","subject":"[PATCH v3 0/8] ci: add GitLab CI definition","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-30T12:14:39Z","receivedAt":"2023-10-30T12:14:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the third version of this patch series that adds GitLab CI\ndefinitions to the Git project. Please refer to the cover letter for v1\nof this series [1] for the motivation and intent -- I won't repeat it\nhere as it's a bit on the longer side.\n\nChanges compared to v2:\n\n - Patch 5: This is a new preparatory step to unify the setup of some\n environment variables. It also fixes some smallish issues, like\n e.g. the fact that some envvars were set _after_ the export.\n\n - Patch 6: Another new preparatory step. It makes our infra around\n certain helper functions that have the intent to interact with the\n project repository, e.g. to cache good trees. We now detect the\n case when there is no Git or when the project is not a Git repo\n and bail out gracefully.\n\n - Patch 7: The last new preparatory patch. Installs a bunch of\n dependencies which are required for the test runtime in the Alpine\n based job. This increases test coverage.\n\n - Patch 8: Several smaller improvements:\n - Added a note why we install sudo and shadow in linux-musl now.\n - Fixed an issue where the HOME directory was part of the\n project directory, and thus Git complained about newly added\n untracked files.\n - Added the \"pedantic\" job that does a pednatic compilation on\n Fedora.\n - Added more test time dependencies.\n - Made the \"echo -e\" invocation portable by using printf\n instead.\n\nWhat I didn't address yet is a suggestion by Phillip, namely to unify\nthe logic that sets up unprivileged builds. I don't have the infra\navailable to test any such change that would ultimately also impact\nGitHub Workflows and Azure Pipelines and thus do not feel comfortable\nto refactor this. I agree with the suggestion though, so I propose to\nrather handle it at a later point in time.\n\nA test run of this patch series can be found at [2].\n\nThanks!\n\nPatrick\n\n[1]: \n[2]: https://gitlab.com/gitlab-org/git/-/pipelines/1054750795\n\nPatrick Steinhardt (8):\n ci: reorder definitions for grouping functions\n ci: make grouping setup more generic\n ci: group installation of Docker dependencies\n ci: split out logic to set up failed test artifacts\n ci: unify setup of some environment variables\n ci: squelch warnings when testing with unusable Git repo\n ci: install test dependencies for linux-musl\n ci: add support for GitLab CI\n\n .gitlab-ci.yml | 53 +++++++++\n ci/install-docker-dependencies.sh | 22 +++-\n ci/lib.sh | 187 +++++++++++++++++++++---------\n ci/print-test-failures.sh | 6 +\n t/lib-httpd.sh | 3 +-\n 5 files changed, 215 insertions(+), 56 deletions(-)\n create mode 100644 .gitlab-ci.yml\n\nRange-diff against v2:\n1: 4eb9cfc816b = 1: ef44ed5c3b1 ci: reorder definitions for grouping functions\n2: 85617ef8577 = 2: 77798fa7a7a ci: make grouping setup more generic\n3: 57bbc50e3dc = 3: 4542bd38dc2 ci: group installation of Docker dependencies\n4: 5ab11d5236d = 4: 5fdda7fd83f ci: split out logic to set up failed test artifacts\n-: ----------- > 5: 6af0075fd87 ci: unify setup of some environment variables\n-: ----------- > 6: 78d863bf24e ci: squelch warnings when testing with unusable Git repo\n-: ----------- > 7: f150d61a1ce ci: install test dependencies for linux-musl\n5: 37a507e9b25 ! 8: 5272d66d9f1 ci: add support for GitLab CI\n @@ Commit message\n \n This commit introduces the integration into our regular CI scripts so\n that most of the setup continues to be shared across all of the CI\n - solutions.\n + solutions. Note that as the builds on GitLab CI run as unprivileged\n + user, we need to pull in both sudo and shadow packages to our Alpine\n + based job to set this up.\n \n [1]: https://gitlab.com/gitlab-org/git\n \n @@ .gitlab-ci.yml (new)\n + before_script:\n + - ./ci/install-docker-dependencies.sh\n + script:\n -+ - useradd builder --home-dir \"${CI_PROJECT_DIR}\"\n ++ - useradd builder --create-home\n + - chown -R builder \"${CI_PROJECT_DIR}\"\n + - sudo --preserve-env --set-home --user=builder ./ci/run-build-and-tests.sh\n + after_script:\n @@ .gitlab-ci.yml (new)\n + - jobname: linux-asan-ubsan\n + image: ubuntu:latest\n + CC: clang\n ++ - jobname: pedantic\n ++ image: fedora:latest\n + - jobname: linux-musl\n + image: alpine:latest\n + artifacts:\n @@ ci/install-docker-dependencies.sh: linux32)\n \t;;\n linux-musl)\n -\tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n -+\tapk add --update git shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n - \t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n ++\tapk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n + \t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n + \t\tapache2 bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n \t;;\n +linux-*)\n +\tapt update -q &&\n -+\tapt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}}\n ++\tapt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev \\\n ++\t\tlibcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev \\\n ++\t\tperl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl \\\n ++\t\tlibdbd-sqlite3-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}} \\\n ++\t\tapache2 cvs cvsps gnupg libcgi-pm-perl subversion\n +\t;;\n pedantic)\n \tdnf -yq update >/dev/null &&\n @@ ci/lib.sh: then\n +then\n +\tbegin_group () {\n +\t\tneed_to_end_group=t\n -+\t\techo -e \"\\e[0Ksection_start:$(date +%s):$(echo \"$1\" | tr ' ' _)\\r\\e[0K$1\"\n ++\t\tprintf \"\\e[0Ksection_start:$(date +%s):$(echo \"$1\" | tr ' ' _)\\r\\e[0K$1\\n\"\n +\t\ttrap \"end_group '$1'\" EXIT\n +\t\tset -x\n +\t}\n @@ ci/lib.sh: then\n +\t\ttest -n \"$need_to_end_group\" || return 0\n +\t\tset +x\n +\t\tneed_to_end_group=\n -+\t\techo -e \"\\e[0Ksection_end:$(date +%s):$(echo \"$1\" | tr ' ' _)\\r\\e[0K\"\n ++\t\tprintf \"\\e[0Ksection_end:$(date +%s):$(echo \"$1\" | tr ' ' _)\\r\\e[0K\\n\"\n +\t\ttrap - EXIT\n +\t}\n else\n \tbegin_group () { :; }\n \tend_group () { :; }\n @@ ci/lib.sh: then\n - \tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n - \ttest windows != \"$CI_OS_NAME\" ||\n - \tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n + \tcache_dir=\"$HOME/none\"\n + \n + \tGIT_TEST_OPTS=\"--github-workflow-markup\"\n +elif test true = \"$GITLAB_CI\"\n +then\n +\tCI_TYPE=gitlab-ci\n @@ ci/lib.sh: then\n +\tcase \"$CI_JOB_IMAGE\" in\n +\tmacos-*)\n +\t\tCI_OS_NAME=osx;;\n -+\talpine:*|ubuntu:*)\n ++\talpine:*|fedora:*|ubuntu:*)\n +\t\tCI_OS_NAME=linux;;\n +\t*)\n +\t\techo \"Could not identify OS image\" >&2\n @@ ci/lib.sh: then\n +\tcache_dir=\"$HOME/none\"\n +\n +\truns_on_pool=$(echo \"$CI_JOB_IMAGE\" | tr : -)\n -+\n -+\texport GIT_PROVE_OPTS=\"--timer --jobs $(nproc)\"\n -+\texport GIT_TEST_OPTS=\"--verbose-log -x\"\n -+\tMAKEFLAGS=\"$MAKEFLAGS --jobs=$(nproc)\"\n -+\ttest windows != \"$CI_OS_NAME\" ||\n -+\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n else\n \techo \"Could not identify CI type\" >&2\n \tenv >&2\n-- \n2.42.0\n\n"},{"id":"484099","messageId":"ef44ed5c3b19ec1479b3dceca8d2f64de3409a5c.1698667545.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698667545.git.ps@pks.im","subject":"[PATCH v3 1/8] ci: reorder definitions for grouping functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-30T12:14:43Z","receivedAt":"2023-10-30T12:14:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We define a set of grouping functions that are used to group together\noutput in our CI, where these groups then end up as collapsible sections\nin the respective pipeline platform. The way these functions are defined\nis not easily extensible though as we have an up front check for the CI\n_not_ being GitHub Actions, where we define the non-stub logic in the\nelse branch.\n\nReorder the conditional branches such that we explicitly handle GitHub\nActions.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 20 ++++++++++----------\n 1 file changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 6fbb5bade12..eb384f4e952 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -1,16 +1,7 @@\n # Library of functions shared by all CI scripts\n \n-if test true != \"$GITHUB_ACTIONS\"\n+if test true = \"$GITHUB_ACTIONS\"\n then\n-\tbegin_group () { :; }\n-\tend_group () { :; }\n-\n-\tgroup () {\n-\t\tshift\n-\t\t\"$@\"\n-\t}\n-\tset -x\n-else\n \tbegin_group () {\n \t\tneed_to_end_group=t\n \t\techo \"::group::$1\" >&2\n@@ -42,6 +33,15 @@ else\n \t}\n \n \tbegin_group \"CI setup\"\n+else\n+\tbegin_group () { :; }\n+\tend_group () { :; }\n+\n+\tgroup () {\n+\t\tshift\n+\t\t\"$@\"\n+\t}\n+\tset -x\n fi\n \n # Set 'exit on error' for all CI scripts to let the caller know that\n-- \n2.42.0\n\n"},{"id":"484100","messageId":"77798fa7a7aa084c1bbd9fc9118346c47770125a.1698667545.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698667545.git.ps@pks.im","subject":"[PATCH v3 2/8] ci: make grouping setup more generic","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-30T12:14:48Z","receivedAt":"2023-10-30T12:14:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Make the grouping setup more generic by always calling `begin_group ()`\nand `end_group ()` regardless of whether we have stubbed those functions\nor not. This ensures we can more readily add support for additional CI\nplatforms.\n\nFurthermore, the `group ()` function is made generic so that it is the\nsame for both GitHub Actions and for other platforms. There is a\nsemantic conflict here though: GitHub Actions used to call `set +x` in\n`group ()` whereas the non-GitHub case unconditionally uses `set -x`.\nThe latter would get overriden if we kept the `set +x` in the generic\nversion of `group ()`. To resolve this conflict, we simply drop the `set\n+x` in the generic variant of this function. As `begin_group ()` calls\n`set -x` anyway this is not much of a change though, as the only\ncommands that aren't printed anymore now are the ones between the\nbeginning of `group ()` and the end of `begin_group ()`.\n\nLast, this commit changes `end_group ()` to also accept a parameter that\nindicates _which_ group should end. This will be required by a later\ncommit that introduces support for GitLab CI.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 46 ++++++++++++++++++++++------------------------\n 1 file changed, 22 insertions(+), 24 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex eb384f4e952..b3411afae8e 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -14,36 +14,34 @@ then\n \t\tneed_to_end_group=\n \t\techo '::endgroup::' >&2\n \t}\n-\ttrap end_group EXIT\n-\n-\tgroup () {\n-\t\tset +x\n-\t\tbegin_group \"$1\"\n-\t\tshift\n-\t\t# work around `dash` not supporting `set -o pipefail`\n-\t\t(\n-\t\t\t\"$@\" 2>&1\n-\t\t\techo $? >exit.status\n-\t\t) |\n-\t\tsed 's/^\\(\\([^ ]*\\):\\([0-9]*\\):\\([0-9]*:\\) \\)\\(error\\|warning\\): /::\\5 file=\\2,line=\\3::\\1/'\n-\t\tres=$(cat exit.status)\n-\t\trm exit.status\n-\t\tend_group\n-\t\treturn $res\n-\t}\n-\n-\tbegin_group \"CI setup\"\n else\n \tbegin_group () { :; }\n \tend_group () { :; }\n \n-\tgroup () {\n-\t\tshift\n-\t\t\"$@\"\n-\t}\n \tset -x\n fi\n \n+group () {\n+\tgroup=\"$1\"\n+\tshift\n+\tbegin_group \"$group\"\n+\n+\t# work around `dash` not supporting `set -o pipefail`\n+\t(\n+\t\t\"$@\" 2>&1\n+\t\techo $? >exit.status\n+\t) |\n+\tsed 's/^\\(\\([^ ]*\\):\\([0-9]*\\):\\([0-9]*:\\) \\)\\(error\\|warning\\): /::\\5 file=\\2,line=\\3::\\1/'\n+\tres=$(cat exit.status)\n+\trm exit.status\n+\n+\tend_group \"$group\"\n+\treturn $res\n+}\n+\n+begin_group \"CI setup\"\n+trap \"end_group 'CI setup'\" EXIT\n+\n # Set 'exit on error' for all CI scripts to let the caller know that\n # something went wrong.\n #\n@@ -287,5 +285,5 @@ esac\n \n MAKEFLAGS=\"$MAKEFLAGS CC=${CC:-cc}\"\n \n-end_group\n+end_group \"CI setup\"\n set -x\n-- \n2.42.0\n\n"},{"id":"484101","messageId":"4542bd38dc270735f5a2674cf058f008746adb53.1698667545.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698667545.git.ps@pks.im","subject":"[PATCH v3 3/8] ci: group installation of Docker dependencies","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-30T12:14:52Z","receivedAt":"2023-10-30T12:14:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The output of CI jobs tends to be quite long-winded and hard to digest.\nTo help with this, many CI systems provide the ability to group output\ninto collapsible sections, and we're also doing this in some of our\nscripts.\n\nOne notable omission is the script to install Docker dependencies.\nAddress it to bring more structure to the output for Docker-based jobs.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-docker-dependencies.sh | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\nindex 78b7e326da6..d0bc19d3bb3 100755\n--- a/ci/install-docker-dependencies.sh\n+++ b/ci/install-docker-dependencies.sh\n@@ -3,6 +3,10 @@\n # Install dependencies required to build and test Git inside container\n #\n \n+. ${0%/*}/lib.sh\n+\n+begin_group \"Install dependencies\"\n+\n case \"$jobname\" in\n linux32)\n \tlinux32 --32bit i386 sh -c '\n@@ -20,3 +24,5 @@ pedantic)\n \tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n \t;;\n esac\n+\n+end_group \"Install dependencies\"\n-- \n2.42.0\n\n"},{"id":"484102","messageId":"5fdda7fd83f21422fb47ee86ae5372357cc4d846.1698667545.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698667545.git.ps@pks.im","subject":"[PATCH v3 4/8] ci: split out logic to set up failed test artifacts","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-30T12:14:57Z","receivedAt":"2023-10-30T12:15:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have some logic in place to create a directory with the output from\nfailed tests, which will then subsequently be uploaded as CI artifacts.\nWe're about to add support for GitLab CI, which will want to reuse the\nlogic.\n\nSplit the logic into a separate function so that it is reusable.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 40 ++++++++++++++++++++++------------------\n 1 file changed, 22 insertions(+), 18 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex b3411afae8e..9ffdf743903 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -131,6 +131,27 @@ handle_failed_tests () {\n \treturn 1\n }\n \n+create_failed_test_artifacts () {\n+\tmkdir -p t/failed-test-artifacts\n+\n+\tfor test_exit in t/test-results/*.exit\n+\tdo\n+\t\ttest 0 != \"$(cat \"$test_exit\")\" || continue\n+\n+\t\ttest_name=\"${test_exit%.exit}\"\n+\t\ttest_name=\"${test_name##*/}\"\n+\t\tprintf \"\\\\e[33m\\\\e[1m=== Failed test: ${test_name} ===\\\\e[m\\\\n\"\n+\t\techo \"The full logs are in the 'print test failures' step below.\"\n+\t\techo \"See also the 'failed-tests-*' artifacts attached to this run.\"\n+\t\tcat \"t/test-results/$test_name.markup\"\n+\n+\t\ttrash_dir=\"t/trash directory.$test_name\"\n+\t\tcp \"t/test-results/$test_name.out\" t/failed-test-artifacts/\n+\t\ttar czf t/failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n+\tdone\n+\treturn 1\n+}\n+\n # GitHub Action doesn't set TERM, which is required by tput\n export TERM=${TERM:-dumb}\n \n@@ -171,25 +192,8 @@ then\n \tCC=\"${CC_PACKAGE:-${CC:-gcc}}\"\n \tDONT_SKIP_TAGS=t\n \thandle_failed_tests () {\n-\t\tmkdir -p t/failed-test-artifacts\n \t\techo \"FAILED_TEST_ARTIFACTS=t/failed-test-artifacts\" >>$GITHUB_ENV\n-\n-\t\tfor test_exit in t/test-results/*.exit\n-\t\tdo\n-\t\t\ttest 0 != \"$(cat \"$test_exit\")\" || continue\n-\n-\t\t\ttest_name=\"${test_exit%.exit}\"\n-\t\t\ttest_name=\"${test_name##*/}\"\n-\t\t\tprintf \"\\\\e[33m\\\\e[1m=== Failed test: ${test_name} ===\\\\e[m\\\\n\"\n-\t\t\techo \"The full logs are in the 'print test failures' step below.\"\n-\t\t\techo \"See also the 'failed-tests-*' artifacts attached to this run.\"\n-\t\t\tcat \"t/test-results/$test_name.markup\"\n-\n-\t\t\ttrash_dir=\"t/trash directory.$test_name\"\n-\t\t\tcp \"t/test-results/$test_name.out\" t/failed-test-artifacts/\n-\t\t\ttar czf t/failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n-\t\tdone\n-\t\treturn 1\n+\t\tcreate_failed_test_artifacts\n \t}\n \n \tcache_dir=\"$HOME/none\"\n-- \n2.42.0\n\n"},{"id":"484103","messageId":"6af0075fd875f176e7fdf6c219e7117dac5cd71c.1698667545.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698667545.git.ps@pks.im","subject":"[PATCH v3 5/8] ci: unify setup of some environment variables","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-30T12:15:01Z","receivedAt":"2023-10-30T12:15:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Both GitHub Actions and Azue Pipelines set up the environment variables\nGIT_TEST_OPTS, GIT_PROVE_OPTS and MAKEFLAGS. And while most values are\nactually the same, the setup is completely duplicate. With the upcoming\nsupport for GitLab CI this duplication would only extend even further.\n\nUnify the setup of those environment variables so that only the uncommon\nparts are separated. While at it, we also perform some additional small\nimprovements:\n\n - We use nproc instead of a hardcoded count of jobs for make and\n prove. This ensures that the number of concurrent processes adapts\n to the host automatically.\n\n - We now always pass `--state=failed,slow,save` via GIT_PROVE_OPTS.\n It doesn't hurt on platforms where we don't persist the state, so\n this further reduces boilerplate.\n\n - When running on Windows systems we set `--no-chain-lint` and\n `--no-bin-wrappers`. Interestingly though, we did so _after_\n already having exported the respective environment variables.\n\n - We stop using `export VAR=value` syntax, which is a Bashism. It's\n not quite worth it as we still use this syntax all over the place,\n but it doesn't hurt readability either.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 24 ++++++++++++++----------\n 1 file changed, 14 insertions(+), 10 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 9ffdf743903..c7a716a6e3f 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -175,11 +175,7 @@ then\n \t# among *all* phases)\n \tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n \n-\texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n-\texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n-\tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n-\ttest windows_nt != \"$CI_OS_NAME\" ||\n-\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n+\tGIT_TEST_OPTS=\"--write-junit-xml\"\n elif test true = \"$GITHUB_ACTIONS\"\n then\n \tCI_TYPE=github-actions\n@@ -198,17 +194,25 @@ then\n \n \tcache_dir=\"$HOME/none\"\n \n-\texport GIT_PROVE_OPTS=\"--timer --jobs 10\"\n-\texport GIT_TEST_OPTS=\"--verbose-log -x --github-workflow-markup\"\n-\tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n-\ttest windows != \"$CI_OS_NAME\" ||\n-\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n+\tGIT_TEST_OPTS=\"--github-workflow-markup\"\n else\n \techo \"Could not identify CI type\" >&2\n \tenv >&2\n \texit 1\n fi\n \n+MAKEFLAGS=\"$MAKEFLAGS --jobs=$(nproc)\"\n+GIT_PROVE_OPTS=\"--timer --jobs $(nproc) --state=failed,slow,save\"\n+\n+GIT_TEST_OPTS=\"$GIT_TEST_OPTS --verbose-log -x\"\n+if test windows = \"$CI_OS_NAME\"\n+then\n+\tGIT_TEST_OPTS=\"$GIT_TEST_OPTS --no-chain-lint --no-bin-wrappers\"\n+fi\n+\n+export GIT_TEST_OPTS\n+export GIT_PROVE_OPTS\n+\n good_trees_file=\"$cache_dir/good-trees\"\n \n mkdir -p \"$cache_dir\"\n-- \n2.42.0\n\n"},{"id":"484104","messageId":"78d863bf24ea3190c87240fa0c2f6b42fdcde3c3.1698667545.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698667545.git.ps@pks.im","subject":"[PATCH v3 6/8] ci: squelch warnings when testing with unusable Git repo","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-30T12:15:05Z","receivedAt":"2023-10-30T12:15:10Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Our CI jobs that run on Docker also use mostly the same architecture to\nbuild and test Git via the \"ci/run-build-and-tests.sh\" script. These\nscripts also provide some functionality to massage the Git repository\nwe're supposedly operating in.\n\nIn our Docker-based infrastructure we may not even have a Git repository\navailable though, which leads to warnings when those functions execute.\nMake the helpers exit gracefully in case either there is no Git in our\nPATH, or when not running in a Git repository.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 32 ++++++++++++++++++++++++++++++++\n 1 file changed, 32 insertions(+)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex c7a716a6e3f..c13b6527cac 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -69,10 +69,32 @@ skip_branch_tip_with_tag () {\n \tfi\n }\n \n+# Check whether we can use the path passed via the first argument as Git\n+# repository.\n+is_usable_git_repository () {\n+\t# We require Git in our PATH, otherwise we cannot access repositories\n+\t# at all.\n+\tif ! command -v git >/dev/null\n+\tthen\n+\t\treturn 1\n+\tfi\n+\n+\t# And the target directory needs to be a proper Git repository.\n+\tif ! git -C \"$1\" rev-parse 2>/dev/null\n+\tthen\n+\t\treturn 1\n+\tfi\n+}\n+\n # Save some info about the current commit's tree, so we can skip the build\n # job if we encounter the same tree again and can provide a useful info\n # message.\n save_good_tree () {\n+\tif ! is_usable_git_repository .\n+\tthen\n+\t\treturn\n+\tfi\n+\n \techo \"$(git rev-parse $CI_COMMIT^{tree}) $CI_COMMIT $CI_JOB_NUMBER $CI_JOB_ID\" >>\"$good_trees_file\"\n \t# limit the file size\n \ttail -1000 \"$good_trees_file\" >\"$good_trees_file\".tmp\n@@ -88,6 +110,11 @@ skip_good_tree () {\n \t\treturn\n \tfi\n \n+\tif ! is_usable_git_repository .\n+\tthen\n+\t\treturn\n+\tfi\n+\n \tif ! good_tree_info=\"$(grep \"^$(git rev-parse $CI_COMMIT^{tree}) \" \"$good_trees_file\")\"\n \tthen\n \t\t# Haven't seen this tree yet, or no cached good trees file yet.\n@@ -119,6 +146,11 @@ skip_good_tree () {\n }\n \n check_unignored_build_artifacts () {\n+\tif ! is_usable_git_repository .\n+\tthen\n+\t\treturn\n+\tfi\n+\n \t! git ls-files --other --exclude-standard --error-unmatch \\\n \t\t-- ':/*' 2>/dev/null ||\n \t{\n-- \n2.42.0\n\n"},{"id":"484105","messageId":"f150d61a1ceec5dd6cf370c3f4421a3e2d9f11a3.1698667545.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698667545.git.ps@pks.im","subject":"[PATCH v3 7/8] ci: install test dependencies for linux-musl","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-30T12:15:10Z","receivedAt":"2023-10-30T12:15:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The linux-musl CI job executes tests on Alpine Linux, which is based on\nmusl libc instead of glibc. We're missing some test dependencies though,\nwhich causes us to skip a subset of tests.\n\nInstall these test dependencies to increase our test coverage on this\nplatform. There are still some missing test dependecies, but these do\nnot have a corresponding package in the Alpine repositories:\n\n - p4 and p4d, both parts of the Perforce version control system.\n\n - cvsps, which generates patch sets for CVS.\n\n - Subversion and the SVN::Core Perl library, the latter of which is\n not available in the Alpine repositories. While the tool itself is\n available, all Subversion-related tests are skipped without the\n SVN::Core Perl library anyway.\n\nFurthermore, in order to make the Apache-based tests work, this commit\nalso adds the Alpine-specific modules path of it to the list of known\npaths.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-docker-dependencies.sh | 3 ++-\n t/lib-httpd.sh | 3 ++-\n 2 files changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\nindex d0bc19d3bb3..05dde5c5d40 100755\n--- a/ci/install-docker-dependencies.sh\n+++ b/ci/install-docker-dependencies.sh\n@@ -17,7 +17,8 @@ linux32)\n \t;;\n linux-musl)\n \tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n-\t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n+\t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n+\t\tapache2 bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n \t;;\n pedantic)\n \tdnf -yq update >/dev/null &&\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex 2fb1b2ae561..9791f94b16f 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -67,7 +67,8 @@ for DEFAULT_HTTPD_MODULE_PATH in '/usr/libexec/apache2' \\\n \t\t\t\t '/usr/lib/apache2/modules' \\\n \t\t\t\t '/usr/lib64/httpd/modules' \\\n \t\t\t\t '/usr/lib/httpd/modules' \\\n-\t\t\t\t '/usr/libexec/httpd'\n+\t\t\t\t '/usr/libexec/httpd' \\\n+\t\t\t\t '/usr/lib/apache2'\n do\n \tif test -d \"$DEFAULT_HTTPD_MODULE_PATH\"\n \tthen\n-- \n2.42.0\n\n"},{"id":"484106","messageId":"5272d66d9f138987def342ff7661515b845318bb.1698667545.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698667545.git.ps@pks.im","subject":"[PATCH v3 8/8] ci: add support for GitLab CI","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-30T12:15:14Z","receivedAt":"2023-10-30T12:15:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We already support Azure Pipelines and GitHub Workflows in the Git\nproject, but until now we do not have support for GitLab CI. While it is\narguably not in the interest of the Git project to maintain a ton of\ndifferent CI platforms, GitLab has recently ramped up its efforts and\ntries to contribute to the Git project more regularly.\n\nPart of a problem we hit at GitLab rather frequently is that our own,\ncustom CI setup we have is so different to the setup that the Git\nproject has. More esoteric jobs like \"linux-TEST-vars\" that also set a\ncouple of environment variables do not exist in GitLab's custom CI\nsetup, and maintaining them to keep up with what Git does feels like\nwasted time. The result is that we regularly send patch series upstream\nthat fail to compile or pass tests in GitHub Workflows. We would thus\nlike to integrate the GitLab CI configuration into the Git project to\nhelp us send better patch series upstream and thus reduce overhead for\nthe maintainer.\n\nThe integration does not necessarily have to be a first-class citizen,\nwhich would in practice only add to the fallout that pipeline failures\nhave for the maintainer. That being said, we are happy to maintain this\nalternative CI setup for the Git project and will make test results\navailable as part of our own mirror of the Git project at [1].\n\nThis commit introduces the integration into our regular CI scripts so\nthat most of the setup continues to be shared across all of the CI\nsolutions. Note that as the builds on GitLab CI run as unprivileged\nuser, we need to pull in both sudo and shadow packages to our Alpine\nbased job to set this up.\n\n[1]: https://gitlab.com/gitlab-org/git\n\nSigned-off-by: Patrick Steinhardt \n---\n .gitlab-ci.yml | 53 +++++++++++++++++++++++++++++++\n ci/install-docker-dependencies.sh | 13 +++++++-\n ci/lib.sh | 43 +++++++++++++++++++++++++\n ci/print-test-failures.sh | 6 ++++\n 4 files changed, 114 insertions(+), 1 deletion(-)\n create mode 100644 .gitlab-ci.yml\n\ndiff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\nnew file mode 100644\nindex 00000000000..cd98bcb18aa\n--- /dev/null\n+++ b/.gitlab-ci.yml\n@@ -0,0 +1,53 @@\n+default:\n+ timeout: 2h\n+\n+workflow:\n+ rules:\n+ - if: $CI_PIPELINE_SOURCE == \"merge_request_event\"\n+ - if: $CI_COMMIT_TAG\n+ - if: $CI_COMMIT_REF_PROTECTED == \"true\"\n+\n+test:\n+ image: $image\n+ before_script:\n+ - ./ci/install-docker-dependencies.sh\n+ script:\n+ - useradd builder --create-home\n+ - chown -R builder \"${CI_PROJECT_DIR}\"\n+ - sudo --preserve-env --set-home --user=builder ./ci/run-build-and-tests.sh\n+ after_script:\n+ - |\n+ if test \"$CI_JOB_STATUS\" != 'success'\n+ then\n+ sudo --preserve-env --set-home --user=builder ./ci/print-test-failures.sh\n+ fi\n+ parallel:\n+ matrix:\n+ - jobname: linux-sha256\n+ image: ubuntu:latest\n+ CC: clang\n+ - jobname: linux-gcc\n+ image: ubuntu:20.04\n+ CC: gcc\n+ CC_PACKAGE: gcc-8\n+ - jobname: linux-TEST-vars\n+ image: ubuntu:20.04\n+ CC: gcc\n+ CC_PACKAGE: gcc-8\n+ - jobname: linux-gcc-default\n+ image: ubuntu:latest\n+ CC: gcc\n+ - jobname: linux-leaks\n+ image: ubuntu:latest\n+ CC: gcc\n+ - jobname: linux-asan-ubsan\n+ image: ubuntu:latest\n+ CC: clang\n+ - jobname: pedantic\n+ image: fedora:latest\n+ - jobname: linux-musl\n+ image: alpine:latest\n+ artifacts:\n+ paths:\n+ - t/failed-test-artifacts\n+ when: on_failure\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\nindex 05dde5c5d40..5e28adf55b6 100755\n--- a/ci/install-docker-dependencies.sh\n+++ b/ci/install-docker-dependencies.sh\n@@ -7,6 +7,9 @@\n \n begin_group \"Install dependencies\"\n \n+# Required so that apt doesn't wait for user input on certain packages.\n+export DEBIAN_FRONTEND=noninteractive\n+\n case \"$jobname\" in\n linux32)\n \tlinux32 --32bit i386 sh -c '\n@@ -16,10 +19,18 @@ linux32)\n \t'\n \t;;\n linux-musl)\n-\tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n+\tapk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n \t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n \t\tapache2 bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n \t;;\n+linux-*)\n+\tapt update -q &&\n+\tapt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev \\\n+\t\tlibcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev \\\n+\t\tperl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl \\\n+\t\tlibdbd-sqlite3-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}} \\\n+\t\tapache2 cvs cvsps gnupg libcgi-pm-perl subversion\n+\t;;\n pedantic)\n \tdnf -yq update >/dev/null &&\n \tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex c13b6527cac..d836fad7dd6 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -14,6 +14,22 @@ then\n \t\tneed_to_end_group=\n \t\techo '::endgroup::' >&2\n \t}\n+elif test true = \"$GITLAB_CI\"\n+then\n+\tbegin_group () {\n+\t\tneed_to_end_group=t\n+\t\tprintf \"\\e[0Ksection_start:$(date +%s):$(echo \"$1\" | tr ' ' _)\\r\\e[0K$1\\n\"\n+\t\ttrap \"end_group '$1'\" EXIT\n+\t\tset -x\n+\t}\n+\n+\tend_group () {\n+\t\ttest -n \"$need_to_end_group\" || return 0\n+\t\tset +x\n+\t\tneed_to_end_group=\n+\t\tprintf \"\\e[0Ksection_end:$(date +%s):$(echo \"$1\" | tr ' ' _)\\r\\e[0K\\n\"\n+\t\ttrap - EXIT\n+\t}\n else\n \tbegin_group () { :; }\n \tend_group () { :; }\n@@ -227,6 +243,33 @@ then\n \tcache_dir=\"$HOME/none\"\n \n \tGIT_TEST_OPTS=\"--github-workflow-markup\"\n+elif test true = \"$GITLAB_CI\"\n+then\n+\tCI_TYPE=gitlab-ci\n+\tCI_BRANCH=\"$CI_COMMIT_REF_NAME\"\n+\tCI_COMMIT=\"$CI_COMMIT_SHA\"\n+\tcase \"$CI_JOB_IMAGE\" in\n+\tmacos-*)\n+\t\tCI_OS_NAME=osx;;\n+\talpine:*|fedora:*|ubuntu:*)\n+\t\tCI_OS_NAME=linux;;\n+\t*)\n+\t\techo \"Could not identify OS image\" >&2\n+\t\tenv >&2\n+\t\texit 1\n+\t\t;;\n+\tesac\n+\tCI_REPO_SLUG=\"$CI_PROJECT_PATH\"\n+\tCI_JOB_ID=\"$CI_JOB_ID\"\n+\tCC=\"${CC_PACKAGE:-${CC:-gcc}}\"\n+\tDONT_SKIP_TAGS=t\n+\thandle_failed_tests () {\n+\t\tcreate_failed_test_artifacts\n+\t}\n+\n+\tcache_dir=\"$HOME/none\"\n+\n+\truns_on_pool=$(echo \"$CI_JOB_IMAGE\" | tr : -)\n else\n \techo \"Could not identify CI type\" >&2\n \tenv >&2\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex 57277eefcd0..c33ad4e3a22 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -51,6 +51,12 @@ do\n \t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n \t\t\tcontinue\n \t\t\t;;\n+\t\tgitlab-ci)\n+\t\t\tmkdir -p failed-test-artifacts\n+\t\t\tcp \"${TEST_EXIT%.exit}.out\" failed-test-artifacts/\n+\t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n+\t\t\tcontinue\n+\t\t\t;;\n \t\t*)\n \t\t\techo \"Unhandled CI type: $CI_TYPE\" >&2\n \t\t\texit 1\n-- \n2.42.0\n\n"},{"id":"484107","messageId":"ZT-l4qDDNloizjxl@tanuki","threadId":"60432","inReplyTo":"f150d61a1ceec5dd6cf370c3f4421a3e2d9f11a3.1698667545.git.ps@pks.im","subject":"Re: [PATCH v3 7/8] ci: install test dependencies for linux-musl","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-30T12:47:30Z","receivedAt":"2023-10-30T12:47:38Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Oct 30, 2023 at 01:15:10PM +0100, Patrick Steinhardt wrote:\n> The linux-musl CI job executes tests on Alpine Linux, which is based on\n> musl libc instead of glibc. We're missing some test dependencies though,\n> which causes us to skip a subset of tests.\n> \n> Install these test dependencies to increase our test coverage on this\n> platform. There are still some missing test dependecies, but these do\n> not have a corresponding package in the Alpine repositories:\n> \n> - p4 and p4d, both parts of the Perforce version control system.\n> \n> - cvsps, which generates patch sets for CVS.\n> \n> - Subversion and the SVN::Core Perl library, the latter of which is\n> not available in the Alpine repositories. While the tool itself is\n> available, all Subversion-related tests are skipped without the\n> SVN::Core Perl library anyway.\n> \n> Furthermore, in order to make the Apache-based tests work, this commit\n> also adds the Alpine-specific modules path of it to the list of known\n> paths.\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n> ci/install-docker-dependencies.sh | 3 ++-\n> t/lib-httpd.sh | 3 ++-\n> 2 files changed, 4 insertions(+), 2 deletions(-)\n> \n> diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\n> index d0bc19d3bb3..05dde5c5d40 100755\n> --- a/ci/install-docker-dependencies.sh\n> +++ b/ci/install-docker-dependencies.sh\n> @@ -17,7 +17,8 @@ linux32)\n> \t;;\n> linux-musl)\n> \tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n> -\t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n> +\t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n> +\t\tapache2 bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n\nMeh. I just noticed that I missed a few other dependencies to make\nApache2 work:\n\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\nindex 5e28adf55b..ce910e3f3c 100755\n--- a/ci/install-docker-dependencies.sh\n+++ b/ci/install-docker-dependencies.sh\n@@ -21,7 +21,8 @@ linux32)\n linux-musl)\n \tapk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n \t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n-\t\tapache2 bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n+\t\tapache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav \\\n+\t\tbash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n \t;;\n linux-*)\n \tapt update -q &&\n\nBut once fixed, tests do indeed start to fail:\n\nt5540-http-push-webdav.sh (Wstat: 256 (exited 1) Tests: 20 Failed: 11)\n Failed tests: 5-11, 13, 15-16, 18\n Non-zero exit status: 1\n\nSeems like another thing to fix in a separate patch series.\n\nPatrick\n\n> \t;;\n> pedantic)\n> \tdnf -yq update >/dev/null &&\n> diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\n> index 2fb1b2ae561..9791f94b16f 100644\n> --- a/t/lib-httpd.sh\n> +++ b/t/lib-httpd.sh\n> @@ -67,7 +67,8 @@ for DEFAULT_HTTPD_MODULE_PATH in '/usr/libexec/apache2' \\\n> \t\t\t\t '/usr/lib/apache2/modules' \\\n> \t\t\t\t '/usr/lib64/httpd/modules' \\\n> \t\t\t\t '/usr/lib/httpd/modules' \\\n> -\t\t\t\t '/usr/libexec/httpd'\n> +\t\t\t\t '/usr/libexec/httpd' \\\n> +\t\t\t\t '/usr/lib/apache2'\n> do\n> \tif test -d \"$DEFAULT_HTTPD_MODULE_PATH\"\n> \tthen\n> -- \n> 2.42.0\n> \n\n\n"},{"id":"484108","messageId":"ZT-uGSu-ftFRrn-X@tanuki","threadId":"60432","inReplyTo":"ZT-l4qDDNloizjxl@tanuki","subject":"Re: [PATCH v3 7/8] ci: install test dependencies for linux-musl","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-30T13:22:33Z","receivedAt":"2023-10-30T13:22:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Oct 30, 2023 at 01:47:30PM +0100, Patrick Steinhardt wrote:\n> On Mon, Oct 30, 2023 at 01:15:10PM +0100, Patrick Steinhardt wrote:\n> > The linux-musl CI job executes tests on Alpine Linux, which is based on\n> > musl libc instead of glibc. We're missing some test dependencies though,\n> > which causes us to skip a subset of tests.\n> > \n> > Install these test dependencies to increase our test coverage on this\n> > platform. There are still some missing test dependecies, but these do\n> > not have a corresponding package in the Alpine repositories:\n> > \n> > - p4 and p4d, both parts of the Perforce version control system.\n> > \n> > - cvsps, which generates patch sets for CVS.\n> > \n> > - Subversion and the SVN::Core Perl library, the latter of which is\n> > not available in the Alpine repositories. While the tool itself is\n> > available, all Subversion-related tests are skipped without the\n> > SVN::Core Perl library anyway.\n> > \n> > Furthermore, in order to make the Apache-based tests work, this commit\n> > also adds the Alpine-specific modules path of it to the list of known\n> > paths.\n> > \n> > Signed-off-by: Patrick Steinhardt \n> > ---\n> > ci/install-docker-dependencies.sh | 3 ++-\n> > t/lib-httpd.sh | 3 ++-\n> > 2 files changed, 4 insertions(+), 2 deletions(-)\n> > \n> > diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\n> > index d0bc19d3bb3..05dde5c5d40 100755\n> > --- a/ci/install-docker-dependencies.sh\n> > +++ b/ci/install-docker-dependencies.sh\n> > @@ -17,7 +17,8 @@ linux32)\n> > \t;;\n> > linux-musl)\n> > \tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n> > -\t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n> > +\t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n> > +\t\tapache2 bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n> \n> Meh. I just noticed that I missed a few other dependencies to make\n> Apache2 work:\n> \n> diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\n> index 5e28adf55b..ce910e3f3c 100755\n> --- a/ci/install-docker-dependencies.sh\n> +++ b/ci/install-docker-dependencies.sh\n> @@ -21,7 +21,8 @@ linux32)\n> linux-musl)\n> \tapk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n> \t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n> -\t\tapache2 bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n> +\t\tapache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav \\\n> +\t\tbash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n> \t;;\n> linux-*)\n> \tapt update -q &&\n> \n> But once fixed, tests do indeed start to fail:\n> \n> t5540-http-push-webdav.sh (Wstat: 256 (exited 1) Tests: 20 Failed: 11)\n> Failed tests: 5-11, 13, 15-16, 18\n> Non-zero exit status: 1\n> \n> Seems like another thing to fix in a separate patch series.\n> \n> Patrick\n\nI've been digging a bit, and the issue comes from the the DAV module\nindeed:\n\n```\n[Mon Oct 30 12:36:19.776149 2023] [dav_fs:crit] [pid 275752] (20019)DSO load failed: AH00576: The DBM driver could not be loaded\n[Mon Oct 30 12:36:19.776168 2023] [dav:error] [pid 275752] [client 127.0.0.1:51388] Could not LOCK /dumb/test_repo.git/info/refs due to a failed precondition (e.g. other locks). [500, #0]\n[Mon Oct 30 12:36:19.776174 2023] [dav:error] [pid 275752] [client 127.0.0.1:51388] The locks could not be queried for verification against a possible \"If:\" header. [500, #0]\n[Mon Oct 30 12:36:19.776177 2023] [dav:error] [pid 275752] [client 127.0.0.1:51388] Could not open the lock database. [500, #400]\n[Mon Oct 30 12:36:19.776181 2023] [dav:error] [pid 275752] (20019)DSO load failed: [client 127.0.0.1:51388] Could not open property database. [500, #1]\n```\n\nThis seems to be a known limitation in Alpine Linux as they do not\npackage apr-util-dbm_db anymore due to license incompatibilities with\nwith Berkely DB [1], and the WebDAV module does rely on it to provide\nlocking.\n\nIn the best case we'd be able to detect this limitation and skip those\ntests automatically so that we can at least execute all the other Apache\ntests. But again, this rather feels like something we should do as a\nfollow up rather than as part of this series.\n\nPatrick\n\n[1]: https://gitlab.alpinelinux.org/alpine/aports/-/issues/12534\n\n> > \t;;\n> > pedantic)\n> > \tdnf -yq update >/dev/null &&\n> > diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\n> > index 2fb1b2ae561..9791f94b16f 100644\n> > --- a/t/lib-httpd.sh\n> > +++ b/t/lib-httpd.sh\n> > @@ -67,7 +67,8 @@ for DEFAULT_HTTPD_MODULE_PATH in '/usr/libexec/apache2' \\\n> > \t\t\t\t '/usr/lib/apache2/modules' \\\n> > \t\t\t\t '/usr/lib64/httpd/modules' \\\n> > \t\t\t\t '/usr/lib/httpd/modules' \\\n> > -\t\t\t\t '/usr/libexec/httpd'\n> > +\t\t\t\t '/usr/libexec/httpd' \\\n> > +\t\t\t\t '/usr/lib/apache2'\n> > do\n> > \tif test -d \"$DEFAULT_HTTPD_MODULE_PATH\"\n> > \tthen\n> > -- \n> > 2.42.0\n> > \n> \n> \n\n\n"},{"id":"484109","messageId":"33788e805b367f9f0c21aed51d7c4da7@manjaro.org","threadId":"60432","inReplyTo":"68c4684b-0ab0-427a-90e5-fca602c9d993@gmail.com","subject":"Re: [PATCH 5/5] ci: add support for GitLab CI","fromName":"Dragan Simic","fromEmail":"dsimic@manjaro.org","sentAt":"2023-10-30T14:04:18Z","receivedAt":"2023-10-30T14:04:24Z","isPatch":true,"sender":{"key":"dsimic@manjaro.org","avatar":null},"body":"On 2023-10-30 10:49, Phillip Wood wrote:\n> On 27/10/2023 18:47, Oswald Buddenhagen wrote:\n>> On Fri, Oct 27, 2023 at 03:32:48PM +0100, Phillip Wood wrote:\n>>> On 27/10/2023 11:43, Oswald Buddenhagen wrote:\n>>>> On Fri, Oct 27, 2023 at 11:22:35AM +0100, Phillip Wood wrote:\n>>>>>>>> +    CI_BRANCH=\"$CI_COMMIT_REF_NAME\"\n>>>>>>>> +    CI_COMMIT=\"$CI_COMMIT_SHA\"\n>>>>>>>> \n>>>>>>> assignments need no quoting to prevent word splitting.\n>>>>>>> repeats below.\n>>>>>>> \n>>>>> I think it is quite common for us to quote variables when it isn't \n>>>>> strictly necessary as it makes it clear to anyone reading the \n>>>>> script that there is no word splitting going on\n>>>> \n>>>>> and ensures that we don't start splitting the variable if the \n>>>>> contents changes in the future.\n>>>>> \n>>>> the point was that it *isn't* content-dependent; it's simply the \n>>>> shell rules.\n>>> \n>>> Oh, I'd misunderstood what you were saying which was that assignment \n>>> and case statements are not subject to field splitting.\n>>> \n>>>> of course, many people (apparently you included) don't know these \n>>>> subtleties\n>>> \n>>> I find this comment to be condescending, needlessly antagonistic and \n>>> completely at odds with the norms of constructive discussion on this \n>>> list.\n>>> \n>> the observation was necessary for the point i subsequently made (which \n>> was basically agreeing with the first part of your response).\n> \n> It was not necessary to phrase it as you did though. Before replying\n> on Friday I showed your comment to someone else and their reaction was\n> \"That's rude\". You could have made your point by saying something like\n> \n> It is hard to remember all the shell's word splitting rules so\n> quoting everywhere is not a bad idea.\n> \n> This is not the first time I've found your comments unnecessarily\n> adversarial and at odds with the norms of constructive discussion and\n> respectful disagreement on this list. I don't think I'm the only one\n> either - in [1] Junio points out an ad-hominem remark and in [2] Marc\n> comments on the unreceptive tone of you review responses.\n> \n> I would urge you to try and strike a more conciliatory tone in your\n> messages - it is perfectly possible to correct or disagree with\n> someone without alienating them in the process.\n\nYeah, I've also noticed a not so great tone is some of Oswald's \nmessages. It's perfectly fine to disagree on something, but it isn't \nthat great to put the other party down while doing that, even to the \npoint of insulting them.\n\nWe're all humans, and we should treat each other with respect. \nFurthermore, disagreeing in a friendly and polite way can many times \nlead to finding a better solution together.\n\n> Best Wishes\n> \n> Phillip\n> \n> [1] https://lore.kernel.org/git/xmqqleeihok5.fsf@gitster.g/\n> [2] \n> https://lore.kernel.org/git/e33f919d-1b6a-4944-ab5d-93ad0d323b68@xiplink.com/\n"},{"id":"484111","messageId":"87430c6c-91c0-4be1-b89d-bf442b3f018b@gmail.com","threadId":"60432","inReplyTo":"6af0075fd875f176e7fdf6c219e7117dac5cd71c.1698667545.git.ps@pks.im","subject":"Re: [PATCH v3 5/8] ci: unify setup of some environment variables","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-10-30T15:09:01Z","receivedAt":"2023-10-30T15:09:06Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Patrick\n\nOn 30/10/2023 12:15, Patrick Steinhardt wrote:\n> Both GitHub Actions and Azue Pipelines set up the environment variables\n> GIT_TEST_OPTS, GIT_PROVE_OPTS and MAKEFLAGS. And while most values are\n> actually the same, the setup is completely duplicate. With the upcoming\n> support for GitLab CI this duplication would only extend even further.\n> \n> Unify the setup of those environment variables so that only the uncommon\n> parts are separated. While at it, we also perform some additional small\n> improvements:\n> \n> - We use nproc instead of a hardcoded count of jobs for make and\n> prove. This ensures that the number of concurrent processes adapts\n> to the host automatically.\n\nSadly this makes the Windows and MacOS jobs fail on GitHub Actions as \nnproc is not installed[1]. Perhaps we could do\n\n\t--jobs=\"$(nproc || echo 2)\"\n\ninstead. (Maybe 2 is a bit low but the current value of 10 seems pretty \nhigh for the number of cores on the runners that we use)\n\n> - We now always pass `--state=failed,slow,save` via GIT_PROVE_OPTS.\n> It doesn't hurt on platforms where we don't persist the state, so\n> this further reduces boilerplate.\n> \n> - When running on Windows systems we set `--no-chain-lint` and\n> `--no-bin-wrappers`. Interestingly though, we did so _after_\n> already having exported the respective environment variables.\n > > - We stop using `export VAR=value` syntax, which is a Bashism. \nIt's\n> not quite worth it as we still use this syntax all over the place,\n> but it doesn't hurt readability either.\n\nI don't mind this change, but the 'export VAR=value' syntax is in POSIX[2]\n\n> Signed-off-by: Patrick Steinhardt \n> ---\n> ci/lib.sh | 24 ++++++++++++++----------\n> 1 file changed, 14 insertions(+), 10 deletions(-)\n> \n> diff --git a/ci/lib.sh b/ci/lib.sh\n> index 9ffdf743903..c7a716a6e3f 100755\n> --- a/ci/lib.sh\n> +++ b/ci/lib.sh\n> @@ -175,11 +175,7 @@ then\n> \t# among *all* phases)\n> \tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n> \n> -\texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n> -\texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n> -\tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n> -\ttest windows_nt != \"$CI_OS_NAME\" ||\n> -\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n> +\tGIT_TEST_OPTS=\"--write-junit-xml\"\n> elif test true = \"$GITHUB_ACTIONS\"\n> then\n> \tCI_TYPE=github-actions\n> @@ -198,17 +194,25 @@ then\n> \n> \tcache_dir=\"$HOME/none\"\n> \n> -\texport GIT_PROVE_OPTS=\"--timer --jobs 10\"\n> -\texport GIT_TEST_OPTS=\"--verbose-log -x --github-workflow-markup\"\n> -\tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n> -\ttest windows != \"$CI_OS_NAME\" ||\n> -\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n> +\tGIT_TEST_OPTS=\"--github-workflow-markup\"\n> else\n> \techo \"Could not identify CI type\" >&2\n> \tenv >&2\n> \texit 1\n> fi\n> \n> +MAKEFLAGS=\"$MAKEFLAGS --jobs=$(nproc)\"\n> +GIT_PROVE_OPTS=\"--timer --jobs $(nproc) --state=failed,slow,save\"\n> +\n> +GIT_TEST_OPTS=\"$GIT_TEST_OPTS --verbose-log -x\"\n> +if test windows = \"$CI_OS_NAME\"\n> +then\n> +\tGIT_TEST_OPTS=\"$GIT_TEST_OPTS --no-chain-lint --no-bin-wrappers\"\n> +fi\n >\n> +export GIT_TEST_OPTS\n> +export GIT_PROVE_OPTS\n\nI was wondering why we don't export MAKEFLAGS here but it is exported \nearlier on before we set it. Apart from the nproc issue this looks like \na nice improvement\n\nBest Wishes\n\nPhillip\n\n[1] https://github.com/phillipwood/git/actions/runs/6694263874\n[2] \nhttps://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#export\n"},{"id":"484112","messageId":"d675b51b-c50b-43e4-b127-650c0e24d3d0@gmail.com","threadId":"60432","inReplyTo":"ZT-l4qDDNloizjxl@tanuki","subject":"Re: [PATCH v3 7/8] ci: install test dependencies for linux-musl","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-10-30T15:13:35Z","receivedAt":"2023-10-30T15:13:40Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Patrick\n\nOn 30/10/2023 12:47, Patrick Steinhardt wrote:\n> On Mon, Oct 30, 2023 at 01:15:10PM +0100, Patrick Steinhardt wrote:\n> But once fixed, tests do indeed start to fail:\n> \n> t5540-http-push-webdav.sh (Wstat: 256 (exited 1) Tests: 20 Failed: 11)\n> Failed tests: 5-11, 13, 15-16, 18\n> Non-zero exit status: 1\n> \n> Seems like another thing to fix in a separate patch series.\n\nYes, or we could just leave it - I had not realized before that it was \nonly the musl job that was not running the httpd tests (I thought \ninstall-docker-dependencies.sh was missing the packages for ubuntu as \nwell). Given that, the status quo does not seem so bad.\n\nBest Wishes\n\nPhillip\n"},{"id":"484113","messageId":"ZT_JcNH79ncQhzPD@tanuki","threadId":"60432","inReplyTo":"87430c6c-91c0-4be1-b89d-bf442b3f018b@gmail.com","subject":"Re: [PATCH v3 5/8] ci: unify setup of some environment variables","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-30T15:19:12Z","receivedAt":"2023-10-30T15:19:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Oct 30, 2023 at 03:09:01PM +0000, Phillip Wood wrote:\n> Hi Patrick\n> \n> On 30/10/2023 12:15, Patrick Steinhardt wrote:\n> > Both GitHub Actions and Azue Pipelines set up the environment variables\n> > GIT_TEST_OPTS, GIT_PROVE_OPTS and MAKEFLAGS. And while most values are\n> > actually the same, the setup is completely duplicate. With the upcoming\n> > support for GitLab CI this duplication would only extend even further.\n> > \n> > Unify the setup of those environment variables so that only the uncommon\n> > parts are separated. While at it, we also perform some additional small\n> > improvements:\n> > \n> > - We use nproc instead of a hardcoded count of jobs for make and\n> > prove. This ensures that the number of concurrent processes adapts\n> > to the host automatically.\n> \n> Sadly this makes the Windows and MacOS jobs fail on GitHub Actions as nproc\n> is not installed[1]. Perhaps we could do\n> \n> \t--jobs=\"$(nproc || echo 2)\"\n> \n> instead. (Maybe 2 is a bit low but the current value of 10 seems pretty high\n> for the number of cores on the runners that we use)\n\nUgh, thanks. I'll update it to keep the hardcoded 10 jobs in place for\nnow for the other pipelines. Trying to do too many things at once is\nonly going to make it harder to get this landed, doubly so when you have\nno easy way to verify what you're doing :)\n\n> > - We now always pass `--state=failed,slow,save` via GIT_PROVE_OPTS.\n> > It doesn't hurt on platforms where we don't persist the state, so\n> > this further reduces boilerplate.\n> > \n> > - When running on Windows systems we set `--no-chain-lint` and\n> > `--no-bin-wrappers`. Interestingly though, we did so _after_\n> > already having exported the respective environment variables.\n> > > - We stop using `export VAR=value` syntax, which is a Bashism. It's\n> > not quite worth it as we still use this syntax all over the place,\n> > but it doesn't hurt readability either.\n> \n> I don't mind this change, but the 'export VAR=value' syntax is in POSIX[2]\n\nI don't quite mind it, either. The reason why I chose to use it is that\nthere was indeed a bug already with the order of exports and\nmodification of the vars after the export. So with that in mind it made\nsense to me to adapt it accordingly.\n\n> > Signed-off-by: Patrick Steinhardt \n> > ---\n> > ci/lib.sh | 24 ++++++++++++++----------\n> > 1 file changed, 14 insertions(+), 10 deletions(-)\n> > \n> > diff --git a/ci/lib.sh b/ci/lib.sh\n> > index 9ffdf743903..c7a716a6e3f 100755\n> > --- a/ci/lib.sh\n> > +++ b/ci/lib.sh\n> > @@ -175,11 +175,7 @@ then\n> > \t# among *all* phases)\n> > \tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n> > -\texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n> > -\texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n> > -\tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n> > -\ttest windows_nt != \"$CI_OS_NAME\" ||\n> > -\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n> > +\tGIT_TEST_OPTS=\"--write-junit-xml\"\n> > elif test true = \"$GITHUB_ACTIONS\"\n> > then\n> > \tCI_TYPE=github-actions\n> > @@ -198,17 +194,25 @@ then\n> > \tcache_dir=\"$HOME/none\"\n> > -\texport GIT_PROVE_OPTS=\"--timer --jobs 10\"\n> > -\texport GIT_TEST_OPTS=\"--verbose-log -x --github-workflow-markup\"\n> > -\tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n> > -\ttest windows != \"$CI_OS_NAME\" ||\n> > -\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n> > +\tGIT_TEST_OPTS=\"--github-workflow-markup\"\n> > else\n> > \techo \"Could not identify CI type\" >&2\n> > \tenv >&2\n> > \texit 1\n> > fi\n> > +MAKEFLAGS=\"$MAKEFLAGS --jobs=$(nproc)\"\n> > +GIT_PROVE_OPTS=\"--timer --jobs $(nproc) --state=failed,slow,save\"\n> > +\n> > +GIT_TEST_OPTS=\"$GIT_TEST_OPTS --verbose-log -x\"\n> > +if test windows = \"$CI_OS_NAME\"\n> > +then\n> > +\tGIT_TEST_OPTS=\"$GIT_TEST_OPTS --no-chain-lint --no-bin-wrappers\"\n> > +fi\n> >\n> > +export GIT_TEST_OPTS\n> > +export GIT_PROVE_OPTS\n> \n> I was wondering why we don't export MAKEFLAGS here but it is exported\n> earlier on before we set it. Apart from the nproc issue this looks like a\n> nice improvement\n\nIn fact, I also noticed later today that we adapt MAKEFLAGS at a later\npoint again without reexporting it, so it's got the same issue. I'll\nadapt it in the same way.\n\nPatrick\n"},{"id":"484114","messageId":"ZT_KWjxkPEIXHEeH@tanuki","threadId":"60432","inReplyTo":"d675b51b-c50b-43e4-b127-650c0e24d3d0@gmail.com","subject":"Re: [PATCH v3 7/8] ci: install test dependencies for linux-musl","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-30T15:23:06Z","receivedAt":"2023-10-30T15:23:13Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Oct 30, 2023 at 03:13:35PM +0000, Phillip Wood wrote:\n> Hi Patrick\n> \n> On 30/10/2023 12:47, Patrick Steinhardt wrote:\n> > On Mon, Oct 30, 2023 at 01:15:10PM +0100, Patrick Steinhardt wrote:\n> > But once fixed, tests do indeed start to fail:\n> > \n> > t5540-http-push-webdav.sh (Wstat: 256 (exited 1) Tests: 20 Failed: 11)\n> > Failed tests: 5-11, 13, 15-16, 18\n> > Non-zero exit status: 1\n> > \n> > Seems like another thing to fix in a separate patch series.\n> \n> Yes, or we could just leave it - I had not realized before that it was only\n> the musl job that was not running the httpd tests (I thought\n> install-docker-dependencies.sh was missing the packages for ubuntu as well).\n> Given that, the status quo does not seem so bad.\n\nI of course couldn't let go. The following would fix this:\n\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\nindex 5e28adf55b6..48cb2e735b5 100755\n--- a/ci/install-docker-dependencies.sh\n+++ b/ci/install-docker-dependencies.sh\n@@ -21,7 +21,8 @@ linux32)\n linux-musl)\n apk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n pcre2-dev python3 musl-libintl perl-utils ncurses \\\n- apache2 bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n+ apache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \\\n+ bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n ;;\n linux-*)\n apt update -q &&\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex 9791f94b16f..9ea74927c40 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -128,6 +128,20 @@ else\n \"Could not identify web server at '$LIB_HTTPD_PATH'\"\n fi\n\n+if test -n \"$LIB_HTTPD_DAV\" && test -f /etc/os-release\n+then\n+ case \"$(grep \"^ID=\" /etc/os-release | cut -d= -f2-)\" in\n+ alpine)\n+ # The WebDAV module in Alpine Linux is broken at least up to\n+ # Alpine v3.16 as the default DBM driver is missing.\n+ #\n+ # https://gitlab.alpinelinux.org/alpine/aports/-/issues/13112\n+ test_skip_or_die GIT_TEST_HTTPD \\\n+ \"Apache WebDAV module does not have default DBM backend driver\"\n+ ;;\n+ esac\n+fi\n+\n install_script () {\n write_script \"$HTTPD_ROOT_PATH/$1\" <\"$TEST_PATH/$1\"\n }\n\nI might as well roll it into this patch series now. It increases test\ncoverage on musl libc and doesn't have any significant downsides. In\nfact, it uncovers that tests on Alpine Linux don't work right now, so it\nfixes real issues to have the test coverage in our pipelines.\n\nPatrick\n"},{"id":"484118","messageId":"ZT/P5Bl9lD9V6ID9@nand.local","threadId":"60432","inReplyTo":"cover.1698305961.git.ps@pks.im","subject":"Re: [PATCH 0/5] ci: add GitLab CI definition","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-30T15:46:44Z","receivedAt":"2023-10-30T15:46:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Oct 26, 2023 at 09:59:59AM +0200, Patrick Steinhardt wrote:\n> And this is exactly what this patch series does: it adds GitLab-specific\n> knowledge to our CI scripts and adds a CI definition that builds on top\n> of those scripts. This is rather straight forward, as the scripts\n> already know to discern Azure Pipelines and GitHub Actions, and adding\n> a third item to this list feels quite natural. And by building on top of\n> the preexisting infra, the actual \".gitlab-ci.yml\" is really quite\n> small.\n>\n> I acknowledge that the Git project may not be willing to fully support\n> GitLab CI, and that's fine with me. If we want to further stress that\n> point then I'd also be perfectly happy to move the definitions into the\n> \"contrib/\" directory -- it would still be a huge win for our workflow.\n> In any case, I'm happy to keep on maintaining the intgeration with\n> GitLab CI, and if things break I'll do my best to fix them fast.\n\nI don't have any strong opinions here, but my preference would probably\nbe to keep any GitLab-specific CI configuration limited to \"contrib\", if\nit lands in the tree at all.\n\nWe already have a rather complicated CI setup on GitHub, which I think\nwe generally consider authoritative in terms of determining whether \"CI\"\nis green. I know we have some Azure remnants in \"ci\", but I'm not aware\nof any of the details there.\n\nSo I have some hesitation about trying to mirror this rather complicated\nset of build rules in another CI environment. My primary concern would\nbe that the two might fall out of sync and a series that is green on\nGitHub would be red on GitLab, or vice-versa. Importantly, this can\nhappen even without changes to the build definitions, since (AFAICT)\nboth forges distribute new images automatically, so the set of packages\ninstalled in GitHub may not exactly match what's in GitLab (and\nvice-versa).\n\nMy other concern is that we're doubling the cost of any new changes to\nour CI definition. Perhaps this is more of an academic concern, but I\nthink my fear would be that one or the other would fall behind on in\nimplementation leading to further divergence between the two.\n\nI think having the new CI definition live in \"contrib\" somewhat\naddresses the \"which CI is authoritative?\" problem, but that it doesn't\naddress the \"we have two of these\" problem.\n\nSo my preference would probably to have this live out of Junio's tree,\nbut I'm curious to hear what others think.\n\nThanks,\nTaylor\n"},{"id":"484121","messageId":"ba7a7a8c-bb0c-4f12-a7fa-d05b1c246d5f@gmail.com","threadId":"60432","inReplyTo":"ZT_KWjxkPEIXHEeH@tanuki","subject":"Re: [PATCH v3 7/8] ci: install test dependencies for linux-musl","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-10-30T16:09:32Z","receivedAt":"2023-10-30T16:09:36Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 30/10/2023 15:23, Patrick Steinhardt wrote:\n> On Mon, Oct 30, 2023 at 03:13:35PM +0000, Phillip Wood wrote:\n>> Hi Patrick\n>>\n>> On 30/10/2023 12:47, Patrick Steinhardt wrote:\n>>> On Mon, Oct 30, 2023 at 01:15:10PM +0100, Patrick Steinhardt wrote:\n>>> But once fixed, tests do indeed start to fail:\n>>>\n>>> t5540-http-push-webdav.sh (Wstat: 256 (exited 1) Tests: 20 Failed: 11)\n>>> Failed tests: 5-11, 13, 15-16, 18\n>>> Non-zero exit status: 1\n>>>\n>>> Seems like another thing to fix in a separate patch series.\n>>\n>> Yes, or we could just leave it - I had not realized before that it was only\n>> the musl job that was not running the httpd tests (I thought\n>> install-docker-dependencies.sh was missing the packages for ubuntu as well).\n>> Given that, the status quo does not seem so bad.\n> \n> I of course couldn't let go. The following would fix this:\n\nGreat, that was fast! As you've got a fix I agree it makes sense to \ninclude it in this series.\n\nBest Wishes\n\nPhillip\n\n> diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\n> index 5e28adf55b6..48cb2e735b5 100755\n> --- a/ci/install-docker-dependencies.sh\n> +++ b/ci/install-docker-dependencies.sh\n> @@ -21,7 +21,8 @@ linux32)\n> linux-musl)\n> apk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n> pcre2-dev python3 musl-libintl perl-utils ncurses \\\n> - apache2 bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n> + apache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \\\n> + bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n> ;;\n> linux-*)\n> apt update -q &&\n> diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\n> index 9791f94b16f..9ea74927c40 100644\n> --- a/t/lib-httpd.sh\n> +++ b/t/lib-httpd.sh\n> @@ -128,6 +128,20 @@ else\n> \"Could not identify web server at '$LIB_HTTPD_PATH'\"\n> fi\n> \n> +if test -n \"$LIB_HTTPD_DAV\" && test -f /etc/os-release\n> +then\n> + case \"$(grep \"^ID=\" /etc/os-release | cut -d= -f2-)\" in\n> + alpine)\n> + # The WebDAV module in Alpine Linux is broken at least up to\n> + # Alpine v3.16 as the default DBM driver is missing.\n> + #\n> + # https://gitlab.alpinelinux.org/alpine/aports/-/issues/13112\n> + test_skip_or_die GIT_TEST_HTTPD \\\n> + \"Apache WebDAV module does not have default DBM backend driver\"\n> + ;;\n> + esac\n> +fi\n> +\n> install_script () {\n> write_script \"$HTTPD_ROOT_PATH/$1\" <\"$TEST_PATH/$1\"\n> }\n> \n> I might as well roll it into this patch series now. It increases test\n> coverage on musl libc and doesn't have any significant downsides. In\n> fact, it uncovers that tests on Alpine Linux don't work right now, so it\n> fixes real issues to have the test coverage in our pipelines.\n> \n> Patrick\n"},{"id":"484137","messageId":"1365af33315c501db15778bc1ecf3852@manjaro.org","threadId":"60432","inReplyTo":"87430c6c-91c0-4be1-b89d-bf442b3f018b@gmail.com","subject":"Re: [PATCH v3 5/8] ci: unify setup of some environment variables","fromName":"Dragan Simic","fromEmail":"dsimic@manjaro.org","sentAt":"2023-10-30T18:22:23Z","receivedAt":"2023-10-30T18:22:54Z","isPatch":true,"sender":{"key":"dsimic@manjaro.org","avatar":null},"body":"On 2023-10-30 16:09, Phillip Wood wrote:\n> On 30/10/2023 12:15, Patrick Steinhardt wrote:\n>> Both GitHub Actions and Azue Pipelines set up the environment \n>> variables\n>> GIT_TEST_OPTS, GIT_PROVE_OPTS and MAKEFLAGS. And while most values are\n>> actually the same, the setup is completely duplicate. With the \n>> upcoming\n>> support for GitLab CI this duplication would only extend even further.\n>> \n>> Unify the setup of those environment variables so that only the \n>> uncommon\n>> parts are separated. While at it, we also perform some additional \n>> small\n>> improvements:\n>> \n>> - We use nproc instead of a hardcoded count of jobs for make and\n>> prove. This ensures that the number of concurrent processes \n>> adapts\n>> to the host automatically.\n> \n> Sadly this makes the Windows and MacOS jobs fail on GitHub Actions as\n> nproc is not installed[1]. Perhaps we could do\n> \n> \t--jobs=\"$(nproc || echo 2)\"\n\nIt would be better to use the following, to also suppress any error \nmessages:\n\n --jobs=$(nproc 2> /dev/null || echo 2)\n\nHaving the quotation marks is also pretty much redundant.\n\n> instead. (Maybe 2 is a bit low but the current value of 10 seems\n> pretty high for the number of cores on the runners that we use)\n"},{"id":"484205","messageId":"ZUCw1B6oQaDWKx3O@tanuki","threadId":"60432","inReplyTo":"ZT/P5Bl9lD9V6ID9@nand.local","subject":"Re: [PATCH 0/5] ci: add GitLab CI definition","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-31T07:46:28Z","receivedAt":"2023-10-31T07:46:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Oct 30, 2023 at 11:46:44AM -0400, Taylor Blau wrote:\n> On Thu, Oct 26, 2023 at 09:59:59AM +0200, Patrick Steinhardt wrote:\n> > And this is exactly what this patch series does: it adds GitLab-specific\n> > knowledge to our CI scripts and adds a CI definition that builds on top\n> > of those scripts. This is rather straight forward, as the scripts\n> > already know to discern Azure Pipelines and GitHub Actions, and adding\n> > a third item to this list feels quite natural. And by building on top of\n> > the preexisting infra, the actual \".gitlab-ci.yml\" is really quite\n> > small.\n> >\n> > I acknowledge that the Git project may not be willing to fully support\n> > GitLab CI, and that's fine with me. If we want to further stress that\n> > point then I'd also be perfectly happy to move the definitions into the\n> > \"contrib/\" directory -- it would still be a huge win for our workflow.\n> > In any case, I'm happy to keep on maintaining the intgeration with\n> > GitLab CI, and if things break I'll do my best to fix them fast.\n> \n> I don't have any strong opinions here, but my preference would probably\n> be to keep any GitLab-specific CI configuration limited to \"contrib\", if\n> it lands in the tree at all.\n\nAs mentioned, I would not mind at all if we wanted to instead carry this\nas part of \"contrib/\".\n\n> We already have a rather complicated CI setup on GitHub, which I think\n> we generally consider authoritative in terms of determining whether \"CI\"\n> is green. I know we have some Azure remnants in \"ci\", but I'm not aware\n> of any of the details there.\n> \n> So I have some hesitation about trying to mirror this rather complicated\n> set of build rules in another CI environment. My primary concern would\n> be that the two might fall out of sync and a series that is green on\n> GitHub would be red on GitLab, or vice-versa. Importantly, this can\n> happen even without changes to the build definitions, since (AFAICT)\n> both forges distribute new images automatically, so the set of packages\n> installed in GitHub may not exactly match what's in GitLab (and\n> vice-versa).\n\nYup, that's a valid concern. As mentioned, this patch series does not\nhave the intent to make GitLab CI a second authoritative CI platform.\nGitHub Actions should remain the source of truth of whether a pipeline\npasses or not. Most importantly, I do not want to require the maintainer\nto now watch both pipelines on GitHub and GitLab. This might be another\nindicator that the pipeline should rather be in \"contrib/\", so that\npeople don't start to treat it as authoritative.\n\n> My other concern is that we're doubling the cost of any new changes to\n> our CI definition. Perhaps this is more of an academic concern, but I\n> think my fear would be that one or the other would fall behind on in\n> implementation leading to further divergence between the two.\n> \n> I think having the new CI definition live in \"contrib\" somewhat\n> addresses the \"which CI is authoritative?\" problem, but that it doesn't\n> address the \"we have two of these\" problem.\n\nI do see that this requires us to be a bit more careful with future\nchanges to our CI definitions. But I think the additional work that this\ncreates is really very limited. Except for the `.gitlab-ci.yml`, there\nare only 54 lines specific to GitLab in our CI scripts now, which I\nthink should be rather manageable.\n\nI also think that it is sensible to ensure that our CI scripts are as\nagnostic to the CI platform as possible, as it ensures that we continue\nto be agile here in the future if we are ever forced to switch due to\nwhatever reason. In the best case, our CI scripts would allow a user to\nalso easily run the tests locally via e.g. Docker. We're not there yet,\nbut this patch series is a good step into that direction already.\n\nLast but not least, I actually think that having multiple supported CI\nplatforms also has the benefit that people can more readily set it up\nfor themselves. In theory, this has the potential to broaden the set of\npeople willing to contribute to our `ci/` scripts, which would in the\nend also benefit GitHub Actions.\n\nIn my opinion, this benefit is demonstrated by this patch series\nalready: besides all the changes that aim to prepare for GitLab CI,\nthere are also patches that deduplicate code and improve test coverage\nfor Alpine Linux. These changes likely wouldn't have happened if it\nwasn't for the GitLab CI.\n\n> So my preference would probably to have this live out of Junio's tree,\n> but I'm curious to hear what others think.\n\nI understand your points, and especially the point about not having a\nsecond authoritative CI platform. I'm very much on the same page as you\nare here, and would be happy to move the definitions to \"contrib/\" if\nyou want me to.\n\nBut I think we should also see the potential benefit of having a second\nCI platform, as it enables a more diverse set of people to contribute.\nwhich can ultimately end up benefitting our CI infra for both GitHub\nActions and GitLab CI.\n\nPatrick\n"},{"id":"484223","messageId":"cover.1698742590.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698305961.git.ps@pks.im","subject":"[PATCH v4 0/8] ci: add GitLab CI definition","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-31T09:04:37Z","receivedAt":"2023-10-31T09:04:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the fourth version of my patch series that introduces support\nfor GitLab CI.\n\nChanges compared to v3:\n\n - Stopped using nproc(1) to figure out the number of builds jobs for\n GitHub Actions and Azure Pipelines. Instead, we now continue to\n use the hardcoded 10 jobs there, whereas on GitLab CI we now use\n nproc. We can adapt GitHub/Azure at a later point as required, but\n I don't feel comfortable doing changes there.\n\n - Improved the linux-musl job. Namely, we now also install all\n required Apache modules, which makes the Apache-based test setup\n work. There is a packaging issue with the WebDAV module though, so\n we now skip tests that depend on it on Alpine Linux.\n\nI still didn't move `.gitlab-ci.yml` to `contrib/`. As Taylor argued\n(and I don't disagree), moving it to `contrib/` would convey the spirit\nthat this is _not_ an authoritative CI pipeline setup. But I wanted to\nhear other opinions first before moving it into `contrib/`.\n\nPatrick\n\nPatrick Steinhardt (8):\n ci: reorder definitions for grouping functions\n ci: make grouping setup more generic\n ci: group installation of Docker dependencies\n ci: split out logic to set up failed test artifacts\n ci: unify setup of some environment variables\n ci: squelch warnings when testing with unusable Git repo\n ci: install test dependencies for linux-musl\n ci: add support for GitLab CI\n\n .gitlab-ci.yml | 53 +++++++++\n ci/install-docker-dependencies.sh | 23 +++-\n ci/lib.sh | 190 +++++++++++++++++++++---------\n ci/print-test-failures.sh | 6 +\n t/lib-httpd.sh | 17 ++-\n 5 files changed, 233 insertions(+), 56 deletions(-)\n create mode 100644 .gitlab-ci.yml\n\nRange-diff against v3:\n1: ef44ed5c3b1 = 1: 8595fe5016a ci: reorder definitions for grouping functions\n2: 77798fa7a7a = 2: 7358a943392 ci: make grouping setup more generic\n3: 4542bd38dc2 = 3: 6d842592c6f ci: group installation of Docker dependencies\n4: 5fdda7fd83f = 4: e15651b3f5d ci: split out logic to set up failed test artifacts\n5: 6af0075fd87 ! 5: a64799b6e25 ci: unify setup of some environment variables\n @@ Commit message\n parts are separated. While at it, we also perform some additional small\n improvements:\n \n - - We use nproc instead of a hardcoded count of jobs for make and\n - prove. This ensures that the number of concurrent processes adapts\n - to the host automatically.\n -\n - We now always pass `--state=failed,slow,save` via GIT_PROVE_OPTS.\n It doesn't hurt on platforms where we don't persist the state, so\n this further reduces boilerplate.\n @@ ci/lib.sh: then\n \texit 1\n fi\n \n -+MAKEFLAGS=\"$MAKEFLAGS --jobs=$(nproc)\"\n -+GIT_PROVE_OPTS=\"--timer --jobs $(nproc) --state=failed,slow,save\"\n ++MAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n ++GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n +\n +GIT_TEST_OPTS=\"$GIT_TEST_OPTS --verbose-log -x\"\n +if test windows = \"$CI_OS_NAME\"\n6: 78d863bf24e = 6: f7d2a8666fe ci: squelch warnings when testing with unusable Git repo\n7: f150d61a1ce ! 7: 9b43b0d90e3 ci: install test dependencies for linux-musl\n @@ Commit message\n available, all Subversion-related tests are skipped without the\n SVN::Core Perl library anyway.\n \n - Furthermore, in order to make the Apache-based tests work, this commit\n - also adds the Alpine-specific modules path of it to the list of known\n - paths.\n + The Apache2-based tests require a bit more care though. For one, the\n + module path is different on Alpine Linux, which requires us to add it to\n + the list of known module paths to detect it. But second, the WebDAV\n + module on Alpine Linux is broken because it does not bundle the default\n + database backend [1]. We thus need to skip the WebDAV-based tests on\n + Alpine Linux for now.\n +\n + [1]: https://gitlab.alpinelinux.org/alpine/aports/-/issues/13112\n \n Signed-off-by: Patrick Steinhardt \n \n @@ ci/install-docker-dependencies.sh: linux32)\n \tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n -\t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n +\t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n -+\t\tapache2 bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n ++\t\tapache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \\\n ++\t\tbash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n \t;;\n pedantic)\n \tdnf -yq update >/dev/null &&\n @@ t/lib-httpd.sh: for DEFAULT_HTTPD_MODULE_PATH in '/usr/libexec/apache2' \\\n do\n \tif test -d \"$DEFAULT_HTTPD_MODULE_PATH\"\n \tthen\n +@@ t/lib-httpd.sh: else\n + \t\t\"Could not identify web server at '$LIB_HTTPD_PATH'\"\n + fi\n + \n ++if test -n \"$LIB_HTTPD_DAV\" && test -f /etc/os-release\n ++then\n ++\tcase \"$(grep \"^ID=\" /etc/os-release | cut -d= -f2-)\" in\n ++\talpine)\n ++\t\t# The WebDAV module in Alpine Linux is broken at least up to\n ++\t\t# Alpine v3.16 as the default DBM driver is missing.\n ++\t\t#\n ++\t\t# https://gitlab.alpinelinux.org/alpine/aports/-/issues/13112\n ++\t\ttest_skip_or_die GIT_TEST_HTTPD \\\n ++\t\t\t\"Apache WebDAV module does not have default DBM backend driver\"\n ++\t\t;;\n ++\tesac\n ++fi\n ++\n + install_script () {\n + \twrite_script \"$HTTPD_ROOT_PATH/$1\" <\"$TEST_PATH/$1\"\n + }\n8: 5272d66d9f1 ! 8: f3f2c98a0dc ci: add support for GitLab CI\n @@ ci/install-docker-dependencies.sh: linux32)\n -\tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n +\tapk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n \t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n - \t\tapache2 bash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n + \t\tapache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \\\n + \t\tbash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n \t;;\n +linux-*)\n +\tapt update -q &&\n @@ ci/lib.sh: then\n else\n \tbegin_group () { :; }\n \tend_group () { :; }\n +@@ ci/lib.sh: then\n + \tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n + \n + \tGIT_TEST_OPTS=\"--write-junit-xml\"\n ++\tJOBS=10\n + elif test true = \"$GITHUB_ACTIONS\"\n + then\n + \tCI_TYPE=github-actions\n @@ ci/lib.sh: then\n \tcache_dir=\"$HOME/none\"\n \n \tGIT_TEST_OPTS=\"--github-workflow-markup\"\n ++\tJOBS=10\n +elif test true = \"$GITLAB_CI\"\n +then\n +\tCI_TYPE=gitlab-ci\n @@ ci/lib.sh: then\n +\tcache_dir=\"$HOME/none\"\n +\n +\truns_on_pool=$(echo \"$CI_JOB_IMAGE\" | tr : -)\n ++\tJOBS=$(nproc)\n else\n \techo \"Could not identify CI type\" >&2\n \tenv >&2\n + \texit 1\n + fi\n + \n +-MAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n +-GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n ++MAKEFLAGS=\"$MAKEFLAGS --jobs=${JOBS}\"\n ++GIT_PROVE_OPTS=\"--timer --jobs ${JOBS} --state=failed,slow,save\"\n + \n + GIT_TEST_OPTS=\"$GIT_TEST_OPTS --verbose-log -x\"\n + if test windows = \"$CI_OS_NAME\"\n \n ## ci/print-test-failures.sh ##\n @@ ci/print-test-failures.sh: do\n-- \n2.42.0\n\n"},{"id":"484224","messageId":"8595fe5016ab99ac3818e132c787303e88bcef84.1698742590.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698742590.git.ps@pks.im","subject":"[PATCH v4 1/8] ci: reorder definitions for grouping functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-31T09:04:41Z","receivedAt":"2023-10-31T09:04:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We define a set of grouping functions that are used to group together\noutput in our CI, where these groups then end up as collapsible sections\nin the respective pipeline platform. The way these functions are defined\nis not easily extensible though as we have an up front check for the CI\n_not_ being GitHub Actions, where we define the non-stub logic in the\nelse branch.\n\nReorder the conditional branches such that we explicitly handle GitHub\nActions.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 20 ++++++++++----------\n 1 file changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 6fbb5bade12..eb384f4e952 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -1,16 +1,7 @@\n # Library of functions shared by all CI scripts\n \n-if test true != \"$GITHUB_ACTIONS\"\n+if test true = \"$GITHUB_ACTIONS\"\n then\n-\tbegin_group () { :; }\n-\tend_group () { :; }\n-\n-\tgroup () {\n-\t\tshift\n-\t\t\"$@\"\n-\t}\n-\tset -x\n-else\n \tbegin_group () {\n \t\tneed_to_end_group=t\n \t\techo \"::group::$1\" >&2\n@@ -42,6 +33,15 @@ else\n \t}\n \n \tbegin_group \"CI setup\"\n+else\n+\tbegin_group () { :; }\n+\tend_group () { :; }\n+\n+\tgroup () {\n+\t\tshift\n+\t\t\"$@\"\n+\t}\n+\tset -x\n fi\n \n # Set 'exit on error' for all CI scripts to let the caller know that\n-- \n2.42.0\n\n"},{"id":"484225","messageId":"7358a9433927aeb61cd9dac1b8ae651f13e11f5d.1698742590.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698742590.git.ps@pks.im","subject":"[PATCH v4 2/8] ci: make grouping setup more generic","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-31T09:04:46Z","receivedAt":"2023-10-31T09:04:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Make the grouping setup more generic by always calling `begin_group ()`\nand `end_group ()` regardless of whether we have stubbed those functions\nor not. This ensures we can more readily add support for additional CI\nplatforms.\n\nFurthermore, the `group ()` function is made generic so that it is the\nsame for both GitHub Actions and for other platforms. There is a\nsemantic conflict here though: GitHub Actions used to call `set +x` in\n`group ()` whereas the non-GitHub case unconditionally uses `set -x`.\nThe latter would get overriden if we kept the `set +x` in the generic\nversion of `group ()`. To resolve this conflict, we simply drop the `set\n+x` in the generic variant of this function. As `begin_group ()` calls\n`set -x` anyway this is not much of a change though, as the only\ncommands that aren't printed anymore now are the ones between the\nbeginning of `group ()` and the end of `begin_group ()`.\n\nLast, this commit changes `end_group ()` to also accept a parameter that\nindicates _which_ group should end. This will be required by a later\ncommit that introduces support for GitLab CI.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 46 ++++++++++++++++++++++------------------------\n 1 file changed, 22 insertions(+), 24 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex eb384f4e952..b3411afae8e 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -14,36 +14,34 @@ then\n \t\tneed_to_end_group=\n \t\techo '::endgroup::' >&2\n \t}\n-\ttrap end_group EXIT\n-\n-\tgroup () {\n-\t\tset +x\n-\t\tbegin_group \"$1\"\n-\t\tshift\n-\t\t# work around `dash` not supporting `set -o pipefail`\n-\t\t(\n-\t\t\t\"$@\" 2>&1\n-\t\t\techo $? >exit.status\n-\t\t) |\n-\t\tsed 's/^\\(\\([^ ]*\\):\\([0-9]*\\):\\([0-9]*:\\) \\)\\(error\\|warning\\): /::\\5 file=\\2,line=\\3::\\1/'\n-\t\tres=$(cat exit.status)\n-\t\trm exit.status\n-\t\tend_group\n-\t\treturn $res\n-\t}\n-\n-\tbegin_group \"CI setup\"\n else\n \tbegin_group () { :; }\n \tend_group () { :; }\n \n-\tgroup () {\n-\t\tshift\n-\t\t\"$@\"\n-\t}\n \tset -x\n fi\n \n+group () {\n+\tgroup=\"$1\"\n+\tshift\n+\tbegin_group \"$group\"\n+\n+\t# work around `dash` not supporting `set -o pipefail`\n+\t(\n+\t\t\"$@\" 2>&1\n+\t\techo $? >exit.status\n+\t) |\n+\tsed 's/^\\(\\([^ ]*\\):\\([0-9]*\\):\\([0-9]*:\\) \\)\\(error\\|warning\\): /::\\5 file=\\2,line=\\3::\\1/'\n+\tres=$(cat exit.status)\n+\trm exit.status\n+\n+\tend_group \"$group\"\n+\treturn $res\n+}\n+\n+begin_group \"CI setup\"\n+trap \"end_group 'CI setup'\" EXIT\n+\n # Set 'exit on error' for all CI scripts to let the caller know that\n # something went wrong.\n #\n@@ -287,5 +285,5 @@ esac\n \n MAKEFLAGS=\"$MAKEFLAGS CC=${CC:-cc}\"\n \n-end_group\n+end_group \"CI setup\"\n set -x\n-- \n2.42.0\n\n"},{"id":"484226","messageId":"6d842592c6fb085b40997466d117637742de44d6.1698742590.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698742590.git.ps@pks.im","subject":"[PATCH v4 3/8] ci: group installation of Docker dependencies","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-31T09:04:50Z","receivedAt":"2023-10-31T09:04:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The output of CI jobs tends to be quite long-winded and hard to digest.\nTo help with this, many CI systems provide the ability to group output\ninto collapsible sections, and we're also doing this in some of our\nscripts.\n\nOne notable omission is the script to install Docker dependencies.\nAddress it to bring more structure to the output for Docker-based jobs.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-docker-dependencies.sh | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\nindex 78b7e326da6..d0bc19d3bb3 100755\n--- a/ci/install-docker-dependencies.sh\n+++ b/ci/install-docker-dependencies.sh\n@@ -3,6 +3,10 @@\n # Install dependencies required to build and test Git inside container\n #\n \n+. ${0%/*}/lib.sh\n+\n+begin_group \"Install dependencies\"\n+\n case \"$jobname\" in\n linux32)\n \tlinux32 --32bit i386 sh -c '\n@@ -20,3 +24,5 @@ pedantic)\n \tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n \t;;\n esac\n+\n+end_group \"Install dependencies\"\n-- \n2.42.0\n\n"},{"id":"484227","messageId":"e15651b3f5d5b7250f73eeb0db33ae1d68cd37c0.1698742590.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698742590.git.ps@pks.im","subject":"[PATCH v4 4/8] ci: split out logic to set up failed test artifacts","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-31T09:04:54Z","receivedAt":"2023-10-31T09:04:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have some logic in place to create a directory with the output from\nfailed tests, which will then subsequently be uploaded as CI artifacts.\nWe're about to add support for GitLab CI, which will want to reuse the\nlogic.\n\nSplit the logic into a separate function so that it is reusable.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 40 ++++++++++++++++++++++------------------\n 1 file changed, 22 insertions(+), 18 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex b3411afae8e..9ffdf743903 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -131,6 +131,27 @@ handle_failed_tests () {\n \treturn 1\n }\n \n+create_failed_test_artifacts () {\n+\tmkdir -p t/failed-test-artifacts\n+\n+\tfor test_exit in t/test-results/*.exit\n+\tdo\n+\t\ttest 0 != \"$(cat \"$test_exit\")\" || continue\n+\n+\t\ttest_name=\"${test_exit%.exit}\"\n+\t\ttest_name=\"${test_name##*/}\"\n+\t\tprintf \"\\\\e[33m\\\\e[1m=== Failed test: ${test_name} ===\\\\e[m\\\\n\"\n+\t\techo \"The full logs are in the 'print test failures' step below.\"\n+\t\techo \"See also the 'failed-tests-*' artifacts attached to this run.\"\n+\t\tcat \"t/test-results/$test_name.markup\"\n+\n+\t\ttrash_dir=\"t/trash directory.$test_name\"\n+\t\tcp \"t/test-results/$test_name.out\" t/failed-test-artifacts/\n+\t\ttar czf t/failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n+\tdone\n+\treturn 1\n+}\n+\n # GitHub Action doesn't set TERM, which is required by tput\n export TERM=${TERM:-dumb}\n \n@@ -171,25 +192,8 @@ then\n \tCC=\"${CC_PACKAGE:-${CC:-gcc}}\"\n \tDONT_SKIP_TAGS=t\n \thandle_failed_tests () {\n-\t\tmkdir -p t/failed-test-artifacts\n \t\techo \"FAILED_TEST_ARTIFACTS=t/failed-test-artifacts\" >>$GITHUB_ENV\n-\n-\t\tfor test_exit in t/test-results/*.exit\n-\t\tdo\n-\t\t\ttest 0 != \"$(cat \"$test_exit\")\" || continue\n-\n-\t\t\ttest_name=\"${test_exit%.exit}\"\n-\t\t\ttest_name=\"${test_name##*/}\"\n-\t\t\tprintf \"\\\\e[33m\\\\e[1m=== Failed test: ${test_name} ===\\\\e[m\\\\n\"\n-\t\t\techo \"The full logs are in the 'print test failures' step below.\"\n-\t\t\techo \"See also the 'failed-tests-*' artifacts attached to this run.\"\n-\t\t\tcat \"t/test-results/$test_name.markup\"\n-\n-\t\t\ttrash_dir=\"t/trash directory.$test_name\"\n-\t\t\tcp \"t/test-results/$test_name.out\" t/failed-test-artifacts/\n-\t\t\ttar czf t/failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n-\t\tdone\n-\t\treturn 1\n+\t\tcreate_failed_test_artifacts\n \t}\n \n \tcache_dir=\"$HOME/none\"\n-- \n2.42.0\n\n"},{"id":"484228","messageId":"a64799b6e25d05e5d5fc7fe3c5602b5ce256d8b9.1698742590.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698742590.git.ps@pks.im","subject":"[PATCH v4 5/8] ci: unify setup of some environment variables","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-31T09:04:58Z","receivedAt":"2023-10-31T09:05:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Both GitHub Actions and Azue Pipelines set up the environment variables\nGIT_TEST_OPTS, GIT_PROVE_OPTS and MAKEFLAGS. And while most values are\nactually the same, the setup is completely duplicate. With the upcoming\nsupport for GitLab CI this duplication would only extend even further.\n\nUnify the setup of those environment variables so that only the uncommon\nparts are separated. While at it, we also perform some additional small\nimprovements:\n\n - We now always pass `--state=failed,slow,save` via GIT_PROVE_OPTS.\n It doesn't hurt on platforms where we don't persist the state, so\n this further reduces boilerplate.\n\n - When running on Windows systems we set `--no-chain-lint` and\n `--no-bin-wrappers`. Interestingly though, we did so _after_\n already having exported the respective environment variables.\n\n - We stop using `export VAR=value` syntax, which is a Bashism. It's\n not quite worth it as we still use this syntax all over the place,\n but it doesn't hurt readability either.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 24 ++++++++++++++----------\n 1 file changed, 14 insertions(+), 10 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 9ffdf743903..9a9b92c05b3 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -175,11 +175,7 @@ then\n \t# among *all* phases)\n \tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n \n-\texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n-\texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n-\tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n-\ttest windows_nt != \"$CI_OS_NAME\" ||\n-\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n+\tGIT_TEST_OPTS=\"--write-junit-xml\"\n elif test true = \"$GITHUB_ACTIONS\"\n then\n \tCI_TYPE=github-actions\n@@ -198,17 +194,25 @@ then\n \n \tcache_dir=\"$HOME/none\"\n \n-\texport GIT_PROVE_OPTS=\"--timer --jobs 10\"\n-\texport GIT_TEST_OPTS=\"--verbose-log -x --github-workflow-markup\"\n-\tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n-\ttest windows != \"$CI_OS_NAME\" ||\n-\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n+\tGIT_TEST_OPTS=\"--github-workflow-markup\"\n else\n \techo \"Could not identify CI type\" >&2\n \tenv >&2\n \texit 1\n fi\n \n+MAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n+GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n+\n+GIT_TEST_OPTS=\"$GIT_TEST_OPTS --verbose-log -x\"\n+if test windows = \"$CI_OS_NAME\"\n+then\n+\tGIT_TEST_OPTS=\"$GIT_TEST_OPTS --no-chain-lint --no-bin-wrappers\"\n+fi\n+\n+export GIT_TEST_OPTS\n+export GIT_PROVE_OPTS\n+\n good_trees_file=\"$cache_dir/good-trees\"\n \n mkdir -p \"$cache_dir\"\n-- \n2.42.0\n\n"},{"id":"484229","messageId":"f7d2a8666fed2e2b98652c78b66871a36b4eddfa.1698742590.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698742590.git.ps@pks.im","subject":"[PATCH v4 6/8] ci: squelch warnings when testing with unusable Git repo","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-31T09:05:02Z","receivedAt":"2023-10-31T09:05:08Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Our CI jobs that run on Docker also use mostly the same architecture to\nbuild and test Git via the \"ci/run-build-and-tests.sh\" script. These\nscripts also provide some functionality to massage the Git repository\nwe're supposedly operating in.\n\nIn our Docker-based infrastructure we may not even have a Git repository\navailable though, which leads to warnings when those functions execute.\nMake the helpers exit gracefully in case either there is no Git in our\nPATH, or when not running in a Git repository.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 32 ++++++++++++++++++++++++++++++++\n 1 file changed, 32 insertions(+)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 9a9b92c05b3..e14b1029fad 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -69,10 +69,32 @@ skip_branch_tip_with_tag () {\n \tfi\n }\n \n+# Check whether we can use the path passed via the first argument as Git\n+# repository.\n+is_usable_git_repository () {\n+\t# We require Git in our PATH, otherwise we cannot access repositories\n+\t# at all.\n+\tif ! command -v git >/dev/null\n+\tthen\n+\t\treturn 1\n+\tfi\n+\n+\t# And the target directory needs to be a proper Git repository.\n+\tif ! git -C \"$1\" rev-parse 2>/dev/null\n+\tthen\n+\t\treturn 1\n+\tfi\n+}\n+\n # Save some info about the current commit's tree, so we can skip the build\n # job if we encounter the same tree again and can provide a useful info\n # message.\n save_good_tree () {\n+\tif ! is_usable_git_repository .\n+\tthen\n+\t\treturn\n+\tfi\n+\n \techo \"$(git rev-parse $CI_COMMIT^{tree}) $CI_COMMIT $CI_JOB_NUMBER $CI_JOB_ID\" >>\"$good_trees_file\"\n \t# limit the file size\n \ttail -1000 \"$good_trees_file\" >\"$good_trees_file\".tmp\n@@ -88,6 +110,11 @@ skip_good_tree () {\n \t\treturn\n \tfi\n \n+\tif ! is_usable_git_repository .\n+\tthen\n+\t\treturn\n+\tfi\n+\n \tif ! good_tree_info=\"$(grep \"^$(git rev-parse $CI_COMMIT^{tree}) \" \"$good_trees_file\")\"\n \tthen\n \t\t# Haven't seen this tree yet, or no cached good trees file yet.\n@@ -119,6 +146,11 @@ skip_good_tree () {\n }\n \n check_unignored_build_artifacts () {\n+\tif ! is_usable_git_repository .\n+\tthen\n+\t\treturn\n+\tfi\n+\n \t! git ls-files --other --exclude-standard --error-unmatch \\\n \t\t-- ':/*' 2>/dev/null ||\n \t{\n-- \n2.42.0\n\n"},{"id":"484230","messageId":"9b43b0d90e30c3ec49fac29060dc19d21274139f.1698742590.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698742590.git.ps@pks.im","subject":"[PATCH v4 7/8] ci: install test dependencies for linux-musl","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-31T09:05:06Z","receivedAt":"2023-10-31T09:05:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The linux-musl CI job executes tests on Alpine Linux, which is based on\nmusl libc instead of glibc. We're missing some test dependencies though,\nwhich causes us to skip a subset of tests.\n\nInstall these test dependencies to increase our test coverage on this\nplatform. There are still some missing test dependecies, but these do\nnot have a corresponding package in the Alpine repositories:\n\n - p4 and p4d, both parts of the Perforce version control system.\n\n - cvsps, which generates patch sets for CVS.\n\n - Subversion and the SVN::Core Perl library, the latter of which is\n not available in the Alpine repositories. While the tool itself is\n available, all Subversion-related tests are skipped without the\n SVN::Core Perl library anyway.\n\nThe Apache2-based tests require a bit more care though. For one, the\nmodule path is different on Alpine Linux, which requires us to add it to\nthe list of known module paths to detect it. But second, the WebDAV\nmodule on Alpine Linux is broken because it does not bundle the default\ndatabase backend [1]. We thus need to skip the WebDAV-based tests on\nAlpine Linux for now.\n\n[1]: https://gitlab.alpinelinux.org/alpine/aports/-/issues/13112\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-docker-dependencies.sh | 4 +++-\n t/lib-httpd.sh | 17 ++++++++++++++++-\n 2 files changed, 19 insertions(+), 2 deletions(-)\n\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\nindex d0bc19d3bb3..6e845283680 100755\n--- a/ci/install-docker-dependencies.sh\n+++ b/ci/install-docker-dependencies.sh\n@@ -17,7 +17,9 @@ linux32)\n \t;;\n linux-musl)\n \tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n-\t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n+\t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n+\t\tapache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \\\n+\t\tbash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n \t;;\n pedantic)\n \tdnf -yq update >/dev/null &&\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex 2fb1b2ae561..9ea74927c40 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -67,7 +67,8 @@ for DEFAULT_HTTPD_MODULE_PATH in '/usr/libexec/apache2' \\\n \t\t\t\t '/usr/lib/apache2/modules' \\\n \t\t\t\t '/usr/lib64/httpd/modules' \\\n \t\t\t\t '/usr/lib/httpd/modules' \\\n-\t\t\t\t '/usr/libexec/httpd'\n+\t\t\t\t '/usr/libexec/httpd' \\\n+\t\t\t\t '/usr/lib/apache2'\n do\n \tif test -d \"$DEFAULT_HTTPD_MODULE_PATH\"\n \tthen\n@@ -127,6 +128,20 @@ else\n \t\t\"Could not identify web server at '$LIB_HTTPD_PATH'\"\n fi\n \n+if test -n \"$LIB_HTTPD_DAV\" && test -f /etc/os-release\n+then\n+\tcase \"$(grep \"^ID=\" /etc/os-release | cut -d= -f2-)\" in\n+\talpine)\n+\t\t# The WebDAV module in Alpine Linux is broken at least up to\n+\t\t# Alpine v3.16 as the default DBM driver is missing.\n+\t\t#\n+\t\t# https://gitlab.alpinelinux.org/alpine/aports/-/issues/13112\n+\t\ttest_skip_or_die GIT_TEST_HTTPD \\\n+\t\t\t\"Apache WebDAV module does not have default DBM backend driver\"\n+\t\t;;\n+\tesac\n+fi\n+\n install_script () {\n \twrite_script \"$HTTPD_ROOT_PATH/$1\" <\"$TEST_PATH/$1\"\n }\n-- \n2.42.0\n\n"},{"id":"484231","messageId":"f3f2c98a0dc6042b7ed5eab9c10bee4f64858f02.1698742590.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698742590.git.ps@pks.im","subject":"[PATCH v4 8/8] ci: add support for GitLab CI","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-31T09:05:11Z","receivedAt":"2023-10-31T09:05:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We already support Azure Pipelines and GitHub Workflows in the Git\nproject, but until now we do not have support for GitLab CI. While it is\narguably not in the interest of the Git project to maintain a ton of\ndifferent CI platforms, GitLab has recently ramped up its efforts and\ntries to contribute to the Git project more regularly.\n\nPart of a problem we hit at GitLab rather frequently is that our own,\ncustom CI setup we have is so different to the setup that the Git\nproject has. More esoteric jobs like \"linux-TEST-vars\" that also set a\ncouple of environment variables do not exist in GitLab's custom CI\nsetup, and maintaining them to keep up with what Git does feels like\nwasted time. The result is that we regularly send patch series upstream\nthat fail to compile or pass tests in GitHub Workflows. We would thus\nlike to integrate the GitLab CI configuration into the Git project to\nhelp us send better patch series upstream and thus reduce overhead for\nthe maintainer.\n\nThe integration does not necessarily have to be a first-class citizen,\nwhich would in practice only add to the fallout that pipeline failures\nhave for the maintainer. That being said, we are happy to maintain this\nalternative CI setup for the Git project and will make test results\navailable as part of our own mirror of the Git project at [1].\n\nThis commit introduces the integration into our regular CI scripts so\nthat most of the setup continues to be shared across all of the CI\nsolutions. Note that as the builds on GitLab CI run as unprivileged\nuser, we need to pull in both sudo and shadow packages to our Alpine\nbased job to set this up.\n\n[1]: https://gitlab.com/gitlab-org/git\n\nSigned-off-by: Patrick Steinhardt \n---\n .gitlab-ci.yml | 53 +++++++++++++++++++++++++++++++\n ci/install-docker-dependencies.sh | 13 +++++++-\n ci/lib.sh | 50 +++++++++++++++++++++++++++--\n ci/print-test-failures.sh | 6 ++++\n 4 files changed, 119 insertions(+), 3 deletions(-)\n create mode 100644 .gitlab-ci.yml\n\ndiff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\nnew file mode 100644\nindex 00000000000..cd98bcb18aa\n--- /dev/null\n+++ b/.gitlab-ci.yml\n@@ -0,0 +1,53 @@\n+default:\n+ timeout: 2h\n+\n+workflow:\n+ rules:\n+ - if: $CI_PIPELINE_SOURCE == \"merge_request_event\"\n+ - if: $CI_COMMIT_TAG\n+ - if: $CI_COMMIT_REF_PROTECTED == \"true\"\n+\n+test:\n+ image: $image\n+ before_script:\n+ - ./ci/install-docker-dependencies.sh\n+ script:\n+ - useradd builder --create-home\n+ - chown -R builder \"${CI_PROJECT_DIR}\"\n+ - sudo --preserve-env --set-home --user=builder ./ci/run-build-and-tests.sh\n+ after_script:\n+ - |\n+ if test \"$CI_JOB_STATUS\" != 'success'\n+ then\n+ sudo --preserve-env --set-home --user=builder ./ci/print-test-failures.sh\n+ fi\n+ parallel:\n+ matrix:\n+ - jobname: linux-sha256\n+ image: ubuntu:latest\n+ CC: clang\n+ - jobname: linux-gcc\n+ image: ubuntu:20.04\n+ CC: gcc\n+ CC_PACKAGE: gcc-8\n+ - jobname: linux-TEST-vars\n+ image: ubuntu:20.04\n+ CC: gcc\n+ CC_PACKAGE: gcc-8\n+ - jobname: linux-gcc-default\n+ image: ubuntu:latest\n+ CC: gcc\n+ - jobname: linux-leaks\n+ image: ubuntu:latest\n+ CC: gcc\n+ - jobname: linux-asan-ubsan\n+ image: ubuntu:latest\n+ CC: clang\n+ - jobname: pedantic\n+ image: fedora:latest\n+ - jobname: linux-musl\n+ image: alpine:latest\n+ artifacts:\n+ paths:\n+ - t/failed-test-artifacts\n+ when: on_failure\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\nindex 6e845283680..48cb2e735b5 100755\n--- a/ci/install-docker-dependencies.sh\n+++ b/ci/install-docker-dependencies.sh\n@@ -7,6 +7,9 @@\n \n begin_group \"Install dependencies\"\n \n+# Required so that apt doesn't wait for user input on certain packages.\n+export DEBIAN_FRONTEND=noninteractive\n+\n case \"$jobname\" in\n linux32)\n \tlinux32 --32bit i386 sh -c '\n@@ -16,11 +19,19 @@ linux32)\n \t'\n \t;;\n linux-musl)\n-\tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n+\tapk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n \t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n \t\tapache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \\\n \t\tbash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n \t;;\n+linux-*)\n+\tapt update -q &&\n+\tapt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev \\\n+\t\tlibcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev \\\n+\t\tperl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl \\\n+\t\tlibdbd-sqlite3-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}} \\\n+\t\tapache2 cvs cvsps gnupg libcgi-pm-perl subversion\n+\t;;\n pedantic)\n \tdnf -yq update >/dev/null &&\n \tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex e14b1029fad..6e3d64004ec 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -14,6 +14,22 @@ then\n \t\tneed_to_end_group=\n \t\techo '::endgroup::' >&2\n \t}\n+elif test true = \"$GITLAB_CI\"\n+then\n+\tbegin_group () {\n+\t\tneed_to_end_group=t\n+\t\tprintf \"\\e[0Ksection_start:$(date +%s):$(echo \"$1\" | tr ' ' _)\\r\\e[0K$1\\n\"\n+\t\ttrap \"end_group '$1'\" EXIT\n+\t\tset -x\n+\t}\n+\n+\tend_group () {\n+\t\ttest -n \"$need_to_end_group\" || return 0\n+\t\tset +x\n+\t\tneed_to_end_group=\n+\t\tprintf \"\\e[0Ksection_end:$(date +%s):$(echo \"$1\" | tr ' ' _)\\r\\e[0K\\n\"\n+\t\ttrap - EXIT\n+\t}\n else\n \tbegin_group () { :; }\n \tend_group () { :; }\n@@ -208,6 +224,7 @@ then\n \tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n \n \tGIT_TEST_OPTS=\"--write-junit-xml\"\n+\tJOBS=10\n elif test true = \"$GITHUB_ACTIONS\"\n then\n \tCI_TYPE=github-actions\n@@ -227,14 +244,43 @@ then\n \tcache_dir=\"$HOME/none\"\n \n \tGIT_TEST_OPTS=\"--github-workflow-markup\"\n+\tJOBS=10\n+elif test true = \"$GITLAB_CI\"\n+then\n+\tCI_TYPE=gitlab-ci\n+\tCI_BRANCH=\"$CI_COMMIT_REF_NAME\"\n+\tCI_COMMIT=\"$CI_COMMIT_SHA\"\n+\tcase \"$CI_JOB_IMAGE\" in\n+\tmacos-*)\n+\t\tCI_OS_NAME=osx;;\n+\talpine:*|fedora:*|ubuntu:*)\n+\t\tCI_OS_NAME=linux;;\n+\t*)\n+\t\techo \"Could not identify OS image\" >&2\n+\t\tenv >&2\n+\t\texit 1\n+\t\t;;\n+\tesac\n+\tCI_REPO_SLUG=\"$CI_PROJECT_PATH\"\n+\tCI_JOB_ID=\"$CI_JOB_ID\"\n+\tCC=\"${CC_PACKAGE:-${CC:-gcc}}\"\n+\tDONT_SKIP_TAGS=t\n+\thandle_failed_tests () {\n+\t\tcreate_failed_test_artifacts\n+\t}\n+\n+\tcache_dir=\"$HOME/none\"\n+\n+\truns_on_pool=$(echo \"$CI_JOB_IMAGE\" | tr : -)\n+\tJOBS=$(nproc)\n else\n \techo \"Could not identify CI type\" >&2\n \tenv >&2\n \texit 1\n fi\n \n-MAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n-GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n+MAKEFLAGS=\"$MAKEFLAGS --jobs=${JOBS}\"\n+GIT_PROVE_OPTS=\"--timer --jobs ${JOBS} --state=failed,slow,save\"\n \n GIT_TEST_OPTS=\"$GIT_TEST_OPTS --verbose-log -x\"\n if test windows = \"$CI_OS_NAME\"\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex 57277eefcd0..c33ad4e3a22 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -51,6 +51,12 @@ do\n \t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n \t\t\tcontinue\n \t\t\t;;\n+\t\tgitlab-ci)\n+\t\t\tmkdir -p failed-test-artifacts\n+\t\t\tcp \"${TEST_EXIT%.exit}.out\" failed-test-artifacts/\n+\t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n+\t\t\tcontinue\n+\t\t\t;;\n \t\t*)\n \t\t\techo \"Unhandled CI type: $CI_TYPE\" >&2\n \t\t\texit 1\n-- \n2.42.0\n\n"},{"id":"484238","messageId":"31ebe4c9-84aa-4d42-9aeb-712e2a6cece3@github.com","threadId":"60432","inReplyTo":"a64799b6e25d05e5d5fc7fe3c5602b5ce256d8b9.1698742590.git.ps@pks.im","subject":"Re: [PATCH v4 5/8] ci: unify setup of some environment variables","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-10-31T17:06:24Z","receivedAt":"2023-10-31T17:06:29Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Patrick Steinhardt wrote:\n> Both GitHub Actions and Azue Pipelines set up the environment variables\n\ns/Azue/Azure\n\n> GIT_TEST_OPTS, GIT_PROVE_OPTS and MAKEFLAGS. And while most values are\n> actually the same, the setup is completely duplicate. With the upcoming\n> support for GitLab CI this duplication would only extend even further.\n> \n> Unify the setup of those environment variables so that only the uncommon\n> parts are separated. While at it, we also perform some additional small\n> improvements:\n> \n> - We now always pass `--state=failed,slow,save` via GIT_PROVE_OPTS.\n> It doesn't hurt on platforms where we don't persist the state, so\n> this further reduces boilerplate.\n> \n> - When running on Windows systems we set `--no-chain-lint` and\n> `--no-bin-wrappers`. Interestingly though, we did so _after_\n> already having exported the respective environment variables.\n> \n> - We stop using `export VAR=value` syntax, which is a Bashism. It's\n> not quite worth it as we still use this syntax all over the place,\n> but it doesn't hurt readability either.\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n> ci/lib.sh | 24 ++++++++++++++----------\n> 1 file changed, 14 insertions(+), 10 deletions(-)\n> \n> diff --git a/ci/lib.sh b/ci/lib.sh\n> index 9ffdf743903..9a9b92c05b3 100755\n> --- a/ci/lib.sh\n> +++ b/ci/lib.sh\n> @@ -175,11 +175,7 @@ then\n> \t# among *all* phases)\n> \tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n> \n> -\texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n> -\texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n> -\tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n> -\ttest windows_nt != \"$CI_OS_NAME\" ||\n> -\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n> +\tGIT_TEST_OPTS=\"--write-junit-xml\"\n> elif test true = \"$GITHUB_ACTIONS\"\n> then\n> \tCI_TYPE=github-actions\n> @@ -198,17 +194,25 @@ then\n> \n> \tcache_dir=\"$HOME/none\"\n> \n> -\texport GIT_PROVE_OPTS=\"--timer --jobs 10\"\n> -\texport GIT_TEST_OPTS=\"--verbose-log -x --github-workflow-markup\"\n> -\tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n> -\ttest windows != \"$CI_OS_NAME\" ||\n> -\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n> +\tGIT_TEST_OPTS=\"--github-workflow-markup\"\n> else\n> \techo \"Could not identify CI type\" >&2\n> \tenv >&2\n> \texit 1\n> fi\n> \n> +MAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n> +GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n> +\n> +GIT_TEST_OPTS=\"$GIT_TEST_OPTS --verbose-log -x\"\n> +if test windows = \"$CI_OS_NAME\"\n\nBased on the deleted lines above, I think this would need to be:\n\n\tif test windows = \"$CI_OS_NAME\" || test windows_nt = \"$CI_OS_NAME\"\n\nI believe these settings are required on all Windows builds, though, so you could \ninstead match on the first 7 characters of $CI_OS_NAME:\n\n\tif test windows = \"$(echo \"$CI_OS_NAME\" | cut -c1-7)\"\n\n(full disclosure: I'm not 100% confident in the correctness of that shell syntax)\n\n> +then\n> +\tGIT_TEST_OPTS=\"$GIT_TEST_OPTS --no-chain-lint --no-bin-wrappers\"\n> +fi\n> +\n> +export GIT_TEST_OPTS\n> +export GIT_PROVE_OPTS\n> +\n> good_trees_file=\"$cache_dir/good-trees\"\n> \n> mkdir -p \"$cache_dir\"\n\n"},{"id":"484239","messageId":"4c8c2f19-1a7e-4524-81e7-c74091e88edf@github.com","threadId":"60432","inReplyTo":"f3f2c98a0dc6042b7ed5eab9c10bee4f64858f02.1698742590.git.ps@pks.im","subject":"Re: [PATCH v4 8/8] ci: add support for GitLab CI","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-10-31T17:47:44Z","receivedAt":"2023-10-31T17:47:57Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Patrick Steinhardt wrote:> diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\n> index 6e845283680..48cb2e735b5 100755\n> --- a/ci/install-docker-dependencies.sh\n> +++ b/ci/install-docker-dependencies.sh\n> @@ -7,6 +7,9 @@\n> \n> begin_group \"Install dependencies\"\n> \n> +# Required so that apt doesn't wait for user input on certain packages.\n> +export DEBIAN_FRONTEND=noninteractive\n> +\n> case \"$jobname\" in\n> linux32)\n> \tlinux32 --32bit i386 sh -c '\n> @@ -16,11 +19,19 @@ linux32)\n> \t'\n> \t;;\n> linux-musl)\n> -\tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n> +\tapk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n> \t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n> \t\tapache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \\\n> \t\tbash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n> \t;;\n> +linux-*)\n> +\tapt update -q &&\n> +\tapt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev \\\n> +\t\tlibcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev \\\n> +\t\tperl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl \\\n> +\t\tlibdbd-sqlite3-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}} \\\n> +\t\tapache2 cvs cvsps gnupg libcgi-pm-perl subversion\n> +\t;;\n> pedantic)\n> \tdnf -yq update >/dev/null &&\n> \tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n\n...\n\n> diff --git a/ci/lib.sh b/ci/lib.sh\n> index e14b1029fad..6e3d64004ec 100755\n> --- a/ci/lib.sh\n> +++ b/ci/lib.sh\n> @@ -208,6 +224,7 @@ then\n> \tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n> \n> \tGIT_TEST_OPTS=\"--write-junit-xml\"\n> +\tJOBS=10\n> elif test true = \"$GITHUB_ACTIONS\"\n> then\n> \tCI_TYPE=github-actions\n\n...\n\n> -MAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n> -GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n> +MAKEFLAGS=\"$MAKEFLAGS --jobs=${JOBS}\"\n> +GIT_PROVE_OPTS=\"--timer --jobs ${JOBS} --state=failed,slow,save\"\n> \n\nOrganizationally, this commit seems to be doing two things at once:\n\n- Adding GitLab-specific CI setup (either in the new .gitlab-ci.yml or in\n conditions gated on \"gitlab-ci\").\n- Updating the common CI scripts with things that are needed for GitLab CI,\n but aren't conditioned on it (i.e. the patch excerpts I've included\n above). \n\nI'd prefer these being separated into two patches, mainly to isolate \"things\nthat affect all CI\" from \"things that affect only GitLab CI\". This is\nultimately a pretty minor nit, though; if you're not planning on re-rolling\n(or just disagree with what I'm suggesting :) ), I'm okay with leaving it\nas-is.\n\nOtherwise, I can't comment on the correctness of the GitLab CI definition (I\nassume you've tested it anyway), but AFAICT the changes above shouldn't break\nGitHub CI.\n\n"},{"id":"484241","messageId":"8e4d111f-3982-4989-90b5-08377fe9c5fd@github.com","threadId":"60432","inReplyTo":"cover.1698742590.git.ps@pks.im","subject":"Re: [PATCH v4 0/8] ci: add GitLab CI definition","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-10-31T18:22:23Z","receivedAt":"2023-10-31T18:22:26Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Patrick Steinhardt wrote:\n> Hi,\n> \n> this is the fourth version of my patch series that introduces support\n> for GitLab CI.\n> \n> Changes compared to v3:\n> \n> - Stopped using nproc(1) to figure out the number of builds jobs for\n> GitHub Actions and Azure Pipelines. Instead, we now continue to\n> use the hardcoded 10 jobs there, whereas on GitLab CI we now use\n> nproc. We can adapt GitHub/Azure at a later point as required, but\n> I don't feel comfortable doing changes there.\n> \n> - Improved the linux-musl job. Namely, we now also install all\n> required Apache modules, which makes the Apache-based test setup\n> work. There is a packaging issue with the WebDAV module though, so\n> we now skip tests that depend on it on Alpine Linux.\n> \n> I still didn't move `.gitlab-ci.yml` to `contrib/`. As Taylor argued\n> (and I don't disagree), moving it to `contrib/` would convey the spirit\n> that this is _not_ an authoritative CI pipeline setup. But I wanted to\n> hear other opinions first before moving it into `contrib/`.\n\nI've read through some of the earlier discussion on this (as well as your\noriginal cover letter [1]), so I'll throw in my 2c.\n\nThe majority of the changes in this patch series aren't conditioned on\nanything that says \"gitlab\", they just improve the flexibility of our CI\nscripts. I personally didn't notice anything too cumbersome added in the\nseries, so I'm happy with all of that (essentially, patches 1-7 & parts of\n8).\n\nAs for adding the GitLab-specific stuff, I'm not opposed to having it in the\nmain tree. For one, there doesn't seem to be a clean way to \"move it into\n`contrib/`\" - '.gitlab-ci.yml' must be at the root of the project [2], and\nmoving the $GITLAB_CI conditions out of the 'ci/*.sh' files into dedicated\nscripts would likely result in a lot of duplicated code (which doesn't solve\nthe maintenance burden issue this series intends to address).\n\nMore generally, there are lots of open source projects that include CI\nconfigurations across different forges, _especially_ those that are\nofficially mirrored across a bunch of them. As long as there are\ncontributors with a vested interest in keeping the GitLab CI definition\nstable (and your cover letter indicates that there are), and the GitLab\nstuff doesn't negatively impact any other CI configurations, I think it\nwarrants the same treatment as e.g. GitHub CI.\n\n[1] https://lore.kernel.org/git/cover.1698305961.git.ps@pks.im/\n[2] https://docs.gitlab.com/ee/ci/index.html#the-gitlab-ciyml-file\n\n> \n> Patrick\n> \n"},{"id":"484248","messageId":"ZUFRmRf7bN7U4afW@nand.local","threadId":"60432","inReplyTo":"ZUCw1B6oQaDWKx3O@tanuki","subject":"Re: [PATCH 0/5] ci: add GitLab CI definition","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-31T19:12:25Z","receivedAt":"2023-10-31T19:12:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Oct 31, 2023 at 08:46:28AM +0100, Patrick Steinhardt wrote:\n> On Mon, Oct 30, 2023 at 11:46:44AM -0400, Taylor Blau wrote:\n> > On Thu, Oct 26, 2023 at 09:59:59AM +0200, Patrick Steinhardt wrote:\n> > > And this is exactly what this patch series does: it adds GitLab-specific\n> > > knowledge to our CI scripts and adds a CI definition that builds on top\n> > > of those scripts. This is rather straight forward, as the scripts\n> > > already know to discern Azure Pipelines and GitHub Actions, and adding\n> > > a third item to this list feels quite natural. And by building on top of\n> > > the preexisting infra, the actual \".gitlab-ci.yml\" is really quite\n> > > small.\n> > >\n> > > I acknowledge that the Git project may not be willing to fully support\n> > > GitLab CI, and that's fine with me. If we want to further stress that\n> > > point then I'd also be perfectly happy to move the definitions into the\n> > > \"contrib/\" directory -- it would still be a huge win for our workflow.\n> > > In any case, I'm happy to keep on maintaining the intgeration with\n> > > GitLab CI, and if things break I'll do my best to fix them fast.\n> >\n> > I don't have any strong opinions here, but my preference would probably\n> > be to keep any GitLab-specific CI configuration limited to \"contrib\", if\n> > it lands in the tree at all.\n>\n> As mentioned, I would not mind at all if we wanted to instead carry this\n> as part of \"contrib/\".\n\nI think my concern with having it in Junio's tree at all is that it\ngives the impression that this is being maintained indefinitely. There\nis definitely some cruft in contrib that we should probably get rid of.\n\nBut since this is coming from an organization that sponsors work on the\nGit project, I think that my concerns there are somewhat more relaxed.\nI'm not worried about this going unmaintained, so I have no objection to\nhaving it live in contrib.\n\n> Yup, that's a valid concern. As mentioned, this patch series does not\n> have the intent to make GitLab CI a second authoritative CI platform.\n> GitHub Actions should remain the source of truth of whether a pipeline\n> passes or not. Most importantly, I do not want to require the maintainer\n> to now watch both pipelines on GitHub and GitLab. This might be another\n> indicator that the pipeline should rather be in \"contrib/\", so that\n> people don't start to treat it as authoritative.\n\nYeah, I agree with everything you said here.\n\n> > My other concern is that we're doubling the cost of any new changes to\n> > our CI definition. Perhaps this is more of an academic concern, but I\n> > think my fear would be that one or the other would fall behind on in\n> > implementation leading to further divergence between the two.\n> >\n> > I think having the new CI definition live in \"contrib\" somewhat\n> > addresses the \"which CI is authoritative?\" problem, but that it doesn't\n> > address the \"we have two of these\" problem.\n>\n> I do see that this requires us to be a bit more careful with future\n> changes to our CI definitions. But I think the additional work that this\n> creates is really very limited. Except for the `.gitlab-ci.yml`, there\n> are only 54 lines specific to GitLab in our CI scripts now, which I\n> think should be rather manageable.\n\nAgreed.\n\n> I also think that it is sensible to ensure that our CI scripts are as\n> agnostic to the CI platform as possible, as it ensures that we continue\n> to be agile here in the future if we are ever forced to switch due to\n> whatever reason. In the best case, our CI scripts would allow a user to\n> also easily run the tests locally via e.g. Docker. We're not there yet,\n> but this patch series is a good step into that direction already.\n\nAgreed there as well.\n\n> Last but not least, I actually think that having multiple supported CI\n> platforms also has the benefit that people can more readily set it up\n> for themselves. In theory, this has the potential to broaden the set of\n> people willing to contribute to our `ci/` scripts, which would in the\n> end also benefit GitHub Actions.\n>\n> In my opinion, this benefit is demonstrated by this patch series\n> already: besides all the changes that aim to prepare for GitLab CI,\n> there are also patches that deduplicate code and improve test coverage\n> for Alpine Linux. These changes likely wouldn't have happened if it\n> wasn't for the GitLab CI.\n\nYeah, I appreciate your careful and patient explanation here. I think\nthat my concerns here are resolved.\n\nThanks,\nTaylor\n"},{"id":"484253","messageId":"20231031193629.GB875658@coredump.intra.peff.net","threadId":"60432","inReplyTo":"d00b02e9-fb05-44bc-90ee-1851ef98dd26@gmail.com","subject":"Re: [PATCH v2 5/5] ci: add support for GitLab CI","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-10-31T19:36:29Z","receivedAt":"2023-10-31T19:36:32Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Oct 27, 2023 at 02:17:02PM +0100, Phillip Wood wrote:\n\n> On 27/10/2023 12:01, Oswald Buddenhagen wrote:\n> > On Fri, Oct 27, 2023 at 11:25:41AM +0200, Patrick Steinhardt wrote:\n> > > +    export GIT_PROVE_OPTS=\"--timer --jobs $(nproc)\"\n> > > +    export GIT_TEST_OPTS=\"--verbose-log -x\"\n> > > \n> > fwiw (as this is again only copied), export with assignment is a\n> > bash-ism\n> \n> Not according to https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#export\n> \n> SYNOPSIS\n> \n> export name[=word]...\n> \n> DESCRIPTION\n> \n> The shell shall give the export attribute to the variables\n> corresponding to the specified names, which shall cause them\n> to be in the environment of subsequently executed commands. If\n> the name of a variable is followed by = word, then the value\n> of that variable shall be set to word.\n> \n> It is true that in our test suite we separate a variable assignment when\n> exporting. Presumably that is because someone reported that their shell did\n> not support the \"export name=WORD\" syntax in the past. As we're already\n> using this syntax with the same docker images in Github Actions I think we\n> can assume it is safe here.\n\nI've wondered about the origin of this myself, and tried to do some\ndigging. All of the commits I found removing \"export var=val\" vaguely\nsay \"unportable\" or \"some shells can't handle\", etc.\n\nThe oldest mention I found on the mailing list was this thread:\n\n https://lore.kernel.org/git/7vfyb0wexo.fsf@assigned-by-dhcp.cox.net/\n\nwhich is even more explicit about its vagueness.\n\nI wouldn't be surprised if SunOS/Solaris /bin/sh had problems with it,\nas that has been a common headache shell in the past. But I think we\nfinally declared it unusable for other reasons (and they ship a more\ncapable shell in /usr/xpg6, if anybody even still wants to build on\nthose operating systems these days).\n\nSo it's possible that avoiding \"export var=val\" is mostly superstition,\nand we could loosen our rules these days. But some things to consider:\n\n 1. Some people may prefer reading the separated form (Junio indicates\n such in the thread linked above, but I don't know how strong a\n preference that is).\n\n 2. We won't really know if there is a odd-ball shell that rejects it\n unless we make a change and wait for a while to see if anybody\n screams. The existing ones in ci/ show that it is not a problem for\n the platforms where we run CI, but I suspect the scripts in t/ see\n a wider audience.\n\nI don't think this has any real bearing on the patches being proposed,\nbut I have been curious about this for our other scripts for a while\nnow.\n\n-Peff\n"},{"id":"484271","messageId":"xmqqttq6xr9k.fsf@gitster.g","threadId":"60432","inReplyTo":"ZUCw1B6oQaDWKx3O@tanuki","subject":"Re: [PATCH 0/5] ci: add GitLab CI definition","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-11-01T00:15:51Z","receivedAt":"2023-11-01T00:15:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> So I have some hesitation about trying to mirror this rather complicated\n>> set of build rules in another CI environment. My primary concern would\n>> be that the two might fall out of sync and a series that is green on\n>> GitHub would be red on GitLab, or vice-versa. Importantly, this can\n>> happen even without changes to the build definitions, since (AFAICT)\n>> both forges distribute new images automatically, so the set of packages\n>> installed in GitHub may not exactly match what's in GitLab (and\n>> vice-versa).\n>\n> Yup, that's a valid concern.\n\nIs it?\n\nI rather naïvely think different set of build options and tools\nrunning the tests would mean we gain wider test coverage. Even with\nthe current setup that relies on whatever GitHub offers, we already\nsee \"this version passes all tests except for the job on macOS\" and\n\"the version that was passing yesterday is not broken today---perhas\nthe image of the test environment has been updated and we need to\nadjust to it\" every once in a while.\n\n> As mentioned, this patch series does not have the intent to make\n> GitLab CI a second authoritative CI platform. GitHub Actions\n> should remain the source of truth of whether a pipeline passes or\n> not.\n\nI am not sure I follow. Often we take a version that happened to\nhave failed Actions tests when we know the reason of the failure\nhas nothing to do with the new code. From time to time people help\nto make CI tests less flakey, but flakes are expected.\n\n> Most importantly, I do not want to require the maintainer\n> to now watch both pipelines on GitHub and GitLab.\n\nI don't even make tests by GitHub Actions force me to do anything,\nso there is no worry here.\n\n> This might be another indicator that the pipeline should rather be\n> in \"contrib/\", so that people don't start to treat it as\n> authoritative.\n\nLet me step back and as more basic questions.\n\n - What do you mean by \"authoritative\"? For an authoritative CI\n test, contributors whose changes do not pass it should take it as\n a sign that their changes need more work? If so, isn't it a\n natural expectation and a good thing? Unless you expect the CI\n tests to be extra flakey, that is.\n\n - Are there reasons why you do not trust the CI tests at GitLab\n more than those run at GitHub?\n\n> Last but not least, I actually think that having multiple supported CI\n> platforms also has the benefit that people can more readily set it up\n> for themselves. In theory, this has the potential to broaden the set of\n> people willing to contribute to our `ci/` scripts, which would in the\n> end also benefit GitHub Actions.\n\nYes, assuming that we can do so without much cutting and pasting but\nwith a clear sharing of the infrastructure code, and the multiple\nsupported CI environments are not too flakey, I am with this rather\nnaïve worldview that the more we have the merrier we would be.\n\n> I understand your points, and especially the point about not having a\n> second authoritative CI platform. I'm very much on the same page as you\n> are here, and would be happy to move the definitions to \"contrib/\" if\n> you want me to.\n>\n> But I think we should also see the potential benefit of having a second\n> CI platform, as it enables a more diverse set of people to contribute.\n> which can ultimately end up benefitting our CI infra for both GitHub\n> Actions and GitLab CI.\n\nI do *not* want to add new things, if we were to use them ourselves,\nto \"contrib/\". We have passed that stage long time ago that keeping\neverything in my tree gives wider exposure and foster cooperation.\n\nThanks.\n"},{"id":"484277","messageId":"xmqqttq6w4f1.fsf@gitster.g","threadId":"60432","inReplyTo":"31ebe4c9-84aa-4d42-9aeb-712e2a6cece3@github.com","subject":"Re: [PATCH v4 5/8] ci: unify setup of some environment variables","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-11-01T03:14:42Z","receivedAt":"2023-11-01T03:14:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Victoria Dye writes:\n\n>> +MAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n>> +GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n>> +\n>> +GIT_TEST_OPTS=\"$GIT_TEST_OPTS --verbose-log -x\"\n>> +if test windows = \"$CI_OS_NAME\"\n>\n> Based on the deleted lines above, I think this would need to be:\n>\n> \tif test windows = \"$CI_OS_NAME\" || test windows_nt = \"$CI_OS_NAME\"\n>\n> I believe these settings are required on all Windows builds, though, so you could \n> instead match on the first 7 characters of $CI_OS_NAME:\n>\n> \tif test windows = \"$(echo \"$CI_OS_NAME\" | cut -c1-7)\"\n>\n> (full disclosure: I'm not 100% confident in the correctness of that shell syntax)\n>> +then\n>> +\tGIT_TEST_OPTS=\"$GIT_TEST_OPTS --no-chain-lint --no-bin-wrappers\"\n>> +fi\n\nEither\n\n\tcase \"$CI_OS_NAME\" in\n\twindows*)\n\t\tGIT_TEST_OPTS=...\n\tesac\n\nor if we want to be more selective, for documentation purposes\nespecially when it is unlikely for us to gain the third variant of\nwindows build:\n\n\tcase ... in\n\twindows | windows_nt)\n\t\tGIT_TEST_OPTS=...\n\tesac\n\n\n\n>> +\n>> +export GIT_TEST_OPTS\n>> +export GIT_PROVE_OPTS\n>> +\n>> good_trees_file=\"$cache_dir/good-trees\"\n>> \n>> mkdir -p \"$cache_dir\"\n"},{"id":"484278","messageId":"xmqqpm0uw41b.fsf@gitster.g","threadId":"60432","inReplyTo":"8e4d111f-3982-4989-90b5-08377fe9c5fd@github.com","subject":"Re: [PATCH v4 0/8] ci: add GitLab CI definition","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-11-01T03:22:56Z","receivedAt":"2023-11-01T03:23:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Victoria Dye writes:\n\n> As for adding the GitLab-specific stuff, I'm not opposed to having it in the\n> main tree. For one, there doesn't seem to be a clean way to \"move it into\n> `contrib/`\" - '.gitlab-ci.yml' must be at the root of the project [2], and\n> moving the $GITLAB_CI conditions out of the 'ci/*.sh' files into dedicated\n> scripts would likely result in a lot of duplicated code (which doesn't solve\n> the maintenance burden issue this series intends to address).\n>\n> More generally, there are lots of open source projects that include CI\n> configurations across different forges, _especially_ those that are\n> officially mirrored across a bunch of them. As long as there are\n> contributors with a vested interest in keeping the GitLab CI definition\n> stable (and your cover letter indicates that there are), and the GitLab\n> stuff doesn't negatively impact any other CI configurations, I think it\n> warrants the same treatment as e.g. GitHub CI.\n\nThanks for expressing this so clearly. I do prefer to add this as\nthe first class citizen (more generally, I do not want to add new\nthings to contrib/ at this point) if we are going to use it.\n\n"},{"id":"484279","messageId":"xmqqil6mw3ja.fsf@gitster.g","threadId":"60432","inReplyTo":"20231031193629.GB875658@coredump.intra.peff.net","subject":"Re: [PATCH v2 5/5] ci: add support for GitLab CI","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-11-01T03:33:45Z","receivedAt":"2023-11-01T03:33:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> So it's possible that avoiding \"export var=val\" is mostly superstition,\n\nThanks for digging up the old thread. I would not be surprised if\nit was already superstition back in the days.\n\n> 2. We won't really know if there is a odd-ball shell that rejects it\n> unless we make a change and wait for a while to see if anybody\n> screams. The existing ones in ci/ show that it is not a problem for\n> the platforms where we run CI, but I suspect the scripts in t/ see\n> a wider audience.\n\nWe could start with a single weather balloon use in t/ somewhere to\nsee if anybody screams. It would be a good place to start if we\nwant to get rid of this particular superstition.\n"},{"id":"484289","messageId":"ZUI6Fqj7QJQG6ox-@tanuki","threadId":"60432","inReplyTo":"xmqqpm0uw41b.fsf@gitster.g","subject":"Re: [PATCH v4 0/8] ci: add GitLab CI definition","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-01T11:44:22Z","receivedAt":"2023-11-01T11:44:36Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Nov 01, 2023 at 12:22:56PM +0900, Junio C Hamano wrote:\n> Victoria Dye writes:\n> \n> > As for adding the GitLab-specific stuff, I'm not opposed to having it in the\n> > main tree. For one, there doesn't seem to be a clean way to \"move it into\n> > `contrib/`\" - '.gitlab-ci.yml' must be at the root of the project [2], and\n> > moving the $GITLAB_CI conditions out of the 'ci/*.sh' files into dedicated\n> > scripts would likely result in a lot of duplicated code (which doesn't solve\n> > the maintenance burden issue this series intends to address).\n\nIt is possible to change the location of the `.gitlab-ci.yml` in GitLab\nprojects, so moving it into `contrib/` would work, too. But it does of\ncourse require additional setup by the project admin.\n\n> > More generally, there are lots of open source projects that include CI\n> > configurations across different forges, _especially_ those that are\n> > officially mirrored across a bunch of them. As long as there are\n> > contributors with a vested interest in keeping the GitLab CI definition\n> > stable (and your cover letter indicates that there are), and the GitLab\n> > stuff doesn't negatively impact any other CI configurations, I think it\n> > warrants the same treatment as e.g. GitHub CI.\n> \n> Thanks for expressing this so clearly. I do prefer to add this as\n> the first class citizen (more generally, I do not want to add new\n> things to contrib/ at this point) if we are going to use it.\n\nI certainly know that we in the Gitaly team will use this CI definition\non a daily basis, which is my main motivation to add and maintain it in\nthe future. I personally don't mind to add this as a first-class\ncitizen, because for my own workflows I'll certainly treat it as such.\nAnd if others can benefit from it, too, then I'm even happier.\n\nI'll keep it in the default location at `.gitlab-ci.yml` then, thanks\nfor your thoughts!\n\nPatrick\n"},{"id":"484290","messageId":"ZUI6I8_r7Lff0ruj@tanuki","threadId":"60432","inReplyTo":"31ebe4c9-84aa-4d42-9aeb-712e2a6cece3@github.com","subject":"Re: [PATCH v4 5/8] ci: unify setup of some environment variables","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-01T11:44:35Z","receivedAt":"2023-11-01T11:44:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Oct 31, 2023 at 10:06:24AM -0700, Victoria Dye wrote:\n> Patrick Steinhardt wrote:\n> > Both GitHub Actions and Azue Pipelines set up the environment variables\n> \n> s/Azue/Azure\n> \n> > GIT_TEST_OPTS, GIT_PROVE_OPTS and MAKEFLAGS. And while most values are\n> > actually the same, the setup is completely duplicate. With the upcoming\n> > support for GitLab CI this duplication would only extend even further.\n> > \n> > Unify the setup of those environment variables so that only the uncommon\n> > parts are separated. While at it, we also perform some additional small\n> > improvements:\n> > \n> > - We now always pass `--state=failed,slow,save` via GIT_PROVE_OPTS.\n> > It doesn't hurt on platforms where we don't persist the state, so\n> > this further reduces boilerplate.\n> > \n> > - When running on Windows systems we set `--no-chain-lint` and\n> > `--no-bin-wrappers`. Interestingly though, we did so _after_\n> > already having exported the respective environment variables.\n> > \n> > - We stop using `export VAR=value` syntax, which is a Bashism. It's\n> > not quite worth it as we still use this syntax all over the place,\n> > but it doesn't hurt readability either.\n> > \n> > Signed-off-by: Patrick Steinhardt \n> > ---\n> > ci/lib.sh | 24 ++++++++++++++----------\n> > 1 file changed, 14 insertions(+), 10 deletions(-)\n> > \n> > diff --git a/ci/lib.sh b/ci/lib.sh\n> > index 9ffdf743903..9a9b92c05b3 100755\n> > --- a/ci/lib.sh\n> > +++ b/ci/lib.sh\n> > @@ -175,11 +175,7 @@ then\n> > \t# among *all* phases)\n> > \tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n> > \n> > -\texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n> > -\texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n> > -\tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n> > -\ttest windows_nt != \"$CI_OS_NAME\" ||\n> > -\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n> > +\tGIT_TEST_OPTS=\"--write-junit-xml\"\n> > elif test true = \"$GITHUB_ACTIONS\"\n> > then\n> > \tCI_TYPE=github-actions\n> > @@ -198,17 +194,25 @@ then\n> > \n> > \tcache_dir=\"$HOME/none\"\n> > \n> > -\texport GIT_PROVE_OPTS=\"--timer --jobs 10\"\n> > -\texport GIT_TEST_OPTS=\"--verbose-log -x --github-workflow-markup\"\n> > -\tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n> > -\ttest windows != \"$CI_OS_NAME\" ||\n> > -\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n> > +\tGIT_TEST_OPTS=\"--github-workflow-markup\"\n> > else\n> > \techo \"Could not identify CI type\" >&2\n> > \tenv >&2\n> > \texit 1\n> > fi\n> > \n> > +MAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n> > +GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n> > +\n> > +GIT_TEST_OPTS=\"$GIT_TEST_OPTS --verbose-log -x\"\n> > +if test windows = \"$CI_OS_NAME\"\n> \n> Based on the deleted lines above, I think this would need to be:\n> \n> \tif test windows = \"$CI_OS_NAME\" || test windows_nt = \"$CI_OS_NAME\"\n> \n> I believe these settings are required on all Windows builds, though, so you could \n> instead match on the first 7 characters of $CI_OS_NAME:\n> \n> \tif test windows = \"$(echo \"$CI_OS_NAME\" | cut -c1-7)\"\n> \n> (full disclosure: I'm not 100% confident in the correctness of that shell syntax)\n\nOh, right. I didn't notice the slight difference between \"windows\" and\n\"windows_nt\". Thanks!\n\nPatrick\n\n> > +then\n> > +\tGIT_TEST_OPTS=\"$GIT_TEST_OPTS --no-chain-lint --no-bin-wrappers\"\n> > +fi\n> > +\n> > +export GIT_TEST_OPTS\n> > +export GIT_PROVE_OPTS\n> > +\n> > good_trees_file=\"$cache_dir/good-trees\"\n> > \n> > mkdir -p \"$cache_dir\"\n> \n"},{"id":"484291","messageId":"ZUI6Nv-HIxXuPMVC@tanuki","threadId":"60432","inReplyTo":"4c8c2f19-1a7e-4524-81e7-c74091e88edf@github.com","subject":"Re: [PATCH v4 8/8] ci: add support for GitLab CI","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-01T11:44:54Z","receivedAt":"2023-11-01T11:45:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Oct 31, 2023 at 10:47:44AM -0700, Victoria Dye wrote:\n> Patrick Steinhardt wrote:> diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\n> > index 6e845283680..48cb2e735b5 100755\n> > --- a/ci/install-docker-dependencies.sh\n> > +++ b/ci/install-docker-dependencies.sh\n> > @@ -7,6 +7,9 @@\n> > \n> > begin_group \"Install dependencies\"\n> > \n> > +# Required so that apt doesn't wait for user input on certain packages.\n> > +export DEBIAN_FRONTEND=noninteractive\n> > +\n> > case \"$jobname\" in\n> > linux32)\n> > \tlinux32 --32bit i386 sh -c '\n> > @@ -16,11 +19,19 @@ linux32)\n> > \t'\n> > \t;;\n> > linux-musl)\n> > -\tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n> > +\tapk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n> > \t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n> > \t\tapache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \\\n> > \t\tbash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n> > \t;;\n> > +linux-*)\n> > +\tapt update -q &&\n> > +\tapt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev \\\n> > +\t\tlibcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev \\\n> > +\t\tperl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl \\\n> > +\t\tlibdbd-sqlite3-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}} \\\n> > +\t\tapache2 cvs cvsps gnupg libcgi-pm-perl subversion\n> > +\t;;\n> > pedantic)\n> > \tdnf -yq update >/dev/null &&\n> > \tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n> \n> ...\n> \n> > diff --git a/ci/lib.sh b/ci/lib.sh\n> > index e14b1029fad..6e3d64004ec 100755\n> > --- a/ci/lib.sh\n> > +++ b/ci/lib.sh\n> > @@ -208,6 +224,7 @@ then\n> > \tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n> > \n> > \tGIT_TEST_OPTS=\"--write-junit-xml\"\n> > +\tJOBS=10\n> > elif test true = \"$GITHUB_ACTIONS\"\n> > then\n> > \tCI_TYPE=github-actions\n> \n> ...\n> \n> > -MAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n> > -GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n> > +MAKEFLAGS=\"$MAKEFLAGS --jobs=${JOBS}\"\n> > +GIT_PROVE_OPTS=\"--timer --jobs ${JOBS} --state=failed,slow,save\"\n> > \n> \n> Organizationally, this commit seems to be doing two things at once:\n> \n> - Adding GitLab-specific CI setup (either in the new .gitlab-ci.yml or in\n> conditions gated on \"gitlab-ci\").\n> - Updating the common CI scripts with things that are needed for GitLab CI,\n> but aren't conditioned on it (i.e. the patch excerpts I've included\n> above). \n> \n> I'd prefer these being separated into two patches, mainly to isolate \"things\n> that affect all CI\" from \"things that affect only GitLab CI\". This is\n> ultimately a pretty minor nit, though; if you're not planning on re-rolling\n> (or just disagree with what I'm suggesting :) ), I'm okay with leaving it\n> as-is.\n\nYeah, the JOBS refactoring can certainly be split out into a preparatory\ncommit where we unify the envvars (currently patch 5). But for the other\nchanges it makes a bit less sense to do so, in my opinion:\n\n - The DEBIAN_FRONTEND variable isn't needed before as the there are\n no Docker-based CI jobs that use apt.\n\n - Adding the shadow and sudo packages to the linux-musl job wouldn't\n be needed either as there are no cases yet where we run\n unprivileged CI builds via Docker.\n\n - Adding the apt packages as a preparatory step doesn't make much\n sense either as there is no Docker job using it.\n\nBut anyway. I will:\n\n - Move around the JOBS variable refactoring to a preparatory patch,\n which feels sensible to me.\n\n - Move the `DEBIAN_FRONTEND` varible into the \"linux-*\" case, which\n should further clarify that this only impacts the newly added and\n thus GitLab-specific infrastructure.\n\nWith these changes, the only thing left in this commit that is not\nguarded by a GitLab CI specific condition is the change to the\n\"linux-musl\" case where we install shadow and sudo now. But I don't feel\nlike it makes sense to move them into a standalone preparatory commit.\n\nThanks!\n\nPatrick\n\n> Otherwise, I can't comment on the correctness of the GitLab CI definition (I\n> assume you've tested it anyway), but AFAICT the changes above shouldn't break\n> GitHub CI.\n\n"},{"id":"484292","messageId":"ZUI8-z2luCAC-XtB@tanuki","threadId":"60432","inReplyTo":"xmqqttq6xr9k.fsf@gitster.g","subject":"Re: [PATCH 0/5] ci: add GitLab CI definition","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-01T11:56:43Z","receivedAt":"2023-11-01T11:56:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Nov 01, 2023 at 09:15:51AM +0900, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> >> So I have some hesitation about trying to mirror this rather complicated\n> >> set of build rules in another CI environment. My primary concern would\n> >> be that the two might fall out of sync and a series that is green on\n> >> GitHub would be red on GitLab, or vice-versa. Importantly, this can\n> >> happen even without changes to the build definitions, since (AFAICT)\n> >> both forges distribute new images automatically, so the set of packages\n> >> installed in GitHub may not exactly match what's in GitLab (and\n> >> vice-versa).\n> >\n> > Yup, that's a valid concern.\n> \n> Is it?\n> \n> I rather naïvely think different set of build options and tools\n> running the tests would mean we gain wider test coverage. Even with\n> the current setup that relies on whatever GitHub offers, we already\n> see \"this version passes all tests except for the job on macOS\" and\n> \"the version that was passing yesterday is not broken today---perhas\n> the image of the test environment has been updated and we need to\n> adjust to it\" every once in a while.\n> \n> > As mentioned, this patch series does not have the intent to make\n> > GitLab CI a second authoritative CI platform. GitHub Actions\n> > should remain the source of truth of whether a pipeline passes or\n> > not.\n> \n> I am not sure I follow. Often we take a version that happened to\n> have failed Actions tests when we know the reason of the failure\n> has nothing to do with the new code. From time to time people help\n> to make CI tests less flakey, but flakes are expected.\n> \n> > Most importantly, I do not want to require the maintainer\n> > to now watch both pipelines on GitHub and GitLab.\n> \n> I don't even make tests by GitHub Actions force me to do anything,\n> so there is no worry here.\n\nOkay.\n\n> > This might be another indicator that the pipeline should rather be\n> > in \"contrib/\", so that people don't start to treat it as\n> > authoritative.\n> \n> Let me step back and as more basic questions.\n> \n> - What do you mean by \"authoritative\"? For an authoritative CI\n> test, contributors whose changes do not pass it should take it as\n> a sign that their changes need more work? If so, isn't it a\n> natural expectation and a good thing? Unless you expect the CI\n> tests to be extra flakey, that is.\n\nI was assuming that GitHub Actions was considered to be \"the\" CI\nplatform of the Git project. But with your explanations above I think\nthat assumption may not necessarily hold, or at least not to the extent\nI assumed.\n\n> - Are there reasons why you do not trust the CI tests at GitLab\n> more than those run at GitHub?\n\nNo. Based on the above assumption I was simply treading carefully here.\nMost importantly, I didn't want to create the impression that either:\n\n - \"Now you have to watch two pipelines\", doubling the effort that CI\n infrastructure creates for you as a maintainer.\n\n - \"I want to eventually replace GitHub Actions\".\n\nThis carefulness probably also comes from the fact that GitLab and\nGitHub are direct competitors, so I was trying to preempt any kind of\nimplied agenda here. There is none, I just want to make sure that it\nbecomes easier for us at GitLab and other potential contributors that\nuse GitLab to contribute to Git.\n\nHope that makes sense.\n\n> > Last but not least, I actually think that having multiple supported CI\n> > platforms also has the benefit that people can more readily set it up\n> > for themselves. In theory, this has the potential to broaden the set of\n> > people willing to contribute to our `ci/` scripts, which would in the\n> > end also benefit GitHub Actions.\n> \n> Yes, assuming that we can do so without much cutting and pasting but\n> with a clear sharing of the infrastructure code, and the multiple\n> supported CI environments are not too flakey, I am with this rather\n> naïve worldview that the more we have the merrier we would be.\n> \n> > I understand your points, and especially the point about not having a\n> > second authoritative CI platform. I'm very much on the same page as you\n> > are here, and would be happy to move the definitions to \"contrib/\" if\n> > you want me to.\n> >\n> > But I think we should also see the potential benefit of having a second\n> > CI platform, as it enables a more diverse set of people to contribute.\n> > which can ultimately end up benefitting our CI infra for both GitHub\n> > Actions and GitLab CI.\n> \n> I do *not* want to add new things, if we were to use them ourselves,\n> to \"contrib/\". We have passed that stage long time ago that keeping\n> everything in my tree gives wider exposure and foster cooperation.\n\nFair enough.\n\nThanks for taking the time to make your thoughts clearer to me!\n\nPatrick\n"},{"id":"484293","messageId":"0ba396f2a331ce1e9f7e7122e13dc683e3ecc752.1698843660.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698843660.git.ps@pks.im","subject":"[PATCH v5 1/8] ci: reorder definitions for grouping functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-01T13:02:43Z","receivedAt":"2023-11-01T13:02:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We define a set of grouping functions that are used to group together\noutput in our CI, where these groups then end up as collapsible sections\nin the respective pipeline platform. The way these functions are defined\nis not easily extensible though as we have an up front check for the CI\n_not_ being GitHub Actions, where we define the non-stub logic in the\nelse branch.\n\nReorder the conditional branches such that we explicitly handle GitHub\nActions.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 20 ++++++++++----------\n 1 file changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 6fbb5bade12..eb384f4e952 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -1,16 +1,7 @@\n # Library of functions shared by all CI scripts\n \n-if test true != \"$GITHUB_ACTIONS\"\n+if test true = \"$GITHUB_ACTIONS\"\n then\n-\tbegin_group () { :; }\n-\tend_group () { :; }\n-\n-\tgroup () {\n-\t\tshift\n-\t\t\"$@\"\n-\t}\n-\tset -x\n-else\n \tbegin_group () {\n \t\tneed_to_end_group=t\n \t\techo \"::group::$1\" >&2\n@@ -42,6 +33,15 @@ else\n \t}\n \n \tbegin_group \"CI setup\"\n+else\n+\tbegin_group () { :; }\n+\tend_group () { :; }\n+\n+\tgroup () {\n+\t\tshift\n+\t\t\"$@\"\n+\t}\n+\tset -x\n fi\n \n # Set 'exit on error' for all CI scripts to let the caller know that\n-- \n2.42.0\n\n"},{"id":"484294","messageId":"821cfcd61254ace6981288a9ad23999502892873.1698843660.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698843660.git.ps@pks.im","subject":"[PATCH v5 2/8] ci: make grouping setup more generic","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-01T13:02:47Z","receivedAt":"2023-11-01T13:02:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Make the grouping setup more generic by always calling `begin_group ()`\nand `end_group ()` regardless of whether we have stubbed those functions\nor not. This ensures we can more readily add support for additional CI\nplatforms.\n\nFurthermore, the `group ()` function is made generic so that it is the\nsame for both GitHub Actions and for other platforms. There is a\nsemantic conflict here though: GitHub Actions used to call `set +x` in\n`group ()` whereas the non-GitHub case unconditionally uses `set -x`.\nThe latter would get overriden if we kept the `set +x` in the generic\nversion of `group ()`. To resolve this conflict, we simply drop the `set\n+x` in the generic variant of this function. As `begin_group ()` calls\n`set -x` anyway this is not much of a change though, as the only\ncommands that aren't printed anymore now are the ones between the\nbeginning of `group ()` and the end of `begin_group ()`.\n\nLast, this commit changes `end_group ()` to also accept a parameter that\nindicates _which_ group should end. This will be required by a later\ncommit that introduces support for GitLab CI.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 46 ++++++++++++++++++++++------------------------\n 1 file changed, 22 insertions(+), 24 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex eb384f4e952..b3411afae8e 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -14,36 +14,34 @@ then\n \t\tneed_to_end_group=\n \t\techo '::endgroup::' >&2\n \t}\n-\ttrap end_group EXIT\n-\n-\tgroup () {\n-\t\tset +x\n-\t\tbegin_group \"$1\"\n-\t\tshift\n-\t\t# work around `dash` not supporting `set -o pipefail`\n-\t\t(\n-\t\t\t\"$@\" 2>&1\n-\t\t\techo $? >exit.status\n-\t\t) |\n-\t\tsed 's/^\\(\\([^ ]*\\):\\([0-9]*\\):\\([0-9]*:\\) \\)\\(error\\|warning\\): /::\\5 file=\\2,line=\\3::\\1/'\n-\t\tres=$(cat exit.status)\n-\t\trm exit.status\n-\t\tend_group\n-\t\treturn $res\n-\t}\n-\n-\tbegin_group \"CI setup\"\n else\n \tbegin_group () { :; }\n \tend_group () { :; }\n \n-\tgroup () {\n-\t\tshift\n-\t\t\"$@\"\n-\t}\n \tset -x\n fi\n \n+group () {\n+\tgroup=\"$1\"\n+\tshift\n+\tbegin_group \"$group\"\n+\n+\t# work around `dash` not supporting `set -o pipefail`\n+\t(\n+\t\t\"$@\" 2>&1\n+\t\techo $? >exit.status\n+\t) |\n+\tsed 's/^\\(\\([^ ]*\\):\\([0-9]*\\):\\([0-9]*:\\) \\)\\(error\\|warning\\): /::\\5 file=\\2,line=\\3::\\1/'\n+\tres=$(cat exit.status)\n+\trm exit.status\n+\n+\tend_group \"$group\"\n+\treturn $res\n+}\n+\n+begin_group \"CI setup\"\n+trap \"end_group 'CI setup'\" EXIT\n+\n # Set 'exit on error' for all CI scripts to let the caller know that\n # something went wrong.\n #\n@@ -287,5 +285,5 @@ esac\n \n MAKEFLAGS=\"$MAKEFLAGS CC=${CC:-cc}\"\n \n-end_group\n+end_group \"CI setup\"\n set -x\n-- \n2.42.0\n\n"},{"id":"484295","messageId":"cover.1698843660.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698305961.git.ps@pks.im","subject":"[PATCH v5 0/8] ci: add support for GitLab CI","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-01T13:02:38Z","receivedAt":"2023-11-01T13:02:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the fifth version of my patch series that introduces support for\nGitLab CI.\n\nThere are only minor changes compared to v4:\n\n - Patch 5: Fixed a typo in the commit message. Furthermore, I've\n dropped the note about `export VAR=value` being a Bashism.\n\n - Patch 5: Fixed the accidentally-dropped \"windows_nt\" case.\n\n - Patch 5: Introduce the JOBS variable right away so that there is\n less churn in the patch that introduces GitLab CI.\n\n - Patch 8: Moved `DEBIAN_FRONTEND` into the \"linux-*\" case to make\n clear that it shouldn't impact anything else.\n\n - Patch 8: Dropped the paragraph about GitLab CI not being a first\n class citizen based on Junio's feedback. This setup is going to be\n actively used (at least) by us at GitLab, and thus we're also\n going to maintain it.\n\nAfter both Junio's and Victoria's feedback I've kept `.gitlab-ci.yml` in\nits canonical location.\n\nThe pipeline for this version can be found at [1].\n\nThanks for all the discussion and feedback so far!\n\nPatrick\n\n[1]: https://gitlab.com/gitlab-org/git/-/pipelines/1057566736\n\nPatrick Steinhardt (8):\n ci: reorder definitions for grouping functions\n ci: make grouping setup more generic\n ci: group installation of Docker dependencies\n ci: split out logic to set up failed test artifacts\n ci: unify setup of some environment variables\n ci: squelch warnings when testing with unusable Git repo\n ci: install test dependencies for linux-musl\n ci: add support for GitLab CI\n\n .gitlab-ci.yml | 53 +++++++++\n ci/install-docker-dependencies.sh | 23 +++-\n ci/lib.sh | 191 +++++++++++++++++++++---------\n ci/print-test-failures.sh | 6 +\n t/lib-httpd.sh | 17 ++-\n 5 files changed, 234 insertions(+), 56 deletions(-)\n create mode 100644 .gitlab-ci.yml\n\nRange-diff against v4:\n1: 8595fe5016a = 1: 0ba396f2a33 ci: reorder definitions for grouping functions\n2: 7358a943392 = 2: 821cfcd6125 ci: make grouping setup more generic\n3: 6d842592c6f = 3: 6e5bcf143c8 ci: group installation of Docker dependencies\n4: e15651b3f5d = 4: 2182acf5bfc ci: split out logic to set up failed test artifacts\n5: a64799b6e25 ! 5: 6078aea246d ci: unify setup of some environment variables\n @@ Metadata\n ## Commit message ##\n ci: unify setup of some environment variables\n \n - Both GitHub Actions and Azue Pipelines set up the environment variables\n + Both GitHub Actions and Azure Pipelines set up the environment variables\n GIT_TEST_OPTS, GIT_PROVE_OPTS and MAKEFLAGS. And while most values are\n actually the same, the setup is completely duplicate. With the upcoming\n support for GitLab CI this duplication would only extend even further.\n @@ Commit message\n `--no-bin-wrappers`. Interestingly though, we did so _after_\n already having exported the respective environment variables.\n \n - - We stop using `export VAR=value` syntax, which is a Bashism. It's\n - not quite worth it as we still use this syntax all over the place,\n - but it doesn't hurt readability either.\n -\n Signed-off-by: Patrick Steinhardt \n \n ## ci/lib.sh ##\n @@ ci/lib.sh: then\n -\ttest windows_nt != \"$CI_OS_NAME\" ||\n -\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n +\tGIT_TEST_OPTS=\"--write-junit-xml\"\n ++\tJOBS=10\n elif test true = \"$GITHUB_ACTIONS\"\n then\n \tCI_TYPE=github-actions\n @@ ci/lib.sh: then\n -\ttest windows != \"$CI_OS_NAME\" ||\n -\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n +\tGIT_TEST_OPTS=\"--github-workflow-markup\"\n ++\tJOBS=10\n else\n \techo \"Could not identify CI type\" >&2\n \tenv >&2\n \texit 1\n fi\n \n -+MAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n -+GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n ++MAKEFLAGS=\"$MAKEFLAGS --jobs=$JOBS\"\n ++GIT_PROVE_OPTS=\"--timer --jobs $JOBS --state=failed,slow,save\"\n +\n +GIT_TEST_OPTS=\"$GIT_TEST_OPTS --verbose-log -x\"\n -+if test windows = \"$CI_OS_NAME\"\n -+then\n ++case \"$CI_OS_NAME\" in\n ++windows|windows_nt)\n +\tGIT_TEST_OPTS=\"$GIT_TEST_OPTS --no-chain-lint --no-bin-wrappers\"\n -+fi\n ++\t;;\n ++esac\n +\n +export GIT_TEST_OPTS\n +export GIT_PROVE_OPTS\n6: f7d2a8666fe = 6: d69bde92f2f ci: squelch warnings when testing with unusable Git repo\n7: 9b43b0d90e3 = 7: b911c005bae ci: install test dependencies for linux-musl\n8: f3f2c98a0dc ! 8: 5784d03a6f1 ci: add support for GitLab CI\n @@ Commit message\n that fail to compile or pass tests in GitHub Workflows. We would thus\n like to integrate the GitLab CI configuration into the Git project to\n help us send better patch series upstream and thus reduce overhead for\n - the maintainer.\n -\n - The integration does not necessarily have to be a first-class citizen,\n - which would in practice only add to the fallout that pipeline failures\n - have for the maintainer. That being said, we are happy to maintain this\n - alternative CI setup for the Git project and will make test results\n - available as part of our own mirror of the Git project at [1].\n + the maintainer. Results of these pipeline runs will be made available\n + (at least) in GitLab's mirror of the Git project at [1].\n \n This commit introduces the integration into our regular CI scripts so\n that most of the setup continues to be shared across all of the CI\n @@ .gitlab-ci.yml (new)\n + when: on_failure\n \n ## ci/install-docker-dependencies.sh ##\n -@@\n - \n - begin_group \"Install dependencies\"\n - \n -+# Required so that apt doesn't wait for user input on certain packages.\n -+export DEBIAN_FRONTEND=noninteractive\n -+\n - case \"$jobname\" in\n - linux32)\n - \tlinux32 --32bit i386 sh -c '\n @@ ci/install-docker-dependencies.sh: linux32)\n \t'\n \t;;\n @@ ci/install-docker-dependencies.sh: linux32)\n \t\tbash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n \t;;\n +linux-*)\n ++\t# Required so that apt doesn't wait for user input on certain packages.\n ++\texport DEBIAN_FRONTEND=noninteractive\n ++\n +\tapt update -q &&\n +\tapt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev \\\n +\t\tlibcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev \\\n @@ ci/lib.sh: then\n \tbegin_group () { :; }\n \tend_group () { :; }\n @@ ci/lib.sh: then\n - \tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n - \n - \tGIT_TEST_OPTS=\"--write-junit-xml\"\n -+\tJOBS=10\n - elif test true = \"$GITHUB_ACTIONS\"\n - then\n - \tCI_TYPE=github-actions\n -@@ ci/lib.sh: then\n - \tcache_dir=\"$HOME/none\"\n \n \tGIT_TEST_OPTS=\"--github-workflow-markup\"\n -+\tJOBS=10\n + \tJOBS=10\n +elif test true = \"$GITLAB_CI\"\n +then\n +\tCI_TYPE=gitlab-ci\n @@ ci/lib.sh: then\n else\n \techo \"Could not identify CI type\" >&2\n \tenv >&2\n - \texit 1\n - fi\n - \n --MAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n --GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n -+MAKEFLAGS=\"$MAKEFLAGS --jobs=${JOBS}\"\n -+GIT_PROVE_OPTS=\"--timer --jobs ${JOBS} --state=failed,slow,save\"\n - \n - GIT_TEST_OPTS=\"$GIT_TEST_OPTS --verbose-log -x\"\n - if test windows = \"$CI_OS_NAME\"\n \n ## ci/print-test-failures.sh ##\n @@ ci/print-test-failures.sh: do\n\nbase-commit: 692be87cbba55e8488f805d236f2ad50483bd7d5\n-- \n2.42.0\n\n"},{"id":"484296","messageId":"6e5bcf143c86650f153de63b0438c9e09c9dc005.1698843660.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698843660.git.ps@pks.im","subject":"[PATCH v5 3/8] ci: group installation of Docker dependencies","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-01T13:02:52Z","receivedAt":"2023-11-01T13:03:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The output of CI jobs tends to be quite long-winded and hard to digest.\nTo help with this, many CI systems provide the ability to group output\ninto collapsible sections, and we're also doing this in some of our\nscripts.\n\nOne notable omission is the script to install Docker dependencies.\nAddress it to bring more structure to the output for Docker-based jobs.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-docker-dependencies.sh | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\nindex 78b7e326da6..d0bc19d3bb3 100755\n--- a/ci/install-docker-dependencies.sh\n+++ b/ci/install-docker-dependencies.sh\n@@ -3,6 +3,10 @@\n # Install dependencies required to build and test Git inside container\n #\n \n+. ${0%/*}/lib.sh\n+\n+begin_group \"Install dependencies\"\n+\n case \"$jobname\" in\n linux32)\n \tlinux32 --32bit i386 sh -c '\n@@ -20,3 +24,5 @@ pedantic)\n \tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n \t;;\n esac\n+\n+end_group \"Install dependencies\"\n-- \n2.42.0\n\n"},{"id":"484297","messageId":"2182acf5bfce5abf00287b035d321a9cb5b18941.1698843660.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698843660.git.ps@pks.im","subject":"[PATCH v5 4/8] ci: split out logic to set up failed test artifacts","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-01T13:02:56Z","receivedAt":"2023-11-01T13:03:01Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have some logic in place to create a directory with the output from\nfailed tests, which will then subsequently be uploaded as CI artifacts.\nWe're about to add support for GitLab CI, which will want to reuse the\nlogic.\n\nSplit the logic into a separate function so that it is reusable.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 40 ++++++++++++++++++++++------------------\n 1 file changed, 22 insertions(+), 18 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex b3411afae8e..9ffdf743903 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -131,6 +131,27 @@ handle_failed_tests () {\n \treturn 1\n }\n \n+create_failed_test_artifacts () {\n+\tmkdir -p t/failed-test-artifacts\n+\n+\tfor test_exit in t/test-results/*.exit\n+\tdo\n+\t\ttest 0 != \"$(cat \"$test_exit\")\" || continue\n+\n+\t\ttest_name=\"${test_exit%.exit}\"\n+\t\ttest_name=\"${test_name##*/}\"\n+\t\tprintf \"\\\\e[33m\\\\e[1m=== Failed test: ${test_name} ===\\\\e[m\\\\n\"\n+\t\techo \"The full logs are in the 'print test failures' step below.\"\n+\t\techo \"See also the 'failed-tests-*' artifacts attached to this run.\"\n+\t\tcat \"t/test-results/$test_name.markup\"\n+\n+\t\ttrash_dir=\"t/trash directory.$test_name\"\n+\t\tcp \"t/test-results/$test_name.out\" t/failed-test-artifacts/\n+\t\ttar czf t/failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n+\tdone\n+\treturn 1\n+}\n+\n # GitHub Action doesn't set TERM, which is required by tput\n export TERM=${TERM:-dumb}\n \n@@ -171,25 +192,8 @@ then\n \tCC=\"${CC_PACKAGE:-${CC:-gcc}}\"\n \tDONT_SKIP_TAGS=t\n \thandle_failed_tests () {\n-\t\tmkdir -p t/failed-test-artifacts\n \t\techo \"FAILED_TEST_ARTIFACTS=t/failed-test-artifacts\" >>$GITHUB_ENV\n-\n-\t\tfor test_exit in t/test-results/*.exit\n-\t\tdo\n-\t\t\ttest 0 != \"$(cat \"$test_exit\")\" || continue\n-\n-\t\t\ttest_name=\"${test_exit%.exit}\"\n-\t\t\ttest_name=\"${test_name##*/}\"\n-\t\t\tprintf \"\\\\e[33m\\\\e[1m=== Failed test: ${test_name} ===\\\\e[m\\\\n\"\n-\t\t\techo \"The full logs are in the 'print test failures' step below.\"\n-\t\t\techo \"See also the 'failed-tests-*' artifacts attached to this run.\"\n-\t\t\tcat \"t/test-results/$test_name.markup\"\n-\n-\t\t\ttrash_dir=\"t/trash directory.$test_name\"\n-\t\t\tcp \"t/test-results/$test_name.out\" t/failed-test-artifacts/\n-\t\t\ttar czf t/failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n-\t\tdone\n-\t\treturn 1\n+\t\tcreate_failed_test_artifacts\n \t}\n \n \tcache_dir=\"$HOME/none\"\n-- \n2.42.0\n\n"},{"id":"484298","messageId":"6078aea246d874ba18bbd9ce2c0a7fb5c03e1564.1698843660.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698843660.git.ps@pks.im","subject":"[PATCH v5 5/8] ci: unify setup of some environment variables","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-01T13:03:00Z","receivedAt":"2023-11-01T13:03:11Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Both GitHub Actions and Azure Pipelines set up the environment variables\nGIT_TEST_OPTS, GIT_PROVE_OPTS and MAKEFLAGS. And while most values are\nactually the same, the setup is completely duplicate. With the upcoming\nsupport for GitLab CI this duplication would only extend even further.\n\nUnify the setup of those environment variables so that only the uncommon\nparts are separated. While at it, we also perform some additional small\nimprovements:\n\n - We now always pass `--state=failed,slow,save` via GIT_PROVE_OPTS.\n It doesn't hurt on platforms where we don't persist the state, so\n this further reduces boilerplate.\n\n - When running on Windows systems we set `--no-chain-lint` and\n `--no-bin-wrappers`. Interestingly though, we did so _after_\n already having exported the respective environment variables.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 27 +++++++++++++++++----------\n 1 file changed, 17 insertions(+), 10 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 9ffdf743903..0b35da3cfdb 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -175,11 +175,8 @@ then\n \t# among *all* phases)\n \tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n \n-\texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n-\texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n-\tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n-\ttest windows_nt != \"$CI_OS_NAME\" ||\n-\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n+\tGIT_TEST_OPTS=\"--write-junit-xml\"\n+\tJOBS=10\n elif test true = \"$GITHUB_ACTIONS\"\n then\n \tCI_TYPE=github-actions\n@@ -198,17 +195,27 @@ then\n \n \tcache_dir=\"$HOME/none\"\n \n-\texport GIT_PROVE_OPTS=\"--timer --jobs 10\"\n-\texport GIT_TEST_OPTS=\"--verbose-log -x --github-workflow-markup\"\n-\tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n-\ttest windows != \"$CI_OS_NAME\" ||\n-\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n+\tGIT_TEST_OPTS=\"--github-workflow-markup\"\n+\tJOBS=10\n else\n \techo \"Could not identify CI type\" >&2\n \tenv >&2\n \texit 1\n fi\n \n+MAKEFLAGS=\"$MAKEFLAGS --jobs=$JOBS\"\n+GIT_PROVE_OPTS=\"--timer --jobs $JOBS --state=failed,slow,save\"\n+\n+GIT_TEST_OPTS=\"$GIT_TEST_OPTS --verbose-log -x\"\n+case \"$CI_OS_NAME\" in\n+windows|windows_nt)\n+\tGIT_TEST_OPTS=\"$GIT_TEST_OPTS --no-chain-lint --no-bin-wrappers\"\n+\t;;\n+esac\n+\n+export GIT_TEST_OPTS\n+export GIT_PROVE_OPTS\n+\n good_trees_file=\"$cache_dir/good-trees\"\n \n mkdir -p \"$cache_dir\"\n-- \n2.42.0\n\n"},{"id":"484299","messageId":"d69bde92f2f30c87e8884873b20862aef10aee28.1698843660.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698843660.git.ps@pks.im","subject":"[PATCH v5 6/8] ci: squelch warnings when testing with unusable Git repo","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-01T13:03:04Z","receivedAt":"2023-11-01T13:03:13Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Our CI jobs that run on Docker also use mostly the same architecture to\nbuild and test Git via the \"ci/run-build-and-tests.sh\" script. These\nscripts also provide some functionality to massage the Git repository\nwe're supposedly operating in.\n\nIn our Docker-based infrastructure we may not even have a Git repository\navailable though, which leads to warnings when those functions execute.\nMake the helpers exit gracefully in case either there is no Git in our\nPATH, or when not running in a Git repository.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 32 ++++++++++++++++++++++++++++++++\n 1 file changed, 32 insertions(+)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 0b35da3cfdb..f0a2f80f094 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -69,10 +69,32 @@ skip_branch_tip_with_tag () {\n \tfi\n }\n \n+# Check whether we can use the path passed via the first argument as Git\n+# repository.\n+is_usable_git_repository () {\n+\t# We require Git in our PATH, otherwise we cannot access repositories\n+\t# at all.\n+\tif ! command -v git >/dev/null\n+\tthen\n+\t\treturn 1\n+\tfi\n+\n+\t# And the target directory needs to be a proper Git repository.\n+\tif ! git -C \"$1\" rev-parse 2>/dev/null\n+\tthen\n+\t\treturn 1\n+\tfi\n+}\n+\n # Save some info about the current commit's tree, so we can skip the build\n # job if we encounter the same tree again and can provide a useful info\n # message.\n save_good_tree () {\n+\tif ! is_usable_git_repository .\n+\tthen\n+\t\treturn\n+\tfi\n+\n \techo \"$(git rev-parse $CI_COMMIT^{tree}) $CI_COMMIT $CI_JOB_NUMBER $CI_JOB_ID\" >>\"$good_trees_file\"\n \t# limit the file size\n \ttail -1000 \"$good_trees_file\" >\"$good_trees_file\".tmp\n@@ -88,6 +110,11 @@ skip_good_tree () {\n \t\treturn\n \tfi\n \n+\tif ! is_usable_git_repository .\n+\tthen\n+\t\treturn\n+\tfi\n+\n \tif ! good_tree_info=\"$(grep \"^$(git rev-parse $CI_COMMIT^{tree}) \" \"$good_trees_file\")\"\n \tthen\n \t\t# Haven't seen this tree yet, or no cached good trees file yet.\n@@ -119,6 +146,11 @@ skip_good_tree () {\n }\n \n check_unignored_build_artifacts () {\n+\tif ! is_usable_git_repository .\n+\tthen\n+\t\treturn\n+\tfi\n+\n \t! git ls-files --other --exclude-standard --error-unmatch \\\n \t\t-- ':/*' 2>/dev/null ||\n \t{\n-- \n2.42.0\n\n"},{"id":"484300","messageId":"b911c005bae115f79bacebff8e5f5f2d534551b8.1698843660.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698843660.git.ps@pks.im","subject":"[PATCH v5 7/8] ci: install test dependencies for linux-musl","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-01T13:03:09Z","receivedAt":"2023-11-01T13:03:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The linux-musl CI job executes tests on Alpine Linux, which is based on\nmusl libc instead of glibc. We're missing some test dependencies though,\nwhich causes us to skip a subset of tests.\n\nInstall these test dependencies to increase our test coverage on this\nplatform. There are still some missing test dependecies, but these do\nnot have a corresponding package in the Alpine repositories:\n\n - p4 and p4d, both parts of the Perforce version control system.\n\n - cvsps, which generates patch sets for CVS.\n\n - Subversion and the SVN::Core Perl library, the latter of which is\n not available in the Alpine repositories. While the tool itself is\n available, all Subversion-related tests are skipped without the\n SVN::Core Perl library anyway.\n\nThe Apache2-based tests require a bit more care though. For one, the\nmodule path is different on Alpine Linux, which requires us to add it to\nthe list of known module paths to detect it. But second, the WebDAV\nmodule on Alpine Linux is broken because it does not bundle the default\ndatabase backend [1]. We thus need to skip the WebDAV-based tests on\nAlpine Linux for now.\n\n[1]: https://gitlab.alpinelinux.org/alpine/aports/-/issues/13112\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-docker-dependencies.sh | 4 +++-\n t/lib-httpd.sh | 17 ++++++++++++++++-\n 2 files changed, 19 insertions(+), 2 deletions(-)\n\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\nindex d0bc19d3bb3..6e845283680 100755\n--- a/ci/install-docker-dependencies.sh\n+++ b/ci/install-docker-dependencies.sh\n@@ -17,7 +17,9 @@ linux32)\n \t;;\n linux-musl)\n \tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n-\t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n+\t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n+\t\tapache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \\\n+\t\tbash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n \t;;\n pedantic)\n \tdnf -yq update >/dev/null &&\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex 2fb1b2ae561..9ea74927c40 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -67,7 +67,8 @@ for DEFAULT_HTTPD_MODULE_PATH in '/usr/libexec/apache2' \\\n \t\t\t\t '/usr/lib/apache2/modules' \\\n \t\t\t\t '/usr/lib64/httpd/modules' \\\n \t\t\t\t '/usr/lib/httpd/modules' \\\n-\t\t\t\t '/usr/libexec/httpd'\n+\t\t\t\t '/usr/libexec/httpd' \\\n+\t\t\t\t '/usr/lib/apache2'\n do\n \tif test -d \"$DEFAULT_HTTPD_MODULE_PATH\"\n \tthen\n@@ -127,6 +128,20 @@ else\n \t\t\"Could not identify web server at '$LIB_HTTPD_PATH'\"\n fi\n \n+if test -n \"$LIB_HTTPD_DAV\" && test -f /etc/os-release\n+then\n+\tcase \"$(grep \"^ID=\" /etc/os-release | cut -d= -f2-)\" in\n+\talpine)\n+\t\t# The WebDAV module in Alpine Linux is broken at least up to\n+\t\t# Alpine v3.16 as the default DBM driver is missing.\n+\t\t#\n+\t\t# https://gitlab.alpinelinux.org/alpine/aports/-/issues/13112\n+\t\ttest_skip_or_die GIT_TEST_HTTPD \\\n+\t\t\t\"Apache WebDAV module does not have default DBM backend driver\"\n+\t\t;;\n+\tesac\n+fi\n+\n install_script () {\n \twrite_script \"$HTTPD_ROOT_PATH/$1\" <\"$TEST_PATH/$1\"\n }\n-- \n2.42.0\n\n"},{"id":"484301","messageId":"5784d03a6f102cac88f7a857e57987968e82a809.1698843660.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698843660.git.ps@pks.im","subject":"[PATCH v5 8/8] ci: add support for GitLab CI","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-01T13:03:13Z","receivedAt":"2023-11-01T13:03:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We already support Azure Pipelines and GitHub Workflows in the Git\nproject, but until now we do not have support for GitLab CI. While it is\narguably not in the interest of the Git project to maintain a ton of\ndifferent CI platforms, GitLab has recently ramped up its efforts and\ntries to contribute to the Git project more regularly.\n\nPart of a problem we hit at GitLab rather frequently is that our own,\ncustom CI setup we have is so different to the setup that the Git\nproject has. More esoteric jobs like \"linux-TEST-vars\" that also set a\ncouple of environment variables do not exist in GitLab's custom CI\nsetup, and maintaining them to keep up with what Git does feels like\nwasted time. The result is that we regularly send patch series upstream\nthat fail to compile or pass tests in GitHub Workflows. We would thus\nlike to integrate the GitLab CI configuration into the Git project to\nhelp us send better patch series upstream and thus reduce overhead for\nthe maintainer. Results of these pipeline runs will be made available\n(at least) in GitLab's mirror of the Git project at [1].\n\nThis commit introduces the integration into our regular CI scripts so\nthat most of the setup continues to be shared across all of the CI\nsolutions. Note that as the builds on GitLab CI run as unprivileged\nuser, we need to pull in both sudo and shadow packages to our Alpine\nbased job to set this up.\n\n[1]: https://gitlab.com/gitlab-org/git\n\nSigned-off-by: Patrick Steinhardt \n---\n .gitlab-ci.yml | 53 +++++++++++++++++++++++++++++++\n ci/install-docker-dependencies.sh | 13 +++++++-\n ci/lib.sh | 44 +++++++++++++++++++++++++\n ci/print-test-failures.sh | 6 ++++\n 4 files changed, 115 insertions(+), 1 deletion(-)\n create mode 100644 .gitlab-ci.yml\n\ndiff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\nnew file mode 100644\nindex 00000000000..cd98bcb18aa\n--- /dev/null\n+++ b/.gitlab-ci.yml\n@@ -0,0 +1,53 @@\n+default:\n+ timeout: 2h\n+\n+workflow:\n+ rules:\n+ - if: $CI_PIPELINE_SOURCE == \"merge_request_event\"\n+ - if: $CI_COMMIT_TAG\n+ - if: $CI_COMMIT_REF_PROTECTED == \"true\"\n+\n+test:\n+ image: $image\n+ before_script:\n+ - ./ci/install-docker-dependencies.sh\n+ script:\n+ - useradd builder --create-home\n+ - chown -R builder \"${CI_PROJECT_DIR}\"\n+ - sudo --preserve-env --set-home --user=builder ./ci/run-build-and-tests.sh\n+ after_script:\n+ - |\n+ if test \"$CI_JOB_STATUS\" != 'success'\n+ then\n+ sudo --preserve-env --set-home --user=builder ./ci/print-test-failures.sh\n+ fi\n+ parallel:\n+ matrix:\n+ - jobname: linux-sha256\n+ image: ubuntu:latest\n+ CC: clang\n+ - jobname: linux-gcc\n+ image: ubuntu:20.04\n+ CC: gcc\n+ CC_PACKAGE: gcc-8\n+ - jobname: linux-TEST-vars\n+ image: ubuntu:20.04\n+ CC: gcc\n+ CC_PACKAGE: gcc-8\n+ - jobname: linux-gcc-default\n+ image: ubuntu:latest\n+ CC: gcc\n+ - jobname: linux-leaks\n+ image: ubuntu:latest\n+ CC: gcc\n+ - jobname: linux-asan-ubsan\n+ image: ubuntu:latest\n+ CC: clang\n+ - jobname: pedantic\n+ image: fedora:latest\n+ - jobname: linux-musl\n+ image: alpine:latest\n+ artifacts:\n+ paths:\n+ - t/failed-test-artifacts\n+ when: on_failure\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\nindex 6e845283680..48c43f0f907 100755\n--- a/ci/install-docker-dependencies.sh\n+++ b/ci/install-docker-dependencies.sh\n@@ -16,11 +16,22 @@ linux32)\n \t'\n \t;;\n linux-musl)\n-\tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n+\tapk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n \t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n \t\tapache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \\\n \t\tbash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n \t;;\n+linux-*)\n+\t# Required so that apt doesn't wait for user input on certain packages.\n+\texport DEBIAN_FRONTEND=noninteractive\n+\n+\tapt update -q &&\n+\tapt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev \\\n+\t\tlibcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev \\\n+\t\tperl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl \\\n+\t\tlibdbd-sqlite3-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}} \\\n+\t\tapache2 cvs cvsps gnupg libcgi-pm-perl subversion\n+\t;;\n pedantic)\n \tdnf -yq update >/dev/null &&\n \tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex f0a2f80f094..2718ce8776c 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -14,6 +14,22 @@ then\n \t\tneed_to_end_group=\n \t\techo '::endgroup::' >&2\n \t}\n+elif test true = \"$GITLAB_CI\"\n+then\n+\tbegin_group () {\n+\t\tneed_to_end_group=t\n+\t\tprintf \"\\e[0Ksection_start:$(date +%s):$(echo \"$1\" | tr ' ' _)\\r\\e[0K$1\\n\"\n+\t\ttrap \"end_group '$1'\" EXIT\n+\t\tset -x\n+\t}\n+\n+\tend_group () {\n+\t\ttest -n \"$need_to_end_group\" || return 0\n+\t\tset +x\n+\t\tneed_to_end_group=\n+\t\tprintf \"\\e[0Ksection_end:$(date +%s):$(echo \"$1\" | tr ' ' _)\\r\\e[0K\\n\"\n+\t\ttrap - EXIT\n+\t}\n else\n \tbegin_group () { :; }\n \tend_group () { :; }\n@@ -229,6 +245,34 @@ then\n \n \tGIT_TEST_OPTS=\"--github-workflow-markup\"\n \tJOBS=10\n+elif test true = \"$GITLAB_CI\"\n+then\n+\tCI_TYPE=gitlab-ci\n+\tCI_BRANCH=\"$CI_COMMIT_REF_NAME\"\n+\tCI_COMMIT=\"$CI_COMMIT_SHA\"\n+\tcase \"$CI_JOB_IMAGE\" in\n+\tmacos-*)\n+\t\tCI_OS_NAME=osx;;\n+\talpine:*|fedora:*|ubuntu:*)\n+\t\tCI_OS_NAME=linux;;\n+\t*)\n+\t\techo \"Could not identify OS image\" >&2\n+\t\tenv >&2\n+\t\texit 1\n+\t\t;;\n+\tesac\n+\tCI_REPO_SLUG=\"$CI_PROJECT_PATH\"\n+\tCI_JOB_ID=\"$CI_JOB_ID\"\n+\tCC=\"${CC_PACKAGE:-${CC:-gcc}}\"\n+\tDONT_SKIP_TAGS=t\n+\thandle_failed_tests () {\n+\t\tcreate_failed_test_artifacts\n+\t}\n+\n+\tcache_dir=\"$HOME/none\"\n+\n+\truns_on_pool=$(echo \"$CI_JOB_IMAGE\" | tr : -)\n+\tJOBS=$(nproc)\n else\n \techo \"Could not identify CI type\" >&2\n \tenv >&2\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex 57277eefcd0..c33ad4e3a22 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -51,6 +51,12 @@ do\n \t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n \t\t\tcontinue\n \t\t\t;;\n+\t\tgitlab-ci)\n+\t\t\tmkdir -p failed-test-artifacts\n+\t\t\tcp \"${TEST_EXIT%.exit}.out\" failed-test-artifacts/\n+\t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n+\t\t\tcontinue\n+\t\t\t;;\n \t\t*)\n \t\t\techo \"Unhandled CI type: $CI_TYPE\" >&2\n \t\t\texit 1\n-- \n2.42.0\n\n"},{"id":"484418","messageId":"CAP8UFD25O9D1WL+CAQOcdqwuPsgdd6qOMNWuVtAw18U3RccYZQ@mail.gmail.com","threadId":"60432","inReplyTo":"4a864a1d174f7d4d36202a375302d450fbe9f2a3.1698305961.git.ps@pks.im","subject":"Re: [PATCH 4/5] ci: split out logic to set up failed test artifacts","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-11-03T22:35:01Z","receivedAt":"2023-11-03T22:35:18Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Oct 26, 2023 at 10:01 AM Patrick Steinhardt wrote:\n>\n> We have some logic in place to create a directory with the output from\n> failed tests, which will then subsequently be uploaded as CI artifact.\n> We're about to add support for GitLab CI, which will want to reuse the\n> logic.\n>\n> Split the logic into a separate function so that it is reusable.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> ci/lib.sh | 40 ++++++++++++++++++++++------------------\n> 1 file changed, 22 insertions(+), 18 deletions(-)\n>\n> diff --git a/ci/lib.sh b/ci/lib.sh\n> index 957fd152d9c..33005854520 100755\n> --- a/ci/lib.sh\n> +++ b/ci/lib.sh\n> @@ -137,6 +137,27 @@ handle_failed_tests () {\n> return 1\n> }\n>\n> +create_failed_test_artifacts () {\n> + mkdir -p t/failed-test-artifacts\n> +\n> + for test_exit in t/test-results/*.exit\n> + do\n> + test 0 != \"$(cat \"$test_exit\")\" || continue\n> +\n> + test_name=\"${test_exit%.exit}\"\n> + test_name=\"${test_name##*/}\"\n> + printf \"\\\\e[33m\\\\e[1m=== Failed test: ${test_name} ===\\\\e[m\\\\n\"\n> + echo \"The full logs are in the 'print test failures' step below.\"\n> + echo \"See also the 'failed-tests-*' artifacts attached to this run.\"\n> + cat \"t/test-results/$test_name.markup\"\n> +\n> + trash_dir=\"t/trash directory.$test_name\"\n> + cp \"t/test-results/$test_name.out\" t/failed-test-artifacts/\n> + tar czf t/failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n> + done\n> + return 1\n\nNit: I think it's more the responsibility of handle_failed_tests() to\n`return 1` than of create_failed_test_artifacts(). I understand that\nputting it into this new function means one more line that is common\nto several CI tools though.\n\nOtherwise everything in this series LGTM. Thanks!\n"},{"id":"484451","messageId":"ZUiS0llx6juV_1M7@tanuki","threadId":"60432","inReplyTo":"CAP8UFD25O9D1WL+CAQOcdqwuPsgdd6qOMNWuVtAw18U3RccYZQ@mail.gmail.com","subject":"Re: [PATCH 4/5] ci: split out logic to set up failed test artifacts","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-06T07:16:50Z","receivedAt":"2023-11-06T07:17:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Nov 03, 2023 at 11:35:01PM +0100, Christian Couder wrote:\n> On Thu, Oct 26, 2023 at 10:01 AM Patrick Steinhardt wrote:\n> >\n> > We have some logic in place to create a directory with the output from\n> > failed tests, which will then subsequently be uploaded as CI artifact.\n> > We're about to add support for GitLab CI, which will want to reuse the\n> > logic.\n> >\n> > Split the logic into a separate function so that it is reusable.\n> >\n> > Signed-off-by: Patrick Steinhardt \n> > ---\n> > ci/lib.sh | 40 ++++++++++++++++++++++------------------\n> > 1 file changed, 22 insertions(+), 18 deletions(-)\n> >\n> > diff --git a/ci/lib.sh b/ci/lib.sh\n> > index 957fd152d9c..33005854520 100755\n> > --- a/ci/lib.sh\n> > +++ b/ci/lib.sh\n> > @@ -137,6 +137,27 @@ handle_failed_tests () {\n> > return 1\n> > }\n> >\n> > +create_failed_test_artifacts () {\n> > + mkdir -p t/failed-test-artifacts\n> > +\n> > + for test_exit in t/test-results/*.exit\n> > + do\n> > + test 0 != \"$(cat \"$test_exit\")\" || continue\n> > +\n> > + test_name=\"${test_exit%.exit}\"\n> > + test_name=\"${test_name##*/}\"\n> > + printf \"\\\\e[33m\\\\e[1m=== Failed test: ${test_name} ===\\\\e[m\\\\n\"\n> > + echo \"The full logs are in the 'print test failures' step below.\"\n> > + echo \"See also the 'failed-tests-*' artifacts attached to this run.\"\n> > + cat \"t/test-results/$test_name.markup\"\n> > +\n> > + trash_dir=\"t/trash directory.$test_name\"\n> > + cp \"t/test-results/$test_name.out\" t/failed-test-artifacts/\n> > + tar czf t/failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n> > + done\n> > + return 1\n> \n> Nit: I think it's more the responsibility of handle_failed_tests() to\n> `return 1` than of create_failed_test_artifacts(). I understand that\n> putting it into this new function means one more line that is common\n> to several CI tools though.\n> \n> Otherwise everything in this series LGTM. Thanks!\n\nGood point indeed. Not sure whether this is worth a reroll though?\n\nPatrick\n"},{"id":"484626","messageId":"cover.1699514143.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1698305961.git.ps@pks.im","subject":"[PATCH v6 0/8] ci: add GitLab CI definition","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-09T08:05:21Z","receivedAt":"2023-11-09T08:05:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the 6th version of my patch series to introduce support for\nGitLab CI into the Git project.\n\nThere's only a single change compared to v5 based on Chris' feedback,\nnamely to move around a `return 1`. The newly extracted helper function\n`create_failed_test_artifacts()` indeed wasn't the correct place to put\nthis error code.\n\nA test run of this pipeline can be found at [1].\n\nThanks!\n\nPatrick\n\n[1]: https://gitlab.com/gitlab-org/git/-/pipelines/1066250852\n\nPatrick Steinhardt (8):\n ci: reorder definitions for grouping functions\n ci: make grouping setup more generic\n ci: group installation of Docker dependencies\n ci: split out logic to set up failed test artifacts\n ci: unify setup of some environment variables\n ci: squelch warnings when testing with unusable Git repo\n ci: install test dependencies for linux-musl\n ci: add support for GitLab CI\n\n .gitlab-ci.yml | 53 +++++++++\n ci/install-docker-dependencies.sh | 23 +++-\n ci/lib.sh | 190 ++++++++++++++++++++++--------\n ci/print-test-failures.sh | 6 +\n t/lib-httpd.sh | 17 ++-\n 5 files changed, 234 insertions(+), 55 deletions(-)\n create mode 100644 .gitlab-ci.yml\n\nRange-diff against v5:\n1: 0ba396f2a33 = 1: a1413b76422 ci: reorder definitions for grouping functions\n2: 821cfcd6125 = 2: 29039d7aa3a ci: make grouping setup more generic\n3: 6e5bcf143c8 = 3: 414655ffb2d ci: group installation of Docker dependencies\n4: 2182acf5bfc ! 4: 96d710faec8 ci: split out logic to set up failed test artifacts\n @@ ci/lib.sh: handle_failed_tests () {\n +\t\tcp \"t/test-results/$test_name.out\" t/failed-test-artifacts/\n +\t\ttar czf t/failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n +\tdone\n -+\treturn 1\n +}\n +\n # GitHub Action doesn't set TERM, which is required by tput\n @@ ci/lib.sh: then\n -\t\t\tcp \"t/test-results/$test_name.out\" t/failed-test-artifacts/\n -\t\t\ttar czf t/failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n -\t\tdone\n --\t\treturn 1\n +\t\tcreate_failed_test_artifacts\n + \t\treturn 1\n \t}\n \n - \tcache_dir=\"$HOME/none\"\n5: 6078aea246d = 5: 486d4bbf8b0 ci: unify setup of some environment variables\n6: d69bde92f2f = 6: 534b14f0262 ci: squelch warnings when testing with unusable Git repo\n7: b911c005bae = 7: a060613f039 ci: install test dependencies for linux-musl\n8: 5784d03a6f1 ! 8: c05ff28cc2c ci: add support for GitLab CI\n @@ ci/lib.sh: then\n +\tDONT_SKIP_TAGS=t\n +\thandle_failed_tests () {\n +\t\tcreate_failed_test_artifacts\n ++\t\treturn 1\n +\t}\n +\n +\tcache_dir=\"$HOME/none\"\n\nbase-commit: dadef801b365989099a9929e995589e455c51fed\n-- \n2.42.0\n\n"},{"id":"484627","messageId":"a1413b76422b85eefaa23a5ee341d6e1e79e708a.1699514143.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1699514143.git.ps@pks.im","subject":"[PATCH v6 1/8] ci: reorder definitions for grouping functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-09T08:05:25Z","receivedAt":"2023-11-09T08:05:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We define a set of grouping functions that are used to group together\noutput in our CI, where these groups then end up as collapsible sections\nin the respective pipeline platform. The way these functions are defined\nis not easily extensible though as we have an up front check for the CI\n_not_ being GitHub Actions, where we define the non-stub logic in the\nelse branch.\n\nReorder the conditional branches such that we explicitly handle GitHub\nActions.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 20 ++++++++++----------\n 1 file changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex bc0b23099df..029819673b4 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -1,16 +1,7 @@\n # Library of functions shared by all CI scripts\n \n-if test true != \"$GITHUB_ACTIONS\"\n+if test true = \"$GITHUB_ACTIONS\"\n then\n-\tbegin_group () { :; }\n-\tend_group () { :; }\n-\n-\tgroup () {\n-\t\tshift\n-\t\t\"$@\"\n-\t}\n-\tset -x\n-else\n \tbegin_group () {\n \t\tneed_to_end_group=t\n \t\techo \"::group::$1\" >&2\n@@ -42,6 +33,15 @@ else\n \t}\n \n \tbegin_group \"CI setup\"\n+else\n+\tbegin_group () { :; }\n+\tend_group () { :; }\n+\n+\tgroup () {\n+\t\tshift\n+\t\t\"$@\"\n+\t}\n+\tset -x\n fi\n \n # Set 'exit on error' for all CI scripts to let the caller know that\n-- \n2.42.0\n\n"},{"id":"484628","messageId":"29039d7aa3ae65f92d3b32dcaf26e753d262869d.1699514143.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1699514143.git.ps@pks.im","subject":"[PATCH v6 2/8] ci: make grouping setup more generic","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-09T08:05:29Z","receivedAt":"2023-11-09T08:05:33Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Make the grouping setup more generic by always calling `begin_group ()`\nand `end_group ()` regardless of whether we have stubbed those functions\nor not. This ensures we can more readily add support for additional CI\nplatforms.\n\nFurthermore, the `group ()` function is made generic so that it is the\nsame for both GitHub Actions and for other platforms. There is a\nsemantic conflict here though: GitHub Actions used to call `set +x` in\n`group ()` whereas the non-GitHub case unconditionally uses `set -x`.\nThe latter would get overriden if we kept the `set +x` in the generic\nversion of `group ()`. To resolve this conflict, we simply drop the `set\n+x` in the generic variant of this function. As `begin_group ()` calls\n`set -x` anyway this is not much of a change though, as the only\ncommands that aren't printed anymore now are the ones between the\nbeginning of `group ()` and the end of `begin_group ()`.\n\nLast, this commit changes `end_group ()` to also accept a parameter that\nindicates _which_ group should end. This will be required by a later\ncommit that introduces support for GitLab CI.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 46 ++++++++++++++++++++++------------------------\n 1 file changed, 22 insertions(+), 24 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 029819673b4..2ee5abeb02d 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -14,36 +14,34 @@ then\n \t\tneed_to_end_group=\n \t\techo '::endgroup::' >&2\n \t}\n-\ttrap end_group EXIT\n-\n-\tgroup () {\n-\t\tset +x\n-\t\tbegin_group \"$1\"\n-\t\tshift\n-\t\t# work around `dash` not supporting `set -o pipefail`\n-\t\t(\n-\t\t\t\"$@\" 2>&1\n-\t\t\techo $? >exit.status\n-\t\t) |\n-\t\tsed 's/^\\(\\([^ ]*\\):\\([0-9]*\\):\\([0-9]*:\\) \\)\\(error\\|warning\\): /::\\5 file=\\2,line=\\3::\\1/'\n-\t\tres=$(cat exit.status)\n-\t\trm exit.status\n-\t\tend_group\n-\t\treturn $res\n-\t}\n-\n-\tbegin_group \"CI setup\"\n else\n \tbegin_group () { :; }\n \tend_group () { :; }\n \n-\tgroup () {\n-\t\tshift\n-\t\t\"$@\"\n-\t}\n \tset -x\n fi\n \n+group () {\n+\tgroup=\"$1\"\n+\tshift\n+\tbegin_group \"$group\"\n+\n+\t# work around `dash` not supporting `set -o pipefail`\n+\t(\n+\t\t\"$@\" 2>&1\n+\t\techo $? >exit.status\n+\t) |\n+\tsed 's/^\\(\\([^ ]*\\):\\([0-9]*\\):\\([0-9]*:\\) \\)\\(error\\|warning\\): /::\\5 file=\\2,line=\\3::\\1/'\n+\tres=$(cat exit.status)\n+\trm exit.status\n+\n+\tend_group \"$group\"\n+\treturn $res\n+}\n+\n+begin_group \"CI setup\"\n+trap \"end_group 'CI setup'\" EXIT\n+\n # Set 'exit on error' for all CI scripts to let the caller know that\n # something went wrong.\n #\n@@ -285,5 +283,5 @@ esac\n \n MAKEFLAGS=\"$MAKEFLAGS CC=${CC:-cc}\"\n \n-end_group\n+end_group \"CI setup\"\n set -x\n-- \n2.42.0\n\n"},{"id":"484629","messageId":"414655ffb2d9342515b26aaf197c797e8d93c646.1699514143.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1699514143.git.ps@pks.im","subject":"[PATCH v6 3/8] ci: group installation of Docker dependencies","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-09T08:05:34Z","receivedAt":"2023-11-09T08:05:38Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The output of CI jobs tends to be quite long-winded and hard to digest.\nTo help with this, many CI systems provide the ability to group output\ninto collapsible sections, and we're also doing this in some of our\nscripts.\n\nOne notable omission is the script to install Docker dependencies.\nAddress it to bring more structure to the output for Docker-based jobs.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-docker-dependencies.sh | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\nindex 78b7e326da6..d0bc19d3bb3 100755\n--- a/ci/install-docker-dependencies.sh\n+++ b/ci/install-docker-dependencies.sh\n@@ -3,6 +3,10 @@\n # Install dependencies required to build and test Git inside container\n #\n \n+. ${0%/*}/lib.sh\n+\n+begin_group \"Install dependencies\"\n+\n case \"$jobname\" in\n linux32)\n \tlinux32 --32bit i386 sh -c '\n@@ -20,3 +24,5 @@ pedantic)\n \tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n \t;;\n esac\n+\n+end_group \"Install dependencies\"\n-- \n2.42.0\n\n"},{"id":"484630","messageId":"96d710faec83daf965cbd7747c703864f4dadd70.1699514143.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1699514143.git.ps@pks.im","subject":"[PATCH v6 4/8] ci: split out logic to set up failed test artifacts","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-09T08:05:38Z","receivedAt":"2023-11-09T08:05:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have some logic in place to create a directory with the output from\nfailed tests, which will then subsequently be uploaded as CI artifacts.\nWe're about to add support for GitLab CI, which will want to reuse the\nlogic.\n\nSplit the logic into a separate function so that it is reusable.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 38 +++++++++++++++++++++-----------------\n 1 file changed, 21 insertions(+), 17 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 2ee5abeb02d..6fe3c08be83 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -131,6 +131,26 @@ handle_failed_tests () {\n \treturn 1\n }\n \n+create_failed_test_artifacts () {\n+\tmkdir -p t/failed-test-artifacts\n+\n+\tfor test_exit in t/test-results/*.exit\n+\tdo\n+\t\ttest 0 != \"$(cat \"$test_exit\")\" || continue\n+\n+\t\ttest_name=\"${test_exit%.exit}\"\n+\t\ttest_name=\"${test_name##*/}\"\n+\t\tprintf \"\\\\e[33m\\\\e[1m=== Failed test: ${test_name} ===\\\\e[m\\\\n\"\n+\t\techo \"The full logs are in the 'print test failures' step below.\"\n+\t\techo \"See also the 'failed-tests-*' artifacts attached to this run.\"\n+\t\tcat \"t/test-results/$test_name.markup\"\n+\n+\t\ttrash_dir=\"t/trash directory.$test_name\"\n+\t\tcp \"t/test-results/$test_name.out\" t/failed-test-artifacts/\n+\t\ttar czf t/failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n+\tdone\n+}\n+\n # GitHub Action doesn't set TERM, which is required by tput\n export TERM=${TERM:-dumb}\n \n@@ -171,24 +191,8 @@ then\n \tCC=\"${CC_PACKAGE:-${CC:-gcc}}\"\n \tDONT_SKIP_TAGS=t\n \thandle_failed_tests () {\n-\t\tmkdir -p t/failed-test-artifacts\n \t\techo \"FAILED_TEST_ARTIFACTS=t/failed-test-artifacts\" >>$GITHUB_ENV\n-\n-\t\tfor test_exit in t/test-results/*.exit\n-\t\tdo\n-\t\t\ttest 0 != \"$(cat \"$test_exit\")\" || continue\n-\n-\t\t\ttest_name=\"${test_exit%.exit}\"\n-\t\t\ttest_name=\"${test_name##*/}\"\n-\t\t\tprintf \"\\\\e[33m\\\\e[1m=== Failed test: ${test_name} ===\\\\e[m\\\\n\"\n-\t\t\techo \"The full logs are in the 'print test failures' step below.\"\n-\t\t\techo \"See also the 'failed-tests-*' artifacts attached to this run.\"\n-\t\t\tcat \"t/test-results/$test_name.markup\"\n-\n-\t\t\ttrash_dir=\"t/trash directory.$test_name\"\n-\t\t\tcp \"t/test-results/$test_name.out\" t/failed-test-artifacts/\n-\t\t\ttar czf t/failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n-\t\tdone\n+\t\tcreate_failed_test_artifacts\n \t\treturn 1\n \t}\n \n-- \n2.42.0\n\n"},{"id":"484631","messageId":"486d4bbf8b085d98326fd1fb1fe82d463ef7af7f.1699514143.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1699514143.git.ps@pks.im","subject":"[PATCH v6 5/8] ci: unify setup of some environment variables","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-09T08:05:42Z","receivedAt":"2023-11-09T08:05:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Both GitHub Actions and Azure Pipelines set up the environment variables\nGIT_TEST_OPTS, GIT_PROVE_OPTS and MAKEFLAGS. And while most values are\nactually the same, the setup is completely duplicate. With the upcoming\nsupport for GitLab CI this duplication would only extend even further.\n\nUnify the setup of those environment variables so that only the uncommon\nparts are separated. While at it, we also perform some additional small\nimprovements:\n\n - We now always pass `--state=failed,slow,save` via GIT_PROVE_OPTS.\n It doesn't hurt on platforms where we don't persist the state, so\n this further reduces boilerplate.\n\n - When running on Windows systems we set `--no-chain-lint` and\n `--no-bin-wrappers`. Interestingly though, we did so _after_\n already having exported the respective environment variables.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 27 +++++++++++++++++----------\n 1 file changed, 17 insertions(+), 10 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 6fe3c08be83..8357ad77e4f 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -174,11 +174,8 @@ then\n \t# among *all* phases)\n \tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n \n-\texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n-\texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n-\tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n-\ttest windows_nt != \"$CI_OS_NAME\" ||\n-\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n+\tGIT_TEST_OPTS=\"--write-junit-xml\"\n+\tJOBS=10\n elif test true = \"$GITHUB_ACTIONS\"\n then\n \tCI_TYPE=github-actions\n@@ -198,17 +195,27 @@ then\n \n \tcache_dir=\"$HOME/none\"\n \n-\texport GIT_PROVE_OPTS=\"--timer --jobs 10\"\n-\texport GIT_TEST_OPTS=\"--verbose-log -x --github-workflow-markup\"\n-\tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n-\ttest windows != \"$CI_OS_NAME\" ||\n-\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n+\tGIT_TEST_OPTS=\"--github-workflow-markup\"\n+\tJOBS=10\n else\n \techo \"Could not identify CI type\" >&2\n \tenv >&2\n \texit 1\n fi\n \n+MAKEFLAGS=\"$MAKEFLAGS --jobs=$JOBS\"\n+GIT_PROVE_OPTS=\"--timer --jobs $JOBS --state=failed,slow,save\"\n+\n+GIT_TEST_OPTS=\"$GIT_TEST_OPTS --verbose-log -x\"\n+case \"$CI_OS_NAME\" in\n+windows|windows_nt)\n+\tGIT_TEST_OPTS=\"$GIT_TEST_OPTS --no-chain-lint --no-bin-wrappers\"\n+\t;;\n+esac\n+\n+export GIT_TEST_OPTS\n+export GIT_PROVE_OPTS\n+\n good_trees_file=\"$cache_dir/good-trees\"\n \n mkdir -p \"$cache_dir\"\n-- \n2.42.0\n\n"},{"id":"484632","messageId":"534b14f02623273163f9d3950380e85d81c6b773.1699514143.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1699514143.git.ps@pks.im","subject":"[PATCH v6 6/8] ci: squelch warnings when testing with unusable Git repo","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-09T08:05:46Z","receivedAt":"2023-11-09T08:05:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Our CI jobs that run on Docker also use mostly the same architecture to\nbuild and test Git via the \"ci/run-build-and-tests.sh\" script. These\nscripts also provide some functionality to massage the Git repository\nwe're supposedly operating in.\n\nIn our Docker-based infrastructure we may not even have a Git repository\navailable though, which leads to warnings when those functions execute.\nMake the helpers exit gracefully in case either there is no Git in our\nPATH, or when not running in a Git repository.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 32 ++++++++++++++++++++++++++++++++\n 1 file changed, 32 insertions(+)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 8357ad77e4f..04997102308 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -69,10 +69,32 @@ skip_branch_tip_with_tag () {\n \tfi\n }\n \n+# Check whether we can use the path passed via the first argument as Git\n+# repository.\n+is_usable_git_repository () {\n+\t# We require Git in our PATH, otherwise we cannot access repositories\n+\t# at all.\n+\tif ! command -v git >/dev/null\n+\tthen\n+\t\treturn 1\n+\tfi\n+\n+\t# And the target directory needs to be a proper Git repository.\n+\tif ! git -C \"$1\" rev-parse 2>/dev/null\n+\tthen\n+\t\treturn 1\n+\tfi\n+}\n+\n # Save some info about the current commit's tree, so we can skip the build\n # job if we encounter the same tree again and can provide a useful info\n # message.\n save_good_tree () {\n+\tif ! is_usable_git_repository .\n+\tthen\n+\t\treturn\n+\tfi\n+\n \techo \"$(git rev-parse $CI_COMMIT^{tree}) $CI_COMMIT $CI_JOB_NUMBER $CI_JOB_ID\" >>\"$good_trees_file\"\n \t# limit the file size\n \ttail -1000 \"$good_trees_file\" >\"$good_trees_file\".tmp\n@@ -88,6 +110,11 @@ skip_good_tree () {\n \t\treturn\n \tfi\n \n+\tif ! is_usable_git_repository .\n+\tthen\n+\t\treturn\n+\tfi\n+\n \tif ! good_tree_info=\"$(grep \"^$(git rev-parse $CI_COMMIT^{tree}) \" \"$good_trees_file\")\"\n \tthen\n \t\t# Haven't seen this tree yet, or no cached good trees file yet.\n@@ -119,6 +146,11 @@ skip_good_tree () {\n }\n \n check_unignored_build_artifacts () {\n+\tif ! is_usable_git_repository .\n+\tthen\n+\t\treturn\n+\tfi\n+\n \t! git ls-files --other --exclude-standard --error-unmatch \\\n \t\t-- ':/*' 2>/dev/null ||\n \t{\n-- \n2.42.0\n\n"},{"id":"484633","messageId":"a060613f039f580206d59f0d42d8004350c1439e.1699514143.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1699514143.git.ps@pks.im","subject":"[PATCH v6 7/8] ci: install test dependencies for linux-musl","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-09T08:05:50Z","receivedAt":"2023-11-09T08:05:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The linux-musl CI job executes tests on Alpine Linux, which is based on\nmusl libc instead of glibc. We're missing some test dependencies though,\nwhich causes us to skip a subset of tests.\n\nInstall these test dependencies to increase our test coverage on this\nplatform. There are still some missing test dependecies, but these do\nnot have a corresponding package in the Alpine repositories:\n\n - p4 and p4d, both parts of the Perforce version control system.\n\n - cvsps, which generates patch sets for CVS.\n\n - Subversion and the SVN::Core Perl library, the latter of which is\n not available in the Alpine repositories. While the tool itself is\n available, all Subversion-related tests are skipped without the\n SVN::Core Perl library anyway.\n\nThe Apache2-based tests require a bit more care though. For one, the\nmodule path is different on Alpine Linux, which requires us to add it to\nthe list of known module paths to detect it. But second, the WebDAV\nmodule on Alpine Linux is broken because it does not bundle the default\ndatabase backend [1]. We thus need to skip the WebDAV-based tests on\nAlpine Linux for now.\n\n[1]: https://gitlab.alpinelinux.org/alpine/aports/-/issues/13112\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-docker-dependencies.sh | 4 +++-\n t/lib-httpd.sh | 17 ++++++++++++++++-\n 2 files changed, 19 insertions(+), 2 deletions(-)\n\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\nindex d0bc19d3bb3..6e845283680 100755\n--- a/ci/install-docker-dependencies.sh\n+++ b/ci/install-docker-dependencies.sh\n@@ -17,7 +17,9 @@ linux32)\n \t;;\n linux-musl)\n \tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n-\t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n+\t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n+\t\tapache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \\\n+\t\tbash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n \t;;\n pedantic)\n \tdnf -yq update >/dev/null &&\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex 5fe3c8ab69d..dbc99775934 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -67,7 +67,8 @@ for DEFAULT_HTTPD_MODULE_PATH in '/usr/libexec/apache2' \\\n \t\t\t\t '/usr/lib/apache2/modules' \\\n \t\t\t\t '/usr/lib64/httpd/modules' \\\n \t\t\t\t '/usr/lib/httpd/modules' \\\n-\t\t\t\t '/usr/libexec/httpd'\n+\t\t\t\t '/usr/libexec/httpd' \\\n+\t\t\t\t '/usr/lib/apache2'\n do\n \tif test -d \"$DEFAULT_HTTPD_MODULE_PATH\"\n \tthen\n@@ -127,6 +128,20 @@ else\n \t\t\"Could not identify web server at '$LIB_HTTPD_PATH'\"\n fi\n \n+if test -n \"$LIB_HTTPD_DAV\" && test -f /etc/os-release\n+then\n+\tcase \"$(grep \"^ID=\" /etc/os-release | cut -d= -f2-)\" in\n+\talpine)\n+\t\t# The WebDAV module in Alpine Linux is broken at least up to\n+\t\t# Alpine v3.16 as the default DBM driver is missing.\n+\t\t#\n+\t\t# https://gitlab.alpinelinux.org/alpine/aports/-/issues/13112\n+\t\ttest_skip_or_die GIT_TEST_HTTPD \\\n+\t\t\t\"Apache WebDAV module does not have default DBM backend driver\"\n+\t\t;;\n+\tesac\n+fi\n+\n install_script () {\n \twrite_script \"$HTTPD_ROOT_PATH/$1\" <\"$TEST_PATH/$1\"\n }\n-- \n2.42.0\n\n"},{"id":"484634","messageId":"c05ff28cc2cbba88088e46caa5d5ba40e583b576.1699514143.git.ps@pks.im","threadId":"60432","inReplyTo":"cover.1699514143.git.ps@pks.im","subject":"[PATCH v6 8/8] ci: add support for GitLab CI","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-11-09T08:05:54Z","receivedAt":"2023-11-09T08:05:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We already support Azure Pipelines and GitHub Workflows in the Git\nproject, but until now we do not have support for GitLab CI. While it is\narguably not in the interest of the Git project to maintain a ton of\ndifferent CI platforms, GitLab has recently ramped up its efforts and\ntries to contribute to the Git project more regularly.\n\nPart of a problem we hit at GitLab rather frequently is that our own,\ncustom CI setup we have is so different to the setup that the Git\nproject has. More esoteric jobs like \"linux-TEST-vars\" that also set a\ncouple of environment variables do not exist in GitLab's custom CI\nsetup, and maintaining them to keep up with what Git does feels like\nwasted time. The result is that we regularly send patch series upstream\nthat fail to compile or pass tests in GitHub Workflows. We would thus\nlike to integrate the GitLab CI configuration into the Git project to\nhelp us send better patch series upstream and thus reduce overhead for\nthe maintainer. Results of these pipeline runs will be made available\n(at least) in GitLab's mirror of the Git project at [1].\n\nThis commit introduces the integration into our regular CI scripts so\nthat most of the setup continues to be shared across all of the CI\nsolutions. Note that as the builds on GitLab CI run as unprivileged\nuser, we need to pull in both sudo and shadow packages to our Alpine\nbased job to set this up.\n\n[1]: https://gitlab.com/gitlab-org/git\n\nSigned-off-by: Patrick Steinhardt \n---\n .gitlab-ci.yml | 53 +++++++++++++++++++++++++++++++\n ci/install-docker-dependencies.sh | 13 +++++++-\n ci/lib.sh | 45 ++++++++++++++++++++++++++\n ci/print-test-failures.sh | 6 ++++\n 4 files changed, 116 insertions(+), 1 deletion(-)\n create mode 100644 .gitlab-ci.yml\n\ndiff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\nnew file mode 100644\nindex 00000000000..cd98bcb18aa\n--- /dev/null\n+++ b/.gitlab-ci.yml\n@@ -0,0 +1,53 @@\n+default:\n+ timeout: 2h\n+\n+workflow:\n+ rules:\n+ - if: $CI_PIPELINE_SOURCE == \"merge_request_event\"\n+ - if: $CI_COMMIT_TAG\n+ - if: $CI_COMMIT_REF_PROTECTED == \"true\"\n+\n+test:\n+ image: $image\n+ before_script:\n+ - ./ci/install-docker-dependencies.sh\n+ script:\n+ - useradd builder --create-home\n+ - chown -R builder \"${CI_PROJECT_DIR}\"\n+ - sudo --preserve-env --set-home --user=builder ./ci/run-build-and-tests.sh\n+ after_script:\n+ - |\n+ if test \"$CI_JOB_STATUS\" != 'success'\n+ then\n+ sudo --preserve-env --set-home --user=builder ./ci/print-test-failures.sh\n+ fi\n+ parallel:\n+ matrix:\n+ - jobname: linux-sha256\n+ image: ubuntu:latest\n+ CC: clang\n+ - jobname: linux-gcc\n+ image: ubuntu:20.04\n+ CC: gcc\n+ CC_PACKAGE: gcc-8\n+ - jobname: linux-TEST-vars\n+ image: ubuntu:20.04\n+ CC: gcc\n+ CC_PACKAGE: gcc-8\n+ - jobname: linux-gcc-default\n+ image: ubuntu:latest\n+ CC: gcc\n+ - jobname: linux-leaks\n+ image: ubuntu:latest\n+ CC: gcc\n+ - jobname: linux-asan-ubsan\n+ image: ubuntu:latest\n+ CC: clang\n+ - jobname: pedantic\n+ image: fedora:latest\n+ - jobname: linux-musl\n+ image: alpine:latest\n+ artifacts:\n+ paths:\n+ - t/failed-test-artifacts\n+ when: on_failure\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\nindex 6e845283680..48c43f0f907 100755\n--- a/ci/install-docker-dependencies.sh\n+++ b/ci/install-docker-dependencies.sh\n@@ -16,11 +16,22 @@ linux32)\n \t'\n \t;;\n linux-musl)\n-\tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n+\tapk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n \t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n \t\tapache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \\\n \t\tbash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n \t;;\n+linux-*)\n+\t# Required so that apt doesn't wait for user input on certain packages.\n+\texport DEBIAN_FRONTEND=noninteractive\n+\n+\tapt update -q &&\n+\tapt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev \\\n+\t\tlibcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev \\\n+\t\tperl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl \\\n+\t\tlibdbd-sqlite3-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}} \\\n+\t\tapache2 cvs cvsps gnupg libcgi-pm-perl subversion\n+\t;;\n pedantic)\n \tdnf -yq update >/dev/null &&\n \tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 04997102308..643e75d0577 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -14,6 +14,22 @@ then\n \t\tneed_to_end_group=\n \t\techo '::endgroup::' >&2\n \t}\n+elif test true = \"$GITLAB_CI\"\n+then\n+\tbegin_group () {\n+\t\tneed_to_end_group=t\n+\t\tprintf \"\\e[0Ksection_start:$(date +%s):$(echo \"$1\" | tr ' ' _)\\r\\e[0K$1\\n\"\n+\t\ttrap \"end_group '$1'\" EXIT\n+\t\tset -x\n+\t}\n+\n+\tend_group () {\n+\t\ttest -n \"$need_to_end_group\" || return 0\n+\t\tset +x\n+\t\tneed_to_end_group=\n+\t\tprintf \"\\e[0Ksection_end:$(date +%s):$(echo \"$1\" | tr ' ' _)\\r\\e[0K\\n\"\n+\t\ttrap - EXIT\n+\t}\n else\n \tbegin_group () { :; }\n \tend_group () { :; }\n@@ -229,6 +245,35 @@ then\n \n \tGIT_TEST_OPTS=\"--github-workflow-markup\"\n \tJOBS=10\n+elif test true = \"$GITLAB_CI\"\n+then\n+\tCI_TYPE=gitlab-ci\n+\tCI_BRANCH=\"$CI_COMMIT_REF_NAME\"\n+\tCI_COMMIT=\"$CI_COMMIT_SHA\"\n+\tcase \"$CI_JOB_IMAGE\" in\n+\tmacos-*)\n+\t\tCI_OS_NAME=osx;;\n+\talpine:*|fedora:*|ubuntu:*)\n+\t\tCI_OS_NAME=linux;;\n+\t*)\n+\t\techo \"Could not identify OS image\" >&2\n+\t\tenv >&2\n+\t\texit 1\n+\t\t;;\n+\tesac\n+\tCI_REPO_SLUG=\"$CI_PROJECT_PATH\"\n+\tCI_JOB_ID=\"$CI_JOB_ID\"\n+\tCC=\"${CC_PACKAGE:-${CC:-gcc}}\"\n+\tDONT_SKIP_TAGS=t\n+\thandle_failed_tests () {\n+\t\tcreate_failed_test_artifacts\n+\t\treturn 1\n+\t}\n+\n+\tcache_dir=\"$HOME/none\"\n+\n+\truns_on_pool=$(echo \"$CI_JOB_IMAGE\" | tr : -)\n+\tJOBS=$(nproc)\n else\n \techo \"Could not identify CI type\" >&2\n \tenv >&2\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex 57277eefcd0..c33ad4e3a22 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -51,6 +51,12 @@ do\n \t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n \t\t\tcontinue\n \t\t\t;;\n+\t\tgitlab-ci)\n+\t\t\tmkdir -p failed-test-artifacts\n+\t\t\tcp \"${TEST_EXIT%.exit}.out\" failed-test-artifacts/\n+\t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n+\t\t\tcontinue\n+\t\t\t;;\n \t\t*)\n \t\t\techo \"Unhandled CI type: $CI_TYPE\" >&2\n \t\t\texit 1\n-- \n2.42.0\n\n"},{"id":"484635","messageId":"xmqqjzqr1bs1.fsf@gitster.g","threadId":"60432","inReplyTo":"cover.1699514143.git.ps@pks.im","subject":"Re: [PATCH v6 0/8] ci: add GitLab CI definition","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-11-09T10:06:22Z","receivedAt":"2023-11-09T10:06:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> There's only a single change compared to v5 based on Chris' feedback,\n> namely to move around a `return 1`. The newly extracted helper function\n> `create_failed_test_artifacts()` indeed wasn't the correct place to put\n> this error code.\n\nThe caller of the helper always signals failure by returning with 1\nafter calling this helper, and if we later add new callers, it is\nvery plausible that the new callers also are calling the helper\nbecause they have noticed that the tests failed.\n\nIf the helper were not named \"create_failed_test_artifacts\" but were\ngiven a more appropriate name, say, \"fail_with_test_artifacts\", we\nwouldn't have said that the helper is not the right place to return\nwith a failure status. So, quite honestly, the difference since the\nprevious round is strictly a Meh to me.\n\nBut let's queue this version, because it is not making it worse ;-)\n\nThanks.\n"}]}