{"thread":{"id":"57451","subject":"[PATCH 01/25] CI: run \"set -ex\" early in ci/lib.sh","startedAt":"2022-02-21T14:46:53Z","lastAt":"2022-05-25T11:27:26Z","messageCount":260,"participants":["Ævar Arnfjörð Bjarmason","Johannes Schindelin","Phillip Wood","Junio C Hamano","SZEDER Gábor","Victoria Dye","Eric Sunshine","Carlo Arenas","Carlo Marcelo Arenas Belón"],"isPatch":true,"patchVersion":1,"patchTotal":25},"messages":[{"id":"448980","messageId":"patch-01.25-970849a227f-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"[PATCH 01/25] CI: run \"set -ex\" early in ci/lib.sh","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:13Z","receivedAt":"2022-02-21T14:46:53Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change ci/lib.sh to run \"set -ex\" before it does anything else, before\nthis we were omitting a few \"export\" commands in our trace output that\nhappened before we enabled tracing.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n ci/lib.sh | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex cbc2f8f1caa..57141d38a85 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -1,5 +1,11 @@\n # Library of functions shared by all CI scripts\n \n+# Set 'exit on error' for all CI scripts to let the caller know that\n+# something went wrong.\n+# Set tracing executed commands, primarily setting environment variables\n+# and installing dependencies.\n+set -ex\n+\n skip_branch_tip_with_tag () {\n \t# Sometimes, a branch is pushed at the same time the tag that points\n \t# at the same commit as the tip of the branch is pushed, and building\n@@ -85,12 +91,6 @@ export TERM=${TERM:-dumb}\n # Clear MAKEFLAGS that may come from the outside world.\n export MAKEFLAGS=\n \n-# Set 'exit on error' for all CI scripts to let the caller know that\n-# something went wrong.\n-# Set tracing executed commands, primarily setting environment variables\n-# and installing dependencies.\n-set -ex\n-\n if test -n \"$SYSTEM_COLLECTIONURI\" || test -n \"$SYSTEM_TASKDEFINITIONSURI\"\n then\n \tCI_TYPE=azure-pipelines\n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"448981","messageId":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":null,"subject":"[PATCH 00/25] CI: run \"make [test]\" directly, use $GITHUB_ENV","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:12Z","receivedAt":"2022-02-21T14:46:55Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"== Applying this\n\nMerging this has a trivial conflct with \"seen\"'s\njs/use-builtin-add-i. The\ns/GIT_TEST_ADD_I_USE_BUILTIN=1/GIT_TEST_ADD_I_USE_BUILTIN=0/g\" there\nneeds to be correspondingly changed (using \"setenv\", no \"export\") in\nci/lib.sh\n\n== Summary\n\nThis series makes the CI less of a special-case by getting rid of\nwrapper scripts like \"ci/run-build-and-tests.sh\" in favor of simply\nrunning \"make\" or \"make test\" directly.\n\nIt also has changes that make us do less special-ness in\nCI. E.g. there's now a \"make ci-static-analysis\" target that can be\nrun locally to do what \"ci/run-static-analysis.sh\" did before. The CI\njust invokes that new \"make\" target, and the\n\"ci/run-static-analysis.sh\" script has been \"git rm'd\".\n\nThe 12/25 here has a more detailed summary[1], but basically the\n\"ci/lib.sh\" is now a script run directly from an early CI step within\na given job. That script runs sets the environment variables needed in\n$GITHUB_ENV (what GitHub CI uses to carry forward the environment\nbetween steps).\n\n== Details\n\nThis series conflicts with and is a proposed alternative to Johannes's\nproposed changes to teach \"ci/lib.sh\" GitHub specific syntax to\n\"group\" certain parts of the output[2]. That series proposes to keep\nthe \"ci/run-build-and-tests.sh\" and similar scripts, but to teach them\nto emit markers indicating when the \"build\" part commences, then\n\"test\" etc.\n\nI think the approach offered here is better as elaborated on in\n12/25[1], especially because as Johannes notes[3] we only have one\nlevel of \"grouping\" within a \"step\" available to us. In his version\nusing it for \"build\" v.s. \"test\" precludes being able to group parts\nof that \"build\" or \"test\" output in the future.\n\nThe following compares CI output between the three with a\nchange-on-top which caused a compilation error (\"this\" is this series,\n\"js\" is Johannes's) when clicking on the \"linux-gcc\" failure:\n\n master: https://github.com/avar/git/runs/5274251909?check_suite_focus=true\n this: https://github.com/avar/git/runs/5274464670?check_suite_focus=true\n js: https://github.com/avar/git/runs/5272239403?check_suite_focus=true\n\n Here we went from 93 lines of output on \"master\" to 47 in \"this\"\n (107 in \"js\"). Note also how in \"this\" we can:\n\n - Expand the \"Run make\" to get the \"MAKEFLAGS\" and other variables\n used in the step. That's now a 3-line summary instead of the\n first 50 lines being the old \"ci/lib.sh\" \"set -x\" output.\n\n - Because \"make\" failed, we have an elided \"make test\" step that's\n not being run below. We can thus see what steps our failure caused\n us to skip.\n\n - Unlike the \"js\" version we'll show the compilation error by\n default (the \"js\" version is grouped and needs to be expanded),\n but our output is now brief enough that that's no longer\n surrounded by other irrelevant output.\n\n - Unlike the \"js\" version there is no \"CI setup\" group within each\n step. That work is in the earlier \"ci/lib.sh --build\" step\n instead, which sets the config for all subsequent steps.\n\nFor other output changes look at the difference between \"master\" and\n\"this\" at:\n\n master: https://github.com/git/git/actions/runs/1866786595\n this: https://github.com/avar/git/actions/runs/1876270588\n\nNot explicitly covered in the summary above is that various other\nparts of ci/* were doing the same sort of within-step setup, but now\ndo so via cross-step passing of variables via $GITHUB_ENV. E.g. on\n\"master\" the test slices for the Windows tests have a lot of verbosity\nbefore they get to running the test itself:\n\n https://github.com/git/git/runs/5254267914?check_suite_focus=true#step:5:56\n\nIn the \"js\" version the test output is hidden (\"grouped\"), but we've\ngot the same amount of verbosity by default:\n\n https://github.com/gitgitgadget/git/runs/4937491771?check_suite_focus=true#step:5:67\n\nWhereas in \"this\" version that verbosity is in the preceding \"select\ntests\" step, with the \"test\" step showing only the relevant end-state\nof the \"$T\" variable we'll use in the Makefile (hidden by default,\nexpanded in this link):\n\n https://github.com/avar/git/runs/5274558869?check_suite_focus=true#step:7:13\n\nThis series does not attempt to replace the use of the \"group\" output\nused for the \"ok\" or \"not ok\" portion of tests in Johannes's\nseries. When a test fails the output in this series (sans config\ndiscovery not being repeated, and now summarized in the $GITHUB_ENV\ndrop-down) is substantially the same as on \"master\".\n\nMy summary at [4] goes into other overlap & non-overlap between the\ntwo. I think using the \"group\" output for those purposes might be\nuseful, although I left some feedback on [5] with problems in the\ncurrent \"js\" implementation.\n\nI do think any such changes should follow behind this series, as doing\nthat sort of grouping once we can effectively free up an extra \"group\"\nlevel (by peeling those off into \"steps\", as is done here) would be\nmuch more useful.\n\n1. https://lore.kernel.org/git/patch-12.25-dfd823f2e7d-20220221T122605Z-avarab@gmail.com\n2. https://lore.kernel.org/git/pull.1117.git.1643050574.gitgitgadget@gmail.com/\n3. https://lore.kernel.org/git/9333ba781b8240f704e739b00d274f8c3d887e39.1643050574.git.gitgitgadget@gmail.com/\n4. https://lore.kernel.org/git/220127.86ilu5cdnf.gmgdl@evledraar.gmail.com/\n5. https://lore.kernel.org/git/220126.86sftbfjl4.gmgdl@evledraar.gmail.com/\n\nÆvar Arnfjörð Bjarmason (25):\n CI: run \"set -ex\" early in ci/lib.sh\n CI: make \"$jobname\" explicit, remove fallback\n CI: remove more dead Travis CI support\n CI: remove dead \"tree skipping\" code\n CI: remove unused Azure ci/* code\n CI: don't have \"git grep\" invoke a pager in tree content check\n CI: have \"static-analysis\" run a \"make ci-static-analysis\" target\n CI: have \"static-analysis\" run \"check-builtins\", not \"documentation\"\n CI: move p4 and git-lfs variables to ci/install-dependencies.sh\n CI: consistently use \"export\" in ci/lib.sh\n CI: export variables via a wrapper\n CI: remove \"run-build-and-tests.sh\", run \"make [test]\" directly\n CI: check ignored unignored build artifacts in \"win[+VS] build\" too\n CI: invoke \"make artifacts-tar\" directly in windows-build\n CI: split up and reduce \"ci/test-documentation.sh\"\n CI: combine ci/install{,-docker}-dependencies.sh\n CI: move \"env\" definitions into ci/lib.sh\n ci/run-test-slice.sh: replace shelling out with \"echo\"\n CI: pre-select test slice in Windows & VS tests\n CI: only invoke ci/lib.sh as \"steps\" in main.yml\n CI: narrow down variable definitions in --build and --test\n CI: add more variables to MAKEFLAGS, except under vs-build\n CI: stop over-setting the $CC variable\n CI: set PYTHON_PATH setting for osx-{clang,gcc} into \"$jobname\" case\n CI: don't use \"set -x\" in \"ci/lib.sh\" output\n\n .github/workflows/main.yml | 100 +++++---\n Makefile | 32 ++-\n ci/check-directional-formatting.bash | 2 +-\n ci/check-unignored-build-artifacts.sh | 20 ++\n ci/install-dependencies.sh | 53 ++++-\n ci/install-docker-dependencies.sh | 22 --\n ci/lib-ci-type.sh | 10 +\n ci/lib-tput.sh | 2 +\n ci/lib.sh | 315 ++++++++++++--------------\n ci/make-test-artifacts.sh | 12 -\n ci/mount-fileshare.sh | 25 --\n ci/print-test-failures.sh | 16 +-\n ci/run-build-and-tests.sh | 54 -----\n ci/run-docker-build.sh | 66 ------\n ci/run-docker.sh | 47 ----\n ci/run-static-analysis.sh | 32 ---\n ci/run-test-slice.sh | 17 --\n ci/select-test-slice.sh | 10 +\n ci/test-documentation.sh | 37 +--\n ci/util/extract-trash-dirs.sh | 50 ----\n 20 files changed, 345 insertions(+), 577 deletions(-)\n create mode 100755 ci/check-unignored-build-artifacts.sh\n delete mode 100755 ci/install-docker-dependencies.sh\n create mode 100644 ci/lib-ci-type.sh\n create mode 100644 ci/lib-tput.sh\n delete mode 100755 ci/make-test-artifacts.sh\n delete mode 100755 ci/mount-fileshare.sh\n delete mode 100755 ci/run-build-and-tests.sh\n delete mode 100755 ci/run-docker-build.sh\n delete mode 100755 ci/run-docker.sh\n delete mode 100755 ci/run-static-analysis.sh\n delete mode 100755 ci/run-test-slice.sh\n create mode 100755 ci/select-test-slice.sh\n delete mode 100755 ci/util/extract-trash-dirs.sh\n\n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"448982","messageId":"patch-02.25-eda7fb18064-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"[PATCH 02/25] CI: make \"$jobname\" explicit, remove fallback","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:14Z","receivedAt":"2022-02-21T14:46:58Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Remove the $jobname fallback last adjusted in b011fabd6e0 (ci/lib.sh:\nencapsulate Travis-specific things, 2019-01-27).\n\nThis makes all of our CI jobs have an explicit $jobname, and removes\nthe special-case of the Windows jobs implicitly setting one (that they\ndon't currently use). In subsequent commits we'll use this new\n$jobname explicitly.\n\nThis is a follow-up to my 25715419bf4 (CI: don't run \"make test\" twice\nin one job, 2021-11-23), since then all the jobs except the Windows\njobs have had an explicit $jobname.\n\nIn the case of the Windows jobs we were using a jobname of windows-gcc\nfor both, due to picking \"gcc\" as a default fallback for \"$CC\" in\nci/lib.sh.\n\nSince we'll need this to run ci/lib.sh it makes sense to move the\ncheck to the top of the file. Subsequent commits will add more such\nassertions to it.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .github/workflows/main.yml | 8 ++++++++\n ci/lib.sh | 12 +++++++-----\n 2 files changed, 15 insertions(+), 5 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex c35200defb9..309b9141249 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -79,6 +79,8 @@ jobs:\n \n windows-build:\n name: win build\n+ env:\n+ jobname: windows-build\n needs: ci-config\n if: needs.ci-config.outputs.enabled == 'yes'\n runs-on: windows-latest\n@@ -100,6 +102,8 @@ jobs:\n path: artifacts\n windows-test:\n name: win test\n+ env:\n+ jobname: windows-test\n runs-on: windows-latest\n needs: [windows-build]\n strategy:\n@@ -131,6 +135,8 @@ jobs:\n path: ${{env.FAILED_TEST_ARTIFACTS}}\n vs-build:\n name: win+VS build\n+ env:\n+ jobname: vs-build\n needs: ci-config\n if: needs.ci-config.outputs.enabled == 'yes'\n env:\n@@ -183,6 +189,8 @@ jobs:\n path: artifacts\n vs-test:\n name: win+VS test\n+ env:\n+ jobname: vs-test\n runs-on: windows-latest\n needs: vs-build\n strategy:\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 57141d38a85..d2a7c33f536 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -6,6 +6,13 @@\n # and installing dependencies.\n set -ex\n \n+# Starting assertions\n+if test -z \"$jobname\"\n+then\n+\techo \"must set a CI jobname\" >&2\n+\texit 1\n+fi\n+\n skip_branch_tip_with_tag () {\n \t# Sometimes, a branch is pushed at the same time the tag that points\n \t# at the same commit as the tip of the branch is pushed, and building\n@@ -146,11 +153,6 @@ test -n \"${DONT_SKIP_TAGS-}\" ||\n skip_branch_tip_with_tag\n skip_good_tree\n \n-if test -z \"$jobname\"\n-then\n-\tjobname=\"$CI_OS_NAME-$CC\"\n-fi\n-\n export DEVELOPER=1\n export DEFAULT_TEST_TARGET=prove\n export GIT_TEST_CLONE_2GB=true\n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"448983","messageId":"patch-03.25-3ee32815cf3-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"[PATCH 03/25] CI: remove more dead Travis CI support","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:15Z","receivedAt":"2022-02-21T14:47:09Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Remove code that I missed in 4a6e4b96026 (CI: remove Travis CI\nsupport, 2021-11-23). This code was only called from or used by the\nnow-removed .travis.yml, or needed by the Travis CI environment.\n\nFor the symlinking in ci/run-build-and-tests.sh: Back when\n3c93b829205 (travis-ci: build Git during the 'script' phase,\n2018-01-08) and 4b060a4d973 (ci: use a junction on Windows instead of\na symlink, 2019-01-27) adjusted this \"ln\" command, the Windows build\nwould use ci/run-build-and-tests.sh.\n\nAs seen in 889cacb6897 (ci: configure GitHub Actions for CI/PR,\n2020-04-11) the current windows build uses a different entry point\nunder the GitHub CI, which doesn't use this .prove caching. Namely\n\"ci/run-test-slice.sh\".\n\nWe can be certain that it's never used in \"ci/run-test-slice.sh\"\nbecause to have a \".prove\" file we'd need to use \"--state=save\", which\nwe only do in the dead Azure codepath in ci/lib.sh. If it were used it\nwould do the wrong thing, because the different test slices would each\ntry to clobber the same \"t/.prove\" file.\n\nIf a subsequent run then used the -\"-state=failed,slow,save\" it would\ndefeat the purpose of \"ci/run-test-slice.sh\", since all slices would\nthen run all tests. I.e. behavior of prove's \"--state\" options is to\nselect tests to run from the provided \"--state\" file, in addition to\nthose specified on the command-line.\n\nFor ci/run-docker{,-build}.sh: It was likewise last referenced in the\n.travis.yml removed in my 4a6e4b96026. The current \"dockerized\" run in\n\".github/workflows/main.yml\" calls the same entry points as the main\n\"regular\" job.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n ci/lib.sh | 10 ++----\n ci/print-test-failures.sh | 3 --\n ci/run-build-and-tests.sh | 5 ---\n ci/run-docker-build.sh | 66 -----------------------------------\n ci/run-docker.sh | 47 -------------------------\n ci/run-test-slice.sh | 5 ---\n ci/util/extract-trash-dirs.sh | 50 --------------------------\n 7 files changed, 2 insertions(+), 184 deletions(-)\n delete mode 100755 ci/run-docker-build.sh\n delete mode 100755 ci/run-docker.sh\n delete mode 100755 ci/util/extract-trash-dirs.sh\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex d2a7c33f536..c3c06d66862 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -106,9 +106,6 @@ then\n \tCI_COMMIT=\"$BUILD_SOURCEVERSION\"\n \tCI_JOB_ID=\"$BUILD_BUILDID\"\n \tCI_JOB_NUMBER=\"$BUILD_BUILDNUMBER\"\n-\tCI_OS_NAME=\"$(echo \"$AGENT_OS\" | tr A-Z a-z)\"\n-\ttest darwin != \"$CI_OS_NAME\" || CI_OS_NAME=osx\n-\tCI_REPO_SLUG=\"$(expr \"$BUILD_REPOSITORY_URI\" : '.*/\\([^/]*/[^/]*\\)$')\"\n \tCC=\"${CC:-gcc}\"\n \n \t# use a subdirectory of the cache dir (because the file share is shared\n@@ -118,16 +115,13 @@ then\n \texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n \texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n \tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n-\ttest windows_nt != \"$CI_OS_NAME\" ||\n+\ttest Windows_NT != \"$AGENT_OS\" ||\n \tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n elif test true = \"$GITHUB_ACTIONS\"\n then\n \tCI_TYPE=github-actions\n \tCI_BRANCH=\"$GITHUB_REF\"\n \tCI_COMMIT=\"$GITHUB_SHA\"\n-\tCI_OS_NAME=\"$(echo \"$RUNNER_OS\" | tr A-Z a-z)\"\n-\ttest macos != \"$CI_OS_NAME\" || CI_OS_NAME=osx\n-\tCI_REPO_SLUG=\"$GITHUB_REPOSITORY\"\n \tCI_JOB_ID=\"$GITHUB_RUN_ID\"\n \tCC=\"${CC:-gcc}\"\n \tDONT_SKIP_TAGS=t\n@@ -137,7 +131,7 @@ then\n \texport GIT_PROVE_OPTS=\"--timer --jobs 10\"\n \texport GIT_TEST_OPTS=\"--verbose-log -x\"\n \tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n-\ttest windows != \"$CI_OS_NAME\" ||\n+\ttest Windows != \"$RUNNER_OS\" ||\n \tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n else\n \techo \"Could not identify CI type\" >&2\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex 57277eefcd0..00fcf27b1b2 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -77,9 +77,6 @@ do\n \t\t\tfi\n \t\t\tcombined_trash_size=$new_combined_trash_size\n \n-\t\t\t# DO NOT modify these two 'echo'-ed strings below\n-\t\t\t# without updating 'ci/util/extract-trash-dirs.sh'\n-\t\t\t# as well.\n \t\t\techo \"$(tput setaf 1)Start of trash directory of '$test_name':$(tput sgr0)\"\n \t\t\tcat \"$trash_tgz_b64\"\n \t\t\techo \"$(tput setaf 1)End of trash directory of '$test_name'$(tput sgr0)\"\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 280dda7d285..0a1ec8c2bae 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -5,11 +5,6 @@\n \n . ${0%/*}/lib.sh\n \n-case \"$CI_OS_NAME\" in\n-windows*) cmd //c mklink //j t\\\\.prove \"$(cygpath -aw \"$cache_dir/.prove\")\";;\n-*) ln -s \"$cache_dir/.prove\" t/.prove;;\n-esac\n-\n export MAKE_TARGETS=\"all test\"\n \n case \"$jobname\" in\ndiff --git a/ci/run-docker-build.sh b/ci/run-docker-build.sh\ndeleted file mode 100755\nindex 6cd832efb9c..00000000000\n--- a/ci/run-docker-build.sh\n+++ /dev/null\n@@ -1,66 +0,0 @@\n-#!/bin/sh\n-#\n-# Build and test Git inside container\n-#\n-# Usage:\n-# run-docker-build.sh \n-#\n-\n-set -ex\n-\n-if test $# -ne 1 || test -z \"$1\"\n-then\n-\techo >&2 \"usage: run-docker-build.sh \"\n-\texit 1\n-fi\n-\n-case \"$jobname\" in\n-linux32)\n-\tswitch_cmd=\"linux32 --32bit i386\"\n-\t;;\n-linux-musl)\n-\tswitch_cmd=\n-\tuseradd () { adduser -D \"$@\"; }\n-\t;;\n-*)\n-\texit 1\n-\t;;\n-esac\n-\n-\"${0%/*}/install-docker-dependencies.sh\"\n-\n-# If this script runs inside a docker container, then all commands are\n-# usually executed as root. Consequently, the host user might not be\n-# able to access the test output files.\n-# If a non 0 host user id is given, then create a user \"ci\" with that\n-# user id to make everything accessible to the host user.\n-HOST_UID=$1\n-if test $HOST_UID -eq 0\n-then\n-\t# Just in case someone does want to run the test suite as root.\n-\tCI_USER=root\n-else\n-\tCI_USER=ci\n-\tif test \"$(id -u $CI_USER 2>/dev/null)\" = $HOST_UID\n-\tthen\n-\t\techo \"user '$CI_USER' already exists with the requested ID $HOST_UID\"\n-\telse\n-\t\tuseradd -u $HOST_UID $CI_USER\n-\tfi\n-fi\n-\n-# Build and test\n-command $switch_cmd su -m -l $CI_USER -c \"\n-\tset -ex\n-\texport DEVELOPER='$DEVELOPER'\n-\texport DEFAULT_TEST_TARGET='$DEFAULT_TEST_TARGET'\n-\texport GIT_PROVE_OPTS='$GIT_PROVE_OPTS'\n-\texport GIT_TEST_OPTS='$GIT_TEST_OPTS'\n-\texport GIT_TEST_CLONE_2GB='$GIT_TEST_CLONE_2GB'\n-\texport MAKEFLAGS='$MAKEFLAGS'\n-\texport cache_dir='$cache_dir'\n-\tcd /usr/src/git\n-\ttest -n '$cache_dir' && ln -s '$cache_dir/.prove' t/.prove\n-\tmake\n-\tmake test\n-\"\ndiff --git a/ci/run-docker.sh b/ci/run-docker.sh\ndeleted file mode 100755\nindex af89d1624a4..00000000000\n--- a/ci/run-docker.sh\n+++ /dev/null\n@@ -1,47 +0,0 @@\n-#!/bin/sh\n-#\n-# Download and run Docker image to build and test Git\n-#\n-\n-. ${0%/*}/lib.sh\n-\n-case \"$jobname\" in\n-linux32)\n-\tCI_CONTAINER=\"daald/ubuntu32:xenial\"\n-\t;;\n-linux-musl)\n-\tCI_CONTAINER=alpine\n-\t;;\n-*)\n-\texit 1\n-\t;;\n-esac\n-\n-docker pull \"$CI_CONTAINER\"\n-\n-# Use the following command to debug the docker build locally:\n-# must be 0 if podman is used as drop-in replacement for docker\n-# $ docker run -itv \"${PWD}:/usr/src/git\" --entrypoint /bin/sh \"$CI_CONTAINER\"\n-# root@container:/# export jobname=\n-# root@container:/# /usr/src/git/ci/run-docker-build.sh \n-\n-container_cache_dir=/tmp/container-cache\n-\n-docker run \\\n-\t--interactive \\\n-\t--env DEVELOPER \\\n-\t--env DEFAULT_TEST_TARGET \\\n-\t--env GIT_PROVE_OPTS \\\n-\t--env GIT_TEST_OPTS \\\n-\t--env GIT_TEST_CLONE_2GB \\\n-\t--env MAKEFLAGS \\\n-\t--env jobname \\\n-\t--env cache_dir=\"$container_cache_dir\" \\\n-\t--volume \"${PWD}:/usr/src/git\" \\\n-\t--volume \"$cache_dir:$container_cache_dir\" \\\n-\t\"$CI_CONTAINER\" \\\n-\t/usr/src/git/ci/run-docker-build.sh $(id -u $USER)\n-\n-check_unignored_build_artifacts\n-\n-save_good_tree\ndiff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh\nindex f8c2c3106a2..b9a682b4bcd 100755\n--- a/ci/run-test-slice.sh\n+++ b/ci/run-test-slice.sh\n@@ -5,11 +5,6 @@\n \n . ${0%/*}/lib.sh\n \n-case \"$CI_OS_NAME\" in\n-windows*) cmd //c mklink //j t\\\\.prove \"$(cygpath -aw \"$cache_dir/.prove\")\";;\n-*) ln -s \"$cache_dir/.prove\" t/.prove;;\n-esac\n-\n make --quiet -C t T=\"$(cd t &&\n \t./helper/test-tool path-utils slice-tests \"$1\" \"$2\" t[0-9]*.sh |\n \ttr '\\n' ' ')\"\ndiff --git a/ci/util/extract-trash-dirs.sh b/ci/util/extract-trash-dirs.sh\ndeleted file mode 100755\nindex 8e67bec21a2..00000000000\n--- a/ci/util/extract-trash-dirs.sh\n+++ /dev/null\n@@ -1,50 +0,0 @@\n-#!/bin/sh\n-\n-error () {\n-\techo >&2 \"error: $@\"\n-\texit 1\n-}\n-\n-find_embedded_trash () {\n-\twhile read -r line\n-\tdo\n-\t\tcase \"$line\" in\n-\t\t*Start\\ of\\ trash\\ directory\\ of\\ \\'t[0-9][0-9][0-9][0-9]-*\\':*)\n-\t\t\ttest_name=\"${line#*\\'}\"\n-\t\t\ttest_name=\"${test_name%\\'*}\"\n-\n-\t\t\treturn 0\n-\t\tesac\n-\tdone\n-\n-\treturn 1\n-}\n-\n-extract_embedded_trash () {\n-\twhile read -r line\n-\tdo\n-\t\tcase \"$line\" in\n-\t\t*End\\ of\\ trash\\ directory\\ of\\ \\'$test_name\\'*)\n-\t\t\treturn\n-\t\t\t;;\n-\t\t*)\n-\t\t\tprintf '%s\\n' \"$line\"\n-\t\t\t;;\n-\t\tesac\n-\tdone\n-\n-\terror \"unexpected end of input\"\n-}\n-\n-# Raw logs from Linux build jobs have CRLF line endings, while OSX\n-# build jobs mostly have CRCRLF, except an odd line every now and\n-# then that has CRCRCRLF. 'base64 -d' from 'coreutils' doesn't like\n-# CRs and complains about \"invalid input\", so remove all CRs at the\n-# end of lines.\n-sed -e 's/\\r*$//' | \\\n-while find_embedded_trash\n-do\n-\techo \"Extracting trash directory of '$test_name'\"\n-\n-\textract_embedded_trash |base64 -d |tar xzp\n-done\n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"448984","messageId":"patch-04.25-c81c1b3f504-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"[PATCH 04/25] CI: remove dead \"tree skipping\" code","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:16Z","receivedAt":"2022-02-21T14:47:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Remove the code related to skipping CI runs if the tree was\nsuccessfully tested before. With the GitHub CI this is done with the\n\"ci-config\" step, see e76eec35540 (ci: allow per-branch config for\nGitHub Actions, 2020-05-07) and 7d78d5fc1a9 (ci: skip GitHub workflow\nruns for already-tested commits/trees, 2020-10-08).\n\nThis code hasn't been used since 4a6e4b96026 (CI: remove Travis CI\nsupport, 2021-11-23), and before that for the retired Azure pipeline\nsupport removed in 6081d3898fe (ci: retire the Azure Pipelines\ndefinition, 2020-04-11).\n\nThis change is needed because this and subsequent commits are turning\n\"ci/lib.sh\" into a dumber library that'll only be tasked with setting\nvariables for CI jobs, or for specific steps within those jobs. Now we\ndon't need to worry about it potentially skipping the run on its own.\n\nThis change also removes a subtle potential logic error introduced in\n0e7696c64db (ci: disallow directional formatting, 2021-11-04). The\n\"ci/check-directional-formatting.bash\" script would have been made to\nrun after the \"save_good_tree\" in invoked in\n\"ci/run-static-analysis.sh\". So if e.g. the \"make coccicheck\" failed\nwe'd still mark the tree as good.\n\nI.e. the addition of \"ci/check-directional-formatting.bash\" didn't\ntake into account that the various users of the \"save_good_tree\"\nfunction in ci/*.sh made a hard assumption that they're the only\nci/*.sh script that's being run (and they use \"set -e\").\n\nThis is entirely academic since we weren't actually running this code,\nbut is something to be careful of if anyone ever needs to resurrect\nparts of this.\n\nFinally, the cache_dir=\"$HOME/none\" added in a3f2eec862b (ci/lib:\nallow running in GitHub Actions, 2020-04-08) for the GitHub CI can be\nremoved. It appears to have been added to appease the subsequent\n'mkdir -p \"$cache_dir\"', which wasn't needed in GitHub CI.\n\n1. https://lore.kernel.org/git/patch-1.1-eec0a8c3164-20211217T000418Z-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n ci/lib.sh | 91 ---------------------------------------\n ci/run-build-and-tests.sh | 2 -\n ci/run-static-analysis.sh | 2 -\n ci/test-documentation.sh | 2 -\n 4 files changed, 97 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex c3c06d66862..5a8938d4b3c 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -13,75 +13,6 @@ then\n \texit 1\n fi\n \n-skip_branch_tip_with_tag () {\n-\t# Sometimes, a branch is pushed at the same time the tag that points\n-\t# at the same commit as the tip of the branch is pushed, and building\n-\t# both at the same time is a waste.\n-\t#\n-\t# When the build is triggered by a push to a tag, $CI_BRANCH will\n-\t# have that tagname, e.g. v2.14.0. Let's see if $CI_BRANCH is\n-\t# exactly at a tag, and if so, if it is different from $CI_BRANCH.\n-\t# That way, we can tell if we are building the tip of a branch that\n-\t# is tagged and we can skip the build because we won't be skipping a\n-\t# build of a tag.\n-\n-\tif TAG=$(git describe --exact-match \"$CI_BRANCH\" 2>/dev/null) &&\n-\t\ttest \"$TAG\" != \"$CI_BRANCH\"\n-\tthen\n-\t\techo \"$(tput setaf 2)Tip of $CI_BRANCH is exactly at $TAG$(tput sgr0)\"\n-\t\texit 0\n-\tfi\n-}\n-\n-# Save some info about the current commit's tree, so we can skip the build\n-# job if we encounter the same tree again and can provide a useful info\n-# message.\n-save_good_tree () {\n-\techo \"$(git rev-parse $CI_COMMIT^{tree}) $CI_COMMIT $CI_JOB_NUMBER $CI_JOB_ID\" >>\"$good_trees_file\"\n-\t# limit the file size\n-\ttail -1000 \"$good_trees_file\" >\"$good_trees_file\".tmp\n-\tmv \"$good_trees_file\".tmp \"$good_trees_file\"\n-}\n-\n-# Skip the build job if the same tree has already been built and tested\n-# successfully before (e.g. because the branch got rebased, changing only\n-# the commit messages).\n-skip_good_tree () {\n-\tif test true = \"$GITHUB_ACTIONS\"\n-\tthen\n-\t\treturn\n-\tfi\n-\n-\tif ! good_tree_info=\"$(grep \"^$(git rev-parse $CI_COMMIT^{tree}) \" \"$good_trees_file\")\"\n-\tthen\n-\t\t# Haven't seen this tree yet, or no cached good trees file yet.\n-\t\t# Continue the build job.\n-\t\treturn\n-\tfi\n-\n-\techo \"$good_tree_info\" | {\n-\t\tread tree prev_good_commit prev_good_job_number prev_good_job_id\n-\n-\t\tif test \"$CI_JOB_ID\" = \"$prev_good_job_id\"\n-\t\tthen\n-\t\t\tcat <<-EOF\n-\t\t\t$(tput setaf 2)Skipping build job for commit $CI_COMMIT.$(tput sgr0)\n-\t\t\tThis commit has already been built and tested successfully by this build job.\n-\t\t\tTo force a re-build delete the branch's cache and then hit 'Restart job'.\n-\t\t\tEOF\n-\t\telse\n-\t\t\tcat <<-EOF\n-\t\t\t$(tput setaf 2)Skipping build job for commit $CI_COMMIT.$(tput sgr0)\n-\t\t\tThis commit's tree has already been built and tested successfully in build job $prev_good_job_number for commit $prev_good_commit.\n-\t\t\tThe log of that build job is available at $SYSTEM_TASKDEFINITIONSURI$SYSTEM_TEAMPROJECT/_build/results?buildId=$prev_good_job_id\n-\t\t\tTo force a re-build delete the branch's cache and then hit 'Restart job'.\n-\t\t\tEOF\n-\t\tfi\n-\t}\n-\n-\texit 0\n-}\n-\n check_unignored_build_artifacts ()\n {\n \t! git ls-files --other --exclude-standard --error-unmatch \\\n@@ -102,16 +33,8 @@ if test -n \"$SYSTEM_COLLECTIONURI\" || test -n \"$SYSTEM_TASKDEFINITIONSURI\"\n then\n \tCI_TYPE=azure-pipelines\n \t# We are running in Azure Pipelines\n-\tCI_BRANCH=\"$BUILD_SOURCEBRANCH\"\n-\tCI_COMMIT=\"$BUILD_SOURCEVERSION\"\n-\tCI_JOB_ID=\"$BUILD_BUILDID\"\n-\tCI_JOB_NUMBER=\"$BUILD_BUILDNUMBER\"\n \tCC=\"${CC:-gcc}\"\n \n-\t# use a subdirectory of the cache dir (because the file share is shared\n-\t# among *all* phases)\n-\tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n-\n \texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n \texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n \tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n@@ -120,13 +43,7 @@ then\n elif test true = \"$GITHUB_ACTIONS\"\n then\n \tCI_TYPE=github-actions\n-\tCI_BRANCH=\"$GITHUB_REF\"\n-\tCI_COMMIT=\"$GITHUB_SHA\"\n-\tCI_JOB_ID=\"$GITHUB_RUN_ID\"\n \tCC=\"${CC:-gcc}\"\n-\tDONT_SKIP_TAGS=t\n-\n-\tcache_dir=\"$HOME/none\"\n \n \texport GIT_PROVE_OPTS=\"--timer --jobs 10\"\n \texport GIT_TEST_OPTS=\"--verbose-log -x\"\n@@ -139,14 +56,6 @@ else\n \texit 1\n fi\n \n-good_trees_file=\"$cache_dir/good-trees\"\n-\n-mkdir -p \"$cache_dir\"\n-\n-test -n \"${DONT_SKIP_TAGS-}\" ||\n-skip_branch_tip_with_tag\n-skip_good_tree\n-\n export DEVELOPER=1\n export DEFAULT_TEST_TARGET=prove\n export GIT_TEST_CLONE_2GB=true\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 0a1ec8c2bae..e5a21985b44 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -45,5 +45,3 @@ esac\n # start running tests.\n make $MAKE_TARGETS\n check_unignored_build_artifacts\n-\n-save_good_tree\ndiff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh\nindex 65bcebda41a..5a87b1cac96 100755\n--- a/ci/run-static-analysis.sh\n+++ b/ci/run-static-analysis.sh\n@@ -28,5 +28,3 @@ fi\n \n make hdr-check ||\n exit 1\n-\n-save_good_tree\ndiff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\nindex de41888430a..9e0652c30dd 100755\n--- a/ci/test-documentation.sh\n+++ b/ci/test-documentation.sh\n@@ -41,5 +41,3 @@ grep '\n---\n Makefile | 3 ++-\n ci/test-documentation.sh | 1 -\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 02cd563e216..0c117d32f48 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -3424,7 +3424,7 @@ check-docs::\n ### Make sure built-ins do not have dups and listed in git.c\n #\n check-builtins::\n-\t./check-builtins.sh\n+\t$(QUIET_CHECK)./check-builtins.sh\n \n ### Test suite coverage testing\n #\n@@ -3514,5 +3514,6 @@ ci-check-directional-formatting:\n \n .PHONY: ci-static-analysis\n ci-static-analysis: ci-check-directional-formatting\n+ci-static-analysis: check-builtins\n ci-static-analysis: check-coccicheck\n ci-static-analysis: hdr-check\ndiff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\nindex 9e0652c30dd..41e2b126311 100755\n--- a/ci/test-documentation.sh\n+++ b/ci/test-documentation.sh\n@@ -14,7 +14,6 @@ filter_log () {\n \t \"$1\"\n }\n \n-make check-builtins\n make check-docs\n \n # Build docs with AsciiDoc\n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"448986","messageId":"patch-06.25-59e4f41e86c-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"[PATCH 06/25] CI: don't have \"git grep\" invoke a pager in tree content check","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:18Z","receivedAt":"2022-02-21T14:47:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Fix the omission of not adding \"-P\" (a.k.a. \"--no-pager\") to the \"git\"\ninvocation that spaws \"git grep\", added in 0e7696c64db (ci: disallow\ndirectional formatting, 2021-11-04).\n\nThis didn't matter for the GitHub CI, but does when running the check\nlocally. A subsequent commit will start invoking it via a new check\nfrom the top-level Makefile, and without this we'd visibly spawn a\npager when doing so.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n ci/check-directional-formatting.bash | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/ci/check-directional-formatting.bash b/ci/check-directional-formatting.bash\nindex e6211b141aa..de8e6e486f0 100755\n--- a/ci/check-directional-formatting.bash\n+++ b/ci/check-directional-formatting.bash\n@@ -23,5 +23,5 @@\n # U+2066..U+2069: LRI, RLI, FSI and PDI\n regex='(\\u202a|\\u202b|\\u202c|\\u202d|\\u202e|\\u2066|\\u2067|\\u2068|\\u2069)'\n \n-! LC_CTYPE=C git grep -El \"$(LC_CTYPE=C.UTF-8 printf \"$regex\")\" \\\n+! LC_CTYPE=C git -P grep -El \"$(LC_CTYPE=C.UTF-8 printf \"$regex\")\" \\\n \t-- ':(exclude,attr:binary)' ':(exclude)*.po'\n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"448987","messageId":"patch-07.25-836ef20fdcc-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"[PATCH 07/25] CI: have \"static-analysis\" run a \"make ci-static-analysis\" target","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:19Z","receivedAt":"2022-02-21T14:47:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Make it easier to run the CI logic locally by creating\n\"ci-static-analysis\" target.\n\nNow the \"check-coccicheck\" logic added in 0860a7641ba (travis-ci: fail\nif Coccinelle static analysis found something to transform,\n2018-07-23) can be easily tested outside of CI, and likewise the\nsanity check added in 0e7696c64db (ci: disallow directional\nformatting, 2021-11-04).\n\nThese targets could be improved, the \"hdr-check\" target will re-do all\nof its work every time it's run, and could with a change similar to my\nc234e8a0ecf (Makefile: make the \"sparse\" target non-.PHONY,\n2021-09-23). The same goes for the new\n\"ci-check-directional-formatting\" target.\n\nBut without those improvements being able to easily run a 1=1\nequivalent of the checks we do in CI during a local build is already\nan improvement. Subsequent commits will expand on this ability.\n\nThis change will also make the CI check faster, since we can take\nadvantage of parallelism. The \"make coccicheck\" takes a very long\ntime, and at the tail end we might only have one job pegging 100% of\none CPU core for a long time. Now any extra cores will be free to run\nthe rest of the targets under \"ci-static-analysis\".\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .github/workflows/main.yml | 3 +--\n Makefile | 29 +++++++++++++++++++++++++++++\n ci/run-static-analysis.sh | 30 ------------------------------\n 3 files changed, 30 insertions(+), 32 deletions(-)\n delete mode 100755 ci/run-static-analysis.sh\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 309b9141249..ad752010102 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -317,8 +317,7 @@ jobs:\n steps:\n - uses: actions/checkout@v2\n - run: ci/install-dependencies.sh\n- - run: ci/run-static-analysis.sh\n- - run: ci/check-directional-formatting.bash\n+ - run: make ci-static-analysis\n sparse:\n needs: ci-config\n if: needs.ci-config.outputs.enabled == 'yes'\ndiff --git a/Makefile b/Makefile\nindex 6f0b4b775fe..02cd563e216 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2005,6 +2005,7 @@ ifndef V\n \tQUIET_HDR = @echo ' ' HDR $(<:hcc=h);\n \tQUIET_RC = @echo ' ' RC $@;\n \tQUIET_SPATCH = @echo ' ' SPATCH $<;\n+\tQUIET_CHECK = @echo ' ' CHECK $@;\n \tQUIET_SUBDIR0 = +@subdir=\n \tQUIET_SUBDIR1 = ;$(NO_SUBDIR) echo ' ' SUBDIR $$subdir; \\\n \t\t\t $(MAKE) $(PRINT_DIR) -C $$subdir\n@@ -3052,6 +3053,20 @@ coccicheck: $(addsuffix .patch,$(filter-out %.pending.cocci,$(wildcard contrib/c\n # See contrib/coccinelle/README\n coccicheck-pending: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.pending.cocci))\n \n+.PHONY: check-coccicheck\n+check-coccicheck: coccicheck\n+\t$(QUIET_CHECK)for cocci_patch in contrib/coccinelle/*.patch; do \\\n+\t\tif test -s \"$$cocci_patch\"; then \\\n+\t\t\techo \"Coccinelle suggests the following changes in '$$cocci_patch':\"; \\\n+\t\t\tcat \"$$cocci_patch\"; \\\n+\t\t\tfail=UnfortunatelyYes; \\\n+\t\tfi \\\n+\tdone; \\\n+\tif test -n \"$$fail\"; then \\\n+\t\techo \"error: Coccinelle suggested some changes\"; \\\n+\t\texit 1; \\\n+\tfi\n+\n .PHONY: coccicheck coccicheck-pending\n \n ### Installation rules\n@@ -3487,3 +3502,17 @@ $(FUZZ_PROGRAMS): all\n \t\t$(XDIFF_OBJS) $(EXTLIBS) git.o $@.o $(LIB_FUZZING_ENGINE) -o $@\n \n fuzz-all: $(FUZZ_PROGRAMS)\n+\n+### CI \"check\" targets\n+#\n+# These targets are run from the CI, see .github/workflows/main.yml,\n+# but can also be run manually to run the same assertions locally.\n+\n+.PHONY: ci-check-directional-formatting\n+ci-check-directional-formatting:\n+\t$(QUIET_CHECK)ci/check-directional-formatting.bash\n+\n+.PHONY: ci-static-analysis\n+ci-static-analysis: ci-check-directional-formatting\n+ci-static-analysis: check-coccicheck\n+ci-static-analysis: hdr-check\ndiff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh\ndeleted file mode 100755\nindex 5a87b1cac96..00000000000\n--- a/ci/run-static-analysis.sh\n+++ /dev/null\n@@ -1,30 +0,0 @@\n-#!/bin/sh\n-#\n-# Perform various static code analysis checks\n-#\n-\n-. ${0%/*}/lib.sh\n-\n-make coccicheck\n-\n-set +x\n-\n-fail=\n-for cocci_patch in contrib/coccinelle/*.patch\n-do\n-\tif test -s \"$cocci_patch\"\n-\tthen\n-\t\techo \"$(tput setaf 1)Coccinelle suggests the following changes in '$cocci_patch':$(tput sgr0)\"\n-\t\tcat \"$cocci_patch\"\n-\t\tfail=UnfortunatelyYes\n-\tfi\n-done\n-\n-if test -n \"$fail\"\n-then\n-\techo \"$(tput setaf 1)error: Coccinelle suggested some changes$(tput sgr0)\"\n-\texit 1\n-fi\n-\n-make hdr-check ||\n-exit 1\n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"448988","messageId":"patch-05.25-4738a22a36d-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"[PATCH 05/25] CI: remove unused Azure ci/* code","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:17Z","receivedAt":"2022-02-21T14:47:28Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Remove Azure-specific code that's been unused since 6081d3898fe (ci:\nretire the Azure Pipelines definition, 2020-04-11). As noted in a\nlarger removal of all of the Azure-supporting code in [1] (although\nthat missed some of this) there is more of it in-tree, but let's focus\non only the ci/* code for now.\n\nThis is needed because in subsequent commits this unused code would\neither need to be changed to accommodate changes in ci/*, or be\nremoved.\n\nAs we'll see in those subsequent commits the end-state we're heading\ntowards will actually make it easier to add new CI types in the\nfuture, even though the only one we're left with now is the GitHub\nCI. I.e. the \"ci/*\" framework will be made to do less, not more. We'll\nbe offloading more of what it does to our generic build and test\nsystem.\n\nWhile I'm at it (since the line needs to be touched anyway) change an\nodd 'if test true == \"$GITHUB_ACTIONS\"' to the more usual style used\nin other places of 'if test \"$GITHUB_ACTIONS\" = \"true\"'.\n\n1. https://lore.kernel.org/git/patch-1.1-eec0a8c3164-20211217T000418Z-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n ci/install-dependencies.sh | 3 ---\n ci/lib.sh | 13 +------------\n ci/mount-fileshare.sh | 25 -------------------------\n ci/print-test-failures.sh | 7 +------\n 4 files changed, 2 insertions(+), 46 deletions(-)\n delete mode 100755 ci/mount-fileshare.sh\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex dbcebad2fb2..e7ea8799411 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -34,8 +34,6 @@ macos-latest)\n \texport HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1\n \t# Uncomment this if you want to run perf tests:\n \t# brew install gnu-time\n-\ttest -z \"$BREW_INSTALL_PACKAGES\" ||\n-\tbrew install $BREW_INSTALL_PACKAGES\n \tbrew link --force gettext\n \tbrew install --cask --no-quarantine perforce || {\n \t\t# Update the definitions and try again\n@@ -69,7 +67,6 @@ Documentation)\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install asciidoc xmlto docbook-xsl-ns make\n \n-\ttest -n \"$ALREADY_HAVE_ASCIIDOCTOR\" ||\n \tsudo gem install --version 1.5.8 asciidoctor\n \t;;\n linux-gcc-default)\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 5a8938d4b3c..ad1aa59a384 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -29,18 +29,7 @@ export TERM=${TERM:-dumb}\n # Clear MAKEFLAGS that may come from the outside world.\n export MAKEFLAGS=\n \n-if test -n \"$SYSTEM_COLLECTIONURI\" || test -n \"$SYSTEM_TASKDEFINITIONSURI\"\n-then\n-\tCI_TYPE=azure-pipelines\n-\t# We are running in Azure Pipelines\n-\tCC=\"${CC:-gcc}\"\n-\n-\texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n-\texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n-\tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n-\ttest Windows_NT != \"$AGENT_OS\" ||\n-\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n-elif test true = \"$GITHUB_ACTIONS\"\n+if test \"$GITHUB_ACTIONS\" = \"true\"\n then\n \tCI_TYPE=github-actions\n \tCC=\"${CC:-gcc}\"\ndiff --git a/ci/mount-fileshare.sh b/ci/mount-fileshare.sh\ndeleted file mode 100755\nindex 26b58a80960..00000000000\n--- a/ci/mount-fileshare.sh\n+++ /dev/null\n@@ -1,25 +0,0 @@\n-#!/bin/sh\n-\n-die () {\n-\techo \"$*\" >&2\n-\texit 1\n-}\n-\n-test $# = 4 ||\n-die \"Usage: $0 \"\n-\n-mkdir -p \"$4\" || die \"Could not create $4\"\n-\n-case \"$(uname -s)\" in\n-Linux)\n-\tsudo mount -t cifs -o vers=3.0,username=\"$2\",password=\"$3\",dir_mode=0777,file_mode=0777,serverino \"$1\" \"$4\"\n-\t;;\n-Darwin)\n-\tpass=\"$(echo \"$3\" | sed -e 's/\\//%2F/g' -e 's/+/%2B/g')\" &&\n-\tmount -t smbfs,soft \"smb://$2:$pass@${1#//}\" \"$4\"\n-\t;;\n-*)\n-\tdie \"No support for $(uname -s)\"\n-\t;;\n-esac ||\n-die \"Could not mount $4\"\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex 00fcf27b1b2..0c63b6f7962 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -38,14 +38,9 @@ do\n \t\ttest_name=\"${TEST_EXIT%.exit}\"\n \t\ttest_name=\"${test_name##*/}\"\n \t\ttrash_dir=\"trash directory.$test_name\"\n+\t\tmkdir -p failed-test-artifacts\n \t\tcase \"$CI_TYPE\" in\n-\t\tazure-pipelines)\n-\t\t\tmkdir -p failed-test-artifacts\n-\t\t\tmv \"$trash_dir\" failed-test-artifacts\n-\t\t\tcontinue\n-\t\t\t;;\n \t\tgithub-actions)\n-\t\t\tmkdir -p failed-test-artifacts\n \t\t\techo \"FAILED_TEST_ARTIFACTS=t/failed-test-artifacts\" >>$GITHUB_ENV\n \t\t\tcp \"${TEST_EXIT%.exit}.out\" failed-test-artifacts/\n \t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"448989","messageId":"patch-10.25-b6a61a786c5-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"[PATCH 10/25] CI: consistently use \"export\" in ci/lib.sh","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:22Z","receivedAt":"2022-02-21T14:47:34Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the \"ci/lib.sh\" script co consistently use \"export\", for\ne.g. MAKEFLAGS we were exporting it, and then assigning to it, let's\ndo it the other way around.\n\nRight now this doesn't matter, since we in\ne.g. \"ci/install-dependencies.sh\" source this file, and don't use\nsomething like \"env(1)\" to retrieve these variables. But in a\nsubsequent commit we'll \"export\" these variables through a wrapper (to\nadditionally write them to a GitHub CI-specific $GITHUB_ENV\nfile). This change makes that subsequent change easier to read, as it\nwon't need to do any control flow refactoring.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n ci/lib.sh | 8 +++++---\n 1 file changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 6b37b10d750..31fe3a476a9 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -27,7 +27,7 @@ check_unignored_build_artifacts ()\n export TERM=${TERM:-dumb}\n \n # Clear MAKEFLAGS that may come from the outside world.\n-export MAKEFLAGS=\n+MAKEFLAGS=\n \n if test \"$GITHUB_ACTIONS\" = \"true\"\n then\n@@ -35,10 +35,12 @@ then\n \tCC=\"${CC:-gcc}\"\n \n \texport GIT_PROVE_OPTS=\"--timer --jobs 10\"\n-\texport GIT_TEST_OPTS=\"--verbose-log -x\"\n+\tGIT_TEST_OPTS=\"--verbose-log -x\"\n \tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n \ttest Windows != \"$RUNNER_OS\" ||\n \tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n+\n+\texport GIT_TEST_OPTS\n else\n \techo \"Could not identify CI type\" >&2\n \tenv >&2\n@@ -92,4 +94,4 @@ linux-leaks)\n \t;;\n esac\n \n-MAKEFLAGS=\"$MAKEFLAGS CC=${CC:-cc}\"\n+export MAKEFLAGS=\"$MAKEFLAGS CC=${CC:-cc}\"\n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"448990","messageId":"patch-09.25-a1d0796259e-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"[PATCH 09/25] CI: move p4 and git-lfs variables to ci/install-dependencies.sh","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:21Z","receivedAt":"2022-02-21T14:47:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the declaration of variables that are only used by the\n\"ubuntu-latest\" block in \"ci/install-dependencies.sh\" there from\n\"ci/lib.sh\".\n\nThis makes the code easier to follow, and changes \"ci/lib.sh\" to a\nlibrary that only exports CI variables for general use, we didn't need\nto export these $P4_PATH and $GIT_LFS_PATH variables.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n ci/install-dependencies.sh | 17 +++++++++++++++--\n ci/lib.sh | 12 ------------\n 2 files changed, 15 insertions(+), 14 deletions(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex e7ea8799411..34b4400f38b 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -5,14 +5,27 @@\n \n . ${0%/*}/lib.sh\n \n-P4WHENCE=http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION\n-LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\n UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl\n libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n case \"$runs_on_pool\" in\n ubuntu-latest)\n+\t# The Linux build installs the defined dependency versions below.\n+\t# The OS X build installs much more recent versions, whichever\n+\t# were recorded in the Homebrew database upon creating the OS X\n+\t# image.\n+\t# Keep that in mind when you encounter a broken OS X build!\n+\tLINUX_P4_VERSION=\"16.2\"\n+\tLINUX_GIT_LFS_VERSION=\"1.5.2\"\n+\n+\tP4_PATH=\"$HOME/custom/p4\"\n+\tGIT_LFS_PATH=\"$HOME/custom/git-lfs\"\n+\texport PATH=\"$GIT_LFS_PATH:$P4_PATH:$PATH\"\n+\n+\tP4WHENCE=http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION\n+\tLFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\n+\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n \t\t$UBUNTU_COMMON_PKGS $CC_PACKAGE\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex ad1aa59a384..6b37b10d750 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -65,18 +65,6 @@ ubuntu-latest)\n \tfi\n \n \texport GIT_TEST_HTTPD=true\n-\n-\t# The Linux build installs the defined dependency versions below.\n-\t# The OS X build installs much more recent versions, whichever\n-\t# were recorded in the Homebrew database upon creating the OS X\n-\t# image.\n-\t# Keep that in mind when you encounter a broken OS X build!\n-\texport LINUX_P4_VERSION=\"16.2\"\n-\texport LINUX_GIT_LFS_VERSION=\"1.5.2\"\n-\n-\tP4_PATH=\"$HOME/custom/p4\"\n-\tGIT_LFS_PATH=\"$HOME/custom/git-lfs\"\n-\texport PATH=\"$GIT_LFS_PATH:$P4_PATH:$PATH\"\n \t;;\n macos-latest)\n \tif [ \"$jobname\" = osx-gcc ]\n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"448991","messageId":"patch-11.25-f2fcee5d6e4-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"[PATCH 11/25] CI: export variables via a wrapper","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:23Z","receivedAt":"2022-02-21T14:47:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add and use a \"setenv\" wrapper to export the variables set in\nci/lib.sh.\n\nFor now this merely adds debugging information about what variables\nare set, but in a subsequent commit we'll expand this to set variables\nin $GITHUB_ENV instead of the environment, and thus carry variable\nsettings between steps in a job.\n\nWe'll also eventually be interested in what type of variable we're\nsetting, and when it needs to be used, because we'll prominently\nexpose what variables we set in what CI \"step\", so having only the\nones we actually use will improve readability.\n\nTo that end call the \"setenv\" function with optional \"--build\",\n\"--test\" and \"--all\" parameters to indicate whether they're needed for\nbuilding, testing or both.\n\nWe currently ignore this parameter, but will make use of it in a\nsubsequent commit. Adding it here reduces the needless churn, as we\nwon't need to change all these lines again for that subsequent change.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n ci/lib.sh | 39 ++++++++++++++++++++++++++++-----------\n ci/run-build-and-tests.sh | 34 +++++++++++++++++-----------------\n 2 files changed, 45 insertions(+), 28 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 31fe3a476a9..1a1fa0e4bb2 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -13,6 +13,23 @@ then\n \texit 1\n fi\n \n+# Helper functions\n+setenv () {\n+\tvarmode=\n+\tcase \"$1\" in\n+\t--*)\n+\t\tvarmode=$1\n+\t\tshift\n+\t\t;;\n+\tesac\n+\n+\tkey=$1\n+\tval=$2\n+\tshift 2\n+\n+\teval \"export $key=\\\"$val\\\"\"\n+}\n+\n check_unignored_build_artifacts ()\n {\n \t! git ls-files --other --exclude-standard --error-unmatch \\\n@@ -24,7 +41,7 @@ check_unignored_build_artifacts ()\n }\n \n # GitHub Action doesn't set TERM, which is required by tput\n-export TERM=${TERM:-dumb}\n+setenv TERM ${TERM:-dumb}\n \n # Clear MAKEFLAGS that may come from the outside world.\n MAKEFLAGS=\n@@ -34,23 +51,23 @@ then\n \tCI_TYPE=github-actions\n \tCC=\"${CC:-gcc}\"\n \n-\texport GIT_PROVE_OPTS=\"--timer --jobs 10\"\n+\tsetenv --test GIT_PROVE_OPTS \"--timer --jobs 10\"\n \tGIT_TEST_OPTS=\"--verbose-log -x\"\n \tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n \ttest Windows != \"$RUNNER_OS\" ||\n \tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n \n-\texport GIT_TEST_OPTS\n+\tsetenv --test GIT_TEST_OPTS \"$GIT_TEST_OPTS\"\n else\n \techo \"Could not identify CI type\" >&2\n \tenv >&2\n \texit 1\n fi\n \n-export DEVELOPER=1\n-export DEFAULT_TEST_TARGET=prove\n-export GIT_TEST_CLONE_2GB=true\n-export SKIP_DASHED_BUILT_INS=YesPlease\n+setenv --build DEVELOPER 1\n+setenv --test DEFAULT_TEST_TARGET prove\n+setenv --test GIT_TEST_CLONE_2GB true\n+setenv --build SKIP_DASHED_BUILT_INS YesPlease\n \n case \"$runs_on_pool\" in\n ubuntu-latest)\n@@ -66,7 +83,7 @@ ubuntu-latest)\n \t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/python2\"\n \tfi\n \n-\texport GIT_TEST_HTTPD=true\n+\tsetenv --test GIT_TEST_HTTPD true\n \t;;\n macos-latest)\n \tif [ \"$jobname\" = osx-gcc ]\n@@ -89,9 +106,9 @@ linux-musl)\n \tMAKEFLAGS=\"$MAKEFLAGS GIT_TEST_UTF8_LOCALE=C.UTF-8\"\n \t;;\n linux-leaks)\n-\texport SANITIZE=leak\n-\texport GIT_TEST_PASSING_SANITIZE_LEAK=true\n+\tsetenv --build SANITIZE leak\n+\tsetenv --test GIT_TEST_PASSING_SANITIZE_LEAK true\n \t;;\n esac\n \n-export MAKEFLAGS=\"$MAKEFLAGS CC=${CC:-cc}\"\n+setenv --all MAKEFLAGS \"$MAKEFLAGS CC=${CC:-cc}\"\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex e5a21985b44..35d45a9373e 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -9,33 +9,33 @@ export MAKE_TARGETS=\"all test\"\n \n case \"$jobname\" in\n linux-gcc)\n-\texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+\tsetenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME main\n \t;;\n linux-TEST-vars)\n-\texport GIT_TEST_SPLIT_INDEX=yes\n-\texport GIT_TEST_MERGE_ALGORITHM=recursive\n-\texport GIT_TEST_FULL_IN_PACK_ARRAY=true\n-\texport GIT_TEST_OE_SIZE=10\n-\texport GIT_TEST_OE_DELTA_SIZE=5\n-\texport GIT_TEST_COMMIT_GRAPH=1\n-\texport GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1\n-\texport GIT_TEST_MULTI_PACK_INDEX=1\n-\texport GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1\n-\texport GIT_TEST_ADD_I_USE_BUILTIN=1\n-\texport GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n-\texport GIT_TEST_WRITE_REV_INDEX=1\n-\texport GIT_TEST_CHECKOUT_WORKERS=2\n+\tsetenv --test GIT_TEST_SPLIT_INDEX yes\n+\tsetenv --test GIT_TEST_MERGE_ALGORITHM recursive\n+\tsetenv --test GIT_TEST_FULL_IN_PACK_ARRAY true\n+\tsetenv --test GIT_TEST_OE_SIZE 10\n+\tsetenv --test GIT_TEST_OE_DELTA_SIZE 5\n+\tsetenv --test GIT_TEST_COMMIT_GRAPH 1\n+\tsetenv --test GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS 1\n+\tsetenv --test GIT_TEST_MULTI_PACK_INDEX 1\n+\tsetenv --test GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP 1\n+\tsetenv --test GIT_TEST_ADD_I_USE_BUILTIN 1\n+\tsetenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME master\n+\tsetenv --test GIT_TEST_WRITE_REV_INDEX 1\n+\tsetenv --test GIT_TEST_CHECKOUT_WORKERS 2\n \t;;\n linux-clang)\n-\texport GIT_TEST_DEFAULT_HASH=sha1\n+\tsetenv --test GIT_TEST_DEFAULT_HASH sha1\n \t;;\n linux-sha256)\n-\texport GIT_TEST_DEFAULT_HASH=sha256\n+\tsetenv --test GIT_TEST_DEFAULT_HASH sha256\n \t;;\n pedantic)\n \t# Don't run the tests; we only care about whether Git can be\n \t# built.\n-\texport DEVOPTS=pedantic\n+\tsetenv --build DEVOPTS pedantic\n \texport MAKE_TARGETS=all\n \t;;\n esac\n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"448992","messageId":"patch-19.25-e9c6c4dd293-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"[PATCH 19/25] CI: pre-select test slice in Windows & VS tests","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:31Z","receivedAt":"2022-02-21T14:47:40Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"In preceding commits the tests have been changed to do their setup via\n$GITHUB_ENV in one step, and to then have subsequent steps that re-use\nthat environment.\n\nLet's change the \"test slice\" tests added in b819f1d2cec (ci:\nparallelize testing on Windows, 2019-01-29) to do the same. These\ntests select 10% of the tests to run in 10 \"test slices\". Now we'll\nselect those in a step that immediately precedes the testing step, and\nthen simply invoke \"make -C t -e\".\n\nThis has the advantage that the tests to be run are now listed in the\nstandard \"Run\" drop-down at the start of the \"test\" step.\n\nSince the \"T\" variable in \"t/Makefile\" doesn't normally accept\noverrides from the environment we need to invoke \"make\" with the \"-e\"\noption (a.k.a. \"--environment-overrides\"). We could also make $(T) in\nt/Makefile be a \"?=\" assigned variable, but this way works, and is\narguably clearer as it's more obvious that we're injecting a special\nlist of tests that override the normal behavior of that Makefile.\n\nNote that we cannot run the top-level \"make test\" here, because of how\nthe Windows CI builds git, i.e. either via CMake or some option that\nwould cause \"make test\" to recompile git itself. Instead we run \"make\n-C t [...]\".\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .github/workflows/main.yml | 10 ++++++++--\n ci/{run-test-slice.sh => select-test-slice.sh} | 4 ++--\n 2 files changed, 10 insertions(+), 4 deletions(-)\n rename ci/{run-test-slice.sh => select-test-slice.sh} (57%)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex fe37e0f1b36..1e13718b989 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -124,9 +124,12 @@ jobs:\n - uses: git-for-windows/setup-git-for-windows-sdk@v1\n - run: ci/lib.sh\n shell: bash\n+ - name: select tests\n+ run: . /etc/profile && ci/select-test-slice.sh ${{matrix.nr}} 10\n+ shell: bash\n - name: test\n shell: bash\n- run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10\n+ run: . /etc/profile && make -C t -e\n - name: ci/print-test-failures.sh\n if: failure()\n shell: bash\n@@ -210,9 +213,12 @@ jobs:\n run: tar xf artifacts.tar.gz && tar xf tracked.tar.gz\n - run: ci/lib.sh\n shell: bash\n+ - name: select tests\n+ run: . /etc/profile && ci/select-test-slice.sh ${{matrix.nr}} 10\n+ shell: bash\n - name: test\n shell: bash\n- run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10\n+ run: . /etc/profile && make -C t -e\n - name: ci/print-test-failures.sh\n if: failure()\n shell: bash\ndiff --git a/ci/run-test-slice.sh b/ci/select-test-slice.sh\nsimilarity index 57%\nrename from ci/run-test-slice.sh\nrename to ci/select-test-slice.sh\nindex 1de6a18ca47..ec602f8a053 100755\n--- a/ci/run-test-slice.sh\n+++ b/ci/select-test-slice.sh\n@@ -1,10 +1,10 @@\n #!/bin/sh\n #\n-# Test Git in parallel\n+# Select a portion of the tests for testing Git in parallel\n #\n \n . ${0%/*}/lib.sh\n \n tests=$(echo $(cd t && ./helper/test-tool path-utils slice-tests \"$1\" \"$2\" \\\n \tt[0-9]*.sh))\n-make --quiet -C t T=\"$tests\"\n+echo T=\"$tests\" >>$GITHUB_ENV\n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"448993","messageId":"patch-16.25-868613a5b06-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"[PATCH 16/25] CI: combine ci/install{,-docker}-dependencies.sh","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:28Z","receivedAt":"2022-02-21T14:47:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Combine the \"ci/install-dependencies.sh\" and\n\"ci/install-docker-dependencies.sh\" into one script. The \"case\"\nstatement in the latter only cared about \"$jobname\", and can be folded\ninto the same \"case\" statement in the former.\n\nThe reason they split up is historical, and because the\n\"ci/install-dependencies.sh\" used \"ci/lib.sh\", which requires\n\"bash\". At least one of the docker containers doesn't have \"bash\". To\nmake the existing code POSIX-compatible we need to replace pushd/popd\nwith a sub-shell, but no other changes were needed.\n\nThis also fixes issue in \"ci/install-docker-dependencies.sh\" where\nwe'd hide errors due to not using \"set -e\". Now that we include\n\"ci/lib.sh\" we won't have that potential issue.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .github/workflows/main.yml | 2 +-\n ci/install-dependencies.sh | 27 ++++++++++++++++++++++-----\n ci/install-docker-dependencies.sh | 22 ----------------------\n 3 files changed, 23 insertions(+), 28 deletions(-)\n delete mode 100755 ci/install-docker-dependencies.sh\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 92b914f16fd..3fd3e4f873c 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -311,7 +311,7 @@ jobs:\n container: ${{matrix.vector.image}}\n steps:\n - uses: actions/checkout@v1\n- - run: ci/install-docker-dependencies.sh\n+ - run: ci/install-dependencies.sh\n - run: ci/lib.sh\n - run: make\n - run: make test\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 01006379fd0..9388289d0ed 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -1,4 +1,4 @@\n-#!/usr/bin/env bash\n+#!/bin/sh\n #\n # Install dependencies required to build and test Git on Linux and macOS\n #\n@@ -34,18 +34,20 @@ ubuntu-latest)\n \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n \t\t$UBUNTU_COMMON_PKGS $CC_PACKAGE\n \tmkdir --parents \"$P4_PATH\"\n-\tpushd \"$P4_PATH\"\n+\t(\n+\t\tcd \"$P4_PATH\"\n \t\twget --quiet \"$P4WHENCE/bin.linux26x86_64/p4d\"\n \t\twget --quiet \"$P4WHENCE/bin.linux26x86_64/p4\"\n \t\tchmod u+x p4d\n \t\tchmod u+x p4\n-\tpopd\n+\t)\n \tmkdir --parents \"$GIT_LFS_PATH\"\n-\tpushd \"$GIT_LFS_PATH\"\n+\t(\n+\t\tcd \"$GIT_LFS_PATH\"\n \t\twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \t\ttar --extract --gunzip --file \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \t\tcp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs .\n-\tpopd\n+\t)\n \t;;\n macos-latest)\n \texport HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1\n@@ -90,6 +92,21 @@ linux-gcc-default)\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install $UBUNTU_COMMON_PKGS\n \t;;\n+linux32)\n+\tlinux32 --32bit i386 sh -c '\n+\t\tapt update >/dev/null &&\n+\t\tapt install -y build-essential libcurl4-openssl-dev \\\n+\t\t\tlibssl-dev libexpat-dev gettext python >/dev/null\n+\t'\n+\t;;\n+linux-musl)\n+\tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n+\t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n+\t;;\n+pedantic)\n+\tdnf -yq update >/dev/null &&\n+\tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n+\t;;\n esac\n \n if type p4d >/dev/null && type p4 >/dev/null\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\ndeleted file mode 100755\nindex 78b7e326da6..00000000000\n--- a/ci/install-docker-dependencies.sh\n+++ /dev/null\n@@ -1,22 +0,0 @@\n-#!/bin/sh\n-#\n-# Install dependencies required to build and test Git inside container\n-#\n-\n-case \"$jobname\" in\n-linux32)\n-\tlinux32 --32bit i386 sh -c '\n-\t\tapt update >/dev/null &&\n-\t\tapt install -y build-essential libcurl4-openssl-dev \\\n-\t\t\tlibssl-dev libexpat-dev gettext python >/dev/null\n-\t'\n-\t;;\n-linux-musl)\n-\tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n-\t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n-\t;;\n-pedantic)\n-\tdnf -yq update >/dev/null &&\n-\tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n-\t;;\n-esac\n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"448994","messageId":"patch-18.25-a6106525b7f-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"[PATCH 18/25] ci/run-test-slice.sh: replace shelling out with \"echo\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:30Z","receivedAt":"2022-02-21T14:47:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Amend this code added in b819f1d2cec (ci: parallelize testing on\nWindows, 2019-01-29) to stop shelling out to \"tr\", all we're trying to\ndo here is to turn '\\n'-delimited output into a ' '-delimited list. To\ndo that we can just use \"echo\".\n\nNone of these files will contain the odd characters using \"echo\" might\nchoke on in the general case (i.e. '\\'-escapes and the like).\n\nThis has the added benefit of not adding a redundant ' ' to the end of\nthe T-parameter, which didn't matter in practice, but as we'll see in\na subsequent commit makes for slightly nicer output, as we'll show\nthis variable using a GitHub CI-native feature.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n ci/run-test-slice.sh | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh\nindex 70326961454..1de6a18ca47 100755\n--- a/ci/run-test-slice.sh\n+++ b/ci/run-test-slice.sh\n@@ -5,6 +5,6 @@\n \n . ${0%/*}/lib.sh\n \n-make --quiet -C t T=\"$(cd t &&\n-\t./helper/test-tool path-utils slice-tests \"$1\" \"$2\" t[0-9]*.sh |\n-\ttr '\\n' ' ')\"\n+tests=$(echo $(cd t && ./helper/test-tool path-utils slice-tests \"$1\" \"$2\" \\\n+\tt[0-9]*.sh))\n+make --quiet -C t T=\"$tests\"\n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"448995","messageId":"patch-17.25-5d854e8ff36-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"[PATCH 17/25] CI: move \"env\" definitions into ci/lib.sh","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:29Z","receivedAt":"2022-02-21T14:47:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Have the GitHub CI jobs use the \"ci/lib\".sh as a source of truth for\nenvironment variables that affect their builds and tests. This brings\nthem in line with the rest of the jobs, and make it easier to see the\nentire CI configuration per jobname at a glance.\n\nTo do this we need to add a ci/lib.sh \"step\" to the remaining jobs\nthat didn't have it added in preceding commits.\n\nThe Makefile parameters are luckily all accepted via the environment,\nso we can export these instead of adding them to MAKEFLAGS.\n\nLet's also use the documented GIT_CONIFG_* mechanism for setting\nconfig instead of the internal GIT_CONFIG_PARAMETERS variable. This\nadjusts code added in 889cacb6897 (ci: configure GitHub Actions for\nCI/PR, 2020-04-11), we could probably use the GIT_AUTHOR_NAME etc, but\nlet's keep setting this via config, just with the documented\nmechanism.\n\nBy setting \"ARTIFACTS_DIRECTORY=artifacts\" once we don't need to\nrepeat it in various places, let's get it from the environment\ninstead.\n\nFinally, the \"DEVELOPER: 1\" was only needed by the \"sparse\" job, which\nwasn't using \"ci/lib.sh\" (and which sets DEVELOPER=1). Let's instead\nhave the \"sparse\" job use the \"ci/lib.sh\" and remove DEVELOPER=1 from\n\".github/workflows/main.yml\".\n\nThis substantially speeds up the \"sparse\" job, since it'll now pick up\nthe \"--jobs\" setting in MAKEFLAGS that we use everywhere else. Before\nit ran in around 4m30s, now in around 2m30s.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .github/workflows/main.yml | 29 +++++++++++------------------\n ci/lib.sh | 21 +++++++++++++++++++++\n 2 files changed, 32 insertions(+), 18 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 3fd3e4f873c..fe37e0f1b36 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -2,9 +2,6 @@ name: CI\n \n on: [push, pull_request]\n \n-env:\n- DEVELOPER: 1\n-\n jobs:\n ci-config:\n name: config\n@@ -93,8 +90,7 @@ jobs:\n shell: bash\n env:\n HOME: ${{runner.workspace}}\n- NO_PERL: 1\n- run: . /etc/profile && make artifacts-tar ARTIFACTS_DIRECTORY=artifacts\n+ run: . /etc/profile && make artifacts-tar\n if: success()\n - run: ci/check-unignored-build-artifacts.sh\n if: success()\n@@ -105,7 +101,7 @@ jobs:\n uses: actions/upload-artifact@v2\n with:\n name: windows-artifacts\n- path: artifacts\n+ path: ${{env.ARTIFACTS_DIRECTORY}}\n windows-test:\n name: win test\n env:\n@@ -126,6 +122,8 @@ jobs:\n shell: bash\n run: tar xf artifacts.tar.gz && tar xf tracked.tar.gz\n - uses: git-for-windows/setup-git-for-windows-sdk@v1\n+ - run: ci/lib.sh\n+ shell: bash\n - name: test\n shell: bash\n run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10\n@@ -145,9 +143,6 @@ jobs:\n jobname: vs-build\n needs: ci-config\n if: needs.ci-config.outputs.enabled == 'yes'\n- env:\n- NO_PERL: 1\n- GIT_CONFIG_PARAMETERS: \"'user.name=CI' 'user.email=ci@git'\"\n runs-on: windows-latest\n steps:\n - uses: actions/checkout@v2\n@@ -171,6 +166,8 @@ jobs:\n - name: copy dlls to root\n shell: cmd\n run: compat\\vcbuild\\vcpkg_copy_dlls.bat release\n+ - run: ci/lib.sh\n+ shell: bash\n - name: generate Visual Studio solution\n shell: bash\n run: |\n@@ -181,21 +178,16 @@ jobs:\n - name: bundle artifact tar\n shell: bash\n env:\n- MSVC: 1\n VCPKG_ROOT: ${{github.workspace}}\\compat\\vcbuild\\vcpkg\n run: |\n- mkdir -p artifacts &&\n- eval \"$(make -n artifacts-tar INCLUDE_DLLS_IN_ARTIFACTS=YesPlease ARTIFACTS_DIRECTORY=artifacts NO_GETTEXT=YesPlease 2>&1 | grep ^tar)\"\n- - run: ci/check-unignored-build-artifacts.sh\n- if: success()\n- shell: bash\n+ eval \"$(make -n artifacts-tar 2>&1 | grep -e ^mkdir -e ^tar)\"\n - name: zip up tracked files\n run: git archive -o artifacts/tracked.tar.gz HEAD\n - name: upload tracked files and build artifacts\n uses: actions/upload-artifact@v2\n with:\n name: vs-artifacts\n- path: artifacts\n+ path: ${{env.ARTIFACTS_DIRECTORY}}\n vs-test:\n name: win+VS test\n env:\n@@ -216,10 +208,10 @@ jobs:\n - name: extract tracked files and build artifacts\n shell: bash\n run: tar xf artifacts.tar.gz && tar xf tracked.tar.gz\n+ - run: ci/lib.sh\n+ shell: bash\n - name: test\n shell: bash\n- env:\n- NO_SVN_TESTS: 1\n run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10\n - name: ci/print-test-failures.sh\n if: failure()\n@@ -354,6 +346,7 @@ jobs:\n - uses: actions/checkout@v2\n - name: Install other dependencies\n run: ci/install-dependencies.sh\n+ - run: ci/lib.sh\n - run: make sparse\n documentation:\n name: documentation\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex fc6ce4d3e04..b882849ed5d 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -95,6 +95,27 @@ macos-latest)\n esac\n \n case \"$jobname\" in\n+windows-build)\n+\tsetenv --build NO_PERL NoThanks\n+\tsetenv --build ARTIFACTS_DIRECTORY artifacts\n+\t;;\n+vs-build)\n+\tsetenv --build NO_PERL NoThanks\n+\tsetenv --build NO_GETTEXT NoThanks\n+\tsetenv --build ARTIFACTS_DIRECTORY artifacts\n+\tsetenv --build INCLUDE_DLLS_IN_ARTIFACTS YesPlease\n+\tsetenv --build MSVC YesPlease\n+\n+\tsetenv --build GIT_CONFIG_COUNT 2\n+\tsetenv --build GIT_CONFIG_KEY_0 user.name\n+\tsetenv --build GIT_CONFIG_VALUE_0 CI\n+\tsetenv --build GIT_CONFIG_KEY_1 user.emailname\n+\tsetenv --build GIT_CONFIG_VALUE_1 ci@git\n+\tsetenv --build GIT_CONFIG_VALUE_1 ci@git\n+\t;;\n+vs-test)\n+\tsetenv --test NO_SVN_TESTS YesPlease\n+\t;;\n linux-gcc)\n \tsetenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME main\n \t;;\n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"448996","messageId":"patch-14.25-aecd3ebaafe-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"[PATCH 14/25] CI: invoke \"make artifacts-tar\" directly in windows-build","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:26Z","receivedAt":"2022-02-21T14:47:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the windows-build job to invoke the \"make artifacts-tar\" step\ndirectly, instead of calling a \"ci/make-test-artifacts.sh\" script.\n\nThe script was needed because \"ci/lib.sh\" would set up various\nenvironment variables for us, but now we can instead use the\n\"ci/lib.sh\" in its script mode.\n\nThe \"mkdir -p\" added in b819f1d2cec (ci: parallelize testing on\nWindows, 2019-01-29) isn't needed, the same commit added that \"mkdir\n-p\" to the \"artifacts-tar\" rule itself, so we can have \"make\" create\nthe directory for us.\n\nThis also has the benefit of making the \"build\" step less chatty,\nsince it won't start with the verbose \"set -x\" output, that's now\ncontained in the \"ci/lib.sh\" step.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .github/workflows/main.yml | 5 ++++-\n ci/make-test-artifacts.sh | 10 ----------\n 2 files changed, 4 insertions(+), 11 deletions(-)\n delete mode 100755 ci/make-test-artifacts.sh\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 05c5771e7f6..2c23a19eac2 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -87,12 +87,15 @@ jobs:\n steps:\n - uses: actions/checkout@v2\n - uses: git-for-windows/setup-git-for-windows-sdk@v1\n+ - run: ci/lib.sh\n+ shell: bash\n - name: build\n shell: bash\n env:\n HOME: ${{runner.workspace}}\n NO_PERL: 1\n- run: . /etc/profile && ci/make-test-artifacts.sh artifacts\n+ run: . /etc/profile && make artifacts-tar ARTIFACTS_DIRECTORY=artifacts\n+ if: success()\n - run: ci/check-unignored-build-artifacts.sh\n if: success()\n shell: bash\ndiff --git a/ci/make-test-artifacts.sh b/ci/make-test-artifacts.sh\ndeleted file mode 100755\nindex 45298562982..00000000000\n--- a/ci/make-test-artifacts.sh\n+++ /dev/null\n@@ -1,10 +0,0 @@\n-#!/bin/sh\n-#\n-# Build Git and store artifacts for testing\n-#\n-\n-mkdir -p \"$1\" # in case ci/lib.sh decides to quit early\n-\n-. ${0%/*}/lib.sh\n-\n-make artifacts-tar ARTIFACTS_DIRECTORY=\"$1\"\n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"448997","messageId":"patch-13.25-46459fff296-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"[PATCH 13/25] CI: check ignored unignored build artifacts in \"win[+VS] build\" too","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:25Z","receivedAt":"2022-02-21T14:48:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Use the newly created \"ci/check-unignored-build-artifacts.sh\" script\nin the \"win build\" and \"win+VS build\" steps too. These steps weren't\ndoing this sanity check of untracked files against .gitignore before,\nbut as the check passes for them let's keep doing that extra sanity\ncheck.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .github/workflows/main.yml | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex c83f6a21ca3..05c5771e7f6 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -93,6 +93,9 @@ jobs:\n HOME: ${{runner.workspace}}\n NO_PERL: 1\n run: . /etc/profile && ci/make-test-artifacts.sh artifacts\n+ - run: ci/check-unignored-build-artifacts.sh\n+ if: success()\n+ shell: bash\n - name: zip up tracked files\n run: git archive -o artifacts/tracked.tar.gz HEAD\n - name: upload tracked files and build artifacts\n@@ -180,6 +183,9 @@ jobs:\n run: |\n mkdir -p artifacts &&\n eval \"$(make -n artifacts-tar INCLUDE_DLLS_IN_ARTIFACTS=YesPlease ARTIFACTS_DIRECTORY=artifacts NO_GETTEXT=YesPlease 2>&1 | grep ^tar)\"\n+ - run: ci/check-unignored-build-artifacts.sh\n+ if: success()\n+ shell: bash\n - name: zip up tracked files\n run: git archive -o artifacts/tracked.tar.gz HEAD\n - name: upload tracked files and build artifacts\n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"448998","messageId":"patch-12.25-dfd823f2e7d-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"[PATCH 12/25] CI: remove \"run-build-and-tests.sh\", run \"make [test]\" directly","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:24Z","receivedAt":"2022-02-21T14:48:01Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Remove the already thin \"ci/run-build-and-tests.sh\" wrapper and\ninstead make the CI run \"make\" or \"make test\" directly. By doing this\nwe'll be able to easily see at a glance whether our failure was in the\ncompilation or testing, whether that's via human eyes or improve\nmachine readability.\n\nWe also need to run our new \"ci/check-unignored-build-artifacts.sh\" on\nsuccess() in the CI now, just like we already had a step conditional\non failure() running ci/print-test-failures.sh.\n\nThe reason we used a \"ci/run-build-and-tests.sh\" wrapper in the first\nplace had to do with Travis CI-specific constraints that no longer\napply to us, as the Travis CI support has been removed.\n\nInstead we can configure the CI in an earlier step by running\n\"ci/lib.sh\", which under GitHub CI will write the environment\nvariables we need to the \"$GITHUB_ENV\" file.\n\nWe'll then have access to them in subsequent steps, and crucially\nthose variables will be prominently visible at the start of each step\nvia an expandable drop-down in the UI.drop-do.\n\nI.e. this changes the CI run from a top-down flow like (pseudocode):\n\n - job:\n - step1:\n - use ci/lib.sh to set env vars\n - run a script like ci/run-build-and-tests.sh\n - step2:\n - if: failure()\n - use ci/lib.sh to set env vars\n - run ci/print-test-failures.sh\n\nTo:\n\n - job:\n - step1:\n - set variables in $GITHUB_ENV using ci/lib.sh\n - step2:\n - make\n - step3:\n - make test\n - step4:\n - if: failure()\n - run ci/print-test-failures.sh\n - step5:\n - if: success()\n - run ci/check-unignored-build-artifacts.sh\n\nThere is a proposal[2] to get some of the benefits of this approach by\nnot re-arranging our variable setup in this way, but to instead use\nthe GitHub CI grouping syntax to focus on the relevant parts of \"make\"\nor \"make test\" when we have failures.\n\nDoing it this way makes for better looking GitHub CI UI, and lays much\nbetter ground work for our CI going forward. Because:\n\n * The CI logic will be more portable to a future CI system, since a\n common feature of them is to run various commands in sequence, but\n a future system won't necessarily support the GitHub-specifics\n syntax of \"grouping\" output within a \"step\".\n\n Even if those systems don't support a \"$GITHUB_ENV\" emulating will\n be much easier than to deal with some CI-specific grouping syntax.\n\n * At the start of every step the GitHub CI presents an expandable\n list of environment variables from \"$GITHUB_ENV\". We'll now see\n exactly what variables affected that step (although we currently\n overshoot that a bit, and always define all variables).\n\n * CI failures will be easier to reproduce locally, as this makes the\n relevant ci/* scripts something that sets up our environment, but\n leaves \"make\" and \"make test\" working as they do locally.\n\n To reproduce a run the user only needs to set the variables\n discussed in the drop-down above, either manually or by running\n \"ci/lib.sh\".\n\n * The output will be less verbose. The \"ci/lib.sh\" script uses \"set\n -x\", and before this e.g. \"ci/static-analysis.sh\" would start with\n 40 lines of trace output, culminating in using \"export\" to export\n the relevant environment variables.\n\n Now that verbosity is in the earlier \"ci/lib.sh\" step, and not in\n any subsequent one. The \"make\" targets then start out with the\n relevant output non-trace output right away.\n\n * If we do want to use the grouping syntax within a \"step\" it'll now\n be easier to do so. It doesn't support nesting, so we'd have to\n make a choice between using it for e.g. \"make\" v.s. \"make test\", or\n individual test failures. See \"sadly\" in [3].\n\n1. https://lore.kernel.org/git/211120.86k0h30zuw.gmgdl@evledraar.gmail.com/\n2. https://lore.kernel.org/git/pull.1117.git.1643050574.gitgitgadget@gmail.com/\n3. https://lore.kernel.org/git/9333ba781b8240f704e739b00d274f8c3d887e39.1643050574.git.gitgitgadget@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .github/workflows/main.yml | 14 ++++++--\n ci/check-unignored-build-artifacts.sh | 8 +++++\n ci/install-dependencies.sh | 4 +++\n ci/lib.sh | 40 ++++++++++++++++++++++-\n ci/make-test-artifacts.sh | 2 --\n ci/run-build-and-tests.sh | 47 ---------------------------\n ci/run-test-slice.sh | 2 --\n ci/test-documentation.sh | 1 -\n 8 files changed, 62 insertions(+), 56 deletions(-)\n create mode 100755 ci/check-unignored-build-artifacts.sh\n delete mode 100755 ci/run-build-and-tests.sh\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex ad752010102..c83f6a21ca3 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -268,7 +268,10 @@ jobs:\n steps:\n - uses: actions/checkout@v2\n - run: ci/install-dependencies.sh\n- - run: ci/run-build-and-tests.sh\n+ - run: ci/lib.sh\n+ - run: make\n+ - run: make test\n+ if: success()\n - run: ci/print-test-failures.sh\n if: failure()\n - name: Upload failed tests' directories\n@@ -292,6 +295,7 @@ jobs:\n image: daald/ubuntu32:xenial\n - jobname: pedantic\n image: fedora\n+ skip-tests: no\n env:\n jobname: ${{matrix.vector.jobname}}\n runs-on: ubuntu-latest\n@@ -299,9 +303,12 @@ jobs:\n steps:\n - uses: actions/checkout@v1\n - run: ci/install-docker-dependencies.sh\n- - run: ci/run-build-and-tests.sh\n+ - run: ci/lib.sh\n+ - run: make\n+ - run: make test\n+ if: success() && matrix.vector.skip-tests != 'no'\n - run: ci/print-test-failures.sh\n- if: failure()\n+ if: failure() && matrix.vector.skip-tests != 'no'\n - name: Upload failed tests' directories\n if: failure() && env.FAILED_TEST_ARTIFACTS != ''\n uses: actions/upload-artifact@v1\n@@ -317,6 +324,7 @@ jobs:\n steps:\n - uses: actions/checkout@v2\n - run: ci/install-dependencies.sh\n+ - run: ci/lib.sh\n - run: make ci-static-analysis\n sparse:\n needs: ci-config\ndiff --git a/ci/check-unignored-build-artifacts.sh b/ci/check-unignored-build-artifacts.sh\nnew file mode 100755\nindex 00000000000..56d04b0db9a\n--- /dev/null\n+++ b/ci/check-unignored-build-artifacts.sh\n@@ -0,0 +1,8 @@\n+#!/bin/sh\n+#\n+# Check whether the build created anything not in our .gitignore\n+#\n+\n+. ${0%/*}/lib.sh\n+\n+check_unignored_build_artifacts\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 34b4400f38b..01006379fd0 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -22,6 +22,10 @@ ubuntu-latest)\n \tP4_PATH=\"$HOME/custom/p4\"\n \tGIT_LFS_PATH=\"$HOME/custom/git-lfs\"\n \texport PATH=\"$GIT_LFS_PATH:$P4_PATH:$PATH\"\n+\tif test -n \"$GITHUB_PATH\"\n+\tthen\n+\t\techo \"$PATH\" >>\"$GITHUB_PATH\"\n+\tfi\n \n \tP4WHENCE=http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION\n \tLFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 1a1fa0e4bb2..05f3dd15e27 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -1,3 +1,5 @@\n+#!/bin/sh\n+\n # Library of functions shared by all CI scripts\n \n # Set 'exit on error' for all CI scripts to let the caller know that\n@@ -27,7 +29,14 @@ setenv () {\n \tval=$2\n \tshift 2\n \n-\teval \"export $key=\\\"$val\\\"\"\n+\tif test -n \"$GITHUB_ENV\"\n+\tthen\n+\t\techo \"$key=$val\" >>\"$GITHUB_ENV\"\n+\telse\n+\t\t# For local debugging. Not used by the GitHub CI\n+\t\t# itself.\n+\t\teval \"export $key=\\\"$val\\\"\"\n+\tfi\n }\n \n check_unignored_build_artifacts ()\n@@ -96,6 +105,35 @@ macos-latest)\n esac\n \n case \"$jobname\" in\n+linux-gcc)\n+\tsetenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME main\n+\t;;\n+linux-TEST-vars)\n+\tsetenv --test GIT_TEST_SPLIT_INDEX yes\n+\tsetenv --test GIT_TEST_MERGE_ALGORITHM recursive\n+\tsetenv --test GIT_TEST_FULL_IN_PACK_ARRAY true\n+\tsetenv --test GIT_TEST_OE_SIZE 10\n+\tsetenv --test GIT_TEST_OE_DELTA_SIZE 5\n+\tsetenv --test GIT_TEST_COMMIT_GRAPH 1\n+\tsetenv --test GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS 1\n+\tsetenv --test GIT_TEST_MULTI_PACK_INDEX 1\n+\tsetenv --test GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP 1\n+\tsetenv --test GIT_TEST_ADD_I_USE_BUILTIN 1\n+\tsetenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME master\n+\tsetenv --test GIT_TEST_WRITE_REV_INDEX 1\n+\tsetenv --test GIT_TEST_CHECKOUT_WORKERS 2\n+\t;;\n+linux-clang)\n+\tsetenv --test GIT_TEST_DEFAULT_HASH sha1\n+\t;;\n+linux-sha256)\n+\tsetenv --test GIT_TEST_DEFAULT_HASH sha256\n+\t;;\n+pedantic)\n+\t# Don't run the tests; we only care about whether Git can be\n+\t# built.\n+\tsetenv --build DEVOPTS pedantic\n+\t;;\n linux32)\n \tCC=gcc\n \t;;\ndiff --git a/ci/make-test-artifacts.sh b/ci/make-test-artifacts.sh\nindex 646967481f6..45298562982 100755\n--- a/ci/make-test-artifacts.sh\n+++ b/ci/make-test-artifacts.sh\n@@ -8,5 +8,3 @@ mkdir -p \"$1\" # in case ci/lib.sh decides to quit early\n . ${0%/*}/lib.sh\n \n make artifacts-tar ARTIFACTS_DIRECTORY=\"$1\"\n-\n-check_unignored_build_artifacts\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\ndeleted file mode 100755\nindex 35d45a9373e..00000000000\n--- a/ci/run-build-and-tests.sh\n+++ /dev/null\n@@ -1,47 +0,0 @@\n-#!/bin/sh\n-#\n-# Build and test Git\n-#\n-\n-. ${0%/*}/lib.sh\n-\n-export MAKE_TARGETS=\"all test\"\n-\n-case \"$jobname\" in\n-linux-gcc)\n-\tsetenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME main\n-\t;;\n-linux-TEST-vars)\n-\tsetenv --test GIT_TEST_SPLIT_INDEX yes\n-\tsetenv --test GIT_TEST_MERGE_ALGORITHM recursive\n-\tsetenv --test GIT_TEST_FULL_IN_PACK_ARRAY true\n-\tsetenv --test GIT_TEST_OE_SIZE 10\n-\tsetenv --test GIT_TEST_OE_DELTA_SIZE 5\n-\tsetenv --test GIT_TEST_COMMIT_GRAPH 1\n-\tsetenv --test GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS 1\n-\tsetenv --test GIT_TEST_MULTI_PACK_INDEX 1\n-\tsetenv --test GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP 1\n-\tsetenv --test GIT_TEST_ADD_I_USE_BUILTIN 1\n-\tsetenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME master\n-\tsetenv --test GIT_TEST_WRITE_REV_INDEX 1\n-\tsetenv --test GIT_TEST_CHECKOUT_WORKERS 2\n-\t;;\n-linux-clang)\n-\tsetenv --test GIT_TEST_DEFAULT_HASH sha1\n-\t;;\n-linux-sha256)\n-\tsetenv --test GIT_TEST_DEFAULT_HASH sha256\n-\t;;\n-pedantic)\n-\t# Don't run the tests; we only care about whether Git can be\n-\t# built.\n-\tsetenv --build DEVOPTS pedantic\n-\texport MAKE_TARGETS=all\n-\t;;\n-esac\n-\n-# Any new \"test\" targets should not go after this \"make\", but should\n-# adjust $MAKE_TARGETS. Otherwise compilation-only targets above will\n-# start running tests.\n-make $MAKE_TARGETS\n-check_unignored_build_artifacts\ndiff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh\nindex b9a682b4bcd..70326961454 100755\n--- a/ci/run-test-slice.sh\n+++ b/ci/run-test-slice.sh\n@@ -8,5 +8,3 @@\n make --quiet -C t T=\"$(cd t &&\n \t./helper/test-tool path-utils slice-tests \"$1\" \"$2\" t[0-9]*.sh |\n \ttr '\\n' ' ')\"\n-\n-check_unignored_build_artifacts\ndiff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\nindex 41e2b126311..6b5cce03bd7 100755\n--- a/ci/test-documentation.sh\n+++ b/ci/test-documentation.sh\n@@ -39,4 +39,3 @@ test -s Documentation/git.html\n grep '\n---\n .github/workflows/main.yml | 16 +++++++++++++-\n ci/check-unignored-build-artifacts.sh | 10 +++++++++\n ci/lib.sh | 10 ---------\n ci/test-documentation.sh | 31 +++++++++------------------\n 4 files changed, 35 insertions(+), 32 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 2c23a19eac2..92b914f16fd 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -365,4 +365,18 @@ jobs:\n steps:\n - uses: actions/checkout@v2\n - run: ci/install-dependencies.sh\n- - run: ci/test-documentation.sh\n+ - run: ci/lib.sh\n+ - run: make check-docs\n+ - run: \"make doc > >(tee stdout.log) 2> >(tee stderr.raw >&2)\"\n+ shell: bash\n+ - run: ci/test-documentation.sh AsciiDoc\n+ if: success()\n+ - run: ci/check-unignored-build-artifacts.sh\n+ if: success()\n+ - run: make clean\n+ - run: \"make USE_ASCIIDOCTOR=1 doc > >(tee stdout.log) 2> >(tee stderr.raw >&2)\"\n+ shell: bash\n+ - run: ci/test-documentation.sh Asciidoctor\n+ if: success()\n+ - run: ci/check-unignored-build-artifacts.sh\n+ if: success()\ndiff --git a/ci/check-unignored-build-artifacts.sh b/ci/check-unignored-build-artifacts.sh\nindex 56d04b0db9a..0bc04f32804 100755\n--- a/ci/check-unignored-build-artifacts.sh\n+++ b/ci/check-unignored-build-artifacts.sh\n@@ -5,4 +5,14 @@\n \n . ${0%/*}/lib.sh\n \n+check_unignored_build_artifacts ()\n+{\n+\t! git ls-files --other --exclude-standard --error-unmatch \\\n+\t\t-- ':/*' 2>/dev/null ||\n+\t{\n+\t\techo \"$(tput setaf 1)error: found unignored build artifacts$(tput sgr0)\"\n+\t\tfalse\n+\t}\n+}\n+\n check_unignored_build_artifacts\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 05f3dd15e27..fc6ce4d3e04 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -39,16 +39,6 @@ setenv () {\n \tfi\n }\n \n-check_unignored_build_artifacts ()\n-{\n-\t! git ls-files --other --exclude-standard --error-unmatch \\\n-\t\t-- ':/*' 2>/dev/null ||\n-\t{\n-\t\techo \"$(tput setaf 1)error: found unignored build artifacts$(tput sgr0)\"\n-\t\tfalse\n-\t}\n-}\n-\n # GitHub Action doesn't set TERM, which is required by tput\n setenv TERM ${TERM:-dumb}\n \ndiff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\nindex 6b5cce03bd7..b8a6a6f664e 100755\n--- a/ci/test-documentation.sh\n+++ b/ci/test-documentation.sh\n@@ -1,10 +1,12 @@\n-#!/usr/bin/env bash\n+#!/bin/sh\n #\n-# Perform sanity checks on documentation and build it.\n+# Perform sanity checks on \"make doc\" output and built documentation\n #\n \n . ${0%/*}/lib.sh\n \n+generator=$1\n+\n filter_log () {\n \tsed -e '/^GIT_VERSION = /d' \\\n \t -e \"/constant Gem::ConfigMap is deprecated/d\" \\\n@@ -14,28 +16,15 @@ filter_log () {\n \t \"$1\"\n }\n \n-make check-docs\n-\n-# Build docs with AsciiDoc\n-make doc > >(tee stdout.log) 2> >(tee stderr.raw >&2)\n-cat stderr.raw\n-filter_log stderr.raw >stderr.log\n-test ! -s stderr.log\n-test -s Documentation/git.html\n-test -s Documentation/git.xml\n-test -s Documentation/git.1\n-grep ' >(tee stdout.log) 2> >(tee stderr.raw >&2)\n cat stderr.raw\n filter_log stderr.raw >stderr.log\n test ! -s stderr.log\n test -s Documentation/git.html\n-grep '\n---\n ci/check-unignored-build-artifacts.sh | 4 +++-\n ci/install-dependencies.sh | 2 +-\n ci/lib-ci-type.sh | 8 ++++++++\n ci/lib-tput.sh | 2 ++\n ci/lib.sh | 27 ++++++++++-----------------\n ci/print-test-failures.sh | 6 +++---\n ci/select-test-slice.sh | 2 +-\n ci/test-documentation.sh | 2 +-\n 8 files changed, 29 insertions(+), 24 deletions(-)\n create mode 100644 ci/lib-ci-type.sh\n create mode 100644 ci/lib-tput.sh\n\ndiff --git a/ci/check-unignored-build-artifacts.sh b/ci/check-unignored-build-artifacts.sh\nindex 0bc04f32804..c27d6a97f45 100755\n--- a/ci/check-unignored-build-artifacts.sh\n+++ b/ci/check-unignored-build-artifacts.sh\n@@ -3,7 +3,9 @@\n # Check whether the build created anything not in our .gitignore\n #\n \n-. ${0%/*}/lib.sh\n+set -ex\n+\n+. ${0%/*}/lib-tput.sh\n \n check_unignored_build_artifacts ()\n {\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 9388289d0ed..ee9af62fc92 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -3,7 +3,7 @@\n # Install dependencies required to build and test Git on Linux and macOS\n #\n \n-. ${0%/*}/lib.sh\n+set -ex\n \n UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl\ndiff --git a/ci/lib-ci-type.sh b/ci/lib-ci-type.sh\nnew file mode 100644\nindex 00000000000..bd6e093c8f4\n--- /dev/null\n+++ b/ci/lib-ci-type.sh\n@@ -0,0 +1,8 @@\n+if test \"$GITHUB_ACTIONS\" = \"true\"\n+then\n+\tCI_TYPE=github-actions\n+else\n+\techo \"Could not identify CI type\" >&2\n+\tenv >&2\n+\texit 1\n+fi\ndiff --git a/ci/lib-tput.sh b/ci/lib-tput.sh\nnew file mode 100644\nindex 00000000000..baed1892f69\n--- /dev/null\n+++ b/ci/lib-tput.sh\n@@ -0,0 +1,2 @@\n+# GitHub Action doesn't set TERM, which is required by tput\n+export TERM=${TERM:-dumb}\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex b882849ed5d..5eadc96de29 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -1,13 +1,9 @@\n #!/bin/sh\n-\n-# Library of functions shared by all CI scripts\n-\n-# Set 'exit on error' for all CI scripts to let the caller know that\n-# something went wrong.\n-# Set tracing executed commands, primarily setting environment variables\n-# and installing dependencies.\n set -ex\n \n+# Helper libraries\n+. ${0%/*}/lib-ci-type.sh\n+\n # Starting assertions\n if test -z \"$jobname\"\n then\n@@ -39,15 +35,11 @@ setenv () {\n \tfi\n }\n \n-# GitHub Action doesn't set TERM, which is required by tput\n-setenv TERM ${TERM:-dumb}\n-\n # Clear MAKEFLAGS that may come from the outside world.\n MAKEFLAGS=\n \n-if test \"$GITHUB_ACTIONS\" = \"true\"\n-then\n-\tCI_TYPE=github-actions\n+case \"$CI_TYPE\" in\n+github-actions)\n \tCC=\"${CC:-gcc}\"\n \n \tsetenv --test GIT_PROVE_OPTS \"--timer --jobs 10\"\n@@ -57,11 +49,12 @@ then\n \tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n \n \tsetenv --test GIT_TEST_OPTS \"$GIT_TEST_OPTS\"\n-else\n-\techo \"Could not identify CI type\" >&2\n-\tenv >&2\n+\t;;\n+*)\n+\techo \"Unhandled CI type: $CI_TYPE\" >&2\n \texit 1\n-fi\n+\t;;\n+esac\n \n setenv --build DEVELOPER 1\n setenv --test DEFAULT_TEST_TARGET prove\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex 0c63b6f7962..452aff35d74 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -3,10 +3,10 @@\n # Print output of failing tests\n #\n \n-. ${0%/*}/lib.sh\n+set -e\n \n-# Tracing executed commands would produce too much noise in the loop below.\n-set +x\n+. ${0%/*}/lib-ci-type.sh\n+. ${0%/*}/lib-tput.sh\n \n cd t/\n \ndiff --git a/ci/select-test-slice.sh b/ci/select-test-slice.sh\nindex ec602f8a053..a0332095bb3 100755\n--- a/ci/select-test-slice.sh\n+++ b/ci/select-test-slice.sh\n@@ -3,7 +3,7 @@\n # Select a portion of the tests for testing Git in parallel\n #\n \n-. ${0%/*}/lib.sh\n+set -ex\n \n tests=$(echo $(cd t && ./helper/test-tool path-utils slice-tests \"$1\" \"$2\" \\\n \tt[0-9]*.sh))\ndiff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\nindex b8a6a6f664e..64ff212cdaa 100755\n--- a/ci/test-documentation.sh\n+++ b/ci/test-documentation.sh\n@@ -3,7 +3,7 @@\n # Perform sanity checks on \"make doc\" output and built documentation\n #\n \n-. ${0%/*}/lib.sh\n+set -ex\n \n generator=$1\n \n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"449001","messageId":"patch-21.25-abf9c504740-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"[PATCH 21/25] CI: narrow down variable definitions in --build and --test","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:33Z","receivedAt":"2022-02-21T14:48:11Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"In a preceding step the \"setvar\" function was made to take a\n\"--build\", \"--test\" or \"--all\" argument to indicate where the\nvariables it sets were used.\n\nLet's make use of that by having the relevant parts of\n\".github/workflows/main.yml\" invoke \"ci/lib.sh\" with those options.\n\nBy doing this the set of variables shown in build-only steps will be\nfewer, which makes diagnosing anything going on there easier, as we\nwon't have to look at a deluge of e.g. GIT_TEST_* variables.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .github/workflows/main.yml | 20 +++++++++++---------\n ci/lib.sh | 17 +++++++++++++++++\n 2 files changed, 28 insertions(+), 9 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 1e13718b989..0787cadc76b 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -84,7 +84,7 @@ jobs:\n steps:\n - uses: actions/checkout@v2\n - uses: git-for-windows/setup-git-for-windows-sdk@v1\n- - run: ci/lib.sh\n+ - run: ci/lib.sh --build\n shell: bash\n - name: build\n shell: bash\n@@ -122,7 +122,7 @@ jobs:\n shell: bash\n run: tar xf artifacts.tar.gz && tar xf tracked.tar.gz\n - uses: git-for-windows/setup-git-for-windows-sdk@v1\n- - run: ci/lib.sh\n+ - run: ci/lib.sh --test\n shell: bash\n - name: select tests\n run: . /etc/profile && ci/select-test-slice.sh ${{matrix.nr}} 10\n@@ -169,7 +169,7 @@ jobs:\n - name: copy dlls to root\n shell: cmd\n run: compat\\vcbuild\\vcpkg_copy_dlls.bat release\n- - run: ci/lib.sh\n+ - run: ci/lib.sh --build\n shell: bash\n - name: generate Visual Studio solution\n shell: bash\n@@ -211,7 +211,7 @@ jobs:\n - name: extract tracked files and build artifacts\n shell: bash\n run: tar xf artifacts.tar.gz && tar xf tracked.tar.gz\n- - run: ci/lib.sh\n+ - run: ci/lib.sh --test\n shell: bash\n - name: select tests\n run: . /etc/profile && ci/select-test-slice.sh ${{matrix.nr}} 10\n@@ -275,8 +275,9 @@ jobs:\n steps:\n - uses: actions/checkout@v2\n - run: ci/install-dependencies.sh\n- - run: ci/lib.sh\n+ - run: ci/lib.sh --build\n - run: make\n+ - run: ci/lib.sh --test\n - run: make test\n if: success()\n - run: ci/print-test-failures.sh\n@@ -310,8 +311,9 @@ jobs:\n steps:\n - uses: actions/checkout@v1\n - run: ci/install-dependencies.sh\n- - run: ci/lib.sh\n+ - run: ci/lib.sh --build\n - run: make\n+ - run: ci/lib.sh --test\n - run: make test\n if: success() && matrix.vector.skip-tests != 'no'\n - run: ci/print-test-failures.sh\n@@ -331,7 +333,7 @@ jobs:\n steps:\n - uses: actions/checkout@v2\n - run: ci/install-dependencies.sh\n- - run: ci/lib.sh\n+ - run: ci/lib.sh --build\n - run: make ci-static-analysis\n sparse:\n needs: ci-config\n@@ -352,7 +354,7 @@ jobs:\n - uses: actions/checkout@v2\n - name: Install other dependencies\n run: ci/install-dependencies.sh\n- - run: ci/lib.sh\n+ - run: ci/lib.sh --build\n - run: make sparse\n documentation:\n name: documentation\n@@ -364,7 +366,7 @@ jobs:\n steps:\n - uses: actions/checkout@v2\n - run: ci/install-dependencies.sh\n- - run: ci/lib.sh\n+ - run: ci/lib.sh --build\n - run: make check-docs\n - run: \"make doc > >(tee stdout.log) 2> >(tee stderr.raw >&2)\"\n shell: bash\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 5eadc96de29..367d1ee05d5 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -5,6 +5,13 @@ set -ex\n . ${0%/*}/lib-ci-type.sh\n \n # Starting assertions\n+mode=$1\n+if test -z \"$mode\"\n+then\n+\techo \"need a $0 mode, e.g. --build or --test\"\n+\texit 1\n+fi\n+\n if test -z \"$jobname\"\n then\n \techo \"must set a CI jobname\" >&2\n@@ -13,9 +20,14 @@ fi\n \n # Helper functions\n setenv () {\n+\tskip=\n \tvarmode=\n \tcase \"$1\" in\n \t--*)\n+\t\tif test \"$1\" != \"$mode\" && test \"$1\" != \"--all\"\n+\t\tthen\n+\t\t\tskip=t\n+\t\tfi\n \t\tvarmode=$1\n \t\tshift\n \t\t;;\n@@ -25,6 +37,11 @@ setenv () {\n \tval=$2\n \tshift 2\n \n+\tif test -n \"$skip\"\n+\tthen\n+\t\treturn 0\n+\tfi\n+\n \tif test -n \"$GITHUB_ENV\"\n \tthen\n \t\techo \"$key=$val\" >>\"$GITHUB_ENV\"\n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"449002","messageId":"patch-22.25-9f4c2798a82-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"[PATCH 22/25] CI: add more variables to MAKEFLAGS, except under vs-build","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:34Z","receivedAt":"2022-02-21T14:48:18Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"It's clearer that \"DEVELOPER\" is a flag that affects the Makefile\nitself in particular if it's put into \"MAKEFLAGS than\" if it\ngenerically sits in the environment. Let's move both it and\n\"SKIP_DASHED_BUILT_INS\" to \"MAKEFLAGS\".\n\nWe can't do this under vs-build, since that invokes cmake. Let's have\nonly that job set these in the environment.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n ci/lib.sh | 12 ++++++++++--\n 1 file changed, 10 insertions(+), 2 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 367d1ee05d5..475e9f63a74 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -55,6 +55,13 @@ setenv () {\n # Clear MAKEFLAGS that may come from the outside world.\n MAKEFLAGS=\n \n+# Common make and cmake build options\n+DEVELOPER=1\n+SKIP_DASHED_BUILT_INS=YesPlease\n+\n+# Use common options for \"make\" (cmake in \"vs-build\" below)\n+MAKEFLAGS=\"DEVELOPER=$DEVELOPER SKIP_DASHED_BUILT_INS=$SKIP_DASHED_BUILT_INS\"\n+\n case \"$CI_TYPE\" in\n github-actions)\n \tCC=\"${CC:-gcc}\"\n@@ -73,10 +80,8 @@ github-actions)\n \t;;\n esac\n \n-setenv --build DEVELOPER 1\n setenv --test DEFAULT_TEST_TARGET prove\n setenv --test GIT_TEST_CLONE_2GB true\n-setenv --build SKIP_DASHED_BUILT_INS YesPlease\n \n case \"$runs_on_pool\" in\n ubuntu-latest)\n@@ -110,6 +115,9 @@ windows-build)\n \tsetenv --build ARTIFACTS_DIRECTORY artifacts\n \t;;\n vs-build)\n+\tsetenv --build DEVELOPER $DEVELOPER\n+\tsetenv --build SKIP_DASHED_BUILT_INS $SKIP_DASHED_BUILT_INS\n+\n \tsetenv --build NO_PERL NoThanks\n \tsetenv --build NO_GETTEXT NoThanks\n \tsetenv --build ARTIFACTS_DIRECTORY artifacts\n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"449003","messageId":"patch-25.25-08a9776c259-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"[PATCH 25/25] CI: don't use \"set -x\" in \"ci/lib.sh\" output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:37Z","receivedAt":"2022-02-21T14:48:22Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Remove the \"set -x\" from the \"ci/lib.sh\" output. Before preceding\ncommits the logic in that file was much more complex, and likely to\nfail in some scenarios.\n\nNow we only task \"ci/lib.sh\" with setting various variables for\nsubsequent steps in our jobs, so we can start emitting more tailored\ndebugging output, which makes what it's doing easier to read.\n\nThis change also changes the output of the \"ci/print-test-failures.sh\"\nscript, since it's the only other user of \"ci/lib-ci-type.sh\". In that\ncase it's also helpful to know what \"$CI_TYPE\" we're using, as that\nscript doesn't \"set -x\" and will act differently depending on the\n$CI_TYPE.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n ci/lib-ci-type.sh | 2 ++\n ci/lib.sh | 9 ++++++++-\n 2 files changed, 10 insertions(+), 1 deletion(-)\n\ndiff --git a/ci/lib-ci-type.sh b/ci/lib-ci-type.sh\nindex bd6e093c8f4..6cfe58596d2 100644\n--- a/ci/lib-ci-type.sh\n+++ b/ci/lib-ci-type.sh\n@@ -6,3 +6,5 @@ else\n \tenv >&2\n \texit 1\n fi\n+\n+echo \"CONFIG: CI_TYPE=$CI_TYPE\" >&2\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 36f79eb6a65..f4323a3dd36 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -1,5 +1,5 @@\n #!/bin/sh\n-set -ex\n+set -e\n \n # Helper libraries\n . ${0%/*}/lib-ci-type.sh\n@@ -11,12 +11,16 @@ then\n \techo \"need a $0 mode, e.g. --build or --test\"\n \texit 1\n fi\n+echo \"CONFIG: mode=$mode\" >&2\n \n if test -z \"$jobname\"\n then\n \techo \"must set a CI jobname\" >&2\n \texit 1\n fi\n+echo \"CONFIG: jobname=$jobname\" >&2\n+echo \"CONFIG: runs_on_pool=$runs_on_pool\" >&2\n+echo \"CONFIG: GITHUB_ENV=$GITHUB_ENV\" >&2\n \n # Helper functions\n setenv () {\n@@ -39,6 +43,7 @@ setenv () {\n \n \tif test -n \"$skip\"\n \tthen\n+\t\techo \"SKIP '$key=$val'\" >&2\n \t\treturn 0\n \tfi\n \n@@ -50,6 +55,8 @@ setenv () {\n \t\t# itself.\n \t\teval \"export $key=\\\"$val\\\"\"\n \tfi\n+\n+\techo \"SET: '$key=$val'\" >&2\n }\n \n # Clear variables that may come from the outside world.\n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"449004","messageId":"patch-23.25-8a8b7ecf16b-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"[PATCH 23/25] CI: stop over-setting the $CC variable","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:35Z","receivedAt":"2022-02-21T14:48:24Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"As detailed in 2c8921db2b8 (travis-ci: build with the right compiler,\n2019-01-17) the reason we started using $CC in $MAKEFLAGS as opposed\nto setting it in the environment was due to Travis CI clobbering $CC\nin the environment.\n\nWe don't need to set it unconditionally to accomplish that, but rather\njust have it set for those jobs that need them. E.g. the \"win+VS\nbuild\" job confusingly has CC=gcc set, even though it builds with\nMSVC.\n\nThis partially reverts my 707d2f2fe86 (CI: use \"$runs_on_pool\", not\n\"$jobname\" to select packages & config, 2021-11-23), i.e. we're now\naiming to only set those variables specific jobs need.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .github/workflows/main.yml | 13 -------------\n ci/lib.sh | 26 ++++++++++++++++++++++----\n 2 files changed, 22 insertions(+), 17 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 0787cadc76b..6d25ec4ae3b 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -238,37 +238,24 @@ jobs:\n matrix:\n vector:\n - jobname: linux-clang\n- cc: clang\n pool: ubuntu-latest\n - jobname: linux-sha256\n- cc: clang\n os: ubuntu\n pool: ubuntu-latest\n - jobname: linux-gcc\n- cc: gcc\n- cc_package: gcc-8\n pool: ubuntu-latest\n - jobname: linux-TEST-vars\n- cc: gcc\n os: ubuntu\n- cc_package: gcc-8\n pool: ubuntu-latest\n - jobname: osx-clang\n- cc: clang\n pool: macos-latest\n - jobname: osx-gcc\n- cc: gcc\n- cc_package: gcc-9\n pool: macos-latest\n - jobname: linux-gcc-default\n- cc: gcc\n pool: ubuntu-latest\n - jobname: linux-leaks\n- cc: gcc\n pool: ubuntu-latest\n env:\n- CC: ${{matrix.vector.cc}}\n- CC_PACKAGE: ${{matrix.vector.cc_package}}\n jobname: ${{matrix.vector.jobname}}\n runs_on_pool: ${{matrix.vector.pool}}\n runs-on: ${{matrix.vector.pool}}\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 475e9f63a74..7064a17cfeb 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -52,7 +52,9 @@ setenv () {\n \tfi\n }\n \n-# Clear MAKEFLAGS that may come from the outside world.\n+# Clear variables that may come from the outside world.\n+CC=\n+CC_PACKAGE=\n MAKEFLAGS=\n \n # Common make and cmake build options\n@@ -64,8 +66,6 @@ MAKEFLAGS=\"DEVELOPER=$DEVELOPER SKIP_DASHED_BUILT_INS=$SKIP_DASHED_BUILT_INS\"\n \n case \"$CI_TYPE\" in\n github-actions)\n-\tCC=\"${CC:-gcc}\"\n-\n \tsetenv --test GIT_PROVE_OPTS \"--timer --jobs 10\"\n \tGIT_TEST_OPTS=\"--verbose-log -x\"\n \tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n@@ -135,9 +135,16 @@ vs-test)\n \tsetenv --test NO_SVN_TESTS YesPlease\n \t;;\n linux-gcc)\n+\tCC=gcc\n+\tCC_PACKAGE=gcc-8\n \tsetenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME main\n \t;;\n+linux-gcc-default)\n+\tCC=gcc\n+\t;;\n linux-TEST-vars)\n+\tCC=gcc\n+\tCC_PACKAGE=gcc-8\n \tsetenv --test GIT_TEST_SPLIT_INDEX yes\n \tsetenv --test GIT_TEST_MERGE_ALGORITHM recursive\n \tsetenv --test GIT_TEST_FULL_IN_PACK_ARRAY true\n@@ -152,10 +159,19 @@ linux-TEST-vars)\n \tsetenv --test GIT_TEST_WRITE_REV_INDEX 1\n \tsetenv --test GIT_TEST_CHECKOUT_WORKERS 2\n \t;;\n+osx-gcc)\n+\tCC=gcc\n+\tCC_PACKAGE=gcc-9\n+\t;;\n+osx-clang)\n+\tCC=clang\n+\t;;\n linux-clang)\n+\tCC=clang\n \tsetenv --test GIT_TEST_DEFAULT_HASH sha1\n \t;;\n linux-sha256)\n+\tCC=clang\n \tsetenv --test GIT_TEST_DEFAULT_HASH sha256\n \t;;\n pedantic)\n@@ -173,9 +189,11 @@ linux-musl)\n \tMAKEFLAGS=\"$MAKEFLAGS GIT_TEST_UTF8_LOCALE=C.UTF-8\"\n \t;;\n linux-leaks)\n+\tCC=gcc\n \tsetenv --build SANITIZE leak\n \tsetenv --test GIT_TEST_PASSING_SANITIZE_LEAK true\n \t;;\n esac\n \n-setenv --all MAKEFLAGS \"$MAKEFLAGS CC=${CC:-cc}\"\n+MAKEFLAGS=\"$MAKEFLAGS${CC:+ CC=$CC}\"\n+setenv --all MAKEFLAGS \"$MAKEFLAGS\"\n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"449005","messageId":"patch-24.25-d7b472b4a52-20220221T143936Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"[PATCH 24/25] CI: set PYTHON_PATH setting for osx-{clang,gcc} into \"$jobname\" case","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T14:46:36Z","receivedAt":"2022-02-21T14:48:26Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the setting of \"PYTHON_PATH=[...]\" to the \"$jobname\" case\nstatement. This partially backs out of my 707d2f2fe86 (CI: use\n\"$runs_on_pool\", not \"$jobname\" to select packages & config,\n2021-11-23), now that we have a \"osx-{clang,gcc}\" anyway for setting\n\"$CC\" we might as well do away with this part of the \"$runs_on_pool\"\ncase.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n ci/lib.sh | 10 ++--------\n 1 file changed, 2 insertions(+), 8 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 7064a17cfeb..36f79eb6a65 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -99,14 +99,6 @@ ubuntu-latest)\n \n \tsetenv --test GIT_TEST_HTTPD true\n \t;;\n-macos-latest)\n-\tif [ \"$jobname\" = osx-gcc ]\n-\tthen\n-\t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python3)\"\n-\telse\n-\t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python2)\"\n-\tfi\n-\t;;\n esac\n \n case \"$jobname\" in\n@@ -160,10 +152,12 @@ linux-TEST-vars)\n \tsetenv --test GIT_TEST_CHECKOUT_WORKERS 2\n \t;;\n osx-gcc)\n+\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python3)\"\n \tCC=gcc\n \tCC_PACKAGE=gcc-9\n \t;;\n osx-clang)\n+\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python2)\"\n \tCC=clang\n \t;;\n linux-clang)\n-- \n2.35.1.1132.ga1fe46f8690\n\n"},{"id":"449097","messageId":"nycvar.QRO.7.76.6.2202221115590.4418@tvgsbejvaqbjf.bet","threadId":"57451","inReplyTo":"patch-05.25-4738a22a36d-20220221T143936Z-avarab@gmail.com","subject":"Re: [PATCH 05/25] CI: remove unused Azure ci/* code","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-22T10:21:57Z","receivedAt":"2022-02-22T10:22:12Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\n\nOn Mon, 21 Feb 2022, Ævar Arnfjörð Bjarmason wrote:\n\n> Remove Azure-specific code that's been unused since 6081d3898fe (ci:\n> retire the Azure Pipelines definition, 2020-04-11). As noted in a\n> larger removal of all of the Azure-supporting code in [1] (although\n> that missed some of this) there is more of it in-tree, but let's focus\n> on only the ci/* code for now.\n>\n> This is needed because in subsequent commits this unused code would\n> either need to be changed to accommodate changes in ci/*, or be\n> removed.\n>\n> As we'll see in those subsequent commits the end-state we're heading\n> towards will actually make it easier to add new CI types in the\n> future, even though the only one we're left with now is the GitHub\n> CI. I.e. the \"ci/*\" framework will be made to do less, not more. We'll\n> be offloading more of what it does to our generic build and test\n> system.\n>\n> While I'm at it (since the line needs to be touched anyway) change an\n> odd 'if test true == \"$GITHUB_ACTIONS\"' to the more usual style used\n> in other places of 'if test \"$GITHUB_ACTIONS\" = \"true\"'.\n>\n> 1. https://lore.kernel.org/git/patch-1.1-eec0a8c3164-20211217T000418Z-avarab@gmail.com/\n\nThis has been discussed before, and I already gave my NAK.\n\nIt is sad that I have to repeat myself: it is a good thing to have the\nAzure Pipelines definition as a fall-back. In the past, this has served us\nvery well especially when we had to run a barrage of security fixes, for a\nslew of backports to previous release trains.\n\nYou seem to have fun to just remove this code, under some assumption that\nit is not needed, despite me pointing out that it is needed.\n\nCiao,\nJohannes\n"},{"id":"449105","messageId":"220222.86k0dn89bm.gmgdl@evledraar.gmail.com","threadId":"57451","inReplyTo":"nycvar.QRO.7.76.6.2202221115590.4418@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 05/25] CI: remove unused Azure ci/* code","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-22T10:27:38Z","receivedAt":"2022-02-22T10:49:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Feb 22 2022, Johannes Schindelin wrote:\n\n> Hi,\n>\n>\n> On Mon, 21 Feb 2022, Ævar Arnfjörð Bjarmason wrote:\n>\n>> Remove Azure-specific code that's been unused since 6081d3898fe (ci:\n>> retire the Azure Pipelines definition, 2020-04-11). As noted in a\n>> larger removal of all of the Azure-supporting code in [1] (although\n>> that missed some of this) there is more of it in-tree, but let's focus\n>> on only the ci/* code for now.\n>>\n>> This is needed because in subsequent commits this unused code would\n>> either need to be changed to accommodate changes in ci/*, or be\n>> removed.\n>>\n>> As we'll see in those subsequent commits the end-state we're heading\n>> towards will actually make it easier to add new CI types in the\n>> future, even though the only one we're left with now is the GitHub\n>> CI. I.e. the \"ci/*\" framework will be made to do less, not more. We'll\n>> be offloading more of what it does to our generic build and test\n>> system.\n>>\n>> While I'm at it (since the line needs to be touched anyway) change an\n>> odd 'if test true == \"$GITHUB_ACTIONS\"' to the more usual style used\n>> in other places of 'if test \"$GITHUB_ACTIONS\" = \"true\"'.\n>>\n>> 1. https://lore.kernel.org/git/patch-1.1-eec0a8c3164-20211217T000418Z-avarab@gmail.com/\n>\n> This has been discussed before, and I already gave my NAK.\n>\n> It is sad that I have to repeat myself: it is a good thing to have the\n> Azure Pipelines definition as a fall-back. In the past, this has served us\n> very well especially when we had to run a barrage of security fixes, for a\n> slew of backports to previous release trains.\n>\n> You seem to have fun to just remove this code, under some assumption that\n> it is not needed, despite me pointing out that it is needed.\n\nI previously submitted a stand-alone patch to remove it[1] after the\nremoval of (most of) the Travis CI was merged in f9b889dd67b (Merge\nbranch 'ab/ci-updates', 2021-12-15).\n\nI submitted that not for \"fun\", but because I had other CI/test-lib.sh\nchanges depending on that.\n\nI.e. I needed to either patch code unused in-tree I couldn't test\nwithout reverting your 6081d3898fe (ci: retire the Azure Pipelines\ndefinition, 2020-04-11) (and presumably set up some Azure CI account\netc.), or remove it.\n\nIt's still unclear to me how Azure CI is being used as a \"fall-back\",\ncan you explain that? I.e. we don't have the azure-pipelines.yml anymore\n(and neither does git-for-windows/git). Is this in-tree code being used\nby some out-of-tree fork of git.git, or do you mean you'd like to keep\nit in case we find a reason to revert your 6081d3898fe?\n\nIn this case (and as I think this series makes clear), we can't easily\nkeep certain Travis + Azure assumptions around *and* simplify how\nci/lib.sh works in the same way. At the tip of this series it's turned\ninto a very dumb variable setting helper with the CI recipe itself\ndriving the test.\n\nWhereas Travis and Azure were using a feature where a \"job\" once running\nwould attempt to skip the rest of the CI run.\n\nAnd in your [1] you mentioned the reasons for keeping it around being:\n \n The reason is that there are still some things that Azure Pipelines can do\n that GitHub workflows cannot, for example:\n \n - present the logs of failed tests in an intuitive manner,\n \n - re-run _only_ failed jobs.\n\nIt seems to me that in your parallel series you're working on 1/2 of\nthose, and with/without those changes we could otherwise improve the\npresentation of the failed test runs enough with the \"grouping\"\netc. feature.\n\nAnd as for 2/2 that any such support would be mostly orthogonal to\nkeeping Azure code that's currently unused around. I.e. a goal of this\nseries is also to make the CI less of a special snowflake, because I'm\nnot only interested in running the CI code on GitHub CI, but also\nlocally.\n\nWhich means that for any future port to Azure, GitLab CI etc. we'd\npresumably just have a thin recipe that ran \"make\" followed by \"make\ntest\", and for a re-run of only that job it would just use the CI itself\nto do that, without us needing to carry any special-cases in ci/*.\n\nI'm not at all opposed to keeping Azure support in-tree. I specifically\nhave a problem with unused code holding other improvements\nhostage.\n\nI.e. neither I nor anyone else can easily change anything surrounding it\nwhile being assured that we haven't broken that unused code. As noted in\n04/25 I think you've also had that issue, i.e. your recent 0e7696c64db\n(it seems to me) introduced what would have been a bug if it were\ncombined with the azure-pipelines.yml code removed in your 6081d3898fe.\n\n1. https://lore.kernel.org/git/nycvar.QRO.7.76.6.2112201834050.347@tvgsbejvaqbjf.bet/\n"},{"id":"449115","messageId":"nycvar.QRO.7.76.6.2202221306190.11118@tvgsbejvaqbjf.bet","threadId":"57451","inReplyTo":"220222.86k0dn89bm.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 05/25] CI: remove unused Azure ci/* code","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-22T12:07:47Z","receivedAt":"2022-02-22T12:08:00Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Tue, 22 Feb 2022, Ævar Arnfjörð Bjarmason wrote:\n\n> It's still unclear to me how Azure CI is being used as a \"fall-back\",\n> can you explain that?\n\nBy reinstating the `azure-pipelines.yml` file from the last known-good\nversion.\n\nCiao,\nJohannes\n"},{"id":"449126","messageId":"220222.86y2236ndp.gmgdl@evledraar.gmail.com","threadId":"57451","inReplyTo":"nycvar.QRO.7.76.6.2202221306190.11118@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 05/25] CI: remove unused Azure ci/* code","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-22T13:05:08Z","receivedAt":"2022-02-22T13:29:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Feb 22 2022, Johannes Schindelin wrote:\n\n> Hi Ævar,\n>\n> On Tue, 22 Feb 2022, Ævar Arnfjörð Bjarmason wrote:\n>\n>> It's still unclear to me how Azure CI is being used as a \"fall-back\",\n>> can you explain that?\n>\n> By reinstating the `azure-pipelines.yml` file from the last known-good\n> version.\n\nOkey, despite what you might thing I'm not on the war path to remove all\nof this at all costs.\n\nIn particular I don't mind much keeping the 05/25 parts around, but the\n04/25 would be a hassle, depending on how you reply to the below.\n\nSo what do you expect me or someone who overtly touches code related to\nthis & the similar (and related) in-tree JUnit support to do? Presumably\none of:\n\n 1. Don't work on any such code at all, in case you'd one day like to\n resurrect Azure support.\n\n 2. Test my patches locally by reverting azure-pipelines.yml and make\n sure any overt changes to Azure-related code still work with that\n reverted commit.\n\n 3. Just \"YOLO hack it\" but leave it in place-as is. E.g. for 04/25 and\n 05/25 in combination with 11/25 (the later s/export/setenv/g change)\n leave a known-bad-but-looking-like-it-was-before Azure branch in-place.\n\n 4. A variant of #2 where I attempt to patch the Azure code branches, but\n the \"testing\" is just eyeballing that they look reasonable, but I haven't\n *really* tested them even once (and I'd note as much in a commit message).\n\nI'm not willing to go for #1 and #2. I could do #3 or #4 if Junio/others\nchime and agree with you, but I really don't think those are worth it\neither.\n\nI could understand your view in your reply back in December when I sent\nthe stand-alone Azure removal patch[1]. Even though I noted that it was\nneeded for subsequent changes it was a stand-alone patch, so it wasn't\neasy to evaluate the trade-off of whether removing it was worth it.\n\nI think here it is quite easy. The end-state of this series\nsignificantly improves the UX for the CI we actually use, and I think\nactually makes it easier to get Azure support back up & running should\nyou ever want that, since the whole structure of it is making CI less\ncomplex and less of a special-case.\n\nIt's been almost 2 years (just around 2 months short of it..) since\nazure-pipelines.yml was removed from \"master\". It would really be quite\neasy to get it or any other new CI target off the ground, particularly\nafter this series.\n\nInsisting that any effort to fix actual CI issues in the actual CI we do\nuse needs to be hamstrung by any change in the area needing to carefully\neyeball:\n\n git show 6081d3898fe^:azure-pipelines.yml\n\nAnd for each change carefully consider them *if* we still ran that just\nseems unreasonably obstructionist.\n\nSure, Azure CI had some neat features, so did Travis CI. If we ever want\nto run either of them again let's just consider that if and when that\nhappens.\n\nMaybe you have relevant feedback queued up, but so far you haven't had\nany meaningful comments on the goals end end-state of this series as\ncompared to yours[2] (to the extent that their approaches differ).\n\nCan we try to focus on that instead? I.e. the actual visible CI changes\nthat'll either make the CI workflow we actually use better or worse?\n\n1. https://lore.kernel.org/git/patch-1.1-eec0a8c3164-20211217T000418Z-avarab@gmail.com/\n2. https://lore.kernel.org/git/pull.1117.git.1643050574.gitgitgadget@gmail.com/\n"},{"id":"449248","messageId":"4d3fecbf-23f9-f6b6-414e-8eaca620aced@gmail.com","threadId":"57451","inReplyTo":"cover-00.25-00000000000-20220221T143936Z-avarab@gmail.com","subject":"Re: [PATCH 00/25] CI: run \"make [test]\" directly, use $GITHUB_ENV","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2022-02-23T13:55:39Z","receivedAt":"2022-02-23T13:55:48Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Ævar\n\nOn 21/02/2022 14:46, Ævar Arnfjörð Bjarmason wrote:\n > [...]\n> == Details\n> \n> This series conflicts with and is a proposed alternative to Johannes's\n> proposed changes to teach \"ci/lib.sh\" GitHub specific syntax to\n> \"group\" certain parts of the output[2]. That series proposes to keep\n> the \"ci/run-build-and-tests.sh\" and similar scripts, but to teach them\n> to emit markers indicating when the \"build\" part commences, then\n> \"test\" etc.\n> \n> I think the approach offered here is better as elaborated on in\n> 12/25[1], especially because as Johannes notes[3] we only have one\n> level of \"grouping\" within a \"step\" available to us. In his version\n> using it for \"build\" v.s. \"test\" precludes being able to group parts\n> of that \"build\" or \"test\" output in the future.\n> \n> The following compares CI output between the three with a\n> change-on-top which caused a compilation error (\"this\" is this series,\n> \"js\" is Johannes's) when clicking on the \"linux-gcc\" failure:\n> \n> master: https://github.com/avar/git/runs/5274251909?check_suite_focus=true\n> this: https://github.com/avar/git/runs/5274464670?check_suite_focus=true\n> js: https://github.com/avar/git/runs/5272239403?check_suite_focus=true\n> \n> Here we went from 93 lines of output on \"master\" to 47 in \"this\"\n> (107 in \"js\"). Note also how in \"this\" we can:\n> \n> - Expand the \"Run make\" to get the \"MAKEFLAGS\" and other variables\n> used in the step. That's now a 3-line summary instead of the\n> first 50 lines being the old \"ci/lib.sh\" \"set -x\" output.\n> \n> - Because \"make\" failed, we have an elided \"make test\" step that's\n> not being run below. We can thus see what steps our failure caused\n> us to skip.\n> \n> - Unlike the \"js\" version we'll show the compilation error by\n> default (the \"js\" version is grouped and needs to be expanded),\n> but our output is now brief enough that that's no longer\n> surrounded by other irrelevant output.\n> \n> - Unlike the \"js\" version there is no \"CI setup\" group within each\n> step. That work is in the earlier \"ci/lib.sh --build\" step\n> instead, which sets the config for all subsequent steps.\n\nI can see that showing compilation errors without having to expand \nanything is useful but in my experience they are relatively rare \ncompared to test failures. If I understand the rest of the message \ncorrectly we're left with having to expand print-test-failures and \nsearching for \"not ok\" to find out what went wrong with this series.\n\n> For other output changes look at the difference between \"master\" and\n> \"this\" at:\n> \n> master: https://github.com/git/git/actions/runs/1866786595\n> this: https://github.com/avar/git/actions/runs/1876270588\n> \n> Not explicitly covered in the summary above is that various other\n> parts of ci/* were doing the same sort of within-step setup, but now\n> do so via cross-step passing of variables via $GITHUB_ENV. E.g. on\n> \"master\" the test slices for the Windows tests have a lot of verbosity\n> before they get to running the test itself:\n> \n> https://github.com/git/git/runs/5254267914?check_suite_focus=true#step:5:56\n> \n> In the \"js\" version the test output is hidden (\"grouped\"), but we've\n> got the same amount of verbosity by default:\n> \n> https://github.com/gitgitgadget/git/runs/4937491771?check_suite_focus=true#step:5:67\n> \n> Whereas in \"this\" version that verbosity is in the preceding \"select\n> tests\" step, with the \"test\" step showing only the relevant end-state\n> of the \"$T\" variable we'll use in the Makefile (hidden by default,\n> expanded in this link):\n> \n> https://github.com/avar/git/runs/5274558869?check_suite_focus=true#step:7:13\n\nAll I can see in the \"$T\" variable I have to scroll to get the prove \noutput on screen\n\n\n> This series does not attempt to replace the use of the \"group\" output\n> used for the \"ok\" or \"not ok\" portion of tests in Johannes's\n> series. When a test fails the output in this series (sans config\n> discovery not being repeated, and now summarized in the $GITHUB_ENV\n> drop-down) is substantially the same as on \"master\".\n> \n> My summary at [4] goes into other overlap & non-overlap between the\n> two. I think using the \"group\" output for those purposes might be\n> useful, although I left some feedback on [5] with problems in the\n> current \"js\" implementation.\n> \n> I do think any such changes should follow behind this series, as doing\n> that sort of grouping once we can effectively free up an extra \"group\"\n> level (by peeling those off into \"steps\", as is done here) would be\n> much more useful.\n\nI can see the attraction of being able to use make directly in the ci \nfrom an implementation point of view but from the point of view of \nsomeone trying to investigate a test failure then unless I've \nmisunderstood I don't think this series improves the current situation. \nWhy can't you build on top of Dscho's series that makes it easier to see \nthe output of the failed tests?\n\nBest Wishes\n\nPhillip\n\n> 1. https://lore.kernel.org/git/patch-12.25-dfd823f2e7d-20220221T122605Z-avarab@gmail.com\n> 2. https://lore.kernel.org/git/pull.1117.git.1643050574.gitgitgadget@gmail.com/\n> 3. https://lore.kernel.org/git/9333ba781b8240f704e739b00d274f8c3d887e39.1643050574.git.gitgitgadget@gmail.com/\n> 4. https://lore.kernel.org/git/220127.86ilu5cdnf.gmgdl@evledraar.gmail.com/\n> 5. https://lore.kernel.org/git/220126.86sftbfjl4.gmgdl@evledraar.gmail.com/\n> \n> Ævar Arnfjörð Bjarmason (25):\n> CI: run \"set -ex\" early in ci/lib.sh\n> CI: make \"$jobname\" explicit, remove fallback\n> CI: remove more dead Travis CI support\n> CI: remove dead \"tree skipping\" code\n> CI: remove unused Azure ci/* code\n> CI: don't have \"git grep\" invoke a pager in tree content check\n> CI: have \"static-analysis\" run a \"make ci-static-analysis\" target\n> CI: have \"static-analysis\" run \"check-builtins\", not \"documentation\"\n> CI: move p4 and git-lfs variables to ci/install-dependencies.sh\n> CI: consistently use \"export\" in ci/lib.sh\n> CI: export variables via a wrapper\n> CI: remove \"run-build-and-tests.sh\", run \"make [test]\" directly\n> CI: check ignored unignored build artifacts in \"win[+VS] build\" too\n> CI: invoke \"make artifacts-tar\" directly in windows-build\n> CI: split up and reduce \"ci/test-documentation.sh\"\n> CI: combine ci/install{,-docker}-dependencies.sh\n> CI: move \"env\" definitions into ci/lib.sh\n> ci/run-test-slice.sh: replace shelling out with \"echo\"\n> CI: pre-select test slice in Windows & VS tests\n> CI: only invoke ci/lib.sh as \"steps\" in main.yml\n> CI: narrow down variable definitions in --build and --test\n> CI: add more variables to MAKEFLAGS, except under vs-build\n> CI: stop over-setting the $CC variable\n> CI: set PYTHON_PATH setting for osx-{clang,gcc} into \"$jobname\" case\n> CI: don't use \"set -x\" in \"ci/lib.sh\" output\n> \n> .github/workflows/main.yml | 100 +++++---\n> Makefile | 32 ++-\n> ci/check-directional-formatting.bash | 2 +-\n> ci/check-unignored-build-artifacts.sh | 20 ++\n> ci/install-dependencies.sh | 53 ++++-\n> ci/install-docker-dependencies.sh | 22 --\n> ci/lib-ci-type.sh | 10 +\n> ci/lib-tput.sh | 2 +\n> ci/lib.sh | 315 ++++++++++++--------------\n> ci/make-test-artifacts.sh | 12 -\n> ci/mount-fileshare.sh | 25 --\n> ci/print-test-failures.sh | 16 +-\n> ci/run-build-and-tests.sh | 54 -----\n> ci/run-docker-build.sh | 66 ------\n> ci/run-docker.sh | 47 ----\n> ci/run-static-analysis.sh | 32 ---\n> ci/run-test-slice.sh | 17 --\n> ci/select-test-slice.sh | 10 +\n> ci/test-documentation.sh | 37 +--\n> ci/util/extract-trash-dirs.sh | 50 ----\n> 20 files changed, 345 insertions(+), 577 deletions(-)\n> create mode 100755 ci/check-unignored-build-artifacts.sh\n> delete mode 100755 ci/install-docker-dependencies.sh\n> create mode 100644 ci/lib-ci-type.sh\n> create mode 100644 ci/lib-tput.sh\n> delete mode 100755 ci/make-test-artifacts.sh\n> delete mode 100755 ci/mount-fileshare.sh\n> delete mode 100755 ci/run-build-and-tests.sh\n> delete mode 100755 ci/run-docker-build.sh\n> delete mode 100755 ci/run-docker.sh\n> delete mode 100755 ci/run-static-analysis.sh\n> delete mode 100755 ci/run-test-slice.sh\n> create mode 100755 ci/select-test-slice.sh\n> delete mode 100755 ci/util/extract-trash-dirs.sh\n> \n\n"},{"id":"449336","messageId":"xmqq8ru1uyty.fsf@gitster.g","threadId":"57451","inReplyTo":"4d3fecbf-23f9-f6b6-414e-8eaca620aced@gmail.com","subject":"Re: [PATCH 00/25] CI: run \"make [test]\" directly, use $GITHUB_ENV","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-23T20:12:25Z","receivedAt":"2022-02-23T20:12:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phillip Wood writes:\n\n> I can see that showing compilation errors without having to expand\n> anything is useful but in my experience they are relatively rare \n> compared to test failures. If I understand the rest of the message\n> correctly we're left with having to expand print-test-failures and \n> searching for \"not ok\" to find out what went wrong with this series.\n\nIt matches my experience that most of the time I have to look for\ntest errors, not compilation errors. Having to expand the other\ngroup and looking for \"not ok\" is something I've got accustomed to,\nso if that part improves, that would be great, but if that part does\nnot become worse, then that is OK, at least to me ;-)\n"},{"id":"449349","messageId":"220223.86pmndb6lj.gmgdl@evledraar.gmail.com","threadId":"57451","inReplyTo":"4d3fecbf-23f9-f6b6-414e-8eaca620aced@gmail.com","subject":"Re: [PATCH 00/25] CI: run \"make [test]\" directly, use $GITHUB_ENV","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-23T21:18:21Z","receivedAt":"2022-02-23T21:45:01Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 23 2022, Phillip Wood wrote:\n\n> On 21/02/2022 14:46, Ævar Arnfjörð Bjarmason wrote:\n>> [...]\n>> == Details\n>> This series conflicts with and is a proposed alternative to\n>> Johannes's\n>> proposed changes to teach \"ci/lib.sh\" GitHub specific syntax to\n>> \"group\" certain parts of the output[2]. That series proposes to keep\n>> the \"ci/run-build-and-tests.sh\" and similar scripts, but to teach them\n>> to emit markers indicating when the \"build\" part commences, then\n>> \"test\" etc.\n>> I think the approach offered here is better as elaborated on in\n>> 12/25[1], especially because as Johannes notes[3] we only have one\n>> level of \"grouping\" within a \"step\" available to us. In his version\n>> using it for \"build\" v.s. \"test\" precludes being able to group parts\n>> of that \"build\" or \"test\" output in the future.\n>> The following compares CI output between the three with a\n>> change-on-top which caused a compilation error (\"this\" is this series,\n>> \"js\" is Johannes's) when clicking on the \"linux-gcc\" failure:\n>> master:\n>> https://github.com/avar/git/runs/5274251909?check_suite_focus=true\n>> this: https://github.com/avar/git/runs/5274464670?check_suite_focus=true\n>> js: https://github.com/avar/git/runs/5272239403?check_suite_focus=true\n>> Here we went from 93 lines of output on \"master\" to 47 in \"this\"\n>> (107 in \"js\"). Note also how in \"this\" we can:\n>> - Expand the \"Run make\" to get the \"MAKEFLAGS\" and other\n>> variables\n>> used in the step. That's now a 3-line summary instead of the\n>> first 50 lines being the old \"ci/lib.sh\" \"set -x\" output.\n>> - Because \"make\" failed, we have an elided \"make test\" step\n>> that's\n>> not being run below. We can thus see what steps our failure caused\n>> us to skip.\n>> - Unlike the \"js\" version we'll show the compilation error by\n>> default (the \"js\" version is grouped and needs to be expanded),\n>> but our output is now brief enough that that's no longer\n>> surrounded by other irrelevant output.\n>> - Unlike the \"js\" version there is no \"CI setup\" group within\n>> each\n>> step. That work is in the earlier \"ci/lib.sh --build\" step\n>> instead, which sets the config for all subsequent steps.\n>\n> I can see that showing compilation errors without having to expand\n> anything is useful but in my experience they are relatively rare \n> compared to test failures. If I understand the rest of the message\n> correctly we're left with having to expand print-test-failures and \n> searching for \"not ok\" to find out what went wrong with this series.\n\nYes, it's a non-goal of this series to directly improve or change the\n\"prove\" output, or to replace or change the functioning of\nci/print-test-failures.sh.\n\nI think that's also a thing worth doing, it's just not what I'm aiming\nfor here. I.e. the goals of this & Johannes's series is only partial.\n\nThe above summary is comparing the parts of the two where those goals do\noverlap, i.e. it's rougly comparable to patches 1-4/9 of Johannes's\nseries. His 5-9/9 are (mostly) orthagonal modifications to t/test-lib.sh\netc.\n\n>> For other output changes look at the difference between \"master\" and\n>> \"this\" at:\n>> master: https://github.com/git/git/actions/runs/1866786595\n>> this: https://github.com/avar/git/actions/runs/1876270588\n>> Not explicitly covered in the summary above is that various other\n>> parts of ci/* were doing the same sort of within-step setup, but now\n>> do so via cross-step passing of variables via $GITHUB_ENV. E.g. on\n>> \"master\" the test slices for the Windows tests have a lot of verbosity\n>> before they get to running the test itself:\n>> https://github.com/git/git/runs/5254267914?check_suite_focus=true#step:5:56\n>> In the \"js\" version the test output is hidden (\"grouped\"), but we've\n>> got the same amount of verbosity by default:\n>> https://github.com/gitgitgadget/git/runs/4937491771?check_suite_focus=true#step:5:67\n>> Whereas in \"this\" version that verbosity is in the preceding \"select\n>> tests\" step, with the \"test\" step showing only the relevant end-state\n>> of the \"$T\" variable we'll use in the Makefile (hidden by default,\n>> expanded in this link):\n>> https://github.com/avar/git/runs/5274558869?check_suite_focus=true#step:7:13\n>\n> All I can see in the \"$T\" variable I have to scroll to get the prove\n> output on screen\n\nYes, what that link is showing you is that for the test run the\nvariables that control the full set of variables that control the test\nrun are now prominently displayed.\n\nAll of the output etc. after that isn't changed.\n\nThe point of those comparisons is to draw your eyes to the part that\n*is* changed v.s. master, and how using that variable passing mechanism\nis making things related to that simpler & less verbose.\n\n>> This series does not attempt to replace the use of the \"group\" output\n>> used for the \"ok\" or \"not ok\" portion of tests in Johannes's\n>> series. When a test fails the output in this series (sans config\n>> discovery not being repeated, and now summarized in the $GITHUB_ENV\n>> drop-down) is substantially the same as on \"master\".\n>> My summary at [4] goes into other overlap & non-overlap between the\n>> two. I think using the \"group\" output for those purposes might be\n>> useful, although I left some feedback on [5] with problems in the\n>> current \"js\" implementation.\n>> I do think any such changes should follow behind this series, as\n>> doing\n>> that sort of grouping once we can effectively free up an extra \"group\"\n>> level (by peeling those off into \"steps\", as is done here) would be\n>> much more useful.\n>\n> I can see the attraction of being able to use make directly in the ci\n> from an implementation point of view but from the point of view of \n> someone trying to investigate a test failure then unless I've\n> misunderstood I don't think this series improves the current\n> situation.\n\nJohannes's series is making use of GitHub's group output with changes to\nci/lib.sh to e.g. distinguish the \"make\" phase from \"make test\".\n\nThis series shows that's not something that's needed to \"group\" those\ntwo.\n\nInstead we can set set the variables and other context that ci/lib.sh\ncurrently sets at the start of every \"step\" in an earlier \"step\" in a\nway that spans later steps.\n\nThose later steps are then a light \"make\" or \"make test\", and because\nsteps are their own \"group\" it accomplishes the same goals for grouping\nthat part of the output, without needing any CI-specific syntax in ci/*.\n\nFurther, if we ever wanted to make use of such syntax for the\n\"ci/lib.sh\" part we could do that on top of this series for\ne.g. \"grouping\" individual errors that \"make\" emits.\n\nDoing that wouldn't be possible with Johannes's approach, since there's\nonly one \"group\" level, you can't nest them. So using it for \"build\" and\n\"test\" means you've already used up your one-level.\n\nIt's also a lot simpler, the resulting code size for the two in terms of\n*.sh code in ci/ is:\n\n $ git grep . avar/ci-unroll-make-commands-to-ci-recipe 'ci/**.sh'|wc -l\n 422\n $ git grep . pr-1117/dscho/use-grouping-in-ci-v1 'ci/**.sh'|wc -l\n 721\n\n> Why can't you build on top of Dscho's series that makes it\n> easier to see the output of the failed tests?\n\nBecause building on top of it would mean adding a bunch of code and\ncomplexity to ci/* to accomplish a goal that this series shows that we\ncan equally get without that complexity and added code, as the diffstat\netc. shows.\n\nWhat we do about t/test-lib.sh, \"prove\" etc. emitting summary output\nabout tests is then a mostly orthagonal change on top of that.\n\n\"Mostly\" except insofar that Johannes's series notes that he's left with\nthe limitation[3] that \"sadly, workflow output currently cannot contain\nany nested groups\".\n\nOf course that's also true after this series (that GitHub CI feature\nwill be the same).\n\nBut the difference is that since we won't have used the \"group\" for\n\"make\" and \"make test\" (as it's \"kicked up\" to the \"step\" level) a\nseries to add \"group\"-ing to the test output wouldn't have the group for\nan individual test failure at the same level as \"make test\".\n\n>> 1. https://lore.kernel.org/git/patch-12.25-dfd823f2e7d-20220221T122605Z-avarab@gmail.com\n>> 2. https://lore.kernel.org/git/pull.1117.git.1643050574.gitgitgadget@gmail.com/\n>> 3. https://lore.kernel.org/git/9333ba781b8240f704e739b00d274f8c3d887e39.1643050574.git.gitgitgadget@gmail.com/\n>> 4. https://lore.kernel.org/git/220127.86ilu5cdnf.gmgdl@evledraar.gmail.com/\n>> 5. https://lore.kernel.org/git/220126.86sftbfjl4.gmgdl@evledraar.gmail.com/\n>> Ævar Arnfjörð Bjarmason (25):\n>> CI: run \"set -ex\" early in ci/lib.sh\n>> CI: make \"$jobname\" explicit, remove fallback\n>> CI: remove more dead Travis CI support\n>> CI: remove dead \"tree skipping\" code\n>> CI: remove unused Azure ci/* code\n>> CI: don't have \"git grep\" invoke a pager in tree content check\n>> CI: have \"static-analysis\" run a \"make ci-static-analysis\" target\n>> CI: have \"static-analysis\" run \"check-builtins\", not \"documentation\"\n>> CI: move p4 and git-lfs variables to ci/install-dependencies.sh\n>> CI: consistently use \"export\" in ci/lib.sh\n>> CI: export variables via a wrapper\n>> CI: remove \"run-build-and-tests.sh\", run \"make [test]\" directly\n>> CI: check ignored unignored build artifacts in \"win[+VS] build\" too\n>> CI: invoke \"make artifacts-tar\" directly in windows-build\n>> CI: split up and reduce \"ci/test-documentation.sh\"\n>> CI: combine ci/install{,-docker}-dependencies.sh\n>> CI: move \"env\" definitions into ci/lib.sh\n>> ci/run-test-slice.sh: replace shelling out with \"echo\"\n>> CI: pre-select test slice in Windows & VS tests\n>> CI: only invoke ci/lib.sh as \"steps\" in main.yml\n>> CI: narrow down variable definitions in --build and --test\n>> CI: add more variables to MAKEFLAGS, except under vs-build\n>> CI: stop over-setting the $CC variable\n>> CI: set PYTHON_PATH setting for osx-{clang,gcc} into \"$jobname\" case\n>> CI: don't use \"set -x\" in \"ci/lib.sh\" output\n>> .github/workflows/main.yml | 100 +++++---\n>> Makefile | 32 ++-\n>> ci/check-directional-formatting.bash | 2 +-\n>> ci/check-unignored-build-artifacts.sh | 20 ++\n>> ci/install-dependencies.sh | 53 ++++-\n>> ci/install-docker-dependencies.sh | 22 --\n>> ci/lib-ci-type.sh | 10 +\n>> ci/lib-tput.sh | 2 +\n>> ci/lib.sh | 315 ++++++++++++--------------\n>> ci/make-test-artifacts.sh | 12 -\n>> ci/mount-fileshare.sh | 25 --\n>> ci/print-test-failures.sh | 16 +-\n>> ci/run-build-and-tests.sh | 54 -----\n>> ci/run-docker-build.sh | 66 ------\n>> ci/run-docker.sh | 47 ----\n>> ci/run-static-analysis.sh | 32 ---\n>> ci/run-test-slice.sh | 17 --\n>> ci/select-test-slice.sh | 10 +\n>> ci/test-documentation.sh | 37 +--\n>> ci/util/extract-trash-dirs.sh | 50 ----\n>> 20 files changed, 345 insertions(+), 577 deletions(-)\n>> create mode 100755 ci/check-unignored-build-artifacts.sh\n>> delete mode 100755 ci/install-docker-dependencies.sh\n>> create mode 100644 ci/lib-ci-type.sh\n>> create mode 100644 ci/lib-tput.sh\n>> delete mode 100755 ci/make-test-artifacts.sh\n>> delete mode 100755 ci/mount-fileshare.sh\n>> delete mode 100755 ci/run-build-and-tests.sh\n>> delete mode 100755 ci/run-docker-build.sh\n>> delete mode 100755 ci/run-docker.sh\n>> delete mode 100755 ci/run-static-analysis.sh\n>> delete mode 100755 ci/run-test-slice.sh\n>> create mode 100755 ci/select-test-slice.sh\n>> delete mode 100755 ci/util/extract-trash-dirs.sh\n>> \n\n"},{"id":"450486","messageId":"20220305081746.GA1655@szeder.dev","threadId":"57451","inReplyTo":"patch-23.25-8a8b7ecf16b-20220221T143936Z-avarab@gmail.com","subject":"Re: [PATCH 23/25] CI: stop over-setting the $CC variable","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2022-03-05T08:17:46Z","receivedAt":"2022-03-05T08:18:00Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Feb 21, 2022 at 03:46:35PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> As detailed in 2c8921db2b8 (travis-ci: build with the right compiler,\n> 2019-01-17) the reason we started using $CC in $MAKEFLAGS as opposed\n> to setting it in the environment was due to Travis CI clobbering $CC\n> in the environment.\n\nThis is not what 2c8921db2b8 detailed; in fact 2c8921db2b8 detailed\nthe exact opposite. Travis CI did not clobber $CC, it set $CC to what\nwe asked it to. We started using $CC in $MAKEFLAGS as opposed to\nsetting it in the environment, because our Makefile _ignores_ $CC in\nthe environment.\n\n"},{"id":"450493","messageId":"220305.8635jw4h78.gmgdl@evledraar.gmail.com","threadId":"57451","inReplyTo":"20220305081746.GA1655@szeder.dev","subject":"Re: [PATCH 23/25] CI: stop over-setting the $CC variable","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-05T14:15:01Z","receivedAt":"2022-03-05T14:17:21Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Mar 05 2022, SZEDER Gábor wrote:\n\n> On Mon, Feb 21, 2022 at 03:46:35PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>> As detailed in 2c8921db2b8 (travis-ci: build with the right compiler,\n>> 2019-01-17) the reason we started using $CC in $MAKEFLAGS as opposed\n>> to setting it in the environment was due to Travis CI clobbering $CC\n>> in the environment.\n>\n> This is not what 2c8921db2b8 detailed; in fact 2c8921db2b8 detailed\n> the exact opposite. Travis CI did not clobber $CC, it set $CC to what\n> we asked it to. We started using $CC in $MAKEFLAGS as opposed to\n> setting it in the environment, because our Makefile _ignores_ $CC in\n> the environment.\n\nThanks. You're completely right, I don't know how I got that wrong when\nwriting this summary.\n\nI'll fix it.\n\nBut as for the body of the change I believe it looks good & is\nconsistent with avoiding that issue. I.e. we'll set it in MAKEFLAGS both\nbefore and after this change.\n\nWe just won't have a separate \"CC\" variable in the environment as far as\nGitHub's environment variable dump is concerned, which IMO makes this a\nlot easier to follow, i.e. we only need CC as a \"make\" argument, so\nlet's add it and other such things to MAKEFLAGS only.\n"},{"id":"450941","messageId":"nycvar.QRO.7.76.6.2203092304320.357@tvgsbejvaqbjf.bet","threadId":"57451","inReplyTo":"220223.86pmndb6lj.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 00/25] CI: run \"make [test]\" directly, use $GITHUB_ENV","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-03-09T22:10:15Z","receivedAt":"2022-03-09T22:10:31Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Wed, 23 Feb 2022, Ævar Arnfjörð Bjarmason wrote:\n\n> On Wed, Feb 23 2022, Phillip Wood wrote:\n>\n> > If I understand the rest of the message correctly we're left with\n> > having to expand print-test-failures and searching for \"not ok\" to\n> > find out what went wrong with this series.\n>\n> Yes, it's a non-goal of this series to directly improve or change the\n> \"prove\" output, or to replace or change the functioning of\n> ci/print-test-failures.sh.\n\nSince it is explicitly _not_ the goal of this series, but of the series I\noffered for review in\nhttps://lore.kernel.org/git/pull.1117.v2.git.1646130289.gitgitgadget@gmail.com/\n(with which this here patch series conflicts rather intentionally), I\nsuggest that your patch series can wait a little longer and build on top\nof mine.\n\nWe do have the rule in the Git project to try to avoid conflicting with\npatch series that are already in flight. It just makes the lives of all\ninvolved parties so much easier.\n\nThanks,\nJohannes\n"},{"id":"452318","messageId":"patch-v2-15.25-dfa91ac8938-20220325T182534Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-v2-00.25-00000000000-20220325T182534Z-avarab@gmail.com","subject":"[PATCH v2 15/25] CI: split up and reduce \"ci/test-documentation.sh\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-25T18:38:06Z","receivedAt":"2022-03-25T19:28:05Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the \"ci/test-documentation.sh\" script to run the bash-specific\nparts in as one command in the CI job itself, and to run the two \"make\ndoc\" commands at the top-level.\n\nIt'll now be obvious from the title of the step if if we failed in the\nasciidoc or asciidoctor step.\n\nSince the \"check_unignored_build_artifacts()\" function is now only\nused in \"ci/check-unignored-build-artifacts.sh\" move that function\nthere.\n\nThe recipe for the job in \".github/workflows/main.yml\" is now a bit\nverbose because it's effectively the same job twice, with a \"make\nclean\" in-between. It would be better for the verbosity to run it via\na matrix as done in the alternate approach in [1] does, but then we'd\nsacrifice overall CPU time for the brevity. It might still be worth\ndoing, but let's go for this simpler approach for now.\n\n1. https://lore.kernel.org/git/patch-v2-6.6-7c423c8283d-20211120T030848Z-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .github/workflows/main.yml | 16 +++++++++++++-\n ci/check-unignored-build-artifacts.sh | 10 +++++++++\n ci/lib.sh | 10 ---------\n ci/test-documentation.sh | 31 +++++++++------------------\n 4 files changed, 35 insertions(+), 32 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 2c23a19eac2..92b914f16fd 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -365,4 +365,18 @@ jobs:\n steps:\n - uses: actions/checkout@v2\n - run: ci/install-dependencies.sh\n- - run: ci/test-documentation.sh\n+ - run: ci/lib.sh\n+ - run: make check-docs\n+ - run: \"make doc > >(tee stdout.log) 2> >(tee stderr.raw >&2)\"\n+ shell: bash\n+ - run: ci/test-documentation.sh AsciiDoc\n+ if: success()\n+ - run: ci/check-unignored-build-artifacts.sh\n+ if: success()\n+ - run: make clean\n+ - run: \"make USE_ASCIIDOCTOR=1 doc > >(tee stdout.log) 2> >(tee stderr.raw >&2)\"\n+ shell: bash\n+ - run: ci/test-documentation.sh Asciidoctor\n+ if: success()\n+ - run: ci/check-unignored-build-artifacts.sh\n+ if: success()\ndiff --git a/ci/check-unignored-build-artifacts.sh b/ci/check-unignored-build-artifacts.sh\nindex 56d04b0db9a..0bc04f32804 100755\n--- a/ci/check-unignored-build-artifacts.sh\n+++ b/ci/check-unignored-build-artifacts.sh\n@@ -5,4 +5,14 @@\n \n . ${0%/*}/lib.sh\n \n+check_unignored_build_artifacts ()\n+{\n+\t! git ls-files --other --exclude-standard --error-unmatch \\\n+\t\t-- ':/*' 2>/dev/null ||\n+\t{\n+\t\techo \"$(tput setaf 1)error: found unignored build artifacts$(tput sgr0)\"\n+\t\tfalse\n+\t}\n+}\n+\n check_unignored_build_artifacts\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 05f3dd15e27..fc6ce4d3e04 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -39,16 +39,6 @@ setenv () {\n \tfi\n }\n \n-check_unignored_build_artifacts ()\n-{\n-\t! git ls-files --other --exclude-standard --error-unmatch \\\n-\t\t-- ':/*' 2>/dev/null ||\n-\t{\n-\t\techo \"$(tput setaf 1)error: found unignored build artifacts$(tput sgr0)\"\n-\t\tfalse\n-\t}\n-}\n-\n # GitHub Action doesn't set TERM, which is required by tput\n setenv TERM ${TERM:-dumb}\n \ndiff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\nindex 6b5cce03bd7..b8a6a6f664e 100755\n--- a/ci/test-documentation.sh\n+++ b/ci/test-documentation.sh\n@@ -1,10 +1,12 @@\n-#!/usr/bin/env bash\n+#!/bin/sh\n #\n-# Perform sanity checks on documentation and build it.\n+# Perform sanity checks on \"make doc\" output and built documentation\n #\n \n . ${0%/*}/lib.sh\n \n+generator=$1\n+\n filter_log () {\n \tsed -e '/^GIT_VERSION = /d' \\\n \t -e \"/constant Gem::ConfigMap is deprecated/d\" \\\n@@ -14,28 +16,15 @@ filter_log () {\n \t \"$1\"\n }\n \n-make check-docs\n-\n-# Build docs with AsciiDoc\n-make doc > >(tee stdout.log) 2> >(tee stderr.raw >&2)\n-cat stderr.raw\n-filter_log stderr.raw >stderr.log\n-test ! -s stderr.log\n-test -s Documentation/git.html\n-test -s Documentation/git.xml\n-test -s Documentation/git.1\n-grep ' >(tee stdout.log) 2> >(tee stderr.raw >&2)\n cat stderr.raw\n filter_log stderr.raw >stderr.log\n test ! -s stderr.log\n test -s Documentation/git.html\n-grep '\n---\n .github/workflows/main.yml | 10 ++++++++--\n ci/{run-test-slice.sh => select-test-slice.sh} | 4 ++--\n 2 files changed, 10 insertions(+), 4 deletions(-)\n rename ci/{run-test-slice.sh => select-test-slice.sh} (57%)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex fe37e0f1b36..1e13718b989 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -124,9 +124,12 @@ jobs:\n - uses: git-for-windows/setup-git-for-windows-sdk@v1\n - run: ci/lib.sh\n shell: bash\n+ - name: select tests\n+ run: . /etc/profile && ci/select-test-slice.sh ${{matrix.nr}} 10\n+ shell: bash\n - name: test\n shell: bash\n- run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10\n+ run: . /etc/profile && make -C t -e\n - name: ci/print-test-failures.sh\n if: failure()\n shell: bash\n@@ -210,9 +213,12 @@ jobs:\n run: tar xf artifacts.tar.gz && tar xf tracked.tar.gz\n - run: ci/lib.sh\n shell: bash\n+ - name: select tests\n+ run: . /etc/profile && ci/select-test-slice.sh ${{matrix.nr}} 10\n+ shell: bash\n - name: test\n shell: bash\n- run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10\n+ run: . /etc/profile && make -C t -e\n - name: ci/print-test-failures.sh\n if: failure()\n shell: bash\ndiff --git a/ci/run-test-slice.sh b/ci/select-test-slice.sh\nsimilarity index 57%\nrename from ci/run-test-slice.sh\nrename to ci/select-test-slice.sh\nindex 1de6a18ca47..ec602f8a053 100755\n--- a/ci/run-test-slice.sh\n+++ b/ci/select-test-slice.sh\n@@ -1,10 +1,10 @@\n #!/bin/sh\n #\n-# Test Git in parallel\n+# Select a portion of the tests for testing Git in parallel\n #\n \n . ${0%/*}/lib.sh\n \n tests=$(echo $(cd t && ./helper/test-tool path-utils slice-tests \"$1\" \"$2\" \\\n \tt[0-9]*.sh))\n-make --quiet -C t T=\"$tests\"\n+echo T=\"$tests\" >>$GITHUB_ENV\n-- \n2.35.1.1517.g20a06c426a7\n\n"},{"id":"452322","messageId":"patch-v2-16.25-ba4ed216769-20220325T182534Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-v2-00.25-00000000000-20220325T182534Z-avarab@gmail.com","subject":"[PATCH v2 16/25] CI: combine ci/install{,-docker}-dependencies.sh","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-25T18:38:07Z","receivedAt":"2022-03-25T19:29:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Combine the \"ci/install-dependencies.sh\" and\n\"ci/install-docker-dependencies.sh\" into one script. The \"case\"\nstatement in the latter only cared about \"$jobname\", and can be folded\ninto the same \"case\" statement in the former.\n\nThe reason they split up is historical, and because the\n\"ci/install-dependencies.sh\" used \"ci/lib.sh\", which requires\n\"bash\". At least one of the docker containers doesn't have \"bash\". To\nmake the existing code POSIX-compatible we need to replace pushd/popd\nwith a sub-shell, but no other changes were needed.\n\nThis also fixes issue in \"ci/install-docker-dependencies.sh\" where\nwe'd hide errors due to not using \"set -e\". Now that we include\n\"ci/lib.sh\" we won't have that potential issue.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .github/workflows/main.yml | 2 +-\n ci/install-dependencies.sh | 27 ++++++++++++++++++++++-----\n ci/install-docker-dependencies.sh | 22 ----------------------\n 3 files changed, 23 insertions(+), 28 deletions(-)\n delete mode 100755 ci/install-docker-dependencies.sh\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 92b914f16fd..3fd3e4f873c 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -311,7 +311,7 @@ jobs:\n container: ${{matrix.vector.image}}\n steps:\n - uses: actions/checkout@v1\n- - run: ci/install-docker-dependencies.sh\n+ - run: ci/install-dependencies.sh\n - run: ci/lib.sh\n - run: make\n - run: make test\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 01006379fd0..9388289d0ed 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -1,4 +1,4 @@\n-#!/usr/bin/env bash\n+#!/bin/sh\n #\n # Install dependencies required to build and test Git on Linux and macOS\n #\n@@ -34,18 +34,20 @@ ubuntu-latest)\n \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n \t\t$UBUNTU_COMMON_PKGS $CC_PACKAGE\n \tmkdir --parents \"$P4_PATH\"\n-\tpushd \"$P4_PATH\"\n+\t(\n+\t\tcd \"$P4_PATH\"\n \t\twget --quiet \"$P4WHENCE/bin.linux26x86_64/p4d\"\n \t\twget --quiet \"$P4WHENCE/bin.linux26x86_64/p4\"\n \t\tchmod u+x p4d\n \t\tchmod u+x p4\n-\tpopd\n+\t)\n \tmkdir --parents \"$GIT_LFS_PATH\"\n-\tpushd \"$GIT_LFS_PATH\"\n+\t(\n+\t\tcd \"$GIT_LFS_PATH\"\n \t\twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \t\ttar --extract --gunzip --file \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \t\tcp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs .\n-\tpopd\n+\t)\n \t;;\n macos-latest)\n \texport HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1\n@@ -90,6 +92,21 @@ linux-gcc-default)\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install $UBUNTU_COMMON_PKGS\n \t;;\n+linux32)\n+\tlinux32 --32bit i386 sh -c '\n+\t\tapt update >/dev/null &&\n+\t\tapt install -y build-essential libcurl4-openssl-dev \\\n+\t\t\tlibssl-dev libexpat-dev gettext python >/dev/null\n+\t'\n+\t;;\n+linux-musl)\n+\tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n+\t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n+\t;;\n+pedantic)\n+\tdnf -yq update >/dev/null &&\n+\tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n+\t;;\n esac\n \n if type p4d >/dev/null && type p4 >/dev/null\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\ndeleted file mode 100755\nindex 78b7e326da6..00000000000\n--- a/ci/install-docker-dependencies.sh\n+++ /dev/null\n@@ -1,22 +0,0 @@\n-#!/bin/sh\n-#\n-# Install dependencies required to build and test Git inside container\n-#\n-\n-case \"$jobname\" in\n-linux32)\n-\tlinux32 --32bit i386 sh -c '\n-\t\tapt update >/dev/null &&\n-\t\tapt install -y build-essential libcurl4-openssl-dev \\\n-\t\t\tlibssl-dev libexpat-dev gettext python >/dev/null\n-\t'\n-\t;;\n-linux-musl)\n-\tapk add --update build-base curl-dev openssl-dev expat-dev gettext \\\n-\t\tpcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null\n-\t;;\n-pedantic)\n-\tdnf -yq update >/dev/null &&\n-\tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n-\t;;\n-esac\n-- \n2.35.1.1517.g20a06c426a7\n\n"},{"id":"452324","messageId":"patch-v2-08.25-81e06f13d84-20220325T182534Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-v2-00.25-00000000000-20220325T182534Z-avarab@gmail.com","subject":"[PATCH v2 08/25] CI: have \"static-analysis\" run \"check-builtins\", not \"documentation\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-25T18:37:59Z","receivedAt":"2022-03-25T19:29:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the \"make check-builtins\" check from the \"documentation\" job to\nthe \"static-analysis\" job.\n\nThe \"check-builtins\" target added in c74390e4a1d (cherry is built-in,\ndo not ship git-cherry.sh, 2006-11-05) is unrelated to the\ndocumentation, so it's odd that b98712b9aa9 (travis-ci: build\ndocumentation, 2016-05-04) added it to the \"documentation\" job.\n\nLet's just move it to the \"ci-static-analysis\" target, and while we're\nat it improve the output with $(QUIET_CHECK).\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Makefile | 3 ++-\n ci/test-documentation.sh | 1 -\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex d30862a23a2..82f4206188a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -3378,7 +3378,7 @@ check-docs::\n ### Make sure built-ins do not have dups and listed in git.c\n #\n check-builtins::\n-\t./check-builtins.sh\n+\t$(QUIET_CHECK)./check-builtins.sh\n \n ### Test suite coverage testing\n #\n@@ -3468,5 +3468,6 @@ ci-check-directional-formatting:\n \n .PHONY: ci-static-analysis\n ci-static-analysis: ci-check-directional-formatting\n+ci-static-analysis: check-builtins\n ci-static-analysis: check-coccicheck\n ci-static-analysis: hdr-check\ndiff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\nindex 9e0652c30dd..41e2b126311 100755\n--- a/ci/test-documentation.sh\n+++ b/ci/test-documentation.sh\n@@ -14,7 +14,6 @@ filter_log () {\n \t \"$1\"\n }\n \n-make check-builtins\n make check-docs\n \n # Build docs with AsciiDoc\n-- \n2.35.1.1517.g20a06c426a7\n\n"},{"id":"452325","messageId":"patch-v2-02.25-b23aa99fd37-20220325T182534Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-v2-00.25-00000000000-20220325T182534Z-avarab@gmail.com","subject":"[PATCH v2 02/25] CI: make \"$jobname\" explicit, remove fallback","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-25T18:37:53Z","receivedAt":"2022-03-25T19:29:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Remove the $jobname fallback last adjusted in b011fabd6e0 (ci/lib.sh:\nencapsulate Travis-specific things, 2019-01-27).\n\nThis makes all of our CI jobs have an explicit $jobname, and removes\nthe special-case of the Windows jobs implicitly setting one (that they\ndon't currently use). In subsequent commits we'll use this new\n$jobname explicitly.\n\nThis is a follow-up to my 25715419bf4 (CI: don't run \"make test\" twice\nin one job, 2021-11-23), since then all the jobs except the Windows\njobs have had an explicit $jobname.\n\nIn the case of the Windows jobs we were using a jobname of windows-gcc\nfor both, due to picking \"gcc\" as a default fallback for \"$CC\" in\nci/lib.sh.\n\nSince we'll need this to run ci/lib.sh it makes sense to move the\ncheck to the top of the file. Subsequent commits will add more such\nassertions to it.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .github/workflows/main.yml | 8 ++++++++\n ci/lib.sh | 12 +++++++-----\n 2 files changed, 15 insertions(+), 5 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex c35200defb9..309b9141249 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -79,6 +79,8 @@ jobs:\n \n windows-build:\n name: win build\n+ env:\n+ jobname: windows-build\n needs: ci-config\n if: needs.ci-config.outputs.enabled == 'yes'\n runs-on: windows-latest\n@@ -100,6 +102,8 @@ jobs:\n path: artifacts\n windows-test:\n name: win test\n+ env:\n+ jobname: windows-test\n runs-on: windows-latest\n needs: [windows-build]\n strategy:\n@@ -131,6 +135,8 @@ jobs:\n path: ${{env.FAILED_TEST_ARTIFACTS}}\n vs-build:\n name: win+VS build\n+ env:\n+ jobname: vs-build\n needs: ci-config\n if: needs.ci-config.outputs.enabled == 'yes'\n env:\n@@ -183,6 +189,8 @@ jobs:\n path: artifacts\n vs-test:\n name: win+VS test\n+ env:\n+ jobname: vs-test\n runs-on: windows-latest\n needs: vs-build\n strategy:\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 57141d38a85..d2a7c33f536 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -6,6 +6,13 @@\n # and installing dependencies.\n set -ex\n \n+# Starting assertions\n+if test -z \"$jobname\"\n+then\n+\techo \"must set a CI jobname\" >&2\n+\texit 1\n+fi\n+\n skip_branch_tip_with_tag () {\n \t# Sometimes, a branch is pushed at the same time the tag that points\n \t# at the same commit as the tip of the branch is pushed, and building\n@@ -146,11 +153,6 @@ test -n \"${DONT_SKIP_TAGS-}\" ||\n skip_branch_tip_with_tag\n skip_good_tree\n \n-if test -z \"$jobname\"\n-then\n-\tjobname=\"$CI_OS_NAME-$CC\"\n-fi\n-\n export DEVELOPER=1\n export DEFAULT_TEST_TARGET=prove\n export GIT_TEST_CLONE_2GB=true\n-- \n2.35.1.1517.g20a06c426a7\n\n"},{"id":"452327","messageId":"patch-v2-13.25-649ad1ae249-20220325T182534Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-v2-00.25-00000000000-20220325T182534Z-avarab@gmail.com","subject":"[PATCH v2 13/25] CI: check ignored unignored build artifacts in \"win[+VS] build\" too","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-25T18:38:04Z","receivedAt":"2022-03-25T19:29:30Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Use the newly created \"ci/check-unignored-build-artifacts.sh\" script\nin the \"win build\" and \"win+VS build\" steps too. These steps weren't\ndoing this sanity check of untracked files against .gitignore before,\nbut as the check passes for them let's keep doing that extra sanity\ncheck.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .github/workflows/main.yml | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex c83f6a21ca3..05c5771e7f6 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -93,6 +93,9 @@ jobs:\n HOME: ${{runner.workspace}}\n NO_PERL: 1\n run: . /etc/profile && ci/make-test-artifacts.sh artifacts\n+ - run: ci/check-unignored-build-artifacts.sh\n+ if: success()\n+ shell: bash\n - name: zip up tracked files\n run: git archive -o artifacts/tracked.tar.gz HEAD\n - name: upload tracked files and build artifacts\n@@ -180,6 +183,9 @@ jobs:\n run: |\n mkdir -p artifacts &&\n eval \"$(make -n artifacts-tar INCLUDE_DLLS_IN_ARTIFACTS=YesPlease ARTIFACTS_DIRECTORY=artifacts NO_GETTEXT=YesPlease 2>&1 | grep ^tar)\"\n+ - run: ci/check-unignored-build-artifacts.sh\n+ if: success()\n+ shell: bash\n - name: zip up tracked files\n run: git archive -o artifacts/tracked.tar.gz HEAD\n - name: upload tracked files and build artifacts\n-- \n2.35.1.1517.g20a06c426a7\n\n"},{"id":"452329","messageId":"patch-v2-17.25-b5e89a33340-20220325T182534Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-v2-00.25-00000000000-20220325T182534Z-avarab@gmail.com","subject":"[PATCH v2 17/25] CI: move \"env\" definitions into ci/lib.sh","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-25T18:38:08Z","receivedAt":"2022-03-25T19:30:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Have the GitHub CI jobs use the \"ci/lib\".sh as a source of truth for\nenvironment variables that affect their builds and tests. This brings\nthem in line with the rest of the jobs, and make it easier to see the\nentire CI configuration per jobname at a glance.\n\nTo do this we need to add a ci/lib.sh \"step\" to the remaining jobs\nthat didn't have it added in preceding commits.\n\nThe Makefile parameters are luckily all accepted via the environment,\nso we can export these instead of adding them to MAKEFLAGS.\n\nLet's also use the documented GIT_CONIFG_* mechanism for setting\nconfig instead of the internal GIT_CONFIG_PARAMETERS variable. This\nadjusts code added in 889cacb6897 (ci: configure GitHub Actions for\nCI/PR, 2020-04-11), we could probably use the GIT_AUTHOR_NAME etc, but\nlet's keep setting this via config, just with the documented\nmechanism.\n\nBy setting \"ARTIFACTS_DIRECTORY=artifacts\" once we don't need to\nrepeat it in various places, let's get it from the environment\ninstead.\n\nFinally, the \"DEVELOPER: 1\" was only needed by the \"sparse\" job, which\nwasn't using \"ci/lib.sh\" (and which sets DEVELOPER=1). Let's instead\nhave the \"sparse\" job use the \"ci/lib.sh\" and remove DEVELOPER=1 from\n\".github/workflows/main.yml\".\n\nThis substantially speeds up the \"sparse\" job, since it'll now pick up\nthe \"--jobs\" setting in MAKEFLAGS that we use everywhere else. Before\nit ran in around 4m30s, now in around 2m30s.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .github/workflows/main.yml | 29 +++++++++++------------------\n ci/lib.sh | 21 +++++++++++++++++++++\n 2 files changed, 32 insertions(+), 18 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 3fd3e4f873c..fe37e0f1b36 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -2,9 +2,6 @@ name: CI\n \n on: [push, pull_request]\n \n-env:\n- DEVELOPER: 1\n-\n jobs:\n ci-config:\n name: config\n@@ -93,8 +90,7 @@ jobs:\n shell: bash\n env:\n HOME: ${{runner.workspace}}\n- NO_PERL: 1\n- run: . /etc/profile && make artifacts-tar ARTIFACTS_DIRECTORY=artifacts\n+ run: . /etc/profile && make artifacts-tar\n if: success()\n - run: ci/check-unignored-build-artifacts.sh\n if: success()\n@@ -105,7 +101,7 @@ jobs:\n uses: actions/upload-artifact@v2\n with:\n name: windows-artifacts\n- path: artifacts\n+ path: ${{env.ARTIFACTS_DIRECTORY}}\n windows-test:\n name: win test\n env:\n@@ -126,6 +122,8 @@ jobs:\n shell: bash\n run: tar xf artifacts.tar.gz && tar xf tracked.tar.gz\n - uses: git-for-windows/setup-git-for-windows-sdk@v1\n+ - run: ci/lib.sh\n+ shell: bash\n - name: test\n shell: bash\n run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10\n@@ -145,9 +143,6 @@ jobs:\n jobname: vs-build\n needs: ci-config\n if: needs.ci-config.outputs.enabled == 'yes'\n- env:\n- NO_PERL: 1\n- GIT_CONFIG_PARAMETERS: \"'user.name=CI' 'user.email=ci@git'\"\n runs-on: windows-latest\n steps:\n - uses: actions/checkout@v2\n@@ -171,6 +166,8 @@ jobs:\n - name: copy dlls to root\n shell: cmd\n run: compat\\vcbuild\\vcpkg_copy_dlls.bat release\n+ - run: ci/lib.sh\n+ shell: bash\n - name: generate Visual Studio solution\n shell: bash\n run: |\n@@ -181,21 +178,16 @@ jobs:\n - name: bundle artifact tar\n shell: bash\n env:\n- MSVC: 1\n VCPKG_ROOT: ${{github.workspace}}\\compat\\vcbuild\\vcpkg\n run: |\n- mkdir -p artifacts &&\n- eval \"$(make -n artifacts-tar INCLUDE_DLLS_IN_ARTIFACTS=YesPlease ARTIFACTS_DIRECTORY=artifacts NO_GETTEXT=YesPlease 2>&1 | grep ^tar)\"\n- - run: ci/check-unignored-build-artifacts.sh\n- if: success()\n- shell: bash\n+ eval \"$(make -n artifacts-tar 2>&1 | grep -e ^mkdir -e ^tar)\"\n - name: zip up tracked files\n run: git archive -o artifacts/tracked.tar.gz HEAD\n - name: upload tracked files and build artifacts\n uses: actions/upload-artifact@v2\n with:\n name: vs-artifacts\n- path: artifacts\n+ path: ${{env.ARTIFACTS_DIRECTORY}}\n vs-test:\n name: win+VS test\n env:\n@@ -216,10 +208,10 @@ jobs:\n - name: extract tracked files and build artifacts\n shell: bash\n run: tar xf artifacts.tar.gz && tar xf tracked.tar.gz\n+ - run: ci/lib.sh\n+ shell: bash\n - name: test\n shell: bash\n- env:\n- NO_SVN_TESTS: 1\n run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10\n - name: ci/print-test-failures.sh\n if: failure()\n@@ -354,6 +346,7 @@ jobs:\n - uses: actions/checkout@v2\n - name: Install other dependencies\n run: ci/install-dependencies.sh\n+ - run: ci/lib.sh\n - run: make sparse\n documentation:\n name: documentation\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex fc6ce4d3e04..b882849ed5d 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -95,6 +95,27 @@ macos-latest)\n esac\n \n case \"$jobname\" in\n+windows-build)\n+\tsetenv --build NO_PERL NoThanks\n+\tsetenv --build ARTIFACTS_DIRECTORY artifacts\n+\t;;\n+vs-build)\n+\tsetenv --build NO_PERL NoThanks\n+\tsetenv --build NO_GETTEXT NoThanks\n+\tsetenv --build ARTIFACTS_DIRECTORY artifacts\n+\tsetenv --build INCLUDE_DLLS_IN_ARTIFACTS YesPlease\n+\tsetenv --build MSVC YesPlease\n+\n+\tsetenv --build GIT_CONFIG_COUNT 2\n+\tsetenv --build GIT_CONFIG_KEY_0 user.name\n+\tsetenv --build GIT_CONFIG_VALUE_0 CI\n+\tsetenv --build GIT_CONFIG_KEY_1 user.emailname\n+\tsetenv --build GIT_CONFIG_VALUE_1 ci@git\n+\tsetenv --build GIT_CONFIG_VALUE_1 ci@git\n+\t;;\n+vs-test)\n+\tsetenv --test NO_SVN_TESTS YesPlease\n+\t;;\n linux-gcc)\n \tsetenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME main\n \t;;\n-- \n2.35.1.1517.g20a06c426a7\n\n"},{"id":"452330","messageId":"patch-v2-24.25-06bf8d9f61b-20220325T182534Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-v2-00.25-00000000000-20220325T182534Z-avarab@gmail.com","subject":"[PATCH v2 24/25] CI: set PYTHON_PATH setting for osx-{clang,gcc} into \"$jobname\" case","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-25T18:38:15Z","receivedAt":"2022-03-25T19:30:16Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Move the setting of \"PYTHON_PATH=[...]\" to the \"$jobname\" case\nstatement. This partially backs out of my 707d2f2fe86 (CI: use\n\"$runs_on_pool\", not \"$jobname\" to select packages & config,\n2021-11-23), now that we have a \"osx-{clang,gcc}\" anyway for setting\n\"$CC\" we might as well do away with this part of the \"$runs_on_pool\"\ncase.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n ci/lib.sh | 10 ++--------\n 1 file changed, 2 insertions(+), 8 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 57c9117a35a..663cbdef9d1 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -99,14 +99,6 @@ ubuntu-latest)\n \n \tsetenv --test GIT_TEST_HTTPD true\n \t;;\n-macos-latest)\n-\tif [ \"$jobname\" = osx-gcc ]\n-\tthen\n-\t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python3)\"\n-\telse\n-\t\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python2)\"\n-\tfi\n-\t;;\n esac\n \n case \"$jobname\" in\n@@ -160,10 +152,12 @@ linux-TEST-vars)\n \tsetenv --test GIT_TEST_CHECKOUT_WORKERS 2\n \t;;\n osx-gcc)\n+\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python3)\"\n \tCC=gcc\n \tCC_PACKAGE=gcc-9\n \t;;\n osx-clang)\n+\tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python2)\"\n \tCC=clang\n \t;;\n linux-clang)\n-- \n2.35.1.1517.g20a06c426a7\n\n"},{"id":"452333","messageId":"patch-v2-05.25-b5e6d538554-20220325T182534Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-v2-00.25-00000000000-20220325T182534Z-avarab@gmail.com","subject":"[PATCH v2 05/25] CI: remove unused Azure ci/* code","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-25T18:37:56Z","receivedAt":"2022-03-25T19:32:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Remove Azure-specific code that's been unused since 6081d3898fe (ci:\nretire the Azure Pipelines definition, 2020-04-11). As noted in a\nlarger removal of all of the Azure-supporting code in [1] (although\nthat missed some of this) there is more of it in-tree, but let's focus\non only the ci/* code for now.\n\nThis is needed because in subsequent commits this unused code would\neither need to be changed to accommodate changes in ci/*, or be\nremoved.\n\nAs we'll see in those subsequent commits the end-state we're heading\ntowards will actually make it easier to add new CI types in the\nfuture, even though the only one we're left with now is the GitHub\nCI. I.e. the \"ci/*\" framework will be made to do less, not more. We'll\nbe offloading more of what it does to our generic build and test\nsystem.\n\nWhile I'm at it (since the line needs to be touched anyway) change an\nodd 'if test true == \"$GITHUB_ACTIONS\"' to the more usual style used\nin other places of 'if test \"$GITHUB_ACTIONS\" = \"true\"'.\n\n1. https://lore.kernel.org/git/patch-1.1-eec0a8c3164-20211217T000418Z-avarab@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n ci/install-dependencies.sh | 3 ---\n ci/lib.sh | 13 +------------\n ci/mount-fileshare.sh | 25 -------------------------\n ci/print-test-failures.sh | 7 +------\n 4 files changed, 2 insertions(+), 46 deletions(-)\n delete mode 100755 ci/mount-fileshare.sh\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex dbcebad2fb2..e7ea8799411 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -34,8 +34,6 @@ macos-latest)\n \texport HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1\n \t# Uncomment this if you want to run perf tests:\n \t# brew install gnu-time\n-\ttest -z \"$BREW_INSTALL_PACKAGES\" ||\n-\tbrew install $BREW_INSTALL_PACKAGES\n \tbrew link --force gettext\n \tbrew install --cask --no-quarantine perforce || {\n \t\t# Update the definitions and try again\n@@ -69,7 +67,6 @@ Documentation)\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install asciidoc xmlto docbook-xsl-ns make\n \n-\ttest -n \"$ALREADY_HAVE_ASCIIDOCTOR\" ||\n \tsudo gem install --version 1.5.8 asciidoctor\n \t;;\n linux-gcc-default)\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 5a8938d4b3c..ad1aa59a384 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -29,18 +29,7 @@ export TERM=${TERM:-dumb}\n # Clear MAKEFLAGS that may come from the outside world.\n export MAKEFLAGS=\n \n-if test -n \"$SYSTEM_COLLECTIONURI\" || test -n \"$SYSTEM_TASKDEFINITIONSURI\"\n-then\n-\tCI_TYPE=azure-pipelines\n-\t# We are running in Azure Pipelines\n-\tCC=\"${CC:-gcc}\"\n-\n-\texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n-\texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n-\tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n-\ttest Windows_NT != \"$AGENT_OS\" ||\n-\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n-elif test true = \"$GITHUB_ACTIONS\"\n+if test \"$GITHUB_ACTIONS\" = \"true\"\n then\n \tCI_TYPE=github-actions\n \tCC=\"${CC:-gcc}\"\ndiff --git a/ci/mount-fileshare.sh b/ci/mount-fileshare.sh\ndeleted file mode 100755\nindex 26b58a80960..00000000000\n--- a/ci/mount-fileshare.sh\n+++ /dev/null\n@@ -1,25 +0,0 @@\n-#!/bin/sh\n-\n-die () {\n-\techo \"$*\" >&2\n-\texit 1\n-}\n-\n-test $# = 4 ||\n-die \"Usage: $0 \"\n-\n-mkdir -p \"$4\" || die \"Could not create $4\"\n-\n-case \"$(uname -s)\" in\n-Linux)\n-\tsudo mount -t cifs -o vers=3.0,username=\"$2\",password=\"$3\",dir_mode=0777,file_mode=0777,serverino \"$1\" \"$4\"\n-\t;;\n-Darwin)\n-\tpass=\"$(echo \"$3\" | sed -e 's/\\//%2F/g' -e 's/+/%2B/g')\" &&\n-\tmount -t smbfs,soft \"smb://$2:$pass@${1#//}\" \"$4\"\n-\t;;\n-*)\n-\tdie \"No support for $(uname -s)\"\n-\t;;\n-esac ||\n-die \"Could not mount $4\"\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex 00fcf27b1b2..0c63b6f7962 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -38,14 +38,9 @@ do\n \t\ttest_name=\"${TEST_EXIT%.exit}\"\n \t\ttest_name=\"${test_name##*/}\"\n \t\ttrash_dir=\"trash directory.$test_name\"\n+\t\tmkdir -p failed-test-artifacts\n \t\tcase \"$CI_TYPE\" in\n-\t\tazure-pipelines)\n-\t\t\tmkdir -p failed-test-artifacts\n-\t\t\tmv \"$trash_dir\" failed-test-artifacts\n-\t\t\tcontinue\n-\t\t\t;;\n \t\tgithub-actions)\n-\t\t\tmkdir -p failed-test-artifacts\n \t\t\techo \"FAILED_TEST_ARTIFACTS=t/failed-test-artifacts\" >>$GITHUB_ENV\n \t\t\tcp \"${TEST_EXIT%.exit}.out\" failed-test-artifacts/\n \t\t\ttar czf failed-test-artifacts/\"$test_name\".trash.tar.gz \"$trash_dir\"\n-- \n2.35.1.1517.g20a06c426a7\n\n"},{"id":"452335","messageId":"patch-v2-25.25-6dc96ba8b82-20220325T182534Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-v2-00.25-00000000000-20220325T182534Z-avarab@gmail.com","subject":"[PATCH v2 25/25] CI: don't use \"set -x\" in \"ci/lib.sh\" output","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-25T18:38:16Z","receivedAt":"2022-03-25T19:32:11Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Remove the \"set -x\" from the \"ci/lib.sh\" output. Before preceding\ncommits the logic in that file was much more complex, and likely to\nfail in some scenarios.\n\nNow we only task \"ci/lib.sh\" with setting various variables for\nsubsequent steps in our jobs, so we can start emitting more tailored\ndebugging output, which makes what it's doing easier to read.\n\nThis change also changes the output of the \"ci/print-test-failures.sh\"\nscript, since it's the only other user of \"ci/lib-ci-type.sh\". In that\ncase it's also helpful to know what \"$CI_TYPE\" we're using, as that\nscript doesn't \"set -x\" and will act differently depending on the\n$CI_TYPE.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n ci/lib-ci-type.sh | 2 ++\n ci/lib.sh | 9 ++++++++-\n 2 files changed, 10 insertions(+), 1 deletion(-)\n\ndiff --git a/ci/lib-ci-type.sh b/ci/lib-ci-type.sh\nindex bd6e093c8f4..6cfe58596d2 100644\n--- a/ci/lib-ci-type.sh\n+++ b/ci/lib-ci-type.sh\n@@ -6,3 +6,5 @@ else\n \tenv >&2\n \texit 1\n fi\n+\n+echo \"CONFIG: CI_TYPE=$CI_TYPE\" >&2\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 663cbdef9d1..304380958bd 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -1,5 +1,5 @@\n #!/bin/sh\n-set -ex\n+set -e\n \n # Helper libraries\n . ${0%/*}/lib-ci-type.sh\n@@ -11,12 +11,16 @@ then\n \techo \"need a $0 mode, e.g. --build or --test\"\n \texit 1\n fi\n+echo \"CONFIG: mode=$mode\" >&2\n \n if test -z \"$jobname\"\n then\n \techo \"must set a CI jobname\" >&2\n \texit 1\n fi\n+echo \"CONFIG: jobname=$jobname\" >&2\n+echo \"CONFIG: runs_on_pool=$runs_on_pool\" >&2\n+echo \"CONFIG: GITHUB_ENV=$GITHUB_ENV\" >&2\n \n # Helper functions\n setenv () {\n@@ -39,6 +43,7 @@ setenv () {\n \n \tif test -n \"$skip\"\n \tthen\n+\t\techo \"SKIP '$key=$val'\" >&2\n \t\treturn 0\n \tfi\n \n@@ -50,6 +55,8 @@ setenv () {\n \t\t# itself.\n \t\teval \"export $key=\\\"$val\\\"\"\n \tfi\n+\n+\techo \"SET: '$key=$val'\" >&2\n }\n \n # Clear variables that may come from the outside world.\n-- \n2.35.1.1517.g20a06c426a7\n\n"},{"id":"452341","messageId":"patch-v2-12.25-86d5a48d59a-20220325T182534Z-avarab@gmail.com","threadId":"57451","inReplyTo":"cover-v2-00.25-00000000000-20220325T182534Z-avarab@gmail.com","subject":"[PATCH v2 12/25] CI: remove \"run-build-and-tests.sh\", run \"make [test]\" directly","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-25T18:38:03Z","receivedAt":"2022-03-25T19:32:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Remove the already thin \"ci/run-build-and-tests.sh\" wrapper and\ninstead make the CI run \"make\" or \"make test\" directly. By doing this\nwe'll be able to easily see at a glance whether our failure was in the\ncompilation or testing, whether that's via human eyes or improve\nmachine readability.\n\nWe also need to run our new \"ci/check-unignored-build-artifacts.sh\" on\nsuccess() in the CI now, just like we already had a step conditional\non failure() running ci/print-test-failures.sh.\n\nThe reason we used a \"ci/run-build-and-tests.sh\" wrapper in the first\nplace had to do with Travis CI-specific constraints that no longer\napply to us, as the Travis CI support has been removed.\n\nInstead we can configure the CI in an earlier step by running\n\"ci/lib.sh\", which under GitHub CI will write the environment\nvariables we need to the \"$GITHUB_ENV\" file.\n\nWe'll then have access to them in subsequent steps, and crucially\nthose variables will be prominently visible at the start of each step\nvia an expandable drop-down in the UI.drop-do.\n\nI.e. this changes the CI run from a top-down flow like (pseudocode):\n\n - job:\n - step1:\n - use ci/lib.sh to set env vars\n - run a script like ci/run-build-and-tests.sh\n - step2:\n - if: failure()\n - use ci/lib.sh to set env vars\n - run ci/print-test-failures.sh\n\nTo:\n\n - job:\n - step1:\n - set variables in $GITHUB_ENV using ci/lib.sh\n - step2:\n - make\n - step3:\n - make test\n - step4:\n - if: failure()\n - run ci/print-test-failures.sh\n - step5:\n - if: success()\n - run ci/check-unignored-build-artifacts.sh\n\nThere is a proposal[2] to get some of the benefits of this approach by\nnot re-arranging our variable setup in this way, but to instead use\nthe GitHub CI grouping syntax to focus on the relevant parts of \"make\"\nor \"make test\" when we have failures.\n\nDoing it this way makes for better looking GitHub CI UI, and lays much\nbetter ground work for our CI going forward. Because:\n\n * The CI logic will be more portable to a future CI system, since a\n common feature of them is to run various commands in sequence, but\n a future system won't necessarily support the GitHub-specifics\n syntax of \"grouping\" output within a \"step\".\n\n Even if those systems don't support a \"$GITHUB_ENV\" emulating will\n be much easier than to deal with some CI-specific grouping syntax.\n\n * At the start of every step the GitHub CI presents an expandable\n list of environment variables from \"$GITHUB_ENV\". We'll now see\n exactly what variables affected that step (although we currently\n overshoot that a bit, and always define all variables).\n\n * CI failures will be easier to reproduce locally, as this makes the\n relevant ci/* scripts something that sets up our environment, but\n leaves \"make\" and \"make test\" working as they do locally.\n\n To reproduce a run the user only needs to set the variables\n discussed in the drop-down above, either manually or by running\n \"ci/lib.sh\".\n\n * The output will be less verbose. The \"ci/lib.sh\" script uses \"set\n -x\", and before this e.g. \"ci/static-analysis.sh\" would start with\n 40 lines of trace output, culminating in using \"export\" to export\n the relevant environment variables.\n\n Now that verbosity is in the earlier \"ci/lib.sh\" step, and not in\n any subsequent one. The \"make\" targets then start out with the\n relevant output non-trace output right away.\n\n * If we do want to use the grouping syntax within a \"step\" it'll now\n be easier to do so. It doesn't support nesting, so we'd have to\n make a choice between using it for e.g. \"make\" v.s. \"make test\", or\n individual test failures. See \"sadly\" in [3].\n\n1. https://lore.kernel.org/git/211120.86k0h30zuw.gmgdl@evledraar.gmail.com/\n2. https://lore.kernel.org/git/pull.1117.git.1643050574.gitgitgadget@gmail.com/\n3. https://lore.kernel.org/git/9333ba781b8240f704e739b00d274f8c3d887e39.1643050574.git.gitgitgadget@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n .github/workflows/main.yml | 14 ++++++--\n ci/check-unignored-build-artifacts.sh | 8 +++++\n ci/install-dependencies.sh | 4 +++\n ci/lib.sh | 40 ++++++++++++++++++++++-\n ci/make-test-artifacts.sh | 2 --\n ci/run-build-and-tests.sh | 47 ---------------------------\n ci/run-test-slice.sh | 2 --\n ci/test-documentation.sh | 1 -\n 8 files changed, 62 insertions(+), 56 deletions(-)\n create mode 100755 ci/check-unignored-build-artifacts.sh\n delete mode 100755 ci/run-build-and-tests.sh\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex ad752010102..c83f6a21ca3 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -268,7 +268,10 @@ jobs:\n steps:\n - uses: actions/checkout@v2\n - run: ci/install-dependencies.sh\n- - run: ci/run-build-and-tests.sh\n+ - run: ci/lib.sh\n+ - run: make\n+ - run: make test\n+ if: success()\n - run: ci/print-test-failures.sh\n if: failure()\n - name: Upload failed tests' directories\n@@ -292,6 +295,7 @@ jobs:\n image: daald/ubuntu32:xenial\n - jobname: pedantic\n image: fedora\n+ skip-tests: no\n env:\n jobname: ${{matrix.vector.jobname}}\n runs-on: ubuntu-latest\n@@ -299,9 +303,12 @@ jobs:\n steps:\n - uses: actions/checkout@v1\n - run: ci/install-docker-dependencies.sh\n- - run: ci/run-build-and-tests.sh\n+ - run: ci/lib.sh\n+ - run: make\n+ - run: make test\n+ if: success() && matrix.vector.skip-tests != 'no'\n - run: ci/print-test-failures.sh\n- if: failure()\n+ if: failure() && matrix.vector.skip-tests != 'no'\n - name: Upload failed tests' directories\n if: failure() && env.FAILED_TEST_ARTIFACTS != ''\n uses: actions/upload-artifact@v1\n@@ -317,6 +324,7 @@ jobs:\n steps:\n - uses: actions/checkout@v2\n - run: ci/install-dependencies.sh\n+ - run: ci/lib.sh\n - run: make ci-static-analysis\n sparse:\n needs: ci-config\ndiff --git a/ci/check-unignored-build-artifacts.sh b/ci/check-unignored-build-artifacts.sh\nnew file mode 100755\nindex 00000000000..56d04b0db9a\n--- /dev/null\n+++ b/ci/check-unignored-build-artifacts.sh\n@@ -0,0 +1,8 @@\n+#!/bin/sh\n+#\n+# Check whether the build created anything not in our .gitignore\n+#\n+\n+. ${0%/*}/lib.sh\n+\n+check_unignored_build_artifacts\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 34b4400f38b..01006379fd0 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -22,6 +22,10 @@ ubuntu-latest)\n \tP4_PATH=\"$HOME/custom/p4\"\n \tGIT_LFS_PATH=\"$HOME/custom/git-lfs\"\n \texport PATH=\"$GIT_LFS_PATH:$P4_PATH:$PATH\"\n+\tif test -n \"$GITHUB_PATH\"\n+\tthen\n+\t\techo \"$PATH\" >>\"$GITHUB_PATH\"\n+\tfi\n \n \tP4WHENCE=http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION\n \tLFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 1a1fa0e4bb2..05f3dd15e27 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -1,3 +1,5 @@\n+#!/bin/sh\n+\n # Library of functions shared by all CI scripts\n \n # Set 'exit on error' for all CI scripts to let the caller know that\n@@ -27,7 +29,14 @@ setenv () {\n \tval=$2\n \tshift 2\n \n-\teval \"export $key=\\\"$val\\\"\"\n+\tif test -n \"$GITHUB_ENV\"\n+\tthen\n+\t\techo \"$key=$val\" >>\"$GITHUB_ENV\"\n+\telse\n+\t\t# For local debugging. Not used by the GitHub CI\n+\t\t# itself.\n+\t\teval \"export $key=\\\"$val\\\"\"\n+\tfi\n }\n \n check_unignored_build_artifacts ()\n@@ -96,6 +105,35 @@ macos-latest)\n esac\n \n case \"$jobname\" in\n+linux-gcc)\n+\tsetenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME main\n+\t;;\n+linux-TEST-vars)\n+\tsetenv --test GIT_TEST_SPLIT_INDEX yes\n+\tsetenv --test GIT_TEST_MERGE_ALGORITHM recursive\n+\tsetenv --test GIT_TEST_FULL_IN_PACK_ARRAY true\n+\tsetenv --test GIT_TEST_OE_SIZE 10\n+\tsetenv --test GIT_TEST_OE_DELTA_SIZE 5\n+\tsetenv --test GIT_TEST_COMMIT_GRAPH 1\n+\tsetenv --test GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS 1\n+\tsetenv --test GIT_TEST_MULTI_PACK_INDEX 1\n+\tsetenv --test GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP 1\n+\tsetenv --test GIT_TEST_ADD_I_USE_BUILTIN 1\n+\tsetenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME master\n+\tsetenv --test GIT_TEST_WRITE_REV_INDEX 1\n+\tsetenv --test GIT_TEST_CHECKOUT_WORKERS 2\n+\t;;\n+linux-clang)\n+\tsetenv --test GIT_TEST_DEFAULT_HASH sha1\n+\t;;\n+linux-sha256)\n+\tsetenv --test GIT_TEST_DEFAULT_HASH sha256\n+\t;;\n+pedantic)\n+\t# Don't run the tests; we only care about whether Git can be\n+\t# built.\n+\tsetenv --build DEVOPTS pedantic\n+\t;;\n linux32)\n \tCC=gcc\n \t;;\ndiff --git a/ci/make-test-artifacts.sh b/ci/make-test-artifacts.sh\nindex 646967481f6..45298562982 100755\n--- a/ci/make-test-artifacts.sh\n+++ b/ci/make-test-artifacts.sh\n@@ -8,5 +8,3 @@ mkdir -p \"$1\" # in case ci/lib.sh decides to quit early\n . ${0%/*}/lib.sh\n \n make artifacts-tar ARTIFACTS_DIRECTORY=\"$1\"\n-\n-check_unignored_build_artifacts\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\ndeleted file mode 100755\nindex 35d45a9373e..00000000000\n--- a/ci/run-build-and-tests.sh\n+++ /dev/null\n@@ -1,47 +0,0 @@\n-#!/bin/sh\n-#\n-# Build and test Git\n-#\n-\n-. ${0%/*}/lib.sh\n-\n-export MAKE_TARGETS=\"all test\"\n-\n-case \"$jobname\" in\n-linux-gcc)\n-\tsetenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME main\n-\t;;\n-linux-TEST-vars)\n-\tsetenv --test GIT_TEST_SPLIT_INDEX yes\n-\tsetenv --test GIT_TEST_MERGE_ALGORITHM recursive\n-\tsetenv --test GIT_TEST_FULL_IN_PACK_ARRAY true\n-\tsetenv --test GIT_TEST_OE_SIZE 10\n-\tsetenv --test GIT_TEST_OE_DELTA_SIZE 5\n-\tsetenv --test GIT_TEST_COMMIT_GRAPH 1\n-\tsetenv --test GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS 1\n-\tsetenv --test GIT_TEST_MULTI_PACK_INDEX 1\n-\tsetenv --test GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP 1\n-\tsetenv --test GIT_TEST_ADD_I_USE_BUILTIN 1\n-\tsetenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME master\n-\tsetenv --test GIT_TEST_WRITE_REV_INDEX 1\n-\tsetenv --test GIT_TEST_CHECKOUT_WORKERS 2\n-\t;;\n-linux-clang)\n-\tsetenv --test GIT_TEST_DEFAULT_HASH sha1\n-\t;;\n-linux-sha256)\n-\tsetenv --test GIT_TEST_DEFAULT_HASH sha256\n-\t;;\n-pedantic)\n-\t# Don't run the tests; we only care about whether Git can be\n-\t# built.\n-\tsetenv --build DEVOPTS pedantic\n-\texport MAKE_TARGETS=all\n-\t;;\n-esac\n-\n-# Any new \"test\" targets should not go after this \"make\", but should\n-# adjust $MAKE_TARGETS. Otherwise compilation-only targets above will\n-# start running tests.\n-make $MAKE_TARGETS\n-check_unignored_build_artifacts\ndiff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh\nindex b9a682b4bcd..70326961454 100755\n--- a/ci/run-test-slice.sh\n+++ b/ci/run-test-slice.sh\n@@ -8,5 +8,3 @@\n make --quiet -C t T=\"$(cd t &&\n \t./helper/test-tool path-utils slice-tests \"$1\" \"$2\" t[0-9]*.sh |\n \ttr '\\n' ' ')\"\n-\n-check_unignored_build_artifacts\ndiff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\nindex 41e2b126311..6b5cce03bd7 100755\n--- a/ci/test-documentation.sh\n+++ b/ci/test-documentation.sh\n@@ -39,4 +39,3 @@ test -s Documentation/git.html\n grep '\n---\n .github/workflows/main.yml | 3 +--\n Makefile | 28 ++++++++++++++++++++++++++++\n ci/run-static-analysis.sh | 30 ------------------------------\n shared.mak | 1 +\n 4 files changed, 30 insertions(+), 32 deletions(-)\n delete mode 100755 ci/run-static-analysis.sh\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 309b9141249..ad752010102 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -317,8 +317,7 @@ jobs:\n steps:\n - uses: actions/checkout@v2\n - run: ci/install-dependencies.sh\n- - run: ci/run-static-analysis.sh\n- - run: ci/check-directional-formatting.bash\n+ - run: make ci-static-analysis\n sparse:\n needs: ci-config\n if: needs.ci-config.outputs.enabled == 'yes'\ndiff --git a/Makefile b/Makefile\nindex 70f0a004e75..d30862a23a2 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -3007,6 +3007,20 @@ coccicheck: $(addsuffix .patch,$(filter-out %.pending.cocci,$(wildcard contrib/c\n # See contrib/coccinelle/README\n coccicheck-pending: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.pending.cocci))\n \n+.PHONY: check-coccicheck\n+check-coccicheck: coccicheck\n+\t$(QUIET_CHECK)for cocci_patch in contrib/coccinelle/*.patch; do \\\n+\t\tif test -s \"$$cocci_patch\"; then \\\n+\t\t\techo \"Coccinelle suggests the following changes in '$$cocci_patch':\"; \\\n+\t\t\tcat \"$$cocci_patch\"; \\\n+\t\t\tfail=UnfortunatelyYes; \\\n+\t\tfi \\\n+\tdone; \\\n+\tif test -n \"$$fail\"; then \\\n+\t\techo \"error: Coccinelle suggested some changes\"; \\\n+\t\texit 1; \\\n+\tfi\n+\n .PHONY: coccicheck coccicheck-pending\n \n ### Installation rules\n@@ -3442,3 +3456,17 @@ $(FUZZ_PROGRAMS): all\n \t\t$(XDIFF_OBJS) $(EXTLIBS) git.o $@.o $(LIB_FUZZING_ENGINE) -o $@\n \n fuzz-all: $(FUZZ_PROGRAMS)\n+\n+### CI \"check\" targets\n+#\n+# These targets are run from the CI, see .github/workflows/main.yml,\n+# but can also be run manually to run the same assertions locally.\n+\n+.PHONY: ci-check-directional-formatting\n+ci-check-directional-formatting:\n+\t$(QUIET_CHECK)ci/check-directional-formatting.bash\n+\n+.PHONY: ci-static-analysis\n+ci-static-analysis: ci-check-directional-formatting\n+ci-static-analysis: check-coccicheck\n+ci-static-analysis: hdr-check\ndiff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh\ndeleted file mode 100755\nindex 5a87b1cac96..00000000000\n--- a/ci/run-static-analysis.sh\n+++ /dev/null\n@@ -1,30 +0,0 @@\n-#!/bin/sh\n-#\n-# Perform various static code analysis checks\n-#\n-\n-. ${0%/*}/lib.sh\n-\n-make coccicheck\n-\n-set +x\n-\n-fail=\n-for cocci_patch in contrib/coccinelle/*.patch\n-do\n-\tif test -s \"$cocci_patch\"\n-\tthen\n-\t\techo \"$(tput setaf 1)Coccinelle suggests the following changes in '$cocci_patch':$(tput sgr0)\"\n-\t\tcat \"$cocci_patch\"\n-\t\tfail=UnfortunatelyYes\n-\tfi\n-done\n-\n-if test -n \"$fail\"\n-then\n-\techo \"$(tput setaf 1)error: Coccinelle suggested some changes$(tput sgr0)\"\n-\texit 1\n-fi\n-\n-make hdr-check ||\n-exit 1\ndiff --git a/shared.mak b/shared.mak\nindex 4e1b62ee99e..0593b3c98be 100644\n--- a/shared.mak\n+++ b/shared.mak\n@@ -68,6 +68,7 @@ ifndef V\n \tQUIET_HDR = @echo ' ' HDR $(<:hcc=h);\n \tQUIET_RC = @echo ' ' RC $@;\n \tQUIET_SPATCH = @echo ' ' SPATCH $<;\n+\tQUIET_CHECK = @echo ' ' CHECK $@;\n \n ## Used in \"Documentation/Makefile\"\n \tQUIET_ASCIIDOC\t= @echo ' ' ASCIIDOC $@;\n-- \n2.35.1.1517.g20a06c426a7\n\n"},{"id":"452344","messageId":"RFC-patch-v3-4.6-429c256ac62-20220325T183946Z-avarab@gmail.com","threadId":"57451","inReplyTo":"RFC-cover-v3-0.6-00000000000-20220325T183946Z-avarab@gmail.com","subject":"[RFC PATCH v3 4/6] ci: optionally mark up output in the GitHub workflow","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-25T19:24:13Z","receivedAt":"2022-03-25T19:33:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: Johannes Schindelin \n\nA couple of commands exist to spruce up the output in GitHub workflows:\nhttps://docs.github.com/en/actions/learn-github-actions/workflow-commands-for-github-actions\n\nIn addition to the `::group::