{"thread":{"id":"62296","subject":"[PATCH 0/5] Wire up Windows-based jobs in GitLab CI","startedAt":"2024-10-09T13:25:20Z","lastAt":"2024-10-10T05:56:24Z","messageCount":8,"participants":["Patrick Steinhardt","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"504557","messageId":"cover.1728480039.git.ps@pks.im","threadId":"62296","inReplyTo":null,"subject":"[PATCH 0/5] Wire up Windows-based jobs in GitLab CI","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-09T13:25:15Z","receivedAt":"2024-10-09T13:25:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthere's been multiple cases where changes I did recently broke Windows\nsystems. This was in the context of reftables, the libification\nrefactorings and other parts. Also, with my investigation of a more\nmodern build system I spent a ton of time in Windows overall. So it was\nhigh time that I finally wire up support for Windows jobs in GitLab CI,\nwhich is something I wanted to do for a long time.\n\nThis patch series does this with the help of Johannes. He helped me out\nquite significantly by starting to publish new build artifacts as part\nof the Git for Windows SDK pipelines that we can now easily download and\nuse. This makes it trivial to reimplement the \"setup-git-for-windows\"\nGitHub Action as a simple PowerShell script such that both GitHub and\nGitLab share the same implementation.\n\nI've created a pull request in GitHub [1] and a merge request in GitLab\n[2] to demonstrate that the pipelines work on both platforms.\n\nThanks!\n\nPatrick\n\n[1]: https://github.com/git/git/pull/1806\n[2]: https://gitlab.com/gitlab-org/git/-/merge_requests/204\n\nPatrick Steinhardt (5):\n t7300: work around platform-specific behaviour with long paths on\n MinGW\n ci: create script to set up Git for Windows SDK\n ci: handle Windows-based CI jobs in GitLab CI\n gitlab-ci: introduce stages and dependencies\n gitlab-ci: exercise Git on Windows\n\n .github/workflows/main.yml | 16 +++++++-----\n .gitlab-ci.yml | 51 ++++++++++++++++++++++++++++++++++++++\n ci/install-sdk.ps1 | 12 +++++++++\n ci/lib.sh | 18 ++++++++++----\n t/t7300-clean.sh | 2 +-\n 5 files changed, 87 insertions(+), 12 deletions(-)\n create mode 100755 ci/install-sdk.ps1\n\n-- \n2.47.0.rc1.33.g90fe3800b9.dirty\n\n"},{"id":"504558","messageId":"bdaf16a06168b7320e8dd1115258f00745a104c7.1728480039.git.ps@pks.im","threadId":"62296","inReplyTo":"cover.1728480039.git.ps@pks.im","subject":"[PATCH 1/5] t7300: work around platform-specific behaviour with long paths on MinGW","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-09T13:25:18Z","receivedAt":"2024-10-09T13:25:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Windows by default has a restriction in place to only allow paths up to\n260 characters. This restriction can nowadays be lifted by setting a\nregistry key, but is still active by default.\n\nIn t7300 we have one test that exercises the behaviour of git-clean(1)\nwith such long paths. Interestingly enough, this test fails on my system\nthat uses Windows 10 with mingw-w64 installed via MSYS2: instead of\nobserving ENAMETOOLONG, we observe ENOENT. This behaviour is consistent\nacross multiple different environments I have tried.\n\nI cannot say why exactly we observe a different error here, but I would\nnot be surprised if this was either dependent on the Windows version,\nthe version of MinGW, the current working directory of Git or any kind\nof combination of these.\n\nWork around the issue by handling both errors.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/t7300-clean.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex 0aae0dee67..12ab25296b 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -747,7 +747,7 @@ test_expect_success MINGW 'handle clean & core.longpaths = false nicely' '\n \ttest_must_fail git clean -xdf 2>.git/err &&\n \t# grepping for a strerror string is unportable but it is OK here with\n \t# MINGW prereq\n-\ttest_grep \"too long\" .git/err\n+\ttest_grep -e \"too long\" -e \"No such file or directory\" .git/err\n '\n \n test_expect_success 'clean untracked paths by pathspec' '\n-- \n2.47.0.rc1.33.g90fe3800b9.dirty\n\n"},{"id":"504559","messageId":"9f36f2125f0f17b173aa0cf9ed2e7df66cd8e76f.1728480039.git.ps@pks.im","threadId":"62296","inReplyTo":"cover.1728480039.git.ps@pks.im","subject":"[PATCH 2/5] ci: create script to set up Git for Windows SDK","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-09T13:25:21Z","receivedAt":"2024-10-09T13:25:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In order to build and test Git, we have to first set up the Git for\nWindows SDK, which contains various required tools and libraries. The\nSDK is basically a clone of [1], but that repository is quite large due\nto all the binaries it contains. We thus use both shallow clones and\nsparse checkouts to speed up the setup. To handle this complexity we use\na GitHub action that is hosted externally at [2].\n\nUnfortunately, this makes it rather hard to reuse the logic for CI\nplatforms other than GitHub Actions. After chatting with Johannes\nSchindelin we came to the conclusion that it would be nice if the Git\nfor Windows SDK would regularly publish releases that one can easily\ndownload and extract, thus moving all of the complexity into that single\nstep. Like this, all that a CI job needs to do is to fetch and extract\nthe resulting archive. This published release comes in the form of a new\n\"ci-artifacts\" tag that gets updated regularly [3].\n\nImplement a new script that knows how to fetch and extract that script\nand convert GitHub Actions to use it.\n\n[1]: https://github.com/git-for-windows/git-sdk-64/\n[2]: https://github.com/git-for-windows/setup-git-for-windows-sdk/\n[3]: https://github.com/git-for-windows/git-sdk-64/releases/tag/ci-artifacts/\n\nHelped-by: Johannes Schindelin \nSigned-off-by: Patrick Steinhardt \n---\n .github/workflows/main.yml | 16 ++++++++++------\n ci/install-sdk.ps1 | 12 ++++++++++++\n 2 files changed, 22 insertions(+), 6 deletions(-)\n create mode 100755 ci/install-sdk.ps1\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 916a64b673..9301a1edd6 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -113,13 +113,15 @@ jobs:\n cancel-in-progress: ${{ needs.ci-config.outputs.skip_concurrent == 'yes' }}\n steps:\n - uses: actions/checkout@v4\n- - uses: git-for-windows/setup-git-for-windows-sdk@v1\n+ - name: setup SDK\n+ shell: powershell\n+ run: ci/install-sdk.ps1\n - name: build\n- shell: bash\n+ shell: powershell\n env:\n HOME: ${{runner.workspace}}\n NO_PERL: 1\n- run: . /etc/profile && ci/make-test-artifacts.sh artifacts\n+ run: git-sdk/usr/bin/bash.exe -l -c 'ci/make-test-artifacts.sh artifacts'\n - name: zip up tracked files\n run: git archive -o artifacts/tracked.tar.gz HEAD\n - name: upload tracked files and build artifacts\n@@ -147,10 +149,12 @@ jobs:\n - name: extract tracked files and build artifacts\n shell: bash\n run: tar xf artifacts.tar.gz && tar xf tracked.tar.gz\n- - uses: git-for-windows/setup-git-for-windows-sdk@v1\n+ - name: setup SDK\n+ shell: powershell\n+ run: ci/install-sdk.ps1\n - name: test\n- shell: bash\n- run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10\n+ shell: powershell\n+ run: git-sdk/usr/bin/bash.exe -l -c 'ci/run-test-slice.sh ${{matrix.nr}} 10'\n - name: print test failures\n if: failure() && env.FAILED_TEST_ARTIFACTS != ''\n shell: bash\ndiff --git a/ci/install-sdk.ps1 b/ci/install-sdk.ps1\nnew file mode 100755\nindex 0000000000..66f24838a4\n--- /dev/null\n+++ b/ci/install-sdk.ps1\n@@ -0,0 +1,12 @@\n+param(\n+ [string]$directory='git-sdk',\n+ [string]$url='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip'\n+)\n+\n+Invoke-WebRequest \"$url\" -OutFile git-sdk.zip\n+Expand-Archive -LiteralPath git-sdk.zip -DestinationPath \"$directory\"\n+Remove-Item -Path git-sdk.zip\n+\n+New-Item -Path .git/info -ItemType Directory -Force\n+New-Item -Path .git/info/exclude -ItemType File -Force\n+Add-Content -Path .git/info/exclude -Value \"/$directory\"\n-- \n2.47.0.rc1.33.g90fe3800b9.dirty\n\n"},{"id":"504560","messageId":"cb8c49b14e723221588c798a8f4a000510b1d5b5.1728480039.git.ps@pks.im","threadId":"62296","inReplyTo":"cover.1728480039.git.ps@pks.im","subject":"[PATCH 3/5] ci: handle Windows-based CI jobs in GitLab CI","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-09T13:25:23Z","receivedAt":"2024-10-09T13:25:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We try to abstract away any differences between different CI platforms\nin \"ci/lib.sh\", such that knowledge specific to e.g. GitHub Actions or\nGitLab CI is neatly encapsulated in a single place. Next to some generic\nvariables, we also set up some variables that are specific to the actual\nplatform that the CI operates on, e.g. Linux or macOS.\n\nWe do not yet support Windows runners on GitLab CI. Unfortunately, those\nsystems do not use the same \"CI_JOB_IMAGE\" environment variable as both\nLinux and macOS do. Instead, we can use the \"OS\" variable, which should\nhave a value of \"Windows_NT\" on Windows platforms.\n\nHandle the combination of \"$OS,$CI_JOB_IMAGE\" and introduce support for\nWindows.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/lib.sh | 18 +++++++++++++-----\n 1 file changed, 13 insertions(+), 5 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 74b430be23..95f39a26ea 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -250,8 +250,13 @@ then\n \tCI_TYPE=gitlab-ci\n \tCI_BRANCH=\"$CI_COMMIT_REF_NAME\"\n \tCI_COMMIT=\"$CI_COMMIT_SHA\"\n-\tcase \"$CI_JOB_IMAGE\" in\n-\tmacos-*)\n+\n+\tcase \"$OS,$CI_JOB_IMAGE\" in\n+\tWindows_NT,*)\n+\t\tCI_OS_NAME=windows\n+\t\tJOBS=$NUMBER_OF_PROCESSORS\n+\t\t;;\n+\t*,macos-*)\n \t\t# GitLab CI has Python installed via multiple package managers,\n \t\t# most notably via asdf and Homebrew. Ensure that our builds\n \t\t# pick up the Homebrew one by prepending it to our PATH as the\n@@ -259,9 +264,12 @@ then\n \t\texport PATH=\"$(brew --prefix)/bin:$PATH\"\n \n \t\tCI_OS_NAME=osx\n+\t\tJOBS=$(nproc)\n+\t\t;;\n+\t*,alpine:*|*,fedora:*|*,ubuntu:*)\n+\t\tCI_OS_NAME=linux\n+\t\tJOBS=$(nproc)\n \t\t;;\n-\talpine:*|fedora:*|ubuntu:*)\n-\t\tCI_OS_NAME=linux;;\n \t*)\n \t\techo \"Could not identify OS image\" >&2\n \t\tenv >&2\n@@ -272,6 +280,7 @@ then\n \tCI_JOB_ID=\"$CI_JOB_ID\"\n \tCC=\"${CC_PACKAGE:-${CC:-gcc}}\"\n \tDONT_SKIP_TAGS=t\n+\n \thandle_failed_tests () {\n \t\tcreate_failed_test_artifacts\n \t\treturn 1\n@@ -280,7 +289,6 @@ then\n \tcache_dir=\"$HOME/none\"\n \n \tdistro=$(echo \"$CI_JOB_IMAGE\" | tr : -)\n-\tJOBS=$(nproc)\n else\n \techo \"Could not identify CI type\" >&2\n \tenv >&2\n-- \n2.47.0.rc1.33.g90fe3800b9.dirty\n\n"},{"id":"504561","messageId":"6ff5aabfe190dfc0e75c7b995a2a63e8f5cc640c.1728480039.git.ps@pks.im","threadId":"62296","inReplyTo":"cover.1728480039.git.ps@pks.im","subject":"[PATCH 4/5] gitlab-ci: introduce stages and dependencies","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-09T13:25:26Z","receivedAt":"2024-10-09T13:25:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're about to add a couple of jobs for Windows. As the Windows runners\nare quite slow, we will split those up across two stages: one stage to\nbuild the artifacts, and one stage that runs test slices in parallel.\n\nIntroduce stages and \"needs\" dependencies for the preexisting jobs as a\npreparatory step. The stages will lead to a more natural representation\nof jobs in the UI, whereas the \"needs\" dependency ensures that jobs do\nnot have to wait for all jobs in the preceding stage to finish.\n\nSigned-off-by: Patrick Steinhardt \n---\n .gitlab-ci.yml | 18 ++++++++++++++++++\n 1 file changed, 18 insertions(+)\n\ndiff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\nindex 4abfbc3e20..e13e8b3866 100644\n--- a/.gitlab-ci.yml\n+++ b/.gitlab-ci.yml\n@@ -1,6 +1,10 @@\n default:\n timeout: 2h\n \n+stages:\n+ - test\n+ - analyze\n+\n workflow:\n rules:\n - if: $CI_PIPELINE_SOURCE == \"merge_request_event\"\n@@ -9,6 +13,8 @@ workflow:\n \n test:linux:\n image: $image\n+ stage: test\n+ needs: [ ]\n tags:\n - saas-linux-medium-amd64\n variables:\n@@ -67,6 +73,8 @@ test:linux:\n \n test:osx:\n image: $image\n+ stage: test\n+ needs: [ ]\n tags:\n - saas-macos-medium-m1\n variables:\n@@ -102,6 +110,8 @@ test:osx:\n \n test:fuzz-smoke-tests:\n image: ubuntu:latest\n+ stage: test\n+ needs: [ ]\n variables:\n CC: clang\n before_script:\n@@ -111,6 +121,8 @@ test:fuzz-smoke-tests:\n \n static-analysis:\n image: ubuntu:22.04\n+ stage: analyze\n+ needs: [ ]\n variables:\n jobname: StaticAnalysis\n before_script:\n@@ -121,6 +133,8 @@ static-analysis:\n \n check-whitespace:\n image: ubuntu:latest\n+ stage: analyze\n+ needs: [ ]\n before_script:\n - ./ci/install-dependencies.sh\n # Since $CI_MERGE_REQUEST_TARGET_BRANCH_SHA is only defined for merged\n@@ -135,6 +149,8 @@ check-whitespace:\n \n check-style:\n image: ubuntu:latest\n+ stage: analyze\n+ needs: [ ]\n allow_failure: true\n variables:\n CC: clang\n@@ -153,6 +169,8 @@ check-style:\n \n documentation:\n image: ubuntu:latest\n+ stage: analyze\n+ needs: [ ]\n variables:\n jobname: Documentation\n before_script:\n-- \n2.47.0.rc1.33.g90fe3800b9.dirty\n\n"},{"id":"504562","messageId":"15b79665ac35c29d1582eebdf0c223fec358f236.1728480039.git.ps@pks.im","threadId":"62296","inReplyTo":"cover.1728480039.git.ps@pks.im","subject":"[PATCH 5/5] gitlab-ci: exercise Git on Windows","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-09T13:25:29Z","receivedAt":"2024-10-09T13:25:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Add jobs that exercise Git on Windows. Unfortunately, building and\nespecially testing Git on Windows is inherently slower compared to other\nUnix-like systems, mostly because spawning processes is way slower. We\nthus use the same layout as we use in GitHub Actions, where we have one\nbuild job, and then pass on the resulting build artifacts to ten test\njobs that split up the work across each other.\n\nUnfortunately, the GitLab runners for Windows machines are embarassingly\nslow by themselves. So while this strategy leads to around 20 minutes of\nbuild time in GitHub Actions, the same pipeline takes around an hour in\nGitLab CI. Still, having late coverage is certainly better than having\nnone at all.\n\nSigned-off-by: Patrick Steinhardt \n---\n .gitlab-ci.yml | 33 +++++++++++++++++++++++++++++++++\n 1 file changed, 33 insertions(+)\n\ndiff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\nindex e13e8b3866..526ecfe030 100644\n--- a/.gitlab-ci.yml\n+++ b/.gitlab-ci.yml\n@@ -2,6 +2,7 @@ default:\n timeout: 2h\n \n stages:\n+ - build\n - test\n - analyze\n \n@@ -108,6 +109,38 @@ test:osx:\n - t/failed-test-artifacts\n when: on_failure\n \n+build:mingw64:\n+ stage: build\n+ tags:\n+ - saas-windows-medium-amd64\n+ variables:\n+ NO_PERL: 1\n+ before_script:\n+ - ./ci/install-sdk.ps1 -directory \"git-sdk\"\n+ script:\n+ - git-sdk/usr/bin/bash.exe -l -c 'ci/make-test-artifacts.sh artifacts'\n+ artifacts:\n+ paths:\n+ - artifacts\n+ - git-sdk\n+\n+test:mingw64:\n+ stage: test\n+ tags:\n+ - saas-windows-medium-amd64\n+ needs:\n+ - job: \"build:mingw64\"\n+ artifacts: true\n+ before_script:\n+ - git-sdk/usr/bin/bash.exe -l -c 'tar xf artifacts/artifacts.tar.gz'\n+ - New-Item -Path .git/info -ItemType Directory\n+ - New-Item .git/info/exclude -ItemType File -Value \"/git-sdk\"\n+ script:\n+ - git-sdk/usr/bin/bash.exe -l -c \"ci/run-test-slice.sh $CI_NODE_INDEX $CI_NODE_TOTAL\"\n+ after_script:\n+ - git-sdk/usr/bin/bash.exe -l -c 'ci/print-test-failures.sh'\n+ parallel: 10\n+\n test:fuzz-smoke-tests:\n image: ubuntu:latest\n stage: test\n-- \n2.47.0.rc1.33.g90fe3800b9.dirty\n\n"},{"id":"504623","messageId":"xmqqr08pruv6.fsf@gitster.g","threadId":"62296","inReplyTo":"bdaf16a06168b7320e8dd1115258f00745a104c7.1728480039.git.ps@pks.im","subject":"Re: [PATCH 1/5] t7300: work around platform-specific behaviour with long paths on MinGW","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-10-09T18:31:25Z","receivedAt":"2024-10-09T18:31:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> diff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\n> index 0aae0dee67..12ab25296b 100755\n> --- a/t/t7300-clean.sh\n> +++ b/t/t7300-clean.sh\n> @@ -747,7 +747,7 @@ test_expect_success MINGW 'handle clean & core.longpaths = false nicely' '\n> \ttest_must_fail git clean -xdf 2>.git/err &&\n> \t# grepping for a strerror string is unportable but it is OK here with\n> \t# MINGW prereq\n\nI find it amusing that this talks about limiting the test to MINGW\nyet we need to accept two variants ;-)\n\n> -\ttest_grep \"too long\" .git/err\n> +\ttest_grep -e \"too long\" -e \"No such file or directory\" .git/err\n> '\n\nThe solution looks obviously good enough.\n\nThanks.\n"},{"id":"504675","messageId":"ZwdsgHdNp108lISP@pks.im","threadId":"62296","inReplyTo":"xmqqr08pruv6.fsf@gitster.g","subject":"Re: [PATCH 1/5] t7300: work around platform-specific behaviour with long paths on MinGW","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-10T05:56:16Z","receivedAt":"2024-10-10T05:56:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Oct 09, 2024 at 11:31:25AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > diff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\n> > index 0aae0dee67..12ab25296b 100755\n> > --- a/t/t7300-clean.sh\n> > +++ b/t/t7300-clean.sh\n> > @@ -747,7 +747,7 @@ test_expect_success MINGW 'handle clean & core.longpaths = false nicely' '\n> > \ttest_must_fail git clean -xdf 2>.git/err &&\n> > \t# grepping for a strerror string is unportable but it is OK here with\n> > \t# MINGW prereq\n> \n> I find it amusing that this talks about limiting the test to MINGW\n> yet we need to accept two variants ;-)\n\nAgreed, I was quite confused by this. I hope that Johannes may shed a\nbit more light on this, as I couldn't make much sense of it. Was this a\nchange on our side? Did MinGW change? Is it something else entirely? No\nidea.\n\nPatrick\n"}]}