git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 1/7] ci: use the new GitHub Action to download git-sdk-64-minimal

From
Johannes Schindelin via GitGitGadget <gitgitgadget@gmail.com>
Date
Jul 3, 2021, 21:26 UTC
Message-ID
<2e4db688deb938592257882c06b658cf3b8463f8.1625347592.git.gitgitgadget@gmail.com>
In-Reply-To
<pull.878.v2.git.1625347592.gitgitgadget@gmail.com>
From: Johannes Schindelin <johannes.schindelin@gmx.de>

In our continuous builds, Windows is the odd cookie that requires a complete development environment to be downloaded because there is no suitable one installed by default on Windows.

Side note: technically, there _is_ a development environment present in GitHub Actions' build agents: MSYS2. But it differs from Git for Windows' SDK in subtle points, unfortunately enough so to prevent Git's test suite from running without failures.

Traditionally, we support downloading this environment (which we nicknamed `git-sdk-64-minimal`) via a PowerShell scriptlet that accesses the build artifacts of a dedicated Azure Pipeline (which packages a tiny subset of the full Git for Windows SDK, containing just enough to build Git and run its test suite).

This PowerShell script is unfortunately not very robust and sometimes fails due to network issues.

Of course, we could add code to detect that situation, wait a little, try again, if it fails again wait a little longer, lather, rinse and repeat.

Instead of doing all of this in Git's own `.github/workflows/`, though, let's offload this logic to the new GitHub Action at https://github.com/marketplace/actions/setup-git-for-windows-sdk

This Action not only downloads and extracts git-sdk-64-minimal _outside_ the worktree (making it no longer necessary to meddle with `.gitignore` or `.git/info/exclude`), it also adds the `bash.exe` to the `PATH` and sets the environment variable `MSYSTEM` (an implementation detail that Git's workflow should never have needed to know about).

This allows us to convert all those funny PowerShell tasks that wanted to call git-sdk-64-minimal's `bash.exe`: they all are now regular `bash` scriptlets.

This finally lets us get rid of the funny quoting and escaping where we had to pay attention not only to quote and escape the Bash scriptlets properly, but also to add a second level of escaping (with backslashes for double quotes and backticks for dollar signs) to stop PowerShell from doing unintended things.

Further, this Action uses a fast caching strategy native to GitHub Actions that should accelerate the download across CI runs: git-sdk-64-minimal is usually updated once per 24h, and needs to be cached only once within that period. Caching it (unfortunately only on a per-branch basis) speeds up the download step, and makes it much more robust at the same time by virtue of accessing a cache location that is closer in the network topology.

With this we can drop the home-rolled caching where we try to accelerate the test phase by uploading git-sdk-64-minimal as a workflow artifact after using it to build Git, and then download it as workflow artifact in the test phase.

Even better: the `vs-test` job no longer needs to depend on the `windows-build` job. The only reason it depended on it was to ensure that the `git-sdk-64-minimal` workflow artifact was available.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 .github/workflows/main.yml | 103 +++++++------------------------------
 1 file changed, 19 insertions(+), 84 deletions(-)
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index 73856bafc9d..c62766e7b1a 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -82,43 +82,18 @@ jobs:
     runs-on: windows-latest
     steps:
     - uses: actions/checkout@v1
-    - name: download git-sdk-64-minimal
-      shell: bash
-      run: |
-        ## Get artifact
-        urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds
-        id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" |
-          jq -r ".value[] | .id")
-        download_url="$(curl "$urlbase/$id/artifacts" |
-          jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')"
-        curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \
-          -o artifacts.zip "$download_url"
-
-        ## Unzip and remove the artifact
-        unzip artifacts.zip
-        rm artifacts.zip
+    - uses: git-for-windows/setup-git-for-windows-sdk@v1
     - name: build
-      shell: powershell
+      shell: bash
       env:
         HOME: ${{runner.workspace}}
-        MSYSTEM: MINGW64
         NO_PERL: 1
-      run: |
-        & .\git-sdk-64-minimal\usr\bin\bash.exe -lc @"
-        printf '%s\n' /git-sdk-64-minimal/ >>.git/info/exclude
-
-          ci/make-test-artifacts.sh artifacts
-        "@
+      run: ci/make-test-artifacts.sh artifacts
     - name: upload build artifacts
       uses: actions/upload-artifact@v1
       with:
         name: windows-artifacts
         path: artifacts
-    - name: upload git-sdk-64-minimal
-      uses: actions/upload-artifact@v1
-      with:
-        name: git-sdk-64-minimal
-        path: git-sdk-64-minimal
   windows-test:
     runs-on: windows-latest
     needs: [windows-build]
@@ -136,25 +111,14 @@ jobs:
     - name: extract build artifacts
       shell: bash
       run: tar xf artifacts.tar.gz
-    - name: download git-sdk-64-minimal
-      uses: actions/download-artifact@v1
-      with:
-        name: git-sdk-64-minimal
-        path: ${{github.workspace}}/git-sdk-64-minimal/
+    - uses: git-for-windows/setup-git-for-windows-sdk@v1
     - name: test
-      shell: powershell
-      run: |
-        & .\git-sdk-64-minimal\usr\bin\bash.exe -lc @"
-          # Let Git ignore the SDK
-          printf '%s\n' /git-sdk-64-minimal/ >>.git/info/exclude
-
-          ci/run-test-slice.sh ${{matrix.nr}} 10
-        "@
+      shell: bash
+      run: ci/run-test-slice.sh ${{matrix.nr}} 10
     - name: ci/print-test-failures.sh
       if: failure()
-      shell: powershell
-      run: |
-        & .\git-sdk-64-minimal\usr\bin\bash.exe -lc ci/print-test-failures.sh
+      shell: bash
+      run: ci/print-test-failures.sh
     - name: Upload failed tests' directories
       if: failure() && env.FAILED_TEST_ARTIFACTS != ''
       uses: actions/upload-artifact@v1
@@ -165,27 +129,12 @@ jobs:
     needs: ci-config
     if: needs.ci-config.outputs.enabled == 'yes'
     env:
-      MSYSTEM: MINGW64
       NO_PERL: 1
       GIT_CONFIG_PARAMETERS: "'user.name=CI' 'user.email=ci@git'"
     runs-on: windows-latest
     steps:
     - uses: actions/checkout@v1
-    - name: download git-sdk-64-minimal
-      shell: bash
-      run: |
-        ## Get artifact
-        urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds
-        id=$(curl "$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\$top=1" |
-          jq -r ".value[] | .id")
-        download_url="$(curl "$urlbase/$id/artifacts" |
-          jq -r '.value[] | select(.name == "git-sdk-64-minimal").resource.downloadUrl')"
-        curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \
-          -o artifacts.zip "$download_url"
-
-        ## Unzip and remove the artifact
-        unzip artifacts.zip
-        rm artifacts.zip
+    - uses: git-for-windows/setup-git-for-windows-sdk@v1
     - name: initialize vcpkg
       uses: actions/checkout@v2
       with:
@@ -211,19 +160,17 @@ jobs:
       shell: bash
       run: |
         cmake `pwd`/contrib/buildsystems/ -DCMAKE_PREFIX_PATH=`pwd`/compat/vcbuild/vcpkg/installed/x64-windows \
-        -DMSGFMT_EXE=`pwd`/git-sdk-64-minimal/mingw64/bin/msgfmt.exe -DPERL_TESTS=OFF -DPYTHON_TESTS=OFF -DCURL_NO_CURL_CMAKE=ON
+        -DMSGFMT_EXE=C:/git-sdk-64-minimal/mingw64/bin/msgfmt.exe -DPERL_TESTS=OFF -DPYTHON_TESTS=OFF -DCURL_NO_CURL_CMAKE=ON
     - name: MSBuild
       run: msbuild git.sln -property:Configuration=Release -property:Platform=x64 -maxCpuCount:4 -property:PlatformToolset=v142
     - name: bundle artifact tar
-      shell: powershell
+      shell: bash
       env:
         MSVC: 1
         VCPKG_ROOT: ${{github.workspace}}\compat\vcbuild\vcpkg
       run: |
-        & git-sdk-64-minimal\usr\bin\bash.exe -lc @"
-          mkdir -p artifacts &&
-          eval \"`$(make -n artifacts-tar INCLUDE_DLLS_IN_ARTIFACTS=YesPlease ARTIFACTS_DIRECTORY=artifacts 2>&1 | grep ^tar)\"
-        "@
+        mkdir -p artifacts &&
+        eval "$(make -n artifacts-tar INCLUDE_DLLS_IN_ARTIFACTS=YesPlease ARTIFACTS_DIRECTORY=artifacts 2>&1 | grep ^tar)"
     - name: upload build artifacts
       uses: actions/upload-artifact@v1
       with:
@@ -231,18 +178,14 @@ jobs:
         path: artifacts
   vs-test:
     runs-on: windows-latest
-    needs: [vs-build, windows-build]
+    needs: vs-build
     strategy:
       fail-fast: false
       matrix:
         nr: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
     steps:
     - uses: actions/checkout@v1
-    - name: download git-sdk-64-minimal
-      uses: actions/download-artifact@v1
-      with:
-        name: git-sdk-64-minimal
-        path: ${{github.workspace}}/git-sdk-64-minimal/
+    - uses: git-for-windows/setup-git-for-windows-sdk@v1
     - name: download build artifacts
       uses: actions/download-artifact@v1
       with:
@@ -252,23 +195,15 @@ jobs:
       shell: bash
       run: tar xf artifacts.tar.gz
     - name: test
-      shell: powershell
+      shell: bash
       env:
-        MSYSTEM: MINGW64
         NO_SVN_TESTS: 1
         GIT_TEST_SKIP_REBASE_P: 1
-      run: |
-        & .\git-sdk-64-minimal\usr\bin\bash.exe -lc @"
-          # Let Git ignore the SDK and the test-cache
-          printf '%s\n' /git-sdk-64-minimal/ /test-cache/ >>.git/info/exclude
-
-          ci/run-test-slice.sh ${{matrix.nr}} 10
-        "@
+      run: ci/run-test-slice.sh ${{matrix.nr}} 10
     - name: ci/print-test-failures.sh
       if: failure()
-      shell: powershell
-      run: |
-        & .\git-sdk-64-minimal\usr\bin\bash.exe -lc ci/print-test-failures.sh
+      shell: bash
+      run: ci/print-test-failures.sh
     - name: Upload failed tests' directories
       if: failure() && env.FAILED_TEST_ARTIFACTS != ''
       uses: actions/upload-artifact@v1

-- 
gitgitgadget
Previous: Johannes Schindelin via GitGitGadgetNext: Dennis Ameling via GitGitGadget
Message 18 of 41 in “ci: speed-up the Windows parts of our GitHub workflow”
  1. 0/6 ci: speed-up the Windows parts of our GitHub workflowJohannes Schindelin via GitGitGadget, Jun 23, 2021
  2. 2/6 ci (vs-build): use `cmd` to copy the DLLs, not `powershell`Johannes Schindelin via GitGitGadget, Jun 23, 2021
  3. 1/6 ci: use the new GitHub Action to download git-sdk-64-minimalJohannes Schindelin via GitGitGadget, Jun 23, 2021
  4. 5/6 ci(vs-build): build with NO_GETTEXTDennis Ameling via GitGitGadget, Jun 23, 2021
  5. Ævar Arnfjörð BjarmasonJul 4, 2021
  6. Ævar Arnfjörð BjarmasonJul 4, 2021
  7. Johannes SchindelinJul 13, 2021
  8. Philip OakleyJul 13, 2021
  9. Johannes SchindelinJul 14, 2021
  10. Ævar Arnfjörð BjarmasonJul 14, 2021
  11. 3/6 ci: upgrade to using actions/{up,down}load-artifacts v2Johannes Schindelin via GitGitGadget, Jun 23, 2021
  12. 4/6 ci(windows): transfer also the Git-tracked files to the test jobsJohannes Schindelin via GitGitGadget, Jun 23, 2021
  13. 6/6 ci: accelerate the checkoutJohannes Schindelin via GitGitGadget, Jun 23, 2021
  14. Junio C HamanoJul 1, 2021
  15. 0/7 ci: speed-up the Windows parts of our GitHub workflowJohannes Schindelin via GitGitGadget, Jul 3, 2021
  16. 2/7 ci (vs-build): use `cmd` to copy the DLLs, not `powershell`Johannes Schindelin via GitGitGadget, Jul 3, 2021
  17. 3/7 ci: upgrade to using actions/{up,down}load-artifacts v2Johannes Schindelin via GitGitGadget, Jul 3, 2021
  18. 1/7 ci: use the new GitHub Action to download git-sdk-64-minimalJohannes Schindelin via GitGitGadget, Jul 3, 2021
  19. 6/7 ci(vs-build): build with NO_GETTEXTDennis Ameling via GitGitGadget, Jul 3, 2021
  20. 5/7 artifacts-tar: respect NO_GETTEXTJohannes Schindelin via GitGitGadget, Jul 3, 2021
  21. Ævar Arnfjörð BjarmasonJul 4, 2021
  22. Johannes SchindelinJul 4, 2021
  23. Ævar Arnfjörð BjarmasonJul 5, 2021
  24. 7/7 ci: accelerate the checkoutJohannes Schindelin via GitGitGadget, Jul 3, 2021
  25. Ævar Arnfjörð BjarmasonJul 4, 2021
  26. Johannes SchindelinJul 4, 2021
  27. 4/7 ci(windows): transfer also the Git-tracked files to the test jobsJohannes Schindelin via GitGitGadget, Jul 3, 2021
  28. 0/7 ci: speed-up the Windows parts of our GitHub workflowJohannes Schindelin via GitGitGadget, Jul 4, 2021
  29. 1/7 ci: use the new GitHub Action to download git-sdk-64-minimalJohannes Schindelin via GitGitGadget, Jul 4, 2021
  30. Junio C HamanoJul 6, 2021
  31. 2/7 ci (vs-build): use `cmd` to copy the DLLs, not `powershell`Johannes Schindelin via GitGitGadget, Jul 4, 2021
  32. 3/7 ci: upgrade to using actions/{up,down}load-artifacts v2Johannes Schindelin via GitGitGadget, Jul 4, 2021
  33. 4/7 ci(windows): transfer also the Git-tracked files to the test jobsJohannes Schindelin via GitGitGadget, Jul 4, 2021
  34. 7/7 ci: accelerate the checkoutJohannes Schindelin via GitGitGadget, Jul 4, 2021
  35. 5/7 artifacts-tar: respect NO_GETTEXTJohannes Schindelin via GitGitGadget, Jul 4, 2021
  36. 6/7 ci(vs-build): build with NO_GETTEXTDennis Ameling via GitGitGadget, Jul 4, 2021
  37. Ævar Arnfjörð BjarmasonJul 5, 2021
  38. Johannes SchindelinJul 5, 2021
  39. Junio C HamanoJul 6, 2021
  40. Johannes SchindelinJul 14, 2021
  41. Junio C HamanoJul 6, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.