{"thread":{"id":"62656","subject":"[PATCH] GitHub ci(windows): speed up initializing Git for Windows' minimal SDK again","startedAt":"2024-12-17T14:57:42Z","lastAt":"2024-12-19T13:20:55Z","messageCount":4,"participants":["Johannes Schindelin via GitGitGadget","Junio C Hamano","Patrick Steinhardt","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"509223","messageId":"pull.1841.git.1734447458896.gitgitgadget@gmail.com","threadId":"62656","inReplyTo":null,"subject":"[PATCH] GitHub ci(windows): speed up initializing Git for Windows' minimal SDK again","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-12-17T14:57:38Z","receivedAt":"2024-12-17T14:57:42Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin <johannes.schindelin@gmx.de>\n\nIt used to be the case that initializing the minimal SDK (i.e. a\nradically slimmed-down subset of Git for Windows' development\nenvironment intended to perform the CI builds and little else) took\na bit over one minute, would then be cached, and subsequent jobs would\ntake at most half a dozen seconds to initialize said minimal SDK.\n\nIt is important that this step is fast because we have to run the test\nsuite in parallel, in a set of matrix jobs, to offset the slowness of\nthe shell-based test suite, and each and every job has to initialize the\nvery same minimal SDK.\n\nWhile it may sound as if parallelizing the jobs might only waste the\ngenerously-provided build minutes but at least the _wallclock_ time\nwould pass quick, in reality it matters a lot: Frequently Git for\nWindows' or GitGitGadget PRs get stuck waiting for quite a while before\nCI builds start because other PRs' builds still spend substantial\namounts of time to run, blocking due to the concurrency limit being\nreached.\n\nSince 91839a88277 (ci: create script to set up Git for Windows SDK,\n2024-10-09), the situation has worsened: every job that requires the\nminimal Git for Windows SDK spends roughly two-and-a-half minutes doing\nso.\n\nWith the switch away from the GitHub Action `setup-git-for-windows-sdk`,\nwe incurred more downsides:\n\n- It is no longer possible for said Action to fix problems independently\n  from the Git repository, e.g. when new rules about GitHub Actions\n  require changes in the way the minimal SDK is initialized.\n\n- The minimal SDK was installed specifically outside of the worktree so\n  as not to clutter it nor incur an additional cost to verify that the\n  worktree is clean.\n\nTherefore, even if it would be nice to have a shared process between\nGitHub and GitLab based CI builds, let's switch the GitHub-based CI back\nto the tried-and-tested `setup-git-for-windows-sdk` Action.\n\nThis commit partially reverts 91839a88277 (ci: create script to set up\nGit for Windows SDK, 2024-10-09).\n\nSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n---\n    Speed up the Git for Windows SDK initialization again\n    \n    While waiting for way too many builds in\n    https://github.com/gitgitgadget/git/actions to finish, I noticed that\n    the minimal Git for Windows SDK initialization now takes a whopping 2.5\n    minutes. That's way too much. It used to take a little over a minute\n    when uncached, and 2-5 seconds when cached.\n    \n    Let's fix this regression by reverting to using the\n    setup-git-for-windows-sdk GitHub Action (also because that Action will\n    soon see another dramatic speed-up, see\n    https://github.com/git-for-windows/setup-git-for-windows-sdk/pull/965).\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1841%2Fdscho%2Fci-windows-jobs-speedup-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1841/dscho/ci-windows-jobs-speedup-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1841\n\n .github/workflows/main.yml | 16 ++++++----------\n 1 file changed, 6 insertions(+), 10 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 9301a1edd6d..916a64b6736 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -113,15 +113,13 @@ jobs:\n       cancel-in-progress: ${{ needs.ci-config.outputs.skip_concurrent == 'yes' }}\n     steps:\n     - uses: actions/checkout@v4\n-    - name: setup SDK\n-      shell: powershell\n-      run: ci/install-sdk.ps1\n+    - uses: git-for-windows/setup-git-for-windows-sdk@v1\n     - name: build\n-      shell: powershell\n+      shell: bash\n       env:\n         HOME: ${{runner.workspace}}\n         NO_PERL: 1\n-      run: git-sdk/usr/bin/bash.exe -l -c 'ci/make-test-artifacts.sh artifacts'\n+      run: . /etc/profile && ci/make-test-artifacts.sh artifacts\n     - name: zip up tracked files\n       run: git archive -o artifacts/tracked.tar.gz HEAD\n     - name: upload tracked files and build artifacts\n@@ -149,12 +147,10 @@ jobs:\n     - name: extract tracked files and build artifacts\n       shell: bash\n       run: tar xf artifacts.tar.gz && tar xf tracked.tar.gz\n-    - name: setup SDK\n-      shell: powershell\n-      run: ci/install-sdk.ps1\n+    - uses: git-for-windows/setup-git-for-windows-sdk@v1\n     - name: test\n-      shell: powershell\n-      run: git-sdk/usr/bin/bash.exe -l -c 'ci/run-test-slice.sh ${{matrix.nr}} 10'\n+      shell: bash\n+      run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10\n     - name: print test failures\n       if: failure() && env.FAILED_TEST_ARTIFACTS != ''\n       shell: bash\n\nbase-commit: 631ddbbcbd912530e1b78e5d782e72879f7f1fb2\n-- \ngitgitgadget\n"},{"id":"509239","messageId":"xmqqldweyrqx.fsf@gitster.g","threadId":"62656","inReplyTo":"pull.1841.git.1734447458896.gitgitgadget@gmail.com","subject":"Re: [PATCH] GitHub ci(windows): speed up initializing Git for Windows' minimal SDK again","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-17T20:33:10Z","receivedAt":"2024-12-17T20:33:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" <gitgitgadget@gmail.com>\nwrites:\n\n> This commit partially reverts 91839a88277 (ci: create script to set up\n> Git for Windows SDK, 2024-10-09).\n\nThanks, will queue.\n\n> diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\n> index 9301a1edd6d..916a64b6736 100644\n> --- a/.github/workflows/main.yml\n> +++ b/.github/workflows/main.yml\n> @@ -113,15 +113,13 @@ jobs:\n>        cancel-in-progress: ${{ needs.ci-config.outputs.skip_concurrent == 'yes' }}\n>      steps:\n>      - uses: actions/checkout@v4\n> -    - name: setup SDK\n> -      shell: powershell\n> -      run: ci/install-sdk.ps1\n> +    - uses: git-for-windows/setup-git-for-windows-sdk@v1\n>      - name: build\n> -      shell: powershell\n> +      shell: bash\n>        env:\n>          HOME: ${{runner.workspace}}\n>          NO_PERL: 1\n> -      run: git-sdk/usr/bin/bash.exe -l -c 'ci/make-test-artifacts.sh artifacts'\n> +      run: . /etc/profile && ci/make-test-artifacts.sh artifacts\n>      - name: zip up tracked files\n>        run: git archive -o artifacts/tracked.tar.gz HEAD\n>      - name: upload tracked files and build artifacts\n> @@ -149,12 +147,10 @@ jobs:\n>      - name: extract tracked files and build artifacts\n>        shell: bash\n>        run: tar xf artifacts.tar.gz && tar xf tracked.tar.gz\n> -    - name: setup SDK\n> -      shell: powershell\n> -      run: ci/install-sdk.ps1\n> +    - uses: git-for-windows/setup-git-for-windows-sdk@v1\n>      - name: test\n> -      shell: powershell\n> -      run: git-sdk/usr/bin/bash.exe -l -c 'ci/run-test-slice.sh ${{matrix.nr}} 10'\n> +      shell: bash\n> +      run: . /etc/profile && ci/run-test-slice.sh ${{matrix.nr}} 10\n>      - name: print test failures\n>        if: failure() && env.FAILED_TEST_ARTIFACTS != ''\n>        shell: bash\n>\n> base-commit: 631ddbbcbd912530e1b78e5d782e72879f7f1fb2\n"},{"id":"509277","messageId":"Z2JkBv87zXik5QJa@pks.im","threadId":"62656","inReplyTo":"xmqqldweyrqx.fsf@gitster.g","subject":"Re: [PATCH] GitHub ci(windows): speed up initializing Git for Windows' minimal SDK again","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-12-18T05:56:22Z","receivedAt":"2024-12-18T05:56:40Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Dec 17, 2024 at 12:33:10PM -0800, Junio C Hamano wrote:\n> \"Johannes Schindelin via GitGitGadget\" <gitgitgadget@gmail.com>\n> writes:\n> \n> > This commit partially reverts 91839a88277 (ci: create script to set up\n> > Git for Windows SDK, 2024-10-09).\n> \n> Thanks, will queue.\n\nOkay. Too bad that things regressed this badly with the change. I agree\nthat reverting is the right thing to do for now. I may revisit this\nagain in the next release cycle to investigate whether we can get it up\nto par with the GitHub Actions. It would be great if the build infra was\nshared between our CIs, so I think there's some value in it. But if the\nanswer is \"no\" then I guess that's ultimately fine, as well.\n\nThanks!\n\nPatrick\n"},{"id":"509335","messageId":"0176d617-182c-45c4-3f2b-e0d433901c40@gmx.de","threadId":"62656","inReplyTo":"Z2JkBv87zXik5QJa@pks.im","subject":"Re: [PATCH] GitHub ci(windows): speed up initializing Git for Windows' minimal SDK again","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2024-12-19T13:20:50Z","receivedAt":"2024-12-19T13:20:55Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Patrick,\n\nOn Wed, 18 Dec 2024, Patrick Steinhardt wrote:\n\n> On Tue, Dec 17, 2024 at 12:33:10PM -0800, Junio C Hamano wrote:\n> > \"Johannes Schindelin via GitGitGadget\" <gitgitgadget@gmail.com>\n> > writes:\n> >\n> > > This commit partially reverts 91839a88277 (ci: create script to set up\n> > > Git for Windows SDK, 2024-10-09).\n> >\n> > Thanks, will queue.\n>\n> Okay. Too bad that things regressed this badly with the change. I agree\n> that reverting is the right thing to do for now. I may revisit this\n> again in the next release cycle to investigate whether we can get it up\n> to par with the GitHub Actions.\n\nThe way I implemented it should be eminently possible in PowerShell, too.\n\nSomething like `Start-Process`, launching not an `Invoke-WebRequest` but\ninstead `C:\\Windows\\system32\\curl.exe` [*1*] to download the `.tar.gz`\nfile (not the `.zip` file, more about that below). Another `Start-Process`\nshould then be opened that executes `tar.exe` [*2*], and then the `stdout`\nof the former should be piped to the latter [*3*].\n\nI do think that it makes a lot of sense to start extracting as soon as the\nfirst byte arrives instead of storing the archive as a temporary file and\nextracting it only once it has arrived completely.\n\nNow, why not use the `.zip` file instead of the `.tar.gz` file? In my\nanalysis [*4*], I pointed out that the `.zip` file is about 10MB smaller,\nafter all, and BSD tar (at least the version in `C:\\Windows\\system32`) is\nable to handle those, too, right? Not so fast. In my experiments, when\nstreaming the `.zip` file to the `tar.exe -xf -` process, the `etc/`\nand `.sparse/` directories were consistently dropped. A bug, I guess. I\nran out of time to investigate this in more depth.\n\nSince the artifacts are now hosted in a GitHub Release and updated\nregularly, and since those updates cannot be atomic (you can only upload a\nrelease asset if no asset of the same name exists already, read: the\nautomation has to _delete_ that asset before uploading a new version), it\nwould also be good to kind of expect that the file may be intermittently\nabsent and add a back-off strategy [*5*].\n\n> It would be great if the build infra was shared between our CIs, so I\n> think there's some value in it. But if the answer is \"no\" then I guess\n> that's ultimately fine, as well.\n\nIt _could_ be done. But the advantages of having it versioned outside of\nthe Git repository outweigh the benefits of that shared infrastructure, I\nbelieve.\n\nCiao,\nJohannes\n\nFootnote *1*:\nhttps://github.com/git-for-windows/setup-git-for-windows-sdk/pull/965/commits/6db65223de699c4f75ab083f82f43947a53ad6ff#diff-6855ef61b94227f9264adab3ff9f2de95c2d7b4e451019cc0105896d32550eb0R58-R73\n\nFootnote *2*:\nhttps://github.com/git-for-windows/setup-git-for-windows-sdk/pull/965/commits/6db65223de699c4f75ab083f82f43947a53ad6ff#diff-6855ef61b94227f9264adab3ff9f2de95c2d7b4e451019cc0105896d32550eb0R77-R86\n\nFootnote *3*:\nhttps://github.com/git-for-windows/setup-git-for-windows-sdk/pull/965/commits/6db65223de699c4f75ab083f82f43947a53ad6ff#diff-6855ef61b94227f9264adab3ff9f2de95c2d7b4e451019cc0105896d32550eb0R88\n\nFootnote *4*:\nhttps://github.com/git-for-windows/git-sdk-64/pull/87/commits/fdb0cea373893ce7d40bcfcfbeb7fd091a3c4020\n\nFootnote *5*:\nhttps://github.com/git-for-windows/setup-git-for-windows-sdk/pull/965/commits/3d4ea07041d0740b21160a9d9a4181f569e706d8\n"}]}