{"thread":{"id":"53454","subject":"[PATCH] ci: avoid pounding on the poor ci-artifacts container","startedAt":"2020-05-12T20:47:17Z","lastAt":"2020-05-15T21:38:15Z","messageCount":9,"participants":["Johannes Schindelin via GitGitGadget","Junio C Hamano","Đoàn Trần Công Danh","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"397626","messageId":"pull.632.git.1589316430595.gitgitgadget@gmail.com","threadId":"53454","inReplyTo":null,"subject":"[PATCH] ci: avoid pounding on the poor ci-artifacts container","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-05-12T20:47:10Z","receivedAt":"2020-05-12T20:47:17Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin <johannes.schindelin@gmx.de>\n\nWhen this developer tested how the git-sdk-64-minimal artifact could be\nserved to all the GitHub workflow runs that need it, Azure Blobs looked\nlike a pretty good choice: it is reliable, fast and we already use it in\nGit for Windows to serve components like OpenSSL, cURL, etc\n\nIt came as an unpleasant surprise just _how many_ times this artifact\nwas downloaded. It exploded the bandwidth to a point where the free tier\nwould no longer be enough, threatening to block other, essential Git for\nWindows services.\n\nLet's switch back to using the Build Artifacts of our trusty Azure\nPipeline for the time being.\n\nTo avoid unnecessary hammering of the Azure Pipeline artifacts, we use\nthe GitHub Action `actions/upload-artifact` in the `windows-build` job\nand the GitHub Action `actions/download-artifact` in the `windows-test`\nand `vs-test` jobs (the latter now depends on `windows-build` for that\nreason, too).\n\nSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n---\n    ci: avoid pounding on the poor ci-artifacts container\n    \n    I did not think through how I designed the Windows job initializing the\n    minimal subset of Git for Windows' SDK (containing e.g. make, gcc etc):\n    it currently accesses Azure Blobs and uses way too much bandwidth,\n    blowing right through my quota.\n    \n    So for now, all the Windows/Visual Studio builds will be failing to\n    download that SDK subset.\n    \n    With this patch, we use the (slower) method of downloading the\n    git-sdk-64-minimal Build Artifact of our Azure Pipeline again, which\n    fixes that particular problem.\n    \n    To be sure, it is a shame that we now spend around a whole minute\n    downloading that SDK subset instead of those sweet, sweet seven seconds.\n    But what does not work does not work, and that's that.\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-632%2Fdscho%2Favoid-ci-artifacts-for-now-git.git-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-632/dscho/avoid-ci-artifacts-for-now-git.git-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/632\n\n .github/workflows/main.yml | 65 +++++++++++++++++++++++++++++++-------\n 1 file changed, 54 insertions(+), 11 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex fd4df939b50..e2e1611aa22 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -12,7 +12,24 @@ jobs:\n     - uses: actions/checkout@v1\n     - name: download git-sdk-64-minimal\n       shell: bash\n-      run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -\n+      run: |\n+        ## Add `json_pp` to the search path\n+        PATH=$PATH:/usr/bin/core_perl\n+\n+        ## Get artifact\n+        urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds\n+        id=$(curl \"$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\\$top=1\" |\n+          json_pp |\n+          sed -n 's/^         \"id\" : \\([1-9][0-9]*\\).*/\\1/p')\n+        download_url=\"$(curl \"$urlbase/$id/artifacts\" |\n+          json_pp |\n+          sed -n '/^      {/{:1;N;/\\n      }/b2;b1;:2;/\"name\" : \"git-sdk-64-minimal\"/{s/.*\"downloadUrl\" : \"\\([^\\\"]*\\).*/\\1/p}}')\"\n+        curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \\\n+          -o artifacts.zip \"$download_url\"\n+\n+        ## Unzip and remove the artifact\n+        unzip artifacts.zip\n+        rm artifacts.zip\n     - name: build\n       shell: powershell\n       env:\n@@ -30,6 +47,11 @@ jobs:\n       with:\n         name: windows-artifacts\n         path: artifacts\n+    - name: upload git-sdk-64-minimal\n+      uses: actions/upload-artifact@v1\n+      with:\n+        name: git-sdk-64-minimal\n+        path: git-sdk-64-minimal\n   windows-test:\n     runs-on: windows-latest\n     needs: [windows-build]\n@@ -38,9 +60,6 @@ jobs:\n         nr: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n     steps:\n     - uses: actions/checkout@v1\n-    - name: download git-sdk-64-minimal\n-      shell: bash\n-      run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -\n     - name: download build artifacts\n       uses: actions/download-artifact@v1\n       with:\n@@ -49,6 +68,11 @@ jobs:\n     - name: extract build artifacts\n       shell: bash\n       run: tar xf artifacts.tar.gz\n+    - name: download git-sdk-64-minimal\n+      uses: actions/download-artifact@v1\n+      with:\n+        name: git-sdk-64-minimal\n+        path: ${{github.workspace}}/git-sdk-64-minimal/\n     - name: test\n       shell: powershell\n       run: |\n@@ -79,7 +103,24 @@ jobs:\n     - uses: actions/checkout@v1\n     - name: download git-sdk-64-minimal\n       shell: bash\n-      run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -\n+      run: |\n+        ## Add `json_pp` to the search path\n+        PATH=$PATH:/usr/bin/core_perl\n+\n+        ## Get artifact\n+        urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds\n+        id=$(curl \"$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\\$top=1\" |\n+          json_pp |\n+          sed -n 's/^         \"id\" : \\([1-9][0-9]*\\).*/\\1/p')\n+        download_url=\"$(curl \"$urlbase/$id/artifacts\" |\n+          json_pp |\n+          sed -n '/^      {/{:1;N;/\\n      }/b2;b1;:2;/\"name\" : \"git-sdk-64-minimal\"/{s/.*\"downloadUrl\" : \"\\([^\\\"]*\\).*/\\1/p}}')\"\n+        curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \\\n+          -o artifacts.zip \"$download_url\"\n+\n+        ## Unzip and remove the artifact\n+        unzip artifacts.zip\n+        rm artifacts.zip\n     - name: generate Visual Studio solution\n       shell: powershell\n       run: |\n@@ -119,15 +160,17 @@ jobs:\n         path: artifacts\n   vs-test:\n     runs-on: windows-latest\n-    needs: [vs-build]\n+    needs: [vs-build, windows-build]\n     strategy:\n       matrix:\n         nr: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n     steps:\n     - uses: actions/checkout@v1\n-    - name: download git-64-portable\n-      shell: bash\n-      run: a=git-64-portable && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -\n+    - name: download git-sdk-64-minimal\n+      uses: actions/download-artifact@v1\n+      with:\n+        name: git-sdk-64-minimal\n+        path: ${{github.workspace}}/git-sdk-64-minimal/\n     - name: download build artifacts\n       uses: actions/download-artifact@v1\n       with:\n@@ -143,9 +186,9 @@ jobs:\n         NO_SVN_TESTS: 1\n         GIT_TEST_SKIP_REBASE_P: 1\n       run: |\n-        & git-64-portable\\git-cmd.exe --command=usr\\bin\\bash.exe -lc @\"\n+        & .\\git-sdk-64-minimal\\usr\\bin\\bash.exe -lc @\"\n           # Let Git ignore the SDK and the test-cache\n-          printf '%s\\n' /git-64-portable/ /test-cache/ >>.git/info/exclude\n+          printf '%s\\n' /git-sdk-64-minimal/ /test-cache/ >>.git/info/exclude\n \n           cd t &&\n           PATH=\\\"`$PWD/helper:`$PATH\\\" &&\n\nbase-commit: f72f328bc57e1b0db380ef76e0c1e94a9ed0ac7c\n-- \ngitgitgadget\n"},{"id":"397640","messageId":"xmqqimh0afkh.fsf@gitster.c.googlers.com","threadId":"53454","inReplyTo":"pull.632.git.1589316430595.gitgitgadget@gmail.com","subject":"Re: [PATCH] ci: avoid pounding on the poor ci-artifacts container","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-05-12T23:39:26Z","receivedAt":"2020-05-12T23:39:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" <gitgitgadget@gmail.com>\nwrites:\n\n> Let's switch back to using the Build Artifacts of our trusty Azure\n> Pipeline for the time being.\n>\n> To avoid unnecessary hammering of the Azure Pipeline artifacts, we use\n> the GitHub Action `actions/upload-artifact` in the `windows-build` job\n> and the GitHub Action `actions/download-artifact` in the `windows-test`\n> and `vs-test` jobs (the latter now depends on `windows-build` for that\n> reason, too).\n\nI guess this answers a question I sent earlier to the list (our\nmails almost crossed, I guess, as two of us were looking at the same\nproblem at around the same time?).\n\nHopefully when cmake-for-windows-build topic lands, this can go away\naltogether, but that is probably at least 8 weeks away (3 weeks\nremaining before the next cycle opens, plus a half of 10 week per\ncycle for a typical major release).\n\nToday's final integration (these days I'm pushing out twice or three\ntimes a day) contains this one, and it seems to have passed ;-)\n\nThanks.\n"},{"id":"397709","messageId":"20200513124406.GB1982@danh.dev","threadId":"53454","inReplyTo":"pull.632.git.1589316430595.gitgitgadget@gmail.com","subject":"Re: [PATCH] ci: avoid pounding on the poor ci-artifacts container","fromName":"Đoàn Trần Công Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2020-05-13T12:44:06Z","receivedAt":"2020-05-13T12:44:11Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2020-05-12 20:47:10+0000, Johannes Schindelin via GitGitGadget <gitgitgadget@gmail.com> wrote:\n> -      run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -\n> +      run: |\n> +        ## Add `json_pp` to the search path\n> +        PATH=$PATH:/usr/bin/core_perl\n> +\n> +        ## Get artifact\n> +        urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds\n> +        id=$(curl \"$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\\$top=1\" |\n> +          json_pp |\n> +          sed -n 's/^         \"id\" : \\([1-9][0-9]*\\).*/\\1/p')\n> +        download_url=\"$(curl \"$urlbase/$id/artifacts\" |\n> +          json_pp |\n> +          sed -n '/^      {/{:1;N;/\\n      }/b2;b1;:2;/\"name\" : \"git-sdk-64-minimal\"/{s/.*\"downloadUrl\" : \"\\([^\\\"]*\\).*/\\1/p}}')\"\n\nHi Dscho,\n\nI wonder if it's acceptable to introduce jq (already installed in\nGitHub Actions and Travis) into our codebase (only in GitHub Actions).\n\nIf yes, I think this will be easier to follow than depending on static\nnumber of space and sed branching.\n---------------------8<-----------------\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex e2e1611aa2..482df46651 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -13,17 +13,12 @@ jobs:\n     - name: download git-sdk-64-minimal\n       shell: bash\n       run: |\n-        ## Add `json_pp` to the search path\n-        PATH=$PATH:/usr/bin/core_perl\n-\n         ## Get artifact\n         urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds\n         id=$(curl \"$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\\$top=1\" |\n-          json_pp |\n-          sed -n 's/^         \"id\" : \\([1-9][0-9]*\\).*/\\1/p')\n+          jq -r \".value[] | .id\")\n         download_url=\"$(curl \"$urlbase/$id/artifacts\" |\n-          json_pp |\n-          sed -n '/^      {/{:1;N;/\\n      }/b2;b1;:2;/\"name\" : \"git-sdk-64-minimal\"/{s/.*\"downloadUrl\" : \"\\([^\\\"]*\\).*/\\1/p}}')\"\n+          jq -r '.value[] | select(.name == \"git-sdk-64-minimal\").resource.downloadUrl')\"\n         curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \\\n           -o artifacts.zip \"$download_url\"\n \n@@ -104,17 +99,12 @@ jobs:\n     - name: download git-sdk-64-minimal\n       shell: bash\n       run: |\n-        ## Add `json_pp` to the search path\n-        PATH=$PATH:/usr/bin/core_perl\n-\n         ## Get artifact\n         urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds\n         id=$(curl \"$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\\$top=1\" |\n-          json_pp |\n-          sed -n 's/^         \"id\" : \\([1-9][0-9]*\\).*/\\1/p')\n+          jq -r \".value[] | .id\")\n         download_url=\"$(curl \"$urlbase/$id/artifacts\" |\n-          json_pp |\n-          sed -n '/^      {/{:1;N;/\\n      }/b2;b1;:2;/\"name\" : \"git-sdk-64-minimal\"/{s/.*\"downloadUrl\" : \"\\([^\\\"]*\\).*/\\1/p}}')\"\n+          jq -r '.value[] | select(.name == \"git-sdk-64-minimal\").resource.downloadUrl')\"\n         curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \\\n           -o artifacts.zip \"$download_url\"\n \n----------->8------------\n\n-- \nDanh\n"},{"id":"397852","messageId":"pull.632.v2.git.1589529318969.gitgitgadget@gmail.com","threadId":"53454","inReplyTo":"pull.632.git.1589316430595.gitgitgadget@gmail.com","subject":"[PATCH v2] ci: avoid pounding on the poor ci-artifacts container","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-05-15T07:55:18Z","receivedAt":"2020-05-15T07:55:25Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin <johannes.schindelin@gmx.de>\n\nWhen this developer tested how the git-sdk-64-minimal artifact could be\nserved to all the GitHub workflow runs that need it, Azure Blobs looked\nlike a pretty good choice: it is reliable, fast and we already use it in\nGit for Windows to serve components like OpenSSL, cURL, etc\n\nIt came as an unpleasant surprise just _how many_ times this artifact\nwas downloaded. It exploded the bandwidth to a point where the free tier\nwould no longer be enough, threatening to block other, essential Git for\nWindows services.\n\nLet's switch back to using the Build Artifacts of our trusty Azure\nPipeline for the time being.\n\nTo avoid unnecessary hammering of the Azure Pipeline artifacts, we use\nthe GitHub Action `actions/upload-artifact` in the `windows-build` job\nand the GitHub Action `actions/download-artifact` in the `windows-test`\nand `vs-test` jobs (the latter now depends on `windows-build` for that\nreason, too).\n\nHelped-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>\nSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n---\n    ci: avoid pounding on the poor ci-artifacts container\n    \n    I did not think through how I designed the Windows job initializing the\n    minimal subset of Git for Windows' SDK (containing e.g. make, gcc etc):\n    it currently accesses Azure Blobs and uses way too much bandwidth,\n    blowing right through my quota.\n    \n    So for now, all the Windows/Visual Studio builds will be failing to\n    download that SDK subset.\n    \n    With this patch, we use the (slower) method of downloading the\n    git-sdk-64-minimal Build Artifact of our Azure Pipeline again, which\n    fixes that particular problem.\n    \n    To be sure, it is a shame that we now spend around a whole minute\n    downloading that SDK subset instead of those sweet, sweet seven seconds.\n    But what does not work does not work, and that's that.\n    \n    Change since v1:\n    \n     * Simplifying the scripted code by using jq (thanks, Danh!)\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-632%2Fdscho%2Favoid-ci-artifacts-for-now-git.git-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-632/dscho/avoid-ci-artifacts-for-now-git.git-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/632\n\nRange-diff vs v1:\n\n 1:  274ae816be6 ! 1:  14606cc2ed7 ci: avoid pounding on the poor ci-artifacts container\n     @@ Commit message\n          and `vs-test` jobs (the latter now depends on `windows-build` for that\n          reason, too).\n      \n     +    Helped-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>\n          Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n      \n       ## .github/workflows/main.yml ##\n     @@ .github/workflows/main.yml: jobs:\n             shell: bash\n      -      run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -\n      +      run: |\n     -+        ## Add `json_pp` to the search path\n     -+        PATH=$PATH:/usr/bin/core_perl\n     -+\n      +        ## Get artifact\n      +        urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds\n      +        id=$(curl \"$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\\$top=1\" |\n     -+          json_pp |\n     -+          sed -n 's/^         \"id\" : \\([1-9][0-9]*\\).*/\\1/p')\n     ++          jq -r \".value[] | .id\")\n      +        download_url=\"$(curl \"$urlbase/$id/artifacts\" |\n     -+          json_pp |\n     -+          sed -n '/^      {/{:1;N;/\\n      }/b2;b1;:2;/\"name\" : \"git-sdk-64-minimal\"/{s/.*\"downloadUrl\" : \"\\([^\\\"]*\\).*/\\1/p}}')\"\n     ++          jq -r '.value[] | select(.name == \"git-sdk-64-minimal\").resource.downloadUrl')\"\n      +        curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \\\n      +          -o artifacts.zip \"$download_url\"\n      +\n     @@ .github/workflows/main.yml: jobs:\n             shell: bash\n      -      run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -\n      +      run: |\n     -+        ## Add `json_pp` to the search path\n     -+        PATH=$PATH:/usr/bin/core_perl\n     -+\n      +        ## Get artifact\n      +        urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds\n      +        id=$(curl \"$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\\$top=1\" |\n     -+          json_pp |\n     -+          sed -n 's/^         \"id\" : \\([1-9][0-9]*\\).*/\\1/p')\n     ++          jq -r \".value[] | .id\")\n      +        download_url=\"$(curl \"$urlbase/$id/artifacts\" |\n     -+          json_pp |\n     -+          sed -n '/^      {/{:1;N;/\\n      }/b2;b1;:2;/\"name\" : \"git-sdk-64-minimal\"/{s/.*\"downloadUrl\" : \"\\([^\\\"]*\\).*/\\1/p}}')\"\n     ++          jq -r '.value[] | select(.name == \"git-sdk-64-minimal\").resource.downloadUrl')\"\n      +        curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \\\n      +          -o artifacts.zip \"$download_url\"\n      +\n\n\n .github/workflows/main.yml | 55 ++++++++++++++++++++++++++++++--------\n 1 file changed, 44 insertions(+), 11 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex fd4df939b50..482df466510 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -12,7 +12,19 @@ jobs:\n     - uses: actions/checkout@v1\n     - name: download git-sdk-64-minimal\n       shell: bash\n-      run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -\n+      run: |\n+        ## Get artifact\n+        urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds\n+        id=$(curl \"$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\\$top=1\" |\n+          jq -r \".value[] | .id\")\n+        download_url=\"$(curl \"$urlbase/$id/artifacts\" |\n+          jq -r '.value[] | select(.name == \"git-sdk-64-minimal\").resource.downloadUrl')\"\n+        curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \\\n+          -o artifacts.zip \"$download_url\"\n+\n+        ## Unzip and remove the artifact\n+        unzip artifacts.zip\n+        rm artifacts.zip\n     - name: build\n       shell: powershell\n       env:\n@@ -30,6 +42,11 @@ jobs:\n       with:\n         name: windows-artifacts\n         path: artifacts\n+    - name: upload git-sdk-64-minimal\n+      uses: actions/upload-artifact@v1\n+      with:\n+        name: git-sdk-64-minimal\n+        path: git-sdk-64-minimal\n   windows-test:\n     runs-on: windows-latest\n     needs: [windows-build]\n@@ -38,9 +55,6 @@ jobs:\n         nr: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n     steps:\n     - uses: actions/checkout@v1\n-    - name: download git-sdk-64-minimal\n-      shell: bash\n-      run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -\n     - name: download build artifacts\n       uses: actions/download-artifact@v1\n       with:\n@@ -49,6 +63,11 @@ jobs:\n     - name: extract build artifacts\n       shell: bash\n       run: tar xf artifacts.tar.gz\n+    - name: download git-sdk-64-minimal\n+      uses: actions/download-artifact@v1\n+      with:\n+        name: git-sdk-64-minimal\n+        path: ${{github.workspace}}/git-sdk-64-minimal/\n     - name: test\n       shell: powershell\n       run: |\n@@ -79,7 +98,19 @@ jobs:\n     - uses: actions/checkout@v1\n     - name: download git-sdk-64-minimal\n       shell: bash\n-      run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -\n+      run: |\n+        ## Get artifact\n+        urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds\n+        id=$(curl \"$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\\$top=1\" |\n+          jq -r \".value[] | .id\")\n+        download_url=\"$(curl \"$urlbase/$id/artifacts\" |\n+          jq -r '.value[] | select(.name == \"git-sdk-64-minimal\").resource.downloadUrl')\"\n+        curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \\\n+          -o artifacts.zip \"$download_url\"\n+\n+        ## Unzip and remove the artifact\n+        unzip artifacts.zip\n+        rm artifacts.zip\n     - name: generate Visual Studio solution\n       shell: powershell\n       run: |\n@@ -119,15 +150,17 @@ jobs:\n         path: artifacts\n   vs-test:\n     runs-on: windows-latest\n-    needs: [vs-build]\n+    needs: [vs-build, windows-build]\n     strategy:\n       matrix:\n         nr: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n     steps:\n     - uses: actions/checkout@v1\n-    - name: download git-64-portable\n-      shell: bash\n-      run: a=git-64-portable && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -\n+    - name: download git-sdk-64-minimal\n+      uses: actions/download-artifact@v1\n+      with:\n+        name: git-sdk-64-minimal\n+        path: ${{github.workspace}}/git-sdk-64-minimal/\n     - name: download build artifacts\n       uses: actions/download-artifact@v1\n       with:\n@@ -143,9 +176,9 @@ jobs:\n         NO_SVN_TESTS: 1\n         GIT_TEST_SKIP_REBASE_P: 1\n       run: |\n-        & git-64-portable\\git-cmd.exe --command=usr\\bin\\bash.exe -lc @\"\n+        & .\\git-sdk-64-minimal\\usr\\bin\\bash.exe -lc @\"\n           # Let Git ignore the SDK and the test-cache\n-          printf '%s\\n' /git-64-portable/ /test-cache/ >>.git/info/exclude\n+          printf '%s\\n' /git-sdk-64-minimal/ /test-cache/ >>.git/info/exclude\n \n           cd t &&\n           PATH=\\\"`$PWD/helper:`$PATH\\\" &&\n\nbase-commit: f72f328bc57e1b0db380ef76e0c1e94a9ed0ac7c\n-- \ngitgitgadget\n"},{"id":"397874","messageId":"nycvar.QRO.7.76.6.2005151557130.55@tvgsbejvaqbjf.bet","threadId":"53454","inReplyTo":"xmqqimh0afkh.fsf@gitster.c.googlers.com","subject":"Re: [PATCH] ci: avoid pounding on the poor ci-artifacts container","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-05-15T13:59:42Z","receivedAt":"2020-05-15T13:59:57Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Tue, 12 May 2020, Junio C Hamano wrote:\n\n> \"Johannes Schindelin via GitGitGadget\" <gitgitgadget@gmail.com>\n> writes:\n>\n> > Let's switch back to using the Build Artifacts of our trusty Azure\n> > Pipeline for the time being.\n> >\n> > To avoid unnecessary hammering of the Azure Pipeline artifacts, we use\n> > the GitHub Action `actions/upload-artifact` in the `windows-build` job\n> > and the GitHub Action `actions/download-artifact` in the `windows-test`\n> > and `vs-test` jobs (the latter now depends on `windows-build` for that\n> > reason, too).\n>\n> I guess this answers a question I sent earlier to the list (our\n> mails almost crossed, I guess, as two of us were looking at the same\n> problem at around the same time?).\n\nI am terribly sorry, but I did not get to read the Git mailing list at all\nthis week (or for that matter, my private mail). So I would not even have\nseen your message... :-(\n\n> Hopefully when cmake-for-windows-build topic lands, this can go away\n> altogether, but that is probably at least 8 weeks away (3 weeks\n> remaining before the next cycle opens, plus a half of 10 week per\n> cycle for a typical major release).\n\nThe `cmake-for-windows-build` would address only the build part for Visual\nStudio. The regular Windows build, as well as the parallelized tests\n_still_ need the `git-sdk-64-minimal` artifact. With or without CMake.\nThat's because neither CMake nor Visual Studio can accommodate the fact\nthat our test suite is implemented in shell script _and_ requires a\nworking Perl interpreter.\n\n> Today's final integration (these days I'm pushing out twice or three\n> times a day) contains this one, and it seems to have passed ;-)\n\nExcellent!\n\nThanks,\nDscho\n"},{"id":"397875","messageId":"nycvar.QRO.7.76.6.2005151600010.55@tvgsbejvaqbjf.bet","threadId":"53454","inReplyTo":"20200513124406.GB1982@danh.dev","subject":"Re: [PATCH] ci: avoid pounding on the poor ci-artifacts container","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-05-15T14:01:16Z","receivedAt":"2020-05-15T14:01:23Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Danh,\n\nOn Wed, 13 May 2020, Đoàn Trần Công Danh wrote:\n\n> On 2020-05-12 20:47:10+0000, Johannes Schindelin via GitGitGadget <gitgitgadget@gmail.com> wrote:\n> > -      run: a=git-sdk-64-minimal && mkdir -p $a && curl -# https://wingit.blob.core.windows.net/ci-artifacts/$a.tar.xz | tar -C $a -xJf -\n> > +      run: |\n> > +        ## Add `json_pp` to the search path\n> > +        PATH=$PATH:/usr/bin/core_perl\n> > +\n> > +        ## Get artifact\n> > +        urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds\n> > +        id=$(curl \"$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\\$top=1\" |\n> > +          json_pp |\n> > +          sed -n 's/^         \"id\" : \\([1-9][0-9]*\\).*/\\1/p')\n> > +        download_url=\"$(curl \"$urlbase/$id/artifacts\" |\n> > +          json_pp |\n> > +          sed -n '/^      {/{:1;N;/\\n      }/b2;b1;:2;/\"name\" : \"git-sdk-64-minimal\"/{s/.*\"downloadUrl\" : \"\\([^\\\"]*\\).*/\\1/p}}')\"\n>\n> Hi Dscho,\n>\n> I wonder if it's acceptable to introduce jq (already installed in\n> GitHub Actions and Travis) into our codebase (only in GitHub Actions).\n>\n> If yes, I think this will be easier to follow than depending on static\n> number of space and sed branching.\n> ---------------------8<-----------------\n> diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\n> index e2e1611aa2..482df46651 100644\n> --- a/.github/workflows/main.yml\n> +++ b/.github/workflows/main.yml\n> @@ -13,17 +13,12 @@ jobs:\n>      - name: download git-sdk-64-minimal\n>        shell: bash\n>        run: |\n> -        ## Add `json_pp` to the search path\n> -        PATH=$PATH:/usr/bin/core_perl\n> -\n>          ## Get artifact\n>          urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds\n>          id=$(curl \"$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\\$top=1\" |\n> -          json_pp |\n> -          sed -n 's/^         \"id\" : \\([1-9][0-9]*\\).*/\\1/p')\n> +          jq -r \".value[] | .id\")\n>          download_url=\"$(curl \"$urlbase/$id/artifacts\" |\n> -          json_pp |\n> -          sed -n '/^      {/{:1;N;/\\n      }/b2;b1;:2;/\"name\" : \"git-sdk-64-minimal\"/{s/.*\"downloadUrl\" : \"\\([^\\\"]*\\).*/\\1/p}}')\"\n> +          jq -r '.value[] | select(.name == \"git-sdk-64-minimal\").resource.downloadUrl')\"\n>          curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \\\n>            -o artifacts.zip \"$download_url\"\n>\n> @@ -104,17 +99,12 @@ jobs:\n>      - name: download git-sdk-64-minimal\n>        shell: bash\n>        run: |\n> -        ## Add `json_pp` to the search path\n> -        PATH=$PATH:/usr/bin/core_perl\n> -\n>          ## Get artifact\n>          urlbase=https://dev.azure.com/git-for-windows/git/_apis/build/builds\n>          id=$(curl \"$urlbase?definitions=22&statusFilter=completed&resultFilter=succeeded&\\$top=1\" |\n> -          json_pp |\n> -          sed -n 's/^         \"id\" : \\([1-9][0-9]*\\).*/\\1/p')\n> +          jq -r \".value[] | .id\")\n>          download_url=\"$(curl \"$urlbase/$id/artifacts\" |\n> -          json_pp |\n> -          sed -n '/^      {/{:1;N;/\\n      }/b2;b1;:2;/\"name\" : \"git-sdk-64-minimal\"/{s/.*\"downloadUrl\" : \"\\([^\\\"]*\\).*/\\1/p}}')\"\n> +          jq -r '.value[] | select(.name == \"git-sdk-64-minimal\").resource.downloadUrl')\"\n>          curl --connect-timeout 10 --retry 5 --retry-delay 0 --retry-max-time 240 \\\n>            -o artifacts.zip \"$download_url\"\n>\n> ----------->8------------\n\nThank you for that. Indeed, I should have checked whether `jq` is\navailable on the Windows build agents (and then I should have learned how\nto hold that tool right).\n\nI sent out a v2 with this change, and I already merged that version into\nGit for Windows' `master` branch in preparation for v2.27.0-rc0.\n\nThanks,\nDscho\n"},{"id":"397883","messageId":"xmqq4kshnsjz.fsf@gitster.c.googlers.com","threadId":"53454","inReplyTo":"pull.632.v2.git.1589529318969.gitgitgadget@gmail.com","subject":"Re: [PATCH v2] ci: avoid pounding on the poor ci-artifacts container","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-05-15T15:09:36Z","receivedAt":"2020-05-15T15:11:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" <gitgitgadget@gmail.com>\nwrites:\n\n> From: Johannes Schindelin <johannes.schindelin@gmx.de>\n>\n> When this developer tested how the git-sdk-64-minimal artifact could be\n> served to all the GitHub workflow runs that need it, Azure Blobs looked\n> like a pretty good choice: it is reliable, fast and we already use it in\n> Git for Windows to serve components like OpenSSL, cURL, etc\n>\n> It came as an unpleasant surprise just _how many_ times this artifact\n> was downloaded. It exploded the bandwidth to a point where the free tier\n> would no longer be enough, threatening to block other, essential Git for\n> Windows services.\n>\n> Let's switch back to using the Build Artifacts of our trusty Azure\n> Pipeline for the time being.\n>\n> To avoid unnecessary hammering of the Azure Pipeline artifacts, we use\n> the GitHub Action `actions/upload-artifact` in the `windows-build` job\n> and the GitHub Action `actions/download-artifact` in the `windows-test`\n> and `vs-test` jobs (the latter now depends on `windows-build` for that\n> reason, too).\n>\n> Helped-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>\n> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n> ---\n>     \n>     Change since v1:\n>     \n>      * Simplifying the scripted code by using jq (thanks, Danh!)\n\nThanks, both.  Will replace and let's merge it down before -rc1.\n\n"},{"id":"397938","messageId":"nycvar.QRO.7.76.6.2005152334550.55@tvgsbejvaqbjf.bet","threadId":"53454","inReplyTo":"xmqq4kshnsjz.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2] ci: avoid pounding on the poor ci-artifacts container","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-05-15T21:35:06Z","receivedAt":"2020-05-15T21:35:16Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Fri, 15 May 2020, Junio C Hamano wrote:\n\n> \"Johannes Schindelin via GitGitGadget\" <gitgitgadget@gmail.com>\n> writes:\n>\n> > From: Johannes Schindelin <johannes.schindelin@gmx.de>\n> >\n> > When this developer tested how the git-sdk-64-minimal artifact could be\n> > served to all the GitHub workflow runs that need it, Azure Blobs looked\n> > like a pretty good choice: it is reliable, fast and we already use it in\n> > Git for Windows to serve components like OpenSSL, cURL, etc\n> >\n> > It came as an unpleasant surprise just _how many_ times this artifact\n> > was downloaded. It exploded the bandwidth to a point where the free tier\n> > would no longer be enough, threatening to block other, essential Git for\n> > Windows services.\n> >\n> > Let's switch back to using the Build Artifacts of our trusty Azure\n> > Pipeline for the time being.\n> >\n> > To avoid unnecessary hammering of the Azure Pipeline artifacts, we use\n> > the GitHub Action `actions/upload-artifact` in the `windows-build` job\n> > and the GitHub Action `actions/download-artifact` in the `windows-test`\n> > and `vs-test` jobs (the latter now depends on `windows-build` for that\n> > reason, too).\n> >\n> > Helped-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>\n> > Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n> > ---\n> >\n> >     Change since v1:\n> >\n> >      * Simplifying the scripted code by using jq (thanks, Danh!)\n>\n> Thanks, both.  Will replace and let's merge it down before -rc1.\n\nThank you!\nDscho\n"},{"id":"397939","messageId":"xmqqimgwlvzx.fsf@gitster.c.googlers.com","threadId":"53454","inReplyTo":"nycvar.QRO.7.76.6.2005152334550.55@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2] ci: avoid pounding on the poor ci-artifacts container","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-05-15T21:38:10Z","receivedAt":"2020-05-15T21:38:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> Hi Junio,\n>\n> On Fri, 15 May 2020, Junio C Hamano wrote:\n>\n>> Thanks, both.  Will replace and let's merge it down before -rc1.\n\n'next' passes with this\n\n https://github.com/git/git/runs/679473788\n https://github.com/git/git/runs/679473866\n\netc.\n\nThanks.\n"}]}