[PATCH v3 0/7] ci: some speedups for slow tests
- From
Patrick Steinhardt <ps@pks.im>
- Date
- Nov 11, 2026, 10:47 UTC
- Message-ID
- <20261111-pks-gitlab-ci-speedups-v3-0-d1e34117ee2e@pks.im>
- In-Reply-To
- <20261106-pks-gitlab-ci-speedups-v1-0-1fa0374c7287@pks.im>
Hi,
I was staring at CI way more often recently, and that got me annoyed real fast because it's just painstakingly slow. So I decided to have a look and try do something about it, the result of which is this patch series.
The series contains a couple of bits that are specific to GitLab, and a couple of fixes that are generic. Some highlights:
- Installing the Git for Windows SDK speeds up from 3 minutes to 10
seconds by using tar(1) instead of Extract-Archive. - t0027 speeds up from >10 minutes to 6 minutes on Windows by avoiding
the overhead of spawning redundant processes. - t5551 and t5559 both speed up by a factor of 10 on macOS, going from
20 minutes each to 2 minutes each by using reftables in a repo with
100k refs. "osx-meson" and "osx-clang" thus go from 29 minutes to 18
minutes end-to-end. - "fedora-breaking-changes-meson" speeds up by a factor of 2, going
from 35 minutes to 16 minutes. Turns out our SHA256 implementation
is quite slow with "-O0".Unfortunately, our Windows runners at GitLab are still atrociously slow by themselves. So the critical path only improves by about ~10 minutes, going from 60-70 minutes to 53 minutes end-to-end. But hey, at least the other jobs now give feedback a bit faster. And I guess on GitHub it may be a different story because Windows runners are way better there and are typically not on the critical path.
A test run can be found at [1].
Changes in v3: - Document why Windows' tar version can extract ZIP archives. - Clear `paths` variable. - Link to v2: https://patch.msgid.link/20261109-pks-gitlab-ci-speedups-v2-0-508832602d0b@pks.im
Changes in v2: - Drop the cache for the GfW SDK again. - Drop repetitive paths in t0027. - Link to v1: https://patch.msgid.link/20261106-pks-gitlab-ci-speedups-v1-0-1fa0374c7287@pks.im
Thanks!
Patrick
[1]: https://gitlab.com/gitlab-org/git/-/merge_requests/734
---
Patrick Steinhardt (7):
gitlab-ci: speed up installation of Git for Windows SDK
gitlab-ci: exclude intermediate outputs from Windows artifacts
ci: build optimized binaries with Meson
t0027: optimize how we reset repository state
t0027: cache expected output files
t1092: set up repositories from templates
t5551: use reftables for repositories with many refs.gitlab-ci.yml | 4 +++- ci/install-sdk.ps1 | 6 +++++- ci/run-build-and-tests.sh | 1 + t/t0027-auto-crlf.sh | 27 ++++++++++++++----------- t/t1092-sparse-checkout-compatibility.sh | 34 +++++++++++++++++++------------- t/t5551-http-fetch-smart.sh | 6 ++++-- 6 files changed, 48 insertions(+), 30 deletions(-)
Range-diff versus v2:
1: 41c3a0e8d7d ! 1: 8fabb336df3 gitlab-ci: speed up installation of Git for Windows SDK
@@ Commit message
Windows SDK on every run and then extracts it via `Expand-Archive`,
which is expensive.
- Improve this by using tar(1) instead of `Expand-Archive`. While at it,
- make the script error on failures and suppress progress output.
+ Improve this by using tar(1) instead of `Expand-Archive`. Note that
+ we're using it to extract a ZIP archive, which may seem puzzling at
+ first. But Windows ships bsdtar by default [1], which also supports ZIP
+ and 7zip files.
+
+ While at it, make the script error on failures and suppress progress
+ output.
+
+ [1]: https://learn.microsoft.com/en-us/windows/tar/
Signed-off-by: Patrick Steinhardt <ps@pks.im>
2: 439c12cb15d = 2: 4a4d1e6f2a8 gitlab-ci: exclude intermediate outputs from Windows artifacts
3: 070bef6fecf = 3: c23b7a7c58d ci: build optimized binaries with Meson
4: 598a143fc24 ! 4: 942d70be750 t0027: optimize how we reset repository state
@@ Commit message
## t/t0027-auto-crlf.sh ##
@@ t/t0027-auto-crlf.sh: checkout_files () {
+ git config core.autocrlf $crlf
+ '
pfx=eol_${ceol}_crlf_${crlf}_attr_${attr}_ &&
++ paths=
for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
do
- test_expect_success "setup $f checkout ${ceol:+ with -c core.eol=$ceol}" '
5: a325efbaddb = 5: 460cd0c88fd t0027: cache expected output files
6: a2d33441e50 = 6: 44c77717749 t1092: set up repositories from templates
7: bfc93fdfb4e = 7: d64a61efec2 t5551: use reftables for repositories with many refs--- base-commit: 0ffa3e35c7c82ab8d0de226149fb03b3f7674d05 change-id: 20261106-pks-gitlab-ci-speedups-09c3841eb178