git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 0/7] ci: some speedups for slow tests

From
Patrick Steinhardt <ps@pks.im>
Date
Nov 11, 2026, 10:47 UTC
Message-ID
<20261111-pks-gitlab-ci-speedups-v3-0-d1e34117ee2e@pks.im>
In-Reply-To
<20261106-pks-gitlab-ci-speedups-v1-0-1fa0374c7287@pks.im>
Hi,

I was staring at CI way more often recently, and that got me annoyed real fast because it's just painstakingly slow. So I decided to have a look and try do something about it, the result of which is this patch series.

The series contains a couple of bits that are specific to GitLab, and a couple of fixes that are generic. Some highlights:

  - Installing the Git for Windows SDK speeds up from 3 minutes to 10
    seconds by using tar(1) instead of Extract-Archive.
  - t0027 speeds up from >10 minutes to 6 minutes on Windows by avoiding
    the overhead of spawning redundant processes.
  - t5551 and t5559 both speed up by a factor of 10 on macOS, going from
    20 minutes each to 2 minutes each by using reftables in a repo with
    100k refs. "osx-meson" and "osx-clang" thus go from 29 minutes to 18
    minutes end-to-end.
  - "fedora-breaking-changes-meson" speeds up by a factor of 2, going
    from 35 minutes to 16 minutes. Turns out our SHA256 implementation
    is quite slow with "-O0".

Unfortunately, our Windows runners at GitLab are still atrociously slow by themselves. So the critical path only improves by about ~10 minutes, going from 60-70 minutes to 53 minutes end-to-end. But hey, at least the other jobs now give feedback a bit faster. And I guess on GitHub it may be a different story because Windows runners are way better there and are typically not on the critical path.

A test run can be found at [1].
Changes in v3:
  - Document why Windows' tar version can extract ZIP archives.
  - Clear `paths` variable.
  - Link to v2: https://patch.msgid.link/20261109-pks-gitlab-ci-speedups-v2-0-508832602d0b@pks.im
Changes in v2:
  - Drop the cache for the GfW SDK again.
  - Drop repetitive paths in t0027.
  - Link to v1: https://patch.msgid.link/20261106-pks-gitlab-ci-speedups-v1-0-1fa0374c7287@pks.im
Thanks!
Patrick
[1]: https://gitlab.com/gitlab-org/git/-/merge_requests/734
---
Patrick Steinhardt (7):
      gitlab-ci: speed up installation of Git for Windows SDK
      gitlab-ci: exclude intermediate outputs from Windows artifacts
      ci: build optimized binaries with Meson
      t0027: optimize how we reset repository state
      t0027: cache expected output files
      t1092: set up repositories from templates
      t5551: use reftables for repositories with many refs
 .gitlab-ci.yml                           |  4 +++-
 ci/install-sdk.ps1                       |  6 +++++-
 ci/run-build-and-tests.sh                |  1 +
 t/t0027-auto-crlf.sh                     | 27 ++++++++++++++-----------
 t/t1092-sparse-checkout-compatibility.sh | 34 +++++++++++++++++++-------------
 t/t5551-http-fetch-smart.sh              |  6 ++++--
 6 files changed, 48 insertions(+), 30 deletions(-)
Range-diff versus v2:
1:  41c3a0e8d7d ! 1:  8fabb336df3 gitlab-ci: speed up installation of Git for Windows SDK
    @@ Commit message
         Windows SDK on every run and then extracts it via `Expand-Archive`,
         which is expensive.
     
    -    Improve this by using tar(1) instead of `Expand-Archive`. While at it,
    -    make the script error on failures and suppress progress output.
    +    Improve this by using tar(1) instead of `Expand-Archive`. Note that
    +    we're using it to extract a ZIP archive, which may seem puzzling at
    +    first. But Windows ships bsdtar by default [1], which also supports ZIP
    +    and 7zip files.
    +
    +    While at it, make the script error on failures and suppress progress
    +    output.
    +
    +    [1]: https://learn.microsoft.com/en-us/windows/tar/
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
2:  439c12cb15d = 2:  4a4d1e6f2a8 gitlab-ci: exclude intermediate outputs from Windows artifacts
3:  070bef6fecf = 3:  c23b7a7c58d ci: build optimized binaries with Meson
4:  598a143fc24 ! 4:  942d70be750 t0027: optimize how we reset repository state
    @@ Commit message
     
      ## t/t0027-auto-crlf.sh ##
     @@ t/t0027-auto-crlf.sh: checkout_files () {
    + 		git config core.autocrlf $crlf
    + 	'
      	pfx=eol_${ceol}_crlf_${crlf}_attr_${attr}_ &&
    ++	paths=
      	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
      	do
     -		test_expect_success "setup $f checkout ${ceol:+ with -c core.eol=$ceol}"  '
5:  a325efbaddb = 5:  460cd0c88fd t0027: cache expected output files
6:  a2d33441e50 = 6:  44c77717749 t1092: set up repositories from templates
7:  bfc93fdfb4e = 7:  d64a61efec2 t5551: use reftables for repositories with many refs

--- base-commit: 0ffa3e35c7c82ab8d0de226149fb03b3f7674d05 change-id: 20261106-pks-gitlab-ci-speedups-09c3841eb178

Previous: Johannes SixtNext: Patrick Steinhardt
Message 33 of 40 in “ci: some speedups for slow tests”
  1. 0/7 ci: some speedups for slow testsPatrick Steinhardt, Nov 6, 2026
  2. 1/7 gitlab-ci: speed up installation of Git for Windows SDKPatrick Steinhardt, Nov 6, 2026
  3. Junio C HamanoNov 6, 2026
  4. Patrick SteinhardtNov 9, 2026
  5. 2/7 gitlab-ci: exclude intermediate outputs from Windows artifactsPatrick Steinhardt, Nov 6, 2026
  6. 3/7 ci: build optimized binaries with MesonPatrick Steinhardt, Nov 6, 2026
  7. Junio C HamanoNov 6, 2026
  8. 4/7 t0027: optimize how we reset repository statePatrick Steinhardt, Nov 6, 2026
  9. Junio C HamanoNov 6, 2026
  10. Torsten BögershausenNov 8, 2026
  11. Patrick SteinhardtNov 9, 2026
  12. 5/7 t0027: cache expected output filesPatrick Steinhardt, Nov 6, 2026
  13. 6/7 t1092: set up repositories from templatesPatrick Steinhardt, Nov 6, 2026
  14. 7/7 t5551: use reftables for repositories with many refsPatrick Steinhardt, Nov 6, 2026
  15. Junio C HamanoNov 6, 2026
  16. 0/7 ci: some speedups for slow testsPatrick Steinhardt, Nov 9, 2026
  17. 1/7 gitlab-ci: speed up installation of Git for Windows SDKPatrick Steinhardt, Nov 9, 2026
  18. Junio C HamanoNov 9, 2026
  19. Patrick SteinhardtNov 10, 2026
  20. Junio C HamanoNov 10, 2026
  21. 2/7 gitlab-ci: exclude intermediate outputs from Windows artifactsPatrick Steinhardt, Nov 9, 2026
  22. 3/7 ci: build optimized binaries with MesonPatrick Steinhardt, Nov 9, 2026
  23. 4/7 t0027: optimize how we reset repository statePatrick Steinhardt, Nov 9, 2026
  24. Junio C HamanoNov 9, 2026
  25. Patrick SteinhardtNov 10, 2026
  26. 5/7 t0027: cache expected output filesPatrick Steinhardt, Nov 9, 2026
  27. Torsten BögershausenNov 9, 2026
  28. Patrick SteinhardtNov 10, 2026
  29. Torsten BögershausenNov 10, 2026
  30. 6/7 t1092: set up repositories from templatesPatrick Steinhardt, Nov 9, 2026
  31. 7/7 t5551: use reftables for repositories with many refsPatrick Steinhardt, Nov 9, 2026
  32. Johannes SixtNov 12, 2026
  33. 0/7 ci: some speedups for slow testsPatrick Steinhardt, Nov 11, 2026
  34. 1/7 gitlab-ci: speed up installation of Git for Windows SDKPatrick Steinhardt, Nov 11, 2026
  35. 2/7 gitlab-ci: exclude intermediate outputs from Windows artifactsPatrick Steinhardt, Nov 11, 2026
  36. 3/7 ci: build optimized binaries with MesonPatrick Steinhardt, Nov 11, 2026
  37. 4/7 t0027: optimize how we reset repository statePatrick Steinhardt, Nov 11, 2026
  38. 5/7 t0027: cache expected output filesPatrick Steinhardt, Nov 11, 2026
  39. 6/7 t1092: set up repositories from templatesPatrick Steinhardt, Nov 11, 2026
  40. 7/7 t5551: use reftables for repositories with many refsPatrick Steinhardt, Nov 11, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.