threads / patch / 66787

patch, 7 partsci: some speedups for slow tests

Subject: [PATCH 0/7] ci: some speedups for slow tests

## tl;dr

Patrick Steinhardt's seven patches aim to cut CI time, with reported gains of minutes to tens of minutes on some jobs. Read the story.

replies: 39people: 4as markdown or json

Patrick Steinhardt· Nov 6, 2026, 10:30 UTC · lore
Hi,

I was staring at CI way more often recently, and that got me annoyed real fast because it's just painstakingly slow. So I decided to have a look and try do something about it, the result of which is this patch series.

The series contains a couple of bits that are specific to GitLab, and a couple of fixes that are generic. Some highlights:

  - Installing the Git for Windows SDK speeds up from 3 minutes to 10
    seconds by using tar(1) instead of Extract-Archive.
  - t0027 speeds up from >10 minutes to 6 minutes on Windows by avoiding
    the overhead of spawning redundant processes.
  - t5551 and t5559 both speed up by a factor of 10 on macOS, going from
    20 minutes each to 2 minutes each by using reftables in a repo with
    100k refs. "osx-meson" and "osx-clang" thus go from 29 minutes to 18
    minutes end-to-end.
  - "fedora-breaking-changes-meson" speeds up by a factor of 2, going
    from 35 minutes to 16 minutes. Turns out our SHA256 implementation
    is quite slow with "-O0".

Unfortunately, our Windows runners at GitLab are still atrociously slow by themselves. So the critical path only improves by about ~10 minutes, going from 60-70 minutes to 53 minutes end-to-end. But hey, at least the other jobs now give feedback a bit faster. And I guess on GitHub it may be a different story because Windows runners are way better there and are typically not on the critical path.

A test run can be found at [1].
Thanks!
Patrick
[1]: https://gitlab.com/gitlab-org/git/-/merge_requests/734
---
Patrick Steinhardt (7):
      gitlab-ci: speed up installation of Git for Windows SDK
      gitlab-ci: exclude intermediate outputs from Windows artifacts
      ci: build optimized binaries with Meson
      t0027: optimize how we reset repository state
      t0027: cache expected output files
      t1092: set up repositories from templates
      t5551: use reftables for repositories with many refs
 .gitlab-ci.yml                           | 13 ++++++++++--
 ci/install-sdk.ps1                       | 26 ++++++++++++++++++------
 ci/run-build-and-tests.sh                |  1 +
 t/t0027-auto-crlf.sh                     | 33 +++++++++++++++++--------------
 t/t1092-sparse-checkout-compatibility.sh | 34 +++++++++++++++++++-------------
 t/t5551-http-fetch-smart.sh              |  6 ++++--
 6 files changed, 74 insertions(+), 39 deletions(-)

--- base-commit: 0ffa3e35c7c82ab8d0de226149fb03b3f7674d05 change-id: 20261106-pks-gitlab-ci-speedups-09c3841eb178

Patrick Steinhardt· Nov 6, 2026, 10:30 UTC · re: Patrick Steinhardt · lore

[PATCH 1/7] gitlab-ci: speed up installation of Git for Windows SDK

The "build:mingw64" job on GitLab CI downloads the minimal Git for Windows SDK on every run and then extracts it via `Expand-Archive`, which is expensive.

Improve this by caching the SDK and by using tar(1) instead of `Expand-Archive`. Note that caching actually only gives us a win of a couple seconds. The bigger win here is using tar(1), which speeds up extracting the archive from 3 minutes to about 10 seconds. Regardless of that though, caching the SDK feels like a sensible thing to do as it removes one step that may spuriously fail.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml     |  9 ++++++++-
 ci/install-sdk.ps1 | 26 ++++++++++++++++++++------
 2 files changed, 28 insertions(+), 7 deletions(-)
Show changes to 2 files +28 −7

.gitlab-ci.yml, ci/install-sdk.ps1

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 9b7698a0037..15419a954b3 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -127,9 +127,16 @@ build:mingw64:
     - saas-windows-medium-amd64
   variables:
     NO_PERL: 1
+  cache:
+    key:
+      files:
+        - ci/install-sdk.ps1
+        - ci/install-dependencies.ps1
+    paths:
+      - .dependencies
   before_script:
     - *windows_before_script
-    - ./ci/install-sdk.ps1 -directory "git-sdk"
+    - ./ci/install-sdk.ps1 -Directory "git-sdk"
     - ./ci/install-dependencies.ps1 -Mingw
   script:
     # The minimal SDK's profile resets PATH.
diff --git a/ci/install-sdk.ps1 b/ci/install-sdk.ps1
index 66f24838a40..1dbcef04a18 100755
--- a/ci/install-sdk.ps1
+++ b/ci/install-sdk.ps1
@@ -1,12 +1,26 @@
 param(
-    [string]$directory='git-sdk',
-    [string]$url='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip'
+    [string]$Directory='git-sdk',
+    [string]$URL='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip',
+    [string]$DownloadDirectory='.dependencies'
 )
 
-Invoke-WebRequest "$url" -OutFile git-sdk.zip
-Expand-Archive -LiteralPath git-sdk.zip -DestinationPath "$directory"
-Remove-Item -Path git-sdk.zip
+$ErrorActionPreference = 'Stop'
+$ProgressPreference = 'SilentlyContinue'
+
+New-Item -Path $DownloadDirectory -ItemType Directory -Force
+
+$archive = Join-Path $DownloadDirectory 'git-sdk.zip'
+if (-not (Test-Path $archive)) {
+    Invoke-WebRequest "$URL" -OutFile $archive -TimeoutSec 300
+}
+
+New-Item -Path $Directory -ItemType Directory -Force
+tar -xf $archive -C $Directory
+if ($LASTEXITCODE -ne 0) {
+    throw "Failed to extract $archive with exit code $LASTEXITCODE"
+}
 
 New-Item -Path .git/info -ItemType Directory -Force
 New-Item -Path .git/info/exclude -ItemType File -Force
-Add-Content -Path .git/info/exclude -Value "/$directory"
+Add-Content -Path .git/info/exclude -Value "/$Directory"
+Add-Content -Path .git/info/exclude -Value "/$DownloadDirectory"
-- 
2.56.0.500.g5d06dc02cd.dirty
Junio C Hamano· Nov 6, 2026, 17:20 UTC · re: Patrick Steinhardt · lore

Re: [PATCH 1/7] gitlab-ci: speed up installation of Git for Windows SDK

Patrick Steinhardt <ps@pks.im> writes:
Show 9 quoted lines
> The "build:mingw64" job on GitLab CI downloads the minimal Git for
> Windows SDK on every run and then extracts it via `Expand-Archive`,
> which is expensive.
>
> Improve this by caching the SDK and by using tar(1) instead of
> `Expand-Archive`.
> Note that caching actually only gives us a win of a
> couple seconds. The bigger win here is using tar(1), which speeds up
> extracting the archive from 3 minutes to about 10 seconds. Regardless of

Wow, is a tool to move the bits from one part of a rusted iron surface to another part really so much more expensive compared to another tool to move the same bits, to make a difference that is worth doing this?

> that though, caching the SDK feels like a sensible thing to do as it
> removes one step that may spuriously fail.

One thing that may make readers curious is if we need to worry about this cached copy going stale.

Show 25 quoted lines
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  .gitlab-ci.yml     |  9 ++++++++-
>  ci/install-sdk.ps1 | 26 ++++++++++++++++++++------
>  2 files changed, 28 insertions(+), 7 deletions(-)
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 9b7698a0037..15419a954b3 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -127,9 +127,16 @@ build:mingw64:
>      - saas-windows-medium-amd64
>    variables:
>      NO_PERL: 1
> +  cache:
> +    key:
> +      files:
> +        - ci/install-sdk.ps1
> +        - ci/install-dependencies.ps1
> +    paths:
> +      - .dependencies
>    before_script:
>      - *windows_before_script
> -    - ./ci/install-sdk.ps1 -directory "git-sdk"
> +    - ./ci/install-sdk.ps1 -Directory "git-sdk"

Is this change essential to understand this patch, or is it an unrelated general clean-up to follow some style rules? I notice that ...

Show 14 quoted lines
>      - ./ci/install-dependencies.ps1 -Mingw
>    script:
>      # The minimal SDK's profile resets PATH.
> diff --git a/ci/install-sdk.ps1 b/ci/install-sdk.ps1
> index 66f24838a40..1dbcef04a18 100755
> --- a/ci/install-sdk.ps1
> +++ b/ci/install-sdk.ps1
> @@ -1,12 +1,26 @@
>  param(
> -    [string]$directory='git-sdk',
> -    [string]$url='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip'
> +    [string]$Directory='git-sdk',
> +    [string]$URL='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip',
> +    [string]$DownloadDirectory='.dependencies'

... we are making a matching change here to upcase the parameter names here.

Patrick Steinhardt· Nov 9, 2026, 06:58 UTC · re: Junio C Hamano · lore

Re: [PATCH 1/7] gitlab-ci: speed up installation of Git for Windows SDK

On Fri, Nov 06, 2026 at 09:20:57AM -0800, Junio C Hamano wrote:
Show 16 quoted lines
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > The "build:mingw64" job on GitLab CI downloads the minimal Git for
> > Windows SDK on every run and then extracts it via `Expand-Archive`,
> > which is expensive.
> >
> > Improve this by caching the SDK and by using tar(1) instead of
> > `Expand-Archive`.
> > Note that caching actually only gives us a win of a
> > couple seconds. The bigger win here is using tar(1), which speeds up
> > extracting the archive from 3 minutes to about 10 seconds. Regardless of
> 
> Wow, is a tool to move the bits from one part of a rusted iron
> surface to another part really so much more expensive compared to
> another tool to move the same bits, to make a difference that is
> worth doing this?
I was surprised by this, as well.
Show 5 quoted lines
> > that though, caching the SDK feels like a sensible thing to do as it
> > removes one step that may spuriously fail.
> 
> One thing that may make readers curious is if we need to worry about
> this cached copy going stale.

Yes, it may go stale, but I'm not even sure whether that really matters. I decided to just do this as-is and Cc Johannes to maybe facilitate his input on this part. But arguably, we can just skip the cache altogether as it didn't really buy much anyway.

Show 41 quoted lines
> > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > index 9b7698a0037..15419a954b3 100644
> > --- a/.gitlab-ci.yml
> > +++ b/.gitlab-ci.yml
> > @@ -127,9 +127,16 @@ build:mingw64:
> >      - saas-windows-medium-amd64
> >    variables:
> >      NO_PERL: 1
> > +  cache:
> > +    key:
> > +      files:
> > +        - ci/install-sdk.ps1
> > +        - ci/install-dependencies.ps1
> > +    paths:
> > +      - .dependencies
> >    before_script:
> >      - *windows_before_script
> > -    - ./ci/install-sdk.ps1 -directory "git-sdk"
> > +    - ./ci/install-sdk.ps1 -Directory "git-sdk"
> 
> Is this change essential to understand this patch, or is it an
> unrelated general clean-up to follow some style rules?  I notice
> that ...
> 
> >      - ./ci/install-dependencies.ps1 -Mingw
> >    script:
> >      # The minimal SDK's profile resets PATH.
> > diff --git a/ci/install-sdk.ps1 b/ci/install-sdk.ps1
> > index 66f24838a40..1dbcef04a18 100755
> > --- a/ci/install-sdk.ps1
> > +++ b/ci/install-sdk.ps1
> > @@ -1,12 +1,26 @@
> >  param(
> > -    [string]$directory='git-sdk',
> > -    [string]$url='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip'
> > +    [string]$Directory='git-sdk',
> > +    [string]$URL='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip',
> > +    [string]$DownloadDirectory='.dependencies'
> 
> ... we are making a matching change here to upcase the parameter
> names here.

Well, with the new DownloadDirectory parameter everything else would have been inconsistent, so I made those changes while at it. I'll dropt he cache in v2 though, and with that we can drop these changes, too.

Patrick
Patrick Steinhardt· Nov 6, 2026, 10:30 UTC · re: Patrick Steinhardt · lore

[PATCH 2/7] gitlab-ci: exclude intermediate outputs from Windows artifacts

As running tests is very slow on Windows we shard our CI tests across 10 runners. To avoid rebuilding Git on each of those runners we have a preliminary step which builds Git and then passes on the build artifacts to the test jobs.

This archive contains a ton of intermediate "*.obj" build artifacts that aren't needed by the test jobs: the artifact contains ~2300 files, out of which ~1800 are "*.obj" files.

Exclude those files from the build artifact, which reduces the time it takes to download and extract them from 34 seconds to 17 seconds.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml | 2 ++
 1 file changed, 2 insertions(+)
Show changes to .gitlab-ci.yml +2 −0
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 15419a954b3..036d2f3c7f2 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -194,6 +194,8 @@ build:msvc-meson:
   artifacts:
     paths:
       - build
+    exclude:
+      - build/**/*.obj
 
 test:msvc-meson:
   extends: .msvc-meson
-- 
2.56.0.500.g5d06dc02cd.dirty
Patrick Steinhardt· Nov 6, 2026, 10:30 UTC · re: Patrick Steinhardt · lore

[PATCH 3/7] ci: build optimized binaries with Meson

When taking a look at our CI jobs one will quickly notice that across the Linux-based jobs, the "fedora-breaking-changes-meson" job is running significantly longer than any of the other jobs. While most Linux jobs only run around 15 minutes, that particular job runs roughly twice as long.

The root cause of this is a combination of two different things:
  - The job uses Meson to compile Git, and Meson by default uses the
    "debug" build type that translates into "-O0". Our Makefile in
    contrast to that uses "-g -O2" by default.
  - The job enables breaking changes and thus runs with SHA256 by
    default with the "block" backend, which benefits heavily from
    compiler optimizations. For example, hashing a 500MB blob with
    SHA256 takes 9.6 seconds with "-O0", whereas it takes only 1.5
    seconds with "-O2". The SHA1 code path only slows down from 0.9
    seconds to 2.1 seconds with "-O0".

Of course, compiling without optimizations also has an impact on all the other jobs, but the Fedora job is by far the most impacted one.

Speed up executing the tests by using the "debugoptimized" build type instead. This reduces the runtime of "fedora-breaking-changes-meson" from 2100 to 1000 seconds. Other Linux- and macOS-based jobs benefit as well, but they only speed up by around ~10%.

On Windows though this is a bit more muddy, as the build step gets almost two minutes slower, whereas tests only run ~10% faster. Combined though the total run time is roughly the same before and after this change.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml            | 2 +-
 ci/run-build-and-tests.sh | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
Show changes to 2 files +2 −1

.gitlab-ci.yml, ci/run-build-and-tests.sh

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 036d2f3c7f2..df82cd3a5df 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -189,7 +189,7 @@ build:msvc-meson:
   extends: .msvc-meson
   stage: build
   script:
-    - meson setup build --vsenv -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
+    - meson setup build --vsenv --buildtype=debugoptimized -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
     - meson compile -C build
   artifacts:
     paths:
diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh
index 68b91e31439..fd8136af342 100755
--- a/ci/run-build-and-tests.sh
+++ b/ci/run-build-and-tests.sh
@@ -55,6 +55,7 @@ case "$jobname" in
 		--fatal-meson-warnings \
 		--warnlevel 2 --werror \
 		--wrap-mode nofallback \
+		--buildtype=debugoptimized \
 		-Dfuzzers=true \
 		-Dtest_output_directory="${TEST_OUTPUT_DIRECTORY:-$(pwd)/t}" \
 		$MESONFLAGS
-- 
2.56.0.500.g5d06dc02cd.dirty
Junio C Hamano· Nov 6, 2026, 17:24 UTC · re: Patrick Steinhardt · lore

Re: [PATCH 3/7] ci: build optimized binaries with Meson

Patrick Steinhardt <ps@pks.im> writes:
> Speed up executing the tests by using the "debugoptimized" build type
> instead. This reduces the runtime of "fedora-breaking-changes-meson"
> from 2100 to 1000 seconds. Other Linux- and macOS-based jobs benefit as
> well, but they only speed up by around ~10%.
That is an impressive improvement.
Show 36 quoted lines
> On Windows though this is a bit more muddy, as the build step gets
> almost two minutes slower, whereas tests only run ~10% faster. Combined
> though the total run time is roughly the same before and after this
> change.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  .gitlab-ci.yml            | 2 +-
>  ci/run-build-and-tests.sh | 1 +
>  2 files changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 036d2f3c7f2..df82cd3a5df 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -189,7 +189,7 @@ build:msvc-meson:
>    extends: .msvc-meson
>    stage: build
>    script:
> -    - meson setup build --vsenv -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
> +    - meson setup build --vsenv --buildtype=debugoptimized -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
>      - meson compile -C build
>    artifacts:
>      paths:
> diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh
> index 68b91e31439..fd8136af342 100755
> --- a/ci/run-build-and-tests.sh
> +++ b/ci/run-build-and-tests.sh
> @@ -55,6 +55,7 @@ case "$jobname" in
>  		--fatal-meson-warnings \
>  		--warnlevel 2 --werror \
>  		--wrap-mode nofallback \
> +		--buildtype=debugoptimized \
>  		-Dfuzzers=true \
>  		-Dtest_output_directory="${TEST_OUTPUT_DIRECTORY:-$(pwd)/t}" \
>  		$MESONFLAGS
Patrick Steinhardt· Nov 6, 2026, 10:30 UTC · re: Patrick Steinhardt · lore

[PATCH 4/7] t0027: optimize how we reset repository state

In t0027 we verify our logic to convert CRLF for all kinds of combinations of attributes and configuration, which ultimately results in 2600 test cases. This is of course on the heavier side, so on Windows this takes more than 10 minutes to run.

One source of slowness is `checkout_files ()`, which gets called for a lot of our test to set up the initial working directory. As part of that we both remove the old files that we care about and then check them out again via git-checkout(1), but we do this one by one for every single file.

This is of course rather wasteful, as we have to spawn ten different processes every time the function is invoked. And on top of that, we also have to rewrite the complete index in git-checkout(1) every time.

Speed this up by instead using a single invocation of rm(1) and git-checkout(1). This reduces the runtime by about 20% on all platforms, and on Windows this reduces the runtime by ~2.5 minutes.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t0027-auto-crlf.sh | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)
Show changes to t/t0027-auto-crlf.sh +9 −7
diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
index 49dbf09da77..eefb346d11f 100755
--- a/t/t0027-auto-crlf.sh
+++ b/t/t0027-auto-crlf.sh
@@ -303,13 +303,15 @@ checkout_files () {
 		git config core.autocrlf $crlf
 	'
 	pfx=eol_${ceol}_crlf_${crlf}_attr_${attr}_ &&
-	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
-	do
-		test_expect_success "setup $f checkout ${ceol:+ with -c core.eol=$ceol}"  '
-			rm -f crlf_false_attr__$f.txt &&
-			git ${ceol:+-c core.eol=$ceol} checkout -- crlf_false_attr__$f.txt
-		'
-	done
+	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
+		rm -f crlf_false_attr__LF.txt crlf_false_attr__CRLF.txt \
+			crlf_false_attr__LF_mix_CR.txt crlf_false_attr__CRLF_mix_LF.txt \
+			crlf_false_attr__LF_nul.txt &&
+		git ${ceol:+-c core.eol=$ceol} checkout -- \
+			crlf_false_attr__LF.txt crlf_false_attr__CRLF.txt \
+			crlf_false_attr__LF_mix_CR.txt crlf_false_attr__CRLF_mix_LF.txt \
+			crlf_false_attr__LF_nul.txt
+	'
 
 	test_expect_success "ls-files --eol attr=$attr $ident aeol=$aeol core.autocrlf=$crlf core.eol=$ceol" '
 		test_when_finished "rm expect actual" &&
-- 
2.56.0.500.g5d06dc02cd.dirty
Junio C Hamano· Nov 6, 2026, 17:33 UTC · re: Patrick Steinhardt · lore

Re: [PATCH 4/7] t0027: optimize how we reset repository state

Patrick Steinhardt <ps@pks.im> writes:
Show 17 quoted lines
>  	pfx=eol_${ceol}_crlf_${crlf}_attr_${attr}_ &&
> -	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
> -	do
> -		test_expect_success "setup $f checkout ${ceol:+ with -c core.eol=$ceol}"  '
> -			rm -f crlf_false_attr__$f.txt &&
> -			git ${ceol:+-c core.eol=$ceol} checkout -- crlf_false_attr__$f.txt
> -		'
> -	done
> +	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
> +		rm -f crlf_false_attr__LF.txt crlf_false_attr__CRLF.txt \
> +			crlf_false_attr__LF_mix_CR.txt crlf_false_attr__CRLF_mix_LF.txt \
> +			crlf_false_attr__LF_nul.txt &&
> +		git ${ceol:+-c core.eol=$ceol} checkout -- \
> +			crlf_false_attr__LF.txt crlf_false_attr__CRLF.txt \
> +			crlf_false_attr__LF_mix_CR.txt crlf_false_attr__CRLF_mix_LF.txt \
> +			crlf_false_attr__LF_nul.txt
> +	'

Hmph, don't we want avoid hand-expand the list of files even though we may want to use one "rm -f" and another "git checkout" command to work on them? IOW,

	paths=
	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
	do
		paths=${paths:+"$paths "}crlf_false_attr__$f.txt
	done
	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
		rm -f $paths &&
		git ${ceol:+-c core.eol=$ceol} checkout -- $paths
	'
or something?
>  
>  	test_expect_success "ls-files --eol attr=$attr $ident aeol=$aeol core.autocrlf=$crlf core.eol=$ceol" '
>  		test_when_finished "rm expect actual" &&
Torsten Bögershausen· Nov 8, 2026, 16:49 UTC · re: Junio C Hamano · lore

Re: [PATCH 4/7] t0027: optimize how we reset repository state

On Fri, Nov 06, 2026 at 09:33:01AM -0800, Junio C Hamano wrote:
Show 13 quoted lines
> Hmph, don't we want avoid hand-expand the list of files even though
> we may want to use one "rm -f" and another "git checkout" command
> to work on them?  IOW,
> 
> 	paths=
> 	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
> 	do
> 		paths=${paths:+"$paths "}crlf_false_attr__$f.txt
> 	done
> 	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
> 		rm -f $paths &&
> 		git ${ceol:+-c core.eol=$ceol} checkout -- $paths
> 	'
"Wow, so it can go" - a well working, but slow code is speed up so much.

I used the following (before reading the suggestion from Junio) and the execution time here on an older MacOs dropped from 71 seconds to 54 seconds.

    test_expect_success "setup file checkout ${ceol:+ with -c core.eol=$ceol}"  '
        files="\
           crlf_false_attr__LF.txt \
           crlf_false_attr__CRLF.txt \
           crlf_false_attr__LF_mix_CR.txt \
           crlf_false_attr__CRLF_mix_LF.txt \
           crlf_false_attr__LF_nul.txt" &&
        rm -fv $files &&
        git ${ceol:+-c core.eol=$ceol} checkout -- $files
    '
Patrick Steinhardt· Nov 9, 2026, 06:30 UTC · re: Torsten Bögershausen · lore

Re: [PATCH 4/7] t0027: optimize how we reset repository state

On Sun, Nov 08, 2026 at 05:49:01PM +0100, Torsten Bögershausen wrote:
Show 30 quoted lines
> On Fri, Nov 06, 2026 at 09:33:01AM -0800, Junio C Hamano wrote:
> > Hmph, don't we want avoid hand-expand the list of files even though
> > we may want to use one "rm -f" and another "git checkout" command
> > to work on them?  IOW,
> > 
> > 	paths=
> > 	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
> > 	do
> > 		paths=${paths:+"$paths "}crlf_false_attr__$f.txt
> > 	done
> > 	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
> > 		rm -f $paths &&
> > 		git ${ceol:+-c core.eol=$ceol} checkout -- $paths
> > 	'
> 
> "Wow, so it can go" - a well working, but slow code is speed up so much.
> 
> I used the following (before reading the suggestion from Junio)
> and the execution time here on an older MacOs dropped from 71 seconds to 54 seconds.
> 
>     test_expect_success "setup file checkout ${ceol:+ with -c core.eol=$ceol}"  '
>         files="\
>            crlf_false_attr__LF.txt \
>            crlf_false_attr__CRLF.txt \
>            crlf_false_attr__LF_mix_CR.txt \
>            crlf_false_attr__CRLF_mix_LF.txt \
>            crlf_false_attr__LF_nul.txt" &&
>         rm -fv $files &&
>         git ${ceol:+-c core.eol=$ceol} checkout -- $files
>     '

Hmm. I think I like Junio's alternative a bit better, but both are definitely an improvement over my take. Will add to the next version, thanks.

Patrick
Patrick Steinhardt· Nov 6, 2026, 10:30 UTC · re: Patrick Steinhardt · lore

[PATCH 5/7] t0027: cache expected output files

In the preceding commit we have optimized one aspect of t0027 already. Another inefficiency is `compare_files ()` and `compare_ws_file ()`, which are both called more than a 1000 times.

These functions will repeatedly generate the same expected outcome even though there's only a small handful of input files that the expected outcome is computed from. Recomputing those every time is of course quite pointless.

Improve this by computing the expected outcome once, only, to reduce the number of spawned processes. This reduces the runtime by around 20-25%, and on Windows the runtime decreases from 480 to 380 seconds.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t0027-auto-crlf.sh | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)
Show changes to t/t0027-auto-crlf.sh +9 −8
diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
index eefb346d11f..25ed6b33704 100755
--- a/t/t0027-auto-crlf.sh
+++ b/t/t0027-auto-crlf.sh
@@ -5,22 +5,23 @@ test_description='CRLF conversion all combinations'
 . ./test-lib.sh
 
 compare_files () {
-	tr '\015\000' QN <"$1" >"$1".expect &&
+	{ test -f "$1".expect || tr '\015\000' QN <"$1" >"$1".expect; } &&
 	tr '\015\000' QN <"$2" | tr -d 'Z' >"$2".actual &&
-	test_cmp "$1".expect "$2".actual &&
-	rm "$1".expect "$2".actual
+	test_cmp "$1".expect "$2".actual
 }
 
 compare_ws_file () {
 	pfx=$1
-	exp=$2.expect
+	exp=$2.ws-expect
 	act=$pfx.actual.$3
-	tr '\015\000abcdef0123456789' QN00000000000000000 <"$2" |
-		sed -e "s/0000*/$ZERO_OID/" >"$exp" &&
+	{
+		test -f "$exp" ||
+		tr '\015\000abcdef0123456789' QN00000000000000000 <"$2" |
+			sed -e "s/0000*/$ZERO_OID/" >"$exp" ;
+	} &&
 	tr '\015\000abcdef0123456789' QN00000000000000000 <"$3" |
 		sed -e "s/0000*/$ZERO_OID/" >"$act" &&
-	test_cmp "$exp" "$act" &&
-	rm "$exp" "$act"
+	test_cmp "$exp" "$act"
 }
 
 create_gitattributes () {
-- 
2.56.0.500.g5d06dc02cd.dirty
Patrick Steinhardt· Nov 6, 2026, 10:30 UTC · re: Patrick Steinhardt · lore

[PATCH 6/7] t1092: set up repositories from templates

In t1092 we verify the behaviour of sparse checkouts. We use three different repositories to do so: one with a full checkout, one with a sparse checkout and one with a sparse index, and then verify that the result is the same across all these configurations.

Those repositories get set up via `init_repos ()`, which initializes them from scratch every single time with a bunch of processes. The function is called more than a 100 times though, so the overhead adds up quite a bit. While the tests only require around 30 seconds on Linux, it takes more than 10 minutes on Windows.

Optimize this by instead creating the repositories once, only. So instead of recreating them from scratch, we can then just copy the template repositories into place and thus save a bunch of processes. This speeds up tests on Windows by about 3 minutes.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t1092-sparse-checkout-compatibility.sh | 34 +++++++++++++++++++-------------
 1 file changed, 20 insertions(+), 14 deletions(-)
Show changes to t/t1092-sparse-checkout-compatibility.sh +20 −14
diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh
index 05b54062b3b..5d81ec54c23 100755
--- a/t/t1092-sparse-checkout-compatibility.sh
+++ b/t/t1092-sparse-checkout-compatibility.sh
@@ -141,28 +141,34 @@ test_expect_success 'setup' '
 	)
 '
 
+test_expect_success 'setup template repositories' '
+	cp -r initial-repo full-checkout.template &&
+	cp -r initial-repo sparse-checkout.template &&
+	cp -r initial-repo sparse-index.template &&
+
+	# initialize sparse-checkout definitions
+	git -C sparse-checkout.template sparse-checkout init --cone &&
+	git -C sparse-checkout.template sparse-checkout set deep &&
+	git -C sparse-index.template sparse-checkout init --cone --sparse-index &&
+	test_cmp_config -C sparse-index.template true index.sparse &&
+	git -C sparse-index.template sparse-checkout set deep &&
+
+	# Disable this message to keep stderr the same.
+	git -C sparse-index.template config advice.sparseIndexExpanded false
+'
+
 init_repos () {
 	rm -rf full-checkout sparse-checkout sparse-index &&
 
 	# create repos in initial state
-	cp -r initial-repo full-checkout &&
+	cp -r full-checkout.template full-checkout &&
 	git -C full-checkout reset --hard &&
 
-	cp -r initial-repo sparse-checkout &&
+	cp -r sparse-checkout.template sparse-checkout &&
 	git -C sparse-checkout reset --hard &&
 
-	cp -r initial-repo sparse-index &&
-	git -C sparse-index reset --hard &&
-
-	# initialize sparse-checkout definitions
-	git -C sparse-checkout sparse-checkout init --cone &&
-	git -C sparse-checkout sparse-checkout set deep &&
-	git -C sparse-index sparse-checkout init --cone --sparse-index &&
-	test_cmp_config -C sparse-index true index.sparse &&
-	git -C sparse-index sparse-checkout set deep &&
-
-	# Disable this message to keep stderr the same.
-	git -C sparse-index config advice.sparseIndexExpanded false
+	cp -r sparse-index.template sparse-index &&
+	git -C sparse-index reset --hard
 }
 
 init_repos_as_submodules () {
-- 
2.56.0.500.g5d06dc02cd.dirty
Patrick Steinhardt· Nov 6, 2026, 10:30 UTC · re: Patrick Steinhardt · lore

[PATCH 7/7] t5551: use reftables for repositories with many refs

Some of the tests in t5551 creates almost a 100,000 references and then verify that those can be fetched. Naturally, creating this many refs is quite involved with the "files" backend, as it means that we have to create this many individual loose files. This is tolerable on Linux, where the test takes around 2 minutes in our CI. But on macOS, the same test takes almost 20 minutes to complete. And to add insult to injury, we execute these tests a second time via t5559.

The system under test here is the HTTP negotiation, so we don't actually care about whether or not the ref backend handles this alright. So adapt the test to instead create repositories with the reftable backend, which is significantly faster in this case as we only need to write a single table.

And sure enough, this reduces the runtime of this test on macOS by a factor of 10, going from 20 minutes to 2 minutes. On Linux, the tests at least speed up by a factor of two.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t5551-http-fetch-smart.sh | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
Show changes to t/t5551-http-fetch-smart.sh +4 −2
diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh
index 130053c57aa..34945059531 100755
--- a/t/t5551-http-fetch-smart.sh
+++ b/t/t5551-http-fetch-smart.sh
@@ -412,7 +412,8 @@ create_tags () {
 }
 
 test_expect_success 'create 2,000 tags in the repo' '
-	git init --bare "$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
+	git init --bare --ref-format=reftable \
+		"$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
 	(
 		cd "$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
 		create_tags 1 2000
@@ -421,7 +422,8 @@ test_expect_success 'create 2,000 tags in the repo' '
 
 test_expect_success CMDLINE_LIMIT \
 	'clone the 2,000 tag repo to check OS command line overflow' '
-	run_with_limited_cmdline git clone $HTTPD_URL/smart/many-tags.git too-many-refs &&
+	run_with_limited_cmdline git clone --ref-format=reftable \
+		$HTTPD_URL/smart/many-tags.git too-many-refs &&
 	(
 		cd too-many-refs &&
 		git for-each-ref refs/tags >actual &&
-- 
2.56.0.500.g5d06dc02cd.dirty
Junio C Hamano· Nov 6, 2026, 17:39 UTC · re: Patrick Steinhardt · lore

Re: [PATCH 0/7] ci: some speedups for slow tests

Patrick Steinhardt <ps@pks.im> writes:
> I was staring at CI way more often recently, and that got me annoyed
> real fast because it's just painstakingly slow. So I decided to have a
> look and try do something about it, the result of which is this patch
> series.

Thanks for these patches. I didn't leave comments on later parts of the series, but all of them were interesting read and fully made sense to me.

Patrick Steinhardt· Nov 9, 2026, 09:58 UTC · re: Patrick Steinhardt · lore

[PATCH v2 0/7] ci: some speedups for slow tests

Hi,

I was staring at CI way more often recently, and that got me annoyed real fast because it's just painstakingly slow. So I decided to have a look and try do something about it, the result of which is this patch series.

The series contains a couple of bits that are specific to GitLab, and a couple of fixes that are generic. Some highlights:

  - Installing the Git for Windows SDK speeds up from 3 minutes to 10
    seconds by using tar(1) instead of Extract-Archive.
  - t0027 speeds up from >10 minutes to 6 minutes on Windows by avoiding
    the overhead of spawning redundant processes.
  - t5551 and t5559 both speed up by a factor of 10 on macOS, going from
    20 minutes each to 2 minutes each by using reftables in a repo with
    100k refs. "osx-meson" and "osx-clang" thus go from 29 minutes to 18
    minutes end-to-end.
  - "fedora-breaking-changes-meson" speeds up by a factor of 2, going
    from 35 minutes to 16 minutes. Turns out our SHA256 implementation
    is quite slow with "-O0".

Unfortunately, our Windows runners at GitLab are still atrociously slow by themselves. So the critical path only improves by about ~10 minutes, going from 60-70 minutes to 53 minutes end-to-end. But hey, at least the other jobs now give feedback a bit faster. And I guess on GitHub it may be a different story because Windows runners are way better there and are typically not on the critical path.

A test run can be found at [1].
Changes in v2:
  - Drop the cache for the GfW SDK again.
  - Drop repetitive paths in t0027.
  - Link to v1: https://patch.msgid.link/20261106-pks-gitlab-ci-speedups-v1-0-1fa0374c7287@pks.im
Thanks!
Patrick
[1]: https://gitlab.com/gitlab-org/git/-/merge_requests/734
---
Patrick Steinhardt (7):
      gitlab-ci: speed up installation of Git for Windows SDK
      gitlab-ci: exclude intermediate outputs from Windows artifacts
      ci: build optimized binaries with Meson
      t0027: optimize how we reset repository state
      t0027: cache expected output files
      t1092: set up repositories from templates
      t5551: use reftables for repositories with many refs
 .gitlab-ci.yml                           |  4 +++-
 ci/install-sdk.ps1                       |  6 +++++-
 ci/run-build-and-tests.sh                |  1 +
 t/t0027-auto-crlf.sh                     | 26 +++++++++++++-----------
 t/t1092-sparse-checkout-compatibility.sh | 34 +++++++++++++++++++-------------
 t/t5551-http-fetch-smart.sh              |  6 ++++--
 6 files changed, 47 insertions(+), 30 deletions(-)
Range-diff versus v1:

1: e25adb1b8c8 < -: ----------- gitlab-ci: speed up installation of Git for Windows SDK -: ----------- > 1: 582a9ab6468 gitlab-ci: speed up installation of Git for Windows SDK 2: 116c8b8a716 = 2: bff0a624127 gitlab-ci: exclude intermediate outputs from Windows artifacts 3: a035b8a902c = 3: 713e69bfb59 ci: build optimized binaries with Meson 4: 1a1238f4be7 < -: ----------- t0027: optimize how we reset repository state -: ----------- > 4: e421eb2a275 t0027: optimize how we reset repository state 5: c652f5b380a = 5: 16905b5be62 t0027: cache expected output files 6: 9a90272c620 = 6: 43f355fbb02 t1092: set up repositories from templates 7: 187128e7d5d = 7: 960716da178 t5551: use reftables for repositories with many refs

--- base-commit: 0ffa3e35c7c82ab8d0de226149fb03b3f7674d05 change-id: 20261106-pks-gitlab-ci-speedups-09c3841eb178

Patrick Steinhardt· Nov 9, 2026, 09:58 UTC · re: Patrick Steinhardt · lore

[PATCH v2 1/7] gitlab-ci: speed up installation of Git for Windows SDK

The "build:mingw64" job on GitLab CI downloads the minimal Git for Windows SDK on every run and then extracts it via `Expand-Archive`, which is expensive.

Improve this by using tar(1) instead of `Expand-Archive`. While at it, make the script error on failures and suppress progress output.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 ci/install-sdk.ps1 | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
Show changes to ci/install-sdk.ps1 +5 −1
diff --git a/ci/install-sdk.ps1 b/ci/install-sdk.ps1
index 66f24838a40..bb332c1c10f 100755
--- a/ci/install-sdk.ps1
+++ b/ci/install-sdk.ps1
@@ -3,8 +3,12 @@ param(
     [string]$url='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip'
 )
 
+$ErrorActionPreference = 'Stop'
+$ProgressPreference = 'SilentlyContinue'
+
 Invoke-WebRequest "$url" -OutFile git-sdk.zip
-Expand-Archive -LiteralPath git-sdk.zip -DestinationPath "$directory"
+New-Item -Path $directory -ItemType Directory -Force
+tar -xf git-sdk.zip -C $directory
 Remove-Item -Path git-sdk.zip
 
 New-Item -Path .git/info -ItemType Directory -Force
-- 
2.56.0.611.g1030f0b940.dirty
Junio C Hamano· Nov 9, 2026, 16:53 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v2 1/7] gitlab-ci: speed up installation of Git for Windows SDK

Patrick Steinhardt <ps@pks.im> writes:
> +tar -xf git-sdk.zip -C $directory
Wow.  You learn new things every day.

This is in ci/install-sdk.ps1 file. Right now that script is referred to only from .gitlab-ci.yml so as long as GitLab CI environment has tar that works this way we would be fine, but I thought that ci/* was designed to be portable across CI and other environments. At least we would want a comment next to it that says we assume BSD derived tar that can grok zip archives here, probably?

Patrick Steinhardt· Nov 10, 2026, 06:53 UTC · re: Junio C Hamano · lore

Re: [PATCH v2 1/7] gitlab-ci: speed up installation of Git for Windows SDK

On Mon, Nov 09, 2026 at 08:53:53AM -0800, Junio C Hamano wrote:
Show 5 quoted lines
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > +tar -xf git-sdk.zip -C $directory
> 
> Wow.  You learn new things every day.

Yeah, it's fun. Windows uses bsdtar, which is in turn based on libarchive, and hence it also supports ZIP and 7zip files [1].

Show 6 quoted lines
> This is in ci/install-sdk.ps1 file.  Right now that script is
> referred to only from .gitlab-ci.yml so as long as GitLab CI
> environment has tar that works this way we would be fine, but I
> thought that ci/* was designed to be portable across CI and other
> environments.  At least we would want a comment next to it that says
> we assume BSD derived tar that can grok zip archives here, probably?

Well, bsdtar is nowadays distributed with Windows by default, so this should be portable to other CI environments, too. So I don't think it really warrants a comment as it should just work everwhere. That's certainly something to point out in the commit message though.

In fact I wonder whether we should maybe adapt GitHub Workflows to use that script, too. That was my intent back when introducing this script, and for a brief time we in fact did that. But we eventually figured that it was too slow in contrast to the git-for-windows/setup-git-for-windows-sdk step, so we reverted that change on GitHub again in 55d62306eea (GitHub ci(windows): speed up initializing Git for Windows' minimal SDK again, 2024-12-17).

But now that we have found a way to remove the slowness we could unify both systems again. I intentionally left that out of this patch series though, as we can still do that as a subsequent step if we want to. In any case, I want Johannes' opinion on this topic before doing such a change.

Patrick
[1]: https://learn.microsoft.com/en-us/windows/tar/
Junio C Hamano· Nov 10, 2026, 16:11 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v2 1/7] gitlab-ci: speed up installation of Git for Windows SDK

Patrick Steinhardt <ps@pks.im> writes:
Show 21 quoted lines
> On Mon, Nov 09, 2026 at 08:53:53AM -0800, Junio C Hamano wrote:
>> Patrick Steinhardt <ps@pks.im> writes:
>> 
>> > +tar -xf git-sdk.zip -C $directory
>> 
>> Wow.  You learn new things every day.
>
> Yeah, it's fun. Windows uses bsdtar, which is in turn based on
> libarchive, and hence it also supports ZIP and 7zip files [1].
>
>> This is in ci/install-sdk.ps1 file.  Right now that script is
>> referred to only from .gitlab-ci.yml so as long as GitLab CI
>> environment has tar that works this way we would be fine, but I
>> thought that ci/* was designed to be portable across CI and other
>> environments.  At least we would want a comment next to it that says
>> we assume BSD derived tar that can grok zip archives here, probably?
>
> Well, bsdtar is nowadays distributed with Windows by default, so this
> should be portable to other CI environments, too. So I don't think it
> really warrants a comment as it should just work everwhere. That's
> certainly something to point out in the commit message though.

Yeah, this installer is about Windows, so any current or future CI that uses this script would end up running BSD tar, which means we can assume readers know what the script is doing and why it works.

Show 5 quoted lines
> But now that we have found a way to remove the slowness we could unify
> both systems again. I intentionally left that out of this patch series
> though, as we can still do that as a subsequent step if we want to. In
> any case, I want Johannes' opinion on this topic before doing such a
> change.
Yes, indeed.  Thanks for these patches.
Patrick Steinhardt· Nov 9, 2026, 09:58 UTC · re: Patrick Steinhardt · lore

[PATCH v2 2/7] gitlab-ci: exclude intermediate outputs from Windows artifacts

As running tests is very slow on Windows we shard our CI tests across 10 runners. To avoid rebuilding Git on each of those runners we have a preliminary step which builds Git and then passes on the build artifacts to the test jobs.

This archive contains a ton of intermediate "*.obj" build artifacts that aren't needed by the test jobs: the artifact contains ~2300 files, out of which ~1800 are "*.obj" files.

Exclude those files from the build artifact, which reduces the time it takes to download and extract them from 34 seconds to 17 seconds.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml | 2 ++
 1 file changed, 2 insertions(+)
Show changes to .gitlab-ci.yml +2 −0
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 9b7698a0037..3631393a8cc 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -187,6 +187,8 @@ build:msvc-meson:
   artifacts:
     paths:
       - build
+    exclude:
+      - build/**/*.obj
 
 test:msvc-meson:
   extends: .msvc-meson
-- 
2.56.0.611.g1030f0b940.dirty
Patrick Steinhardt· Nov 9, 2026, 09:58 UTC · re: Patrick Steinhardt · lore

[PATCH v2 3/7] ci: build optimized binaries with Meson

When taking a look at our CI jobs one will quickly notice that across the Linux-based jobs, the "fedora-breaking-changes-meson" job is running significantly longer than any of the other jobs. While most Linux jobs only run around 15 minutes, that particular job runs roughly twice as long.

The root cause of this is a combination of two different things:
  - The job uses Meson to compile Git, and Meson by default uses the
    "debug" build type that translates into "-O0". Our Makefile in
    contrast to that uses "-g -O2" by default.
  - The job enables breaking changes and thus runs with SHA256 by
    default with the "block" backend, which benefits heavily from
    compiler optimizations. For example, hashing a 500MB blob with
    SHA256 takes 9.6 seconds with "-O0", whereas it takes only 1.5
    seconds with "-O2". The SHA1 code path only slows down from 0.9
    seconds to 2.1 seconds with "-O0".

Of course, compiling without optimizations also has an impact on all the other jobs, but the Fedora job is by far the most impacted one.

Speed up executing the tests by using the "debugoptimized" build type instead. This reduces the runtime of "fedora-breaking-changes-meson" from 2100 to 1000 seconds. Other Linux- and macOS-based jobs benefit as well, but they only speed up by around ~10%.

On Windows though this is a bit more muddy, as the build step gets almost two minutes slower, whereas tests only run ~10% faster. Combined though the total run time is roughly the same before and after this change.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml            | 2 +-
 ci/run-build-and-tests.sh | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
Show changes to 2 files +2 −1

.gitlab-ci.yml, ci/run-build-and-tests.sh

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 3631393a8cc..449fed1b392 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -182,7 +182,7 @@ build:msvc-meson:
   extends: .msvc-meson
   stage: build
   script:
-    - meson setup build --vsenv -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
+    - meson setup build --vsenv --buildtype=debugoptimized -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
     - meson compile -C build
   artifacts:
     paths:
diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh
index 68b91e31439..fd8136af342 100755
--- a/ci/run-build-and-tests.sh
+++ b/ci/run-build-and-tests.sh
@@ -55,6 +55,7 @@ case "$jobname" in
 		--fatal-meson-warnings \
 		--warnlevel 2 --werror \
 		--wrap-mode nofallback \
+		--buildtype=debugoptimized \
 		-Dfuzzers=true \
 		-Dtest_output_directory="${TEST_OUTPUT_DIRECTORY:-$(pwd)/t}" \
 		$MESONFLAGS
-- 
2.56.0.611.g1030f0b940.dirty
Patrick Steinhardt· Nov 9, 2026, 09:58 UTC · re: Patrick Steinhardt · lore

[PATCH v2 4/7] t0027: optimize how we reset repository state

In t0027 we verify our logic to convert CRLF for all kinds of combinations of attributes and configuration, which ultimately results in 2600 test cases. This is of course on the heavier side, so on Windows this takes more than 10 minutes to run.

One source of slowness is `checkout_files ()`, which gets called for a lot of our test to set up the initial working directory. As part of that we both remove the old files that we care about and then check them out again via git-checkout(1), but we do this one by one for every single file.

This is of course rather wasteful, as we have to spawn ten different processes every time the function is invoked. And on top of that, we also have to rewrite the complete index in git-checkout(1) every time.

Speed this up by instead using a single invocation of rm(1) and git-checkout(1). This reduces the runtime by about 20% on all platforms, and on Windows this reduces the runtime by ~2.5 minutes.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t0027-auto-crlf.sh | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
Show changes to t/t0027-auto-crlf.sh +5 −4
diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
index 49dbf09da77..504a7ea54bb 100755
--- a/t/t0027-auto-crlf.sh
+++ b/t/t0027-auto-crlf.sh
@@ -305,11 +305,12 @@ checkout_files () {
 	pfx=eol_${ceol}_crlf_${crlf}_attr_${attr}_ &&
 	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
 	do
-		test_expect_success "setup $f checkout ${ceol:+ with -c core.eol=$ceol}"  '
-			rm -f crlf_false_attr__$f.txt &&
-			git ${ceol:+-c core.eol=$ceol} checkout -- crlf_false_attr__$f.txt
-		'
+		paths="${paths:+"$paths "}crlf_false_attr__$f.txt"
 	done
+	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
+		rm -f $paths &&
+		git ${ceol:+-c core.eol=$ceol} checkout -- $paths
+	'
 
 	test_expect_success "ls-files --eol attr=$attr $ident aeol=$aeol core.autocrlf=$crlf core.eol=$ceol" '
 		test_when_finished "rm expect actual" &&
-- 
2.56.0.611.g1030f0b940.dirty
Junio C Hamano· Nov 9, 2026, 16:59 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v2 4/7] t0027: optimize how we reset repository state

Patrick Steinhardt <ps@pks.im> writes:
Show 5 quoted lines
> diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
> index 49dbf09da77..504a7ea54bb 100755
> --- a/t/t0027-auto-crlf.sh
> +++ b/t/t0027-auto-crlf.sh
> @@ -305,11 +305,12 @@ checkout_files () {
A micronit.
Show 9 quoted lines
>  	pfx=eol_${ceol}_crlf_${crlf}_attr_${attr}_ &&
>  	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
>  	do
> -		test_expect_success "setup $f checkout ${ceol:+ with -c core.eol=$ceol}"  '
> -			rm -f crlf_false_attr__$f.txt &&
> -			git ${ceol:+-c core.eol=$ceol} checkout -- crlf_false_attr__$f.txt
> -		'
> +		paths="${paths:+"$paths "}crlf_false_attr__$f.txt"
>  	done

We may know that nobody has touched variable "$paths" elsewhere in this program, but I would prefer to have an explicit "let's start with an empty $paths" assignment very close to this loop to help future readers. That way they do not have to wonder and look all over to find out after what string we are appending these filenames.

Show 7 quoted lines
> +	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
> +		rm -f $paths &&
> +		git ${ceol:+-c core.eol=$ceol} checkout -- $paths
> +	'
>  
>  	test_expect_success "ls-files --eol attr=$attr $ident aeol=$aeol core.autocrlf=$crlf core.eol=$ceol" '
>  		test_when_finished "rm expect actual" &&
Patrick Steinhardt· Nov 10, 2026, 06:53 UTC · re: Junio C Hamano · lore

Re: [PATCH v2 4/7] t0027: optimize how we reset repository state

On Mon, Nov 09, 2026 at 08:59:39AM -0800, Junio C Hamano wrote:
Show 25 quoted lines
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
> > index 49dbf09da77..504a7ea54bb 100755
> > --- a/t/t0027-auto-crlf.sh
> > +++ b/t/t0027-auto-crlf.sh
> > @@ -305,11 +305,12 @@ checkout_files () {
> 
> A micronit.
> 
> >  	pfx=eol_${ceol}_crlf_${crlf}_attr_${attr}_ &&
> >  	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
> >  	do
> > -		test_expect_success "setup $f checkout ${ceol:+ with -c core.eol=$ceol}"  '
> > -			rm -f crlf_false_attr__$f.txt &&
> > -			git ${ceol:+-c core.eol=$ceol} checkout -- crlf_false_attr__$f.txt
> > -		'
> > +		paths="${paths:+"$paths "}crlf_false_attr__$f.txt"
> >  	done
> 
> We may know that nobody has touched variable "$paths" elsewhere in
> this program, but I would prefer to have an explicit "let's start
> with an empty $paths" assignment very close to this loop to help
> future readers.  That way they do not have to wonder and look all
> over to find out after what string we are appending these filenames.
Yeah, makes sense.
Patrick
Patrick Steinhardt· Nov 9, 2026, 09:58 UTC · re: Patrick Steinhardt · lore

[PATCH v2 5/7] t0027: cache expected output files

In the preceding commit we have optimized one aspect of t0027 already. Another inefficiency is `compare_files ()` and `compare_ws_file ()`, which are both called more than a 1000 times.

These functions will repeatedly generate the same expected outcome even though there's only a small handful of input files that the expected outcome is computed from. Recomputing those every time is of course quite pointless.

Improve this by computing the expected outcome once, only, to reduce the number of spawned processes. This reduces the runtime by around 20-25%, and on Windows the runtime decreases from 480 to 380 seconds.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t0027-auto-crlf.sh | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)
Show changes to t/t0027-auto-crlf.sh +9 −8
diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
index 504a7ea54bb..b021f0855a8 100755
--- a/t/t0027-auto-crlf.sh
+++ b/t/t0027-auto-crlf.sh
@@ -5,22 +5,23 @@ test_description='CRLF conversion all combinations'
 . ./test-lib.sh
 
 compare_files () {
-	tr '\015\000' QN <"$1" >"$1".expect &&
+	{ test -f "$1".expect || tr '\015\000' QN <"$1" >"$1".expect; } &&
 	tr '\015\000' QN <"$2" | tr -d 'Z' >"$2".actual &&
-	test_cmp "$1".expect "$2".actual &&
-	rm "$1".expect "$2".actual
+	test_cmp "$1".expect "$2".actual
 }
 
 compare_ws_file () {
 	pfx=$1
-	exp=$2.expect
+	exp=$2.ws-expect
 	act=$pfx.actual.$3
-	tr '\015\000abcdef0123456789' QN00000000000000000 <"$2" |
-		sed -e "s/0000*/$ZERO_OID/" >"$exp" &&
+	{
+		test -f "$exp" ||
+		tr '\015\000abcdef0123456789' QN00000000000000000 <"$2" |
+			sed -e "s/0000*/$ZERO_OID/" >"$exp" ;
+	} &&
 	tr '\015\000abcdef0123456789' QN00000000000000000 <"$3" |
 		sed -e "s/0000*/$ZERO_OID/" >"$act" &&
-	test_cmp "$exp" "$act" &&
-	rm "$exp" "$act"
+	test_cmp "$exp" "$act"
 }
 
 create_gitattributes () {
-- 
2.56.0.611.g1030f0b940.dirty
Torsten Bögershausen· Nov 9, 2026, 14:46 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v2 5/7] t0027: cache expected output files

On Mon, Nov 09, 2026 at 10:58:26AM +0100, Patrick Steinhardt wrote:
Show 34 quoted lines
> In the preceding commit we have optimized one aspect of t0027 already.
> Another inefficiency is `compare_files ()` and `compare_ws_file ()`,
> which are both called more than a 1000 times.
> 
> These functions will repeatedly generate the same expected outcome even
> though there's only a small handful of input files that the expected
> outcome is computed from. Recomputing those every time is of course
> quite pointless.
> 
> Improve this by computing the expected outcome once, only, to reduce the
> number of spawned processes. This reduces the runtime by around 20-25%,
> and on Windows the runtime decreases from 480 to 380 seconds.
> 
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  t/t0027-auto-crlf.sh | 17 +++++++++--------
>  1 file changed, 9 insertions(+), 8 deletions(-)
> 
> diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
> index 504a7ea54bb..b021f0855a8 100755
> --- a/t/t0027-auto-crlf.sh
> +++ b/t/t0027-auto-crlf.sh
> @@ -5,22 +5,23 @@ test_description='CRLF conversion all combinations'
>  . ./test-lib.sh
>  
>  compare_files () {
> -	tr '\015\000' QN <"$1" >"$1".expect &&
> +	{ test -f "$1".expect || tr '\015\000' QN <"$1" >"$1".expect; } &&
>  	tr '\015\000' QN <"$2" | tr -d 'Z' >"$2".actual &&
> -	test_cmp "$1".expect "$2".actual &&
> -	rm "$1".expect "$2".actual
> +	test_cmp "$1".expect "$2".actual
>  }
>  

Nice. If you want to save even more CPU/SSD-cycles, you can prepare the expect files like this: (No, I didn't do any benchmarks, nor look at ws_compare)

Show changes to t/t0027-auto-crlf.sh +21 −3
diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
index 49dbf09da7..3563336162 100755
--- a/t/t0027-auto-crlf.sh
+++ b/t/t0027-auto-crlf.sh
@@ -5,10 +5,9 @@ test_description='CRLF conversion all combinations'
 . ./test-lib.sh
 
 compare_files () {
-	tr '\015\000' QN <"$1" >"$1".expect &&
 	tr '\015\000' QN <"$2" | tr -d 'Z' >"$2".actual &&
-	test_cmp "$1".expect "$2".actual &&
-	rm "$1".expect "$2".actual
+	test_cmp "expd/$1".expect "$2".actual &&
+	rm "$2".actual
 }
 
 compare_ws_file () {
@@ -343,6 +342,25 @@ checkout_files () {
 	"
 }
 
+test_expect_success 'create expect files' '
+	printf "\$Id: 0000000000000000000000000000000000000000 \$\nLINEONE\nLINETWO\nLINETHREE"     >LF &&
+	printf "\$Id: 0000000000000000000000000000000000000000 \$\r\nLINEONE\r\nLINETWO\r\nLINETHREE" >CRLF &&
+	printf "\$Id: 0000000000000000000000000000000000000000 \$\nLINEONE\r\nLINETWO\nLINETHREE"   >CRLF_mix_LF &&
+	printf "\$Id: 0000000000000000000000000000000000000000 \$\nLINEONE\nLINETWO\rLINETHREE"     >LF_mix_CR &&
+	printf "\$Id: 0000000000000000000000000000000000000000 \$\r\nLINEONE\r\nLINETWO\rLINETHREE"   >CRLF_mix_CR &&
+	printf "\$Id: 0000000000000000000000000000000000000000 \$\r\nLINEONEQ\r\nLINETWO\r\nLINETHREE" | q_to_nul >CRLF_nul &&
+	printf "\$Id: 0000000000000000000000000000000000000000 \$\nLINEONEQ\nLINETWO\nLINETHREE" | q_to_nul >LF_nul &&
+  mkdir expd &&
+  echo "* -text" >expd/.gitattributes &&
+	tr "\015\000" QN <LF >expd/LF.expect &&
+	tr "\015\000" QN <CRLF >expd/CRLF.expect &&
+	tr "\015\000" QN <CRLF_mix_LF >expd/CRLF_mix_LF.expect &&
+	tr "\015\000" QN <LF_mix_CR >expd/LF_mix_CR.expect &&
+	tr "\015\000" QN <CRLF_mix_CR >expd/CRLF_mix_CR.expect &&
+	tr "\015\000" QN <CRLF_nul >expd/CRLF_nul.expect &&
+	tr "\015\000" QN <LF_nul >expd/LF_nul.expect
+  '
+
 # Test control characters
 # NUL SOH CR EOF==^Z
 test_expect_success 'ls-files --eol -o Text/Binary' '



[snip]
Patrick Steinhardt· Nov 10, 2026, 06:53 UTC · re: Torsten Bögershausen · lore

Re: [PATCH v2 5/7] t0027: cache expected output files

On Mon, Nov 09, 2026 at 03:46:51PM +0100, Torsten Bögershausen wrote:
Show 40 quoted lines
> On Mon, Nov 09, 2026 at 10:58:26AM +0100, Patrick Steinhardt wrote:
> > In the preceding commit we have optimized one aspect of t0027 already.
> > Another inefficiency is `compare_files ()` and `compare_ws_file ()`,
> > which are both called more than a 1000 times.
> > 
> > These functions will repeatedly generate the same expected outcome even
> > though there's only a small handful of input files that the expected
> > outcome is computed from. Recomputing those every time is of course
> > quite pointless.
> > 
> > Improve this by computing the expected outcome once, only, to reduce the
> > number of spawned processes. This reduces the runtime by around 20-25%,
> > and on Windows the runtime decreases from 480 to 380 seconds.
> > 
> > Signed-off-by: Patrick Steinhardt <ps@pks.im>
> > ---
> >  t/t0027-auto-crlf.sh | 17 +++++++++--------
> >  1 file changed, 9 insertions(+), 8 deletions(-)
> > 
> > diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
> > index 504a7ea54bb..b021f0855a8 100755
> > --- a/t/t0027-auto-crlf.sh
> > +++ b/t/t0027-auto-crlf.sh
> > @@ -5,22 +5,23 @@ test_description='CRLF conversion all combinations'
> >  . ./test-lib.sh
> >  
> >  compare_files () {
> > -	tr '\015\000' QN <"$1" >"$1".expect &&
> > +	{ test -f "$1".expect || tr '\015\000' QN <"$1" >"$1".expect; } &&
> >  	tr '\015\000' QN <"$2" | tr -d 'Z' >"$2".actual &&
> > -	test_cmp "$1".expect "$2".actual &&
> > -	rm "$1".expect "$2".actual
> > +	test_cmp "$1".expect "$2".actual
> >  }
> >  
> 
> Nice.
> If you want to save even more CPU/SSD-cycles,
> you can prepare the expect files like this:
> (No, I didn't do any benchmarks, nor look at ws_compare)

Thanks! I benchmarked this and it doesn't seem to make much of a difference. So I'd propose to stick with the simple change for now, if that's alright with you.

Patrick
Torsten Bögershausen· Nov 10, 2026, 10:29 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v2 5/7] t0027: cache expected output files

> 
> Thanks! I benchmarked this and it doesn't seem to make much of a
> difference. So I'd propose to stick with the simple change for now, if
> that's alright with you.

Good to hear that the benchmark was done. OK to stick your patch "as is"

Patrick Steinhardt· Nov 9, 2026, 09:58 UTC · re: Patrick Steinhardt · lore

[PATCH v2 6/7] t1092: set up repositories from templates

In t1092 we verify the behaviour of sparse checkouts. We use three different repositories to do so: one with a full checkout, one with a sparse checkout and one with a sparse index, and then verify that the result is the same across all these configurations.

Those repositories get set up via `init_repos ()`, which initializes them from scratch every single time with a bunch of processes. The function is called more than a 100 times though, so the overhead adds up quite a bit. While the tests only require around 30 seconds on Linux, it takes more than 10 minutes on Windows.

Optimize this by instead creating the repositories once, only. So instead of recreating them from scratch, we can then just copy the template repositories into place and thus save a bunch of processes. This speeds up tests on Windows by about 3 minutes.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t1092-sparse-checkout-compatibility.sh | 34 +++++++++++++++++++-------------
 1 file changed, 20 insertions(+), 14 deletions(-)
Show changes to t/t1092-sparse-checkout-compatibility.sh +20 −14
diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh
index 05b54062b3b..5d81ec54c23 100755
--- a/t/t1092-sparse-checkout-compatibility.sh
+++ b/t/t1092-sparse-checkout-compatibility.sh
@@ -141,28 +141,34 @@ test_expect_success 'setup' '
 	)
 '
 
+test_expect_success 'setup template repositories' '
+	cp -r initial-repo full-checkout.template &&
+	cp -r initial-repo sparse-checkout.template &&
+	cp -r initial-repo sparse-index.template &&
+
+	# initialize sparse-checkout definitions
+	git -C sparse-checkout.template sparse-checkout init --cone &&
+	git -C sparse-checkout.template sparse-checkout set deep &&
+	git -C sparse-index.template sparse-checkout init --cone --sparse-index &&
+	test_cmp_config -C sparse-index.template true index.sparse &&
+	git -C sparse-index.template sparse-checkout set deep &&
+
+	# Disable this message to keep stderr the same.
+	git -C sparse-index.template config advice.sparseIndexExpanded false
+'
+
 init_repos () {
 	rm -rf full-checkout sparse-checkout sparse-index &&
 
 	# create repos in initial state
-	cp -r initial-repo full-checkout &&
+	cp -r full-checkout.template full-checkout &&
 	git -C full-checkout reset --hard &&
 
-	cp -r initial-repo sparse-checkout &&
+	cp -r sparse-checkout.template sparse-checkout &&
 	git -C sparse-checkout reset --hard &&
 
-	cp -r initial-repo sparse-index &&
-	git -C sparse-index reset --hard &&
-
-	# initialize sparse-checkout definitions
-	git -C sparse-checkout sparse-checkout init --cone &&
-	git -C sparse-checkout sparse-checkout set deep &&
-	git -C sparse-index sparse-checkout init --cone --sparse-index &&
-	test_cmp_config -C sparse-index true index.sparse &&
-	git -C sparse-index sparse-checkout set deep &&
-
-	# Disable this message to keep stderr the same.
-	git -C sparse-index config advice.sparseIndexExpanded false
+	cp -r sparse-index.template sparse-index &&
+	git -C sparse-index reset --hard
 }
 
 init_repos_as_submodules () {
-- 
2.56.0.611.g1030f0b940.dirty
Patrick Steinhardt· Nov 9, 2026, 09:58 UTC · re: Patrick Steinhardt · lore

[PATCH v2 7/7] t5551: use reftables for repositories with many refs

Some of the tests in t5551 creates almost a 100,000 references and then verify that those can be fetched. Naturally, creating this many refs is quite involved with the "files" backend, as it means that we have to create this many individual loose files. This is tolerable on Linux, where the test takes around 2 minutes in our CI. But on macOS, the same test takes almost 20 minutes to complete. And to add insult to injury, we execute these tests a second time via t5559.

The system under test here is the HTTP negotiation, so we don't actually care about whether or not the ref backend handles this alright. So adapt the test to instead create repositories with the reftable backend, which is significantly faster in this case as we only need to write a single table.

And sure enough, this reduces the runtime of this test on macOS by a factor of 10, going from 20 minutes to 2 minutes. On Linux, the tests at least speed up by a factor of two.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t5551-http-fetch-smart.sh | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
Show changes to t/t5551-http-fetch-smart.sh +4 −2
diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh
index 130053c57aa..34945059531 100755
--- a/t/t5551-http-fetch-smart.sh
+++ b/t/t5551-http-fetch-smart.sh
@@ -412,7 +412,8 @@ create_tags () {
 }
 
 test_expect_success 'create 2,000 tags in the repo' '
-	git init --bare "$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
+	git init --bare --ref-format=reftable \
+		"$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
 	(
 		cd "$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
 		create_tags 1 2000
@@ -421,7 +422,8 @@ test_expect_success 'create 2,000 tags in the repo' '
 
 test_expect_success CMDLINE_LIMIT \
 	'clone the 2,000 tag repo to check OS command line overflow' '
-	run_with_limited_cmdline git clone $HTTPD_URL/smart/many-tags.git too-many-refs &&
+	run_with_limited_cmdline git clone --ref-format=reftable \
+		$HTTPD_URL/smart/many-tags.git too-many-refs &&
 	(
 		cd too-many-refs &&
 		git for-each-ref refs/tags >actual &&
-- 
2.56.0.611.g1030f0b940.dirty
Johannes Sixt· Nov 12, 2026, 08:19 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v2 0/7] ci: some speedups for slow tests

Am 09.11.26 um 10:58 schrieb Patrick Steinhardt:
Show 24 quoted lines
> The series contains a couple of bits that are specific to GitLab, and a
> couple of fixes that are generic. Some highlights:
> 
>   - Installing the Git for Windows SDK speeds up from 3 minutes to 10
>     seconds by using tar(1) instead of Extract-Archive.
> 
>   - t0027 speeds up from >10 minutes to 6 minutes on Windows by avoiding
>     the overhead of spawning redundant processes.
> 
>   - t5551 and t5559 both speed up by a factor of 10 on macOS, going from
>     20 minutes each to 2 minutes each by using reftables in a repo with
>     100k refs. "osx-meson" and "osx-clang" thus go from 29 minutes to 18
>     minutes end-to-end.
> 
>   - "fedora-breaking-changes-meson" speeds up by a factor of 2, going
>     from 35 minutes to 16 minutes. Turns out our SHA256 implementation
>     is quite slow with "-O0".
> 
> Unfortunately, our Windows runners at GitLab are still atrociously slow
> by themselves. So the critical path only improves by about ~10 minutes,
> going from 60-70 minutes to 53 minutes end-to-end. But hey, at least the
> other jobs now give feedback a bit faster. And I guess on GitHub it may
> be a different story because Windows runners are way better there and
> are typically not on the critical path.
I've tested this series on my Windows notebook and it is looking good.

The runtime of t0027 went down from 4:45 to 3:14, and that of t1092 from 6:15 to 5:33. t5551 is skipped in my setup.

Thanks!
-- Hannes
Patrick Steinhardt· Nov 11, 2026, 10:47 UTC · re: Patrick Steinhardt · lore

[PATCH v3 0/7] ci: some speedups for slow tests

Hi,

I was staring at CI way more often recently, and that got me annoyed real fast because it's just painstakingly slow. So I decided to have a look and try do something about it, the result of which is this patch series.

The series contains a couple of bits that are specific to GitLab, and a couple of fixes that are generic. Some highlights:

  - Installing the Git for Windows SDK speeds up from 3 minutes to 10
    seconds by using tar(1) instead of Extract-Archive.
  - t0027 speeds up from >10 minutes to 6 minutes on Windows by avoiding
    the overhead of spawning redundant processes.
  - t5551 and t5559 both speed up by a factor of 10 on macOS, going from
    20 minutes each to 2 minutes each by using reftables in a repo with
    100k refs. "osx-meson" and "osx-clang" thus go from 29 minutes to 18
    minutes end-to-end.
  - "fedora-breaking-changes-meson" speeds up by a factor of 2, going
    from 35 minutes to 16 minutes. Turns out our SHA256 implementation
    is quite slow with "-O0".

Unfortunately, our Windows runners at GitLab are still atrociously slow by themselves. So the critical path only improves by about ~10 minutes, going from 60-70 minutes to 53 minutes end-to-end. But hey, at least the other jobs now give feedback a bit faster. And I guess on GitHub it may be a different story because Windows runners are way better there and are typically not on the critical path.

A test run can be found at [1].
Changes in v3:
  - Document why Windows' tar version can extract ZIP archives.
  - Clear `paths` variable.
  - Link to v2: https://patch.msgid.link/20261109-pks-gitlab-ci-speedups-v2-0-508832602d0b@pks.im
Changes in v2:
  - Drop the cache for the GfW SDK again.
  - Drop repetitive paths in t0027.
  - Link to v1: https://patch.msgid.link/20261106-pks-gitlab-ci-speedups-v1-0-1fa0374c7287@pks.im
Thanks!
Patrick
[1]: https://gitlab.com/gitlab-org/git/-/merge_requests/734
---
Patrick Steinhardt (7):
      gitlab-ci: speed up installation of Git for Windows SDK
      gitlab-ci: exclude intermediate outputs from Windows artifacts
      ci: build optimized binaries with Meson
      t0027: optimize how we reset repository state
      t0027: cache expected output files
      t1092: set up repositories from templates
      t5551: use reftables for repositories with many refs
 .gitlab-ci.yml                           |  4 +++-
 ci/install-sdk.ps1                       |  6 +++++-
 ci/run-build-and-tests.sh                |  1 +
 t/t0027-auto-crlf.sh                     | 27 ++++++++++++++-----------
 t/t1092-sparse-checkout-compatibility.sh | 34 +++++++++++++++++++-------------
 t/t5551-http-fetch-smart.sh              |  6 ++++--
 6 files changed, 48 insertions(+), 30 deletions(-)
Range-diff versus v2:
1:  41c3a0e8d7d ! 1:  8fabb336df3 gitlab-ci: speed up installation of Git for Windows SDK
    @@ Commit message
         Windows SDK on every run and then extracts it via `Expand-Archive`,
         which is expensive.
     
    -    Improve this by using tar(1) instead of `Expand-Archive`. While at it,
    -    make the script error on failures and suppress progress output.
    +    Improve this by using tar(1) instead of `Expand-Archive`. Note that
    +    we're using it to extract a ZIP archive, which may seem puzzling at
    +    first. But Windows ships bsdtar by default [1], which also supports ZIP
    +    and 7zip files.
    +
    +    While at it, make the script error on failures and suppress progress
    +    output.
    +
    +    [1]: https://learn.microsoft.com/en-us/windows/tar/
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
2:  439c12cb15d = 2:  4a4d1e6f2a8 gitlab-ci: exclude intermediate outputs from Windows artifacts
3:  070bef6fecf = 3:  c23b7a7c58d ci: build optimized binaries with Meson
4:  598a143fc24 ! 4:  942d70be750 t0027: optimize how we reset repository state
    @@ Commit message
     
      ## t/t0027-auto-crlf.sh ##
     @@ t/t0027-auto-crlf.sh: checkout_files () {
    + 		git config core.autocrlf $crlf
    + 	'
      	pfx=eol_${ceol}_crlf_${crlf}_attr_${attr}_ &&
    ++	paths=
      	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
      	do
     -		test_expect_success "setup $f checkout ${ceol:+ with -c core.eol=$ceol}"  '
5:  a325efbaddb = 5:  460cd0c88fd t0027: cache expected output files
6:  a2d33441e50 = 6:  44c77717749 t1092: set up repositories from templates
7:  bfc93fdfb4e = 7:  d64a61efec2 t5551: use reftables for repositories with many refs

--- base-commit: 0ffa3e35c7c82ab8d0de226149fb03b3f7674d05 change-id: 20261106-pks-gitlab-ci-speedups-09c3841eb178

Patrick Steinhardt· Nov 11, 2026, 10:47 UTC · re: Patrick Steinhardt · lore

[PATCH v3 1/7] gitlab-ci: speed up installation of Git for Windows SDK

The "build:mingw64" job on GitLab CI downloads the minimal Git for Windows SDK on every run and then extracts it via `Expand-Archive`, which is expensive.

Improve this by using tar(1) instead of `Expand-Archive`. Note that we're using it to extract a ZIP archive, which may seem puzzling at first. But Windows ships bsdtar by default [1], which also supports ZIP and 7zip files.

While at it, make the script error on failures and suppress progress output.

[1]: https://learn.microsoft.com/en-us/windows/tar/
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 ci/install-sdk.ps1 | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
Show changes to ci/install-sdk.ps1 +5 −1
diff --git a/ci/install-sdk.ps1 b/ci/install-sdk.ps1
index 66f24838a40..bb332c1c10f 100755
--- a/ci/install-sdk.ps1
+++ b/ci/install-sdk.ps1
@@ -3,8 +3,12 @@ param(
     [string]$url='https://github.com/git-for-windows/git-sdk-64/releases/download/ci-artifacts/git-sdk-x86_64-minimal.zip'
 )
 
+$ErrorActionPreference = 'Stop'
+$ProgressPreference = 'SilentlyContinue'
+
 Invoke-WebRequest "$url" -OutFile git-sdk.zip
-Expand-Archive -LiteralPath git-sdk.zip -DestinationPath "$directory"
+New-Item -Path $directory -ItemType Directory -Force
+tar -xf git-sdk.zip -C $directory
 Remove-Item -Path git-sdk.zip
 
 New-Item -Path .git/info -ItemType Directory -Force
-- 
2.56.0.611.g1030f0b940.dirty
Patrick Steinhardt· Nov 11, 2026, 10:47 UTC · re: Patrick Steinhardt · lore

[PATCH v3 2/7] gitlab-ci: exclude intermediate outputs from Windows artifacts

As running tests is very slow on Windows we shard our CI tests across 10 runners. To avoid rebuilding Git on each of those runners we have a preliminary step which builds Git and then passes on the build artifacts to the test jobs.

This archive contains a ton of intermediate "*.obj" build artifacts that aren't needed by the test jobs: the artifact contains ~2300 files, out of which ~1800 are "*.obj" files.

Exclude those files from the build artifact, which reduces the time it takes to download and extract them from 34 seconds to 17 seconds.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml | 2 ++
 1 file changed, 2 insertions(+)
Show changes to .gitlab-ci.yml +2 −0
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 9b7698a0037..3631393a8cc 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -187,6 +187,8 @@ build:msvc-meson:
   artifacts:
     paths:
       - build
+    exclude:
+      - build/**/*.obj
 
 test:msvc-meson:
   extends: .msvc-meson
-- 
2.56.0.611.g1030f0b940.dirty
Patrick Steinhardt· Nov 11, 2026, 10:47 UTC · re: Patrick Steinhardt · lore

[PATCH v3 3/7] ci: build optimized binaries with Meson

When taking a look at our CI jobs one will quickly notice that across the Linux-based jobs, the "fedora-breaking-changes-meson" job is running significantly longer than any of the other jobs. While most Linux jobs only run around 15 minutes, that particular job runs roughly twice as long.

The root cause of this is a combination of two different things:
  - The job uses Meson to compile Git, and Meson by default uses the
    "debug" build type that translates into "-O0". Our Makefile in
    contrast to that uses "-g -O2" by default.
  - The job enables breaking changes and thus runs with SHA256 by
    default with the "block" backend, which benefits heavily from
    compiler optimizations. For example, hashing a 500MB blob with
    SHA256 takes 9.6 seconds with "-O0", whereas it takes only 1.5
    seconds with "-O2". The SHA1 code path only slows down from 0.9
    seconds to 2.1 seconds with "-O0".

Of course, compiling without optimizations also has an impact on all the other jobs, but the Fedora job is by far the most impacted one.

Speed up executing the tests by using the "debugoptimized" build type instead. This reduces the runtime of "fedora-breaking-changes-meson" from 2100 to 1000 seconds. Other Linux- and macOS-based jobs benefit as well, but they only speed up by around ~10%.

On Windows though this is a bit more muddy, as the build step gets almost two minutes slower, whereas tests only run ~10% faster. Combined though the total run time is roughly the same before and after this change.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml            | 2 +-
 ci/run-build-and-tests.sh | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
Show changes to 2 files +2 −1

.gitlab-ci.yml, ci/run-build-and-tests.sh

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 3631393a8cc..449fed1b392 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -182,7 +182,7 @@ build:msvc-meson:
   extends: .msvc-meson
   stage: build
   script:
-    - meson setup build --vsenv -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
+    - meson setup build --vsenv --buildtype=debugoptimized -Dperl=disabled -Dbackend_max_links=1 -Dcredential_helpers=wincred -Dtest_output_directory="$TEST_OUTPUT_DIRECTORY"
     - meson compile -C build
   artifacts:
     paths:
diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh
index 68b91e31439..fd8136af342 100755
--- a/ci/run-build-and-tests.sh
+++ b/ci/run-build-and-tests.sh
@@ -55,6 +55,7 @@ case "$jobname" in
 		--fatal-meson-warnings \
 		--warnlevel 2 --werror \
 		--wrap-mode nofallback \
+		--buildtype=debugoptimized \
 		-Dfuzzers=true \
 		-Dtest_output_directory="${TEST_OUTPUT_DIRECTORY:-$(pwd)/t}" \
 		$MESONFLAGS
-- 
2.56.0.611.g1030f0b940.dirty
Patrick Steinhardt· Nov 11, 2026, 10:47 UTC · re: Patrick Steinhardt · lore

[PATCH v3 4/7] t0027: optimize how we reset repository state

In t0027 we verify our logic to convert CRLF for all kinds of combinations of attributes and configuration, which ultimately results in 2600 test cases. This is of course on the heavier side, so on Windows this takes more than 10 minutes to run.

One source of slowness is `checkout_files ()`, which gets called for a lot of our test to set up the initial working directory. As part of that we both remove the old files that we care about and then check them out again via git-checkout(1), but we do this one by one for every single file.

This is of course rather wasteful, as we have to spawn ten different processes every time the function is invoked. And on top of that, we also have to rewrite the complete index in git-checkout(1) every time.

Speed this up by instead using a single invocation of rm(1) and git-checkout(1). This reduces the runtime by about 20% on all platforms, and on Windows this reduces the runtime by ~2.5 minutes.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t0027-auto-crlf.sh | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
Show changes to t/t0027-auto-crlf.sh +6 −4
diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
index 49dbf09da77..a3904b729b3 100755
--- a/t/t0027-auto-crlf.sh
+++ b/t/t0027-auto-crlf.sh
@@ -303,13 +303,15 @@ checkout_files () {
 		git config core.autocrlf $crlf
 	'
 	pfx=eol_${ceol}_crlf_${crlf}_attr_${attr}_ &&
+	paths=
 	for f in LF CRLF LF_mix_CR CRLF_mix_LF LF_nul
 	do
-		test_expect_success "setup $f checkout ${ceol:+ with -c core.eol=$ceol}"  '
-			rm -f crlf_false_attr__$f.txt &&
-			git ${ceol:+-c core.eol=$ceol} checkout -- crlf_false_attr__$f.txt
-		'
+		paths="${paths:+"$paths "}crlf_false_attr__$f.txt"
 	done
+	test_expect_success "setup checkout ${ceol:+ with -c core.eol=$ceol}" '
+		rm -f $paths &&
+		git ${ceol:+-c core.eol=$ceol} checkout -- $paths
+	'
 
 	test_expect_success "ls-files --eol attr=$attr $ident aeol=$aeol core.autocrlf=$crlf core.eol=$ceol" '
 		test_when_finished "rm expect actual" &&
-- 
2.56.0.611.g1030f0b940.dirty
Patrick Steinhardt· Nov 11, 2026, 10:47 UTC · re: Patrick Steinhardt · lore

[PATCH v3 5/7] t0027: cache expected output files

In the preceding commit we have optimized one aspect of t0027 already. Another inefficiency is `compare_files ()` and `compare_ws_file ()`, which are both called more than a 1000 times.

These functions will repeatedly generate the same expected outcome even though there's only a small handful of input files that the expected outcome is computed from. Recomputing those every time is of course quite pointless.

Improve this by computing the expected outcome once, only, to reduce the number of spawned processes. This reduces the runtime by around 20-25%, and on Windows the runtime decreases from 480 to 380 seconds.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t0027-auto-crlf.sh | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)
Show changes to t/t0027-auto-crlf.sh +9 −8
diff --git a/t/t0027-auto-crlf.sh b/t/t0027-auto-crlf.sh
index a3904b729b3..38ce06cbc1e 100755
--- a/t/t0027-auto-crlf.sh
+++ b/t/t0027-auto-crlf.sh
@@ -5,22 +5,23 @@ test_description='CRLF conversion all combinations'
 . ./test-lib.sh
 
 compare_files () {
-	tr '\015\000' QN <"$1" >"$1".expect &&
+	{ test -f "$1".expect || tr '\015\000' QN <"$1" >"$1".expect; } &&
 	tr '\015\000' QN <"$2" | tr -d 'Z' >"$2".actual &&
-	test_cmp "$1".expect "$2".actual &&
-	rm "$1".expect "$2".actual
+	test_cmp "$1".expect "$2".actual
 }
 
 compare_ws_file () {
 	pfx=$1
-	exp=$2.expect
+	exp=$2.ws-expect
 	act=$pfx.actual.$3
-	tr '\015\000abcdef0123456789' QN00000000000000000 <"$2" |
-		sed -e "s/0000*/$ZERO_OID/" >"$exp" &&
+	{
+		test -f "$exp" ||
+		tr '\015\000abcdef0123456789' QN00000000000000000 <"$2" |
+			sed -e "s/0000*/$ZERO_OID/" >"$exp" ;
+	} &&
 	tr '\015\000abcdef0123456789' QN00000000000000000 <"$3" |
 		sed -e "s/0000*/$ZERO_OID/" >"$act" &&
-	test_cmp "$exp" "$act" &&
-	rm "$exp" "$act"
+	test_cmp "$exp" "$act"
 }
 
 create_gitattributes () {
-- 
2.56.0.611.g1030f0b940.dirty
Patrick Steinhardt· Nov 11, 2026, 10:47 UTC · re: Patrick Steinhardt · lore

[PATCH v3 6/7] t1092: set up repositories from templates

In t1092 we verify the behaviour of sparse checkouts. We use three different repositories to do so: one with a full checkout, one with a sparse checkout and one with a sparse index, and then verify that the result is the same across all these configurations.

Those repositories get set up via `init_repos ()`, which initializes them from scratch every single time with a bunch of processes. The function is called more than a 100 times though, so the overhead adds up quite a bit. While the tests only require around 30 seconds on Linux, it takes more than 10 minutes on Windows.

Optimize this by instead creating the repositories once, only. So instead of recreating them from scratch, we can then just copy the template repositories into place and thus save a bunch of processes. This speeds up tests on Windows by about 3 minutes.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t1092-sparse-checkout-compatibility.sh | 34 +++++++++++++++++++-------------
 1 file changed, 20 insertions(+), 14 deletions(-)
Show changes to t/t1092-sparse-checkout-compatibility.sh +20 −14
diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh
index 05b54062b3b..5d81ec54c23 100755
--- a/t/t1092-sparse-checkout-compatibility.sh
+++ b/t/t1092-sparse-checkout-compatibility.sh
@@ -141,28 +141,34 @@ test_expect_success 'setup' '
 	)
 '
 
+test_expect_success 'setup template repositories' '
+	cp -r initial-repo full-checkout.template &&
+	cp -r initial-repo sparse-checkout.template &&
+	cp -r initial-repo sparse-index.template &&
+
+	# initialize sparse-checkout definitions
+	git -C sparse-checkout.template sparse-checkout init --cone &&
+	git -C sparse-checkout.template sparse-checkout set deep &&
+	git -C sparse-index.template sparse-checkout init --cone --sparse-index &&
+	test_cmp_config -C sparse-index.template true index.sparse &&
+	git -C sparse-index.template sparse-checkout set deep &&
+
+	# Disable this message to keep stderr the same.
+	git -C sparse-index.template config advice.sparseIndexExpanded false
+'
+
 init_repos () {
 	rm -rf full-checkout sparse-checkout sparse-index &&
 
 	# create repos in initial state
-	cp -r initial-repo full-checkout &&
+	cp -r full-checkout.template full-checkout &&
 	git -C full-checkout reset --hard &&
 
-	cp -r initial-repo sparse-checkout &&
+	cp -r sparse-checkout.template sparse-checkout &&
 	git -C sparse-checkout reset --hard &&
 
-	cp -r initial-repo sparse-index &&
-	git -C sparse-index reset --hard &&
-
-	# initialize sparse-checkout definitions
-	git -C sparse-checkout sparse-checkout init --cone &&
-	git -C sparse-checkout sparse-checkout set deep &&
-	git -C sparse-index sparse-checkout init --cone --sparse-index &&
-	test_cmp_config -C sparse-index true index.sparse &&
-	git -C sparse-index sparse-checkout set deep &&
-
-	# Disable this message to keep stderr the same.
-	git -C sparse-index config advice.sparseIndexExpanded false
+	cp -r sparse-index.template sparse-index &&
+	git -C sparse-index reset --hard
 }
 
 init_repos_as_submodules () {
-- 
2.56.0.611.g1030f0b940.dirty
Patrick Steinhardt· Nov 11, 2026, 10:47 UTC · re: Patrick Steinhardt · lore

[PATCH v3 7/7] t5551: use reftables for repositories with many refs

Some of the tests in t5551 creates almost a 100,000 references and then verify that those can be fetched. Naturally, creating this many refs is quite involved with the "files" backend, as it means that we have to create this many individual loose files. This is tolerable on Linux, where the test takes around 2 minutes in our CI. But on macOS, the same test takes almost 20 minutes to complete. And to add insult to injury, we execute these tests a second time via t5559.

The system under test here is the HTTP negotiation, so we don't actually care about whether or not the ref backend handles this alright. So adapt the test to instead create repositories with the reftable backend, which is significantly faster in this case as we only need to write a single table.

And sure enough, this reduces the runtime of this test on macOS by a factor of 10, going from 20 minutes to 2 minutes. On Linux, the tests at least speed up by a factor of two.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t5551-http-fetch-smart.sh | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
Show changes to t/t5551-http-fetch-smart.sh +4 −2
diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh
index 130053c57aa..34945059531 100755
--- a/t/t5551-http-fetch-smart.sh
+++ b/t/t5551-http-fetch-smart.sh
@@ -412,7 +412,8 @@ create_tags () {
 }
 
 test_expect_success 'create 2,000 tags in the repo' '
-	git init --bare "$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
+	git init --bare --ref-format=reftable \
+		"$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
 	(
 		cd "$HTTPD_DOCUMENT_ROOT_PATH/many-tags.git" &&
 		create_tags 1 2000
@@ -421,7 +422,8 @@ test_expect_success 'create 2,000 tags in the repo' '
 
 test_expect_success CMDLINE_LIMIT \
 	'clone the 2,000 tag repo to check OS command line overflow' '
-	run_with_limited_cmdline git clone $HTTPD_URL/smart/many-tags.git too-many-refs &&
+	run_with_limited_cmdline git clone --ref-format=reftable \
+		$HTTPD_URL/smart/many-tags.git too-many-refs &&
 	(
 		cd too-many-refs &&
 		git for-each-ref refs/tags >actual &&
-- 
2.56.0.611.g1030f0b940.dirty

← back to recent threads