The Git project uses CI systems from both GitHub and GitLab. While both of these systems are extensively used in day-to-day work, we only have a link to the GitHub Workflows in our README, which makes the GitLab CI hard to discover.

Improve the situation by adding a second badge for GitLab CI to our README.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 README.md | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Hi,

this series started out as a simple two-patch series that wired up the GitLab CI badge in our README and GIT_TEST_LONG for GitLab CI. But as it typically goes, tests broke on GitLab CI, which made me realize that they are broken even on GitHub's master branch right now. Some tests are failing in the linux32 job, and we only didn't notice because the whole pipeline hangs.

So I had to go down the rabbit hole a bit, the result of which is this patch series.

Changes in v3:
  - Fix commit subjects to mention correct prerequisite.
  - Link to v2: https://patch.msgid.link/20260703-b4-pks-t-fixes-for-GIT-TEST-LONG-v2-0-79076a7e0c62@pks.im

Changes in v2:
  - Reinstate the EXPENSIVE prerequisite on t4141, as we still end up
    reading 1GB of data into memory.
  - Improve a rather hand-wavy commit message.
  - Link to v1: https://patch.msgid.link/20260702-b4-pks-t-fixes-for-GIT-TEST-LONG-v1-0-76b4d7bab3d0@pks.im

Thanks!

Patrick

[1]: <akIfsaVMB_S6kfJQ@pks.im>

---
Patrick Steinhardt (9):
      README: add GitLab CI badge to make it more discoverable
      t0021: skip EXPENSIVE test that is broken without SIZE_T_IS_64BIT
      t4141: fix inefficient use of dd(1)
      t5608: reduce maximum disk usage
      t7508: skip EXPENSIVE test that is broken without SIZE_T_IS_64BIT
      t7900: clean up large EXPENSIVE repository
      t: use `test_bool_env` to parse GIT_TEST_LONG
      gitlab-ci: disable RAM disk on macOS jobs
      gitlab-ci: enable "GIT_TEST_LONG"

 .gitlab-ci.yml             | 13 ++++-----
 README.md                  |  3 ++-
 ci/lib.sh                  | 12 +++++++--
 t/t0021-conversion.sh      |  2 +-
 t/t4141-apply-too-large.sh |  5 ++--
 t/t5608-clone-2gb.sh       | 66 ++++++++++++++++++++++++----------------------
 t/t7508-status.sh          |  2 +-
 t/t7900-maintenance.sh     | 56 +++++++++++++++++++++------------------
 t/test-lib.sh              |  4 +--
 9 files changed, 91 insertions(+), 72 deletions(-)

Range-diff versus v2:

 1:  e4add14ea7 =  1:  afc7563e22 README: add GitLab CI badge to make it more discoverable
 2:  d762b4d46e !  2:  753e950eaf t0021: skip EXPENSIVE test that is broken without SIZE_T_IS_32BIT
    @@ Metadata
     Author: Patrick Steinhardt <ps@pks.im>
     
      ## Commit message ##
    -    t0021: skip EXPENSIVE test that is broken without SIZE_T_IS_32BIT
    +    t0021: skip EXPENSIVE test that is broken without SIZE_T_IS_64BIT
     
         One of the tests in t0021 writes a 2GB file and then roundtrips it
         through the clean/sumdge filters. This test is broken on 32 bit
 3:  8d43eb2819 =  3:  f776e0fb5f t4141: fix inefficient use of dd(1)
 4:  fcd048f6f7 =  4:  9754b96a43 t5608: reduce maximum disk usage
 5:  11df7f2cb9 !  5:  0f2e28dc11 t7508: skip EXPENSIVE test that is broken without SIZE_T_IS_32BIT
    @@ Metadata
     Author: Patrick Steinhardt <ps@pks.im>
     
      ## Commit message ##
    -    t7508: skip EXPENSIVE test that is broken without SIZE_T_IS_32BIT
    +    t7508: skip EXPENSIVE test that is broken without SIZE_T_IS_64BIT
     
         One of the tests in t7508 is marked as EXPENSIVE because it ends up
         creating and adding files that are multiple gigabytes in size. This
 6:  a16bc1754b =  6:  d329a2cd40 t7900: clean up large EXPENSIVE repository
 7:  b2e6b0d517 =  7:  a336d4ce9e t: use `test_bool_env` to parse GIT_TEST_LONG
 8:  9632b19164 =  8:  cfff94c79e gitlab-ci: disable RAM disk on macOS jobs
 9:  a42c613012 =  9:  ed5e8807fe gitlab-ci: enable "GIT_TEST_LONG"

--- base-commit: e9019fcafe0040228b8631c30f97ae1adb61bcdc change-id: 20260701-b4-pks-t-fixes-for-GIT-TEST-LONG-78e538bf0e06

One of the tests in t0021 writes a 2GB file and then roundtrips it through the clean/sumdge filters. This test is broken on 32 bit platforms because they typically don't handle files larger then `SSIZE_MAX` well at all.

While our CI has a "linux32" job that should in theory hit this issue, we never noticed it because we didn't use to run EXPENSIVE tests until 7a094d68a2 (ci: run expensive tests on push builds to integration branches, 2026-05-08). And after that commit, the test does not fail but instead hangs completely.

Ideally, we'd of course properly detect this situation and then test for it. In practice, this turns out to be hard as the test failure are not reliable as they often (but not always) run into ENOMEM errors.

Instead, skip the test altogether.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t0021-conversion.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

In t4141 we generate a patch that is roughly 1GB in size to verify that git-apply(1) indeed rejects that patch. We generate that patch by prepending a patch header and then executing `test-tool genzeros` without a limit. This causes us to print infinitely many zeros, and we limit the overall amount of generated bytes via `test_copy_bytes`.

This test setup is extremely expensive, as `test_copy_bytes` is implemented via `dd ibs=1 count="$1"`, which copies data one byte at a time. So as we write 1GB of data, we end up doing 1 billion reads and writes. This naturally takes a while: it takes 6 minutes on my system, and around 40 minutes in some CI jobs!

We can do much better though, as genzeros already knows to handle an optional limit of how much data it is supposed to write, which allows us to remove the call to `test_copy_bytes`. Furthermore, it has already been optimized to generate the data fast.

And indeed, doing this conversion drops the test execution to less than a second on my machine. That means that in theory it becomes feasible to drop the EXPENSIVE prerequisite now. But git-apply(1) still soaks up 1GB of data into memory, which may count as being expensive. Consequently, we keep the prerequisite intact.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t4141-apply-too-large.sh | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

The tests in t5608 perform a couple of clones of repositories that are somewhat large. Ultimately, we end up creating:

  - A setup repository that contains 2GB of uncompressed pack data.

  - A bare clone that contains the same 2GB of data.

  - A clone with worktree writes a 2GB packfile and a 2GB worktree.

  - A second setup repository that contains a 4GB packfile.

  - Two 4GB clone of that repository.

Some of these clones ultimately hardlink files, which ensures that we at least don't end up with more than 20GB of data. But at the end of the test we still have around 16GB of data, which is only a tiny bit better.

Refactor the test to prune repositories after they have no use anymore. This reduced the peak disk usage of this test to 8GB.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t5608-clone-2gb.sh | 66 ++++++++++++++++++++++++++++------------------------
 1 file changed, 35 insertions(+), 31 deletions(-)

One of the tests in t7508 is marked as EXPENSIVE because it ends up creating and adding files that are multiple gigabytes in size. This takes a while to complete, hence the EXPENSIVE prerequisite.

Besides being expensive though the test can only work on systems where `size_t` is at least 64 bit. This is because one of the created files is larger than 4GB, and because Git tracks object size via `size_t` it will eventually blow up.

This test has also been blowing up in the "linux32" CI job in GitHub Workflows since 7a094d68a2 (ci: run expensive tests on push builds to integration branches, 2026-05-08). But that job doesn't only fail, it also hangs, and that has been concealing the failure.

Fix the issue by marking the test as requiring 64 bit `size_t`.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t7508-status.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

One of the tests in t7900 is marked with EXPENSIVE because we create a repository with 2GB of data that we end up repacking. We never clean up that repository though, so we occupy the full 2GB of data until the end of the test suite.

Besides clogging our disk, having an EXPENSIVE test that alters the repository's state used by subsequent tests is also a bad idea, as it can easily have an impact on the heuristics used by other maintenance tasks.

Adapt the test so that we create the data in a standalone repository that we clean up at the end of the test. While at it, also disable auto-maintenance so that it does not race with our manual maintenance.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t7900-maintenance.sh | 56 ++++++++++++++++++++++++++++----------------------
 1 file changed, 31 insertions(+), 25 deletions(-)

It's currently hard to explicitly disable GIT_TEST_LONG by setting it to `false`. Fix this by using `test_bool_env` instead.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 ci/lib.sh     | 2 +-
 t/test-lib.sh | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

When we added the macOS jobs to GitLab CI in 56090a35ab (ci: add macOS jobs to GitLab CI, 2024-01-18) we had to work around some very slow disks. This workaround essentially creates a RAM disk that we mount, where all test data is being written into RAM instead of the real disk.

In the next commit though we're about to enable "GIT_TEST_LONG", which will make tests run that are marked with the "EXPENSIVE" prerequisite. This change will make a couple of tests run that write up to 8GB of data into the test output directory. As our RAM disk is only 4GB in size, this change will cause ENOSPC errors.

We could accommodate for this by increasing the size of the RAM disk. In c9d708b7fc (gitlab-ci: upgrade macOS runners, 2026-05-21) we have upgraded our runners to use the "large" runners, which have 16GB of RAM available. So we could easily expand the RAM disk to a capacity of for example 12GB. But some test runs have shown that this is still quite flaky overall, as we get quite close to our limits.

Instead, drop the workaround completely. This does indeed slow down execution of the test jobs:

  - osx-clang goes from 18 minutes to 25 minutes

  - osx-meson goes from 21 minutes to 33 minutes

  - osx-reftable stays at 21 minutes

The last one seems like an outlier. The only explanation that I have is that we end up writing significantly less files with the reftable backend, which ultimately causes less I/O.

Overall though, it's preferable to have something that works with the least amount of flakiness compared to having something else that is faster but unstable. Despite that, the macOS jobs aren't even the slowest jobs, so this doesn't extend the overall pipeline's length.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)

Starting with 7a094d68a2 (ci: run expensive tests on push builds to integration branches, 2026-05-08) we run expensive tests in our CI for certain events. So far, this has only been wired up for GitHub Workflows though, which creates a test gap for GitLab CI.

Plug this gap by also making this work for the latter.

Note that these tests cannot be run on the Windows runners, as they only have 7.5GB of RAM. This is insufficient for some of the EXPENSIVE tests, so we explicitly disable "GIT_TEST_LONG" on these jobs.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .gitlab-ci.yml |  6 ++++++
 ci/lib.sh      | 12 ++++++++++--
 2 files changed, 16 insertions(+), 2 deletions(-)

Patrick Steinhardt <ps@pks.im> writes:

The interdiff looks trivially correct ;-).

Hopefully we are now ready to declare victory and plan to merge this to 'next'?

Thanks.

On Mon, Jul 06, 2026 at 01:29:45PM -0700, Junio C Hamano wrote:

Yeah. I wouldn't say I did a super-deep review, but I did look over the original and have no complaints.

-Peff