git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 3/9] t4141: fix inefficient use of dd(1)

From
Patrick Steinhardt <ps@pks.im>
Date
Jul 3, 2026, 05:42 UTC
Message-ID
<akdLqNHW3pGThQat@pks.im>
In-Reply-To
<20260702211614.GB2051171@coredump.intra.peff.net>
On Thu, Jul 02, 2026 at 05:16:14PM -0400, Jeff King wrote:
Show 21 quoted lines
> On Thu, Jul 02, 2026 at 02:00:56PM +0200, Patrick Steinhardt wrote:
> 
> > This test setup is extremely expensive, as `test_copy_bytes` is
> > implemented via `dd ibs=1 count="$1"`, which copies data one byte at a
> > time. So as we write 1GB of data, we end up doing 1 billion reads and
> > writes. This naturally takes a while: it takes 6 minutes on my system,
> > and around 40 minutes in some CI jobs!
> > 
> > We can do much better though, as genzeros already knows to handle an
> > optional limit of how much data it is supposed to write, which allows us
> > to remove the call to `test_copy_bytes`. Furthermore, it has already
> > been optimized to generate the data fast.
> 
> Seems like a good fix for this case, where we can skip the extra process
> entirely.
> 
> It feels like test_copy_bytes should be able to do much better in
> general. The obvious thing to reach for is "head -c", but the function
> was originally added because that wasn't portable. The "-c" option is
> not in POSIX, though the original comment claims IRIX was the problem,
> so I wonder if "head -c" is de facto portable these days.

An alternative could be to implement a simple helper as part of our test-tool. But I doubt it's really worth it: almost all callers only want to copy a small number of bytes. The only exception seems to be t0021, where we copy up to 65kB. But that whole test suite still only takes ~3 seconds, so optimizing that feels like wasted time to me.

Patrick
Previous: Jeff KingNext: Jeff King
Message 8 of 33 in “t: fixes and improvements for GIT_TEST_LONG”
  1. 0/9 t: fixes and improvements for GIT_TEST_LONGPatrick Steinhardt, Jul 2, 2026
  2. 1/9 README: add GitLab CI badge to make it more discoverablePatrick Steinhardt, Jul 2, 2026
  3. 2/9 t0021: skip EXPENSIVE test that is broken without SIZE_T_IS_32BITPatrick Steinhardt, Jul 2, 2026
  4. 3/9 t4141: fix inefficient use of dd(1)Patrick Steinhardt, Jul 2, 2026
  5. SZEDER GáborJul 2, 2026
  6. Patrick SteinhardtJul 3, 2026
  7. Jeff KingJul 2, 2026
  8. Patrick SteinhardtJul 3, 2026
  9. Jeff KingJul 3, 2026
  10. 4/9 t5608: reduce maximum disk usagePatrick Steinhardt, Jul 2, 2026
  11. 5/9 t7508: skip EXPENSIVE test that is broken without SIZE_T_IS_32BITPatrick Steinhardt, Jul 2, 2026
  12. Jeff KingJul 2, 2026
  13. Jeff KingJul 2, 2026
  14. 6/9 t7900: clean up large EXPENSIVE repositoryPatrick Steinhardt, Jul 2, 2026
  15. Jeff KingJul 2, 2026
  16. Patrick SteinhardtJul 3, 2026
  17. Jeff KingJul 3, 2026
  18. 7/9 t: use `test_bool_env` to parse GIT_TEST_LONGPatrick Steinhardt, Jul 2, 2026
  19. 8/9 gitlab-ci: disable RAM disk on macOS jobsPatrick Steinhardt, Jul 2, 2026
  20. 9/9 gitlab-ci: enable "GIT_TEST_LONG"Patrick Steinhardt, Jul 2, 2026
  21. 0/9 t: fixes and improvements for GIT_TEST_LONGPatrick Steinhardt, Jul 3, 2026
  22. 1/9 README: add GitLab CI badge to make it more discoverablePatrick Steinhardt, Jul 3, 2026
  23. 2/9 t0021: skip EXPENSIVE test that is broken without SIZE_T_IS_32BITPatrick Steinhardt, Jul 3, 2026
  24. Junio C HamanoJul 3, 2026
  25. Patrick SteinhardtJul 6, 2026
  26. 3/9 t4141: fix inefficient use of dd(1)Patrick Steinhardt, Jul 3, 2026
  27. 4/9 t5608: reduce maximum disk usagePatrick Steinhardt, Jul 3, 2026
  28. 5/9 t7508: skip EXPENSIVE test that is broken without SIZE_T_IS_32BITPatrick Steinhardt, Jul 3, 2026
  29. Junio C HamanoJul 3, 2026
  30. 6/9 t7900: clean up large EXPENSIVE repositoryPatrick Steinhardt, Jul 3, 2026
  31. 7/9 t: use `test_bool_env` to parse GIT_TEST_LONGPatrick Steinhardt, Jul 3, 2026
  32. 8/9 gitlab-ci: disable RAM disk on macOS jobsPatrick Steinhardt, Jul 3, 2026
  33. 9/9 gitlab-ci: enable "GIT_TEST_LONG"Patrick Steinhardt, Jul 3, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.