Re: [PATCH 3/9] t4141: fix inefficient use of dd(1)
- From
Jeff King <peff@peff.net>
- Date
- Jul 2, 2026, 21:16 UTC
- Message-ID
- <20260702211614.GB2051171@coredump.intra.peff.net>
- In-Reply-To
- <20260702-b4-pks-t-fixes-for-GIT-TEST-LONG-v1-3-76b4d7bab3d0@pks.im>
On Thu, Jul 02, 2026 at 02:00:56PM +0200, Patrick Steinhardt wrote:
Show 10 quoted lines
> This test setup is extremely expensive, as `test_copy_bytes` is > implemented via `dd ibs=1 count="$1"`, which copies data one byte at a > time. So as we write 1GB of data, we end up doing 1 billion reads and > writes. This naturally takes a while: it takes 6 minutes on my system, > and around 40 minutes in some CI jobs! > > We can do much better though, as genzeros already knows to handle an > optional limit of how much data it is supposed to write, which allows us > to remove the call to `test_copy_bytes`. Furthermore, it has already > been optimized to generate the data fast.
Seems like a good fix for this case, where we can skip the extra process entirely.
It feels like test_copy_bytes should be able to do much better in general. The obvious thing to reach for is "head -c", but the function was originally added because that wasn't portable. The "-c" option is not in POSIX, though the original comment claims IRIX was the problem, so I wonder if "head -c" is de facto portable these days.
I'd use perl of course. ;) The history here is somewhat amusing. We originally did use dd, but that changed in 4de0bbd898 (t9300: use perl "head -c" clone in place of "dd bs=1 count=16000" kluge, 2010-12-13) because dd was slow. The code moved to test-lib.sh in 48860819e8 (t9300: factor out portable "head -c" replacement, 2016-06-30), where I rejected the dd solution because it was slow. And then the perl turned back into dd in 01486b5de8 (t: adapt `test_copy_bytes()` to not use Perl, 2025-04-03), becoming slow again.
Chesterton's fence at work?
-Peff