From: Collin Funk Date: Tue, 10 Nov 2026 06:00:53 GMT Subject: Re: [PATCH 2/3] t: add "test-tool genbytes" to avoid slow tr(1) on macOS Message-ID: <8733t9z0mi.fsf@gmail.com> In-Reply-To: Kristofer Karlsson writes: > Nice trick, and it would work for t4205, which does not care about > the byte value. I tried it for the 2147483649 bytes there on Linux: > > printf time peak memory > dash 6.5s 2.1GB > bash 6.0s 3MB > coreutils 3.7s 2.1GB > > It is definitely fast enough (on the variants I tested). > The annoying thing (but not a real blocker) is the extra memory > usage required and I am not sure I fully trust that all > implementations will behave the same for absurdly large padding. > > (Particularly for paddings larger than 2^31, as you noted too.) > > It feels a bit like a hack, and tweaking the existing test-tool > feels like a more sane option that is easier to reason about. > > A gotcha here is that the printf hack only works for producing > "0", so for the cases we truly need NUL characters it does not > help us, and I think that is also a point in favor of > simply generalizing test-tool genzeros instead. I've mentioned this idea on coreutils or gnulib's mailing lists before, but I forget where and when. I've wanted to create a printf variant that doesn't allocate much memory and will just use putc_unlocked for each padding character, which would avoid the memory issue you bring up. Most platforms will be limited to INT_MAX there, since that is what POSIX specifies is supplied with '*' [1]: A field width, or precision, or both, may be indicated by an ('*'). In this case an argument of type int supplies the field width or precision. E.g., in an invocation like this where the field width is supplied with another argument: $ printf '%0*d\n' 5 1 00001 I think my original reason for wanting a different printf variant was escaping that limit. For generating bytes where the values don't matter, in GNU coreutils we frequently use 'yes' for that. As a result, the implementation is very overengineered. E.g., it will use zero-copy since coreutils-9.11. Before that, it would reuse ARGV if it was contiguous and the arguments used to determine the printed string were long enough such that we weren't calling read/write on a tiny string, or it would generate a large buffer before using read and write. Here is the performance from before and after zero-copy was introduced on a beefy Power10 machine, stolen from the commit message [2]: IBM,9043-MRX $ taskset 1 yes | taskset 2 pv > /dev/null ... [11.6GiB/s] IBM,9043-MRX $ taskset 1 src/yes | taskset 2 pv > /dev/null ... [175GiB/s] This was introduced around the time of the copy-fail vulnerability and various zero-copy vulnerabilities, where there was serious consideration given to nerfing splice and vmsplice. Luckily for us, that never happened. FreeBSD, and therefore, I assume, macOS use something similar to our previous read/write method, which is fast enough. I think OpenBSD and NetBSD use puts on ARGV or the "y" string, so it's relatively slow. GNU 'tr' doesn't handle multi-byte characters, so you can adjust that output if needed and remain relatively quick: $ timeout 10 yes 0 | LC_ALL=C tr -d '\n' | pv -r > /dev/null [ 616MiB/s] I'm not sure if macOS 'tr' handles multibyte characters. But it might be worth checking if they do, and if they keep a fast case for single-byte locales. Doing that without duplicating code is quite tricky, which is part of the reason why we haven't done it in GNU 'tr' yet (but recent releases have improved multi-byte character support for other utilities, so hopefully it is handled soon). Probably more information than you need, but hopefully it helps you generate slop sometimes needed for testing. :) Collin [1] https://pubs.opengroup.org/onlinepubs/9799919799/functions/printf.html [2] https://github.com/coreutils/coreutils/commit/2b1c059e6a06eebbb721d010b1221ec54200cc33