Re: [PATCH 2/3] t: add "test-tool genbytes" to avoid slow tr(1) on macOS
- From
Kristofer Karlsson <krka@spotify.com>
- Date
- Nov 8, 2026, 09:18 UTC
- Message-ID
- <CAL71e4NFyAShDZPW9JRB7Bpg3aLY5mY4wdn5m_ZYoLQjHZw_jA@mail.gmail.com>
- In-Reply-To
- <50b6c35f-e9c3-47ef-8fca-aebd2620623a@kdbg.org>
On Sun, 8 Nov 2026 at 09:54, Johannes Sixt <j6t@kdbg.org> wrote:
Show 37 quoted lines
> > Am 07.11.26 um 15:59 schrieb Kristofer Karlsson via GitGitGadget: > > Several tests create fixture data consisting of a single repeated > > character with "test-tool genzeros <count> | tr '\000' <char>". This > > works well on platforms where tr(1) is implemented efficiently. > > > > The implementation of tr(1) on macOS supports multi-byte characters, > > and processes its input one character at a time through > > getwchar() and putwchar(), even in the C locale. For small inputs > > that does not matter, but t4205 generates more than 2GB of data this > > way, which makes that test much slower on macOS than on Linux: on the > > macOS CI runners, tr(1) alone spends 95 to 123 seconds of CPU time on > > it, compared to about 6 seconds on Linux. > > > > Fix this by adding "test-tool genbytes <byte> [<count>]", which writes > > the requested byte directly, and turn "genzeros" into a thin > > wrapper around the same code. > > All cases that do not need a particular character can use > > printf %0.500d 0 > > Though, on my Linux box, > > printf %0.2147483649d 0 > > produces nothing, but > > printf 000000000%0.2147483640d 0 > > has 2147483649 bytes of output. > > If this isn't insanely slow on macOS, use this pattern instead of a > highly engineered helper program? ;-) > > -- Hannes >
Nice trick, and it would work for t4205, which does not care about the byte value. I tried it for the 2147483649 bytes there on Linux:
printf time peak memory dash 6.5s 2.1GB bash 6.0s 3MB coreutils 3.7s 2.1GB
It is definitely fast enough (on the variants I tested). The annoying thing (but not a real blocker) is the extra memory usage required and I am not sure I fully trust that all implementations will behave the same for absurdly large padding.
(Particularly for paddings larger than 2^31, as you noted too.)
It feels a bit like a hack, and tweaking the existing test-tool feels like a more sane option that is easier to reason about.
A gotcha here is that the printf hack only works for producing "0", so for the cases we truly need NUL characters it does not help us, and I think that is also a point in favor of simply generalizing test-tool genzeros instead.
Thanks, Kristofer