git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 2/3] t: add "test-tool genbytes" to avoid slow tr(1) on macOS

From
Collin Funk <collin.funk1@gmail.com>
Date
Nov 10, 2026, 06:00 UTC
Message-ID
<8733t9z0mi.fsf@gmail.com>
In-Reply-To
<CAL71e4NFyAShDZPW9JRB7Bpg3aLY5mY4wdn5m_ZYoLQjHZw_jA@mail.gmail.com>
Kristofer Karlsson <krka@spotify.com> writes:
Show 22 quoted lines
> Nice trick, and it would work for t4205, which does not care about
> the byte value.  I tried it for the 2147483649 bytes there on Linux:
>
>   printf           time   peak memory
>   dash             6.5s   2.1GB
>   bash             6.0s   3MB
>   coreutils        3.7s   2.1GB
>
> It is definitely fast enough (on the variants I tested).
> The annoying thing (but not a real blocker) is the extra memory
> usage required and I am not sure I fully trust that all
> implementations will behave the same for absurdly large padding.
>
> (Particularly for paddings larger than 2^31, as you noted too.)
>
> It feels a bit like a hack, and tweaking the existing test-tool
> feels like a more sane option that is easier to reason about.
>
> A gotcha here is that the printf hack only works for producing
> "0", so for the cases we truly need NUL characters it does not
> help us, and I think that is also a point in favor of
> simply generalizing test-tool genzeros instead.

I've mentioned this idea on coreutils or gnulib's mailing lists before, but I forget where and when. I've wanted to create a printf variant that doesn't allocate much memory and will just use putc_unlocked for each padding character, which would avoid the memory issue you bring up.

Most platforms will be limited to INT_MAX there, since that is what POSIX specifies is supplied with '*' [1]:

    A field width, or precision, or both, may be indicated by an
    <asterisk> ('*'). In this case an argument of type int supplies the
    field width or precision.

E.g., in an invocation like this where the field width is supplied with another argument:

    $ printf '%0*d\n' 5 1
    00001

I think my original reason for wanting a different printf variant was escaping that limit.

For generating bytes where the values don't matter, in GNU coreutils we frequently use 'yes' for that. As a result, the implementation is very overengineered. E.g., it will use zero-copy since coreutils-9.11. Before that, it would reuse ARGV if it was contiguous and the arguments used to determine the printed string were long enough such that we weren't calling read/write on a tiny string, or it would generate a large buffer before using read and write. Here is the performance from before and after zero-copy was introduced on a beefy Power10 machine, stolen from the commit message [2]:

   IBM,9043-MRX $ taskset 1 yes | taskset 2 pv > /dev/null
   ... [11.6GiB/s]
   IBM,9043-MRX $ taskset 1 src/yes | taskset 2 pv > /dev/null
   ... [175GiB/s]

This was introduced around the time of the copy-fail vulnerability and various zero-copy vulnerabilities, where there was serious consideration given to nerfing splice and vmsplice. Luckily for us, that never happened.

FreeBSD, and therefore, I assume, macOS use something similar to our previous read/write method, which is fast enough. I think OpenBSD and NetBSD use puts on ARGV or the "y" string, so it's relatively slow.

GNU 'tr' doesn't handle multi-byte characters, so you can adjust that output if needed and remain relatively quick:

    $ timeout 10 yes 0 | LC_ALL=C tr -d '\n' | pv -r > /dev/null
    [ 616MiB/s]

I'm not sure if macOS 'tr' handles multibyte characters. But it might be worth checking if they do, and if they keep a fast case for single-byte locales. Doing that without duplicating code is quite tricky, which is part of the reason why we haven't done it in GNU 'tr' yet (but recent releases have improved multi-byte character support for other utilities, so hopefully it is handled soon).

Probably more information than you need, but hopefully it helps you generate slop sometimes needed for testing. :)

Collin

[1] https://pubs.opengroup.org/onlinepubs/9799919799/functions/printf.html [2] https://github.com/coreutils/coreutils/commit/2b1c059e6a06eebbb721d010b1221ec54200cc33

Previous: Johannes SixtNext: Kristofer Karlsson
Message 17 of 39 in “t: add "test-tool genbytes" to avoid slow tr(1) on macOS”
  1. 0/3 t: add "test-tool genbytes" to avoid slow tr(1) on macOSKristofer Karlsson via GitGitGadget, Nov 7, 2026
  2. 1/3 t/helper: rename test-genzeros.c to test-genbytes.cKristofer Karlsson via GitGitGadget, Nov 7, 2026
  3. Junio C HamanoNov 7, 2026
  4. Junio C HamanoNov 8, 2026
  5. Kristofer KarlssonNov 8, 2026
  6. Junio C HamanoNov 8, 2026
  7. Kristofer KarlssonNov 8, 2026
  8. 2/3 t: add "test-tool genbytes" to avoid slow tr(1) on macOSKristofer Karlsson via GitGitGadget, Nov 7, 2026
  9. Johannes SixtNov 8, 2026
  10. Kristofer KarlssonNov 8, 2026
  11. Johannes SixtNov 8, 2026
  12. Kristofer KarlssonNov 8, 2026
  13. Johannes SixtNov 8, 2026
  14. Johannes SixtNov 9, 2026
  15. Kristofer KarlssonNov 9, 2026
  16. Johannes SixtNov 9, 2026
  17. Collin FunkNov 10, 2026
  18. Kristofer KarlssonNov 10, 2026
  19. Junio C HamanoNov 8, 2026
  20. 3/3 t: replace "test-tool genzeros" with "genbytes NUL"Kristofer Karlsson via GitGitGadget, Nov 7, 2026
  21. Junio C HamanoNov 7, 2026
  22. Kristofer KarlssonNov 8, 2026
  23. Junio C HamanoNov 7, 2026
  24. Kristofer KarlssonNov 8, 2026
  25. 0/2 t: add "test-tool genbytes" to avoid slow tr(1) on macOSKristofer Karlsson via GitGitGadget, Nov 8, 2026
  26. 1/2 t: add "test-tool genbytes" to avoid slow tr(1) on macOSKristofer Karlsson via GitGitGadget, Nov 8, 2026
  27. Junio C HamanoNov 9, 2026
  28. Kristofer KarlssonNov 9, 2026
  29. Junio C HamanoNov 9, 2026
  30. Patrick SteinhardtNov 9, 2026
  31. Kristofer KarlssonNov 9, 2026
  32. 2/2 t: replace "test-tool genzeros" with "genbytes 00"Kristofer Karlsson via GitGitGadget, Nov 8, 2026
  33. t: add "test-tool genbytes" to avoid slow tr(1) on macOSKristofer Karlsson via GitGitGadget, Nov 10, 2026
  34. Junio C HamanoNov 10, 2026
  35. Patrick SteinhardtNov 11, 2026
  36. Junio C HamanoNov 11, 2026
  37. Kristofer KarlssonNov 11, 2026
  38. Junio C HamanoNov 11, 2026
  39. t: add "test-tool genbytes" to avoid slow tr(1) on macOSKristofer Karlsson via GitGitGadget, Nov 11, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.