git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v6 10/13] convert: generate large test files only once

From
Jeff King <peff@peff.net>
Date
Aug 30, 2016, 16:37 UTC
Message-ID
<20160830163749.it4nedannqzqeh5c@sigill.intra.peff.net>
In-Reply-To
<F976C223-0CF3-43FB-ABF2-FFE7C600138E@gmail.com>
On Tue, Aug 30, 2016 at 01:41:59PM +0200, Lars Schneider wrote:
Show 14 quoted lines
> >> +	git checkout -- test test.t test.i &&
> >> +
> >> +	mkdir generated-test-data &&
> >> +	for i in $(test_seq 1 $T0021_LARGE_FILE_SIZE)
> >> +	do
> >> +		RANDOM_STRING="$(test-genrandom end $i | tr -dc "A-Za-z0-9" )"
> >> +		ROT_RANDOM_STRING="$(echo $RANDOM_STRING | ./rot13.sh )"
> > 
> > In earlier iteration of loop with lower $i, what guarantees that
> > some bytes survive "tr -dc"?
> 
> Nothing really, good catch! The seed "end" produces as first character always a 
> "S" which would survive "tr -dc". However, that is clunky. I will always set "1"
> as first character in $RANDOM_STRING to mitigate the problem.

It seems odd that you would generate a larger set of random bytes and then throw most of them away (about 1 in 5, on average). So you don't actually know how long your inputs are, and you're wasting time generating bytes which are discarded.

The goal looks like it is just to clean up the string to only-ASCII characters. Perhaps converting to to base64 or hex would be conceptually simpler? Like:

  test-genrandom ... |
  perl -pe 's/./hex(ord($&))/sge'
Show 17 quoted lines
> > I do not quite get the point of this complexity.  You are using
> > exactly the same seed "end" every time, so in the first round you
> > have 1M of SP, letter '1', letter 'S' (from the genrandom), then
> > in the second round you have 1M of SP, letter '1', letter 'S' and
> > letter 'p' (the last two from the genrandom), and go on.  Is it
> > significant for the purpose of your test that the cruft inserted
> > between the repetition of 1M of SP gets longer by one byte but they
> > all share the same prefix (e.g. "1S", "1Sp", "1SpZ", "1SpZT",
> > ... are what you insert between a large run of spaces)?
> 
> The pktline packets have a constant size. If the cruft between 1M of SP 
> has a constant size as well then the generated packets for the test data
> would repeat themselves. That's why I increased the length after every 1M
> of SP.
> 
> However, I realized that this test complexity is not necessary. I'll
> simplify it in the next round.

I was also confused by this, and wondered if other patterns (perhaps using a single longer genrandom) might be applicable. Simplification (or explanation in comments about what properties the content _needs_ to have) would be welcome. :)

-Peff
Previous: Lars SchneiderNext: larsxschneider@gmail.com
Message 10 of 66 in “Git filter protocol”
  1. 00/13 Git filter protocollarsxschneider@gmail.com, Aug 25, 2016
  2. 10/13 convert: generate large test files only oncelarsxschneider@gmail.com, Aug 25, 2016
  3. Stefan BellerAug 25, 2016
  4. Lars SchneiderAug 25, 2016
  5. Junio C HamanoAug 29, 2016
  6. Lars SchneiderAug 30, 2016
  7. Junio C HamanoAug 30, 2016
  8. Junio C HamanoAug 29, 2016
  9. Lars SchneiderAug 30, 2016
  10. Jeff KingAug 30, 2016
  11. 06/13 pkt-line: add functions to read/write flush terminated packet streamslarsxschneider@gmail.com, Aug 25, 2016
  12. Stefan BellerAug 25, 2016
  13. Junio C HamanoAug 25, 2016
  14. Jacob KellerAug 26, 2016
  15. Stefan BellerAug 26, 2016
  16. Jeff KingAug 26, 2016
  17. Junio C HamanoAug 26, 2016
  18. Lars SchneiderAug 25, 2016
  19. Junio C HamanoAug 25, 2016
  20. Lars SchneiderAug 26, 2016
  21. Junio C HamanoAug 26, 2016
  22. Lars SchneiderAug 29, 2016
  23. 01/13 pkt-line: rename packet_write() to packet_write_fmt()larsxschneider@gmail.com, Aug 25, 2016
  24. 03/13 pkt-line: add packet_write_fmt_gently()larsxschneider@gmail.com, Aug 25, 2016
  25. Stefan BellerAug 25, 2016
  26. Lars SchneiderAug 25, 2016
  27. Junio C HamanoAug 25, 2016
  28. Lars SchneiderAug 26, 2016
  29. Junio C HamanoAug 26, 2016
  30. Jeff KingAug 26, 2016
  31. 11/13 convert: make apply_filter() adhere to standard Git error handlinglarsxschneider@gmail.com, Aug 25, 2016
  32. 12/13 convert: add filter.<driver>.process optionlarsxschneider@gmail.com, Aug 25, 2016
  33. Junio C HamanoAug 29, 2016
  34. Lars SchneiderAug 30, 2016
  35. Junio C HamanoAug 30, 2016
  36. Lars SchneiderAug 30, 2016
  37. Junio C HamanoAug 30, 2016
  38. Torsten BögershausenAug 31, 2016
  39. Jakub NarębskiAug 31, 2016
  40. Jakub NarębskiAug 30, 2016
  41. Lars SchneiderSep 5, 2016
  42. 05/13 pkt-line: add packet_write_gently()larsxschneider@gmail.com, Aug 25, 2016
  43. Junio C HamanoAug 25, 2016
  44. Lars SchneiderAug 26, 2016
  45. Junio C HamanoAug 26, 2016
  46. Lars SchneiderAug 29, 2016
  47. 13/13 read-cache: make sure file handles are not inherited by child processeslarsxschneider@gmail.com, Aug 25, 2016
  48. Junio C HamanoAug 29, 2016
  49. Lars SchneiderAug 29, 2016
  50. Junio C HamanoAug 29, 2016
  51. Lars SchneiderAug 30, 2016
  52. Torsten BögershausenAug 30, 2016
  53. Junio C HamanoSep 1, 2016
  54. 02/13 pkt-line: extract set_packet_header()larsxschneider@gmail.com, Aug 25, 2016
  55. 09/13 convert: modernize testslarsxschneider@gmail.com, Aug 25, 2016
  56. Junio C HamanoAug 26, 2016
  57. Lars SchneiderAug 29, 2016
  58. 08/13 convert: quote filter names in error messageslarsxschneider@gmail.com, Aug 25, 2016
  59. Junio C HamanoAug 26, 2016
  60. 04/13 pkt-line: add packet_flush_gently()larsxschneider@gmail.com, Aug 25, 2016
  61. 07/13 pack-protocol: fix maximum pkt-line sizelarsxschneider@gmail.com, Aug 25, 2016
  62. Stefan BellerAug 25, 2016
  63. Lars SchneiderAug 25, 2016
  64. Junio C HamanoAug 26, 2016
  65. Lars SchneiderAug 29, 2016
  66. Junio C HamanoAug 29, 2016

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.