git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 2/5] t/helper/test-hashmap.c: avoid using `strtok()`

From
Taylor Blau <me@ttaylorr.com>
Date
Apr 18, 2023, 18:06 UTC
Message-ID
<ZD7cGx2zKQ2KYnLS@nand.local>
In-Reply-To
<20230418102320.GB508219@coredump.intra.peff.net>
On Tue, Apr 18, 2023 at 06:23:20AM -0400, Jeff King wrote:
Show 19 quoted lines
> On Thu, Apr 13, 2023 at 07:31:46PM -0400, Taylor Blau wrote:
>
> > Avoid using the non-reentrant `strtok()` to separate the parts of each
> > incoming command. Instead of replacing it with `strtok_r()`, let's
> > instead use the more friendly `string_list_split_in_place_multi()`.
>
> Junio mentioned this offhand in his response, but I wanted to highlight
> one difference in before/after behavior here.
>
>   [before]
>   $ printf 'add foo    bar\niterate\n' | t/helper/test-tool hashmap
>   foo bar
>
>   [after]
>   $ printf 'add foo    bar\niterate\n' | t/helper/test-tool hashmap
>   foo    bar
>
> I think that's fine for this test script, but it may be an indication
> that we want string-list's split to support different semantics.

I agree that it's OK for the test scripts, but probably isn't the right thing for all cases. In the reroll I'll send shortly, I reimplemented this in a way that (a) doesn't change the behavior of `string_list_split_in_place()`, and (b) allows us to easily change the semantics for `string_list_split_in_place_multi()`.

Show 37 quoted lines
> > @@ -159,21 +161,34 @@ int cmd__hashmap(int argc, const char **argv)
> >
> >  	/* process commands from stdin */
> >  	while (strbuf_getline(&line, stdin) != EOF) {
> > -		char *cmd, *p1 = NULL, *p2 = NULL;
> > +		char *cmd, *p1, *p2;
> >  		unsigned int hash = 0;
> >  		struct test_entry *entry;
> >
> > +		/*
> > +		 * Because we memdup() the arguments out of the
> > +		 * string_list before inserting them into the hashmap,
> > +		 * it's OK to set its length back to zero to avoid
> > +		 * re-allocating the items array once per line.
> > +		 *
> > +		 * By doing so, we'll instead overwrite the existing
> > +		 * entries and avoid re-allocating.
> > +		 */
> > +		parts.nr = 0;
>
> I think this is OK, but I wonder if we should wrap it in a function that
> makes sure the strings aren't owned by the strdup (and thus aren't being
> leaked). Something like:
>
>   void string_list_setlen(struct string_list *sl, size_t nr)
>   {
> 	/* alternatively, I guess we could actually free nr..sl->nr */
> 	if (sl->strdup_strings)
> 		BUG("you can't setlen on a string-list which owns its strings");
> 	if (nr > sl->nr)
> 		BUG("you can't grow a string-list with setlen");
> 	sl->nr = nr;
>   }
>
> That is probably overkill for these two test helpers, but I wonder if
> the pattern might escape into "real" code (especially if we suggest
> using string-list instead of strtok).

I like this suggestion. I added a new patch (for which you are listed under Co-authored-by) which adds this function as-is.

Thanks, Taylor

Previous: Jeff KingNext: Taylor Blau
Message 7 of 47 in “banned: mark `strok()`, `strtok_r()` as banned”
  1. 0/5 banned: mark `strok()`, `strtok_r()` as bannedTaylor Blau, Apr 13, 2023
  2. 1/5 string-list: introduce `string_list_split_in_place_multi()`Taylor Blau, Apr 13, 2023
  3. Jeff KingApr 18, 2023
  4. Taylor BlauApr 18, 2023
  5. 2/5 t/helper/test-hashmap.c: avoid using `strtok()`Taylor Blau, Apr 13, 2023
  6. Jeff KingApr 18, 2023
  7. Taylor BlauApr 18, 2023
  8. 3/5 t/helper/test-oidmap.c: avoid using `strtok()`Taylor Blau, Apr 13, 2023
  9. 4/5 t/helper/test-json-writer.c: avoid using `strtok()`Taylor Blau, Apr 13, 2023
  10. 5/5 banned.h: mark `strtok()`, `strtok_r()` as bannedTaylor Blau, Apr 13, 2023
  11. Junio C HamanoApr 14, 2023
  12. Chris TorekApr 14, 2023
  13. Taylor BlauApr 14, 2023
  14. 0/6 banned: mark `strok()` as bannedTaylor Blau, Apr 18, 2023
  15. 1/6 string-list: introduce `string_list_split_in_place_multi()`Taylor Blau, Apr 18, 2023
  16. Junio C HamanoApr 18, 2023
  17. Taylor BlauApr 18, 2023
  18. Jeff KingApr 22, 2023
  19. René ScharfeApr 22, 2023
  20. Jeff KingApr 23, 2023
  21. Junio C HamanoApr 24, 2023
  22. Taylor BlauApr 23, 2023
  23. Taylor BlauApr 23, 2023
  24. 2/6 string-list: introduce `string_list_setlen()`Taylor Blau, Apr 18, 2023
  25. Jeff KingApr 22, 2023
  26. 3/6 t/helper/test-hashmap.c: avoid using `strtok()`Taylor Blau, Apr 18, 2023
  27. Jeff KingApr 22, 2023
  28. Taylor BlauApr 24, 2023
  29. 4/6 t/helper/test-oidmap.c: avoid using `strtok()`Taylor Blau, Apr 18, 2023
  30. 5/6 t/helper/test-json-writer.c: avoid using `strtok()`Taylor Blau, Apr 18, 2023
  31. 6/6 banned.h: mark `strtok()` as bannedTaylor Blau, Apr 18, 2023
  32. 0/6 banned: mark `strok()`, `strtok_r()` as bannedTaylor Blau, Apr 24, 2023
  33. 1/6 string-list: multi-delimiter `string_list_split_in_place()`Taylor Blau, Apr 24, 2023
  34. 2/6 string-list: introduce `string_list_setlen()`Taylor Blau, Apr 24, 2023
  35. Jeff KingApr 25, 2023
  36. Taylor BlauApr 25, 2023
  37. 3/6 t/helper/test-hashmap.c: avoid using `strtok()`Taylor Blau, Apr 24, 2023
  38. 4/6 t/helper/test-oidmap.c: avoid using `strtok()`Taylor Blau, Apr 24, 2023
  39. 5/6 t/helper/test-json-writer.c: avoid using `strtok()`Taylor Blau, Apr 24, 2023
  40. Jeff HostetlerApr 25, 2023
  41. 6/6 banned.h: mark `strtok()` and `strtok_r()` as bannedTaylor Blau, Apr 24, 2023
  42. Chris TorekApr 24, 2023
  43. Taylor BlauApr 24, 2023
  44. Jeff KingApr 25, 2023
  45. Taylor BlauApr 25, 2023
  46. Jeff KingApr 25, 2023
  47. Taylor BlauApr 25, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.