git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 2/2] hex: make hash_to_hex_algop() and friends thread-safe

From
Johannes Schindelin <johannes.schindelin@gmx.de>
Date
Jul 16, 2020, 11:29 UTC
Message-ID
<nycvar.QRO.7.76.6.2007161214270.54@tvgsbejvaqbjf.bet>
In-Reply-To
<b47445fa1cef6d4523dd0ca336f7ee22bce89466.1593208411.git.matheus.bernardino@usp.br>
Hi Matheus,
On Fri, 26 Jun 2020, Matheus Tavares wrote:
Show 42 quoted lines
> @@ -136,12 +137,49 @@ char *oid_to_hex_r(char *buffer, const struct object_id *oid)
>  	return hash_to_hex_algop_r(buffer, oid->hash, the_hash_algo);
>  }
>
> +struct hexbuf_array {
> +	int idx;
> +	char bufs[4][GIT_MAX_HEXSZ + 1];
> +};
> +
> +#ifdef HAVE_THREADS
> +static pthread_key_t hexbuf_array_key;
> +static pthread_once_t hexbuf_array_once = PTHREAD_ONCE_INIT;
> +
> +static void init_hexbuf_array_key(void)
> +{
> +	if (pthread_key_create(&hexbuf_array_key, free))
> +		die(_("failed to initialize threads' key for hash to hex conversion"));
> +}
> +
> +#else
> +static struct hexbuf_array default_hexbuf_array;
> +#endif
> +
>  char *hash_to_hex_algop(const unsigned char *hash, const struct git_hash_algo *algop)
>  {
> -	static int bufno;
> -	static char hexbuffer[4][GIT_MAX_HEXSZ + 1];
> -	bufno = (bufno + 1) % ARRAY_SIZE(hexbuffer);
> -	return hash_to_hex_algop_r(hexbuffer[bufno], hash, algop);
> +	struct hexbuf_array *ha;
> +
> +#ifdef HAVE_THREADS
> +	void *value;
> +
> +	if (pthread_once(&hexbuf_array_once, init_hexbuf_array_key))
> +		die(_("failed to initialize threads' key for hash to hex conversion"));
> +
> +	value = pthread_getspecific(hexbuf_array_key);
> +	if (value) {
> +		ha = (struct hexbuf_array *) value;
> +	} else {
> +		ha = xmalloc(sizeof(*ha));

I just realized (while trying to debug something independent) that this leaves `ha->idx` uninitialized. So you will need at least this patch to fix a bug that currently haunts `seen`'s CI builds (you can use `--valgrind`, like I did, to identify the problem):

-- snip --
diff --git a/hex.c b/hex.c
index 4f2f163d5e7..365ba94ab11 100644
--- a/hex.c
+++ b/hex.c
@@ -171,6 +171,7 @@ char *hash_to_hex_algop(const unsigned char *hash, const struct git_hash_algo *a
 		ha = (struct hexbuf_array *) value;
 	} else {
 		ha = xmalloc(sizeof(*ha));
+		ha->idx = 0;
 		if (pthread_setspecific(hexbuf_array_key, (void *)ha))
 			die(_("failed to set thread buffer for hash to hex conversion"));
 	}
-- snap --

But as I mentioned before, I would be much more in favor of abandoning
this thread-local idea (because it is _still_ fragile, as the same thread
could try to make use of more than four hex values in the same `printf()`,
for example) and instead using Coccinelle to convert all those
`oid_to_hex()` calls to `oid_to_hex_r()` calls.

Now, I am _far_ from knowing what I'm doing with Coccinelle, but I think
this here semantic patch should get you going:

-- snipsnap --
@@
expression E;
@@
  {
++   char hex[GIT_MAX_HEXSZ + 1];
     ...
-    oid_to_hex(E)
+    oid_to_hex_r(hex, E)
     ...
  }

@@
expression E1, E2;
@@
  {
++   char hex1[GIT_MAX_HEXSZ + 1], hex2[GIT_MAX_HEXSZ + 1];
     ...
-    oid_to_hex(E1)
+    oid_to_hex_r(hex1, E1)
     ...
-    oid_to_hex(E2)
+    oid_to_hex_r(hex2, E2)
     ...
  }
Previous: Johannes SchindelinNext: Matheus Tavares Bernardino
Message 21 of 25 in “[RFC] Thread safety in some low-level functions”
  1. Matheus Tavares BernardinoJun 24, 2020
  2. Matheus Tavares BernardinoJun 24, 2020
  3. brian m. carlsonJun 25, 2020
  4. 0/2 Make oid_to_hex() thread-safeMatheus Tavares, Jun 25, 2020
  5. 1/2 compat/win32/pthread: add pthread_once()Matheus Tavares, Jun 25, 2020
  6. Christian CouderJun 26, 2020
  7. Matheus Tavares BernardinoJun 26, 2020
  8. 2/2 hex: make hash_to_hex_algop() and friends thread-safeMatheus Tavares, Jun 25, 2020
  9. brian m. carlsonJun 25, 2020
  10. Matheus Tavares BernardinoJun 25, 2020
  11. Matheus Tavares BernardinoJun 26, 2020
  12. Christian CouderJun 26, 2020
  13. Christian CouderJun 26, 2020
  14. Matheus Tavares BernardinoJun 26, 2020
  15. 0/2 Make oid_to_hex() thread-safeMatheus Tavares, Jun 26, 2020
  16. 1/2 compat/win32/pthread: add pthread_once()Matheus Tavares, Jun 26, 2020
  17. 2/2 hex: make hash_to_hex_algop() and friends thread-safeMatheus Tavares, Jun 26, 2020
  18. Johannes SchindelinJun 29, 2020
  19. Matheus Tavares BernardinoJun 30, 2020
  20. Johannes SchindelinJul 1, 2020
  21. Johannes SchindelinJul 16, 2020
  22. Matheus Tavares BernardinoJul 18, 2020
  23. Johannes SchindelinAug 10, 2020
  24. Matheus TavaresJul 18, 2020
  25. René ScharfeJul 26, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.