git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 20/21] refs: add LMDB refs backend

From
David Turner <dturner@twopensource.com>
Date
Jan 14, 2016, 22:30 UTC
Message-ID
<1452810655.16226.21.camel@twopensource.com>
In-Reply-To
<20160114205259.GA10440@sigill.intra.peff.net>
On Thu, 2016-01-14 at 15:52 -0500, Jeff King wrote:
Show 61 quoted lines
> On Mon, Jan 11, 2016 at 08:22:19PM -0500, David Turner wrote:
> 
> > +static int rename_reflog_ent(unsigned char *osha1, unsigned char
> > *nsha1,
> > +			     const char *email, unsigned long
> > timestamp, int tz,
> > +			     const char *message, void *cb_data)
> > +{
> > +
> > +	const char *newrefname = cb_data;
> > +	MDB_val key, new_key, val;
> > +
> > +	assert(transaction.cursor);
> > +
> > +	if (mdb_cursor_get_or_die(transaction.cursor, &key, &val,
> > MDB_GET_CURRENT))
> > +		die("renaming ref: mdb_cursor_get failed to get
> > current");
> > +
> > +	new_key.mv_size = strlen(newrefname) + 5 + 1 + 8;
> > +	new_key.mv_data = xmalloc(new_key.mv_size);
> > +	strcpy(new_key.mv_data, "logs/");
> > +	strcpy((char *)new_key.mv_data + 5, newrefname);
> > +	memcpy((char *)new_key.mv_data + new_key.mv_size - 8,
> > +	       (const char *)key.mv_data + key.mv_size - 8, 8);
> > +	mdb_put_or_die(&transaction, &new_key, &val, 0);
> > +	mdb_cursor_del_or_die(transaction.cursor, 0);
> > +	free(new_key.mv_data);
> > +	return 0;
> 
> When you re-roll, do you mind avoiding strcpy here? I know that your
> malloc is big enough, but:
> 
>   1. Avoiding strcpy makes auditing easier.
> 
>   2. We can probably come up with a solution that avoids the magic
>      numbers, making it more pleasant to read.
> 
>   3. Manual computation plus a strcpy can be vulnerable to integer
>      overflows in the size (I didn't check the types on MDB_val to
> see
>      if that is feasible or not, but again, it's nice to avoid for
> audit
>      purposes).
> 
> Since we free the memory immediately-ish, I think using a strbuf
> would
> be a good fit. Something like:
> 
>   struct strbuf path = STRBUF_INIT;
>   ...
>   strbuf_addf(&path, "logs/%s", newrefname);
>   strbuf_add(&path, (const char *)key.mv_data + key.mv_size - 8, 8);
>   new_key.mv_size = path.len;
>   new_key.mv_data = path.buf;
>   ... mdb_put, etc ...
>   strbuf_release(&path);
> 
> (I hope I'm reading the 8-byte thing right; should we also be
> asserting
> that key.mv_size >= 8?).
Did both.
Show 23 quoted lines
> > +static int lmdb_for_each_reflog_ent_order(const char *refname,
> > +					  each_reflog_ent_fn fn,
> > +					  void *cb_data, int
> > reverse)
> > +{
> > +	MDB_val key, val;
> > +	char *search_key;
> > +	char *log_path;
> > +	int len;
> > +	MDB_cursor *cursor;
> > +	int ret = 0;
> > +	struct strbuf sb = STRBUF_INIT;
> > +	enum MDB_cursor_op direction = reverse ? MDB_PREV :
> > MDB_NEXT;
> > +	uint64_t zero = 0ULL;
> > +
> > +	len = strlen(refname) + 6;
> > +	log_path = xmalloc(len);
> > +	search_key = xmalloc(len + 1);
> > +	sprintf(log_path, "logs/%s", refname);
> > +	strcpy(search_key, log_path);
> 
> Ditto here (and for sprintf, too). You can do these with xstrfmt:
OK.
Show 21 quoted lines
>   log_path = xstrfmt("logs/%s", refname);
>   len = strlen(log_path); /* or use a strbuf to avoid the extra
> strlen */
> 
> The search_key one looks like an extra off-by-one, but the extra byte
> gets used below. So maybe:
> 
>   /* \0 may be rewritten as \1 for reverse search below */
>   search_key = xstrfmt("%s\0", log_path);
> 
> though I think:
> 
>   if (reverse) {
> 	/* explanation ... */
> 	search_key = xstrfmt("%s\1", log_path);
>   } else {
> 	search_key = xstrdup(log_path);
>   }
> 
> might be clearer to a reader. There are a few other sprintfs and
> strcpys, but I think they can all use similar techniques.
OK, I went ahead and did all of these.
Thanks for the review.
Previous: Jeff KingNext: David Turner
Message 30 of 35 in “refs backend reroll”
  1. 00/21 refs backend rerollDavid Turner, Jan 12, 2016
  2. 01/21 refs: add a backend method structure with transaction functionsDavid Turner, Jan 12, 2016
  3. 02/21 refs: add methods for misc ref operationsDavid Turner, Jan 12, 2016
  4. 03/21 refs: add methods for the ref iteratorsDavid Turner, Jan 12, 2016
  5. 04/21 refs: add do_for_each_per_worktree_refDavid Turner, Jan 12, 2016
  6. 05/21 refs: add methods for reflogDavid Turner, Jan 12, 2016
  7. 06/21 refs: add method for initial ref transaction commitDavid Turner, Jan 12, 2016
  8. 07/21 refs: add method for delete_refsDavid Turner, Jan 12, 2016
  9. 08/21 refs: add methods to init refs dbDavid Turner, Jan 12, 2016
  10. 09/21 refs: add method to rename refsDavid Turner, Jan 12, 2016
  11. 10/21 refs: make lock genericDavid Turner, Jan 12, 2016
  12. 11/21 refs: move duplicate check to common codeDavid Turner, Jan 12, 2016
  13. 12/21 refs: allow log-only updatesDavid Turner, Jan 12, 2016
  14. 13/21 refs: resolve symbolic refs firstDavid Turner, Jan 12, 2016
  15. Junio C HamanoJan 13, 2016
  16. David TurnerJan 13, 2016
  17. 14/21 refs: always handle non-normal refs in files backendDavid Turner, Jan 12, 2016
  18. Junio C HamanoJan 13, 2016
  19. David TurnerJan 13, 2016
  20. 15/21 init: allow alternate backends to be set for new reposDavid Turner, Jan 12, 2016
  21. 16/21 refs: check submodules ref storage configDavid Turner, Jan 12, 2016
  22. Junio C HamanoJan 13, 2016
  23. David TurnerJan 13, 2016
  24. Junio C HamanoJan 13, 2016
  25. 17/21 clone: use child_process for recursive checkoutsDavid Turner, Jan 12, 2016
  26. 18/21 refs: allow ref backend to be set for cloneDavid Turner, Jan 12, 2016
  27. 19/21 svn: learn ref-storage argumentDavid Turner, Jan 12, 2016
  28. 20/21 refs: add LMDB refs backendDavid Turner, Jan 12, 2016
  29. Jeff KingJan 14, 2016
  30. David TurnerJan 14, 2016
  31. 21/21 refs: tests for lmdb backendDavid Turner, Jan 12, 2016
  32. Junio C HamanoJan 13, 2016
  33. Jeff KingJan 14, 2016
  34. David TurnerJan 14, 2016
  35. Jeff KingJan 14, 2016

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.