git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 3/8] refs/reftable: read references via `struct reftable_backend`

From
Patrick Steinhardt <ps@pks.im>
Date
Nov 12, 2024, 09:05 UTC
Message-ID
<ZzMaY-lqwKIak7Y3@pks.im>
In-Reply-To
<xmqqed3g6hg1.fsf@gitster.g>
On Tue, Nov 12, 2024 at 04:26:38PM +0900, Junio C Hamano wrote:
Show 117 quoted lines
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > Refactor `read_ref_without_reload()` to accept a `struct reftable_stack`
> > as input instead of accepting a `struct reftable_stack`. This allows us
> > to implement an additional caching layer when reading refs where we can
> > reuse reftable iterators.
> >
> > Signed-off-by: Patrick Steinhardt <ps@pks.im>
> > ---
> >  refs/reftable-backend.c   | 110 ++++++++++++++++++++------------------
> >  reftable/reftable-stack.h |   3 ++
> >  reftable/stack.c          |   5 ++
> >  3 files changed, 67 insertions(+), 51 deletions(-)
> >
> > diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
> > index 4a28dc8a9d..230adb690d 100644
> > --- a/refs/reftable-backend.c
> > +++ b/refs/reftable-backend.c
> > @@ -51,6 +51,50 @@ static void reftable_backend_release(struct reftable_backend *be)
> >  	be->stack = NULL;
> >  }
> >  
> > +static int reftable_backend_read_ref(struct reftable_backend *be,
> > +				     const char *refname,
> > +				     struct object_id *oid,
> > +				     struct strbuf *referent,
> > +				     unsigned int *type)
> > +{
> > +	struct reftable_ref_record ref = {0};
> > +	int ret;
> > +
> > +	ret = reftable_stack_read_ref(be->stack, refname, &ref);
> > +	if (ret)
> > +		goto done;
> > +
> > +	if (ref.value_type == REFTABLE_REF_SYMREF) {
> > +		strbuf_reset(referent);
> > +		strbuf_addstr(referent, ref.value.symref);
> > +		*type |= REF_ISSYMREF;
> > +	} else if (reftable_ref_record_val1(&ref)) {
> > +		unsigned int hash_id;
> > +
> > +		switch (reftable_stack_hash_id(be->stack)) {
> 
> So, relative to the original, instead of relying on the repository
> and its knowledge of what hash function is used, we ask the stack
> what hash function is in use and use that instead.
> 
> > +		case REFTABLE_HASH_SHA1:
> > +			hash_id = GIT_HASH_SHA1;
> > +			break;
> > +		case REFTABLE_HASH_SHA256:
> > +			hash_id = GIT_HASH_SHA256;
> > +			break;
> > +		default:
> > +			BUG("unhandled hash ID %d", reftable_stack_hash_id(be->stack));
> > +		}
> > +
> > +		oidread(oid, reftable_ref_record_val1(&ref),
> > +			&hash_algos[hash_id]);
> > +	} else {
> > +		/* We got a tombstone, which should not happen. */
> > +		BUG("unhandled reference value type %d", ref.value_type);
> > +	}
> > +
> > +done:
> > +	assert(ret != REFTABLE_API_ERROR);
> > +	reftable_ref_record_release(&ref);
> > +	return ret;
> > +}
> 
> Here is the original that got replaced.  Since ...
> 
> > -static int read_ref_without_reload(struct reftable_ref_store *refs,
> > -				   struct reftable_stack *stack,
> > -				   const char *refname,
> > -				   struct object_id *oid,
> > -				   struct strbuf *referent,
> > -				   unsigned int *type)
> > -{
> > -	struct reftable_ref_record ref = {0};
> > -	int ret;
> > -
> > -	ret = reftable_stack_read_ref(stack, refname, &ref);
> > -	if (ret)
> > -		goto done;
> > -
> > -	if (ref.value_type == REFTABLE_REF_SYMREF) {
> > -		strbuf_reset(referent);
> > -		strbuf_addstr(referent, ref.value.symref);
> > -		*type |= REF_ISSYMREF;
> > -	} else if (reftable_ref_record_val1(&ref)) {
> > -		oidread(oid, reftable_ref_record_val1(&ref),
> > -			refs->base.repo->hash_algo);
> 
> ... we have access to "refs", which is a ref_store, that knows its
> repository, it was just a few pointer references away to get the
> hash id of the Git side.  But of course we use REFTABLE_HASH_*NAME*
> to identify the algorithm at this layer, so we need to translate it
> back to the ide on the Git side before asking oidread() to read it.
> 
> > -	} else {
> > -		/* We got a tombstone, which should not happen. */
> > -		BUG("unhandled reference value type %d", ref.value_type);
> > -	}
> > -
> > -done:
> > -	assert(ret != REFTABLE_API_ERROR);
> > -	reftable_ref_record_release(&ref);
> > -	return ret;
> > -}
> 
> There is one thing that is curious about this step.
> 
> It isn't like we teach stack what hash it uses in this step---the
> reftable_stack_hash_id() could have been implemented as early as
> 59343984 (reftable/system: stop depending on "hash.h", 2024-11-08).

In theory we could've implemented it even earlier than that: the commit only introduces the reftable-specific hashes, and we had the Git-specific hashes available before that. Like that we wouldn't even have to translate between the different hashes in the first place.

Show 5 quoted lines
> Other than that this step introduces the first caller of
> reftable_stack_hash_id() in the series, the remaining hunks of this
> patch do not have to be part of this patch, but could have been a
> separate step.  Not a suggestion to split it out, but merely an
> observation (to make sure I am reading the code correctly).
Yup, your understanding matches mine.
Patrick
Previous: Junio C HamanoNext: Patrick Steinhardt
Message 26 of 57 in “refs/reftable: reuse iterators when reading refs”
  1. 0/8 refs/reftable: reuse iterators when reading refsPatrick Steinhardt, Nov 4, 2024
  2. 1/8 refs/reftable: encapsulate reftable stackPatrick Steinhardt, Nov 4, 2024
  3. karthik nayakNov 5, 2024
  4. 2/8 refs/reftable: handle reloading stacks in the reftable backendPatrick Steinhardt, Nov 4, 2024
  5. karthik nayakNov 5, 2024
  6. Patrick SteinhardtNov 6, 2024
  7. 3/8 refs/reftable: read references via `struct reftable_backend`Patrick Steinhardt, Nov 4, 2024
  8. karthik nayakNov 5, 2024
  9. 4/8 refs/reftable: refactor reading symbolic refs to use reftable backendPatrick Steinhardt, Nov 4, 2024
  10. 5/8 refs/reftable: refactor reflog expiry to use reftable backendPatrick Steinhardt, Nov 4, 2024
  11. 6/8 reftable/stack: add mechanism to notify callers on reloadPatrick Steinhardt, Nov 4, 2024
  12. 7/8 reftable/merged: drain priority queue on reseekPatrick Steinhardt, Nov 4, 2024
  13. Junio C HamanoNov 5, 2024
  14. Junio C HamanoNov 5, 2024
  15. Patrick SteinhardtNov 5, 2024
  16. 8/8 refs/reftable: reuse iterators when reading refsPatrick Steinhardt, Nov 4, 2024
  17. Junio C HamanoNov 5, 2024
  18. 0/8 refs/reftable: reuse iterators when reading refsPatrick Steinhardt, Nov 5, 2024
  19. 1/8 refs/reftable: encapsulate reftable stackPatrick Steinhardt, Nov 5, 2024
  20. Junio C HamanoNov 12, 2024
  21. 2/8 refs/reftable: handle reloading stacks in the reftable backendPatrick Steinhardt, Nov 5, 2024
  22. Junio C HamanoNov 12, 2024
  23. Patrick SteinhardtNov 12, 2024
  24. 3/8 refs/reftable: read references via `struct reftable_backend`Patrick Steinhardt, Nov 5, 2024
  25. Junio C HamanoNov 12, 2024
  26. Patrick SteinhardtNov 12, 2024
  27. 4/8 refs/reftable: refactor reading symbolic refs to use reftable backendPatrick Steinhardt, Nov 5, 2024
  28. 5/8 refs/reftable: refactor reflog expiry to use reftable backendPatrick Steinhardt, Nov 5, 2024
  29. 6/8 reftable/stack: add mechanism to notify callers on reloadPatrick Steinhardt, Nov 5, 2024
  30. 7/8 reftable/merged: drain priority queue on reseekPatrick Steinhardt, Nov 5, 2024
  31. 8/8 refs/reftable: reuse iterators when reading refsPatrick Steinhardt, Nov 5, 2024
  32. 0/9 refs/reftable: reuse iterators when reading refsPatrick Steinhardt, Nov 25, 2024
  33. 1/9 refs/reftable: encapsulate reftable stackPatrick Steinhardt, Nov 25, 2024
  34. 2/9 refs/reftable: handle reloading stacks in the reftable backendPatrick Steinhardt, Nov 25, 2024
  35. Junio C HamanoNov 26, 2024
  36. 3/9 reftable/stack: add accessor for the hash IDPatrick Steinhardt, Nov 25, 2024
  37. 4/9 refs/reftable: read references via `struct reftable_backend`Patrick Steinhardt, Nov 25, 2024
  38. Junio C HamanoNov 26, 2024
  39. Patrick SteinhardtNov 26, 2024
  40. 5/9 refs/reftable: refactor reading symbolic refs to use reftable backendPatrick Steinhardt, Nov 25, 2024
  41. 6/9 refs/reftable: refactor reflog expiry to use reftable backendPatrick Steinhardt, Nov 25, 2024
  42. 7/9 reftable/stack: add mechanism to notify callers on reloadPatrick Steinhardt, Nov 25, 2024
  43. 8/9 reftable/merged: drain priority queue on reseekPatrick Steinhardt, Nov 25, 2024
  44. 9/9 refs/reftable: reuse iterators when reading refsPatrick Steinhardt, Nov 25, 2024
  45. Christian CouderNov 25, 2024
  46. Patrick SteinhardtNov 25, 2024
  47. 00/10 refs/reftable: reuse iterators when reading refsPatrick Steinhardt, Nov 26, 2024
  48. 01/10 refs/reftable: encapsulate reftable stackPatrick Steinhardt, Nov 26, 2024
  49. 03/10 reftable/stack: add accessor for the hash IDPatrick Steinhardt, Nov 26, 2024
  50. 02/10 refs/reftable: handle reloading stacks in the reftable backendPatrick Steinhardt, Nov 26, 2024
  51. 05/10 refs/reftable: read references via `struct reftable_backend`Patrick Steinhardt, Nov 26, 2024
  52. 04/10 refs/reftable: figure out hash via `reftable_stack`Patrick Steinhardt, Nov 26, 2024
  53. 06/10 refs/reftable: refactor reading symbolic refs to use reftable backendPatrick Steinhardt, Nov 26, 2024
  54. 07/10 refs/reftable: refactor reflog expiry to use reftable backendPatrick Steinhardt, Nov 26, 2024
  55. 08/10 reftable/stack: add mechanism to notify callers on reloadPatrick Steinhardt, Nov 26, 2024
  56. 09/10 reftable/merged: drain priority queue on reseekPatrick Steinhardt, Nov 26, 2024
  57. 10/10 refs/reftable: reuse iterators when reading refsPatrick Steinhardt, Nov 26, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.