git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 09/13] reftable/generic: move seeking of records into the iterator

From
Justin Tobler <jltobler@gmail.com>
Date
May 10, 2024, 21:44 UTC
Message-ID
<4pyzm53ioaqt5men72ti4ffu7zjbpigytfgcyg4h2q7657zoji@7hlrvlt2cws2>
In-Reply-To
<859b399e71e92dd7aa0fa7fc9f2d9ca6f389d34a.1715166175.git.ps@pks.im>
On 24/05/08 01:04PM, Patrick Steinhardt wrote:
Show 16 quoted lines
> Reftable iterators are created by seeking on the parent structure of a
> corresponding record. For example, to create an iterator for the merged
> table you would call `reftable_merged_table_seek_ref()`. Most notably,
> it is not posible to create an iterator and then seek it afterwards.
> 
> While this may be a bit easier to reason about, it comes with two
> significant downsides. The first downside is that the logic to find
> records is split up between the parent data structure and the iterator
> itself. Conceptually, it is more straight forward if all that logic was
> contained in a single place, which should be the iterator.
> 
> The second and more significant downside is that it is impossible to
> reuse iterators for multiple seeks. Whenever you want to look up a
> record, you need to re-create the whole infrastructure again, which is
> quite a waste of time. Furthermore, it is impossible to for example
> optimize seeks, for example when seeking the same record multiple times.
The last setence could use some rewording.

"Furthermore, it is impossible to optimize seeks, such as when seeking the same record multiple times."

Show 29 quoted lines
> 
> To address this, we essentially split up the concerns properly such that
> the parent data structure is responsible for setting up the iterator via
> a new `init_iter()` callback, whereas the iterator handles seeks via a
> new `seek()` callback. This will eventually allow us to call `seek()` on
> the iterator multiple times, where every iterator can potentially
> optimize for certain cases.
> 
> Note that at this point in time we are not yet ready to reuse the
> iterators. This will be left for a future patch series.
> 
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  reftable/generic.c |  47 +++++++++++++----
>  reftable/generic.h |   9 +++-
>  reftable/iter.c    |  15 ++++++
>  reftable/merged.c  |  97 +++++++++++++++++-----------------
>  reftable/reader.c  | 126 +++++++++++++++++++++++++--------------------
>  5 files changed, 177 insertions(+), 117 deletions(-)
> 
> diff --git a/reftable/generic.c b/reftable/generic.c
> index b9f1c7c18a..1cf68fe124 100644
> --- a/reftable/generic.c
> +++ b/reftable/generic.c
> @@ -12,25 +12,39 @@ license that can be found in the LICENSE file or at
>  #include "reftable-iterator.h"
>  #include "reftable-generic.h"
>  
> +void table_init_iter(struct reftable_table *tab,

The following table related functions are prefixed with `reftable_`. Do we want to do the same here?

Show 46 quoted lines
> +		     struct reftable_iterator *it,
> +		     uint8_t typ)
> +{
> +
> +	tab->ops->init_iter(tab->table_arg, it, typ);
> +}
> +
>  int reftable_table_seek_ref(struct reftable_table *tab,
>  			    struct reftable_iterator *it, const char *name)
>  {
> -	struct reftable_record rec = { .type = BLOCK_TYPE_REF,
> -				       .u.ref = {
> -					       .refname = (char *)name,
> -				       } };
> -	return tab->ops->seek_record(tab->table_arg, it, &rec);
> +	struct reftable_record want = {
> +		.type = BLOCK_TYPE_REF,
> +		.u.ref = {
> +			.refname = (char *)name,
> +		},
> +	};
> +	table_init_iter(tab, it, BLOCK_TYPE_REF);
> +	return it->ops->seek(it->iter_arg, &want);
>  }
>  
>  int reftable_table_seek_log(struct reftable_table *tab,
>  			    struct reftable_iterator *it, const char *name)
>  {
> -	struct reftable_record rec = { .type = BLOCK_TYPE_LOG,
> -				       .u.log = {
> -					       .refname = (char *)name,
> -					       .update_index = ~((uint64_t)0),
> -				       } };
> -	return tab->ops->seek_record(tab->table_arg, it, &rec);
> +	struct reftable_record want = {
> +		.type = BLOCK_TYPE_LOG,
> +		.u.log = {
> +			.refname = (char *)name,
> +			.update_index = ~((uint64_t)0),
> +		},
> +	};
> +	table_init_iter(tab, it, BLOCK_TYPE_LOG);
> +	return it->ops->seek(it->iter_arg, &want);
>  }
>  
>  int reftable_table_read_ref(struct reftable_table *tab, const char *name,
...
Show 10 quoted lines
> @@ -23,6 +23,13 @@ static void filtering_ref_iterator_close(void *iter_arg)
>  	reftable_iterator_destroy(&fri->it);
>  }
>  
> +static int filtering_ref_iterator_seek(void *iter_arg,
> +				       struct reftable_record *want)
> +{
> +	struct filtering_ref_iterator *fri = iter_arg;
> +	return iterator_seek(&fri->it, want);
> +}

I've found the `filtering_ref_iterator_seek()` here to be a little confusing. At first, I assumed that the `filtering_ref_iterator` would have referenced `filtering_ref_iterator_vtable` thus resulting in a cycle, but on closer inspection this does not seem to be the case and is in face always set to some other iterator operation.

Am I understanding this correctly?
-Justin
Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 15 of 50 in “reftable: prepare for re-seekable iterators”
  1. 00/13 reftable: prepare for re-seekable iteratorsPatrick Steinhardt, May 8, 2024
  2. 01/13 reftable/block: use `size_t` to track restart point indexPatrick Steinhardt, May 8, 2024
  3. 02/13 reftable/reader: avoid copying index iteratorPatrick Steinhardt, May 8, 2024
  4. 03/13 reftable/reader: unify indexed and linear seekingPatrick Steinhardt, May 8, 2024
  5. 04/13 reftable/reader: separate concerns of table iter and reftable readerPatrick Steinhardt, May 8, 2024
  6. 05/13 reftable/reader: inline `reader_seek_internal()`Patrick Steinhardt, May 8, 2024
  7. 06/13 reftable/reader: set up the reader when initializing table iteratorPatrick Steinhardt, May 8, 2024
  8. 07/13 reftable/merged: split up initialization and seeking of recordsPatrick Steinhardt, May 8, 2024
  9. Justin ToblerMay 10, 2024
  10. Patrick SteinhardtMay 13, 2024
  11. 08/13 reftable/merged: simplify indices for subiteratorsPatrick Steinhardt, May 8, 2024
  12. Justin ToblerMay 10, 2024
  13. Patrick SteinhardtMay 13, 2024
  14. 09/13 reftable/generic: move seeking of records into the iteratorPatrick Steinhardt, May 8, 2024
  15. Justin ToblerMay 10, 2024
  16. Patrick SteinhardtMay 13, 2024
  17. 10/13 reftable/generic: adapt interface to allow reuse of iteratorsPatrick Steinhardt, May 8, 2024
  18. 11/13 reftable/reader: adapt interface to allow reuse of iteratorsPatrick Steinhardt, May 8, 2024
  19. Justin ToblerMay 10, 2024
  20. Patrick SteinhardtMay 13, 2024
  21. 12/13 reftable/stack: provide convenience functions to create iteratorsPatrick Steinhardt, May 8, 2024
  22. 13/13 reftable/merged: adapt interface to allow reuse of iteratorsPatrick Steinhardt, May 8, 2024
  23. Junio C HamanoMay 8, 2024
  24. Junio C HamanoMay 9, 2024
  25. Patrick SteinhardtMay 10, 2024
  26. Junio C HamanoMay 10, 2024
  27. Patrick SteinhardtMay 10, 2024
  28. Junio C HamanoMay 10, 2024
  29. 00/13 reftable: prepare for re-seekable iteratorsPatrick Steinhardt, May 13, 2024
  30. 01/13 reftable/block: use `size_t` to track restart point indexPatrick Steinhardt, May 13, 2024
  31. Karthik NayakMay 21, 2024
  32. Patrick SteinhardtMay 22, 2024
  33. 02/13 reftable/reader: avoid copying index iteratorPatrick Steinhardt, May 13, 2024
  34. 03/13 reftable/reader: unify indexed and linear seekingPatrick Steinhardt, May 13, 2024
  35. Karthik NayakMay 21, 2024
  36. Patrick SteinhardtMay 22, 2024
  37. Karthik NayakMay 22, 2024
  38. 04/13 reftable/reader: separate concerns of table iter and reftable readerPatrick Steinhardt, May 13, 2024
  39. 05/13 reftable/reader: inline `reader_seek_internal()`Patrick Steinhardt, May 13, 2024
  40. 06/13 reftable/reader: set up the reader when initializing table iteratorPatrick Steinhardt, May 13, 2024
  41. 07/13 reftable/merged: split up initialization and seeking of recordsPatrick Steinhardt, May 13, 2024
  42. 08/13 reftable/merged: simplify indices for subiteratorsPatrick Steinhardt, May 13, 2024
  43. 09/13 reftable/generic: move seeking of records into the iteratorPatrick Steinhardt, May 13, 2024
  44. 10/13 reftable/generic: adapt interface to allow reuse of iteratorsPatrick Steinhardt, May 13, 2024
  45. 11/13 reftable/reader: adapt interface to allow reuse of iteratorsPatrick Steinhardt, May 13, 2024
  46. 12/13 reftable/stack: provide convenience functions to create iteratorsPatrick Steinhardt, May 13, 2024
  47. 13/13 reftable/merged: adapt interface to allow reuse of iteratorsPatrick Steinhardt, May 13, 2024
  48. Justin ToblerMay 15, 2024
  49. Karthik NayakMay 21, 2024
  50. Patrick SteinhardtMay 22, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.