{"thread":{"id":"62444","subject":"[PATCH 0/8] refs/reftable: reuse iterators when reading refs","startedAt":"2024-11-04T15:11:44Z","lastAt":"2024-11-26T06:43:18Z","messageCount":57,"participants":["Patrick Steinhardt","Junio C Hamano","karthik nayak","Christian Couder"],"isPatch":true,"patchVersion":1,"patchTotal":8},"messages":[{"id":"506532","messageId":"cover.1730732881.git.ps@pks.im","threadId":"62444","inReplyTo":null,"subject":"[PATCH 0/8] refs/reftable: reuse iterators when reading refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-04T15:11:32Z","receivedAt":"2024-11-04T15:11:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series refactors the reftable backend to reuse reftable\niterators when reading random references. This removes the overhead of\nhaving to recreate the iterator on every read and thus leads to better\nperformance and less allocation churn. It also gives us the ability to\nfurther optimize reads by optimizing re-seeking iterators in the future.\n\nOverall this leads to a 7% speedup when creating many refs in a\ntransaction, which performs many random reads. But this change also\npositively impacts other usecases.\n\nThanks!\n\nPatrick\n\nPatrick Steinhardt (8):\n  refs/reftable: encapsulate reftable stack\n  refs/reftable: handle reloading stacks in the reftable backend\n  refs/reftable: read references via `struct reftable_backend`\n  refs/reftable: refactor reading symbolic refs to use reftable backend\n  refs/reftable: refactor reflog expiry to use reftable backend\n  reftable/stack: add mechanism to notify callers on reload\n  reftable/merged: drain priority queue on reseek\n  refs/reftable: reuse iterators when reading refs\n\n refs/reftable-backend.c          | 356 ++++++++++++++++++-------------\n reftable/merged.c                |   2 +\n reftable/reftable-stack.h        |   3 +\n reftable/reftable-writer.h       |   9 +\n reftable/stack.c                 |   9 +\n t/unit-tests/t-reftable-merged.c |  73 +++++++\n 6 files changed, 308 insertions(+), 144 deletions(-)\n\n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506533","messageId":"b599bcdac191fd3cfb7797accebd79211e757c47.1730732881.git.ps@pks.im","threadId":"62444","inReplyTo":"cover.1730732881.git.ps@pks.im","subject":"[PATCH 1/8] refs/reftable: encapsulate reftable stack","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-04T15:11:35Z","receivedAt":"2024-11-04T15:11:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable ref store needs to keep track of multiple stacks, one for\nthe main worktree and an arbitrary number of stacks for worktrees. This\nis done by storing pointers to `struct reftable_stack`, which we then\naccess directly.\n\nWrap the stack in a new `struct reftable_backend`. This will allow us to\nattach more data to each respective stack in subsequent commits.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 129 +++++++++++++++++++++++-----------------\n 1 file changed, 73 insertions(+), 56 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 38eb14d591..43cba53cb1 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -32,6 +32,23 @@\n  */\n #define REF_UPDATE_VIA_HEAD (1 << 8)\n \n+struct reftable_backend {\n+\tstruct reftable_stack *stack;\n+};\n+\n+static int reftable_backend_init(struct reftable_backend *be,\n+\t\t\t\t const char *path,\n+\t\t\t\t const struct reftable_write_options *opts)\n+{\n+\treturn reftable_new_stack(&be->stack, path, opts);\n+}\n+\n+static void reftable_backend_release(struct reftable_backend *be)\n+{\n+\treftable_stack_destroy(be->stack);\n+\tbe->stack = NULL;\n+}\n+\n struct reftable_ref_store {\n \tstruct ref_store base;\n \n@@ -39,17 +56,17 @@ struct reftable_ref_store {\n \t * The main stack refers to the common dir and thus contains common\n \t * refs as well as refs of the main repository.\n \t */\n-\tstruct reftable_stack *main_stack;\n+\tstruct reftable_backend main_backend;\n \t/*\n \t * The worktree stack refers to the gitdir in case the refdb is opened\n \t * via a worktree. It thus contains the per-worktree refs.\n \t */\n-\tstruct reftable_stack *worktree_stack;\n+\tstruct reftable_backend worktree_backend;\n \t/*\n \t * Map of worktree stacks by their respective worktree names. The map\n \t * is populated lazily when we try to resolve `worktrees/$worktree` refs.\n \t */\n-\tstruct strmap worktree_stacks;\n+\tstruct strmap worktree_backends;\n \tstruct reftable_write_options write_options;\n \n \tunsigned int store_flags;\n@@ -95,21 +112,21 @@ static struct reftable_ref_store *reftable_be_downcast(struct ref_store *ref_sto\n  * like `worktrees/$worktree/refs/heads/foo` as worktree stacks will store\n  * those references in their normalized form.\n  */\n-static struct reftable_stack *stack_for(struct reftable_ref_store *store,\n-\t\t\t\t\tconst char *refname,\n-\t\t\t\t\tconst char **rewritten_ref)\n+static struct reftable_backend *backend_for(struct reftable_ref_store *store,\n+\t\t\t\t\t    const char *refname,\n+\t\t\t\t\t    const char **rewritten_ref)\n {\n \tconst char *wtname;\n \tint wtname_len;\n \n \tif (!refname)\n-\t\treturn store->main_stack;\n+\t\treturn &store->main_backend;\n \n \tswitch (parse_worktree_ref(refname, &wtname, &wtname_len, rewritten_ref)) {\n \tcase REF_WORKTREE_OTHER: {\n \t\tstatic struct strbuf wtname_buf = STRBUF_INIT;\n \t\tstruct strbuf wt_dir = STRBUF_INIT;\n-\t\tstruct reftable_stack *stack;\n+\t\tstruct reftable_backend *be;\n \n \t\t/*\n \t\t * We're using a static buffer here so that we don't need to\n@@ -123,37 +140,39 @@ static struct reftable_stack *stack_for(struct reftable_ref_store *store,\n \t\t/*\n \t\t * There is an edge case here: when the worktree references the\n \t\t * current worktree, then we set up the stack once via\n-\t\t * `worktree_stacks` and once via `worktree_stack`. This is\n+\t\t * `worktree_backends` and once via `worktree_backend`. This is\n \t\t * wasteful, but in the reading case it shouldn't matter. And\n \t\t * in the writing case we would notice that the stack is locked\n \t\t * already and error out when trying to write a reference via\n \t\t * both stacks.\n \t\t */\n-\t\tstack = strmap_get(&store->worktree_stacks, wtname_buf.buf);\n-\t\tif (!stack) {\n+\t\tbe = strmap_get(&store->worktree_backends, wtname_buf.buf);\n+\t\tif (!be) {\n \t\t\tstrbuf_addf(&wt_dir, \"%s/worktrees/%s/reftable\",\n \t\t\t\t    store->base.repo->commondir, wtname_buf.buf);\n \n-\t\t\tstore->err = reftable_new_stack(&stack, wt_dir.buf,\n-\t\t\t\t\t\t\t&store->write_options);\n+\t\t\tCALLOC_ARRAY(be, 1);\n+\t\t\tstore->err = reftable_backend_init(be, wt_dir.buf,\n+\t\t\t\t\t\t\t   &store->write_options);\n \t\t\tassert(store->err != REFTABLE_API_ERROR);\n-\t\t\tstrmap_put(&store->worktree_stacks, wtname_buf.buf, stack);\n+\n+\t\t\tstrmap_put(&store->worktree_backends, wtname_buf.buf, be);\n \t\t}\n \n \t\tstrbuf_release(&wt_dir);\n-\t\treturn stack;\n+\t\treturn be;\n \t}\n \tcase REF_WORKTREE_CURRENT:\n \t\t/*\n \t\t * If there is no worktree stack then we're currently in the\n \t\t * main worktree. We thus return the main stack in that case.\n \t\t */\n-\t\tif (!store->worktree_stack)\n-\t\t\treturn store->main_stack;\n-\t\treturn store->worktree_stack;\n+\t\tif (!store->worktree_backend.stack)\n+\t\t\treturn &store->main_backend;\n+\t\treturn &store->worktree_backend;\n \tcase REF_WORKTREE_MAIN:\n \tcase REF_WORKTREE_SHARED:\n-\t\treturn store->main_stack;\n+\t\treturn &store->main_backend;\n \tdefault:\n \t\tBUG(\"unhandled worktree reference type\");\n \t}\n@@ -285,7 +304,7 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \tumask(mask);\n \n \tbase_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable);\n-\tstrmap_init(&refs->worktree_stacks);\n+\tstrmap_init(&refs->worktree_backends);\n \trefs->store_flags = store_flags;\n \trefs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);\n \n@@ -320,8 +339,8 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \t\tstrbuf_realpath(&path, gitdir, 0);\n \t}\n \tstrbuf_addstr(&path, \"/reftable\");\n-\trefs->err = reftable_new_stack(&refs->main_stack, path.buf,\n-\t\t\t\t       &refs->write_options);\n+\trefs->err = reftable_backend_init(&refs->main_backend, path.buf,\n+\t\t\t\t\t  &refs->write_options);\n \tif (refs->err)\n \t\tgoto done;\n \n@@ -337,8 +356,8 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \t\tstrbuf_reset(&path);\n \t\tstrbuf_addf(&path, \"%s/reftable\", gitdir);\n \n-\t\trefs->err = reftable_new_stack(&refs->worktree_stack, path.buf,\n-\t\t\t\t\t       &refs->write_options);\n+\t\trefs->err = reftable_backend_init(&refs->worktree_backend, path.buf,\n+\t\t\t\t\t\t  &refs->write_options);\n \t\tif (refs->err)\n \t\t\tgoto done;\n \t}\n@@ -357,19 +376,17 @@ static void reftable_be_release(struct ref_store *ref_store)\n \tstruct strmap_entry *entry;\n \tstruct hashmap_iter iter;\n \n-\tif (refs->main_stack) {\n-\t\treftable_stack_destroy(refs->main_stack);\n-\t\trefs->main_stack = NULL;\n-\t}\n+\tif (refs->main_backend.stack)\n+\t\treftable_backend_release(&refs->main_backend);\n+\tif (refs->worktree_backend.stack)\n+\t\treftable_backend_release(&refs->worktree_backend);\n \n-\tif (refs->worktree_stack) {\n-\t\treftable_stack_destroy(refs->worktree_stack);\n-\t\trefs->worktree_stack = NULL;\n+\tstrmap_for_each_entry(&refs->worktree_backends, &iter, entry) {\n+\t\tstruct reftable_backend *be = entry->value;\n+\t\treftable_backend_release(be);\n+\t\tfree(be);\n \t}\n-\n-\tstrmap_for_each_entry(&refs->worktree_stacks, &iter, entry)\n-\t\treftable_stack_destroy(entry->value);\n-\tstrmap_clear(&refs->worktree_stacks, 0);\n+\tstrmap_clear(&refs->worktree_backends, 0);\n }\n \n static int reftable_be_create_on_disk(struct ref_store *ref_store,\n@@ -764,7 +781,7 @@ static struct ref_iterator *reftable_be_iterator_begin(struct ref_store *ref_sto\n \t\trequired_flags |= REF_STORE_ODB;\n \trefs = reftable_be_downcast(ref_store, required_flags, \"ref_iterator_begin\");\n \n-\tmain_iter = ref_iterator_for_stack(refs, refs->main_stack, prefix,\n+\tmain_iter = ref_iterator_for_stack(refs, refs->main_backend.stack, prefix,\n \t\t\t\t\t   exclude_patterns, flags);\n \n \t/*\n@@ -772,14 +789,14 @@ static struct ref_iterator *reftable_be_iterator_begin(struct ref_store *ref_sto\n \t * right now. If we aren't, then we return the common reftable\n \t * iterator, only.\n \t */\n-\t if (!refs->worktree_stack)\n+\t if (!refs->worktree_backend.stack)\n \t\treturn &main_iter->base;\n \n \t/*\n \t * Otherwise we merge both the common and the per-worktree refs into a\n \t * single iterator.\n \t */\n-\tworktree_iter = ref_iterator_for_stack(refs, refs->worktree_stack, prefix,\n+\tworktree_iter = ref_iterator_for_stack(refs, refs->worktree_backend.stack, prefix,\n \t\t\t\t\t       exclude_patterns, flags);\n \treturn merge_ref_iterator_begin(&worktree_iter->base, &main_iter->base,\n \t\t\t\t\tref_iterator_select, NULL);\n@@ -794,7 +811,7 @@ static int reftable_be_read_raw_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_raw_ref\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tint ret;\n \n \tif (refs->err < 0)\n@@ -821,7 +838,7 @@ static int reftable_be_read_symbolic_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_symbolic_ref\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_ref_record ref = {0};\n \tint ret;\n \n@@ -881,7 +898,7 @@ static int prepare_transaction_update(struct write_transaction_table_arg **out,\n \t\t\t\t      struct ref_update *update,\n \t\t\t\t      struct strbuf *err)\n {\n-\tstruct reftable_stack *stack = stack_for(refs, update->refname, NULL);\n+\tstruct reftable_stack *stack = backend_for(refs, update->refname, NULL)->stack;\n \tstruct write_transaction_table_arg *arg = NULL;\n \tsize_t i;\n \tint ret;\n@@ -1014,7 +1031,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\tgoto done;\n \t}\n \n-\tret = read_ref_without_reload(refs, stack_for(refs, \"HEAD\", NULL), \"HEAD\",\n+\tret = read_ref_without_reload(refs, backend_for(refs, \"HEAD\", NULL)->stack, \"HEAD\",\n \t\t\t\t      &head_oid, &head_referent, &head_type);\n \tif (ret < 0)\n \t\tgoto done;\n@@ -1026,7 +1043,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\tstruct reftable_stack *stack;\n \t\tconst char *rewritten_ref;\n \n-\t\tstack = stack_for(refs, u->refname, &rewritten_ref);\n+\t\tstack = backend_for(refs, u->refname, &rewritten_ref)->stack;\n \n \t\t/* Verify that the new object ID is valid. */\n \t\tif ((u->flags & REF_HAVE_NEW) && !is_null_oid(&u->new_oid) &&\n@@ -1508,9 +1525,9 @@ static int reftable_be_pack_refs(struct ref_store *ref_store,\n \tif (refs->err)\n \t\treturn refs->err;\n \n-\tstack = refs->worktree_stack;\n+\tstack = refs->worktree_backend.stack;\n \tif (!stack)\n-\t\tstack = refs->main_stack;\n+\t\tstack = refs->main_backend.stack;\n \n \tif (opts->flags & PACK_REFS_AUTO)\n \t\tret = reftable_stack_auto_compact(stack);\n@@ -1765,7 +1782,7 @@ static int reftable_be_rename_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"rename_ref\");\n-\tstruct reftable_stack *stack = stack_for(refs, newrefname, &newrefname);\n+\tstruct reftable_stack *stack = backend_for(refs, newrefname, &newrefname)->stack;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n \t\t.stack = stack,\n@@ -1797,7 +1814,7 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"copy_ref\");\n-\tstruct reftable_stack *stack = stack_for(refs, newrefname, &newrefname);\n+\tstruct reftable_stack *stack = backend_for(refs, newrefname, &newrefname)->stack;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n \t\t.stack = stack,\n@@ -1935,11 +1952,11 @@ static struct ref_iterator *reftable_be_reflog_iterator_begin(struct ref_store *\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_iterator_begin\");\n \tstruct reftable_reflog_iterator *main_iter, *worktree_iter;\n \n-\tmain_iter = reflog_iterator_for_stack(refs, refs->main_stack);\n-\tif (!refs->worktree_stack)\n+\tmain_iter = reflog_iterator_for_stack(refs, refs->main_backend.stack);\n+\tif (!refs->worktree_backend.stack)\n \t\treturn &main_iter->base;\n \n-\tworktree_iter = reflog_iterator_for_stack(refs, refs->worktree_stack);\n+\tworktree_iter = reflog_iterator_for_stack(refs, refs->worktree_backend.stack);\n \n \treturn merge_ref_iterator_begin(&worktree_iter->base, &main_iter->base,\n \t\t\t\t\tref_iterator_select, NULL);\n@@ -1978,7 +1995,7 @@ static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent_reverse\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record log = {0};\n \tstruct reftable_iterator it = {0};\n \tint ret;\n@@ -2018,7 +2035,7 @@ static int reftable_be_for_each_reflog_ent(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record *logs = NULL;\n \tstruct reftable_iterator it = {0};\n \tsize_t logs_alloc = 0, logs_nr = 0, i;\n@@ -2067,7 +2084,7 @@ static int reftable_be_reflog_exists(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_exists\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record log = {0};\n \tstruct reftable_iterator it = {0};\n \tint ret;\n@@ -2152,7 +2169,7 @@ static int reftable_be_create_reflog(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"create_reflog\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct write_reflog_existence_arg arg = {\n \t\t.refs = refs,\n \t\t.stack = stack,\n@@ -2226,7 +2243,7 @@ static int reftable_be_delete_reflog(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"delete_reflog\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct write_reflog_delete_arg arg = {\n \t\t.stack = stack,\n \t\t.refname = refname,\n@@ -2335,7 +2352,7 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \t */\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"reflog_expire\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record *logs = NULL;\n \tstruct reftable_log_record *rewritten = NULL;\n \tstruct reftable_ref_record ref_record = {0};\n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506534","messageId":"b81ce63589e0380baf2b977a6abd706a66b08bee.1730732881.git.ps@pks.im","threadId":"62444","inReplyTo":"cover.1730732881.git.ps@pks.im","subject":"[PATCH 2/8] refs/reftable: handle reloading stacks in the reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-04T15:11:37Z","receivedAt":"2024-11-04T15:11:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When accessing a stack we almost always have to reload the stack before\nreading data from it. This is mostly because Git does not have a\nnotification mechanism for when underlying data has been changed, and\nthus we are forced to opportunistically reload the stack every single\ntime to account for any changes that may have happened concurrently.\n\nHandle the reload internally in `backend_for()`. For one this forces\ncallsites to think about whether or not they need to reload the stack.\nBut second this makes the logic to access stacks more self-contained by\nletting the `struct reftable_backend` manage themselves.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 150 +++++++++++++++++++++++++---------------\n 1 file changed, 93 insertions(+), 57 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 43cba53cb1..93f3602faa 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -112,21 +112,25 @@ static struct reftable_ref_store *reftable_be_downcast(struct ref_store *ref_sto\n  * like `worktrees/$worktree/refs/heads/foo` as worktree stacks will store\n  * those references in their normalized form.\n  */\n-static struct reftable_backend *backend_for(struct reftable_ref_store *store,\n-\t\t\t\t\t    const char *refname,\n-\t\t\t\t\t    const char **rewritten_ref)\n+static int backend_for(struct reftable_backend **out,\n+\t\t       struct reftable_ref_store *store,\n+\t\t       const char *refname,\n+\t\t       const char **rewritten_ref,\n+\t\t       int reload)\n {\n+\tstruct reftable_backend *be;\n \tconst char *wtname;\n \tint wtname_len;\n \n-\tif (!refname)\n-\t\treturn &store->main_backend;\n+\tif (!refname) {\n+\t\tbe = &store->main_backend;\n+\t\tgoto out;\n+\t}\n \n \tswitch (parse_worktree_ref(refname, &wtname, &wtname_len, rewritten_ref)) {\n \tcase REF_WORKTREE_OTHER: {\n \t\tstatic struct strbuf wtname_buf = STRBUF_INIT;\n \t\tstruct strbuf wt_dir = STRBUF_INIT;\n-\t\tstruct reftable_backend *be;\n \n \t\t/*\n \t\t * We're using a static buffer here so that we don't need to\n@@ -160,7 +164,7 @@ static struct reftable_backend *backend_for(struct reftable_ref_store *store,\n \t\t}\n \n \t\tstrbuf_release(&wt_dir);\n-\t\treturn be;\n+\t\tgoto out;\n \t}\n \tcase REF_WORKTREE_CURRENT:\n \t\t/*\n@@ -168,14 +172,27 @@ static struct reftable_backend *backend_for(struct reftable_ref_store *store,\n \t\t * main worktree. We thus return the main stack in that case.\n \t\t */\n \t\tif (!store->worktree_backend.stack)\n-\t\t\treturn &store->main_backend;\n-\t\treturn &store->worktree_backend;\n+\t\t\tbe = &store->main_backend;\n+\t\telse\n+\t\t\tbe = &store->worktree_backend;\n+\t\tgoto out;\n \tcase REF_WORKTREE_MAIN:\n \tcase REF_WORKTREE_SHARED:\n-\t\treturn &store->main_backend;\n+\t\tbe = &store->main_backend;\n+\t\tgoto out;\n \tdefault:\n \t\tBUG(\"unhandled worktree reference type\");\n \t}\n+\n+out:\n+\tif (reload) {\n+\t\tint ret = reftable_stack_reload(be->stack);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n+\t*out = be;\n+\n+\treturn 0;\n }\n \n static int should_write_log(struct reftable_ref_store *refs, const char *refname)\n@@ -811,17 +828,17 @@ static int reftable_be_read_raw_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_raw_ref\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n+\tstruct reftable_backend *be;\n \tint ret;\n \n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret)\n \t\treturn ret;\n \n-\tret = read_ref_without_reload(refs, stack, refname, oid, referent, type);\n+\tret = read_ref_without_reload(refs, be->stack, refname, oid, referent, type);\n \tif (ret < 0)\n \t\treturn ret;\n \tif (ret > 0) {\n@@ -838,15 +855,15 @@ static int reftable_be_read_symbolic_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_symbolic_ref\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_ref_record ref = {0};\n+\tstruct reftable_backend *be;\n \tint ret;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret)\n \t\treturn ret;\n \n-\tret = reftable_stack_read_ref(stack, refname, &ref);\n+\tret = reftable_stack_read_ref(be->stack, refname, &ref);\n \tif (ret == 0 && ref.value_type == REFTABLE_REF_SYMREF)\n \t\tstrbuf_addstr(referent, ref.value.symref);\n \telse\n@@ -863,7 +880,7 @@ struct reftable_transaction_update {\n \n struct write_transaction_table_arg {\n \tstruct reftable_ref_store *refs;\n-\tstruct reftable_stack *stack;\n+\tstruct reftable_backend *be;\n \tstruct reftable_addition *addition;\n \tstruct reftable_transaction_update *updates;\n \tsize_t updates_nr;\n@@ -898,27 +915,31 @@ static int prepare_transaction_update(struct write_transaction_table_arg **out,\n \t\t\t\t      struct ref_update *update,\n \t\t\t\t      struct strbuf *err)\n {\n-\tstruct reftable_stack *stack = backend_for(refs, update->refname, NULL)->stack;\n \tstruct write_transaction_table_arg *arg = NULL;\n+\tstruct reftable_backend *be;\n \tsize_t i;\n \tint ret;\n \n+\tret = backend_for(&be, refs, update->refname, NULL, 0);\n+\tif (ret)\n+\t\treturn ret;\n+\n \t/*\n \t * Search for a preexisting stack update. If there is one then we add\n \t * the update to it, otherwise we set up a new stack update.\n \t */\n \tfor (i = 0; !arg && i < tx_data->args_nr; i++)\n-\t\tif (tx_data->args[i].stack == stack)\n+\t\tif (tx_data->args[i].be == be)\n \t\t\targ = &tx_data->args[i];\n \n \tif (!arg) {\n \t\tstruct reftable_addition *addition;\n \n-\t\tret = reftable_stack_reload(stack);\n+\t\tret = backend_for(&be, refs, update->refname, NULL, 1);\n \t\tif (ret)\n \t\t\treturn ret;\n \n-\t\tret = reftable_stack_new_addition(&addition, stack,\n+\t\tret = reftable_stack_new_addition(&addition, be->stack,\n \t\t\t\t\t\t  REFTABLE_STACK_NEW_ADDITION_RELOAD);\n \t\tif (ret) {\n \t\t\tif (ret == REFTABLE_LOCK_ERROR)\n@@ -930,7 +951,7 @@ static int prepare_transaction_update(struct write_transaction_table_arg **out,\n \t\t\t   tx_data->args_alloc);\n \t\targ = &tx_data->args[tx_data->args_nr++];\n \t\targ->refs = refs;\n-\t\targ->stack = stack;\n+\t\targ->be = be;\n \t\targ->addition = addition;\n \t\targ->updates = NULL;\n \t\targ->updates_nr = 0;\n@@ -985,6 +1006,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \tstruct strbuf referent = STRBUF_INIT, head_referent = STRBUF_INIT;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n \tstruct reftable_transaction_data *tx_data = NULL;\n+\tstruct reftable_backend *be;\n \tstruct object_id head_oid;\n \tunsigned int head_type = 0;\n \tsize_t i;\n@@ -1031,7 +1053,11 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\tgoto done;\n \t}\n \n-\tret = read_ref_without_reload(refs, backend_for(refs, \"HEAD\", NULL)->stack, \"HEAD\",\n+\tret = backend_for(&be, refs, \"HEAD\", NULL, 0);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = read_ref_without_reload(refs, be->stack, \"HEAD\",\n \t\t\t\t      &head_oid, &head_referent, &head_type);\n \tif (ret < 0)\n \t\tgoto done;\n@@ -1040,10 +1066,11 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \tfor (i = 0; i < transaction->nr; i++) {\n \t\tstruct ref_update *u = transaction->updates[i];\n \t\tstruct object_id current_oid = {0};\n-\t\tstruct reftable_stack *stack;\n \t\tconst char *rewritten_ref;\n \n-\t\tstack = backend_for(refs, u->refname, &rewritten_ref)->stack;\n+\t\tret = backend_for(&be, refs, u->refname, &rewritten_ref, 0);\n+\t\tif (ret)\n+\t\t\tgoto done;\n \n \t\t/* Verify that the new object ID is valid. */\n \t\tif ((u->flags & REF_HAVE_NEW) && !is_null_oid(&u->new_oid) &&\n@@ -1099,7 +1126,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t\tstring_list_insert(&affected_refnames, new_update->refname);\n \t\t}\n \n-\t\tret = read_ref_without_reload(refs, stack, rewritten_ref,\n+\t\tret = read_ref_without_reload(refs, be->stack, rewritten_ref,\n \t\t\t\t\t      &current_oid, &referent, &u->type);\n \t\tif (ret < 0)\n \t\t\tgoto done;\n@@ -1301,7 +1328,7 @@ static int transaction_update_cmp(const void *a, const void *b)\n static int write_transaction_table(struct reftable_writer *writer, void *cb_data)\n {\n \tstruct write_transaction_table_arg *arg = cb_data;\n-\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tuint64_t ts = reftable_stack_next_update_index(arg->be->stack);\n \tstruct reftable_log_record *logs = NULL;\n \tstruct ident_split committer_ident = {0};\n \tsize_t logs_nr = 0, logs_alloc = 0, i;\n@@ -1337,7 +1364,7 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n \t\t\tstruct reftable_log_record log = {0};\n \t\t\tstruct reftable_iterator it = {0};\n \n-\t\t\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\t\t\tret = reftable_stack_init_log_iterator(arg->be->stack, &it);\n \t\t\tif (ret < 0)\n \t\t\t\tgoto done;\n \n@@ -1782,10 +1809,9 @@ static int reftable_be_rename_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"rename_ref\");\n-\tstruct reftable_stack *stack = backend_for(refs, newrefname, &newrefname)->stack;\n+\tstruct reftable_backend *be;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n-\t\t.stack = stack,\n \t\t.oldname = oldrefname,\n \t\t.newname = newrefname,\n \t\t.logmsg = logmsg,\n@@ -1797,10 +1823,11 @@ static int reftable_be_rename_ref(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, newrefname, &newrefname, 1);\n \tif (ret)\n \t\tgoto done;\n-\tret = reftable_stack_add(stack, &write_copy_table, &arg);\n+\targ.stack = be->stack;\n+\tret = reftable_stack_add(be->stack, &write_copy_table, &arg);\n \n done:\n \tassert(ret != REFTABLE_API_ERROR);\n@@ -1814,10 +1841,9 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"copy_ref\");\n-\tstruct reftable_stack *stack = backend_for(refs, newrefname, &newrefname)->stack;\n+\tstruct reftable_backend *be;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n-\t\t.stack = stack,\n \t\t.oldname = oldrefname,\n \t\t.newname = newrefname,\n \t\t.logmsg = logmsg,\n@@ -1828,10 +1854,11 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, newrefname, &newrefname, 1);\n \tif (ret)\n \t\tgoto done;\n-\tret = reftable_stack_add(stack, &write_copy_table, &arg);\n+\targ.stack = be->stack;\n+\tret = reftable_stack_add(be->stack, &write_copy_table, &arg);\n \n done:\n \tassert(ret != REFTABLE_API_ERROR);\n@@ -1995,15 +2022,19 @@ static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent_reverse\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record log = {0};\n \tstruct reftable_iterator it = {0};\n+\tstruct reftable_backend *be;\n \tint ret;\n \n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tret = backend_for(&be, refs, refname, &refname, 0);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_init_log_iterator(be->stack, &it);\n \tif (ret < 0)\n \t\tgoto done;\n \n@@ -2035,16 +2066,20 @@ static int reftable_be_for_each_reflog_ent(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record *logs = NULL;\n \tstruct reftable_iterator it = {0};\n+\tstruct reftable_backend *be;\n \tsize_t logs_alloc = 0, logs_nr = 0, i;\n \tint ret;\n \n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tret = backend_for(&be, refs, refname, &refname, 0);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_init_log_iterator(be->stack, &it);\n \tif (ret < 0)\n \t\tgoto done;\n \n@@ -2084,20 +2119,20 @@ static int reftable_be_reflog_exists(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_exists\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record log = {0};\n \tstruct reftable_iterator it = {0};\n+\tstruct reftable_backend *be;\n \tint ret;\n \n \tret = refs->err;\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(be->stack, &it);\n \tif (ret < 0)\n \t\tgoto done;\n \n@@ -2169,10 +2204,9 @@ static int reftable_be_create_reflog(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"create_reflog\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n+\tstruct reftable_backend *be;\n \tstruct write_reflog_existence_arg arg = {\n \t\t.refs = refs,\n-\t\t.stack = stack,\n \t\t.refname = refname,\n \t};\n \tint ret;\n@@ -2181,11 +2215,12 @@ static int reftable_be_create_reflog(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret)\n \t\tgoto done;\n+\targ.stack = be->stack;\n \n-\tret = reftable_stack_add(stack, &write_reflog_existence_table, &arg);\n+\tret = reftable_stack_add(be->stack, &write_reflog_existence_table, &arg);\n \n done:\n \treturn ret;\n@@ -2243,17 +2278,18 @@ static int reftable_be_delete_reflog(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"delete_reflog\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n+\tstruct reftable_backend *be;\n \tstruct write_reflog_delete_arg arg = {\n-\t\t.stack = stack,\n \t\t.refname = refname,\n \t};\n \tint ret;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret)\n \t\treturn ret;\n-\tret = reftable_stack_add(stack, &write_reflog_delete_table, &arg);\n+\targ.stack = be->stack;\n+\n+\tret = reftable_stack_add(be->stack, &write_reflog_delete_table, &arg);\n \n \tassert(ret != REFTABLE_API_ERROR);\n \treturn ret;\n@@ -2352,13 +2388,13 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \t */\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"reflog_expire\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record *logs = NULL;\n \tstruct reftable_log_record *rewritten = NULL;\n \tstruct reftable_ref_record ref_record = {0};\n \tstruct reftable_iterator it = {0};\n \tstruct reftable_addition *add = NULL;\n \tstruct reflog_expiry_arg arg = {0};\n+\tstruct reftable_backend *be;\n \tstruct object_id oid = {0};\n \tuint8_t *last_hash = NULL;\n \tsize_t logs_nr = 0, logs_alloc = 0, i;\n@@ -2367,11 +2403,11 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(be->stack, &it);\n \tif (ret < 0)\n \t\tgoto done;\n \n@@ -2379,11 +2415,11 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_new_addition(&add, stack, 0);\n+\tret = reftable_stack_new_addition(&add, be->stack, 0);\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_read_ref(stack, refname, &ref_record);\n+\tret = reftable_stack_read_ref(be->stack, refname, &ref_record);\n \tif (ret < 0)\n \t\tgoto done;\n \tif (reftable_ref_record_val1(&ref_record))\n@@ -2462,7 +2498,7 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \targ.refs = refs;\n \targ.records = rewritten;\n \targ.len = logs_nr;\n-\targ.stack = stack,\n+\targ.stack = be->stack,\n \targ.refname = refname,\n \n \tret = reftable_addition_add(add, &write_reflog_expiry_table, &arg);\n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506535","messageId":"00fdf392a64f2b0f2e9bb7b398565a87c2e187b5.1730732881.git.ps@pks.im","threadId":"62444","inReplyTo":"cover.1730732881.git.ps@pks.im","subject":"[PATCH 3/8] refs/reftable: read references via `struct reftable_backend`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-04T15:11:40Z","receivedAt":"2024-11-04T15:11:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor `read_ref_without_reload()` to accept a `struct reftable_stack`\nas input instead of accepting a `struct reftable_stack`. This allows us\nto implement an additional caching layer when reading refs where we can\nreuse reftable iterators.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c   | 97 +++++++++++++++++++--------------------\n reftable/reftable-stack.h |  3 ++\n reftable/stack.c          |  5 ++\n 3 files changed, 54 insertions(+), 51 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 93f3602faa..99caa9d5e6 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -49,6 +49,37 @@ static void reftable_backend_release(struct reftable_backend *be)\n \tbe->stack = NULL;\n }\n \n+static int reftable_backend_read_ref(struct reftable_backend *be,\n+\t\t\t\t     const char *refname,\n+\t\t\t\t     struct object_id *oid,\n+\t\t\t\t     struct strbuf *referent,\n+\t\t\t\t     unsigned int *type)\n+{\n+\tstruct reftable_ref_record ref = {0};\n+\tint ret;\n+\n+\tret = reftable_stack_read_ref(be->stack, refname, &ref);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tstrbuf_reset(referent);\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\t\t*type |= REF_ISSYMREF;\n+\t} else if (reftable_ref_record_val1(&ref)) {\n+\t\toidread(oid, reftable_ref_record_val1(&ref),\n+\t\t\t&hash_algos[hash_algo_by_id(reftable_stack_hash_id(be->stack))]);\n+\t} else {\n+\t\t/* We got a tombstone, which should not happen. */\n+\t\tBUG(\"unhandled reference value type %d\", ref.value_type);\n+\t}\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn ret;\n+}\n+\n struct reftable_ref_store {\n \tstruct ref_store base;\n \n@@ -241,38 +272,6 @@ static void fill_reftable_log_record(struct reftable_log_record *log, const stru\n \tlog->value.update.tz_offset = sign * atoi(tz_begin);\n }\n \n-static int read_ref_without_reload(struct reftable_ref_store *refs,\n-\t\t\t\t   struct reftable_stack *stack,\n-\t\t\t\t   const char *refname,\n-\t\t\t\t   struct object_id *oid,\n-\t\t\t\t   struct strbuf *referent,\n-\t\t\t\t   unsigned int *type)\n-{\n-\tstruct reftable_ref_record ref = {0};\n-\tint ret;\n-\n-\tret = reftable_stack_read_ref(stack, refname, &ref);\n-\tif (ret)\n-\t\tgoto done;\n-\n-\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n-\t\tstrbuf_reset(referent);\n-\t\tstrbuf_addstr(referent, ref.value.symref);\n-\t\t*type |= REF_ISSYMREF;\n-\t} else if (reftable_ref_record_val1(&ref)) {\n-\t\toidread(oid, reftable_ref_record_val1(&ref),\n-\t\t\trefs->base.repo->hash_algo);\n-\t} else {\n-\t\t/* We got a tombstone, which should not happen. */\n-\t\tBUG(\"unhandled reference value type %d\", ref.value_type);\n-\t}\n-\n-done:\n-\tassert(ret != REFTABLE_API_ERROR);\n-\treftable_ref_record_release(&ref);\n-\treturn ret;\n-}\n-\n static int reftable_be_config(const char *var, const char *value,\n \t\t\t      const struct config_context *ctx,\n \t\t\t      void *_opts)\n@@ -838,7 +837,7 @@ static int reftable_be_read_raw_ref(struct ref_store *ref_store,\n \tif (ret)\n \t\treturn ret;\n \n-\tret = read_ref_without_reload(refs, be->stack, refname, oid, referent, type);\n+\tret = reftable_backend_read_ref(be, refname, oid, referent, type);\n \tif (ret < 0)\n \t\treturn ret;\n \tif (ret > 0) {\n@@ -1057,8 +1056,8 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \tif (ret)\n \t\tgoto done;\n \n-\tret = read_ref_without_reload(refs, be->stack, \"HEAD\",\n-\t\t\t\t      &head_oid, &head_referent, &head_type);\n+\tret = reftable_backend_read_ref(be, \"HEAD\", &head_oid,\n+\t\t\t\t\t&head_referent, &head_type);\n \tif (ret < 0)\n \t\tgoto done;\n \tret = 0;\n@@ -1126,8 +1125,8 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t\tstring_list_insert(&affected_refnames, new_update->refname);\n \t\t}\n \n-\t\tret = read_ref_without_reload(refs, be->stack, rewritten_ref,\n-\t\t\t\t\t      &current_oid, &referent, &u->type);\n+\t\tret = reftable_backend_read_ref(be, rewritten_ref,\n+\t\t\t\t\t\t&current_oid, &referent, &u->type);\n \t\tif (ret < 0)\n \t\t\tgoto done;\n \t\tif (ret > 0 && !ref_update_expects_existing_old_ref(u)) {\n@@ -1585,7 +1584,7 @@ struct write_create_symref_arg {\n \n struct write_copy_arg {\n \tstruct reftable_ref_store *refs;\n-\tstruct reftable_stack *stack;\n+\tstruct reftable_backend *be;\n \tconst char *oldname;\n \tconst char *newname;\n \tconst char *logmsg;\n@@ -1610,7 +1609,7 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n \tif (split_ident_line(&committer_ident, committer_info, strlen(committer_info)))\n \t\tBUG(\"failed splitting committer info\");\n \n-\tif (reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref)) {\n+\tif (reftable_stack_read_ref(arg->be->stack, arg->oldname, &old_ref)) {\n \t\tret = error(_(\"refname %s not found\"), arg->oldname);\n \t\tgoto done;\n \t}\n@@ -1649,7 +1648,7 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n \t * the old branch and the creation of the new branch, and we cannot do\n \t * two changes to a reflog in a single update.\n \t */\n-\tdeletion_ts = creation_ts = reftable_stack_next_update_index(arg->stack);\n+\tdeletion_ts = creation_ts = reftable_stack_next_update_index(arg->be->stack);\n \tif (arg->delete_old)\n \t\tcreation_ts++;\n \treftable_writer_set_limits(writer, deletion_ts, creation_ts);\n@@ -1692,8 +1691,8 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n \t\tmemcpy(logs[logs_nr].value.update.old_hash, old_ref.value.val1, GIT_MAX_RAWSZ);\n \t\tlogs_nr++;\n \n-\t\tret = read_ref_without_reload(arg->refs, arg->stack, \"HEAD\", &head_oid,\n-\t\t\t\t\t      &head_referent, &head_type);\n+\t\tret = reftable_backend_read_ref(arg->be, \"HEAD\", &head_oid,\n+\t\t\t\t\t\t&head_referent, &head_type);\n \t\tif (ret < 0)\n \t\t\tgoto done;\n \t\tappend_head_reflog = (head_type & REF_ISSYMREF) && !strcmp(head_referent.buf, arg->oldname);\n@@ -1736,7 +1735,7 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n \t * copy over all log entries from the old reflog. Last but not least,\n \t * when renaming we also have to delete all the old reflog entries.\n \t */\n-\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\tret = reftable_stack_init_log_iterator(arg->be->stack, &it);\n \tif (ret < 0)\n \t\tgoto done;\n \n@@ -1809,7 +1808,6 @@ static int reftable_be_rename_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"rename_ref\");\n-\tstruct reftable_backend *be;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n \t\t.oldname = oldrefname,\n@@ -1823,11 +1821,10 @@ static int reftable_be_rename_ref(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = backend_for(&be, refs, newrefname, &newrefname, 1);\n+\tret = backend_for(&arg.be, refs, newrefname, &newrefname, 1);\n \tif (ret)\n \t\tgoto done;\n-\targ.stack = be->stack;\n-\tret = reftable_stack_add(be->stack, &write_copy_table, &arg);\n+\tret = reftable_stack_add(arg.be->stack, &write_copy_table, &arg);\n \n done:\n \tassert(ret != REFTABLE_API_ERROR);\n@@ -1841,7 +1838,6 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"copy_ref\");\n-\tstruct reftable_backend *be;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n \t\t.oldname = oldrefname,\n@@ -1854,11 +1850,10 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = backend_for(&be, refs, newrefname, &newrefname, 1);\n+\tret = backend_for(&arg.be, refs, newrefname, &newrefname, 1);\n \tif (ret)\n \t\tgoto done;\n-\targ.stack = be->stack;\n-\tret = reftable_stack_add(be->stack, &write_copy_table, &arg);\n+\tret = reftable_stack_add(arg.be->stack, &write_copy_table, &arg);\n \n done:\n \tassert(ret != REFTABLE_API_ERROR);\ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nindex 54787f2ef5..6dfd22e3f5 100644\n--- a/reftable/reftable-stack.h\n+++ b/reftable/reftable-stack.h\n@@ -149,4 +149,7 @@ struct reftable_compaction_stats {\n struct reftable_compaction_stats *\n reftable_stack_compaction_stats(struct reftable_stack *st);\n \n+/* return the hash ID of the merged table. */\n+uint32_t reftable_stack_hash_id(struct reftable_stack *st);\n+\n #endif\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex c33979536e..530ba2d927 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1790,3 +1790,8 @@ int reftable_stack_clean(struct reftable_stack *st)\n \treftable_addition_destroy(add);\n \treturn err;\n }\n+\n+uint32_t reftable_stack_hash_id(struct reftable_stack *st)\n+{\n+\treturn reftable_merged_table_hash_id(st->merged);\n+}\n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506536","messageId":"142081cb0c22702dca018198f2e315109278d3a8.1730732881.git.ps@pks.im","threadId":"62444","inReplyTo":"cover.1730732881.git.ps@pks.im","subject":"[PATCH 4/8] refs/reftable: refactor reading symbolic refs to use reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-04T15:11:43Z","receivedAt":"2024-11-04T15:11:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor the callback function that reads symbolic references in the\nreftable backend to use `reftable_backend_read_ref()` instead of\naccessing the reftable stack directly. This ensures that the function\nwill benefit from the new caching layer that we're about to introduce.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 11 ++++-------\n 1 file changed, 4 insertions(+), 7 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 99caa9d5e6..3912431111 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -854,21 +854,18 @@ static int reftable_be_read_symbolic_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_symbolic_ref\");\n-\tstruct reftable_ref_record ref = {0};\n \tstruct reftable_backend *be;\n+\tstruct object_id oid;\n+\tunsigned int type = 0;\n \tint ret;\n \n \tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret)\n \t\treturn ret;\n \n-\tret = reftable_stack_read_ref(be->stack, refname, &ref);\n-\tif (ret == 0 && ref.value_type == REFTABLE_REF_SYMREF)\n-\t\tstrbuf_addstr(referent, ref.value.symref);\n-\telse\n+\tret = reftable_backend_read_ref(be, refname, &oid, referent, &type);\n+\tif (type != REF_ISSYMREF)\n \t\tret = -1;\n-\n-\treftable_ref_record_release(&ref);\n \treturn ret;\n }\n \n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506537","messageId":"44f4adce9adb1ef16017da6631f9b957929cf842.1730732881.git.ps@pks.im","threadId":"62444","inReplyTo":"cover.1730732881.git.ps@pks.im","subject":"[PATCH 5/8] refs/reftable: refactor reflog expiry to use reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-04T15:11:48Z","receivedAt":"2024-11-04T15:11:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor the callback function that expires reflog entries in the\nreftable backend to use `reftable_backend_read_ref()` instead of\naccessing the reftable stack directly. This ensures that the function\nwill benefit from the new caching layer that we're about to introduce.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 13 +++++--------\n 1 file changed, 5 insertions(+), 8 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 3912431111..98a070f5a7 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2382,14 +2382,15 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"reflog_expire\");\n \tstruct reftable_log_record *logs = NULL;\n \tstruct reftable_log_record *rewritten = NULL;\n-\tstruct reftable_ref_record ref_record = {0};\n \tstruct reftable_iterator it = {0};\n \tstruct reftable_addition *add = NULL;\n \tstruct reflog_expiry_arg arg = {0};\n \tstruct reftable_backend *be;\n \tstruct object_id oid = {0};\n+\tstruct strbuf referent = STRBUF_INIT;\n \tuint8_t *last_hash = NULL;\n \tsize_t logs_nr = 0, logs_alloc = 0, i;\n+\tunsigned int type = 0;\n \tint ret;\n \n \tif (refs->err < 0)\n@@ -2411,12 +2412,9 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_read_ref(be->stack, refname, &ref_record);\n+\tret = reftable_backend_read_ref(be, refname, &oid, &referent, &type);\n \tif (ret < 0)\n \t\tgoto done;\n-\tif (reftable_ref_record_val1(&ref_record))\n-\t\toidread(&oid, reftable_ref_record_val1(&ref_record),\n-\t\t\tref_store->repo->hash_algo);\n \tprepare_fn(refname, &oid, policy_cb_data);\n \n \twhile (1) {\n@@ -2483,8 +2481,7 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \t\t}\n \t}\n \n-\tif (flags & EXPIRE_REFLOGS_UPDATE_REF && last_hash &&\n-\t    reftable_ref_record_val1(&ref_record))\n+\tif (flags & EXPIRE_REFLOGS_UPDATE_REF && last_hash && !is_null_oid(&oid))\n \t\toidread(&arg.update_oid, last_hash, ref_store->repo->hash_algo);\n \n \targ.refs = refs;\n@@ -2509,11 +2506,11 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \t\tcleanup_fn(policy_cb_data);\n \tassert(ret != REFTABLE_API_ERROR);\n \n-\treftable_ref_record_release(&ref_record);\n \treftable_iterator_destroy(&it);\n \treftable_addition_destroy(add);\n \tfor (i = 0; i < logs_nr; i++)\n \t\treftable_log_record_release(&logs[i]);\n+\tstrbuf_release(&referent);\n \tfree(logs);\n \tfree(rewritten);\n \treturn ret;\n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506538","messageId":"0a294b577f915ccc901c6cd5e04f05ef24f34b15.1730732881.git.ps@pks.im","threadId":"62444","inReplyTo":"cover.1730732881.git.ps@pks.im","subject":"[PATCH 6/8] reftable/stack: add mechanism to notify callers on reload","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-04T15:11:50Z","receivedAt":"2024-11-04T15:11:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Reftable stacks are reloaded in two cases:\n\n  - When calling `reftable_stack_reload()`, if the stat-cache tells us\n    that the stack has been modified.\n\n  - When committing a reftable addition.\n\nWhile callers can figure out the second case, they do not have a\nmechanism to figure out whether `reftable_stack_reload()` led to an\nactual reload of the on-disk data. All they can do is thus to assume\nthat data is always being reloaded in that case.\n\nImprove the situation by introducing a new `on_reload()` callback to the\nreftable options. If provided, the function will be invoked every time\nthe stack has indeed been reloaded. This allows callers to invalidate\ndata that depends on the current stack data.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n reftable/reftable-writer.h | 9 +++++++++\n reftable/stack.c           | 4 ++++\n 2 files changed, 13 insertions(+)\n\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex e4fc953788..d027f7070b 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -62,6 +62,15 @@ struct reftable_write_options {\n \t * negative value will cause us to block indefinitely.\n \t */\n \tlong lock_timeout_ms;\n+\n+\t/*\n+\t * Callback function to execute whenever the stack is being reloaded.\n+\t * This can be used e.g. to discard cached information that relies on\n+\t * the old stack's data. The payload data will be passed as argument to\n+\t * the callback.\n+\t */\n+\tvoid (*on_reload)(void *payload);\n+\tvoid *on_reload_payload;\n };\n \n /* reftable_block_stats holds statistics for a single block type */\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 530ba2d927..f943fc5b85 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -539,6 +539,10 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n \t\tclose(fd);\n \tfree_names(names);\n \tfree_names(names_after);\n+\n+\tif (st->opts.on_reload)\n+\t\tst->opts.on_reload(st->opts.on_reload_payload);\n+\n \treturn err;\n }\n \n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506539","messageId":"45f397b563fe88524a2063bde8cad31ab9a2ccbb.1730732881.git.ps@pks.im","threadId":"62444","inReplyTo":"cover.1730732881.git.ps@pks.im","subject":"[PATCH 7/8] reftable/merged: drain priority queue on reseek","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-04T15:11:53Z","receivedAt":"2024-11-04T15:12:01Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In 5bf96e0c39 (reftable/generic: move seeking of records into the\niterator, 2024-05-13) we have refactored the reftable codebase such that\niterators can be initialized once and then re-seeked multiple times.\nThis feature is used by 1869525066 (refs/reftable: wire up support for\nexclude patterns, 2024-09-16) in order to skip records based on exclude\npatterns provided by the caller.\n\nThe logic to re-seek the merged iterator is insufficient though because\nwe don't drain the priority queue on a re-seek. This means that the\nqueue may contain stale entries and thus reading the next record in the\nqueue will return the wrong entry. While this is an obvious bug, it is\nharmless in the context of above exclude patterns:\n\n  - If the queue contained stale entries that match the pattern then the\n    caller would already know to filter out such refs. This is because\n    our codebase is prepared to handle backends that don't have a way to\n    efficiently implement exclude patterns.\n\n  - If the queue contained stale entries that don't match the pattern\n    we'd eventually filter out any duplicates. This is because the\n    reftable code discards items with the same ref name and sorts any\n    remaining entries properly.\n\nSo things happen to work in this context regardless of the bug, and\nthere is no other use case yet where we re-seek iterators. We're about\nto introduce a caching mechanism though where iterators are reused by\nthe reftable backend, and that will expose the bug.\n\nFix the issue by draining the priority queue when seeking and add a\ntestcase that surfaces the issue.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n reftable/merged.c                |  2 +\n t/unit-tests/t-reftable-merged.c | 73 ++++++++++++++++++++++++++++++++\n 2 files changed, 75 insertions(+)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 514d6facf4..84d6e933f4 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -66,6 +66,8 @@ static int merged_iter_seek(struct merged_iter *mi, struct reftable_record *want\n \tint err;\n \n \tmi->advance_index = -1;\n+\twhile (!merged_iter_pqueue_is_empty(mi->pq))\n+\t\tmerged_iter_pqueue_remove(&mi->pq);\n \n \tfor (size_t i = 0; i < mi->subiters_len; i++) {\n \t\terr = iterator_seek(&mi->subiters[i].iter, want);\ndiff --git a/t/unit-tests/t-reftable-merged.c b/t/unit-tests/t-reftable-merged.c\nindex 4c25ee5334..99ca33f973 100644\n--- a/t/unit-tests/t-reftable-merged.c\n+++ b/t/unit-tests/t-reftable-merged.c\n@@ -273,6 +273,78 @@ static void t_merged_seek_multiple_times(void)\n \treftable_free(sources);\n }\n \n+static void t_merged_seek_multiple_times_without_draining(void)\n+{\n+\tstruct reftable_ref_record r1[] = {\n+\t\t{\n+\t\t\t.refname = (char *) \"a\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = { 1 },\n+\t\t},\n+\t\t{\n+\t\t\t.refname = (char *) \"c\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = { 2 },\n+\t\t}\n+\t};\n+\tstruct reftable_ref_record r2[] = {\n+\t\t{\n+\t\t\t.refname = (char *) \"b\",\n+\t\t\t.update_index = 2,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = { 3 },\n+\t\t},\n+\t\t{\n+\t\t\t.refname = (char *) \"d\",\n+\t\t\t.update_index = 2,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = { 4 },\n+\t\t},\n+\t};\n+\tstruct reftable_ref_record *refs[] = {\n+\t\tr1, r2,\n+\t};\n+\tsize_t sizes[] = {\n+\t\tARRAY_SIZE(r1), ARRAY_SIZE(r2),\n+\t};\n+\tstruct reftable_buf bufs[] = {\n+\t\tREFTABLE_BUF_INIT, REFTABLE_BUF_INIT,\n+\t};\n+\tstruct reftable_block_source *sources = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_ref_record rec = { 0 };\n+\tstruct reftable_iterator it = { 0 };\n+\tstruct reftable_merged_table *mt;\n+\tint err;\n+\n+\tmt = merged_table_from_records(refs, &sources, &readers, sizes, bufs, 2);\n+\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\n+\terr = reftable_iterator_seek_ref(&it, \"b\");\n+\tcheck(!err);\n+\terr = reftable_iterator_next_ref(&it, &rec);\n+\tcheck(!err);\n+\terr = reftable_ref_record_equal(&rec, &r2[0], GIT_SHA1_RAWSZ);\n+\tcheck(err == 1);\n+\n+\terr = reftable_iterator_seek_ref(&it, \"a\");\n+\tcheck(!err);\n+\terr = reftable_iterator_next_ref(&it, &rec);\n+\tcheck(!err);\n+\terr = reftable_ref_record_equal(&rec, &r1[0], GIT_SHA1_RAWSZ);\n+\tcheck(err == 1);\n+\n+\tfor (size_t i = 0; i < ARRAY_SIZE(bufs); i++)\n+\t\treftable_buf_release(&bufs[i]);\n+\treaders_destroy(readers, ARRAY_SIZE(refs));\n+\treftable_ref_record_release(&rec);\n+\treftable_iterator_destroy(&it);\n+\treftable_merged_table_free(mt);\n+\treftable_free(sources);\n+}\n+\n static struct reftable_merged_table *\n merged_table_from_log_records(struct reftable_log_record **logs,\n \t\t\t      struct reftable_block_source **source,\n@@ -467,6 +539,7 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \tTEST(t_merged_logs(), \"merged table with multiple log updates for same ref\");\n \tTEST(t_merged_refs(), \"merged table with multiple updates to same ref\");\n \tTEST(t_merged_seek_multiple_times(), \"merged table can seek multiple times\");\n+\tTEST(t_merged_seek_multiple_times_without_draining(), \"merged table can seek multiple times without draining\");\n \tTEST(t_merged_single_record(), \"ref occurring in only one record can be fetched\");\n \n \treturn test_done();\n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506540","messageId":"feb4e6a36f960914ee6967b95d877d97065b5384.1730732881.git.ps@pks.im","threadId":"62444","inReplyTo":"cover.1730732881.git.ps@pks.im","subject":"[PATCH 8/8] refs/reftable: reuse iterators when reading refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-04T15:11:56Z","receivedAt":"2024-11-04T15:12:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When reading references the reftable backend has to:\n\n  1. Create a new ref iterator.\n\n  2. Seek the iterator to the record we're searching for.\n\n  3. Read the record.\n\nWe cannot really avoid the last two steps, but re-creating the iterator\nevery single time we want to read a reference is kind of expensive and a\nwaste of resources. We couldn't help it in the past though because it\nwas not possible to reuse iterators. But starting with 5bf96e0c39\n(reftable/generic: move seeking of records into the iterator,\n2024-05-13) we have split up the iterator lifecycle such that creating\nthe iterator and seeking are two different concerns.\n\nRefactor the code such that we cache iterators in the reftable backend.\nThis cache is invalidated whenever the respective stack is reloaded such\nthat we know to recreate the iterator in that case. This leads to a\nsizeable speedup when creating many refs, which requires a lot of random\nreference reads:\n\n    Benchmark 1: update-ref: create many refs (refcount = 100000, revision = master)\n      Time (mean ± σ):      1.793 s ±  0.010 s    [User: 0.954 s, System: 0.835 s]\n      Range (min … max):    1.781 s …  1.811 s    10 runs\n\n    Benchmark 2: update-ref: create many refs (refcount = 100000, revision = HEAD)\n      Time (mean ± σ):      1.680 s ±  0.013 s    [User: 0.846 s, System: 0.831 s]\n      Range (min … max):    1.664 s …  1.702 s    10 runs\n\n    Summary\n      update-ref: create many refs (refcount = 100000, revision = HEAD) ran\n        1.07 ± 0.01 times faster than update-ref: create many refs (refcount = 100000, revision = master)\n\nWhile 7% is not a huge win, you have to consider that the benchmark is\n_writing_ data, so _reading_ references is only one part of what we do.\nFlame graphs show that we spend around 40% of our time reading refs, so\nthe speedup when reading refs is approximately ~2.5x that. I could not\nfind better benchmarks where we perform a lot of random ref reads.\n\nYou can also see a sizeable impact on memory usage when creating 100k\nreferences. Before this change:\n\n    HEAP SUMMARY:\n        in use at exit: 19,112,538 bytes in 200,170 blocks\n      total heap usage: 8,400,426 allocs, 8,200,256 frees, 454,367,048 bytes allocated\n\nAfter this change:\n\n    HEAP SUMMARY:\n        in use at exit: 674,416 bytes in 169 blocks\n      total heap usage: 7,929,872 allocs, 7,929,703 frees, 281,509,985 bytes allocated\n\nAs an additional factor, this refactoring opens up the possibility for\nmore performance optimizations in how we re-seek iterators. Any change\nthat allows us to optimize re-seeking by e.g. reusing data structures\nwould thus also directly speed up random reads.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 32 +++++++++++++++++++++++++++++---\n 1 file changed, 29 insertions(+), 3 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 98a070f5a7..e0577d666f 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -34,19 +34,30 @@\n \n struct reftable_backend {\n \tstruct reftable_stack *stack;\n+\tstruct reftable_iterator it;\n };\n \n+static void reftable_backend_on_reload(void *payload)\n+{\n+\tstruct reftable_backend *be = payload;\n+\treftable_iterator_destroy(&be->it);\n+}\n+\n static int reftable_backend_init(struct reftable_backend *be,\n \t\t\t\t const char *path,\n-\t\t\t\t const struct reftable_write_options *opts)\n+\t\t\t\t const struct reftable_write_options *_opts)\n {\n-\treturn reftable_new_stack(&be->stack, path, opts);\n+\tstruct reftable_write_options opts = *_opts;\n+\topts.on_reload = reftable_backend_on_reload;\n+\topts.on_reload_payload = be;\n+\treturn reftable_new_stack(&be->stack, path, &opts);\n }\n \n static void reftable_backend_release(struct reftable_backend *be)\n {\n \treftable_stack_destroy(be->stack);\n \tbe->stack = NULL;\n+\treftable_iterator_destroy(&be->it);\n }\n \n static int reftable_backend_read_ref(struct reftable_backend *be,\n@@ -58,10 +69,25 @@ static int reftable_backend_read_ref(struct reftable_backend *be,\n \tstruct reftable_ref_record ref = {0};\n \tint ret;\n \n-\tret = reftable_stack_read_ref(be->stack, refname, &ref);\n+\tif (!be->it.ops) {\n+\t\tret = reftable_stack_init_ref_iterator(be->stack, &be->it);\n+\t\tif (ret)\n+\t\t\tgoto done;\n+\t}\n+\n+\tret = reftable_iterator_seek_ref(&be->it, refname);\n \tif (ret)\n \t\tgoto done;\n \n+\tret = reftable_iterator_next_ref(&be->it, &ref);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tif (strcmp(ref.refname, refname)) {\n+\t\tret = 1;\n+\t\tgoto done;\n+\t}\n+\n \tif (ref.value_type == REFTABLE_REF_SYMREF) {\n \t\tstrbuf_reset(referent);\n \t\tstrbuf_addstr(referent, ref.value.symref);\n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506598","messageId":"xmqqv7x2tlnk.fsf@gitster.g","threadId":"62444","inReplyTo":"45f397b563fe88524a2063bde8cad31ab9a2ccbb.1730732881.git.ps@pks.im","subject":"Re: [PATCH 7/8] reftable/merged: drain priority queue on reseek","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-05T03:16:47Z","receivedAt":"2024-11-05T03:16:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> diff --git a/t/unit-tests/t-reftable-merged.c b/t/unit-tests/t-reftable-merged.c\n> index 4c25ee5334..99ca33f973 100644\n> --- a/t/unit-tests/t-reftable-merged.c\n> +++ b/t/unit-tests/t-reftable-merged.c\n> @@ -273,6 +273,78 @@ static void t_merged_seek_multiple_times(void)\n>  \treftable_free(sources);\n>  }\n>  \n> +static void t_merged_seek_multiple_times_without_draining(void)\n> +{\n> ...\n> +\terr = reftable_ref_record_equal(&rec, &r2[0], GIT_SHA1_RAWSZ);\n> +\tcheck(err == 1);\n> +\n> +\terr = reftable_iterator_seek_ref(&it, \"a\");\n> +\tcheck(!err);\n> +\terr = reftable_iterator_next_ref(&it, &rec);\n> +\tcheck(!err);\n> +\terr = reftable_ref_record_equal(&rec, &r1[0], GIT_SHA1_RAWSZ);\n\nDid you mean REFTABLE_HASH_SIZE_SHA1 instead?\n\ndiff --git i/t/unit-tests/t-reftable-merged.c w/t/unit-tests/t-reftable-merged.c\nindex 620803e0ed..a12bd0e1a3 100644\n--- i/t/unit-tests/t-reftable-merged.c\n+++ w/t/unit-tests/t-reftable-merged.c\n@@ -326,14 +326,14 @@ static void t_merged_seek_multiple_times_without_draining(void)\n \tcheck(!err);\n \terr = reftable_iterator_next_ref(&it, &rec);\n \tcheck(!err);\n-\terr = reftable_ref_record_equal(&rec, &r2[0], GIT_SHA1_RAWSZ);\n+\terr = reftable_ref_record_equal(&rec, &r2[0], REFTABLE_HASH_SIZE_SHA1);\n \tcheck(err == 1);\n \n \terr = reftable_iterator_seek_ref(&it, \"a\");\n \tcheck(!err);\n \terr = reftable_iterator_next_ref(&it, &rec);\n \tcheck(!err);\n-\terr = reftable_ref_record_equal(&rec, &r1[0], GIT_SHA1_RAWSZ);\n+\terr = reftable_ref_record_equal(&rec, &r1[0], REFTABLE_HASH_SIZE_SHA1);\n \tcheck(err == 1);\n \n \tfor (size_t i = 0; i < ARRAY_SIZE(bufs); i++)\n"},{"id":"506599","messageId":"xmqqr07qtlcm.fsf@gitster.g","threadId":"62444","inReplyTo":"xmqqv7x2tlnk.fsf@gitster.g","subject":"Re: [PATCH 7/8] reftable/merged: drain priority queue on reseek","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-05T03:23:21Z","receivedAt":"2024-11-05T03:23:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Did you mean REFTABLE_HASH_SIZE_SHA1 instead?\n\nAh, that transition hasn't happened yet on 'master'.  I'll carry the\nsemantic conflict resoluion in merge-fix hierarchy then.\n\n> diff --git i/t/unit-tests/t-reftable-merged.c w/t/unit-tests/t-reftable-merged.c\n> index 620803e0ed..a12bd0e1a3 100644\n> --- i/t/unit-tests/t-reftable-merged.c\n> +++ w/t/unit-tests/t-reftable-merged.c\n> @@ -326,14 +326,14 @@ static void t_merged_seek_multiple_times_without_draining(void)\n>  \tcheck(!err);\n>  \terr = reftable_iterator_next_ref(&it, &rec);\n>  \tcheck(!err);\n> -\terr = reftable_ref_record_equal(&rec, &r2[0], GIT_SHA1_RAWSZ);\n> +\terr = reftable_ref_record_equal(&rec, &r2[0], REFTABLE_HASH_SIZE_SHA1);\n>  \tcheck(err == 1);\n>  \n>  \terr = reftable_iterator_seek_ref(&it, \"a\");\n>  \tcheck(!err);\n>  \terr = reftable_iterator_next_ref(&it, &rec);\n>  \tcheck(!err);\n> -\terr = reftable_ref_record_equal(&rec, &r1[0], GIT_SHA1_RAWSZ);\n> +\terr = reftable_ref_record_equal(&rec, &r1[0], REFTABLE_HASH_SIZE_SHA1);\n>  \tcheck(err == 1);\n>  \n>  \tfor (size_t i = 0; i < ARRAY_SIZE(bufs); i++)\n\n"},{"id":"506601","messageId":"xmqqmsiethdp.fsf@gitster.g","threadId":"62444","inReplyTo":"cover.1730732881.git.ps@pks.im","subject":"Re: [PATCH 0/8] refs/reftable: reuse iterators when reading refs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-05T04:49:06Z","receivedAt":"2024-11-05T04:49:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> this patch series refactors the reftable backend to reuse reftable\n> iterators when reading random references. This removes the overhead of\n> having to recreate the iterator on every read and thus leads to better\n> performance and less allocation churn. It also gives us the ability to\n> further optimize reads by optimizing re-seeking iterators in the future.\n>\n> Overall this leads to a 7% speedup when creating many refs in a\n> transaction, which performs many random reads. But this change also\n> positively impacts other usecases.\n\nWhen merged to 'seen', this seems to break a handful of tests.  I am\nreasonably sure that it is interaction with other topics, as these\ntests passes in this topic itself without other topics merged.\n\nThanks.\n\n\nTest Summary Report\n-------------------\nt0611-reftable-httpd.sh             (Wstat: 256 (exited 1) Tests: 1 Failed: 1)\n  Failed test:  1\n  Non-zero exit status: 1\nt0613-reftable-write-options.sh     (Wstat: 256 (exited 1) Tests: 11 Failed: 9)\n  Failed tests:  1-3, 5-8, 10-11\n  Non-zero exit status: 1\nt7424-submodule-mixed-ref-formats.sh (Wstat: 256 (exited 1) Tests: 7 Failed: 6)\n  Failed tests:  2-7\n  Non-zero exit status: 1\nt1460-refs-migrate.sh               (Wstat: 256 (exited 1) Tests: 30 Failed: 16)\n  Failed tests:  9-15, 21-27, 29-30\n  Non-zero exit status: 1\nt0610-reftable-basics.sh            (Wstat: 256 (exited 1) Tests: 89 Failed: 53)\n  Failed tests:  8, 10, 12, 14, 16, 18, 20, 22, 24-27, 38-42\n                44, 47-52, 55-56, 58, 62-71, 73, 75-89\n  Non-zero exit status: 1\n"},{"id":"506640","messageId":"ZynF3Ai9FDVRAYOc@pks.im","threadId":"62444","inReplyTo":"xmqqr07qtlcm.fsf@gitster.g","subject":"Re: [PATCH 7/8] reftable/merged: drain priority queue on reseek","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-05T07:14:36Z","receivedAt":"2024-11-05T07:14:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Nov 04, 2024 at 07:23:21PM -0800, Junio C Hamano wrote:\n> Junio C Hamano <gitster@pobox.com> writes:\n> \n> > Did you mean REFTABLE_HASH_SIZE_SHA1 instead?\n> \n> Ah, that transition hasn't happened yet on 'master'.  I'll carry the\n> semantic conflict resoluion in merge-fix hierarchy then.\n> \n> > diff --git i/t/unit-tests/t-reftable-merged.c w/t/unit-tests/t-reftable-merged.c\n> > index 620803e0ed..a12bd0e1a3 100644\n> > --- i/t/unit-tests/t-reftable-merged.c\n> > +++ w/t/unit-tests/t-reftable-merged.c\n> > @@ -326,14 +326,14 @@ static void t_merged_seek_multiple_times_without_draining(void)\n> >  \tcheck(!err);\n> >  \terr = reftable_iterator_next_ref(&it, &rec);\n> >  \tcheck(!err);\n> > -\terr = reftable_ref_record_equal(&rec, &r2[0], GIT_SHA1_RAWSZ);\n> > +\terr = reftable_ref_record_equal(&rec, &r2[0], REFTABLE_HASH_SIZE_SHA1);\n> >  \tcheck(err == 1);\n> >  \n> >  \terr = reftable_iterator_seek_ref(&it, \"a\");\n> >  \tcheck(!err);\n> >  \terr = reftable_iterator_next_ref(&it, &rec);\n> >  \tcheck(!err);\n> > -\terr = reftable_ref_record_equal(&rec, &r1[0], GIT_SHA1_RAWSZ);\n> > +\terr = reftable_ref_record_equal(&rec, &r1[0], REFTABLE_HASH_SIZE_SHA1);\n> >  \tcheck(err == 1);\n> >  \n> >  \tfor (size_t i = 0; i < ARRAY_SIZE(bufs); i++)\n\nThis looks good to me. I guess I may end up resending this topic with a\nnew merge base in case the other failures are caused by that semantic\nmerge conflict, too.\n\nThanks!\n\nPatrick\n"},{"id":"506643","messageId":"cover.1730792627.git.ps@pks.im","threadId":"62444","inReplyTo":"cover.1730732881.git.ps@pks.im","subject":"[PATCH v2 0/8] refs/reftable: reuse iterators when reading refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-05T09:11:48Z","receivedAt":"2024-11-05T09:12:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the second version of my patch series that refactors the\nreftable backend to reuse iterators when reading random references. This\nremoves the overhead of having to recreate the iterator on every read\nand thus leads to better performance and less allocation churn.\n\nThe only change compared to v2 is that I've rebased the series on top of\n8f8d6eee53 (The seventh batch, 2024-11-01) with ps/reftable-detach at\n3740325472 (reftable/system: provide thin wrapper for lockfile\nsubsystem, 2024-10-23) merged into it. This was done to fix textual and\nsemantic conflicts with that series.\n\nThanks!\n\nPatrick\n\nPatrick Steinhardt (8):\n  refs/reftable: encapsulate reftable stack\n  refs/reftable: handle reloading stacks in the reftable backend\n  refs/reftable: read references via `struct reftable_backend`\n  refs/reftable: refactor reading symbolic refs to use reftable backend\n  refs/reftable: refactor reflog expiry to use reftable backend\n  reftable/stack: add mechanism to notify callers on reload\n  reftable/merged: drain priority queue on reseek\n  refs/reftable: reuse iterators when reading refs\n\n refs/reftable-backend.c          | 369 +++++++++++++++++++------------\n reftable/merged.c                |   2 +\n reftable/reftable-stack.h        |   3 +\n reftable/reftable-writer.h       |   9 +\n reftable/stack.c                 |   9 +\n t/unit-tests/t-reftable-merged.c |  73 ++++++\n 6 files changed, 321 insertions(+), 144 deletions(-)\n\nRange-diff against v1:\n1:  b599bcdac1 = 1:  ac01c06c41 refs/reftable: encapsulate reftable stack\n2:  b81ce63589 = 2:  bab837e373 refs/reftable: handle reloading stacks in the reftable backend\n3:  00fdf392a6 ! 3:  1b50655202 refs/reftable: read references via `struct reftable_backend`\n    @@ refs/reftable-backend.c: static void reftable_backend_release(struct reftable_ba\n     +\t\tstrbuf_addstr(referent, ref.value.symref);\n     +\t\t*type |= REF_ISSYMREF;\n     +\t} else if (reftable_ref_record_val1(&ref)) {\n    ++\t\tunsigned int hash_id;\n    ++\n    ++\t\tswitch (reftable_stack_hash_id(be->stack)) {\n    ++\t\tcase REFTABLE_HASH_SHA1:\n    ++\t\t\thash_id = GIT_HASH_SHA1;\n    ++\t\t\tbreak;\n    ++\t\tcase REFTABLE_HASH_SHA256:\n    ++\t\t\thash_id = GIT_HASH_SHA256;\n    ++\t\t\tbreak;\n    ++\t\tdefault:\n    ++\t\t\tBUG(\"unhandled hash ID %d\", reftable_stack_hash_id(be->stack));\n    ++\t\t}\n    ++\n     +\t\toidread(oid, reftable_ref_record_val1(&ref),\n    -+\t\t\t&hash_algos[hash_algo_by_id(reftable_stack_hash_id(be->stack))]);\n    ++\t\t\t&hash_algos[hash_id]);\n     +\t} else {\n     +\t\t/* We got a tombstone, which should not happen. */\n     +\t\tBUG(\"unhandled reference value type %d\", ref.value_type);\n    @@ reftable/reftable-stack.h: struct reftable_compaction_stats {\n      struct reftable_compaction_stats *\n      reftable_stack_compaction_stats(struct reftable_stack *st);\n      \n    -+/* return the hash ID of the merged table. */\n    -+uint32_t reftable_stack_hash_id(struct reftable_stack *st);\n    ++/* Return the hash of the stack. */\n    ++enum reftable_hash reftable_stack_hash_id(struct reftable_stack *st);\n     +\n      #endif\n     \n    @@ reftable/stack.c: int reftable_stack_clean(struct reftable_stack *st)\n      \treturn err;\n      }\n     +\n    -+uint32_t reftable_stack_hash_id(struct reftable_stack *st)\n    ++enum reftable_hash reftable_stack_hash_id(struct reftable_stack *st)\n     +{\n     +\treturn reftable_merged_table_hash_id(st->merged);\n     +}\n4:  142081cb0c = 4:  0906b04fc6 refs/reftable: refactor reading symbolic refs to use reftable backend\n5:  44f4adce9a = 5:  355557ec95 refs/reftable: refactor reflog expiry to use reftable backend\n6:  0a294b577f ! 6:  71ad6c80b0 reftable/stack: add mechanism to notify callers on reload\n    @@ Commit message\n     \n      ## reftable/reftable-writer.h ##\n     @@ reftable/reftable-writer.h: struct reftable_write_options {\n    - \t * negative value will cause us to block indefinitely.\n    + \t * fsync(3P) when unset.\n      \t */\n    - \tlong lock_timeout_ms;\n    + \tint (*fsync)(int fd);\n     +\n     +\t/*\n     +\t * Callback function to execute whenever the stack is being reloaded.\n7:  45f397b563 ! 7:  93efd11886 reftable/merged: drain priority queue on reseek\n    @@ t/unit-tests/t-reftable-merged.c: static void t_merged_seek_multiple_times(void)\n     +\tcheck(!err);\n     +\terr = reftable_iterator_next_ref(&it, &rec);\n     +\tcheck(!err);\n    -+\terr = reftable_ref_record_equal(&rec, &r2[0], GIT_SHA1_RAWSZ);\n    ++\terr = reftable_ref_record_equal(&rec, &r2[0], REFTABLE_HASH_SIZE_SHA1);\n     +\tcheck(err == 1);\n     +\n     +\terr = reftable_iterator_seek_ref(&it, \"a\");\n     +\tcheck(!err);\n     +\terr = reftable_iterator_next_ref(&it, &rec);\n     +\tcheck(!err);\n    -+\terr = reftable_ref_record_equal(&rec, &r1[0], GIT_SHA1_RAWSZ);\n    ++\terr = reftable_ref_record_equal(&rec, &r1[0], REFTABLE_HASH_SIZE_SHA1);\n     +\tcheck(err == 1);\n     +\n     +\tfor (size_t i = 0; i < ARRAY_SIZE(bufs); i++)\n8:  feb4e6a36f = 8:  276c27e770 refs/reftable: reuse iterators when reading refs\n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506642","messageId":"ac01c06c4177d1b07678972115648d125c2bdc66.1730792627.git.ps@pks.im","threadId":"62444","inReplyTo":"cover.1730792627.git.ps@pks.im","subject":"[PATCH v2 1/8] refs/reftable: encapsulate reftable stack","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-05T09:11:59Z","receivedAt":"2024-11-05T09:12:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable ref store needs to keep track of multiple stacks, one for\nthe main worktree and an arbitrary number of stacks for worktrees. This\nis done by storing pointers to `struct reftable_stack`, which we then\naccess directly.\n\nWrap the stack in a new `struct reftable_backend`. This will allow us to\nattach more data to each respective stack in subsequent commits.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 129 +++++++++++++++++++++++-----------------\n 1 file changed, 73 insertions(+), 56 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex f560bc2b67..116cc5ec23 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -34,6 +34,23 @@\n  */\n #define REF_UPDATE_VIA_HEAD (1 << 8)\n \n+struct reftable_backend {\n+\tstruct reftable_stack *stack;\n+};\n+\n+static int reftable_backend_init(struct reftable_backend *be,\n+\t\t\t\t const char *path,\n+\t\t\t\t const struct reftable_write_options *opts)\n+{\n+\treturn reftable_new_stack(&be->stack, path, opts);\n+}\n+\n+static void reftable_backend_release(struct reftable_backend *be)\n+{\n+\treftable_stack_destroy(be->stack);\n+\tbe->stack = NULL;\n+}\n+\n struct reftable_ref_store {\n \tstruct ref_store base;\n \n@@ -41,17 +58,17 @@ struct reftable_ref_store {\n \t * The main stack refers to the common dir and thus contains common\n \t * refs as well as refs of the main repository.\n \t */\n-\tstruct reftable_stack *main_stack;\n+\tstruct reftable_backend main_backend;\n \t/*\n \t * The worktree stack refers to the gitdir in case the refdb is opened\n \t * via a worktree. It thus contains the per-worktree refs.\n \t */\n-\tstruct reftable_stack *worktree_stack;\n+\tstruct reftable_backend worktree_backend;\n \t/*\n \t * Map of worktree stacks by their respective worktree names. The map\n \t * is populated lazily when we try to resolve `worktrees/$worktree` refs.\n \t */\n-\tstruct strmap worktree_stacks;\n+\tstruct strmap worktree_backends;\n \tstruct reftable_write_options write_options;\n \n \tunsigned int store_flags;\n@@ -97,21 +114,21 @@ static struct reftable_ref_store *reftable_be_downcast(struct ref_store *ref_sto\n  * like `worktrees/$worktree/refs/heads/foo` as worktree stacks will store\n  * those references in their normalized form.\n  */\n-static struct reftable_stack *stack_for(struct reftable_ref_store *store,\n-\t\t\t\t\tconst char *refname,\n-\t\t\t\t\tconst char **rewritten_ref)\n+static struct reftable_backend *backend_for(struct reftable_ref_store *store,\n+\t\t\t\t\t    const char *refname,\n+\t\t\t\t\t    const char **rewritten_ref)\n {\n \tconst char *wtname;\n \tint wtname_len;\n \n \tif (!refname)\n-\t\treturn store->main_stack;\n+\t\treturn &store->main_backend;\n \n \tswitch (parse_worktree_ref(refname, &wtname, &wtname_len, rewritten_ref)) {\n \tcase REF_WORKTREE_OTHER: {\n \t\tstatic struct strbuf wtname_buf = STRBUF_INIT;\n \t\tstruct strbuf wt_dir = STRBUF_INIT;\n-\t\tstruct reftable_stack *stack;\n+\t\tstruct reftable_backend *be;\n \n \t\t/*\n \t\t * We're using a static buffer here so that we don't need to\n@@ -125,37 +142,39 @@ static struct reftable_stack *stack_for(struct reftable_ref_store *store,\n \t\t/*\n \t\t * There is an edge case here: when the worktree references the\n \t\t * current worktree, then we set up the stack once via\n-\t\t * `worktree_stacks` and once via `worktree_stack`. This is\n+\t\t * `worktree_backends` and once via `worktree_backend`. This is\n \t\t * wasteful, but in the reading case it shouldn't matter. And\n \t\t * in the writing case we would notice that the stack is locked\n \t\t * already and error out when trying to write a reference via\n \t\t * both stacks.\n \t\t */\n-\t\tstack = strmap_get(&store->worktree_stacks, wtname_buf.buf);\n-\t\tif (!stack) {\n+\t\tbe = strmap_get(&store->worktree_backends, wtname_buf.buf);\n+\t\tif (!be) {\n \t\t\tstrbuf_addf(&wt_dir, \"%s/worktrees/%s/reftable\",\n \t\t\t\t    store->base.repo->commondir, wtname_buf.buf);\n \n-\t\t\tstore->err = reftable_new_stack(&stack, wt_dir.buf,\n-\t\t\t\t\t\t\t&store->write_options);\n+\t\t\tCALLOC_ARRAY(be, 1);\n+\t\t\tstore->err = reftable_backend_init(be, wt_dir.buf,\n+\t\t\t\t\t\t\t   &store->write_options);\n \t\t\tassert(store->err != REFTABLE_API_ERROR);\n-\t\t\tstrmap_put(&store->worktree_stacks, wtname_buf.buf, stack);\n+\n+\t\t\tstrmap_put(&store->worktree_backends, wtname_buf.buf, be);\n \t\t}\n \n \t\tstrbuf_release(&wt_dir);\n-\t\treturn stack;\n+\t\treturn be;\n \t}\n \tcase REF_WORKTREE_CURRENT:\n \t\t/*\n \t\t * If there is no worktree stack then we're currently in the\n \t\t * main worktree. We thus return the main stack in that case.\n \t\t */\n-\t\tif (!store->worktree_stack)\n-\t\t\treturn store->main_stack;\n-\t\treturn store->worktree_stack;\n+\t\tif (!store->worktree_backend.stack)\n+\t\t\treturn &store->main_backend;\n+\t\treturn &store->worktree_backend;\n \tcase REF_WORKTREE_MAIN:\n \tcase REF_WORKTREE_SHARED:\n-\t\treturn store->main_stack;\n+\t\treturn &store->main_backend;\n \tdefault:\n \t\tBUG(\"unhandled worktree reference type\");\n \t}\n@@ -292,7 +311,7 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \tumask(mask);\n \n \tbase_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable);\n-\tstrmap_init(&refs->worktree_stacks);\n+\tstrmap_init(&refs->worktree_backends);\n \trefs->store_flags = store_flags;\n \trefs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);\n \n@@ -337,8 +356,8 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \t\tstrbuf_realpath(&path, gitdir, 0);\n \t}\n \tstrbuf_addstr(&path, \"/reftable\");\n-\trefs->err = reftable_new_stack(&refs->main_stack, path.buf,\n-\t\t\t\t       &refs->write_options);\n+\trefs->err = reftable_backend_init(&refs->main_backend, path.buf,\n+\t\t\t\t\t  &refs->write_options);\n \tif (refs->err)\n \t\tgoto done;\n \n@@ -354,8 +373,8 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \t\tstrbuf_reset(&path);\n \t\tstrbuf_addf(&path, \"%s/reftable\", gitdir);\n \n-\t\trefs->err = reftable_new_stack(&refs->worktree_stack, path.buf,\n-\t\t\t\t\t       &refs->write_options);\n+\t\trefs->err = reftable_backend_init(&refs->worktree_backend, path.buf,\n+\t\t\t\t\t\t  &refs->write_options);\n \t\tif (refs->err)\n \t\t\tgoto done;\n \t}\n@@ -374,19 +393,17 @@ static void reftable_be_release(struct ref_store *ref_store)\n \tstruct strmap_entry *entry;\n \tstruct hashmap_iter iter;\n \n-\tif (refs->main_stack) {\n-\t\treftable_stack_destroy(refs->main_stack);\n-\t\trefs->main_stack = NULL;\n-\t}\n+\tif (refs->main_backend.stack)\n+\t\treftable_backend_release(&refs->main_backend);\n+\tif (refs->worktree_backend.stack)\n+\t\treftable_backend_release(&refs->worktree_backend);\n \n-\tif (refs->worktree_stack) {\n-\t\treftable_stack_destroy(refs->worktree_stack);\n-\t\trefs->worktree_stack = NULL;\n+\tstrmap_for_each_entry(&refs->worktree_backends, &iter, entry) {\n+\t\tstruct reftable_backend *be = entry->value;\n+\t\treftable_backend_release(be);\n+\t\tfree(be);\n \t}\n-\n-\tstrmap_for_each_entry(&refs->worktree_stacks, &iter, entry)\n-\t\treftable_stack_destroy(entry->value);\n-\tstrmap_clear(&refs->worktree_stacks, 0);\n+\tstrmap_clear(&refs->worktree_backends, 0);\n }\n \n static int reftable_be_create_on_disk(struct ref_store *ref_store,\n@@ -781,7 +798,7 @@ static struct ref_iterator *reftable_be_iterator_begin(struct ref_store *ref_sto\n \t\trequired_flags |= REF_STORE_ODB;\n \trefs = reftable_be_downcast(ref_store, required_flags, \"ref_iterator_begin\");\n \n-\tmain_iter = ref_iterator_for_stack(refs, refs->main_stack, prefix,\n+\tmain_iter = ref_iterator_for_stack(refs, refs->main_backend.stack, prefix,\n \t\t\t\t\t   exclude_patterns, flags);\n \n \t/*\n@@ -789,14 +806,14 @@ static struct ref_iterator *reftable_be_iterator_begin(struct ref_store *ref_sto\n \t * right now. If we aren't, then we return the common reftable\n \t * iterator, only.\n \t */\n-\t if (!refs->worktree_stack)\n+\t if (!refs->worktree_backend.stack)\n \t\treturn &main_iter->base;\n \n \t/*\n \t * Otherwise we merge both the common and the per-worktree refs into a\n \t * single iterator.\n \t */\n-\tworktree_iter = ref_iterator_for_stack(refs, refs->worktree_stack, prefix,\n+\tworktree_iter = ref_iterator_for_stack(refs, refs->worktree_backend.stack, prefix,\n \t\t\t\t\t       exclude_patterns, flags);\n \treturn merge_ref_iterator_begin(&worktree_iter->base, &main_iter->base,\n \t\t\t\t\tref_iterator_select, NULL);\n@@ -811,7 +828,7 @@ static int reftable_be_read_raw_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_raw_ref\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tint ret;\n \n \tif (refs->err < 0)\n@@ -838,7 +855,7 @@ static int reftable_be_read_symbolic_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_symbolic_ref\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_ref_record ref = {0};\n \tint ret;\n \n@@ -898,7 +915,7 @@ static int prepare_transaction_update(struct write_transaction_table_arg **out,\n \t\t\t\t      struct ref_update *update,\n \t\t\t\t      struct strbuf *err)\n {\n-\tstruct reftable_stack *stack = stack_for(refs, update->refname, NULL);\n+\tstruct reftable_stack *stack = backend_for(refs, update->refname, NULL)->stack;\n \tstruct write_transaction_table_arg *arg = NULL;\n \tsize_t i;\n \tint ret;\n@@ -1031,7 +1048,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\tgoto done;\n \t}\n \n-\tret = read_ref_without_reload(refs, stack_for(refs, \"HEAD\", NULL), \"HEAD\",\n+\tret = read_ref_without_reload(refs, backend_for(refs, \"HEAD\", NULL)->stack, \"HEAD\",\n \t\t\t\t      &head_oid, &head_referent, &head_type);\n \tif (ret < 0)\n \t\tgoto done;\n@@ -1043,7 +1060,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\tstruct reftable_stack *stack;\n \t\tconst char *rewritten_ref;\n \n-\t\tstack = stack_for(refs, u->refname, &rewritten_ref);\n+\t\tstack = backend_for(refs, u->refname, &rewritten_ref)->stack;\n \n \t\t/* Verify that the new object ID is valid. */\n \t\tif ((u->flags & REF_HAVE_NEW) && !is_null_oid(&u->new_oid) &&\n@@ -1525,9 +1542,9 @@ static int reftable_be_pack_refs(struct ref_store *ref_store,\n \tif (refs->err)\n \t\treturn refs->err;\n \n-\tstack = refs->worktree_stack;\n+\tstack = refs->worktree_backend.stack;\n \tif (!stack)\n-\t\tstack = refs->main_stack;\n+\t\tstack = refs->main_backend.stack;\n \n \tif (opts->flags & PACK_REFS_AUTO)\n \t\tret = reftable_stack_auto_compact(stack);\n@@ -1782,7 +1799,7 @@ static int reftable_be_rename_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"rename_ref\");\n-\tstruct reftable_stack *stack = stack_for(refs, newrefname, &newrefname);\n+\tstruct reftable_stack *stack = backend_for(refs, newrefname, &newrefname)->stack;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n \t\t.stack = stack,\n@@ -1814,7 +1831,7 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"copy_ref\");\n-\tstruct reftable_stack *stack = stack_for(refs, newrefname, &newrefname);\n+\tstruct reftable_stack *stack = backend_for(refs, newrefname, &newrefname)->stack;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n \t\t.stack = stack,\n@@ -1952,11 +1969,11 @@ static struct ref_iterator *reftable_be_reflog_iterator_begin(struct ref_store *\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_iterator_begin\");\n \tstruct reftable_reflog_iterator *main_iter, *worktree_iter;\n \n-\tmain_iter = reflog_iterator_for_stack(refs, refs->main_stack);\n-\tif (!refs->worktree_stack)\n+\tmain_iter = reflog_iterator_for_stack(refs, refs->main_backend.stack);\n+\tif (!refs->worktree_backend.stack)\n \t\treturn &main_iter->base;\n \n-\tworktree_iter = reflog_iterator_for_stack(refs, refs->worktree_stack);\n+\tworktree_iter = reflog_iterator_for_stack(refs, refs->worktree_backend.stack);\n \n \treturn merge_ref_iterator_begin(&worktree_iter->base, &main_iter->base,\n \t\t\t\t\tref_iterator_select, NULL);\n@@ -1995,7 +2012,7 @@ static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent_reverse\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record log = {0};\n \tstruct reftable_iterator it = {0};\n \tint ret;\n@@ -2035,7 +2052,7 @@ static int reftable_be_for_each_reflog_ent(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record *logs = NULL;\n \tstruct reftable_iterator it = {0};\n \tsize_t logs_alloc = 0, logs_nr = 0, i;\n@@ -2084,7 +2101,7 @@ static int reftable_be_reflog_exists(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_exists\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record log = {0};\n \tstruct reftable_iterator it = {0};\n \tint ret;\n@@ -2169,7 +2186,7 @@ static int reftable_be_create_reflog(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"create_reflog\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct write_reflog_existence_arg arg = {\n \t\t.refs = refs,\n \t\t.stack = stack,\n@@ -2243,7 +2260,7 @@ static int reftable_be_delete_reflog(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"delete_reflog\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct write_reflog_delete_arg arg = {\n \t\t.stack = stack,\n \t\t.refname = refname,\n@@ -2352,7 +2369,7 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \t */\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"reflog_expire\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record *logs = NULL;\n \tstruct reftable_log_record *rewritten = NULL;\n \tstruct reftable_ref_record ref_record = {0};\n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506644","messageId":"bab837e3733a982973bb96eedca15d073089693a.1730792627.git.ps@pks.im","threadId":"62444","inReplyTo":"cover.1730792627.git.ps@pks.im","subject":"[PATCH v2 2/8] refs/reftable: handle reloading stacks in the reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-05T09:12:02Z","receivedAt":"2024-11-05T09:12:11Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When accessing a stack we almost always have to reload the stack before\nreading data from it. This is mostly because Git does not have a\nnotification mechanism for when underlying data has been changed, and\nthus we are forced to opportunistically reload the stack every single\ntime to account for any changes that may have happened concurrently.\n\nHandle the reload internally in `backend_for()`. For one this forces\ncallsites to think about whether or not they need to reload the stack.\nBut second this makes the logic to access stacks more self-contained by\nletting the `struct reftable_backend` manage themselves.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 150 +++++++++++++++++++++++++---------------\n 1 file changed, 93 insertions(+), 57 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 116cc5ec23..4a28dc8a9d 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -114,21 +114,25 @@ static struct reftable_ref_store *reftable_be_downcast(struct ref_store *ref_sto\n  * like `worktrees/$worktree/refs/heads/foo` as worktree stacks will store\n  * those references in their normalized form.\n  */\n-static struct reftable_backend *backend_for(struct reftable_ref_store *store,\n-\t\t\t\t\t    const char *refname,\n-\t\t\t\t\t    const char **rewritten_ref)\n+static int backend_for(struct reftable_backend **out,\n+\t\t       struct reftable_ref_store *store,\n+\t\t       const char *refname,\n+\t\t       const char **rewritten_ref,\n+\t\t       int reload)\n {\n+\tstruct reftable_backend *be;\n \tconst char *wtname;\n \tint wtname_len;\n \n-\tif (!refname)\n-\t\treturn &store->main_backend;\n+\tif (!refname) {\n+\t\tbe = &store->main_backend;\n+\t\tgoto out;\n+\t}\n \n \tswitch (parse_worktree_ref(refname, &wtname, &wtname_len, rewritten_ref)) {\n \tcase REF_WORKTREE_OTHER: {\n \t\tstatic struct strbuf wtname_buf = STRBUF_INIT;\n \t\tstruct strbuf wt_dir = STRBUF_INIT;\n-\t\tstruct reftable_backend *be;\n \n \t\t/*\n \t\t * We're using a static buffer here so that we don't need to\n@@ -162,7 +166,7 @@ static struct reftable_backend *backend_for(struct reftable_ref_store *store,\n \t\t}\n \n \t\tstrbuf_release(&wt_dir);\n-\t\treturn be;\n+\t\tgoto out;\n \t}\n \tcase REF_WORKTREE_CURRENT:\n \t\t/*\n@@ -170,14 +174,27 @@ static struct reftable_backend *backend_for(struct reftable_ref_store *store,\n \t\t * main worktree. We thus return the main stack in that case.\n \t\t */\n \t\tif (!store->worktree_backend.stack)\n-\t\t\treturn &store->main_backend;\n-\t\treturn &store->worktree_backend;\n+\t\t\tbe = &store->main_backend;\n+\t\telse\n+\t\t\tbe = &store->worktree_backend;\n+\t\tgoto out;\n \tcase REF_WORKTREE_MAIN:\n \tcase REF_WORKTREE_SHARED:\n-\t\treturn &store->main_backend;\n+\t\tbe = &store->main_backend;\n+\t\tgoto out;\n \tdefault:\n \t\tBUG(\"unhandled worktree reference type\");\n \t}\n+\n+out:\n+\tif (reload) {\n+\t\tint ret = reftable_stack_reload(be->stack);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n+\t*out = be;\n+\n+\treturn 0;\n }\n \n static int should_write_log(struct reftable_ref_store *refs, const char *refname)\n@@ -828,17 +845,17 @@ static int reftable_be_read_raw_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_raw_ref\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n+\tstruct reftable_backend *be;\n \tint ret;\n \n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret)\n \t\treturn ret;\n \n-\tret = read_ref_without_reload(refs, stack, refname, oid, referent, type);\n+\tret = read_ref_without_reload(refs, be->stack, refname, oid, referent, type);\n \tif (ret < 0)\n \t\treturn ret;\n \tif (ret > 0) {\n@@ -855,15 +872,15 @@ static int reftable_be_read_symbolic_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_symbolic_ref\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_ref_record ref = {0};\n+\tstruct reftable_backend *be;\n \tint ret;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret)\n \t\treturn ret;\n \n-\tret = reftable_stack_read_ref(stack, refname, &ref);\n+\tret = reftable_stack_read_ref(be->stack, refname, &ref);\n \tif (ret == 0 && ref.value_type == REFTABLE_REF_SYMREF)\n \t\tstrbuf_addstr(referent, ref.value.symref);\n \telse\n@@ -880,7 +897,7 @@ struct reftable_transaction_update {\n \n struct write_transaction_table_arg {\n \tstruct reftable_ref_store *refs;\n-\tstruct reftable_stack *stack;\n+\tstruct reftable_backend *be;\n \tstruct reftable_addition *addition;\n \tstruct reftable_transaction_update *updates;\n \tsize_t updates_nr;\n@@ -915,27 +932,31 @@ static int prepare_transaction_update(struct write_transaction_table_arg **out,\n \t\t\t\t      struct ref_update *update,\n \t\t\t\t      struct strbuf *err)\n {\n-\tstruct reftable_stack *stack = backend_for(refs, update->refname, NULL)->stack;\n \tstruct write_transaction_table_arg *arg = NULL;\n+\tstruct reftable_backend *be;\n \tsize_t i;\n \tint ret;\n \n+\tret = backend_for(&be, refs, update->refname, NULL, 0);\n+\tif (ret)\n+\t\treturn ret;\n+\n \t/*\n \t * Search for a preexisting stack update. If there is one then we add\n \t * the update to it, otherwise we set up a new stack update.\n \t */\n \tfor (i = 0; !arg && i < tx_data->args_nr; i++)\n-\t\tif (tx_data->args[i].stack == stack)\n+\t\tif (tx_data->args[i].be == be)\n \t\t\targ = &tx_data->args[i];\n \n \tif (!arg) {\n \t\tstruct reftable_addition *addition;\n \n-\t\tret = reftable_stack_reload(stack);\n+\t\tret = backend_for(&be, refs, update->refname, NULL, 1);\n \t\tif (ret)\n \t\t\treturn ret;\n \n-\t\tret = reftable_stack_new_addition(&addition, stack,\n+\t\tret = reftable_stack_new_addition(&addition, be->stack,\n \t\t\t\t\t\t  REFTABLE_STACK_NEW_ADDITION_RELOAD);\n \t\tif (ret) {\n \t\t\tif (ret == REFTABLE_LOCK_ERROR)\n@@ -947,7 +968,7 @@ static int prepare_transaction_update(struct write_transaction_table_arg **out,\n \t\t\t   tx_data->args_alloc);\n \t\targ = &tx_data->args[tx_data->args_nr++];\n \t\targ->refs = refs;\n-\t\targ->stack = stack;\n+\t\targ->be = be;\n \t\targ->addition = addition;\n \t\targ->updates = NULL;\n \t\targ->updates_nr = 0;\n@@ -1002,6 +1023,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \tstruct strbuf referent = STRBUF_INIT, head_referent = STRBUF_INIT;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n \tstruct reftable_transaction_data *tx_data = NULL;\n+\tstruct reftable_backend *be;\n \tstruct object_id head_oid;\n \tunsigned int head_type = 0;\n \tsize_t i;\n@@ -1048,7 +1070,11 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\tgoto done;\n \t}\n \n-\tret = read_ref_without_reload(refs, backend_for(refs, \"HEAD\", NULL)->stack, \"HEAD\",\n+\tret = backend_for(&be, refs, \"HEAD\", NULL, 0);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = read_ref_without_reload(refs, be->stack, \"HEAD\",\n \t\t\t\t      &head_oid, &head_referent, &head_type);\n \tif (ret < 0)\n \t\tgoto done;\n@@ -1057,10 +1083,11 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \tfor (i = 0; i < transaction->nr; i++) {\n \t\tstruct ref_update *u = transaction->updates[i];\n \t\tstruct object_id current_oid = {0};\n-\t\tstruct reftable_stack *stack;\n \t\tconst char *rewritten_ref;\n \n-\t\tstack = backend_for(refs, u->refname, &rewritten_ref)->stack;\n+\t\tret = backend_for(&be, refs, u->refname, &rewritten_ref, 0);\n+\t\tif (ret)\n+\t\t\tgoto done;\n \n \t\t/* Verify that the new object ID is valid. */\n \t\tif ((u->flags & REF_HAVE_NEW) && !is_null_oid(&u->new_oid) &&\n@@ -1116,7 +1143,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t\tstring_list_insert(&affected_refnames, new_update->refname);\n \t\t}\n \n-\t\tret = read_ref_without_reload(refs, stack, rewritten_ref,\n+\t\tret = read_ref_without_reload(refs, be->stack, rewritten_ref,\n \t\t\t\t\t      &current_oid, &referent, &u->type);\n \t\tif (ret < 0)\n \t\t\tgoto done;\n@@ -1318,7 +1345,7 @@ static int transaction_update_cmp(const void *a, const void *b)\n static int write_transaction_table(struct reftable_writer *writer, void *cb_data)\n {\n \tstruct write_transaction_table_arg *arg = cb_data;\n-\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tuint64_t ts = reftable_stack_next_update_index(arg->be->stack);\n \tstruct reftable_log_record *logs = NULL;\n \tstruct ident_split committer_ident = {0};\n \tsize_t logs_nr = 0, logs_alloc = 0, i;\n@@ -1354,7 +1381,7 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n \t\t\tstruct reftable_log_record log = {0};\n \t\t\tstruct reftable_iterator it = {0};\n \n-\t\t\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\t\t\tret = reftable_stack_init_log_iterator(arg->be->stack, &it);\n \t\t\tif (ret < 0)\n \t\t\t\tgoto done;\n \n@@ -1799,10 +1826,9 @@ static int reftable_be_rename_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"rename_ref\");\n-\tstruct reftable_stack *stack = backend_for(refs, newrefname, &newrefname)->stack;\n+\tstruct reftable_backend *be;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n-\t\t.stack = stack,\n \t\t.oldname = oldrefname,\n \t\t.newname = newrefname,\n \t\t.logmsg = logmsg,\n@@ -1814,10 +1840,11 @@ static int reftable_be_rename_ref(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, newrefname, &newrefname, 1);\n \tif (ret)\n \t\tgoto done;\n-\tret = reftable_stack_add(stack, &write_copy_table, &arg);\n+\targ.stack = be->stack;\n+\tret = reftable_stack_add(be->stack, &write_copy_table, &arg);\n \n done:\n \tassert(ret != REFTABLE_API_ERROR);\n@@ -1831,10 +1858,9 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"copy_ref\");\n-\tstruct reftable_stack *stack = backend_for(refs, newrefname, &newrefname)->stack;\n+\tstruct reftable_backend *be;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n-\t\t.stack = stack,\n \t\t.oldname = oldrefname,\n \t\t.newname = newrefname,\n \t\t.logmsg = logmsg,\n@@ -1845,10 +1871,11 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, newrefname, &newrefname, 1);\n \tif (ret)\n \t\tgoto done;\n-\tret = reftable_stack_add(stack, &write_copy_table, &arg);\n+\targ.stack = be->stack;\n+\tret = reftable_stack_add(be->stack, &write_copy_table, &arg);\n \n done:\n \tassert(ret != REFTABLE_API_ERROR);\n@@ -2012,15 +2039,19 @@ static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent_reverse\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record log = {0};\n \tstruct reftable_iterator it = {0};\n+\tstruct reftable_backend *be;\n \tint ret;\n \n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tret = backend_for(&be, refs, refname, &refname, 0);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_init_log_iterator(be->stack, &it);\n \tif (ret < 0)\n \t\tgoto done;\n \n@@ -2052,16 +2083,20 @@ static int reftable_be_for_each_reflog_ent(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record *logs = NULL;\n \tstruct reftable_iterator it = {0};\n+\tstruct reftable_backend *be;\n \tsize_t logs_alloc = 0, logs_nr = 0, i;\n \tint ret;\n \n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tret = backend_for(&be, refs, refname, &refname, 0);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_init_log_iterator(be->stack, &it);\n \tif (ret < 0)\n \t\tgoto done;\n \n@@ -2101,20 +2136,20 @@ static int reftable_be_reflog_exists(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_exists\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record log = {0};\n \tstruct reftable_iterator it = {0};\n+\tstruct reftable_backend *be;\n \tint ret;\n \n \tret = refs->err;\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(be->stack, &it);\n \tif (ret < 0)\n \t\tgoto done;\n \n@@ -2186,10 +2221,9 @@ static int reftable_be_create_reflog(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"create_reflog\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n+\tstruct reftable_backend *be;\n \tstruct write_reflog_existence_arg arg = {\n \t\t.refs = refs,\n-\t\t.stack = stack,\n \t\t.refname = refname,\n \t};\n \tint ret;\n@@ -2198,11 +2232,12 @@ static int reftable_be_create_reflog(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret)\n \t\tgoto done;\n+\targ.stack = be->stack;\n \n-\tret = reftable_stack_add(stack, &write_reflog_existence_table, &arg);\n+\tret = reftable_stack_add(be->stack, &write_reflog_existence_table, &arg);\n \n done:\n \treturn ret;\n@@ -2260,17 +2295,18 @@ static int reftable_be_delete_reflog(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"delete_reflog\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n+\tstruct reftable_backend *be;\n \tstruct write_reflog_delete_arg arg = {\n-\t\t.stack = stack,\n \t\t.refname = refname,\n \t};\n \tint ret;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret)\n \t\treturn ret;\n-\tret = reftable_stack_add(stack, &write_reflog_delete_table, &arg);\n+\targ.stack = be->stack;\n+\n+\tret = reftable_stack_add(be->stack, &write_reflog_delete_table, &arg);\n \n \tassert(ret != REFTABLE_API_ERROR);\n \treturn ret;\n@@ -2369,13 +2405,13 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \t */\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"reflog_expire\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record *logs = NULL;\n \tstruct reftable_log_record *rewritten = NULL;\n \tstruct reftable_ref_record ref_record = {0};\n \tstruct reftable_iterator it = {0};\n \tstruct reftable_addition *add = NULL;\n \tstruct reflog_expiry_arg arg = {0};\n+\tstruct reftable_backend *be;\n \tstruct object_id oid = {0};\n \tuint8_t *last_hash = NULL;\n \tsize_t logs_nr = 0, logs_alloc = 0, i;\n@@ -2384,11 +2420,11 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(be->stack, &it);\n \tif (ret < 0)\n \t\tgoto done;\n \n@@ -2396,11 +2432,11 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_new_addition(&add, stack, 0);\n+\tret = reftable_stack_new_addition(&add, be->stack, 0);\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_read_ref(stack, refname, &ref_record);\n+\tret = reftable_stack_read_ref(be->stack, refname, &ref_record);\n \tif (ret < 0)\n \t\tgoto done;\n \tif (reftable_ref_record_val1(&ref_record))\n@@ -2479,7 +2515,7 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \targ.refs = refs;\n \targ.records = rewritten;\n \targ.len = logs_nr;\n-\targ.stack = stack,\n+\targ.stack = be->stack,\n \targ.refname = refname,\n \n \tret = reftable_addition_add(add, &write_reflog_expiry_table, &arg);\n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506645","messageId":"1b50655202f311c6a6ded61d4d50b1f287761d84.1730792627.git.ps@pks.im","threadId":"62444","inReplyTo":"cover.1730792627.git.ps@pks.im","subject":"[PATCH v2 3/8] refs/reftable: read references via `struct reftable_backend`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-05T09:12:04Z","receivedAt":"2024-11-05T09:12:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor `read_ref_without_reload()` to accept a `struct reftable_stack`\nas input instead of accepting a `struct reftable_stack`. This allows us\nto implement an additional caching layer when reading refs where we can\nreuse reftable iterators.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c   | 110 ++++++++++++++++++++------------------\n reftable/reftable-stack.h |   3 ++\n reftable/stack.c          |   5 ++\n 3 files changed, 67 insertions(+), 51 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 4a28dc8a9d..230adb690d 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -51,6 +51,50 @@ static void reftable_backend_release(struct reftable_backend *be)\n \tbe->stack = NULL;\n }\n \n+static int reftable_backend_read_ref(struct reftable_backend *be,\n+\t\t\t\t     const char *refname,\n+\t\t\t\t     struct object_id *oid,\n+\t\t\t\t     struct strbuf *referent,\n+\t\t\t\t     unsigned int *type)\n+{\n+\tstruct reftable_ref_record ref = {0};\n+\tint ret;\n+\n+\tret = reftable_stack_read_ref(be->stack, refname, &ref);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tstrbuf_reset(referent);\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\t\t*type |= REF_ISSYMREF;\n+\t} else if (reftable_ref_record_val1(&ref)) {\n+\t\tunsigned int hash_id;\n+\n+\t\tswitch (reftable_stack_hash_id(be->stack)) {\n+\t\tcase REFTABLE_HASH_SHA1:\n+\t\t\thash_id = GIT_HASH_SHA1;\n+\t\t\tbreak;\n+\t\tcase REFTABLE_HASH_SHA256:\n+\t\t\thash_id = GIT_HASH_SHA256;\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tBUG(\"unhandled hash ID %d\", reftable_stack_hash_id(be->stack));\n+\t\t}\n+\n+\t\toidread(oid, reftable_ref_record_val1(&ref),\n+\t\t\t&hash_algos[hash_id]);\n+\t} else {\n+\t\t/* We got a tombstone, which should not happen. */\n+\t\tBUG(\"unhandled reference value type %d\", ref.value_type);\n+\t}\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn ret;\n+}\n+\n struct reftable_ref_store {\n \tstruct ref_store base;\n \n@@ -243,38 +287,6 @@ static void fill_reftable_log_record(struct reftable_log_record *log, const stru\n \tlog->value.update.tz_offset = sign * atoi(tz_begin);\n }\n \n-static int read_ref_without_reload(struct reftable_ref_store *refs,\n-\t\t\t\t   struct reftable_stack *stack,\n-\t\t\t\t   const char *refname,\n-\t\t\t\t   struct object_id *oid,\n-\t\t\t\t   struct strbuf *referent,\n-\t\t\t\t   unsigned int *type)\n-{\n-\tstruct reftable_ref_record ref = {0};\n-\tint ret;\n-\n-\tret = reftable_stack_read_ref(stack, refname, &ref);\n-\tif (ret)\n-\t\tgoto done;\n-\n-\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n-\t\tstrbuf_reset(referent);\n-\t\tstrbuf_addstr(referent, ref.value.symref);\n-\t\t*type |= REF_ISSYMREF;\n-\t} else if (reftable_ref_record_val1(&ref)) {\n-\t\toidread(oid, reftable_ref_record_val1(&ref),\n-\t\t\trefs->base.repo->hash_algo);\n-\t} else {\n-\t\t/* We got a tombstone, which should not happen. */\n-\t\tBUG(\"unhandled reference value type %d\", ref.value_type);\n-\t}\n-\n-done:\n-\tassert(ret != REFTABLE_API_ERROR);\n-\treftable_ref_record_release(&ref);\n-\treturn ret;\n-}\n-\n static int reftable_be_config(const char *var, const char *value,\n \t\t\t      const struct config_context *ctx,\n \t\t\t      void *_opts)\n@@ -855,7 +867,7 @@ static int reftable_be_read_raw_ref(struct ref_store *ref_store,\n \tif (ret)\n \t\treturn ret;\n \n-\tret = read_ref_without_reload(refs, be->stack, refname, oid, referent, type);\n+\tret = reftable_backend_read_ref(be, refname, oid, referent, type);\n \tif (ret < 0)\n \t\treturn ret;\n \tif (ret > 0) {\n@@ -1074,8 +1086,8 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \tif (ret)\n \t\tgoto done;\n \n-\tret = read_ref_without_reload(refs, be->stack, \"HEAD\",\n-\t\t\t\t      &head_oid, &head_referent, &head_type);\n+\tret = reftable_backend_read_ref(be, \"HEAD\", &head_oid,\n+\t\t\t\t\t&head_referent, &head_type);\n \tif (ret < 0)\n \t\tgoto done;\n \tret = 0;\n@@ -1143,8 +1155,8 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t\tstring_list_insert(&affected_refnames, new_update->refname);\n \t\t}\n \n-\t\tret = read_ref_without_reload(refs, be->stack, rewritten_ref,\n-\t\t\t\t\t      &current_oid, &referent, &u->type);\n+\t\tret = reftable_backend_read_ref(be, rewritten_ref,\n+\t\t\t\t\t\t&current_oid, &referent, &u->type);\n \t\tif (ret < 0)\n \t\t\tgoto done;\n \t\tif (ret > 0 && !ref_update_expects_existing_old_ref(u)) {\n@@ -1602,7 +1614,7 @@ struct write_create_symref_arg {\n \n struct write_copy_arg {\n \tstruct reftable_ref_store *refs;\n-\tstruct reftable_stack *stack;\n+\tstruct reftable_backend *be;\n \tconst char *oldname;\n \tconst char *newname;\n \tconst char *logmsg;\n@@ -1627,7 +1639,7 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n \tif (split_ident_line(&committer_ident, committer_info, strlen(committer_info)))\n \t\tBUG(\"failed splitting committer info\");\n \n-\tif (reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref)) {\n+\tif (reftable_stack_read_ref(arg->be->stack, arg->oldname, &old_ref)) {\n \t\tret = error(_(\"refname %s not found\"), arg->oldname);\n \t\tgoto done;\n \t}\n@@ -1666,7 +1678,7 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n \t * the old branch and the creation of the new branch, and we cannot do\n \t * two changes to a reflog in a single update.\n \t */\n-\tdeletion_ts = creation_ts = reftable_stack_next_update_index(arg->stack);\n+\tdeletion_ts = creation_ts = reftable_stack_next_update_index(arg->be->stack);\n \tif (arg->delete_old)\n \t\tcreation_ts++;\n \treftable_writer_set_limits(writer, deletion_ts, creation_ts);\n@@ -1709,8 +1721,8 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n \t\tmemcpy(logs[logs_nr].value.update.old_hash, old_ref.value.val1, GIT_MAX_RAWSZ);\n \t\tlogs_nr++;\n \n-\t\tret = read_ref_without_reload(arg->refs, arg->stack, \"HEAD\", &head_oid,\n-\t\t\t\t\t      &head_referent, &head_type);\n+\t\tret = reftable_backend_read_ref(arg->be, \"HEAD\", &head_oid,\n+\t\t\t\t\t\t&head_referent, &head_type);\n \t\tif (ret < 0)\n \t\t\tgoto done;\n \t\tappend_head_reflog = (head_type & REF_ISSYMREF) && !strcmp(head_referent.buf, arg->oldname);\n@@ -1753,7 +1765,7 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n \t * copy over all log entries from the old reflog. Last but not least,\n \t * when renaming we also have to delete all the old reflog entries.\n \t */\n-\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\tret = reftable_stack_init_log_iterator(arg->be->stack, &it);\n \tif (ret < 0)\n \t\tgoto done;\n \n@@ -1826,7 +1838,6 @@ static int reftable_be_rename_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"rename_ref\");\n-\tstruct reftable_backend *be;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n \t\t.oldname = oldrefname,\n@@ -1840,11 +1851,10 @@ static int reftable_be_rename_ref(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = backend_for(&be, refs, newrefname, &newrefname, 1);\n+\tret = backend_for(&arg.be, refs, newrefname, &newrefname, 1);\n \tif (ret)\n \t\tgoto done;\n-\targ.stack = be->stack;\n-\tret = reftable_stack_add(be->stack, &write_copy_table, &arg);\n+\tret = reftable_stack_add(arg.be->stack, &write_copy_table, &arg);\n \n done:\n \tassert(ret != REFTABLE_API_ERROR);\n@@ -1858,7 +1868,6 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"copy_ref\");\n-\tstruct reftable_backend *be;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n \t\t.oldname = oldrefname,\n@@ -1871,11 +1880,10 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = backend_for(&be, refs, newrefname, &newrefname, 1);\n+\tret = backend_for(&arg.be, refs, newrefname, &newrefname, 1);\n \tif (ret)\n \t\tgoto done;\n-\targ.stack = be->stack;\n-\tret = reftable_stack_add(be->stack, &write_copy_table, &arg);\n+\tret = reftable_stack_add(arg.be->stack, &write_copy_table, &arg);\n \n done:\n \tassert(ret != REFTABLE_API_ERROR);\ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nindex 54787f2ef5..ae14270ea7 100644\n--- a/reftable/reftable-stack.h\n+++ b/reftable/reftable-stack.h\n@@ -149,4 +149,7 @@ struct reftable_compaction_stats {\n struct reftable_compaction_stats *\n reftable_stack_compaction_stats(struct reftable_stack *st);\n \n+/* Return the hash of the stack. */\n+enum reftable_hash reftable_stack_hash_id(struct reftable_stack *st);\n+\n #endif\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 1fffd75630..d97b64a40d 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1791,3 +1791,8 @@ int reftable_stack_clean(struct reftable_stack *st)\n \treftable_addition_destroy(add);\n \treturn err;\n }\n+\n+enum reftable_hash reftable_stack_hash_id(struct reftable_stack *st)\n+{\n+\treturn reftable_merged_table_hash_id(st->merged);\n+}\n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506646","messageId":"0906b04fc6da205cf67cca4d0dedac6f0d0adbc0.1730792627.git.ps@pks.im","threadId":"62444","inReplyTo":"cover.1730792627.git.ps@pks.im","subject":"[PATCH v2 4/8] refs/reftable: refactor reading symbolic refs to use reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-05T09:12:10Z","receivedAt":"2024-11-05T09:12:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor the callback function that reads symbolic references in the\nreftable backend to use `reftable_backend_read_ref()` instead of\naccessing the reftable stack directly. This ensures that the function\nwill benefit from the new caching layer that we're about to introduce.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 11 ++++-------\n 1 file changed, 4 insertions(+), 7 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 230adb690d..dfe94ff969 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -884,21 +884,18 @@ static int reftable_be_read_symbolic_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_symbolic_ref\");\n-\tstruct reftable_ref_record ref = {0};\n \tstruct reftable_backend *be;\n+\tstruct object_id oid;\n+\tunsigned int type = 0;\n \tint ret;\n \n \tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret)\n \t\treturn ret;\n \n-\tret = reftable_stack_read_ref(be->stack, refname, &ref);\n-\tif (ret == 0 && ref.value_type == REFTABLE_REF_SYMREF)\n-\t\tstrbuf_addstr(referent, ref.value.symref);\n-\telse\n+\tret = reftable_backend_read_ref(be, refname, &oid, referent, &type);\n+\tif (type != REF_ISSYMREF)\n \t\tret = -1;\n-\n-\treftable_ref_record_release(&ref);\n \treturn ret;\n }\n \n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506647","messageId":"355557ec951372a16b70558706981867a5206c87.1730792627.git.ps@pks.im","threadId":"62444","inReplyTo":"cover.1730792627.git.ps@pks.im","subject":"[PATCH v2 5/8] refs/reftable: refactor reflog expiry to use reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-05T09:12:13Z","receivedAt":"2024-11-05T09:12:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor the callback function that expires reflog entries in the\nreftable backend to use `reftable_backend_read_ref()` instead of\naccessing the reftable stack directly. This ensures that the function\nwill benefit from the new caching layer that we're about to introduce.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 13 +++++--------\n 1 file changed, 5 insertions(+), 8 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex dfe94ff969..9c6e9c8374 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2412,14 +2412,15 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"reflog_expire\");\n \tstruct reftable_log_record *logs = NULL;\n \tstruct reftable_log_record *rewritten = NULL;\n-\tstruct reftable_ref_record ref_record = {0};\n \tstruct reftable_iterator it = {0};\n \tstruct reftable_addition *add = NULL;\n \tstruct reflog_expiry_arg arg = {0};\n \tstruct reftable_backend *be;\n \tstruct object_id oid = {0};\n+\tstruct strbuf referent = STRBUF_INIT;\n \tuint8_t *last_hash = NULL;\n \tsize_t logs_nr = 0, logs_alloc = 0, i;\n+\tunsigned int type = 0;\n \tint ret;\n \n \tif (refs->err < 0)\n@@ -2441,12 +2442,9 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_read_ref(be->stack, refname, &ref_record);\n+\tret = reftable_backend_read_ref(be, refname, &oid, &referent, &type);\n \tif (ret < 0)\n \t\tgoto done;\n-\tif (reftable_ref_record_val1(&ref_record))\n-\t\toidread(&oid, reftable_ref_record_val1(&ref_record),\n-\t\t\tref_store->repo->hash_algo);\n \tprepare_fn(refname, &oid, policy_cb_data);\n \n \twhile (1) {\n@@ -2513,8 +2511,7 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \t\t}\n \t}\n \n-\tif (flags & EXPIRE_REFLOGS_UPDATE_REF && last_hash &&\n-\t    reftable_ref_record_val1(&ref_record))\n+\tif (flags & EXPIRE_REFLOGS_UPDATE_REF && last_hash && !is_null_oid(&oid))\n \t\toidread(&arg.update_oid, last_hash, ref_store->repo->hash_algo);\n \n \targ.refs = refs;\n@@ -2539,11 +2536,11 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \t\tcleanup_fn(policy_cb_data);\n \tassert(ret != REFTABLE_API_ERROR);\n \n-\treftable_ref_record_release(&ref_record);\n \treftable_iterator_destroy(&it);\n \treftable_addition_destroy(add);\n \tfor (i = 0; i < logs_nr; i++)\n \t\treftable_log_record_release(&logs[i]);\n+\tstrbuf_release(&referent);\n \tfree(logs);\n \tfree(rewritten);\n \treturn ret;\n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506648","messageId":"71ad6c80b0218cb61394fcaea8618ce888c505ab.1730792627.git.ps@pks.im","threadId":"62444","inReplyTo":"cover.1730792627.git.ps@pks.im","subject":"[PATCH v2 6/8] reftable/stack: add mechanism to notify callers on reload","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-05T09:12:15Z","receivedAt":"2024-11-05T09:12:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Reftable stacks are reloaded in two cases:\n\n  - When calling `reftable_stack_reload()`, if the stat-cache tells us\n    that the stack has been modified.\n\n  - When committing a reftable addition.\n\nWhile callers can figure out the second case, they do not have a\nmechanism to figure out whether `reftable_stack_reload()` led to an\nactual reload of the on-disk data. All they can do is thus to assume\nthat data is always being reloaded in that case.\n\nImprove the situation by introducing a new `on_reload()` callback to the\nreftable options. If provided, the function will be invoked every time\nthe stack has indeed been reloaded. This allows callers to invalidate\ndata that depends on the current stack data.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n reftable/reftable-writer.h | 9 +++++++++\n reftable/stack.c           | 4 ++++\n 2 files changed, 13 insertions(+)\n\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex c85ef5a5bd..5f9afa620b 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -68,6 +68,15 @@ struct reftable_write_options {\n \t * fsync(3P) when unset.\n \t */\n \tint (*fsync)(int fd);\n+\n+\t/*\n+\t * Callback function to execute whenever the stack is being reloaded.\n+\t * This can be used e.g. to discard cached information that relies on\n+\t * the old stack's data. The payload data will be passed as argument to\n+\t * the callback.\n+\t */\n+\tvoid (*on_reload)(void *payload);\n+\tvoid *on_reload_payload;\n };\n \n /* reftable_block_stats holds statistics for a single block type */\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex d97b64a40d..5384ca9de0 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -548,6 +548,10 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n \t\tclose(fd);\n \tfree_names(names);\n \tfree_names(names_after);\n+\n+\tif (st->opts.on_reload)\n+\t\tst->opts.on_reload(st->opts.on_reload_payload);\n+\n \treturn err;\n }\n \n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506649","messageId":"93efd118864c96b51eb84e46d1eff358eb007a3a.1730792627.git.ps@pks.im","threadId":"62444","inReplyTo":"cover.1730792627.git.ps@pks.im","subject":"[PATCH v2 7/8] reftable/merged: drain priority queue on reseek","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-05T09:12:18Z","receivedAt":"2024-11-05T09:12:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In 5bf96e0c39 (reftable/generic: move seeking of records into the\niterator, 2024-05-13) we have refactored the reftable codebase such that\niterators can be initialized once and then re-seeked multiple times.\nThis feature is used by 1869525066 (refs/reftable: wire up support for\nexclude patterns, 2024-09-16) in order to skip records based on exclude\npatterns provided by the caller.\n\nThe logic to re-seek the merged iterator is insufficient though because\nwe don't drain the priority queue on a re-seek. This means that the\nqueue may contain stale entries and thus reading the next record in the\nqueue will return the wrong entry. While this is an obvious bug, it is\nharmless in the context of above exclude patterns:\n\n  - If the queue contained stale entries that match the pattern then the\n    caller would already know to filter out such refs. This is because\n    our codebase is prepared to handle backends that don't have a way to\n    efficiently implement exclude patterns.\n\n  - If the queue contained stale entries that don't match the pattern\n    we'd eventually filter out any duplicates. This is because the\n    reftable code discards items with the same ref name and sorts any\n    remaining entries properly.\n\nSo things happen to work in this context regardless of the bug, and\nthere is no other use case yet where we re-seek iterators. We're about\nto introduce a caching mechanism though where iterators are reused by\nthe reftable backend, and that will expose the bug.\n\nFix the issue by draining the priority queue when seeking and add a\ntestcase that surfaces the issue.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n reftable/merged.c                |  2 +\n t/unit-tests/t-reftable-merged.c | 73 ++++++++++++++++++++++++++++++++\n 2 files changed, 75 insertions(+)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 5b93e20f42..bb0836e344 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -66,6 +66,8 @@ static int merged_iter_seek(struct merged_iter *mi, struct reftable_record *want\n \tint err;\n \n \tmi->advance_index = -1;\n+\twhile (!merged_iter_pqueue_is_empty(mi->pq))\n+\t\tmerged_iter_pqueue_remove(&mi->pq);\n \n \tfor (size_t i = 0; i < mi->subiters_len; i++) {\n \t\terr = iterator_seek(&mi->subiters[i].iter, want);\ndiff --git a/t/unit-tests/t-reftable-merged.c b/t/unit-tests/t-reftable-merged.c\nindex 2591b5e597..a12bd0e1a3 100644\n--- a/t/unit-tests/t-reftable-merged.c\n+++ b/t/unit-tests/t-reftable-merged.c\n@@ -273,6 +273,78 @@ static void t_merged_seek_multiple_times(void)\n \treftable_free(sources);\n }\n \n+static void t_merged_seek_multiple_times_without_draining(void)\n+{\n+\tstruct reftable_ref_record r1[] = {\n+\t\t{\n+\t\t\t.refname = (char *) \"a\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = { 1 },\n+\t\t},\n+\t\t{\n+\t\t\t.refname = (char *) \"c\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = { 2 },\n+\t\t}\n+\t};\n+\tstruct reftable_ref_record r2[] = {\n+\t\t{\n+\t\t\t.refname = (char *) \"b\",\n+\t\t\t.update_index = 2,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = { 3 },\n+\t\t},\n+\t\t{\n+\t\t\t.refname = (char *) \"d\",\n+\t\t\t.update_index = 2,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = { 4 },\n+\t\t},\n+\t};\n+\tstruct reftable_ref_record *refs[] = {\n+\t\tr1, r2,\n+\t};\n+\tsize_t sizes[] = {\n+\t\tARRAY_SIZE(r1), ARRAY_SIZE(r2),\n+\t};\n+\tstruct reftable_buf bufs[] = {\n+\t\tREFTABLE_BUF_INIT, REFTABLE_BUF_INIT,\n+\t};\n+\tstruct reftable_block_source *sources = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_ref_record rec = { 0 };\n+\tstruct reftable_iterator it = { 0 };\n+\tstruct reftable_merged_table *mt;\n+\tint err;\n+\n+\tmt = merged_table_from_records(refs, &sources, &readers, sizes, bufs, 2);\n+\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\n+\terr = reftable_iterator_seek_ref(&it, \"b\");\n+\tcheck(!err);\n+\terr = reftable_iterator_next_ref(&it, &rec);\n+\tcheck(!err);\n+\terr = reftable_ref_record_equal(&rec, &r2[0], REFTABLE_HASH_SIZE_SHA1);\n+\tcheck(err == 1);\n+\n+\terr = reftable_iterator_seek_ref(&it, \"a\");\n+\tcheck(!err);\n+\terr = reftable_iterator_next_ref(&it, &rec);\n+\tcheck(!err);\n+\terr = reftable_ref_record_equal(&rec, &r1[0], REFTABLE_HASH_SIZE_SHA1);\n+\tcheck(err == 1);\n+\n+\tfor (size_t i = 0; i < ARRAY_SIZE(bufs); i++)\n+\t\treftable_buf_release(&bufs[i]);\n+\treaders_destroy(readers, ARRAY_SIZE(refs));\n+\treftable_ref_record_release(&rec);\n+\treftable_iterator_destroy(&it);\n+\treftable_merged_table_free(mt);\n+\treftable_free(sources);\n+}\n+\n static struct reftable_merged_table *\n merged_table_from_log_records(struct reftable_log_record **logs,\n \t\t\t      struct reftable_block_source **source,\n@@ -467,6 +539,7 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \tTEST(t_merged_logs(), \"merged table with multiple log updates for same ref\");\n \tTEST(t_merged_refs(), \"merged table with multiple updates to same ref\");\n \tTEST(t_merged_seek_multiple_times(), \"merged table can seek multiple times\");\n+\tTEST(t_merged_seek_multiple_times_without_draining(), \"merged table can seek multiple times without draining\");\n \tTEST(t_merged_single_record(), \"ref occurring in only one record can be fetched\");\n \n \treturn test_done();\n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506650","messageId":"276c27e770ace7030bdd83106c05f4fe5b07dbe5.1730792627.git.ps@pks.im","threadId":"62444","inReplyTo":"cover.1730792627.git.ps@pks.im","subject":"[PATCH v2 8/8] refs/reftable: reuse iterators when reading refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-05T09:12:20Z","receivedAt":"2024-11-05T09:12:29Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When reading references the reftable backend has to:\n\n  1. Create a new ref iterator.\n\n  2. Seek the iterator to the record we're searching for.\n\n  3. Read the record.\n\nWe cannot really avoid the last two steps, but re-creating the iterator\nevery single time we want to read a reference is kind of expensive and a\nwaste of resources. We couldn't help it in the past though because it\nwas not possible to reuse iterators. But starting with 5bf96e0c39\n(reftable/generic: move seeking of records into the iterator,\n2024-05-13) we have split up the iterator lifecycle such that creating\nthe iterator and seeking are two different concerns.\n\nRefactor the code such that we cache iterators in the reftable backend.\nThis cache is invalidated whenever the respective stack is reloaded such\nthat we know to recreate the iterator in that case. This leads to a\nsizeable speedup when creating many refs, which requires a lot of random\nreference reads:\n\n    Benchmark 1: update-ref: create many refs (refcount = 100000, revision = master)\n      Time (mean ± σ):      1.793 s ±  0.010 s    [User: 0.954 s, System: 0.835 s]\n      Range (min … max):    1.781 s …  1.811 s    10 runs\n\n    Benchmark 2: update-ref: create many refs (refcount = 100000, revision = HEAD)\n      Time (mean ± σ):      1.680 s ±  0.013 s    [User: 0.846 s, System: 0.831 s]\n      Range (min … max):    1.664 s …  1.702 s    10 runs\n\n    Summary\n      update-ref: create many refs (refcount = 100000, revision = HEAD) ran\n        1.07 ± 0.01 times faster than update-ref: create many refs (refcount = 100000, revision = master)\n\nWhile 7% is not a huge win, you have to consider that the benchmark is\n_writing_ data, so _reading_ references is only one part of what we do.\nFlame graphs show that we spend around 40% of our time reading refs, so\nthe speedup when reading refs is approximately ~2.5x that. I could not\nfind better benchmarks where we perform a lot of random ref reads.\n\nYou can also see a sizeable impact on memory usage when creating 100k\nreferences. Before this change:\n\n    HEAP SUMMARY:\n        in use at exit: 19,112,538 bytes in 200,170 blocks\n      total heap usage: 8,400,426 allocs, 8,200,256 frees, 454,367,048 bytes allocated\n\nAfter this change:\n\n    HEAP SUMMARY:\n        in use at exit: 674,416 bytes in 169 blocks\n      total heap usage: 7,929,872 allocs, 7,929,703 frees, 281,509,985 bytes allocated\n\nAs an additional factor, this refactoring opens up the possibility for\nmore performance optimizations in how we re-seek iterators. Any change\nthat allows us to optimize re-seeking by e.g. reusing data structures\nwould thus also directly speed up random reads.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 32 +++++++++++++++++++++++++++++---\n 1 file changed, 29 insertions(+), 3 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 9c6e9c8374..4942363712 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -36,19 +36,30 @@\n \n struct reftable_backend {\n \tstruct reftable_stack *stack;\n+\tstruct reftable_iterator it;\n };\n \n+static void reftable_backend_on_reload(void *payload)\n+{\n+\tstruct reftable_backend *be = payload;\n+\treftable_iterator_destroy(&be->it);\n+}\n+\n static int reftable_backend_init(struct reftable_backend *be,\n \t\t\t\t const char *path,\n-\t\t\t\t const struct reftable_write_options *opts)\n+\t\t\t\t const struct reftable_write_options *_opts)\n {\n-\treturn reftable_new_stack(&be->stack, path, opts);\n+\tstruct reftable_write_options opts = *_opts;\n+\topts.on_reload = reftable_backend_on_reload;\n+\topts.on_reload_payload = be;\n+\treturn reftable_new_stack(&be->stack, path, &opts);\n }\n \n static void reftable_backend_release(struct reftable_backend *be)\n {\n \treftable_stack_destroy(be->stack);\n \tbe->stack = NULL;\n+\treftable_iterator_destroy(&be->it);\n }\n \n static int reftable_backend_read_ref(struct reftable_backend *be,\n@@ -60,10 +71,25 @@ static int reftable_backend_read_ref(struct reftable_backend *be,\n \tstruct reftable_ref_record ref = {0};\n \tint ret;\n \n-\tret = reftable_stack_read_ref(be->stack, refname, &ref);\n+\tif (!be->it.ops) {\n+\t\tret = reftable_stack_init_ref_iterator(be->stack, &be->it);\n+\t\tif (ret)\n+\t\t\tgoto done;\n+\t}\n+\n+\tret = reftable_iterator_seek_ref(&be->it, refname);\n \tif (ret)\n \t\tgoto done;\n \n+\tret = reftable_iterator_next_ref(&be->it, &ref);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tif (strcmp(ref.refname, refname)) {\n+\t\tret = 1;\n+\t\tgoto done;\n+\t}\n+\n \tif (ref.value_type == REFTABLE_REF_SYMREF) {\n \t\tstrbuf_reset(referent);\n \t\tstrbuf_addstr(referent, ref.value.symref);\n-- \n2.47.0.229.g8f8d6eee53.dirty\n\n"},{"id":"506655","messageId":"CAOLa=ZR89CRuJ7nwBfbGhi-wFuwEc96qNoid8sNKHD5ssmrOXg@mail.gmail.com","threadId":"62444","inReplyTo":"b599bcdac191fd3cfb7797accebd79211e757c47.1730732881.git.ps@pks.im","subject":"Re: [PATCH 1/8] refs/reftable: encapsulate reftable stack","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-11-05T11:03:00Z","receivedAt":"2024-11-05T11:03:02Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> The reftable ref store needs to keep track of multiple stacks, one for\n> the main worktree and an arbitrary number of stacks for worktrees. This\n> is done by storing pointers to `struct reftable_stack`, which we then\n> access directly.\n>\n> Wrap the stack in a new `struct reftable_backend`. This will allow us to\n> attach more data to each respective stack in subsequent commits.\n>\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  refs/reftable-backend.c | 129 +++++++++++++++++++++++-----------------\n>  1 file changed, 73 insertions(+), 56 deletions(-)\n>\n> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> index 38eb14d591..43cba53cb1 100644\n> --- a/refs/reftable-backend.c\n> +++ b/refs/reftable-backend.c\n> @@ -32,6 +32,23 @@\n>   */\n>  #define REF_UPDATE_VIA_HEAD (1 << 8)\n>\n> +struct reftable_backend {\n> +\tstruct reftable_stack *stack;\n> +};\n> +\n> +static int reftable_backend_init(struct reftable_backend *be,\n> +\t\t\t\t const char *path,\n> +\t\t\t\t const struct reftable_write_options *opts)\n> +{\n> +\treturn reftable_new_stack(&be->stack, path, opts);\n> +}\n> +\n> +static void reftable_backend_release(struct reftable_backend *be)\n> +{\n> +\treftable_stack_destroy(be->stack);\n> +\tbe->stack = NULL;\n> +}\n> +\n>  struct reftable_ref_store {\n>  \tstruct ref_store base;\n>\n> @@ -39,17 +56,17 @@ struct reftable_ref_store {\n>  \t * The main stack refers to the common dir and thus contains common\n>  \t * refs as well as refs of the main repository.\n>  \t */\n\nShouldn't these comments be updated to say s/stack/backend, while the\nbackend contains the stack, it is confusing to read stack and see\nbackend.\n\n> -\tstruct reftable_stack *main_stack;\n> +\tstruct reftable_backend main_backend;\n>  \t/*\n>  \t * The worktree stack refers to the gitdir in case the refdb is opened\n>  \t * via a worktree. It thus contains the per-worktree refs.\n>  \t */\n\nHere too.\n\n> -\tstruct reftable_stack *worktree_stack;\n> +\tstruct reftable_backend worktree_backend;\n>  \t/*\n>  \t * Map of worktree stacks by their respective worktree names. The map\n>  \t * is populated lazily when we try to resolve `worktrees/$worktree` refs.\n>  \t */\n\nHere too.\n\n> -\tstruct strmap worktree_stacks;\n> +\tstruct strmap worktree_backends;\n>  \tstruct reftable_write_options write_options;\n>\n>  \tunsigned int store_flags;\n\n[snip]\n\n> @@ -772,14 +789,14 @@ static struct ref_iterator *reftable_be_iterator_begin(struct ref_store *ref_sto\n>  \t * right now. If we aren't, then we return the common reftable\n>  \t * iterator, only.\n>  \t */\n> -\t if (!refs->worktree_stack)\n> +\t if (!refs->worktree_backend.stack)\n\nNit: Not your fault, but this is misaligned no?\n\n>  \t\treturn &main_iter->base;\n>\n\n[snip]\n"},{"id":"506657","messageId":"CAOLa=ZQwUGzjmvCv8TKuwbeePTG6+HNjU3ngO6hD8=xSOQ5kaQ@mail.gmail.com","threadId":"62444","inReplyTo":"b81ce63589e0380baf2b977a6abd706a66b08bee.1730732881.git.ps@pks.im","subject":"Re: [PATCH 2/8] refs/reftable: handle reloading stacks in the reftable backend","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-11-05T11:14:15Z","receivedAt":"2024-11-05T11:14:17Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n[snip]\n\n> @@ -898,27 +915,31 @@ static int prepare_transaction_update(struct write_transaction_table_arg **out,\n>  \t\t\t\t      struct ref_update *update,\n>  \t\t\t\t      struct strbuf *err)\n>  {\n> -\tstruct reftable_stack *stack = backend_for(refs, update->refname, NULL)->stack;\n>  \tstruct write_transaction_table_arg *arg = NULL;\n> +\tstruct reftable_backend *be;\n>  \tsize_t i;\n>  \tint ret;\n>\n> +\tret = backend_for(&be, refs, update->refname, NULL, 0);\n\nSo here, we don't reload the stack, it would be nice to add a comment\nwhy, in the places we don't. Here, it seems to be because we possibly\nalready have an update which would have pushed us to reload the stack.\n\n> +\tif (ret)\n> +\t\treturn ret;\n> +\n\n[snip]\n\n> @@ -1995,15 +2022,19 @@ static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n>  {\n>  \tstruct reftable_ref_store *refs =\n>  \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent_reverse\");\n> -\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n>  \tstruct reftable_log_record log = {0};\n>  \tstruct reftable_iterator it = {0};\n> +\tstruct reftable_backend *be;\n>  \tint ret;\n>\n>  \tif (refs->err < 0)\n>  \t\treturn refs->err;\n>\n> -\tret = reftable_stack_init_log_iterator(stack, &it);\n> +\tret = backend_for(&be, refs, refname, &refname, 0);\n\nLike here, I'm not entirely sure why we don't reload the stack.\n\n> +\tif (ret)\n> +\t\tgoto done;\n> +\n> +\tret = reftable_stack_init_log_iterator(be->stack, &it);\n>  \tif (ret < 0)\n>  \t\tgoto done;\n>\n\n[snip]\n\n> @@ -2462,7 +2498,7 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n>  \targ.refs = refs;\n>  \targ.records = rewritten;\n>  \targ.len = logs_nr;\n> -\targ.stack = stack,\n> +\targ.stack = be->stack,\n>  \targ.refname = refname,\n>\n\nShouldn't these lines end with ';'?\n\n>  \tret = reftable_addition_add(add, &write_reflog_expiry_table, &arg);\n> --\n> 2.47.0.229.g8f8d6eee53.dirty\n"},{"id":"506658","messageId":"CAOLa=ZRO_F93DDZefForD4Jx-ReXDKBnHeL8yOkHwPNtJ6S_bQ@mail.gmail.com","threadId":"62444","inReplyTo":"00fdf392a64f2b0f2e9bb7b398565a87c2e187b5.1730732881.git.ps@pks.im","subject":"Re: [PATCH 3/8] refs/reftable: read references via `struct reftable_backend`","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-11-05T11:20:19Z","receivedAt":"2024-11-05T11:20:22Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> Refactor `read_ref_without_reload()` to accept a `struct reftable_stack`\n\nDo you mean `struct reftable_backend` here?\n\n> as input instead of accepting a `struct reftable_stack`. This allows us\n> to implement an additional caching layer when reading refs where we can\n> reuse reftable iterators.\n>\n\nNit: This commit also refactors the function to use the newly defined\n'REFTABLE_HASH_*'. Maybe worth mentioning in the commit.\n\nThe patch itself looks good!\n\n[snip]\n"},{"id":"506707","messageId":"ZytISDjXYp41caZW@pks.im","threadId":"62444","inReplyTo":"CAOLa=ZQwUGzjmvCv8TKuwbeePTG6+HNjU3ngO6hD8=xSOQ5kaQ@mail.gmail.com","subject":"Re: [PATCH 2/8] refs/reftable: handle reloading stacks in the reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-06T10:43:24Z","receivedAt":"2024-11-06T10:43:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 05, 2024 at 05:14:15AM -0600, karthik nayak wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n> \n> [snip]\n> \n> > @@ -898,27 +915,31 @@ static int prepare_transaction_update(struct write_transaction_table_arg **out,\n> >  \t\t\t\t      struct ref_update *update,\n> >  \t\t\t\t      struct strbuf *err)\n> >  {\n> > -\tstruct reftable_stack *stack = backend_for(refs, update->refname, NULL)->stack;\n> >  \tstruct write_transaction_table_arg *arg = NULL;\n> > +\tstruct reftable_backend *be;\n> >  \tsize_t i;\n> >  \tint ret;\n> >\n> > +\tret = backend_for(&be, refs, update->refname, NULL, 0);\n> \n> So here, we don't reload the stack, it would be nice to add a comment\n> why, in the places we don't. Here, it seems to be because we possibly\n> already have an update which would have pushed us to reload the stack.\n\nI'll add comments.\n\n> > @@ -2462,7 +2498,7 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n> >  \targ.refs = refs;\n> >  \targ.records = rewritten;\n> >  \targ.len = logs_nr;\n> > -\targ.stack = stack,\n> > +\targ.stack = be->stack,\n> >  \targ.refname = refname,\n> >\n> \n> Shouldn't these lines end with ';'?\n\nYup, they should. It's harmless, but certainly confusing.\n\nPatrick\n"},{"id":"507106","messageId":"xmqqcyj16l3k.fsf@gitster.g","threadId":"62444","inReplyTo":"ac01c06c4177d1b07678972115648d125c2bdc66.1730792627.git.ps@pks.im","subject":"Re: [PATCH v2 1/8] refs/reftable: encapsulate reftable stack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-12T06:07:43Z","receivedAt":"2024-11-12T06:07:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> +static void reftable_backend_release(struct reftable_backend *be)\n> +{\n> +\treftable_stack_destroy(be->stack);\n> +\tbe->stack = NULL;\n> +}\n\nOK, this turned out to be the required reading for the remainder of\nthe patch ;-).  Everything else was pretty much a mechanical\nreplacement from reftable_stack to reftable_backend, but to get rid\nof the latter, we need to do this _release() thing where we used to\ndo the _destroy on a bare reftable_stack instance.\n\nLooking good and understandable.\n"},{"id":"507108","messageId":"xmqqo72l54yb.fsf@gitster.g","threadId":"62444","inReplyTo":"bab837e3733a982973bb96eedca15d073089693a.1730792627.git.ps@pks.im","subject":"Re: [PATCH v2 2/8] refs/reftable: handle reloading stacks in the reftable backend","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-12T06:41:48Z","receivedAt":"2024-11-12T06:41:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> +static int backend_for(struct reftable_backend **out,\n> +\t\t       struct reftable_ref_store *store,\n> +\t\t       const char *refname,\n> +\t\t       const char **rewritten_ref,\n> +\t\t       int reload)\n>  {\n> +\tstruct reftable_backend *be;\n>  \tconst char *wtname;\n>  \tint wtname_len;\n>  \n> -\tif (!refname)\n> -\t\treturn &store->main_backend;\n> +\tif (!refname) {\n> +\t\tbe = &store->main_backend;\n> +\t\tgoto out;\n> +\t}\n>  \n>  \tswitch (parse_worktree_ref(refname, &wtname, &wtname_len, rewritten_ref)) {\n>  \tcase REF_WORKTREE_OTHER: {\n>  \t\tstatic struct strbuf wtname_buf = STRBUF_INIT;\n>  \t\tstruct strbuf wt_dir = STRBUF_INIT;\n> -\t\tstruct reftable_backend *be;\n>  \n>  \t\t/*\n>  \t\t * We're using a static buffer here so that we don't need to\n> @@ -162,7 +166,7 @@ static struct reftable_backend *backend_for(struct reftable_ref_store *store,\n>  \t\t}\n>  \n>  \t\tstrbuf_release(&wt_dir);\n> -\t\treturn be;\n> +\t\tgoto out;\n\nAn interesting part of this function is not shown in the above\ncontext, but we look up an existing backend from a strmap, and\nallocate one if there isn't.  In either case, be points at the\nbackend to use.  Now be is not local to this block, we can access it\nafter jumping to \"out\" label.\n\n> +out:\n> +\tif (reload) {\n> +\t\tint ret = reftable_stack_reload(be->stack);\n> +\t\tif (ret)\n> +\t\t\treturn ret;\n> +\t}\n> +\t*out = be;\n> +\n> +\treturn 0;\n>  }\n\n> @@ -828,17 +845,17 @@ static int reftable_be_read_raw_ref(struct ref_store *ref_store,\n>  {\n>  \tstruct reftable_ref_store *refs =\n>  \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_raw_ref\");\n> -\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n> +\tstruct reftable_backend *be;\n>  \tint ret;\n>  \n>  \tif (refs->err < 0)\n>  \t\treturn refs->err;\n>  \n> -\tret = reftable_stack_reload(stack);\n> +\tret = backend_for(&be, refs, refname, &refname, 1);\n>  \tif (ret)\n>  \t\treturn ret;\n\nThis one chooses to reload, so that the next one, i.e.\n\"without-reload\", would not read stale information?\n\n> -\tret = read_ref_without_reload(refs, stack, refname, oid, referent, type);\n> +\tret = read_ref_without_reload(refs, be->stack, refname, oid, referent, type);\n\n\nThe following bit is curious.\n\n> +\tret = backend_for(&be, refs, update->refname, NULL, 0);\n> +\tif (ret)\n> +\t\treturn ret;\n> +\n\nWe locate one without reloading, and ...\n\n>  \t/*\n>  \t * Search for a preexisting stack update. If there is one then we add\n>  \t * the update to it, otherwise we set up a new stack update.\n>  \t */\n>  \tfor (i = 0; !arg && i < tx_data->args_nr; i++)\n> -\t\tif (tx_data->args[i].stack == stack)\n> +\t\tif (tx_data->args[i].be == be)\n>  \t\t\targ = &tx_data->args[i];\n>  \tif (!arg) {\n\n... only when we cannot reuse preexisting one, ...\n\n>  \t\tstruct reftable_addition *addition;\n>  \n> -\t\tret = reftable_stack_reload(stack);\n> +\t\tret = backend_for(&be, refs, update->refname, NULL, 1);\n>  \t\tif (ret)\n>  \t\t\treturn ret;\n\n... instead of directly doing reload on the instance we already\nhave, we do another _for() to locate one, this time reload set to 1.\n\nThat looks like doing some redundant work?  I am confused.\n\n> @@ -1048,7 +1070,11 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n>  \t\tgoto done;\n>  \t}\n>  \n> -\tret = read_ref_without_reload(refs, backend_for(refs, \"HEAD\", NULL)->stack, \"HEAD\",\n> +\tret = backend_for(&be, refs, \"HEAD\", NULL, 0);\n> +\tif (ret)\n> +\t\tgoto done;\n> +\n> +\tret = read_ref_without_reload(refs, be->stack, \"HEAD\",\n>  \t\t\t\t      &head_oid, &head_referent, &head_type);\n\nThis now takes into account the possibility that backend_for() might\nfail.  The original code would have segfaulted when it happened, I\nguess.\n\n> @@ -1057,10 +1083,11 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n>  \tfor (i = 0; i < transaction->nr; i++) {\n>  \t\tstruct ref_update *u = transaction->updates[i];\n>  \t\tstruct object_id current_oid = {0};\n> -\t\tstruct reftable_stack *stack;\n>  \t\tconst char *rewritten_ref;\n>  \n> -\t\tstack = backend_for(refs, u->refname, &rewritten_ref)->stack;\n> +\t\tret = backend_for(&be, refs, u->refname, &rewritten_ref, 0);\n> +\t\tif (ret)\n> +\t\t\tgoto done;\n\nDitto, we would have segfaulted in the next hunk when stack got NULL\nhere ...\n\n> @@ -1116,7 +1143,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n>  \t\t\tstring_list_insert(&affected_refnames, new_update->refname);\n>  \t\t}\n>  \n> -\t\tret = read_ref_without_reload(refs, stack, rewritten_ref,\n> +\t\tret = read_ref_without_reload(refs, be->stack, rewritten_ref,\n>  \t\t\t\t\t      &current_oid, &referent, &u->type);\n\n... here.\n\n> @@ -1831,10 +1858,9 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n>  {\n>  \tstruct reftable_ref_store *refs =\n>  \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"copy_ref\");\n> -\tstruct reftable_stack *stack = backend_for(refs, newrefname, &newrefname)->stack;\n> +\tstruct reftable_backend *be;\n>  \tstruct write_copy_arg arg = {\n>  \t\t.refs = refs,\n> -\t\t.stack = stack,\n>  \t\t.oldname = oldrefname,\n>  \t\t.newname = newrefname,\n>  \t\t.logmsg = logmsg,\n> @@ -1845,10 +1871,11 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n>  \tif (ret < 0)\n>  \t\tgoto done;\n>  \n> -\tret = reftable_stack_reload(stack);\n> +\tret = backend_for(&be, refs, newrefname, &newrefname, 1);\n>  \tif (ret)\n>  \t\tgoto done;\n\nWe used to grab \"stack\" upfront and then called reload here; we now\ndo backend_for() and let it do the reload as well, so they should be\nequivalent.\n\n> -\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n>  \tstruct reftable_log_record log = {0};\n>  \tstruct reftable_iterator it = {0};\n> +\tstruct reftable_backend *be;\n>  \tint ret;\n>  \n>  \tif (refs->err < 0)\n>  \t\treturn refs->err;\n>  \n> -\tret = reftable_stack_init_log_iterator(stack, &it);\n> +\tret = backend_for(&be, refs, refname, &refname, 0);\n> +\tif (ret)\n> +\t\tgoto done;\n> +\n> +\tret = reftable_stack_init_log_iterator(be->stack, &it);\n\nAgain, other than the fact that the new code carefully prepares for\nthe case where backend_for() fails to find be, the versions of the\ncode with and without the patch are equivalent.\n\n> @@ -2052,16 +2083,20 @@ static int reftable_be_for_each_reflog_ent(struct ref_store *ref_store,\n>  {\n>  \tstruct reftable_ref_store *refs =\n>  \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent\");\n> -\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n>  \tstruct reftable_log_record *logs = NULL;\n>  \tstruct reftable_iterator it = {0};\n> +\tstruct reftable_backend *be;\n>  \tsize_t logs_alloc = 0, logs_nr = 0, i;\n>  \tint ret;\n>  \n>  \tif (refs->err < 0)\n>  \t\treturn refs->err;\n>  \n> -\tret = reftable_stack_init_log_iterator(stack, &it);\n> +\tret = backend_for(&be, refs, refname, &refname, 0);\n> +\tif (ret)\n> +\t\tgoto done;\n> +\n> +\tret = reftable_stack_init_log_iterator(be->stack, &it);\n\nDitto.\n\n> @@ -2101,20 +2136,20 @@ static int reftable_be_reflog_exists(struct ref_store *ref_store,\n>  {\n>  \tstruct reftable_ref_store *refs =\n>  \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_exists\");\n> -\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n>  \tstruct reftable_log_record log = {0};\n>  \tstruct reftable_iterator it = {0};\n> +\tstruct reftable_backend *be;\n>  \tint ret;\n>  \n>  \tret = refs->err;\n>  \tif (ret < 0)\n>  \t\tgoto done;\n>  \n> -\tret = reftable_stack_reload(stack);\n> +\tret = backend_for(&be, refs, refname, &refname, 1);\n>  \tif (ret < 0)\n>  \t\tgoto done;\n>  \n> -\tret = reftable_stack_init_log_iterator(stack, &it);\n> +\tret = reftable_stack_init_log_iterator(be->stack, &it);\n>  \tif (ret < 0)\n>  \t\tgoto done;\n\nDitto.\n\nOverall they seem to be mostly equivalent, except that the new code\nis a bit more careful against failing backend_for().  One part of\nthe code confused me (and still I am unsure), but other than that it\nwas a pleasant read.\n\nThanks.\n"},{"id":"507110","messageId":"xmqqed3g6hg1.fsf@gitster.g","threadId":"62444","inReplyTo":"1b50655202f311c6a6ded61d4d50b1f287761d84.1730792627.git.ps@pks.im","subject":"Re: [PATCH v2 3/8] refs/reftable: read references via `struct reftable_backend`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-12T07:26:38Z","receivedAt":"2024-11-12T07:26:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> Refactor `read_ref_without_reload()` to accept a `struct reftable_stack`\n> as input instead of accepting a `struct reftable_stack`. This allows us\n> to implement an additional caching layer when reading refs where we can\n> reuse reftable iterators.\n>\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n>  refs/reftable-backend.c   | 110 ++++++++++++++++++++------------------\n>  reftable/reftable-stack.h |   3 ++\n>  reftable/stack.c          |   5 ++\n>  3 files changed, 67 insertions(+), 51 deletions(-)\n>\n> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> index 4a28dc8a9d..230adb690d 100644\n> --- a/refs/reftable-backend.c\n> +++ b/refs/reftable-backend.c\n> @@ -51,6 +51,50 @@ static void reftable_backend_release(struct reftable_backend *be)\n>  \tbe->stack = NULL;\n>  }\n>  \n> +static int reftable_backend_read_ref(struct reftable_backend *be,\n> +\t\t\t\t     const char *refname,\n> +\t\t\t\t     struct object_id *oid,\n> +\t\t\t\t     struct strbuf *referent,\n> +\t\t\t\t     unsigned int *type)\n> +{\n> +\tstruct reftable_ref_record ref = {0};\n> +\tint ret;\n> +\n> +\tret = reftable_stack_read_ref(be->stack, refname, &ref);\n> +\tif (ret)\n> +\t\tgoto done;\n> +\n> +\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n> +\t\tstrbuf_reset(referent);\n> +\t\tstrbuf_addstr(referent, ref.value.symref);\n> +\t\t*type |= REF_ISSYMREF;\n> +\t} else if (reftable_ref_record_val1(&ref)) {\n> +\t\tunsigned int hash_id;\n> +\n> +\t\tswitch (reftable_stack_hash_id(be->stack)) {\n\nSo, relative to the original, instead of relying on the repository\nand its knowledge of what hash function is used, we ask the stack\nwhat hash function is in use and use that instead.\n\n> +\t\tcase REFTABLE_HASH_SHA1:\n> +\t\t\thash_id = GIT_HASH_SHA1;\n> +\t\t\tbreak;\n> +\t\tcase REFTABLE_HASH_SHA256:\n> +\t\t\thash_id = GIT_HASH_SHA256;\n> +\t\t\tbreak;\n> +\t\tdefault:\n> +\t\t\tBUG(\"unhandled hash ID %d\", reftable_stack_hash_id(be->stack));\n> +\t\t}\n> +\n> +\t\toidread(oid, reftable_ref_record_val1(&ref),\n> +\t\t\t&hash_algos[hash_id]);\n> +\t} else {\n> +\t\t/* We got a tombstone, which should not happen. */\n> +\t\tBUG(\"unhandled reference value type %d\", ref.value_type);\n> +\t}\n> +\n> +done:\n> +\tassert(ret != REFTABLE_API_ERROR);\n> +\treftable_ref_record_release(&ref);\n> +\treturn ret;\n> +}\n\nHere is the original that got replaced.  Since ...\n\n> -static int read_ref_without_reload(struct reftable_ref_store *refs,\n> -\t\t\t\t   struct reftable_stack *stack,\n> -\t\t\t\t   const char *refname,\n> -\t\t\t\t   struct object_id *oid,\n> -\t\t\t\t   struct strbuf *referent,\n> -\t\t\t\t   unsigned int *type)\n> -{\n> -\tstruct reftable_ref_record ref = {0};\n> -\tint ret;\n> -\n> -\tret = reftable_stack_read_ref(stack, refname, &ref);\n> -\tif (ret)\n> -\t\tgoto done;\n> -\n> -\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n> -\t\tstrbuf_reset(referent);\n> -\t\tstrbuf_addstr(referent, ref.value.symref);\n> -\t\t*type |= REF_ISSYMREF;\n> -\t} else if (reftable_ref_record_val1(&ref)) {\n> -\t\toidread(oid, reftable_ref_record_val1(&ref),\n> -\t\t\trefs->base.repo->hash_algo);\n\n... we have access to \"refs\", which is a ref_store, that knows its\nrepository, it was just a few pointer references away to get the\nhash id of the Git side.  But of course we use REFTABLE_HASH_*NAME*\nto identify the algorithm at this layer, so we need to translate it\nback to the ide on the Git side before asking oidread() to read it.\n\n> -\t} else {\n> -\t\t/* We got a tombstone, which should not happen. */\n> -\t\tBUG(\"unhandled reference value type %d\", ref.value_type);\n> -\t}\n> -\n> -done:\n> -\tassert(ret != REFTABLE_API_ERROR);\n> -\treftable_ref_record_release(&ref);\n> -\treturn ret;\n> -}\n\nThere is one thing that is curious about this step.\n\nIt isn't like we teach stack what hash it uses in this step---the\nreftable_stack_hash_id() could have been implemented as early as\n59343984 (reftable/system: stop depending on \"hash.h\", 2024-11-08).\n\nOther than that this step introduces the first caller of\nreftable_stack_hash_id() in the series, the remaining hunks of this\npatch do not have to be part of this patch, but could have been a\nseparate step.  Not a suggestion to split it out, but merely an\nobservation (to make sure I am reading the code correctly).\n\n> diff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\n> index 54787f2ef5..ae14270ea7 100644\n> --- a/reftable/reftable-stack.h\n> +++ b/reftable/reftable-stack.h\n> @@ -149,4 +149,7 @@ struct reftable_compaction_stats {\n>  struct reftable_compaction_stats *\n>  reftable_stack_compaction_stats(struct reftable_stack *st);\n>  \n> +/* Return the hash of the stack. */\n> +enum reftable_hash reftable_stack_hash_id(struct reftable_stack *st);\n> +\n>  #endif\n> diff --git a/reftable/stack.c b/reftable/stack.c\n> index 1fffd75630..d97b64a40d 100644\n> --- a/reftable/stack.c\n> +++ b/reftable/stack.c\n> @@ -1791,3 +1791,8 @@ int reftable_stack_clean(struct reftable_stack *st)\n>  \treftable_addition_destroy(add);\n>  \treturn err;\n>  }\n> +\n> +enum reftable_hash reftable_stack_hash_id(struct reftable_stack *st)\n> +{\n> +\treturn reftable_merged_table_hash_id(st->merged);\n> +}\n"},{"id":"507122","messageId":"ZzMaX8HCj3GO5JUJ@pks.im","threadId":"62444","inReplyTo":"xmqqo72l54yb.fsf@gitster.g","subject":"Re: [PATCH v2 2/8] refs/reftable: handle reloading stacks in the reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-12T09:05:35Z","receivedAt":"2024-11-12T09:05:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 12, 2024 at 03:41:48PM +0900, Junio C Hamano wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n> > -\tret = read_ref_without_reload(refs, stack, refname, oid, referent, type);\n> > +\tret = read_ref_without_reload(refs, be->stack, refname, oid, referent, type);\n> \n> \n> The following bit is curious.\n> \n> > +\tret = backend_for(&be, refs, update->refname, NULL, 0);\n> > +\tif (ret)\n> > +\t\treturn ret;\n> > +\n> \n> We locate one without reloading, and ...\n> \n> >  \t/*\n> >  \t * Search for a preexisting stack update. If there is one then we add\n> >  \t * the update to it, otherwise we set up a new stack update.\n> >  \t */\n> >  \tfor (i = 0; !arg && i < tx_data->args_nr; i++)\n> > -\t\tif (tx_data->args[i].stack == stack)\n> > +\t\tif (tx_data->args[i].be == be)\n> >  \t\t\targ = &tx_data->args[i];\n> >  \tif (!arg) {\n> \n> ... only when we cannot reuse preexisting one, ...\n> \n> >  \t\tstruct reftable_addition *addition;\n> >  \n> > -\t\tret = reftable_stack_reload(stack);\n> > +\t\tret = backend_for(&be, refs, update->refname, NULL, 1);\n> >  \t\tif (ret)\n> >  \t\t\treturn ret;\n> \n> ... instead of directly doing reload on the instance we already\n> have, we do another _for() to locate one, this time reload set to 1.\n> \n> That looks like doing some redundant work?  I am confused.\n\nIt indeed is redundant work, yes. And in fact it is redundant work that\nisn't really required anymore. My first iteration didn't yet have the\n`reftable_write_options::on_reload()` callback, and I instead tried to\ncatch reloads via `backend_for()`, so it was required to reload via that\nfunction.\n\nBut now that we do have the callback that isn't needed anymore, and thus\nwe don't have to call `backend_for()` a second time here. I'll adapt\nthis accordingly.\n\nPatrick\n"},{"id":"507123","messageId":"ZzMaY-lqwKIak7Y3@pks.im","threadId":"62444","inReplyTo":"xmqqed3g6hg1.fsf@gitster.g","subject":"Re: [PATCH v2 3/8] refs/reftable: read references via `struct reftable_backend`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-12T09:05:39Z","receivedAt":"2024-11-12T09:05:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 12, 2024 at 04:26:38PM +0900, Junio C Hamano wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n> \n> > Refactor `read_ref_without_reload()` to accept a `struct reftable_stack`\n> > as input instead of accepting a `struct reftable_stack`. This allows us\n> > to implement an additional caching layer when reading refs where we can\n> > reuse reftable iterators.\n> >\n> > Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> > ---\n> >  refs/reftable-backend.c   | 110 ++++++++++++++++++++------------------\n> >  reftable/reftable-stack.h |   3 ++\n> >  reftable/stack.c          |   5 ++\n> >  3 files changed, 67 insertions(+), 51 deletions(-)\n> >\n> > diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> > index 4a28dc8a9d..230adb690d 100644\n> > --- a/refs/reftable-backend.c\n> > +++ b/refs/reftable-backend.c\n> > @@ -51,6 +51,50 @@ static void reftable_backend_release(struct reftable_backend *be)\n> >  \tbe->stack = NULL;\n> >  }\n> >  \n> > +static int reftable_backend_read_ref(struct reftable_backend *be,\n> > +\t\t\t\t     const char *refname,\n> > +\t\t\t\t     struct object_id *oid,\n> > +\t\t\t\t     struct strbuf *referent,\n> > +\t\t\t\t     unsigned int *type)\n> > +{\n> > +\tstruct reftable_ref_record ref = {0};\n> > +\tint ret;\n> > +\n> > +\tret = reftable_stack_read_ref(be->stack, refname, &ref);\n> > +\tif (ret)\n> > +\t\tgoto done;\n> > +\n> > +\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n> > +\t\tstrbuf_reset(referent);\n> > +\t\tstrbuf_addstr(referent, ref.value.symref);\n> > +\t\t*type |= REF_ISSYMREF;\n> > +\t} else if (reftable_ref_record_val1(&ref)) {\n> > +\t\tunsigned int hash_id;\n> > +\n> > +\t\tswitch (reftable_stack_hash_id(be->stack)) {\n> \n> So, relative to the original, instead of relying on the repository\n> and its knowledge of what hash function is used, we ask the stack\n> what hash function is in use and use that instead.\n> \n> > +\t\tcase REFTABLE_HASH_SHA1:\n> > +\t\t\thash_id = GIT_HASH_SHA1;\n> > +\t\t\tbreak;\n> > +\t\tcase REFTABLE_HASH_SHA256:\n> > +\t\t\thash_id = GIT_HASH_SHA256;\n> > +\t\t\tbreak;\n> > +\t\tdefault:\n> > +\t\t\tBUG(\"unhandled hash ID %d\", reftable_stack_hash_id(be->stack));\n> > +\t\t}\n> > +\n> > +\t\toidread(oid, reftable_ref_record_val1(&ref),\n> > +\t\t\t&hash_algos[hash_id]);\n> > +\t} else {\n> > +\t\t/* We got a tombstone, which should not happen. */\n> > +\t\tBUG(\"unhandled reference value type %d\", ref.value_type);\n> > +\t}\n> > +\n> > +done:\n> > +\tassert(ret != REFTABLE_API_ERROR);\n> > +\treftable_ref_record_release(&ref);\n> > +\treturn ret;\n> > +}\n> \n> Here is the original that got replaced.  Since ...\n> \n> > -static int read_ref_without_reload(struct reftable_ref_store *refs,\n> > -\t\t\t\t   struct reftable_stack *stack,\n> > -\t\t\t\t   const char *refname,\n> > -\t\t\t\t   struct object_id *oid,\n> > -\t\t\t\t   struct strbuf *referent,\n> > -\t\t\t\t   unsigned int *type)\n> > -{\n> > -\tstruct reftable_ref_record ref = {0};\n> > -\tint ret;\n> > -\n> > -\tret = reftable_stack_read_ref(stack, refname, &ref);\n> > -\tif (ret)\n> > -\t\tgoto done;\n> > -\n> > -\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n> > -\t\tstrbuf_reset(referent);\n> > -\t\tstrbuf_addstr(referent, ref.value.symref);\n> > -\t\t*type |= REF_ISSYMREF;\n> > -\t} else if (reftable_ref_record_val1(&ref)) {\n> > -\t\toidread(oid, reftable_ref_record_val1(&ref),\n> > -\t\t\trefs->base.repo->hash_algo);\n> \n> ... we have access to \"refs\", which is a ref_store, that knows its\n> repository, it was just a few pointer references away to get the\n> hash id of the Git side.  But of course we use REFTABLE_HASH_*NAME*\n> to identify the algorithm at this layer, so we need to translate it\n> back to the ide on the Git side before asking oidread() to read it.\n> \n> > -\t} else {\n> > -\t\t/* We got a tombstone, which should not happen. */\n> > -\t\tBUG(\"unhandled reference value type %d\", ref.value_type);\n> > -\t}\n> > -\n> > -done:\n> > -\tassert(ret != REFTABLE_API_ERROR);\n> > -\treftable_ref_record_release(&ref);\n> > -\treturn ret;\n> > -}\n> \n> There is one thing that is curious about this step.\n> \n> It isn't like we teach stack what hash it uses in this step---the\n> reftable_stack_hash_id() could have been implemented as early as\n> 59343984 (reftable/system: stop depending on \"hash.h\", 2024-11-08).\n\nIn theory we could've implemented it even earlier than that: the commit\nonly introduces the reftable-specific hashes, and we had the\nGit-specific hashes available before that. Like that we wouldn't even\nhave to translate between the different hashes in the first place.\n\n> Other than that this step introduces the first caller of\n> reftable_stack_hash_id() in the series, the remaining hunks of this\n> patch do not have to be part of this patch, but could have been a\n> separate step.  Not a suggestion to split it out, but merely an\n> observation (to make sure I am reading the code correctly).\n\nYup, your understanding matches mine.\n\nPatrick\n"},{"id":"508040","messageId":"20241125-pks-reftable-backend-reuse-iter-v3-0-1d7b658e3e9e@pks.im","threadId":"62444","inReplyTo":"cover.1730732881.git.ps@pks.im","subject":"[PATCH v3 0/9] refs/reftable: reuse iterators when reading refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-25T07:38:22Z","receivedAt":"2024-11-25T07:38:40Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the second version of my patch series that refactors the\nreftable backend to reuse iterators when reading random references. This\nremoves the overhead of having to recreate the iterator on every read\nand thus leads to better performance and less allocation churn.\n\nChanges in v3:\n\n  - Adapt some comments to refer to the \"backend\" instead of to the\n    \"stack\".\n  - Fix indentation of a statement while at it.\n  - Explain why callsites don't want to reload the stack.\n  - Optimize `prepare_transaction_update()` by not using `backend_for()`\n    twice, but instead reload the stack manually.\n  - Split out the change that adds `reftable_stack_hash_id()` into a\n    separate commit.\n  - Link to v2: https://lore.kernel.org/r/cover.1730792627.git.ps@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (9):\n      refs/reftable: encapsulate reftable stack\n      refs/reftable: handle reloading stacks in the reftable backend\n      reftable/stack: add accessor for the hash ID\n      refs/reftable: read references via `struct reftable_backend`\n      refs/reftable: refactor reading symbolic refs to use reftable backend\n      refs/reftable: refactor reflog expiry to use reftable backend\n      reftable/stack: add mechanism to notify callers on reload\n      reftable/merged: drain priority queue on reseek\n      refs/reftable: reuse iterators when reading refs\n\n refs/reftable-backend.c          | 409 +++++++++++++++++++++++++--------------\n reftable/merged.c                |   2 +\n reftable/reftable-stack.h        |   3 +\n reftable/reftable-writer.h       |   9 +\n reftable/stack.c                 |   9 +\n t/unit-tests/t-reftable-merged.c |  73 +++++++\n 6 files changed, 357 insertions(+), 148 deletions(-)\n\nRange-diff versus v2:\n\n 1:  9854214fe9 !  1:  21071ae5a5 refs/reftable: encapsulate reftable stack\n    @@ refs/reftable-backend.c\n      struct reftable_ref_store {\n      \tstruct ref_store base;\n      \n    -@@ refs/reftable-backend.c: struct reftable_ref_store {\n    - \t * The main stack refers to the common dir and thus contains common\n    + \t/*\n    +-\t * The main stack refers to the common dir and thus contains common\n    ++\t * The main backend refers to the common dir and thus contains common\n      \t * refs as well as refs of the main repository.\n      \t */\n     -\tstruct reftable_stack *main_stack;\n     +\tstruct reftable_backend main_backend;\n      \t/*\n    - \t * The worktree stack refers to the gitdir in case the refdb is opened\n    +-\t * The worktree stack refers to the gitdir in case the refdb is opened\n    ++\t * The worktree backend refers to the gitdir in case the refdb is opened\n      \t * via a worktree. It thus contains the per-worktree refs.\n      \t */\n     -\tstruct reftable_stack *worktree_stack;\n     +\tstruct reftable_backend worktree_backend;\n      \t/*\n    - \t * Map of worktree stacks by their respective worktree names. The map\n    +-\t * Map of worktree stacks by their respective worktree names. The map\n    ++\t * Map of worktree backends by their respective worktree names. The map\n      \t * is populated lazily when we try to resolve `worktrees/$worktree` refs.\n      \t */\n     -\tstruct strmap worktree_stacks;\n    @@ refs/reftable-backend.c: static struct ref_iterator *reftable_be_iterator_begin(\n      \t * iterator, only.\n      \t */\n     -\t if (!refs->worktree_stack)\n    -+\t if (!refs->worktree_backend.stack)\n    ++\tif (!refs->worktree_backend.stack)\n      \t\treturn &main_iter->base;\n      \n      \t/*\n 2:  18265dfafc !  2:  a9588125c7 refs/reftable: handle reloading stacks in the reftable backend\n    @@ Commit message\n         But second this makes the logic to access stacks more self-contained by\n         letting the `struct reftable_backend` manage themselves.\n     \n    +    Update callsites where we don't reload the stack to document why we\n    +    don't. In some cases it's unclear whether it is the right thing to do in\n    +    the first place, but fixing that is outside of the scope of this patch\n    +    series.\n    +\n         Signed-off-by: Patrick Steinhardt <ps@pks.im>\n     \n      ## refs/reftable-backend.c ##\n    @@ refs/reftable-backend.c: static int prepare_transaction_update(struct write_tran\n      \tsize_t i;\n      \tint ret;\n      \n    ++\t/*\n    ++\t * This function gets called in a loop, and we don't want to repeatedly\n    ++\t * reload the stack for every single ref update. Instead, we manually\n    ++\t * reload further down in the case where we haven't yet prepared the\n    ++\t * specific `reftable_backend`.\n    ++\t */\n     +\tret = backend_for(&be, refs, update->refname, NULL, 0);\n     +\tif (ret)\n     +\t\treturn ret;\n    @@ refs/reftable-backend.c: static int prepare_transaction_update(struct write_tran\n      \t\tstruct reftable_addition *addition;\n      \n     -\t\tret = reftable_stack_reload(stack);\n    -+\t\tret = backend_for(&be, refs, update->refname, NULL, 1);\n    ++\t\tret = reftable_stack_reload(be->stack);\n      \t\tif (ret)\n      \t\t\treturn ret;\n      \n    @@ refs/reftable-backend.c: static int reftable_be_transaction_prepare(struct ref_s\n      \t}\n      \n     -\tret = read_ref_without_reload(refs, backend_for(refs, \"HEAD\", NULL)->stack, \"HEAD\",\n    ++\t/*\n    ++\t * TODO: it's dubious whether we should reload the stack that \"HEAD\"\n    ++\t * belongs to or not. In theory, it may happen that we only modify\n    ++\t * stacks which are _not_ part of the \"HEAD\" stack. In that case we\n    ++\t * wouldn't have prepared any transaction for its stack and would not\n    ++\t * have reloaded it, which may mean that it is stale.\n    ++\t *\n    ++\t * On the other hand, reloading that stack without locking it feels\n    ++\t * wrong to, as the value of \"HEAD\" could be modified concurrently at\n    ++\t * any point in time.\n    ++\t */\n     +\tret = backend_for(&be, refs, \"HEAD\", NULL, 0);\n     +\tif (ret)\n     +\t\tgoto done;\n    @@ refs/reftable-backend.c: static int reftable_be_transaction_prepare(struct ref_s\n      \t\tconst char *rewritten_ref;\n      \n     -\t\tstack = backend_for(refs, u->refname, &rewritten_ref)->stack;\n    ++\t\t/*\n    ++\t\t * There is no need to reload the respective backends here as\n    ++\t\t * we have already reloaded them when preparing the transaction\n    ++\t\t * update. And given that the stacks have been locked there\n    ++\t\t * shouldn't have been any concurrent modifications of the\n    ++\t\t * stack.\n    ++\t\t */\n     +\t\tret = backend_for(&be, refs, u->refname, &rewritten_ref, 0);\n     +\t\tif (ret)\n     +\t\t\tgoto done;\n    @@ refs/reftable-backend.c: static int reftable_be_for_each_reflog_ent_reverse(stru\n      \t\treturn refs->err;\n      \n     -\tret = reftable_stack_init_log_iterator(stack, &it);\n    ++\t/*\n    ++\t * TODO: we should adapt this callsite to reload the stack. There is no\n    ++\t * obvious reason why we shouldn't.\n    ++\t */\n     +\tret = backend_for(&be, refs, refname, &refname, 0);\n     +\tif (ret)\n     +\t\tgoto done;\n    @@ refs/reftable-backend.c: static int reftable_be_for_each_reflog_ent(struct ref_s\n      \t\treturn refs->err;\n      \n     -\tret = reftable_stack_init_log_iterator(stack, &it);\n    ++\t/*\n    ++\t * TODO: we should adapt this callsite to reload the stack. There is no\n    ++\t * obvious reason why we shouldn't.\n    ++\t */\n     +\tret = backend_for(&be, refs, refname, &refname, 0);\n     +\tif (ret)\n     +\t\tgoto done;\n    @@ refs/reftable-backend.c: static int reftable_be_reflog_expire(struct ref_store *\n      \targ.records = rewritten;\n      \targ.len = logs_nr;\n     -\targ.stack = stack,\n    -+\targ.stack = be->stack,\n    - \targ.refname = refname,\n    +-\targ.refname = refname,\n    ++\targ.stack = be->stack;\n    ++\targ.refname = refname;\n      \n      \tret = reftable_addition_add(add, &write_reflog_expiry_table, &arg);\n    + \tif (ret < 0)\n -:  ---------- >  3:  76f7ff40d2 reftable/stack: add accessor for the hash ID\n 3:  c33093e73a !  4:  19da5f587c refs/reftable: read references via `struct reftable_backend`\n    @@ refs/reftable-backend.c: static int reftable_be_copy_ref(struct ref_store *ref_s\n      \n      done:\n      \tassert(ret != REFTABLE_API_ERROR);\n    -\n    - ## reftable/reftable-stack.h ##\n    -@@ reftable/reftable-stack.h: struct reftable_compaction_stats {\n    - struct reftable_compaction_stats *\n    - reftable_stack_compaction_stats(struct reftable_stack *st);\n    - \n    -+/* Return the hash of the stack. */\n    -+enum reftable_hash reftable_stack_hash_id(struct reftable_stack *st);\n    -+\n    - #endif\n    -\n    - ## reftable/stack.c ##\n    -@@ reftable/stack.c: int reftable_stack_clean(struct reftable_stack *st)\n    - \treftable_addition_destroy(add);\n    - \treturn err;\n    - }\n    -+\n    -+enum reftable_hash reftable_stack_hash_id(struct reftable_stack *st)\n    -+{\n    -+\treturn reftable_merged_table_hash_id(st->merged);\n    -+}\n 4:  8489e32d87 =  5:  ff4f02dda7 refs/reftable: refactor reading symbolic refs to use reftable backend\n 5:  b1afd63785 =  6:  ed8963a520 refs/reftable: refactor reflog expiry to use reftable backend\n 6:  1e754ccde8 =  7:  5b91ca48e3 reftable/stack: add mechanism to notify callers on reload\n 7:  6755cf9ec9 =  8:  613f794fe6 reftable/merged: drain priority queue on reseek\n 8:  e3b29b2035 =  9:  a44911e4a4 refs/reftable: reuse iterators when reading refs\n\n---\nbase-commit: 455ddbf8c6a694968c1089fb6c7ffb1d31d97e9d\nchange-id: 20241125-pks-reftable-backend-reuse-iter-3a2e92428789\n\n"},{"id":"508041","messageId":"20241125-pks-reftable-backend-reuse-iter-v3-1-1d7b658e3e9e@pks.im","threadId":"62444","inReplyTo":"20241125-pks-reftable-backend-reuse-iter-v3-0-1d7b658e3e9e@pks.im","subject":"[PATCH v3 1/9] refs/reftable: encapsulate reftable stack","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-25T07:38:23Z","receivedAt":"2024-11-25T07:38:41Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable ref store needs to keep track of multiple stacks, one for\nthe main worktree and an arbitrary number of stacks for worktrees. This\nis done by storing pointers to `struct reftable_stack`, which we then\naccess directly.\n\nWrap the stack in a new `struct reftable_backend`. This will allow us to\nattach more data to each respective stack in subsequent commits.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 135 +++++++++++++++++++++++++++---------------------\n 1 file changed, 76 insertions(+), 59 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex f560bc2b67857d785294e6b5699383a256f30813..acd26f8928d18396f78a2d39ad0e0c1796d5a409 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -34,24 +34,41 @@\n  */\n #define REF_UPDATE_VIA_HEAD (1 << 8)\n \n+struct reftable_backend {\n+\tstruct reftable_stack *stack;\n+};\n+\n+static int reftable_backend_init(struct reftable_backend *be,\n+\t\t\t\t const char *path,\n+\t\t\t\t const struct reftable_write_options *opts)\n+{\n+\treturn reftable_new_stack(&be->stack, path, opts);\n+}\n+\n+static void reftable_backend_release(struct reftable_backend *be)\n+{\n+\treftable_stack_destroy(be->stack);\n+\tbe->stack = NULL;\n+}\n+\n struct reftable_ref_store {\n \tstruct ref_store base;\n \n \t/*\n-\t * The main stack refers to the common dir and thus contains common\n+\t * The main backend refers to the common dir and thus contains common\n \t * refs as well as refs of the main repository.\n \t */\n-\tstruct reftable_stack *main_stack;\n+\tstruct reftable_backend main_backend;\n \t/*\n-\t * The worktree stack refers to the gitdir in case the refdb is opened\n+\t * The worktree backend refers to the gitdir in case the refdb is opened\n \t * via a worktree. It thus contains the per-worktree refs.\n \t */\n-\tstruct reftable_stack *worktree_stack;\n+\tstruct reftable_backend worktree_backend;\n \t/*\n-\t * Map of worktree stacks by their respective worktree names. The map\n+\t * Map of worktree backends by their respective worktree names. The map\n \t * is populated lazily when we try to resolve `worktrees/$worktree` refs.\n \t */\n-\tstruct strmap worktree_stacks;\n+\tstruct strmap worktree_backends;\n \tstruct reftable_write_options write_options;\n \n \tunsigned int store_flags;\n@@ -97,21 +114,21 @@ static struct reftable_ref_store *reftable_be_downcast(struct ref_store *ref_sto\n  * like `worktrees/$worktree/refs/heads/foo` as worktree stacks will store\n  * those references in their normalized form.\n  */\n-static struct reftable_stack *stack_for(struct reftable_ref_store *store,\n-\t\t\t\t\tconst char *refname,\n-\t\t\t\t\tconst char **rewritten_ref)\n+static struct reftable_backend *backend_for(struct reftable_ref_store *store,\n+\t\t\t\t\t    const char *refname,\n+\t\t\t\t\t    const char **rewritten_ref)\n {\n \tconst char *wtname;\n \tint wtname_len;\n \n \tif (!refname)\n-\t\treturn store->main_stack;\n+\t\treturn &store->main_backend;\n \n \tswitch (parse_worktree_ref(refname, &wtname, &wtname_len, rewritten_ref)) {\n \tcase REF_WORKTREE_OTHER: {\n \t\tstatic struct strbuf wtname_buf = STRBUF_INIT;\n \t\tstruct strbuf wt_dir = STRBUF_INIT;\n-\t\tstruct reftable_stack *stack;\n+\t\tstruct reftable_backend *be;\n \n \t\t/*\n \t\t * We're using a static buffer here so that we don't need to\n@@ -125,37 +142,39 @@ static struct reftable_stack *stack_for(struct reftable_ref_store *store,\n \t\t/*\n \t\t * There is an edge case here: when the worktree references the\n \t\t * current worktree, then we set up the stack once via\n-\t\t * `worktree_stacks` and once via `worktree_stack`. This is\n+\t\t * `worktree_backends` and once via `worktree_backend`. This is\n \t\t * wasteful, but in the reading case it shouldn't matter. And\n \t\t * in the writing case we would notice that the stack is locked\n \t\t * already and error out when trying to write a reference via\n \t\t * both stacks.\n \t\t */\n-\t\tstack = strmap_get(&store->worktree_stacks, wtname_buf.buf);\n-\t\tif (!stack) {\n+\t\tbe = strmap_get(&store->worktree_backends, wtname_buf.buf);\n+\t\tif (!be) {\n \t\t\tstrbuf_addf(&wt_dir, \"%s/worktrees/%s/reftable\",\n \t\t\t\t    store->base.repo->commondir, wtname_buf.buf);\n \n-\t\t\tstore->err = reftable_new_stack(&stack, wt_dir.buf,\n-\t\t\t\t\t\t\t&store->write_options);\n+\t\t\tCALLOC_ARRAY(be, 1);\n+\t\t\tstore->err = reftable_backend_init(be, wt_dir.buf,\n+\t\t\t\t\t\t\t   &store->write_options);\n \t\t\tassert(store->err != REFTABLE_API_ERROR);\n-\t\t\tstrmap_put(&store->worktree_stacks, wtname_buf.buf, stack);\n+\n+\t\t\tstrmap_put(&store->worktree_backends, wtname_buf.buf, be);\n \t\t}\n \n \t\tstrbuf_release(&wt_dir);\n-\t\treturn stack;\n+\t\treturn be;\n \t}\n \tcase REF_WORKTREE_CURRENT:\n \t\t/*\n \t\t * If there is no worktree stack then we're currently in the\n \t\t * main worktree. We thus return the main stack in that case.\n \t\t */\n-\t\tif (!store->worktree_stack)\n-\t\t\treturn store->main_stack;\n-\t\treturn store->worktree_stack;\n+\t\tif (!store->worktree_backend.stack)\n+\t\t\treturn &store->main_backend;\n+\t\treturn &store->worktree_backend;\n \tcase REF_WORKTREE_MAIN:\n \tcase REF_WORKTREE_SHARED:\n-\t\treturn store->main_stack;\n+\t\treturn &store->main_backend;\n \tdefault:\n \t\tBUG(\"unhandled worktree reference type\");\n \t}\n@@ -292,7 +311,7 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \tumask(mask);\n \n \tbase_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable);\n-\tstrmap_init(&refs->worktree_stacks);\n+\tstrmap_init(&refs->worktree_backends);\n \trefs->store_flags = store_flags;\n \trefs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);\n \n@@ -337,8 +356,8 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \t\tstrbuf_realpath(&path, gitdir, 0);\n \t}\n \tstrbuf_addstr(&path, \"/reftable\");\n-\trefs->err = reftable_new_stack(&refs->main_stack, path.buf,\n-\t\t\t\t       &refs->write_options);\n+\trefs->err = reftable_backend_init(&refs->main_backend, path.buf,\n+\t\t\t\t\t  &refs->write_options);\n \tif (refs->err)\n \t\tgoto done;\n \n@@ -354,8 +373,8 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \t\tstrbuf_reset(&path);\n \t\tstrbuf_addf(&path, \"%s/reftable\", gitdir);\n \n-\t\trefs->err = reftable_new_stack(&refs->worktree_stack, path.buf,\n-\t\t\t\t\t       &refs->write_options);\n+\t\trefs->err = reftable_backend_init(&refs->worktree_backend, path.buf,\n+\t\t\t\t\t\t  &refs->write_options);\n \t\tif (refs->err)\n \t\t\tgoto done;\n \t}\n@@ -374,19 +393,17 @@ static void reftable_be_release(struct ref_store *ref_store)\n \tstruct strmap_entry *entry;\n \tstruct hashmap_iter iter;\n \n-\tif (refs->main_stack) {\n-\t\treftable_stack_destroy(refs->main_stack);\n-\t\trefs->main_stack = NULL;\n-\t}\n+\tif (refs->main_backend.stack)\n+\t\treftable_backend_release(&refs->main_backend);\n+\tif (refs->worktree_backend.stack)\n+\t\treftable_backend_release(&refs->worktree_backend);\n \n-\tif (refs->worktree_stack) {\n-\t\treftable_stack_destroy(refs->worktree_stack);\n-\t\trefs->worktree_stack = NULL;\n+\tstrmap_for_each_entry(&refs->worktree_backends, &iter, entry) {\n+\t\tstruct reftable_backend *be = entry->value;\n+\t\treftable_backend_release(be);\n+\t\tfree(be);\n \t}\n-\n-\tstrmap_for_each_entry(&refs->worktree_stacks, &iter, entry)\n-\t\treftable_stack_destroy(entry->value);\n-\tstrmap_clear(&refs->worktree_stacks, 0);\n+\tstrmap_clear(&refs->worktree_backends, 0);\n }\n \n static int reftable_be_create_on_disk(struct ref_store *ref_store,\n@@ -781,7 +798,7 @@ static struct ref_iterator *reftable_be_iterator_begin(struct ref_store *ref_sto\n \t\trequired_flags |= REF_STORE_ODB;\n \trefs = reftable_be_downcast(ref_store, required_flags, \"ref_iterator_begin\");\n \n-\tmain_iter = ref_iterator_for_stack(refs, refs->main_stack, prefix,\n+\tmain_iter = ref_iterator_for_stack(refs, refs->main_backend.stack, prefix,\n \t\t\t\t\t   exclude_patterns, flags);\n \n \t/*\n@@ -789,14 +806,14 @@ static struct ref_iterator *reftable_be_iterator_begin(struct ref_store *ref_sto\n \t * right now. If we aren't, then we return the common reftable\n \t * iterator, only.\n \t */\n-\t if (!refs->worktree_stack)\n+\tif (!refs->worktree_backend.stack)\n \t\treturn &main_iter->base;\n \n \t/*\n \t * Otherwise we merge both the common and the per-worktree refs into a\n \t * single iterator.\n \t */\n-\tworktree_iter = ref_iterator_for_stack(refs, refs->worktree_stack, prefix,\n+\tworktree_iter = ref_iterator_for_stack(refs, refs->worktree_backend.stack, prefix,\n \t\t\t\t\t       exclude_patterns, flags);\n \treturn merge_ref_iterator_begin(&worktree_iter->base, &main_iter->base,\n \t\t\t\t\tref_iterator_select, NULL);\n@@ -811,7 +828,7 @@ static int reftable_be_read_raw_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_raw_ref\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tint ret;\n \n \tif (refs->err < 0)\n@@ -838,7 +855,7 @@ static int reftable_be_read_symbolic_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_symbolic_ref\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_ref_record ref = {0};\n \tint ret;\n \n@@ -898,7 +915,7 @@ static int prepare_transaction_update(struct write_transaction_table_arg **out,\n \t\t\t\t      struct ref_update *update,\n \t\t\t\t      struct strbuf *err)\n {\n-\tstruct reftable_stack *stack = stack_for(refs, update->refname, NULL);\n+\tstruct reftable_stack *stack = backend_for(refs, update->refname, NULL)->stack;\n \tstruct write_transaction_table_arg *arg = NULL;\n \tsize_t i;\n \tint ret;\n@@ -1031,7 +1048,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\tgoto done;\n \t}\n \n-\tret = read_ref_without_reload(refs, stack_for(refs, \"HEAD\", NULL), \"HEAD\",\n+\tret = read_ref_without_reload(refs, backend_for(refs, \"HEAD\", NULL)->stack, \"HEAD\",\n \t\t\t\t      &head_oid, &head_referent, &head_type);\n \tif (ret < 0)\n \t\tgoto done;\n@@ -1043,7 +1060,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\tstruct reftable_stack *stack;\n \t\tconst char *rewritten_ref;\n \n-\t\tstack = stack_for(refs, u->refname, &rewritten_ref);\n+\t\tstack = backend_for(refs, u->refname, &rewritten_ref)->stack;\n \n \t\t/* Verify that the new object ID is valid. */\n \t\tif ((u->flags & REF_HAVE_NEW) && !is_null_oid(&u->new_oid) &&\n@@ -1525,9 +1542,9 @@ static int reftable_be_pack_refs(struct ref_store *ref_store,\n \tif (refs->err)\n \t\treturn refs->err;\n \n-\tstack = refs->worktree_stack;\n+\tstack = refs->worktree_backend.stack;\n \tif (!stack)\n-\t\tstack = refs->main_stack;\n+\t\tstack = refs->main_backend.stack;\n \n \tif (opts->flags & PACK_REFS_AUTO)\n \t\tret = reftable_stack_auto_compact(stack);\n@@ -1782,7 +1799,7 @@ static int reftable_be_rename_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"rename_ref\");\n-\tstruct reftable_stack *stack = stack_for(refs, newrefname, &newrefname);\n+\tstruct reftable_stack *stack = backend_for(refs, newrefname, &newrefname)->stack;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n \t\t.stack = stack,\n@@ -1814,7 +1831,7 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"copy_ref\");\n-\tstruct reftable_stack *stack = stack_for(refs, newrefname, &newrefname);\n+\tstruct reftable_stack *stack = backend_for(refs, newrefname, &newrefname)->stack;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n \t\t.stack = stack,\n@@ -1952,11 +1969,11 @@ static struct ref_iterator *reftable_be_reflog_iterator_begin(struct ref_store *\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_iterator_begin\");\n \tstruct reftable_reflog_iterator *main_iter, *worktree_iter;\n \n-\tmain_iter = reflog_iterator_for_stack(refs, refs->main_stack);\n-\tif (!refs->worktree_stack)\n+\tmain_iter = reflog_iterator_for_stack(refs, refs->main_backend.stack);\n+\tif (!refs->worktree_backend.stack)\n \t\treturn &main_iter->base;\n \n-\tworktree_iter = reflog_iterator_for_stack(refs, refs->worktree_stack);\n+\tworktree_iter = reflog_iterator_for_stack(refs, refs->worktree_backend.stack);\n \n \treturn merge_ref_iterator_begin(&worktree_iter->base, &main_iter->base,\n \t\t\t\t\tref_iterator_select, NULL);\n@@ -1995,7 +2012,7 @@ static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent_reverse\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record log = {0};\n \tstruct reftable_iterator it = {0};\n \tint ret;\n@@ -2035,7 +2052,7 @@ static int reftable_be_for_each_reflog_ent(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record *logs = NULL;\n \tstruct reftable_iterator it = {0};\n \tsize_t logs_alloc = 0, logs_nr = 0, i;\n@@ -2084,7 +2101,7 @@ static int reftable_be_reflog_exists(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_exists\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record log = {0};\n \tstruct reftable_iterator it = {0};\n \tint ret;\n@@ -2169,7 +2186,7 @@ static int reftable_be_create_reflog(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"create_reflog\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct write_reflog_existence_arg arg = {\n \t\t.refs = refs,\n \t\t.stack = stack,\n@@ -2243,7 +2260,7 @@ static int reftable_be_delete_reflog(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"delete_reflog\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct write_reflog_delete_arg arg = {\n \t\t.stack = stack,\n \t\t.refname = refname,\n@@ -2352,7 +2369,7 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \t */\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"reflog_expire\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record *logs = NULL;\n \tstruct reftable_log_record *rewritten = NULL;\n \tstruct reftable_ref_record ref_record = {0};\n\n-- \n2.47.0.274.g962d0b743d.dirty\n\n"},{"id":"508042","messageId":"20241125-pks-reftable-backend-reuse-iter-v3-2-1d7b658e3e9e@pks.im","threadId":"62444","inReplyTo":"20241125-pks-reftable-backend-reuse-iter-v3-0-1d7b658e3e9e@pks.im","subject":"[PATCH v3 2/9] refs/reftable: handle reloading stacks in the reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-25T07:38:24Z","receivedAt":"2024-11-25T07:38:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When accessing a stack we almost always have to reload the stack before\nreading data from it. This is mostly because Git does not have a\nnotification mechanism for when underlying data has been changed, and\nthus we are forced to opportunistically reload the stack every single\ntime to account for any changes that may have happened concurrently.\n\nHandle the reload internally in `backend_for()`. For one this forces\ncallsites to think about whether or not they need to reload the stack.\nBut second this makes the logic to access stacks more self-contained by\nletting the `struct reftable_backend` manage themselves.\n\nUpdate callsites where we don't reload the stack to document why we\ndon't. In some cases it's unclear whether it is the right thing to do in\nthe first place, but fixing that is outside of the scope of this patch\nseries.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 184 +++++++++++++++++++++++++++++++++---------------\n 1 file changed, 126 insertions(+), 58 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex acd26f8928d18396f78a2d39ad0e0c1796d5a409..64fe8fd02d8ec932f7980cdb7d7d5223c3c83b73 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -114,21 +114,25 @@ static struct reftable_ref_store *reftable_be_downcast(struct ref_store *ref_sto\n  * like `worktrees/$worktree/refs/heads/foo` as worktree stacks will store\n  * those references in their normalized form.\n  */\n-static struct reftable_backend *backend_for(struct reftable_ref_store *store,\n-\t\t\t\t\t    const char *refname,\n-\t\t\t\t\t    const char **rewritten_ref)\n+static int backend_for(struct reftable_backend **out,\n+\t\t       struct reftable_ref_store *store,\n+\t\t       const char *refname,\n+\t\t       const char **rewritten_ref,\n+\t\t       int reload)\n {\n+\tstruct reftable_backend *be;\n \tconst char *wtname;\n \tint wtname_len;\n \n-\tif (!refname)\n-\t\treturn &store->main_backend;\n+\tif (!refname) {\n+\t\tbe = &store->main_backend;\n+\t\tgoto out;\n+\t}\n \n \tswitch (parse_worktree_ref(refname, &wtname, &wtname_len, rewritten_ref)) {\n \tcase REF_WORKTREE_OTHER: {\n \t\tstatic struct strbuf wtname_buf = STRBUF_INIT;\n \t\tstruct strbuf wt_dir = STRBUF_INIT;\n-\t\tstruct reftable_backend *be;\n \n \t\t/*\n \t\t * We're using a static buffer here so that we don't need to\n@@ -162,7 +166,7 @@ static struct reftable_backend *backend_for(struct reftable_ref_store *store,\n \t\t}\n \n \t\tstrbuf_release(&wt_dir);\n-\t\treturn be;\n+\t\tgoto out;\n \t}\n \tcase REF_WORKTREE_CURRENT:\n \t\t/*\n@@ -170,14 +174,27 @@ static struct reftable_backend *backend_for(struct reftable_ref_store *store,\n \t\t * main worktree. We thus return the main stack in that case.\n \t\t */\n \t\tif (!store->worktree_backend.stack)\n-\t\t\treturn &store->main_backend;\n-\t\treturn &store->worktree_backend;\n+\t\t\tbe = &store->main_backend;\n+\t\telse\n+\t\t\tbe = &store->worktree_backend;\n+\t\tgoto out;\n \tcase REF_WORKTREE_MAIN:\n \tcase REF_WORKTREE_SHARED:\n-\t\treturn &store->main_backend;\n+\t\tbe = &store->main_backend;\n+\t\tgoto out;\n \tdefault:\n \t\tBUG(\"unhandled worktree reference type\");\n \t}\n+\n+out:\n+\tif (reload) {\n+\t\tint ret = reftable_stack_reload(be->stack);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n+\t*out = be;\n+\n+\treturn 0;\n }\n \n static int should_write_log(struct reftable_ref_store *refs, const char *refname)\n@@ -828,17 +845,17 @@ static int reftable_be_read_raw_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_raw_ref\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n+\tstruct reftable_backend *be;\n \tint ret;\n \n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret)\n \t\treturn ret;\n \n-\tret = read_ref_without_reload(refs, stack, refname, oid, referent, type);\n+\tret = read_ref_without_reload(refs, be->stack, refname, oid, referent, type);\n \tif (ret < 0)\n \t\treturn ret;\n \tif (ret > 0) {\n@@ -855,15 +872,15 @@ static int reftable_be_read_symbolic_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_symbolic_ref\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_ref_record ref = {0};\n+\tstruct reftable_backend *be;\n \tint ret;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret)\n \t\treturn ret;\n \n-\tret = reftable_stack_read_ref(stack, refname, &ref);\n+\tret = reftable_stack_read_ref(be->stack, refname, &ref);\n \tif (ret == 0 && ref.value_type == REFTABLE_REF_SYMREF)\n \t\tstrbuf_addstr(referent, ref.value.symref);\n \telse\n@@ -880,7 +897,7 @@ struct reftable_transaction_update {\n \n struct write_transaction_table_arg {\n \tstruct reftable_ref_store *refs;\n-\tstruct reftable_stack *stack;\n+\tstruct reftable_backend *be;\n \tstruct reftable_addition *addition;\n \tstruct reftable_transaction_update *updates;\n \tsize_t updates_nr;\n@@ -915,27 +932,37 @@ static int prepare_transaction_update(struct write_transaction_table_arg **out,\n \t\t\t\t      struct ref_update *update,\n \t\t\t\t      struct strbuf *err)\n {\n-\tstruct reftable_stack *stack = backend_for(refs, update->refname, NULL)->stack;\n \tstruct write_transaction_table_arg *arg = NULL;\n+\tstruct reftable_backend *be;\n \tsize_t i;\n \tint ret;\n \n+\t/*\n+\t * This function gets called in a loop, and we don't want to repeatedly\n+\t * reload the stack for every single ref update. Instead, we manually\n+\t * reload further down in the case where we haven't yet prepared the\n+\t * specific `reftable_backend`.\n+\t */\n+\tret = backend_for(&be, refs, update->refname, NULL, 0);\n+\tif (ret)\n+\t\treturn ret;\n+\n \t/*\n \t * Search for a preexisting stack update. If there is one then we add\n \t * the update to it, otherwise we set up a new stack update.\n \t */\n \tfor (i = 0; !arg && i < tx_data->args_nr; i++)\n-\t\tif (tx_data->args[i].stack == stack)\n+\t\tif (tx_data->args[i].be == be)\n \t\t\targ = &tx_data->args[i];\n \n \tif (!arg) {\n \t\tstruct reftable_addition *addition;\n \n-\t\tret = reftable_stack_reload(stack);\n+\t\tret = reftable_stack_reload(be->stack);\n \t\tif (ret)\n \t\t\treturn ret;\n \n-\t\tret = reftable_stack_new_addition(&addition, stack,\n+\t\tret = reftable_stack_new_addition(&addition, be->stack,\n \t\t\t\t\t\t  REFTABLE_STACK_NEW_ADDITION_RELOAD);\n \t\tif (ret) {\n \t\t\tif (ret == REFTABLE_LOCK_ERROR)\n@@ -947,7 +974,7 @@ static int prepare_transaction_update(struct write_transaction_table_arg **out,\n \t\t\t   tx_data->args_alloc);\n \t\targ = &tx_data->args[tx_data->args_nr++];\n \t\targ->refs = refs;\n-\t\targ->stack = stack;\n+\t\targ->be = be;\n \t\targ->addition = addition;\n \t\targ->updates = NULL;\n \t\targ->updates_nr = 0;\n@@ -1002,6 +1029,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \tstruct strbuf referent = STRBUF_INIT, head_referent = STRBUF_INIT;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n \tstruct reftable_transaction_data *tx_data = NULL;\n+\tstruct reftable_backend *be;\n \tstruct object_id head_oid;\n \tunsigned int head_type = 0;\n \tsize_t i;\n@@ -1048,7 +1076,22 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\tgoto done;\n \t}\n \n-\tret = read_ref_without_reload(refs, backend_for(refs, \"HEAD\", NULL)->stack, \"HEAD\",\n+\t/*\n+\t * TODO: it's dubious whether we should reload the stack that \"HEAD\"\n+\t * belongs to or not. In theory, it may happen that we only modify\n+\t * stacks which are _not_ part of the \"HEAD\" stack. In that case we\n+\t * wouldn't have prepared any transaction for its stack and would not\n+\t * have reloaded it, which may mean that it is stale.\n+\t *\n+\t * On the other hand, reloading that stack without locking it feels\n+\t * wrong to, as the value of \"HEAD\" could be modified concurrently at\n+\t * any point in time.\n+\t */\n+\tret = backend_for(&be, refs, \"HEAD\", NULL, 0);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = read_ref_without_reload(refs, be->stack, \"HEAD\",\n \t\t\t\t      &head_oid, &head_referent, &head_type);\n \tif (ret < 0)\n \t\tgoto done;\n@@ -1057,10 +1100,18 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \tfor (i = 0; i < transaction->nr; i++) {\n \t\tstruct ref_update *u = transaction->updates[i];\n \t\tstruct object_id current_oid = {0};\n-\t\tstruct reftable_stack *stack;\n \t\tconst char *rewritten_ref;\n \n-\t\tstack = backend_for(refs, u->refname, &rewritten_ref)->stack;\n+\t\t/*\n+\t\t * There is no need to reload the respective backends here as\n+\t\t * we have already reloaded them when preparing the transaction\n+\t\t * update. And given that the stacks have been locked there\n+\t\t * shouldn't have been any concurrent modifications of the\n+\t\t * stack.\n+\t\t */\n+\t\tret = backend_for(&be, refs, u->refname, &rewritten_ref, 0);\n+\t\tif (ret)\n+\t\t\tgoto done;\n \n \t\t/* Verify that the new object ID is valid. */\n \t\tif ((u->flags & REF_HAVE_NEW) && !is_null_oid(&u->new_oid) &&\n@@ -1116,7 +1167,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t\tstring_list_insert(&affected_refnames, new_update->refname);\n \t\t}\n \n-\t\tret = read_ref_without_reload(refs, stack, rewritten_ref,\n+\t\tret = read_ref_without_reload(refs, be->stack, rewritten_ref,\n \t\t\t\t\t      &current_oid, &referent, &u->type);\n \t\tif (ret < 0)\n \t\t\tgoto done;\n@@ -1318,7 +1369,7 @@ static int transaction_update_cmp(const void *a, const void *b)\n static int write_transaction_table(struct reftable_writer *writer, void *cb_data)\n {\n \tstruct write_transaction_table_arg *arg = cb_data;\n-\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tuint64_t ts = reftable_stack_next_update_index(arg->be->stack);\n \tstruct reftable_log_record *logs = NULL;\n \tstruct ident_split committer_ident = {0};\n \tsize_t logs_nr = 0, logs_alloc = 0, i;\n@@ -1354,7 +1405,7 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n \t\t\tstruct reftable_log_record log = {0};\n \t\t\tstruct reftable_iterator it = {0};\n \n-\t\t\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\t\t\tret = reftable_stack_init_log_iterator(arg->be->stack, &it);\n \t\t\tif (ret < 0)\n \t\t\t\tgoto done;\n \n@@ -1799,10 +1850,9 @@ static int reftable_be_rename_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"rename_ref\");\n-\tstruct reftable_stack *stack = backend_for(refs, newrefname, &newrefname)->stack;\n+\tstruct reftable_backend *be;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n-\t\t.stack = stack,\n \t\t.oldname = oldrefname,\n \t\t.newname = newrefname,\n \t\t.logmsg = logmsg,\n@@ -1814,10 +1864,11 @@ static int reftable_be_rename_ref(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, newrefname, &newrefname, 1);\n \tif (ret)\n \t\tgoto done;\n-\tret = reftable_stack_add(stack, &write_copy_table, &arg);\n+\targ.stack = be->stack;\n+\tret = reftable_stack_add(be->stack, &write_copy_table, &arg);\n \n done:\n \tassert(ret != REFTABLE_API_ERROR);\n@@ -1831,10 +1882,9 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"copy_ref\");\n-\tstruct reftable_stack *stack = backend_for(refs, newrefname, &newrefname)->stack;\n+\tstruct reftable_backend *be;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n-\t\t.stack = stack,\n \t\t.oldname = oldrefname,\n \t\t.newname = newrefname,\n \t\t.logmsg = logmsg,\n@@ -1845,10 +1895,11 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, newrefname, &newrefname, 1);\n \tif (ret)\n \t\tgoto done;\n-\tret = reftable_stack_add(stack, &write_copy_table, &arg);\n+\targ.stack = be->stack;\n+\tret = reftable_stack_add(be->stack, &write_copy_table, &arg);\n \n done:\n \tassert(ret != REFTABLE_API_ERROR);\n@@ -2012,15 +2063,23 @@ static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent_reverse\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record log = {0};\n \tstruct reftable_iterator it = {0};\n+\tstruct reftable_backend *be;\n \tint ret;\n \n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\tret = reftable_stack_init_log_iterator(stack, &it);\n+\t/*\n+\t * TODO: we should adapt this callsite to reload the stack. There is no\n+\t * obvious reason why we shouldn't.\n+\t */\n+\tret = backend_for(&be, refs, refname, &refname, 0);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_init_log_iterator(be->stack, &it);\n \tif (ret < 0)\n \t\tgoto done;\n \n@@ -2052,16 +2111,24 @@ static int reftable_be_for_each_reflog_ent(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record *logs = NULL;\n \tstruct reftable_iterator it = {0};\n+\tstruct reftable_backend *be;\n \tsize_t logs_alloc = 0, logs_nr = 0, i;\n \tint ret;\n \n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\tret = reftable_stack_init_log_iterator(stack, &it);\n+\t/*\n+\t * TODO: we should adapt this callsite to reload the stack. There is no\n+\t * obvious reason why we shouldn't.\n+\t */\n+\tret = backend_for(&be, refs, refname, &refname, 0);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_init_log_iterator(be->stack, &it);\n \tif (ret < 0)\n \t\tgoto done;\n \n@@ -2101,20 +2168,20 @@ static int reftable_be_reflog_exists(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_exists\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record log = {0};\n \tstruct reftable_iterator it = {0};\n+\tstruct reftable_backend *be;\n \tint ret;\n \n \tret = refs->err;\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(be->stack, &it);\n \tif (ret < 0)\n \t\tgoto done;\n \n@@ -2186,10 +2253,9 @@ static int reftable_be_create_reflog(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"create_reflog\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n+\tstruct reftable_backend *be;\n \tstruct write_reflog_existence_arg arg = {\n \t\t.refs = refs,\n-\t\t.stack = stack,\n \t\t.refname = refname,\n \t};\n \tint ret;\n@@ -2198,11 +2264,12 @@ static int reftable_be_create_reflog(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret)\n \t\tgoto done;\n+\targ.stack = be->stack;\n \n-\tret = reftable_stack_add(stack, &write_reflog_existence_table, &arg);\n+\tret = reftable_stack_add(be->stack, &write_reflog_existence_table, &arg);\n \n done:\n \treturn ret;\n@@ -2260,17 +2327,18 @@ static int reftable_be_delete_reflog(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"delete_reflog\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n+\tstruct reftable_backend *be;\n \tstruct write_reflog_delete_arg arg = {\n-\t\t.stack = stack,\n \t\t.refname = refname,\n \t};\n \tint ret;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret)\n \t\treturn ret;\n-\tret = reftable_stack_add(stack, &write_reflog_delete_table, &arg);\n+\targ.stack = be->stack;\n+\n+\tret = reftable_stack_add(be->stack, &write_reflog_delete_table, &arg);\n \n \tassert(ret != REFTABLE_API_ERROR);\n \treturn ret;\n@@ -2369,13 +2437,13 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \t */\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"reflog_expire\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record *logs = NULL;\n \tstruct reftable_log_record *rewritten = NULL;\n \tstruct reftable_ref_record ref_record = {0};\n \tstruct reftable_iterator it = {0};\n \tstruct reftable_addition *add = NULL;\n \tstruct reflog_expiry_arg arg = {0};\n+\tstruct reftable_backend *be;\n \tstruct object_id oid = {0};\n \tuint8_t *last_hash = NULL;\n \tsize_t logs_nr = 0, logs_alloc = 0, i;\n@@ -2384,11 +2452,11 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(be->stack, &it);\n \tif (ret < 0)\n \t\tgoto done;\n \n@@ -2396,11 +2464,11 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_new_addition(&add, stack, 0);\n+\tret = reftable_stack_new_addition(&add, be->stack, 0);\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_read_ref(stack, refname, &ref_record);\n+\tret = reftable_stack_read_ref(be->stack, refname, &ref_record);\n \tif (ret < 0)\n \t\tgoto done;\n \tif (reftable_ref_record_val1(&ref_record))\n@@ -2479,8 +2547,8 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \targ.refs = refs;\n \targ.records = rewritten;\n \targ.len = logs_nr;\n-\targ.stack = stack,\n-\targ.refname = refname,\n+\targ.stack = be->stack;\n+\targ.refname = refname;\n \n \tret = reftable_addition_add(add, &write_reflog_expiry_table, &arg);\n \tif (ret < 0)\n\n-- \n2.47.0.274.g962d0b743d.dirty\n\n"},{"id":"508043","messageId":"20241125-pks-reftable-backend-reuse-iter-v3-3-1d7b658e3e9e@pks.im","threadId":"62444","inReplyTo":"20241125-pks-reftable-backend-reuse-iter-v3-0-1d7b658e3e9e@pks.im","subject":"[PATCH v3 3/9] reftable/stack: add accessor for the hash ID","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-25T07:38:25Z","receivedAt":"2024-11-25T07:38:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Add an accessor function that allows callers to access the hash ID of a\nreftable stack. This function will be used in a subsequent commit.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n reftable/reftable-stack.h | 3 +++\n reftable/stack.c          | 5 +++++\n 2 files changed, 8 insertions(+)\n\ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nindex 54787f2ef530406a7970db058c3a0cf456897978..ae14270ea74108cd4c314ec38e7d5c9a4e731481 100644\n--- a/reftable/reftable-stack.h\n+++ b/reftable/reftable-stack.h\n@@ -149,4 +149,7 @@ struct reftable_compaction_stats {\n struct reftable_compaction_stats *\n reftable_stack_compaction_stats(struct reftable_stack *st);\n \n+/* Return the hash of the stack. */\n+enum reftable_hash reftable_stack_hash_id(struct reftable_stack *st);\n+\n #endif\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 1fffd75630266c0d3d1e4a2d037b5cf90455529d..d97b64a40d4ad05cfd9e6f33e8ba1e713281ef6d 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1791,3 +1791,8 @@ int reftable_stack_clean(struct reftable_stack *st)\n \treftable_addition_destroy(add);\n \treturn err;\n }\n+\n+enum reftable_hash reftable_stack_hash_id(struct reftable_stack *st)\n+{\n+\treturn reftable_merged_table_hash_id(st->merged);\n+}\n\n-- \n2.47.0.274.g962d0b743d.dirty\n\n"},{"id":"508044","messageId":"20241125-pks-reftable-backend-reuse-iter-v3-4-1d7b658e3e9e@pks.im","threadId":"62444","inReplyTo":"20241125-pks-reftable-backend-reuse-iter-v3-0-1d7b658e3e9e@pks.im","subject":"[PATCH v3 4/9] refs/reftable: read references via `struct reftable_backend`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-25T07:38:26Z","receivedAt":"2024-11-25T07:38:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor `read_ref_without_reload()` to accept a `struct reftable_stack`\nas input instead of accepting a `struct reftable_stack`. This allows us\nto implement an additional caching layer when reading refs where we can\nreuse reftable iterators.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 110 ++++++++++++++++++++++++++----------------------\n 1 file changed, 59 insertions(+), 51 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 64fe8fd02d8ec932f7980cdb7d7d5223c3c83b73..5933c561f5c422d12b616514ed76b75c52a13477 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -51,6 +51,50 @@ static void reftable_backend_release(struct reftable_backend *be)\n \tbe->stack = NULL;\n }\n \n+static int reftable_backend_read_ref(struct reftable_backend *be,\n+\t\t\t\t     const char *refname,\n+\t\t\t\t     struct object_id *oid,\n+\t\t\t\t     struct strbuf *referent,\n+\t\t\t\t     unsigned int *type)\n+{\n+\tstruct reftable_ref_record ref = {0};\n+\tint ret;\n+\n+\tret = reftable_stack_read_ref(be->stack, refname, &ref);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tstrbuf_reset(referent);\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\t\t*type |= REF_ISSYMREF;\n+\t} else if (reftable_ref_record_val1(&ref)) {\n+\t\tunsigned int hash_id;\n+\n+\t\tswitch (reftable_stack_hash_id(be->stack)) {\n+\t\tcase REFTABLE_HASH_SHA1:\n+\t\t\thash_id = GIT_HASH_SHA1;\n+\t\t\tbreak;\n+\t\tcase REFTABLE_HASH_SHA256:\n+\t\t\thash_id = GIT_HASH_SHA256;\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tBUG(\"unhandled hash ID %d\", reftable_stack_hash_id(be->stack));\n+\t\t}\n+\n+\t\toidread(oid, reftable_ref_record_val1(&ref),\n+\t\t\t&hash_algos[hash_id]);\n+\t} else {\n+\t\t/* We got a tombstone, which should not happen. */\n+\t\tBUG(\"unhandled reference value type %d\", ref.value_type);\n+\t}\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn ret;\n+}\n+\n struct reftable_ref_store {\n \tstruct ref_store base;\n \n@@ -243,38 +287,6 @@ static void fill_reftable_log_record(struct reftable_log_record *log, const stru\n \tlog->value.update.tz_offset = sign * atoi(tz_begin);\n }\n \n-static int read_ref_without_reload(struct reftable_ref_store *refs,\n-\t\t\t\t   struct reftable_stack *stack,\n-\t\t\t\t   const char *refname,\n-\t\t\t\t   struct object_id *oid,\n-\t\t\t\t   struct strbuf *referent,\n-\t\t\t\t   unsigned int *type)\n-{\n-\tstruct reftable_ref_record ref = {0};\n-\tint ret;\n-\n-\tret = reftable_stack_read_ref(stack, refname, &ref);\n-\tif (ret)\n-\t\tgoto done;\n-\n-\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n-\t\tstrbuf_reset(referent);\n-\t\tstrbuf_addstr(referent, ref.value.symref);\n-\t\t*type |= REF_ISSYMREF;\n-\t} else if (reftable_ref_record_val1(&ref)) {\n-\t\toidread(oid, reftable_ref_record_val1(&ref),\n-\t\t\trefs->base.repo->hash_algo);\n-\t} else {\n-\t\t/* We got a tombstone, which should not happen. */\n-\t\tBUG(\"unhandled reference value type %d\", ref.value_type);\n-\t}\n-\n-done:\n-\tassert(ret != REFTABLE_API_ERROR);\n-\treftable_ref_record_release(&ref);\n-\treturn ret;\n-}\n-\n static int reftable_be_config(const char *var, const char *value,\n \t\t\t      const struct config_context *ctx,\n \t\t\t      void *_opts)\n@@ -855,7 +867,7 @@ static int reftable_be_read_raw_ref(struct ref_store *ref_store,\n \tif (ret)\n \t\treturn ret;\n \n-\tret = read_ref_without_reload(refs, be->stack, refname, oid, referent, type);\n+\tret = reftable_backend_read_ref(be, refname, oid, referent, type);\n \tif (ret < 0)\n \t\treturn ret;\n \tif (ret > 0) {\n@@ -1091,8 +1103,8 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \tif (ret)\n \t\tgoto done;\n \n-\tret = read_ref_without_reload(refs, be->stack, \"HEAD\",\n-\t\t\t\t      &head_oid, &head_referent, &head_type);\n+\tret = reftable_backend_read_ref(be, \"HEAD\", &head_oid,\n+\t\t\t\t\t&head_referent, &head_type);\n \tif (ret < 0)\n \t\tgoto done;\n \tret = 0;\n@@ -1167,8 +1179,8 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t\tstring_list_insert(&affected_refnames, new_update->refname);\n \t\t}\n \n-\t\tret = read_ref_without_reload(refs, be->stack, rewritten_ref,\n-\t\t\t\t\t      &current_oid, &referent, &u->type);\n+\t\tret = reftable_backend_read_ref(be, rewritten_ref,\n+\t\t\t\t\t\t&current_oid, &referent, &u->type);\n \t\tif (ret < 0)\n \t\t\tgoto done;\n \t\tif (ret > 0 && !ref_update_expects_existing_old_ref(u)) {\n@@ -1626,7 +1638,7 @@ struct write_create_symref_arg {\n \n struct write_copy_arg {\n \tstruct reftable_ref_store *refs;\n-\tstruct reftable_stack *stack;\n+\tstruct reftable_backend *be;\n \tconst char *oldname;\n \tconst char *newname;\n \tconst char *logmsg;\n@@ -1651,7 +1663,7 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n \tif (split_ident_line(&committer_ident, committer_info, strlen(committer_info)))\n \t\tBUG(\"failed splitting committer info\");\n \n-\tif (reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref)) {\n+\tif (reftable_stack_read_ref(arg->be->stack, arg->oldname, &old_ref)) {\n \t\tret = error(_(\"refname %s not found\"), arg->oldname);\n \t\tgoto done;\n \t}\n@@ -1690,7 +1702,7 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n \t * the old branch and the creation of the new branch, and we cannot do\n \t * two changes to a reflog in a single update.\n \t */\n-\tdeletion_ts = creation_ts = reftable_stack_next_update_index(arg->stack);\n+\tdeletion_ts = creation_ts = reftable_stack_next_update_index(arg->be->stack);\n \tif (arg->delete_old)\n \t\tcreation_ts++;\n \treftable_writer_set_limits(writer, deletion_ts, creation_ts);\n@@ -1733,8 +1745,8 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n \t\tmemcpy(logs[logs_nr].value.update.old_hash, old_ref.value.val1, GIT_MAX_RAWSZ);\n \t\tlogs_nr++;\n \n-\t\tret = read_ref_without_reload(arg->refs, arg->stack, \"HEAD\", &head_oid,\n-\t\t\t\t\t      &head_referent, &head_type);\n+\t\tret = reftable_backend_read_ref(arg->be, \"HEAD\", &head_oid,\n+\t\t\t\t\t\t&head_referent, &head_type);\n \t\tif (ret < 0)\n \t\t\tgoto done;\n \t\tappend_head_reflog = (head_type & REF_ISSYMREF) && !strcmp(head_referent.buf, arg->oldname);\n@@ -1777,7 +1789,7 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n \t * copy over all log entries from the old reflog. Last but not least,\n \t * when renaming we also have to delete all the old reflog entries.\n \t */\n-\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\tret = reftable_stack_init_log_iterator(arg->be->stack, &it);\n \tif (ret < 0)\n \t\tgoto done;\n \n@@ -1850,7 +1862,6 @@ static int reftable_be_rename_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"rename_ref\");\n-\tstruct reftable_backend *be;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n \t\t.oldname = oldrefname,\n@@ -1864,11 +1875,10 @@ static int reftable_be_rename_ref(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = backend_for(&be, refs, newrefname, &newrefname, 1);\n+\tret = backend_for(&arg.be, refs, newrefname, &newrefname, 1);\n \tif (ret)\n \t\tgoto done;\n-\targ.stack = be->stack;\n-\tret = reftable_stack_add(be->stack, &write_copy_table, &arg);\n+\tret = reftable_stack_add(arg.be->stack, &write_copy_table, &arg);\n \n done:\n \tassert(ret != REFTABLE_API_ERROR);\n@@ -1882,7 +1892,6 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"copy_ref\");\n-\tstruct reftable_backend *be;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n \t\t.oldname = oldrefname,\n@@ -1895,11 +1904,10 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = backend_for(&be, refs, newrefname, &newrefname, 1);\n+\tret = backend_for(&arg.be, refs, newrefname, &newrefname, 1);\n \tif (ret)\n \t\tgoto done;\n-\targ.stack = be->stack;\n-\tret = reftable_stack_add(be->stack, &write_copy_table, &arg);\n+\tret = reftable_stack_add(arg.be->stack, &write_copy_table, &arg);\n \n done:\n \tassert(ret != REFTABLE_API_ERROR);\n\n-- \n2.47.0.274.g962d0b743d.dirty\n\n"},{"id":"508045","messageId":"20241125-pks-reftable-backend-reuse-iter-v3-5-1d7b658e3e9e@pks.im","threadId":"62444","inReplyTo":"20241125-pks-reftable-backend-reuse-iter-v3-0-1d7b658e3e9e@pks.im","subject":"[PATCH v3 5/9] refs/reftable: refactor reading symbolic refs to use reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-25T07:38:27Z","receivedAt":"2024-11-25T07:38:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor the callback function that reads symbolic references in the\nreftable backend to use `reftable_backend_read_ref()` instead of\naccessing the reftable stack directly. This ensures that the function\nwill benefit from the new caching layer that we're about to introduce.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 11 ++++-------\n 1 file changed, 4 insertions(+), 7 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 5933c561f5c422d12b616514ed76b75c52a13477..498bc9f932673e6089bd3b27e1bb7ed8d0e36a4c 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -884,21 +884,18 @@ static int reftable_be_read_symbolic_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_symbolic_ref\");\n-\tstruct reftable_ref_record ref = {0};\n \tstruct reftable_backend *be;\n+\tstruct object_id oid;\n+\tunsigned int type = 0;\n \tint ret;\n \n \tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret)\n \t\treturn ret;\n \n-\tret = reftable_stack_read_ref(be->stack, refname, &ref);\n-\tif (ret == 0 && ref.value_type == REFTABLE_REF_SYMREF)\n-\t\tstrbuf_addstr(referent, ref.value.symref);\n-\telse\n+\tret = reftable_backend_read_ref(be, refname, &oid, referent, &type);\n+\tif (type != REF_ISSYMREF)\n \t\tret = -1;\n-\n-\treftable_ref_record_release(&ref);\n \treturn ret;\n }\n \n\n-- \n2.47.0.274.g962d0b743d.dirty\n\n"},{"id":"508046","messageId":"20241125-pks-reftable-backend-reuse-iter-v3-6-1d7b658e3e9e@pks.im","threadId":"62444","inReplyTo":"20241125-pks-reftable-backend-reuse-iter-v3-0-1d7b658e3e9e@pks.im","subject":"[PATCH v3 6/9] refs/reftable: refactor reflog expiry to use reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-25T07:38:28Z","receivedAt":"2024-11-25T07:38:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor the callback function that expires reflog entries in the\nreftable backend to use `reftable_backend_read_ref()` instead of\naccessing the reftable stack directly. This ensures that the function\nwill benefit from the new caching layer that we're about to introduce.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 13 +++++--------\n 1 file changed, 5 insertions(+), 8 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 498bc9f932673e6089bd3b27e1bb7ed8d0e36a4c..ff49c27a6a0f4cb1e5b3bfaa3d34d3302c1bdb2e 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2444,14 +2444,15 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"reflog_expire\");\n \tstruct reftable_log_record *logs = NULL;\n \tstruct reftable_log_record *rewritten = NULL;\n-\tstruct reftable_ref_record ref_record = {0};\n \tstruct reftable_iterator it = {0};\n \tstruct reftable_addition *add = NULL;\n \tstruct reflog_expiry_arg arg = {0};\n \tstruct reftable_backend *be;\n \tstruct object_id oid = {0};\n+\tstruct strbuf referent = STRBUF_INIT;\n \tuint8_t *last_hash = NULL;\n \tsize_t logs_nr = 0, logs_alloc = 0, i;\n+\tunsigned int type = 0;\n \tint ret;\n \n \tif (refs->err < 0)\n@@ -2473,12 +2474,9 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_read_ref(be->stack, refname, &ref_record);\n+\tret = reftable_backend_read_ref(be, refname, &oid, &referent, &type);\n \tif (ret < 0)\n \t\tgoto done;\n-\tif (reftable_ref_record_val1(&ref_record))\n-\t\toidread(&oid, reftable_ref_record_val1(&ref_record),\n-\t\t\tref_store->repo->hash_algo);\n \tprepare_fn(refname, &oid, policy_cb_data);\n \n \twhile (1) {\n@@ -2545,8 +2543,7 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \t\t}\n \t}\n \n-\tif (flags & EXPIRE_REFLOGS_UPDATE_REF && last_hash &&\n-\t    reftable_ref_record_val1(&ref_record))\n+\tif (flags & EXPIRE_REFLOGS_UPDATE_REF && last_hash && !is_null_oid(&oid))\n \t\toidread(&arg.update_oid, last_hash, ref_store->repo->hash_algo);\n \n \targ.refs = refs;\n@@ -2571,11 +2568,11 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \t\tcleanup_fn(policy_cb_data);\n \tassert(ret != REFTABLE_API_ERROR);\n \n-\treftable_ref_record_release(&ref_record);\n \treftable_iterator_destroy(&it);\n \treftable_addition_destroy(add);\n \tfor (i = 0; i < logs_nr; i++)\n \t\treftable_log_record_release(&logs[i]);\n+\tstrbuf_release(&referent);\n \tfree(logs);\n \tfree(rewritten);\n \treturn ret;\n\n-- \n2.47.0.274.g962d0b743d.dirty\n\n"},{"id":"508047","messageId":"20241125-pks-reftable-backend-reuse-iter-v3-7-1d7b658e3e9e@pks.im","threadId":"62444","inReplyTo":"20241125-pks-reftable-backend-reuse-iter-v3-0-1d7b658e3e9e@pks.im","subject":"[PATCH v3 7/9] reftable/stack: add mechanism to notify callers on reload","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-25T07:38:29Z","receivedAt":"2024-11-25T07:38:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Reftable stacks are reloaded in two cases:\n\n  - When calling `reftable_stack_reload()`, if the stat-cache tells us\n    that the stack has been modified.\n\n  - When committing a reftable addition.\n\nWhile callers can figure out the second case, they do not have a\nmechanism to figure out whether `reftable_stack_reload()` led to an\nactual reload of the on-disk data. All they can do is thus to assume\nthat data is always being reloaded in that case.\n\nImprove the situation by introducing a new `on_reload()` callback to the\nreftable options. If provided, the function will be invoked every time\nthe stack has indeed been reloaded. This allows callers to invalidate\ndata that depends on the current stack data.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n reftable/reftable-writer.h | 9 +++++++++\n reftable/stack.c           | 4 ++++\n 2 files changed, 13 insertions(+)\n\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex c85ef5a5bd14595d75f99457fef4407040e197c5..5f9afa620bb00de66c311765fb0ae8c6f56401ae 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -68,6 +68,15 @@ struct reftable_write_options {\n \t * fsync(3P) when unset.\n \t */\n \tint (*fsync)(int fd);\n+\n+\t/*\n+\t * Callback function to execute whenever the stack is being reloaded.\n+\t * This can be used e.g. to discard cached information that relies on\n+\t * the old stack's data. The payload data will be passed as argument to\n+\t * the callback.\n+\t */\n+\tvoid (*on_reload)(void *payload);\n+\tvoid *on_reload_payload;\n };\n \n /* reftable_block_stats holds statistics for a single block type */\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex d97b64a40d4ad05cfd9e6f33e8ba1e713281ef6d..5384ca9de0d1f064aebcb09308a74cc397b37463 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -548,6 +548,10 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n \t\tclose(fd);\n \tfree_names(names);\n \tfree_names(names_after);\n+\n+\tif (st->opts.on_reload)\n+\t\tst->opts.on_reload(st->opts.on_reload_payload);\n+\n \treturn err;\n }\n \n\n-- \n2.47.0.274.g962d0b743d.dirty\n\n"},{"id":"508048","messageId":"20241125-pks-reftable-backend-reuse-iter-v3-8-1d7b658e3e9e@pks.im","threadId":"62444","inReplyTo":"20241125-pks-reftable-backend-reuse-iter-v3-0-1d7b658e3e9e@pks.im","subject":"[PATCH v3 8/9] reftable/merged: drain priority queue on reseek","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-25T07:38:30Z","receivedAt":"2024-11-25T07:38:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In 5bf96e0c39 (reftable/generic: move seeking of records into the\niterator, 2024-05-13) we have refactored the reftable codebase such that\niterators can be initialized once and then re-seeked multiple times.\nThis feature is used by 1869525066 (refs/reftable: wire up support for\nexclude patterns, 2024-09-16) in order to skip records based on exclude\npatterns provided by the caller.\n\nThe logic to re-seek the merged iterator is insufficient though because\nwe don't drain the priority queue on a re-seek. This means that the\nqueue may contain stale entries and thus reading the next record in the\nqueue will return the wrong entry. While this is an obvious bug, it is\nharmless in the context of above exclude patterns:\n\n  - If the queue contained stale entries that match the pattern then the\n    caller would already know to filter out such refs. This is because\n    our codebase is prepared to handle backends that don't have a way to\n    efficiently implement exclude patterns.\n\n  - If the queue contained stale entries that don't match the pattern\n    we'd eventually filter out any duplicates. This is because the\n    reftable code discards items with the same ref name and sorts any\n    remaining entries properly.\n\nSo things happen to work in this context regardless of the bug, and\nthere is no other use case yet where we re-seek iterators. We're about\nto introduce a caching mechanism though where iterators are reused by\nthe reftable backend, and that will expose the bug.\n\nFix the issue by draining the priority queue when seeking and add a\ntestcase that surfaces the issue.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n reftable/merged.c                |  2 ++\n t/unit-tests/t-reftable-merged.c | 73 ++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 75 insertions(+)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 5b93e20f42945300abbc1a036bbdf067fced7854..bb0836e3443271f9c0d5ba5582c78694d437ddc2 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -66,6 +66,8 @@ static int merged_iter_seek(struct merged_iter *mi, struct reftable_record *want\n \tint err;\n \n \tmi->advance_index = -1;\n+\twhile (!merged_iter_pqueue_is_empty(mi->pq))\n+\t\tmerged_iter_pqueue_remove(&mi->pq);\n \n \tfor (size_t i = 0; i < mi->subiters_len; i++) {\n \t\terr = iterator_seek(&mi->subiters[i].iter, want);\ndiff --git a/t/unit-tests/t-reftable-merged.c b/t/unit-tests/t-reftable-merged.c\nindex 2591b5e59745536a205271491f747875e04c5a3f..a12bd0e1a3bdeda82bcbc6259e679df4f232e3e2 100644\n--- a/t/unit-tests/t-reftable-merged.c\n+++ b/t/unit-tests/t-reftable-merged.c\n@@ -273,6 +273,78 @@ static void t_merged_seek_multiple_times(void)\n \treftable_free(sources);\n }\n \n+static void t_merged_seek_multiple_times_without_draining(void)\n+{\n+\tstruct reftable_ref_record r1[] = {\n+\t\t{\n+\t\t\t.refname = (char *) \"a\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = { 1 },\n+\t\t},\n+\t\t{\n+\t\t\t.refname = (char *) \"c\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = { 2 },\n+\t\t}\n+\t};\n+\tstruct reftable_ref_record r2[] = {\n+\t\t{\n+\t\t\t.refname = (char *) \"b\",\n+\t\t\t.update_index = 2,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = { 3 },\n+\t\t},\n+\t\t{\n+\t\t\t.refname = (char *) \"d\",\n+\t\t\t.update_index = 2,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = { 4 },\n+\t\t},\n+\t};\n+\tstruct reftable_ref_record *refs[] = {\n+\t\tr1, r2,\n+\t};\n+\tsize_t sizes[] = {\n+\t\tARRAY_SIZE(r1), ARRAY_SIZE(r2),\n+\t};\n+\tstruct reftable_buf bufs[] = {\n+\t\tREFTABLE_BUF_INIT, REFTABLE_BUF_INIT,\n+\t};\n+\tstruct reftable_block_source *sources = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_ref_record rec = { 0 };\n+\tstruct reftable_iterator it = { 0 };\n+\tstruct reftable_merged_table *mt;\n+\tint err;\n+\n+\tmt = merged_table_from_records(refs, &sources, &readers, sizes, bufs, 2);\n+\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\n+\terr = reftable_iterator_seek_ref(&it, \"b\");\n+\tcheck(!err);\n+\terr = reftable_iterator_next_ref(&it, &rec);\n+\tcheck(!err);\n+\terr = reftable_ref_record_equal(&rec, &r2[0], REFTABLE_HASH_SIZE_SHA1);\n+\tcheck(err == 1);\n+\n+\terr = reftable_iterator_seek_ref(&it, \"a\");\n+\tcheck(!err);\n+\terr = reftable_iterator_next_ref(&it, &rec);\n+\tcheck(!err);\n+\terr = reftable_ref_record_equal(&rec, &r1[0], REFTABLE_HASH_SIZE_SHA1);\n+\tcheck(err == 1);\n+\n+\tfor (size_t i = 0; i < ARRAY_SIZE(bufs); i++)\n+\t\treftable_buf_release(&bufs[i]);\n+\treaders_destroy(readers, ARRAY_SIZE(refs));\n+\treftable_ref_record_release(&rec);\n+\treftable_iterator_destroy(&it);\n+\treftable_merged_table_free(mt);\n+\treftable_free(sources);\n+}\n+\n static struct reftable_merged_table *\n merged_table_from_log_records(struct reftable_log_record **logs,\n \t\t\t      struct reftable_block_source **source,\n@@ -467,6 +539,7 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \tTEST(t_merged_logs(), \"merged table with multiple log updates for same ref\");\n \tTEST(t_merged_refs(), \"merged table with multiple updates to same ref\");\n \tTEST(t_merged_seek_multiple_times(), \"merged table can seek multiple times\");\n+\tTEST(t_merged_seek_multiple_times_without_draining(), \"merged table can seek multiple times without draining\");\n \tTEST(t_merged_single_record(), \"ref occurring in only one record can be fetched\");\n \n \treturn test_done();\n\n-- \n2.47.0.274.g962d0b743d.dirty\n\n"},{"id":"508049","messageId":"20241125-pks-reftable-backend-reuse-iter-v3-9-1d7b658e3e9e@pks.im","threadId":"62444","inReplyTo":"20241125-pks-reftable-backend-reuse-iter-v3-0-1d7b658e3e9e@pks.im","subject":"[PATCH v3 9/9] refs/reftable: reuse iterators when reading refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-25T07:38:31Z","receivedAt":"2024-11-25T07:38:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When reading references the reftable backend has to:\n\n  1. Create a new ref iterator.\n\n  2. Seek the iterator to the record we're searching for.\n\n  3. Read the record.\n\nWe cannot really avoid the last two steps, but re-creating the iterator\nevery single time we want to read a reference is kind of expensive and a\nwaste of resources. We couldn't help it in the past though because it\nwas not possible to reuse iterators. But starting with 5bf96e0c39\n(reftable/generic: move seeking of records into the iterator,\n2024-05-13) we have split up the iterator lifecycle such that creating\nthe iterator and seeking are two different concerns.\n\nRefactor the code such that we cache iterators in the reftable backend.\nThis cache is invalidated whenever the respective stack is reloaded such\nthat we know to recreate the iterator in that case. This leads to a\nsizeable speedup when creating many refs, which requires a lot of random\nreference reads:\n\n    Benchmark 1: update-ref: create many refs (refcount = 100000, revision = master)\n      Time (mean ± σ):      1.793 s ±  0.010 s    [User: 0.954 s, System: 0.835 s]\n      Range (min … max):    1.781 s …  1.811 s    10 runs\n\n    Benchmark 2: update-ref: create many refs (refcount = 100000, revision = HEAD)\n      Time (mean ± σ):      1.680 s ±  0.013 s    [User: 0.846 s, System: 0.831 s]\n      Range (min … max):    1.664 s …  1.702 s    10 runs\n\n    Summary\n      update-ref: create many refs (refcount = 100000, revision = HEAD) ran\n        1.07 ± 0.01 times faster than update-ref: create many refs (refcount = 100000, revision = master)\n\nWhile 7% is not a huge win, you have to consider that the benchmark is\n_writing_ data, so _reading_ references is only one part of what we do.\nFlame graphs show that we spend around 40% of our time reading refs, so\nthe speedup when reading refs is approximately ~2.5x that. I could not\nfind better benchmarks where we perform a lot of random ref reads.\n\nYou can also see a sizeable impact on memory usage when creating 100k\nreferences. Before this change:\n\n    HEAP SUMMARY:\n        in use at exit: 19,112,538 bytes in 200,170 blocks\n      total heap usage: 8,400,426 allocs, 8,200,256 frees, 454,367,048 bytes allocated\n\nAfter this change:\n\n    HEAP SUMMARY:\n        in use at exit: 674,416 bytes in 169 blocks\n      total heap usage: 7,929,872 allocs, 7,929,703 frees, 281,509,985 bytes allocated\n\nAs an additional factor, this refactoring opens up the possibility for\nmore performance optimizations in how we re-seek iterators. Any change\nthat allows us to optimize re-seeking by e.g. reusing data structures\nwould thus also directly speed up random reads.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 32 +++++++++++++++++++++++++++++---\n 1 file changed, 29 insertions(+), 3 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex ff49c27a6a0f4cb1e5b3bfaa3d34d3302c1bdb2e..417c6c4955672ef69fe956e2e6dd8dbd1dd15a3c 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -36,19 +36,30 @@\n \n struct reftable_backend {\n \tstruct reftable_stack *stack;\n+\tstruct reftable_iterator it;\n };\n \n+static void reftable_backend_on_reload(void *payload)\n+{\n+\tstruct reftable_backend *be = payload;\n+\treftable_iterator_destroy(&be->it);\n+}\n+\n static int reftable_backend_init(struct reftable_backend *be,\n \t\t\t\t const char *path,\n-\t\t\t\t const struct reftable_write_options *opts)\n+\t\t\t\t const struct reftable_write_options *_opts)\n {\n-\treturn reftable_new_stack(&be->stack, path, opts);\n+\tstruct reftable_write_options opts = *_opts;\n+\topts.on_reload = reftable_backend_on_reload;\n+\topts.on_reload_payload = be;\n+\treturn reftable_new_stack(&be->stack, path, &opts);\n }\n \n static void reftable_backend_release(struct reftable_backend *be)\n {\n \treftable_stack_destroy(be->stack);\n \tbe->stack = NULL;\n+\treftable_iterator_destroy(&be->it);\n }\n \n static int reftable_backend_read_ref(struct reftable_backend *be,\n@@ -60,10 +71,25 @@ static int reftable_backend_read_ref(struct reftable_backend *be,\n \tstruct reftable_ref_record ref = {0};\n \tint ret;\n \n-\tret = reftable_stack_read_ref(be->stack, refname, &ref);\n+\tif (!be->it.ops) {\n+\t\tret = reftable_stack_init_ref_iterator(be->stack, &be->it);\n+\t\tif (ret)\n+\t\t\tgoto done;\n+\t}\n+\n+\tret = reftable_iterator_seek_ref(&be->it, refname);\n \tif (ret)\n \t\tgoto done;\n \n+\tret = reftable_iterator_next_ref(&be->it, &ref);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tif (strcmp(ref.refname, refname)) {\n+\t\tret = 1;\n+\t\tgoto done;\n+\t}\n+\n \tif (ref.value_type == REFTABLE_REF_SYMREF) {\n \t\tstrbuf_reset(referent);\n \t\tstrbuf_addstr(referent, ref.value.symref);\n\n-- \n2.47.0.274.g962d0b743d.dirty\n\n"},{"id":"508060","messageId":"CAP8UFD2woZV1O-B9vxEdM-1_R-oTDgt-k1RKt1wJJsofZHb9rQ@mail.gmail.com","threadId":"62444","inReplyTo":"20241125-pks-reftable-backend-reuse-iter-v3-0-1d7b658e3e9e@pks.im","subject":"Re: [PATCH v3 0/9] refs/reftable: reuse iterators when reading refs","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-11-25T09:47:57Z","receivedAt":"2024-11-25T09:48:12Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Hi,\n\nOn Mon, Nov 25, 2024 at 8:38 AM Patrick Steinhardt <ps@pks.im> wrote:\n>\n> Hi,\n>\n> this is the second version of my patch series that refactors the\n\nDo you mean that despite the \"v3\" mark, it's only the second version\nbecause \"v2\" was just rebasing \"v1\" on top of a better base?\n\n> reftable backend to reuse iterators when reading random references. This\n> removes the overhead of having to recreate the iterator on every read\n> and thus leads to better performance and less allocation churn.\n>\n> Changes in v3:\n>\n>   - Adapt some comments to refer to the \"backend\" instead of to the\n>     \"stack\".\n>   - Fix indentation of a statement while at it.\n>   - Explain why callsites don't want to reload the stack.\n>   - Optimize `prepare_transaction_update()` by not using `backend_for()`\n>     twice, but instead reload the stack manually.\n>   - Split out the change that adds `reftable_stack_hash_id()` into a\n>     separate commit.\n>   - Link to v2: https://lore.kernel.org/r/cover.1730792627.git.ps@pks.im\n\nThanks!\n"},{"id":"508065","messageId":"Z0RIyXF2C-YFGCQA@pks.im","threadId":"62444","inReplyTo":"CAP8UFD2woZV1O-B9vxEdM-1_R-oTDgt-k1RKt1wJJsofZHb9rQ@mail.gmail.com","subject":"Re: [PATCH v3 0/9] refs/reftable: reuse iterators when reading refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-25T09:52:09Z","receivedAt":"2024-11-25T09:52:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Nov 25, 2024 at 10:47:57AM +0100, Christian Couder wrote:\n> Hi,\n> \n> On Mon, Nov 25, 2024 at 8:38 AM Patrick Steinhardt <ps@pks.im> wrote:\n> >\n> > Hi,\n> >\n> > this is the second version of my patch series that refactors the\n> \n> Do you mean that despite the \"v3\" mark, it's only the second version\n> because \"v2\" was just rebasing \"v1\" on top of a better base?\n\nNah, I simply forgot to update this introduction :)\n\nPatrick\n"},{"id":"508119","messageId":"xmqqa5dm4yzm.fsf@gitster.g","threadId":"62444","inReplyTo":"20241125-pks-reftable-backend-reuse-iter-v3-2-1d7b658e3e9e@pks.im","subject":"Re: [PATCH v3 2/9] refs/reftable: handle reloading stacks in the reftable backend","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-26T00:31:25Z","receivedAt":"2024-11-26T00:31:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> +\t * On the other hand, reloading that stack without locking it feels\n> +\t * wrong to, as the value of \"HEAD\" could be modified concurrently at\n> +\t * any point in time.\n> +\t */\n\n\"wrong to,\" -> \"wrong too,\", probably.\n"},{"id":"508120","messageId":"xmqq34je4y6m.fsf@gitster.g","threadId":"62444","inReplyTo":"20241125-pks-reftable-backend-reuse-iter-v3-4-1d7b658e3e9e@pks.im","subject":"Re: [PATCH v3 4/9] refs/reftable: read references via `struct reftable_backend`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-26T00:48:49Z","receivedAt":"2024-11-26T00:48:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> Refactor `read_ref_without_reload()` to accept a `struct reftable_stack`\n> as input instead of accepting a `struct reftable_stack`.\n\nHmph...  am I supposed to be puzzled that the patch replaces A with\nA?\n\nAll callers of read_ref_without_reload() now call\nreftable_backend_read_ref().  \n\nThe former took <reftable_ref_store, reftable_stack> while the\nlatter take <reftable_backend>.  They both take <refname, oid,\nreferent, type>, of course, because the former is replaced by the\nlatter.\n\nOK, so we introduce a new function, and instead of passing ref-store\nand stack, the new function only takes reftable-backend (which has a\nstack contained in it).\n\nThe old function used ref-store only to find out the hash algorithm\nvia its base repository.  Since the hash algorithm can be found from\nthe stack that is in the backend, the new function does not take a\nref-store.  FWIW, the old function did not have to take one either,\nbut since we are getting rid of the old function altogether, that is\nfine ;-)\n\n> This allows us\n> to implement an additional caching layer when reading refs where we can\n> reuse reftable iterators.\n\nOK.\n\n>\n"},{"id":"508137","messageId":"Z0VtfJpOsMCEBuA_@pks.im","threadId":"62444","inReplyTo":"xmqq34je4y6m.fsf@gitster.g","subject":"Re: [PATCH v3 4/9] refs/reftable: read references via `struct reftable_backend`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-26T06:41:00Z","receivedAt":"2024-11-26T06:41:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Nov 26, 2024 at 09:48:49AM +0900, Junio C Hamano wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n> \n> > Refactor `read_ref_without_reload()` to accept a `struct reftable_stack`\n> > as input instead of accepting a `struct reftable_stack`.\n> \n> Hmph...  am I supposed to be puzzled that the patch replaces A with\n> A?\n\nOops, no. The first one should of course be `struct reftable_backend`.\n\n> All callers of read_ref_without_reload() now call\n> reftable_backend_read_ref().  \n> \n> The former took <reftable_ref_store, reftable_stack> while the\n> latter take <reftable_backend>.  They both take <refname, oid,\n> referent, type>, of course, because the former is replaced by the\n> latter.\n> \n> OK, so we introduce a new function, and instead of passing ref-store\n> and stack, the new function only takes reftable-backend (which has a\n> stack contained in it).\n\nYes.\n\n> The old function used ref-store only to find out the hash algorithm\n> via its base repository.  Since the hash algorithm can be found from\n> the stack that is in the backend, the new function does not take a\n> ref-store.  FWIW, the old function did not have to take one either,\n> but since we are getting rid of the old function altogether, that is\n> fine ;-)\n\nYou know, let me maybe split out this change into a separate commit.\nWith your comments it's rather obvious that this commit does too many\nthings at once.\n\nPatrick\n"},{"id":"508138","messageId":"20241126-pks-reftable-backend-reuse-iter-v4-0-b17fd27df126@pks.im","threadId":"62444","inReplyTo":"cover.1730732881.git.ps@pks.im","subject":"[PATCH v4 00/10] refs/reftable: reuse iterators when reading refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-26T06:42:51Z","receivedAt":"2024-11-26T06:43:09Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series refactors the reftable backend to reuse reftable\niterators when reading random references. This removes the overhead of\nhaving to recreate the iterator on every read and thus leads to better\nperformance and less allocation churn. It also gives us the ability to\nfurther optimize reads by optimizing re-seeking iterators in the future.\n\nOverall this leads to a 7% speedup when creating many refs in a\ntransaction, which performs many random reads. But this change also\npositively impacts other usecases.\n\nChanges in v3:\n\n  - Adapt some comments to refer to the \"backend\" instead of to the\n    \"stack\".\n  - Fix indentation of a statement while at it.\n  - Explain why callsites don't want to reload the stack.\n  - Optimize `prepare_transaction_update()` by not using `backend_for()`\n    twice, but instead reload the stack manually.\n  - Split out the change that adds `reftable_stack_hash_id()` into a\n    separate commit.\n  - Link to v2: https://lore.kernel.org/r/cover.1730792627.git.ps@pks.im\n\nChanges in v4:\n\n  - Split up the introduction of `reftable_backend_read_ref()` into two\n    commits: one that gets rid of the `struct reftable_ref_store`\n    parameter and one that converts the function to accept a `struct\n    reftable_backend`.\n  - Fix a comment typo.\n  - Link to v3: https://lore.kernel.org/r/20241125-pks-reftable-backend-reuse-iter-v3-0-1d7b658e3e9e@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (10):\n      refs/reftable: encapsulate reftable stack\n      refs/reftable: handle reloading stacks in the reftable backend\n      reftable/stack: add accessor for the hash ID\n      refs/reftable: figure out hash via `reftable_stack`\n      refs/reftable: read references via `struct reftable_backend`\n      refs/reftable: refactor reading symbolic refs to use reftable backend\n      refs/reftable: refactor reflog expiry to use reftable backend\n      reftable/stack: add mechanism to notify callers on reload\n      reftable/merged: drain priority queue on reseek\n      refs/reftable: reuse iterators when reading refs\n\n refs/reftable-backend.c          | 409 +++++++++++++++++++++++++--------------\n reftable/merged.c                |   2 +\n reftable/reftable-stack.h        |   3 +\n reftable/reftable-writer.h       |   9 +\n reftable/stack.c                 |   9 +\n t/unit-tests/t-reftable-merged.c |  73 +++++++\n 6 files changed, 357 insertions(+), 148 deletions(-)\n\nRange-diff versus v3:\n\n 1:  3db0ba3eb5 =  1:  ec0b7e35c8 refs/reftable: encapsulate reftable stack\n 2:  556eb8301c !  2:  437304908b refs/reftable: handle reloading stacks in the reftable backend\n    @@ refs/reftable-backend.c: static int reftable_be_transaction_prepare(struct ref_s\n     +\t * have reloaded it, which may mean that it is stale.\n     +\t *\n     +\t * On the other hand, reloading that stack without locking it feels\n    -+\t * wrong to, as the value of \"HEAD\" could be modified concurrently at\n    ++\t * wrong, too, as the value of \"HEAD\" could be modified concurrently at\n     +\t * any point in time.\n     +\t */\n     +\tret = backend_for(&be, refs, \"HEAD\", NULL, 0);\n 3:  e966b9acf2 =  3:  0bd9aea884 reftable/stack: add accessor for the hash ID\n -:  ---------- >  4:  d8b801503f refs/reftable: figure out hash via `reftable_stack`\n 4:  de9f52a5b2 !  5:  a57210ac46 refs/reftable: read references via `struct reftable_backend`\n    @@ Metadata\n      ## Commit message ##\n         refs/reftable: read references via `struct reftable_backend`\n     \n    -    Refactor `read_ref_without_reload()` to accept a `struct reftable_stack`\n    -    as input instead of accepting a `struct reftable_stack`. This allows us\n    -    to implement an additional caching layer when reading refs where we can\n    -    reuse reftable iterators.\n    +    Refactor `read_ref_without_reload()` to accept `struct reftable_backend`\n    +    as parameter instead of `struct reftable_stack`. Rename the function to\n    +    `reftable_backend_read_ref()` to clarify its scope and move it close to\n    +    other functions operating on `struct reftable_backend`.\n    +\n    +    This change allows us to implement an additional caching layer when\n    +    reading refs where we can reuse reftable iterators.\n     \n         Signed-off-by: Patrick Steinhardt <ps@pks.im>\n     \n    @@ refs/reftable-backend.c: static void fill_reftable_log_record(struct reftable_lo\n      \tlog->value.update.tz_offset = sign * atoi(tz_begin);\n      }\n      \n    --static int read_ref_without_reload(struct reftable_ref_store *refs,\n    --\t\t\t\t   struct reftable_stack *stack,\n    +-static int read_ref_without_reload(struct reftable_stack *stack,\n     -\t\t\t\t   const char *refname,\n     -\t\t\t\t   struct object_id *oid,\n     -\t\t\t\t   struct strbuf *referent,\n    @@ refs/reftable-backend.c: static void fill_reftable_log_record(struct reftable_lo\n     -\t\tstrbuf_addstr(referent, ref.value.symref);\n     -\t\t*type |= REF_ISSYMREF;\n     -\t} else if (reftable_ref_record_val1(&ref)) {\n    +-\t\tunsigned int hash_id;\n    +-\n    +-\t\tswitch (reftable_stack_hash_id(stack)) {\n    +-\t\tcase REFTABLE_HASH_SHA1:\n    +-\t\t\thash_id = GIT_HASH_SHA1;\n    +-\t\t\tbreak;\n    +-\t\tcase REFTABLE_HASH_SHA256:\n    +-\t\t\thash_id = GIT_HASH_SHA256;\n    +-\t\t\tbreak;\n    +-\t\tdefault:\n    +-\t\t\tBUG(\"unhandled hash ID %d\", reftable_stack_hash_id(stack));\n    +-\t\t}\n    +-\n     -\t\toidread(oid, reftable_ref_record_val1(&ref),\n    --\t\t\trefs->base.repo->hash_algo);\n    +-\t\t\t&hash_algos[hash_id]);\n     -\t} else {\n     -\t\t/* We got a tombstone, which should not happen. */\n     -\t\tBUG(\"unhandled reference value type %d\", ref.value_type);\n    @@ refs/reftable-backend.c: static int reftable_be_read_raw_ref(struct ref_store *r\n      \tif (ret)\n      \t\treturn ret;\n      \n    --\tret = read_ref_without_reload(refs, be->stack, refname, oid, referent, type);\n    +-\tret = read_ref_without_reload(be->stack, refname, oid, referent, type);\n     +\tret = reftable_backend_read_ref(be, refname, oid, referent, type);\n      \tif (ret < 0)\n      \t\treturn ret;\n    @@ refs/reftable-backend.c: static int reftable_be_transaction_prepare(struct ref_s\n      \tif (ret)\n      \t\tgoto done;\n      \n    --\tret = read_ref_without_reload(refs, be->stack, \"HEAD\",\n    +-\tret = read_ref_without_reload(be->stack, \"HEAD\",\n     -\t\t\t\t      &head_oid, &head_referent, &head_type);\n     +\tret = reftable_backend_read_ref(be, \"HEAD\", &head_oid,\n     +\t\t\t\t\t&head_referent, &head_type);\n    @@ refs/reftable-backend.c: static int reftable_be_transaction_prepare(struct ref_s\n      \t\t\tstring_list_insert(&affected_refnames, new_update->refname);\n      \t\t}\n      \n    --\t\tret = read_ref_without_reload(refs, be->stack, rewritten_ref,\n    +-\t\tret = read_ref_without_reload(be->stack, rewritten_ref,\n     -\t\t\t\t\t      &current_oid, &referent, &u->type);\n     +\t\tret = reftable_backend_read_ref(be, rewritten_ref,\n     +\t\t\t\t\t\t&current_oid, &referent, &u->type);\n    @@ refs/reftable-backend.c: static int write_copy_table(struct reftable_writer *wri\n      \t\tmemcpy(logs[logs_nr].value.update.old_hash, old_ref.value.val1, GIT_MAX_RAWSZ);\n      \t\tlogs_nr++;\n      \n    --\t\tret = read_ref_without_reload(arg->refs, arg->stack, \"HEAD\", &head_oid,\n    +-\t\tret = read_ref_without_reload(arg->stack, \"HEAD\", &head_oid,\n     -\t\t\t\t\t      &head_referent, &head_type);\n     +\t\tret = reftable_backend_read_ref(arg->be, \"HEAD\", &head_oid,\n     +\t\t\t\t\t\t&head_referent, &head_type);\n 5:  6e7db8e3fc =  6:  432f75ef01 refs/reftable: refactor reading symbolic refs to use reftable backend\n 6:  687afb781a =  7:  1ca28e00b1 refs/reftable: refactor reflog expiry to use reftable backend\n 7:  e1d4bdc3e8 =  8:  a95b67f1f3 reftable/stack: add mechanism to notify callers on reload\n 8:  8eccf308bd =  9:  cbaf95ff15 reftable/merged: drain priority queue on reseek\n 9:  942957447a = 10:  013f05e08b refs/reftable: reuse iterators when reading refs\n\n---\nbase-commit: 455ddbf8c6a694968c1089fb6c7ffb1d31d97e9d\nchange-id: 20241125-pks-reftable-backend-reuse-iter-3a2e92428789\n\n"},{"id":"508140","messageId":"20241126-pks-reftable-backend-reuse-iter-v4-1-b17fd27df126@pks.im","threadId":"62444","inReplyTo":"20241126-pks-reftable-backend-reuse-iter-v4-0-b17fd27df126@pks.im","subject":"[PATCH v4 01/10] refs/reftable: encapsulate reftable stack","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-26T06:42:52Z","receivedAt":"2024-11-26T06:43:11Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The reftable ref store needs to keep track of multiple stacks, one for\nthe main worktree and an arbitrary number of stacks for worktrees. This\nis done by storing pointers to `struct reftable_stack`, which we then\naccess directly.\n\nWrap the stack in a new `struct reftable_backend`. This will allow us to\nattach more data to each respective stack in subsequent commits.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 135 +++++++++++++++++++++++++++---------------------\n 1 file changed, 76 insertions(+), 59 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex f560bc2b67857d785294e6b5699383a256f30813..acd26f8928d18396f78a2d39ad0e0c1796d5a409 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -34,24 +34,41 @@\n  */\n #define REF_UPDATE_VIA_HEAD (1 << 8)\n \n+struct reftable_backend {\n+\tstruct reftable_stack *stack;\n+};\n+\n+static int reftable_backend_init(struct reftable_backend *be,\n+\t\t\t\t const char *path,\n+\t\t\t\t const struct reftable_write_options *opts)\n+{\n+\treturn reftable_new_stack(&be->stack, path, opts);\n+}\n+\n+static void reftable_backend_release(struct reftable_backend *be)\n+{\n+\treftable_stack_destroy(be->stack);\n+\tbe->stack = NULL;\n+}\n+\n struct reftable_ref_store {\n \tstruct ref_store base;\n \n \t/*\n-\t * The main stack refers to the common dir and thus contains common\n+\t * The main backend refers to the common dir and thus contains common\n \t * refs as well as refs of the main repository.\n \t */\n-\tstruct reftable_stack *main_stack;\n+\tstruct reftable_backend main_backend;\n \t/*\n-\t * The worktree stack refers to the gitdir in case the refdb is opened\n+\t * The worktree backend refers to the gitdir in case the refdb is opened\n \t * via a worktree. It thus contains the per-worktree refs.\n \t */\n-\tstruct reftable_stack *worktree_stack;\n+\tstruct reftable_backend worktree_backend;\n \t/*\n-\t * Map of worktree stacks by their respective worktree names. The map\n+\t * Map of worktree backends by their respective worktree names. The map\n \t * is populated lazily when we try to resolve `worktrees/$worktree` refs.\n \t */\n-\tstruct strmap worktree_stacks;\n+\tstruct strmap worktree_backends;\n \tstruct reftable_write_options write_options;\n \n \tunsigned int store_flags;\n@@ -97,21 +114,21 @@ static struct reftable_ref_store *reftable_be_downcast(struct ref_store *ref_sto\n  * like `worktrees/$worktree/refs/heads/foo` as worktree stacks will store\n  * those references in their normalized form.\n  */\n-static struct reftable_stack *stack_for(struct reftable_ref_store *store,\n-\t\t\t\t\tconst char *refname,\n-\t\t\t\t\tconst char **rewritten_ref)\n+static struct reftable_backend *backend_for(struct reftable_ref_store *store,\n+\t\t\t\t\t    const char *refname,\n+\t\t\t\t\t    const char **rewritten_ref)\n {\n \tconst char *wtname;\n \tint wtname_len;\n \n \tif (!refname)\n-\t\treturn store->main_stack;\n+\t\treturn &store->main_backend;\n \n \tswitch (parse_worktree_ref(refname, &wtname, &wtname_len, rewritten_ref)) {\n \tcase REF_WORKTREE_OTHER: {\n \t\tstatic struct strbuf wtname_buf = STRBUF_INIT;\n \t\tstruct strbuf wt_dir = STRBUF_INIT;\n-\t\tstruct reftable_stack *stack;\n+\t\tstruct reftable_backend *be;\n \n \t\t/*\n \t\t * We're using a static buffer here so that we don't need to\n@@ -125,37 +142,39 @@ static struct reftable_stack *stack_for(struct reftable_ref_store *store,\n \t\t/*\n \t\t * There is an edge case here: when the worktree references the\n \t\t * current worktree, then we set up the stack once via\n-\t\t * `worktree_stacks` and once via `worktree_stack`. This is\n+\t\t * `worktree_backends` and once via `worktree_backend`. This is\n \t\t * wasteful, but in the reading case it shouldn't matter. And\n \t\t * in the writing case we would notice that the stack is locked\n \t\t * already and error out when trying to write a reference via\n \t\t * both stacks.\n \t\t */\n-\t\tstack = strmap_get(&store->worktree_stacks, wtname_buf.buf);\n-\t\tif (!stack) {\n+\t\tbe = strmap_get(&store->worktree_backends, wtname_buf.buf);\n+\t\tif (!be) {\n \t\t\tstrbuf_addf(&wt_dir, \"%s/worktrees/%s/reftable\",\n \t\t\t\t    store->base.repo->commondir, wtname_buf.buf);\n \n-\t\t\tstore->err = reftable_new_stack(&stack, wt_dir.buf,\n-\t\t\t\t\t\t\t&store->write_options);\n+\t\t\tCALLOC_ARRAY(be, 1);\n+\t\t\tstore->err = reftable_backend_init(be, wt_dir.buf,\n+\t\t\t\t\t\t\t   &store->write_options);\n \t\t\tassert(store->err != REFTABLE_API_ERROR);\n-\t\t\tstrmap_put(&store->worktree_stacks, wtname_buf.buf, stack);\n+\n+\t\t\tstrmap_put(&store->worktree_backends, wtname_buf.buf, be);\n \t\t}\n \n \t\tstrbuf_release(&wt_dir);\n-\t\treturn stack;\n+\t\treturn be;\n \t}\n \tcase REF_WORKTREE_CURRENT:\n \t\t/*\n \t\t * If there is no worktree stack then we're currently in the\n \t\t * main worktree. We thus return the main stack in that case.\n \t\t */\n-\t\tif (!store->worktree_stack)\n-\t\t\treturn store->main_stack;\n-\t\treturn store->worktree_stack;\n+\t\tif (!store->worktree_backend.stack)\n+\t\t\treturn &store->main_backend;\n+\t\treturn &store->worktree_backend;\n \tcase REF_WORKTREE_MAIN:\n \tcase REF_WORKTREE_SHARED:\n-\t\treturn store->main_stack;\n+\t\treturn &store->main_backend;\n \tdefault:\n \t\tBUG(\"unhandled worktree reference type\");\n \t}\n@@ -292,7 +311,7 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \tumask(mask);\n \n \tbase_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable);\n-\tstrmap_init(&refs->worktree_stacks);\n+\tstrmap_init(&refs->worktree_backends);\n \trefs->store_flags = store_flags;\n \trefs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);\n \n@@ -337,8 +356,8 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \t\tstrbuf_realpath(&path, gitdir, 0);\n \t}\n \tstrbuf_addstr(&path, \"/reftable\");\n-\trefs->err = reftable_new_stack(&refs->main_stack, path.buf,\n-\t\t\t\t       &refs->write_options);\n+\trefs->err = reftable_backend_init(&refs->main_backend, path.buf,\n+\t\t\t\t\t  &refs->write_options);\n \tif (refs->err)\n \t\tgoto done;\n \n@@ -354,8 +373,8 @@ static struct ref_store *reftable_be_init(struct repository *repo,\n \t\tstrbuf_reset(&path);\n \t\tstrbuf_addf(&path, \"%s/reftable\", gitdir);\n \n-\t\trefs->err = reftable_new_stack(&refs->worktree_stack, path.buf,\n-\t\t\t\t\t       &refs->write_options);\n+\t\trefs->err = reftable_backend_init(&refs->worktree_backend, path.buf,\n+\t\t\t\t\t\t  &refs->write_options);\n \t\tif (refs->err)\n \t\t\tgoto done;\n \t}\n@@ -374,19 +393,17 @@ static void reftable_be_release(struct ref_store *ref_store)\n \tstruct strmap_entry *entry;\n \tstruct hashmap_iter iter;\n \n-\tif (refs->main_stack) {\n-\t\treftable_stack_destroy(refs->main_stack);\n-\t\trefs->main_stack = NULL;\n-\t}\n+\tif (refs->main_backend.stack)\n+\t\treftable_backend_release(&refs->main_backend);\n+\tif (refs->worktree_backend.stack)\n+\t\treftable_backend_release(&refs->worktree_backend);\n \n-\tif (refs->worktree_stack) {\n-\t\treftable_stack_destroy(refs->worktree_stack);\n-\t\trefs->worktree_stack = NULL;\n+\tstrmap_for_each_entry(&refs->worktree_backends, &iter, entry) {\n+\t\tstruct reftable_backend *be = entry->value;\n+\t\treftable_backend_release(be);\n+\t\tfree(be);\n \t}\n-\n-\tstrmap_for_each_entry(&refs->worktree_stacks, &iter, entry)\n-\t\treftable_stack_destroy(entry->value);\n-\tstrmap_clear(&refs->worktree_stacks, 0);\n+\tstrmap_clear(&refs->worktree_backends, 0);\n }\n \n static int reftable_be_create_on_disk(struct ref_store *ref_store,\n@@ -781,7 +798,7 @@ static struct ref_iterator *reftable_be_iterator_begin(struct ref_store *ref_sto\n \t\trequired_flags |= REF_STORE_ODB;\n \trefs = reftable_be_downcast(ref_store, required_flags, \"ref_iterator_begin\");\n \n-\tmain_iter = ref_iterator_for_stack(refs, refs->main_stack, prefix,\n+\tmain_iter = ref_iterator_for_stack(refs, refs->main_backend.stack, prefix,\n \t\t\t\t\t   exclude_patterns, flags);\n \n \t/*\n@@ -789,14 +806,14 @@ static struct ref_iterator *reftable_be_iterator_begin(struct ref_store *ref_sto\n \t * right now. If we aren't, then we return the common reftable\n \t * iterator, only.\n \t */\n-\t if (!refs->worktree_stack)\n+\tif (!refs->worktree_backend.stack)\n \t\treturn &main_iter->base;\n \n \t/*\n \t * Otherwise we merge both the common and the per-worktree refs into a\n \t * single iterator.\n \t */\n-\tworktree_iter = ref_iterator_for_stack(refs, refs->worktree_stack, prefix,\n+\tworktree_iter = ref_iterator_for_stack(refs, refs->worktree_backend.stack, prefix,\n \t\t\t\t\t       exclude_patterns, flags);\n \treturn merge_ref_iterator_begin(&worktree_iter->base, &main_iter->base,\n \t\t\t\t\tref_iterator_select, NULL);\n@@ -811,7 +828,7 @@ static int reftable_be_read_raw_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_raw_ref\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tint ret;\n \n \tif (refs->err < 0)\n@@ -838,7 +855,7 @@ static int reftable_be_read_symbolic_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_symbolic_ref\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_ref_record ref = {0};\n \tint ret;\n \n@@ -898,7 +915,7 @@ static int prepare_transaction_update(struct write_transaction_table_arg **out,\n \t\t\t\t      struct ref_update *update,\n \t\t\t\t      struct strbuf *err)\n {\n-\tstruct reftable_stack *stack = stack_for(refs, update->refname, NULL);\n+\tstruct reftable_stack *stack = backend_for(refs, update->refname, NULL)->stack;\n \tstruct write_transaction_table_arg *arg = NULL;\n \tsize_t i;\n \tint ret;\n@@ -1031,7 +1048,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\tgoto done;\n \t}\n \n-\tret = read_ref_without_reload(refs, stack_for(refs, \"HEAD\", NULL), \"HEAD\",\n+\tret = read_ref_without_reload(refs, backend_for(refs, \"HEAD\", NULL)->stack, \"HEAD\",\n \t\t\t\t      &head_oid, &head_referent, &head_type);\n \tif (ret < 0)\n \t\tgoto done;\n@@ -1043,7 +1060,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\tstruct reftable_stack *stack;\n \t\tconst char *rewritten_ref;\n \n-\t\tstack = stack_for(refs, u->refname, &rewritten_ref);\n+\t\tstack = backend_for(refs, u->refname, &rewritten_ref)->stack;\n \n \t\t/* Verify that the new object ID is valid. */\n \t\tif ((u->flags & REF_HAVE_NEW) && !is_null_oid(&u->new_oid) &&\n@@ -1525,9 +1542,9 @@ static int reftable_be_pack_refs(struct ref_store *ref_store,\n \tif (refs->err)\n \t\treturn refs->err;\n \n-\tstack = refs->worktree_stack;\n+\tstack = refs->worktree_backend.stack;\n \tif (!stack)\n-\t\tstack = refs->main_stack;\n+\t\tstack = refs->main_backend.stack;\n \n \tif (opts->flags & PACK_REFS_AUTO)\n \t\tret = reftable_stack_auto_compact(stack);\n@@ -1782,7 +1799,7 @@ static int reftable_be_rename_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"rename_ref\");\n-\tstruct reftable_stack *stack = stack_for(refs, newrefname, &newrefname);\n+\tstruct reftable_stack *stack = backend_for(refs, newrefname, &newrefname)->stack;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n \t\t.stack = stack,\n@@ -1814,7 +1831,7 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"copy_ref\");\n-\tstruct reftable_stack *stack = stack_for(refs, newrefname, &newrefname);\n+\tstruct reftable_stack *stack = backend_for(refs, newrefname, &newrefname)->stack;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n \t\t.stack = stack,\n@@ -1952,11 +1969,11 @@ static struct ref_iterator *reftable_be_reflog_iterator_begin(struct ref_store *\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_iterator_begin\");\n \tstruct reftable_reflog_iterator *main_iter, *worktree_iter;\n \n-\tmain_iter = reflog_iterator_for_stack(refs, refs->main_stack);\n-\tif (!refs->worktree_stack)\n+\tmain_iter = reflog_iterator_for_stack(refs, refs->main_backend.stack);\n+\tif (!refs->worktree_backend.stack)\n \t\treturn &main_iter->base;\n \n-\tworktree_iter = reflog_iterator_for_stack(refs, refs->worktree_stack);\n+\tworktree_iter = reflog_iterator_for_stack(refs, refs->worktree_backend.stack);\n \n \treturn merge_ref_iterator_begin(&worktree_iter->base, &main_iter->base,\n \t\t\t\t\tref_iterator_select, NULL);\n@@ -1995,7 +2012,7 @@ static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent_reverse\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record log = {0};\n \tstruct reftable_iterator it = {0};\n \tint ret;\n@@ -2035,7 +2052,7 @@ static int reftable_be_for_each_reflog_ent(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record *logs = NULL;\n \tstruct reftable_iterator it = {0};\n \tsize_t logs_alloc = 0, logs_nr = 0, i;\n@@ -2084,7 +2101,7 @@ static int reftable_be_reflog_exists(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_exists\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record log = {0};\n \tstruct reftable_iterator it = {0};\n \tint ret;\n@@ -2169,7 +2186,7 @@ static int reftable_be_create_reflog(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"create_reflog\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct write_reflog_existence_arg arg = {\n \t\t.refs = refs,\n \t\t.stack = stack,\n@@ -2243,7 +2260,7 @@ static int reftable_be_delete_reflog(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"delete_reflog\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct write_reflog_delete_arg arg = {\n \t\t.stack = stack,\n \t\t.refname = refname,\n@@ -2352,7 +2369,7 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \t */\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"reflog_expire\");\n-\tstruct reftable_stack *stack = stack_for(refs, refname, &refname);\n+\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record *logs = NULL;\n \tstruct reftable_log_record *rewritten = NULL;\n \tstruct reftable_ref_record ref_record = {0};\n\n-- \n2.47.0.366.gd4f858ca17.dirty\n\n"},{"id":"508139","messageId":"20241126-pks-reftable-backend-reuse-iter-v4-3-b17fd27df126@pks.im","threadId":"62444","inReplyTo":"20241126-pks-reftable-backend-reuse-iter-v4-0-b17fd27df126@pks.im","subject":"[PATCH v4 03/10] reftable/stack: add accessor for the hash ID","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-26T06:42:54Z","receivedAt":"2024-11-26T06:43:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Add an accessor function that allows callers to access the hash ID of a\nreftable stack. This function will be used in a subsequent commit.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n reftable/reftable-stack.h | 3 +++\n reftable/stack.c          | 5 +++++\n 2 files changed, 8 insertions(+)\n\ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nindex 54787f2ef530406a7970db058c3a0cf456897978..ae14270ea74108cd4c314ec38e7d5c9a4e731481 100644\n--- a/reftable/reftable-stack.h\n+++ b/reftable/reftable-stack.h\n@@ -149,4 +149,7 @@ struct reftable_compaction_stats {\n struct reftable_compaction_stats *\n reftable_stack_compaction_stats(struct reftable_stack *st);\n \n+/* Return the hash of the stack. */\n+enum reftable_hash reftable_stack_hash_id(struct reftable_stack *st);\n+\n #endif\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 1fffd75630266c0d3d1e4a2d037b5cf90455529d..d97b64a40d4ad05cfd9e6f33e8ba1e713281ef6d 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -1791,3 +1791,8 @@ int reftable_stack_clean(struct reftable_stack *st)\n \treftable_addition_destroy(add);\n \treturn err;\n }\n+\n+enum reftable_hash reftable_stack_hash_id(struct reftable_stack *st)\n+{\n+\treturn reftable_merged_table_hash_id(st->merged);\n+}\n\n-- \n2.47.0.366.gd4f858ca17.dirty\n\n"},{"id":"508141","messageId":"20241126-pks-reftable-backend-reuse-iter-v4-2-b17fd27df126@pks.im","threadId":"62444","inReplyTo":"20241126-pks-reftable-backend-reuse-iter-v4-0-b17fd27df126@pks.im","subject":"[PATCH v4 02/10] refs/reftable: handle reloading stacks in the reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-26T06:42:53Z","receivedAt":"2024-11-26T06:43:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When accessing a stack we almost always have to reload the stack before\nreading data from it. This is mostly because Git does not have a\nnotification mechanism for when underlying data has been changed, and\nthus we are forced to opportunistically reload the stack every single\ntime to account for any changes that may have happened concurrently.\n\nHandle the reload internally in `backend_for()`. For one this forces\ncallsites to think about whether or not they need to reload the stack.\nBut second this makes the logic to access stacks more self-contained by\nletting the `struct reftable_backend` manage themselves.\n\nUpdate callsites where we don't reload the stack to document why we\ndon't. In some cases it's unclear whether it is the right thing to do in\nthe first place, but fixing that is outside of the scope of this patch\nseries.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 184 +++++++++++++++++++++++++++++++++---------------\n 1 file changed, 126 insertions(+), 58 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex acd26f8928d18396f78a2d39ad0e0c1796d5a409..8be9cc43c990212b0b4f8d0b473f84270ff4787d 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -114,21 +114,25 @@ static struct reftable_ref_store *reftable_be_downcast(struct ref_store *ref_sto\n  * like `worktrees/$worktree/refs/heads/foo` as worktree stacks will store\n  * those references in their normalized form.\n  */\n-static struct reftable_backend *backend_for(struct reftable_ref_store *store,\n-\t\t\t\t\t    const char *refname,\n-\t\t\t\t\t    const char **rewritten_ref)\n+static int backend_for(struct reftable_backend **out,\n+\t\t       struct reftable_ref_store *store,\n+\t\t       const char *refname,\n+\t\t       const char **rewritten_ref,\n+\t\t       int reload)\n {\n+\tstruct reftable_backend *be;\n \tconst char *wtname;\n \tint wtname_len;\n \n-\tif (!refname)\n-\t\treturn &store->main_backend;\n+\tif (!refname) {\n+\t\tbe = &store->main_backend;\n+\t\tgoto out;\n+\t}\n \n \tswitch (parse_worktree_ref(refname, &wtname, &wtname_len, rewritten_ref)) {\n \tcase REF_WORKTREE_OTHER: {\n \t\tstatic struct strbuf wtname_buf = STRBUF_INIT;\n \t\tstruct strbuf wt_dir = STRBUF_INIT;\n-\t\tstruct reftable_backend *be;\n \n \t\t/*\n \t\t * We're using a static buffer here so that we don't need to\n@@ -162,7 +166,7 @@ static struct reftable_backend *backend_for(struct reftable_ref_store *store,\n \t\t}\n \n \t\tstrbuf_release(&wt_dir);\n-\t\treturn be;\n+\t\tgoto out;\n \t}\n \tcase REF_WORKTREE_CURRENT:\n \t\t/*\n@@ -170,14 +174,27 @@ static struct reftable_backend *backend_for(struct reftable_ref_store *store,\n \t\t * main worktree. We thus return the main stack in that case.\n \t\t */\n \t\tif (!store->worktree_backend.stack)\n-\t\t\treturn &store->main_backend;\n-\t\treturn &store->worktree_backend;\n+\t\t\tbe = &store->main_backend;\n+\t\telse\n+\t\t\tbe = &store->worktree_backend;\n+\t\tgoto out;\n \tcase REF_WORKTREE_MAIN:\n \tcase REF_WORKTREE_SHARED:\n-\t\treturn &store->main_backend;\n+\t\tbe = &store->main_backend;\n+\t\tgoto out;\n \tdefault:\n \t\tBUG(\"unhandled worktree reference type\");\n \t}\n+\n+out:\n+\tif (reload) {\n+\t\tint ret = reftable_stack_reload(be->stack);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n+\t*out = be;\n+\n+\treturn 0;\n }\n \n static int should_write_log(struct reftable_ref_store *refs, const char *refname)\n@@ -828,17 +845,17 @@ static int reftable_be_read_raw_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_raw_ref\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n+\tstruct reftable_backend *be;\n \tint ret;\n \n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret)\n \t\treturn ret;\n \n-\tret = read_ref_without_reload(refs, stack, refname, oid, referent, type);\n+\tret = read_ref_without_reload(refs, be->stack, refname, oid, referent, type);\n \tif (ret < 0)\n \t\treturn ret;\n \tif (ret > 0) {\n@@ -855,15 +872,15 @@ static int reftable_be_read_symbolic_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_symbolic_ref\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_ref_record ref = {0};\n+\tstruct reftable_backend *be;\n \tint ret;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret)\n \t\treturn ret;\n \n-\tret = reftable_stack_read_ref(stack, refname, &ref);\n+\tret = reftable_stack_read_ref(be->stack, refname, &ref);\n \tif (ret == 0 && ref.value_type == REFTABLE_REF_SYMREF)\n \t\tstrbuf_addstr(referent, ref.value.symref);\n \telse\n@@ -880,7 +897,7 @@ struct reftable_transaction_update {\n \n struct write_transaction_table_arg {\n \tstruct reftable_ref_store *refs;\n-\tstruct reftable_stack *stack;\n+\tstruct reftable_backend *be;\n \tstruct reftable_addition *addition;\n \tstruct reftable_transaction_update *updates;\n \tsize_t updates_nr;\n@@ -915,27 +932,37 @@ static int prepare_transaction_update(struct write_transaction_table_arg **out,\n \t\t\t\t      struct ref_update *update,\n \t\t\t\t      struct strbuf *err)\n {\n-\tstruct reftable_stack *stack = backend_for(refs, update->refname, NULL)->stack;\n \tstruct write_transaction_table_arg *arg = NULL;\n+\tstruct reftable_backend *be;\n \tsize_t i;\n \tint ret;\n \n+\t/*\n+\t * This function gets called in a loop, and we don't want to repeatedly\n+\t * reload the stack for every single ref update. Instead, we manually\n+\t * reload further down in the case where we haven't yet prepared the\n+\t * specific `reftable_backend`.\n+\t */\n+\tret = backend_for(&be, refs, update->refname, NULL, 0);\n+\tif (ret)\n+\t\treturn ret;\n+\n \t/*\n \t * Search for a preexisting stack update. If there is one then we add\n \t * the update to it, otherwise we set up a new stack update.\n \t */\n \tfor (i = 0; !arg && i < tx_data->args_nr; i++)\n-\t\tif (tx_data->args[i].stack == stack)\n+\t\tif (tx_data->args[i].be == be)\n \t\t\targ = &tx_data->args[i];\n \n \tif (!arg) {\n \t\tstruct reftable_addition *addition;\n \n-\t\tret = reftable_stack_reload(stack);\n+\t\tret = reftable_stack_reload(be->stack);\n \t\tif (ret)\n \t\t\treturn ret;\n \n-\t\tret = reftable_stack_new_addition(&addition, stack,\n+\t\tret = reftable_stack_new_addition(&addition, be->stack,\n \t\t\t\t\t\t  REFTABLE_STACK_NEW_ADDITION_RELOAD);\n \t\tif (ret) {\n \t\t\tif (ret == REFTABLE_LOCK_ERROR)\n@@ -947,7 +974,7 @@ static int prepare_transaction_update(struct write_transaction_table_arg **out,\n \t\t\t   tx_data->args_alloc);\n \t\targ = &tx_data->args[tx_data->args_nr++];\n \t\targ->refs = refs;\n-\t\targ->stack = stack;\n+\t\targ->be = be;\n \t\targ->addition = addition;\n \t\targ->updates = NULL;\n \t\targ->updates_nr = 0;\n@@ -1002,6 +1029,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \tstruct strbuf referent = STRBUF_INIT, head_referent = STRBUF_INIT;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n \tstruct reftable_transaction_data *tx_data = NULL;\n+\tstruct reftable_backend *be;\n \tstruct object_id head_oid;\n \tunsigned int head_type = 0;\n \tsize_t i;\n@@ -1048,7 +1076,22 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\tgoto done;\n \t}\n \n-\tret = read_ref_without_reload(refs, backend_for(refs, \"HEAD\", NULL)->stack, \"HEAD\",\n+\t/*\n+\t * TODO: it's dubious whether we should reload the stack that \"HEAD\"\n+\t * belongs to or not. In theory, it may happen that we only modify\n+\t * stacks which are _not_ part of the \"HEAD\" stack. In that case we\n+\t * wouldn't have prepared any transaction for its stack and would not\n+\t * have reloaded it, which may mean that it is stale.\n+\t *\n+\t * On the other hand, reloading that stack without locking it feels\n+\t * wrong, too, as the value of \"HEAD\" could be modified concurrently at\n+\t * any point in time.\n+\t */\n+\tret = backend_for(&be, refs, \"HEAD\", NULL, 0);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = read_ref_without_reload(refs, be->stack, \"HEAD\",\n \t\t\t\t      &head_oid, &head_referent, &head_type);\n \tif (ret < 0)\n \t\tgoto done;\n@@ -1057,10 +1100,18 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \tfor (i = 0; i < transaction->nr; i++) {\n \t\tstruct ref_update *u = transaction->updates[i];\n \t\tstruct object_id current_oid = {0};\n-\t\tstruct reftable_stack *stack;\n \t\tconst char *rewritten_ref;\n \n-\t\tstack = backend_for(refs, u->refname, &rewritten_ref)->stack;\n+\t\t/*\n+\t\t * There is no need to reload the respective backends here as\n+\t\t * we have already reloaded them when preparing the transaction\n+\t\t * update. And given that the stacks have been locked there\n+\t\t * shouldn't have been any concurrent modifications of the\n+\t\t * stack.\n+\t\t */\n+\t\tret = backend_for(&be, refs, u->refname, &rewritten_ref, 0);\n+\t\tif (ret)\n+\t\t\tgoto done;\n \n \t\t/* Verify that the new object ID is valid. */\n \t\tif ((u->flags & REF_HAVE_NEW) && !is_null_oid(&u->new_oid) &&\n@@ -1116,7 +1167,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t\tstring_list_insert(&affected_refnames, new_update->refname);\n \t\t}\n \n-\t\tret = read_ref_without_reload(refs, stack, rewritten_ref,\n+\t\tret = read_ref_without_reload(refs, be->stack, rewritten_ref,\n \t\t\t\t\t      &current_oid, &referent, &u->type);\n \t\tif (ret < 0)\n \t\t\tgoto done;\n@@ -1318,7 +1369,7 @@ static int transaction_update_cmp(const void *a, const void *b)\n static int write_transaction_table(struct reftable_writer *writer, void *cb_data)\n {\n \tstruct write_transaction_table_arg *arg = cb_data;\n-\tuint64_t ts = reftable_stack_next_update_index(arg->stack);\n+\tuint64_t ts = reftable_stack_next_update_index(arg->be->stack);\n \tstruct reftable_log_record *logs = NULL;\n \tstruct ident_split committer_ident = {0};\n \tsize_t logs_nr = 0, logs_alloc = 0, i;\n@@ -1354,7 +1405,7 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data\n \t\t\tstruct reftable_log_record log = {0};\n \t\t\tstruct reftable_iterator it = {0};\n \n-\t\t\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\t\t\tret = reftable_stack_init_log_iterator(arg->be->stack, &it);\n \t\t\tif (ret < 0)\n \t\t\t\tgoto done;\n \n@@ -1799,10 +1850,9 @@ static int reftable_be_rename_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"rename_ref\");\n-\tstruct reftable_stack *stack = backend_for(refs, newrefname, &newrefname)->stack;\n+\tstruct reftable_backend *be;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n-\t\t.stack = stack,\n \t\t.oldname = oldrefname,\n \t\t.newname = newrefname,\n \t\t.logmsg = logmsg,\n@@ -1814,10 +1864,11 @@ static int reftable_be_rename_ref(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, newrefname, &newrefname, 1);\n \tif (ret)\n \t\tgoto done;\n-\tret = reftable_stack_add(stack, &write_copy_table, &arg);\n+\targ.stack = be->stack;\n+\tret = reftable_stack_add(be->stack, &write_copy_table, &arg);\n \n done:\n \tassert(ret != REFTABLE_API_ERROR);\n@@ -1831,10 +1882,9 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"copy_ref\");\n-\tstruct reftable_stack *stack = backend_for(refs, newrefname, &newrefname)->stack;\n+\tstruct reftable_backend *be;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n-\t\t.stack = stack,\n \t\t.oldname = oldrefname,\n \t\t.newname = newrefname,\n \t\t.logmsg = logmsg,\n@@ -1845,10 +1895,11 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, newrefname, &newrefname, 1);\n \tif (ret)\n \t\tgoto done;\n-\tret = reftable_stack_add(stack, &write_copy_table, &arg);\n+\targ.stack = be->stack;\n+\tret = reftable_stack_add(be->stack, &write_copy_table, &arg);\n \n done:\n \tassert(ret != REFTABLE_API_ERROR);\n@@ -2012,15 +2063,23 @@ static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent_reverse\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record log = {0};\n \tstruct reftable_iterator it = {0};\n+\tstruct reftable_backend *be;\n \tint ret;\n \n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\tret = reftable_stack_init_log_iterator(stack, &it);\n+\t/*\n+\t * TODO: we should adapt this callsite to reload the stack. There is no\n+\t * obvious reason why we shouldn't.\n+\t */\n+\tret = backend_for(&be, refs, refname, &refname, 0);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_init_log_iterator(be->stack, &it);\n \tif (ret < 0)\n \t\tgoto done;\n \n@@ -2052,16 +2111,24 @@ static int reftable_be_for_each_reflog_ent(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"for_each_reflog_ent\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record *logs = NULL;\n \tstruct reftable_iterator it = {0};\n+\tstruct reftable_backend *be;\n \tsize_t logs_alloc = 0, logs_nr = 0, i;\n \tint ret;\n \n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\tret = reftable_stack_init_log_iterator(stack, &it);\n+\t/*\n+\t * TODO: we should adapt this callsite to reload the stack. There is no\n+\t * obvious reason why we shouldn't.\n+\t */\n+\tret = backend_for(&be, refs, refname, &refname, 0);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_stack_init_log_iterator(be->stack, &it);\n \tif (ret < 0)\n \t\tgoto done;\n \n@@ -2101,20 +2168,20 @@ static int reftable_be_reflog_exists(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"reflog_exists\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record log = {0};\n \tstruct reftable_iterator it = {0};\n+\tstruct reftable_backend *be;\n \tint ret;\n \n \tret = refs->err;\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(be->stack, &it);\n \tif (ret < 0)\n \t\tgoto done;\n \n@@ -2186,10 +2253,9 @@ static int reftable_be_create_reflog(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"create_reflog\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n+\tstruct reftable_backend *be;\n \tstruct write_reflog_existence_arg arg = {\n \t\t.refs = refs,\n-\t\t.stack = stack,\n \t\t.refname = refname,\n \t};\n \tint ret;\n@@ -2198,11 +2264,12 @@ static int reftable_be_create_reflog(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret)\n \t\tgoto done;\n+\targ.stack = be->stack;\n \n-\tret = reftable_stack_add(stack, &write_reflog_existence_table, &arg);\n+\tret = reftable_stack_add(be->stack, &write_reflog_existence_table, &arg);\n \n done:\n \treturn ret;\n@@ -2260,17 +2327,18 @@ static int reftable_be_delete_reflog(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"delete_reflog\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n+\tstruct reftable_backend *be;\n \tstruct write_reflog_delete_arg arg = {\n-\t\t.stack = stack,\n \t\t.refname = refname,\n \t};\n \tint ret;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret)\n \t\treturn ret;\n-\tret = reftable_stack_add(stack, &write_reflog_delete_table, &arg);\n+\targ.stack = be->stack;\n+\n+\tret = reftable_stack_add(be->stack, &write_reflog_delete_table, &arg);\n \n \tassert(ret != REFTABLE_API_ERROR);\n \treturn ret;\n@@ -2369,13 +2437,13 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \t */\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"reflog_expire\");\n-\tstruct reftable_stack *stack = backend_for(refs, refname, &refname)->stack;\n \tstruct reftable_log_record *logs = NULL;\n \tstruct reftable_log_record *rewritten = NULL;\n \tstruct reftable_ref_record ref_record = {0};\n \tstruct reftable_iterator it = {0};\n \tstruct reftable_addition *add = NULL;\n \tstruct reflog_expiry_arg arg = {0};\n+\tstruct reftable_backend *be;\n \tstruct object_id oid = {0};\n \tuint8_t *last_hash = NULL;\n \tsize_t logs_nr = 0, logs_alloc = 0, i;\n@@ -2384,11 +2452,11 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \tif (refs->err < 0)\n \t\treturn refs->err;\n \n-\tret = reftable_stack_reload(stack);\n+\tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_init_log_iterator(stack, &it);\n+\tret = reftable_stack_init_log_iterator(be->stack, &it);\n \tif (ret < 0)\n \t\tgoto done;\n \n@@ -2396,11 +2464,11 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_new_addition(&add, stack, 0);\n+\tret = reftable_stack_new_addition(&add, be->stack, 0);\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_read_ref(stack, refname, &ref_record);\n+\tret = reftable_stack_read_ref(be->stack, refname, &ref_record);\n \tif (ret < 0)\n \t\tgoto done;\n \tif (reftable_ref_record_val1(&ref_record))\n@@ -2479,8 +2547,8 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \targ.refs = refs;\n \targ.records = rewritten;\n \targ.len = logs_nr;\n-\targ.stack = stack,\n-\targ.refname = refname,\n+\targ.stack = be->stack;\n+\targ.refname = refname;\n \n \tret = reftable_addition_add(add, &write_reflog_expiry_table, &arg);\n \tif (ret < 0)\n\n-- \n2.47.0.366.gd4f858ca17.dirty\n\n"},{"id":"508142","messageId":"20241126-pks-reftable-backend-reuse-iter-v4-5-b17fd27df126@pks.im","threadId":"62444","inReplyTo":"20241126-pks-reftable-backend-reuse-iter-v4-0-b17fd27df126@pks.im","subject":"[PATCH v4 05/10] refs/reftable: read references via `struct reftable_backend`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-26T06:42:56Z","receivedAt":"2024-11-26T06:43:13Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor `read_ref_without_reload()` to accept `struct reftable_backend`\nas parameter instead of `struct reftable_stack`. Rename the function to\n`reftable_backend_read_ref()` to clarify its scope and move it close to\nother functions operating on `struct reftable_backend`.\n\nThis change allows us to implement an additional caching layer when\nreading refs where we can reuse reftable iterators.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 122 +++++++++++++++++++++++-------------------------\n 1 file changed, 59 insertions(+), 63 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 302de622d44ccbe8eceee0654f11ad7184791eba..88910207b87b1fa5bbbedebc8817f8afab937b77 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -51,6 +51,50 @@ static void reftable_backend_release(struct reftable_backend *be)\n \tbe->stack = NULL;\n }\n \n+static int reftable_backend_read_ref(struct reftable_backend *be,\n+\t\t\t\t     const char *refname,\n+\t\t\t\t     struct object_id *oid,\n+\t\t\t\t     struct strbuf *referent,\n+\t\t\t\t     unsigned int *type)\n+{\n+\tstruct reftable_ref_record ref = {0};\n+\tint ret;\n+\n+\tret = reftable_stack_read_ref(be->stack, refname, &ref);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n+\t\tstrbuf_reset(referent);\n+\t\tstrbuf_addstr(referent, ref.value.symref);\n+\t\t*type |= REF_ISSYMREF;\n+\t} else if (reftable_ref_record_val1(&ref)) {\n+\t\tunsigned int hash_id;\n+\n+\t\tswitch (reftable_stack_hash_id(be->stack)) {\n+\t\tcase REFTABLE_HASH_SHA1:\n+\t\t\thash_id = GIT_HASH_SHA1;\n+\t\t\tbreak;\n+\t\tcase REFTABLE_HASH_SHA256:\n+\t\t\thash_id = GIT_HASH_SHA256;\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tBUG(\"unhandled hash ID %d\", reftable_stack_hash_id(be->stack));\n+\t\t}\n+\n+\t\toidread(oid, reftable_ref_record_val1(&ref),\n+\t\t\t&hash_algos[hash_id]);\n+\t} else {\n+\t\t/* We got a tombstone, which should not happen. */\n+\t\tBUG(\"unhandled reference value type %d\", ref.value_type);\n+\t}\n+\n+done:\n+\tassert(ret != REFTABLE_API_ERROR);\n+\treftable_ref_record_release(&ref);\n+\treturn ret;\n+}\n+\n struct reftable_ref_store {\n \tstruct ref_store base;\n \n@@ -243,50 +287,6 @@ static void fill_reftable_log_record(struct reftable_log_record *log, const stru\n \tlog->value.update.tz_offset = sign * atoi(tz_begin);\n }\n \n-static int read_ref_without_reload(struct reftable_stack *stack,\n-\t\t\t\t   const char *refname,\n-\t\t\t\t   struct object_id *oid,\n-\t\t\t\t   struct strbuf *referent,\n-\t\t\t\t   unsigned int *type)\n-{\n-\tstruct reftable_ref_record ref = {0};\n-\tint ret;\n-\n-\tret = reftable_stack_read_ref(stack, refname, &ref);\n-\tif (ret)\n-\t\tgoto done;\n-\n-\tif (ref.value_type == REFTABLE_REF_SYMREF) {\n-\t\tstrbuf_reset(referent);\n-\t\tstrbuf_addstr(referent, ref.value.symref);\n-\t\t*type |= REF_ISSYMREF;\n-\t} else if (reftable_ref_record_val1(&ref)) {\n-\t\tunsigned int hash_id;\n-\n-\t\tswitch (reftable_stack_hash_id(stack)) {\n-\t\tcase REFTABLE_HASH_SHA1:\n-\t\t\thash_id = GIT_HASH_SHA1;\n-\t\t\tbreak;\n-\t\tcase REFTABLE_HASH_SHA256:\n-\t\t\thash_id = GIT_HASH_SHA256;\n-\t\t\tbreak;\n-\t\tdefault:\n-\t\t\tBUG(\"unhandled hash ID %d\", reftable_stack_hash_id(stack));\n-\t\t}\n-\n-\t\toidread(oid, reftable_ref_record_val1(&ref),\n-\t\t\t&hash_algos[hash_id]);\n-\t} else {\n-\t\t/* We got a tombstone, which should not happen. */\n-\t\tBUG(\"unhandled reference value type %d\", ref.value_type);\n-\t}\n-\n-done:\n-\tassert(ret != REFTABLE_API_ERROR);\n-\treftable_ref_record_release(&ref);\n-\treturn ret;\n-}\n-\n static int reftable_be_config(const char *var, const char *value,\n \t\t\t      const struct config_context *ctx,\n \t\t\t      void *_opts)\n@@ -867,7 +867,7 @@ static int reftable_be_read_raw_ref(struct ref_store *ref_store,\n \tif (ret)\n \t\treturn ret;\n \n-\tret = read_ref_without_reload(be->stack, refname, oid, referent, type);\n+\tret = reftable_backend_read_ref(be, refname, oid, referent, type);\n \tif (ret < 0)\n \t\treturn ret;\n \tif (ret > 0) {\n@@ -1103,8 +1103,8 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \tif (ret)\n \t\tgoto done;\n \n-\tret = read_ref_without_reload(be->stack, \"HEAD\",\n-\t\t\t\t      &head_oid, &head_referent, &head_type);\n+\tret = reftable_backend_read_ref(be, \"HEAD\", &head_oid,\n+\t\t\t\t\t&head_referent, &head_type);\n \tif (ret < 0)\n \t\tgoto done;\n \tret = 0;\n@@ -1179,8 +1179,8 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t\tstring_list_insert(&affected_refnames, new_update->refname);\n \t\t}\n \n-\t\tret = read_ref_without_reload(be->stack, rewritten_ref,\n-\t\t\t\t\t      &current_oid, &referent, &u->type);\n+\t\tret = reftable_backend_read_ref(be, rewritten_ref,\n+\t\t\t\t\t\t&current_oid, &referent, &u->type);\n \t\tif (ret < 0)\n \t\t\tgoto done;\n \t\tif (ret > 0 && !ref_update_expects_existing_old_ref(u)) {\n@@ -1638,7 +1638,7 @@ struct write_create_symref_arg {\n \n struct write_copy_arg {\n \tstruct reftable_ref_store *refs;\n-\tstruct reftable_stack *stack;\n+\tstruct reftable_backend *be;\n \tconst char *oldname;\n \tconst char *newname;\n \tconst char *logmsg;\n@@ -1663,7 +1663,7 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n \tif (split_ident_line(&committer_ident, committer_info, strlen(committer_info)))\n \t\tBUG(\"failed splitting committer info\");\n \n-\tif (reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref)) {\n+\tif (reftable_stack_read_ref(arg->be->stack, arg->oldname, &old_ref)) {\n \t\tret = error(_(\"refname %s not found\"), arg->oldname);\n \t\tgoto done;\n \t}\n@@ -1702,7 +1702,7 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n \t * the old branch and the creation of the new branch, and we cannot do\n \t * two changes to a reflog in a single update.\n \t */\n-\tdeletion_ts = creation_ts = reftable_stack_next_update_index(arg->stack);\n+\tdeletion_ts = creation_ts = reftable_stack_next_update_index(arg->be->stack);\n \tif (arg->delete_old)\n \t\tcreation_ts++;\n \treftable_writer_set_limits(writer, deletion_ts, creation_ts);\n@@ -1745,8 +1745,8 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n \t\tmemcpy(logs[logs_nr].value.update.old_hash, old_ref.value.val1, GIT_MAX_RAWSZ);\n \t\tlogs_nr++;\n \n-\t\tret = read_ref_without_reload(arg->stack, \"HEAD\", &head_oid,\n-\t\t\t\t\t      &head_referent, &head_type);\n+\t\tret = reftable_backend_read_ref(arg->be, \"HEAD\", &head_oid,\n+\t\t\t\t\t\t&head_referent, &head_type);\n \t\tif (ret < 0)\n \t\t\tgoto done;\n \t\tappend_head_reflog = (head_type & REF_ISSYMREF) && !strcmp(head_referent.buf, arg->oldname);\n@@ -1789,7 +1789,7 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n \t * copy over all log entries from the old reflog. Last but not least,\n \t * when renaming we also have to delete all the old reflog entries.\n \t */\n-\tret = reftable_stack_init_log_iterator(arg->stack, &it);\n+\tret = reftable_stack_init_log_iterator(arg->be->stack, &it);\n \tif (ret < 0)\n \t\tgoto done;\n \n@@ -1862,7 +1862,6 @@ static int reftable_be_rename_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"rename_ref\");\n-\tstruct reftable_backend *be;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n \t\t.oldname = oldrefname,\n@@ -1876,11 +1875,10 @@ static int reftable_be_rename_ref(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = backend_for(&be, refs, newrefname, &newrefname, 1);\n+\tret = backend_for(&arg.be, refs, newrefname, &newrefname, 1);\n \tif (ret)\n \t\tgoto done;\n-\targ.stack = be->stack;\n-\tret = reftable_stack_add(be->stack, &write_copy_table, &arg);\n+\tret = reftable_stack_add(arg.be->stack, &write_copy_table, &arg);\n \n done:\n \tassert(ret != REFTABLE_API_ERROR);\n@@ -1894,7 +1892,6 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"copy_ref\");\n-\tstruct reftable_backend *be;\n \tstruct write_copy_arg arg = {\n \t\t.refs = refs,\n \t\t.oldname = oldrefname,\n@@ -1907,11 +1904,10 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = backend_for(&be, refs, newrefname, &newrefname, 1);\n+\tret = backend_for(&arg.be, refs, newrefname, &newrefname, 1);\n \tif (ret)\n \t\tgoto done;\n-\targ.stack = be->stack;\n-\tret = reftable_stack_add(be->stack, &write_copy_table, &arg);\n+\tret = reftable_stack_add(arg.be->stack, &write_copy_table, &arg);\n \n done:\n \tassert(ret != REFTABLE_API_ERROR);\n\n-- \n2.47.0.366.gd4f858ca17.dirty\n\n"},{"id":"508143","messageId":"20241126-pks-reftable-backend-reuse-iter-v4-4-b17fd27df126@pks.im","threadId":"62444","inReplyTo":"20241126-pks-reftable-backend-reuse-iter-v4-0-b17fd27df126@pks.im","subject":"[PATCH v4 04/10] refs/reftable: figure out hash via `reftable_stack`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-26T06:42:55Z","receivedAt":"2024-11-26T06:43:13Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The function `read_ref_without_reload()` accepts a ref store as input\nonly so that we can figure out the hash function used by it. This is\nduplicate information though because the reftable stack knows about its\nhash function, too.\n\nDrop the superfluous parameter to simplify the calling convention a bit.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 26 +++++++++++++++++++-------\n 1 file changed, 19 insertions(+), 7 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 8be9cc43c990212b0b4f8d0b473f84270ff4787d..302de622d44ccbe8eceee0654f11ad7184791eba 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -243,8 +243,7 @@ static void fill_reftable_log_record(struct reftable_log_record *log, const stru\n \tlog->value.update.tz_offset = sign * atoi(tz_begin);\n }\n \n-static int read_ref_without_reload(struct reftable_ref_store *refs,\n-\t\t\t\t   struct reftable_stack *stack,\n+static int read_ref_without_reload(struct reftable_stack *stack,\n \t\t\t\t   const char *refname,\n \t\t\t\t   struct object_id *oid,\n \t\t\t\t   struct strbuf *referent,\n@@ -262,8 +261,21 @@ static int read_ref_without_reload(struct reftable_ref_store *refs,\n \t\tstrbuf_addstr(referent, ref.value.symref);\n \t\t*type |= REF_ISSYMREF;\n \t} else if (reftable_ref_record_val1(&ref)) {\n+\t\tunsigned int hash_id;\n+\n+\t\tswitch (reftable_stack_hash_id(stack)) {\n+\t\tcase REFTABLE_HASH_SHA1:\n+\t\t\thash_id = GIT_HASH_SHA1;\n+\t\t\tbreak;\n+\t\tcase REFTABLE_HASH_SHA256:\n+\t\t\thash_id = GIT_HASH_SHA256;\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tBUG(\"unhandled hash ID %d\", reftable_stack_hash_id(stack));\n+\t\t}\n+\n \t\toidread(oid, reftable_ref_record_val1(&ref),\n-\t\t\trefs->base.repo->hash_algo);\n+\t\t\t&hash_algos[hash_id]);\n \t} else {\n \t\t/* We got a tombstone, which should not happen. */\n \t\tBUG(\"unhandled reference value type %d\", ref.value_type);\n@@ -855,7 +867,7 @@ static int reftable_be_read_raw_ref(struct ref_store *ref_store,\n \tif (ret)\n \t\treturn ret;\n \n-\tret = read_ref_without_reload(refs, be->stack, refname, oid, referent, type);\n+\tret = read_ref_without_reload(be->stack, refname, oid, referent, type);\n \tif (ret < 0)\n \t\treturn ret;\n \tif (ret > 0) {\n@@ -1091,7 +1103,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \tif (ret)\n \t\tgoto done;\n \n-\tret = read_ref_without_reload(refs, be->stack, \"HEAD\",\n+\tret = read_ref_without_reload(be->stack, \"HEAD\",\n \t\t\t\t      &head_oid, &head_referent, &head_type);\n \tif (ret < 0)\n \t\tgoto done;\n@@ -1167,7 +1179,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t\tstring_list_insert(&affected_refnames, new_update->refname);\n \t\t}\n \n-\t\tret = read_ref_without_reload(refs, be->stack, rewritten_ref,\n+\t\tret = read_ref_without_reload(be->stack, rewritten_ref,\n \t\t\t\t\t      &current_oid, &referent, &u->type);\n \t\tif (ret < 0)\n \t\t\tgoto done;\n@@ -1733,7 +1745,7 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data)\n \t\tmemcpy(logs[logs_nr].value.update.old_hash, old_ref.value.val1, GIT_MAX_RAWSZ);\n \t\tlogs_nr++;\n \n-\t\tret = read_ref_without_reload(arg->refs, arg->stack, \"HEAD\", &head_oid,\n+\t\tret = read_ref_without_reload(arg->stack, \"HEAD\", &head_oid,\n \t\t\t\t\t      &head_referent, &head_type);\n \t\tif (ret < 0)\n \t\t\tgoto done;\n\n-- \n2.47.0.366.gd4f858ca17.dirty\n\n"},{"id":"508144","messageId":"20241126-pks-reftable-backend-reuse-iter-v4-6-b17fd27df126@pks.im","threadId":"62444","inReplyTo":"20241126-pks-reftable-backend-reuse-iter-v4-0-b17fd27df126@pks.im","subject":"[PATCH v4 06/10] refs/reftable: refactor reading symbolic refs to use reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-26T06:42:57Z","receivedAt":"2024-11-26T06:43:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor the callback function that reads symbolic references in the\nreftable backend to use `reftable_backend_read_ref()` instead of\naccessing the reftable stack directly. This ensures that the function\nwill benefit from the new caching layer that we're about to introduce.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 11 ++++-------\n 1 file changed, 4 insertions(+), 7 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 88910207b87b1fa5bbbedebc8817f8afab937b77..2d06620ac8b1b7f07783cd5873dbe0fe67b84bd6 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -884,21 +884,18 @@ static int reftable_be_read_symbolic_ref(struct ref_store *ref_store,\n {\n \tstruct reftable_ref_store *refs =\n \t\treftable_be_downcast(ref_store, REF_STORE_READ, \"read_symbolic_ref\");\n-\tstruct reftable_ref_record ref = {0};\n \tstruct reftable_backend *be;\n+\tstruct object_id oid;\n+\tunsigned int type = 0;\n \tint ret;\n \n \tret = backend_for(&be, refs, refname, &refname, 1);\n \tif (ret)\n \t\treturn ret;\n \n-\tret = reftable_stack_read_ref(be->stack, refname, &ref);\n-\tif (ret == 0 && ref.value_type == REFTABLE_REF_SYMREF)\n-\t\tstrbuf_addstr(referent, ref.value.symref);\n-\telse\n+\tret = reftable_backend_read_ref(be, refname, &oid, referent, &type);\n+\tif (type != REF_ISSYMREF)\n \t\tret = -1;\n-\n-\treftable_ref_record_release(&ref);\n \treturn ret;\n }\n \n\n-- \n2.47.0.366.gd4f858ca17.dirty\n\n"},{"id":"508145","messageId":"20241126-pks-reftable-backend-reuse-iter-v4-7-b17fd27df126@pks.im","threadId":"62444","inReplyTo":"20241126-pks-reftable-backend-reuse-iter-v4-0-b17fd27df126@pks.im","subject":"[PATCH v4 07/10] refs/reftable: refactor reflog expiry to use reftable backend","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-26T06:42:58Z","receivedAt":"2024-11-26T06:43:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor the callback function that expires reflog entries in the\nreftable backend to use `reftable_backend_read_ref()` instead of\naccessing the reftable stack directly. This ensures that the function\nwill benefit from the new caching layer that we're about to introduce.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 13 +++++--------\n 1 file changed, 5 insertions(+), 8 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 2d06620ac8b1b7f07783cd5873dbe0fe67b84bd6..b6638d43028d11ae7621dcd4e0dbf1d3174743b7 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -2444,14 +2444,15 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE, \"reflog_expire\");\n \tstruct reftable_log_record *logs = NULL;\n \tstruct reftable_log_record *rewritten = NULL;\n-\tstruct reftable_ref_record ref_record = {0};\n \tstruct reftable_iterator it = {0};\n \tstruct reftable_addition *add = NULL;\n \tstruct reflog_expiry_arg arg = {0};\n \tstruct reftable_backend *be;\n \tstruct object_id oid = {0};\n+\tstruct strbuf referent = STRBUF_INIT;\n \tuint8_t *last_hash = NULL;\n \tsize_t logs_nr = 0, logs_alloc = 0, i;\n+\tunsigned int type = 0;\n \tint ret;\n \n \tif (refs->err < 0)\n@@ -2473,12 +2474,9 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_read_ref(be->stack, refname, &ref_record);\n+\tret = reftable_backend_read_ref(be, refname, &oid, &referent, &type);\n \tif (ret < 0)\n \t\tgoto done;\n-\tif (reftable_ref_record_val1(&ref_record))\n-\t\toidread(&oid, reftable_ref_record_val1(&ref_record),\n-\t\t\tref_store->repo->hash_algo);\n \tprepare_fn(refname, &oid, policy_cb_data);\n \n \twhile (1) {\n@@ -2545,8 +2543,7 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \t\t}\n \t}\n \n-\tif (flags & EXPIRE_REFLOGS_UPDATE_REF && last_hash &&\n-\t    reftable_ref_record_val1(&ref_record))\n+\tif (flags & EXPIRE_REFLOGS_UPDATE_REF && last_hash && !is_null_oid(&oid))\n \t\toidread(&arg.update_oid, last_hash, ref_store->repo->hash_algo);\n \n \targ.refs = refs;\n@@ -2571,11 +2568,11 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \t\tcleanup_fn(policy_cb_data);\n \tassert(ret != REFTABLE_API_ERROR);\n \n-\treftable_ref_record_release(&ref_record);\n \treftable_iterator_destroy(&it);\n \treftable_addition_destroy(add);\n \tfor (i = 0; i < logs_nr; i++)\n \t\treftable_log_record_release(&logs[i]);\n+\tstrbuf_release(&referent);\n \tfree(logs);\n \tfree(rewritten);\n \treturn ret;\n\n-- \n2.47.0.366.gd4f858ca17.dirty\n\n"},{"id":"508146","messageId":"20241126-pks-reftable-backend-reuse-iter-v4-8-b17fd27df126@pks.im","threadId":"62444","inReplyTo":"20241126-pks-reftable-backend-reuse-iter-v4-0-b17fd27df126@pks.im","subject":"[PATCH v4 08/10] reftable/stack: add mechanism to notify callers on reload","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-26T06:42:59Z","receivedAt":"2024-11-26T06:43:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Reftable stacks are reloaded in two cases:\n\n  - When calling `reftable_stack_reload()`, if the stat-cache tells us\n    that the stack has been modified.\n\n  - When committing a reftable addition.\n\nWhile callers can figure out the second case, they do not have a\nmechanism to figure out whether `reftable_stack_reload()` led to an\nactual reload of the on-disk data. All they can do is thus to assume\nthat data is always being reloaded in that case.\n\nImprove the situation by introducing a new `on_reload()` callback to the\nreftable options. If provided, the function will be invoked every time\nthe stack has indeed been reloaded. This allows callers to invalidate\ndata that depends on the current stack data.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n reftable/reftable-writer.h | 9 +++++++++\n reftable/stack.c           | 4 ++++\n 2 files changed, 13 insertions(+)\n\ndiff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h\nindex c85ef5a5bd14595d75f99457fef4407040e197c5..5f9afa620bb00de66c311765fb0ae8c6f56401ae 100644\n--- a/reftable/reftable-writer.h\n+++ b/reftable/reftable-writer.h\n@@ -68,6 +68,15 @@ struct reftable_write_options {\n \t * fsync(3P) when unset.\n \t */\n \tint (*fsync)(int fd);\n+\n+\t/*\n+\t * Callback function to execute whenever the stack is being reloaded.\n+\t * This can be used e.g. to discard cached information that relies on\n+\t * the old stack's data. The payload data will be passed as argument to\n+\t * the callback.\n+\t */\n+\tvoid (*on_reload)(void *payload);\n+\tvoid *on_reload_payload;\n };\n \n /* reftable_block_stats holds statistics for a single block type */\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex d97b64a40d4ad05cfd9e6f33e8ba1e713281ef6d..5384ca9de0d1f064aebcb09308a74cc397b37463 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -548,6 +548,10 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,\n \t\tclose(fd);\n \tfree_names(names);\n \tfree_names(names_after);\n+\n+\tif (st->opts.on_reload)\n+\t\tst->opts.on_reload(st->opts.on_reload_payload);\n+\n \treturn err;\n }\n \n\n-- \n2.47.0.366.gd4f858ca17.dirty\n\n"},{"id":"508147","messageId":"20241126-pks-reftable-backend-reuse-iter-v4-9-b17fd27df126@pks.im","threadId":"62444","inReplyTo":"20241126-pks-reftable-backend-reuse-iter-v4-0-b17fd27df126@pks.im","subject":"[PATCH v4 09/10] reftable/merged: drain priority queue on reseek","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-26T06:43:00Z","receivedAt":"2024-11-26T06:43:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In 5bf96e0c39 (reftable/generic: move seeking of records into the\niterator, 2024-05-13) we have refactored the reftable codebase such that\niterators can be initialized once and then re-seeked multiple times.\nThis feature is used by 1869525066 (refs/reftable: wire up support for\nexclude patterns, 2024-09-16) in order to skip records based on exclude\npatterns provided by the caller.\n\nThe logic to re-seek the merged iterator is insufficient though because\nwe don't drain the priority queue on a re-seek. This means that the\nqueue may contain stale entries and thus reading the next record in the\nqueue will return the wrong entry. While this is an obvious bug, it is\nharmless in the context of above exclude patterns:\n\n  - If the queue contained stale entries that match the pattern then the\n    caller would already know to filter out such refs. This is because\n    our codebase is prepared to handle backends that don't have a way to\n    efficiently implement exclude patterns.\n\n  - If the queue contained stale entries that don't match the pattern\n    we'd eventually filter out any duplicates. This is because the\n    reftable code discards items with the same ref name and sorts any\n    remaining entries properly.\n\nSo things happen to work in this context regardless of the bug, and\nthere is no other use case yet where we re-seek iterators. We're about\nto introduce a caching mechanism though where iterators are reused by\nthe reftable backend, and that will expose the bug.\n\nFix the issue by draining the priority queue when seeking and add a\ntestcase that surfaces the issue.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n reftable/merged.c                |  2 ++\n t/unit-tests/t-reftable-merged.c | 73 ++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 75 insertions(+)\n\ndiff --git a/reftable/merged.c b/reftable/merged.c\nindex 5b93e20f42945300abbc1a036bbdf067fced7854..bb0836e3443271f9c0d5ba5582c78694d437ddc2 100644\n--- a/reftable/merged.c\n+++ b/reftable/merged.c\n@@ -66,6 +66,8 @@ static int merged_iter_seek(struct merged_iter *mi, struct reftable_record *want\n \tint err;\n \n \tmi->advance_index = -1;\n+\twhile (!merged_iter_pqueue_is_empty(mi->pq))\n+\t\tmerged_iter_pqueue_remove(&mi->pq);\n \n \tfor (size_t i = 0; i < mi->subiters_len; i++) {\n \t\terr = iterator_seek(&mi->subiters[i].iter, want);\ndiff --git a/t/unit-tests/t-reftable-merged.c b/t/unit-tests/t-reftable-merged.c\nindex 2591b5e59745536a205271491f747875e04c5a3f..a12bd0e1a3bdeda82bcbc6259e679df4f232e3e2 100644\n--- a/t/unit-tests/t-reftable-merged.c\n+++ b/t/unit-tests/t-reftable-merged.c\n@@ -273,6 +273,78 @@ static void t_merged_seek_multiple_times(void)\n \treftable_free(sources);\n }\n \n+static void t_merged_seek_multiple_times_without_draining(void)\n+{\n+\tstruct reftable_ref_record r1[] = {\n+\t\t{\n+\t\t\t.refname = (char *) \"a\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = { 1 },\n+\t\t},\n+\t\t{\n+\t\t\t.refname = (char *) \"c\",\n+\t\t\t.update_index = 1,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = { 2 },\n+\t\t}\n+\t};\n+\tstruct reftable_ref_record r2[] = {\n+\t\t{\n+\t\t\t.refname = (char *) \"b\",\n+\t\t\t.update_index = 2,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = { 3 },\n+\t\t},\n+\t\t{\n+\t\t\t.refname = (char *) \"d\",\n+\t\t\t.update_index = 2,\n+\t\t\t.value_type = REFTABLE_REF_VAL1,\n+\t\t\t.value.val1 = { 4 },\n+\t\t},\n+\t};\n+\tstruct reftable_ref_record *refs[] = {\n+\t\tr1, r2,\n+\t};\n+\tsize_t sizes[] = {\n+\t\tARRAY_SIZE(r1), ARRAY_SIZE(r2),\n+\t};\n+\tstruct reftable_buf bufs[] = {\n+\t\tREFTABLE_BUF_INIT, REFTABLE_BUF_INIT,\n+\t};\n+\tstruct reftable_block_source *sources = NULL;\n+\tstruct reftable_reader **readers = NULL;\n+\tstruct reftable_ref_record rec = { 0 };\n+\tstruct reftable_iterator it = { 0 };\n+\tstruct reftable_merged_table *mt;\n+\tint err;\n+\n+\tmt = merged_table_from_records(refs, &sources, &readers, sizes, bufs, 2);\n+\tmerged_table_init_iter(mt, &it, BLOCK_TYPE_REF);\n+\n+\terr = reftable_iterator_seek_ref(&it, \"b\");\n+\tcheck(!err);\n+\terr = reftable_iterator_next_ref(&it, &rec);\n+\tcheck(!err);\n+\terr = reftable_ref_record_equal(&rec, &r2[0], REFTABLE_HASH_SIZE_SHA1);\n+\tcheck(err == 1);\n+\n+\terr = reftable_iterator_seek_ref(&it, \"a\");\n+\tcheck(!err);\n+\terr = reftable_iterator_next_ref(&it, &rec);\n+\tcheck(!err);\n+\terr = reftable_ref_record_equal(&rec, &r1[0], REFTABLE_HASH_SIZE_SHA1);\n+\tcheck(err == 1);\n+\n+\tfor (size_t i = 0; i < ARRAY_SIZE(bufs); i++)\n+\t\treftable_buf_release(&bufs[i]);\n+\treaders_destroy(readers, ARRAY_SIZE(refs));\n+\treftable_ref_record_release(&rec);\n+\treftable_iterator_destroy(&it);\n+\treftable_merged_table_free(mt);\n+\treftable_free(sources);\n+}\n+\n static struct reftable_merged_table *\n merged_table_from_log_records(struct reftable_log_record **logs,\n \t\t\t      struct reftable_block_source **source,\n@@ -467,6 +539,7 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED)\n \tTEST(t_merged_logs(), \"merged table with multiple log updates for same ref\");\n \tTEST(t_merged_refs(), \"merged table with multiple updates to same ref\");\n \tTEST(t_merged_seek_multiple_times(), \"merged table can seek multiple times\");\n+\tTEST(t_merged_seek_multiple_times_without_draining(), \"merged table can seek multiple times without draining\");\n \tTEST(t_merged_single_record(), \"ref occurring in only one record can be fetched\");\n \n \treturn test_done();\n\n-- \n2.47.0.366.gd4f858ca17.dirty\n\n"},{"id":"508148","messageId":"20241126-pks-reftable-backend-reuse-iter-v4-10-b17fd27df126@pks.im","threadId":"62444","inReplyTo":"20241126-pks-reftable-backend-reuse-iter-v4-0-b17fd27df126@pks.im","subject":"[PATCH v4 10/10] refs/reftable: reuse iterators when reading refs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-26T06:43:01Z","receivedAt":"2024-11-26T06:43:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When reading references the reftable backend has to:\n\n  1. Create a new ref iterator.\n\n  2. Seek the iterator to the record we're searching for.\n\n  3. Read the record.\n\nWe cannot really avoid the last two steps, but re-creating the iterator\nevery single time we want to read a reference is kind of expensive and a\nwaste of resources. We couldn't help it in the past though because it\nwas not possible to reuse iterators. But starting with 5bf96e0c39\n(reftable/generic: move seeking of records into the iterator,\n2024-05-13) we have split up the iterator lifecycle such that creating\nthe iterator and seeking are two different concerns.\n\nRefactor the code such that we cache iterators in the reftable backend.\nThis cache is invalidated whenever the respective stack is reloaded such\nthat we know to recreate the iterator in that case. This leads to a\nsizeable speedup when creating many refs, which requires a lot of random\nreference reads:\n\n    Benchmark 1: update-ref: create many refs (refcount = 100000, revision = master)\n      Time (mean ± σ):      1.793 s ±  0.010 s    [User: 0.954 s, System: 0.835 s]\n      Range (min … max):    1.781 s …  1.811 s    10 runs\n\n    Benchmark 2: update-ref: create many refs (refcount = 100000, revision = HEAD)\n      Time (mean ± σ):      1.680 s ±  0.013 s    [User: 0.846 s, System: 0.831 s]\n      Range (min … max):    1.664 s …  1.702 s    10 runs\n\n    Summary\n      update-ref: create many refs (refcount = 100000, revision = HEAD) ran\n        1.07 ± 0.01 times faster than update-ref: create many refs (refcount = 100000, revision = master)\n\nWhile 7% is not a huge win, you have to consider that the benchmark is\n_writing_ data, so _reading_ references is only one part of what we do.\nFlame graphs show that we spend around 40% of our time reading refs, so\nthe speedup when reading refs is approximately ~2.5x that. I could not\nfind better benchmarks where we perform a lot of random ref reads.\n\nYou can also see a sizeable impact on memory usage when creating 100k\nreferences. Before this change:\n\n    HEAP SUMMARY:\n        in use at exit: 19,112,538 bytes in 200,170 blocks\n      total heap usage: 8,400,426 allocs, 8,200,256 frees, 454,367,048 bytes allocated\n\nAfter this change:\n\n    HEAP SUMMARY:\n        in use at exit: 674,416 bytes in 169 blocks\n      total heap usage: 7,929,872 allocs, 7,929,703 frees, 281,509,985 bytes allocated\n\nAs an additional factor, this refactoring opens up the possibility for\nmore performance optimizations in how we re-seek iterators. Any change\nthat allows us to optimize re-seeking by e.g. reusing data structures\nwould thus also directly speed up random reads.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n refs/reftable-backend.c | 32 +++++++++++++++++++++++++++++---\n 1 file changed, 29 insertions(+), 3 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex b6638d43028d11ae7621dcd4e0dbf1d3174743b7..a80c334f384acd0d72a1c4ef9e59c93600dd6db4 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -36,19 +36,30 @@\n \n struct reftable_backend {\n \tstruct reftable_stack *stack;\n+\tstruct reftable_iterator it;\n };\n \n+static void reftable_backend_on_reload(void *payload)\n+{\n+\tstruct reftable_backend *be = payload;\n+\treftable_iterator_destroy(&be->it);\n+}\n+\n static int reftable_backend_init(struct reftable_backend *be,\n \t\t\t\t const char *path,\n-\t\t\t\t const struct reftable_write_options *opts)\n+\t\t\t\t const struct reftable_write_options *_opts)\n {\n-\treturn reftable_new_stack(&be->stack, path, opts);\n+\tstruct reftable_write_options opts = *_opts;\n+\topts.on_reload = reftable_backend_on_reload;\n+\topts.on_reload_payload = be;\n+\treturn reftable_new_stack(&be->stack, path, &opts);\n }\n \n static void reftable_backend_release(struct reftable_backend *be)\n {\n \treftable_stack_destroy(be->stack);\n \tbe->stack = NULL;\n+\treftable_iterator_destroy(&be->it);\n }\n \n static int reftable_backend_read_ref(struct reftable_backend *be,\n@@ -60,10 +71,25 @@ static int reftable_backend_read_ref(struct reftable_backend *be,\n \tstruct reftable_ref_record ref = {0};\n \tint ret;\n \n-\tret = reftable_stack_read_ref(be->stack, refname, &ref);\n+\tif (!be->it.ops) {\n+\t\tret = reftable_stack_init_ref_iterator(be->stack, &be->it);\n+\t\tif (ret)\n+\t\t\tgoto done;\n+\t}\n+\n+\tret = reftable_iterator_seek_ref(&be->it, refname);\n \tif (ret)\n \t\tgoto done;\n \n+\tret = reftable_iterator_next_ref(&be->it, &ref);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tif (strcmp(ref.refname, refname)) {\n+\t\tret = 1;\n+\t\tgoto done;\n+\t}\n+\n \tif (ref.value_type == REFTABLE_REF_SYMREF) {\n \t\tstrbuf_reset(referent);\n \t\tstrbuf_addstr(referent, ref.value.symref);\n\n-- \n2.47.0.366.gd4f858ca17.dirty\n\n"}]}