{"thread":{"id":"66211","subject":"[PATCH v2 1/4] reftable/stack: remove `REFTABLE_STACK_NEW_ADDITION_RELOAD`","startedAt":"2026-08-24T09:31:17Z","lastAt":"2026-08-26T09:47:12Z","messageCount":9,"participants":["Karthik Nayak","Jeff King","Patrick Steinhardt"],"isPatch":true,"patchVersion":2,"patchTotal":4},"messages":[{"id":"551116","messageId":"20260824-740-optimize-reloading-the-reftable-stack-v2-1-9c9de2eb0af7@gmail.com","threadId":"66211","inReplyTo":"20260824-740-optimize-reloading-the-reftable-stack-v2-0-9c9de2eb0af7@gmail.com","subject":"[PATCH v2 1/4] reftable/stack: remove `REFTABLE_STACK_NEW_ADDITION_RELOAD`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-08-24T09:30:59Z","receivedAt":"2026-08-24T09:31:17Z","isPatch":true,"body":"In 80e7342ea8 (reftable/stack: allow locking of outdated stacks,\n2024-09-24), the `REFTABLE_STACK_NEW_ADDITION_RELOAD` was introduced so\nthat callers of `reftable_stack_init_addition()` can also reload the\nstack if there was a concurrent update made before the lock was\nobtained.\n\nThen 16684b6fae (refs/reftable: always reload stacks when creating\nlock, 2025-08-12) updated all of the remaining call-sites to propagate\nthis flag to ensure that we always reload the stack whenever there was a\nconcurrent update.\n\nAs all calls to `reftable_stack_init_addition()` inevitably propagate\nthe flag, it is safe to remove the flag and its associated code and make\nthe reloading of the stack the default flow. This makes it easier to\nfollow the flow and simplifies the logic.\n\nThe only exceptions are:\n\n  1. Unit tests, where we explicitly do not propagate the flag. These\n     tests are now modified with the new status quo.\n\n  2. `reftable_stack_clean()`, which was propagating 0 to\n     `reftable_stack_new_addition()` but was then manually reloading the\n     stack after. Here the new flow will achieve the same, while also\n     allowing us to remove the manual reload.\n\nThis also makes two checks for 'REFTABLE_OUTDATED_ERROR' redundant, so\nremove them also.\n\nSigned-off-by: Karthik Nayak <karthik.188@gmail.com>\n---\n refs/reftable-backend.c         | 18 ++++-------\n reftable/reftable-stack.h       | 17 ++--------\n reftable/stack.c                | 37 ++++++----------------\n t/unit-tests/u-reftable-stack.c | 69 ++++++++++++++++++-----------------------\n 4 files changed, 49 insertions(+), 92 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 028f0211af..5c87fd2d68 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1003,8 +1003,7 @@ static int prepare_transaction_update(struct write_transaction_table_arg **out,\n \t\tstruct reftable_addition *addition;\n \n \t\tret = reftable_stack_new_addition(&addition, be->stack,\n-\t\t\t\t\t\t  &reftable_be_write_options(refs)->opts,\n-\t\t\t\t\t\t  REFTABLE_STACK_NEW_ADDITION_RELOAD);\n+\t\t\t\t\t\t  &reftable_be_write_options(refs)->opts);\n \t\tif (ret) {\n \t\t\tif (ret == REFTABLE_LOCK_ERROR)\n \t\t\t\tstrbuf_addstr(err, \"cannot lock references\");\n@@ -2010,8 +2009,7 @@ static int reftable_be_rename_ref(struct ref_store *ref_store,\n \tif (ret)\n \t\tgoto done;\n \tret = reftable_stack_add(arg.be->stack, &write_copy_table, &arg,\n-\t\t\t\t &reftable_be_write_options(refs)->opts,\n-\t\t\t\t REFTABLE_STACK_NEW_ADDITION_RELOAD);\n+\t\t\t\t &reftable_be_write_options(refs)->opts);\n \n done:\n \tassert(ret != REFTABLE_API_ERROR);\n@@ -2041,8 +2039,7 @@ static int reftable_be_copy_ref(struct ref_store *ref_store,\n \tif (ret)\n \t\tgoto done;\n \tret = reftable_stack_add(arg.be->stack, &write_copy_table, &arg,\n-\t\t\t\t &reftable_be_write_options(refs)->opts,\n-\t\t\t\t REFTABLE_STACK_NEW_ADDITION_RELOAD);\n+\t\t\t\t &reftable_be_write_options(refs)->opts);\n \n done:\n \tassert(ret != REFTABLE_API_ERROR);\n@@ -2424,8 +2421,7 @@ static int reftable_be_create_reflog(struct ref_store *ref_store,\n \targ.stack = be->stack;\n \n \tret = reftable_stack_add(be->stack, &write_reflog_existence_table, &arg,\n-\t\t\t\t &reftable_be_write_options(refs)->opts,\n-\t\t\t\t REFTABLE_STACK_NEW_ADDITION_RELOAD);\n+\t\t\t\t &reftable_be_write_options(refs)->opts);\n \n done:\n \treturn ret;\n@@ -2499,8 +2495,7 @@ static int reftable_be_delete_reflog(struct ref_store *ref_store,\n \targ.stack = be->stack;\n \n \tret = reftable_stack_add(be->stack, &write_reflog_delete_table, &arg,\n-\t\t\t\t &reftable_be_write_options(refs)->opts,\n-\t\t\t\t REFTABLE_STACK_NEW_ADDITION_RELOAD);\n+\t\t\t\t &reftable_be_write_options(refs)->opts);\n \n \tassert(ret != REFTABLE_API_ERROR);\n \treturn ret;\n@@ -2622,8 +2617,7 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \t\tgoto done;\n \n \tret = reftable_stack_new_addition(&add, be->stack,\n-\t\t\t\t\t  &reftable_be_write_options(refs)->opts,\n-\t\t\t\t\t  REFTABLE_STACK_NEW_ADDITION_RELOAD);\n+\t\t\t\t\t  &reftable_be_write_options(refs)->opts);\n \tif (ret < 0)\n \t\tgoto done;\n \ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nindex 5d22d84e80..5d224f8079 100644\n--- a/reftable/reftable-stack.h\n+++ b/reftable/reftable-stack.h\n@@ -58,22 +58,13 @@ uint64_t reftable_stack_next_update_index(struct reftable_stack *st);\n /* holds a transaction to add tables at the top of a stack. */\n struct reftable_addition;\n \n-enum {\n-\t/*\n-\t * Reload the stack when the stack is out-of-date after locking it.\n-\t */\n-\tREFTABLE_STACK_NEW_ADDITION_RELOAD = (1 << 0),\n-};\n-\n /*\n  * returns a new transaction to add reftables to the given stack. As a side\n- * effect, the ref database is locked. Accepts REFTABLE_STACK_NEW_ADDITION_*\n- * flags.\n+ * effect, the ref database is locked.\n  */\n int reftable_stack_new_addition(struct reftable_addition **dest,\n \t\t\t\tstruct reftable_stack *st,\n-\t\t\t\tconst struct reftable_write_options *opts,\n-\t\t\t\tunsigned int flags);\n+\t\t\t\tconst struct reftable_write_options *opts);\n \n /* Adds a reftable to transaction. */\n int reftable_addition_add(struct reftable_addition *add,\n@@ -93,14 +84,12 @@ void reftable_addition_destroy(struct reftable_addition *add);\n /*\n  * Add a new table to the stack. The write_table function must call\n  * reftable_writer_set_limits, add refs and return an error value.\n- * The flags are passed through to `reftable_stack_new_addition()`.\n  */\n int reftable_stack_add(struct reftable_stack *st,\n \t\t       int (*write_table)(struct reftable_writer *wr,\n \t\t\t\t\t  void *write_arg),\n \t\t       void *write_arg,\n-\t\t       const struct reftable_write_options *opts,\n-\t\t       unsigned flags);\n+\t\t       const struct reftable_write_options *opts);\n \n struct reftable_iterator;\n \ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 308f9578f0..540f5e77ac 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -659,8 +659,7 @@ static void reftable_addition_close(struct reftable_addition *add)\n \n static int reftable_stack_init_addition(struct reftable_addition *add,\n \t\t\t\t\tstruct reftable_stack *st,\n-\t\t\t\t\tconst struct reftable_write_options *opts,\n-\t\t\t\t\tunsigned int flags)\n+\t\t\t\t\tconst struct reftable_write_options *opts)\n {\n \tstruct reftable_buf lock_file_name = REFTABLE_BUF_INIT;\n \tint err;\n@@ -686,15 +685,11 @@ static int reftable_stack_init_addition(struct reftable_addition *add,\n \terr = stack_uptodate(st);\n \tif (err < 0)\n \t\tgoto done;\n-\tif (err > 0 && flags & REFTABLE_STACK_NEW_ADDITION_RELOAD) {\n+\tif (err > 0) {\n \t\terr = reftable_stack_reload_maybe_reuse(add->stack, 1);\n \t\tif (err)\n \t\t\tgoto done;\n \t}\n-\tif (err > 0) {\n-\t\terr = REFTABLE_OUTDATED_ERROR;\n-\t\tgoto done;\n-\t}\n \n \tadd->next_update_index = reftable_stack_next_update_index(st);\n done:\n@@ -708,13 +703,12 @@ static int stack_try_add(struct reftable_stack *st,\n \t\t\t int (*write_table)(struct reftable_writer *wr,\n \t\t\t\t\t    void *arg),\n \t\t\t void *arg,\n-\t\t\t const struct reftable_write_options *opts,\n-\t\t\t unsigned flags)\n+\t\t\t const struct reftable_write_options *opts)\n {\n \tstruct reftable_addition add;\n \tint err;\n \n-\terr = reftable_stack_init_addition(&add, st, opts, flags);\n+\terr = reftable_stack_init_addition(&add, st, opts);\n \tif (err < 0)\n \t\tgoto done;\n \n@@ -731,17 +725,10 @@ static int stack_try_add(struct reftable_stack *st,\n int reftable_stack_add(struct reftable_stack *st,\n \t\t       int (*write)(struct reftable_writer *wr, void *arg),\n \t\t       void *arg,\n-\t\t       const struct reftable_write_options *opts,\n-\t\t       unsigned flags)\n+\t\t       const struct reftable_write_options *opts)\n {\n-\tint err = stack_try_add(st, write, arg, opts, flags);\n+\tint err = stack_try_add(st, write, arg, opts);\n \tif (err < 0) {\n-\t\tif (err == REFTABLE_OUTDATED_ERROR) {\n-\t\t\t/* Ignore error return, we want to propagate\n-\t\t\t   REFTABLE_OUTDATED_ERROR.\n-\t\t\t*/\n-\t\t\treftable_stack_reload(st);\n-\t\t}\n \t\treturn err;\n \t}\n \n@@ -843,8 +830,7 @@ int reftable_addition_commit(struct reftable_addition *add)\n \n int reftable_stack_new_addition(struct reftable_addition **dest,\n \t\t\t\tstruct reftable_stack *st,\n-\t\t\t\tconst struct reftable_write_options *opts,\n-\t\t\t\tunsigned int flags)\n+\t\t\t\tconst struct reftable_write_options *opts)\n {\n \tint err;\n \n@@ -852,7 +838,7 @@ int reftable_stack_new_addition(struct reftable_addition **dest,\n \tif (!*dest)\n \t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n \n-\terr = reftable_stack_init_addition(*dest, st, opts, flags);\n+\terr = reftable_stack_init_addition(*dest, st, opts);\n \tif (err) {\n \t\treftable_free(*dest);\n \t\t*dest = NULL;\n@@ -1840,12 +1826,7 @@ static int reftable_stack_clean_locked(struct reftable_stack *st)\n int reftable_stack_clean(struct reftable_stack *st)\n {\n \tstruct reftable_addition *add = NULL;\n-\tint err = reftable_stack_new_addition(&add, st, NULL, 0);\n-\tif (err < 0) {\n-\t\tgoto done;\n-\t}\n-\n-\terr = reftable_stack_reload(st);\n+\tint err = reftable_stack_new_addition(&add, st, NULL);\n \tif (err < 0) {\n \t\tgoto done;\n \t}\ndiff --git a/t/unit-tests/u-reftable-stack.c b/t/unit-tests/u-reftable-stack.c\nindex e6c1635940..c6254190e6 100644\n--- a/t/unit-tests/u-reftable-stack.c\n+++ b/t/unit-tests/u-reftable-stack.c\n@@ -127,7 +127,7 @@ static void write_n_ref_tables(struct reftable_stack *st,\n \t\tcl_reftable_set_hash(ref.value.val1, i, REFTABLE_HASH_SHA1);\n \n \t\tcl_assert_equal_i(reftable_stack_add(st,\n-\t\t\t\t\t\t     &write_test_ref, &ref, &opts, 0), 0);\n+\t\t\t\t\t\t     &write_test_ref, &ref, &opts), 0);\n \t}\n }\n \n@@ -168,7 +168,7 @@ void test_reftable_stack__add_one(void)\n \terr = reftable_new_stack(&st, dir, NULL);\n \tcl_assert(!err);\n \n-\terr = reftable_stack_add(st, write_test_ref, &ref, &opts, 0);\n+\terr = reftable_stack_add(st, write_test_ref, &ref, &opts);\n \tcl_assert(!err);\n \n \terr = reftable_stack_read_ref(st, ref.refname, &dest);\n@@ -231,12 +231,9 @@ void test_reftable_stack__uptodate(void)\n \tcl_assert_equal_i(reftable_new_stack(&st1, dir, NULL), 0);\n \tcl_assert_equal_i(reftable_new_stack(&st2, dir, NULL), 0);\n \tcl_assert_equal_i(reftable_stack_add(st1, write_test_ref,\n-\t\t\t\t\t     &ref1, NULL, 0), 0);\n+\t\t\t\t\t     &ref1, NULL), 0);\n \tcl_assert_equal_i(reftable_stack_add(st2, write_test_ref,\n-\t\t\t\t\t     &ref2, NULL, 0), REFTABLE_OUTDATED_ERROR);\n-\tcl_assert_equal_i(reftable_stack_reload(st2), 0);\n-\tcl_assert_equal_i(reftable_stack_add(st2, write_test_ref,\n-\t\t\t\t\t     &ref2, NULL, 0), 0);\n+\t\t\t\t\t     &ref2, NULL), 0);\n \treftable_stack_destroy(st1);\n \treftable_stack_destroy(st2);\n \tclear_dir(dir);\n@@ -260,7 +257,7 @@ void test_reftable_stack__transaction_api(void)\n \n \treftable_addition_destroy(add);\n \n-\tcl_assert_equal_i(reftable_stack_new_addition(&add, st, NULL, 0), 0);\n+\tcl_assert_equal_i(reftable_stack_new_addition(&add, st, NULL), 0);\n \tcl_assert_equal_i(reftable_addition_add(add, write_test_ref,\n \t\t\t\t\t\t&ref), 0);\n \tcl_assert_equal_i(reftable_addition_commit(add), 0);\n@@ -301,21 +298,17 @@ void test_reftable_stack__transaction_with_reload(void)\n \n \tcl_assert_equal_i(reftable_new_stack(&st1, dir, NULL), 0);\n \tcl_assert_equal_i(reftable_new_stack(&st2, dir, NULL), 0);\n-\tcl_assert_equal_i(reftable_stack_new_addition(&add, st1, NULL, 0), 0);\n+\tcl_assert_equal_i(reftable_stack_new_addition(&add, st1, NULL), 0);\n \tcl_assert_equal_i(reftable_addition_add(add, write_test_ref,\n \t\t\t\t\t\t&refs[0]), 0);\n \tcl_assert_equal_i(reftable_addition_commit(add), 0);\n \treftable_addition_destroy(add);\n \n \t/*\n-\t * The second stack is now outdated, which we should notice. We do not\n-\t * create the addition and lock the stack by default, but allow the\n-\t * reload to happen when REFTABLE_STACK_NEW_ADDITION_RELOAD is set.\n+\t * The second stack is now outdated, but it should automatically reload it\n+\t * with the newer updates.\n \t */\n-\tcl_assert_equal_i(reftable_stack_new_addition(&add, st2, NULL, 0),\n-\t\t\t\t\t\t      REFTABLE_OUTDATED_ERROR);\n-\tcl_assert_equal_i(reftable_stack_new_addition(&add, st2, NULL,\n-\t\t\t\t\t\t      REFTABLE_STACK_NEW_ADDITION_RELOAD), 0);\n+\tcl_assert_equal_i(reftable_stack_new_addition(&add, st2, NULL), 0);\n \tcl_assert_equal_i(reftable_addition_add(add, write_test_ref,\n \t\t\t\t\t\t&refs[1]), 0);\n \tcl_assert_equal_i(reftable_addition_commit(add), 0);\n@@ -363,7 +356,7 @@ void test_reftable_stack__transaction_api_performs_auto_compaction(void)\n \t\t * better control over when exactly auto compaction runs.\n \t\t */\n \t\tcl_assert_equal_i(reftable_stack_new_addition(&add,\n-\t\t\t\t\t\t\t      st, &write_opts, 0), 0);\n+\t\t\t\t\t\t\t      st, &write_opts), 0);\n \t\tcl_assert_equal_i(reftable_addition_add(add,\n \t\t\t\t\t\t\twrite_test_ref, &ref), 0);\n \t\tcl_assert_equal_i(reftable_addition_commit(add), 0);\n@@ -400,7 +393,7 @@ void test_reftable_stack__auto_compaction_fails_gracefully(void)\n \n \tcl_assert_equal_i(reftable_new_stack(&st, dir, NULL), 0);\n \tcl_assert_equal_i(reftable_stack_add(st, write_test_ref,\n-\t\t\t\t\t     &ref, NULL, 0), 0);\n+\t\t\t\t\t     &ref, NULL), 0);\n \tcl_assert_equal_i(st->merged->tables_len, 1);\n \tcl_assert_equal_i(st->stats.attempts, 0);\n \tcl_assert_equal_i(st->stats.failures, 0);\n@@ -418,7 +411,7 @@ void test_reftable_stack__auto_compaction_fails_gracefully(void)\n \twrite_file_buf(table_path.buf, \"\", 0);\n \n \tref.update_index = 2;\n-\terr = reftable_stack_add(st, write_test_ref, &ref, NULL, 0);\n+\terr = reftable_stack_add(st, write_test_ref, &ref, NULL);\n \tcl_assert(!err);\n \tcl_assert_equal_i(st->merged->tables_len, 2);\n \tcl_assert_equal_i(st->stats.attempts, 1);\n@@ -453,9 +446,9 @@ void test_reftable_stack__update_index_check(void)\n \n \tcl_assert_equal_i(reftable_new_stack(&st, dir, NULL), 0);\n \tcl_assert_equal_i(reftable_stack_add(st, write_test_ref,\n-\t\t\t\t\t     &ref1, NULL, 0), 0);\n+\t\t\t\t\t     &ref1, NULL), 0);\n \tcl_assert_equal_i(reftable_stack_add(st, write_test_ref,\n-\t\t\t\t\t     &ref2, NULL, 0), REFTABLE_API_ERROR);\n+\t\t\t\t\t     &ref2, NULL), REFTABLE_API_ERROR);\n \treftable_stack_destroy(st);\n \tclear_dir(dir);\n }\n@@ -469,7 +462,7 @@ void test_reftable_stack__lock_failure(void)\n \tcl_assert_equal_i(reftable_new_stack(&st, dir, NULL), 0);\n \tfor (i = -1; i != REFTABLE_EMPTY_TABLE_ERROR; i--)\n \t\tcl_assert_equal_i(reftable_stack_add(st, write_error,\n-\t\t\t\t\t\t     &i, NULL, 0), i);\n+\t\t\t\t\t\t     &i, NULL), i);\n \n \treftable_stack_destroy(st);\n \tclear_dir(dir);\n@@ -513,7 +506,7 @@ void test_reftable_stack__add(void)\n \n \tfor (i = 0; i < N; i++)\n \t\tcl_assert_equal_i(reftable_stack_add(st, write_test_ref,\n-\t\t\t\t\t\t     &refs[i], &opts, 0), 0);\n+\t\t\t\t\t\t     &refs[i], &opts), 0);\n \n \tfor (i = 0; i < N; i++) {\n \t\tstruct write_log_arg arg = {\n@@ -521,7 +514,7 @@ void test_reftable_stack__add(void)\n \t\t\t.update_index = reftable_stack_next_update_index(st),\n \t\t};\n \t\tcl_assert_equal_i(reftable_stack_add(st, write_test_log,\n-\t\t\t\t\t\t     &arg, &opts, 0), 0);\n+\t\t\t\t\t\t     &arg, &opts), 0);\n \t}\n \n \tcl_assert_equal_i(reftable_stack_compact_all(st, &opts, NULL), 0);\n@@ -604,7 +597,7 @@ void test_reftable_stack__iterator(void)\n \n \tfor (i = 0; i < N; i++)\n \t\tcl_assert_equal_i(reftable_stack_add(st, write_test_ref,\n-\t\t\t\t\t\t     &refs[i], NULL, 0), 0);\n+\t\t\t\t\t\t     &refs[i], NULL), 0);\n \n \tfor (i = 0; i < N; i++) {\n \t\tstruct write_log_arg arg = {\n@@ -613,7 +606,7 @@ void test_reftable_stack__iterator(void)\n \t\t};\n \n \t\tcl_assert_equal_i(reftable_stack_add(st, write_test_log,\n-\t\t\t\t\t\t     &arg, NULL, 0), 0);\n+\t\t\t\t\t\t     &arg, NULL), 0);\n \t}\n \n \treftable_stack_init_ref_iterator(st, &it);\n@@ -685,11 +678,11 @@ void test_reftable_stack__log_normalize(void)\n \n \tinput.value.update.message = (char *) \"one\\ntwo\";\n \tcl_assert_equal_i(reftable_stack_add(st, write_test_log,\n-\t\t\t\t\t     &arg, NULL, 0), REFTABLE_API_ERROR);\n+\t\t\t\t\t     &arg, NULL), REFTABLE_API_ERROR);\n \n \tinput.value.update.message = (char *) \"one\";\n \tcl_assert_equal_i(reftable_stack_add(st, write_test_log,\n-\t\t\t\t\t     &arg, NULL, 0), 0);\n+\t\t\t\t\t     &arg, NULL), 0);\n \tcl_assert_equal_i(reftable_stack_read_log(st, input.refname,\n \t\t\t\t\t\t  &dest), 0);\n \tcl_assert_equal_s(dest.value.update.message, \"one\\n\");\n@@ -697,7 +690,7 @@ void test_reftable_stack__log_normalize(void)\n \tinput.value.update.message = (char *) \"two\\n\";\n \targ.update_index = 2;\n \tcl_assert_equal_i(reftable_stack_add(st, write_test_log,\n-\t\t\t\t\t     &arg, NULL, 0), 0);\n+\t\t\t\t\t     &arg, NULL), 0);\n \tcl_assert_equal_i(reftable_stack_read_log(st, input.refname,\n \t\t\t\t\t\t  &dest), 0);\n \tcl_assert_equal_s(dest.value.update.message, \"two\\n\");\n@@ -747,7 +740,7 @@ void test_reftable_stack__tombstone(void)\n \t}\n \tfor (i = 0; i < N; i++)\n \t\tcl_assert_equal_i(reftable_stack_add(st, write_test_ref,\n-\t\t\t\t\t\t     &refs[i], NULL, 0), 0);\n+\t\t\t\t\t\t     &refs[i], NULL), 0);\n \n \tfor (i = 0; i < N; i++) {\n \t\tstruct write_log_arg arg = {\n@@ -755,7 +748,7 @@ void test_reftable_stack__tombstone(void)\n \t\t\t.update_index = reftable_stack_next_update_index(st),\n \t\t};\n \t\tcl_assert_equal_i(reftable_stack_add(st, write_test_log,\n-\t\t\t\t\t\t     &arg, NULL, 0), 0);\n+\t\t\t\t\t\t     &arg, NULL), 0);\n \t}\n \n \tcl_assert_equal_i(reftable_stack_read_ref(st, \"branch\",\n@@ -801,7 +794,7 @@ void test_reftable_stack__hash_id(void)\n \n \tcl_assert_equal_i(reftable_new_stack(&st, dir, NULL), 0);\n \tcl_assert_equal_i(reftable_stack_add(st, write_test_ref,\n-\t\t\t\t\t     &ref, NULL, 0), 0);\n+\t\t\t\t\t     &ref, NULL), 0);\n \n \t/* can't read it with the wrong hash ID. */\n \tcl_assert_equal_i(reftable_new_stack(&st32, dir,\n@@ -869,7 +862,7 @@ void test_reftable_stack__reflog_expire(void)\n \t\t\t.update_index = reftable_stack_next_update_index(st),\n \t\t};\n \t\tcl_assert_equal_i(reftable_stack_add(st, write_test_log,\n-\t\t\t\t\t\t     &arg, NULL, 0), 0);\n+\t\t\t\t\t\t     &arg, NULL), 0);\n \t}\n \n \tcl_assert_equal_i(reftable_stack_compact_all(st, NULL, NULL), 0);\n@@ -908,7 +901,7 @@ void test_reftable_stack__empty_add(void)\n \n \tcl_assert_equal_i(reftable_new_stack(&st, dir, NULL), 0);\n \tcl_assert_equal_i(reftable_stack_add(st, write_nothing,\n-\t\t\t\t\t     NULL, NULL, 0), 0);\n+\t\t\t\t\t     NULL, NULL), 0);\n \tcl_assert_equal_i(reftable_new_stack(&st2, dir, NULL), 0);\n \tclear_dir(dir);\n \treftable_stack_destroy(st);\n@@ -947,7 +940,7 @@ void test_reftable_stack__auto_compaction(void)\n \t\t};\n \t\tsnprintf(name, sizeof(name), \"branch%04\"PRIuMAX, (uintmax_t)i);\n \n-\t\terr = reftable_stack_add(st, write_test_ref, &ref, &opts, 0);\n+\t\terr = reftable_stack_add(st, write_test_ref, &ref, &opts);\n \t\tcl_assert(!err);\n \n \t\terr = reftable_stack_auto_compact(st, &opts);\n@@ -983,7 +976,7 @@ void test_reftable_stack__auto_compaction_factor(void)\n \t\t};\n \t\txsnprintf(name, sizeof(name), \"branch%04\"PRIuMAX, (uintmax_t)i);\n \n-\t\terr = reftable_stack_add(st, &write_test_ref, &ref, &opts, 0);\n+\t\terr = reftable_stack_add(st, &write_test_ref, &ref, &opts);\n \t\tcl_assert(!err);\n \n \t\tcl_assert(i < 5 || st->merged->tables_len < 5 * fastlogN(i, 5));\n@@ -1064,7 +1057,7 @@ void test_reftable_stack__add_performs_auto_compaction(void)\n \t\tref.refname = buf;\n \n \t\tcl_assert_equal_i(reftable_stack_add(st, write_test_ref,\n-\t\t\t\t\t\t     &ref, &write_opts, 0), 0);\n+\t\t\t\t\t\t     &ref, &write_opts), 0);\n \n \t\t/*\n \t\t * The stack length should grow continuously for all runs where\n@@ -1303,7 +1296,7 @@ void test_reftable_stack__invalid_limit_updates(void)\n \n \treftable_addition_destroy(add);\n \n-\tcl_assert_equal_i(reftable_stack_new_addition(&add, st, &opts, 0), 0);\n+\tcl_assert_equal_i(reftable_stack_new_addition(&add, st, &opts), 0);\n \n \t/*\n \t * write_limits_after_ref also updates the update indexes after adding\n\n-- \n2.55.GIT\n\n"},{"id":"551117","messageId":"20260824-740-optimize-reloading-the-reftable-stack-v2-2-9c9de2eb0af7@gmail.com","threadId":"66211","inReplyTo":"20260824-740-optimize-reloading-the-reftable-stack-v2-0-9c9de2eb0af7@gmail.com","subject":"[PATCH v2 2/4] reftable/stack: rename reftable_stack_new_addition()","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-08-24T09:31:00Z","receivedAt":"2026-08-24T09:31:18Z","isPatch":true,"body":"Rename the function `reftable_stack_new_addition()` to\n`reftable_stack_addition_new()` to be more inline with our naming\nscheme.\n\nSigned-off-by: Karthik Nayak <karthik.188@gmail.com>\n---\n refs/reftable-backend.c         |  4 ++--\n reftable/reftable-stack.h       |  2 +-\n reftable/stack.c                |  4 ++--\n t/unit-tests/u-reftable-stack.c | 10 +++++-----\n 4 files changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 5c87fd2d68..73cd794fc6 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1002,7 +1002,7 @@ static int prepare_transaction_update(struct write_transaction_table_arg **out,\n \tif (!arg) {\n \t\tstruct reftable_addition *addition;\n \n-\t\tret = reftable_stack_new_addition(&addition, be->stack,\n+\t\tret = reftable_stack_addition_new(&addition, be->stack,\n \t\t\t\t\t\t  &reftable_be_write_options(refs)->opts);\n \t\tif (ret) {\n \t\t\tif (ret == REFTABLE_LOCK_ERROR)\n@@ -2616,7 +2616,7 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,\n \tif (ret < 0)\n \t\tgoto done;\n \n-\tret = reftable_stack_new_addition(&add, be->stack,\n+\tret = reftable_stack_addition_new(&add, be->stack,\n \t\t\t\t\t  &reftable_be_write_options(refs)->opts);\n \tif (ret < 0)\n \t\tgoto done;\ndiff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h\nindex 5d224f8079..875d09d241 100644\n--- a/reftable/reftable-stack.h\n+++ b/reftable/reftable-stack.h\n@@ -62,7 +62,7 @@ struct reftable_addition;\n  * returns a new transaction to add reftables to the given stack. As a side\n  * effect, the ref database is locked.\n  */\n-int reftable_stack_new_addition(struct reftable_addition **dest,\n+int reftable_stack_addition_new(struct reftable_addition **dest,\n \t\t\t\tstruct reftable_stack *st,\n \t\t\t\tconst struct reftable_write_options *opts);\n \ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 540f5e77ac..703548417c 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -828,7 +828,7 @@ int reftable_addition_commit(struct reftable_addition *add)\n \treturn err;\n }\n \n-int reftable_stack_new_addition(struct reftable_addition **dest,\n+int reftable_stack_addition_new(struct reftable_addition **dest,\n \t\t\t\tstruct reftable_stack *st,\n \t\t\t\tconst struct reftable_write_options *opts)\n {\n@@ -1826,7 +1826,7 @@ static int reftable_stack_clean_locked(struct reftable_stack *st)\n int reftable_stack_clean(struct reftable_stack *st)\n {\n \tstruct reftable_addition *add = NULL;\n-\tint err = reftable_stack_new_addition(&add, st, NULL);\n+\tint err = reftable_stack_addition_new(&add, st, NULL);\n \tif (err < 0) {\n \t\tgoto done;\n \t}\ndiff --git a/t/unit-tests/u-reftable-stack.c b/t/unit-tests/u-reftable-stack.c\nindex c6254190e6..04927113c2 100644\n--- a/t/unit-tests/u-reftable-stack.c\n+++ b/t/unit-tests/u-reftable-stack.c\n@@ -257,7 +257,7 @@ void test_reftable_stack__transaction_api(void)\n \n \treftable_addition_destroy(add);\n \n-\tcl_assert_equal_i(reftable_stack_new_addition(&add, st, NULL), 0);\n+\tcl_assert_equal_i(reftable_stack_addition_new(&add, st, NULL), 0);\n \tcl_assert_equal_i(reftable_addition_add(add, write_test_ref,\n \t\t\t\t\t\t&ref), 0);\n \tcl_assert_equal_i(reftable_addition_commit(add), 0);\n@@ -298,7 +298,7 @@ void test_reftable_stack__transaction_with_reload(void)\n \n \tcl_assert_equal_i(reftable_new_stack(&st1, dir, NULL), 0);\n \tcl_assert_equal_i(reftable_new_stack(&st2, dir, NULL), 0);\n-\tcl_assert_equal_i(reftable_stack_new_addition(&add, st1, NULL), 0);\n+\tcl_assert_equal_i(reftable_stack_addition_new(&add, st1, NULL), 0);\n \tcl_assert_equal_i(reftable_addition_add(add, write_test_ref,\n \t\t\t\t\t\t&refs[0]), 0);\n \tcl_assert_equal_i(reftable_addition_commit(add), 0);\n@@ -308,7 +308,7 @@ void test_reftable_stack__transaction_with_reload(void)\n \t * The second stack is now outdated, but it should automatically reload it\n \t * with the newer updates.\n \t */\n-\tcl_assert_equal_i(reftable_stack_new_addition(&add, st2, NULL), 0);\n+\tcl_assert_equal_i(reftable_stack_addition_new(&add, st2, NULL), 0);\n \tcl_assert_equal_i(reftable_addition_add(add, write_test_ref,\n \t\t\t\t\t\t&refs[1]), 0);\n \tcl_assert_equal_i(reftable_addition_commit(add), 0);\n@@ -355,7 +355,7 @@ void test_reftable_stack__transaction_api_performs_auto_compaction(void)\n \t\t * we can ensure that we indeed honor this setting and have\n \t\t * better control over when exactly auto compaction runs.\n \t\t */\n-\t\tcl_assert_equal_i(reftable_stack_new_addition(&add,\n+\t\tcl_assert_equal_i(reftable_stack_addition_new(&add,\n \t\t\t\t\t\t\t      st, &write_opts), 0);\n \t\tcl_assert_equal_i(reftable_addition_add(add,\n \t\t\t\t\t\t\twrite_test_ref, &ref), 0);\n@@ -1296,7 +1296,7 @@ void test_reftable_stack__invalid_limit_updates(void)\n \n \treftable_addition_destroy(add);\n \n-\tcl_assert_equal_i(reftable_stack_new_addition(&add, st, &opts), 0);\n+\tcl_assert_equal_i(reftable_stack_addition_new(&add, st, &opts), 0);\n \n \t/*\n \t * write_limits_after_ref also updates the update indexes after adding\n\n-- \n2.55.GIT\n\n"},{"id":"551118","messageId":"20260824-740-optimize-reloading-the-reftable-stack-v2-0-9c9de2eb0af7@gmail.com","threadId":"66211","inReplyTo":"20260819-740-optimize-reloading-the-reftable-stack-v1-0-6bf5305d4e43@gmail.com","subject":"[PATCH v2 0/4] reftable/stack: avoid reloading the stack when locked","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-08-24T09:30:58Z","receivedAt":"2026-08-24T09:31:18Z","isPatch":true,"body":"This patch series is based on the report by Jeff [1], where he noticed\nthat when creating a lot of refs within a single reference transaction,\nthe majority of the time was spent on fstat().\n\nThe issue stems from the fact that within the reftable library we do not\ntrack Git reference transactions, as such any calls within the library\nwould potentially reload the stack to ensure that there are no\nconcurrent updates made to the stack. While this makes sense outside of\na reference transaction, within one, the stack is locked, so reloading\nthe stack is a no-op. The only time we want to reload the stack is\nimmediately after locking the list file, which is to catch any\nconcurrent updates made to the stack.\n\nThe first patch in this small series, cleans up the flow of reloading\nthe stack by providing a flag explicitly. The patch argues that since\nall flows reload the stack, the flag can be safely removed. This\nsimplifies the flow of when to reload the stack.\n\nThe next two commits move the lock variable to the reftable_stack\nstructure and then use this information to decide if reloading of the\nstack is necessary.\n\nDuring benchmarking, I first tried to benchmark adding new references\nagainst HEAD. This kicks in the DWIM ref resolution, and we iterate over\nsiz difference candidate ref names before settling on a match. Each such\nlookup reloads the stack. This happens before the reference transaction\nis created. I quickly realized that this would dominate the benchmarks,\nso the benchmarks in the third patch are against a static commit OID.\n\nBenchmarking with a fixed, non-symbolic target OID in the 'refs/tags/'\nnamespace (since it triggers a stack reload when checking if reflog\nexists for the given tag name), shows a consistent 15-20% improvement\nwith these patches:\n\n   refcount   master     patch     speedup\n   --------   -------    -------   -------\n   2,000       18.5 ms    16.6 ms   1.11x\n   20,000     120.7 ms   102.8 ms   1.17x\n   50,000     296.5 ms   247.1 ms   1.20x\n\nThe syscall counts show that now the calls to `newfstatat()` stay\nconstant at around 55 calls regardless of the number of refs to be\ncreated. Before this would grow linearly with the number of refs.\n\n[1]: https://lore.kernel.org/git/20260629203527.GA1895313@coredump.intra.peff.net/\n\nSigned-off-by: Karthik Nayak <karthik.188@gmail.com>\n---\nChanges in v2:\n- Fix typos in commit messages and add additional information.\n- To prevent multiple `reftable_addition`'s from releasing each others\n  list lock, add a bit field to track ownership. Add related unit test.\n- Add correct benchmark results in the last commit, my earlier results\n  were based on using 'refs/heads' which doesn't check for reflogs when\n  using the default config. Use 'refs/tags' as suggested by Peff. This\n  shows a good 15-20% improvement.\n- Link to v1: https://patch.msgid.link/20260819-740-optimize-reloading-the-reftable-stack-v1-0-6bf5305d4e43@gmail.com\n\n---\nKarthik Nayak (4):\n      reftable/stack: remove `REFTABLE_STACK_NEW_ADDITION_RELOAD`\n      reftable/stack: rename reftable_stack_new_addition()\n      reftable/stack: move list lock to `struct reftable_stack`\n      reftable/stack: avoid reloading the stack when already locked\n\n refs/reftable-backend.c         | 22 ++++-----\n reftable/reftable-stack.h       | 19 ++------\n reftable/stack.c                | 82 +++++++++++++++++-----------------\n reftable/stack.h                |  7 ++-\n t/unit-tests/u-reftable-stack.c | 99 +++++++++++++++++++++++++----------------\n 5 files changed, 119 insertions(+), 110 deletions(-)\n\nRange-diff versus v1:\n\n1:  63f46a1517 ! 1:  8ec5578141 reftable/stack: remove `REFTABLE_STACK_NEW_ADDITION_RELOAD`\n    @@ Commit message\n           1. Unit tests, where we explicitly do not propagate the flag. These\n              tests are now modified with the new status quo.\n     \n    -      2. `reftable_stack_clean_locked()`, which was propagating 0 to\n    +      2. `reftable_stack_clean()`, which was propagating 0 to\n              `reftable_stack_new_addition()` but was then manually reloading the\n              stack after. Here the new flow will achieve the same, while also\n              allowing us to remove the manual reload.\n-:  ---------- > 2:  45d965abd3 reftable/stack: rename reftable_stack_new_addition()\n2:  2796229716 ! 3:  434bba1bee reftable/stack: move list lock to `struct reftable_stack`\n    @@ Commit message\n         on top of `struct reftable_stack`, and we would miss the optimization in\n         such paths.\n     \n    +    Since the lock is now shared across all additions on the same stack, a\n    +    second `reftable_addition` that fails to acquire the already held lock\n    +    would still call `reftable_addition_close()`, which will release the\n    +    `stack->list_lock` which is still held by the first addition. To avoid\n    +    this, add a new bit field `locked` to `reftable_addition` that tracks\n    +    whether a particular addition is the one holding the lock, and only\n    +    release it in that case. Add a unit test to validate this behavior.\n    +\n         While here, remove an unused header file from 'reftable/stack.h'.\n     \n         Signed-off-by: Karthik Nayak <karthik.188@gmail.com>\n    @@ reftable/stack.c: int reftable_stack_reload(struct reftable_stack *st)\n      \tstruct reftable_stack *stack;\n      \tstruct reftable_write_options opts;\n      \n    ++\t/*\n    ++\t * While the list lock is acquired on the stack, we need to distinguish\n    ++\t * which 'reftable_addition' is responsible for the lock. This avoids\n    ++\t * clearing the lock of another 'reftable_addition'.\n    ++\t */\n    ++\tunsigned int locked : 1;\n    ++\n    + \tchar **new_tables;\n    + \tsize_t new_tables_len, new_tables_cap;\n    + \tuint64_t next_update_index;\n     @@ reftable/stack.c: static void reftable_addition_close(struct reftable_addition *add)\n      \tadd->new_tables_len = 0;\n      \tadd->new_tables_cap = 0;\n      \n     -\tflock_release(&add->tables_list_lock);\n    -+\tflock_release(&add->stack->list_lock);\n    ++\tif (add->locked)\n    ++\t\tflock_release(&add->stack->list_lock);\n    ++\tadd->locked = 0;\n      \treftable_buf_release(&nm);\n      }\n      \n    @@ reftable/stack.c: static int reftable_stack_init_addition(struct reftable_additi\n      \t\t\t    add->opts.lock_timeout_ms);\n      \tif (err < 0)\n      \t\tgoto done;\n    ++\tadd->locked = 1;\n      \n      \tif (add->opts.default_permissions) {\n     -\t\tif (chmod(add->tables_list_lock.path,\n    @@ reftable/stack.c: int reftable_addition_commit(struct reftable_addition *add)\n      \tif (err < 0) {\n      \t\terr = REFTABLE_IO_ERROR;\n      \t\tgoto done;\n    + \t}\n    ++\tadd->locked = 0;\n    + \n    + \t/* success, no more state to clean up. */\n    + \tfor (i = 0; i < add->new_tables_len; i++)\n     \n      ## reftable/stack.h ##\n     @@\n    @@ reftable/stack.h: struct reftable_stack {\n      \tchar *reftable_dir;\n      \n      \tstruct reftable_stack_options opts;\n    +\n    + ## t/unit-tests/u-reftable-stack.c ##\n    +@@ t/unit-tests/u-reftable-stack.c: void test_reftable_stack__invalid_limit_updates(void)\n    + \treftable_stack_destroy(st);\n    + \tclear_dir(dir);\n    + }\n    ++\n    ++void test_reftable_stack__two_additions(void)\n    ++{\n    ++\tstruct reftable_stack *st = NULL;\n    ++\tchar *dir = get_tmp_dir(__LINE__);\n    ++\tstruct reftable_addition *add1 = NULL;\n    ++\tstruct reftable_addition *add2 = NULL;\n    ++\n    ++\tstruct reftable_ref_record ref = {\n    ++\t\t.refname = (char *) \"HEAD\",\n    ++\t\t.update_index = 1,\n    ++\t\t.value_type = REFTABLE_REF_SYMREF,\n    ++\t\t.value.symref = (char *) \"master\",\n    ++\t};\n    ++\n    ++\tcl_assert_equal_i(reftable_new_stack(&st, dir, NULL), 0);\n    ++\n    ++\tcl_assert_equal_i(reftable_stack_addition_new(&add1, st, NULL), 0);\n    ++\tcl_assert_equal_i(reftable_stack_addition_new(&add2, st, NULL), REFTABLE_LOCK_ERROR);\n    ++\n    ++\tcl_assert_equal_i(reftable_addition_add(add1, write_test_ref, &ref), 0);\n    ++\n    ++\tcl_assert_equal_i(reftable_addition_commit(add1), 0);\n    ++\n    ++\treftable_addition_destroy(add1);\n    ++\treftable_stack_destroy(st);\n    ++\tclear_dir(dir);\n    ++}\n3:  7d7e4a7a33 ! 4:  bd038809b9 reftable/stack: avoid reloading the stack when already locked\n    @@ Commit message\n         no-op. Now that the struct has access to the lock file status, simply\n         skip reloading if the lock is present.\n     \n    -    Benchmarking with a fixed, non-symbolic target OID shows a modest but\n    -    consistent ~1-2% improvement in clock time for `update-ref` across ref\n    -    counts ranging from 2,000 to 100,000.\n    +    Benchmarking with a fixed, non-symbolic target OID in the 'refs/tags/'\n    +    namespace (since it triggers a stack reload when checking if reflog\n    +    exists for the given tag name), shows a consistent 15-20% improvement\n    +    with these patches:\n     \n    -    We can see better improvements in the number of syscall counts. On\n    +      refcount   master     patch     speedup\n    +      --------   -------    -------   -------\n    +      2,000       18.5 ms    16.6 ms   1.11x\n    +      20,000     120.7 ms   102.8 ms   1.17x\n    +      50,000     296.5 ms   247.1 ms   1.20x\n    +\n    +    We can also see the improvements in the number of syscall counts. On\n         master, the number of calls to `newfstatat()` grows linearly with the\n         number of refs created. With this patch, the number is now a constant:\n     \n\n---\nbase-commit: 18e66859d87fb4b76599f73460b54f0848c76b16\nchange-id: 20260814-740-optimize-reloading-the-reftable-stack-f5f3adf0a0c0\n\n\nThanks\n- Karthik\n\n"},{"id":"551119","messageId":"20260824-740-optimize-reloading-the-reftable-stack-v2-3-9c9de2eb0af7@gmail.com","threadId":"66211","inReplyTo":"20260824-740-optimize-reloading-the-reftable-stack-v2-0-9c9de2eb0af7@gmail.com","subject":"[PATCH v2 3/4] reftable/stack: move list lock to `struct reftable_stack`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-08-24T09:31:01Z","receivedAt":"2026-08-24T09:31:19Z","isPatch":true,"body":"The struct `reftable_addition` is used to modify a given stack, as such,\nit also includes a `struct reftable_flock` used to obtain the lock to\nthe list file. While the scope of the field lies within this struct, it\ndoesn't allow for optimizations to be made on `struct reftable_stack`\nitself.\n\nMove the field to `struct reftable_stack`, allowing us to make a simple\noptimization around avoiding a stack reload when we have already\nobtained a lock. While this is currently possible in the write path, the\nwrite path also contains multiple branches to reads which only work\non top of `struct reftable_stack`, and we would miss the optimization in\nsuch paths.\n\nSince the lock is now shared across all additions on the same stack, a\nsecond `reftable_addition` that fails to acquire the already held lock\nwould still call `reftable_addition_close()`, which will release the\n`stack->list_lock` which is still held by the first addition. To avoid\nthis, add a new bit field `locked` to `reftable_addition` that tracks\nwhether a particular addition is the one holding the lock, and only\nrelease it in that case. Add a unit test to validate this behavior.\n\nWhile here, remove an unused header file from 'reftable/stack.h'.\n\nSigned-off-by: Karthik Nayak <karthik.188@gmail.com>\n---\n reftable/stack.c                | 26 +++++++++++++++++++-------\n reftable/stack.h                |  7 ++++++-\n t/unit-tests/u-reftable-stack.c | 28 ++++++++++++++++++++++++++++\n 3 files changed, 53 insertions(+), 8 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex 703548417c..c3d4deff29 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -536,6 +536,8 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \t\tgoto out;\n \t}\n \n+\tp->list_lock = REFTABLE_FLOCK_INIT;\n+\n \terr = reftable_stack_reload_maybe_reuse(p, 1);\n \tif (err < 0)\n \t\tgoto out;\n@@ -628,10 +630,16 @@ int reftable_stack_reload(struct reftable_stack *st)\n }\n \n struct reftable_addition {\n-\tstruct reftable_flock tables_list_lock;\n \tstruct reftable_stack *stack;\n \tstruct reftable_write_options opts;\n \n+\t/*\n+\t * While the list lock is acquired on the stack, we need to distinguish\n+\t * which 'reftable_addition' is responsible for the lock. This avoids\n+\t * clearing the lock of another 'reftable_addition'.\n+\t */\n+\tunsigned int locked : 1;\n+\n \tchar **new_tables;\n \tsize_t new_tables_len, new_tables_cap;\n \tuint64_t next_update_index;\n@@ -653,7 +661,9 @@ static void reftable_addition_close(struct reftable_addition *add)\n \tadd->new_tables_len = 0;\n \tadd->new_tables_cap = 0;\n \n-\tflock_release(&add->tables_list_lock);\n+\tif (add->locked)\n+\t\tflock_release(&add->stack->list_lock);\n+\tadd->locked = 0;\n \treftable_buf_release(&nm);\n }\n \n@@ -669,13 +679,14 @@ static int reftable_stack_init_addition(struct reftable_addition *add,\n \tif (opts)\n \t\tadd->opts = *opts;\n \n-\terr = flock_acquire(&add->tables_list_lock, st->list_file,\n+\terr = flock_acquire(&add->stack->list_lock, st->list_file,\n \t\t\t    add->opts.lock_timeout_ms);\n \tif (err < 0)\n \t\tgoto done;\n+\tadd->locked = 1;\n \n \tif (add->opts.default_permissions) {\n-\t\tif (chmod(add->tables_list_lock.path,\n+\t\tif (chmod(add->stack->list_lock.path,\n \t\t\t  add->opts.default_permissions) < 0) {\n \t\t\terr = REFTABLE_IO_ERROR;\n \t\t\tgoto done;\n@@ -774,7 +785,7 @@ int reftable_addition_commit(struct reftable_addition *add)\n \t\t\tgoto done;\n \t}\n \n-\terr = reftable_write_data(add->tables_list_lock.fd,\n+\terr = reftable_write_data(add->stack->list_lock.fd,\n \t\t\t\t  table_list.buf, table_list.len);\n \treftable_buf_release(&table_list);\n \tif (err < 0) {\n@@ -782,17 +793,18 @@ int reftable_addition_commit(struct reftable_addition *add)\n \t\tgoto done;\n \t}\n \n-\terr = fsync(add->tables_list_lock.fd);\n+\terr = fsync(add->stack->list_lock.fd);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n \n-\terr = flock_commit(&add->tables_list_lock);\n+\terr = flock_commit(&add->stack->list_lock);\n \tif (err < 0) {\n \t\terr = REFTABLE_IO_ERROR;\n \t\tgoto done;\n \t}\n+\tadd->locked = 0;\n \n \t/* success, no more state to clean up. */\n \tfor (i = 0; i < add->new_tables_len; i++)\ndiff --git a/reftable/stack.h b/reftable/stack.h\nindex f7901e6c6f..52e07ad551 100644\n--- a/reftable/stack.h\n+++ b/reftable/stack.h\n@@ -10,7 +10,6 @@\n #define STACK_H\n \n #include \"system.h\"\n-#include \"reftable-writer.h\"\n #include \"reftable-stack.h\"\n \n struct reftable_stack {\n@@ -18,6 +17,12 @@ struct reftable_stack {\n \tchar *list_file;\n \tint list_fd;\n \n+\t/*\n+\t * Set while an addition holds the stack locked. Used by\n+\t * stack_uptodate() to skip reload checks while locked.\n+\t */\n+\tstruct reftable_flock list_lock;\n+\n \tchar *reftable_dir;\n \n \tstruct reftable_stack_options opts;\ndiff --git a/t/unit-tests/u-reftable-stack.c b/t/unit-tests/u-reftable-stack.c\nindex 04927113c2..b6f1c6cc52 100644\n--- a/t/unit-tests/u-reftable-stack.c\n+++ b/t/unit-tests/u-reftable-stack.c\n@@ -1310,3 +1310,31 @@ void test_reftable_stack__invalid_limit_updates(void)\n \treftable_stack_destroy(st);\n \tclear_dir(dir);\n }\n+\n+void test_reftable_stack__two_additions(void)\n+{\n+\tstruct reftable_stack *st = NULL;\n+\tchar *dir = get_tmp_dir(__LINE__);\n+\tstruct reftable_addition *add1 = NULL;\n+\tstruct reftable_addition *add2 = NULL;\n+\n+\tstruct reftable_ref_record ref = {\n+\t\t.refname = (char *) \"HEAD\",\n+\t\t.update_index = 1,\n+\t\t.value_type = REFTABLE_REF_SYMREF,\n+\t\t.value.symref = (char *) \"master\",\n+\t};\n+\n+\tcl_assert_equal_i(reftable_new_stack(&st, dir, NULL), 0);\n+\n+\tcl_assert_equal_i(reftable_stack_addition_new(&add1, st, NULL), 0);\n+\tcl_assert_equal_i(reftable_stack_addition_new(&add2, st, NULL), REFTABLE_LOCK_ERROR);\n+\n+\tcl_assert_equal_i(reftable_addition_add(add1, write_test_ref, &ref), 0);\n+\n+\tcl_assert_equal_i(reftable_addition_commit(add1), 0);\n+\n+\treftable_addition_destroy(add1);\n+\treftable_stack_destroy(st);\n+\tclear_dir(dir);\n+}\n\n-- \n2.55.GIT\n\n"},{"id":"551120","messageId":"20260824-740-optimize-reloading-the-reftable-stack-v2-4-9c9de2eb0af7@gmail.com","threadId":"66211","inReplyTo":"20260824-740-optimize-reloading-the-reftable-stack-v2-0-9c9de2eb0af7@gmail.com","subject":"[PATCH v2 4/4] reftable/stack: avoid reloading the stack when already locked","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-08-24T09:31:02Z","receivedAt":"2026-08-24T09:31:21Z","isPatch":true,"body":"When making modifications to the reftable stack, the stack obtains a\nlock to the list file and removes the lock after the commit phase. Since\nmost operations reload the stack to ensure we have the latest state, any\nbranched operation during the locked phase could trigger a state reload.\n\nTo prevent data loss due to concurrent writes, state reload is necessary\nright after obtaining the lock. But any reloads after that are just a\nno-op. Now that the struct has access to the lock file status, simply\nskip reloading if the lock is present.\n\nBenchmarking with a fixed, non-symbolic target OID in the 'refs/tags/'\nnamespace (since it triggers a stack reload when checking if reflog\nexists for the given tag name), shows a consistent 15-20% improvement\nwith these patches:\n\n  refcount   master     patch     speedup\n  --------   -------    -------   -------\n  2,000       18.5 ms    16.6 ms   1.11x\n  20,000     120.7 ms   102.8 ms   1.17x\n  50,000     296.5 ms   247.1 ms   1.20x\n\nWe can also see the improvements in the number of syscall counts. On\nmaster, the number of calls to `newfstatat()` grows linearly with the\nnumber of refs created. With this patch, the number is now a constant:\n\n  refcount   master   patch\n  --------   ------   ------\n  1,000      1,059       55\n  5,000      5,059       55\n  10,000     10,059      55\n  20,000     20,059      55\n\nReported-by: Jeff King <peff@peff.net>\nSigned-off-by: Karthik Nayak <karthik.188@gmail.com>\n---\n reftable/stack.c | 17 ++++++++++++-----\n 1 file changed, 12 insertions(+), 5 deletions(-)\n\ndiff --git a/reftable/stack.c b/reftable/stack.c\nindex c3d4deff29..47a60db079 100644\n--- a/reftable/stack.c\n+++ b/reftable/stack.c\n@@ -553,14 +553,21 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir,\n \n /*\n  * Check whether the given stack is up-to-date with what we have in memory.\n+ * If skip_if_locked is set skip stack reloading if the stack is currently\n+ * locked. Stack reloading must _not_ be skipped right after obtaining the\n+ * lock, to check for concurrent updates which may have happened.\n+ *\n  * Returns 0 if so, 1 if the stack is out-of-date or a negative error code\n  * otherwise.\n  */\n-static int stack_uptodate(struct reftable_stack *st)\n+static int stack_uptodate(struct reftable_stack *st, int skip_if_locked)\n {\n \tchar **names = NULL;\n \tint err;\n \n+\tif (skip_if_locked && st->list_lock.fd != -1)\n+\t\treturn 0;\n+\n \t/*\n \t * When we have cached stat information available then we use it to\n \t * verify whether the file has been rewritten.\n@@ -623,7 +630,7 @@ static int stack_uptodate(struct reftable_stack *st)\n \n int reftable_stack_reload(struct reftable_stack *st)\n {\n-\tint err = stack_uptodate(st);\n+\tint err = stack_uptodate(st, 1);\n \tif (err > 0)\n \t\treturn reftable_stack_reload_maybe_reuse(st, 1);\n \treturn err;\n@@ -693,7 +700,7 @@ static int reftable_stack_init_addition(struct reftable_addition *add,\n \t\t}\n \t}\n \n-\terr = stack_uptodate(st);\n+\terr = stack_uptodate(st, 0);\n \tif (err < 0)\n \t\tgoto done;\n \tif (err > 0) {\n@@ -1200,7 +1207,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t * we could check that relevant tables still exist. But for now it's\n \t * good enough to just abort.\n \t */\n-\terr = stack_uptodate(st);\n+\terr = stack_uptodate(st, 0);\n \tif (err < 0)\n \t\tgoto done;\n \tif (err > 0) {\n@@ -1319,7 +1326,7 @@ static int stack_compact_range(struct reftable_stack *st,\n \t * tables with our compacted version. If they don't, then we need to\n \t * abort.\n \t */\n-\terr = stack_uptodate(st);\n+\terr = stack_uptodate(st, 0);\n \tif (err < 0)\n \t\tgoto done;\n \tif (err > 0) {\n\n-- \n2.55.GIT\n\n"},{"id":"551161","messageId":"20260824225202.GA190620@coredump.intra.peff.net","threadId":"66211","inReplyTo":"20260824-740-optimize-reloading-the-reftable-stack-v2-4-9c9de2eb0af7@gmail.com","subject":"Re: [PATCH v2 4/4] reftable/stack: avoid reloading the stack when already locked","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-08-24T22:52:02Z","receivedAt":"2026-08-24T22:52:04Z","isPatch":true,"body":"On Mon, Aug 24, 2026 at 11:31:02AM +0200, Karthik Nayak wrote:\n\n> Benchmarking with a fixed, non-symbolic target OID in the 'refs/tags/'\n> namespace (since it triggers a stack reload when checking if reflog\n> exists for the given tag name), shows a consistent 15-20% improvement\n> with these patches:\n> \n>   refcount   master     patch     speedup\n>   --------   -------    -------   -------\n>   2,000       18.5 ms    16.6 ms   1.11x\n>   20,000     120.7 ms   102.8 ms   1.17x\n>   50,000     296.5 ms   247.1 ms   1.20x\n\nMuch nicer. :)\n\nI'm not very familiar with the reftable code, so you can take my review\nwith a huge grain of salt. But I think the idea is sound, and I didn't\nsee anything obviously wrong. Thanks for working on this.\n\n-Peff\n"},{"id":"551172","messageId":"ao1uqpCxFHlOyTV-@pks.im","threadId":"66211","inReplyTo":"20260824-740-optimize-reloading-the-reftable-stack-v2-3-9c9de2eb0af7@gmail.com","subject":"Re: [PATCH v2 3/4] reftable/stack: move list lock to `struct reftable_stack`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-25T10:30:02Z","receivedAt":"2026-08-25T10:30:16Z","isPatch":true,"body":"On Mon, Aug 24, 2026 at 11:31:01AM +0200, Karthik Nayak wrote:\n> diff --git a/reftable/stack.c b/reftable/stack.c\n> index 703548417c..c3d4deff29 100644\n> --- a/reftable/stack.c\n> +++ b/reftable/stack.c\n> @@ -628,10 +630,16 @@ int reftable_stack_reload(struct reftable_stack *st)\n>  }\n>  \n>  struct reftable_addition {\n> -\tstruct reftable_flock tables_list_lock;\n>  \tstruct reftable_stack *stack;\n>  \tstruct reftable_write_options opts;\n>  \n> +\t/*\n> +\t * While the list lock is acquired on the stack, we need to distinguish\n> +\t * which 'reftable_addition' is responsible for the lock. This avoids\n> +\t * clearing the lock of another 'reftable_addition'.\n> +\t */\n> +\tunsigned int locked : 1;\n> +\n>  \tchar **new_tables;\n>  \tsize_t new_tables_len, new_tables_cap;\n>  \tuint64_t next_update_index;\n\nThis feels somewhat fragile, as tracking state via a bit is very easy to\nget wrong. I would have preferred a solution where we didn't have to\nhave this field and instead a more direct proxy. But there's not that\nmany spots where we have to do this, so this may be okay-ish.\n\nOther than that this series looks good to me, thanks!\n\nPatrick\n"},{"id":"551269","messageId":"CAOLa=ZSbOFmsNUfhsEESsD5rBqUq0iX=Ls_TgfRkeo5FQpamtQ@mail.gmail.com","threadId":"66211","inReplyTo":"ao1uqpCxFHlOyTV-@pks.im","subject":"Re: [PATCH v2 3/4] reftable/stack: move list lock to `struct reftable_stack`","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-08-26T09:46:20Z","receivedAt":"2026-08-26T09:46:23Z","isPatch":true,"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> On Mon, Aug 24, 2026 at 11:31:01AM +0200, Karthik Nayak wrote:\n>> diff --git a/reftable/stack.c b/reftable/stack.c\n>> index 703548417c..c3d4deff29 100644\n>> --- a/reftable/stack.c\n>> +++ b/reftable/stack.c\n>> @@ -628,10 +630,16 @@ int reftable_stack_reload(struct reftable_stack *st)\n>>  }\n>>\n>>  struct reftable_addition {\n>> -\tstruct reftable_flock tables_list_lock;\n>>  \tstruct reftable_stack *stack;\n>>  \tstruct reftable_write_options opts;\n>>\n>> +\t/*\n>> +\t * While the list lock is acquired on the stack, we need to distinguish\n>> +\t * which 'reftable_addition' is responsible for the lock. This avoids\n>> +\t * clearing the lock of another 'reftable_addition'.\n>> +\t */\n>> +\tunsigned int locked : 1;\n>> +\n>>  \tchar **new_tables;\n>>  \tsize_t new_tables_len, new_tables_cap;\n>>  \tuint64_t next_update_index;\n>\n> This feels somewhat fragile, as tracking state via a bit is very easy to\n> get wrong. I would have preferred a solution where we didn't have to\n> have this field and instead a more direct proxy. But there's not that\n> many spots where we have to do this, so this may be okay-ish.\n>\n> Other than that this series looks good to me, thanks!\n>\n> Patrick\n\nI agree, I couldn't find another solution. The alternative being, we\nmove the bitfield to the 'reftable_stack', but that is more fragile\nsince there are more touch fields. So I think this is okay too.\n\nThanks\n"},{"id":"551270","messageId":"CAOLa=ZRanhvBragQEcsLostB=4YGdDT8js7x8Ck5nJvLpsRZOQ@mail.gmail.com","threadId":"66211","inReplyTo":"20260824225202.GA190620@coredump.intra.peff.net","subject":"Re: [PATCH v2 4/4] reftable/stack: avoid reloading the stack when already locked","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2026-08-26T09:47:06Z","receivedAt":"2026-08-26T09:47:12Z","isPatch":true,"body":"Jeff King <peff@peff.net> writes:\n\n> On Mon, Aug 24, 2026 at 11:31:02AM +0200, Karthik Nayak wrote:\n>\n>> Benchmarking with a fixed, non-symbolic target OID in the 'refs/tags/'\n>> namespace (since it triggers a stack reload when checking if reflog\n>> exists for the given tag name), shows a consistent 15-20% improvement\n>> with these patches:\n>>\n>>   refcount   master     patch     speedup\n>>   --------   -------    -------   -------\n>>   2,000       18.5 ms    16.6 ms   1.11x\n>>   20,000     120.7 ms   102.8 ms   1.17x\n>>   50,000     296.5 ms   247.1 ms   1.20x\n>\n> Much nicer. :)\n>\n> I'm not very familiar with the reftable code, so you can take my review\n> with a huge grain of salt. But I think the idea is sound, and I didn't\n> see anything obviously wrong. Thanks for working on this.\n>\n> -Peff\n\nThanks for the report and the review :)\n"}]}