{"thread":{"id":"62965","subject":"[PATCH 00/14] refs: batch refname availability checks","startedAt":"2025-02-17T15:50:26Z","lastAt":"2025-04-04T10:59:46Z","messageCount":169,"participants":["Patrick Steinhardt","Karthik Nayak","shejialuo","brian m. carlson","Justin Tobler","Jeff King","Junio C Hamano","Elijah Newren"],"isPatch":true,"patchVersion":1,"patchTotal":14},"messages":[{"id":"512527","messageId":"20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im","threadId":"62965","inReplyTo":null,"subject":"[PATCH 00/14] refs: batch refname availability checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-17T15:50:14Z","receivedAt":"2025-02-17T15:50:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series has been inspired by brian's report that the reftable\nbackend is significantly slower when writing many references compared to\nthe files backend. As explained in that thread, the underlying issue is\nthe design of tombstone references: when we first delete all references\nin a repository and then recreate them, we still have all the tombstones\nand thus we need to churn through all of them to figure out that they\nhave been deleted in the first place. The files backend does not have\nthis issue.\n\nI consider the benchmark itself to be kind of broken, as it stems from\nus deleting all refs and then recreating them. And if you pack refs in\nbetween then the \"reftable\" backend outperforms the \"files\" backend.\n\nBut there are a couple of opportunities here anyway. While we cannot\nmake the underlying issue of tombstones being less efficient go away,\nthis has prompted me to have a deeper look at where we spend all the\ntime. There are three ideas in this series:\n\n - git-update-ref(1) performs ambiguity checks for any full-size object\n ID, which triggers a lot of reads. This is somewhat pointless though\n given that the manpage explicitly points out that the command is\n about object IDs, even though it does know to parse refs. But being\n part of plumbing, emitting the warning here does not make a ton of\n sense, and favoring object IDs over references in these cases is the\n obvious thing to do anyway.\n\n - For each ref \"refs/heads/bar\", we need to verify that neither\n \"refs/heads\" nor \"refs\" exists. This was repeated for every refname,\n but because most refnames use common prefixes this made us re-check\n a lot of prefixes. This is addressed by using a `strset` of already\n checked prefixes.\n\n - For each ref \"refs/heads/bar\", we need to verify that no ref\n \"refs/heads/bar/*\" exists. We always created a new ref iterator for\n this check, which requires us to discard all internal state and then\n recreate it. The reftable library has already been refactored though\n to have reseekable iterators, so we backfill this functionality to\n all the other iterators and then reuse the iterator.\n\nWith the (somewhat broken) benchmark we small speedup with the \"files\"\nbackend:\n\n Benchmark 1: update-ref (refformat = files, revision = master)\n Time (mean ± σ): 233.8 ms ± 1.4 ms [User: 81.2 ms, System: 151.5 ms]\n Range (min … max): 232.2 ms … 236.0 ms 10 runs\n\n Benchmark 2: update-ref (refformat = files, revision = HEAD)\n Time (mean ± σ): 192.3 ms ± 1.5 ms [User: 67.2 ms, System: 124.0 ms]\n Range (min … max): 190.1 ms … 194.2 ms 10 runs\n\n Summary\n update-ref (refformat = files, revision = HEAD) ran\n 1.22 ± 0.01 times faster than update-ref (refformat = files, revision = master)\n\nAnd a huge speedup with the \"reftable\" backend:\n\n Benchmark 1: update-ref (refformat = reftable, revision = master)\n Time (mean ± σ): 16.852 s ± 0.061 s [User: 16.754 s, System: 0.059 s]\n Range (min … max): 16.785 s … 16.982 s 10 runs\n\n Benchmark 2: update-ref (refformat = reftable, revision = HEAD)\n Time (mean ± σ): 2.230 s ± 0.009 s [User: 2.192 s, System: 0.029 s]\n Range (min … max): 2.215 s … 2.244 s 10 runs\n\n Summary\n update-ref (refformat = reftable, revision = HEAD) ran\n 7.56 ± 0.04 times faster than update-ref (refformat = reftable, revision = master)\n\nWe're still not up to speed with the \"files\" backend, but considerably\nbetter. Given that this is an extreme edge case and not reflective of\nthe general case I'm okay with this result for now.\n\nBut more importantly, this refactoring also has a positive effect when\nupdating references in a repository with preexisting refs, which I\nconsider to be the more realistic scenario. The following benchmark\ncreates 10k refs with 100k preexisting refs.\n\nWith the \"files\" backend we see a modest improvement:\n\n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = master)\n Time (mean ± σ): 470.1 ms ± 5.4 ms [User: 104.5 ms, System: 363.1 ms]\n Range (min … max): 465.7 ms … 484.3 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 407.8 ms ± 5.4 ms [User: 66.0 ms, System: 340.0 ms]\n Range (min … max): 399.9 ms … 417.6 ms 10 runs\n\n Summary\n update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.15 ± 0.02 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = master)\n\nBut with the \"reftable\" backend we see an almost 5x improvement, where\nit's now ~15x faster than the \"files\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = master)\n Time (mean ± σ): 153.9 ms ± 2.0 ms [User: 96.5 ms, System: 56.6 ms]\n Range (min … max): 150.5 ms … 158.4 ms 18 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 32.2 ms ± 1.2 ms [User: 27.6 ms, System: 4.3 ms]\n Range (min … max): 29.8 ms … 38.6 ms 71 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 4.78 ± 0.19 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = master)\n\nThe series is structured as follows:\n\n - Patches 1 to 4 implement the logic to skip ambiguity checks in\n git-update-ref(1).\n\n - Patch 5 and 6 introduce batched checks.\n\n - Patch 7 deduplicates the ref prefix checks.\n\n - Patch 8 to 14 implement the infrastructure to reseek iterators.\n\n - Patch 15 starts to reuse iterators for nested ref checks.\n\nThanks!\n\nPatrick\n\n[1]: \n\n---\nPatrick Steinhardt (14):\n object-name: introduce `repo_get_oid_with_flags()`\n object-name: allow skipping ambiguity checks in `get_oid()` family\n builtin/update-ref: skip ambiguity checks when parsing object IDs\n refs: introduce function to batch refname availability checks\n refs/reftable: start using `refs_verify_refnames_available()`\n refs: stop re-verifying common prefixes for availability\n refs/iterator: separate lifecycle from iteration\n refs/iterator: provide infrastructure to re-seek iterators\n refs/iterator: implement seeking for merged iterators\n refs/iterator: implement seeking for reftable iterators\n refs/iterator: implement seeking for ref-cache iterators\n refs/iterator: implement seeking for `packed-ref` iterators\n refs/iterator: implement seeking for \"files\" iterators\n refs: reuse iterators when determining refname availability\n\n builtin/clone.c | 2 +\n builtin/update-ref.c | 12 ++-\n dir-iterator.c | 24 +++---\n dir-iterator.h | 13 +--\n hash.h | 1 +\n object-name.c | 18 +++--\n object-name.h | 6 ++\n refs.c | 186 ++++++++++++++++++++++++++-----------------\n refs.h | 12 +++\n refs/debug.c | 20 +++--\n refs/files-backend.c | 52 ++++++------\n refs/iterator.c | 145 +++++++++++++++++----------------\n refs/packed-backend.c | 89 ++++++++++++---------\n refs/ref-cache.c | 83 +++++++++++--------\n refs/refs-internal.h | 54 ++++++++-----\n refs/reftable-backend.c | 85 +++++++++++---------\n t/helper/test-dir-iterator.c | 1 +\n 17 files changed, 471 insertions(+), 332 deletions(-)\n\n\n---\nbase-commit: e2067b49ecaef9b7f51a17ce251f9207f72ef52d\nchange-id: 20250217-pks-update-ref-optimization-15c795e66e2b\n\n"},{"id":"512528","messageId":"20250217-pks-update-ref-optimization-v1-1-a2b6d87a24af@pks.im","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im","subject":"[PATCH 01/14] object-name: introduce `repo_get_oid_with_flags()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-17T15:50:15Z","receivedAt":"2025-02-17T15:50:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new function `repo_get_oid_with_flags()`. This function\nbehaves the same as `repo_get_oid()`, except that it takes an extra\n`flags` parameter that it ends up passing to `get_oid_with_context()`.\n\nThis function will be used in a subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-name.c | 14 ++++++++------\n object-name.h | 6 ++++++\n 2 files changed, 14 insertions(+), 6 deletions(-)\n\ndiff --git a/object-name.c b/object-name.c\nindex 945d5bdef25..bc0265ad2a1 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -1794,18 +1794,20 @@ void object_context_release(struct object_context *ctx)\n \tstrbuf_release(&ctx->symlink_path);\n }\n \n-/*\n- * This is like \"get_oid_basic()\", except it allows \"object ID expressions\",\n- * notably \"xyz^\" for \"parent of xyz\"\n- */\n-int repo_get_oid(struct repository *r, const char *name, struct object_id *oid)\n+int repo_get_oid_with_flags(struct repository *r, const char *name, struct object_id *oid,\n+\t\t\t unsigned flags)\n {\n \tstruct object_context unused;\n-\tint ret = get_oid_with_context(r, name, 0, oid, &unused);\n+\tint ret = get_oid_with_context(r, name, flags, oid, &unused);\n \tobject_context_release(&unused);\n \treturn ret;\n }\n \n+int repo_get_oid(struct repository *r, const char *name, struct object_id *oid)\n+{\n+\treturn repo_get_oid_with_flags(r, name, oid, 0);\n+}\n+\n /*\n * This returns a non-zero value if the string (built using printf\n * format and the given arguments) is not a valid object.\ndiff --git a/object-name.h b/object-name.h\nindex 8dba4a47a47..fb5a97b2c8e 100644\n--- a/object-name.h\n+++ b/object-name.h\n@@ -51,6 +51,12 @@ void strbuf_repo_add_unique_abbrev(struct strbuf *sb, struct repository *repo,\n void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n \t\t\t int abbrev_len);\n \n+/*\n+ * This is like \"get_oid_basic()\", except it allows \"object ID expressions\",\n+ * notably \"xyz^\" for \"parent of xyz\". Accepts GET_OID_* flags.\n+ */\n+int repo_get_oid_with_flags(struct repository *r, const char *str, struct object_id *oid,\n+\t\t\t unsigned flags);\n int repo_get_oid(struct repository *r, const char *str, struct object_id *oid);\n __attribute__((format (printf, 2, 3)))\n int get_oidf(struct object_id *oid, const char *fmt, ...);\n\n-- \n2.48.1.666.gff9fcf71b7.dirty\n\n"},{"id":"512529","messageId":"20250217-pks-update-ref-optimization-v1-2-a2b6d87a24af@pks.im","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im","subject":"[PATCH 02/14] object-name: allow skipping ambiguity checks in `get_oid()` family","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-17T15:50:16Z","receivedAt":"2025-02-17T15:50:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When reading an object ID via `get_oid_basic()` or any of its related\nfunctions we perform a check whether the object ID is ambiguous, which\ncan be the case when a reference with the same name exists. While the\ncheck is generally helpful, there are cases where it only adds to the\nruntime overhead without providing much of a benefit.\n\nAdd a new flag that allows us to disable the check. The flag will be\nused in a subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n hash.h | 1 +\n object-name.c | 4 +++-\n 2 files changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/hash.h b/hash.h\nindex 4367acfec50..aeb705d6850 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -204,6 +204,7 @@ struct object_id {\n #define GET_OID_ONLY_TO_DIE 04000\n #define GET_OID_REQUIRE_PATH 010000\n #define GET_OID_HASH_ANY 020000\n+#define GET_OID_HASH_SKIP_AMBIGUITY_CHECK 040000\n \n #define GET_OID_DISAMBIGUATORS \\\n \t(GET_OID_COMMIT | GET_OID_COMMITTISH | \\\ndiff --git a/object-name.c b/object-name.c\nindex bc0265ad2a1..0b23a65d2ed 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -961,7 +961,9 @@ static int get_oid_basic(struct repository *r, const char *str, int len,\n \tint fatal = !(flags & GET_OID_QUIETLY);\n \n \tif (len == r->hash_algo->hexsz && !get_oid_hex(str, oid)) {\n-\t\tif (repo_settings_get_warn_ambiguous_refs(r) && warn_on_object_refname_ambiguity) {\n+\t\tif (!(flags & GET_OID_HASH_SKIP_AMBIGUITY_CHECK) &&\n+\t\t repo_settings_get_warn_ambiguous_refs(r) &&\n+\t\t warn_on_object_refname_ambiguity) {\n \t\t\trefs_found = repo_dwim_ref(r, str, len, &tmp_oid, &real_ref, 0);\n \t\t\tif (refs_found > 0) {\n \t\t\t\twarning(warn_msg, len, str);\n\n-- \n2.48.1.666.gff9fcf71b7.dirty\n\n"},{"id":"512530","messageId":"20250217-pks-update-ref-optimization-v1-3-a2b6d87a24af@pks.im","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im","subject":"[PATCH 03/14] builtin/update-ref: skip ambiguity checks when parsing object IDs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-17T15:50:17Z","receivedAt":"2025-02-17T15:50:29Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Most of the commands in git-update-ref(1) accept an old and/or new\nobject ID to update a specific reference to. These object IDs get parsed\nvia `repo_get_oid()`, which not only handles plain object IDs, but also\nthose that have a suffix like \"~\" or \"^2\". More surprisingly though, it\neven knows to resolve references, despite the fact that its manpage does\nnot mention this fact even once.\n\nOne consequence of this is that we also check for ambiguous references:\nwhen parsing a full object ID where the DWIM mechanism would also cause\nus to resolve it as a branch, we'd end up printing a warning. While this\ncheck makes sense to have in general, it is arguably less useful in the\ncontext of git-update-ref(1). This is out of two reasons:\n\n - The manpage is explicitly structured around object IDs. So if we see\n a fully blown object ID, the intent should be quite clear in\n general.\n\n - The command is part of our plumbing layer and not a tool that users\n would generally use in interactive workflows. As such, the warning\n will likely not be visible to anybody in the first place.\n\nFurthermore, this check can be quite expensive when updating lots of\nreferences via `--stdin`, because we try to read multiple references per\nobject ID that we parse according to the DWIM rules. This effect can be\nseen both with the \"files\" and \"reftable\" backend.\n\nDisable the warning in git-update-ref(1), which provides a significant\nspeedup with both backends. The following benchmark creates 10000 new\nreferences with a 100000 preexisting refs with the \"files\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 467.3 ms ± 5.1 ms [User: 100.0 ms, System: 365.1 ms]\n Range (min … max): 461.9 ms … 479.3 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 394.1 ms ± 5.8 ms [User: 63.3 ms, System: 327.6 ms]\n Range (min … max): 384.9 ms … 405.7 ms 10 runs\n\n Summary\n update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.19 ± 0.02 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n\nAnd with the \"reftable\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 146.9 ms ± 2.2 ms [User: 90.4 ms, System: 56.0 ms]\n Range (min … max): 142.7 ms … 150.8 ms 19 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 63.2 ms ± 1.1 ms [User: 41.0 ms, System: 21.8 ms]\n Range (min … max): 61.1 ms … 66.6 ms 41 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 2.32 ± 0.05 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n\nNote that the absolute improvement with both backends is roughly in the\nsame ballpark, but the relative improvement for the \"reftable\" backend\nis more significant because writing the new table to disk is faster in\nthe first place.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/update-ref.c | 12 ++++++++----\n 1 file changed, 8 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/update-ref.c b/builtin/update-ref.c\nindex 4d35bdc4b4b..ac330748244 100644\n--- a/builtin/update-ref.c\n+++ b/builtin/update-ref.c\n@@ -179,7 +179,8 @@ static int parse_next_oid(const char **next, const char *end,\n \t\t(*next)++;\n \t\t*next = parse_arg(*next, &arg);\n \t\tif (arg.len) {\n-\t\t\tif (repo_get_oid(the_repository, arg.buf, oid))\n+\t\t\tif (repo_get_oid_with_flags(the_repository, arg.buf, oid,\n+\t\t\t\t\t\t GET_OID_HASH_SKIP_AMBIGUITY_CHECK))\n \t\t\t\tgoto invalid;\n \t\t} else {\n \t\t\t/* Without -z, an empty value means all zeros: */\n@@ -197,7 +198,8 @@ static int parse_next_oid(const char **next, const char *end,\n \t\t*next += arg.len;\n \n \t\tif (arg.len) {\n-\t\t\tif (repo_get_oid(the_repository, arg.buf, oid))\n+\t\t\tif (repo_get_oid_with_flags(the_repository, arg.buf, oid,\n+\t\t\t\t\t\t GET_OID_HASH_SKIP_AMBIGUITY_CHECK))\n \t\t\t\tgoto invalid;\n \t\t} else if (flags & PARSE_SHA1_ALLOW_EMPTY) {\n \t\t\t/* With -z, treat an empty value as all zeros: */\n@@ -772,7 +774,8 @@ int cmd_update_ref(int argc,\n \t\trefname = argv[0];\n \t\tvalue = argv[1];\n \t\toldval = argv[2];\n-\t\tif (repo_get_oid(the_repository, value, &oid))\n+\t\tif (repo_get_oid_with_flags(the_repository, value, &oid,\n+\t\t\t\t\t GET_OID_HASH_SKIP_AMBIGUITY_CHECK))\n \t\t\tdie(\"%s: not a valid SHA1\", value);\n \t}\n \n@@ -783,7 +786,8 @@ int cmd_update_ref(int argc,\n \t\t\t * must not already exist:\n \t\t\t */\n \t\t\toidclr(&oldoid, the_repository->hash_algo);\n-\t\telse if (repo_get_oid(the_repository, oldval, &oldoid))\n+\t\telse if (repo_get_oid_with_flags(the_repository, oldval, &oldoid,\n+\t\t\t\t\t\t GET_OID_HASH_SKIP_AMBIGUITY_CHECK))\n \t\t\tdie(\"%s: not a valid old SHA1\", oldval);\n \t}\n \n\n-- \n2.48.1.666.gff9fcf71b7.dirty\n\n"},{"id":"512531","messageId":"20250217-pks-update-ref-optimization-v1-4-a2b6d87a24af@pks.im","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im","subject":"[PATCH 04/14] refs: introduce function to batch refname availability checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-17T15:50:18Z","receivedAt":"2025-02-17T15:50:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `refs_verify_refname_available()` functions checks whether a\nreference update can be committed or whether it would conflict with\neither a prefix or suffix thereof. This function needs to be called once\nper reference that one wants to check, which requires us to redo a\ncouple of checks every time the function is called.\n\nIntroduce a new function `refs_verify_refnames_available()` that does\nthe same, but for a list of references. For now, the new function uses\nthe exact same implementation, except that we loop through all refnames\nprovided by the caller. This will be tuned in subsequent commits.\n\nThe existing `refs_verify_refname_available()` function is reimplemented\non top of the new function. As such, the diff is best viewed with the\n`--ignore-space-change option`.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs.c | 169 +++++++++++++++++++++++++++++++++++++----------------------------\n refs.h | 12 +++++\n 2 files changed, 109 insertions(+), 72 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex f4094a326a9..5a9b0f2fa1e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2467,19 +2467,15 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \treturn ret;\n }\n \n-int refs_verify_refname_available(struct ref_store *refs,\n-\t\t\t\t const char *refname,\n-\t\t\t\t const struct string_list *extras,\n-\t\t\t\t const struct string_list *skip,\n-\t\t\t\t unsigned int initial_transaction,\n-\t\t\t\t struct strbuf *err)\n+int refs_verify_refnames_available(struct ref_store *refs,\n+\t\t\t\t const struct string_list *refnames,\n+\t\t\t\t const struct string_list *extras,\n+\t\t\t\t const struct string_list *skip,\n+\t\t\t\t unsigned int initial_transaction,\n+\t\t\t\t struct strbuf *err)\n {\n-\tconst char *slash;\n-\tconst char *extra_refname;\n \tstruct strbuf dirname = STRBUF_INIT;\n \tstruct strbuf referent = STRBUF_INIT;\n-\tstruct object_id oid;\n-\tunsigned int type;\n \tint ret = -1;\n \n \t/*\n@@ -2489,79 +2485,91 @@ int refs_verify_refname_available(struct ref_store *refs,\n \n \tassert(err);\n \n-\tstrbuf_grow(&dirname, strlen(refname) + 1);\n-\tfor (slash = strchr(refname, '/'); slash; slash = strchr(slash + 1, '/')) {\n-\t\t/*\n-\t\t * Just saying \"Is a directory\" when we e.g. can't\n-\t\t * lock some multi-level ref isn't very informative,\n-\t\t * the user won't be told *what* is a directory, so\n-\t\t * let's not use strerror() below.\n-\t\t */\n-\t\tint ignore_errno;\n-\t\t/* Expand dirname to the new prefix, not including the trailing slash: */\n-\t\tstrbuf_add(&dirname, refname + dirname.len, slash - refname - dirname.len);\n+\tfor (size_t i = 0; i < refnames->nr; i++) {\n+\t\tconst char *refname = refnames->items[i].string;\n+\t\tconst char *extra_refname;\n+\t\tstruct object_id oid;\n+\t\tunsigned int type;\n+\t\tconst char *slash;\n+\n+\t\tstrbuf_reset(&dirname);\n+\n+\t\tfor (slash = strchr(refname, '/'); slash; slash = strchr(slash + 1, '/')) {\n+\t\t\t/*\n+\t\t\t * Just saying \"Is a directory\" when we e.g. can't\n+\t\t\t * lock some multi-level ref isn't very informative,\n+\t\t\t * the user won't be told *what* is a directory, so\n+\t\t\t * let's not use strerror() below.\n+\t\t\t */\n+\t\t\tint ignore_errno;\n+\n+\t\t\t/* Expand dirname to the new prefix, not including the trailing slash: */\n+\t\t\tstrbuf_add(&dirname, refname + dirname.len, slash - refname - dirname.len);\n+\n+\t\t\t/*\n+\t\t\t * We are still at a leading dir of the refname (e.g.,\n+\t\t\t * \"refs/foo\"; if there is a reference with that name,\n+\t\t\t * it is a conflict, *unless* it is in skip.\n+\t\t\t */\n+\t\t\tif (skip && string_list_has_string(skip, dirname.buf))\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (!initial_transaction &&\n+\t\t\t !refs_read_raw_ref(refs, dirname.buf, &oid, &referent,\n+\t\t\t\t\t &type, &ignore_errno)) {\n+\t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n+\t\t\t\t\t dirname.buf, refname);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\n+\t\t\tif (extras && string_list_has_string(extras, dirname.buf)) {\n+\t\t\t\tstrbuf_addf(err, _(\"cannot process '%s' and '%s' at the same time\"),\n+\t\t\t\t\t refname, dirname.buf);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\t\t}\n \n \t\t/*\n-\t\t * We are still at a leading dir of the refname (e.g.,\n-\t\t * \"refs/foo\"; if there is a reference with that name,\n-\t\t * it is a conflict, *unless* it is in skip.\n+\t\t * We are at the leaf of our refname (e.g., \"refs/foo/bar\").\n+\t\t * There is no point in searching for a reference with that\n+\t\t * name, because a refname isn't considered to conflict with\n+\t\t * itself. But we still need to check for references whose\n+\t\t * names are in the \"refs/foo/bar/\" namespace, because they\n+\t\t * *do* conflict.\n \t\t */\n-\t\tif (skip && string_list_has_string(skip, dirname.buf))\n-\t\t\tcontinue;\n+\t\tstrbuf_addstr(&dirname, refname + dirname.len);\n+\t\tstrbuf_addch(&dirname, '/');\n+\n+\t\tif (!initial_transaction) {\n+\t\t\tstruct ref_iterator *iter;\n+\t\t\tint ok;\n+\n+\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n+\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n+\t\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n+\t\t\t\tif (skip &&\n+\t\t\t\t string_list_has_string(skip, iter->refname))\n+\t\t\t\t\tcontinue;\n+\n+\t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n+\t\t\t\t\t iter->refname, refname);\n+\t\t\t\tref_iterator_abort(iter);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \n-\t\tif (!initial_transaction &&\n-\t\t !refs_read_raw_ref(refs, dirname.buf, &oid, &referent,\n-\t\t\t\t &type, &ignore_errno)) {\n-\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n-\t\t\t\t dirname.buf, refname);\n-\t\t\tgoto cleanup;\n+\t\t\tif (ok != ITER_DONE)\n+\t\t\t\tBUG(\"error while iterating over references\");\n \t\t}\n \n-\t\tif (extras && string_list_has_string(extras, dirname.buf)) {\n+\t\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n+\t\tif (extra_refname) {\n \t\t\tstrbuf_addf(err, _(\"cannot process '%s' and '%s' at the same time\"),\n-\t\t\t\t refname, dirname.buf);\n+\t\t\t\t refname, extra_refname);\n \t\t\tgoto cleanup;\n \t\t}\n \t}\n \n-\t/*\n-\t * We are at the leaf of our refname (e.g., \"refs/foo/bar\").\n-\t * There is no point in searching for a reference with that\n-\t * name, because a refname isn't considered to conflict with\n-\t * itself. But we still need to check for references whose\n-\t * names are in the \"refs/foo/bar/\" namespace, because they\n-\t * *do* conflict.\n-\t */\n-\tstrbuf_addstr(&dirname, refname + dirname.len);\n-\tstrbuf_addch(&dirname, '/');\n-\n-\tif (!initial_transaction) {\n-\t\tstruct ref_iterator *iter;\n-\t\tint ok;\n-\n-\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n-\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n-\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n-\t\t\tif (skip &&\n-\t\t\t string_list_has_string(skip, iter->refname))\n-\t\t\t\tcontinue;\n-\n-\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n-\t\t\t\t iter->refname, refname);\n-\t\t\tref_iterator_abort(iter);\n-\t\t\tgoto cleanup;\n-\t\t}\n-\n-\t\tif (ok != ITER_DONE)\n-\t\t\tBUG(\"error while iterating over references\");\n-\t}\n-\n-\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n-\tif (extra_refname)\n-\t\tstrbuf_addf(err, _(\"cannot process '%s' and '%s' at the same time\"),\n-\t\t\t refname, extra_refname);\n-\telse\n-\t\tret = 0;\n+\tret = 0;\n \n cleanup:\n \tstrbuf_release(&referent);\n@@ -2569,6 +2577,23 @@ int refs_verify_refname_available(struct ref_store *refs,\n \treturn ret;\n }\n \n+int refs_verify_refname_available(struct ref_store *refs,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const struct string_list *extras,\n+\t\t\t\t const struct string_list *skip,\n+\t\t\t\t unsigned int initial_transaction,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct string_list_item item = { .string = (char *) refname };\n+\tstruct string_list refnames = {\n+\t\t.items = &item,\n+\t\t.nr = 1,\n+\t};\n+\n+\treturn refs_verify_refnames_available(refs, &refnames, extras, skip,\n+\t\t\t\t\t initial_transaction, err);\n+}\n+\n struct do_for_each_reflog_help {\n \teach_reflog_fn *fn;\n \tvoid *cb_data;\ndiff --git a/refs.h b/refs.h\nindex a0cdd99250e..185aed5a461 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -124,6 +124,18 @@ int refs_verify_refname_available(struct ref_store *refs,\n \t\t\t\t unsigned int initial_transaction,\n \t\t\t\t struct strbuf *err);\n \n+/*\n+ * Same as `refs_verify_refname_available()`, but checking for a list of\n+ * refnames instead of only a single item. This is more efficient in the case\n+ * where one needs to check multiple refnames.\n+ */\n+int refs_verify_refnames_available(struct ref_store *refs,\n+\t\t\t\t const struct string_list *refnames,\n+\t\t\t\t const struct string_list *extras,\n+\t\t\t\t const struct string_list *skip,\n+\t\t\t\t unsigned int initial_transaction,\n+\t\t\t\t struct strbuf *err);\n+\n int refs_ref_exists(struct ref_store *refs, const char *refname);\n \n int should_autocreate_reflog(enum log_refs_config log_all_ref_updates,\n\n-- \n2.48.1.666.gff9fcf71b7.dirty\n\n"},{"id":"512532","messageId":"20250217-pks-update-ref-optimization-v1-5-a2b6d87a24af@pks.im","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im","subject":"[PATCH 05/14] refs/reftable: start using `refs_verify_refnames_available()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-17T15:50:19Z","receivedAt":"2025-02-17T15:50:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor the \"reftable\" backend to batch the availability check for\nrefnames. This does not yet have an effect on performance as we\nessentially still call `refs_verify_refname_available()` in a loop, but\nthis will change in subsequent commits.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 16 ++++++++++------\n 1 file changed, 10 insertions(+), 6 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex d39a14c5a46..2a90e7cb391 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1069,6 +1069,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE|REF_STORE_MAIN, \"ref_transaction_prepare\");\n \tstruct strbuf referent = STRBUF_INIT, head_referent = STRBUF_INIT;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n \tstruct reftable_transaction_data *tx_data = NULL;\n \tstruct reftable_backend *be;\n \tstruct object_id head_oid;\n@@ -1224,12 +1225,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t\t * can output a proper error message instead of failing\n \t\t\t * at a later point.\n \t\t\t */\n-\t\t\tret = refs_verify_refname_available(ref_store, u->refname,\n-\t\t\t\t\t\t\t &affected_refnames, NULL,\n-\t\t\t\t\t\t\t transaction->flags & REF_TRANSACTION_FLAG_INITIAL,\n-\t\t\t\t\t\t\t err);\n-\t\t\tif (ret < 0)\n-\t\t\t\tgoto done;\n+\t\t\tstring_list_append(&refnames_to_check, u->refname);\n \n \t\t\t/*\n \t\t\t * There is no need to write the reference deletion\n@@ -1379,6 +1375,13 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t}\n \t}\n \n+\tstring_list_sort(&refnames_to_check);\n+\tret = refs_verify_refnames_available(ref_store, &refnames_to_check, &affected_refnames, NULL,\n+\t\t\t\t\t transaction->flags & REF_TRANSACTION_FLAG_INITIAL,\n+\t\t\t\t\t err);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \ttransaction->backend_data = tx_data;\n \ttransaction->state = REF_TRANSACTION_PREPARED;\n \n@@ -1394,6 +1397,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \tstring_list_clear(&affected_refnames, 0);\n \tstrbuf_release(&referent);\n \tstrbuf_release(&head_referent);\n+\tstring_list_clear(&refnames_to_check, 0);\n \n \treturn ret;\n }\n\n-- \n2.48.1.666.gff9fcf71b7.dirty\n\n"},{"id":"512533","messageId":"20250217-pks-update-ref-optimization-v1-6-a2b6d87a24af@pks.im","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im","subject":"[PATCH 06/14] refs: stop re-verifying common prefixes for availability","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-17T15:50:20Z","receivedAt":"2025-02-17T15:50:33Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"One of the checks done by `refs_verify_refnames_available()` is whether\nany of the prefixes of a reference already exists. For example, given a\nreference \"refs/heads/main\", we'd check whether \"refs/heads\" or \"refs\"\nalready exist, and if so we'd abort the transaction.\n\nWhen updating multiple references at once, this check is performed for\neach of the references individually. Consequently, because references\ntend to have common prefixes like \"refs/heads/\" or refs/tags/\", we\nevaluate the availability of these prefixes repeatedly. Naturally this\nis a waste of compute, as the availability of those prefixes should in\ngeneral not change in the middle of a transaction. And if it would,\nbackends would notice at a later point in time.\n\nOptimize this pattern by storing prefixes in a `strset` so that we can\ntrivially track those prefixes that we have already checked. This leads\nto a significant speedup when creating many references that all share a\ncommon prefix:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 63.1 ms ± 1.8 ms [User: 41.0 ms, System: 21.6 ms]\n Range (min … max): 60.6 ms … 69.5 ms 38 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 40.0 ms ± 1.3 ms [User: 29.3 ms, System: 10.3 ms]\n Range (min … max): 38.1 ms … 47.3 ms 61 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.58 ± 0.07 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n\nNote that the same speedup cannot be observed for the \"files\" backend\nbecause it still performs availability check per reference.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs.c | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/refs.c b/refs.c\nindex 5a9b0f2fa1e..eaf41421f50 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2476,6 +2476,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n {\n \tstruct strbuf dirname = STRBUF_INIT;\n \tstruct strbuf referent = STRBUF_INIT;\n+\tstruct strset dirnames;\n \tint ret = -1;\n \n \t/*\n@@ -2485,6 +2486,8 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \n \tassert(err);\n \n+\tstrset_init(&dirnames);\n+\n \tfor (size_t i = 0; i < refnames->nr; i++) {\n \t\tconst char *refname = refnames->items[i].string;\n \t\tconst char *extra_refname;\n@@ -2514,6 +2517,14 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \t\t\tif (skip && string_list_has_string(skip, dirname.buf))\n \t\t\t\tcontinue;\n \n+\t\t\t/*\n+\t\t\t * If we've already seen the directory we don't need to\n+\t\t\t * process it again. Skip it to avoid checking checking\n+\t\t\t * common prefixes like \"refs/heads/\" repeatedly.\n+\t\t\t */\n+\t\t\tif (!strset_add(&dirnames, dirname.buf))\n+\t\t\t\tcontinue;\n+\n \t\t\tif (!initial_transaction &&\n \t\t\t !refs_read_raw_ref(refs, dirname.buf, &oid, &referent,\n \t\t\t\t\t &type, &ignore_errno)) {\n@@ -2574,6 +2585,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n cleanup:\n \tstrbuf_release(&referent);\n \tstrbuf_release(&dirname);\n+\tstrset_clear(&dirnames);\n \treturn ret;\n }\n \n\n-- \n2.48.1.666.gff9fcf71b7.dirty\n\n"},{"id":"512534","messageId":"20250217-pks-update-ref-optimization-v1-7-a2b6d87a24af@pks.im","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im","subject":"[PATCH 07/14] refs/iterator: separate lifecycle from iteration","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-17T15:50:21Z","receivedAt":"2025-02-17T15:50:33Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The ref and reflog iterators have their lifecycle attached to iteration:\nonce the iterator reaches its end, it is automatically released and the\ncaller doesn't have to care about that anymore. When the iterator should\nbe released before it has been exhausted, callers must explicitly abort\nthe iterator via `ref_iterator_abort()`.\n\nThis lifecycle is somewhat unusual in the Git codebase and creates two\nproblems:\n\n - Callsites need to be very careful about when exactly they call\n `ref_iterator_abort()`, as calling the function is only valid when\n the iterator itself still is. This leads to somewhat awkward calling\n patterns in some situations.\n\n - It is impossible to reuse iterators and re-seek them to a different\n prefix. This feature isn't supported by any iterator implementation\n except for the reftable iterators anyway, but if it was implemented\n it would allow us to optimize cases where we need to search for\n specific references repeatedly by reusing internal state.\n\nDetangle the lifecycle from iteration so that we don't deallocate the\niterator anymore once it is exhausted. Instead, callers are now expected\nto always call a newly introduce `ref_iterator_free()` function that\ndeallocates the iterator and its internal state.\n\nWhile at it, drop the return value of `ref_iterator_abort()`, which\nwasn't really required by any of the iterator implementations anyway.\nFurthermore, stop calling `base_ref_iterator_free()` in any of the\nbackends, but instead call it in `ref_iterator_free()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/clone.c | 2 +\n dir-iterator.c | 24 +++++------\n dir-iterator.h | 13 ++----\n refs.c | 7 +++-\n refs/debug.c | 9 ++---\n refs/files-backend.c | 36 +++++------------\n refs/iterator.c | 95 ++++++++++++++------------------------------\n refs/packed-backend.c | 27 ++++++-------\n refs/ref-cache.c | 9 ++---\n refs/refs-internal.h | 31 +++++----------\n refs/reftable-backend.c | 34 ++++------------\n t/helper/test-dir-iterator.c | 1 +\n 12 files changed, 99 insertions(+), 189 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex fd001d800c6..ac3e84b2b18 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -426,6 +426,8 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\tstrbuf_setlen(src, src_len);\n \t\tdie(_(\"failed to iterate over '%s'\"), src->buf);\n \t}\n+\n+\tdir_iterator_free(iter);\n }\n \n static void clone_local(const char *src_repo, const char *dest_repo)\ndiff --git a/dir-iterator.c b/dir-iterator.c\nindex de619846f29..857e1d9bdaf 100644\n--- a/dir-iterator.c\n+++ b/dir-iterator.c\n@@ -193,9 +193,9 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \n \tif (S_ISDIR(iter->base.st.st_mode) && push_level(iter)) {\n \t\tif (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC)\n-\t\t\tgoto error_out;\n+\t\t\treturn ITER_ERROR;\n \t\tif (iter->levels_nr == 0)\n-\t\t\tgoto error_out;\n+\t\t\treturn ITER_ERROR;\n \t}\n \n \t/* Loop until we find an entry that we can give back to the caller. */\n@@ -211,11 +211,11 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\tint ret = next_directory_entry(level->dir, iter->base.path.buf, &de);\n \t\t\tif (ret < 0) {\n \t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n-\t\t\t\t\tgoto error_out;\n+\t\t\t\t\treturn ITER_ERROR;\n \t\t\t\tcontinue;\n \t\t\t} else if (ret > 0) {\n \t\t\t\tif (pop_level(iter) == 0)\n-\t\t\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\t\t\t\treturn ITER_DONE;\n \t\t\t\tcontinue;\n \t\t\t}\n \n@@ -223,7 +223,7 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t} else {\n \t\t\tif (level->entries_idx >= level->entries.nr) {\n \t\t\t\tif (pop_level(iter) == 0)\n-\t\t\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\t\t\t\treturn ITER_DONE;\n \t\t\t\tcontinue;\n \t\t\t}\n \n@@ -232,22 +232,21 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \n \t\tif (prepare_next_entry_data(iter, name)) {\n \t\t\tif (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC)\n-\t\t\t\tgoto error_out;\n+\t\t\t\treturn ITER_ERROR;\n \t\t\tcontinue;\n \t\t}\n \n \t\treturn ITER_OK;\n \t}\n-\n-error_out:\n-\tdir_iterator_abort(dir_iterator);\n-\treturn ITER_ERROR;\n }\n \n-int dir_iterator_abort(struct dir_iterator *dir_iterator)\n+void dir_iterator_free(struct dir_iterator *dir_iterator)\n {\n \tstruct dir_iterator_int *iter = (struct dir_iterator_int *)dir_iterator;\n \n+\tif (!iter)\n+\t\treturn;\n+\n \tfor (; iter->levels_nr; iter->levels_nr--) {\n \t\tstruct dir_iterator_level *level =\n \t\t\t&iter->levels[iter->levels_nr - 1];\n@@ -266,7 +265,6 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator)\n \tfree(iter->levels);\n \tstrbuf_release(&iter->base.path);\n \tfree(iter);\n-\treturn ITER_DONE;\n }\n \n struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags)\n@@ -301,7 +299,7 @@ struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags)\n \treturn dir_iterator;\n \n error_out:\n-\tdir_iterator_abort(dir_iterator);\n+\tdir_iterator_free(dir_iterator);\n \terrno = saved_errno;\n \treturn NULL;\n }\ndiff --git a/dir-iterator.h b/dir-iterator.h\nindex 6d438809b6e..01f51f6bac1 100644\n--- a/dir-iterator.h\n+++ b/dir-iterator.h\n@@ -27,10 +27,8 @@\n * goto error_handler;\n *\n * while ((ok = dir_iterator_advance(iter)) == ITER_OK) {\n- * if (want_to_stop_iteration()) {\n- * ok = dir_iterator_abort(iter);\n+ * if (want_to_stop_iteration())\n * break;\n- * }\n *\n * // Access information about the current path:\n * if (S_ISDIR(iter->st.st_mode))\n@@ -39,6 +37,7 @@\n *\n * if (ok != ITER_DONE)\n * handle_error();\n+ * dir_iterator_release(iter);\n *\n * Callers are allowed to modify iter->path while they are working,\n * but they must restore it to its original contents before calling\n@@ -107,11 +106,7 @@ struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags);\n */\n int dir_iterator_advance(struct dir_iterator *iterator);\n \n-/*\n- * End the iteration before it has been exhausted. Free the\n- * dir_iterator and any associated resources and return ITER_DONE. On\n- * error, free the dir_iterator and return ITER_ERROR.\n- */\n-int dir_iterator_abort(struct dir_iterator *iterator);\n+/* Free the dir_iterator and any associated resources. */\n+void dir_iterator_free(struct dir_iterator *iterator);\n \n #endif\ndiff --git a/refs.c b/refs.c\nindex eaf41421f50..8eff60a2186 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2476,6 +2476,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n {\n \tstruct strbuf dirname = STRBUF_INIT;\n \tstruct strbuf referent = STRBUF_INIT;\n+\tstruct ref_iterator *iter = NULL;\n \tstruct strset dirnames;\n \tint ret = -1;\n \n@@ -2552,7 +2553,6 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \t\tstrbuf_addch(&dirname, '/');\n \n \t\tif (!initial_transaction) {\n-\t\t\tstruct ref_iterator *iter;\n \t\t\tint ok;\n \n \t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n@@ -2564,12 +2564,14 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \n \t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n \t\t\t\t\t iter->refname, refname);\n-\t\t\t\tref_iterator_abort(iter);\n \t\t\t\tgoto cleanup;\n \t\t\t}\n \n \t\t\tif (ok != ITER_DONE)\n \t\t\t\tBUG(\"error while iterating over references\");\n+\n+\t\t\tref_iterator_free(iter);\n+\t\t\titer = NULL;\n \t\t}\n \n \t\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n@@ -2586,6 +2588,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \tstrbuf_release(&referent);\n \tstrbuf_release(&dirname);\n \tstrset_clear(&dirnames);\n+\tref_iterator_free(iter);\n \treturn ret;\n }\n \ndiff --git a/refs/debug.c b/refs/debug.c\nindex fbc4df08b43..a9786da4ba1 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -179,19 +179,18 @@ static int debug_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn res;\n }\n \n-static int debug_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void debug_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct debug_ref_iterator *diter =\n \t\t(struct debug_ref_iterator *)ref_iterator;\n-\tint res = diter->iter->vtable->abort(diter->iter);\n-\ttrace_printf_key(&trace_refs, \"iterator_abort: %d\\n\", res);\n-\treturn res;\n+\tditer->iter->vtable->release(diter->iter);\n+\ttrace_printf_key(&trace_refs, \"iterator_abort\\n\");\n }\n \n static struct ref_iterator_vtable debug_ref_iterator_vtable = {\n \t.advance = debug_ref_iterator_advance,\n \t.peel = debug_ref_iterator_peel,\n-\t.abort = debug_ref_iterator_abort,\n+\t.release = debug_ref_iterator_release,\n };\n \n static struct ref_iterator *\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 29f08dced40..9511b6f3448 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -919,10 +919,6 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\titer->iter0 = NULL;\n-\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\tok = ITER_ERROR;\n-\n \treturn ok;\n }\n \n@@ -935,23 +931,17 @@ static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator_peel(iter->iter0, peeled);\n }\n \n-static int files_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void files_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct files_ref_iterator *iter =\n \t\t(struct files_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->iter0)\n-\t\tok = ref_iterator_abort(iter->iter0);\n-\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n+\tref_iterator_free(iter->iter0);\n }\n \n static struct ref_iterator_vtable files_ref_iterator_vtable = {\n \t.advance = files_ref_iterator_advance,\n \t.peel = files_ref_iterator_peel,\n-\t.abort = files_ref_iterator_abort,\n+\t.release = files_ref_iterator_release,\n };\n \n static struct ref_iterator *files_ref_iterator_begin(\n@@ -1382,7 +1372,7 @@ static int should_pack_refs(struct files_ref_store *refs,\n \t\t\t\t iter->flags, opts))\n \t\t\trefcount++;\n \t\tif (refcount >= limit) {\n-\t\t\tref_iterator_abort(iter);\n+\t\t\tref_iterator_free(iter);\n \t\t\treturn 1;\n \t\t}\n \t}\n@@ -1390,6 +1380,7 @@ static int should_pack_refs(struct files_ref_store *refs,\n \tif (ret != ITER_DONE)\n \t\tdie(\"error while iterating over references\");\n \n+\tref_iterator_free(iter);\n \treturn 0;\n }\n \n@@ -1456,6 +1447,7 @@ static int files_pack_refs(struct ref_store *ref_store,\n \tpacked_refs_unlock(refs->packed_ref_store);\n \n \tprune_refs(refs, &refs_to_prune);\n+\tref_iterator_free(iter);\n \tstrbuf_release(&err);\n \treturn 0;\n }\n@@ -2303,9 +2295,6 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\titer->dir_iterator = NULL;\n-\tif (ref_iterator_abort(ref_iterator) == ITER_ERROR)\n-\t\tok = ITER_ERROR;\n \treturn ok;\n }\n \n@@ -2315,23 +2304,17 @@ static int files_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \tBUG(\"ref_iterator_peel() called for reflog_iterator\");\n }\n \n-static int files_reflog_iterator_abort(struct ref_iterator *ref_iterator)\n+static void files_reflog_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct files_reflog_iterator *iter =\n \t\t(struct files_reflog_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->dir_iterator)\n-\t\tok = dir_iterator_abort(iter->dir_iterator);\n-\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n+\tdir_iterator_free(iter->dir_iterator);\n }\n \n static struct ref_iterator_vtable files_reflog_iterator_vtable = {\n \t.advance = files_reflog_iterator_advance,\n \t.peel = files_reflog_iterator_peel,\n-\t.abort = files_reflog_iterator_abort,\n+\t.release = files_reflog_iterator_release,\n };\n \n static struct ref_iterator *reflog_iterator_begin(struct ref_store *ref_store,\n@@ -3808,6 +3791,7 @@ static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n \n out:\n+\tdir_iterator_free(iter);\n \tstrbuf_release(&sb);\n \tstrbuf_release(&refname);\n \treturn ret;\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex d25e568bf0b..aaeff270437 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -21,9 +21,14 @@ int ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator->vtable->peel(ref_iterator, peeled);\n }\n \n-int ref_iterator_abort(struct ref_iterator *ref_iterator)\n+void ref_iterator_free(struct ref_iterator *ref_iterator)\n {\n-\treturn ref_iterator->vtable->abort(ref_iterator);\n+\tif (ref_iterator) {\n+\t\tref_iterator->vtable->release(ref_iterator);\n+\t\t/* Help make use-after-free bugs fail quickly: */\n+\t\tref_iterator->vtable = NULL;\n+\t\tfree(ref_iterator);\n+\t}\n }\n \n void base_ref_iterator_init(struct ref_iterator *iter,\n@@ -36,20 +41,13 @@ void base_ref_iterator_init(struct ref_iterator *iter,\n \titer->flags = 0;\n }\n \n-void base_ref_iterator_free(struct ref_iterator *iter)\n-{\n-\t/* Help make use-after-free bugs fail quickly: */\n-\titer->vtable = NULL;\n-\tfree(iter);\n-}\n-\n struct empty_ref_iterator {\n \tstruct ref_iterator base;\n };\n \n-static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)\n {\n-\treturn ref_iterator_abort(ref_iterator);\n+\treturn ITER_DONE;\n }\n \n static int empty_ref_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n@@ -58,16 +56,14 @@ static int empty_ref_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \tBUG(\"peel called for empty iterator\");\n }\n \n-static int empty_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void empty_ref_iterator_release(struct ref_iterator *ref_iterator UNUSED)\n {\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable empty_ref_iterator_vtable = {\n \t.advance = empty_ref_iterator_advance,\n \t.peel = empty_ref_iterator_peel,\n-\t.abort = empty_ref_iterator_abort,\n+\t.release = empty_ref_iterator_release,\n };\n \n struct ref_iterator *empty_ref_iterator_begin(void)\n@@ -151,11 +147,13 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tif (!iter->current) {\n \t\t/* Initialize: advance both iterators to their first entries */\n \t\tif ((ok = ref_iterator_advance(iter->iter0)) != ITER_OK) {\n+\t\t\tref_iterator_free(iter->iter0);\n \t\t\titer->iter0 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n \t\t}\n \t\tif ((ok = ref_iterator_advance(iter->iter1)) != ITER_OK) {\n+\t\t\tref_iterator_free(iter->iter1);\n \t\t\titer->iter1 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -166,6 +164,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t * entry:\n \t\t */\n \t\tif ((ok = ref_iterator_advance(*iter->current)) != ITER_OK) {\n+\t\t\tref_iterator_free(*iter->current);\n \t\t\t*iter->current = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -179,9 +178,8 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t\titer->select(iter->iter0, iter->iter1, iter->cb_data);\n \n \t\tif (selection == ITER_SELECT_DONE) {\n-\t\t\treturn ref_iterator_abort(ref_iterator);\n+\t\t\treturn ITER_DONE;\n \t\t} else if (selection == ITER_SELECT_ERROR) {\n-\t\t\tref_iterator_abort(ref_iterator);\n \t\t\treturn ITER_ERROR;\n \t\t}\n \n@@ -195,6 +193,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \n \t\tif (selection & ITER_SKIP_SECONDARY) {\n \t\t\tif ((ok = ref_iterator_advance(*secondary)) != ITER_OK) {\n+\t\t\t\tref_iterator_free(*secondary);\n \t\t\t\t*secondary = NULL;\n \t\t\t\tif (ok == ITER_ERROR)\n \t\t\t\t\tgoto error;\n@@ -211,7 +210,6 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t}\n \n error:\n-\tref_iterator_abort(ref_iterator);\n \treturn ITER_ERROR;\n }\n \n@@ -227,28 +225,18 @@ static int merge_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator_peel(*iter->current, peeled);\n }\n \n-static int merge_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void merge_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct merge_ref_iterator *iter =\n \t\t(struct merge_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->iter0) {\n-\t\tif (ref_iterator_abort(iter->iter0) != ITER_DONE)\n-\t\t\tok = ITER_ERROR;\n-\t}\n-\tif (iter->iter1) {\n-\t\tif (ref_iterator_abort(iter->iter1) != ITER_DONE)\n-\t\t\tok = ITER_ERROR;\n-\t}\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n+\tref_iterator_free(iter->iter0);\n+\tref_iterator_free(iter->iter1);\n }\n \n static struct ref_iterator_vtable merge_ref_iterator_vtable = {\n \t.advance = merge_ref_iterator_advance,\n \t.peel = merge_ref_iterator_peel,\n-\t.abort = merge_ref_iterator_abort,\n+\t.release = merge_ref_iterator_release,\n };\n \n struct ref_iterator *merge_ref_iterator_begin(\n@@ -310,10 +298,10 @@ struct ref_iterator *overlay_ref_iterator_begin(\n \t * them.\n \t */\n \tif (is_empty_ref_iterator(front)) {\n-\t\tref_iterator_abort(front);\n+\t\tref_iterator_free(front);\n \t\treturn back;\n \t} else if (is_empty_ref_iterator(back)) {\n-\t\tref_iterator_abort(back);\n+\t\tref_iterator_free(back);\n \t\treturn front;\n \t}\n \n@@ -350,19 +338,10 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \n \twhile ((ok = ref_iterator_advance(iter->iter0)) == ITER_OK) {\n \t\tint cmp = compare_prefix(iter->iter0->refname, iter->prefix);\n-\n \t\tif (cmp < 0)\n \t\t\tcontinue;\n-\n-\t\tif (cmp > 0) {\n-\t\t\t/*\n-\t\t\t * As the source iterator is ordered, we\n-\t\t\t * can stop the iteration as soon as we see a\n-\t\t\t * refname that comes after the prefix:\n-\t\t\t */\n-\t\t\tok = ref_iterator_abort(iter->iter0);\n-\t\t\tbreak;\n-\t\t}\n+\t\tif (cmp > 0)\n+\t\t\treturn ITER_DONE;\n \n \t\tif (iter->trim) {\n \t\t\t/*\n@@ -386,9 +365,6 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\titer->iter0 = NULL;\n-\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\treturn ITER_ERROR;\n \treturn ok;\n }\n \n@@ -401,23 +377,18 @@ static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator_peel(iter->iter0, peeled);\n }\n \n-static int prefix_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void prefix_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct prefix_ref_iterator *iter =\n \t\t(struct prefix_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->iter0)\n-\t\tok = ref_iterator_abort(iter->iter0);\n+\tref_iterator_free(iter->iter0);\n \tfree(iter->prefix);\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n }\n \n static struct ref_iterator_vtable prefix_ref_iterator_vtable = {\n \t.advance = prefix_ref_iterator_advance,\n \t.peel = prefix_ref_iterator_peel,\n-\t.abort = prefix_ref_iterator_abort,\n+\t.release = prefix_ref_iterator_release,\n };\n \n struct ref_iterator *prefix_ref_iterator_begin(struct ref_iterator *iter0,\n@@ -453,20 +424,14 @@ int do_for_each_ref_iterator(struct ref_iterator *iter,\n \tcurrent_ref_iter = iter;\n \twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\tretval = fn(iter->refname, iter->referent, iter->oid, iter->flags, cb_data);\n-\t\tif (retval) {\n-\t\t\t/*\n-\t\t\t * If ref_iterator_abort() returns ITER_ERROR,\n-\t\t\t * we ignore that error in deference to the\n-\t\t\t * callback function's return value.\n-\t\t\t */\n-\t\t\tref_iterator_abort(iter);\n+\t\tif (retval)\n \t\t\tgoto out;\n-\t\t}\n \t}\n \n out:\n \tcurrent_ref_iter = old_ref_iter;\n \tif (ok == ITER_ERROR)\n-\t\treturn -1;\n+\t\tretval = -1;\n+\tref_iterator_free(iter);\n \treturn retval;\n }\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex a7b6f74b6e3..38a1956d1a8 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -954,9 +954,6 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\tok = ITER_ERROR;\n-\n \treturn ok;\n }\n \n@@ -976,23 +973,19 @@ static int packed_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t}\n }\n \n-static int packed_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void packed_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct packed_ref_iterator *iter =\n \t\t(struct packed_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n \tstrbuf_release(&iter->refname_buf);\n \tfree(iter->jump);\n \trelease_snapshot(iter->snapshot);\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n }\n \n static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \t.advance = packed_ref_iterator_advance,\n \t.peel = packed_ref_iterator_peel,\n-\t.abort = packed_ref_iterator_abort\n+\t.release = packed_ref_iterator_release,\n };\n \n static int jump_list_entry_cmp(const void *va, const void *vb)\n@@ -1362,8 +1355,10 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t */\n \titer = packed_ref_iterator_begin(&refs->base, \"\", NULL,\n \t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n-\tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n+\tif ((ok = ref_iterator_advance(iter)) != ITER_OK) {\n+\t\tref_iterator_free(iter);\n \t\titer = NULL;\n+\t}\n \n \ti = 0;\n \n@@ -1411,8 +1406,10 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t\t\t\t * the iterator over the unneeded\n \t\t\t\t * value.\n \t\t\t\t */\n-\t\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n+\t\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK) {\n+\t\t\t\t\tref_iterator_free(iter);\n \t\t\t\t\titer = NULL;\n+\t\t\t\t}\n \t\t\t\tcmp = +1;\n \t\t\t} else {\n \t\t\t\t/*\n@@ -1449,8 +1446,10 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t\t\t\t\t peel_error ? NULL : &peeled))\n \t\t\t\tgoto write_error;\n \n-\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n+\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK) {\n+\t\t\t\tref_iterator_free(iter);\n \t\t\t\titer = NULL;\n+\t\t\t}\n \t\t} else if (is_null_oid(&update->new_oid)) {\n \t\t\t/*\n \t\t\t * The update wants to delete the reference,\n@@ -1499,9 +1498,7 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t\t get_tempfile_path(refs->tempfile), strerror(errno));\n \n error:\n-\tif (iter)\n-\t\tref_iterator_abort(iter);\n-\n+\tref_iterator_free(iter);\n \tdelete_tempfile(&refs->tempfile);\n \treturn -1;\n }\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 02f09e4df88..6457e02c1ea 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -409,7 +409,7 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\tif (++level->index == level->dir->nr) {\n \t\t\t/* This level is exhausted; pop up a level */\n \t\t\tif (--iter->levels_nr == 0)\n-\t\t\t\treturn ref_iterator_abort(ref_iterator);\n+\t\t\t\treturn ITER_DONE;\n \n \t\t\tcontinue;\n \t\t}\n@@ -452,21 +452,18 @@ static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn peel_object(iter->repo, ref_iterator->oid, peeled) ? -1 : 0;\n }\n \n-static int cache_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void cache_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct cache_ref_iterator *iter =\n \t\t(struct cache_ref_iterator *)ref_iterator;\n-\n \tfree((char *)iter->prefix);\n \tfree(iter->levels);\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable cache_ref_iterator_vtable = {\n \t.advance = cache_ref_iterator_advance,\n \t.peel = cache_ref_iterator_peel,\n-\t.abort = cache_ref_iterator_abort\n+\t.release = cache_ref_iterator_release,\n };\n \n struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex aaab711bb96..27ff822cf43 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -273,11 +273,11 @@ enum do_for_each_ref_flags {\n * the next reference and returns ITER_OK. The data pointed at by\n * refname and oid belong to the iterator; if you want to retain them\n * after calling ref_iterator_advance() again or calling\n- * ref_iterator_abort(), you must make a copy. When the iteration has\n+ * ref_iterator_free(), you must make a copy. When the iteration has\n * been exhausted, ref_iterator_advance() releases any resources\n * associated with the iteration, frees the ref_iterator object, and\n * returns ITER_DONE. If you want to abort the iteration early, call\n- * ref_iterator_abort(), which also frees the ref_iterator object and\n+ * ref_iterator_free(), which also frees the ref_iterator object and\n * any associated resources. If there was an internal error advancing\n * to the next entry, ref_iterator_advance() aborts the iteration,\n * frees the ref_iterator, and returns ITER_ERROR.\n@@ -292,10 +292,8 @@ enum do_for_each_ref_flags {\n * struct ref_iterator *iter = ...;\n *\n * while ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n- * if (want_to_stop_iteration()) {\n- * ok = ref_iterator_abort(iter);\n+ * if (want_to_stop_iteration())\n * break;\n- * }\n *\n * // Access information about the current reference:\n * if (!(iter->flags & REF_ISSYMREF))\n@@ -307,6 +305,7 @@ enum do_for_each_ref_flags {\n *\n * if (ok != ITER_DONE)\n * handle_error();\n+ * ref_iterator_free(iter);\n */\n struct ref_iterator {\n \tstruct ref_iterator_vtable *vtable;\n@@ -333,12 +332,8 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator);\n int ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t struct object_id *peeled);\n \n-/*\n- * End the iteration before it has been exhausted, freeing the\n- * reference iterator and any associated resources and returning\n- * ITER_DONE. If the abort itself failed, return ITER_ERROR.\n- */\n-int ref_iterator_abort(struct ref_iterator *ref_iterator);\n+/* Free the reference iterator and any associated resources. */\n+void ref_iterator_free(struct ref_iterator *ref_iterator);\n \n /*\n * An iterator over nothing (its first ref_iterator_advance() call\n@@ -438,13 +433,6 @@ struct ref_iterator *prefix_ref_iterator_begin(struct ref_iterator *iter0,\n void base_ref_iterator_init(struct ref_iterator *iter,\n \t\t\t struct ref_iterator_vtable *vtable);\n \n-/*\n- * Base class destructor for ref_iterators. Destroy the ref_iterator\n- * part of iter and shallow-free the object. This is meant to be\n- * called only by the destructors of derived classes.\n- */\n-void base_ref_iterator_free(struct ref_iterator *iter);\n-\n /* Virtual function declarations for ref_iterators: */\n \n /*\n@@ -463,15 +451,14 @@ typedef int ref_iterator_peel_fn(struct ref_iterator *ref_iterator,\n \n /*\n * Implementations of this function should free any resources specific\n- * to the derived class, then call base_ref_iterator_free() to clean\n- * up and free the ref_iterator object.\n+ * to the derived class.\n */\n-typedef int ref_iterator_abort_fn(struct ref_iterator *ref_iterator);\n+typedef void ref_iterator_release_fn(struct ref_iterator *ref_iterator);\n \n struct ref_iterator_vtable {\n \tref_iterator_advance_fn *advance;\n \tref_iterator_peel_fn *peel;\n-\tref_iterator_abort_fn *abort;\n+\tref_iterator_release_fn *release;\n };\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 2a90e7cb391..06543f79c64 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -711,17 +711,10 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\tbreak;\n \t}\n \n-\tif (iter->err > 0) {\n-\t\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\t\treturn ITER_ERROR;\n+\tif (iter->err > 0)\n \t\treturn ITER_DONE;\n-\t}\n-\n-\tif (iter->err < 0) {\n-\t\tref_iterator_abort(ref_iterator);\n+\tif (iter->err < 0)\n \t\treturn ITER_ERROR;\n-\t}\n-\n \treturn ITER_OK;\n }\n \n@@ -740,7 +733,7 @@ static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn -1;\n }\n \n-static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void reftable_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct reftable_ref_iterator *iter =\n \t\t(struct reftable_ref_iterator *)ref_iterator;\n@@ -751,14 +744,12 @@ static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n \t\t\tfree(iter->exclude_patterns[i]);\n \t\tfree(iter->exclude_patterns);\n \t}\n-\tfree(iter);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n \t.advance = reftable_ref_iterator_advance,\n \t.peel = reftable_ref_iterator_peel,\n-\t.abort = reftable_ref_iterator_abort\n+\t.release = reftable_ref_iterator_release,\n };\n \n static int qsort_strcmp(const void *va, const void *vb)\n@@ -2017,17 +2008,10 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \t\tbreak;\n \t}\n \n-\tif (iter->err > 0) {\n-\t\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\t\treturn ITER_ERROR;\n+\tif (iter->err > 0)\n \t\treturn ITER_DONE;\n-\t}\n-\n-\tif (iter->err < 0) {\n-\t\tref_iterator_abort(ref_iterator);\n+\tif (iter->err < 0)\n \t\treturn ITER_ERROR;\n-\t}\n-\n \treturn ITER_OK;\n }\n \n@@ -2038,21 +2022,19 @@ static int reftable_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSE\n \treturn -1;\n }\n \n-static int reftable_reflog_iterator_abort(struct ref_iterator *ref_iterator)\n+static void reftable_reflog_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct reftable_reflog_iterator *iter =\n \t\t(struct reftable_reflog_iterator *)ref_iterator;\n \treftable_log_record_release(&iter->log);\n \treftable_iterator_destroy(&iter->iter);\n \tstrbuf_release(&iter->last_name);\n-\tfree(iter);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable reftable_reflog_iterator_vtable = {\n \t.advance = reftable_reflog_iterator_advance,\n \t.peel = reftable_reflog_iterator_peel,\n-\t.abort = reftable_reflog_iterator_abort\n+\t.release = reftable_reflog_iterator_release,\n };\n \n static struct reftable_reflog_iterator *reflog_iterator_for_stack(struct reftable_ref_store *refs,\ndiff --git a/t/helper/test-dir-iterator.c b/t/helper/test-dir-iterator.c\nindex 6b297bd7536..8d46e8ba409 100644\n--- a/t/helper/test-dir-iterator.c\n+++ b/t/helper/test-dir-iterator.c\n@@ -53,6 +53,7 @@ int cmd__dir_iterator(int argc, const char **argv)\n \t\tprintf(\"(%s) [%s] %s\\n\", diter->relative_path, diter->basename,\n \t\t diter->path.buf);\n \t}\n+\tdir_iterator_free(diter);\n \n \tif (iter_status != ITER_DONE) {\n \t\tprintf(\"dir_iterator_advance failure\\n\");\n\n-- \n2.48.1.666.gff9fcf71b7.dirty\n\n"},{"id":"512535","messageId":"20250217-pks-update-ref-optimization-v1-8-a2b6d87a24af@pks.im","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im","subject":"[PATCH 08/14] refs/iterator: provide infrastructure to re-seek iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-17T15:50:22Z","receivedAt":"2025-02-17T15:50:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Reftable iterators need to be scrapped after they have either been\nexhausted or aren't useful to the caller anymore, and it is explicitly\nnot possible to reuse them for iterations. But enabling for reuse of\niterators may allow us to tune them by reusing internal state of an\niterator. The reftable iterators for example can already be reused\ninternally, but we're not able to expose this to any users outside of\nthe reftable backend.\n\nIntroduce a new `.seek` function in the ref iterator vtable that allows\ncallers to re-seek an iterator. It is expected to be functionally the\nsame as calling `refs_ref_iterator_begin()` with a different (or the\nsame) prefix.\n\nImplement the callback for trivial cases. The other iterators will be\nimplemented in subsequent commits.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/debug.c | 11 +++++++++++\n refs/iterator.c | 24 ++++++++++++++++++++++++\n refs/refs-internal.h | 23 +++++++++++++++++++++++\n 3 files changed, 58 insertions(+)\n\ndiff --git a/refs/debug.c b/refs/debug.c\nindex a9786da4ba1..5390fa9c187 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -169,6 +169,16 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn res;\n }\n \n+static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct debug_ref_iterator *diter =\n+\t\t(struct debug_ref_iterator *)ref_iterator;\n+\tint res = diter->iter->vtable->seek(diter->iter, prefix);\n+\ttrace_printf_key(&trace_refs, \"iterator_seek: %s: %d\\n\", prefix ? prefix : \"\", res);\n+\treturn res;\n+}\n+\n static int debug_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -189,6 +199,7 @@ static void debug_ref_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable debug_ref_iterator_vtable = {\n \t.advance = debug_ref_iterator_advance,\n+\t.seek = debug_ref_iterator_seek,\n \t.peel = debug_ref_iterator_peel,\n \t.release = debug_ref_iterator_release,\n };\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex aaeff270437..757b105261a 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -15,6 +15,12 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ref_iterator->vtable->advance(ref_iterator);\n }\n \n+int ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t const char *prefix)\n+{\n+\treturn ref_iterator->vtable->seek(ref_iterator, prefix);\n+}\n+\n int ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t struct object_id *peeled)\n {\n@@ -50,6 +56,12 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)\n \treturn ITER_DONE;\n }\n \n+static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n+\t\t\t\t const char *prefix UNUSED)\n+{\n+\treturn 0;\n+}\n+\n static int empty_ref_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \t\t\t\t struct object_id *peeled UNUSED)\n {\n@@ -62,6 +74,7 @@ static void empty_ref_iterator_release(struct ref_iterator *ref_iterator UNUSED)\n \n static struct ref_iterator_vtable empty_ref_iterator_vtable = {\n \t.advance = empty_ref_iterator_advance,\n+\t.seek = empty_ref_iterator_seek,\n \t.peel = empty_ref_iterator_peel,\n \t.release = empty_ref_iterator_release,\n };\n@@ -368,6 +381,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n+static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct prefix_ref_iterator *iter =\n+\t\t(struct prefix_ref_iterator *)ref_iterator;\n+\tfree(iter->prefix);\n+\titer->prefix = xstrdup_or_null(prefix);\n+\treturn ref_iterator_seek(iter->iter0, prefix);\n+}\n+\n static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -387,6 +410,7 @@ static void prefix_ref_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable prefix_ref_iterator_vtable = {\n \t.advance = prefix_ref_iterator_advance,\n+\t.seek = prefix_ref_iterator_seek,\n \t.peel = prefix_ref_iterator_peel,\n \t.release = prefix_ref_iterator_release,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 27ff822cf43..5fade7e8408 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -325,6 +325,21 @@ struct ref_iterator {\n */\n int ref_iterator_advance(struct ref_iterator *ref_iterator);\n \n+/*\n+ * Seek the iterator to the first reference with the given prefix.\n+ * The prefix is matched as a literal string, without regard for path\n+ * separators. If prefix is NULL or the empty string, seek the iterator to the\n+ * first reference again.\n+ *\n+ * This function is expected to behave as if a new ref iterator with the same\n+ * prefix had been created, but allows reuse of iterators and thus may allow\n+ * the backend to optimize.\n+ *\n+ * Returns 0 on success, a negative error code otherwise.\n+ */\n+int ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t const char *prefix);\n+\n /*\n * If possible, peel the reference currently being viewed by the\n * iterator. Return 0 on success.\n@@ -443,6 +458,13 @@ void base_ref_iterator_init(struct ref_iterator *iter,\n */\n typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);\n \n+/*\n+ * Seek the iterator to the first reference matching the given prefix. Should\n+ * behave the same as if a new iterator was created with the same prefix.\n+ */\n+typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix);\n+\n /*\n * Peels the current ref, returning 0 for success or -1 for failure.\n */\n@@ -457,6 +479,7 @@ typedef void ref_iterator_release_fn(struct ref_iterator *ref_iterator);\n \n struct ref_iterator_vtable {\n \tref_iterator_advance_fn *advance;\n+\tref_iterator_seek_fn *seek;\n \tref_iterator_peel_fn *peel;\n \tref_iterator_release_fn *release;\n };\n\n-- \n2.48.1.666.gff9fcf71b7.dirty\n\n"},{"id":"512536","messageId":"20250217-pks-update-ref-optimization-v1-9-a2b6d87a24af@pks.im","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im","subject":"[PATCH 09/14] refs/iterator: implement seeking for merged iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-17T15:50:23Z","receivedAt":"2025-02-17T15:50:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking on merged iterators. The implementation is rather\nstraight forward, with the only exception that we must not deallocate\nthe underlying iterators once they have been exhausted.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/iterator.c | 38 +++++++++++++++++++++++++++++---------\n 1 file changed, 29 insertions(+), 9 deletions(-)\n\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex 757b105261a..63608ef9907 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -96,7 +96,8 @@ int is_empty_ref_iterator(struct ref_iterator *ref_iterator)\n struct merge_ref_iterator {\n \tstruct ref_iterator base;\n \n-\tstruct ref_iterator *iter0, *iter1;\n+\tstruct ref_iterator *iter0, *iter0_owned;\n+\tstruct ref_iterator *iter1, *iter1_owned;\n \n \tref_iterator_select_fn *select;\n \tvoid *cb_data;\n@@ -160,13 +161,11 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tif (!iter->current) {\n \t\t/* Initialize: advance both iterators to their first entries */\n \t\tif ((ok = ref_iterator_advance(iter->iter0)) != ITER_OK) {\n-\t\t\tref_iterator_free(iter->iter0);\n \t\t\titer->iter0 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n \t\t}\n \t\tif ((ok = ref_iterator_advance(iter->iter1)) != ITER_OK) {\n-\t\t\tref_iterator_free(iter->iter1);\n \t\t\titer->iter1 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -177,7 +176,6 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t * entry:\n \t\t */\n \t\tif ((ok = ref_iterator_advance(*iter->current)) != ITER_OK) {\n-\t\t\tref_iterator_free(*iter->current);\n \t\t\t*iter->current = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -206,7 +204,6 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \n \t\tif (selection & ITER_SKIP_SECONDARY) {\n \t\t\tif ((ok = ref_iterator_advance(*secondary)) != ITER_OK) {\n-\t\t\t\tref_iterator_free(*secondary);\n \t\t\t\t*secondary = NULL;\n \t\t\t\tif (ok == ITER_ERROR)\n \t\t\t\t\tgoto error;\n@@ -226,6 +223,28 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ITER_ERROR;\n }\n \n+static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct merge_ref_iterator *iter =\n+\t\t(struct merge_ref_iterator *)ref_iterator;\n+\tint ret;\n+\n+\titer->current = NULL;\n+\titer->iter0 = iter->iter0_owned;\n+\titer->iter1 = iter->iter1_owned;\n+\n+\tret = ref_iterator_seek(iter->iter0, prefix);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n+\tret = ref_iterator_seek(iter->iter1, prefix);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n+\treturn 0;\n+}\n+\n static int merge_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -242,12 +261,13 @@ static void merge_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct merge_ref_iterator *iter =\n \t\t(struct merge_ref_iterator *)ref_iterator;\n-\tref_iterator_free(iter->iter0);\n-\tref_iterator_free(iter->iter1);\n+\tref_iterator_free(iter->iter0_owned);\n+\tref_iterator_free(iter->iter1_owned);\n }\n \n static struct ref_iterator_vtable merge_ref_iterator_vtable = {\n \t.advance = merge_ref_iterator_advance,\n+\t.seek = merge_ref_iterator_seek,\n \t.peel = merge_ref_iterator_peel,\n \t.release = merge_ref_iterator_release,\n };\n@@ -268,8 +288,8 @@ struct ref_iterator *merge_ref_iterator_begin(\n \t */\n \n \tbase_ref_iterator_init(ref_iterator, &merge_ref_iterator_vtable);\n-\titer->iter0 = iter0;\n-\titer->iter1 = iter1;\n+\titer->iter0 = iter->iter0_owned = iter0;\n+\titer->iter1 = iter->iter1_owned = iter1;\n \titer->select = select;\n \titer->cb_data = cb_data;\n \titer->current = NULL;\n\n-- \n2.48.1.666.gff9fcf71b7.dirty\n\n"},{"id":"512537","messageId":"20250217-pks-update-ref-optimization-v1-10-a2b6d87a24af@pks.im","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im","subject":"[PATCH 10/14] refs/iterator: implement seeking for reftable iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-17T15:50:24Z","receivedAt":"2025-02-17T15:50:36Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking of reftable iterators. As the low-level reftable\niterators already support seeking this change is straight-forward. Two\nnotes though:\n\n - We do not support seeking on reflog iterators.\n\n - We start to check whether `reftable_stack_init_ref_iterator()` is\n successful.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 35 ++++++++++++++++++++++++++++++-----\n 1 file changed, 30 insertions(+), 5 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 06543f79c64..b0c09f34433 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -547,7 +547,7 @@ struct reftable_ref_iterator {\n \tstruct reftable_ref_record ref;\n \tstruct object_id oid;\n \n-\tconst char *prefix;\n+\tchar *prefix;\n \tsize_t prefix_len;\n \tchar **exclude_patterns;\n \tsize_t exclude_patterns_index;\n@@ -718,6 +718,20 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ITER_OK;\n }\n \n+static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct reftable_ref_iterator *iter =\n+\t\t(struct reftable_ref_iterator *)ref_iterator;\n+\n+\tfree(iter->prefix);\n+\titer->prefix = xstrdup_or_null(prefix);\n+\titer->prefix_len = prefix ? strlen(prefix) : 0;\n+\titer->err = reftable_iterator_seek_ref(&iter->iter, prefix);\n+\n+\treturn iter->err;\n+}\n+\n static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -744,10 +758,12 @@ static void reftable_ref_iterator_release(struct ref_iterator *ref_iterator)\n \t\t\tfree(iter->exclude_patterns[i]);\n \t\tfree(iter->exclude_patterns);\n \t}\n+\tfree(iter->prefix);\n }\n \n static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n \t.advance = reftable_ref_iterator_advance,\n+\t.seek = reftable_ref_iterator_seek,\n \t.peel = reftable_ref_iterator_peel,\n \t.release = reftable_ref_iterator_release,\n };\n@@ -806,8 +822,6 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_\n \n \titer = xcalloc(1, sizeof(*iter));\n \tbase_ref_iterator_init(&iter->base, &reftable_ref_iterator_vtable);\n-\titer->prefix = prefix;\n-\titer->prefix_len = prefix ? strlen(prefix) : 0;\n \titer->base.oid = &iter->oid;\n \titer->flags = flags;\n \titer->refs = refs;\n@@ -821,8 +835,11 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_\n \tif (ret)\n \t\tgoto done;\n \n-\treftable_stack_init_ref_iterator(stack, &iter->iter);\n-\tret = reftable_iterator_seek_ref(&iter->iter, prefix);\n+\tret = reftable_stack_init_ref_iterator(stack, &iter->iter);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_ref_iterator_seek(&iter->base, prefix);\n \tif (ret)\n \t\tgoto done;\n \n@@ -2015,6 +2032,13 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ITER_OK;\n }\n \n+static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n+\t\t\t\t\t const char *prefix UNUSED)\n+{\n+\tBUG(\"reftable reflog iterator cannot be seeked\");\n+\treturn -1;\n+}\n+\n static int reftable_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \t\t\t\t\t struct object_id *peeled UNUSED)\n {\n@@ -2033,6 +2057,7 @@ static void reftable_reflog_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable reftable_reflog_iterator_vtable = {\n \t.advance = reftable_reflog_iterator_advance,\n+\t.seek = reftable_reflog_iterator_seek,\n \t.peel = reftable_reflog_iterator_peel,\n \t.release = reftable_reflog_iterator_release,\n };\n\n-- \n2.48.1.666.gff9fcf71b7.dirty\n\n"},{"id":"512538","messageId":"20250217-pks-update-ref-optimization-v1-11-a2b6d87a24af@pks.im","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im","subject":"[PATCH 11/14] refs/iterator: implement seeking for ref-cache iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-17T15:50:25Z","receivedAt":"2025-02-17T15:50:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking of ref-cache iterators. This is done by splitting most\nof the logic to seek iterators out of `cache_ref_iterator_begin()` and\nputting it into `cache_ref_iterator_seek()` so that we can reuse the\nlogic.\n\nNote that we cannot use the optimization anymore where we return an\nempty ref iterator when there aren't any references, as otherwise it\nwouldn't be possible to reseek the iterator to a different prefix that\nmay exist. This shouldn't be much of a performance corncern though as we\nnow start to bail out early in case `advance()` sees that there are no\nmore directories to be searched.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/ref-cache.c | 74 ++++++++++++++++++++++++++++++++++++--------------------\n 1 file changed, 48 insertions(+), 26 deletions(-)\n\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 6457e02c1ea..b54547d71ee 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -362,9 +362,7 @@ struct cache_ref_iterator {\n \tstruct ref_iterator base;\n \n \t/*\n-\t * The number of levels currently on the stack. This is always\n-\t * at least 1, because when it becomes zero the iteration is\n-\t * ended and this struct is freed.\n+\t * The number of levels currently on the stack.\n \t */\n \tsize_t levels_nr;\n \n@@ -389,6 +387,9 @@ struct cache_ref_iterator {\n \tstruct cache_ref_iterator_level *levels;\n \n \tstruct repository *repo;\n+\tstruct ref_cache *cache;\n+\n+\tint prime_dir;\n };\n \n static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n@@ -396,6 +397,9 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tstruct cache_ref_iterator *iter =\n \t\t(struct cache_ref_iterator *)ref_iterator;\n \n+\tif (!iter->levels_nr)\n+\t\treturn ITER_DONE;\n+\n \twhile (1) {\n \t\tstruct cache_ref_iterator_level *level =\n \t\t\t&iter->levels[iter->levels_nr - 1];\n@@ -444,6 +448,40 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t}\n }\n \n+static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct cache_ref_iterator *iter =\n+\t\t(struct cache_ref_iterator *)ref_iterator;\n+\tstruct ref_dir *dir;\n+\n+\tdir = get_ref_dir(iter->cache->root);\n+\tif (prefix && *prefix)\n+\t\tdir = find_containing_dir(dir, prefix);\n+\n+\tif (dir) {\n+\t\tstruct cache_ref_iterator_level *level;\n+\n+\t\tif (iter->prime_dir)\n+\t\t\tprime_ref_dir(dir, prefix);\n+\t\titer->levels_nr = 1;\n+\t\tlevel = &iter->levels[0];\n+\t\tlevel->index = -1;\n+\t\tlevel->dir = dir;\n+\n+\t\tif (prefix && *prefix) {\n+\t\t\titer->prefix = xstrdup(prefix);\n+\t\t\tlevel->prefix_state = PREFIX_WITHIN_DIR;\n+\t\t} else {\n+\t\t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n+\t\t}\n+\t} else {\n+\t\titer->levels_nr = 0;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -462,6 +500,7 @@ static void cache_ref_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable cache_ref_iterator_vtable = {\n \t.advance = cache_ref_iterator_advance,\n+\t.seek = cache_ref_iterator_seek,\n \t.peel = cache_ref_iterator_peel,\n \t.release = cache_ref_iterator_release,\n };\n@@ -471,39 +510,22 @@ struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\n \t\t\t\t\t struct repository *repo,\n \t\t\t\t\t int prime_dir)\n {\n-\tstruct ref_dir *dir;\n \tstruct cache_ref_iterator *iter;\n \tstruct ref_iterator *ref_iterator;\n-\tstruct cache_ref_iterator_level *level;\n-\n-\tdir = get_ref_dir(cache->root);\n-\tif (prefix && *prefix)\n-\t\tdir = find_containing_dir(dir, prefix);\n-\tif (!dir)\n-\t\t/* There's nothing to iterate over. */\n-\t\treturn empty_ref_iterator_begin();\n-\n-\tif (prime_dir)\n-\t\tprime_ref_dir(dir, prefix);\n \n \tCALLOC_ARRAY(iter, 1);\n \tref_iterator = &iter->base;\n \tbase_ref_iterator_init(ref_iterator, &cache_ref_iterator_vtable);\n \tALLOC_GROW(iter->levels, 10, iter->levels_alloc);\n \n-\titer->levels_nr = 1;\n-\tlevel = &iter->levels[0];\n-\tlevel->index = -1;\n-\tlevel->dir = dir;\n+\titer->repo = repo;\n+\titer->cache = cache;\n+\titer->prime_dir = prime_dir;\n \n-\tif (prefix && *prefix) {\n-\t\titer->prefix = xstrdup(prefix);\n-\t\tlevel->prefix_state = PREFIX_WITHIN_DIR;\n-\t} else {\n-\t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n+\tif (cache_ref_iterator_seek(&iter->base, prefix) < 0) {\n+\t\tref_iterator_free(&iter->base);\n+\t\treturn NULL;\n \t}\n \n-\titer->repo = repo;\n-\n \treturn ref_iterator;\n }\n\n-- \n2.48.1.666.gff9fcf71b7.dirty\n\n"},{"id":"512539","messageId":"20250217-pks-update-ref-optimization-v1-12-a2b6d87a24af@pks.im","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im","subject":"[PATCH 12/14] refs/iterator: implement seeking for `packed-ref` iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-17T15:50:26Z","receivedAt":"2025-02-17T15:50:38Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking of `packed-ref` iterators. The implementation is again\nstraight forward, except that we cannot continue to use the prefix\niterator as we would otherwise not be able to reseek the iterator\nanymore in case one first asks for an empty and then for a non-empty\nprefix. Instead, we open-code the logic to in `advance()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/packed-backend.c | 62 +++++++++++++++++++++++++++++++++------------------\n 1 file changed, 40 insertions(+), 22 deletions(-)\n\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 38a1956d1a8..71a38acfedc 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -819,6 +819,8 @@ struct packed_ref_iterator {\n \n \tstruct snapshot *snapshot;\n \n+\tchar *prefix;\n+\n \t/* The current position in the snapshot's buffer: */\n \tconst char *pos;\n \n@@ -841,11 +843,9 @@ struct packed_ref_iterator {\n };\n \n /*\n- * Move the iterator to the next record in the snapshot, without\n- * respect for whether the record is actually required by the current\n- * iteration. Adjust the fields in `iter` and return `ITER_OK` or\n- * `ITER_DONE`. This function does not free the iterator in the case\n- * of `ITER_DONE`.\n+ * Move the iterator to the next record in the snapshot. Adjust the fields in\n+ * `iter` and return `ITER_OK` or `ITER_DONE`. This function does not free the\n+ * iterator in the case of `ITER_DONE`.\n */\n static int next_record(struct packed_ref_iterator *iter)\n {\n@@ -942,6 +942,9 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tint ok;\n \n \twhile ((ok = next_record(iter)) == ITER_OK) {\n+\t\tconst char *refname = iter->base.refname;\n+\t\tconst char *prefix = iter->prefix;\n+\n \t\tif (iter->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n \t\t !is_per_worktree_ref(iter->base.refname))\n \t\t\tcontinue;\n@@ -951,12 +954,41 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t\t\t\t &iter->oid, iter->flags))\n \t\t\tcontinue;\n \n+\t\twhile (prefix && *prefix) {\n+\t\t\tif (*refname < *prefix)\n+\t\t\t\tBUG(\"packed-refs backend yielded reference preceding its prefix\");\n+\t\t\telse if (*refname > *prefix)\n+\t\t\t\treturn ITER_DONE;\n+\t\t\tprefix++;\n+\t\t\trefname++;\n+\t\t}\n+\n \t\treturn ITER_OK;\n \t}\n \n \treturn ok;\n }\n \n+static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct packed_ref_iterator *iter =\n+\t\t(struct packed_ref_iterator *)ref_iterator;\n+\tconst char *start;\n+\n+\tif (prefix && *prefix)\n+\t\tstart = find_reference_location(iter->snapshot, prefix, 0);\n+\telse\n+\t\tstart = iter->snapshot->start;\n+\n+\tfree(iter->prefix);\n+\titer->prefix = xstrdup_or_null(prefix);\n+\titer->pos = start;\n+\titer->eof = iter->snapshot->eof;\n+\n+\treturn 0;\n+}\n+\n static int packed_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -979,11 +1011,13 @@ static void packed_ref_iterator_release(struct ref_iterator *ref_iterator)\n \t\t(struct packed_ref_iterator *)ref_iterator;\n \tstrbuf_release(&iter->refname_buf);\n \tfree(iter->jump);\n+\tfree(iter->prefix);\n \trelease_snapshot(iter->snapshot);\n }\n \n static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \t.advance = packed_ref_iterator_advance,\n+\t.seek = packed_ref_iterator_seek,\n \t.peel = packed_ref_iterator_peel,\n \t.release = packed_ref_iterator_release,\n };\n@@ -1097,7 +1131,6 @@ static struct ref_iterator *packed_ref_iterator_begin(\n {\n \tstruct packed_ref_store *refs;\n \tstruct snapshot *snapshot;\n-\tconst char *start;\n \tstruct packed_ref_iterator *iter;\n \tstruct ref_iterator *ref_iterator;\n \tunsigned int required_flags = REF_STORE_READ;\n@@ -1113,14 +1146,6 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \t */\n \tsnapshot = get_snapshot(refs);\n \n-\tif (prefix && *prefix)\n-\t\tstart = find_reference_location(snapshot, prefix, 0);\n-\telse\n-\t\tstart = snapshot->start;\n-\n-\tif (start == snapshot->eof)\n-\t\treturn empty_ref_iterator_begin();\n-\n \tCALLOC_ARRAY(iter, 1);\n \tref_iterator = &iter->base;\n \tbase_ref_iterator_init(ref_iterator, &packed_ref_iterator_vtable);\n@@ -1130,19 +1155,12 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \n \titer->snapshot = snapshot;\n \tacquire_snapshot(snapshot);\n-\n-\titer->pos = start;\n-\titer->eof = snapshot->eof;\n \tstrbuf_init(&iter->refname_buf, 0);\n-\n \titer->base.oid = &iter->oid;\n-\n \titer->repo = ref_store->repo;\n \titer->flags = flags;\n \n-\tif (prefix && *prefix)\n-\t\t/* Stop iteration after we've gone *past* prefix: */\n-\t\tref_iterator = prefix_ref_iterator_begin(ref_iterator, prefix, 0);\n+\tpacked_ref_iterator_seek(&iter->base, prefix);\n \n \treturn ref_iterator;\n }\n\n-- \n2.48.1.666.gff9fcf71b7.dirty\n\n"},{"id":"512540","messageId":"20250217-pks-update-ref-optimization-v1-13-a2b6d87a24af@pks.im","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im","subject":"[PATCH 13/14] refs/iterator: implement seeking for \"files\" iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-17T15:50:27Z","receivedAt":"2025-02-17T15:50:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking for \"files\" iterators. As we simply use a ref-cache\niterator under the hood the implementation is straight-forward. Note\nthat we do not implement seeking on reflog iterators, same as with the\n\"reftable\" backend.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/files-backend.c | 16 ++++++++++++++++\n 1 file changed, 16 insertions(+)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 9511b6f3448..acc28e1ad81 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -922,6 +922,14 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n+static int files_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct files_ref_iterator *iter =\n+\t\t(struct files_ref_iterator *)ref_iterator;\n+\treturn ref_iterator_seek(iter->iter0, prefix);\n+}\n+\n static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -940,6 +948,7 @@ static void files_ref_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable files_ref_iterator_vtable = {\n \t.advance = files_ref_iterator_advance,\n+\t.seek = files_ref_iterator_seek,\n \t.peel = files_ref_iterator_peel,\n \t.release = files_ref_iterator_release,\n };\n@@ -2298,6 +2307,12 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n+static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n+\t\t\t\t const char *prefix UNUSED)\n+{\n+\tBUG(\"ref_iterator_seek() called for reflog_iterator\");\n+}\n+\n static int files_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \t\t\t\t struct object_id *peeled UNUSED)\n {\n@@ -2313,6 +2328,7 @@ static void files_reflog_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable files_reflog_iterator_vtable = {\n \t.advance = files_reflog_iterator_advance,\n+\t.seek = files_reflog_iterator_seek,\n \t.peel = files_reflog_iterator_peel,\n \t.release = files_reflog_iterator_release,\n };\n\n-- \n2.48.1.666.gff9fcf71b7.dirty\n\n"},{"id":"512541","messageId":"20250217-pks-update-ref-optimization-v1-14-a2b6d87a24af@pks.im","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im","subject":"[PATCH 14/14] refs: reuse iterators when determining refname availability","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-17T15:50:28Z","receivedAt":"2025-02-17T15:50:40Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When verifying whether refnames are available we have to verify whether\nany reference exists that is nested under the current reference. E.g.\ngiven a reference \"refs/heads/foo\", we must make sure that there is no\nother reference \"refs/heads/foo/*\".\n\nThis check is performed using a ref iterator with the prefix set to the\nnested reference namespace. Until now it used to not be possible to\nreseek iterators, so we always had to reallocate the iterator for every\nsingle reference we're about to check. This keeps us from reusing state\nthat the iterator may have and that may make it work more efficiently.\n\nRefactor the logic to reseek iterators. This leads to a speedup with the\nreftable backend, which is the only backend that knows to batch refname\navailability checks:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 39.8 ms ± 0.9 ms [User: 29.7 ms, System: 9.8 ms]\n Range (min … max): 38.4 ms … 42.0 ms 62 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 31.9 ms ± 1.1 ms [User: 27.0 ms, System: 4.5 ms]\n Range (min … max): 29.8 ms … 34.3 ms 74 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.25 ± 0.05 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n\nSigned-off-by: Patrick Steinhardt \n---\n refs.c | 12 +++++++-----\n 1 file changed, 7 insertions(+), 5 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 8eff60a2186..6cbb9decdb0 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2555,8 +2555,13 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \t\tif (!initial_transaction) {\n \t\t\tint ok;\n \n-\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n-\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n+\t\t\tif (!iter) {\n+\t\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n+\t\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n+\t\t\t} else if (ref_iterator_seek(iter, dirname.buf) < 0) {\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\n \t\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\t\t\tif (skip &&\n \t\t\t\t string_list_has_string(skip, iter->refname))\n@@ -2569,9 +2574,6 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \n \t\t\tif (ok != ITER_DONE)\n \t\t\t\tBUG(\"error while iterating over references\");\n-\n-\t\t\tref_iterator_free(iter);\n-\t\t\titer = NULL;\n \t\t}\n \n \t\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n\n-- \n2.48.1.666.gff9fcf71b7.dirty\n\n"},{"id":"512609","messageId":"CAOLa=ZRJLDKKu4R0TVn7z4LVZ6pirCN+5VpiehJ2grdfHGe7ew@mail.gmail.com","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-3-a2b6d87a24af@pks.im","subject":"Re: [PATCH 03/14] builtin/update-ref: skip ambiguity checks when parsing object IDs","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-02-18T16:04:29Z","receivedAt":"2025-02-18T16:04:31Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n[snip]\n\n> diff --git a/builtin/update-ref.c b/builtin/update-ref.c\n> index 4d35bdc4b4b..ac330748244 100644\n> --- a/builtin/update-ref.c\n> +++ b/builtin/update-ref.c\n> @@ -179,7 +179,8 @@ static int parse_next_oid(const char **next, const char *end,\n> \t\t(*next)++;\n> \t\t*next = parse_arg(*next, &arg);\n> \t\tif (arg.len) {\n> -\t\t\tif (repo_get_oid(the_repository, arg.buf, oid))\n> +\t\t\tif (repo_get_oid_with_flags(the_repository, arg.buf, oid,\n> +\t\t\t\t\t\t GET_OID_HASH_SKIP_AMBIGUITY_CHECK))\n> \t\t\t\tgoto invalid;\n> \t\t} else {\n> \t\t\t/* Without -z, an empty value means all zeros: */\n> @@ -197,7 +198,8 @@ static int parse_next_oid(const char **next, const char *end,\n> \t\t*next += arg.len;\n>\n> \t\tif (arg.len) {\n> -\t\t\tif (repo_get_oid(the_repository, arg.buf, oid))\n> +\t\t\tif (repo_get_oid_with_flags(the_repository, arg.buf, oid,\n> +\t\t\t\t\t\t GET_OID_HASH_SKIP_AMBIGUITY_CHECK))\n> \t\t\t\tgoto invalid;\n> \t\t} else if (flags & PARSE_SHA1_ALLOW_EMPTY) {\n> \t\t\t/* With -z, treat an empty value as all zeros: */\n\nSo the above two instances are used within the individual sub-commands\nfor `--stdin` mode. The symref commands use `parse_refname()` for\nparsing refnames, so all good.\n\n> @@ -772,7 +774,8 @@ int cmd_update_ref(int argc,\n> \t\trefname = argv[0];\n> \t\tvalue = argv[1];\n> \t\toldval = argv[2];\n> -\t\tif (repo_get_oid(the_repository, value, &oid))\n> +\t\tif (repo_get_oid_with_flags(the_repository, value, &oid,\n> +\t\t\t\t\t GET_OID_HASH_SKIP_AMBIGUITY_CHECK))\n> \t\t\tdie(\"%s: not a valid SHA1\", value);\n> \t}\n>\n> @@ -783,7 +786,8 @@ int cmd_update_ref(int argc,\n> \t\t\t * must not already exist:\n> \t\t\t */\n> \t\t\toidclr(&oldoid, the_repository->hash_algo);\n> -\t\telse if (repo_get_oid(the_repository, oldval, &oldoid))\n> +\t\telse if (repo_get_oid_with_flags(the_repository, oldval, &oldoid,\n> +\t\t\t\t\t\t GET_OID_HASH_SKIP_AMBIGUITY_CHECK))\n> \t\t\tdie(\"%s: not a valid old SHA1\", oldval);\n> \t}\n>\n\nThis is when the user uses 'git update-ref' directly. Makes sense.\n\n>\n> --\n> 2.48.1.666.gff9fcf71b7.dirty\n"},{"id":"512610","messageId":"CAOLa=ZQC+UXQGjOqot=pTopkd8mOjduixQ=rBnsis9g_3_HOqw@mail.gmail.com","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-6-a2b6d87a24af@pks.im","subject":"Re: [PATCH 06/14] refs: stop re-verifying common prefixes for availability","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-02-18T16:12:05Z","receivedAt":"2025-02-18T16:12:07Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> One of the checks done by `refs_verify_refnames_available()` is whether\n> any of the prefixes of a reference already exists. For example, given a\n> reference \"refs/heads/main\", we'd check whether \"refs/heads\" or \"refs\"\n> already exist, and if so we'd abort the transaction.\n>\n> When updating multiple references at once, this check is performed for\n> each of the references individually. Consequently, because references\n> tend to have common prefixes like \"refs/heads/\" or refs/tags/\", we\n> evaluate the availability of these prefixes repeatedly. Naturally this\n> is a waste of compute, as the availability of those prefixes should in\n> general not change in the middle of a transaction. And if it would,\n> backends would notice at a later point in time.\n>\n> Optimize this pattern by storing prefixes in a `strset` so that we can\n> trivially track those prefixes that we have already checked. This leads\n> to a significant speedup when creating many references that all share a\n> common prefix:\n>\n> Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n> Time (mean ± σ): 63.1 ms ± 1.8 ms [User: 41.0 ms, System: 21.6 ms]\n> Range (min … max): 60.6 ms … 69.5 ms 38 runs\n>\n> Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n> Time (mean ± σ): 40.0 ms ± 1.3 ms [User: 29.3 ms, System: 10.3 ms]\n> Range (min … max): 38.1 ms … 47.3 ms 61 runs\n>\n> Summary\n> update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n> 1.58 ± 0.07 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n>\n> Note that the same speedup cannot be observed for the \"files\" backend\n> because it still performs availability check per reference.\n>\n\nIn the previous commit, you started using the new function in the\nreftable backend, can we not make a similar change to the files backend?\n\n> Signed-off-by: Patrick Steinhardt \n> ---\n> refs.c | 12 ++++++++++++\n> 1 file changed, 12 insertions(+)\n>\n> diff --git a/refs.c b/refs.c\n> index 5a9b0f2fa1e..eaf41421f50 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -2476,6 +2476,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n> {\n> \tstruct strbuf dirname = STRBUF_INIT;\n> \tstruct strbuf referent = STRBUF_INIT;\n> +\tstruct strset dirnames;\n> \tint ret = -1;\n>\n> \t/*\n> @@ -2485,6 +2486,8 @@ int refs_verify_refnames_available(struct ref_store *refs,\n>\n> \tassert(err);\n>\n> +\tstrset_init(&dirnames);\n> +\n> \tfor (size_t i = 0; i < refnames->nr; i++) {\n> \t\tconst char *refname = refnames->items[i].string;\n> \t\tconst char *extra_refname;\n> @@ -2514,6 +2517,14 @@ int refs_verify_refnames_available(struct ref_store *refs,\n> \t\t\tif (skip && string_list_has_string(skip, dirname.buf))\n> \t\t\t\tcontinue;\n>\n> +\t\t\t/*\n> +\t\t\t * If we've already seen the directory we don't need to\n> +\t\t\t * process it again. Skip it to avoid checking checking\n> +\t\t\t * common prefixes like \"refs/heads/\" repeatedly.\n> +\t\t\t */\n> +\t\t\tif (!strset_add(&dirnames, dirname.buf))\n> +\t\t\t\tcontinue;\n> +\n\nThis was simple and neat. Nice.\n\n> \t\t\tif (!initial_transaction &&\n> \t\t\t !refs_read_raw_ref(refs, dirname.buf, &oid, &referent,\n> \t\t\t\t\t &type, &ignore_errno)) {\n> @@ -2574,6 +2585,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n> cleanup:\n> \tstrbuf_release(&referent);\n> \tstrbuf_release(&dirname);\n> +\tstrset_clear(&dirnames);\n> \treturn ret;\n> }\n>\n>\n> --\n> 2.48.1.666.gff9fcf71b7.dirty\n"},{"id":"512618","messageId":"Z7S6xzmPb3lK-SdT@ArchLinux","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-7-a2b6d87a24af@pks.im","subject":"Re: [PATCH 07/14] refs/iterator: separate lifecycle from iteration","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-02-18T16:52:23Z","receivedAt":"2025-02-18T16:52:26Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Mon, Feb 17, 2025 at 04:50:21PM +0100, Patrick Steinhardt wrote:\n> The ref and reflog iterators have their lifecycle attached to iteration:\n> once the iterator reaches its end, it is automatically released and the\n> caller doesn't have to care about that anymore. When the iterator should\n> be released before it has been exhausted, callers must explicitly abort\n> the iterator via `ref_iterator_abort()`.\n> \n> This lifecycle is somewhat unusual in the Git codebase and creates two\n> problems:\n> \n> - Callsites need to be very careful about when exactly they call\n> `ref_iterator_abort()`, as calling the function is only valid when\n> the iterator itself still is. This leads to somewhat awkward calling\n> patterns in some situations.\n> \n\nIn what situations, the iterator has been disappeared when we call\n`ref_iterator_abort`? Why is this awkward? When reading, I am really\ncurious about this.\n\n> - It is impossible to reuse iterators and re-seek them to a different\n> prefix. This feature isn't supported by any iterator implementation\n> except for the reftable iterators anyway, but if it was implemented\n> it would allow us to optimize cases where we need to search for\n> specific references repeatedly by reusing internal state.\n> \n\nSo, the reason why we cannot reuse the iterator is that we will\ndeallocate the iterator? So, below we want to detangle the lifecycle. I\ndon't know whether my understanding is correct.\n\n> Detangle the lifecycle from iteration so that we don't deallocate the\n> iterator anymore once it is exhausted. Instead, callers are now expected\n> to always call a newly introduce `ref_iterator_free()` function that\n> deallocates the iterator and its internal state.\n> \n\nA design question: why do not just introduce a variable, for example,\n`unsigned int free` to indicate whether we need to free the iterator?\n\n> While at it, drop the return value of `ref_iterator_abort()`, which\n> wasn't really required by any of the iterator implementations anyway.\n> Furthermore, stop calling `base_ref_iterator_free()` in any of the\n> backends, but instead call it in `ref_iterator_free()`.\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n> builtin/clone.c | 2 +\n> dir-iterator.c | 24 +++++------\n> dir-iterator.h | 13 ++----\n> refs.c | 7 +++-\n> refs/debug.c | 9 ++---\n> refs/files-backend.c | 36 +++++------------\n> refs/iterator.c | 95 ++++++++++++++------------------------------\n> refs/packed-backend.c | 27 ++++++-------\n> refs/ref-cache.c | 9 ++---\n> refs/refs-internal.h | 31 +++++----------\n> refs/reftable-backend.c | 34 ++++------------\n> t/helper/test-dir-iterator.c | 1 +\n> 12 files changed, 99 insertions(+), 189 deletions(-)\n> \n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index fd001d800c6..ac3e84b2b18 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -426,6 +426,8 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n> \t\tstrbuf_setlen(src, src_len);\n> \t\tdie(_(\"failed to iterate over '%s'\"), src->buf);\n> \t}\n> +\n> +\tdir_iterator_free(iter);\n\nHere, we explicitly free the iterator. This is a must, because we will\nnever free the iterator anymore.\n\n> }\n> \n> static void clone_local(const char *src_repo, const char *dest_repo)\n> diff --git a/dir-iterator.c b/dir-iterator.c\n> index de619846f29..857e1d9bdaf 100644\n> --- a/dir-iterator.c\n> +++ b/dir-iterator.c\n> @@ -193,9 +193,9 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n> \n> \tif (S_ISDIR(iter->base.st.st_mode) && push_level(iter)) {\n> \t\tif (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC)\n> -\t\t\tgoto error_out;\n> +\t\t\treturn ITER_ERROR;\n\nHere, we just return `ITER_ERROR` instead of going `error_out` label\nwhich will call `dir_iterator_abort` function to free the iterator, we\nshould make the caller free.\n\n> \t\tif (iter->levels_nr == 0)\n> -\t\t\tgoto error_out;\n> +\t\t\treturn ITER_ERROR;\n> \t}\n> \n> \t/* Loop until we find an entry that we can give back to the caller. */\n> @@ -211,11 +211,11 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n> \t\t\tint ret = next_directory_entry(level->dir, iter->base.path.buf, &de);\n> \t\t\tif (ret < 0) {\n> \t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n> -\t\t\t\t\tgoto error_out;\n> +\t\t\t\t\treturn ITER_ERROR;\n> \t\t\t\tcontinue;\n> \t\t\t} else if (ret > 0) {\n> \t\t\t\tif (pop_level(iter) == 0)\n> -\t\t\t\t\treturn dir_iterator_abort(dir_iterator);\n> +\t\t\t\t\treturn ITER_DONE;\n\nInstead of calling `dir_iterator_abort`, we just return `ITER_DONE`.\nHowever, this does not make sense. We break the semantics of\n`ITER_DONE`. Let me cite the comment from \"iterator.h\":\n\n /*\n * The iterator is exhausted and has been freed.\n */\n #define ITER_DONE -1\n\nHowever, we don't free the iterator here.\n\n> \t\t\t\tcontinue;\n> \t\t\t}\n> \n> @@ -223,7 +223,7 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n> \t\t} else {\n> \t\t\tif (level->entries_idx >= level->entries.nr) {\n> \t\t\t\tif (pop_level(iter) == 0)\n> -\t\t\t\t\treturn dir_iterator_abort(dir_iterator);\n> +\t\t\t\t\treturn ITER_DONE;\n> \t\t\t\tcontinue;\n> \t\t\t}\n> \n> @@ -232,22 +232,21 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n> \n> \t\tif (prepare_next_entry_data(iter, name)) {\n> \t\t\tif (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC)\n> -\t\t\t\tgoto error_out;\n> +\t\t\t\treturn ITER_ERROR;\n> \t\t\tcontinue;\n> \t\t}\n> \n> \t\treturn ITER_OK;\n> \t}\n> -\n> -error_out:\n> -\tdir_iterator_abort(dir_iterator);\n> -\treturn ITER_ERROR;\n> }\n> \n> -int dir_iterator_abort(struct dir_iterator *dir_iterator)\n> +void dir_iterator_free(struct dir_iterator *dir_iterator)\n\nWe rename `dir_iterator_abort` to `dir_iterator_free`. And we drop the\nreturn value.\n\n> {\n> \tstruct dir_iterator_int *iter = (struct dir_iterator_int *)dir_iterator;\n> \n> +\tif (!iter)\n> +\t\treturn;\n> +\n\nMake sense, because we have no idea whether the `dir_iterator` is\nexhausted. So we need to check whether it is valid.\n\n> \tfor (; iter->levels_nr; iter->levels_nr--) {\n> \t\tstruct dir_iterator_level *level =\n> \t\t\t&iter->levels[iter->levels_nr - 1];\n> @@ -266,7 +265,6 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator)\n> \tfree(iter->levels);\n> \tstrbuf_release(&iter->base.path);\n> \tfree(iter);\n> -\treturn ITER_DONE;\n\nI am confused that why we don't provide `dir_iterator_release` for dir\niterator? For other ref related iterators, we will rename \"*abort\" to\n\"*release\" and drop the operation to free the iterator. However, for dir\niterator, we rename its \"*abort\" to \"*free\".\n\nI think this does not make sense. It causes inconsistency. Although dir\niterator does _not_ have \"peel\" method. But it does have \"advance\" and\n\"abort\" methods just like ref iterators.\n\nI think you have already considered this problem. I guess that's the\nreason why in the below comment you typed `dir_iterator_release` instead\nof `dir_iterator_free`.\n\n> }\n> \n> struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags)\n> @@ -301,7 +299,7 @@ struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags)\n> \treturn dir_iterator;\n> \n> error_out:\n> -\tdir_iterator_abort(dir_iterator);\n> +\tdir_iterator_free(dir_iterator);\n> \terrno = saved_errno;\n> \treturn NULL;\n> }\n> diff --git a/dir-iterator.h b/dir-iterator.h\n> index 6d438809b6e..01f51f6bac1 100644\n> --- a/dir-iterator.h\n> +++ b/dir-iterator.h\n> @@ -27,10 +27,8 @@\n> * goto error_handler;\n> *\n> * while ((ok = dir_iterator_advance(iter)) == ITER_OK) {\n> - * if (want_to_stop_iteration()) {\n> - * ok = dir_iterator_abort(iter);\n> + * if (want_to_stop_iteration())\n> * break;\n> - * }\n\nIs this correct? If `want_to_stop_iteration()` is true, the `ok` will be\n`ITER_OK` and then it breaks the loop to jump to check whether the `ok`\nis `ITER_DONE`. Of course it is not, it will call `handle_error`. At\nleast, we should assign `ok = ITER_DONE`.\n\n> *\n> * // Access information about the current path:\n> * if (S_ISDIR(iter->st.st_mode))\n> @@ -39,6 +37,7 @@\n> *\n> * if (ok != ITER_DONE)\n> * handle_error();\n> + * dir_iterator_release(iter);\n\nI think this is a typo. Should `dir_iterator_release` be\n`dir_iterator_free`?\n\n> *\n> * Callers are allowed to modify iter->path while they are working,\n> * but they must restore it to its original contents before calling\n> @@ -107,11 +106,7 @@ struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags);\n> */\n> int dir_iterator_advance(struct dir_iterator *iterator);\n> \n> -/*\n> - * End the iteration before it has been exhausted. Free the\n> - * dir_iterator and any associated resources and return ITER_DONE. On\n> - * error, free the dir_iterator and return ITER_ERROR.\n> - */\n> -int dir_iterator_abort(struct dir_iterator *iterator);\n> +/* Free the dir_iterator and any associated resources. */\n> +void dir_iterator_free(struct dir_iterator *iterator);\n> \n> #endif\n> diff --git a/refs.c b/refs.c\n> index eaf41421f50..8eff60a2186 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -2476,6 +2476,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n> {\n> \tstruct strbuf dirname = STRBUF_INIT;\n> \tstruct strbuf referent = STRBUF_INIT;\n> +\tstruct ref_iterator *iter = NULL;\n> \tstruct strset dirnames;\n> \tint ret = -1;\n> \n> @@ -2552,7 +2553,6 @@ int refs_verify_refnames_available(struct ref_store *refs,\n> \t\tstrbuf_addch(&dirname, '/');\n> \n> \t\tif (!initial_transaction) {\n> -\t\t\tstruct ref_iterator *iter;\n> \t\t\tint ok;\n> \n> \t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n> @@ -2564,12 +2564,14 @@ int refs_verify_refnames_available(struct ref_store *refs,\n> \n> \t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n> \t\t\t\t\t iter->refname, refname);\n> -\t\t\t\tref_iterator_abort(iter);\n> \t\t\t\tgoto cleanup;\n> \t\t\t}\n> \n> \t\t\tif (ok != ITER_DONE)\n> \t\t\t\tBUG(\"error while iterating over references\");\n> +\n> +\t\t\tref_iterator_free(iter);\n> +\t\t\titer = NULL;\n> \t\t}\n> \n> \t\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n> @@ -2586,6 +2588,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n> \tstrbuf_release(&referent);\n> \tstrbuf_release(&dirname);\n> \tstrset_clear(&dirnames);\n> +\tref_iterator_free(iter);\n> \treturn ret;\n> }\n> \n> diff --git a/refs/debug.c b/refs/debug.c\n> index fbc4df08b43..a9786da4ba1 100644\n> --- a/refs/debug.c\n> +++ b/refs/debug.c\n> @@ -179,19 +179,18 @@ static int debug_ref_iterator_peel(struct ref_iterator *ref_iterator,\n> \treturn res;\n> }\n> \n> -static int debug_ref_iterator_abort(struct ref_iterator *ref_iterator)\n> +static void debug_ref_iterator_release(struct ref_iterator *ref_iterator)\n\nHere, we rename \"*abort\" to \"*release\" for debug ref and call the\ncorresponding `release` method.\n\n> {\n> \tstruct debug_ref_iterator *diter =\n> \t\t(struct debug_ref_iterator *)ref_iterator;\n> -\tint res = diter->iter->vtable->abort(diter->iter);\n> -\ttrace_printf_key(&trace_refs, \"iterator_abort: %d\\n\", res);\n> -\treturn res;\n> +\tditer->iter->vtable->release(diter->iter);\n> +\ttrace_printf_key(&trace_refs, \"iterator_abort\\n\");\n> }\n> \n> static struct ref_iterator_vtable debug_ref_iterator_vtable = {\n> \t.advance = debug_ref_iterator_advance,\n> \t.peel = debug_ref_iterator_peel,\n> -\t.abort = debug_ref_iterator_abort,\n> +\t.release = debug_ref_iterator_release,\n> };\n> \n> static struct ref_iterator *\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index 29f08dced40..9511b6f3448 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -919,10 +919,6 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> \t\treturn ITER_OK;\n> \t}\n> \n> -\titer->iter0 = NULL;\n> -\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n> -\t\tok = ITER_ERROR;\n> -\n\nWe don't abort the iterator in `advance`. Is this because we want to\nreuse this iterator?\n\n> \treturn ok;\n> }\n> \n\n[snip]\n\n> @@ -935,23 +931,17 @@ static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n> \treturn ref_iterator_peel(iter->iter0, peeled);\n> }\n> \n> -int ref_iterator_abort(struct ref_iterator *ref_iterator)\n> +void ref_iterator_free(struct ref_iterator *ref_iterator)\n> {\n> -\treturn ref_iterator->vtable->abort(ref_iterator);\n> +\tif (ref_iterator) {\n> +\t\tref_iterator->vtable->release(ref_iterator);\n> +\t\t/* Help make use-after-free bugs fail quickly: */\n> +\t\tref_iterator->vtable = NULL;\n> +\t\tfree(ref_iterator);\n> +\t}\n> }\n> \n\nSo, when calling `ref_iterator_free`, we will call corresponding\n\"release\" method to release the resources associated and then we free\nthe iterator. Would this be too complicated? From my view, we could just\nmake the `ref_iterator_abort` name unchanged but add a new variable such\nas \"unsigned int free_iterator\". And we change each \"abort\" callback to\navoid free the iterator. This would be much simpler.\n\n[snip]\n\n> diff --git a/refs/ref-cache.c b/refs/ref-cache.c\n> index 02f09e4df88..6457e02c1ea 100644\n> --- a/refs/ref-cache.c\n> +++ b/refs/ref-cache.c\n> @@ -409,7 +409,7 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> \t\tif (++level->index == level->dir->nr) {\n> \t\t\t/* This level is exhausted; pop up a level */\n> \t\t\tif (--iter->levels_nr == 0)\n> -\t\t\t\treturn ref_iterator_abort(ref_iterator);\n> +\t\t\t\treturn ITER_DONE;\n\nAs I have said, simply return `ITER_DONE` breaks the semantics of the\n`ITER_DONE`.\n\n> \n> \t\t\tcontinue;\n> \t\t}\n> @@ -452,21 +452,18 @@ static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n> \treturn peel_object(iter->repo, ref_iterator->oid, peeled) ? -1 : 0;\n> }\n> \n> -static int cache_ref_iterator_abort(struct ref_iterator *ref_iterator)\n> +static void cache_ref_iterator_release(struct ref_iterator *ref_iterator)\n> {\n> \tstruct cache_ref_iterator *iter =\n> \t\t(struct cache_ref_iterator *)ref_iterator;\n> -\n> \tfree((char *)iter->prefix);\n> \tfree(iter->levels);\n> -\tbase_ref_iterator_free(ref_iterator);\n> -\treturn ITER_DONE;\n> }\n> \n> static struct ref_iterator_vtable cache_ref_iterator_vtable = {\n> \t.advance = cache_ref_iterator_advance,\n> \t.peel = cache_ref_iterator_peel,\n> -\t.abort = cache_ref_iterator_abort\n> +\t.release = cache_ref_iterator_release,\n> };\n> \n> struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\n> diff --git a/refs/refs-internal.h b/refs/refs-internal.h\n> index aaab711bb96..27ff822cf43 100644\n> --- a/refs/refs-internal.h\n> +++ b/refs/refs-internal.h\n> @@ -273,11 +273,11 @@ enum do_for_each_ref_flags {\n> * the next reference and returns ITER_OK. The data pointed at by\n> * refname and oid belong to the iterator; if you want to retain them\n> * after calling ref_iterator_advance() again or calling\n> - * ref_iterator_abort(), you must make a copy. When the iteration has\n> + * ref_iterator_free(), you must make a copy. When the iteration has\n> * been exhausted, ref_iterator_advance() releases any resources\n> * associated with the iteration, frees the ref_iterator object, and\n> * returns ITER_DONE. If you want to abort the iteration early, call\n> - * ref_iterator_abort(), which also frees the ref_iterator object and\n> + * ref_iterator_free(), which also frees the ref_iterator object and\n> * any associated resources. If there was an internal error advancing\n> * to the next entry, ref_iterator_advance() aborts the iteration,\n> * frees the ref_iterator, and returns ITER_ERROR.\n> @@ -292,10 +292,8 @@ enum do_for_each_ref_flags {\n> * struct ref_iterator *iter = ...;\n> *\n> * while ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n> - * if (want_to_stop_iteration()) {\n> - * ok = ref_iterator_abort(iter);\n> + * if (want_to_stop_iteration())\n> * break;\n> - * }\n\nI also think here we have problem, at least we should set `ok =\nITER_DONE`.\n\n> *\n> * // Access information about the current reference:\n> * if (!(iter->flags & REF_ISSYMREF))\n> @@ -307,6 +305,7 @@ enum do_for_each_ref_flags {\n> *\n> * if (ok != ITER_DONE)\n> * handle_error();\n> + * ref_iterator_free(iter);\n> */\n> struct ref_iterator {\n> \tstruct ref_iterator_vtable *vtable;\n> @@ -333,12 +332,8 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator);\n> int ref_iterator_peel(struct ref_iterator *ref_iterator,\n> \t\t struct object_id *peeled);\n> \n> -/*\n> - * End the iteration before it has been exhausted, freeing the\n> - * reference iterator and any associated resources and returning\n> - * ITER_DONE. If the abort itself failed, return ITER_ERROR.\n> - */\n> -int ref_iterator_abort(struct ref_iterator *ref_iterator);\n> +/* Free the reference iterator and any associated resources. */\n> +void ref_iterator_free(struct ref_iterator *ref_iterator);\n> \n> /*\n> * An iterator over nothing (its first ref_iterator_advance() call\n> @@ -438,13 +433,6 @@ struct ref_iterator *prefix_ref_iterator_begin(struct ref_iterator *iter0,\n> void base_ref_iterator_init(struct ref_iterator *iter,\n> \t\t\t struct ref_iterator_vtable *vtable);\n> \n> -/*\n> - * Base class destructor for ref_iterators. Destroy the ref_iterator\n> - * part of iter and shallow-free the object. This is meant to be\n> - * called only by the destructors of derived classes.\n> - */\n> -void base_ref_iterator_free(struct ref_iterator *iter);\n> -\n\nOK, here we delete `base_ref_iterator_free`. Because we will use\n`ref_iterator_free`.\n\n> /* Virtual function declarations for ref_iterators: */\n> \n> /*\n> @@ -463,15 +451,14 @@ typedef int ref_iterator_peel_fn(struct ref_iterator *ref_iterator,\n> \n> /*\n> * Implementations of this function should free any resources specific\n> - * to the derived class, then call base_ref_iterator_free() to clean\n> - * up and free the ref_iterator object.\n> + * to the derived class.\n> */\n> -typedef int ref_iterator_abort_fn(struct ref_iterator *ref_iterator);\n> +typedef void ref_iterator_release_fn(struct ref_iterator *ref_iterator);\n\nSo here why we name this function to `release` is that we want to\nrelease any resource related to iterator but we don't free iterator\nitself.\n\n[snip]\n\n\n---\n\nSome my thinking after reading this whole patch:\n\n1. We somehow misuses the \"ITER_DONE\". We either need to adjust its\nmeaning (we just delete the iterator is freed part) or introduces a new\nstate to represent that the iteration is done but we don't release its\nresource and free the iterator itself.\n\n2. I don't think we need to make things complicated. From my\nunderstanding, the motivation here is that we don't want to `advance`\ncallback to call `abort` callback. I want to ask an _important_ question\nhere: what is the motivation we rename `abort` to `release` in the first\nplace? As far as I know, we only call this callback in the newly created\n\"ref_iterator_free\". Although release may be more accurate, this change\ntruly causes overhead of this patch.\n\n3. If the motivation is that we don't want to `advance` callback to call\n`abort` callback. I think we could just let the user call `abort`\ncallback for the following two situations:\n\n 1. We have exhausted the iteration. It returns `ITER_OK`.\n 2. We encountered the error, it returns `ITER_ERROR`.\n\nAnd we give the freedom to the caller. It's their duty to call\n`ref_iterator_abort` which cleans the resource and free the iterator.\n\nWriting here, I have always thought that there is a situation that we\njust want to release the resources but not want to free the iterator\nitself. That's why I am wondering why just add a new variable to do.\nHowever, if we just want to make the lifecycle out, we just delete each\n\"abort\" code where it frees the iterator. And we free the iterator in\nthe \"ref_iterator_abort\". Should this be enough?\n\nThanks,\nJialuo\n"},{"id":"512622","messageId":"Z7S-_iA73twblpvP@tapette.crustytoothpaste.net","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im","subject":"Re: [PATCH 00/14] refs: batch refname availability checks","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2025-02-18T17:10:22Z","receivedAt":"2025-02-18T17:10:30Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2025-02-17 at 15:50:14, Patrick Steinhardt wrote:\n> But more importantly, this refactoring also has a positive effect when\n> updating references in a repository with preexisting refs, which I\n> consider to be the more realistic scenario. The following benchmark\n> creates 10k refs with 100k preexisting refs.\n> \n> With the \"files\" backend we see a modest improvement:\n> \n> Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = master)\n> Time (mean ± σ): 470.1 ms ± 5.4 ms [User: 104.5 ms, System: 363.1 ms]\n> Range (min … max): 465.7 ms … 484.3 ms 10 runs\n> \n> Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n> Time (mean ± σ): 407.8 ms ± 5.4 ms [User: 66.0 ms, System: 340.0 ms]\n> Range (min … max): 399.9 ms … 417.6 ms 10 runs\n> \n> Summary\n> update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n> 1.15 ± 0.02 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = master)\n> \n> But with the \"reftable\" backend we see an almost 5x improvement, where\n> it's now ~15x faster than the \"files\" backend:\n> \n> Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = master)\n> Time (mean ± σ): 153.9 ms ± 2.0 ms [User: 96.5 ms, System: 56.6 ms]\n> Range (min … max): 150.5 ms … 158.4 ms 18 runs\n> \n> Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n> Time (mean ± σ): 32.2 ms ± 1.2 ms [User: 27.6 ms, System: 4.3 ms]\n> Range (min … max): 29.8 ms … 38.6 ms 71 runs\n> \n> Summary\n> update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n> 4.78 ± 0.19 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = master)\n\nI'm glad to see this performance speedup. That's a really nice\nimprovement.\n\n> The series is structured as follows:\n> \n> - Patches 1 to 4 implement the logic to skip ambiguity checks in\n> git-update-ref(1).\n> \n> - Patch 5 and 6 introduce batched checks.\n> \n> - Patch 7 deduplicates the ref prefix checks.\n> \n> - Patch 8 to 14 implement the infrastructure to reseek iterators.\n> \n> - Patch 15 starts to reuse iterators for nested ref checks.\n\nI took a look at this series and I didn't find anything that stood out\nto me as a problem. I will say that the reftable code isn't my forte,\nso please don't take this as a formal review, but I am definitely\npositive on the series in the general sense.\n-- \nbrian m. carlson (they/them or he/him)\nToronto, Ontario, CA\n"},{"id":"512623","messageId":"CAOLa=ZQzEwWci350dhhTFiG3WZB67JUB4O9L3coiXeexYGTNZw@mail.gmail.com","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-7-a2b6d87a24af@pks.im","subject":"Re: [PATCH 07/14] refs/iterator: separate lifecycle from iteration","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-02-18T17:13:55Z","receivedAt":"2025-02-18T17:13:57Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> The ref and reflog iterators have their lifecycle attached to iteration:\n> once the iterator reaches its end, it is automatically released and the\n> caller doesn't have to care about that anymore. When the iterator should\n> be released before it has been exhausted, callers must explicitly abort\n> the iterator via `ref_iterator_abort()`.\n>\n> This lifecycle is somewhat unusual in the Git codebase and creates two\n> problems:\n>\n> - Callsites need to be very careful about when exactly they call\n> `ref_iterator_abort()`, as calling the function is only valid when\n> the iterator itself still is. This leads to somewhat awkward calling\n> patterns in some situations.\n>\n> - It is impossible to reuse iterators and re-seek them to a different\n> prefix. This feature isn't supported by any iterator implementation\n> except for the reftable iterators anyway, but if it was implemented\n> it would allow us to optimize cases where we need to search for\n> specific references repeatedly by reusing internal state.\n>\n> Detangle the lifecycle from iteration so that we don't deallocate the\n> iterator anymore once it is exhausted. Instead, callers are now expected\n> to always call a newly introduce `ref_iterator_free()` function that\n> deallocates the iterator and its internal state.\n>\n> While at it, drop the return value of `ref_iterator_abort()`, which\n> wasn't really required by any of the iterator implementations anyway.\n> Furthermore, stop calling `base_ref_iterator_free()` in any of the\n> backends, but instead call it in `ref_iterator_free()`.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> builtin/clone.c | 2 +\n> dir-iterator.c | 24 +++++------\n> dir-iterator.h | 13 ++----\n> refs.c | 7 +++-\n> refs/debug.c | 9 ++---\n> refs/files-backend.c | 36 +++++------------\n> refs/iterator.c | 95 ++++++++++++++------------------------------\n> refs/packed-backend.c | 27 ++++++-------\n> refs/ref-cache.c | 9 ++---\n> refs/refs-internal.h | 31 +++++----------\n> refs/reftable-backend.c | 34 ++++------------\n> t/helper/test-dir-iterator.c | 1 +\n> 12 files changed, 99 insertions(+), 189 deletions(-)\n>\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index fd001d800c6..ac3e84b2b18 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -426,6 +426,8 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n> \t\tstrbuf_setlen(src, src_len);\n> \t\tdie(_(\"failed to iterate over '%s'\"), src->buf);\n> \t}\n> +\n> +\tdir_iterator_free(iter);\n> }\n>\n\nA bit puzzled to see `dir_iterator_*` change here, I'm assuming it's\nlinked to the 'files-backend' and perhaps similar to the changes\nmentioned about `ref_iterator_*` in the commit message. Would be nice to\ncall out in the commit message too.\n\n[snip]\n\n> @@ -223,7 +223,7 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n> \t\t} else {\n> \t\t\tif (level->entries_idx >= level->entries.nr) {\n> \t\t\t\tif (pop_level(iter) == 0)\n> -\t\t\t\t\treturn dir_iterator_abort(dir_iterator);\n> +\t\t\t\t\treturn ITER_DONE;\n> \t\t\t\tcontinue;\n> \t\t\t}\n>\n> @@ -232,22 +232,21 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n>\n> \t\tif (prepare_next_entry_data(iter, name)) {\n> \t\t\tif (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC)\n> -\t\t\t\tgoto error_out;\n> +\t\t\t\treturn ITER_ERROR;\n> \t\t\tcontinue;\n> \t\t}\n>\n> \t\treturn ITER_OK;\n> \t}\n> -\n> -error_out:\n> -\tdir_iterator_abort(dir_iterator);\n> -\treturn ITER_ERROR;\n> }\n\nOkay yeah, we're getting rid of `dir_iterator_abort` so potentially add\n`dir_iterator_free` below\n\n>\n> -int dir_iterator_abort(struct dir_iterator *dir_iterator)\n> +void dir_iterator_free(struct dir_iterator *dir_iterator)\n> {\n> \tstruct dir_iterator_int *iter = (struct dir_iterator_int *)dir_iterator;\n>\n> +\tif (!iter)\n> +\t\treturn;\n> +\n> \tfor (; iter->levels_nr; iter->levels_nr--) {\n> \t\tstruct dir_iterator_level *level =\n> \t\t\t&iter->levels[iter->levels_nr - 1];\n> @@ -266,7 +265,6 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator)\n> \tfree(iter->levels);\n> \tstrbuf_release(&iter->base.path);\n> \tfree(iter);\n> -\treturn ITER_DONE;\n> }\n>\n> struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags)\n\nOkay this makes sense!\n\n[snip]\n\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index 29f08dced40..9511b6f3448 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -919,10 +919,6 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> \t\treturn ITER_OK;\n> \t}\n>\n> -\titer->iter0 = NULL;\n> -\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n> -\t\tok = ITER_ERROR;\n> -\n>\n\nSince we're explicitly going to call `ref_iterator_free`, this makes sense.\n\n> \treturn ok;\n> }\n>\n> @@ -935,23 +931,17 @@ static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n> \treturn ref_iterator_peel(iter->iter0, peeled);\n> }\n>\n> -static int files_ref_iterator_abort(struct ref_iterator *ref_iterator)\n> +static void files_ref_iterator_release(struct ref_iterator *ref_iterator)\n> {\n> \tstruct files_ref_iterator *iter =\n> \t\t(struct files_ref_iterator *)ref_iterator;\n> -\tint ok = ITER_DONE;\n> -\n> -\tif (iter->iter0)\n> -\t\tok = ref_iterator_abort(iter->iter0);\n> -\n> -\tbase_ref_iterator_free(ref_iterator);\n> -\treturn ok;\n> +\tref_iterator_free(iter->iter0);\n> }\n>\n\nI like how much more cleaner it looks now.\n\n[snip]\n\n> diff --git a/refs/packed-backend.c b/refs/packed-backend.c\n> index a7b6f74b6e3..38a1956d1a8 100644\n> --- a/refs/packed-backend.c\n> +++ b/refs/packed-backend.c\n> @@ -954,9 +954,6 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> \t\treturn ITER_OK;\n> \t}\n>\n> -\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n> -\t\tok = ITER_ERROR;\n> -\n> \treturn ok;\n> }\n>\n\nThe merged_iterator is used to combine the files and packed backend\niterators to provide a uniform view over them. Likewise the changes here\nseem similar too.\n\n> @@ -976,23 +973,19 @@ static int packed_ref_iterator_peel(struct ref_iterator *ref_iterator,\n> \t}\n> }\n>\n> -static int packed_ref_iterator_abort(struct ref_iterator *ref_iterator)\n> +static void packed_ref_iterator_release(struct ref_iterator *ref_iterator)\n> {\n> \tstruct packed_ref_iterator *iter =\n> \t\t(struct packed_ref_iterator *)ref_iterator;\n> -\tint ok = ITER_DONE;\n> -\n> \tstrbuf_release(&iter->refname_buf);\n> \tfree(iter->jump);\n> \trelease_snapshot(iter->snapshot);\n> -\tbase_ref_iterator_free(ref_iterator);\n> -\treturn ok;\n> }\n>\n> static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n> \t.advance = packed_ref_iterator_advance,\n> \t.peel = packed_ref_iterator_peel,\n> -\t.abort = packed_ref_iterator_abort\n> +\t.release = packed_ref_iterator_release,\n> };\n>\n> static int jump_list_entry_cmp(const void *va, const void *vb)\n> @@ -1362,8 +1355,10 @@ static int write_with_updates(struct packed_ref_store *refs,\n> \t */\n> \titer = packed_ref_iterator_begin(&refs->base, \"\", NULL,\n> \t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n> -\tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n> +\tif ((ok = ref_iterator_advance(iter)) != ITER_OK) {\n> +\t\tref_iterator_free(iter);\n\nNit: Since we don't return early here, wouldn't the `ref_iterator_free`\nat the end of the function be sufficient? I think the only early return\nwhen `iter == NULL` is towards the end of the function, where it might\nbe better to add a `goto error`.\n\n[snip]\n"},{"id":"512663","messageId":"Z7XF5pGsa42jrIcN@pks.im","threadId":"62965","inReplyTo":"Z7S6xzmPb3lK-SdT@ArchLinux","subject":"Re: [PATCH 07/14] refs/iterator: separate lifecycle from iteration","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T11:52:06Z","receivedAt":"2025-02-19T11:52:11Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Feb 19, 2025 at 12:52:23AM +0800, shejialuo wrote:\n> On Mon, Feb 17, 2025 at 04:50:21PM +0100, Patrick Steinhardt wrote:\n> > The ref and reflog iterators have their lifecycle attached to iteration:\n> > once the iterator reaches its end, it is automatically released and the\n> > caller doesn't have to care about that anymore. When the iterator should\n> > be released before it has been exhausted, callers must explicitly abort\n> > the iterator via `ref_iterator_abort()`.\n> > \n> > This lifecycle is somewhat unusual in the Git codebase and creates two\n> > problems:\n> > \n> > - Callsites need to be very careful about when exactly they call\n> > `ref_iterator_abort()`, as calling the function is only valid when\n> > the iterator itself still is. This leads to somewhat awkward calling\n> > patterns in some situations.\n> > \n> \n> In what situations, the iterator has been disappeared when we call\n> `ref_iterator_abort`? Why is this awkward? When reading, I am really\n> curious about this.\n\nIt leads to patterns where you have to call `ref_iterator_abort()`, but\nonly in a subset of code paths. You need to be really careful about the\nlifetime of the iterator, and I had several occasions where I was doing\nthe wrong thing because I missed this subtlety. Compared to that, it is\ntrivial (and less code as demonstrated by the patch) to unconditionally\ncall `ref_iterator_free()`.\n\n> > - It is impossible to reuse iterators and re-seek them to a different\n> > prefix. This feature isn't supported by any iterator implementation\n> > except for the reftable iterators anyway, but if it was implemented\n> > it would allow us to optimize cases where we need to search for\n> > specific references repeatedly by reusing internal state.\n> > \n> \n> So, the reason why we cannot reuse the iterator is that we will\n> deallocate the iterator? So, below we want to detangle the lifecycle. I\n> don't know whether my understanding is correct.\n\nYup, your understanding is correct. A deallocated iterator cannot be\nreused.\n\n> > Detangle the lifecycle from iteration so that we don't deallocate the\n> > iterator anymore once it is exhausted. Instead, callers are now expected\n> > to always call a newly introduce `ref_iterator_free()` function that\n> > deallocates the iterator and its internal state.\n> > \n> \n> A design question: why do not just introduce a variable, for example,\n> `unsigned int free` to indicate whether we need to free the iterator?\n\nBecause that would make the code even more subtle than it already is.\nDoing things unconditionally is always easier to reason about than doing\nthem conditionally.\n\n> > diff --git a/dir-iterator.c b/dir-iterator.c\n> > index de619846f29..857e1d9bdaf 100644\n> > --- a/dir-iterator.c\n> > +++ b/dir-iterator.c\n> > @@ -211,11 +211,11 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n> > \t\t\tint ret = next_directory_entry(level->dir, iter->base.path.buf, &de);\n> > \t\t\tif (ret < 0) {\n> > \t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n> > -\t\t\t\t\tgoto error_out;\n> > +\t\t\t\t\treturn ITER_ERROR;\n> > \t\t\t\tcontinue;\n> > \t\t\t} else if (ret > 0) {\n> > \t\t\t\tif (pop_level(iter) == 0)\n> > -\t\t\t\t\treturn dir_iterator_abort(dir_iterator);\n> > +\t\t\t\t\treturn ITER_DONE;\n> \n> Instead of calling `dir_iterator_abort`, we just return `ITER_DONE`.\n> However, this does not make sense. We break the semantics of\n> `ITER_DONE`. Let me cite the comment from \"iterator.h\":\n> \n> /*\n> * The iterator is exhausted and has been freed.\n> */\n> #define ITER_DONE -1\n> \n> However, we don't free the iterator here.\n\nOh, yeah, I'll have to adapt that comment, thanks for pointing it out.\n\n> > @@ -266,7 +265,6 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator)\n> > \tfree(iter->levels);\n> > \tstrbuf_release(&iter->base.path);\n> > \tfree(iter);\n> > -\treturn ITER_DONE;\n> \n> I am confused that why we don't provide `dir_iterator_release` for dir\n> iterator? For other ref related iterators, we will rename \"*abort\" to\n> \"*release\" and drop the operation to free the iterator. However, for dir\n> iterator, we rename its \"*abort\" to \"*free\".\n> \n> I think this does not make sense. It causes inconsistency. Although dir\n> iterator does _not_ have \"peel\" method. But it does have \"advance\" and\n> \"abort\" methods just like ref iterators.\n> \n> I think you have already considered this problem. I guess that's the\n> reason why in the below comment you typed `dir_iterator_release` instead\n> of `dir_iterator_free`.\n\nThe `dir_iterator` itself is already being inconsistent with every other\niterator that we have because it is not a `ref_iterator`. It is _used_\nto implement ref iterators, but doesn't provide the same interface. And\nbecause of that we cannot rely on `ref_iterator_free()` to free it, but\nmust instead provide `dir_iterator_free()` to do so.\n\nI have amended the commit message to explain why this one is special.\n\n> > diff --git a/dir-iterator.h b/dir-iterator.h\n> > index 6d438809b6e..01f51f6bac1 100644\n> > --- a/dir-iterator.h\n> > +++ b/dir-iterator.h\n> > @@ -27,10 +27,8 @@\n> > * goto error_handler;\n> > *\n> > * while ((ok = dir_iterator_advance(iter)) == ITER_OK) {\n> > - * if (want_to_stop_iteration()) {\n> > - * ok = dir_iterator_abort(iter);\n> > + * if (want_to_stop_iteration())\n> > * break;\n> > - * }\n> \n> Is this correct? If `want_to_stop_iteration()` is true, the `ok` will be\n> `ITER_OK` and then it breaks the loop to jump to check whether the `ok`\n> is `ITER_DONE`. Of course it is not, it will call `handle_error`. At\n> least, we should assign `ok = ITER_DONE`.\n\nTrue, fixed.\n\n> > *\n> > * // Access information about the current path:\n> > * if (S_ISDIR(iter->st.st_mode))\n> > @@ -39,6 +37,7 @@\n> > *\n> > * if (ok != ITER_DONE)\n> > * handle_error();\n> > + * dir_iterator_release(iter);\n> \n> I think this is a typo. Should `dir_iterator_release` be\n> `dir_iterator_free`?\n\nYup, fixed.\n\n> > diff --git a/refs/files-backend.c b/refs/files-backend.c\n> > index 29f08dced40..9511b6f3448 100644\n> > --- a/refs/files-backend.c\n> > +++ b/refs/files-backend.c\n> > @@ -919,10 +919,6 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> > \t\treturn ITER_OK;\n> > \t}\n> > \n> > -\titer->iter0 = NULL;\n> > -\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n> > -\t\tok = ITER_ERROR;\n> > -\n> \n> We don't abort the iterator in `advance`. Is this because we want to\n> reuse this iterator?\n\nExactly, this is basically the gist of this whole patch.\n\n> > @@ -935,23 +931,17 @@ static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n> > \treturn ref_iterator_peel(iter->iter0, peeled);\n> > }\n> > \n> > -int ref_iterator_abort(struct ref_iterator *ref_iterator)\n> > +void ref_iterator_free(struct ref_iterator *ref_iterator)\n> > {\n> > -\treturn ref_iterator->vtable->abort(ref_iterator);\n> > +\tif (ref_iterator) {\n> > +\t\tref_iterator->vtable->release(ref_iterator);\n> > +\t\t/* Help make use-after-free bugs fail quickly: */\n> > +\t\tref_iterator->vtable = NULL;\n> > +\t\tfree(ref_iterator);\n> > +\t}\n> > }\n> > \n> \n> So, when calling `ref_iterator_free`, we will call corresponding\n> \"release\" method to release the resources associated and then we free\n> the iterator. Would this be too complicated? From my view, we could just\n> make the `ref_iterator_abort` name unchanged but add a new variable such\n> as \"unsigned int free_iterator\". And we change each \"abort\" callback to\n> avoid free the iterator. This would be much simpler.\n\nI think adding a separate variable to track whether or not things should\nbe freed would make things way more complicated. I would claim the\nopposite: the fact that the patch removes 100 lines of code demonstrates\nquite neatly that the new design is way simpler and needs less logic.\n\nIt's also simpler to reason about from my perspective: you allocate an\niterator, you free it. No conditionals, no nothing.\n\n> [snip]\n> \n> > diff --git a/refs/ref-cache.c b/refs/ref-cache.c\n> > index 02f09e4df88..6457e02c1ea 100644\n> > --- a/refs/ref-cache.c\n> > +++ b/refs/ref-cache.c\n> > @@ -409,7 +409,7 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> > \t\tif (++level->index == level->dir->nr) {\n> > \t\t\t/* This level is exhausted; pop up a level */\n> > \t\t\tif (--iter->levels_nr == 0)\n> > -\t\t\t\treturn ref_iterator_abort(ref_iterator);\n> > +\t\t\t\treturn ITER_DONE;\n> \n> As I have said, simply return `ITER_DONE` breaks the semantics of the\n> `ITER_DONE`.\n\nIt doesn't: `ITER_DONE` indicates that the iterator has been exhausted.\nThe current comment is stale though, as it claims that the iterator will\nalso have been free'd.\n\n> > diff --git a/refs/refs-internal.h b/refs/refs-internal.h\n> > index aaab711bb96..27ff822cf43 100644\n> > --- a/refs/refs-internal.h\n> > +++ b/refs/refs-internal.h\n> > @@ -292,10 +292,8 @@ enum do_for_each_ref_flags {\n> > * struct ref_iterator *iter = ...;\n> > *\n> > * while ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n> > - * if (want_to_stop_iteration()) {\n> > - * ok = ref_iterator_abort(iter);\n> > + * if (want_to_stop_iteration())\n> > * break;\n> > - * }\n> \n> I also think here we have problem, at least we should set `ok =\n> ITER_DONE`.\n\nYup, same as in the other comment indeed.\n\n> Some my thinking after reading this whole patch:\n> \n> 1. We somehow misuses the \"ITER_DONE\". We either need to adjust its\n> meaning (we just delete the iterator is freed part) or introduces a new\n> state to represent that the iteration is done but we don't release its\n> resource and free the iterator itself.\n\nYup, addressed.\n\n> 2. I don't think we need to make things complicated. From my\n> understanding, the motivation here is that we don't want to `advance`\n> callback to call `abort` callback. I want to ask an _important_ question\n> here: what is the motivation we rename `abort` to `release` in the first\n> place? As far as I know, we only call this callback in the newly created\n> \"ref_iterator_free\". Although release may be more accurate, this change\n> truly causes overhead of this patch.\n\nWe have to touch up all of these functions anyway, so renaming them at\nthe same point doesn't feel like it adds any more complexity.\n\n> 3. If the motivation is that we don't want to `advance` callback to call\n> `abort` callback. I think we could just let the user call `abort`\n> callback for the following two situations:\n> \n> 1. We have exhausted the iteration. It returns `ITER_OK`.\n\nThis is impossible because the caller wouldn't be able to discern an\nexhausted iterator from an iterator that still has entries. We have to\ndiscern `ITER_OK` and `ITER_DONE`.\n\n> 2. We encountered the error, it returns `ITER_ERROR`.\n\nYup.\n\n> And we give the freedom to the caller. It's their duty to call\n> `ref_iterator_abort` which cleans the resource and free the iterator.\n\nYup.\n\n> Writing here, I have always thought that there is a situation that we\n> just want to release the resources but not want to free the iterator\n> itself. That's why I am wondering why just add a new variable to do.\n> However, if we just want to make the lifecycle out, we just delete each\n> \"abort\" code where it frees the iterator. And we free the iterator in\n> the \"ref_iterator_abort\". Should this be enough?\n\nAs mentioned, I don't think a new variable would lead to a simplified\narchitecture. With re-seekable iterators the caller is the one who needs\nto control whether the iterator should or should not be free'd, as they\nare the only one who knows whether they want to reuse it. So making it\nthe responsibility of the callers to release the iterator is the proper\nway to do it.\n\nI also don't quite see the complexity argument. The patch rather clearly\nshows that we're _reducing_ complexity as it allows us to drop around a\n100 lines of code. We can stop worrying about whether or not we have to\ncall `ref_iterator_abort()` and don't have to worry about any conditions\nthat leak from the iterator subsystem to the callers. We just free the\niterator once we're done with it and call it a day.\n\nThanks for your input!\n\nPatrick\n"},{"id":"512664","messageId":"Z7XF6-FbetYl2AXZ@pks.im","threadId":"62965","inReplyTo":"CAOLa=ZQzEwWci350dhhTFiG3WZB67JUB4O9L3coiXeexYGTNZw@mail.gmail.com","subject":"Re: [PATCH 07/14] refs/iterator: separate lifecycle from iteration","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T11:52:11Z","receivedAt":"2025-02-19T11:52:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Feb 18, 2025 at 09:13:55AM -0800, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> > diff --git a/builtin/clone.c b/builtin/clone.c\n> > index fd001d800c6..ac3e84b2b18 100644\n> > --- a/builtin/clone.c\n> > +++ b/builtin/clone.c\n> > @@ -426,6 +426,8 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n> > \t\tstrbuf_setlen(src, src_len);\n> > \t\tdie(_(\"failed to iterate over '%s'\"), src->buf);\n> > \t}\n> > +\n> > +\tdir_iterator_free(iter);\n> > }\n> >\n> \n> A bit puzzled to see `dir_iterator_*` change here, I'm assuming it's\n> linked to the 'files-backend' and perhaps similar to the changes\n> mentioned about `ref_iterator_*` in the commit message. Would be nice to\n> call out in the commit message too.\n\nYeah, that's the reason. I've added a note to the commit message.\n\n> > diff --git a/refs/packed-backend.c b/refs/packed-backend.c\n> > index a7b6f74b6e3..38a1956d1a8 100644\n> > --- a/refs/packed-backend.c\n> > +++ b/refs/packed-backend.c\n> > @@ -1362,8 +1355,10 @@ static int write_with_updates(struct packed_ref_store *refs,\n> > \t */\n> > \titer = packed_ref_iterator_begin(&refs->base, \"\", NULL,\n> > \t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n> > -\tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n> > +\tif ((ok = ref_iterator_advance(iter)) != ITER_OK) {\n> > +\t\tref_iterator_free(iter);\n> \n> Nit: Since we don't return early here, wouldn't the `ref_iterator_free`\n> at the end of the function be sufficient? I think the only early return\n> when `iter == NULL` is towards the end of the function, where it might\n> be better to add a `goto error`.\n\nYeah, the code here could definitely be improved with the new semantics.\nBut I was aiming to keep the required refactoring work at callsites to\nthe bare minimum, so I'd rather prefer to keep this unchanged.\n\nPatrick\n"},{"id":"512665","messageId":"Z7XF77yKUgENdbA-@pks.im","threadId":"62965","inReplyTo":"CAOLa=ZQC+UXQGjOqot=pTopkd8mOjduixQ=rBnsis9g_3_HOqw@mail.gmail.com","subject":"Re: [PATCH 06/14] refs: stop re-verifying common prefixes for availability","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T11:52:15Z","receivedAt":"2025-02-19T11:52:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Feb 18, 2025 at 08:12:05AM -0800, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> \n> > One of the checks done by `refs_verify_refnames_available()` is whether\n> > any of the prefixes of a reference already exists. For example, given a\n> > reference \"refs/heads/main\", we'd check whether \"refs/heads\" or \"refs\"\n> > already exist, and if so we'd abort the transaction.\n> >\n> > When updating multiple references at once, this check is performed for\n> > each of the references individually. Consequently, because references\n> > tend to have common prefixes like \"refs/heads/\" or refs/tags/\", we\n> > evaluate the availability of these prefixes repeatedly. Naturally this\n> > is a waste of compute, as the availability of those prefixes should in\n> > general not change in the middle of a transaction. And if it would,\n> > backends would notice at a later point in time.\n> >\n> > Optimize this pattern by storing prefixes in a `strset` so that we can\n> > trivially track those prefixes that we have already checked. This leads\n> > to a significant speedup when creating many references that all share a\n> > common prefix:\n> >\n> > Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n> > Time (mean ± σ): 63.1 ms ± 1.8 ms [User: 41.0 ms, System: 21.6 ms]\n> > Range (min … max): 60.6 ms … 69.5 ms 38 runs\n> >\n> > Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n> > Time (mean ± σ): 40.0 ms ± 1.3 ms [User: 29.3 ms, System: 10.3 ms]\n> > Range (min … max): 38.1 ms … 47.3 ms 61 runs\n> >\n> > Summary\n> > update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n> > 1.58 ± 0.07 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n> >\n> > Note that the same speedup cannot be observed for the \"files\" backend\n> > because it still performs availability check per reference.\n> >\n> \n> In the previous commit, you started using the new function in the\n> reftable backend, can we not make a similar change to the files backend?\n\nIt's quite a bit more intricate in the \"files\" backend because the\ncreation of the lockfiles and calls to `refs_verify_refname_available()`\nare intertwined with one another:\n\n - `lock_raw_ref()` verifies availability when it hits either EEXISTS\n or EISDIR to generate error messages. This is probably nothing we\n have to care about too much, as these are irrelevant in the good\n path.\n\n - `lock_raw_ref()` also verifies availability though in the case where\n it _could_ create the lockfile to check whether it is conflicting\n with any packed refs. This one could potentially be batched. It's a\n curious thing in the first place as we do not have the packed refs\n locked at this point in time, so this check might even be racy.\n\n - `lock_ref_oid_basic()` also checks availability with packed refs, so\n this is another case where we might batch the checks. But the\n function is only used when copying/renaming references or when\n expiring reflogs, so it won't be called for many refs.\n\n - We call it in `refs_refname_ref_available()`, which is executed when\n copying/renaming references. Uninteresting due to the same reason as\n the previous entry.\n\n - We call it in `files_transaction_finish_initial()`. This one should\n be rather trivial to batch. Again though, no locking with packed\n refs, so the checks are racy.\n\nSo... it's a bit more complicated here compared to the reftable backend,\nand I didn't feel like opening a can of worms with the potentially-racy\nchecks with the packed backend.\n\nAnyway, I think we still can and probably should use the new mechanism\nin two cases:\n\n - During normal transactions to batch the availability checks with the\n packed backend. I will have to ignore the issue of a potential race,\n but other than that the change is straight forward and the result is\n a slight speedup:\n\n Benchmark 1: update-ref: create many refs (preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 393.4 ms ± 4.0 ms [User: 64.1 ms, System: 327.5 ms]\n Range (min … max): 387.8 ms … 398.7 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 373.3 ms ± 3.4 ms [User: 48.8 ms, System: 322.7 ms]\n Range (min … max): 368.7 ms … 378.6 ms 10 runs\n\n Summary\n update-ref: create many refs (preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.05 ± 0.01 times faster than update-ref: create many refs (preexisting = 100000, new = 10000, revision = HEAD~)\n\n - During the initial transaction. Here the change is even more trivial\n and we can also fix the race as we eventually lock the packed-refs\n file anyway. This leads to a noticeable speedup when migrating from\n the reftable backend to the files backend:\n\n Benchmark 1: migrate reftable:files (refcount = 1000000, revision = HEAD~)\n Time (mean ± σ): 980.6 ms ± 10.9 ms [User: 801.8 ms, System: 172.4 ms]\n Range (min … max): 964.7 ms … 995.3 ms 10 runs\n\n Benchmark 2: migrate reftable:files (refcount = 1000000, revision = HEAD)\n Time (mean ± σ): 739.7 ms ± 6.6 ms [User: 551.9 ms, System: 181.9 ms]\n Range (min … max): 727.9 ms … 747.2 ms 10 runs\n\n Summary\n migrate reftable:files (refcount = 1000000, revision = HEAD) ran\n 1.33 ± 0.02 times faster than migrate reftable:files (refcount = 1000000, revision = HEAD~)\n\nI'll include these changes in the next version, thanks for questioning\nwhy I skipped over the \"files\" backend.\n\nPatrick\n"},{"id":"512666","messageId":"Z7XRX1gfo942QdNR@ArchLinux","threadId":"62965","inReplyTo":"Z7XF5pGsa42jrIcN@pks.im","subject":"Re: [PATCH 07/14] refs/iterator: separate lifecycle from iteration","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-02-19T12:41:03Z","receivedAt":"2025-02-19T12:41:04Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Wed, Feb 19, 2025 at 12:52:06PM +0100, Patrick Steinhardt wrote:\n> > > @@ -935,23 +931,17 @@ static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n> > > \treturn ref_iterator_peel(iter->iter0, peeled);\n> > > }\n> > > \n> > > -int ref_iterator_abort(struct ref_iterator *ref_iterator)\n> > > +void ref_iterator_free(struct ref_iterator *ref_iterator)\n> > > {\n> > > -\treturn ref_iterator->vtable->abort(ref_iterator);\n> > > +\tif (ref_iterator) {\n> > > +\t\tref_iterator->vtable->release(ref_iterator);\n> > > +\t\t/* Help make use-after-free bugs fail quickly: */\n> > > +\t\tref_iterator->vtable = NULL;\n> > > +\t\tfree(ref_iterator);\n> > > +\t}\n> > > }\n> > > \n> > \n> > So, when calling `ref_iterator_free`, we will call corresponding\n> > \"release\" method to release the resources associated and then we free\n> > the iterator. Would this be too complicated? From my view, we could just\n> > make the `ref_iterator_abort` name unchanged but add a new variable such\n> > as \"unsigned int free_iterator\". And we change each \"abort\" callback to\n> > avoid free the iterator. This would be much simpler.\n> \n> I think adding a separate variable to track whether or not things should\n> be freed would make things way more complicated. I would claim the\n> opposite: the fact that the patch removes 100 lines of code demonstrates\n> quite neatly that the new design is way simpler and needs less logic.\n> \n\nI agree with you with the current implementation, we don't need to worry\nabout calling \"abort\" callback in \"advance\". And the logic is simpler.\nWhen writing this comment, I have thought that there is a situation that\nwe just want to call \"release\" callback. So, that's the reason why I\nhave asked you why not just add a new variable.\n\nHowever, we don't call \"release\" callback expect in `ref_iterator_free`.\n\n[snip]\n\n> > 2. I don't think we need to make things complicated. From my\n> > understanding, the motivation here is that we don't want to `advance`\n> > callback to call `abort` callback. I want to ask an _important_ question\n> > here: what is the motivation we rename `abort` to `release` in the first\n> > place? As far as I know, we only call this callback in the newly created\n> > \"ref_iterator_free\". Although release may be more accurate, this change\n> > truly causes overhead of this patch.\n> \n> We have to touch up all of these functions anyway, so renaming them at\n> the same point doesn't feel like it adds any more complexity.\n> \n\nI will explain this in the later.\n\n[snip]\n\n> > Writing here, I have always thought that there is a situation that we\n> > just want to release the resources but not want to free the iterator\n> > itself. That's why I am wondering why just add a new variable to do.\n> > However, if we just want to make the lifecycle out, we just delete each\n> > \"abort\" code where it frees the iterator. And we free the iterator in\n> > the \"ref_iterator_abort\". Should this be enough?\n> \n> As mentioned, I don't think a new variable would lead to a simplified\n> architecture. With re-seekable iterators the caller is the one who needs\n> to control whether the iterator should or should not be free'd, as they\n> are the only one who knows whether they want to reuse it. So making it\n> the responsibility of the callers to release the iterator is the proper\n> way to do it.\n> \n\nYes, actually I think you have misunderstood my meaning here. I have\nabandoned the idea to add a new variable when writing here. After\nreading through the patch, I know that your motivation. My point is\n\"However, ...\" part.\n\n> I also don't quite see the complexity argument. The patch rather clearly\n> shows that we're _reducing_ complexity as it allows us to drop around a\n> 100 lines of code. We can stop worrying about whether or not we have to\n> call `ref_iterator_abort()` and don't have to worry about any conditions\n> that leak from the iterator subsystem to the callers. We just free the\n> iterator once we're done with it and call it a day.\n> \n\nYes, I agree with you that we truly reduce complexity here. And as you\nhave said, the user allocate the iterator and free the iterator. With\nthis, we make call sequence clearer.\n\nBut there is one thing I want to argue with. I don't think we need to\nrename \"abort\" callback to \"release\" and also \"ref_iterator_abort\" to\n\"ref_iterator_free\" for the following reasons:\n\n1. We never call \"release\" expect in the \"ref_iterator_free\" function.\nFor other exposed functions \"ref_iterator_advance\", \"ref_iterator_peel\"\nand the original \"ref_iterator_abort\". We will just call the registered\ncallback \"advance\", \"peel\" or \"abort\" via virtual table. I somehow think\nwe should follow this pattern. But I don't know actually.\n2. When I read the patch yesterday, I really wonder what is the\ndifference between \"release\" and \"free\". Why do we only change the\n\"ref_iterator_abort\" to \"ref_iterator_free\" but for the callback, we\nrename \"abort\" to \"release\". I know that you want to distinguish to\nemphasis that we won't free the iterator but only release its resource\nfor ref iterator. But could abort also mean this?\n\nThanks,\nJialuo\n"},{"id":"512668","messageId":"Z7XVoeNMjXJnlrmX@pks.im","threadId":"62965","inReplyTo":"Z7XRX1gfo942QdNR@ArchLinux","subject":"Re: [PATCH 07/14] refs/iterator: separate lifecycle from iteration","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T12:59:13Z","receivedAt":"2025-02-19T12:59:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Feb 19, 2025 at 08:41:03PM +0800, shejialuo wrote:\n> But there is one thing I want to argue with. I don't think we need to\n> rename \"abort\" callback to \"release\" and also \"ref_iterator_abort\" to\n> \"ref_iterator_free\" for the following reasons:\n> \n> 1. We never call \"release\" expect in the \"ref_iterator_free\" function.\n> For other exposed functions \"ref_iterator_advance\", \"ref_iterator_peel\"\n> and the original \"ref_iterator_abort\". We will just call the registered\n> callback \"advance\", \"peel\" or \"abort\" via virtual table. I somehow think\n> we should follow this pattern. But I don't know actually.\n> 2. When I read the patch yesterday, I really wonder what is the\n> difference between \"release\" and \"free\". Why do we only change the\n> \"ref_iterator_abort\" to \"ref_iterator_free\" but for the callback, we\n> rename \"abort\" to \"release\". I know that you want to distinguish to\n> emphasis that we won't free the iterator but only release its resource\n> for ref iterator. But could abort also mean this?\n\nThe difference between \"release\" and \"free\" is explicitly documented in\nour CodingGuidelines. Quoting the relevant parts:\n\n - `S_release()` releases a structure's contents without freeing the\n structure.\n\n - `S_free()` releases a structure's contents and frees the\n structure.\n\nSo following these coding guidelines, we have to call the underlying\nimplementations that are specific to the iterators `release()` because\nthey don't free the iterator itself. And because the generic part _does_\nfree the iterator itself in addition to releasing its state, it has to\nbe called `free()`.\n\nRegarding the question why to even rename `ref_iterator_abort()` itself:\nthis is done to avoid confusion going forward. Previously it really only\nhad to be called when you actually wanted to abort an ongoing iteration\nover its yielded references. This is not the case anymore, and now you\nhave to call it unconditionally after you're done with the iterator. So\nwhile the naming previously made sense, now it doesn't anymore.\n\nPatrick\n"},{"id":"512669","messageId":"Z7XXcp9o0fb7FloS@ArchLinux","threadId":"62965","inReplyTo":"Z7XVoeNMjXJnlrmX@pks.im","subject":"Re: [PATCH 07/14] refs/iterator: separate lifecycle from iteration","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-02-19T13:06:58Z","receivedAt":"2025-02-19T13:06:59Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Wed, Feb 19, 2025 at 01:59:13PM +0100, Patrick Steinhardt wrote:\n> On Wed, Feb 19, 2025 at 08:41:03PM +0800, shejialuo wrote:\n> > But there is one thing I want to argue with. I don't think we need to\n> > rename \"abort\" callback to \"release\" and also \"ref_iterator_abort\" to\n> > \"ref_iterator_free\" for the following reasons:\n> > \n> > 1. We never call \"release\" expect in the \"ref_iterator_free\" function.\n> > For other exposed functions \"ref_iterator_advance\", \"ref_iterator_peel\"\n> > and the original \"ref_iterator_abort\". We will just call the registered\n> > callback \"advance\", \"peel\" or \"abort\" via virtual table. I somehow think\n> > we should follow this pattern. But I don't know actually.\n> > 2. When I read the patch yesterday, I really wonder what is the\n> > difference between \"release\" and \"free\". Why do we only change the\n> > \"ref_iterator_abort\" to \"ref_iterator_free\" but for the callback, we\n> > rename \"abort\" to \"release\". I know that you want to distinguish to\n> > emphasis that we won't free the iterator but only release its resource\n> > for ref iterator. But could abort also mean this?\n> \n> The difference between \"release\" and \"free\" is explicitly documented in\n> our CodingGuidelines. Quoting the relevant parts:\n> \n> - `S_release()` releases a structure's contents without freeing the\n> structure.\n> \n> - `S_free()` releases a structure's contents and frees the\n> structure.\n> \n> So following these coding guidelines, we have to call the underlying\n> implementations that are specific to the iterators `release()` because\n> they don't free the iterator itself. And because the generic part _does_\n> free the iterator itself in addition to releasing its state, it has to\n> be called `free()`.\n> \n\nMake sense.\n\n> Regarding the question why to even rename `ref_iterator_abort()` itself:\n> this is done to avoid confusion going forward. Previously it really only\n> had to be called when you actually wanted to abort an ongoing iteration\n> over its yielded references. This is not the case anymore, and now you\n> have to call it unconditionally after you're done with the iterator. So\n> while the naming previously made sense, now it doesn't anymore.\n> \n\nGood point, I didn't realise this part. Thanks for the detailed\nexplanation. I will continue to review the later patches. However, I\nwon't touch the oid part, because I am not familiar with this. By the\nway, I think we miss out one thing in this patch:\n\nWe forget to free the dir iterator defined in the\n\"files-backend.c::files_fsck_refs_dir\". I have just remembered that I\nuse dir iterator when checking the ref consistency.\n\nThanks,\nJialuo\n\n> Patrick\n"},{"id":"512681","messageId":"Z7XZ02obxTyj1tGO@pks.im","threadId":"62965","inReplyTo":"Z7XXcp9o0fb7FloS@ArchLinux","subject":"Re: [PATCH 07/14] refs/iterator: separate lifecycle from iteration","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T13:17:07Z","receivedAt":"2025-02-19T13:17:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Feb 19, 2025 at 09:06:58PM +0800, shejialuo wrote:\n> On Wed, Feb 19, 2025 at 01:59:13PM +0100, Patrick Steinhardt wrote:\n> > Regarding the question why to even rename `ref_iterator_abort()` itself:\n> > this is done to avoid confusion going forward. Previously it really only\n> > had to be called when you actually wanted to abort an ongoing iteration\n> > over its yielded references. This is not the case anymore, and now you\n> > have to call it unconditionally after you're done with the iterator. So\n> > while the naming previously made sense, now it doesn't anymore.\n> > \n> \n> Good point, I didn't realise this part. Thanks for the detailed\n> explanation. I will continue to review the later patches. However, I\n> won't touch the oid part, because I am not familiar with this. By the\n> way, I think we miss out one thing in this patch:\n> \n> We forget to free the dir iterator defined in the\n> \"files-backend.c::files_fsck_refs_dir\". I have just remembered that I\n> use dir iterator when checking the ref consistency.\n\nHm, good point. Why doesn't CI complain about this leak...? I'll\ninvestigate, thanks for the hint!\n\nPatrick\n"},{"id":"512682","messageId":"Z7Xaj78hBkMt00CN@pks.im","threadId":"62965","inReplyTo":"Z7XZ02obxTyj1tGO@pks.im","subject":"Re: [PATCH 07/14] refs/iterator: separate lifecycle from iteration","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T13:20:15Z","receivedAt":"2025-02-19T13:20:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Feb 19, 2025 at 02:17:07PM +0100, Patrick Steinhardt wrote:\n> On Wed, Feb 19, 2025 at 09:06:58PM +0800, shejialuo wrote:\n> > On Wed, Feb 19, 2025 at 01:59:13PM +0100, Patrick Steinhardt wrote:\n> > > Regarding the question why to even rename `ref_iterator_abort()` itself:\n> > > this is done to avoid confusion going forward. Previously it really only\n> > > had to be called when you actually wanted to abort an ongoing iteration\n> > > over its yielded references. This is not the case anymore, and now you\n> > > have to call it unconditionally after you're done with the iterator. So\n> > > while the naming previously made sense, now it doesn't anymore.\n> > > \n> > \n> > Good point, I didn't realise this part. Thanks for the detailed\n> > explanation. I will continue to review the later patches. However, I\n> > won't touch the oid part, because I am not familiar with this. By the\n> > way, I think we miss out one thing in this patch:\n> > \n> > We forget to free the dir iterator defined in the\n> > \"files-backend.c::files_fsck_refs_dir\". I have just remembered that I\n> > use dir iterator when checking the ref consistency.\n> \n> Hm, good point. Why doesn't CI complain about this leak...? I'll\n> investigate, thanks for the hint!\n\nWait, no, I had been looking at the wrong branch. We do free the\niterator:\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 11a620ea11a..859f1c11941 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3837,6 +3820,7 @@ static int files_fsck_refs_dir(struct ref_store *ref_store,\n ret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n\n out:\n+ dir_iterator_free(iter);\n strbuf_release(&sb);\n strbuf_release(&refname);\n return ret;\n\nPatrick\n"},{"id":"512683","messageId":"Z7XbP6O7Ul8Ln3WD@ArchLinux","threadId":"62965","inReplyTo":"Z7Xaj78hBkMt00CN@pks.im","subject":"Re: [PATCH 07/14] refs/iterator: separate lifecycle from iteration","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-02-19T13:23:11Z","receivedAt":"2025-02-19T13:23:12Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Wed, Feb 19, 2025 at 02:20:15PM +0100, Patrick Steinhardt wrote:\n> On Wed, Feb 19, 2025 at 02:17:07PM +0100, Patrick Steinhardt wrote:\n> > On Wed, Feb 19, 2025 at 09:06:58PM +0800, shejialuo wrote:\n> > > On Wed, Feb 19, 2025 at 01:59:13PM +0100, Patrick Steinhardt wrote:\n> > > > Regarding the question why to even rename `ref_iterator_abort()` itself:\n> > > > this is done to avoid confusion going forward. Previously it really only\n> > > > had to be called when you actually wanted to abort an ongoing iteration\n> > > > over its yielded references. This is not the case anymore, and now you\n> > > > have to call it unconditionally after you're done with the iterator. So\n> > > > while the naming previously made sense, now it doesn't anymore.\n> > > > \n> > > \n> > > Good point, I didn't realise this part. Thanks for the detailed\n> > > explanation. I will continue to review the later patches. However, I\n> > > won't touch the oid part, because I am not familiar with this. By the\n> > > way, I think we miss out one thing in this patch:\n> > > \n> > > We forget to free the dir iterator defined in the\n> > > \"files-backend.c::files_fsck_refs_dir\". I have just remembered that I\n> > > use dir iterator when checking the ref consistency.\n> > \n> > Hm, good point. Why doesn't CI complain about this leak...? I'll\n> > investigate, thanks for the hint!\n> \n> Wait, no, I had been looking at the wrong branch. We do free the\n> iterator:\n> \n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index 11a620ea11a..859f1c11941 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -3837,6 +3820,7 @@ static int files_fsck_refs_dir(struct ref_store *ref_store,\n> ret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n> \n> out:\n> + dir_iterator_free(iter);\n> strbuf_release(&sb);\n> strbuf_release(&refname);\n> return ret;\n> \n> Patrick\n\nOh, my mistake. I omit that part during review... Sorry here.\n\n"},{"id":"512684","messageId":"20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im","subject":"[PATCH v2 00/16] refs: batch refname availability checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T13:23:27Z","receivedAt":"2025-02-19T13:23:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series has been inspired by brian's report that the reftable\nbackend is significantly slower when writing many references compared to\nthe files backend. As explained in that thread, the underlying issue is\nthe design of tombstone references: when we first delete all references\nin a repository and then recreate them, we still have all the tombstones\nand thus we need to churn through all of them to figure out that they\nhave been deleted in the first place. The files backend does not have\nthis issue.\n\nI consider the benchmark itself to be kind of broken, as it stems from\nus deleting all refs and then recreating them. And if you pack refs in\nbetween then the \"reftable\" backend outperforms the \"files\" backend.\n\nBut there are a couple of opportunities here anyway. While we cannot\nmake the underlying issue of tombstones being less efficient go away,\nthis has prompted me to have a deeper look at where we spend all the\ntime. There are three ideas in this series:\n\n - git-update-ref(1) performs ambiguity checks for any full-size object\n ID, which triggers a lot of reads. This is somewhat pointless though\n given that the manpage explicitly points out that the command is\n about object IDs, even though it does know to parse refs. But being\n part of plumbing, emitting the warning here does not make a ton of\n sense, and favoring object IDs over references in these cases is the\n obvious thing to do anyway.\n\n - For each ref \"refs/heads/bar\", we need to verify that neither\n \"refs/heads\" nor \"refs\" exists. This was repeated for every refname,\n but because most refnames use common prefixes this made us re-check\n a lot of prefixes. This is addressed by using a `strset` of already\n checked prefixes.\n\n - For each ref \"refs/heads/bar\", we need to verify that no ref\n \"refs/heads/bar/*\" exists. We always created a new ref iterator for\n this check, which requires us to discard all internal state and then\n recreate it. The reftable library has already been refactored though\n to have reseekable iterators, so we backfill this functionality to\n all the other iterators and then reuse the iterator.\n\nWith the (somewhat broken) benchmark we see a small speedup with the\n\"files\" backend:\n\n Benchmark 1: update-ref (refformat = files, revision = master)\n Time (mean ± σ): 234.4 ms ± 1.9 ms [User: 75.6 ms, System: 157.2 ms]\n Range (min … max): 232.2 ms … 236.9 ms 10 runs\n\n Benchmark 2: update-ref (refformat = files, revision = HEAD)\n Time (mean ± σ): 184.2 ms ± 2.0 ms [User: 62.8 ms, System: 119.9 ms]\n Range (min … max): 181.1 ms … 187.0 ms 10 runs\n\n Summary\n update-ref (refformat = files, revision = HEAD) ran\n 1.27 ± 0.02 times faster than update-ref (refformat = files, revision = master)\n\nAnd a huge speedup with the \"reftable\" backend:\n\n Benchmark 1: update-ref (refformat = reftable, revision = master)\n Time (mean ± σ): 16.852 s ± 0.061 s [User: 16.754 s, System: 0.059 s]\n Range (min … max): 16.785 s … 16.982 s 10 runs\n\n Benchmark 2: update-ref (refformat = reftable, revision = HEAD)\n Time (mean ± σ): 2.230 s ± 0.009 s [User: 2.192 s, System: 0.029 s]\n Range (min … max): 2.215 s … 2.244 s 10 runs\n\n Summary\n update-ref (refformat = reftable, revision = HEAD) ran\n 7.56 ± 0.04 times faster than update-ref (refformat = reftable, revision = master)\n\nWe're still not up to speed with the \"files\" backend, but considerably\nbetter. Given that this is an extreme edge case and not reflective of\nthe general case I'm okay with this result for now.\n\nBut more importantly, this refactoring also has a positive effect when\nupdating references in a repository with preexisting refs, which I\nconsider to be the more realistic scenario. The following benchmark\ncreates 10k refs with 100k preexisting refs.\n\nWith the \"files\" backend we see a modest improvement:\n\n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = master)\n Time (mean ± σ): 478.4 ms ± 11.9 ms [User: 96.7 ms, System: 379.6 ms]\n Range (min … max): 465.4 ms … 496.6 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 388.5 ms ± 10.3 ms [User: 52.0 ms, System: 333.8 ms]\n Range (min … max): 376.5 ms … 403.1 ms 10 runs\n\n Summary\n update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.23 ± 0.04 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = master)\n\nBut with the \"reftable\" backend we see an almost 5x improvement, where\nit's now ~15x faster than the \"files\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = master)\n Time (mean ± σ): 153.9 ms ± 2.0 ms [User: 96.5 ms, System: 56.6 ms]\n Range (min … max): 150.5 ms … 158.4 ms 18 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 32.2 ms ± 1.2 ms [User: 27.6 ms, System: 4.3 ms]\n Range (min … max): 29.8 ms … 38.6 ms 71 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 4.78 ± 0.19 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = master)\n\nThe series is structured as follows:\n\n - Patches 1 to 4 implement the logic to skip ambiguity checks in\n git-update-ref(1).\n\n - Patch 5 to 8 introduce batched checks.\n\n - Patch 9 deduplicates the ref prefix checks.\n\n - Patch 10 to 16 implement the infrastructure to reseek iterators.\n\n - Patch 17 starts to reuse iterators for nested ref checks.\n\nChanges in v2:\n - Point out why we also have to touch up the `dir_iterator`.\n - Fix up the comment explaining `ITER_DONE`.\n - Fix up comments that show usage patterns of the ref and dir iterator\n interfaces.\n - Start batching availability checks in the \"files\" backend, as well.\n - Improve the commit message that drops the ambiguity check so that we\n also point to 25fba78d36b (cat-file: disable object/refname\n ambiguity check for batch mode, 2013-07-12).\n - Link to v1: https://lore.kernel.org/r/20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im\n\nThanks!\n\nPatrick\n\n[1]: \n\n---\nPatrick Steinhardt (16):\n object-name: introduce `repo_get_oid_with_flags()`\n object-name: allow skipping ambiguity checks in `get_oid()` family\n builtin/update-ref: skip ambiguity checks when parsing object IDs\n refs: introduce function to batch refname availability checks\n refs/reftable: batch refname availability checks\n refs/files: batch refname availability checks for normal transactions\n refs/files: batch refname availability checks for initial transactions\n refs: stop re-verifying common prefixes for availability\n refs/iterator: separate lifecycle from iteration\n refs/iterator: provide infrastructure to re-seek iterators\n refs/iterator: implement seeking for merged iterators\n refs/iterator: implement seeking for reftable iterators\n refs/iterator: implement seeking for ref-cache iterators\n refs/iterator: implement seeking for `packed-ref` iterators\n refs/iterator: implement seeking for \"files\" iterators\n refs: reuse iterators when determining refname availability\n\n builtin/clone.c | 2 +\n builtin/update-ref.c | 12 ++-\n dir-iterator.c | 24 +++---\n dir-iterator.h | 11 +--\n hash.h | 1 +\n iterator.h | 2 +-\n object-name.c | 18 +++--\n object-name.h | 6 ++\n refs.c | 186 ++++++++++++++++++++++++++-----------------\n refs.h | 12 +++\n refs/debug.c | 20 +++--\n refs/files-backend.c | 117 +++++++++++++++++----------\n refs/iterator.c | 145 +++++++++++++++++----------------\n refs/packed-backend.c | 89 ++++++++++++---------\n refs/ref-cache.c | 83 +++++++++++--------\n refs/refs-internal.h | 52 +++++++-----\n refs/reftable-backend.c | 85 +++++++++++---------\n t/helper/test-dir-iterator.c | 1 +\n 18 files changed, 519 insertions(+), 347 deletions(-)\n\nRange-diff versus v1:\n\n 1: 313d86f4274 = 1: 22de0f21a9f object-name: introduce `repo_get_oid_with_flags()`\n 2: a0ca8e62a81 ! 2: 4aa573a50d4 object-name: allow skipping ambiguity checks in `get_oid()` family\n @@ hash.h: struct object_id {\n #define GET_OID_ONLY_TO_DIE 04000\n #define GET_OID_REQUIRE_PATH 010000\n #define GET_OID_HASH_ANY 020000\n -+#define GET_OID_HASH_SKIP_AMBIGUITY_CHECK 040000\n ++#define GET_OID_SKIP_AMBIGUITY_CHECK 040000\n \n #define GET_OID_DISAMBIGUATORS \\\n \t(GET_OID_COMMIT | GET_OID_COMMITTISH | \\\n @@ object-name.c: static int get_oid_basic(struct repository *r, const char *str, i\n \n \tif (len == r->hash_algo->hexsz && !get_oid_hex(str, oid)) {\n -\t\tif (repo_settings_get_warn_ambiguous_refs(r) && warn_on_object_refname_ambiguity) {\n -+\t\tif (!(flags & GET_OID_HASH_SKIP_AMBIGUITY_CHECK) &&\n ++\t\tif (!(flags & GET_OID_SKIP_AMBIGUITY_CHECK) &&\n +\t\t repo_settings_get_warn_ambiguous_refs(r) &&\n +\t\t warn_on_object_refname_ambiguity) {\n \t\t\trefs_found = repo_dwim_ref(r, str, len, &tmp_oid, &real_ref, 0);\n 3: 7029057b07f ! 3: 073392c4371 builtin/update-ref: skip ambiguity checks when parsing object IDs\n @@ Commit message\n object ID that we parse according to the DWIM rules. This effect can be\n seen both with the \"files\" and \"reftable\" backend.\n \n + The issue is not unique to git-update-ref(1), but was also an issue in\n + git-cat-file(1), where it was addressed by disabling the ambiguity check\n + in 25fba78d36b (cat-file: disable object/refname ambiguity check for\n + batch mode, 2013-07-12).\n +\n Disable the warning in git-update-ref(1), which provides a significant\n speedup with both backends. The following benchmark creates 10000 new\n references with a 100000 preexisting refs with the \"files\" backend:\n @@ builtin/update-ref.c: static int parse_next_oid(const char **next, const char *e\n \t\tif (arg.len) {\n -\t\t\tif (repo_get_oid(the_repository, arg.buf, oid))\n +\t\t\tif (repo_get_oid_with_flags(the_repository, arg.buf, oid,\n -+\t\t\t\t\t\t GET_OID_HASH_SKIP_AMBIGUITY_CHECK))\n ++\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\t\tgoto invalid;\n \t\t} else {\n \t\t\t/* Without -z, an empty value means all zeros: */\n @@ builtin/update-ref.c: static int parse_next_oid(const char **next, const char *e\n \t\tif (arg.len) {\n -\t\t\tif (repo_get_oid(the_repository, arg.buf, oid))\n +\t\t\tif (repo_get_oid_with_flags(the_repository, arg.buf, oid,\n -+\t\t\t\t\t\t GET_OID_HASH_SKIP_AMBIGUITY_CHECK))\n ++\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\t\tgoto invalid;\n \t\t} else if (flags & PARSE_SHA1_ALLOW_EMPTY) {\n \t\t\t/* With -z, treat an empty value as all zeros: */\n @@ builtin/update-ref.c: int cmd_update_ref(int argc,\n \t\toldval = argv[2];\n -\t\tif (repo_get_oid(the_repository, value, &oid))\n +\t\tif (repo_get_oid_with_flags(the_repository, value, &oid,\n -+\t\t\t\t\t GET_OID_HASH_SKIP_AMBIGUITY_CHECK))\n ++\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\tdie(\"%s: not a valid SHA1\", value);\n \t}\n \n @@ builtin/update-ref.c: int cmd_update_ref(int argc,\n \t\t\toidclr(&oldoid, the_repository->hash_algo);\n -\t\telse if (repo_get_oid(the_repository, oldval, &oldoid))\n +\t\telse if (repo_get_oid_with_flags(the_repository, oldval, &oldoid,\n -+\t\t\t\t\t\t GET_OID_HASH_SKIP_AMBIGUITY_CHECK))\n ++\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\tdie(\"%s: not a valid old SHA1\", oldval);\n \t}\n \n 4: 768cb058d6e = 4: e61aab15188 refs: introduce function to batch refname availability checks\n 5: 6eed3559d68 ! 5: ef3df1044b4 refs/reftable: start using `refs_verify_refnames_available()`\n @@ Metadata\n Author: Patrick Steinhardt \n \n ## Commit message ##\n - refs/reftable: start using `refs_verify_refnames_available()`\n + refs/reftable: batch refname availability checks\n \n Refactor the \"reftable\" backend to batch the availability check for\n refnames. This does not yet have an effect on performance as we\n -: ----------- > 6: 822bfc7bdee refs/files: batch refname availability checks for normal transactions\n -: ----------- > 7: f5dc7eaa97e refs/files: batch refname availability checks for initial transactions\n 6: 5d78f40c460 ! 8: 20a74a89045 refs: stop re-verifying common prefixes for availability\n @@ Commit message\n \n Optimize this pattern by storing prefixes in a `strset` so that we can\n trivially track those prefixes that we have already checked. This leads\n - to a significant speedup when creating many references that all share a\n - common prefix:\n + to a significant speedup with the \"reftable\" backend when creating many\n + references that all share a common prefix:\n \n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 63.1 ms ± 1.8 ms [User: 41.0 ms, System: 21.6 ms]\n @@ Commit message\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.58 ± 0.07 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n \n - Note that the same speedup cannot be observed for the \"files\" backend\n - because it still performs availability check per reference.\n + For the \"files\" backend we see an improvement, but a much smaller one:\n +\n + Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n + Time (mean ± σ): 395.8 ms ± 5.3 ms [User: 63.6 ms, System: 330.5 ms]\n + Range (min … max): 387.0 ms … 404.6 ms 10 runs\n +\n + Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n + Time (mean ± σ): 386.0 ms ± 4.0 ms [User: 51.5 ms, System: 332.8 ms]\n + Range (min … max): 380.8 ms … 392.6 ms 10 runs\n +\n + Summary\n + update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n + 1.03 ± 0.02 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n +\n + This change also leads to a modest improvement when writing references\n + with \"initial\" semantics, for example when migrating references. The\n + following benchmarks are migrating 1m references from the \"reftable\" to\n + the \"files\" backend:\n +\n + Benchmark 1: migrate reftable:files (refcount = 1000000, revision = HEAD~)\n + Time (mean ± σ): 836.6 ms ± 5.6 ms [User: 645.2 ms, System: 185.2 ms]\n + Range (min … max): 829.6 ms … 845.9 ms 10 runs\n +\n + Benchmark 2: migrate reftable:files (refcount = 1000000, revision = HEAD)\n + Time (mean ± σ): 759.8 ms ± 5.1 ms [User: 574.9 ms, System: 178.9 ms]\n + Range (min … max): 753.1 ms … 768.8 ms 10 runs\n +\n + Summary\n + migrate reftable:files (refcount = 1000000, revision = HEAD) ran\n + 1.10 ± 0.01 times faster than migrate reftable:files (refcount = 1000000, revision = HEAD~)\n +\n + And vice versa:\n +\n + Benchmark 1: migrate files:reftable (refcount = 1000000, revision = HEAD~)\n + Time (mean ± σ): 870.7 ms ± 5.7 ms [User: 735.2 ms, System: 127.4 ms]\n + Range (min … max): 861.6 ms … 883.2 ms 10 runs\n +\n + Benchmark 2: migrate files:reftable (refcount = 1000000, revision = HEAD)\n + Time (mean ± σ): 799.1 ms ± 8.5 ms [User: 661.1 ms, System: 130.2 ms]\n + Range (min … max): 787.5 ms … 812.6 ms 10 runs\n +\n + Summary\n + migrate files:reftable (refcount = 1000000, revision = HEAD) ran\n + 1.09 ± 0.01 times faster than migrate files:reftable (refcount = 1000000, revision = HEAD~)\n +\n + The impact here is significantly smaller given that we don't perform any\n + reference reads with \"initial\" semantics, so the speedup only comes from\n + us doing less string list lookups.\n \n Signed-off-by: Patrick Steinhardt \n \n 7: a1da57bf37d ! 9: b6524ecfe0c refs/iterator: separate lifecycle from iteration\n @@ Commit message\n to always call a newly introduce `ref_iterator_free()` function that\n deallocates the iterator and its internal state.\n \n + Note that the `dir_iterator` is somewhat special because it does not\n + implement the `ref_iterator` interface, but is only used to implement\n + other iterators. Consequently, we have to provide `dir_iterator_free()`\n + instead of `dir_iterator_release()` as the allocated structure itself is\n + managed by the `dir_iterator` interfaces, as well, and not freed by\n + `ref_iterator_free()` like in all the other cases.\n +\n While at it, drop the return value of `ref_iterator_abort()`, which\n wasn't really required by any of the iterator implementations anyway.\n Furthermore, stop calling `base_ref_iterator_free()` in any of the\n @@ dir-iterator.c: struct dir_iterator *dir_iterator_begin(const char *path, unsign\n \n ## dir-iterator.h ##\n @@\n - * goto error_handler;\n *\n * while ((ok = dir_iterator_advance(iter)) == ITER_OK) {\n -- * if (want_to_stop_iteration()) {\n + * if (want_to_stop_iteration()) {\n - * ok = dir_iterator_abort(iter);\n -+ * if (want_to_stop_iteration())\n ++ * ok = ITER_DONE;\n * break;\n -- * }\n + * }\n *\n - * // Access information about the current path:\n - * if (S_ISDIR(iter->st.st_mode))\n @@\n *\n * if (ok != ITER_DONE)\n * handle_error();\n -+ * dir_iterator_release(iter);\n ++ * dir_iterator_free(iter);\n *\n * Callers are allowed to modify iter->path while they are working,\n * but they must restore it to its original contents before calling\n @@ dir-iterator.h: struct dir_iterator *dir_iterator_begin(const char *path, unsign\n \n #endif\n \n + ## iterator.h ##\n +@@\n + #define ITER_OK 0\n + \n + /*\n +- * The iterator is exhausted and has been freed.\n ++ * The iterator is exhausted.\n + */\n + #define ITER_DONE -1\n + \n +\n ## refs.c ##\n @@ refs.c: int refs_verify_refnames_available(struct ref_store *refs,\n {\n @@ refs/refs-internal.h: enum do_for_each_ref_flags {\n * to the next entry, ref_iterator_advance() aborts the iteration,\n * frees the ref_iterator, and returns ITER_ERROR.\n @@ refs/refs-internal.h: enum do_for_each_ref_flags {\n - * struct ref_iterator *iter = ...;\n *\n * while ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n -- * if (want_to_stop_iteration()) {\n + * if (want_to_stop_iteration()) {\n - * ok = ref_iterator_abort(iter);\n -+ * if (want_to_stop_iteration())\n ++ * ok = ITER_DONE;\n * break;\n -- * }\n + * }\n *\n - * // Access information about the current reference:\n - * if (!(iter->flags & REF_ISSYMREF))\n @@ refs/refs-internal.h: enum do_for_each_ref_flags {\n *\n * if (ok != ITER_DONE)\n 8: 4618e6d5959 = 10: d2763859826 refs/iterator: provide infrastructure to re-seek iterators\n 9: 527caf0bae2 = 11: 3a56f55e2c4 refs/iterator: implement seeking for merged iterators\n10: 61eddf82887 = 12: 33a04cc0a80 refs/iterator: implement seeking for reftable iterators\n11: d6a9792cb4c = 13: 5380a6f57dc refs/iterator: implement seeking for ref-cache iterators\n12: 72ac2f31c39 = 14: 7a26532dd1f refs/iterator: implement seeking for `packed-ref` iterators\n13: 916ec77de21 = 15: dad520cf933 refs/iterator: implement seeking for \"files\" iterators\n14: 7d40945d157 ! 16: c300e7f049e refs: reuse iterators when determining refname availability\n @@ Commit message\n single reference we're about to check. This keeps us from reusing state\n that the iterator may have and that may make it work more efficiently.\n \n - Refactor the logic to reseek iterators. This leads to a speedup with the\n - reftable backend, which is the only backend that knows to batch refname\n - availability checks:\n + Refactor the logic to reseek iterators. This leads to a sizeable speedup\n + with the \"reftable\" backend:\n \n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 39.8 ms ± 0.9 ms [User: 29.7 ms, System: 9.8 ms]\n @@ Commit message\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.25 ± 0.05 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n \n + The \"files\" backend doesn't really show a huge impact:\n +\n + Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n + Time (mean ± σ): 392.3 ms ± 7.1 ms [User: 59.7 ms, System: 328.8 ms]\n + Range (min … max): 384.6 ms … 404.5 ms 10 runs\n +\n + Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n + Time (mean ± σ): 387.7 ms ± 7.4 ms [User: 54.6 ms, System: 329.6 ms]\n + Range (min … max): 377.0 ms … 397.7 ms 10 runs\n +\n + Summary\n + update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n + 1.01 ± 0.03 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n +\n + This is mostly because it is way slower to begin with because it has to\n + create a separate file for each new reference, so the milliseconds we\n + shave off by reseeking the iterator doesn't really translate into a\n + significant relative improvement.\n +\n Signed-off-by: Patrick Steinhardt \n \n ## refs.c ##\n\n---\nbase-commit: e2067b49ecaef9b7f51a17ce251f9207f72ef52d\nchange-id: 20250217-pks-update-ref-optimization-15c795e66e2b\n\n"},{"id":"512685","messageId":"20250219-pks-update-ref-optimization-v2-1-e696e7220b22@pks.im","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im","subject":"[PATCH v2 01/16] object-name: introduce `repo_get_oid_with_flags()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T13:23:28Z","receivedAt":"2025-02-19T13:23:40Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new function `repo_get_oid_with_flags()`. This function\nbehaves the same as `repo_get_oid()`, except that it takes an extra\n`flags` parameter that it ends up passing to `get_oid_with_context()`.\n\nThis function will be used in a subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-name.c | 14 ++++++++------\n object-name.h | 6 ++++++\n 2 files changed, 14 insertions(+), 6 deletions(-)\n\ndiff --git a/object-name.c b/object-name.c\nindex 945d5bdef25..bc0265ad2a1 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -1794,18 +1794,20 @@ void object_context_release(struct object_context *ctx)\n \tstrbuf_release(&ctx->symlink_path);\n }\n \n-/*\n- * This is like \"get_oid_basic()\", except it allows \"object ID expressions\",\n- * notably \"xyz^\" for \"parent of xyz\"\n- */\n-int repo_get_oid(struct repository *r, const char *name, struct object_id *oid)\n+int repo_get_oid_with_flags(struct repository *r, const char *name, struct object_id *oid,\n+\t\t\t unsigned flags)\n {\n \tstruct object_context unused;\n-\tint ret = get_oid_with_context(r, name, 0, oid, &unused);\n+\tint ret = get_oid_with_context(r, name, flags, oid, &unused);\n \tobject_context_release(&unused);\n \treturn ret;\n }\n \n+int repo_get_oid(struct repository *r, const char *name, struct object_id *oid)\n+{\n+\treturn repo_get_oid_with_flags(r, name, oid, 0);\n+}\n+\n /*\n * This returns a non-zero value if the string (built using printf\n * format and the given arguments) is not a valid object.\ndiff --git a/object-name.h b/object-name.h\nindex 8dba4a47a47..fb5a97b2c8e 100644\n--- a/object-name.h\n+++ b/object-name.h\n@@ -51,6 +51,12 @@ void strbuf_repo_add_unique_abbrev(struct strbuf *sb, struct repository *repo,\n void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n \t\t\t int abbrev_len);\n \n+/*\n+ * This is like \"get_oid_basic()\", except it allows \"object ID expressions\",\n+ * notably \"xyz^\" for \"parent of xyz\". Accepts GET_OID_* flags.\n+ */\n+int repo_get_oid_with_flags(struct repository *r, const char *str, struct object_id *oid,\n+\t\t\t unsigned flags);\n int repo_get_oid(struct repository *r, const char *str, struct object_id *oid);\n __attribute__((format (printf, 2, 3)))\n int get_oidf(struct object_id *oid, const char *fmt, ...);\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512687","messageId":"20250219-pks-update-ref-optimization-v2-2-e696e7220b22@pks.im","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im","subject":"[PATCH v2 02/16] object-name: allow skipping ambiguity checks in `get_oid()` family","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T13:23:29Z","receivedAt":"2025-02-19T13:23:41Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When reading an object ID via `get_oid_basic()` or any of its related\nfunctions we perform a check whether the object ID is ambiguous, which\ncan be the case when a reference with the same name exists. While the\ncheck is generally helpful, there are cases where it only adds to the\nruntime overhead without providing much of a benefit.\n\nAdd a new flag that allows us to disable the check. The flag will be\nused in a subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n hash.h | 1 +\n object-name.c | 4 +++-\n 2 files changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/hash.h b/hash.h\nindex 4367acfec50..79419016513 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -204,6 +204,7 @@ struct object_id {\n #define GET_OID_ONLY_TO_DIE 04000\n #define GET_OID_REQUIRE_PATH 010000\n #define GET_OID_HASH_ANY 020000\n+#define GET_OID_SKIP_AMBIGUITY_CHECK 040000\n \n #define GET_OID_DISAMBIGUATORS \\\n \t(GET_OID_COMMIT | GET_OID_COMMITTISH | \\\ndiff --git a/object-name.c b/object-name.c\nindex bc0265ad2a1..3e0b7edea11 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -961,7 +961,9 @@ static int get_oid_basic(struct repository *r, const char *str, int len,\n \tint fatal = !(flags & GET_OID_QUIETLY);\n \n \tif (len == r->hash_algo->hexsz && !get_oid_hex(str, oid)) {\n-\t\tif (repo_settings_get_warn_ambiguous_refs(r) && warn_on_object_refname_ambiguity) {\n+\t\tif (!(flags & GET_OID_SKIP_AMBIGUITY_CHECK) &&\n+\t\t repo_settings_get_warn_ambiguous_refs(r) &&\n+\t\t warn_on_object_refname_ambiguity) {\n \t\t\trefs_found = repo_dwim_ref(r, str, len, &tmp_oid, &real_ref, 0);\n \t\t\tif (refs_found > 0) {\n \t\t\t\twarning(warn_msg, len, str);\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512686","messageId":"20250219-pks-update-ref-optimization-v2-3-e696e7220b22@pks.im","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im","subject":"[PATCH v2 03/16] builtin/update-ref: skip ambiguity checks when parsing object IDs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T13:23:30Z","receivedAt":"2025-02-19T13:23:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Most of the commands in git-update-ref(1) accept an old and/or new\nobject ID to update a specific reference to. These object IDs get parsed\nvia `repo_get_oid()`, which not only handles plain object IDs, but also\nthose that have a suffix like \"~\" or \"^2\". More surprisingly though, it\neven knows to resolve references, despite the fact that its manpage does\nnot mention this fact even once.\n\nOne consequence of this is that we also check for ambiguous references:\nwhen parsing a full object ID where the DWIM mechanism would also cause\nus to resolve it as a branch, we'd end up printing a warning. While this\ncheck makes sense to have in general, it is arguably less useful in the\ncontext of git-update-ref(1). This is out of two reasons:\n\n - The manpage is explicitly structured around object IDs. So if we see\n a fully blown object ID, the intent should be quite clear in\n general.\n\n - The command is part of our plumbing layer and not a tool that users\n would generally use in interactive workflows. As such, the warning\n will likely not be visible to anybody in the first place.\n\nFurthermore, this check can be quite expensive when updating lots of\nreferences via `--stdin`, because we try to read multiple references per\nobject ID that we parse according to the DWIM rules. This effect can be\nseen both with the \"files\" and \"reftable\" backend.\n\nThe issue is not unique to git-update-ref(1), but was also an issue in\ngit-cat-file(1), where it was addressed by disabling the ambiguity check\nin 25fba78d36b (cat-file: disable object/refname ambiguity check for\nbatch mode, 2013-07-12).\n\nDisable the warning in git-update-ref(1), which provides a significant\nspeedup with both backends. The following benchmark creates 10000 new\nreferences with a 100000 preexisting refs with the \"files\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 467.3 ms ± 5.1 ms [User: 100.0 ms, System: 365.1 ms]\n Range (min … max): 461.9 ms … 479.3 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 394.1 ms ± 5.8 ms [User: 63.3 ms, System: 327.6 ms]\n Range (min … max): 384.9 ms … 405.7 ms 10 runs\n\n Summary\n update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.19 ± 0.02 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n\nAnd with the \"reftable\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 146.9 ms ± 2.2 ms [User: 90.4 ms, System: 56.0 ms]\n Range (min … max): 142.7 ms … 150.8 ms 19 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 63.2 ms ± 1.1 ms [User: 41.0 ms, System: 21.8 ms]\n Range (min … max): 61.1 ms … 66.6 ms 41 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 2.32 ± 0.05 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n\nNote that the absolute improvement with both backends is roughly in the\nsame ballpark, but the relative improvement for the \"reftable\" backend\nis more significant because writing the new table to disk is faster in\nthe first place.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/update-ref.c | 12 ++++++++----\n 1 file changed, 8 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/update-ref.c b/builtin/update-ref.c\nindex 4d35bdc4b4b..d603f54b770 100644\n--- a/builtin/update-ref.c\n+++ b/builtin/update-ref.c\n@@ -179,7 +179,8 @@ static int parse_next_oid(const char **next, const char *end,\n \t\t(*next)++;\n \t\t*next = parse_arg(*next, &arg);\n \t\tif (arg.len) {\n-\t\t\tif (repo_get_oid(the_repository, arg.buf, oid))\n+\t\t\tif (repo_get_oid_with_flags(the_repository, arg.buf, oid,\n+\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\t\tgoto invalid;\n \t\t} else {\n \t\t\t/* Without -z, an empty value means all zeros: */\n@@ -197,7 +198,8 @@ static int parse_next_oid(const char **next, const char *end,\n \t\t*next += arg.len;\n \n \t\tif (arg.len) {\n-\t\t\tif (repo_get_oid(the_repository, arg.buf, oid))\n+\t\t\tif (repo_get_oid_with_flags(the_repository, arg.buf, oid,\n+\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\t\tgoto invalid;\n \t\t} else if (flags & PARSE_SHA1_ALLOW_EMPTY) {\n \t\t\t/* With -z, treat an empty value as all zeros: */\n@@ -772,7 +774,8 @@ int cmd_update_ref(int argc,\n \t\trefname = argv[0];\n \t\tvalue = argv[1];\n \t\toldval = argv[2];\n-\t\tif (repo_get_oid(the_repository, value, &oid))\n+\t\tif (repo_get_oid_with_flags(the_repository, value, &oid,\n+\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\tdie(\"%s: not a valid SHA1\", value);\n \t}\n \n@@ -783,7 +786,8 @@ int cmd_update_ref(int argc,\n \t\t\t * must not already exist:\n \t\t\t */\n \t\t\toidclr(&oldoid, the_repository->hash_algo);\n-\t\telse if (repo_get_oid(the_repository, oldval, &oldoid))\n+\t\telse if (repo_get_oid_with_flags(the_repository, oldval, &oldoid,\n+\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\tdie(\"%s: not a valid old SHA1\", oldval);\n \t}\n \n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512688","messageId":"20250219-pks-update-ref-optimization-v2-4-e696e7220b22@pks.im","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im","subject":"[PATCH v2 04/16] refs: introduce function to batch refname availability checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T13:23:31Z","receivedAt":"2025-02-19T13:23:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `refs_verify_refname_available()` functions checks whether a\nreference update can be committed or whether it would conflict with\neither a prefix or suffix thereof. This function needs to be called once\nper reference that one wants to check, which requires us to redo a\ncouple of checks every time the function is called.\n\nIntroduce a new function `refs_verify_refnames_available()` that does\nthe same, but for a list of references. For now, the new function uses\nthe exact same implementation, except that we loop through all refnames\nprovided by the caller. This will be tuned in subsequent commits.\n\nThe existing `refs_verify_refname_available()` function is reimplemented\non top of the new function. As such, the diff is best viewed with the\n`--ignore-space-change option`.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs.c | 169 +++++++++++++++++++++++++++++++++++++----------------------------\n refs.h | 12 +++++\n 2 files changed, 109 insertions(+), 72 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex f4094a326a9..5a9b0f2fa1e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2467,19 +2467,15 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \treturn ret;\n }\n \n-int refs_verify_refname_available(struct ref_store *refs,\n-\t\t\t\t const char *refname,\n-\t\t\t\t const struct string_list *extras,\n-\t\t\t\t const struct string_list *skip,\n-\t\t\t\t unsigned int initial_transaction,\n-\t\t\t\t struct strbuf *err)\n+int refs_verify_refnames_available(struct ref_store *refs,\n+\t\t\t\t const struct string_list *refnames,\n+\t\t\t\t const struct string_list *extras,\n+\t\t\t\t const struct string_list *skip,\n+\t\t\t\t unsigned int initial_transaction,\n+\t\t\t\t struct strbuf *err)\n {\n-\tconst char *slash;\n-\tconst char *extra_refname;\n \tstruct strbuf dirname = STRBUF_INIT;\n \tstruct strbuf referent = STRBUF_INIT;\n-\tstruct object_id oid;\n-\tunsigned int type;\n \tint ret = -1;\n \n \t/*\n@@ -2489,79 +2485,91 @@ int refs_verify_refname_available(struct ref_store *refs,\n \n \tassert(err);\n \n-\tstrbuf_grow(&dirname, strlen(refname) + 1);\n-\tfor (slash = strchr(refname, '/'); slash; slash = strchr(slash + 1, '/')) {\n-\t\t/*\n-\t\t * Just saying \"Is a directory\" when we e.g. can't\n-\t\t * lock some multi-level ref isn't very informative,\n-\t\t * the user won't be told *what* is a directory, so\n-\t\t * let's not use strerror() below.\n-\t\t */\n-\t\tint ignore_errno;\n-\t\t/* Expand dirname to the new prefix, not including the trailing slash: */\n-\t\tstrbuf_add(&dirname, refname + dirname.len, slash - refname - dirname.len);\n+\tfor (size_t i = 0; i < refnames->nr; i++) {\n+\t\tconst char *refname = refnames->items[i].string;\n+\t\tconst char *extra_refname;\n+\t\tstruct object_id oid;\n+\t\tunsigned int type;\n+\t\tconst char *slash;\n+\n+\t\tstrbuf_reset(&dirname);\n+\n+\t\tfor (slash = strchr(refname, '/'); slash; slash = strchr(slash + 1, '/')) {\n+\t\t\t/*\n+\t\t\t * Just saying \"Is a directory\" when we e.g. can't\n+\t\t\t * lock some multi-level ref isn't very informative,\n+\t\t\t * the user won't be told *what* is a directory, so\n+\t\t\t * let's not use strerror() below.\n+\t\t\t */\n+\t\t\tint ignore_errno;\n+\n+\t\t\t/* Expand dirname to the new prefix, not including the trailing slash: */\n+\t\t\tstrbuf_add(&dirname, refname + dirname.len, slash - refname - dirname.len);\n+\n+\t\t\t/*\n+\t\t\t * We are still at a leading dir of the refname (e.g.,\n+\t\t\t * \"refs/foo\"; if there is a reference with that name,\n+\t\t\t * it is a conflict, *unless* it is in skip.\n+\t\t\t */\n+\t\t\tif (skip && string_list_has_string(skip, dirname.buf))\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (!initial_transaction &&\n+\t\t\t !refs_read_raw_ref(refs, dirname.buf, &oid, &referent,\n+\t\t\t\t\t &type, &ignore_errno)) {\n+\t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n+\t\t\t\t\t dirname.buf, refname);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\n+\t\t\tif (extras && string_list_has_string(extras, dirname.buf)) {\n+\t\t\t\tstrbuf_addf(err, _(\"cannot process '%s' and '%s' at the same time\"),\n+\t\t\t\t\t refname, dirname.buf);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\t\t}\n \n \t\t/*\n-\t\t * We are still at a leading dir of the refname (e.g.,\n-\t\t * \"refs/foo\"; if there is a reference with that name,\n-\t\t * it is a conflict, *unless* it is in skip.\n+\t\t * We are at the leaf of our refname (e.g., \"refs/foo/bar\").\n+\t\t * There is no point in searching for a reference with that\n+\t\t * name, because a refname isn't considered to conflict with\n+\t\t * itself. But we still need to check for references whose\n+\t\t * names are in the \"refs/foo/bar/\" namespace, because they\n+\t\t * *do* conflict.\n \t\t */\n-\t\tif (skip && string_list_has_string(skip, dirname.buf))\n-\t\t\tcontinue;\n+\t\tstrbuf_addstr(&dirname, refname + dirname.len);\n+\t\tstrbuf_addch(&dirname, '/');\n+\n+\t\tif (!initial_transaction) {\n+\t\t\tstruct ref_iterator *iter;\n+\t\t\tint ok;\n+\n+\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n+\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n+\t\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n+\t\t\t\tif (skip &&\n+\t\t\t\t string_list_has_string(skip, iter->refname))\n+\t\t\t\t\tcontinue;\n+\n+\t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n+\t\t\t\t\t iter->refname, refname);\n+\t\t\t\tref_iterator_abort(iter);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \n-\t\tif (!initial_transaction &&\n-\t\t !refs_read_raw_ref(refs, dirname.buf, &oid, &referent,\n-\t\t\t\t &type, &ignore_errno)) {\n-\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n-\t\t\t\t dirname.buf, refname);\n-\t\t\tgoto cleanup;\n+\t\t\tif (ok != ITER_DONE)\n+\t\t\t\tBUG(\"error while iterating over references\");\n \t\t}\n \n-\t\tif (extras && string_list_has_string(extras, dirname.buf)) {\n+\t\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n+\t\tif (extra_refname) {\n \t\t\tstrbuf_addf(err, _(\"cannot process '%s' and '%s' at the same time\"),\n-\t\t\t\t refname, dirname.buf);\n+\t\t\t\t refname, extra_refname);\n \t\t\tgoto cleanup;\n \t\t}\n \t}\n \n-\t/*\n-\t * We are at the leaf of our refname (e.g., \"refs/foo/bar\").\n-\t * There is no point in searching for a reference with that\n-\t * name, because a refname isn't considered to conflict with\n-\t * itself. But we still need to check for references whose\n-\t * names are in the \"refs/foo/bar/\" namespace, because they\n-\t * *do* conflict.\n-\t */\n-\tstrbuf_addstr(&dirname, refname + dirname.len);\n-\tstrbuf_addch(&dirname, '/');\n-\n-\tif (!initial_transaction) {\n-\t\tstruct ref_iterator *iter;\n-\t\tint ok;\n-\n-\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n-\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n-\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n-\t\t\tif (skip &&\n-\t\t\t string_list_has_string(skip, iter->refname))\n-\t\t\t\tcontinue;\n-\n-\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n-\t\t\t\t iter->refname, refname);\n-\t\t\tref_iterator_abort(iter);\n-\t\t\tgoto cleanup;\n-\t\t}\n-\n-\t\tif (ok != ITER_DONE)\n-\t\t\tBUG(\"error while iterating over references\");\n-\t}\n-\n-\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n-\tif (extra_refname)\n-\t\tstrbuf_addf(err, _(\"cannot process '%s' and '%s' at the same time\"),\n-\t\t\t refname, extra_refname);\n-\telse\n-\t\tret = 0;\n+\tret = 0;\n \n cleanup:\n \tstrbuf_release(&referent);\n@@ -2569,6 +2577,23 @@ int refs_verify_refname_available(struct ref_store *refs,\n \treturn ret;\n }\n \n+int refs_verify_refname_available(struct ref_store *refs,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const struct string_list *extras,\n+\t\t\t\t const struct string_list *skip,\n+\t\t\t\t unsigned int initial_transaction,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct string_list_item item = { .string = (char *) refname };\n+\tstruct string_list refnames = {\n+\t\t.items = &item,\n+\t\t.nr = 1,\n+\t};\n+\n+\treturn refs_verify_refnames_available(refs, &refnames, extras, skip,\n+\t\t\t\t\t initial_transaction, err);\n+}\n+\n struct do_for_each_reflog_help {\n \teach_reflog_fn *fn;\n \tvoid *cb_data;\ndiff --git a/refs.h b/refs.h\nindex a0cdd99250e..185aed5a461 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -124,6 +124,18 @@ int refs_verify_refname_available(struct ref_store *refs,\n \t\t\t\t unsigned int initial_transaction,\n \t\t\t\t struct strbuf *err);\n \n+/*\n+ * Same as `refs_verify_refname_available()`, but checking for a list of\n+ * refnames instead of only a single item. This is more efficient in the case\n+ * where one needs to check multiple refnames.\n+ */\n+int refs_verify_refnames_available(struct ref_store *refs,\n+\t\t\t\t const struct string_list *refnames,\n+\t\t\t\t const struct string_list *extras,\n+\t\t\t\t const struct string_list *skip,\n+\t\t\t\t unsigned int initial_transaction,\n+\t\t\t\t struct strbuf *err);\n+\n int refs_ref_exists(struct ref_store *refs, const char *refname);\n \n int should_autocreate_reflog(enum log_refs_config log_all_ref_updates,\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512689","messageId":"20250219-pks-update-ref-optimization-v2-5-e696e7220b22@pks.im","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im","subject":"[PATCH v2 05/16] refs/reftable: batch refname availability checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T13:23:32Z","receivedAt":"2025-02-19T13:23:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor the \"reftable\" backend to batch the availability check for\nrefnames. This does not yet have an effect on performance as we\nessentially still call `refs_verify_refname_available()` in a loop, but\nthis will change in subsequent commits.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 16 ++++++++++------\n 1 file changed, 10 insertions(+), 6 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex d39a14c5a46..2a90e7cb391 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1069,6 +1069,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE|REF_STORE_MAIN, \"ref_transaction_prepare\");\n \tstruct strbuf referent = STRBUF_INIT, head_referent = STRBUF_INIT;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n \tstruct reftable_transaction_data *tx_data = NULL;\n \tstruct reftable_backend *be;\n \tstruct object_id head_oid;\n@@ -1224,12 +1225,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t\t * can output a proper error message instead of failing\n \t\t\t * at a later point.\n \t\t\t */\n-\t\t\tret = refs_verify_refname_available(ref_store, u->refname,\n-\t\t\t\t\t\t\t &affected_refnames, NULL,\n-\t\t\t\t\t\t\t transaction->flags & REF_TRANSACTION_FLAG_INITIAL,\n-\t\t\t\t\t\t\t err);\n-\t\t\tif (ret < 0)\n-\t\t\t\tgoto done;\n+\t\t\tstring_list_append(&refnames_to_check, u->refname);\n \n \t\t\t/*\n \t\t\t * There is no need to write the reference deletion\n@@ -1379,6 +1375,13 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t}\n \t}\n \n+\tstring_list_sort(&refnames_to_check);\n+\tret = refs_verify_refnames_available(ref_store, &refnames_to_check, &affected_refnames, NULL,\n+\t\t\t\t\t transaction->flags & REF_TRANSACTION_FLAG_INITIAL,\n+\t\t\t\t\t err);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \ttransaction->backend_data = tx_data;\n \ttransaction->state = REF_TRANSACTION_PREPARED;\n \n@@ -1394,6 +1397,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \tstring_list_clear(&affected_refnames, 0);\n \tstrbuf_release(&referent);\n \tstrbuf_release(&head_referent);\n+\tstring_list_clear(&refnames_to_check, 0);\n \n \treturn ret;\n }\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512690","messageId":"20250219-pks-update-ref-optimization-v2-6-e696e7220b22@pks.im","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im","subject":"[PATCH v2 06/16] refs/files: batch refname availability checks for normal transactions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T13:23:33Z","receivedAt":"2025-02-19T13:23:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Same as the \"reftable\" backend that we have adapted in the preceding\ncommit to use batched refname availability checks we can also do so for\nthe \"files\" backend. Things are a bit more intricate here though, as we\ncall `refs_verify_refname_available()` in a set of different contexts:\n\n 1. `lock_raw_ref()` when it hits either EEXISTS or EISDIR when creating\n a new reference, mostly to create a nice, user-readable error\n message. This is nothing we have to care about too much, as we only\n hit this code path at most once when we hit a conflict.\n\n 2. `lock_raw_ref()` when it _could_ create the lockfile to check\n whether it is conflicting with any packed refs. In the general case,\n this code path will be hit once for every (successful) reference\n update.\n\n 3. `lock_ref_oid_basic()`, but it is only executed when copying or\n renaming references or when expiring reflogs. It will thus not be\n called in contexts where we have many references queued up.\n\n 4. `refs_refname_ref_available()`, but again only when copying or\n renaming references. It is thus not interesting due to the same\n reason as the previous case.\n\n 5. `files_transaction_finish_initial()`, which is only executed when\n creating a new repository or migrating references.\n\nSo out of these, only (2) and (5) are viable candidates to use the\nbatched checks.\n\nAdapt `lock_raw_ref()` accordingly by queueing up reference names that\nneed to be checked for availability and then checking them after we have\nprocessed all updates. This check is done before we (optionally) lock\nthe `packed-refs` file, which is somewhat flawed because it means that\nthe `packed-refs` could still change after the availability check and\nthus create an undetected conflict. But unconditionally locking the file\nwould change semantics that users are likely to rely on, so we keep the\ncurrent locking sequence intact, even if it's suboptmial.\n\nThe refactoring of `files_transaction_finish_initial()` will be done in\nthe next commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/files-backend.c | 42 +++++++++++++++++++++++++++++++-----------\n 1 file changed, 31 insertions(+), 11 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 29f08dced40..6ce79cf0791 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -678,6 +678,7 @@ static void unlock_ref(struct ref_lock *lock)\n */\n static int lock_raw_ref(struct files_ref_store *refs,\n \t\t\tconst char *refname, int mustexist,\n+\t\t\tstruct string_list *refnames_to_check,\n \t\t\tconst struct string_list *extras,\n \t\t\tstruct ref_lock **lock_p,\n \t\t\tstruct strbuf *referent,\n@@ -855,16 +856,11 @@ static int lock_raw_ref(struct files_ref_store *refs,\n \t\t}\n \n \t\t/*\n-\t\t * If the ref did not exist and we are creating it,\n-\t\t * make sure there is no existing packed ref that\n-\t\t * conflicts with refname:\n+\t\t * If the ref did not exist and we are creating it, we have to\n+\t\t * make sure there is no existing packed ref that conflicts\n+\t\t * with refname. This check is deferred so that we can batch it.\n \t\t */\n-\t\tif (refs_verify_refname_available(\n-\t\t\t\t refs->packed_ref_store, refname,\n-\t\t\t\t extras, NULL, 0, err)) {\n-\t\t\tret = TRANSACTION_NAME_CONFLICT;\n-\t\t\tgoto error_return;\n-\t\t}\n+\t\tstring_list_insert(refnames_to_check, refname);\n \t}\n \n \tret = 0;\n@@ -2569,6 +2565,7 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n \t\t\t struct ref_update *update,\n \t\t\t struct ref_transaction *transaction,\n \t\t\t const char *head_ref,\n+\t\t\t struct string_list *refnames_to_check,\n \t\t\t struct string_list *affected_refnames,\n \t\t\t struct strbuf *err)\n {\n@@ -2597,7 +2594,7 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n \t\tlock->count++;\n \t} else {\n \t\tret = lock_raw_ref(refs, update->refname, mustexist,\n-\t\t\t\t affected_refnames,\n+\t\t\t\t refnames_to_check, affected_refnames,\n \t\t\t\t &lock, &referent,\n \t\t\t\t &update->type, err);\n \t\tif (ret) {\n@@ -2811,6 +2808,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \tsize_t i;\n \tint ret = 0;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n \tchar *head_ref = NULL;\n \tint head_type;\n \tstruct files_transaction_backend_data *backend_data;\n@@ -2898,7 +2896,8 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\tstruct ref_update *update = transaction->updates[i];\n \n \t\tret = lock_ref_for_update(refs, update, transaction,\n-\t\t\t\t\t head_ref, &affected_refnames, err);\n+\t\t\t\t\t head_ref, &refnames_to_check,\n+\t\t\t\t\t &affected_refnames, err);\n \t\tif (ret)\n \t\t\tgoto cleanup;\n \n@@ -2930,6 +2929,26 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\t}\n \t}\n \n+\t/*\n+\t * Verify that none of the loose reference that we're about to write\n+\t * conflict with any existing packed references. Ideally, we'd do this\n+\t * check after the packed-refs are locked so that the file cannot\n+\t * change underneath our feet. But introducing such a lock now would\n+\t * probably do more harm than good as users rely on there not being a\n+\t * global lock with the \"files\" backend.\n+\t *\n+\t * Another alternative would be to do the check after the (optional)\n+\t * lock, but that would extend the time we spend in the globally-locked\n+\t * state.\n+\t *\n+\t * So instead, we accept the race for now.\n+\t */\n+\tif (refs_verify_refnames_available(refs->packed_ref_store, &refnames_to_check,\n+\t\t\t\t\t &affected_refnames, NULL, 0, err)) {\n+\t\tret = TRANSACTION_NAME_CONFLICT;\n+\t\tgoto cleanup;\n+\t}\n+\n \tif (packed_transaction) {\n \t\tif (packed_refs_lock(refs->packed_ref_store, 0, err)) {\n \t\t\tret = TRANSACTION_GENERIC_ERROR;\n@@ -2972,6 +2991,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n cleanup:\n \tfree(head_ref);\n \tstring_list_clear(&affected_refnames, 0);\n+\tstring_list_clear(&refnames_to_check, 0);\n \n \tif (ret)\n \t\tfiles_transaction_cleanup(refs, transaction);\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512691","messageId":"20250219-pks-update-ref-optimization-v2-7-e696e7220b22@pks.im","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im","subject":"[PATCH v2 07/16] refs/files: batch refname availability checks for initial transactions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T13:23:34Z","receivedAt":"2025-02-19T13:23:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"files\" backend explicitly carves out special logic for its initial\ntransaction so that it can avoid writing out every single reference as\na loose reference. While the assumption is that there shouldn't be any\npreexisting references, we still have to verify that none of the newly\nwritten references will conflict with any other new reference in the\nsame transaction.\n\nRefactor the initial transaction to use batched refname availability\nchecks. This does not yet have an effect on performance as we still call\n`refs_verify_refname_available()` in a loop. But this will change in\nsubsequent commits and then impact performance when cloning a repository\nwith many references or when migrating references to the \"files\" format.\n\nThis doesn't yet have an effect on performance as the underlying\nlogic simply calls This will improve performance when cloning a repository with\nmany references or when migrating references from any format to the\n\"files\" format.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/files-backend.c | 23 ++++++++++++++++-------\n 1 file changed, 16 insertions(+), 7 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 6ce79cf0791..11a620ea11a 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3056,6 +3056,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \tsize_t i;\n \tint ret = 0;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n \tstruct ref_transaction *packed_transaction = NULL;\n \tstruct ref_transaction *loose_transaction = NULL;\n \n@@ -3105,11 +3106,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\t !is_null_oid(&update->old_oid))\n \t\t\tBUG(\"initial ref transaction with old_sha1 set\");\n \n-\t\tif (refs_verify_refname_available(&refs->base, update->refname,\n-\t\t\t\t\t\t &affected_refnames, NULL, 1, err)) {\n-\t\t\tret = TRANSACTION_NAME_CONFLICT;\n-\t\t\tgoto cleanup;\n-\t\t}\n+\t\tstring_list_append(&refnames_to_check, update->refname);\n \n \t\t/*\n \t\t * packed-refs don't support symbolic refs, root refs and reflogs,\n@@ -3145,8 +3142,19 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\t}\n \t}\n \n-\tif (packed_refs_lock(refs->packed_ref_store, 0, err) ||\n-\t ref_transaction_commit(packed_transaction, err)) {\n+\tif (packed_refs_lock(refs->packed_ref_store, 0, err)) {\n+\t\tret = TRANSACTION_GENERIC_ERROR;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif (refs_verify_refnames_available(&refs->base, &refnames_to_check,\n+\t\t\t\t\t &affected_refnames, NULL, 1, err)) {\n+\t\tpacked_refs_unlock(refs->packed_ref_store);\n+\t\tret = TRANSACTION_NAME_CONFLICT;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif (ref_transaction_commit(packed_transaction, err)) {\n \t\tret = TRANSACTION_GENERIC_ERROR;\n \t\tgoto cleanup;\n \t}\n@@ -3167,6 +3175,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\tref_transaction_free(packed_transaction);\n \ttransaction->state = REF_TRANSACTION_CLOSED;\n \tstring_list_clear(&affected_refnames, 0);\n+\tstring_list_clear(&refnames_to_check, 0);\n \treturn ret;\n }\n \n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512692","messageId":"20250219-pks-update-ref-optimization-v2-8-e696e7220b22@pks.im","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im","subject":"[PATCH v2 08/16] refs: stop re-verifying common prefixes for availability","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T13:23:35Z","receivedAt":"2025-02-19T13:23:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"One of the checks done by `refs_verify_refnames_available()` is whether\nany of the prefixes of a reference already exists. For example, given a\nreference \"refs/heads/main\", we'd check whether \"refs/heads\" or \"refs\"\nalready exist, and if so we'd abort the transaction.\n\nWhen updating multiple references at once, this check is performed for\neach of the references individually. Consequently, because references\ntend to have common prefixes like \"refs/heads/\" or refs/tags/\", we\nevaluate the availability of these prefixes repeatedly. Naturally this\nis a waste of compute, as the availability of those prefixes should in\ngeneral not change in the middle of a transaction. And if it would,\nbackends would notice at a later point in time.\n\nOptimize this pattern by storing prefixes in a `strset` so that we can\ntrivially track those prefixes that we have already checked. This leads\nto a significant speedup with the \"reftable\" backend when creating many\nreferences that all share a common prefix:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 63.1 ms ± 1.8 ms [User: 41.0 ms, System: 21.6 ms]\n Range (min … max): 60.6 ms … 69.5 ms 38 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 40.0 ms ± 1.3 ms [User: 29.3 ms, System: 10.3 ms]\n Range (min … max): 38.1 ms … 47.3 ms 61 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.58 ± 0.07 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n\nFor the \"files\" backend we see an improvement, but a much smaller one:\n\n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 395.8 ms ± 5.3 ms [User: 63.6 ms, System: 330.5 ms]\n Range (min … max): 387.0 ms … 404.6 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 386.0 ms ± 4.0 ms [User: 51.5 ms, System: 332.8 ms]\n Range (min … max): 380.8 ms … 392.6 ms 10 runs\n\n Summary\n update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.03 ± 0.02 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n\nThis change also leads to a modest improvement when writing references\nwith \"initial\" semantics, for example when migrating references. The\nfollowing benchmarks are migrating 1m references from the \"reftable\" to\nthe \"files\" backend:\n\n Benchmark 1: migrate reftable:files (refcount = 1000000, revision = HEAD~)\n Time (mean ± σ): 836.6 ms ± 5.6 ms [User: 645.2 ms, System: 185.2 ms]\n Range (min … max): 829.6 ms … 845.9 ms 10 runs\n\n Benchmark 2: migrate reftable:files (refcount = 1000000, revision = HEAD)\n Time (mean ± σ): 759.8 ms ± 5.1 ms [User: 574.9 ms, System: 178.9 ms]\n Range (min … max): 753.1 ms … 768.8 ms 10 runs\n\n Summary\n migrate reftable:files (refcount = 1000000, revision = HEAD) ran\n 1.10 ± 0.01 times faster than migrate reftable:files (refcount = 1000000, revision = HEAD~)\n\nAnd vice versa:\n\n Benchmark 1: migrate files:reftable (refcount = 1000000, revision = HEAD~)\n Time (mean ± σ): 870.7 ms ± 5.7 ms [User: 735.2 ms, System: 127.4 ms]\n Range (min … max): 861.6 ms … 883.2 ms 10 runs\n\n Benchmark 2: migrate files:reftable (refcount = 1000000, revision = HEAD)\n Time (mean ± σ): 799.1 ms ± 8.5 ms [User: 661.1 ms, System: 130.2 ms]\n Range (min … max): 787.5 ms … 812.6 ms 10 runs\n\n Summary\n migrate files:reftable (refcount = 1000000, revision = HEAD) ran\n 1.09 ± 0.01 times faster than migrate files:reftable (refcount = 1000000, revision = HEAD~)\n\nThe impact here is significantly smaller given that we don't perform any\nreference reads with \"initial\" semantics, so the speedup only comes from\nus doing less string list lookups.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs.c | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/refs.c b/refs.c\nindex 5a9b0f2fa1e..eaf41421f50 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2476,6 +2476,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n {\n \tstruct strbuf dirname = STRBUF_INIT;\n \tstruct strbuf referent = STRBUF_INIT;\n+\tstruct strset dirnames;\n \tint ret = -1;\n \n \t/*\n@@ -2485,6 +2486,8 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \n \tassert(err);\n \n+\tstrset_init(&dirnames);\n+\n \tfor (size_t i = 0; i < refnames->nr; i++) {\n \t\tconst char *refname = refnames->items[i].string;\n \t\tconst char *extra_refname;\n@@ -2514,6 +2517,14 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \t\t\tif (skip && string_list_has_string(skip, dirname.buf))\n \t\t\t\tcontinue;\n \n+\t\t\t/*\n+\t\t\t * If we've already seen the directory we don't need to\n+\t\t\t * process it again. Skip it to avoid checking checking\n+\t\t\t * common prefixes like \"refs/heads/\" repeatedly.\n+\t\t\t */\n+\t\t\tif (!strset_add(&dirnames, dirname.buf))\n+\t\t\t\tcontinue;\n+\n \t\t\tif (!initial_transaction &&\n \t\t\t !refs_read_raw_ref(refs, dirname.buf, &oid, &referent,\n \t\t\t\t\t &type, &ignore_errno)) {\n@@ -2574,6 +2585,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n cleanup:\n \tstrbuf_release(&referent);\n \tstrbuf_release(&dirname);\n+\tstrset_clear(&dirnames);\n \treturn ret;\n }\n \n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512695","messageId":"20250219-pks-update-ref-optimization-v2-9-e696e7220b22@pks.im","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im","subject":"[PATCH v2 09/16] refs/iterator: separate lifecycle from iteration","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T13:23:36Z","receivedAt":"2025-02-19T13:23:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The ref and reflog iterators have their lifecycle attached to iteration:\nonce the iterator reaches its end, it is automatically released and the\ncaller doesn't have to care about that anymore. When the iterator should\nbe released before it has been exhausted, callers must explicitly abort\nthe iterator via `ref_iterator_abort()`.\n\nThis lifecycle is somewhat unusual in the Git codebase and creates two\nproblems:\n\n - Callsites need to be very careful about when exactly they call\n `ref_iterator_abort()`, as calling the function is only valid when\n the iterator itself still is. This leads to somewhat awkward calling\n patterns in some situations.\n\n - It is impossible to reuse iterators and re-seek them to a different\n prefix. This feature isn't supported by any iterator implementation\n except for the reftable iterators anyway, but if it was implemented\n it would allow us to optimize cases where we need to search for\n specific references repeatedly by reusing internal state.\n\nDetangle the lifecycle from iteration so that we don't deallocate the\niterator anymore once it is exhausted. Instead, callers are now expected\nto always call a newly introduce `ref_iterator_free()` function that\ndeallocates the iterator and its internal state.\n\nNote that the `dir_iterator` is somewhat special because it does not\nimplement the `ref_iterator` interface, but is only used to implement\nother iterators. Consequently, we have to provide `dir_iterator_free()`\ninstead of `dir_iterator_release()` as the allocated structure itself is\nmanaged by the `dir_iterator` interfaces, as well, and not freed by\n`ref_iterator_free()` like in all the other cases.\n\nWhile at it, drop the return value of `ref_iterator_abort()`, which\nwasn't really required by any of the iterator implementations anyway.\nFurthermore, stop calling `base_ref_iterator_free()` in any of the\nbackends, but instead call it in `ref_iterator_free()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/clone.c | 2 +\n dir-iterator.c | 24 +++++------\n dir-iterator.h | 11 ++---\n iterator.h | 2 +-\n refs.c | 7 +++-\n refs/debug.c | 9 ++---\n refs/files-backend.c | 36 +++++------------\n refs/iterator.c | 95 ++++++++++++++------------------------------\n refs/packed-backend.c | 27 ++++++-------\n refs/ref-cache.c | 9 ++---\n refs/refs-internal.h | 29 +++++---------\n refs/reftable-backend.c | 34 ++++------------\n t/helper/test-dir-iterator.c | 1 +\n 13 files changed, 100 insertions(+), 186 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex fd001d800c6..ac3e84b2b18 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -426,6 +426,8 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\tstrbuf_setlen(src, src_len);\n \t\tdie(_(\"failed to iterate over '%s'\"), src->buf);\n \t}\n+\n+\tdir_iterator_free(iter);\n }\n \n static void clone_local(const char *src_repo, const char *dest_repo)\ndiff --git a/dir-iterator.c b/dir-iterator.c\nindex de619846f29..857e1d9bdaf 100644\n--- a/dir-iterator.c\n+++ b/dir-iterator.c\n@@ -193,9 +193,9 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \n \tif (S_ISDIR(iter->base.st.st_mode) && push_level(iter)) {\n \t\tif (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC)\n-\t\t\tgoto error_out;\n+\t\t\treturn ITER_ERROR;\n \t\tif (iter->levels_nr == 0)\n-\t\t\tgoto error_out;\n+\t\t\treturn ITER_ERROR;\n \t}\n \n \t/* Loop until we find an entry that we can give back to the caller. */\n@@ -211,11 +211,11 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\tint ret = next_directory_entry(level->dir, iter->base.path.buf, &de);\n \t\t\tif (ret < 0) {\n \t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n-\t\t\t\t\tgoto error_out;\n+\t\t\t\t\treturn ITER_ERROR;\n \t\t\t\tcontinue;\n \t\t\t} else if (ret > 0) {\n \t\t\t\tif (pop_level(iter) == 0)\n-\t\t\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\t\t\t\treturn ITER_DONE;\n \t\t\t\tcontinue;\n \t\t\t}\n \n@@ -223,7 +223,7 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t} else {\n \t\t\tif (level->entries_idx >= level->entries.nr) {\n \t\t\t\tif (pop_level(iter) == 0)\n-\t\t\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\t\t\t\treturn ITER_DONE;\n \t\t\t\tcontinue;\n \t\t\t}\n \n@@ -232,22 +232,21 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \n \t\tif (prepare_next_entry_data(iter, name)) {\n \t\t\tif (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC)\n-\t\t\t\tgoto error_out;\n+\t\t\t\treturn ITER_ERROR;\n \t\t\tcontinue;\n \t\t}\n \n \t\treturn ITER_OK;\n \t}\n-\n-error_out:\n-\tdir_iterator_abort(dir_iterator);\n-\treturn ITER_ERROR;\n }\n \n-int dir_iterator_abort(struct dir_iterator *dir_iterator)\n+void dir_iterator_free(struct dir_iterator *dir_iterator)\n {\n \tstruct dir_iterator_int *iter = (struct dir_iterator_int *)dir_iterator;\n \n+\tif (!iter)\n+\t\treturn;\n+\n \tfor (; iter->levels_nr; iter->levels_nr--) {\n \t\tstruct dir_iterator_level *level =\n \t\t\t&iter->levels[iter->levels_nr - 1];\n@@ -266,7 +265,6 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator)\n \tfree(iter->levels);\n \tstrbuf_release(&iter->base.path);\n \tfree(iter);\n-\treturn ITER_DONE;\n }\n \n struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags)\n@@ -301,7 +299,7 @@ struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags)\n \treturn dir_iterator;\n \n error_out:\n-\tdir_iterator_abort(dir_iterator);\n+\tdir_iterator_free(dir_iterator);\n \terrno = saved_errno;\n \treturn NULL;\n }\ndiff --git a/dir-iterator.h b/dir-iterator.h\nindex 6d438809b6e..ccd6a197343 100644\n--- a/dir-iterator.h\n+++ b/dir-iterator.h\n@@ -28,7 +28,7 @@\n *\n * while ((ok = dir_iterator_advance(iter)) == ITER_OK) {\n * if (want_to_stop_iteration()) {\n- * ok = dir_iterator_abort(iter);\n+ * ok = ITER_DONE;\n * break;\n * }\n *\n@@ -39,6 +39,7 @@\n *\n * if (ok != ITER_DONE)\n * handle_error();\n+ * dir_iterator_free(iter);\n *\n * Callers are allowed to modify iter->path while they are working,\n * but they must restore it to its original contents before calling\n@@ -107,11 +108,7 @@ struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags);\n */\n int dir_iterator_advance(struct dir_iterator *iterator);\n \n-/*\n- * End the iteration before it has been exhausted. Free the\n- * dir_iterator and any associated resources and return ITER_DONE. On\n- * error, free the dir_iterator and return ITER_ERROR.\n- */\n-int dir_iterator_abort(struct dir_iterator *iterator);\n+/* Free the dir_iterator and any associated resources. */\n+void dir_iterator_free(struct dir_iterator *iterator);\n \n #endif\ndiff --git a/iterator.h b/iterator.h\nindex 0f6900e43ad..6b77dcc2626 100644\n--- a/iterator.h\n+++ b/iterator.h\n@@ -12,7 +12,7 @@\n #define ITER_OK 0\n \n /*\n- * The iterator is exhausted and has been freed.\n+ * The iterator is exhausted.\n */\n #define ITER_DONE -1\n \ndiff --git a/refs.c b/refs.c\nindex eaf41421f50..8eff60a2186 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2476,6 +2476,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n {\n \tstruct strbuf dirname = STRBUF_INIT;\n \tstruct strbuf referent = STRBUF_INIT;\n+\tstruct ref_iterator *iter = NULL;\n \tstruct strset dirnames;\n \tint ret = -1;\n \n@@ -2552,7 +2553,6 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \t\tstrbuf_addch(&dirname, '/');\n \n \t\tif (!initial_transaction) {\n-\t\t\tstruct ref_iterator *iter;\n \t\t\tint ok;\n \n \t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n@@ -2564,12 +2564,14 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \n \t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n \t\t\t\t\t iter->refname, refname);\n-\t\t\t\tref_iterator_abort(iter);\n \t\t\t\tgoto cleanup;\n \t\t\t}\n \n \t\t\tif (ok != ITER_DONE)\n \t\t\t\tBUG(\"error while iterating over references\");\n+\n+\t\t\tref_iterator_free(iter);\n+\t\t\titer = NULL;\n \t\t}\n \n \t\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n@@ -2586,6 +2588,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \tstrbuf_release(&referent);\n \tstrbuf_release(&dirname);\n \tstrset_clear(&dirnames);\n+\tref_iterator_free(iter);\n \treturn ret;\n }\n \ndiff --git a/refs/debug.c b/refs/debug.c\nindex fbc4df08b43..a9786da4ba1 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -179,19 +179,18 @@ static int debug_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn res;\n }\n \n-static int debug_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void debug_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct debug_ref_iterator *diter =\n \t\t(struct debug_ref_iterator *)ref_iterator;\n-\tint res = diter->iter->vtable->abort(diter->iter);\n-\ttrace_printf_key(&trace_refs, \"iterator_abort: %d\\n\", res);\n-\treturn res;\n+\tditer->iter->vtable->release(diter->iter);\n+\ttrace_printf_key(&trace_refs, \"iterator_abort\\n\");\n }\n \n static struct ref_iterator_vtable debug_ref_iterator_vtable = {\n \t.advance = debug_ref_iterator_advance,\n \t.peel = debug_ref_iterator_peel,\n-\t.abort = debug_ref_iterator_abort,\n+\t.release = debug_ref_iterator_release,\n };\n \n static struct ref_iterator *\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 11a620ea11a..859f1c11941 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -915,10 +915,6 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\titer->iter0 = NULL;\n-\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\tok = ITER_ERROR;\n-\n \treturn ok;\n }\n \n@@ -931,23 +927,17 @@ static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator_peel(iter->iter0, peeled);\n }\n \n-static int files_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void files_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct files_ref_iterator *iter =\n \t\t(struct files_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->iter0)\n-\t\tok = ref_iterator_abort(iter->iter0);\n-\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n+\tref_iterator_free(iter->iter0);\n }\n \n static struct ref_iterator_vtable files_ref_iterator_vtable = {\n \t.advance = files_ref_iterator_advance,\n \t.peel = files_ref_iterator_peel,\n-\t.abort = files_ref_iterator_abort,\n+\t.release = files_ref_iterator_release,\n };\n \n static struct ref_iterator *files_ref_iterator_begin(\n@@ -1378,7 +1368,7 @@ static int should_pack_refs(struct files_ref_store *refs,\n \t\t\t\t iter->flags, opts))\n \t\t\trefcount++;\n \t\tif (refcount >= limit) {\n-\t\t\tref_iterator_abort(iter);\n+\t\t\tref_iterator_free(iter);\n \t\t\treturn 1;\n \t\t}\n \t}\n@@ -1386,6 +1376,7 @@ static int should_pack_refs(struct files_ref_store *refs,\n \tif (ret != ITER_DONE)\n \t\tdie(\"error while iterating over references\");\n \n+\tref_iterator_free(iter);\n \treturn 0;\n }\n \n@@ -1452,6 +1443,7 @@ static int files_pack_refs(struct ref_store *ref_store,\n \tpacked_refs_unlock(refs->packed_ref_store);\n \n \tprune_refs(refs, &refs_to_prune);\n+\tref_iterator_free(iter);\n \tstrbuf_release(&err);\n \treturn 0;\n }\n@@ -2299,9 +2291,6 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\titer->dir_iterator = NULL;\n-\tif (ref_iterator_abort(ref_iterator) == ITER_ERROR)\n-\t\tok = ITER_ERROR;\n \treturn ok;\n }\n \n@@ -2311,23 +2300,17 @@ static int files_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \tBUG(\"ref_iterator_peel() called for reflog_iterator\");\n }\n \n-static int files_reflog_iterator_abort(struct ref_iterator *ref_iterator)\n+static void files_reflog_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct files_reflog_iterator *iter =\n \t\t(struct files_reflog_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->dir_iterator)\n-\t\tok = dir_iterator_abort(iter->dir_iterator);\n-\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n+\tdir_iterator_free(iter->dir_iterator);\n }\n \n static struct ref_iterator_vtable files_reflog_iterator_vtable = {\n \t.advance = files_reflog_iterator_advance,\n \t.peel = files_reflog_iterator_peel,\n-\t.abort = files_reflog_iterator_abort,\n+\t.release = files_reflog_iterator_release,\n };\n \n static struct ref_iterator *reflog_iterator_begin(struct ref_store *ref_store,\n@@ -3837,6 +3820,7 @@ static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n \n out:\n+\tdir_iterator_free(iter);\n \tstrbuf_release(&sb);\n \tstrbuf_release(&refname);\n \treturn ret;\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex d25e568bf0b..aaeff270437 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -21,9 +21,14 @@ int ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator->vtable->peel(ref_iterator, peeled);\n }\n \n-int ref_iterator_abort(struct ref_iterator *ref_iterator)\n+void ref_iterator_free(struct ref_iterator *ref_iterator)\n {\n-\treturn ref_iterator->vtable->abort(ref_iterator);\n+\tif (ref_iterator) {\n+\t\tref_iterator->vtable->release(ref_iterator);\n+\t\t/* Help make use-after-free bugs fail quickly: */\n+\t\tref_iterator->vtable = NULL;\n+\t\tfree(ref_iterator);\n+\t}\n }\n \n void base_ref_iterator_init(struct ref_iterator *iter,\n@@ -36,20 +41,13 @@ void base_ref_iterator_init(struct ref_iterator *iter,\n \titer->flags = 0;\n }\n \n-void base_ref_iterator_free(struct ref_iterator *iter)\n-{\n-\t/* Help make use-after-free bugs fail quickly: */\n-\titer->vtable = NULL;\n-\tfree(iter);\n-}\n-\n struct empty_ref_iterator {\n \tstruct ref_iterator base;\n };\n \n-static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)\n {\n-\treturn ref_iterator_abort(ref_iterator);\n+\treturn ITER_DONE;\n }\n \n static int empty_ref_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n@@ -58,16 +56,14 @@ static int empty_ref_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \tBUG(\"peel called for empty iterator\");\n }\n \n-static int empty_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void empty_ref_iterator_release(struct ref_iterator *ref_iterator UNUSED)\n {\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable empty_ref_iterator_vtable = {\n \t.advance = empty_ref_iterator_advance,\n \t.peel = empty_ref_iterator_peel,\n-\t.abort = empty_ref_iterator_abort,\n+\t.release = empty_ref_iterator_release,\n };\n \n struct ref_iterator *empty_ref_iterator_begin(void)\n@@ -151,11 +147,13 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tif (!iter->current) {\n \t\t/* Initialize: advance both iterators to their first entries */\n \t\tif ((ok = ref_iterator_advance(iter->iter0)) != ITER_OK) {\n+\t\t\tref_iterator_free(iter->iter0);\n \t\t\titer->iter0 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n \t\t}\n \t\tif ((ok = ref_iterator_advance(iter->iter1)) != ITER_OK) {\n+\t\t\tref_iterator_free(iter->iter1);\n \t\t\titer->iter1 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -166,6 +164,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t * entry:\n \t\t */\n \t\tif ((ok = ref_iterator_advance(*iter->current)) != ITER_OK) {\n+\t\t\tref_iterator_free(*iter->current);\n \t\t\t*iter->current = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -179,9 +178,8 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t\titer->select(iter->iter0, iter->iter1, iter->cb_data);\n \n \t\tif (selection == ITER_SELECT_DONE) {\n-\t\t\treturn ref_iterator_abort(ref_iterator);\n+\t\t\treturn ITER_DONE;\n \t\t} else if (selection == ITER_SELECT_ERROR) {\n-\t\t\tref_iterator_abort(ref_iterator);\n \t\t\treturn ITER_ERROR;\n \t\t}\n \n@@ -195,6 +193,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \n \t\tif (selection & ITER_SKIP_SECONDARY) {\n \t\t\tif ((ok = ref_iterator_advance(*secondary)) != ITER_OK) {\n+\t\t\t\tref_iterator_free(*secondary);\n \t\t\t\t*secondary = NULL;\n \t\t\t\tif (ok == ITER_ERROR)\n \t\t\t\t\tgoto error;\n@@ -211,7 +210,6 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t}\n \n error:\n-\tref_iterator_abort(ref_iterator);\n \treturn ITER_ERROR;\n }\n \n@@ -227,28 +225,18 @@ static int merge_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator_peel(*iter->current, peeled);\n }\n \n-static int merge_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void merge_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct merge_ref_iterator *iter =\n \t\t(struct merge_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->iter0) {\n-\t\tif (ref_iterator_abort(iter->iter0) != ITER_DONE)\n-\t\t\tok = ITER_ERROR;\n-\t}\n-\tif (iter->iter1) {\n-\t\tif (ref_iterator_abort(iter->iter1) != ITER_DONE)\n-\t\t\tok = ITER_ERROR;\n-\t}\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n+\tref_iterator_free(iter->iter0);\n+\tref_iterator_free(iter->iter1);\n }\n \n static struct ref_iterator_vtable merge_ref_iterator_vtable = {\n \t.advance = merge_ref_iterator_advance,\n \t.peel = merge_ref_iterator_peel,\n-\t.abort = merge_ref_iterator_abort,\n+\t.release = merge_ref_iterator_release,\n };\n \n struct ref_iterator *merge_ref_iterator_begin(\n@@ -310,10 +298,10 @@ struct ref_iterator *overlay_ref_iterator_begin(\n \t * them.\n \t */\n \tif (is_empty_ref_iterator(front)) {\n-\t\tref_iterator_abort(front);\n+\t\tref_iterator_free(front);\n \t\treturn back;\n \t} else if (is_empty_ref_iterator(back)) {\n-\t\tref_iterator_abort(back);\n+\t\tref_iterator_free(back);\n \t\treturn front;\n \t}\n \n@@ -350,19 +338,10 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \n \twhile ((ok = ref_iterator_advance(iter->iter0)) == ITER_OK) {\n \t\tint cmp = compare_prefix(iter->iter0->refname, iter->prefix);\n-\n \t\tif (cmp < 0)\n \t\t\tcontinue;\n-\n-\t\tif (cmp > 0) {\n-\t\t\t/*\n-\t\t\t * As the source iterator is ordered, we\n-\t\t\t * can stop the iteration as soon as we see a\n-\t\t\t * refname that comes after the prefix:\n-\t\t\t */\n-\t\t\tok = ref_iterator_abort(iter->iter0);\n-\t\t\tbreak;\n-\t\t}\n+\t\tif (cmp > 0)\n+\t\t\treturn ITER_DONE;\n \n \t\tif (iter->trim) {\n \t\t\t/*\n@@ -386,9 +365,6 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\titer->iter0 = NULL;\n-\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\treturn ITER_ERROR;\n \treturn ok;\n }\n \n@@ -401,23 +377,18 @@ static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator_peel(iter->iter0, peeled);\n }\n \n-static int prefix_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void prefix_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct prefix_ref_iterator *iter =\n \t\t(struct prefix_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->iter0)\n-\t\tok = ref_iterator_abort(iter->iter0);\n+\tref_iterator_free(iter->iter0);\n \tfree(iter->prefix);\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n }\n \n static struct ref_iterator_vtable prefix_ref_iterator_vtable = {\n \t.advance = prefix_ref_iterator_advance,\n \t.peel = prefix_ref_iterator_peel,\n-\t.abort = prefix_ref_iterator_abort,\n+\t.release = prefix_ref_iterator_release,\n };\n \n struct ref_iterator *prefix_ref_iterator_begin(struct ref_iterator *iter0,\n@@ -453,20 +424,14 @@ int do_for_each_ref_iterator(struct ref_iterator *iter,\n \tcurrent_ref_iter = iter;\n \twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\tretval = fn(iter->refname, iter->referent, iter->oid, iter->flags, cb_data);\n-\t\tif (retval) {\n-\t\t\t/*\n-\t\t\t * If ref_iterator_abort() returns ITER_ERROR,\n-\t\t\t * we ignore that error in deference to the\n-\t\t\t * callback function's return value.\n-\t\t\t */\n-\t\t\tref_iterator_abort(iter);\n+\t\tif (retval)\n \t\t\tgoto out;\n-\t\t}\n \t}\n \n out:\n \tcurrent_ref_iter = old_ref_iter;\n \tif (ok == ITER_ERROR)\n-\t\treturn -1;\n+\t\tretval = -1;\n+\tref_iterator_free(iter);\n \treturn retval;\n }\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex a7b6f74b6e3..38a1956d1a8 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -954,9 +954,6 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\tok = ITER_ERROR;\n-\n \treturn ok;\n }\n \n@@ -976,23 +973,19 @@ static int packed_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t}\n }\n \n-static int packed_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void packed_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct packed_ref_iterator *iter =\n \t\t(struct packed_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n \tstrbuf_release(&iter->refname_buf);\n \tfree(iter->jump);\n \trelease_snapshot(iter->snapshot);\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n }\n \n static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \t.advance = packed_ref_iterator_advance,\n \t.peel = packed_ref_iterator_peel,\n-\t.abort = packed_ref_iterator_abort\n+\t.release = packed_ref_iterator_release,\n };\n \n static int jump_list_entry_cmp(const void *va, const void *vb)\n@@ -1362,8 +1355,10 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t */\n \titer = packed_ref_iterator_begin(&refs->base, \"\", NULL,\n \t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n-\tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n+\tif ((ok = ref_iterator_advance(iter)) != ITER_OK) {\n+\t\tref_iterator_free(iter);\n \t\titer = NULL;\n+\t}\n \n \ti = 0;\n \n@@ -1411,8 +1406,10 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t\t\t\t * the iterator over the unneeded\n \t\t\t\t * value.\n \t\t\t\t */\n-\t\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n+\t\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK) {\n+\t\t\t\t\tref_iterator_free(iter);\n \t\t\t\t\titer = NULL;\n+\t\t\t\t}\n \t\t\t\tcmp = +1;\n \t\t\t} else {\n \t\t\t\t/*\n@@ -1449,8 +1446,10 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t\t\t\t\t peel_error ? NULL : &peeled))\n \t\t\t\tgoto write_error;\n \n-\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n+\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK) {\n+\t\t\t\tref_iterator_free(iter);\n \t\t\t\titer = NULL;\n+\t\t\t}\n \t\t} else if (is_null_oid(&update->new_oid)) {\n \t\t\t/*\n \t\t\t * The update wants to delete the reference,\n@@ -1499,9 +1498,7 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t\t get_tempfile_path(refs->tempfile), strerror(errno));\n \n error:\n-\tif (iter)\n-\t\tref_iterator_abort(iter);\n-\n+\tref_iterator_free(iter);\n \tdelete_tempfile(&refs->tempfile);\n \treturn -1;\n }\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 02f09e4df88..6457e02c1ea 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -409,7 +409,7 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\tif (++level->index == level->dir->nr) {\n \t\t\t/* This level is exhausted; pop up a level */\n \t\t\tif (--iter->levels_nr == 0)\n-\t\t\t\treturn ref_iterator_abort(ref_iterator);\n+\t\t\t\treturn ITER_DONE;\n \n \t\t\tcontinue;\n \t\t}\n@@ -452,21 +452,18 @@ static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn peel_object(iter->repo, ref_iterator->oid, peeled) ? -1 : 0;\n }\n \n-static int cache_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void cache_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct cache_ref_iterator *iter =\n \t\t(struct cache_ref_iterator *)ref_iterator;\n-\n \tfree((char *)iter->prefix);\n \tfree(iter->levels);\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable cache_ref_iterator_vtable = {\n \t.advance = cache_ref_iterator_advance,\n \t.peel = cache_ref_iterator_peel,\n-\t.abort = cache_ref_iterator_abort\n+\t.release = cache_ref_iterator_release,\n };\n \n struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex aaab711bb96..74e2c03cef1 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -273,11 +273,11 @@ enum do_for_each_ref_flags {\n * the next reference and returns ITER_OK. The data pointed at by\n * refname and oid belong to the iterator; if you want to retain them\n * after calling ref_iterator_advance() again or calling\n- * ref_iterator_abort(), you must make a copy. When the iteration has\n+ * ref_iterator_free(), you must make a copy. When the iteration has\n * been exhausted, ref_iterator_advance() releases any resources\n * associated with the iteration, frees the ref_iterator object, and\n * returns ITER_DONE. If you want to abort the iteration early, call\n- * ref_iterator_abort(), which also frees the ref_iterator object and\n+ * ref_iterator_free(), which also frees the ref_iterator object and\n * any associated resources. If there was an internal error advancing\n * to the next entry, ref_iterator_advance() aborts the iteration,\n * frees the ref_iterator, and returns ITER_ERROR.\n@@ -293,7 +293,7 @@ enum do_for_each_ref_flags {\n *\n * while ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n * if (want_to_stop_iteration()) {\n- * ok = ref_iterator_abort(iter);\n+ * ok = ITER_DONE;\n * break;\n * }\n *\n@@ -307,6 +307,7 @@ enum do_for_each_ref_flags {\n *\n * if (ok != ITER_DONE)\n * handle_error();\n+ * ref_iterator_free(iter);\n */\n struct ref_iterator {\n \tstruct ref_iterator_vtable *vtable;\n@@ -333,12 +334,8 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator);\n int ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t struct object_id *peeled);\n \n-/*\n- * End the iteration before it has been exhausted, freeing the\n- * reference iterator and any associated resources and returning\n- * ITER_DONE. If the abort itself failed, return ITER_ERROR.\n- */\n-int ref_iterator_abort(struct ref_iterator *ref_iterator);\n+/* Free the reference iterator and any associated resources. */\n+void ref_iterator_free(struct ref_iterator *ref_iterator);\n \n /*\n * An iterator over nothing (its first ref_iterator_advance() call\n@@ -438,13 +435,6 @@ struct ref_iterator *prefix_ref_iterator_begin(struct ref_iterator *iter0,\n void base_ref_iterator_init(struct ref_iterator *iter,\n \t\t\t struct ref_iterator_vtable *vtable);\n \n-/*\n- * Base class destructor for ref_iterators. Destroy the ref_iterator\n- * part of iter and shallow-free the object. This is meant to be\n- * called only by the destructors of derived classes.\n- */\n-void base_ref_iterator_free(struct ref_iterator *iter);\n-\n /* Virtual function declarations for ref_iterators: */\n \n /*\n@@ -463,15 +453,14 @@ typedef int ref_iterator_peel_fn(struct ref_iterator *ref_iterator,\n \n /*\n * Implementations of this function should free any resources specific\n- * to the derived class, then call base_ref_iterator_free() to clean\n- * up and free the ref_iterator object.\n+ * to the derived class.\n */\n-typedef int ref_iterator_abort_fn(struct ref_iterator *ref_iterator);\n+typedef void ref_iterator_release_fn(struct ref_iterator *ref_iterator);\n \n struct ref_iterator_vtable {\n \tref_iterator_advance_fn *advance;\n \tref_iterator_peel_fn *peel;\n-\tref_iterator_abort_fn *abort;\n+\tref_iterator_release_fn *release;\n };\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 2a90e7cb391..06543f79c64 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -711,17 +711,10 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\tbreak;\n \t}\n \n-\tif (iter->err > 0) {\n-\t\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\t\treturn ITER_ERROR;\n+\tif (iter->err > 0)\n \t\treturn ITER_DONE;\n-\t}\n-\n-\tif (iter->err < 0) {\n-\t\tref_iterator_abort(ref_iterator);\n+\tif (iter->err < 0)\n \t\treturn ITER_ERROR;\n-\t}\n-\n \treturn ITER_OK;\n }\n \n@@ -740,7 +733,7 @@ static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn -1;\n }\n \n-static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void reftable_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct reftable_ref_iterator *iter =\n \t\t(struct reftable_ref_iterator *)ref_iterator;\n@@ -751,14 +744,12 @@ static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n \t\t\tfree(iter->exclude_patterns[i]);\n \t\tfree(iter->exclude_patterns);\n \t}\n-\tfree(iter);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n \t.advance = reftable_ref_iterator_advance,\n \t.peel = reftable_ref_iterator_peel,\n-\t.abort = reftable_ref_iterator_abort\n+\t.release = reftable_ref_iterator_release,\n };\n \n static int qsort_strcmp(const void *va, const void *vb)\n@@ -2017,17 +2008,10 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \t\tbreak;\n \t}\n \n-\tif (iter->err > 0) {\n-\t\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\t\treturn ITER_ERROR;\n+\tif (iter->err > 0)\n \t\treturn ITER_DONE;\n-\t}\n-\n-\tif (iter->err < 0) {\n-\t\tref_iterator_abort(ref_iterator);\n+\tif (iter->err < 0)\n \t\treturn ITER_ERROR;\n-\t}\n-\n \treturn ITER_OK;\n }\n \n@@ -2038,21 +2022,19 @@ static int reftable_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSE\n \treturn -1;\n }\n \n-static int reftable_reflog_iterator_abort(struct ref_iterator *ref_iterator)\n+static void reftable_reflog_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct reftable_reflog_iterator *iter =\n \t\t(struct reftable_reflog_iterator *)ref_iterator;\n \treftable_log_record_release(&iter->log);\n \treftable_iterator_destroy(&iter->iter);\n \tstrbuf_release(&iter->last_name);\n-\tfree(iter);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable reftable_reflog_iterator_vtable = {\n \t.advance = reftable_reflog_iterator_advance,\n \t.peel = reftable_reflog_iterator_peel,\n-\t.abort = reftable_reflog_iterator_abort\n+\t.release = reftable_reflog_iterator_release,\n };\n \n static struct reftable_reflog_iterator *reflog_iterator_for_stack(struct reftable_ref_store *refs,\ndiff --git a/t/helper/test-dir-iterator.c b/t/helper/test-dir-iterator.c\nindex 6b297bd7536..8d46e8ba409 100644\n--- a/t/helper/test-dir-iterator.c\n+++ b/t/helper/test-dir-iterator.c\n@@ -53,6 +53,7 @@ int cmd__dir_iterator(int argc, const char **argv)\n \t\tprintf(\"(%s) [%s] %s\\n\", diter->relative_path, diter->basename,\n \t\t diter->path.buf);\n \t}\n+\tdir_iterator_free(diter);\n \n \tif (iter_status != ITER_DONE) {\n \t\tprintf(\"dir_iterator_advance failure\\n\");\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512693","messageId":"20250219-pks-update-ref-optimization-v2-11-e696e7220b22@pks.im","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im","subject":"[PATCH v2 11/16] refs/iterator: implement seeking for merged iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T13:23:38Z","receivedAt":"2025-02-19T13:23:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking on merged iterators. The implementation is rather\nstraight forward, with the only exception that we must not deallocate\nthe underlying iterators once they have been exhausted.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/iterator.c | 38 +++++++++++++++++++++++++++++---------\n 1 file changed, 29 insertions(+), 9 deletions(-)\n\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex 757b105261a..63608ef9907 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -96,7 +96,8 @@ int is_empty_ref_iterator(struct ref_iterator *ref_iterator)\n struct merge_ref_iterator {\n \tstruct ref_iterator base;\n \n-\tstruct ref_iterator *iter0, *iter1;\n+\tstruct ref_iterator *iter0, *iter0_owned;\n+\tstruct ref_iterator *iter1, *iter1_owned;\n \n \tref_iterator_select_fn *select;\n \tvoid *cb_data;\n@@ -160,13 +161,11 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tif (!iter->current) {\n \t\t/* Initialize: advance both iterators to their first entries */\n \t\tif ((ok = ref_iterator_advance(iter->iter0)) != ITER_OK) {\n-\t\t\tref_iterator_free(iter->iter0);\n \t\t\titer->iter0 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n \t\t}\n \t\tif ((ok = ref_iterator_advance(iter->iter1)) != ITER_OK) {\n-\t\t\tref_iterator_free(iter->iter1);\n \t\t\titer->iter1 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -177,7 +176,6 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t * entry:\n \t\t */\n \t\tif ((ok = ref_iterator_advance(*iter->current)) != ITER_OK) {\n-\t\t\tref_iterator_free(*iter->current);\n \t\t\t*iter->current = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -206,7 +204,6 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \n \t\tif (selection & ITER_SKIP_SECONDARY) {\n \t\t\tif ((ok = ref_iterator_advance(*secondary)) != ITER_OK) {\n-\t\t\t\tref_iterator_free(*secondary);\n \t\t\t\t*secondary = NULL;\n \t\t\t\tif (ok == ITER_ERROR)\n \t\t\t\t\tgoto error;\n@@ -226,6 +223,28 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ITER_ERROR;\n }\n \n+static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct merge_ref_iterator *iter =\n+\t\t(struct merge_ref_iterator *)ref_iterator;\n+\tint ret;\n+\n+\titer->current = NULL;\n+\titer->iter0 = iter->iter0_owned;\n+\titer->iter1 = iter->iter1_owned;\n+\n+\tret = ref_iterator_seek(iter->iter0, prefix);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n+\tret = ref_iterator_seek(iter->iter1, prefix);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n+\treturn 0;\n+}\n+\n static int merge_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -242,12 +261,13 @@ static void merge_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct merge_ref_iterator *iter =\n \t\t(struct merge_ref_iterator *)ref_iterator;\n-\tref_iterator_free(iter->iter0);\n-\tref_iterator_free(iter->iter1);\n+\tref_iterator_free(iter->iter0_owned);\n+\tref_iterator_free(iter->iter1_owned);\n }\n \n static struct ref_iterator_vtable merge_ref_iterator_vtable = {\n \t.advance = merge_ref_iterator_advance,\n+\t.seek = merge_ref_iterator_seek,\n \t.peel = merge_ref_iterator_peel,\n \t.release = merge_ref_iterator_release,\n };\n@@ -268,8 +288,8 @@ struct ref_iterator *merge_ref_iterator_begin(\n \t */\n \n \tbase_ref_iterator_init(ref_iterator, &merge_ref_iterator_vtable);\n-\titer->iter0 = iter0;\n-\titer->iter1 = iter1;\n+\titer->iter0 = iter->iter0_owned = iter0;\n+\titer->iter1 = iter->iter1_owned = iter1;\n \titer->select = select;\n \titer->cb_data = cb_data;\n \titer->current = NULL;\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512694","messageId":"20250219-pks-update-ref-optimization-v2-10-e696e7220b22@pks.im","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im","subject":"[PATCH v2 10/16] refs/iterator: provide infrastructure to re-seek iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T13:23:37Z","receivedAt":"2025-02-19T13:23:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Reftable iterators need to be scrapped after they have either been\nexhausted or aren't useful to the caller anymore, and it is explicitly\nnot possible to reuse them for iterations. But enabling for reuse of\niterators may allow us to tune them by reusing internal state of an\niterator. The reftable iterators for example can already be reused\ninternally, but we're not able to expose this to any users outside of\nthe reftable backend.\n\nIntroduce a new `.seek` function in the ref iterator vtable that allows\ncallers to re-seek an iterator. It is expected to be functionally the\nsame as calling `refs_ref_iterator_begin()` with a different (or the\nsame) prefix.\n\nImplement the callback for trivial cases. The other iterators will be\nimplemented in subsequent commits.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/debug.c | 11 +++++++++++\n refs/iterator.c | 24 ++++++++++++++++++++++++\n refs/refs-internal.h | 23 +++++++++++++++++++++++\n 3 files changed, 58 insertions(+)\n\ndiff --git a/refs/debug.c b/refs/debug.c\nindex a9786da4ba1..5390fa9c187 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -169,6 +169,16 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn res;\n }\n \n+static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct debug_ref_iterator *diter =\n+\t\t(struct debug_ref_iterator *)ref_iterator;\n+\tint res = diter->iter->vtable->seek(diter->iter, prefix);\n+\ttrace_printf_key(&trace_refs, \"iterator_seek: %s: %d\\n\", prefix ? prefix : \"\", res);\n+\treturn res;\n+}\n+\n static int debug_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -189,6 +199,7 @@ static void debug_ref_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable debug_ref_iterator_vtable = {\n \t.advance = debug_ref_iterator_advance,\n+\t.seek = debug_ref_iterator_seek,\n \t.peel = debug_ref_iterator_peel,\n \t.release = debug_ref_iterator_release,\n };\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex aaeff270437..757b105261a 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -15,6 +15,12 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ref_iterator->vtable->advance(ref_iterator);\n }\n \n+int ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t const char *prefix)\n+{\n+\treturn ref_iterator->vtable->seek(ref_iterator, prefix);\n+}\n+\n int ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t struct object_id *peeled)\n {\n@@ -50,6 +56,12 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)\n \treturn ITER_DONE;\n }\n \n+static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n+\t\t\t\t const char *prefix UNUSED)\n+{\n+\treturn 0;\n+}\n+\n static int empty_ref_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \t\t\t\t struct object_id *peeled UNUSED)\n {\n@@ -62,6 +74,7 @@ static void empty_ref_iterator_release(struct ref_iterator *ref_iterator UNUSED)\n \n static struct ref_iterator_vtable empty_ref_iterator_vtable = {\n \t.advance = empty_ref_iterator_advance,\n+\t.seek = empty_ref_iterator_seek,\n \t.peel = empty_ref_iterator_peel,\n \t.release = empty_ref_iterator_release,\n };\n@@ -368,6 +381,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n+static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct prefix_ref_iterator *iter =\n+\t\t(struct prefix_ref_iterator *)ref_iterator;\n+\tfree(iter->prefix);\n+\titer->prefix = xstrdup_or_null(prefix);\n+\treturn ref_iterator_seek(iter->iter0, prefix);\n+}\n+\n static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -387,6 +410,7 @@ static void prefix_ref_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable prefix_ref_iterator_vtable = {\n \t.advance = prefix_ref_iterator_advance,\n+\t.seek = prefix_ref_iterator_seek,\n \t.peel = prefix_ref_iterator_peel,\n \t.release = prefix_ref_iterator_release,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 74e2c03cef1..3f6d43110b7 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -327,6 +327,21 @@ struct ref_iterator {\n */\n int ref_iterator_advance(struct ref_iterator *ref_iterator);\n \n+/*\n+ * Seek the iterator to the first reference with the given prefix.\n+ * The prefix is matched as a literal string, without regard for path\n+ * separators. If prefix is NULL or the empty string, seek the iterator to the\n+ * first reference again.\n+ *\n+ * This function is expected to behave as if a new ref iterator with the same\n+ * prefix had been created, but allows reuse of iterators and thus may allow\n+ * the backend to optimize.\n+ *\n+ * Returns 0 on success, a negative error code otherwise.\n+ */\n+int ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t const char *prefix);\n+\n /*\n * If possible, peel the reference currently being viewed by the\n * iterator. Return 0 on success.\n@@ -445,6 +460,13 @@ void base_ref_iterator_init(struct ref_iterator *iter,\n */\n typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);\n \n+/*\n+ * Seek the iterator to the first reference matching the given prefix. Should\n+ * behave the same as if a new iterator was created with the same prefix.\n+ */\n+typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix);\n+\n /*\n * Peels the current ref, returning 0 for success or -1 for failure.\n */\n@@ -459,6 +481,7 @@ typedef void ref_iterator_release_fn(struct ref_iterator *ref_iterator);\n \n struct ref_iterator_vtable {\n \tref_iterator_advance_fn *advance;\n+\tref_iterator_seek_fn *seek;\n \tref_iterator_peel_fn *peel;\n \tref_iterator_release_fn *release;\n };\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512696","messageId":"20250219-pks-update-ref-optimization-v2-12-e696e7220b22@pks.im","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im","subject":"[PATCH v2 12/16] refs/iterator: implement seeking for reftable iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T13:23:39Z","receivedAt":"2025-02-19T13:23:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking of reftable iterators. As the low-level reftable\niterators already support seeking this change is straight-forward. Two\nnotes though:\n\n - We do not support seeking on reflog iterators.\n\n - We start to check whether `reftable_stack_init_ref_iterator()` is\n successful.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 35 ++++++++++++++++++++++++++++++-----\n 1 file changed, 30 insertions(+), 5 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 06543f79c64..b0c09f34433 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -547,7 +547,7 @@ struct reftable_ref_iterator {\n \tstruct reftable_ref_record ref;\n \tstruct object_id oid;\n \n-\tconst char *prefix;\n+\tchar *prefix;\n \tsize_t prefix_len;\n \tchar **exclude_patterns;\n \tsize_t exclude_patterns_index;\n@@ -718,6 +718,20 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ITER_OK;\n }\n \n+static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct reftable_ref_iterator *iter =\n+\t\t(struct reftable_ref_iterator *)ref_iterator;\n+\n+\tfree(iter->prefix);\n+\titer->prefix = xstrdup_or_null(prefix);\n+\titer->prefix_len = prefix ? strlen(prefix) : 0;\n+\titer->err = reftable_iterator_seek_ref(&iter->iter, prefix);\n+\n+\treturn iter->err;\n+}\n+\n static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -744,10 +758,12 @@ static void reftable_ref_iterator_release(struct ref_iterator *ref_iterator)\n \t\t\tfree(iter->exclude_patterns[i]);\n \t\tfree(iter->exclude_patterns);\n \t}\n+\tfree(iter->prefix);\n }\n \n static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n \t.advance = reftable_ref_iterator_advance,\n+\t.seek = reftable_ref_iterator_seek,\n \t.peel = reftable_ref_iterator_peel,\n \t.release = reftable_ref_iterator_release,\n };\n@@ -806,8 +822,6 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_\n \n \titer = xcalloc(1, sizeof(*iter));\n \tbase_ref_iterator_init(&iter->base, &reftable_ref_iterator_vtable);\n-\titer->prefix = prefix;\n-\titer->prefix_len = prefix ? strlen(prefix) : 0;\n \titer->base.oid = &iter->oid;\n \titer->flags = flags;\n \titer->refs = refs;\n@@ -821,8 +835,11 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_\n \tif (ret)\n \t\tgoto done;\n \n-\treftable_stack_init_ref_iterator(stack, &iter->iter);\n-\tret = reftable_iterator_seek_ref(&iter->iter, prefix);\n+\tret = reftable_stack_init_ref_iterator(stack, &iter->iter);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_ref_iterator_seek(&iter->base, prefix);\n \tif (ret)\n \t\tgoto done;\n \n@@ -2015,6 +2032,13 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ITER_OK;\n }\n \n+static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n+\t\t\t\t\t const char *prefix UNUSED)\n+{\n+\tBUG(\"reftable reflog iterator cannot be seeked\");\n+\treturn -1;\n+}\n+\n static int reftable_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \t\t\t\t\t struct object_id *peeled UNUSED)\n {\n@@ -2033,6 +2057,7 @@ static void reftable_reflog_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable reftable_reflog_iterator_vtable = {\n \t.advance = reftable_reflog_iterator_advance,\n+\t.seek = reftable_reflog_iterator_seek,\n \t.peel = reftable_reflog_iterator_peel,\n \t.release = reftable_reflog_iterator_release,\n };\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512697","messageId":"20250219-pks-update-ref-optimization-v2-13-e696e7220b22@pks.im","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im","subject":"[PATCH v2 13/16] refs/iterator: implement seeking for ref-cache iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T13:23:40Z","receivedAt":"2025-02-19T13:23:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking of ref-cache iterators. This is done by splitting most\nof the logic to seek iterators out of `cache_ref_iterator_begin()` and\nputting it into `cache_ref_iterator_seek()` so that we can reuse the\nlogic.\n\nNote that we cannot use the optimization anymore where we return an\nempty ref iterator when there aren't any references, as otherwise it\nwouldn't be possible to reseek the iterator to a different prefix that\nmay exist. This shouldn't be much of a performance corncern though as we\nnow start to bail out early in case `advance()` sees that there are no\nmore directories to be searched.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/ref-cache.c | 74 ++++++++++++++++++++++++++++++++++++--------------------\n 1 file changed, 48 insertions(+), 26 deletions(-)\n\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 6457e02c1ea..b54547d71ee 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -362,9 +362,7 @@ struct cache_ref_iterator {\n \tstruct ref_iterator base;\n \n \t/*\n-\t * The number of levels currently on the stack. This is always\n-\t * at least 1, because when it becomes zero the iteration is\n-\t * ended and this struct is freed.\n+\t * The number of levels currently on the stack.\n \t */\n \tsize_t levels_nr;\n \n@@ -389,6 +387,9 @@ struct cache_ref_iterator {\n \tstruct cache_ref_iterator_level *levels;\n \n \tstruct repository *repo;\n+\tstruct ref_cache *cache;\n+\n+\tint prime_dir;\n };\n \n static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n@@ -396,6 +397,9 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tstruct cache_ref_iterator *iter =\n \t\t(struct cache_ref_iterator *)ref_iterator;\n \n+\tif (!iter->levels_nr)\n+\t\treturn ITER_DONE;\n+\n \twhile (1) {\n \t\tstruct cache_ref_iterator_level *level =\n \t\t\t&iter->levels[iter->levels_nr - 1];\n@@ -444,6 +448,40 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t}\n }\n \n+static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct cache_ref_iterator *iter =\n+\t\t(struct cache_ref_iterator *)ref_iterator;\n+\tstruct ref_dir *dir;\n+\n+\tdir = get_ref_dir(iter->cache->root);\n+\tif (prefix && *prefix)\n+\t\tdir = find_containing_dir(dir, prefix);\n+\n+\tif (dir) {\n+\t\tstruct cache_ref_iterator_level *level;\n+\n+\t\tif (iter->prime_dir)\n+\t\t\tprime_ref_dir(dir, prefix);\n+\t\titer->levels_nr = 1;\n+\t\tlevel = &iter->levels[0];\n+\t\tlevel->index = -1;\n+\t\tlevel->dir = dir;\n+\n+\t\tif (prefix && *prefix) {\n+\t\t\titer->prefix = xstrdup(prefix);\n+\t\t\tlevel->prefix_state = PREFIX_WITHIN_DIR;\n+\t\t} else {\n+\t\t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n+\t\t}\n+\t} else {\n+\t\titer->levels_nr = 0;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -462,6 +500,7 @@ static void cache_ref_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable cache_ref_iterator_vtable = {\n \t.advance = cache_ref_iterator_advance,\n+\t.seek = cache_ref_iterator_seek,\n \t.peel = cache_ref_iterator_peel,\n \t.release = cache_ref_iterator_release,\n };\n@@ -471,39 +510,22 @@ struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\n \t\t\t\t\t struct repository *repo,\n \t\t\t\t\t int prime_dir)\n {\n-\tstruct ref_dir *dir;\n \tstruct cache_ref_iterator *iter;\n \tstruct ref_iterator *ref_iterator;\n-\tstruct cache_ref_iterator_level *level;\n-\n-\tdir = get_ref_dir(cache->root);\n-\tif (prefix && *prefix)\n-\t\tdir = find_containing_dir(dir, prefix);\n-\tif (!dir)\n-\t\t/* There's nothing to iterate over. */\n-\t\treturn empty_ref_iterator_begin();\n-\n-\tif (prime_dir)\n-\t\tprime_ref_dir(dir, prefix);\n \n \tCALLOC_ARRAY(iter, 1);\n \tref_iterator = &iter->base;\n \tbase_ref_iterator_init(ref_iterator, &cache_ref_iterator_vtable);\n \tALLOC_GROW(iter->levels, 10, iter->levels_alloc);\n \n-\titer->levels_nr = 1;\n-\tlevel = &iter->levels[0];\n-\tlevel->index = -1;\n-\tlevel->dir = dir;\n+\titer->repo = repo;\n+\titer->cache = cache;\n+\titer->prime_dir = prime_dir;\n \n-\tif (prefix && *prefix) {\n-\t\titer->prefix = xstrdup(prefix);\n-\t\tlevel->prefix_state = PREFIX_WITHIN_DIR;\n-\t} else {\n-\t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n+\tif (cache_ref_iterator_seek(&iter->base, prefix) < 0) {\n+\t\tref_iterator_free(&iter->base);\n+\t\treturn NULL;\n \t}\n \n-\titer->repo = repo;\n-\n \treturn ref_iterator;\n }\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512698","messageId":"20250219-pks-update-ref-optimization-v2-14-e696e7220b22@pks.im","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im","subject":"[PATCH v2 14/16] refs/iterator: implement seeking for `packed-ref` iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T13:23:41Z","receivedAt":"2025-02-19T13:23:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking of `packed-ref` iterators. The implementation is again\nstraight forward, except that we cannot continue to use the prefix\niterator as we would otherwise not be able to reseek the iterator\nanymore in case one first asks for an empty and then for a non-empty\nprefix. Instead, we open-code the logic to in `advance()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/packed-backend.c | 62 +++++++++++++++++++++++++++++++++------------------\n 1 file changed, 40 insertions(+), 22 deletions(-)\n\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 38a1956d1a8..71a38acfedc 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -819,6 +819,8 @@ struct packed_ref_iterator {\n \n \tstruct snapshot *snapshot;\n \n+\tchar *prefix;\n+\n \t/* The current position in the snapshot's buffer: */\n \tconst char *pos;\n \n@@ -841,11 +843,9 @@ struct packed_ref_iterator {\n };\n \n /*\n- * Move the iterator to the next record in the snapshot, without\n- * respect for whether the record is actually required by the current\n- * iteration. Adjust the fields in `iter` and return `ITER_OK` or\n- * `ITER_DONE`. This function does not free the iterator in the case\n- * of `ITER_DONE`.\n+ * Move the iterator to the next record in the snapshot. Adjust the fields in\n+ * `iter` and return `ITER_OK` or `ITER_DONE`. This function does not free the\n+ * iterator in the case of `ITER_DONE`.\n */\n static int next_record(struct packed_ref_iterator *iter)\n {\n@@ -942,6 +942,9 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tint ok;\n \n \twhile ((ok = next_record(iter)) == ITER_OK) {\n+\t\tconst char *refname = iter->base.refname;\n+\t\tconst char *prefix = iter->prefix;\n+\n \t\tif (iter->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n \t\t !is_per_worktree_ref(iter->base.refname))\n \t\t\tcontinue;\n@@ -951,12 +954,41 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t\t\t\t &iter->oid, iter->flags))\n \t\t\tcontinue;\n \n+\t\twhile (prefix && *prefix) {\n+\t\t\tif (*refname < *prefix)\n+\t\t\t\tBUG(\"packed-refs backend yielded reference preceding its prefix\");\n+\t\t\telse if (*refname > *prefix)\n+\t\t\t\treturn ITER_DONE;\n+\t\t\tprefix++;\n+\t\t\trefname++;\n+\t\t}\n+\n \t\treturn ITER_OK;\n \t}\n \n \treturn ok;\n }\n \n+static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct packed_ref_iterator *iter =\n+\t\t(struct packed_ref_iterator *)ref_iterator;\n+\tconst char *start;\n+\n+\tif (prefix && *prefix)\n+\t\tstart = find_reference_location(iter->snapshot, prefix, 0);\n+\telse\n+\t\tstart = iter->snapshot->start;\n+\n+\tfree(iter->prefix);\n+\titer->prefix = xstrdup_or_null(prefix);\n+\titer->pos = start;\n+\titer->eof = iter->snapshot->eof;\n+\n+\treturn 0;\n+}\n+\n static int packed_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -979,11 +1011,13 @@ static void packed_ref_iterator_release(struct ref_iterator *ref_iterator)\n \t\t(struct packed_ref_iterator *)ref_iterator;\n \tstrbuf_release(&iter->refname_buf);\n \tfree(iter->jump);\n+\tfree(iter->prefix);\n \trelease_snapshot(iter->snapshot);\n }\n \n static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \t.advance = packed_ref_iterator_advance,\n+\t.seek = packed_ref_iterator_seek,\n \t.peel = packed_ref_iterator_peel,\n \t.release = packed_ref_iterator_release,\n };\n@@ -1097,7 +1131,6 @@ static struct ref_iterator *packed_ref_iterator_begin(\n {\n \tstruct packed_ref_store *refs;\n \tstruct snapshot *snapshot;\n-\tconst char *start;\n \tstruct packed_ref_iterator *iter;\n \tstruct ref_iterator *ref_iterator;\n \tunsigned int required_flags = REF_STORE_READ;\n@@ -1113,14 +1146,6 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \t */\n \tsnapshot = get_snapshot(refs);\n \n-\tif (prefix && *prefix)\n-\t\tstart = find_reference_location(snapshot, prefix, 0);\n-\telse\n-\t\tstart = snapshot->start;\n-\n-\tif (start == snapshot->eof)\n-\t\treturn empty_ref_iterator_begin();\n-\n \tCALLOC_ARRAY(iter, 1);\n \tref_iterator = &iter->base;\n \tbase_ref_iterator_init(ref_iterator, &packed_ref_iterator_vtable);\n@@ -1130,19 +1155,12 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \n \titer->snapshot = snapshot;\n \tacquire_snapshot(snapshot);\n-\n-\titer->pos = start;\n-\titer->eof = snapshot->eof;\n \tstrbuf_init(&iter->refname_buf, 0);\n-\n \titer->base.oid = &iter->oid;\n-\n \titer->repo = ref_store->repo;\n \titer->flags = flags;\n \n-\tif (prefix && *prefix)\n-\t\t/* Stop iteration after we've gone *past* prefix: */\n-\t\tref_iterator = prefix_ref_iterator_begin(ref_iterator, prefix, 0);\n+\tpacked_ref_iterator_seek(&iter->base, prefix);\n \n \treturn ref_iterator;\n }\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512699","messageId":"20250219-pks-update-ref-optimization-v2-16-e696e7220b22@pks.im","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im","subject":"[PATCH v2 16/16] refs: reuse iterators when determining refname availability","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T13:23:43Z","receivedAt":"2025-02-19T13:23:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When verifying whether refnames are available we have to verify whether\nany reference exists that is nested under the current reference. E.g.\ngiven a reference \"refs/heads/foo\", we must make sure that there is no\nother reference \"refs/heads/foo/*\".\n\nThis check is performed using a ref iterator with the prefix set to the\nnested reference namespace. Until now it used to not be possible to\nreseek iterators, so we always had to reallocate the iterator for every\nsingle reference we're about to check. This keeps us from reusing state\nthat the iterator may have and that may make it work more efficiently.\n\nRefactor the logic to reseek iterators. This leads to a sizeable speedup\nwith the \"reftable\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 39.8 ms ± 0.9 ms [User: 29.7 ms, System: 9.8 ms]\n Range (min … max): 38.4 ms … 42.0 ms 62 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 31.9 ms ± 1.1 ms [User: 27.0 ms, System: 4.5 ms]\n Range (min … max): 29.8 ms … 34.3 ms 74 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.25 ± 0.05 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n\nThe \"files\" backend doesn't really show a huge impact:\n\n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 392.3 ms ± 7.1 ms [User: 59.7 ms, System: 328.8 ms]\n Range (min … max): 384.6 ms … 404.5 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 387.7 ms ± 7.4 ms [User: 54.6 ms, System: 329.6 ms]\n Range (min … max): 377.0 ms … 397.7 ms 10 runs\n\n Summary\n update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.01 ± 0.03 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n\nThis is mostly because it is way slower to begin with because it has to\ncreate a separate file for each new reference, so the milliseconds we\nshave off by reseeking the iterator doesn't really translate into a\nsignificant relative improvement.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs.c | 12 +++++++-----\n 1 file changed, 7 insertions(+), 5 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 8eff60a2186..6cbb9decdb0 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2555,8 +2555,13 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \t\tif (!initial_transaction) {\n \t\t\tint ok;\n \n-\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n-\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n+\t\t\tif (!iter) {\n+\t\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n+\t\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n+\t\t\t} else if (ref_iterator_seek(iter, dirname.buf) < 0) {\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\n \t\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\t\t\tif (skip &&\n \t\t\t\t string_list_has_string(skip, iter->refname))\n@@ -2569,9 +2574,6 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \n \t\t\tif (ok != ITER_DONE)\n \t\t\t\tBUG(\"error while iterating over references\");\n-\n-\t\t\tref_iterator_free(iter);\n-\t\t\titer = NULL;\n \t\t}\n \n \t\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512700","messageId":"20250219-pks-update-ref-optimization-v2-15-e696e7220b22@pks.im","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im","subject":"[PATCH v2 15/16] refs/iterator: implement seeking for \"files\" iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-19T13:23:42Z","receivedAt":"2025-02-19T13:23:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking for \"files\" iterators. As we simply use a ref-cache\niterator under the hood the implementation is straight-forward. Note\nthat we do not implement seeking on reflog iterators, same as with the\n\"reftable\" backend.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/files-backend.c | 16 ++++++++++++++++\n 1 file changed, 16 insertions(+)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 859f1c11941..4e1c50fead3 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -918,6 +918,14 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n+static int files_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct files_ref_iterator *iter =\n+\t\t(struct files_ref_iterator *)ref_iterator;\n+\treturn ref_iterator_seek(iter->iter0, prefix);\n+}\n+\n static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -936,6 +944,7 @@ static void files_ref_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable files_ref_iterator_vtable = {\n \t.advance = files_ref_iterator_advance,\n+\t.seek = files_ref_iterator_seek,\n \t.peel = files_ref_iterator_peel,\n \t.release = files_ref_iterator_release,\n };\n@@ -2294,6 +2303,12 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n+static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n+\t\t\t\t const char *prefix UNUSED)\n+{\n+\tBUG(\"ref_iterator_seek() called for reflog_iterator\");\n+}\n+\n static int files_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \t\t\t\t struct object_id *peeled UNUSED)\n {\n@@ -2309,6 +2324,7 @@ static void files_reflog_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable files_reflog_iterator_vtable = {\n \t.advance = files_reflog_iterator_advance,\n+\t.seek = files_reflog_iterator_seek,\n \t.peel = files_reflog_iterator_peel,\n \t.release = files_reflog_iterator_release,\n };\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512720","messageId":"5qatn3wcp4ili3dyz6edzvp5ydsydh523cfz5syqz75afiuvbh@67ndxkvkbxnb","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-1-e696e7220b22@pks.im","subject":"Re: [PATCH v2 01/16] object-name: introduce `repo_get_oid_with_flags()`","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-19T17:02:43Z","receivedAt":"2025-02-19T17:05:52Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/02/19 02:23PM, Patrick Steinhardt wrote:\n[snip]\n> diff --git a/object-name.c b/object-name.c\n> index 945d5bdef25..bc0265ad2a1 100644\n> --- a/object-name.c\n> +++ b/object-name.c\n> @@ -1794,18 +1794,20 @@ void object_context_release(struct object_context *ctx)\n> \tstrbuf_release(&ctx->symlink_path);\n> }\n> \n> -/*\n> - * This is like \"get_oid_basic()\", except it allows \"object ID expressions\",\n> - * notably \"xyz^\" for \"parent of xyz\"\n> - */\n> -int repo_get_oid(struct repository *r, const char *name, struct object_id *oid)\n> +int repo_get_oid_with_flags(struct repository *r, const char *name, struct object_id *oid,\n> +\t\t\t unsigned flags)\n\nstyle: The function signature runs a bit long on the first line. Not a\nbig deal, but we could reformat it.\n\n> {\n> \tstruct object_context unused;\n> -\tint ret = get_oid_with_context(r, name, 0, oid, &unused);\n> +\tint ret = get_oid_with_context(r, name, flags, oid, &unused);\n> \tobject_context_release(&unused);\n> \treturn ret;\n> }\n> \n> +int repo_get_oid(struct repository *r, const char *name, struct object_id *oid)\n> +{\n> +\treturn repo_get_oid_with_flags(r, name, oid, 0);\n> +}\n> +\n> /*\n> * This returns a non-zero value if the string (built using printf\n> * format and the given arguments) is not a valid object.\n> diff --git a/object-name.h b/object-name.h\n> index 8dba4a47a47..fb5a97b2c8e 100644\n> --- a/object-name.h\n> +++ b/object-name.h\n> @@ -51,6 +51,12 @@ void strbuf_repo_add_unique_abbrev(struct strbuf *sb, struct repository *repo,\n> void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n> \t\t\t int abbrev_len);\n> \n> +/*\n> + * This is like \"get_oid_basic()\", except it allows \"object ID expressions\",\n> + * notably \"xyz^\" for \"parent of xyz\". Accepts GET_OID_* flags.\n> + */\n> +int repo_get_oid_with_flags(struct repository *r, const char *str, struct object_id *oid,\n> +\t\t\t unsigned flags);\n\nSame here.\n\n> int repo_get_oid(struct repository *r, const char *str, struct object_id *oid);\n> __attribute__((format (printf, 2, 3)))\n> int get_oidf(struct object_id *oid, const char *fmt, ...);\n> \n> -- \n> 2.48.1.683.gf705b3209c.dirty\n> \n> \n"},{"id":"512726","messageId":"7q2ldwq2mc7vhpqfpj4lg5jc2badt5sgcv6x3eeqjcboraiqxn@5bzhi5n6alxm","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-3-e696e7220b22@pks.im","subject":"Re: [PATCH v2 03/16] builtin/update-ref: skip ambiguity checks when parsing object IDs","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-02-19T18:21:44Z","receivedAt":"2025-02-19T18:24:54Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/02/19 02:23PM, Patrick Steinhardt wrote:\n> Most of the commands in git-update-ref(1) accept an old and/or new\n> object ID to update a specific reference to. These object IDs get parsed\n> via `repo_get_oid()`, which not only handles plain object IDs, but also\n> those that have a suffix like \"~\" or \"^2\". More surprisingly though, it\n> even knows to resolve references, despite the fact that its manpage does\n> not mention this fact even once.\n> \n> One consequence of this is that we also check for ambiguous references:\n> when parsing a full object ID where the DWIM mechanism would also cause\n> us to resolve it as a branch, we'd end up printing a warning. While this\n> check makes sense to have in general, it is arguably less useful in the\n> context of git-update-ref(1). This is out of two reasons:\n> \n> - The manpage is explicitly structured around object IDs. So if we see\n> a fully blown object ID, the intent should be quite clear in\n> general.\n\nMakes sense.\n\n> - The command is part of our plumbing layer and not a tool that users\n> would generally use in interactive workflows. As such, the warning\n> will likely not be visible to anybody in the first place.\n\nOk, so in many cases already the warning is not propagated which makes\nits computation wasteful to begin with.\n\n> Furthermore, this check can be quite expensive when updating lots of\n> references via `--stdin`, because we try to read multiple references per\n> object ID that we parse according to the DWIM rules. This effect can be\n> seen both with the \"files\" and \"reftable\" backend.\n> \n> The issue is not unique to git-update-ref(1), but was also an issue in\n> git-cat-file(1), where it was addressed by disabling the ambiguity check\n> in 25fba78d36b (cat-file: disable object/refname ambiguity check for\n> batch mode, 2013-07-12).\n> \n> Disable the warning in git-update-ref(1), which provides a significant\n> speedup with both backends. The following benchmark creates 10000 new\n> references with a 100000 preexisting refs with the \"files\" backend:\n> \n> Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n> Time (mean ± σ): 467.3 ms ± 5.1 ms [User: 100.0 ms, System: 365.1 ms]\n> Range (min … max): 461.9 ms … 479.3 ms 10 runs\n> \n> Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n> Time (mean ± σ): 394.1 ms ± 5.8 ms [User: 63.3 ms, System: 327.6 ms]\n> Range (min … max): 384.9 ms … 405.7 ms 10 runs\n> \n> Summary\n> update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n> 1.19 ± 0.02 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n> \n> And with the \"reftable\" backend:\n> \n> Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n> Time (mean ± σ): 146.9 ms ± 2.2 ms [User: 90.4 ms, System: 56.0 ms]\n> Range (min … max): 142.7 ms … 150.8 ms 19 runs\n> \n> Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n> Time (mean ± σ): 63.2 ms ± 1.1 ms [User: 41.0 ms, System: 21.8 ms]\n> Range (min … max): 61.1 ms … 66.6 ms 41 runs\n> \n> Summary\n> update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n> 2.32 ± 0.05 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n> \n> Note that the absolute improvement with both backends is roughly in the\n> same ballpark, but the relative improvement for the \"reftable\" backend\n> is more significant because writing the new table to disk is faster in\n> the first place.\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n> builtin/update-ref.c | 12 ++++++++----\n> 1 file changed, 8 insertions(+), 4 deletions(-)\n> \n> diff --git a/builtin/update-ref.c b/builtin/update-ref.c\n> index 4d35bdc4b4b..d603f54b770 100644\n> --- a/builtin/update-ref.c\n> +++ b/builtin/update-ref.c\n> @@ -179,7 +179,8 @@ static int parse_next_oid(const char **next, const char *end,\n> \t\t(*next)++;\n> \t\t*next = parse_arg(*next, &arg);\n> \t\tif (arg.len) {\n> -\t\t\tif (repo_get_oid(the_repository, arg.buf, oid))\n> +\t\t\tif (repo_get_oid_with_flags(the_repository, arg.buf, oid,\n> +\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n> \t\t\t\tgoto invalid;\n> \t\t} else {\n> \t\t\t/* Without -z, an empty value means all zeros: */\n> @@ -197,7 +198,8 @@ static int parse_next_oid(const char **next, const char *end,\n> \t\t*next += arg.len;\n> \n> \t\tif (arg.len) {\n> -\t\t\tif (repo_get_oid(the_repository, arg.buf, oid))\n> +\t\t\tif (repo_get_oid_with_flags(the_repository, arg.buf, oid,\n> +\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n> \t\t\t\tgoto invalid;\n> \t\t} else if (flags & PARSE_SHA1_ALLOW_EMPTY) {\n> \t\t\t/* With -z, treat an empty value as all zeros: */\n> @@ -772,7 +774,8 @@ int cmd_update_ref(int argc,\n> \t\trefname = argv[0];\n> \t\tvalue = argv[1];\n> \t\toldval = argv[2];\n> -\t\tif (repo_get_oid(the_repository, value, &oid))\n> +\t\tif (repo_get_oid_with_flags(the_repository, value, &oid,\n> +\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n> \t\t\tdie(\"%s: not a valid SHA1\", value);\n> \t}\n> \n> @@ -783,7 +786,8 @@ int cmd_update_ref(int argc,\n> \t\t\t * must not already exist:\n> \t\t\t */\n> \t\t\toidclr(&oldoid, the_repository->hash_algo);\n> -\t\telse if (repo_get_oid(the_repository, oldval, &oldoid))\n> +\t\telse if (repo_get_oid_with_flags(the_repository, oldval, &oldoid,\n> +\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n> \t\t\tdie(\"%s: not a valid old SHA1\", oldval);\n> \t}\n\nIn builtin/update-ref.c all uses of repo_get_oid() have been converted\nto repo_get_oid_with_flags() with the GET_OID_SKIP_AMBIGUITY_CHECK flag\nexcept for one in parse_cmd_symref_update(). Is there reason to leave\nthat one untouched?\n\n-Justin\n"},{"id":"512727","messageId":"CAOLa=ZQCRrAXeucZjqMpa-Uhb+d+oxjnDpYX00Qwt6nKGyJ6Xw@mail.gmail.com","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-9-a2b6d87a24af@pks.im","subject":"Re: [PATCH 09/14] refs/iterator: implement seeking for merged iterators","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-02-19T20:10:54Z","receivedAt":"2025-02-19T20:10:56Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> Implement seeking on merged iterators. The implementation is rather\n> straight forward, with the only exception that we must not deallocate\n> the underlying iterators once they have been exhausted.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> refs/iterator.c | 38 +++++++++++++++++++++++++++++---------\n> 1 file changed, 29 insertions(+), 9 deletions(-)\n>\n> diff --git a/refs/iterator.c b/refs/iterator.c\n> index 757b105261a..63608ef9907 100644\n> --- a/refs/iterator.c\n> +++ b/refs/iterator.c\n> @@ -96,7 +96,8 @@ int is_empty_ref_iterator(struct ref_iterator *ref_iterator)\n> struct merge_ref_iterator {\n> \tstruct ref_iterator base;\n>\n> -\tstruct ref_iterator *iter0, *iter1;\n> +\tstruct ref_iterator *iter0, *iter0_owned;\n> +\tstruct ref_iterator *iter1, *iter1_owned;\n>\n> \tref_iterator_select_fn *select;\n> \tvoid *cb_data;\n> @@ -160,13 +161,11 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> \tif (!iter->current) {\n> \t\t/* Initialize: advance both iterators to their first entries */\n> \t\tif ((ok = ref_iterator_advance(iter->iter0)) != ITER_OK) {\n> -\t\t\tref_iterator_free(iter->iter0);\n> \t\t\titer->iter0 = NULL;\n\nOkay so if advancing the iterator fails, we set the current iterator to\nNULL but the underlying pointer `iter0_owned` still holds. Okay makes sense.\n\nNow it's just a matter of ensuring that we reuse the original iterator\nwhen needed and seems like that' what we do in this patch. Looks good!\n\n[snip]\n"},{"id":"512728","messageId":"CAOLa=ZSgpeTCayxCs01hdVJ3FsXMPCzSg+RV5tgMc0Agz9nAqg@mail.gmail.com","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-10-a2b6d87a24af@pks.im","subject":"Re: [PATCH 10/14] refs/iterator: implement seeking for reftable iterators","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-02-19T20:13:48Z","receivedAt":"2025-02-19T20:13:50Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> Implement seeking of reftable iterators. As the low-level reftable\n> iterators already support seeking this change is straight-forward. Two\n> notes though:\n>\n> - We do not support seeking on reflog iterators.\n>\n\nNit: this doesn't explain the reason and it would be nice to state why\nwe do not support reflog iterators here\n\n> - We start to check whether `reftable_stack_init_ref_iterator()` is\n> successful.\n>\n\nThe patch looks good!\n\n[snip]\n"},{"id":"512756","messageId":"Z7biPX-FAefLanh5@pks.im","threadId":"62965","inReplyTo":"7q2ldwq2mc7vhpqfpj4lg5jc2badt5sgcv6x3eeqjcboraiqxn@5bzhi5n6alxm","subject":"Re: [PATCH v2 03/16] builtin/update-ref: skip ambiguity checks when parsing object IDs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-20T08:05:17Z","receivedAt":"2025-02-20T08:05:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Feb 19, 2025 at 12:21:44PM -0600, Justin Tobler wrote:\n> > diff --git a/builtin/update-ref.c b/builtin/update-ref.c\n> > index 4d35bdc4b4b..d603f54b770 100644\n> > --- a/builtin/update-ref.c\n> > +++ b/builtin/update-ref.c\n> > @@ -179,7 +179,8 @@ static int parse_next_oid(const char **next, const char *end,\n> > @@ -783,7 +786,8 @@ int cmd_update_ref(int argc,\n> > \t\t\t * must not already exist:\n> > \t\t\t */\n> > \t\t\toidclr(&oldoid, the_repository->hash_algo);\n> > -\t\telse if (repo_get_oid(the_repository, oldval, &oldoid))\n> > +\t\telse if (repo_get_oid_with_flags(the_repository, oldval, &oldoid,\n> > +\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n> > \t\t\tdie(\"%s: not a valid old SHA1\", oldval);\n> > \t}\n> \n> In builtin/update-ref.c all uses of repo_get_oid() have been converted\n> to repo_get_oid_with_flags() with the GET_OID_SKIP_AMBIGUITY_CHECK flag\n> except for one in parse_cmd_symref_update(). Is there reason to leave\n> that one untouched?\n\nAh, no, this was a mere oversight. Good catch, fixed.\n\nPatrick\n"},{"id":"512806","messageId":"20250221080003.GD1988395@coredump.intra.peff.net","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-2-e696e7220b22@pks.im","subject":"Re: [PATCH v2 02/16] object-name: allow skipping ambiguity checks in `get_oid()` family","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-02-21T08:00:03Z","receivedAt":"2025-02-21T08:00:04Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Feb 19, 2025 at 02:23:29PM +0100, Patrick Steinhardt wrote:\n\n> When reading an object ID via `get_oid_basic()` or any of its related\n> functions we perform a check whether the object ID is ambiguous, which\n> can be the case when a reference with the same name exists. While the\n> check is generally helpful, there are cases where it only adds to the\n> runtime overhead without providing much of a benefit.\n> \n> Add a new flag that allows us to disable the check. The flag will be\n> used in a subsequent commit.\n\nIf we are going to switch to this and get rid of the global\nwarn_on_object_refname_ambiguity flag, I could see it being worth it.\n\nBut when I looked into doing that, it did not make much sense (there are\ntoo many code paths that share the same get_oid calls, and you'd have to\nplumb the flags through the stack).\n\nSo if we are going to leave the global flag anyway, and if your patch 3\nis just changing all of update-ref to pass the per-call flag in every\ncall, why don't we just skip this new mechanism and have update-ref\nunset the warn_on_object_refname_ambiguity flag?\n\nThat makes patch 3 a one-liner, and patches 1 and 2 can go away.\n\n-Peff\n\nPS Sorry, I haven't looked carefully at the rest of the series. I've\n been moving houses and am way back-logged on Git stuff, so don't\n count on me reviewing it anytime soon.\n"},{"id":"512810","messageId":"Z7g69C7Y-CXIUrRn@pks.im","threadId":"62965","inReplyTo":"20250221080003.GD1988395@coredump.intra.peff.net","subject":"Re: [PATCH v2 02/16] object-name: allow skipping ambiguity checks in `get_oid()` family","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-21T08:36:04Z","receivedAt":"2025-02-21T08:36:09Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Feb 21, 2025 at 03:00:03AM -0500, Jeff King wrote:\n> On Wed, Feb 19, 2025 at 02:23:29PM +0100, Patrick Steinhardt wrote:\n> \n> > When reading an object ID via `get_oid_basic()` or any of its related\n> > functions we perform a check whether the object ID is ambiguous, which\n> > can be the case when a reference with the same name exists. While the\n> > check is generally helpful, there are cases where it only adds to the\n> > runtime overhead without providing much of a benefit.\n> > \n> > Add a new flag that allows us to disable the check. The flag will be\n> > used in a subsequent commit.\n> \n> If we are going to switch to this and get rid of the global\n> warn_on_object_refname_ambiguity flag, I could see it being worth it.\n> \n> But when I looked into doing that, it did not make much sense (there are\n> too many code paths that share the same get_oid calls, and you'd have to\n> plumb the flags through the stack).\n> \n> So if we are going to leave the global flag anyway, and if your patch 3\n> is just changing all of update-ref to pass the per-call flag in every\n> call, why don't we just skip this new mechanism and have update-ref\n> unset the warn_on_object_refname_ambiguity flag?\n> \n> That makes patch 3 a one-liner, and patches 1 and 2 can go away.\n> \n> -Peff\n> \n> PS Sorry, I haven't looked carefully at the rest of the series. I've\n> been moving houses and am way back-logged on Git stuff, so don't\n> count on me reviewing it anytime soon.\n\nSpoiler alert: I do have a patch series locally that gets rid of the\nglobal variable completely, and that series builds on top of the new\nflag I'm introducing here. So I'd prefer to keep it so that we can\neventually have less callsites that rely on global state.\n\nPatrick\n"},{"id":"512813","messageId":"20250221090625.GA2518234@coredump.intra.peff.net","threadId":"62965","inReplyTo":"Z7g69C7Y-CXIUrRn@pks.im","subject":"Re: [PATCH v2 02/16] object-name: allow skipping ambiguity checks in `get_oid()` family","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-02-21T09:06:25Z","receivedAt":"2025-02-21T09:06:26Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Feb 21, 2025 at 09:36:04AM +0100, Patrick Steinhardt wrote:\n\n> Spoiler alert: I do have a patch series locally that gets rid of the\n> global variable completely, and that series builds on top of the new\n> flag I'm introducing here. So I'd prefer to keep it so that we can\n> eventually have less callsites that rely on global state.\n\nOK. I do agree that getting rid of the global would be nice. I just\nwasn't sure how ugly it would be to pass the flags through the stack\ninto handle_revision().\n\n-Peff\n"},{"id":"512893","messageId":"Z7xvWPjHjppNp167@ArchLinux","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-10-e696e7220b22@pks.im","subject":"Re: [PATCH v2 10/16] refs/iterator: provide infrastructure to re-seek iterators","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-02-24T13:08:40Z","receivedAt":"2025-02-24T13:08:34Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Wed, Feb 19, 2025 at 02:23:37PM +0100, Patrick Steinhardt wrote:\n> Reftable iterators need to be scrapped after they have either been\n> exhausted or aren't useful to the caller anymore, and it is explicitly\n> not possible to reuse them for iterations. But enabling for reuse of\n> iterators may allow us to tune them by reusing internal state of an\n> iterator. The reftable iterators for example can already be reused\n> internally, but we're not able to expose this to any users outside of\n> the reftable backend.\n> \n\nOut of curiosity, is there any benefits for reusing iterators for files\nbackend?\n\n> Introduce a new `.seek` function in the ref iterator vtable that allows\n> callers to re-seek an iterator. It is expected to be functionally the\n\nIt's a bit strange that we use \"re-seek\". I think we just want to see an\niterator. Isn't it? Don't worth a reroll.\n\n> same as calling `refs_ref_iterator_begin()` with a different (or the\n> same) prefix.\n> \n> Implement the callback for trivial cases. The other iterators will be\n> implemented in subsequent commits.\n> \n\n[snip]\n\n> @@ -368,6 +381,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> \treturn ok;\n> }\n> \n> +static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,\n> +\t\t\t\t const char *prefix)\n> +{\n> +\tstruct prefix_ref_iterator *iter =\n> +\t\t(struct prefix_ref_iterator *)ref_iterator;\n> +\tfree(iter->prefix);\n\nHere, we need to free the \"iter->prefix\". We don't know whether the\ncaller would call `prefix_ref_iterator_seek` many times for the same ref\niterator. So, we need to restore the state/context.\n\nI want to ask a question here, why don't we care about \"trim\" parameter\nwhich is declared in the \"prefix_ref_iterator_begin\"? From my\nunderstanding, we may want to reuse \"trim\" state in the original state.\nSo, when we want to reuse iterator, we only consider about the \"prefix\"\nbut leave the other stats the same. Is my understanding correct?\n\n> +\titer->prefix = xstrdup_or_null(prefix);\n> +\treturn ref_iterator_seek(iter->iter0, prefix);\n> +}\n> +\n\n> diff --git a/refs/refs-internal.h b/refs/refs-internal.h\n> index 74e2c03cef1..3f6d43110b7 100644\n> --- a/refs/refs-internal.h\n> +++ b/refs/refs-internal.h\n> @@ -327,6 +327,21 @@ struct ref_iterator {\n> */\n> int ref_iterator_advance(struct ref_iterator *ref_iterator);\n> \n> +/*\n> + * Seek the iterator to the first reference with the given prefix.\n> + * The prefix is matched as a literal string, without regard for path\n> + * separators. If prefix is NULL or the empty string, seek the iterator to the\n> + * first reference again.\n> + *\n> + * This function is expected to behave as if a new ref iterator with the same\n> + * prefix had been created, but allows reuse of iterators and thus may allow\n> + * the backend to optimize.\n\nI somehow think we may emphasis that we want to reuse some internal\nstates of the ref iterator except the prefix. However, I am not sure.\nJust think about this.\n\n> + *\n> + * Returns 0 on success, a negative error code otherwise.\n> + */\n> +int ref_iterator_seek(struct ref_iterator *ref_iterator,\n> +\t\t const char *prefix);\n> +\n> /*\n> * If possible, peel the reference currently being viewed by the\n> * iterator. Return 0 on success.\n> @@ -445,6 +460,13 @@ void base_ref_iterator_init(struct ref_iterator *iter,\n> */\n> typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);\n> \n> +/*\n> + * Seek the iterator to the first reference matching the given prefix. Should\n\nMaybe which should?\n\n> + * behave the same as if a new iterator was created with the same prefix.\n> + */\n\nThis statement makes me a little confused. I think there are something\ndifference between `seek` and `begin`? For prefix ref iterator, we will\npass \"trim\" when calling `begin`, but for seeking, we don't care about\n\"trim\". Although the prefix maybe the same, but the internal state may\nbe different.\n\nThanks,\nJialuo\n"},{"id":"512894","messageId":"Z7x2IEdRP4fzdXMo@ArchLinux","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-11-e696e7220b22@pks.im","subject":"Re: [PATCH v2 11/16] refs/iterator: implement seeking for merged iterators","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-02-24T13:37:36Z","receivedAt":"2025-02-24T13:37:30Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Wed, Feb 19, 2025 at 02:23:38PM +0100, Patrick Steinhardt wrote:\n> Implement seeking on merged iterators. The implementation is rather\n> straight forward, with the only exception that we must not deallocate\n> the underlying iterators once they have been exhausted.\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n> refs/iterator.c | 38 +++++++++++++++++++++++++++++---------\n> 1 file changed, 29 insertions(+), 9 deletions(-)\n> \n> diff --git a/refs/iterator.c b/refs/iterator.c\n> index 757b105261a..63608ef9907 100644\n> --- a/refs/iterator.c\n> +++ b/refs/iterator.c\n> @@ -96,7 +96,8 @@ int is_empty_ref_iterator(struct ref_iterator *ref_iterator)\n> struct merge_ref_iterator {\n> \tstruct ref_iterator base;\n> \n> -\tstruct ref_iterator *iter0, *iter1;\n> +\tstruct ref_iterator *iter0, *iter0_owned;\n> +\tstruct ref_iterator *iter1, *iter1_owned;\n\nWe would always free `iter0_owned` and `iter1_owned`. That's the reason\nwhy in the below code, we could drop `ref_iterator_free`. Make sense.\n\n[snip]\n\n> +static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,\n> +\t\t\t\t const char *prefix)\n> +{\n> +\tstruct merge_ref_iterator *iter =\n> +\t\t(struct merge_ref_iterator *)ref_iterator;\n> +\tint ret;\n> +\n> +\titer->current = NULL;\n> +\titer->iter0 = iter->iter0_owned;\n> +\titer->iter1 = iter->iter1_owned;\n> +\n> +\tret = ref_iterator_seek(iter->iter0, prefix);\n> +\tif (ret < 0)\n> +\t\treturn ret;\n> +\n> +\tret = ref_iterator_seek(iter->iter1, prefix);\n> +\tif (ret < 0)\n> +\t\treturn ret;\n\nWe could simply use a single `if` statement to handle this. Is the\nreason why we design this is that we want to return the exact error code\nfor each case?\n\n> +\n> +\treturn 0;\n> +}\n> +\n> static int merge_ref_iterator_peel(struct ref_iterator *ref_iterator,\n> \t\t\t\t struct object_id *peeled)\n> {\n> @@ -242,12 +261,13 @@ static void merge_ref_iterator_release(struct ref_iterator *ref_iterator)\n> {\n> \tstruct merge_ref_iterator *iter =\n> \t\t(struct merge_ref_iterator *)ref_iterator;\n> -\tref_iterator_free(iter->iter0);\n> -\tref_iterator_free(iter->iter1);\n> +\tref_iterator_free(iter->iter0_owned);\n> +\tref_iterator_free(iter->iter1_owned);\n\nWe free the internal pointer but not the pointer exposed to the caller.\nMake sense.\n\n> }\n> \n> static struct ref_iterator_vtable merge_ref_iterator_vtable = {\n> \t.advance = merge_ref_iterator_advance,\n> +\t.seek = merge_ref_iterator_seek,\n> \t.peel = merge_ref_iterator_peel,\n> \t.release = merge_ref_iterator_release,\n> };\n> @@ -268,8 +288,8 @@ struct ref_iterator *merge_ref_iterator_begin(\n> \t */\n> \n> \tbase_ref_iterator_init(ref_iterator, &merge_ref_iterator_vtable);\n> -\titer->iter0 = iter0;\n> -\titer->iter1 = iter1;\n> +\titer->iter0 = iter->iter0_owned = iter0;\n> +\titer->iter1 = iter->iter1_owned = iter1;\n\nOK, we would assign `iter0` to `iter0_owned` and `iter1` to `iter1_owned`.\n\nThanks,\nJialuo\n"},{"id":"512895","messageId":"Z7x7fDiW0iN7qGIg@ArchLinux","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-12-e696e7220b22@pks.im","subject":"Re: [PATCH v2 12/16] refs/iterator: implement seeking for reftable iterators","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-02-24T14:00:28Z","receivedAt":"2025-02-24T14:00:25Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Wed, Feb 19, 2025 at 02:23:39PM +0100, Patrick Steinhardt wrote:\n> Implement seeking of reftable iterators. As the low-level reftable\n> iterators already support seeking this change is straight-forward. Two\n> notes though:\n> \n> - We do not support seeking on reflog iterators.\n> \n> - We start to check whether `reftable_stack_init_ref_iterator()` is\n> successful.\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n> refs/reftable-backend.c | 35 ++++++++++++++++++++++++++++++-----\n> 1 file changed, 30 insertions(+), 5 deletions(-)\n> \n> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> index 06543f79c64..b0c09f34433 100644\n> --- a/refs/reftable-backend.c\n> +++ b/refs/reftable-backend.c\n> @@ -547,7 +547,7 @@ struct reftable_ref_iterator {\n> \tstruct reftable_ref_record ref;\n> \tstruct object_id oid;\n> \n> -\tconst char *prefix;\n> +\tchar *prefix;\n> \tsize_t prefix_len;\n> \tchar **exclude_patterns;\n> \tsize_t exclude_patterns_index;\n> @@ -718,6 +718,20 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> \treturn ITER_OK;\n> }\n> \n> +static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,\n> +\t\t\t\t const char *prefix)\n> +{\n> +\tstruct reftable_ref_iterator *iter =\n> +\t\t(struct reftable_ref_iterator *)ref_iterator;\n> +\n> +\tfree(iter->prefix);\n> +\titer->prefix = xstrdup_or_null(prefix);\n> +\titer->prefix_len = prefix ? strlen(prefix) : 0;\n> +\titer->err = reftable_iterator_seek_ref(&iter->iter, prefix);\n\nShould we rename this function `reftable_iterator_seek_ref` by the way?\nIt is a little strange that we have two functions which are so similar:\n\n1. reftable_ref_iterator_seek\n2. reftable_iterator_seek_ref\n\nHowever, don't worth a reroll.\n\n> +\n> +\treturn iter->err;\n> +}\n> +\n> static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n> \t\t\t\t struct object_id *peeled)\n> {\n> @@ -744,10 +758,12 @@ static void reftable_ref_iterator_release(struct ref_iterator *ref_iterator)\n> \t\t\tfree(iter->exclude_patterns[i]);\n> \t\tfree(iter->exclude_patterns);\n> \t}\n> +\tfree(iter->prefix);\n> }\n> \n> static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n> \t.advance = reftable_ref_iterator_advance,\n> +\t.seek = reftable_ref_iterator_seek,\n> \t.peel = reftable_ref_iterator_peel,\n> \t.release = reftable_ref_iterator_release,\n> };\n> @@ -806,8 +822,6 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_\n> \n> \titer = xcalloc(1, sizeof(*iter));\n> \tbase_ref_iterator_init(&iter->base, &reftable_ref_iterator_vtable);\n> -\titer->prefix = prefix;\n> -\titer->prefix_len = prefix ? strlen(prefix) : 0;\n\nWe don't assign `iter->prefix` and `iter->prefix_len` here. This is\nbecause we want to use the new defined function\n`reftable_ref_iterator_seek`. In the fist glance, I am worried that\n\"iter->prefix\" might not be `NULL`. However, because we use `xcalloc`,\n\"iter->prefix\" would be `NULL` by default.\n\nThanks,\nJialuo\n"},{"id":"512905","messageId":"Z7yG6q44rBccInPt@ArchLinux","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-13-e696e7220b22@pks.im","subject":"Re: [PATCH v2 13/16] refs/iterator: implement seeking for ref-cache iterators","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-02-24T14:49:14Z","receivedAt":"2025-02-24T14:49:09Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Wed, Feb 19, 2025 at 02:23:40PM +0100, Patrick Steinhardt wrote:\n> Implement seeking of ref-cache iterators. This is done by splitting most\n> of the logic to seek iterators out of `cache_ref_iterator_begin()` and\n> putting it into `cache_ref_iterator_seek()` so that we can reuse the\n> logic.\n> \n> Note that we cannot use the optimization anymore where we return an\n> empty ref iterator when there aren't any references, as otherwise it\n> wouldn't be possible to reseek the iterator to a different prefix that\n> may exist. This shouldn't be much of a performance corncern though as we\n> now start to bail out early in case `advance()` sees that there are no\n> more directories to be searched.\n> \n\nBit: corncern/concern. Don't worth a reroll.\n\n> Signed-off-by: Patrick Steinhardt \n> ---\n> refs/ref-cache.c | 74 ++++++++++++++++++++++++++++++++++++--------------------\n> 1 file changed, 48 insertions(+), 26 deletions(-)\n> \n> diff --git a/refs/ref-cache.c b/refs/ref-cache.c\n> index 6457e02c1ea..b54547d71ee 100644\n> --- a/refs/ref-cache.c\n> +++ b/refs/ref-cache.c\n> @@ -362,9 +362,7 @@ struct cache_ref_iterator {\n> \tstruct ref_iterator base;\n> \n> \t/*\n> -\t * The number of levels currently on the stack. This is always\n> -\t * at least 1, because when it becomes zero the iteration is\n> -\t * ended and this struct is freed.\n> +\t * The number of levels currently on the stack.\n> \t */\n\nSo, this value could be zero? We want to use this to optimize because\nthat we don't return the empty ref iterator any more.\n\n> \tsize_t levels_nr;\n> \n> @@ -389,6 +387,9 @@ struct cache_ref_iterator {\n> \tstruct cache_ref_iterator_level *levels;\n> \n> \tstruct repository *repo;\n> +\tstruct ref_cache *cache;\n> +\n> +\tint prime_dir;\n\nThe reason why we needs to add these two states is that when using\n`cache_ref_iterator_begin`, we need to pass `ref_cache` and\n`prime_dir`. So, we need to store the state when reusing the ref\niterator.\n\n> };\n> \n> static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> @@ -396,6 +397,9 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> \tstruct cache_ref_iterator *iter =\n> \t\t(struct cache_ref_iterator *)ref_iterator;\n> \n> +\tif (!iter->levels_nr)\n> +\t\treturn ITER_DONE;\n> +\n\nOk, we will check whether the cache ref iterator is exhausted.\n\n> \twhile (1) {\n> \t\tstruct cache_ref_iterator_level *level =\n> \t\t\t&iter->levels[iter->levels_nr - 1];\n> @@ -444,6 +448,40 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> \t}\n> }\n> \n> +static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n> +\t\t\t\t const char *prefix)\n> +{\n> +\tstruct cache_ref_iterator *iter =\n> +\t\t(struct cache_ref_iterator *)ref_iterator;\n> +\tstruct ref_dir *dir;\n> +\n> +\tdir = get_ref_dir(iter->cache->root);\n> +\tif (prefix && *prefix)\n> +\t\tdir = find_containing_dir(dir, prefix);\n> +\n> +\tif (dir) {\n> +\t\tstruct cache_ref_iterator_level *level;\n> +\n> +\t\tif (iter->prime_dir)\n> +\t\t\tprime_ref_dir(dir, prefix);\n> +\t\titer->levels_nr = 1;\n> +\t\tlevel = &iter->levels[0];\n> +\t\tlevel->index = -1;\n> +\t\tlevel->dir = dir;\n> +\n> +\t\tif (prefix && *prefix) {\n> +\t\t\titer->prefix = xstrdup(prefix);\n\nShould we free the original `iter->prefix` before we assign the new\n`prefix`? I have seen this pattern in previous patch. If the caller\ncalls this function multiple times, there would be memory leak.\n\n> +\t\t\tlevel->prefix_state = PREFIX_WITHIN_DIR;\n> +\t\t} else {\n> +\t\t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n> +\t\t}\n> +\t} else {\n> +\t\titer->levels_nr = 0;\n> +\t}\n\nWhen we cannot find the dir, we set the `iter->levels_nr = 0`. Could we\nfirst check\n\n if (!dir) {\n\titer->levels_nr = 0;\n\treturn 0;\n }\n\nAnd thus we could avoid indentation. However, it seems that we always\nreturn 0. So, maybe we should not change.\n\n> +\n> +\treturn 0;\n\nI know your motivation that you want to normally return the ref iterator\nthus we can reuse later. The original behavior is that we return an\nempty ref iterator but empty ref iterator cannot be reused. So, we will\nalways get the cache ref iterator. If the level is 0, we still have a\nvalid cache ref iterator. Make sense.\n\n> +}\n> +\n\nThanks,\nJialuo\n"},{"id":"512906","messageId":"Z7yLrKVxCF5qb7gP@ArchLinux","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-14-e696e7220b22@pks.im","subject":"Re: [PATCH v2 14/16] refs/iterator: implement seeking for `packed-ref` iterators","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-02-24T15:09:32Z","receivedAt":"2025-02-24T15:09:26Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Wed, Feb 19, 2025 at 02:23:41PM +0100, Patrick Steinhardt wrote:\n> Implement seeking of `packed-ref` iterators. The implementation is again\n> straight forward, except that we cannot continue to use the prefix\n> iterator as we would otherwise not be able to reseek the iterator\n> anymore in case one first asks for an empty and then for a non-empty\n> prefix. Instead, we open-code the logic to in `advance()`.\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n> refs/packed-backend.c | 62 +++++++++++++++++++++++++++++++++------------------\n> 1 file changed, 40 insertions(+), 22 deletions(-)\n> \n> diff --git a/refs/packed-backend.c b/refs/packed-backend.c\n> index 38a1956d1a8..71a38acfedc 100644\n> --- a/refs/packed-backend.c\n> +++ b/refs/packed-backend.c\n> @@ -819,6 +819,8 @@ struct packed_ref_iterator {\n> \n> \tstruct snapshot *snapshot;\n> \n> +\tchar *prefix;\n> +\n> \t/* The current position in the snapshot's buffer: */\n> \tconst char *pos;\n> \n> @@ -841,11 +843,9 @@ struct packed_ref_iterator {\n> };\n> \n> /*\n> - * Move the iterator to the next record in the snapshot, without\n> - * respect for whether the record is actually required by the current\n> - * iteration. Adjust the fields in `iter` and return `ITER_OK` or\n> - * `ITER_DONE`. This function does not free the iterator in the case\n> - * of `ITER_DONE`.\n> + * Move the iterator to the next record in the snapshot. Adjust the fields in\n> + * `iter` and return `ITER_OK` or `ITER_DONE`. This function does not free the\n> + * iterator in the case of `ITER_DONE`.\n> */\n> static int next_record(struct packed_ref_iterator *iter)\n> {\n> @@ -942,6 +942,9 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> \tint ok;\n> \n> \twhile ((ok = next_record(iter)) == ITER_OK) {\n> +\t\tconst char *refname = iter->base.refname;\n> +\t\tconst char *prefix = iter->prefix;\n> +\n> \t\tif (iter->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n> \t\t !is_per_worktree_ref(iter->base.refname))\n> \t\t\tcontinue;\n> @@ -951,12 +954,41 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> \t\t\t\t\t &iter->oid, iter->flags))\n> \t\t\tcontinue;\n> \n> +\t\twhile (prefix && *prefix) {\n> +\t\t\tif (*refname < *prefix)\n> +\t\t\t\tBUG(\"packed-refs backend yielded reference preceding its prefix\");\n> +\t\t\telse if (*refname > *prefix)\n> +\t\t\t\treturn ITER_DONE;\n> +\t\t\tprefix++;\n> +\t\t\trefname++;\n> +\t\t}\n\nAlthough I cannot understand the code, I want to ask a question here, we\nwe need to do this in `advance`? Should we check this for\n`packed_ref_iterator_seek` or in the `next_record` function?\n\nBefore we introduce `seek`, we don't need this logic. I somehow think we\nshould do this in `packed_ref_iterator_seek`.\n\n> +\n> \t\treturn ITER_OK;\n> \t}\n> \n> \treturn ok;\n> }\n> \n> +static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,\n> +\t\t\t\t const char *prefix)\n> +{\n> +\tstruct packed_ref_iterator *iter =\n> +\t\t(struct packed_ref_iterator *)ref_iterator;\n> +\tconst char *start;\n> +\n> +\tif (prefix && *prefix)\n> +\t\tstart = find_reference_location(iter->snapshot, prefix, 0);\n> +\telse\n> +\t\tstart = iter->snapshot->start;\n> +\n> +\tfree(iter->prefix);\n> +\titer->prefix = xstrdup_or_null(prefix);\n> +\titer->pos = start;\n> +\titer->eof = iter->snapshot->eof;\n> +\n> +\treturn 0;\n> +}\n> +\n> static int packed_ref_iterator_peel(struct ref_iterator *ref_iterator,\n> \t\t\t\t struct object_id *peeled)\n> {\n> @@ -979,11 +1011,13 @@ static void packed_ref_iterator_release(struct ref_iterator *ref_iterator)\n> \t\t(struct packed_ref_iterator *)ref_iterator;\n> \tstrbuf_release(&iter->refname_buf);\n> \tfree(iter->jump);\n> +\tfree(iter->prefix);\n> \trelease_snapshot(iter->snapshot);\n> }\n> \n> static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n> \t.advance = packed_ref_iterator_advance,\n> +\t.seek = packed_ref_iterator_seek,\n> \t.peel = packed_ref_iterator_peel,\n> \t.release = packed_ref_iterator_release,\n> };\n> @@ -1097,7 +1131,6 @@ static struct ref_iterator *packed_ref_iterator_begin(\n> {\n> \tstruct packed_ref_store *refs;\n> \tstruct snapshot *snapshot;\n> -\tconst char *start;\n> \tstruct packed_ref_iterator *iter;\n> \tstruct ref_iterator *ref_iterator;\n> \tunsigned int required_flags = REF_STORE_READ;\n> @@ -1113,14 +1146,6 @@ static struct ref_iterator *packed_ref_iterator_begin(\n> \t */\n> \tsnapshot = get_snapshot(refs);\n> \n> -\tif (prefix && *prefix)\n> -\t\tstart = find_reference_location(snapshot, prefix, 0);\n> -\telse\n> -\t\tstart = snapshot->start;\n> -\n> -\tif (start == snapshot->eof)\n> -\t\treturn empty_ref_iterator_begin();\n> -\n\nSo, we don't return empty ref iterator. This is the same motivation like\nthe previous patch.\n\n> \tCALLOC_ARRAY(iter, 1);\n> \tref_iterator = &iter->base;\n> \tbase_ref_iterator_init(ref_iterator, &packed_ref_iterator_vtable);\n> @@ -1130,19 +1155,12 @@ static struct ref_iterator *packed_ref_iterator_begin(\n> \n> \titer->snapshot = snapshot;\n> \tacquire_snapshot(snapshot);\n> -\n> -\titer->pos = start;\n> -\titer->eof = snapshot->eof;\n> \tstrbuf_init(&iter->refname_buf, 0);\n> -\n> \titer->base.oid = &iter->oid;\n> -\n> \titer->repo = ref_store->repo;\n> \titer->flags = flags;\n> \n> -\tif (prefix && *prefix)\n> -\t\t/* Stop iteration after we've gone *past* prefix: */\n> -\t\tref_iterator = prefix_ref_iterator_begin(ref_iterator, prefix, 0);\n> +\tpacked_ref_iterator_seek(&iter->base, prefix);\n\nWhy don't we check the return value here? Actually, in the previous\npatch, `cache_ref_iterator_seek` will always return 0, but you still\ncheck. I have thought that you just want to be more defensive.\n\nThanks,\nJialuo\n"},{"id":"512907","messageId":"Z7yMuEzzRdt1GLRo@ArchLinux","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-16-e696e7220b22@pks.im","subject":"Re: [PATCH v2 16/16] refs: reuse iterators when determining refname availability","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-02-24T15:14:00Z","receivedAt":"2025-02-24T15:13:53Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Wed, Feb 19, 2025 at 02:23:43PM +0100, Patrick Steinhardt wrote:\n> When verifying whether refnames are available we have to verify whether\n> any reference exists that is nested under the current reference. E.g.\n> given a reference \"refs/heads/foo\", we must make sure that there is no\n> other reference \"refs/heads/foo/*\".\n> \n> This check is performed using a ref iterator with the prefix set to the\n> nested reference namespace. Until now it used to not be possible to\n> reseek iterators, so we always had to reallocate the iterator for every\n> single reference we're about to check. This keeps us from reusing state\n> that the iterator may have and that may make it work more efficiently.\n> \n> Refactor the logic to reseek iterators. This leads to a sizeable speedup\n> with the \"reftable\" backend:\n> \n> Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n> Time (mean ± σ): 39.8 ms ± 0.9 ms [User: 29.7 ms, System: 9.8 ms]\n> Range (min … max): 38.4 ms … 42.0 ms 62 runs\n> \n> Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n> Time (mean ± σ): 31.9 ms ± 1.1 ms [User: 27.0 ms, System: 4.5 ms]\n> Range (min … max): 29.8 ms … 34.3 ms 74 runs\n> \n> Summary\n> update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n> 1.25 ± 0.05 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n> \n> The \"files\" backend doesn't really show a huge impact:\n> \n> Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n> Time (mean ± σ): 392.3 ms ± 7.1 ms [User: 59.7 ms, System: 328.8 ms]\n> Range (min … max): 384.6 ms … 404.5 ms 10 runs\n> \n> Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n> Time (mean ± σ): 387.7 ms ± 7.4 ms [User: 54.6 ms, System: 329.6 ms]\n> Range (min … max): 377.0 ms … 397.7 ms 10 runs\n> \n> Summary\n> update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n> 1.01 ± 0.03 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n> \n> This is mostly because it is way slower to begin with because it has to\n> create a separate file for each new reference, so the milliseconds we\n> shave off by reseeking the iterator doesn't really translate into a\n> significant relative improvement.\n\nInteresting, because there are many I/O operations which hides the\ncompute latency. Even though we improve the compute speed, the I/O\noperations would still delay the process.\n\nThanks,\nJialuo\n"},{"id":"512908","messageId":"Z7yNsqd_28PE6dVI@ArchLinux","threadId":"62965","inReplyTo":"20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im","subject":"Re: [PATCH v2 00/16] refs: batch refname availability checks","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-02-24T15:18:10Z","receivedAt":"2025-02-24T15:18:04Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Wed, Feb 19, 2025 at 02:23:27PM +0100, Patrick Steinhardt wrote:\n> Hi,\n> \n> this patch series has been inspired by brian's report that the reftable\n> backend is significantly slower when writing many references compared to\n> the files backend. As explained in that thread, the underlying issue is\n> the design of tombstone references: when we first delete all references\n> in a repository and then recreate them, we still have all the tombstones\n> and thus we need to churn through all of them to figure out that they\n> have been deleted in the first place. The files backend does not have\n> this issue.\n> \n> I consider the benchmark itself to be kind of broken, as it stems from\n> us deleting all refs and then recreating them. And if you pack refs in\n> between then the \"reftable\" backend outperforms the \"files\" backend.\n> \n> But there are a couple of opportunities here anyway. While we cannot\n> make the underlying issue of tombstones being less efficient go away,\n> this has prompted me to have a deeper look at where we spend all the\n> time. There are three ideas in this series:\n> \n> - git-update-ref(1) performs ambiguity checks for any full-size object\n> ID, which triggers a lot of reads. This is somewhat pointless though\n> given that the manpage explicitly points out that the command is\n> about object IDs, even though it does know to parse refs. But being\n> part of plumbing, emitting the warning here does not make a ton of\n> sense, and favoring object IDs over references in these cases is the\n> obvious thing to do anyway.\n> \n> - For each ref \"refs/heads/bar\", we need to verify that neither\n> \"refs/heads\" nor \"refs\" exists. This was repeated for every refname,\n> but because most refnames use common prefixes this made us re-check\n> a lot of prefixes. This is addressed by using a `strset` of already\n> checked prefixes.\n> \n> - For each ref \"refs/heads/bar\", we need to verify that no ref\n> \"refs/heads/bar/*\" exists. We always created a new ref iterator for\n> this check, which requires us to discard all internal state and then\n> recreate it. The reftable library has already been refactored though\n> to have reseekable iterators, so we backfill this functionality to\n> all the other iterators and then reuse the iterator.\n> \n> With the (somewhat broken) benchmark we see a small speedup with the\n> \"files\" backend:\n> \n> Benchmark 1: update-ref (refformat = files, revision = master)\n> Time (mean ± σ): 234.4 ms ± 1.9 ms [User: 75.6 ms, System: 157.2 ms]\n> Range (min … max): 232.2 ms … 236.9 ms 10 runs\n> \n> Benchmark 2: update-ref (refformat = files, revision = HEAD)\n> Time (mean ± σ): 184.2 ms ± 2.0 ms [User: 62.8 ms, System: 119.9 ms]\n> Range (min … max): 181.1 ms … 187.0 ms 10 runs\n> \n> Summary\n> update-ref (refformat = files, revision = HEAD) ran\n> 1.27 ± 0.02 times faster than update-ref (refformat = files, revision = master)\n> \n> And a huge speedup with the \"reftable\" backend:\n> \n> Benchmark 1: update-ref (refformat = reftable, revision = master)\n> Time (mean ± σ): 16.852 s ± 0.061 s [User: 16.754 s, System: 0.059 s]\n> Range (min … max): 16.785 s … 16.982 s 10 runs\n> \n> Benchmark 2: update-ref (refformat = reftable, revision = HEAD)\n> Time (mean ± σ): 2.230 s ± 0.009 s [User: 2.192 s, System: 0.029 s]\n> Range (min … max): 2.215 s … 2.244 s 10 runs\n> \n> Summary\n> update-ref (refformat = reftable, revision = HEAD) ran\n> 7.56 ± 0.04 times faster than update-ref (refformat = reftable, revision = master)\n> \n> We're still not up to speed with the \"files\" backend, but considerably\n> better. Given that this is an extreme edge case and not reflective of\n> the general case I'm okay with this result for now.\n> \n> But more importantly, this refactoring also has a positive effect when\n> updating references in a repository with preexisting refs, which I\n> consider to be the more realistic scenario. The following benchmark\n> creates 10k refs with 100k preexisting refs.\n> \n> With the \"files\" backend we see a modest improvement:\n> \n> Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = master)\n> Time (mean ± σ): 478.4 ms ± 11.9 ms [User: 96.7 ms, System: 379.6 ms]\n> Range (min … max): 465.4 ms … 496.6 ms 10 runs\n> \n> Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n> Time (mean ± σ): 388.5 ms ± 10.3 ms [User: 52.0 ms, System: 333.8 ms]\n> Range (min … max): 376.5 ms … 403.1 ms 10 runs\n> \n> Summary\n> update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n> 1.23 ± 0.04 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = master)\n> \n> But with the \"reftable\" backend we see an almost 5x improvement, where\n> it's now ~15x faster than the \"files\" backend:\n> \n> Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = master)\n> Time (mean ± σ): 153.9 ms ± 2.0 ms [User: 96.5 ms, System: 56.6 ms]\n> Range (min … max): 150.5 ms … 158.4 ms 18 runs\n> \n> Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n> Time (mean ± σ): 32.2 ms ± 1.2 ms [User: 27.6 ms, System: 4.3 ms]\n> Range (min … max): 29.8 ms … 38.6 ms 71 runs\n> \n> Summary\n> update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n> 4.78 ± 0.19 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = master)\n> \n\nThis is an amazing improvement.\n\n> The series is structured as follows:\n> \n> - Patches 1 to 4 implement the logic to skip ambiguity checks in\n> git-update-ref(1).\n> \n> - Patch 5 to 8 introduce batched checks.\n> \n> - Patch 9 deduplicates the ref prefix checks.\n> \n> - Patch 10 to 16 implement the infrastructure to reseek iterators.\n> \n> - Patch 17 starts to reuse iterators for nested ref checks.\n> \n> Changes in v2:\n> - Point out why we also have to touch up the `dir_iterator`.\n> - Fix up the comment explaining `ITER_DONE`.\n> - Fix up comments that show usage patterns of the ref and dir iterator\n> interfaces.\n> - Start batching availability checks in the \"files\" backend, as well.\n> - Improve the commit message that drops the ambiguity check so that we\n> also point to 25fba78d36b (cat-file: disable object/refname\n> ambiguity check for batch mode, 2013-07-12).\n> - Link to v1: https://lore.kernel.org/r/20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im\n> \n> Thanks!\n> \n> Patrick\n> \n> [1]: \n\nI have reviewed [PATCH v2 09/16] - [PATCH v2 16/16], leave some\ncomments. For other patches, I don't have energy to review. So maybe\nothers could help.\n\nThanks,\nJialuo\n"},{"id":"512959","messageId":"Z71zwGNcKaX7mGAl@pks.im","threadId":"62965","inReplyTo":"Z7x2IEdRP4fzdXMo@ArchLinux","subject":"Re: [PATCH v2 11/16] refs/iterator: implement seeking for merged iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T07:39:44Z","receivedAt":"2025-02-25T07:39:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Feb 24, 2025 at 09:37:36PM +0800, shejialuo wrote:\n> On Wed, Feb 19, 2025 at 02:23:38PM +0100, Patrick Steinhardt wrote:\n> > diff --git a/refs/iterator.c b/refs/iterator.c\n> > index 757b105261a..63608ef9907 100644\n> > --- a/refs/iterator.c\n> > +++ b/refs/iterator.c\n> > @@ -96,7 +96,8 @@ int is_empty_ref_iterator(struct ref_iterator *ref_iterator)\n> > +static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,\n> > +\t\t\t\t const char *prefix)\n> > +{\n> > +\tstruct merge_ref_iterator *iter =\n> > +\t\t(struct merge_ref_iterator *)ref_iterator;\n> > +\tint ret;\n> > +\n> > +\titer->current = NULL;\n> > +\titer->iter0 = iter->iter0_owned;\n> > +\titer->iter1 = iter->iter1_owned;\n> > +\n> > +\tret = ref_iterator_seek(iter->iter0, prefix);\n> > +\tif (ret < 0)\n> > +\t\treturn ret;\n> > +\n> > +\tret = ref_iterator_seek(iter->iter1, prefix);\n> > +\tif (ret < 0)\n> > +\t\treturn ret;\n> \n> We could simply use a single `if` statement to handle this. Is the\n> reason why we design this is that we want to return the exact error code\n> for each case?\n\nYup, I don't want to loose the error code. We could write this as:\n\n if ((ret = ref_iterator_seek(iter->iter0, prefix)) < 0 ||\n (ret = ref_iterator_seek(iter->iter0, prefix)) < 0)\n return ret;\n\nBut assigning to variables in conditions is not something we typically\ndo in the Git codebase.\n\nPatrick\n"},{"id":"512960","messageId":"Z71zua2Aoq8DC6ve@pks.im","threadId":"62965","inReplyTo":"Z7xvWPjHjppNp167@ArchLinux","subject":"Re: [PATCH v2 10/16] refs/iterator: provide infrastructure to re-seek iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T07:39:37Z","receivedAt":"2025-02-25T07:39:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Feb 24, 2025 at 09:08:40PM +0800, shejialuo wrote:\n> On Wed, Feb 19, 2025 at 02:23:37PM +0100, Patrick Steinhardt wrote:\n> > Reftable iterators need to be scrapped after they have either been\n> > exhausted or aren't useful to the caller anymore, and it is explicitly\n> > not possible to reuse them for iterations. But enabling for reuse of\n> > iterators may allow us to tune them by reusing internal state of an\n> > iterator. The reftable iterators for example can already be reused\n> > internally, but we're not able to expose this to any users outside of\n> > the reftable backend.\n> > \n> \n> Out of curiosity, is there any benefits for reusing iterators for files\n> backend?\n\nWe see a small improvement there due to reduced allocation churn, but\noverall the benefit is more limited there, as shown by the last commit\nin this series where we start to reuse iterators. The refactorings may\nunlock further optimization potential by reusing more of the iterators'\nstate, but I haven't checked for the \"files\" backend.\n\n> > Introduce a new `.seek` function in the ref iterator vtable that allows\n> > callers to re-seek an iterator. It is expected to be functionally the\n> \n> It's a bit strange that we use \"re-seek\". I think we just want to see an\n> iterator. Isn't it? Don't worth a reroll.\n\nWell, it's really only relevant in the case where we want to seek on an\niterator multiple times because we already seek when creating the\niterator. I'll reformulate this slightly.\n\n> > @@ -368,6 +381,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> > \treturn ok;\n> > }\n> > \n> > +static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,\n> > +\t\t\t\t const char *prefix)\n> > +{\n> > +\tstruct prefix_ref_iterator *iter =\n> > +\t\t(struct prefix_ref_iterator *)ref_iterator;\n> > +\tfree(iter->prefix);\n> \n> Here, we need to free the \"iter->prefix\". We don't know whether the\n> caller would call `prefix_ref_iterator_seek` many times for the same ref\n> iterator. So, we need to restore the state/context.\n> \n> I want to ask a question here, why don't we care about \"trim\" parameter\n> which is declared in the \"prefix_ref_iterator_begin\"? From my\n> understanding, we may want to reuse \"trim\" state in the original state.\n> So, when we want to reuse iterator, we only consider about the \"prefix\"\n> but leave the other stats the same. Is my understanding correct?\n\nYeah, it's correct. I couldn't find a usecase for also adjusting \"trim\"\nwhen seeking multiple times, so I didn't introduce this functionality.\nLet me mention this in the commit message.\n\n> > +\titer->prefix = xstrdup_or_null(prefix);\n> > +\treturn ref_iterator_seek(iter->iter0, prefix);\n> > +}\n> > +\n> \n> > diff --git a/refs/refs-internal.h b/refs/refs-internal.h\n> > index 74e2c03cef1..3f6d43110b7 100644\n> > --- a/refs/refs-internal.h\n> > +++ b/refs/refs-internal.h\n> > @@ -327,6 +327,21 @@ struct ref_iterator {\n> > */\n> > int ref_iterator_advance(struct ref_iterator *ref_iterator);\n> > \n> > +/*\n> > + * Seek the iterator to the first reference with the given prefix.\n> > + * The prefix is matched as a literal string, without regard for path\n> > + * separators. If prefix is NULL or the empty string, seek the iterator to the\n> > + * first reference again.\n> > + *\n> > + * This function is expected to behave as if a new ref iterator with the same\n> > + * prefix had been created, but allows reuse of iterators and thus may allow\n> > + * the backend to optimize.\n> \n> I somehow think we may emphasis that we want to reuse some internal\n> states of the ref iterator except the prefix. However, I am not sure.\n> Just think about this.\n\nYup, I'll improve the comment.\n\n> > @@ -445,6 +460,13 @@ void base_ref_iterator_init(struct ref_iterator *iter,\n> > */\n> > typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);\n> > \n> > +/*\n> > + * Seek the iterator to the first reference matching the given prefix. Should\n> \n> Maybe which should?\n> \n> > + * behave the same as if a new iterator was created with the same prefix.\n> > + */\n> \n> This statement makes me a little confused. I think there are something\n> difference between `seek` and `begin`? For prefix ref iterator, we will\n> pass \"trim\" when calling `begin`, but for seeking, we don't care about\n> \"trim\". Although the prefix maybe the same, but the internal state may\n> be different.\n\nHere, as well.\n\nPatrick\n"},{"id":"512961","messageId":"Z71zxD17Ib5iydyr@pks.im","threadId":"62965","inReplyTo":"Z7x7fDiW0iN7qGIg@ArchLinux","subject":"Re: [PATCH v2 12/16] refs/iterator: implement seeking for reftable iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T07:39:48Z","receivedAt":"2025-02-25T07:39:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Feb 24, 2025 at 10:00:28PM +0800, shejialuo wrote:\n> On Wed, Feb 19, 2025 at 02:23:39PM +0100, Patrick Steinhardt wrote:\n> > diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> > index 06543f79c64..b0c09f34433 100644\n> > --- a/refs/reftable-backend.c\n> > +++ b/refs/reftable-backend.c\n> > @@ -718,6 +718,20 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> > \treturn ITER_OK;\n> > }\n> > \n> > +static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,\n> > +\t\t\t\t const char *prefix)\n> > +{\n> > +\tstruct reftable_ref_iterator *iter =\n> > +\t\t(struct reftable_ref_iterator *)ref_iterator;\n> > +\n> > +\tfree(iter->prefix);\n> > +\titer->prefix = xstrdup_or_null(prefix);\n> > +\titer->prefix_len = prefix ? strlen(prefix) : 0;\n> > +\titer->err = reftable_iterator_seek_ref(&iter->iter, prefix);\n> \n> Should we rename this function `reftable_iterator_seek_ref` by the way?\n> It is a little strange that we have two functions which are so similar:\n> \n> 1. reftable_ref_iterator_seek\n> 2. reftable_iterator_seek_ref\n> \n> However, don't worth a reroll.\n\nWell, they do similar things, but at different levels:\n\n - `reftable_ref_iterator_seek()` operates on the high-level generic\n `struct ref_iterator`.\n\n - `reftable_iterator_seek_ref()` operates on the low-level `struct\n reftable_ref_iterator` provided by the reftable library.\n\nAs such I think that they are named appropriately as their prefixes tell\nus which structure they operate on.\n\nPatrick\n"},{"id":"512962","messageId":"Z71zxwWA3VzM82Gr@pks.im","threadId":"62965","inReplyTo":"Z7yG6q44rBccInPt@ArchLinux","subject":"Re: [PATCH v2 13/16] refs/iterator: implement seeking for ref-cache iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T07:39:51Z","receivedAt":"2025-02-25T07:39:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Feb 24, 2025 at 10:49:14PM +0800, shejialuo wrote:\n> On Wed, Feb 19, 2025 at 02:23:40PM +0100, Patrick Steinhardt wrote:\n> > diff --git a/refs/ref-cache.c b/refs/ref-cache.c\n> > index 6457e02c1ea..b54547d71ee 100644\n> > --- a/refs/ref-cache.c\n> > +++ b/refs/ref-cache.c\n> > @@ -362,9 +362,7 @@ struct cache_ref_iterator {\n> > \tstruct ref_iterator base;\n> > \n> > \t/*\n> > -\t * The number of levels currently on the stack. This is always\n> > -\t * at least 1, because when it becomes zero the iteration is\n> > -\t * ended and this struct is freed.\n> > +\t * The number of levels currently on the stack.\n> > \t */\n> \n> So, this value could be zero? We want to use this to optimize because\n> that we don't return the empty ref iterator any more.\n\nNow it can, yes. Before it couldn't as we returned an empty iterator in\nthat case, but we cannot do anymore because you cannot re-seek an empty\niterator.\n\n> > @@ -444,6 +448,40 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> > \t}\n> > }\n> > \n> > +static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n> > +\t\t\t\t const char *prefix)\n> > +{\n> > +\tstruct cache_ref_iterator *iter =\n> > +\t\t(struct cache_ref_iterator *)ref_iterator;\n> > +\tstruct ref_dir *dir;\n> > +\n> > +\tdir = get_ref_dir(iter->cache->root);\n> > +\tif (prefix && *prefix)\n> > +\t\tdir = find_containing_dir(dir, prefix);\n> > +\n> > +\tif (dir) {\n> > +\t\tstruct cache_ref_iterator_level *level;\n> > +\n> > +\t\tif (iter->prime_dir)\n> > +\t\t\tprime_ref_dir(dir, prefix);\n> > +\t\titer->levels_nr = 1;\n> > +\t\tlevel = &iter->levels[0];\n> > +\t\tlevel->index = -1;\n> > +\t\tlevel->dir = dir;\n> > +\n> > +\t\tif (prefix && *prefix) {\n> > +\t\t\titer->prefix = xstrdup(prefix);\n> \n> Should we free the original `iter->prefix` before we assign the new\n> `prefix`? I have seen this pattern in previous patch. If the caller\n> calls this function multiple times, there would be memory leak.\n\nOh, good catch, yes.\n\n> > +\t\t\tlevel->prefix_state = PREFIX_WITHIN_DIR;\n> > +\t\t} else {\n> > +\t\t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n> > +\t\t}\n> > +\t} else {\n> > +\t\titer->levels_nr = 0;\n> > +\t}\n> \n> When we cannot find the dir, we set the `iter->levels_nr = 0`. Could we\n> first check\n> \n> if (!dir) {\n> \titer->levels_nr = 0;\n> \treturn 0;\n> }\n> \n> And thus we could avoid indentation. However, it seems that we always\n> return 0. So, maybe we should not change.\n\nGood idea.\n\nPatrick\n"},{"id":"512963","messageId":"Z71zyrai9f9dPZRK@pks.im","threadId":"62965","inReplyTo":"Z7yLrKVxCF5qb7gP@ArchLinux","subject":"Re: [PATCH v2 14/16] refs/iterator: implement seeking for `packed-ref` iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T07:39:54Z","receivedAt":"2025-02-25T07:39:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Feb 24, 2025 at 11:09:32PM +0800, shejialuo wrote:\n> On Wed, Feb 19, 2025 at 02:23:41PM +0100, Patrick Steinhardt wrote:\n> > Implement seeking of `packed-ref` iterators. The implementation is again\n> > straight forward, except that we cannot continue to use the prefix\n> > iterator as we would otherwise not be able to reseek the iterator\n> > anymore in case one first asks for an empty and then for a non-empty\n> > prefix. Instead, we open-code the logic to in `advance()`.\n> > \n> > Signed-off-by: Patrick Steinhardt \n> > ---\n> > refs/packed-backend.c | 62 +++++++++++++++++++++++++++++++++------------------\n> > 1 file changed, 40 insertions(+), 22 deletions(-)\n> > \n> > diff --git a/refs/packed-backend.c b/refs/packed-backend.c\n> > index 38a1956d1a8..71a38acfedc 100644\n> > --- a/refs/packed-backend.c\n> > +++ b/refs/packed-backend.c\n> > @@ -951,12 +954,41 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> > \t\t\t\t\t &iter->oid, iter->flags))\n> > \t\t\tcontinue;\n> > \n> > +\t\twhile (prefix && *prefix) {\n> > +\t\t\tif (*refname < *prefix)\n> > +\t\t\t\tBUG(\"packed-refs backend yielded reference preceding its prefix\");\n> > +\t\t\telse if (*refname > *prefix)\n> > +\t\t\t\treturn ITER_DONE;\n> > +\t\t\tprefix++;\n> > +\t\t\trefname++;\n> > +\t\t}\n> \n> Although I cannot understand the code, I want to ask a question here, we\n> we need to do this in `advance`? Should we check this for\n> `packed_ref_iterator_seek` or in the `next_record` function?\n> \n> Before we introduce `seek`, we don't need this logic. I somehow think we\n> should do this in `packed_ref_iterator_seek`.\n\nWe cannot do this in `packed_ref_iterator_seek()` because we need to do\nit for every single record that we yield from the iterator. We _could_\ndo it in `next_record()`, but that function is rather complex already\nand really only cares about yielding the next record. On the other hand,\n`advance()` already knows to skip certain entries, so putting the logic\nin there to also handle termination feels like a natural fit to me.\n\n> > @@ -1130,19 +1155,12 @@ static struct ref_iterator *packed_ref_iterator_begin(\n> > \n> > \titer->snapshot = snapshot;\n> > \tacquire_snapshot(snapshot);\n> > -\n> > -\titer->pos = start;\n> > -\titer->eof = snapshot->eof;\n> > \tstrbuf_init(&iter->refname_buf, 0);\n> > -\n> > \titer->base.oid = &iter->oid;\n> > -\n> > \titer->repo = ref_store->repo;\n> > \titer->flags = flags;\n> > \n> > -\tif (prefix && *prefix)\n> > -\t\t/* Stop iteration after we've gone *past* prefix: */\n> > -\t\tref_iterator = prefix_ref_iterator_begin(ref_iterator, prefix, 0);\n> > +\tpacked_ref_iterator_seek(&iter->base, prefix);\n> \n> Why don't we check the return value here? Actually, in the previous\n> patch, `cache_ref_iterator_seek` will always return 0, but you still\n> check. I have thought that you just want to be more defensive.\n\nYeah, let's have the check over here, as well.\n\nPatrick\n"},{"id":"512964","messageId":"Z71zzeDc0AS4ujPS@pks.im","threadId":"62965","inReplyTo":"Z7yNsqd_28PE6dVI@ArchLinux","subject":"Re: [PATCH v2 00/16] refs: batch refname availability checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T07:39:57Z","receivedAt":"2025-02-25T07:40:01Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Feb 24, 2025 at 11:18:10PM +0800, shejialuo wrote:\n> I have reviewed [PATCH v2 09/16] - [PATCH v2 16/16], leave some\n> comments. For other patches, I don't have energy to review. So maybe\n> others could help.\n\nThanks a lot for your review!\n\nPatrick\n"},{"id":"512972","messageId":"20250225-pks-update-ref-optimization-v3-0-77c3687cda75@pks.im","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im","subject":"[PATCH v3 00/16] refs: batch refname availability checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T08:55:46Z","receivedAt":"2025-02-25T08:56:01Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series has been inspired by brian's report that the reftable\nbackend is significantly slower when writing many references compared to\nthe files backend. As explained in that thread, the underlying issue is\nthe design of tombstone references: when we first delete all references\nin a repository and then recreate them, we still have all the tombstones\nand thus we need to churn through all of them to figure out that they\nhave been deleted in the first place. The files backend does not have\nthis issue.\n\nI consider the benchmark itself to be kind of broken, as it stems from\nus deleting all refs and then recreating them. And if you pack refs in\nbetween then the \"reftable\" backend outperforms the \"files\" backend.\n\nBut there are a couple of opportunities here anyway. While we cannot\nmake the underlying issue of tombstones being less efficient go away,\nthis has prompted me to have a deeper look at where we spend all the\ntime. There are three ideas in this series:\n\n - git-update-ref(1) performs ambiguity checks for any full-size object\n ID, which triggers a lot of reads. This is somewhat pointless though\n given that the manpage explicitly points out that the command is\n about object IDs, even though it does know to parse refs. But being\n part of plumbing, emitting the warning here does not make a ton of\n sense, and favoring object IDs over references in these cases is the\n obvious thing to do anyway.\n\n - For each ref \"refs/heads/bar\", we need to verify that neither\n \"refs/heads\" nor \"refs\" exists. This was repeated for every refname,\n but because most refnames use common prefixes this made us re-check\n a lot of prefixes. This is addressed by using a `strset` of already\n checked prefixes.\n\n - For each ref \"refs/heads/bar\", we need to verify that no ref\n \"refs/heads/bar/*\" exists. We always created a new ref iterator for\n this check, which requires us to discard all internal state and then\n recreate it. The reftable library has already been refactored though\n to have reseekable iterators, so we backfill this functionality to\n all the other iterators and then reuse the iterator.\n\nWith the (somewhat broken) benchmark we see a small speedup with the\n\"files\" backend:\n\n Benchmark 1: update-ref (refformat = files, revision = master)\n Time (mean ± σ): 234.4 ms ± 1.9 ms [User: 75.6 ms, System: 157.2 ms]\n Range (min … max): 232.2 ms … 236.9 ms 10 runs\n\n Benchmark 2: update-ref (refformat = files, revision = HEAD)\n Time (mean ± σ): 184.2 ms ± 2.0 ms [User: 62.8 ms, System: 119.9 ms]\n Range (min … max): 181.1 ms … 187.0 ms 10 runs\n\n Summary\n update-ref (refformat = files, revision = HEAD) ran\n 1.27 ± 0.02 times faster than update-ref (refformat = files, revision = master)\n\nAnd a huge speedup with the \"reftable\" backend:\n\n Benchmark 1: update-ref (refformat = reftable, revision = master)\n Time (mean ± σ): 16.852 s ± 0.061 s [User: 16.754 s, System: 0.059 s]\n Range (min … max): 16.785 s … 16.982 s 10 runs\n\n Benchmark 2: update-ref (refformat = reftable, revision = HEAD)\n Time (mean ± σ): 2.230 s ± 0.009 s [User: 2.192 s, System: 0.029 s]\n Range (min … max): 2.215 s … 2.244 s 10 runs\n\n Summary\n update-ref (refformat = reftable, revision = HEAD) ran\n 7.56 ± 0.04 times faster than update-ref (refformat = reftable, revision = master)\n\nWe're still not up to speed with the \"files\" backend, but considerably\nbetter. Given that this is an extreme edge case and not reflective of\nthe general case I'm okay with this result for now.\n\nBut more importantly, this refactoring also has a positive effect when\nupdating references in a repository with preexisting refs, which I\nconsider to be the more realistic scenario. The following benchmark\ncreates 10k refs with 100k preexisting refs.\n\nWith the \"files\" backend we see a modest improvement:\n\n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = master)\n Time (mean ± σ): 478.4 ms ± 11.9 ms [User: 96.7 ms, System: 379.6 ms]\n Range (min … max): 465.4 ms … 496.6 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 388.5 ms ± 10.3 ms [User: 52.0 ms, System: 333.8 ms]\n Range (min … max): 376.5 ms … 403.1 ms 10 runs\n\n Summary\n update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.23 ± 0.04 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = master)\n\nBut with the \"reftable\" backend we see an almost 5x improvement, where\nit's now ~15x faster than the \"files\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = master)\n Time (mean ± σ): 153.9 ms ± 2.0 ms [User: 96.5 ms, System: 56.6 ms]\n Range (min … max): 150.5 ms … 158.4 ms 18 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 32.2 ms ± 1.2 ms [User: 27.6 ms, System: 4.3 ms]\n Range (min … max): 29.8 ms … 38.6 ms 71 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 4.78 ± 0.19 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = master)\n\nThe series is structured as follows:\n\n - Patches 1 to 4 implement the logic to skip ambiguity checks in\n git-update-ref(1).\n\n - Patch 5 to 8 introduce batched checks.\n\n - Patch 9 deduplicates the ref prefix checks.\n\n - Patch 10 to 16 implement the infrastructure to reseek iterators.\n\n - Patch 17 starts to reuse iterators for nested ref checks.\n\nChanges in v2:\n - Point out why we also have to touch up the `dir_iterator`.\n - Fix up the comment explaining `ITER_DONE`.\n - Fix up comments that show usage patterns of the ref and dir iterator\n interfaces.\n - Start batching availability checks in the \"files\" backend, as well.\n - Improve the commit message that drops the ambiguity check so that we\n also point to 25fba78d36b (cat-file: disable object/refname\n ambiguity check for batch mode, 2013-07-12).\n - Link to v1: https://lore.kernel.org/r/20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im\n\nChanges in v3:\n - Fix one case where we didn't skip ambiguity checks in\n git-update-ref(1).\n - Document better that only the prefix can change on reseeking\n iterators. Other internal state will remain the same.\n - Fix a memory leak in the ref-cache iterator.\n - Don't ignore errors returned by `packed_ref_iterator_seek()`.\n - Link to v2: https://lore.kernel.org/r/20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im\n\nThanks!\n\nPatrick\n\n[1]: \n\n---\nPatrick Steinhardt (16):\n object-name: introduce `repo_get_oid_with_flags()`\n object-name: allow skipping ambiguity checks in `get_oid()` family\n builtin/update-ref: skip ambiguity checks when parsing object IDs\n refs: introduce function to batch refname availability checks\n refs/reftable: batch refname availability checks\n refs/files: batch refname availability checks for normal transactions\n refs/files: batch refname availability checks for initial transactions\n refs: stop re-verifying common prefixes for availability\n refs/iterator: separate lifecycle from iteration\n refs/iterator: provide infrastructure to re-seek iterators\n refs/iterator: implement seeking for merged iterators\n refs/iterator: implement seeking for reftable iterators\n refs/iterator: implement seeking for ref-cache iterators\n refs/iterator: implement seeking for packed-ref iterators\n refs/iterator: implement seeking for files iterators\n refs: reuse iterators when determining refname availability\n\n builtin/clone.c | 2 +\n builtin/update-ref.c | 15 ++--\n dir-iterator.c | 24 +++---\n dir-iterator.h | 11 +--\n hash.h | 1 +\n iterator.h | 2 +-\n object-name.c | 18 +++--\n object-name.h | 6 ++\n refs.c | 186 ++++++++++++++++++++++++++-----------------\n refs.h | 12 +++\n refs/debug.c | 20 +++--\n refs/files-backend.c | 117 +++++++++++++++++----------\n refs/iterator.c | 145 +++++++++++++++++----------------\n refs/packed-backend.c | 92 ++++++++++++---------\n refs/ref-cache.c | 88 ++++++++++++--------\n refs/refs-internal.h | 53 +++++++-----\n refs/reftable-backend.c | 85 +++++++++++---------\n t/helper/test-dir-iterator.c | 1 +\n 18 files changed, 528 insertions(+), 350 deletions(-)\n\nRange-diff versus v2:\n\n 1: b7b3e057628 ! 1: 34198fbc1c0 object-name: introduce `repo_get_oid_with_flags()`\n @@ object-name.c: void object_context_release(struct object_context *ctx)\n - * notably \"xyz^\" for \"parent of xyz\"\n - */\n -int repo_get_oid(struct repository *r, const char *name, struct object_id *oid)\n -+int repo_get_oid_with_flags(struct repository *r, const char *name, struct object_id *oid,\n -+\t\t\t unsigned flags)\n ++int repo_get_oid_with_flags(struct repository *r, const char *name,\n ++\t\t\t struct object_id *oid, unsigned flags)\n {\n \tstruct object_context unused;\n -\tint ret = get_oid_with_context(r, name, 0, oid, &unused);\n @@ object-name.h: void strbuf_repo_add_unique_abbrev(struct strbuf *sb, struct repo\n + * This is like \"get_oid_basic()\", except it allows \"object ID expressions\",\n + * notably \"xyz^\" for \"parent of xyz\". Accepts GET_OID_* flags.\n + */\n -+int repo_get_oid_with_flags(struct repository *r, const char *str, struct object_id *oid,\n -+\t\t\t unsigned flags);\n ++int repo_get_oid_with_flags(struct repository *r, const char *str,\n ++\t\t\t struct object_id *oid, unsigned flags);\n int repo_get_oid(struct repository *r, const char *str, struct object_id *oid);\n __attribute__((format (printf, 2, 3)))\n int get_oidf(struct object_id *oid, const char *fmt, ...);\n 2: 3cba1ffa8aa = 2: f5d4d2a67ce object-name: allow skipping ambiguity checks in `get_oid()` family\n 3: 80dfc2ee6b7 ! 3: 0512c256641 builtin/update-ref: skip ambiguity checks when parsing object IDs\n @@ builtin/update-ref.c: static int parse_next_oid(const char **next, const char *e\n \t\t\t\tgoto invalid;\n \t\t} else if (flags & PARSE_SHA1_ALLOW_EMPTY) {\n \t\t\t/* With -z, treat an empty value as all zeros: */\n +@@ builtin/update-ref.c: static void parse_cmd_symref_update(struct ref_transaction *transaction,\n + \t\t\tdie(\"symref-update %s: expected old value\", refname);\n + \n + \t\tif (!strcmp(old_arg, \"oid\")) {\n +-\t\t\tif (repo_get_oid(the_repository, old_target, &old_oid))\n ++\t\t\tif (repo_get_oid_with_flags(the_repository, old_target, &old_oid,\n ++\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n + \t\t\t\tdie(\"symref-update %s: invalid oid: %s\", refname, old_target);\n + \n + \t\t\thave_old_oid = 1;\n @@ builtin/update-ref.c: int cmd_update_ref(int argc,\n \t\trefname = argv[0];\n \t\tvalue = argv[1];\n 4: 9bd05801ac0 = 4: ad7bf4bed31 refs: introduce function to batch refname availability checks\n 5: 0104c9759aa = 5: 132f09c4584 refs/reftable: batch refname availability checks\n 6: fba06e5fcb7 = 6: bd7e5fa7bf1 refs/files: batch refname availability checks for normal transactions\n 7: dfb4be26147 = 7: a907f62d2c2 refs/files: batch refname availability checks for initial transactions\n 8: e7abe4bae25 = 8: 866e5f4b4cc refs: stop re-verifying common prefixes for availability\n 9: 1deae95c53a = 9: 736f1bd9afd refs/iterator: separate lifecycle from iteration\n10: 8b942563e65 ! 10: 71d4c4c4655 refs/iterator: provide infrastructure to re-seek iterators\n @@ Commit message\n the reftable backend.\n \n Introduce a new `.seek` function in the ref iterator vtable that allows\n - callers to re-seek an iterator. It is expected to be functionally the\n - same as calling `refs_ref_iterator_begin()` with a different (or the\n - same) prefix.\n + callers to seek an iterator multiple times. It is expected to be\n + functionally the same as calling `refs_ref_iterator_begin()` with a\n + different (or the same) prefix.\n +\n + Note that it is not possible to adjust parameters other than the seeked\n + prefix for now, so exclude patterns, trimmed prefixes and flags will\n + remain unchanged. We do not have a usecase for changing these parameters\n + right now, but if we ever find one we can adapt accordingly.\n \n Implement the callback for trivial cases. The other iterators will be\n implemented in subsequent commits.\n @@ refs/refs-internal.h: struct ref_iterator {\n + *\n + * This function is expected to behave as if a new ref iterator with the same\n + * prefix had been created, but allows reuse of iterators and thus may allow\n -+ * the backend to optimize.\n ++ * the backend to optimize. Parameters other than the prefix that have been\n ++ * passed when creating the iterator will remain unchanged.\n + *\n + * Returns 0 on success, a negative error code otherwise.\n + */\n11: ad4f063ef06 = 11: 5a0412d754b refs/iterator: implement seeking for merged iterators\n12: ddac957862f ! 12: ece7e500ecd refs/iterator: implement seeking for reftable iterators\n @@ Commit message\n iterators already support seeking this change is straight-forward. Two\n notes though:\n \n - - We do not support seeking on reflog iterators.\n + - We do not support seeking on reflog iterators. It is unclear what\n + seeking would even look like in this context, as you typically would\n + want to seek to a specific entry in the reflog for a specific ref.\n + There is not currently a usecase for this, but if there ever is we\n + can implement seeking in the future.\n \n - We start to check whether `reftable_stack_init_ref_iterator()` is\n successful.\n13: 87b81552acf ! 13: f693de656b5 refs/iterator: implement seeking for ref-cache iterators\n @@ Commit message\n Note that we cannot use the optimization anymore where we return an\n empty ref iterator when there aren't any references, as otherwise it\n wouldn't be possible to reseek the iterator to a different prefix that\n - may exist. This shouldn't be much of a performance corncern though as we\n + may exist. This shouldn't be much of a performance concern though as we\n now start to bail out early in case `advance()` sees that there are no\n more directories to be searched.\n \n @@ refs/ref-cache.c: struct cache_ref_iterator {\n \t */\n \tsize_t levels_nr;\n \n +@@ refs/ref-cache.c: struct cache_ref_iterator {\n + \t * The prefix is matched textually, without regard for path\n + \t * component boundaries.\n + \t */\n +-\tconst char *prefix;\n ++\tchar *prefix;\n + \n + \t/*\n + \t * A stack of levels. levels[0] is the uppermost level that is\n @@ refs/ref-cache.c: struct cache_ref_iterator {\n \tstruct cache_ref_iterator_level *levels;\n \n @@ refs/ref-cache.c: static int cache_ref_iterator_advance(struct ref_iterator *ref\n +{\n +\tstruct cache_ref_iterator *iter =\n +\t\t(struct cache_ref_iterator *)ref_iterator;\n ++\tstruct cache_ref_iterator_level *level;\n +\tstruct ref_dir *dir;\n +\n +\tdir = get_ref_dir(iter->cache->root);\n +\tif (prefix && *prefix)\n +\t\tdir = find_containing_dir(dir, prefix);\n ++\tif (!dir) {\n ++\t\titer->levels_nr = 0;\n ++\t\treturn 0;\n ++\t}\n +\n -+\tif (dir) {\n -+\t\tstruct cache_ref_iterator_level *level;\n -+\n -+\t\tif (iter->prime_dir)\n -+\t\t\tprime_ref_dir(dir, prefix);\n -+\t\titer->levels_nr = 1;\n -+\t\tlevel = &iter->levels[0];\n -+\t\tlevel->index = -1;\n -+\t\tlevel->dir = dir;\n ++\tif (iter->prime_dir)\n ++\t\tprime_ref_dir(dir, prefix);\n ++\titer->levels_nr = 1;\n ++\tlevel = &iter->levels[0];\n ++\tlevel->index = -1;\n ++\tlevel->dir = dir;\n +\n -+\t\tif (prefix && *prefix) {\n -+\t\t\titer->prefix = xstrdup(prefix);\n -+\t\t\tlevel->prefix_state = PREFIX_WITHIN_DIR;\n -+\t\t} else {\n -+\t\t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n -+\t\t}\n ++\tif (prefix && *prefix) {\n ++\t\tfree(iter->prefix);\n ++\t\titer->prefix = xstrdup(prefix);\n ++\t\tlevel->prefix_state = PREFIX_WITHIN_DIR;\n +\t} else {\n -+\t\titer->levels_nr = 0;\n ++\t\tFREE_AND_NULL(iter->prefix);\n ++\t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n +\t}\n +\n +\treturn 0;\n @@ refs/ref-cache.c: static int cache_ref_iterator_advance(struct ref_iterator *ref\n \t\t\t\t struct object_id *peeled)\n {\n @@ refs/ref-cache.c: static void cache_ref_iterator_release(struct ref_iterator *ref_iterator)\n + {\n + \tstruct cache_ref_iterator *iter =\n + \t\t(struct cache_ref_iterator *)ref_iterator;\n +-\tfree((char *)iter->prefix);\n ++\tfree(iter->prefix);\n + \tfree(iter->levels);\n + }\n \n static struct ref_iterator_vtable cache_ref_iterator_vtable = {\n \t.advance = cache_ref_iterator_advance,\n14: 2619de30fe1 ! 14: ac71647ee94 refs/iterator: implement seeking for `packed-ref` iterators\n @@ Metadata\n Author: Patrick Steinhardt \n \n ## Commit message ##\n - refs/iterator: implement seeking for `packed-ref` iterators\n + refs/iterator: implement seeking for packed-ref iterators\n \n Implement seeking of `packed-ref` iterators. The implementation is again\n straight forward, except that we cannot continue to use the prefix\n @@ refs/packed-backend.c: static struct ref_iterator *packed_ref_iterator_begin(\n -\tif (prefix && *prefix)\n -\t\t/* Stop iteration after we've gone *past* prefix: */\n -\t\tref_iterator = prefix_ref_iterator_begin(ref_iterator, prefix, 0);\n -+\tpacked_ref_iterator_seek(&iter->base, prefix);\n ++\tif (packed_ref_iterator_seek(&iter->base, prefix) < 0) {\n ++\t\tref_iterator_free(&iter->base);\n ++\t\treturn NULL;\n ++\t}\n \n \treturn ref_iterator;\n }\n15: d4f76e6480b ! 15: 02cafca513c refs/iterator: implement seeking for \"files\" iterators\n @@ Metadata\n Author: Patrick Steinhardt \n \n ## Commit message ##\n - refs/iterator: implement seeking for \"files\" iterators\n + refs/iterator: implement seeking for files iterators\n \n Implement seeking for \"files\" iterators. As we simply use a ref-cache\n iterator under the hood the implementation is straight-forward. Note\n16: 49017050289 = 16: baed7615a97 refs: reuse iterators when determining refname availability\n\n---\nbase-commit: e2067b49ecaef9b7f51a17ce251f9207f72ef52d\nchange-id: 20250217-pks-update-ref-optimization-15c795e66e2b\n\n"},{"id":"512974","messageId":"20250225-pks-update-ref-optimization-v3-1-77c3687cda75@pks.im","threadId":"62965","inReplyTo":"20250225-pks-update-ref-optimization-v3-0-77c3687cda75@pks.im","subject":"[PATCH v3 01/16] object-name: introduce `repo_get_oid_with_flags()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T08:55:47Z","receivedAt":"2025-02-25T08:56:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new function `repo_get_oid_with_flags()`. This function\nbehaves the same as `repo_get_oid()`, except that it takes an extra\n`flags` parameter that it ends up passing to `get_oid_with_context()`.\n\nThis function will be used in a subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-name.c | 14 ++++++++------\n object-name.h | 6 ++++++\n 2 files changed, 14 insertions(+), 6 deletions(-)\n\ndiff --git a/object-name.c b/object-name.c\nindex 945d5bdef25..233f3f861e3 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -1794,18 +1794,20 @@ void object_context_release(struct object_context *ctx)\n \tstrbuf_release(&ctx->symlink_path);\n }\n \n-/*\n- * This is like \"get_oid_basic()\", except it allows \"object ID expressions\",\n- * notably \"xyz^\" for \"parent of xyz\"\n- */\n-int repo_get_oid(struct repository *r, const char *name, struct object_id *oid)\n+int repo_get_oid_with_flags(struct repository *r, const char *name,\n+\t\t\t struct object_id *oid, unsigned flags)\n {\n \tstruct object_context unused;\n-\tint ret = get_oid_with_context(r, name, 0, oid, &unused);\n+\tint ret = get_oid_with_context(r, name, flags, oid, &unused);\n \tobject_context_release(&unused);\n \treturn ret;\n }\n \n+int repo_get_oid(struct repository *r, const char *name, struct object_id *oid)\n+{\n+\treturn repo_get_oid_with_flags(r, name, oid, 0);\n+}\n+\n /*\n * This returns a non-zero value if the string (built using printf\n * format and the given arguments) is not a valid object.\ndiff --git a/object-name.h b/object-name.h\nindex 8dba4a47a47..cda4934cd5f 100644\n--- a/object-name.h\n+++ b/object-name.h\n@@ -51,6 +51,12 @@ void strbuf_repo_add_unique_abbrev(struct strbuf *sb, struct repository *repo,\n void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n \t\t\t int abbrev_len);\n \n+/*\n+ * This is like \"get_oid_basic()\", except it allows \"object ID expressions\",\n+ * notably \"xyz^\" for \"parent of xyz\". Accepts GET_OID_* flags.\n+ */\n+int repo_get_oid_with_flags(struct repository *r, const char *str,\n+\t\t\t struct object_id *oid, unsigned flags);\n int repo_get_oid(struct repository *r, const char *str, struct object_id *oid);\n __attribute__((format (printf, 2, 3)))\n int get_oidf(struct object_id *oid, const char *fmt, ...);\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512973","messageId":"20250225-pks-update-ref-optimization-v3-2-77c3687cda75@pks.im","threadId":"62965","inReplyTo":"20250225-pks-update-ref-optimization-v3-0-77c3687cda75@pks.im","subject":"[PATCH v3 02/16] object-name: allow skipping ambiguity checks in `get_oid()` family","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T08:55:48Z","receivedAt":"2025-02-25T08:56:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When reading an object ID via `get_oid_basic()` or any of its related\nfunctions we perform a check whether the object ID is ambiguous, which\ncan be the case when a reference with the same name exists. While the\ncheck is generally helpful, there are cases where it only adds to the\nruntime overhead without providing much of a benefit.\n\nAdd a new flag that allows us to disable the check. The flag will be\nused in a subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n hash.h | 1 +\n object-name.c | 4 +++-\n 2 files changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/hash.h b/hash.h\nindex 4367acfec50..79419016513 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -204,6 +204,7 @@ struct object_id {\n #define GET_OID_ONLY_TO_DIE 04000\n #define GET_OID_REQUIRE_PATH 010000\n #define GET_OID_HASH_ANY 020000\n+#define GET_OID_SKIP_AMBIGUITY_CHECK 040000\n \n #define GET_OID_DISAMBIGUATORS \\\n \t(GET_OID_COMMIT | GET_OID_COMMITTISH | \\\ndiff --git a/object-name.c b/object-name.c\nindex 233f3f861e3..85444dbb15b 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -961,7 +961,9 @@ static int get_oid_basic(struct repository *r, const char *str, int len,\n \tint fatal = !(flags & GET_OID_QUIETLY);\n \n \tif (len == r->hash_algo->hexsz && !get_oid_hex(str, oid)) {\n-\t\tif (repo_settings_get_warn_ambiguous_refs(r) && warn_on_object_refname_ambiguity) {\n+\t\tif (!(flags & GET_OID_SKIP_AMBIGUITY_CHECK) &&\n+\t\t repo_settings_get_warn_ambiguous_refs(r) &&\n+\t\t warn_on_object_refname_ambiguity) {\n \t\t\trefs_found = repo_dwim_ref(r, str, len, &tmp_oid, &real_ref, 0);\n \t\t\tif (refs_found > 0) {\n \t\t\t\twarning(warn_msg, len, str);\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512975","messageId":"20250225-pks-update-ref-optimization-v3-3-77c3687cda75@pks.im","threadId":"62965","inReplyTo":"20250225-pks-update-ref-optimization-v3-0-77c3687cda75@pks.im","subject":"[PATCH v3 03/16] builtin/update-ref: skip ambiguity checks when parsing object IDs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T08:55:49Z","receivedAt":"2025-02-25T08:56:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Most of the commands in git-update-ref(1) accept an old and/or new\nobject ID to update a specific reference to. These object IDs get parsed\nvia `repo_get_oid()`, which not only handles plain object IDs, but also\nthose that have a suffix like \"~\" or \"^2\". More surprisingly though, it\neven knows to resolve references, despite the fact that its manpage does\nnot mention this fact even once.\n\nOne consequence of this is that we also check for ambiguous references:\nwhen parsing a full object ID where the DWIM mechanism would also cause\nus to resolve it as a branch, we'd end up printing a warning. While this\ncheck makes sense to have in general, it is arguably less useful in the\ncontext of git-update-ref(1). This is out of two reasons:\n\n - The manpage is explicitly structured around object IDs. So if we see\n a fully blown object ID, the intent should be quite clear in\n general.\n\n - The command is part of our plumbing layer and not a tool that users\n would generally use in interactive workflows. As such, the warning\n will likely not be visible to anybody in the first place.\n\nFurthermore, this check can be quite expensive when updating lots of\nreferences via `--stdin`, because we try to read multiple references per\nobject ID that we parse according to the DWIM rules. This effect can be\nseen both with the \"files\" and \"reftable\" backend.\n\nThe issue is not unique to git-update-ref(1), but was also an issue in\ngit-cat-file(1), where it was addressed by disabling the ambiguity check\nin 25fba78d36b (cat-file: disable object/refname ambiguity check for\nbatch mode, 2013-07-12).\n\nDisable the warning in git-update-ref(1), which provides a significant\nspeedup with both backends. The following benchmark creates 10000 new\nreferences with a 100000 preexisting refs with the \"files\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 467.3 ms ± 5.1 ms [User: 100.0 ms, System: 365.1 ms]\n Range (min … max): 461.9 ms … 479.3 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 394.1 ms ± 5.8 ms [User: 63.3 ms, System: 327.6 ms]\n Range (min … max): 384.9 ms … 405.7 ms 10 runs\n\n Summary\n update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.19 ± 0.02 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n\nAnd with the \"reftable\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 146.9 ms ± 2.2 ms [User: 90.4 ms, System: 56.0 ms]\n Range (min … max): 142.7 ms … 150.8 ms 19 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 63.2 ms ± 1.1 ms [User: 41.0 ms, System: 21.8 ms]\n Range (min … max): 61.1 ms … 66.6 ms 41 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 2.32 ± 0.05 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n\nNote that the absolute improvement with both backends is roughly in the\nsame ballpark, but the relative improvement for the \"reftable\" backend\nis more significant because writing the new table to disk is faster in\nthe first place.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/update-ref.c | 15 ++++++++++-----\n 1 file changed, 10 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/update-ref.c b/builtin/update-ref.c\nindex 4d35bdc4b4b..1d541e13ade 100644\n--- a/builtin/update-ref.c\n+++ b/builtin/update-ref.c\n@@ -179,7 +179,8 @@ static int parse_next_oid(const char **next, const char *end,\n \t\t(*next)++;\n \t\t*next = parse_arg(*next, &arg);\n \t\tif (arg.len) {\n-\t\t\tif (repo_get_oid(the_repository, arg.buf, oid))\n+\t\t\tif (repo_get_oid_with_flags(the_repository, arg.buf, oid,\n+\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\t\tgoto invalid;\n \t\t} else {\n \t\t\t/* Without -z, an empty value means all zeros: */\n@@ -197,7 +198,8 @@ static int parse_next_oid(const char **next, const char *end,\n \t\t*next += arg.len;\n \n \t\tif (arg.len) {\n-\t\t\tif (repo_get_oid(the_repository, arg.buf, oid))\n+\t\t\tif (repo_get_oid_with_flags(the_repository, arg.buf, oid,\n+\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\t\tgoto invalid;\n \t\t} else if (flags & PARSE_SHA1_ALLOW_EMPTY) {\n \t\t\t/* With -z, treat an empty value as all zeros: */\n@@ -299,7 +301,8 @@ static void parse_cmd_symref_update(struct ref_transaction *transaction,\n \t\t\tdie(\"symref-update %s: expected old value\", refname);\n \n \t\tif (!strcmp(old_arg, \"oid\")) {\n-\t\t\tif (repo_get_oid(the_repository, old_target, &old_oid))\n+\t\t\tif (repo_get_oid_with_flags(the_repository, old_target, &old_oid,\n+\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\t\tdie(\"symref-update %s: invalid oid: %s\", refname, old_target);\n \n \t\t\thave_old_oid = 1;\n@@ -772,7 +775,8 @@ int cmd_update_ref(int argc,\n \t\trefname = argv[0];\n \t\tvalue = argv[1];\n \t\toldval = argv[2];\n-\t\tif (repo_get_oid(the_repository, value, &oid))\n+\t\tif (repo_get_oid_with_flags(the_repository, value, &oid,\n+\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\tdie(\"%s: not a valid SHA1\", value);\n \t}\n \n@@ -783,7 +787,8 @@ int cmd_update_ref(int argc,\n \t\t\t * must not already exist:\n \t\t\t */\n \t\t\toidclr(&oldoid, the_repository->hash_algo);\n-\t\telse if (repo_get_oid(the_repository, oldval, &oldoid))\n+\t\telse if (repo_get_oid_with_flags(the_repository, oldval, &oldoid,\n+\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\tdie(\"%s: not a valid old SHA1\", oldval);\n \t}\n \n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512976","messageId":"20250225-pks-update-ref-optimization-v3-4-77c3687cda75@pks.im","threadId":"62965","inReplyTo":"20250225-pks-update-ref-optimization-v3-0-77c3687cda75@pks.im","subject":"[PATCH v3 04/16] refs: introduce function to batch refname availability checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T08:55:50Z","receivedAt":"2025-02-25T08:56:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `refs_verify_refname_available()` functions checks whether a\nreference update can be committed or whether it would conflict with\neither a prefix or suffix thereof. This function needs to be called once\nper reference that one wants to check, which requires us to redo a\ncouple of checks every time the function is called.\n\nIntroduce a new function `refs_verify_refnames_available()` that does\nthe same, but for a list of references. For now, the new function uses\nthe exact same implementation, except that we loop through all refnames\nprovided by the caller. This will be tuned in subsequent commits.\n\nThe existing `refs_verify_refname_available()` function is reimplemented\non top of the new function. As such, the diff is best viewed with the\n`--ignore-space-change option`.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs.c | 169 +++++++++++++++++++++++++++++++++++++----------------------------\n refs.h | 12 +++++\n 2 files changed, 109 insertions(+), 72 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex f4094a326a9..5a9b0f2fa1e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2467,19 +2467,15 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \treturn ret;\n }\n \n-int refs_verify_refname_available(struct ref_store *refs,\n-\t\t\t\t const char *refname,\n-\t\t\t\t const struct string_list *extras,\n-\t\t\t\t const struct string_list *skip,\n-\t\t\t\t unsigned int initial_transaction,\n-\t\t\t\t struct strbuf *err)\n+int refs_verify_refnames_available(struct ref_store *refs,\n+\t\t\t\t const struct string_list *refnames,\n+\t\t\t\t const struct string_list *extras,\n+\t\t\t\t const struct string_list *skip,\n+\t\t\t\t unsigned int initial_transaction,\n+\t\t\t\t struct strbuf *err)\n {\n-\tconst char *slash;\n-\tconst char *extra_refname;\n \tstruct strbuf dirname = STRBUF_INIT;\n \tstruct strbuf referent = STRBUF_INIT;\n-\tstruct object_id oid;\n-\tunsigned int type;\n \tint ret = -1;\n \n \t/*\n@@ -2489,79 +2485,91 @@ int refs_verify_refname_available(struct ref_store *refs,\n \n \tassert(err);\n \n-\tstrbuf_grow(&dirname, strlen(refname) + 1);\n-\tfor (slash = strchr(refname, '/'); slash; slash = strchr(slash + 1, '/')) {\n-\t\t/*\n-\t\t * Just saying \"Is a directory\" when we e.g. can't\n-\t\t * lock some multi-level ref isn't very informative,\n-\t\t * the user won't be told *what* is a directory, so\n-\t\t * let's not use strerror() below.\n-\t\t */\n-\t\tint ignore_errno;\n-\t\t/* Expand dirname to the new prefix, not including the trailing slash: */\n-\t\tstrbuf_add(&dirname, refname + dirname.len, slash - refname - dirname.len);\n+\tfor (size_t i = 0; i < refnames->nr; i++) {\n+\t\tconst char *refname = refnames->items[i].string;\n+\t\tconst char *extra_refname;\n+\t\tstruct object_id oid;\n+\t\tunsigned int type;\n+\t\tconst char *slash;\n+\n+\t\tstrbuf_reset(&dirname);\n+\n+\t\tfor (slash = strchr(refname, '/'); slash; slash = strchr(slash + 1, '/')) {\n+\t\t\t/*\n+\t\t\t * Just saying \"Is a directory\" when we e.g. can't\n+\t\t\t * lock some multi-level ref isn't very informative,\n+\t\t\t * the user won't be told *what* is a directory, so\n+\t\t\t * let's not use strerror() below.\n+\t\t\t */\n+\t\t\tint ignore_errno;\n+\n+\t\t\t/* Expand dirname to the new prefix, not including the trailing slash: */\n+\t\t\tstrbuf_add(&dirname, refname + dirname.len, slash - refname - dirname.len);\n+\n+\t\t\t/*\n+\t\t\t * We are still at a leading dir of the refname (e.g.,\n+\t\t\t * \"refs/foo\"; if there is a reference with that name,\n+\t\t\t * it is a conflict, *unless* it is in skip.\n+\t\t\t */\n+\t\t\tif (skip && string_list_has_string(skip, dirname.buf))\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (!initial_transaction &&\n+\t\t\t !refs_read_raw_ref(refs, dirname.buf, &oid, &referent,\n+\t\t\t\t\t &type, &ignore_errno)) {\n+\t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n+\t\t\t\t\t dirname.buf, refname);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\n+\t\t\tif (extras && string_list_has_string(extras, dirname.buf)) {\n+\t\t\t\tstrbuf_addf(err, _(\"cannot process '%s' and '%s' at the same time\"),\n+\t\t\t\t\t refname, dirname.buf);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\t\t}\n \n \t\t/*\n-\t\t * We are still at a leading dir of the refname (e.g.,\n-\t\t * \"refs/foo\"; if there is a reference with that name,\n-\t\t * it is a conflict, *unless* it is in skip.\n+\t\t * We are at the leaf of our refname (e.g., \"refs/foo/bar\").\n+\t\t * There is no point in searching for a reference with that\n+\t\t * name, because a refname isn't considered to conflict with\n+\t\t * itself. But we still need to check for references whose\n+\t\t * names are in the \"refs/foo/bar/\" namespace, because they\n+\t\t * *do* conflict.\n \t\t */\n-\t\tif (skip && string_list_has_string(skip, dirname.buf))\n-\t\t\tcontinue;\n+\t\tstrbuf_addstr(&dirname, refname + dirname.len);\n+\t\tstrbuf_addch(&dirname, '/');\n+\n+\t\tif (!initial_transaction) {\n+\t\t\tstruct ref_iterator *iter;\n+\t\t\tint ok;\n+\n+\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n+\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n+\t\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n+\t\t\t\tif (skip &&\n+\t\t\t\t string_list_has_string(skip, iter->refname))\n+\t\t\t\t\tcontinue;\n+\n+\t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n+\t\t\t\t\t iter->refname, refname);\n+\t\t\t\tref_iterator_abort(iter);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \n-\t\tif (!initial_transaction &&\n-\t\t !refs_read_raw_ref(refs, dirname.buf, &oid, &referent,\n-\t\t\t\t &type, &ignore_errno)) {\n-\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n-\t\t\t\t dirname.buf, refname);\n-\t\t\tgoto cleanup;\n+\t\t\tif (ok != ITER_DONE)\n+\t\t\t\tBUG(\"error while iterating over references\");\n \t\t}\n \n-\t\tif (extras && string_list_has_string(extras, dirname.buf)) {\n+\t\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n+\t\tif (extra_refname) {\n \t\t\tstrbuf_addf(err, _(\"cannot process '%s' and '%s' at the same time\"),\n-\t\t\t\t refname, dirname.buf);\n+\t\t\t\t refname, extra_refname);\n \t\t\tgoto cleanup;\n \t\t}\n \t}\n \n-\t/*\n-\t * We are at the leaf of our refname (e.g., \"refs/foo/bar\").\n-\t * There is no point in searching for a reference with that\n-\t * name, because a refname isn't considered to conflict with\n-\t * itself. But we still need to check for references whose\n-\t * names are in the \"refs/foo/bar/\" namespace, because they\n-\t * *do* conflict.\n-\t */\n-\tstrbuf_addstr(&dirname, refname + dirname.len);\n-\tstrbuf_addch(&dirname, '/');\n-\n-\tif (!initial_transaction) {\n-\t\tstruct ref_iterator *iter;\n-\t\tint ok;\n-\n-\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n-\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n-\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n-\t\t\tif (skip &&\n-\t\t\t string_list_has_string(skip, iter->refname))\n-\t\t\t\tcontinue;\n-\n-\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n-\t\t\t\t iter->refname, refname);\n-\t\t\tref_iterator_abort(iter);\n-\t\t\tgoto cleanup;\n-\t\t}\n-\n-\t\tif (ok != ITER_DONE)\n-\t\t\tBUG(\"error while iterating over references\");\n-\t}\n-\n-\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n-\tif (extra_refname)\n-\t\tstrbuf_addf(err, _(\"cannot process '%s' and '%s' at the same time\"),\n-\t\t\t refname, extra_refname);\n-\telse\n-\t\tret = 0;\n+\tret = 0;\n \n cleanup:\n \tstrbuf_release(&referent);\n@@ -2569,6 +2577,23 @@ int refs_verify_refname_available(struct ref_store *refs,\n \treturn ret;\n }\n \n+int refs_verify_refname_available(struct ref_store *refs,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const struct string_list *extras,\n+\t\t\t\t const struct string_list *skip,\n+\t\t\t\t unsigned int initial_transaction,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct string_list_item item = { .string = (char *) refname };\n+\tstruct string_list refnames = {\n+\t\t.items = &item,\n+\t\t.nr = 1,\n+\t};\n+\n+\treturn refs_verify_refnames_available(refs, &refnames, extras, skip,\n+\t\t\t\t\t initial_transaction, err);\n+}\n+\n struct do_for_each_reflog_help {\n \teach_reflog_fn *fn;\n \tvoid *cb_data;\ndiff --git a/refs.h b/refs.h\nindex a0cdd99250e..185aed5a461 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -124,6 +124,18 @@ int refs_verify_refname_available(struct ref_store *refs,\n \t\t\t\t unsigned int initial_transaction,\n \t\t\t\t struct strbuf *err);\n \n+/*\n+ * Same as `refs_verify_refname_available()`, but checking for a list of\n+ * refnames instead of only a single item. This is more efficient in the case\n+ * where one needs to check multiple refnames.\n+ */\n+int refs_verify_refnames_available(struct ref_store *refs,\n+\t\t\t\t const struct string_list *refnames,\n+\t\t\t\t const struct string_list *extras,\n+\t\t\t\t const struct string_list *skip,\n+\t\t\t\t unsigned int initial_transaction,\n+\t\t\t\t struct strbuf *err);\n+\n int refs_ref_exists(struct ref_store *refs, const char *refname);\n \n int should_autocreate_reflog(enum log_refs_config log_all_ref_updates,\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512977","messageId":"20250225-pks-update-ref-optimization-v3-5-77c3687cda75@pks.im","threadId":"62965","inReplyTo":"20250225-pks-update-ref-optimization-v3-0-77c3687cda75@pks.im","subject":"[PATCH v3 05/16] refs/reftable: batch refname availability checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T08:55:51Z","receivedAt":"2025-02-25T08:56:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor the \"reftable\" backend to batch the availability check for\nrefnames. This does not yet have an effect on performance as we\nessentially still call `refs_verify_refname_available()` in a loop, but\nthis will change in subsequent commits.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 16 ++++++++++------\n 1 file changed, 10 insertions(+), 6 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex d39a14c5a46..2a90e7cb391 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1069,6 +1069,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE|REF_STORE_MAIN, \"ref_transaction_prepare\");\n \tstruct strbuf referent = STRBUF_INIT, head_referent = STRBUF_INIT;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n \tstruct reftable_transaction_data *tx_data = NULL;\n \tstruct reftable_backend *be;\n \tstruct object_id head_oid;\n@@ -1224,12 +1225,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t\t * can output a proper error message instead of failing\n \t\t\t * at a later point.\n \t\t\t */\n-\t\t\tret = refs_verify_refname_available(ref_store, u->refname,\n-\t\t\t\t\t\t\t &affected_refnames, NULL,\n-\t\t\t\t\t\t\t transaction->flags & REF_TRANSACTION_FLAG_INITIAL,\n-\t\t\t\t\t\t\t err);\n-\t\t\tif (ret < 0)\n-\t\t\t\tgoto done;\n+\t\t\tstring_list_append(&refnames_to_check, u->refname);\n \n \t\t\t/*\n \t\t\t * There is no need to write the reference deletion\n@@ -1379,6 +1375,13 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t}\n \t}\n \n+\tstring_list_sort(&refnames_to_check);\n+\tret = refs_verify_refnames_available(ref_store, &refnames_to_check, &affected_refnames, NULL,\n+\t\t\t\t\t transaction->flags & REF_TRANSACTION_FLAG_INITIAL,\n+\t\t\t\t\t err);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \ttransaction->backend_data = tx_data;\n \ttransaction->state = REF_TRANSACTION_PREPARED;\n \n@@ -1394,6 +1397,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \tstring_list_clear(&affected_refnames, 0);\n \tstrbuf_release(&referent);\n \tstrbuf_release(&head_referent);\n+\tstring_list_clear(&refnames_to_check, 0);\n \n \treturn ret;\n }\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512978","messageId":"20250225-pks-update-ref-optimization-v3-6-77c3687cda75@pks.im","threadId":"62965","inReplyTo":"20250225-pks-update-ref-optimization-v3-0-77c3687cda75@pks.im","subject":"[PATCH v3 06/16] refs/files: batch refname availability checks for normal transactions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T08:55:52Z","receivedAt":"2025-02-25T08:56:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Same as the \"reftable\" backend that we have adapted in the preceding\ncommit to use batched refname availability checks we can also do so for\nthe \"files\" backend. Things are a bit more intricate here though, as we\ncall `refs_verify_refname_available()` in a set of different contexts:\n\n 1. `lock_raw_ref()` when it hits either EEXISTS or EISDIR when creating\n a new reference, mostly to create a nice, user-readable error\n message. This is nothing we have to care about too much, as we only\n hit this code path at most once when we hit a conflict.\n\n 2. `lock_raw_ref()` when it _could_ create the lockfile to check\n whether it is conflicting with any packed refs. In the general case,\n this code path will be hit once for every (successful) reference\n update.\n\n 3. `lock_ref_oid_basic()`, but it is only executed when copying or\n renaming references or when expiring reflogs. It will thus not be\n called in contexts where we have many references queued up.\n\n 4. `refs_refname_ref_available()`, but again only when copying or\n renaming references. It is thus not interesting due to the same\n reason as the previous case.\n\n 5. `files_transaction_finish_initial()`, which is only executed when\n creating a new repository or migrating references.\n\nSo out of these, only (2) and (5) are viable candidates to use the\nbatched checks.\n\nAdapt `lock_raw_ref()` accordingly by queueing up reference names that\nneed to be checked for availability and then checking them after we have\nprocessed all updates. This check is done before we (optionally) lock\nthe `packed-refs` file, which is somewhat flawed because it means that\nthe `packed-refs` could still change after the availability check and\nthus create an undetected conflict. But unconditionally locking the file\nwould change semantics that users are likely to rely on, so we keep the\ncurrent locking sequence intact, even if it's suboptmial.\n\nThe refactoring of `files_transaction_finish_initial()` will be done in\nthe next commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/files-backend.c | 42 +++++++++++++++++++++++++++++++-----------\n 1 file changed, 31 insertions(+), 11 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 29f08dced40..6ce79cf0791 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -678,6 +678,7 @@ static void unlock_ref(struct ref_lock *lock)\n */\n static int lock_raw_ref(struct files_ref_store *refs,\n \t\t\tconst char *refname, int mustexist,\n+\t\t\tstruct string_list *refnames_to_check,\n \t\t\tconst struct string_list *extras,\n \t\t\tstruct ref_lock **lock_p,\n \t\t\tstruct strbuf *referent,\n@@ -855,16 +856,11 @@ static int lock_raw_ref(struct files_ref_store *refs,\n \t\t}\n \n \t\t/*\n-\t\t * If the ref did not exist and we are creating it,\n-\t\t * make sure there is no existing packed ref that\n-\t\t * conflicts with refname:\n+\t\t * If the ref did not exist and we are creating it, we have to\n+\t\t * make sure there is no existing packed ref that conflicts\n+\t\t * with refname. This check is deferred so that we can batch it.\n \t\t */\n-\t\tif (refs_verify_refname_available(\n-\t\t\t\t refs->packed_ref_store, refname,\n-\t\t\t\t extras, NULL, 0, err)) {\n-\t\t\tret = TRANSACTION_NAME_CONFLICT;\n-\t\t\tgoto error_return;\n-\t\t}\n+\t\tstring_list_insert(refnames_to_check, refname);\n \t}\n \n \tret = 0;\n@@ -2569,6 +2565,7 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n \t\t\t struct ref_update *update,\n \t\t\t struct ref_transaction *transaction,\n \t\t\t const char *head_ref,\n+\t\t\t struct string_list *refnames_to_check,\n \t\t\t struct string_list *affected_refnames,\n \t\t\t struct strbuf *err)\n {\n@@ -2597,7 +2594,7 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n \t\tlock->count++;\n \t} else {\n \t\tret = lock_raw_ref(refs, update->refname, mustexist,\n-\t\t\t\t affected_refnames,\n+\t\t\t\t refnames_to_check, affected_refnames,\n \t\t\t\t &lock, &referent,\n \t\t\t\t &update->type, err);\n \t\tif (ret) {\n@@ -2811,6 +2808,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \tsize_t i;\n \tint ret = 0;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n \tchar *head_ref = NULL;\n \tint head_type;\n \tstruct files_transaction_backend_data *backend_data;\n@@ -2898,7 +2896,8 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\tstruct ref_update *update = transaction->updates[i];\n \n \t\tret = lock_ref_for_update(refs, update, transaction,\n-\t\t\t\t\t head_ref, &affected_refnames, err);\n+\t\t\t\t\t head_ref, &refnames_to_check,\n+\t\t\t\t\t &affected_refnames, err);\n \t\tif (ret)\n \t\t\tgoto cleanup;\n \n@@ -2930,6 +2929,26 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\t}\n \t}\n \n+\t/*\n+\t * Verify that none of the loose reference that we're about to write\n+\t * conflict with any existing packed references. Ideally, we'd do this\n+\t * check after the packed-refs are locked so that the file cannot\n+\t * change underneath our feet. But introducing such a lock now would\n+\t * probably do more harm than good as users rely on there not being a\n+\t * global lock with the \"files\" backend.\n+\t *\n+\t * Another alternative would be to do the check after the (optional)\n+\t * lock, but that would extend the time we spend in the globally-locked\n+\t * state.\n+\t *\n+\t * So instead, we accept the race for now.\n+\t */\n+\tif (refs_verify_refnames_available(refs->packed_ref_store, &refnames_to_check,\n+\t\t\t\t\t &affected_refnames, NULL, 0, err)) {\n+\t\tret = TRANSACTION_NAME_CONFLICT;\n+\t\tgoto cleanup;\n+\t}\n+\n \tif (packed_transaction) {\n \t\tif (packed_refs_lock(refs->packed_ref_store, 0, err)) {\n \t\t\tret = TRANSACTION_GENERIC_ERROR;\n@@ -2972,6 +2991,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n cleanup:\n \tfree(head_ref);\n \tstring_list_clear(&affected_refnames, 0);\n+\tstring_list_clear(&refnames_to_check, 0);\n \n \tif (ret)\n \t\tfiles_transaction_cleanup(refs, transaction);\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512979","messageId":"20250225-pks-update-ref-optimization-v3-7-77c3687cda75@pks.im","threadId":"62965","inReplyTo":"20250225-pks-update-ref-optimization-v3-0-77c3687cda75@pks.im","subject":"[PATCH v3 07/16] refs/files: batch refname availability checks for initial transactions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T08:55:53Z","receivedAt":"2025-02-25T08:56:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"files\" backend explicitly carves out special logic for its initial\ntransaction so that it can avoid writing out every single reference as\na loose reference. While the assumption is that there shouldn't be any\npreexisting references, we still have to verify that none of the newly\nwritten references will conflict with any other new reference in the\nsame transaction.\n\nRefactor the initial transaction to use batched refname availability\nchecks. This does not yet have an effect on performance as we still call\n`refs_verify_refname_available()` in a loop. But this will change in\nsubsequent commits and then impact performance when cloning a repository\nwith many references or when migrating references to the \"files\" format.\n\nThis doesn't yet have an effect on performance as the underlying\nlogic simply calls This will improve performance when cloning a repository with\nmany references or when migrating references from any format to the\n\"files\" format.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/files-backend.c | 23 ++++++++++++++++-------\n 1 file changed, 16 insertions(+), 7 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 6ce79cf0791..11a620ea11a 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3056,6 +3056,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \tsize_t i;\n \tint ret = 0;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n \tstruct ref_transaction *packed_transaction = NULL;\n \tstruct ref_transaction *loose_transaction = NULL;\n \n@@ -3105,11 +3106,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\t !is_null_oid(&update->old_oid))\n \t\t\tBUG(\"initial ref transaction with old_sha1 set\");\n \n-\t\tif (refs_verify_refname_available(&refs->base, update->refname,\n-\t\t\t\t\t\t &affected_refnames, NULL, 1, err)) {\n-\t\t\tret = TRANSACTION_NAME_CONFLICT;\n-\t\t\tgoto cleanup;\n-\t\t}\n+\t\tstring_list_append(&refnames_to_check, update->refname);\n \n \t\t/*\n \t\t * packed-refs don't support symbolic refs, root refs and reflogs,\n@@ -3145,8 +3142,19 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\t}\n \t}\n \n-\tif (packed_refs_lock(refs->packed_ref_store, 0, err) ||\n-\t ref_transaction_commit(packed_transaction, err)) {\n+\tif (packed_refs_lock(refs->packed_ref_store, 0, err)) {\n+\t\tret = TRANSACTION_GENERIC_ERROR;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif (refs_verify_refnames_available(&refs->base, &refnames_to_check,\n+\t\t\t\t\t &affected_refnames, NULL, 1, err)) {\n+\t\tpacked_refs_unlock(refs->packed_ref_store);\n+\t\tret = TRANSACTION_NAME_CONFLICT;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif (ref_transaction_commit(packed_transaction, err)) {\n \t\tret = TRANSACTION_GENERIC_ERROR;\n \t\tgoto cleanup;\n \t}\n@@ -3167,6 +3175,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\tref_transaction_free(packed_transaction);\n \ttransaction->state = REF_TRANSACTION_CLOSED;\n \tstring_list_clear(&affected_refnames, 0);\n+\tstring_list_clear(&refnames_to_check, 0);\n \treturn ret;\n }\n \n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512980","messageId":"20250225-pks-update-ref-optimization-v3-8-77c3687cda75@pks.im","threadId":"62965","inReplyTo":"20250225-pks-update-ref-optimization-v3-0-77c3687cda75@pks.im","subject":"[PATCH v3 08/16] refs: stop re-verifying common prefixes for availability","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T08:55:54Z","receivedAt":"2025-02-25T08:56:08Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"One of the checks done by `refs_verify_refnames_available()` is whether\nany of the prefixes of a reference already exists. For example, given a\nreference \"refs/heads/main\", we'd check whether \"refs/heads\" or \"refs\"\nalready exist, and if so we'd abort the transaction.\n\nWhen updating multiple references at once, this check is performed for\neach of the references individually. Consequently, because references\ntend to have common prefixes like \"refs/heads/\" or refs/tags/\", we\nevaluate the availability of these prefixes repeatedly. Naturally this\nis a waste of compute, as the availability of those prefixes should in\ngeneral not change in the middle of a transaction. And if it would,\nbackends would notice at a later point in time.\n\nOptimize this pattern by storing prefixes in a `strset` so that we can\ntrivially track those prefixes that we have already checked. This leads\nto a significant speedup with the \"reftable\" backend when creating many\nreferences that all share a common prefix:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 63.1 ms ± 1.8 ms [User: 41.0 ms, System: 21.6 ms]\n Range (min … max): 60.6 ms … 69.5 ms 38 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 40.0 ms ± 1.3 ms [User: 29.3 ms, System: 10.3 ms]\n Range (min … max): 38.1 ms … 47.3 ms 61 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.58 ± 0.07 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n\nFor the \"files\" backend we see an improvement, but a much smaller one:\n\n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 395.8 ms ± 5.3 ms [User: 63.6 ms, System: 330.5 ms]\n Range (min … max): 387.0 ms … 404.6 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 386.0 ms ± 4.0 ms [User: 51.5 ms, System: 332.8 ms]\n Range (min … max): 380.8 ms … 392.6 ms 10 runs\n\n Summary\n update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.03 ± 0.02 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n\nThis change also leads to a modest improvement when writing references\nwith \"initial\" semantics, for example when migrating references. The\nfollowing benchmarks are migrating 1m references from the \"reftable\" to\nthe \"files\" backend:\n\n Benchmark 1: migrate reftable:files (refcount = 1000000, revision = HEAD~)\n Time (mean ± σ): 836.6 ms ± 5.6 ms [User: 645.2 ms, System: 185.2 ms]\n Range (min … max): 829.6 ms … 845.9 ms 10 runs\n\n Benchmark 2: migrate reftable:files (refcount = 1000000, revision = HEAD)\n Time (mean ± σ): 759.8 ms ± 5.1 ms [User: 574.9 ms, System: 178.9 ms]\n Range (min … max): 753.1 ms … 768.8 ms 10 runs\n\n Summary\n migrate reftable:files (refcount = 1000000, revision = HEAD) ran\n 1.10 ± 0.01 times faster than migrate reftable:files (refcount = 1000000, revision = HEAD~)\n\nAnd vice versa:\n\n Benchmark 1: migrate files:reftable (refcount = 1000000, revision = HEAD~)\n Time (mean ± σ): 870.7 ms ± 5.7 ms [User: 735.2 ms, System: 127.4 ms]\n Range (min … max): 861.6 ms … 883.2 ms 10 runs\n\n Benchmark 2: migrate files:reftable (refcount = 1000000, revision = HEAD)\n Time (mean ± σ): 799.1 ms ± 8.5 ms [User: 661.1 ms, System: 130.2 ms]\n Range (min … max): 787.5 ms … 812.6 ms 10 runs\n\n Summary\n migrate files:reftable (refcount = 1000000, revision = HEAD) ran\n 1.09 ± 0.01 times faster than migrate files:reftable (refcount = 1000000, revision = HEAD~)\n\nThe impact here is significantly smaller given that we don't perform any\nreference reads with \"initial\" semantics, so the speedup only comes from\nus doing less string list lookups.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs.c | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/refs.c b/refs.c\nindex 5a9b0f2fa1e..eaf41421f50 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2476,6 +2476,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n {\n \tstruct strbuf dirname = STRBUF_INIT;\n \tstruct strbuf referent = STRBUF_INIT;\n+\tstruct strset dirnames;\n \tint ret = -1;\n \n \t/*\n@@ -2485,6 +2486,8 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \n \tassert(err);\n \n+\tstrset_init(&dirnames);\n+\n \tfor (size_t i = 0; i < refnames->nr; i++) {\n \t\tconst char *refname = refnames->items[i].string;\n \t\tconst char *extra_refname;\n@@ -2514,6 +2517,14 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \t\t\tif (skip && string_list_has_string(skip, dirname.buf))\n \t\t\t\tcontinue;\n \n+\t\t\t/*\n+\t\t\t * If we've already seen the directory we don't need to\n+\t\t\t * process it again. Skip it to avoid checking checking\n+\t\t\t * common prefixes like \"refs/heads/\" repeatedly.\n+\t\t\t */\n+\t\t\tif (!strset_add(&dirnames, dirname.buf))\n+\t\t\t\tcontinue;\n+\n \t\t\tif (!initial_transaction &&\n \t\t\t !refs_read_raw_ref(refs, dirname.buf, &oid, &referent,\n \t\t\t\t\t &type, &ignore_errno)) {\n@@ -2574,6 +2585,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n cleanup:\n \tstrbuf_release(&referent);\n \tstrbuf_release(&dirname);\n+\tstrset_clear(&dirnames);\n \treturn ret;\n }\n \n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512981","messageId":"20250225-pks-update-ref-optimization-v3-10-77c3687cda75@pks.im","threadId":"62965","inReplyTo":"20250225-pks-update-ref-optimization-v3-0-77c3687cda75@pks.im","subject":"[PATCH v3 10/16] refs/iterator: provide infrastructure to re-seek iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T08:55:56Z","receivedAt":"2025-02-25T08:56:09Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Reftable iterators need to be scrapped after they have either been\nexhausted or aren't useful to the caller anymore, and it is explicitly\nnot possible to reuse them for iterations. But enabling for reuse of\niterators may allow us to tune them by reusing internal state of an\niterator. The reftable iterators for example can already be reused\ninternally, but we're not able to expose this to any users outside of\nthe reftable backend.\n\nIntroduce a new `.seek` function in the ref iterator vtable that allows\ncallers to seek an iterator multiple times. It is expected to be\nfunctionally the same as calling `refs_ref_iterator_begin()` with a\ndifferent (or the same) prefix.\n\nNote that it is not possible to adjust parameters other than the seeked\nprefix for now, so exclude patterns, trimmed prefixes and flags will\nremain unchanged. We do not have a usecase for changing these parameters\nright now, but if we ever find one we can adapt accordingly.\n\nImplement the callback for trivial cases. The other iterators will be\nimplemented in subsequent commits.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/debug.c | 11 +++++++++++\n refs/iterator.c | 24 ++++++++++++++++++++++++\n refs/refs-internal.h | 24 ++++++++++++++++++++++++\n 3 files changed, 59 insertions(+)\n\ndiff --git a/refs/debug.c b/refs/debug.c\nindex a9786da4ba1..5390fa9c187 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -169,6 +169,16 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn res;\n }\n \n+static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct debug_ref_iterator *diter =\n+\t\t(struct debug_ref_iterator *)ref_iterator;\n+\tint res = diter->iter->vtable->seek(diter->iter, prefix);\n+\ttrace_printf_key(&trace_refs, \"iterator_seek: %s: %d\\n\", prefix ? prefix : \"\", res);\n+\treturn res;\n+}\n+\n static int debug_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -189,6 +199,7 @@ static void debug_ref_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable debug_ref_iterator_vtable = {\n \t.advance = debug_ref_iterator_advance,\n+\t.seek = debug_ref_iterator_seek,\n \t.peel = debug_ref_iterator_peel,\n \t.release = debug_ref_iterator_release,\n };\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex aaeff270437..757b105261a 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -15,6 +15,12 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ref_iterator->vtable->advance(ref_iterator);\n }\n \n+int ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t const char *prefix)\n+{\n+\treturn ref_iterator->vtable->seek(ref_iterator, prefix);\n+}\n+\n int ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t struct object_id *peeled)\n {\n@@ -50,6 +56,12 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)\n \treturn ITER_DONE;\n }\n \n+static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n+\t\t\t\t const char *prefix UNUSED)\n+{\n+\treturn 0;\n+}\n+\n static int empty_ref_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \t\t\t\t struct object_id *peeled UNUSED)\n {\n@@ -62,6 +74,7 @@ static void empty_ref_iterator_release(struct ref_iterator *ref_iterator UNUSED)\n \n static struct ref_iterator_vtable empty_ref_iterator_vtable = {\n \t.advance = empty_ref_iterator_advance,\n+\t.seek = empty_ref_iterator_seek,\n \t.peel = empty_ref_iterator_peel,\n \t.release = empty_ref_iterator_release,\n };\n@@ -368,6 +381,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n+static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct prefix_ref_iterator *iter =\n+\t\t(struct prefix_ref_iterator *)ref_iterator;\n+\tfree(iter->prefix);\n+\titer->prefix = xstrdup_or_null(prefix);\n+\treturn ref_iterator_seek(iter->iter0, prefix);\n+}\n+\n static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -387,6 +410,7 @@ static void prefix_ref_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable prefix_ref_iterator_vtable = {\n \t.advance = prefix_ref_iterator_advance,\n+\t.seek = prefix_ref_iterator_seek,\n \t.peel = prefix_ref_iterator_peel,\n \t.release = prefix_ref_iterator_release,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 74e2c03cef1..8f18274a165 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -327,6 +327,22 @@ struct ref_iterator {\n */\n int ref_iterator_advance(struct ref_iterator *ref_iterator);\n \n+/*\n+ * Seek the iterator to the first reference with the given prefix.\n+ * The prefix is matched as a literal string, without regard for path\n+ * separators. If prefix is NULL or the empty string, seek the iterator to the\n+ * first reference again.\n+ *\n+ * This function is expected to behave as if a new ref iterator with the same\n+ * prefix had been created, but allows reuse of iterators and thus may allow\n+ * the backend to optimize. Parameters other than the prefix that have been\n+ * passed when creating the iterator will remain unchanged.\n+ *\n+ * Returns 0 on success, a negative error code otherwise.\n+ */\n+int ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t const char *prefix);\n+\n /*\n * If possible, peel the reference currently being viewed by the\n * iterator. Return 0 on success.\n@@ -445,6 +461,13 @@ void base_ref_iterator_init(struct ref_iterator *iter,\n */\n typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);\n \n+/*\n+ * Seek the iterator to the first reference matching the given prefix. Should\n+ * behave the same as if a new iterator was created with the same prefix.\n+ */\n+typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix);\n+\n /*\n * Peels the current ref, returning 0 for success or -1 for failure.\n */\n@@ -459,6 +482,7 @@ typedef void ref_iterator_release_fn(struct ref_iterator *ref_iterator);\n \n struct ref_iterator_vtable {\n \tref_iterator_advance_fn *advance;\n+\tref_iterator_seek_fn *seek;\n \tref_iterator_peel_fn *peel;\n \tref_iterator_release_fn *release;\n };\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512982","messageId":"20250225-pks-update-ref-optimization-v3-9-77c3687cda75@pks.im","threadId":"62965","inReplyTo":"20250225-pks-update-ref-optimization-v3-0-77c3687cda75@pks.im","subject":"[PATCH v3 09/16] refs/iterator: separate lifecycle from iteration","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T08:55:55Z","receivedAt":"2025-02-25T08:56:09Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The ref and reflog iterators have their lifecycle attached to iteration:\nonce the iterator reaches its end, it is automatically released and the\ncaller doesn't have to care about that anymore. When the iterator should\nbe released before it has been exhausted, callers must explicitly abort\nthe iterator via `ref_iterator_abort()`.\n\nThis lifecycle is somewhat unusual in the Git codebase and creates two\nproblems:\n\n - Callsites need to be very careful about when exactly they call\n `ref_iterator_abort()`, as calling the function is only valid when\n the iterator itself still is. This leads to somewhat awkward calling\n patterns in some situations.\n\n - It is impossible to reuse iterators and re-seek them to a different\n prefix. This feature isn't supported by any iterator implementation\n except for the reftable iterators anyway, but if it was implemented\n it would allow us to optimize cases where we need to search for\n specific references repeatedly by reusing internal state.\n\nDetangle the lifecycle from iteration so that we don't deallocate the\niterator anymore once it is exhausted. Instead, callers are now expected\nto always call a newly introduce `ref_iterator_free()` function that\ndeallocates the iterator and its internal state.\n\nNote that the `dir_iterator` is somewhat special because it does not\nimplement the `ref_iterator` interface, but is only used to implement\nother iterators. Consequently, we have to provide `dir_iterator_free()`\ninstead of `dir_iterator_release()` as the allocated structure itself is\nmanaged by the `dir_iterator` interfaces, as well, and not freed by\n`ref_iterator_free()` like in all the other cases.\n\nWhile at it, drop the return value of `ref_iterator_abort()`, which\nwasn't really required by any of the iterator implementations anyway.\nFurthermore, stop calling `base_ref_iterator_free()` in any of the\nbackends, but instead call it in `ref_iterator_free()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/clone.c | 2 +\n dir-iterator.c | 24 +++++------\n dir-iterator.h | 11 ++---\n iterator.h | 2 +-\n refs.c | 7 +++-\n refs/debug.c | 9 ++---\n refs/files-backend.c | 36 +++++------------\n refs/iterator.c | 95 ++++++++++++++------------------------------\n refs/packed-backend.c | 27 ++++++-------\n refs/ref-cache.c | 9 ++---\n refs/refs-internal.h | 29 +++++---------\n refs/reftable-backend.c | 34 ++++------------\n t/helper/test-dir-iterator.c | 1 +\n 13 files changed, 100 insertions(+), 186 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex fd001d800c6..ac3e84b2b18 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -426,6 +426,8 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\tstrbuf_setlen(src, src_len);\n \t\tdie(_(\"failed to iterate over '%s'\"), src->buf);\n \t}\n+\n+\tdir_iterator_free(iter);\n }\n \n static void clone_local(const char *src_repo, const char *dest_repo)\ndiff --git a/dir-iterator.c b/dir-iterator.c\nindex de619846f29..857e1d9bdaf 100644\n--- a/dir-iterator.c\n+++ b/dir-iterator.c\n@@ -193,9 +193,9 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \n \tif (S_ISDIR(iter->base.st.st_mode) && push_level(iter)) {\n \t\tif (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC)\n-\t\t\tgoto error_out;\n+\t\t\treturn ITER_ERROR;\n \t\tif (iter->levels_nr == 0)\n-\t\t\tgoto error_out;\n+\t\t\treturn ITER_ERROR;\n \t}\n \n \t/* Loop until we find an entry that we can give back to the caller. */\n@@ -211,11 +211,11 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\tint ret = next_directory_entry(level->dir, iter->base.path.buf, &de);\n \t\t\tif (ret < 0) {\n \t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n-\t\t\t\t\tgoto error_out;\n+\t\t\t\t\treturn ITER_ERROR;\n \t\t\t\tcontinue;\n \t\t\t} else if (ret > 0) {\n \t\t\t\tif (pop_level(iter) == 0)\n-\t\t\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\t\t\t\treturn ITER_DONE;\n \t\t\t\tcontinue;\n \t\t\t}\n \n@@ -223,7 +223,7 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t} else {\n \t\t\tif (level->entries_idx >= level->entries.nr) {\n \t\t\t\tif (pop_level(iter) == 0)\n-\t\t\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\t\t\t\treturn ITER_DONE;\n \t\t\t\tcontinue;\n \t\t\t}\n \n@@ -232,22 +232,21 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \n \t\tif (prepare_next_entry_data(iter, name)) {\n \t\t\tif (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC)\n-\t\t\t\tgoto error_out;\n+\t\t\t\treturn ITER_ERROR;\n \t\t\tcontinue;\n \t\t}\n \n \t\treturn ITER_OK;\n \t}\n-\n-error_out:\n-\tdir_iterator_abort(dir_iterator);\n-\treturn ITER_ERROR;\n }\n \n-int dir_iterator_abort(struct dir_iterator *dir_iterator)\n+void dir_iterator_free(struct dir_iterator *dir_iterator)\n {\n \tstruct dir_iterator_int *iter = (struct dir_iterator_int *)dir_iterator;\n \n+\tif (!iter)\n+\t\treturn;\n+\n \tfor (; iter->levels_nr; iter->levels_nr--) {\n \t\tstruct dir_iterator_level *level =\n \t\t\t&iter->levels[iter->levels_nr - 1];\n@@ -266,7 +265,6 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator)\n \tfree(iter->levels);\n \tstrbuf_release(&iter->base.path);\n \tfree(iter);\n-\treturn ITER_DONE;\n }\n \n struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags)\n@@ -301,7 +299,7 @@ struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags)\n \treturn dir_iterator;\n \n error_out:\n-\tdir_iterator_abort(dir_iterator);\n+\tdir_iterator_free(dir_iterator);\n \terrno = saved_errno;\n \treturn NULL;\n }\ndiff --git a/dir-iterator.h b/dir-iterator.h\nindex 6d438809b6e..ccd6a197343 100644\n--- a/dir-iterator.h\n+++ b/dir-iterator.h\n@@ -28,7 +28,7 @@\n *\n * while ((ok = dir_iterator_advance(iter)) == ITER_OK) {\n * if (want_to_stop_iteration()) {\n- * ok = dir_iterator_abort(iter);\n+ * ok = ITER_DONE;\n * break;\n * }\n *\n@@ -39,6 +39,7 @@\n *\n * if (ok != ITER_DONE)\n * handle_error();\n+ * dir_iterator_free(iter);\n *\n * Callers are allowed to modify iter->path while they are working,\n * but they must restore it to its original contents before calling\n@@ -107,11 +108,7 @@ struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags);\n */\n int dir_iterator_advance(struct dir_iterator *iterator);\n \n-/*\n- * End the iteration before it has been exhausted. Free the\n- * dir_iterator and any associated resources and return ITER_DONE. On\n- * error, free the dir_iterator and return ITER_ERROR.\n- */\n-int dir_iterator_abort(struct dir_iterator *iterator);\n+/* Free the dir_iterator and any associated resources. */\n+void dir_iterator_free(struct dir_iterator *iterator);\n \n #endif\ndiff --git a/iterator.h b/iterator.h\nindex 0f6900e43ad..6b77dcc2626 100644\n--- a/iterator.h\n+++ b/iterator.h\n@@ -12,7 +12,7 @@\n #define ITER_OK 0\n \n /*\n- * The iterator is exhausted and has been freed.\n+ * The iterator is exhausted.\n */\n #define ITER_DONE -1\n \ndiff --git a/refs.c b/refs.c\nindex eaf41421f50..8eff60a2186 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2476,6 +2476,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n {\n \tstruct strbuf dirname = STRBUF_INIT;\n \tstruct strbuf referent = STRBUF_INIT;\n+\tstruct ref_iterator *iter = NULL;\n \tstruct strset dirnames;\n \tint ret = -1;\n \n@@ -2552,7 +2553,6 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \t\tstrbuf_addch(&dirname, '/');\n \n \t\tif (!initial_transaction) {\n-\t\t\tstruct ref_iterator *iter;\n \t\t\tint ok;\n \n \t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n@@ -2564,12 +2564,14 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \n \t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n \t\t\t\t\t iter->refname, refname);\n-\t\t\t\tref_iterator_abort(iter);\n \t\t\t\tgoto cleanup;\n \t\t\t}\n \n \t\t\tif (ok != ITER_DONE)\n \t\t\t\tBUG(\"error while iterating over references\");\n+\n+\t\t\tref_iterator_free(iter);\n+\t\t\titer = NULL;\n \t\t}\n \n \t\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n@@ -2586,6 +2588,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \tstrbuf_release(&referent);\n \tstrbuf_release(&dirname);\n \tstrset_clear(&dirnames);\n+\tref_iterator_free(iter);\n \treturn ret;\n }\n \ndiff --git a/refs/debug.c b/refs/debug.c\nindex fbc4df08b43..a9786da4ba1 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -179,19 +179,18 @@ static int debug_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn res;\n }\n \n-static int debug_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void debug_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct debug_ref_iterator *diter =\n \t\t(struct debug_ref_iterator *)ref_iterator;\n-\tint res = diter->iter->vtable->abort(diter->iter);\n-\ttrace_printf_key(&trace_refs, \"iterator_abort: %d\\n\", res);\n-\treturn res;\n+\tditer->iter->vtable->release(diter->iter);\n+\ttrace_printf_key(&trace_refs, \"iterator_abort\\n\");\n }\n \n static struct ref_iterator_vtable debug_ref_iterator_vtable = {\n \t.advance = debug_ref_iterator_advance,\n \t.peel = debug_ref_iterator_peel,\n-\t.abort = debug_ref_iterator_abort,\n+\t.release = debug_ref_iterator_release,\n };\n \n static struct ref_iterator *\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 11a620ea11a..859f1c11941 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -915,10 +915,6 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\titer->iter0 = NULL;\n-\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\tok = ITER_ERROR;\n-\n \treturn ok;\n }\n \n@@ -931,23 +927,17 @@ static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator_peel(iter->iter0, peeled);\n }\n \n-static int files_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void files_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct files_ref_iterator *iter =\n \t\t(struct files_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->iter0)\n-\t\tok = ref_iterator_abort(iter->iter0);\n-\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n+\tref_iterator_free(iter->iter0);\n }\n \n static struct ref_iterator_vtable files_ref_iterator_vtable = {\n \t.advance = files_ref_iterator_advance,\n \t.peel = files_ref_iterator_peel,\n-\t.abort = files_ref_iterator_abort,\n+\t.release = files_ref_iterator_release,\n };\n \n static struct ref_iterator *files_ref_iterator_begin(\n@@ -1378,7 +1368,7 @@ static int should_pack_refs(struct files_ref_store *refs,\n \t\t\t\t iter->flags, opts))\n \t\t\trefcount++;\n \t\tif (refcount >= limit) {\n-\t\t\tref_iterator_abort(iter);\n+\t\t\tref_iterator_free(iter);\n \t\t\treturn 1;\n \t\t}\n \t}\n@@ -1386,6 +1376,7 @@ static int should_pack_refs(struct files_ref_store *refs,\n \tif (ret != ITER_DONE)\n \t\tdie(\"error while iterating over references\");\n \n+\tref_iterator_free(iter);\n \treturn 0;\n }\n \n@@ -1452,6 +1443,7 @@ static int files_pack_refs(struct ref_store *ref_store,\n \tpacked_refs_unlock(refs->packed_ref_store);\n \n \tprune_refs(refs, &refs_to_prune);\n+\tref_iterator_free(iter);\n \tstrbuf_release(&err);\n \treturn 0;\n }\n@@ -2299,9 +2291,6 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\titer->dir_iterator = NULL;\n-\tif (ref_iterator_abort(ref_iterator) == ITER_ERROR)\n-\t\tok = ITER_ERROR;\n \treturn ok;\n }\n \n@@ -2311,23 +2300,17 @@ static int files_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \tBUG(\"ref_iterator_peel() called for reflog_iterator\");\n }\n \n-static int files_reflog_iterator_abort(struct ref_iterator *ref_iterator)\n+static void files_reflog_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct files_reflog_iterator *iter =\n \t\t(struct files_reflog_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->dir_iterator)\n-\t\tok = dir_iterator_abort(iter->dir_iterator);\n-\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n+\tdir_iterator_free(iter->dir_iterator);\n }\n \n static struct ref_iterator_vtable files_reflog_iterator_vtable = {\n \t.advance = files_reflog_iterator_advance,\n \t.peel = files_reflog_iterator_peel,\n-\t.abort = files_reflog_iterator_abort,\n+\t.release = files_reflog_iterator_release,\n };\n \n static struct ref_iterator *reflog_iterator_begin(struct ref_store *ref_store,\n@@ -3837,6 +3820,7 @@ static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n \n out:\n+\tdir_iterator_free(iter);\n \tstrbuf_release(&sb);\n \tstrbuf_release(&refname);\n \treturn ret;\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex d25e568bf0b..aaeff270437 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -21,9 +21,14 @@ int ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator->vtable->peel(ref_iterator, peeled);\n }\n \n-int ref_iterator_abort(struct ref_iterator *ref_iterator)\n+void ref_iterator_free(struct ref_iterator *ref_iterator)\n {\n-\treturn ref_iterator->vtable->abort(ref_iterator);\n+\tif (ref_iterator) {\n+\t\tref_iterator->vtable->release(ref_iterator);\n+\t\t/* Help make use-after-free bugs fail quickly: */\n+\t\tref_iterator->vtable = NULL;\n+\t\tfree(ref_iterator);\n+\t}\n }\n \n void base_ref_iterator_init(struct ref_iterator *iter,\n@@ -36,20 +41,13 @@ void base_ref_iterator_init(struct ref_iterator *iter,\n \titer->flags = 0;\n }\n \n-void base_ref_iterator_free(struct ref_iterator *iter)\n-{\n-\t/* Help make use-after-free bugs fail quickly: */\n-\titer->vtable = NULL;\n-\tfree(iter);\n-}\n-\n struct empty_ref_iterator {\n \tstruct ref_iterator base;\n };\n \n-static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)\n {\n-\treturn ref_iterator_abort(ref_iterator);\n+\treturn ITER_DONE;\n }\n \n static int empty_ref_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n@@ -58,16 +56,14 @@ static int empty_ref_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \tBUG(\"peel called for empty iterator\");\n }\n \n-static int empty_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void empty_ref_iterator_release(struct ref_iterator *ref_iterator UNUSED)\n {\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable empty_ref_iterator_vtable = {\n \t.advance = empty_ref_iterator_advance,\n \t.peel = empty_ref_iterator_peel,\n-\t.abort = empty_ref_iterator_abort,\n+\t.release = empty_ref_iterator_release,\n };\n \n struct ref_iterator *empty_ref_iterator_begin(void)\n@@ -151,11 +147,13 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tif (!iter->current) {\n \t\t/* Initialize: advance both iterators to their first entries */\n \t\tif ((ok = ref_iterator_advance(iter->iter0)) != ITER_OK) {\n+\t\t\tref_iterator_free(iter->iter0);\n \t\t\titer->iter0 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n \t\t}\n \t\tif ((ok = ref_iterator_advance(iter->iter1)) != ITER_OK) {\n+\t\t\tref_iterator_free(iter->iter1);\n \t\t\titer->iter1 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -166,6 +164,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t * entry:\n \t\t */\n \t\tif ((ok = ref_iterator_advance(*iter->current)) != ITER_OK) {\n+\t\t\tref_iterator_free(*iter->current);\n \t\t\t*iter->current = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -179,9 +178,8 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t\titer->select(iter->iter0, iter->iter1, iter->cb_data);\n \n \t\tif (selection == ITER_SELECT_DONE) {\n-\t\t\treturn ref_iterator_abort(ref_iterator);\n+\t\t\treturn ITER_DONE;\n \t\t} else if (selection == ITER_SELECT_ERROR) {\n-\t\t\tref_iterator_abort(ref_iterator);\n \t\t\treturn ITER_ERROR;\n \t\t}\n \n@@ -195,6 +193,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \n \t\tif (selection & ITER_SKIP_SECONDARY) {\n \t\t\tif ((ok = ref_iterator_advance(*secondary)) != ITER_OK) {\n+\t\t\t\tref_iterator_free(*secondary);\n \t\t\t\t*secondary = NULL;\n \t\t\t\tif (ok == ITER_ERROR)\n \t\t\t\t\tgoto error;\n@@ -211,7 +210,6 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t}\n \n error:\n-\tref_iterator_abort(ref_iterator);\n \treturn ITER_ERROR;\n }\n \n@@ -227,28 +225,18 @@ static int merge_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator_peel(*iter->current, peeled);\n }\n \n-static int merge_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void merge_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct merge_ref_iterator *iter =\n \t\t(struct merge_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->iter0) {\n-\t\tif (ref_iterator_abort(iter->iter0) != ITER_DONE)\n-\t\t\tok = ITER_ERROR;\n-\t}\n-\tif (iter->iter1) {\n-\t\tif (ref_iterator_abort(iter->iter1) != ITER_DONE)\n-\t\t\tok = ITER_ERROR;\n-\t}\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n+\tref_iterator_free(iter->iter0);\n+\tref_iterator_free(iter->iter1);\n }\n \n static struct ref_iterator_vtable merge_ref_iterator_vtable = {\n \t.advance = merge_ref_iterator_advance,\n \t.peel = merge_ref_iterator_peel,\n-\t.abort = merge_ref_iterator_abort,\n+\t.release = merge_ref_iterator_release,\n };\n \n struct ref_iterator *merge_ref_iterator_begin(\n@@ -310,10 +298,10 @@ struct ref_iterator *overlay_ref_iterator_begin(\n \t * them.\n \t */\n \tif (is_empty_ref_iterator(front)) {\n-\t\tref_iterator_abort(front);\n+\t\tref_iterator_free(front);\n \t\treturn back;\n \t} else if (is_empty_ref_iterator(back)) {\n-\t\tref_iterator_abort(back);\n+\t\tref_iterator_free(back);\n \t\treturn front;\n \t}\n \n@@ -350,19 +338,10 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \n \twhile ((ok = ref_iterator_advance(iter->iter0)) == ITER_OK) {\n \t\tint cmp = compare_prefix(iter->iter0->refname, iter->prefix);\n-\n \t\tif (cmp < 0)\n \t\t\tcontinue;\n-\n-\t\tif (cmp > 0) {\n-\t\t\t/*\n-\t\t\t * As the source iterator is ordered, we\n-\t\t\t * can stop the iteration as soon as we see a\n-\t\t\t * refname that comes after the prefix:\n-\t\t\t */\n-\t\t\tok = ref_iterator_abort(iter->iter0);\n-\t\t\tbreak;\n-\t\t}\n+\t\tif (cmp > 0)\n+\t\t\treturn ITER_DONE;\n \n \t\tif (iter->trim) {\n \t\t\t/*\n@@ -386,9 +365,6 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\titer->iter0 = NULL;\n-\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\treturn ITER_ERROR;\n \treturn ok;\n }\n \n@@ -401,23 +377,18 @@ static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator_peel(iter->iter0, peeled);\n }\n \n-static int prefix_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void prefix_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct prefix_ref_iterator *iter =\n \t\t(struct prefix_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->iter0)\n-\t\tok = ref_iterator_abort(iter->iter0);\n+\tref_iterator_free(iter->iter0);\n \tfree(iter->prefix);\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n }\n \n static struct ref_iterator_vtable prefix_ref_iterator_vtable = {\n \t.advance = prefix_ref_iterator_advance,\n \t.peel = prefix_ref_iterator_peel,\n-\t.abort = prefix_ref_iterator_abort,\n+\t.release = prefix_ref_iterator_release,\n };\n \n struct ref_iterator *prefix_ref_iterator_begin(struct ref_iterator *iter0,\n@@ -453,20 +424,14 @@ int do_for_each_ref_iterator(struct ref_iterator *iter,\n \tcurrent_ref_iter = iter;\n \twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\tretval = fn(iter->refname, iter->referent, iter->oid, iter->flags, cb_data);\n-\t\tif (retval) {\n-\t\t\t/*\n-\t\t\t * If ref_iterator_abort() returns ITER_ERROR,\n-\t\t\t * we ignore that error in deference to the\n-\t\t\t * callback function's return value.\n-\t\t\t */\n-\t\t\tref_iterator_abort(iter);\n+\t\tif (retval)\n \t\t\tgoto out;\n-\t\t}\n \t}\n \n out:\n \tcurrent_ref_iter = old_ref_iter;\n \tif (ok == ITER_ERROR)\n-\t\treturn -1;\n+\t\tretval = -1;\n+\tref_iterator_free(iter);\n \treturn retval;\n }\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex a7b6f74b6e3..38a1956d1a8 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -954,9 +954,6 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\tok = ITER_ERROR;\n-\n \treturn ok;\n }\n \n@@ -976,23 +973,19 @@ static int packed_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t}\n }\n \n-static int packed_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void packed_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct packed_ref_iterator *iter =\n \t\t(struct packed_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n \tstrbuf_release(&iter->refname_buf);\n \tfree(iter->jump);\n \trelease_snapshot(iter->snapshot);\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n }\n \n static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \t.advance = packed_ref_iterator_advance,\n \t.peel = packed_ref_iterator_peel,\n-\t.abort = packed_ref_iterator_abort\n+\t.release = packed_ref_iterator_release,\n };\n \n static int jump_list_entry_cmp(const void *va, const void *vb)\n@@ -1362,8 +1355,10 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t */\n \titer = packed_ref_iterator_begin(&refs->base, \"\", NULL,\n \t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n-\tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n+\tif ((ok = ref_iterator_advance(iter)) != ITER_OK) {\n+\t\tref_iterator_free(iter);\n \t\titer = NULL;\n+\t}\n \n \ti = 0;\n \n@@ -1411,8 +1406,10 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t\t\t\t * the iterator over the unneeded\n \t\t\t\t * value.\n \t\t\t\t */\n-\t\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n+\t\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK) {\n+\t\t\t\t\tref_iterator_free(iter);\n \t\t\t\t\titer = NULL;\n+\t\t\t\t}\n \t\t\t\tcmp = +1;\n \t\t\t} else {\n \t\t\t\t/*\n@@ -1449,8 +1446,10 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t\t\t\t\t peel_error ? NULL : &peeled))\n \t\t\t\tgoto write_error;\n \n-\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n+\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK) {\n+\t\t\t\tref_iterator_free(iter);\n \t\t\t\titer = NULL;\n+\t\t\t}\n \t\t} else if (is_null_oid(&update->new_oid)) {\n \t\t\t/*\n \t\t\t * The update wants to delete the reference,\n@@ -1499,9 +1498,7 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t\t get_tempfile_path(refs->tempfile), strerror(errno));\n \n error:\n-\tif (iter)\n-\t\tref_iterator_abort(iter);\n-\n+\tref_iterator_free(iter);\n \tdelete_tempfile(&refs->tempfile);\n \treturn -1;\n }\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 02f09e4df88..6457e02c1ea 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -409,7 +409,7 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\tif (++level->index == level->dir->nr) {\n \t\t\t/* This level is exhausted; pop up a level */\n \t\t\tif (--iter->levels_nr == 0)\n-\t\t\t\treturn ref_iterator_abort(ref_iterator);\n+\t\t\t\treturn ITER_DONE;\n \n \t\t\tcontinue;\n \t\t}\n@@ -452,21 +452,18 @@ static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn peel_object(iter->repo, ref_iterator->oid, peeled) ? -1 : 0;\n }\n \n-static int cache_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void cache_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct cache_ref_iterator *iter =\n \t\t(struct cache_ref_iterator *)ref_iterator;\n-\n \tfree((char *)iter->prefix);\n \tfree(iter->levels);\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable cache_ref_iterator_vtable = {\n \t.advance = cache_ref_iterator_advance,\n \t.peel = cache_ref_iterator_peel,\n-\t.abort = cache_ref_iterator_abort\n+\t.release = cache_ref_iterator_release,\n };\n \n struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex aaab711bb96..74e2c03cef1 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -273,11 +273,11 @@ enum do_for_each_ref_flags {\n * the next reference and returns ITER_OK. The data pointed at by\n * refname and oid belong to the iterator; if you want to retain them\n * after calling ref_iterator_advance() again or calling\n- * ref_iterator_abort(), you must make a copy. When the iteration has\n+ * ref_iterator_free(), you must make a copy. When the iteration has\n * been exhausted, ref_iterator_advance() releases any resources\n * associated with the iteration, frees the ref_iterator object, and\n * returns ITER_DONE. If you want to abort the iteration early, call\n- * ref_iterator_abort(), which also frees the ref_iterator object and\n+ * ref_iterator_free(), which also frees the ref_iterator object and\n * any associated resources. If there was an internal error advancing\n * to the next entry, ref_iterator_advance() aborts the iteration,\n * frees the ref_iterator, and returns ITER_ERROR.\n@@ -293,7 +293,7 @@ enum do_for_each_ref_flags {\n *\n * while ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n * if (want_to_stop_iteration()) {\n- * ok = ref_iterator_abort(iter);\n+ * ok = ITER_DONE;\n * break;\n * }\n *\n@@ -307,6 +307,7 @@ enum do_for_each_ref_flags {\n *\n * if (ok != ITER_DONE)\n * handle_error();\n+ * ref_iterator_free(iter);\n */\n struct ref_iterator {\n \tstruct ref_iterator_vtable *vtable;\n@@ -333,12 +334,8 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator);\n int ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t struct object_id *peeled);\n \n-/*\n- * End the iteration before it has been exhausted, freeing the\n- * reference iterator and any associated resources and returning\n- * ITER_DONE. If the abort itself failed, return ITER_ERROR.\n- */\n-int ref_iterator_abort(struct ref_iterator *ref_iterator);\n+/* Free the reference iterator and any associated resources. */\n+void ref_iterator_free(struct ref_iterator *ref_iterator);\n \n /*\n * An iterator over nothing (its first ref_iterator_advance() call\n@@ -438,13 +435,6 @@ struct ref_iterator *prefix_ref_iterator_begin(struct ref_iterator *iter0,\n void base_ref_iterator_init(struct ref_iterator *iter,\n \t\t\t struct ref_iterator_vtable *vtable);\n \n-/*\n- * Base class destructor for ref_iterators. Destroy the ref_iterator\n- * part of iter and shallow-free the object. This is meant to be\n- * called only by the destructors of derived classes.\n- */\n-void base_ref_iterator_free(struct ref_iterator *iter);\n-\n /* Virtual function declarations for ref_iterators: */\n \n /*\n@@ -463,15 +453,14 @@ typedef int ref_iterator_peel_fn(struct ref_iterator *ref_iterator,\n \n /*\n * Implementations of this function should free any resources specific\n- * to the derived class, then call base_ref_iterator_free() to clean\n- * up and free the ref_iterator object.\n+ * to the derived class.\n */\n-typedef int ref_iterator_abort_fn(struct ref_iterator *ref_iterator);\n+typedef void ref_iterator_release_fn(struct ref_iterator *ref_iterator);\n \n struct ref_iterator_vtable {\n \tref_iterator_advance_fn *advance;\n \tref_iterator_peel_fn *peel;\n-\tref_iterator_abort_fn *abort;\n+\tref_iterator_release_fn *release;\n };\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 2a90e7cb391..06543f79c64 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -711,17 +711,10 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\tbreak;\n \t}\n \n-\tif (iter->err > 0) {\n-\t\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\t\treturn ITER_ERROR;\n+\tif (iter->err > 0)\n \t\treturn ITER_DONE;\n-\t}\n-\n-\tif (iter->err < 0) {\n-\t\tref_iterator_abort(ref_iterator);\n+\tif (iter->err < 0)\n \t\treturn ITER_ERROR;\n-\t}\n-\n \treturn ITER_OK;\n }\n \n@@ -740,7 +733,7 @@ static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn -1;\n }\n \n-static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void reftable_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct reftable_ref_iterator *iter =\n \t\t(struct reftable_ref_iterator *)ref_iterator;\n@@ -751,14 +744,12 @@ static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n \t\t\tfree(iter->exclude_patterns[i]);\n \t\tfree(iter->exclude_patterns);\n \t}\n-\tfree(iter);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n \t.advance = reftable_ref_iterator_advance,\n \t.peel = reftable_ref_iterator_peel,\n-\t.abort = reftable_ref_iterator_abort\n+\t.release = reftable_ref_iterator_release,\n };\n \n static int qsort_strcmp(const void *va, const void *vb)\n@@ -2017,17 +2008,10 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \t\tbreak;\n \t}\n \n-\tif (iter->err > 0) {\n-\t\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\t\treturn ITER_ERROR;\n+\tif (iter->err > 0)\n \t\treturn ITER_DONE;\n-\t}\n-\n-\tif (iter->err < 0) {\n-\t\tref_iterator_abort(ref_iterator);\n+\tif (iter->err < 0)\n \t\treturn ITER_ERROR;\n-\t}\n-\n \treturn ITER_OK;\n }\n \n@@ -2038,21 +2022,19 @@ static int reftable_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSE\n \treturn -1;\n }\n \n-static int reftable_reflog_iterator_abort(struct ref_iterator *ref_iterator)\n+static void reftable_reflog_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct reftable_reflog_iterator *iter =\n \t\t(struct reftable_reflog_iterator *)ref_iterator;\n \treftable_log_record_release(&iter->log);\n \treftable_iterator_destroy(&iter->iter);\n \tstrbuf_release(&iter->last_name);\n-\tfree(iter);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable reftable_reflog_iterator_vtable = {\n \t.advance = reftable_reflog_iterator_advance,\n \t.peel = reftable_reflog_iterator_peel,\n-\t.abort = reftable_reflog_iterator_abort\n+\t.release = reftable_reflog_iterator_release,\n };\n \n static struct reftable_reflog_iterator *reflog_iterator_for_stack(struct reftable_ref_store *refs,\ndiff --git a/t/helper/test-dir-iterator.c b/t/helper/test-dir-iterator.c\nindex 6b297bd7536..8d46e8ba409 100644\n--- a/t/helper/test-dir-iterator.c\n+++ b/t/helper/test-dir-iterator.c\n@@ -53,6 +53,7 @@ int cmd__dir_iterator(int argc, const char **argv)\n \t\tprintf(\"(%s) [%s] %s\\n\", diter->relative_path, diter->basename,\n \t\t diter->path.buf);\n \t}\n+\tdir_iterator_free(diter);\n \n \tif (iter_status != ITER_DONE) {\n \t\tprintf(\"dir_iterator_advance failure\\n\");\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512983","messageId":"20250225-pks-update-ref-optimization-v3-11-77c3687cda75@pks.im","threadId":"62965","inReplyTo":"20250225-pks-update-ref-optimization-v3-0-77c3687cda75@pks.im","subject":"[PATCH v3 11/16] refs/iterator: implement seeking for merged iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T08:55:57Z","receivedAt":"2025-02-25T08:56:10Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking on merged iterators. The implementation is rather\nstraight forward, with the only exception that we must not deallocate\nthe underlying iterators once they have been exhausted.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/iterator.c | 38 +++++++++++++++++++++++++++++---------\n 1 file changed, 29 insertions(+), 9 deletions(-)\n\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex 757b105261a..63608ef9907 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -96,7 +96,8 @@ int is_empty_ref_iterator(struct ref_iterator *ref_iterator)\n struct merge_ref_iterator {\n \tstruct ref_iterator base;\n \n-\tstruct ref_iterator *iter0, *iter1;\n+\tstruct ref_iterator *iter0, *iter0_owned;\n+\tstruct ref_iterator *iter1, *iter1_owned;\n \n \tref_iterator_select_fn *select;\n \tvoid *cb_data;\n@@ -160,13 +161,11 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tif (!iter->current) {\n \t\t/* Initialize: advance both iterators to their first entries */\n \t\tif ((ok = ref_iterator_advance(iter->iter0)) != ITER_OK) {\n-\t\t\tref_iterator_free(iter->iter0);\n \t\t\titer->iter0 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n \t\t}\n \t\tif ((ok = ref_iterator_advance(iter->iter1)) != ITER_OK) {\n-\t\t\tref_iterator_free(iter->iter1);\n \t\t\titer->iter1 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -177,7 +176,6 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t * entry:\n \t\t */\n \t\tif ((ok = ref_iterator_advance(*iter->current)) != ITER_OK) {\n-\t\t\tref_iterator_free(*iter->current);\n \t\t\t*iter->current = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -206,7 +204,6 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \n \t\tif (selection & ITER_SKIP_SECONDARY) {\n \t\t\tif ((ok = ref_iterator_advance(*secondary)) != ITER_OK) {\n-\t\t\t\tref_iterator_free(*secondary);\n \t\t\t\t*secondary = NULL;\n \t\t\t\tif (ok == ITER_ERROR)\n \t\t\t\t\tgoto error;\n@@ -226,6 +223,28 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ITER_ERROR;\n }\n \n+static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct merge_ref_iterator *iter =\n+\t\t(struct merge_ref_iterator *)ref_iterator;\n+\tint ret;\n+\n+\titer->current = NULL;\n+\titer->iter0 = iter->iter0_owned;\n+\titer->iter1 = iter->iter1_owned;\n+\n+\tret = ref_iterator_seek(iter->iter0, prefix);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n+\tret = ref_iterator_seek(iter->iter1, prefix);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n+\treturn 0;\n+}\n+\n static int merge_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -242,12 +261,13 @@ static void merge_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct merge_ref_iterator *iter =\n \t\t(struct merge_ref_iterator *)ref_iterator;\n-\tref_iterator_free(iter->iter0);\n-\tref_iterator_free(iter->iter1);\n+\tref_iterator_free(iter->iter0_owned);\n+\tref_iterator_free(iter->iter1_owned);\n }\n \n static struct ref_iterator_vtable merge_ref_iterator_vtable = {\n \t.advance = merge_ref_iterator_advance,\n+\t.seek = merge_ref_iterator_seek,\n \t.peel = merge_ref_iterator_peel,\n \t.release = merge_ref_iterator_release,\n };\n@@ -268,8 +288,8 @@ struct ref_iterator *merge_ref_iterator_begin(\n \t */\n \n \tbase_ref_iterator_init(ref_iterator, &merge_ref_iterator_vtable);\n-\titer->iter0 = iter0;\n-\titer->iter1 = iter1;\n+\titer->iter0 = iter->iter0_owned = iter0;\n+\titer->iter1 = iter->iter1_owned = iter1;\n \titer->select = select;\n \titer->cb_data = cb_data;\n \titer->current = NULL;\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512984","messageId":"20250225-pks-update-ref-optimization-v3-12-77c3687cda75@pks.im","threadId":"62965","inReplyTo":"20250225-pks-update-ref-optimization-v3-0-77c3687cda75@pks.im","subject":"[PATCH v3 12/16] refs/iterator: implement seeking for reftable iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T08:55:58Z","receivedAt":"2025-02-25T08:56:11Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking of reftable iterators. As the low-level reftable\niterators already support seeking this change is straight-forward. Two\nnotes though:\n\n - We do not support seeking on reflog iterators. It is unclear what\n seeking would even look like in this context, as you typically would\n want to seek to a specific entry in the reflog for a specific ref.\n There is not currently a usecase for this, but if there ever is we\n can implement seeking in the future.\n\n - We start to check whether `reftable_stack_init_ref_iterator()` is\n successful.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 35 ++++++++++++++++++++++++++++++-----\n 1 file changed, 30 insertions(+), 5 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 06543f79c64..b0c09f34433 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -547,7 +547,7 @@ struct reftable_ref_iterator {\n \tstruct reftable_ref_record ref;\n \tstruct object_id oid;\n \n-\tconst char *prefix;\n+\tchar *prefix;\n \tsize_t prefix_len;\n \tchar **exclude_patterns;\n \tsize_t exclude_patterns_index;\n@@ -718,6 +718,20 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ITER_OK;\n }\n \n+static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct reftable_ref_iterator *iter =\n+\t\t(struct reftable_ref_iterator *)ref_iterator;\n+\n+\tfree(iter->prefix);\n+\titer->prefix = xstrdup_or_null(prefix);\n+\titer->prefix_len = prefix ? strlen(prefix) : 0;\n+\titer->err = reftable_iterator_seek_ref(&iter->iter, prefix);\n+\n+\treturn iter->err;\n+}\n+\n static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -744,10 +758,12 @@ static void reftable_ref_iterator_release(struct ref_iterator *ref_iterator)\n \t\t\tfree(iter->exclude_patterns[i]);\n \t\tfree(iter->exclude_patterns);\n \t}\n+\tfree(iter->prefix);\n }\n \n static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n \t.advance = reftable_ref_iterator_advance,\n+\t.seek = reftable_ref_iterator_seek,\n \t.peel = reftable_ref_iterator_peel,\n \t.release = reftable_ref_iterator_release,\n };\n@@ -806,8 +822,6 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_\n \n \titer = xcalloc(1, sizeof(*iter));\n \tbase_ref_iterator_init(&iter->base, &reftable_ref_iterator_vtable);\n-\titer->prefix = prefix;\n-\titer->prefix_len = prefix ? strlen(prefix) : 0;\n \titer->base.oid = &iter->oid;\n \titer->flags = flags;\n \titer->refs = refs;\n@@ -821,8 +835,11 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_\n \tif (ret)\n \t\tgoto done;\n \n-\treftable_stack_init_ref_iterator(stack, &iter->iter);\n-\tret = reftable_iterator_seek_ref(&iter->iter, prefix);\n+\tret = reftable_stack_init_ref_iterator(stack, &iter->iter);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_ref_iterator_seek(&iter->base, prefix);\n \tif (ret)\n \t\tgoto done;\n \n@@ -2015,6 +2032,13 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ITER_OK;\n }\n \n+static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n+\t\t\t\t\t const char *prefix UNUSED)\n+{\n+\tBUG(\"reftable reflog iterator cannot be seeked\");\n+\treturn -1;\n+}\n+\n static int reftable_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \t\t\t\t\t struct object_id *peeled UNUSED)\n {\n@@ -2033,6 +2057,7 @@ static void reftable_reflog_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable reftable_reflog_iterator_vtable = {\n \t.advance = reftable_reflog_iterator_advance,\n+\t.seek = reftable_reflog_iterator_seek,\n \t.peel = reftable_reflog_iterator_peel,\n \t.release = reftable_reflog_iterator_release,\n };\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512985","messageId":"20250225-pks-update-ref-optimization-v3-13-77c3687cda75@pks.im","threadId":"62965","inReplyTo":"20250225-pks-update-ref-optimization-v3-0-77c3687cda75@pks.im","subject":"[PATCH v3 13/16] refs/iterator: implement seeking for ref-cache iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T08:55:59Z","receivedAt":"2025-02-25T08:56:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking of ref-cache iterators. This is done by splitting most\nof the logic to seek iterators out of `cache_ref_iterator_begin()` and\nputting it into `cache_ref_iterator_seek()` so that we can reuse the\nlogic.\n\nNote that we cannot use the optimization anymore where we return an\nempty ref iterator when there aren't any references, as otherwise it\nwouldn't be possible to reseek the iterator to a different prefix that\nmay exist. This shouldn't be much of a performance concern though as we\nnow start to bail out early in case `advance()` sees that there are no\nmore directories to be searched.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/ref-cache.c | 79 ++++++++++++++++++++++++++++++++++++--------------------\n 1 file changed, 51 insertions(+), 28 deletions(-)\n\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 6457e02c1ea..c1f1bab1d50 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -362,9 +362,7 @@ struct cache_ref_iterator {\n \tstruct ref_iterator base;\n \n \t/*\n-\t * The number of levels currently on the stack. This is always\n-\t * at least 1, because when it becomes zero the iteration is\n-\t * ended and this struct is freed.\n+\t * The number of levels currently on the stack.\n \t */\n \tsize_t levels_nr;\n \n@@ -376,7 +374,7 @@ struct cache_ref_iterator {\n \t * The prefix is matched textually, without regard for path\n \t * component boundaries.\n \t */\n-\tconst char *prefix;\n+\tchar *prefix;\n \n \t/*\n \t * A stack of levels. levels[0] is the uppermost level that is\n@@ -389,6 +387,9 @@ struct cache_ref_iterator {\n \tstruct cache_ref_iterator_level *levels;\n \n \tstruct repository *repo;\n+\tstruct ref_cache *cache;\n+\n+\tint prime_dir;\n };\n \n static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n@@ -396,6 +397,9 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tstruct cache_ref_iterator *iter =\n \t\t(struct cache_ref_iterator *)ref_iterator;\n \n+\tif (!iter->levels_nr)\n+\t\treturn ITER_DONE;\n+\n \twhile (1) {\n \t\tstruct cache_ref_iterator_level *level =\n \t\t\t&iter->levels[iter->levels_nr - 1];\n@@ -444,6 +448,41 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t}\n }\n \n+static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct cache_ref_iterator *iter =\n+\t\t(struct cache_ref_iterator *)ref_iterator;\n+\tstruct cache_ref_iterator_level *level;\n+\tstruct ref_dir *dir;\n+\n+\tdir = get_ref_dir(iter->cache->root);\n+\tif (prefix && *prefix)\n+\t\tdir = find_containing_dir(dir, prefix);\n+\tif (!dir) {\n+\t\titer->levels_nr = 0;\n+\t\treturn 0;\n+\t}\n+\n+\tif (iter->prime_dir)\n+\t\tprime_ref_dir(dir, prefix);\n+\titer->levels_nr = 1;\n+\tlevel = &iter->levels[0];\n+\tlevel->index = -1;\n+\tlevel->dir = dir;\n+\n+\tif (prefix && *prefix) {\n+\t\tfree(iter->prefix);\n+\t\titer->prefix = xstrdup(prefix);\n+\t\tlevel->prefix_state = PREFIX_WITHIN_DIR;\n+\t} else {\n+\t\tFREE_AND_NULL(iter->prefix);\n+\t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -456,12 +495,13 @@ static void cache_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct cache_ref_iterator *iter =\n \t\t(struct cache_ref_iterator *)ref_iterator;\n-\tfree((char *)iter->prefix);\n+\tfree(iter->prefix);\n \tfree(iter->levels);\n }\n \n static struct ref_iterator_vtable cache_ref_iterator_vtable = {\n \t.advance = cache_ref_iterator_advance,\n+\t.seek = cache_ref_iterator_seek,\n \t.peel = cache_ref_iterator_peel,\n \t.release = cache_ref_iterator_release,\n };\n@@ -471,39 +511,22 @@ struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\n \t\t\t\t\t struct repository *repo,\n \t\t\t\t\t int prime_dir)\n {\n-\tstruct ref_dir *dir;\n \tstruct cache_ref_iterator *iter;\n \tstruct ref_iterator *ref_iterator;\n-\tstruct cache_ref_iterator_level *level;\n-\n-\tdir = get_ref_dir(cache->root);\n-\tif (prefix && *prefix)\n-\t\tdir = find_containing_dir(dir, prefix);\n-\tif (!dir)\n-\t\t/* There's nothing to iterate over. */\n-\t\treturn empty_ref_iterator_begin();\n-\n-\tif (prime_dir)\n-\t\tprime_ref_dir(dir, prefix);\n \n \tCALLOC_ARRAY(iter, 1);\n \tref_iterator = &iter->base;\n \tbase_ref_iterator_init(ref_iterator, &cache_ref_iterator_vtable);\n \tALLOC_GROW(iter->levels, 10, iter->levels_alloc);\n \n-\titer->levels_nr = 1;\n-\tlevel = &iter->levels[0];\n-\tlevel->index = -1;\n-\tlevel->dir = dir;\n+\titer->repo = repo;\n+\titer->cache = cache;\n+\titer->prime_dir = prime_dir;\n \n-\tif (prefix && *prefix) {\n-\t\titer->prefix = xstrdup(prefix);\n-\t\tlevel->prefix_state = PREFIX_WITHIN_DIR;\n-\t} else {\n-\t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n+\tif (cache_ref_iterator_seek(&iter->base, prefix) < 0) {\n+\t\tref_iterator_free(&iter->base);\n+\t\treturn NULL;\n \t}\n \n-\titer->repo = repo;\n-\n \treturn ref_iterator;\n }\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512986","messageId":"20250225-pks-update-ref-optimization-v3-14-77c3687cda75@pks.im","threadId":"62965","inReplyTo":"20250225-pks-update-ref-optimization-v3-0-77c3687cda75@pks.im","subject":"[PATCH v3 14/16] refs/iterator: implement seeking for packed-ref iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T08:56:00Z","receivedAt":"2025-02-25T08:56:13Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking of `packed-ref` iterators. The implementation is again\nstraight forward, except that we cannot continue to use the prefix\niterator as we would otherwise not be able to reseek the iterator\nanymore in case one first asks for an empty and then for a non-empty\nprefix. Instead, we open-code the logic to in `advance()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/packed-backend.c | 65 ++++++++++++++++++++++++++++++++++-----------------\n 1 file changed, 43 insertions(+), 22 deletions(-)\n\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 38a1956d1a8..f4c82ba2c7d 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -819,6 +819,8 @@ struct packed_ref_iterator {\n \n \tstruct snapshot *snapshot;\n \n+\tchar *prefix;\n+\n \t/* The current position in the snapshot's buffer: */\n \tconst char *pos;\n \n@@ -841,11 +843,9 @@ struct packed_ref_iterator {\n };\n \n /*\n- * Move the iterator to the next record in the snapshot, without\n- * respect for whether the record is actually required by the current\n- * iteration. Adjust the fields in `iter` and return `ITER_OK` or\n- * `ITER_DONE`. This function does not free the iterator in the case\n- * of `ITER_DONE`.\n+ * Move the iterator to the next record in the snapshot. Adjust the fields in\n+ * `iter` and return `ITER_OK` or `ITER_DONE`. This function does not free the\n+ * iterator in the case of `ITER_DONE`.\n */\n static int next_record(struct packed_ref_iterator *iter)\n {\n@@ -942,6 +942,9 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tint ok;\n \n \twhile ((ok = next_record(iter)) == ITER_OK) {\n+\t\tconst char *refname = iter->base.refname;\n+\t\tconst char *prefix = iter->prefix;\n+\n \t\tif (iter->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n \t\t !is_per_worktree_ref(iter->base.refname))\n \t\t\tcontinue;\n@@ -951,12 +954,41 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t\t\t\t &iter->oid, iter->flags))\n \t\t\tcontinue;\n \n+\t\twhile (prefix && *prefix) {\n+\t\t\tif (*refname < *prefix)\n+\t\t\t\tBUG(\"packed-refs backend yielded reference preceding its prefix\");\n+\t\t\telse if (*refname > *prefix)\n+\t\t\t\treturn ITER_DONE;\n+\t\t\tprefix++;\n+\t\t\trefname++;\n+\t\t}\n+\n \t\treturn ITER_OK;\n \t}\n \n \treturn ok;\n }\n \n+static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct packed_ref_iterator *iter =\n+\t\t(struct packed_ref_iterator *)ref_iterator;\n+\tconst char *start;\n+\n+\tif (prefix && *prefix)\n+\t\tstart = find_reference_location(iter->snapshot, prefix, 0);\n+\telse\n+\t\tstart = iter->snapshot->start;\n+\n+\tfree(iter->prefix);\n+\titer->prefix = xstrdup_or_null(prefix);\n+\titer->pos = start;\n+\titer->eof = iter->snapshot->eof;\n+\n+\treturn 0;\n+}\n+\n static int packed_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -979,11 +1011,13 @@ static void packed_ref_iterator_release(struct ref_iterator *ref_iterator)\n \t\t(struct packed_ref_iterator *)ref_iterator;\n \tstrbuf_release(&iter->refname_buf);\n \tfree(iter->jump);\n+\tfree(iter->prefix);\n \trelease_snapshot(iter->snapshot);\n }\n \n static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \t.advance = packed_ref_iterator_advance,\n+\t.seek = packed_ref_iterator_seek,\n \t.peel = packed_ref_iterator_peel,\n \t.release = packed_ref_iterator_release,\n };\n@@ -1097,7 +1131,6 @@ static struct ref_iterator *packed_ref_iterator_begin(\n {\n \tstruct packed_ref_store *refs;\n \tstruct snapshot *snapshot;\n-\tconst char *start;\n \tstruct packed_ref_iterator *iter;\n \tstruct ref_iterator *ref_iterator;\n \tunsigned int required_flags = REF_STORE_READ;\n@@ -1113,14 +1146,6 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \t */\n \tsnapshot = get_snapshot(refs);\n \n-\tif (prefix && *prefix)\n-\t\tstart = find_reference_location(snapshot, prefix, 0);\n-\telse\n-\t\tstart = snapshot->start;\n-\n-\tif (start == snapshot->eof)\n-\t\treturn empty_ref_iterator_begin();\n-\n \tCALLOC_ARRAY(iter, 1);\n \tref_iterator = &iter->base;\n \tbase_ref_iterator_init(ref_iterator, &packed_ref_iterator_vtable);\n@@ -1130,19 +1155,15 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \n \titer->snapshot = snapshot;\n \tacquire_snapshot(snapshot);\n-\n-\titer->pos = start;\n-\titer->eof = snapshot->eof;\n \tstrbuf_init(&iter->refname_buf, 0);\n-\n \titer->base.oid = &iter->oid;\n-\n \titer->repo = ref_store->repo;\n \titer->flags = flags;\n \n-\tif (prefix && *prefix)\n-\t\t/* Stop iteration after we've gone *past* prefix: */\n-\t\tref_iterator = prefix_ref_iterator_begin(ref_iterator, prefix, 0);\n+\tif (packed_ref_iterator_seek(&iter->base, prefix) < 0) {\n+\t\tref_iterator_free(&iter->base);\n+\t\treturn NULL;\n+\t}\n \n \treturn ref_iterator;\n }\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512987","messageId":"20250225-pks-update-ref-optimization-v3-15-77c3687cda75@pks.im","threadId":"62965","inReplyTo":"20250225-pks-update-ref-optimization-v3-0-77c3687cda75@pks.im","subject":"[PATCH v3 15/16] refs/iterator: implement seeking for files iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T08:56:01Z","receivedAt":"2025-02-25T08:56:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking for \"files\" iterators. As we simply use a ref-cache\niterator under the hood the implementation is straight-forward. Note\nthat we do not implement seeking on reflog iterators, same as with the\n\"reftable\" backend.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/files-backend.c | 16 ++++++++++++++++\n 1 file changed, 16 insertions(+)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 859f1c11941..4e1c50fead3 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -918,6 +918,14 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n+static int files_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct files_ref_iterator *iter =\n+\t\t(struct files_ref_iterator *)ref_iterator;\n+\treturn ref_iterator_seek(iter->iter0, prefix);\n+}\n+\n static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -936,6 +944,7 @@ static void files_ref_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable files_ref_iterator_vtable = {\n \t.advance = files_ref_iterator_advance,\n+\t.seek = files_ref_iterator_seek,\n \t.peel = files_ref_iterator_peel,\n \t.release = files_ref_iterator_release,\n };\n@@ -2294,6 +2303,12 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n+static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n+\t\t\t\t const char *prefix UNUSED)\n+{\n+\tBUG(\"ref_iterator_seek() called for reflog_iterator\");\n+}\n+\n static int files_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \t\t\t\t struct object_id *peeled UNUSED)\n {\n@@ -2309,6 +2324,7 @@ static void files_reflog_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable files_reflog_iterator_vtable = {\n \t.advance = files_reflog_iterator_advance,\n+\t.seek = files_reflog_iterator_seek,\n \t.peel = files_reflog_iterator_peel,\n \t.release = files_reflog_iterator_release,\n };\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"512988","messageId":"20250225-pks-update-ref-optimization-v3-16-77c3687cda75@pks.im","threadId":"62965","inReplyTo":"20250225-pks-update-ref-optimization-v3-0-77c3687cda75@pks.im","subject":"[PATCH v3 16/16] refs: reuse iterators when determining refname availability","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-25T08:56:02Z","receivedAt":"2025-02-25T08:56:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When verifying whether refnames are available we have to verify whether\nany reference exists that is nested under the current reference. E.g.\ngiven a reference \"refs/heads/foo\", we must make sure that there is no\nother reference \"refs/heads/foo/*\".\n\nThis check is performed using a ref iterator with the prefix set to the\nnested reference namespace. Until now it used to not be possible to\nreseek iterators, so we always had to reallocate the iterator for every\nsingle reference we're about to check. This keeps us from reusing state\nthat the iterator may have and that may make it work more efficiently.\n\nRefactor the logic to reseek iterators. This leads to a sizeable speedup\nwith the \"reftable\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 39.8 ms ± 0.9 ms [User: 29.7 ms, System: 9.8 ms]\n Range (min … max): 38.4 ms … 42.0 ms 62 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 31.9 ms ± 1.1 ms [User: 27.0 ms, System: 4.5 ms]\n Range (min … max): 29.8 ms … 34.3 ms 74 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.25 ± 0.05 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n\nThe \"files\" backend doesn't really show a huge impact:\n\n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 392.3 ms ± 7.1 ms [User: 59.7 ms, System: 328.8 ms]\n Range (min … max): 384.6 ms … 404.5 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 387.7 ms ± 7.4 ms [User: 54.6 ms, System: 329.6 ms]\n Range (min … max): 377.0 ms … 397.7 ms 10 runs\n\n Summary\n update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.01 ± 0.03 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n\nThis is mostly because it is way slower to begin with because it has to\ncreate a separate file for each new reference, so the milliseconds we\nshave off by reseeking the iterator doesn't really translate into a\nsignificant relative improvement.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs.c | 12 +++++++-----\n 1 file changed, 7 insertions(+), 5 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 8eff60a2186..6cbb9decdb0 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2555,8 +2555,13 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \t\tif (!initial_transaction) {\n \t\t\tint ok;\n \n-\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n-\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n+\t\t\tif (!iter) {\n+\t\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n+\t\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n+\t\t\t} else if (ref_iterator_seek(iter, dirname.buf) < 0) {\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\n \t\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\t\t\tif (skip &&\n \t\t\t\t string_list_has_string(skip, iter->refname))\n@@ -2569,9 +2574,6 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \n \t\t\tif (ok != ITER_DONE)\n \t\t\t\tBUG(\"error while iterating over references\");\n-\n-\t\t\tref_iterator_free(iter);\n-\t\t\titer = NULL;\n \t\t}\n \n \t\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n\n-- \n2.48.1.683.gf705b3209c.dirty\n\n"},{"id":"513007","messageId":"Z72yfF1samLJeSZd@ArchLinux","threadId":"62965","inReplyTo":"Z71zyrai9f9dPZRK@pks.im","subject":"Re: [PATCH v2 14/16] refs/iterator: implement seeking for `packed-ref` iterators","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-02-25T12:07:24Z","receivedAt":"2025-02-25T12:07:17Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Tue, Feb 25, 2025 at 08:39:54AM +0100, Patrick Steinhardt wrote:\n> On Mon, Feb 24, 2025 at 11:09:32PM +0800, shejialuo wrote:\n> > On Wed, Feb 19, 2025 at 02:23:41PM +0100, Patrick Steinhardt wrote:\n> > > Implement seeking of `packed-ref` iterators. The implementation is again\n> > > straight forward, except that we cannot continue to use the prefix\n> > > iterator as we would otherwise not be able to reseek the iterator\n> > > anymore in case one first asks for an empty and then for a non-empty\n> > > prefix. Instead, we open-code the logic to in `advance()`.\n> > > \n> > > Signed-off-by: Patrick Steinhardt \n> > > ---\n> > > refs/packed-backend.c | 62 +++++++++++++++++++++++++++++++++------------------\n> > > 1 file changed, 40 insertions(+), 22 deletions(-)\n> > > \n> > > diff --git a/refs/packed-backend.c b/refs/packed-backend.c\n> > > index 38a1956d1a8..71a38acfedc 100644\n> > > --- a/refs/packed-backend.c\n> > > +++ b/refs/packed-backend.c\n> > > @@ -951,12 +954,41 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> > > \t\t\t\t\t &iter->oid, iter->flags))\n> > > \t\t\tcontinue;\n> > > \n> > > +\t\twhile (prefix && *prefix) {\n> > > +\t\t\tif (*refname < *prefix)\n> > > +\t\t\t\tBUG(\"packed-refs backend yielded reference preceding its prefix\");\n> > > +\t\t\telse if (*refname > *prefix)\n> > > +\t\t\t\treturn ITER_DONE;\n> > > +\t\t\tprefix++;\n> > > +\t\t\trefname++;\n> > > +\t\t}\n> > \n> > Although I cannot understand the code, I want to ask a question here, we\n> > we need to do this in `advance`? Should we check this for\n> > `packed_ref_iterator_seek` or in the `next_record` function?\n> > \n> > Before we introduce `seek`, we don't need this logic. I somehow think we\n> > should do this in `packed_ref_iterator_seek`.\n> \n> We cannot do this in `packed_ref_iterator_seek()` because we need to do\n> it for every single record that we yield from the iterator. We _could_\n> do it in `next_record()`, but that function is rather complex already\n> and really only cares about yielding the next record. On the other hand,\n> `advance()` already knows to skip certain entries, so putting the logic\n> in there to also handle termination feels like a natural fit to me.\n> \n\nThanks for the detailed explanation.\n"},{"id":"513123","messageId":"xmqq5xkwbavg.fsf@gitster.g","threadId":"62965","inReplyTo":"20250225-pks-update-ref-optimization-v3-3-77c3687cda75@pks.im","subject":"Re: [PATCH v3 03/16] builtin/update-ref: skip ambiguity checks when parsing object IDs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-02-26T22:26:59Z","receivedAt":"2025-02-26T22:27:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> Most of the commands in git-update-ref(1) accept an old and/or new\n> object ID to update a specific reference to. These object IDs get parsed\n> via `repo_get_oid()`, which not only handles plain object IDs, but also\n> those that have a suffix like \"~\" or \"^2\". More surprisingly though, it\n> even knows to resolve references, despite the fact that its manpage does\n> not mention this fact even once.\n\nAre you referring to and other placeholders with \"oid\" in\ntheir names? I do think \"oid\" in our documentation implies that\nonly full hexadecimal object names are allowed. The glossary agrees\nby saying that is a synonym for that is\nusually 40-hex SHA-1. However, that is not strictly enforced and we\nsay (or its typed variants like ) even when a\ncommand takes any extended SHA-1 expression, as described in\nDocumentation/revisions.{txt,adoc}, not limited to full hexadecimal\nobject name.\n\nSo, I am somewhat sympathetic to your confusion, but not that much.\nWhen we wrote the command and documented it back in 2005, we did\nmean to take any object name that is spelled in any way, not just\nfull hexadecimal. You may want to update the manual to emphasize\nthat we encourage the use of full hexadecimal for this command and\nelsewhere where it is more appropriate.\n\n> One consequence of this is that we also check for ambiguous references:\n> when parsing a full object ID where the DWIM mechanism would also cause\n> us to resolve it as a branch, we'd end up printing a warning. While this\n> check makes sense to have in general, it is arguably less useful in the\n> context of git-update-ref(1).\n>\n> - The manpage is explicitly structured around object IDs. So if we see\n> a fully blown object ID, the intent should be quite clear in\n> general.\n>\n> - The command is part of our plumbing layer and not a tool that users\n> would generally use in interactive workflows. As such, the warning\n> will likely not be visible to anybody in the first place.\n\nIn addition, if the user meant to refer to a ref, it is possible to\ndisambiguate by prefixing refs/tags/ or whatever. So squelching the\nwarning unconditionally might make sense. We will yield the value\nof the full hexadecimal object name, instead of the value of the ref\nthat is confusingly named, so there is no material change in the\nbehaviour here.\n\nOK.\n\n"},{"id":"513143","messageId":"Z8BTKcUG6ogajfyq@pks.im","threadId":"62965","inReplyTo":"xmqq5xkwbavg.fsf@gitster.g","subject":"Re: [PATCH v3 03/16] builtin/update-ref: skip ambiguity checks when parsing object IDs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-27T11:57:29Z","receivedAt":"2025-02-27T11:57:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Feb 26, 2025 at 02:26:59PM -0800, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > Most of the commands in git-update-ref(1) accept an old and/or new\n> > object ID to update a specific reference to. These object IDs get parsed\n> > via `repo_get_oid()`, which not only handles plain object IDs, but also\n> > those that have a suffix like \"~\" or \"^2\". More surprisingly though, it\n> > even knows to resolve references, despite the fact that its manpage does\n> > not mention this fact even once.\n> \n> Are you referring to and other placeholders with \"oid\" in\n> their names? I do think \"oid\" in our documentation implies that\n> only full hexadecimal object names are allowed. The glossary agrees\n> by saying that is a synonym for that is\n> usually 40-hex SHA-1. However, that is not strictly enforced and we\n> say (or its typed variants like ) even when a\n> command takes any extended SHA-1 expression, as described in\n> Documentation/revisions.{txt,adoc}, not limited to full hexadecimal\n> object name.\n> \n> So, I am somewhat sympathetic to your confusion, but not that much.\n> When we wrote the command and documented it back in 2005, we did\n> mean to take any object name that is spelled in any way, not just\n> full hexadecimal. You may want to update the manual to emphasize\n> that we encourage the use of full hexadecimal for this command and\n> elsewhere where it is more appropriate.\n\nYeah. I have been aware of the behaviour beforehand, but an unsuspecting\nuser that reads through the manpage wouldn't be able to figure out at\nall that this is the case. I guess this is something we should improve,\nbut I think it's outside of the scope of this series. #leftoverbits\n\nPatrick\n"},{"id":"513242","messageId":"20250228-pks-update-ref-optimization-v4-0-6425c04268b5@pks.im","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im","subject":"[PATCH v4 00/16] refs: batch refname availability checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T09:26:37Z","receivedAt":"2025-02-28T09:26:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series has been inspired by brian's report that the reftable\nbackend is significantly slower when writing many references compared to\nthe files backend. As explained in that thread, the underlying issue is\nthe design of tombstone references: when we first delete all references\nin a repository and then recreate them, we still have all the tombstones\nand thus we need to churn through all of them to figure out that they\nhave been deleted in the first place. The files backend does not have\nthis issue.\n\nI consider the benchmark itself to be kind of broken, as it stems from\nus deleting all refs and then recreating them. And if you pack refs in\nbetween then the \"reftable\" backend outperforms the \"files\" backend.\n\nBut there are a couple of opportunities here anyway. While we cannot\nmake the underlying issue of tombstones being less efficient go away,\nthis has prompted me to have a deeper look at where we spend all the\ntime. There are three ideas in this series:\n\n - git-update-ref(1) performs ambiguity checks for any full-size object\n ID, which triggers a lot of reads. This is somewhat pointless though\n given that the manpage explicitly points out that the command is\n about object IDs, even though it does know to parse refs. But being\n part of plumbing, emitting the warning here does not make a ton of\n sense, and favoring object IDs over references in these cases is the\n obvious thing to do anyway.\n\n - For each ref \"refs/heads/bar\", we need to verify that neither\n \"refs/heads\" nor \"refs\" exists. This was repeated for every refname,\n but because most refnames use common prefixes this made us re-check\n a lot of prefixes. This is addressed by using a `strset` of already\n checked prefixes.\n\n - For each ref \"refs/heads/bar\", we need to verify that no ref\n \"refs/heads/bar/*\" exists. We always created a new ref iterator for\n this check, which requires us to discard all internal state and then\n recreate it. The reftable library has already been refactored though\n to have reseekable iterators, so we backfill this functionality to\n all the other iterators and then reuse the iterator.\n\nWith the (somewhat broken) benchmark we see a small speedup with the\n\"files\" backend:\n\n Benchmark 1: update-ref (refformat = files, revision = master)\n Time (mean ± σ): 234.4 ms ± 1.9 ms [User: 75.6 ms, System: 157.2 ms]\n Range (min … max): 232.2 ms … 236.9 ms 10 runs\n\n Benchmark 2: update-ref (refformat = files, revision = HEAD)\n Time (mean ± σ): 184.2 ms ± 2.0 ms [User: 62.8 ms, System: 119.9 ms]\n Range (min … max): 181.1 ms … 187.0 ms 10 runs\n\n Summary\n update-ref (refformat = files, revision = HEAD) ran\n 1.27 ± 0.02 times faster than update-ref (refformat = files, revision = master)\n\nAnd a huge speedup with the \"reftable\" backend:\n\n Benchmark 1: update-ref (refformat = reftable, revision = master)\n Time (mean ± σ): 16.852 s ± 0.061 s [User: 16.754 s, System: 0.059 s]\n Range (min … max): 16.785 s … 16.982 s 10 runs\n\n Benchmark 2: update-ref (refformat = reftable, revision = HEAD)\n Time (mean ± σ): 2.230 s ± 0.009 s [User: 2.192 s, System: 0.029 s]\n Range (min … max): 2.215 s … 2.244 s 10 runs\n\n Summary\n update-ref (refformat = reftable, revision = HEAD) ran\n 7.56 ± 0.04 times faster than update-ref (refformat = reftable, revision = master)\n\nWe're still not up to speed with the \"files\" backend, but considerably\nbetter. Given that this is an extreme edge case and not reflective of\nthe general case I'm okay with this result for now.\n\nBut more importantly, this refactoring also has a positive effect when\nupdating references in a repository with preexisting refs, which I\nconsider to be the more realistic scenario. The following benchmark\ncreates 10k refs with 100k preexisting refs.\n\nWith the \"files\" backend we see a modest improvement:\n\n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = master)\n Time (mean ± σ): 478.4 ms ± 11.9 ms [User: 96.7 ms, System: 379.6 ms]\n Range (min … max): 465.4 ms … 496.6 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 388.5 ms ± 10.3 ms [User: 52.0 ms, System: 333.8 ms]\n Range (min … max): 376.5 ms … 403.1 ms 10 runs\n\n Summary\n update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.23 ± 0.04 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = master)\n\nBut with the \"reftable\" backend we see an almost 5x improvement, where\nit's now ~15x faster than the \"files\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = master)\n Time (mean ± σ): 153.9 ms ± 2.0 ms [User: 96.5 ms, System: 56.6 ms]\n Range (min … max): 150.5 ms … 158.4 ms 18 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 32.2 ms ± 1.2 ms [User: 27.6 ms, System: 4.3 ms]\n Range (min … max): 29.8 ms … 38.6 ms 71 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 4.78 ± 0.19 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = master)\n\nThe series is structured as follows:\n\n - Patches 1 to 4 implement the logic to skip ambiguity checks in\n git-update-ref(1).\n\n - Patch 5 to 8 introduce batched checks.\n\n - Patch 9 deduplicates the ref prefix checks.\n\n - Patch 10 to 16 implement the infrastructure to reseek iterators.\n\n - Patch 17 starts to reuse iterators for nested ref checks.\n\nChanges in v2:\n - Point out why we also have to touch up the `dir_iterator`.\n - Fix up the comment explaining `ITER_DONE`.\n - Fix up comments that show usage patterns of the ref and dir iterator\n interfaces.\n - Start batching availability checks in the \"files\" backend, as well.\n - Improve the commit message that drops the ambiguity check so that we\n also point to 25fba78d36b (cat-file: disable object/refname\n ambiguity check for batch mode, 2013-07-12).\n - Link to v1: https://lore.kernel.org/r/20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im\n\nChanges in v3:\n - Fix one case where we didn't skip ambiguity checks in\n git-update-ref(1).\n - Document better that only the prefix can change on reseeking\n iterators. Other internal state will remain the same.\n - Fix a memory leak in the ref-cache iterator.\n - Don't ignore errors returned by `packed_ref_iterator_seek()`.\n - Link to v2: https://lore.kernel.org/r/20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im\n\nChanges in v4:\n - A couple of clarifications in the commit message that disabled\n ambiguity warnings.\n - Link to v3: https://lore.kernel.org/r/20250225-pks-update-ref-optimization-v3-0-77c3687cda75@pks.im\n\nThanks!\n\nPatrick\n\n[1]: \n\n---\nPatrick Steinhardt (16):\n object-name: introduce `repo_get_oid_with_flags()`\n object-name: allow skipping ambiguity checks in `get_oid()` family\n builtin/update-ref: skip ambiguity checks when parsing object IDs\n refs: introduce function to batch refname availability checks\n refs/reftable: batch refname availability checks\n refs/files: batch refname availability checks for normal transactions\n refs/files: batch refname availability checks for initial transactions\n refs: stop re-verifying common prefixes for availability\n refs/iterator: separate lifecycle from iteration\n refs/iterator: provide infrastructure to re-seek iterators\n refs/iterator: implement seeking for merged iterators\n refs/iterator: implement seeking for reftable iterators\n refs/iterator: implement seeking for ref-cache iterators\n refs/iterator: implement seeking for packed-ref iterators\n refs/iterator: implement seeking for files iterators\n refs: reuse iterators when determining refname availability\n\n builtin/clone.c | 2 +\n builtin/update-ref.c | 15 ++--\n dir-iterator.c | 24 +++---\n dir-iterator.h | 11 +--\n hash.h | 1 +\n iterator.h | 2 +-\n object-name.c | 18 +++--\n object-name.h | 6 ++\n refs.c | 186 ++++++++++++++++++++++++++-----------------\n refs.h | 12 +++\n refs/debug.c | 20 +++--\n refs/files-backend.c | 117 +++++++++++++++++----------\n refs/iterator.c | 145 +++++++++++++++++----------------\n refs/packed-backend.c | 92 ++++++++++++---------\n refs/ref-cache.c | 88 ++++++++++++--------\n refs/refs-internal.h | 53 +++++++-----\n refs/reftable-backend.c | 85 +++++++++++---------\n t/helper/test-dir-iterator.c | 1 +\n 18 files changed, 528 insertions(+), 350 deletions(-)\n\nRange-diff versus v3:\n\n 1: 9fa53bcfcc4 = 1: 9b4f39fb07f object-name: introduce `repo_get_oid_with_flags()`\n 2: 5f5d7fe8f2f = 2: dbd0e3d3da5 object-name: allow skipping ambiguity checks in `get_oid()` family\n 3: 0feb7829db9 ! 3: 63587d1c6ee builtin/update-ref: skip ambiguity checks when parsing object IDs\n @@ Commit message\n object ID to update a specific reference to. These object IDs get parsed\n via `repo_get_oid()`, which not only handles plain object IDs, but also\n those that have a suffix like \"~\" or \"^2\". More surprisingly though, it\n - even knows to resolve references, despite the fact that its manpage does\n - not mention this fact even once.\n + even knows to resolve arbitrary revisions, despite the fact that its\n + manpage does not mention this fact even once.\n \n One consequence of this is that we also check for ambiguous references:\n when parsing a full object ID where the DWIM mechanism would also cause\n us to resolve it as a branch, we'd end up printing a warning. While this\n check makes sense to have in general, it is arguably less useful in the\n - context of git-update-ref(1). This is out of two reasons:\n + context of git-update-ref(1). This is due to multiple reasons:\n \n - The manpage is explicitly structured around object IDs. So if we see\n a fully blown object ID, the intent should be quite clear in\n @@ Commit message\n would generally use in interactive workflows. As such, the warning\n will likely not be visible to anybody in the first place.\n \n + - Users can and should use the fully-qualified refname in case there\n + is any potential for ambiguity. And given that this command is part\n + of our plumbing layer, one should always try to be as defensive as\n + possible and use fully-qualified refnames.\n +\n Furthermore, this check can be quite expensive when updating lots of\n references via `--stdin`, because we try to read multiple references per\n object ID that we parse according to the DWIM rules. This effect can be\n @@ Commit message\n batch mode, 2013-07-12).\n \n Disable the warning in git-update-ref(1), which provides a significant\n - speedup with both backends. The following benchmark creates 10000 new\n - references with a 100000 preexisting refs with the \"files\" backend:\n + speedup with both backends. The user-visible outcome is unchanged even\n + when ambiguity exists, except that we don't show the warning anymore.\n +\n + The following benchmark creates 10000 new references with a 100000\n + preexisting refs with the \"files\" backend:\n \n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 467.3 ms ± 5.1 ms [User: 100.0 ms, System: 365.1 ms]\n 4: 4202541a160 = 4: 0f964bfc3af refs: introduce function to batch refname availability checks\n 5: e1ff37d766b = 5: 59c5f21a17d refs/reftable: batch refname availability checks\n 6: 90363250575 = 6: 395d008b3c8 refs/files: batch refname availability checks for normal transactions\n 7: a019888cd65 = 7: a8e33c64ece refs/files: batch refname availability checks for initial transactions\n 8: dab106f4530 = 8: 15b3cdae4ee refs: stop re-verifying common prefixes for availability\n 9: dc03794e974 = 9: 0f6901fe637 refs/iterator: separate lifecycle from iteration\n10: c3eb4ecb35f = 10: c4a9ae51590 refs/iterator: provide infrastructure to re-seek iterators\n11: a0b626e3fd6 = 11: d9398eb9f48 refs/iterator: implement seeking for merged iterators\n12: e0fe9998d7f = 12: b653eb7f663 refs/iterator: implement seeking for reftable iterators\n13: 84f6d1fc512 = 13: 6a616d0e328 refs/iterator: implement seeking for ref-cache iterators\n14: 4ccc5a7c6e3 = 14: 747188e9b81 refs/iterator: implement seeking for packed-ref iterators\n15: 7b25a9033d9 = 15: ebda415ae47 refs/iterator: implement seeking for files iterators\n16: 9688d6bc5cf = 16: 80e50011c28 refs: reuse iterators when determining refname availability\n\n---\nbase-commit: e2067b49ecaef9b7f51a17ce251f9207f72ef52d\nchange-id: 20250217-pks-update-ref-optimization-15c795e66e2b\n\n"},{"id":"513243","messageId":"20250228-pks-update-ref-optimization-v4-1-6425c04268b5@pks.im","threadId":"62965","inReplyTo":"20250228-pks-update-ref-optimization-v4-0-6425c04268b5@pks.im","subject":"[PATCH v4 01/16] object-name: introduce `repo_get_oid_with_flags()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T09:26:38Z","receivedAt":"2025-02-28T09:26:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new function `repo_get_oid_with_flags()`. This function\nbehaves the same as `repo_get_oid()`, except that it takes an extra\n`flags` parameter that it ends up passing to `get_oid_with_context()`.\n\nThis function will be used in a subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-name.c | 14 ++++++++------\n object-name.h | 6 ++++++\n 2 files changed, 14 insertions(+), 6 deletions(-)\n\ndiff --git a/object-name.c b/object-name.c\nindex 945d5bdef25..233f3f861e3 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -1794,18 +1794,20 @@ void object_context_release(struct object_context *ctx)\n \tstrbuf_release(&ctx->symlink_path);\n }\n \n-/*\n- * This is like \"get_oid_basic()\", except it allows \"object ID expressions\",\n- * notably \"xyz^\" for \"parent of xyz\"\n- */\n-int repo_get_oid(struct repository *r, const char *name, struct object_id *oid)\n+int repo_get_oid_with_flags(struct repository *r, const char *name,\n+\t\t\t struct object_id *oid, unsigned flags)\n {\n \tstruct object_context unused;\n-\tint ret = get_oid_with_context(r, name, 0, oid, &unused);\n+\tint ret = get_oid_with_context(r, name, flags, oid, &unused);\n \tobject_context_release(&unused);\n \treturn ret;\n }\n \n+int repo_get_oid(struct repository *r, const char *name, struct object_id *oid)\n+{\n+\treturn repo_get_oid_with_flags(r, name, oid, 0);\n+}\n+\n /*\n * This returns a non-zero value if the string (built using printf\n * format and the given arguments) is not a valid object.\ndiff --git a/object-name.h b/object-name.h\nindex 8dba4a47a47..cda4934cd5f 100644\n--- a/object-name.h\n+++ b/object-name.h\n@@ -51,6 +51,12 @@ void strbuf_repo_add_unique_abbrev(struct strbuf *sb, struct repository *repo,\n void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n \t\t\t int abbrev_len);\n \n+/*\n+ * This is like \"get_oid_basic()\", except it allows \"object ID expressions\",\n+ * notably \"xyz^\" for \"parent of xyz\". Accepts GET_OID_* flags.\n+ */\n+int repo_get_oid_with_flags(struct repository *r, const char *str,\n+\t\t\t struct object_id *oid, unsigned flags);\n int repo_get_oid(struct repository *r, const char *str, struct object_id *oid);\n __attribute__((format (printf, 2, 3)))\n int get_oidf(struct object_id *oid, const char *fmt, ...);\n\n-- \n2.49.0.rc0.375.gae4b89d849.dirty\n\n"},{"id":"513244","messageId":"20250228-pks-update-ref-optimization-v4-2-6425c04268b5@pks.im","threadId":"62965","inReplyTo":"20250228-pks-update-ref-optimization-v4-0-6425c04268b5@pks.im","subject":"[PATCH v4 02/16] object-name: allow skipping ambiguity checks in `get_oid()` family","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T09:26:39Z","receivedAt":"2025-02-28T09:26:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When reading an object ID via `get_oid_basic()` or any of its related\nfunctions we perform a check whether the object ID is ambiguous, which\ncan be the case when a reference with the same name exists. While the\ncheck is generally helpful, there are cases where it only adds to the\nruntime overhead without providing much of a benefit.\n\nAdd a new flag that allows us to disable the check. The flag will be\nused in a subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n hash.h | 1 +\n object-name.c | 4 +++-\n 2 files changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/hash.h b/hash.h\nindex 4367acfec50..79419016513 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -204,6 +204,7 @@ struct object_id {\n #define GET_OID_ONLY_TO_DIE 04000\n #define GET_OID_REQUIRE_PATH 010000\n #define GET_OID_HASH_ANY 020000\n+#define GET_OID_SKIP_AMBIGUITY_CHECK 040000\n \n #define GET_OID_DISAMBIGUATORS \\\n \t(GET_OID_COMMIT | GET_OID_COMMITTISH | \\\ndiff --git a/object-name.c b/object-name.c\nindex 233f3f861e3..85444dbb15b 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -961,7 +961,9 @@ static int get_oid_basic(struct repository *r, const char *str, int len,\n \tint fatal = !(flags & GET_OID_QUIETLY);\n \n \tif (len == r->hash_algo->hexsz && !get_oid_hex(str, oid)) {\n-\t\tif (repo_settings_get_warn_ambiguous_refs(r) && warn_on_object_refname_ambiguity) {\n+\t\tif (!(flags & GET_OID_SKIP_AMBIGUITY_CHECK) &&\n+\t\t repo_settings_get_warn_ambiguous_refs(r) &&\n+\t\t warn_on_object_refname_ambiguity) {\n \t\t\trefs_found = repo_dwim_ref(r, str, len, &tmp_oid, &real_ref, 0);\n \t\t\tif (refs_found > 0) {\n \t\t\t\twarning(warn_msg, len, str);\n\n-- \n2.49.0.rc0.375.gae4b89d849.dirty\n\n"},{"id":"513245","messageId":"20250228-pks-update-ref-optimization-v4-3-6425c04268b5@pks.im","threadId":"62965","inReplyTo":"20250228-pks-update-ref-optimization-v4-0-6425c04268b5@pks.im","subject":"[PATCH v4 03/16] builtin/update-ref: skip ambiguity checks when parsing object IDs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T09:26:40Z","receivedAt":"2025-02-28T09:26:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Most of the commands in git-update-ref(1) accept an old and/or new\nobject ID to update a specific reference to. These object IDs get parsed\nvia `repo_get_oid()`, which not only handles plain object IDs, but also\nthose that have a suffix like \"~\" or \"^2\". More surprisingly though, it\neven knows to resolve arbitrary revisions, despite the fact that its\nmanpage does not mention this fact even once.\n\nOne consequence of this is that we also check for ambiguous references:\nwhen parsing a full object ID where the DWIM mechanism would also cause\nus to resolve it as a branch, we'd end up printing a warning. While this\ncheck makes sense to have in general, it is arguably less useful in the\ncontext of git-update-ref(1). This is due to multiple reasons:\n\n - The manpage is explicitly structured around object IDs. So if we see\n a fully blown object ID, the intent should be quite clear in\n general.\n\n - The command is part of our plumbing layer and not a tool that users\n would generally use in interactive workflows. As such, the warning\n will likely not be visible to anybody in the first place.\n\n - Users can and should use the fully-qualified refname in case there\n is any potential for ambiguity. And given that this command is part\n of our plumbing layer, one should always try to be as defensive as\n possible and use fully-qualified refnames.\n\nFurthermore, this check can be quite expensive when updating lots of\nreferences via `--stdin`, because we try to read multiple references per\nobject ID that we parse according to the DWIM rules. This effect can be\nseen both with the \"files\" and \"reftable\" backend.\n\nThe issue is not unique to git-update-ref(1), but was also an issue in\ngit-cat-file(1), where it was addressed by disabling the ambiguity check\nin 25fba78d36b (cat-file: disable object/refname ambiguity check for\nbatch mode, 2013-07-12).\n\nDisable the warning in git-update-ref(1), which provides a significant\nspeedup with both backends. The user-visible outcome is unchanged even\nwhen ambiguity exists, except that we don't show the warning anymore.\n\nThe following benchmark creates 10000 new references with a 100000\npreexisting refs with the \"files\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 467.3 ms ± 5.1 ms [User: 100.0 ms, System: 365.1 ms]\n Range (min … max): 461.9 ms … 479.3 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 394.1 ms ± 5.8 ms [User: 63.3 ms, System: 327.6 ms]\n Range (min … max): 384.9 ms … 405.7 ms 10 runs\n\n Summary\n update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.19 ± 0.02 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n\nAnd with the \"reftable\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 146.9 ms ± 2.2 ms [User: 90.4 ms, System: 56.0 ms]\n Range (min … max): 142.7 ms … 150.8 ms 19 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 63.2 ms ± 1.1 ms [User: 41.0 ms, System: 21.8 ms]\n Range (min … max): 61.1 ms … 66.6 ms 41 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 2.32 ± 0.05 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n\nNote that the absolute improvement with both backends is roughly in the\nsame ballpark, but the relative improvement for the \"reftable\" backend\nis more significant because writing the new table to disk is faster in\nthe first place.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/update-ref.c | 15 ++++++++++-----\n 1 file changed, 10 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/update-ref.c b/builtin/update-ref.c\nindex 4d35bdc4b4b..1d541e13ade 100644\n--- a/builtin/update-ref.c\n+++ b/builtin/update-ref.c\n@@ -179,7 +179,8 @@ static int parse_next_oid(const char **next, const char *end,\n \t\t(*next)++;\n \t\t*next = parse_arg(*next, &arg);\n \t\tif (arg.len) {\n-\t\t\tif (repo_get_oid(the_repository, arg.buf, oid))\n+\t\t\tif (repo_get_oid_with_flags(the_repository, arg.buf, oid,\n+\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\t\tgoto invalid;\n \t\t} else {\n \t\t\t/* Without -z, an empty value means all zeros: */\n@@ -197,7 +198,8 @@ static int parse_next_oid(const char **next, const char *end,\n \t\t*next += arg.len;\n \n \t\tif (arg.len) {\n-\t\t\tif (repo_get_oid(the_repository, arg.buf, oid))\n+\t\t\tif (repo_get_oid_with_flags(the_repository, arg.buf, oid,\n+\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\t\tgoto invalid;\n \t\t} else if (flags & PARSE_SHA1_ALLOW_EMPTY) {\n \t\t\t/* With -z, treat an empty value as all zeros: */\n@@ -299,7 +301,8 @@ static void parse_cmd_symref_update(struct ref_transaction *transaction,\n \t\t\tdie(\"symref-update %s: expected old value\", refname);\n \n \t\tif (!strcmp(old_arg, \"oid\")) {\n-\t\t\tif (repo_get_oid(the_repository, old_target, &old_oid))\n+\t\t\tif (repo_get_oid_with_flags(the_repository, old_target, &old_oid,\n+\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\t\tdie(\"symref-update %s: invalid oid: %s\", refname, old_target);\n \n \t\t\thave_old_oid = 1;\n@@ -772,7 +775,8 @@ int cmd_update_ref(int argc,\n \t\trefname = argv[0];\n \t\tvalue = argv[1];\n \t\toldval = argv[2];\n-\t\tif (repo_get_oid(the_repository, value, &oid))\n+\t\tif (repo_get_oid_with_flags(the_repository, value, &oid,\n+\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\tdie(\"%s: not a valid SHA1\", value);\n \t}\n \n@@ -783,7 +787,8 @@ int cmd_update_ref(int argc,\n \t\t\t * must not already exist:\n \t\t\t */\n \t\t\toidclr(&oldoid, the_repository->hash_algo);\n-\t\telse if (repo_get_oid(the_repository, oldval, &oldoid))\n+\t\telse if (repo_get_oid_with_flags(the_repository, oldval, &oldoid,\n+\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\tdie(\"%s: not a valid old SHA1\", oldval);\n \t}\n \n\n-- \n2.49.0.rc0.375.gae4b89d849.dirty\n\n"},{"id":"513246","messageId":"20250228-pks-update-ref-optimization-v4-4-6425c04268b5@pks.im","threadId":"62965","inReplyTo":"20250228-pks-update-ref-optimization-v4-0-6425c04268b5@pks.im","subject":"[PATCH v4 04/16] refs: introduce function to batch refname availability checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T09:26:41Z","receivedAt":"2025-02-28T09:26:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `refs_verify_refname_available()` functions checks whether a\nreference update can be committed or whether it would conflict with\neither a prefix or suffix thereof. This function needs to be called once\nper reference that one wants to check, which requires us to redo a\ncouple of checks every time the function is called.\n\nIntroduce a new function `refs_verify_refnames_available()` that does\nthe same, but for a list of references. For now, the new function uses\nthe exact same implementation, except that we loop through all refnames\nprovided by the caller. This will be tuned in subsequent commits.\n\nThe existing `refs_verify_refname_available()` function is reimplemented\non top of the new function. As such, the diff is best viewed with the\n`--ignore-space-change option`.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs.c | 169 +++++++++++++++++++++++++++++++++++++----------------------------\n refs.h | 12 +++++\n 2 files changed, 109 insertions(+), 72 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex f4094a326a9..5a9b0f2fa1e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2467,19 +2467,15 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \treturn ret;\n }\n \n-int refs_verify_refname_available(struct ref_store *refs,\n-\t\t\t\t const char *refname,\n-\t\t\t\t const struct string_list *extras,\n-\t\t\t\t const struct string_list *skip,\n-\t\t\t\t unsigned int initial_transaction,\n-\t\t\t\t struct strbuf *err)\n+int refs_verify_refnames_available(struct ref_store *refs,\n+\t\t\t\t const struct string_list *refnames,\n+\t\t\t\t const struct string_list *extras,\n+\t\t\t\t const struct string_list *skip,\n+\t\t\t\t unsigned int initial_transaction,\n+\t\t\t\t struct strbuf *err)\n {\n-\tconst char *slash;\n-\tconst char *extra_refname;\n \tstruct strbuf dirname = STRBUF_INIT;\n \tstruct strbuf referent = STRBUF_INIT;\n-\tstruct object_id oid;\n-\tunsigned int type;\n \tint ret = -1;\n \n \t/*\n@@ -2489,79 +2485,91 @@ int refs_verify_refname_available(struct ref_store *refs,\n \n \tassert(err);\n \n-\tstrbuf_grow(&dirname, strlen(refname) + 1);\n-\tfor (slash = strchr(refname, '/'); slash; slash = strchr(slash + 1, '/')) {\n-\t\t/*\n-\t\t * Just saying \"Is a directory\" when we e.g. can't\n-\t\t * lock some multi-level ref isn't very informative,\n-\t\t * the user won't be told *what* is a directory, so\n-\t\t * let's not use strerror() below.\n-\t\t */\n-\t\tint ignore_errno;\n-\t\t/* Expand dirname to the new prefix, not including the trailing slash: */\n-\t\tstrbuf_add(&dirname, refname + dirname.len, slash - refname - dirname.len);\n+\tfor (size_t i = 0; i < refnames->nr; i++) {\n+\t\tconst char *refname = refnames->items[i].string;\n+\t\tconst char *extra_refname;\n+\t\tstruct object_id oid;\n+\t\tunsigned int type;\n+\t\tconst char *slash;\n+\n+\t\tstrbuf_reset(&dirname);\n+\n+\t\tfor (slash = strchr(refname, '/'); slash; slash = strchr(slash + 1, '/')) {\n+\t\t\t/*\n+\t\t\t * Just saying \"Is a directory\" when we e.g. can't\n+\t\t\t * lock some multi-level ref isn't very informative,\n+\t\t\t * the user won't be told *what* is a directory, so\n+\t\t\t * let's not use strerror() below.\n+\t\t\t */\n+\t\t\tint ignore_errno;\n+\n+\t\t\t/* Expand dirname to the new prefix, not including the trailing slash: */\n+\t\t\tstrbuf_add(&dirname, refname + dirname.len, slash - refname - dirname.len);\n+\n+\t\t\t/*\n+\t\t\t * We are still at a leading dir of the refname (e.g.,\n+\t\t\t * \"refs/foo\"; if there is a reference with that name,\n+\t\t\t * it is a conflict, *unless* it is in skip.\n+\t\t\t */\n+\t\t\tif (skip && string_list_has_string(skip, dirname.buf))\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (!initial_transaction &&\n+\t\t\t !refs_read_raw_ref(refs, dirname.buf, &oid, &referent,\n+\t\t\t\t\t &type, &ignore_errno)) {\n+\t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n+\t\t\t\t\t dirname.buf, refname);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\n+\t\t\tif (extras && string_list_has_string(extras, dirname.buf)) {\n+\t\t\t\tstrbuf_addf(err, _(\"cannot process '%s' and '%s' at the same time\"),\n+\t\t\t\t\t refname, dirname.buf);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\t\t}\n \n \t\t/*\n-\t\t * We are still at a leading dir of the refname (e.g.,\n-\t\t * \"refs/foo\"; if there is a reference with that name,\n-\t\t * it is a conflict, *unless* it is in skip.\n+\t\t * We are at the leaf of our refname (e.g., \"refs/foo/bar\").\n+\t\t * There is no point in searching for a reference with that\n+\t\t * name, because a refname isn't considered to conflict with\n+\t\t * itself. But we still need to check for references whose\n+\t\t * names are in the \"refs/foo/bar/\" namespace, because they\n+\t\t * *do* conflict.\n \t\t */\n-\t\tif (skip && string_list_has_string(skip, dirname.buf))\n-\t\t\tcontinue;\n+\t\tstrbuf_addstr(&dirname, refname + dirname.len);\n+\t\tstrbuf_addch(&dirname, '/');\n+\n+\t\tif (!initial_transaction) {\n+\t\t\tstruct ref_iterator *iter;\n+\t\t\tint ok;\n+\n+\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n+\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n+\t\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n+\t\t\t\tif (skip &&\n+\t\t\t\t string_list_has_string(skip, iter->refname))\n+\t\t\t\t\tcontinue;\n+\n+\t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n+\t\t\t\t\t iter->refname, refname);\n+\t\t\t\tref_iterator_abort(iter);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \n-\t\tif (!initial_transaction &&\n-\t\t !refs_read_raw_ref(refs, dirname.buf, &oid, &referent,\n-\t\t\t\t &type, &ignore_errno)) {\n-\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n-\t\t\t\t dirname.buf, refname);\n-\t\t\tgoto cleanup;\n+\t\t\tif (ok != ITER_DONE)\n+\t\t\t\tBUG(\"error while iterating over references\");\n \t\t}\n \n-\t\tif (extras && string_list_has_string(extras, dirname.buf)) {\n+\t\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n+\t\tif (extra_refname) {\n \t\t\tstrbuf_addf(err, _(\"cannot process '%s' and '%s' at the same time\"),\n-\t\t\t\t refname, dirname.buf);\n+\t\t\t\t refname, extra_refname);\n \t\t\tgoto cleanup;\n \t\t}\n \t}\n \n-\t/*\n-\t * We are at the leaf of our refname (e.g., \"refs/foo/bar\").\n-\t * There is no point in searching for a reference with that\n-\t * name, because a refname isn't considered to conflict with\n-\t * itself. But we still need to check for references whose\n-\t * names are in the \"refs/foo/bar/\" namespace, because they\n-\t * *do* conflict.\n-\t */\n-\tstrbuf_addstr(&dirname, refname + dirname.len);\n-\tstrbuf_addch(&dirname, '/');\n-\n-\tif (!initial_transaction) {\n-\t\tstruct ref_iterator *iter;\n-\t\tint ok;\n-\n-\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n-\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n-\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n-\t\t\tif (skip &&\n-\t\t\t string_list_has_string(skip, iter->refname))\n-\t\t\t\tcontinue;\n-\n-\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n-\t\t\t\t iter->refname, refname);\n-\t\t\tref_iterator_abort(iter);\n-\t\t\tgoto cleanup;\n-\t\t}\n-\n-\t\tif (ok != ITER_DONE)\n-\t\t\tBUG(\"error while iterating over references\");\n-\t}\n-\n-\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n-\tif (extra_refname)\n-\t\tstrbuf_addf(err, _(\"cannot process '%s' and '%s' at the same time\"),\n-\t\t\t refname, extra_refname);\n-\telse\n-\t\tret = 0;\n+\tret = 0;\n \n cleanup:\n \tstrbuf_release(&referent);\n@@ -2569,6 +2577,23 @@ int refs_verify_refname_available(struct ref_store *refs,\n \treturn ret;\n }\n \n+int refs_verify_refname_available(struct ref_store *refs,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const struct string_list *extras,\n+\t\t\t\t const struct string_list *skip,\n+\t\t\t\t unsigned int initial_transaction,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct string_list_item item = { .string = (char *) refname };\n+\tstruct string_list refnames = {\n+\t\t.items = &item,\n+\t\t.nr = 1,\n+\t};\n+\n+\treturn refs_verify_refnames_available(refs, &refnames, extras, skip,\n+\t\t\t\t\t initial_transaction, err);\n+}\n+\n struct do_for_each_reflog_help {\n \teach_reflog_fn *fn;\n \tvoid *cb_data;\ndiff --git a/refs.h b/refs.h\nindex a0cdd99250e..185aed5a461 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -124,6 +124,18 @@ int refs_verify_refname_available(struct ref_store *refs,\n \t\t\t\t unsigned int initial_transaction,\n \t\t\t\t struct strbuf *err);\n \n+/*\n+ * Same as `refs_verify_refname_available()`, but checking for a list of\n+ * refnames instead of only a single item. This is more efficient in the case\n+ * where one needs to check multiple refnames.\n+ */\n+int refs_verify_refnames_available(struct ref_store *refs,\n+\t\t\t\t const struct string_list *refnames,\n+\t\t\t\t const struct string_list *extras,\n+\t\t\t\t const struct string_list *skip,\n+\t\t\t\t unsigned int initial_transaction,\n+\t\t\t\t struct strbuf *err);\n+\n int refs_ref_exists(struct ref_store *refs, const char *refname);\n \n int should_autocreate_reflog(enum log_refs_config log_all_ref_updates,\n\n-- \n2.49.0.rc0.375.gae4b89d849.dirty\n\n"},{"id":"513247","messageId":"20250228-pks-update-ref-optimization-v4-5-6425c04268b5@pks.im","threadId":"62965","inReplyTo":"20250228-pks-update-ref-optimization-v4-0-6425c04268b5@pks.im","subject":"[PATCH v4 05/16] refs/reftable: batch refname availability checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T09:26:42Z","receivedAt":"2025-02-28T09:26:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor the \"reftable\" backend to batch the availability check for\nrefnames. This does not yet have an effect on performance as we\nessentially still call `refs_verify_refname_available()` in a loop, but\nthis will change in subsequent commits.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 16 ++++++++++------\n 1 file changed, 10 insertions(+), 6 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex d39a14c5a46..2a90e7cb391 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1069,6 +1069,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE|REF_STORE_MAIN, \"ref_transaction_prepare\");\n \tstruct strbuf referent = STRBUF_INIT, head_referent = STRBUF_INIT;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n \tstruct reftable_transaction_data *tx_data = NULL;\n \tstruct reftable_backend *be;\n \tstruct object_id head_oid;\n@@ -1224,12 +1225,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t\t * can output a proper error message instead of failing\n \t\t\t * at a later point.\n \t\t\t */\n-\t\t\tret = refs_verify_refname_available(ref_store, u->refname,\n-\t\t\t\t\t\t\t &affected_refnames, NULL,\n-\t\t\t\t\t\t\t transaction->flags & REF_TRANSACTION_FLAG_INITIAL,\n-\t\t\t\t\t\t\t err);\n-\t\t\tif (ret < 0)\n-\t\t\t\tgoto done;\n+\t\t\tstring_list_append(&refnames_to_check, u->refname);\n \n \t\t\t/*\n \t\t\t * There is no need to write the reference deletion\n@@ -1379,6 +1375,13 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t}\n \t}\n \n+\tstring_list_sort(&refnames_to_check);\n+\tret = refs_verify_refnames_available(ref_store, &refnames_to_check, &affected_refnames, NULL,\n+\t\t\t\t\t transaction->flags & REF_TRANSACTION_FLAG_INITIAL,\n+\t\t\t\t\t err);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \ttransaction->backend_data = tx_data;\n \ttransaction->state = REF_TRANSACTION_PREPARED;\n \n@@ -1394,6 +1397,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \tstring_list_clear(&affected_refnames, 0);\n \tstrbuf_release(&referent);\n \tstrbuf_release(&head_referent);\n+\tstring_list_clear(&refnames_to_check, 0);\n \n \treturn ret;\n }\n\n-- \n2.49.0.rc0.375.gae4b89d849.dirty\n\n"},{"id":"513248","messageId":"20250228-pks-update-ref-optimization-v4-6-6425c04268b5@pks.im","threadId":"62965","inReplyTo":"20250228-pks-update-ref-optimization-v4-0-6425c04268b5@pks.im","subject":"[PATCH v4 06/16] refs/files: batch refname availability checks for normal transactions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T09:26:43Z","receivedAt":"2025-02-28T09:26:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Same as the \"reftable\" backend that we have adapted in the preceding\ncommit to use batched refname availability checks we can also do so for\nthe \"files\" backend. Things are a bit more intricate here though, as we\ncall `refs_verify_refname_available()` in a set of different contexts:\n\n 1. `lock_raw_ref()` when it hits either EEXISTS or EISDIR when creating\n a new reference, mostly to create a nice, user-readable error\n message. This is nothing we have to care about too much, as we only\n hit this code path at most once when we hit a conflict.\n\n 2. `lock_raw_ref()` when it _could_ create the lockfile to check\n whether it is conflicting with any packed refs. In the general case,\n this code path will be hit once for every (successful) reference\n update.\n\n 3. `lock_ref_oid_basic()`, but it is only executed when copying or\n renaming references or when expiring reflogs. It will thus not be\n called in contexts where we have many references queued up.\n\n 4. `refs_refname_ref_available()`, but again only when copying or\n renaming references. It is thus not interesting due to the same\n reason as the previous case.\n\n 5. `files_transaction_finish_initial()`, which is only executed when\n creating a new repository or migrating references.\n\nSo out of these, only (2) and (5) are viable candidates to use the\nbatched checks.\n\nAdapt `lock_raw_ref()` accordingly by queueing up reference names that\nneed to be checked for availability and then checking them after we have\nprocessed all updates. This check is done before we (optionally) lock\nthe `packed-refs` file, which is somewhat flawed because it means that\nthe `packed-refs` could still change after the availability check and\nthus create an undetected conflict. But unconditionally locking the file\nwould change semantics that users are likely to rely on, so we keep the\ncurrent locking sequence intact, even if it's suboptmial.\n\nThe refactoring of `files_transaction_finish_initial()` will be done in\nthe next commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/files-backend.c | 42 +++++++++++++++++++++++++++++++-----------\n 1 file changed, 31 insertions(+), 11 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 29f08dced40..6ce79cf0791 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -678,6 +678,7 @@ static void unlock_ref(struct ref_lock *lock)\n */\n static int lock_raw_ref(struct files_ref_store *refs,\n \t\t\tconst char *refname, int mustexist,\n+\t\t\tstruct string_list *refnames_to_check,\n \t\t\tconst struct string_list *extras,\n \t\t\tstruct ref_lock **lock_p,\n \t\t\tstruct strbuf *referent,\n@@ -855,16 +856,11 @@ static int lock_raw_ref(struct files_ref_store *refs,\n \t\t}\n \n \t\t/*\n-\t\t * If the ref did not exist and we are creating it,\n-\t\t * make sure there is no existing packed ref that\n-\t\t * conflicts with refname:\n+\t\t * If the ref did not exist and we are creating it, we have to\n+\t\t * make sure there is no existing packed ref that conflicts\n+\t\t * with refname. This check is deferred so that we can batch it.\n \t\t */\n-\t\tif (refs_verify_refname_available(\n-\t\t\t\t refs->packed_ref_store, refname,\n-\t\t\t\t extras, NULL, 0, err)) {\n-\t\t\tret = TRANSACTION_NAME_CONFLICT;\n-\t\t\tgoto error_return;\n-\t\t}\n+\t\tstring_list_insert(refnames_to_check, refname);\n \t}\n \n \tret = 0;\n@@ -2569,6 +2565,7 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n \t\t\t struct ref_update *update,\n \t\t\t struct ref_transaction *transaction,\n \t\t\t const char *head_ref,\n+\t\t\t struct string_list *refnames_to_check,\n \t\t\t struct string_list *affected_refnames,\n \t\t\t struct strbuf *err)\n {\n@@ -2597,7 +2594,7 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n \t\tlock->count++;\n \t} else {\n \t\tret = lock_raw_ref(refs, update->refname, mustexist,\n-\t\t\t\t affected_refnames,\n+\t\t\t\t refnames_to_check, affected_refnames,\n \t\t\t\t &lock, &referent,\n \t\t\t\t &update->type, err);\n \t\tif (ret) {\n@@ -2811,6 +2808,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \tsize_t i;\n \tint ret = 0;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n \tchar *head_ref = NULL;\n \tint head_type;\n \tstruct files_transaction_backend_data *backend_data;\n@@ -2898,7 +2896,8 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\tstruct ref_update *update = transaction->updates[i];\n \n \t\tret = lock_ref_for_update(refs, update, transaction,\n-\t\t\t\t\t head_ref, &affected_refnames, err);\n+\t\t\t\t\t head_ref, &refnames_to_check,\n+\t\t\t\t\t &affected_refnames, err);\n \t\tif (ret)\n \t\t\tgoto cleanup;\n \n@@ -2930,6 +2929,26 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\t}\n \t}\n \n+\t/*\n+\t * Verify that none of the loose reference that we're about to write\n+\t * conflict with any existing packed references. Ideally, we'd do this\n+\t * check after the packed-refs are locked so that the file cannot\n+\t * change underneath our feet. But introducing such a lock now would\n+\t * probably do more harm than good as users rely on there not being a\n+\t * global lock with the \"files\" backend.\n+\t *\n+\t * Another alternative would be to do the check after the (optional)\n+\t * lock, but that would extend the time we spend in the globally-locked\n+\t * state.\n+\t *\n+\t * So instead, we accept the race for now.\n+\t */\n+\tif (refs_verify_refnames_available(refs->packed_ref_store, &refnames_to_check,\n+\t\t\t\t\t &affected_refnames, NULL, 0, err)) {\n+\t\tret = TRANSACTION_NAME_CONFLICT;\n+\t\tgoto cleanup;\n+\t}\n+\n \tif (packed_transaction) {\n \t\tif (packed_refs_lock(refs->packed_ref_store, 0, err)) {\n \t\t\tret = TRANSACTION_GENERIC_ERROR;\n@@ -2972,6 +2991,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n cleanup:\n \tfree(head_ref);\n \tstring_list_clear(&affected_refnames, 0);\n+\tstring_list_clear(&refnames_to_check, 0);\n \n \tif (ret)\n \t\tfiles_transaction_cleanup(refs, transaction);\n\n-- \n2.49.0.rc0.375.gae4b89d849.dirty\n\n"},{"id":"513250","messageId":"20250228-pks-update-ref-optimization-v4-7-6425c04268b5@pks.im","threadId":"62965","inReplyTo":"20250228-pks-update-ref-optimization-v4-0-6425c04268b5@pks.im","subject":"[PATCH v4 07/16] refs/files: batch refname availability checks for initial transactions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T09:26:44Z","receivedAt":"2025-02-28T09:26:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"files\" backend explicitly carves out special logic for its initial\ntransaction so that it can avoid writing out every single reference as\na loose reference. While the assumption is that there shouldn't be any\npreexisting references, we still have to verify that none of the newly\nwritten references will conflict with any other new reference in the\nsame transaction.\n\nRefactor the initial transaction to use batched refname availability\nchecks. This does not yet have an effect on performance as we still call\n`refs_verify_refname_available()` in a loop. But this will change in\nsubsequent commits and then impact performance when cloning a repository\nwith many references or when migrating references to the \"files\" format.\n\nThis doesn't yet have an effect on performance as the underlying\nlogic simply calls This will improve performance when cloning a repository with\nmany references or when migrating references from any format to the\n\"files\" format.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/files-backend.c | 23 ++++++++++++++++-------\n 1 file changed, 16 insertions(+), 7 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 6ce79cf0791..11a620ea11a 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3056,6 +3056,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \tsize_t i;\n \tint ret = 0;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n \tstruct ref_transaction *packed_transaction = NULL;\n \tstruct ref_transaction *loose_transaction = NULL;\n \n@@ -3105,11 +3106,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\t !is_null_oid(&update->old_oid))\n \t\t\tBUG(\"initial ref transaction with old_sha1 set\");\n \n-\t\tif (refs_verify_refname_available(&refs->base, update->refname,\n-\t\t\t\t\t\t &affected_refnames, NULL, 1, err)) {\n-\t\t\tret = TRANSACTION_NAME_CONFLICT;\n-\t\t\tgoto cleanup;\n-\t\t}\n+\t\tstring_list_append(&refnames_to_check, update->refname);\n \n \t\t/*\n \t\t * packed-refs don't support symbolic refs, root refs and reflogs,\n@@ -3145,8 +3142,19 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\t}\n \t}\n \n-\tif (packed_refs_lock(refs->packed_ref_store, 0, err) ||\n-\t ref_transaction_commit(packed_transaction, err)) {\n+\tif (packed_refs_lock(refs->packed_ref_store, 0, err)) {\n+\t\tret = TRANSACTION_GENERIC_ERROR;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif (refs_verify_refnames_available(&refs->base, &refnames_to_check,\n+\t\t\t\t\t &affected_refnames, NULL, 1, err)) {\n+\t\tpacked_refs_unlock(refs->packed_ref_store);\n+\t\tret = TRANSACTION_NAME_CONFLICT;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif (ref_transaction_commit(packed_transaction, err)) {\n \t\tret = TRANSACTION_GENERIC_ERROR;\n \t\tgoto cleanup;\n \t}\n@@ -3167,6 +3175,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\tref_transaction_free(packed_transaction);\n \ttransaction->state = REF_TRANSACTION_CLOSED;\n \tstring_list_clear(&affected_refnames, 0);\n+\tstring_list_clear(&refnames_to_check, 0);\n \treturn ret;\n }\n \n\n-- \n2.49.0.rc0.375.gae4b89d849.dirty\n\n"},{"id":"513249","messageId":"20250228-pks-update-ref-optimization-v4-8-6425c04268b5@pks.im","threadId":"62965","inReplyTo":"20250228-pks-update-ref-optimization-v4-0-6425c04268b5@pks.im","subject":"[PATCH v4 08/16] refs: stop re-verifying common prefixes for availability","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T09:26:45Z","receivedAt":"2025-02-28T09:26:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"One of the checks done by `refs_verify_refnames_available()` is whether\nany of the prefixes of a reference already exists. For example, given a\nreference \"refs/heads/main\", we'd check whether \"refs/heads\" or \"refs\"\nalready exist, and if so we'd abort the transaction.\n\nWhen updating multiple references at once, this check is performed for\neach of the references individually. Consequently, because references\ntend to have common prefixes like \"refs/heads/\" or refs/tags/\", we\nevaluate the availability of these prefixes repeatedly. Naturally this\nis a waste of compute, as the availability of those prefixes should in\ngeneral not change in the middle of a transaction. And if it would,\nbackends would notice at a later point in time.\n\nOptimize this pattern by storing prefixes in a `strset` so that we can\ntrivially track those prefixes that we have already checked. This leads\nto a significant speedup with the \"reftable\" backend when creating many\nreferences that all share a common prefix:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 63.1 ms ± 1.8 ms [User: 41.0 ms, System: 21.6 ms]\n Range (min … max): 60.6 ms … 69.5 ms 38 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 40.0 ms ± 1.3 ms [User: 29.3 ms, System: 10.3 ms]\n Range (min … max): 38.1 ms … 47.3 ms 61 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.58 ± 0.07 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n\nFor the \"files\" backend we see an improvement, but a much smaller one:\n\n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 395.8 ms ± 5.3 ms [User: 63.6 ms, System: 330.5 ms]\n Range (min … max): 387.0 ms … 404.6 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 386.0 ms ± 4.0 ms [User: 51.5 ms, System: 332.8 ms]\n Range (min … max): 380.8 ms … 392.6 ms 10 runs\n\n Summary\n update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.03 ± 0.02 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n\nThis change also leads to a modest improvement when writing references\nwith \"initial\" semantics, for example when migrating references. The\nfollowing benchmarks are migrating 1m references from the \"reftable\" to\nthe \"files\" backend:\n\n Benchmark 1: migrate reftable:files (refcount = 1000000, revision = HEAD~)\n Time (mean ± σ): 836.6 ms ± 5.6 ms [User: 645.2 ms, System: 185.2 ms]\n Range (min … max): 829.6 ms … 845.9 ms 10 runs\n\n Benchmark 2: migrate reftable:files (refcount = 1000000, revision = HEAD)\n Time (mean ± σ): 759.8 ms ± 5.1 ms [User: 574.9 ms, System: 178.9 ms]\n Range (min … max): 753.1 ms … 768.8 ms 10 runs\n\n Summary\n migrate reftable:files (refcount = 1000000, revision = HEAD) ran\n 1.10 ± 0.01 times faster than migrate reftable:files (refcount = 1000000, revision = HEAD~)\n\nAnd vice versa:\n\n Benchmark 1: migrate files:reftable (refcount = 1000000, revision = HEAD~)\n Time (mean ± σ): 870.7 ms ± 5.7 ms [User: 735.2 ms, System: 127.4 ms]\n Range (min … max): 861.6 ms … 883.2 ms 10 runs\n\n Benchmark 2: migrate files:reftable (refcount = 1000000, revision = HEAD)\n Time (mean ± σ): 799.1 ms ± 8.5 ms [User: 661.1 ms, System: 130.2 ms]\n Range (min … max): 787.5 ms … 812.6 ms 10 runs\n\n Summary\n migrate files:reftable (refcount = 1000000, revision = HEAD) ran\n 1.09 ± 0.01 times faster than migrate files:reftable (refcount = 1000000, revision = HEAD~)\n\nThe impact here is significantly smaller given that we don't perform any\nreference reads with \"initial\" semantics, so the speedup only comes from\nus doing less string list lookups.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs.c | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/refs.c b/refs.c\nindex 5a9b0f2fa1e..eaf41421f50 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2476,6 +2476,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n {\n \tstruct strbuf dirname = STRBUF_INIT;\n \tstruct strbuf referent = STRBUF_INIT;\n+\tstruct strset dirnames;\n \tint ret = -1;\n \n \t/*\n@@ -2485,6 +2486,8 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \n \tassert(err);\n \n+\tstrset_init(&dirnames);\n+\n \tfor (size_t i = 0; i < refnames->nr; i++) {\n \t\tconst char *refname = refnames->items[i].string;\n \t\tconst char *extra_refname;\n@@ -2514,6 +2517,14 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \t\t\tif (skip && string_list_has_string(skip, dirname.buf))\n \t\t\t\tcontinue;\n \n+\t\t\t/*\n+\t\t\t * If we've already seen the directory we don't need to\n+\t\t\t * process it again. Skip it to avoid checking checking\n+\t\t\t * common prefixes like \"refs/heads/\" repeatedly.\n+\t\t\t */\n+\t\t\tif (!strset_add(&dirnames, dirname.buf))\n+\t\t\t\tcontinue;\n+\n \t\t\tif (!initial_transaction &&\n \t\t\t !refs_read_raw_ref(refs, dirname.buf, &oid, &referent,\n \t\t\t\t\t &type, &ignore_errno)) {\n@@ -2574,6 +2585,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n cleanup:\n \tstrbuf_release(&referent);\n \tstrbuf_release(&dirname);\n+\tstrset_clear(&dirnames);\n \treturn ret;\n }\n \n\n-- \n2.49.0.rc0.375.gae4b89d849.dirty\n\n"},{"id":"513251","messageId":"20250228-pks-update-ref-optimization-v4-9-6425c04268b5@pks.im","threadId":"62965","inReplyTo":"20250228-pks-update-ref-optimization-v4-0-6425c04268b5@pks.im","subject":"[PATCH v4 09/16] refs/iterator: separate lifecycle from iteration","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T09:26:46Z","receivedAt":"2025-02-28T09:26:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The ref and reflog iterators have their lifecycle attached to iteration:\nonce the iterator reaches its end, it is automatically released and the\ncaller doesn't have to care about that anymore. When the iterator should\nbe released before it has been exhausted, callers must explicitly abort\nthe iterator via `ref_iterator_abort()`.\n\nThis lifecycle is somewhat unusual in the Git codebase and creates two\nproblems:\n\n - Callsites need to be very careful about when exactly they call\n `ref_iterator_abort()`, as calling the function is only valid when\n the iterator itself still is. This leads to somewhat awkward calling\n patterns in some situations.\n\n - It is impossible to reuse iterators and re-seek them to a different\n prefix. This feature isn't supported by any iterator implementation\n except for the reftable iterators anyway, but if it was implemented\n it would allow us to optimize cases where we need to search for\n specific references repeatedly by reusing internal state.\n\nDetangle the lifecycle from iteration so that we don't deallocate the\niterator anymore once it is exhausted. Instead, callers are now expected\nto always call a newly introduce `ref_iterator_free()` function that\ndeallocates the iterator and its internal state.\n\nNote that the `dir_iterator` is somewhat special because it does not\nimplement the `ref_iterator` interface, but is only used to implement\nother iterators. Consequently, we have to provide `dir_iterator_free()`\ninstead of `dir_iterator_release()` as the allocated structure itself is\nmanaged by the `dir_iterator` interfaces, as well, and not freed by\n`ref_iterator_free()` like in all the other cases.\n\nWhile at it, drop the return value of `ref_iterator_abort()`, which\nwasn't really required by any of the iterator implementations anyway.\nFurthermore, stop calling `base_ref_iterator_free()` in any of the\nbackends, but instead call it in `ref_iterator_free()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/clone.c | 2 +\n dir-iterator.c | 24 +++++------\n dir-iterator.h | 11 ++---\n iterator.h | 2 +-\n refs.c | 7 +++-\n refs/debug.c | 9 ++---\n refs/files-backend.c | 36 +++++------------\n refs/iterator.c | 95 ++++++++++++++------------------------------\n refs/packed-backend.c | 27 ++++++-------\n refs/ref-cache.c | 9 ++---\n refs/refs-internal.h | 29 +++++---------\n refs/reftable-backend.c | 34 ++++------------\n t/helper/test-dir-iterator.c | 1 +\n 13 files changed, 100 insertions(+), 186 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex fd001d800c6..ac3e84b2b18 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -426,6 +426,8 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\tstrbuf_setlen(src, src_len);\n \t\tdie(_(\"failed to iterate over '%s'\"), src->buf);\n \t}\n+\n+\tdir_iterator_free(iter);\n }\n \n static void clone_local(const char *src_repo, const char *dest_repo)\ndiff --git a/dir-iterator.c b/dir-iterator.c\nindex de619846f29..857e1d9bdaf 100644\n--- a/dir-iterator.c\n+++ b/dir-iterator.c\n@@ -193,9 +193,9 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \n \tif (S_ISDIR(iter->base.st.st_mode) && push_level(iter)) {\n \t\tif (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC)\n-\t\t\tgoto error_out;\n+\t\t\treturn ITER_ERROR;\n \t\tif (iter->levels_nr == 0)\n-\t\t\tgoto error_out;\n+\t\t\treturn ITER_ERROR;\n \t}\n \n \t/* Loop until we find an entry that we can give back to the caller. */\n@@ -211,11 +211,11 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\tint ret = next_directory_entry(level->dir, iter->base.path.buf, &de);\n \t\t\tif (ret < 0) {\n \t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n-\t\t\t\t\tgoto error_out;\n+\t\t\t\t\treturn ITER_ERROR;\n \t\t\t\tcontinue;\n \t\t\t} else if (ret > 0) {\n \t\t\t\tif (pop_level(iter) == 0)\n-\t\t\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\t\t\t\treturn ITER_DONE;\n \t\t\t\tcontinue;\n \t\t\t}\n \n@@ -223,7 +223,7 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t} else {\n \t\t\tif (level->entries_idx >= level->entries.nr) {\n \t\t\t\tif (pop_level(iter) == 0)\n-\t\t\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\t\t\t\treturn ITER_DONE;\n \t\t\t\tcontinue;\n \t\t\t}\n \n@@ -232,22 +232,21 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \n \t\tif (prepare_next_entry_data(iter, name)) {\n \t\t\tif (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC)\n-\t\t\t\tgoto error_out;\n+\t\t\t\treturn ITER_ERROR;\n \t\t\tcontinue;\n \t\t}\n \n \t\treturn ITER_OK;\n \t}\n-\n-error_out:\n-\tdir_iterator_abort(dir_iterator);\n-\treturn ITER_ERROR;\n }\n \n-int dir_iterator_abort(struct dir_iterator *dir_iterator)\n+void dir_iterator_free(struct dir_iterator *dir_iterator)\n {\n \tstruct dir_iterator_int *iter = (struct dir_iterator_int *)dir_iterator;\n \n+\tif (!iter)\n+\t\treturn;\n+\n \tfor (; iter->levels_nr; iter->levels_nr--) {\n \t\tstruct dir_iterator_level *level =\n \t\t\t&iter->levels[iter->levels_nr - 1];\n@@ -266,7 +265,6 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator)\n \tfree(iter->levels);\n \tstrbuf_release(&iter->base.path);\n \tfree(iter);\n-\treturn ITER_DONE;\n }\n \n struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags)\n@@ -301,7 +299,7 @@ struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags)\n \treturn dir_iterator;\n \n error_out:\n-\tdir_iterator_abort(dir_iterator);\n+\tdir_iterator_free(dir_iterator);\n \terrno = saved_errno;\n \treturn NULL;\n }\ndiff --git a/dir-iterator.h b/dir-iterator.h\nindex 6d438809b6e..ccd6a197343 100644\n--- a/dir-iterator.h\n+++ b/dir-iterator.h\n@@ -28,7 +28,7 @@\n *\n * while ((ok = dir_iterator_advance(iter)) == ITER_OK) {\n * if (want_to_stop_iteration()) {\n- * ok = dir_iterator_abort(iter);\n+ * ok = ITER_DONE;\n * break;\n * }\n *\n@@ -39,6 +39,7 @@\n *\n * if (ok != ITER_DONE)\n * handle_error();\n+ * dir_iterator_free(iter);\n *\n * Callers are allowed to modify iter->path while they are working,\n * but they must restore it to its original contents before calling\n@@ -107,11 +108,7 @@ struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags);\n */\n int dir_iterator_advance(struct dir_iterator *iterator);\n \n-/*\n- * End the iteration before it has been exhausted. Free the\n- * dir_iterator and any associated resources and return ITER_DONE. On\n- * error, free the dir_iterator and return ITER_ERROR.\n- */\n-int dir_iterator_abort(struct dir_iterator *iterator);\n+/* Free the dir_iterator and any associated resources. */\n+void dir_iterator_free(struct dir_iterator *iterator);\n \n #endif\ndiff --git a/iterator.h b/iterator.h\nindex 0f6900e43ad..6b77dcc2626 100644\n--- a/iterator.h\n+++ b/iterator.h\n@@ -12,7 +12,7 @@\n #define ITER_OK 0\n \n /*\n- * The iterator is exhausted and has been freed.\n+ * The iterator is exhausted.\n */\n #define ITER_DONE -1\n \ndiff --git a/refs.c b/refs.c\nindex eaf41421f50..8eff60a2186 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2476,6 +2476,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n {\n \tstruct strbuf dirname = STRBUF_INIT;\n \tstruct strbuf referent = STRBUF_INIT;\n+\tstruct ref_iterator *iter = NULL;\n \tstruct strset dirnames;\n \tint ret = -1;\n \n@@ -2552,7 +2553,6 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \t\tstrbuf_addch(&dirname, '/');\n \n \t\tif (!initial_transaction) {\n-\t\t\tstruct ref_iterator *iter;\n \t\t\tint ok;\n \n \t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n@@ -2564,12 +2564,14 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \n \t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n \t\t\t\t\t iter->refname, refname);\n-\t\t\t\tref_iterator_abort(iter);\n \t\t\t\tgoto cleanup;\n \t\t\t}\n \n \t\t\tif (ok != ITER_DONE)\n \t\t\t\tBUG(\"error while iterating over references\");\n+\n+\t\t\tref_iterator_free(iter);\n+\t\t\titer = NULL;\n \t\t}\n \n \t\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n@@ -2586,6 +2588,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \tstrbuf_release(&referent);\n \tstrbuf_release(&dirname);\n \tstrset_clear(&dirnames);\n+\tref_iterator_free(iter);\n \treturn ret;\n }\n \ndiff --git a/refs/debug.c b/refs/debug.c\nindex fbc4df08b43..a9786da4ba1 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -179,19 +179,18 @@ static int debug_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn res;\n }\n \n-static int debug_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void debug_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct debug_ref_iterator *diter =\n \t\t(struct debug_ref_iterator *)ref_iterator;\n-\tint res = diter->iter->vtable->abort(diter->iter);\n-\ttrace_printf_key(&trace_refs, \"iterator_abort: %d\\n\", res);\n-\treturn res;\n+\tditer->iter->vtable->release(diter->iter);\n+\ttrace_printf_key(&trace_refs, \"iterator_abort\\n\");\n }\n \n static struct ref_iterator_vtable debug_ref_iterator_vtable = {\n \t.advance = debug_ref_iterator_advance,\n \t.peel = debug_ref_iterator_peel,\n-\t.abort = debug_ref_iterator_abort,\n+\t.release = debug_ref_iterator_release,\n };\n \n static struct ref_iterator *\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 11a620ea11a..859f1c11941 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -915,10 +915,6 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\titer->iter0 = NULL;\n-\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\tok = ITER_ERROR;\n-\n \treturn ok;\n }\n \n@@ -931,23 +927,17 @@ static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator_peel(iter->iter0, peeled);\n }\n \n-static int files_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void files_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct files_ref_iterator *iter =\n \t\t(struct files_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->iter0)\n-\t\tok = ref_iterator_abort(iter->iter0);\n-\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n+\tref_iterator_free(iter->iter0);\n }\n \n static struct ref_iterator_vtable files_ref_iterator_vtable = {\n \t.advance = files_ref_iterator_advance,\n \t.peel = files_ref_iterator_peel,\n-\t.abort = files_ref_iterator_abort,\n+\t.release = files_ref_iterator_release,\n };\n \n static struct ref_iterator *files_ref_iterator_begin(\n@@ -1378,7 +1368,7 @@ static int should_pack_refs(struct files_ref_store *refs,\n \t\t\t\t iter->flags, opts))\n \t\t\trefcount++;\n \t\tif (refcount >= limit) {\n-\t\t\tref_iterator_abort(iter);\n+\t\t\tref_iterator_free(iter);\n \t\t\treturn 1;\n \t\t}\n \t}\n@@ -1386,6 +1376,7 @@ static int should_pack_refs(struct files_ref_store *refs,\n \tif (ret != ITER_DONE)\n \t\tdie(\"error while iterating over references\");\n \n+\tref_iterator_free(iter);\n \treturn 0;\n }\n \n@@ -1452,6 +1443,7 @@ static int files_pack_refs(struct ref_store *ref_store,\n \tpacked_refs_unlock(refs->packed_ref_store);\n \n \tprune_refs(refs, &refs_to_prune);\n+\tref_iterator_free(iter);\n \tstrbuf_release(&err);\n \treturn 0;\n }\n@@ -2299,9 +2291,6 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\titer->dir_iterator = NULL;\n-\tif (ref_iterator_abort(ref_iterator) == ITER_ERROR)\n-\t\tok = ITER_ERROR;\n \treturn ok;\n }\n \n@@ -2311,23 +2300,17 @@ static int files_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \tBUG(\"ref_iterator_peel() called for reflog_iterator\");\n }\n \n-static int files_reflog_iterator_abort(struct ref_iterator *ref_iterator)\n+static void files_reflog_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct files_reflog_iterator *iter =\n \t\t(struct files_reflog_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->dir_iterator)\n-\t\tok = dir_iterator_abort(iter->dir_iterator);\n-\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n+\tdir_iterator_free(iter->dir_iterator);\n }\n \n static struct ref_iterator_vtable files_reflog_iterator_vtable = {\n \t.advance = files_reflog_iterator_advance,\n \t.peel = files_reflog_iterator_peel,\n-\t.abort = files_reflog_iterator_abort,\n+\t.release = files_reflog_iterator_release,\n };\n \n static struct ref_iterator *reflog_iterator_begin(struct ref_store *ref_store,\n@@ -3837,6 +3820,7 @@ static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n \n out:\n+\tdir_iterator_free(iter);\n \tstrbuf_release(&sb);\n \tstrbuf_release(&refname);\n \treturn ret;\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex d25e568bf0b..aaeff270437 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -21,9 +21,14 @@ int ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator->vtable->peel(ref_iterator, peeled);\n }\n \n-int ref_iterator_abort(struct ref_iterator *ref_iterator)\n+void ref_iterator_free(struct ref_iterator *ref_iterator)\n {\n-\treturn ref_iterator->vtable->abort(ref_iterator);\n+\tif (ref_iterator) {\n+\t\tref_iterator->vtable->release(ref_iterator);\n+\t\t/* Help make use-after-free bugs fail quickly: */\n+\t\tref_iterator->vtable = NULL;\n+\t\tfree(ref_iterator);\n+\t}\n }\n \n void base_ref_iterator_init(struct ref_iterator *iter,\n@@ -36,20 +41,13 @@ void base_ref_iterator_init(struct ref_iterator *iter,\n \titer->flags = 0;\n }\n \n-void base_ref_iterator_free(struct ref_iterator *iter)\n-{\n-\t/* Help make use-after-free bugs fail quickly: */\n-\titer->vtable = NULL;\n-\tfree(iter);\n-}\n-\n struct empty_ref_iterator {\n \tstruct ref_iterator base;\n };\n \n-static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)\n {\n-\treturn ref_iterator_abort(ref_iterator);\n+\treturn ITER_DONE;\n }\n \n static int empty_ref_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n@@ -58,16 +56,14 @@ static int empty_ref_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \tBUG(\"peel called for empty iterator\");\n }\n \n-static int empty_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void empty_ref_iterator_release(struct ref_iterator *ref_iterator UNUSED)\n {\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable empty_ref_iterator_vtable = {\n \t.advance = empty_ref_iterator_advance,\n \t.peel = empty_ref_iterator_peel,\n-\t.abort = empty_ref_iterator_abort,\n+\t.release = empty_ref_iterator_release,\n };\n \n struct ref_iterator *empty_ref_iterator_begin(void)\n@@ -151,11 +147,13 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tif (!iter->current) {\n \t\t/* Initialize: advance both iterators to their first entries */\n \t\tif ((ok = ref_iterator_advance(iter->iter0)) != ITER_OK) {\n+\t\t\tref_iterator_free(iter->iter0);\n \t\t\titer->iter0 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n \t\t}\n \t\tif ((ok = ref_iterator_advance(iter->iter1)) != ITER_OK) {\n+\t\t\tref_iterator_free(iter->iter1);\n \t\t\titer->iter1 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -166,6 +164,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t * entry:\n \t\t */\n \t\tif ((ok = ref_iterator_advance(*iter->current)) != ITER_OK) {\n+\t\t\tref_iterator_free(*iter->current);\n \t\t\t*iter->current = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -179,9 +178,8 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t\titer->select(iter->iter0, iter->iter1, iter->cb_data);\n \n \t\tif (selection == ITER_SELECT_DONE) {\n-\t\t\treturn ref_iterator_abort(ref_iterator);\n+\t\t\treturn ITER_DONE;\n \t\t} else if (selection == ITER_SELECT_ERROR) {\n-\t\t\tref_iterator_abort(ref_iterator);\n \t\t\treturn ITER_ERROR;\n \t\t}\n \n@@ -195,6 +193,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \n \t\tif (selection & ITER_SKIP_SECONDARY) {\n \t\t\tif ((ok = ref_iterator_advance(*secondary)) != ITER_OK) {\n+\t\t\t\tref_iterator_free(*secondary);\n \t\t\t\t*secondary = NULL;\n \t\t\t\tif (ok == ITER_ERROR)\n \t\t\t\t\tgoto error;\n@@ -211,7 +210,6 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t}\n \n error:\n-\tref_iterator_abort(ref_iterator);\n \treturn ITER_ERROR;\n }\n \n@@ -227,28 +225,18 @@ static int merge_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator_peel(*iter->current, peeled);\n }\n \n-static int merge_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void merge_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct merge_ref_iterator *iter =\n \t\t(struct merge_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->iter0) {\n-\t\tif (ref_iterator_abort(iter->iter0) != ITER_DONE)\n-\t\t\tok = ITER_ERROR;\n-\t}\n-\tif (iter->iter1) {\n-\t\tif (ref_iterator_abort(iter->iter1) != ITER_DONE)\n-\t\t\tok = ITER_ERROR;\n-\t}\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n+\tref_iterator_free(iter->iter0);\n+\tref_iterator_free(iter->iter1);\n }\n \n static struct ref_iterator_vtable merge_ref_iterator_vtable = {\n \t.advance = merge_ref_iterator_advance,\n \t.peel = merge_ref_iterator_peel,\n-\t.abort = merge_ref_iterator_abort,\n+\t.release = merge_ref_iterator_release,\n };\n \n struct ref_iterator *merge_ref_iterator_begin(\n@@ -310,10 +298,10 @@ struct ref_iterator *overlay_ref_iterator_begin(\n \t * them.\n \t */\n \tif (is_empty_ref_iterator(front)) {\n-\t\tref_iterator_abort(front);\n+\t\tref_iterator_free(front);\n \t\treturn back;\n \t} else if (is_empty_ref_iterator(back)) {\n-\t\tref_iterator_abort(back);\n+\t\tref_iterator_free(back);\n \t\treturn front;\n \t}\n \n@@ -350,19 +338,10 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \n \twhile ((ok = ref_iterator_advance(iter->iter0)) == ITER_OK) {\n \t\tint cmp = compare_prefix(iter->iter0->refname, iter->prefix);\n-\n \t\tif (cmp < 0)\n \t\t\tcontinue;\n-\n-\t\tif (cmp > 0) {\n-\t\t\t/*\n-\t\t\t * As the source iterator is ordered, we\n-\t\t\t * can stop the iteration as soon as we see a\n-\t\t\t * refname that comes after the prefix:\n-\t\t\t */\n-\t\t\tok = ref_iterator_abort(iter->iter0);\n-\t\t\tbreak;\n-\t\t}\n+\t\tif (cmp > 0)\n+\t\t\treturn ITER_DONE;\n \n \t\tif (iter->trim) {\n \t\t\t/*\n@@ -386,9 +365,6 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\titer->iter0 = NULL;\n-\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\treturn ITER_ERROR;\n \treturn ok;\n }\n \n@@ -401,23 +377,18 @@ static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator_peel(iter->iter0, peeled);\n }\n \n-static int prefix_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void prefix_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct prefix_ref_iterator *iter =\n \t\t(struct prefix_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->iter0)\n-\t\tok = ref_iterator_abort(iter->iter0);\n+\tref_iterator_free(iter->iter0);\n \tfree(iter->prefix);\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n }\n \n static struct ref_iterator_vtable prefix_ref_iterator_vtable = {\n \t.advance = prefix_ref_iterator_advance,\n \t.peel = prefix_ref_iterator_peel,\n-\t.abort = prefix_ref_iterator_abort,\n+\t.release = prefix_ref_iterator_release,\n };\n \n struct ref_iterator *prefix_ref_iterator_begin(struct ref_iterator *iter0,\n@@ -453,20 +424,14 @@ int do_for_each_ref_iterator(struct ref_iterator *iter,\n \tcurrent_ref_iter = iter;\n \twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\tretval = fn(iter->refname, iter->referent, iter->oid, iter->flags, cb_data);\n-\t\tif (retval) {\n-\t\t\t/*\n-\t\t\t * If ref_iterator_abort() returns ITER_ERROR,\n-\t\t\t * we ignore that error in deference to the\n-\t\t\t * callback function's return value.\n-\t\t\t */\n-\t\t\tref_iterator_abort(iter);\n+\t\tif (retval)\n \t\t\tgoto out;\n-\t\t}\n \t}\n \n out:\n \tcurrent_ref_iter = old_ref_iter;\n \tif (ok == ITER_ERROR)\n-\t\treturn -1;\n+\t\tretval = -1;\n+\tref_iterator_free(iter);\n \treturn retval;\n }\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex a7b6f74b6e3..38a1956d1a8 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -954,9 +954,6 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\tok = ITER_ERROR;\n-\n \treturn ok;\n }\n \n@@ -976,23 +973,19 @@ static int packed_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t}\n }\n \n-static int packed_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void packed_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct packed_ref_iterator *iter =\n \t\t(struct packed_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n \tstrbuf_release(&iter->refname_buf);\n \tfree(iter->jump);\n \trelease_snapshot(iter->snapshot);\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n }\n \n static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \t.advance = packed_ref_iterator_advance,\n \t.peel = packed_ref_iterator_peel,\n-\t.abort = packed_ref_iterator_abort\n+\t.release = packed_ref_iterator_release,\n };\n \n static int jump_list_entry_cmp(const void *va, const void *vb)\n@@ -1362,8 +1355,10 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t */\n \titer = packed_ref_iterator_begin(&refs->base, \"\", NULL,\n \t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n-\tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n+\tif ((ok = ref_iterator_advance(iter)) != ITER_OK) {\n+\t\tref_iterator_free(iter);\n \t\titer = NULL;\n+\t}\n \n \ti = 0;\n \n@@ -1411,8 +1406,10 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t\t\t\t * the iterator over the unneeded\n \t\t\t\t * value.\n \t\t\t\t */\n-\t\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n+\t\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK) {\n+\t\t\t\t\tref_iterator_free(iter);\n \t\t\t\t\titer = NULL;\n+\t\t\t\t}\n \t\t\t\tcmp = +1;\n \t\t\t} else {\n \t\t\t\t/*\n@@ -1449,8 +1446,10 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t\t\t\t\t peel_error ? NULL : &peeled))\n \t\t\t\tgoto write_error;\n \n-\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n+\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK) {\n+\t\t\t\tref_iterator_free(iter);\n \t\t\t\titer = NULL;\n+\t\t\t}\n \t\t} else if (is_null_oid(&update->new_oid)) {\n \t\t\t/*\n \t\t\t * The update wants to delete the reference,\n@@ -1499,9 +1498,7 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t\t get_tempfile_path(refs->tempfile), strerror(errno));\n \n error:\n-\tif (iter)\n-\t\tref_iterator_abort(iter);\n-\n+\tref_iterator_free(iter);\n \tdelete_tempfile(&refs->tempfile);\n \treturn -1;\n }\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 02f09e4df88..6457e02c1ea 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -409,7 +409,7 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\tif (++level->index == level->dir->nr) {\n \t\t\t/* This level is exhausted; pop up a level */\n \t\t\tif (--iter->levels_nr == 0)\n-\t\t\t\treturn ref_iterator_abort(ref_iterator);\n+\t\t\t\treturn ITER_DONE;\n \n \t\t\tcontinue;\n \t\t}\n@@ -452,21 +452,18 @@ static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn peel_object(iter->repo, ref_iterator->oid, peeled) ? -1 : 0;\n }\n \n-static int cache_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void cache_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct cache_ref_iterator *iter =\n \t\t(struct cache_ref_iterator *)ref_iterator;\n-\n \tfree((char *)iter->prefix);\n \tfree(iter->levels);\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable cache_ref_iterator_vtable = {\n \t.advance = cache_ref_iterator_advance,\n \t.peel = cache_ref_iterator_peel,\n-\t.abort = cache_ref_iterator_abort\n+\t.release = cache_ref_iterator_release,\n };\n \n struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex aaab711bb96..74e2c03cef1 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -273,11 +273,11 @@ enum do_for_each_ref_flags {\n * the next reference and returns ITER_OK. The data pointed at by\n * refname and oid belong to the iterator; if you want to retain them\n * after calling ref_iterator_advance() again or calling\n- * ref_iterator_abort(), you must make a copy. When the iteration has\n+ * ref_iterator_free(), you must make a copy. When the iteration has\n * been exhausted, ref_iterator_advance() releases any resources\n * associated with the iteration, frees the ref_iterator object, and\n * returns ITER_DONE. If you want to abort the iteration early, call\n- * ref_iterator_abort(), which also frees the ref_iterator object and\n+ * ref_iterator_free(), which also frees the ref_iterator object and\n * any associated resources. If there was an internal error advancing\n * to the next entry, ref_iterator_advance() aborts the iteration,\n * frees the ref_iterator, and returns ITER_ERROR.\n@@ -293,7 +293,7 @@ enum do_for_each_ref_flags {\n *\n * while ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n * if (want_to_stop_iteration()) {\n- * ok = ref_iterator_abort(iter);\n+ * ok = ITER_DONE;\n * break;\n * }\n *\n@@ -307,6 +307,7 @@ enum do_for_each_ref_flags {\n *\n * if (ok != ITER_DONE)\n * handle_error();\n+ * ref_iterator_free(iter);\n */\n struct ref_iterator {\n \tstruct ref_iterator_vtable *vtable;\n@@ -333,12 +334,8 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator);\n int ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t struct object_id *peeled);\n \n-/*\n- * End the iteration before it has been exhausted, freeing the\n- * reference iterator and any associated resources and returning\n- * ITER_DONE. If the abort itself failed, return ITER_ERROR.\n- */\n-int ref_iterator_abort(struct ref_iterator *ref_iterator);\n+/* Free the reference iterator and any associated resources. */\n+void ref_iterator_free(struct ref_iterator *ref_iterator);\n \n /*\n * An iterator over nothing (its first ref_iterator_advance() call\n@@ -438,13 +435,6 @@ struct ref_iterator *prefix_ref_iterator_begin(struct ref_iterator *iter0,\n void base_ref_iterator_init(struct ref_iterator *iter,\n \t\t\t struct ref_iterator_vtable *vtable);\n \n-/*\n- * Base class destructor for ref_iterators. Destroy the ref_iterator\n- * part of iter and shallow-free the object. This is meant to be\n- * called only by the destructors of derived classes.\n- */\n-void base_ref_iterator_free(struct ref_iterator *iter);\n-\n /* Virtual function declarations for ref_iterators: */\n \n /*\n@@ -463,15 +453,14 @@ typedef int ref_iterator_peel_fn(struct ref_iterator *ref_iterator,\n \n /*\n * Implementations of this function should free any resources specific\n- * to the derived class, then call base_ref_iterator_free() to clean\n- * up and free the ref_iterator object.\n+ * to the derived class.\n */\n-typedef int ref_iterator_abort_fn(struct ref_iterator *ref_iterator);\n+typedef void ref_iterator_release_fn(struct ref_iterator *ref_iterator);\n \n struct ref_iterator_vtable {\n \tref_iterator_advance_fn *advance;\n \tref_iterator_peel_fn *peel;\n-\tref_iterator_abort_fn *abort;\n+\tref_iterator_release_fn *release;\n };\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 2a90e7cb391..06543f79c64 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -711,17 +711,10 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\tbreak;\n \t}\n \n-\tif (iter->err > 0) {\n-\t\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\t\treturn ITER_ERROR;\n+\tif (iter->err > 0)\n \t\treturn ITER_DONE;\n-\t}\n-\n-\tif (iter->err < 0) {\n-\t\tref_iterator_abort(ref_iterator);\n+\tif (iter->err < 0)\n \t\treturn ITER_ERROR;\n-\t}\n-\n \treturn ITER_OK;\n }\n \n@@ -740,7 +733,7 @@ static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn -1;\n }\n \n-static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void reftable_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct reftable_ref_iterator *iter =\n \t\t(struct reftable_ref_iterator *)ref_iterator;\n@@ -751,14 +744,12 @@ static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n \t\t\tfree(iter->exclude_patterns[i]);\n \t\tfree(iter->exclude_patterns);\n \t}\n-\tfree(iter);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n \t.advance = reftable_ref_iterator_advance,\n \t.peel = reftable_ref_iterator_peel,\n-\t.abort = reftable_ref_iterator_abort\n+\t.release = reftable_ref_iterator_release,\n };\n \n static int qsort_strcmp(const void *va, const void *vb)\n@@ -2017,17 +2008,10 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \t\tbreak;\n \t}\n \n-\tif (iter->err > 0) {\n-\t\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\t\treturn ITER_ERROR;\n+\tif (iter->err > 0)\n \t\treturn ITER_DONE;\n-\t}\n-\n-\tif (iter->err < 0) {\n-\t\tref_iterator_abort(ref_iterator);\n+\tif (iter->err < 0)\n \t\treturn ITER_ERROR;\n-\t}\n-\n \treturn ITER_OK;\n }\n \n@@ -2038,21 +2022,19 @@ static int reftable_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSE\n \treturn -1;\n }\n \n-static int reftable_reflog_iterator_abort(struct ref_iterator *ref_iterator)\n+static void reftable_reflog_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct reftable_reflog_iterator *iter =\n \t\t(struct reftable_reflog_iterator *)ref_iterator;\n \treftable_log_record_release(&iter->log);\n \treftable_iterator_destroy(&iter->iter);\n \tstrbuf_release(&iter->last_name);\n-\tfree(iter);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable reftable_reflog_iterator_vtable = {\n \t.advance = reftable_reflog_iterator_advance,\n \t.peel = reftable_reflog_iterator_peel,\n-\t.abort = reftable_reflog_iterator_abort\n+\t.release = reftable_reflog_iterator_release,\n };\n \n static struct reftable_reflog_iterator *reflog_iterator_for_stack(struct reftable_ref_store *refs,\ndiff --git a/t/helper/test-dir-iterator.c b/t/helper/test-dir-iterator.c\nindex 6b297bd7536..8d46e8ba409 100644\n--- a/t/helper/test-dir-iterator.c\n+++ b/t/helper/test-dir-iterator.c\n@@ -53,6 +53,7 @@ int cmd__dir_iterator(int argc, const char **argv)\n \t\tprintf(\"(%s) [%s] %s\\n\", diter->relative_path, diter->basename,\n \t\t diter->path.buf);\n \t}\n+\tdir_iterator_free(diter);\n \n \tif (iter_status != ITER_DONE) {\n \t\tprintf(\"dir_iterator_advance failure\\n\");\n\n-- \n2.49.0.rc0.375.gae4b89d849.dirty\n\n"},{"id":"513252","messageId":"20250228-pks-update-ref-optimization-v4-10-6425c04268b5@pks.im","threadId":"62965","inReplyTo":"20250228-pks-update-ref-optimization-v4-0-6425c04268b5@pks.im","subject":"[PATCH v4 10/16] refs/iterator: provide infrastructure to re-seek iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T09:26:47Z","receivedAt":"2025-02-28T09:26:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Reftable iterators need to be scrapped after they have either been\nexhausted or aren't useful to the caller anymore, and it is explicitly\nnot possible to reuse them for iterations. But enabling for reuse of\niterators may allow us to tune them by reusing internal state of an\niterator. The reftable iterators for example can already be reused\ninternally, but we're not able to expose this to any users outside of\nthe reftable backend.\n\nIntroduce a new `.seek` function in the ref iterator vtable that allows\ncallers to seek an iterator multiple times. It is expected to be\nfunctionally the same as calling `refs_ref_iterator_begin()` with a\ndifferent (or the same) prefix.\n\nNote that it is not possible to adjust parameters other than the seeked\nprefix for now, so exclude patterns, trimmed prefixes and flags will\nremain unchanged. We do not have a usecase for changing these parameters\nright now, but if we ever find one we can adapt accordingly.\n\nImplement the callback for trivial cases. The other iterators will be\nimplemented in subsequent commits.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/debug.c | 11 +++++++++++\n refs/iterator.c | 24 ++++++++++++++++++++++++\n refs/refs-internal.h | 24 ++++++++++++++++++++++++\n 3 files changed, 59 insertions(+)\n\ndiff --git a/refs/debug.c b/refs/debug.c\nindex a9786da4ba1..5390fa9c187 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -169,6 +169,16 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn res;\n }\n \n+static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct debug_ref_iterator *diter =\n+\t\t(struct debug_ref_iterator *)ref_iterator;\n+\tint res = diter->iter->vtable->seek(diter->iter, prefix);\n+\ttrace_printf_key(&trace_refs, \"iterator_seek: %s: %d\\n\", prefix ? prefix : \"\", res);\n+\treturn res;\n+}\n+\n static int debug_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -189,6 +199,7 @@ static void debug_ref_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable debug_ref_iterator_vtable = {\n \t.advance = debug_ref_iterator_advance,\n+\t.seek = debug_ref_iterator_seek,\n \t.peel = debug_ref_iterator_peel,\n \t.release = debug_ref_iterator_release,\n };\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex aaeff270437..757b105261a 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -15,6 +15,12 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ref_iterator->vtable->advance(ref_iterator);\n }\n \n+int ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t const char *prefix)\n+{\n+\treturn ref_iterator->vtable->seek(ref_iterator, prefix);\n+}\n+\n int ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t struct object_id *peeled)\n {\n@@ -50,6 +56,12 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)\n \treturn ITER_DONE;\n }\n \n+static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n+\t\t\t\t const char *prefix UNUSED)\n+{\n+\treturn 0;\n+}\n+\n static int empty_ref_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \t\t\t\t struct object_id *peeled UNUSED)\n {\n@@ -62,6 +74,7 @@ static void empty_ref_iterator_release(struct ref_iterator *ref_iterator UNUSED)\n \n static struct ref_iterator_vtable empty_ref_iterator_vtable = {\n \t.advance = empty_ref_iterator_advance,\n+\t.seek = empty_ref_iterator_seek,\n \t.peel = empty_ref_iterator_peel,\n \t.release = empty_ref_iterator_release,\n };\n@@ -368,6 +381,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n+static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct prefix_ref_iterator *iter =\n+\t\t(struct prefix_ref_iterator *)ref_iterator;\n+\tfree(iter->prefix);\n+\titer->prefix = xstrdup_or_null(prefix);\n+\treturn ref_iterator_seek(iter->iter0, prefix);\n+}\n+\n static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -387,6 +410,7 @@ static void prefix_ref_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable prefix_ref_iterator_vtable = {\n \t.advance = prefix_ref_iterator_advance,\n+\t.seek = prefix_ref_iterator_seek,\n \t.peel = prefix_ref_iterator_peel,\n \t.release = prefix_ref_iterator_release,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 74e2c03cef1..8f18274a165 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -327,6 +327,22 @@ struct ref_iterator {\n */\n int ref_iterator_advance(struct ref_iterator *ref_iterator);\n \n+/*\n+ * Seek the iterator to the first reference with the given prefix.\n+ * The prefix is matched as a literal string, without regard for path\n+ * separators. If prefix is NULL or the empty string, seek the iterator to the\n+ * first reference again.\n+ *\n+ * This function is expected to behave as if a new ref iterator with the same\n+ * prefix had been created, but allows reuse of iterators and thus may allow\n+ * the backend to optimize. Parameters other than the prefix that have been\n+ * passed when creating the iterator will remain unchanged.\n+ *\n+ * Returns 0 on success, a negative error code otherwise.\n+ */\n+int ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t const char *prefix);\n+\n /*\n * If possible, peel the reference currently being viewed by the\n * iterator. Return 0 on success.\n@@ -445,6 +461,13 @@ void base_ref_iterator_init(struct ref_iterator *iter,\n */\n typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);\n \n+/*\n+ * Seek the iterator to the first reference matching the given prefix. Should\n+ * behave the same as if a new iterator was created with the same prefix.\n+ */\n+typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix);\n+\n /*\n * Peels the current ref, returning 0 for success or -1 for failure.\n */\n@@ -459,6 +482,7 @@ typedef void ref_iterator_release_fn(struct ref_iterator *ref_iterator);\n \n struct ref_iterator_vtable {\n \tref_iterator_advance_fn *advance;\n+\tref_iterator_seek_fn *seek;\n \tref_iterator_peel_fn *peel;\n \tref_iterator_release_fn *release;\n };\n\n-- \n2.49.0.rc0.375.gae4b89d849.dirty\n\n"},{"id":"513253","messageId":"20250228-pks-update-ref-optimization-v4-11-6425c04268b5@pks.im","threadId":"62965","inReplyTo":"20250228-pks-update-ref-optimization-v4-0-6425c04268b5@pks.im","subject":"[PATCH v4 11/16] refs/iterator: implement seeking for merged iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T09:26:48Z","receivedAt":"2025-02-28T09:26:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking on merged iterators. The implementation is rather\nstraight forward, with the only exception that we must not deallocate\nthe underlying iterators once they have been exhausted.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/iterator.c | 38 +++++++++++++++++++++++++++++---------\n 1 file changed, 29 insertions(+), 9 deletions(-)\n\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex 757b105261a..63608ef9907 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -96,7 +96,8 @@ int is_empty_ref_iterator(struct ref_iterator *ref_iterator)\n struct merge_ref_iterator {\n \tstruct ref_iterator base;\n \n-\tstruct ref_iterator *iter0, *iter1;\n+\tstruct ref_iterator *iter0, *iter0_owned;\n+\tstruct ref_iterator *iter1, *iter1_owned;\n \n \tref_iterator_select_fn *select;\n \tvoid *cb_data;\n@@ -160,13 +161,11 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tif (!iter->current) {\n \t\t/* Initialize: advance both iterators to their first entries */\n \t\tif ((ok = ref_iterator_advance(iter->iter0)) != ITER_OK) {\n-\t\t\tref_iterator_free(iter->iter0);\n \t\t\titer->iter0 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n \t\t}\n \t\tif ((ok = ref_iterator_advance(iter->iter1)) != ITER_OK) {\n-\t\t\tref_iterator_free(iter->iter1);\n \t\t\titer->iter1 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -177,7 +176,6 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t * entry:\n \t\t */\n \t\tif ((ok = ref_iterator_advance(*iter->current)) != ITER_OK) {\n-\t\t\tref_iterator_free(*iter->current);\n \t\t\t*iter->current = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -206,7 +204,6 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \n \t\tif (selection & ITER_SKIP_SECONDARY) {\n \t\t\tif ((ok = ref_iterator_advance(*secondary)) != ITER_OK) {\n-\t\t\t\tref_iterator_free(*secondary);\n \t\t\t\t*secondary = NULL;\n \t\t\t\tif (ok == ITER_ERROR)\n \t\t\t\t\tgoto error;\n@@ -226,6 +223,28 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ITER_ERROR;\n }\n \n+static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct merge_ref_iterator *iter =\n+\t\t(struct merge_ref_iterator *)ref_iterator;\n+\tint ret;\n+\n+\titer->current = NULL;\n+\titer->iter0 = iter->iter0_owned;\n+\titer->iter1 = iter->iter1_owned;\n+\n+\tret = ref_iterator_seek(iter->iter0, prefix);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n+\tret = ref_iterator_seek(iter->iter1, prefix);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n+\treturn 0;\n+}\n+\n static int merge_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -242,12 +261,13 @@ static void merge_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct merge_ref_iterator *iter =\n \t\t(struct merge_ref_iterator *)ref_iterator;\n-\tref_iterator_free(iter->iter0);\n-\tref_iterator_free(iter->iter1);\n+\tref_iterator_free(iter->iter0_owned);\n+\tref_iterator_free(iter->iter1_owned);\n }\n \n static struct ref_iterator_vtable merge_ref_iterator_vtable = {\n \t.advance = merge_ref_iterator_advance,\n+\t.seek = merge_ref_iterator_seek,\n \t.peel = merge_ref_iterator_peel,\n \t.release = merge_ref_iterator_release,\n };\n@@ -268,8 +288,8 @@ struct ref_iterator *merge_ref_iterator_begin(\n \t */\n \n \tbase_ref_iterator_init(ref_iterator, &merge_ref_iterator_vtable);\n-\titer->iter0 = iter0;\n-\titer->iter1 = iter1;\n+\titer->iter0 = iter->iter0_owned = iter0;\n+\titer->iter1 = iter->iter1_owned = iter1;\n \titer->select = select;\n \titer->cb_data = cb_data;\n \titer->current = NULL;\n\n-- \n2.49.0.rc0.375.gae4b89d849.dirty\n\n"},{"id":"513254","messageId":"20250228-pks-update-ref-optimization-v4-12-6425c04268b5@pks.im","threadId":"62965","inReplyTo":"20250228-pks-update-ref-optimization-v4-0-6425c04268b5@pks.im","subject":"[PATCH v4 12/16] refs/iterator: implement seeking for reftable iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T09:26:49Z","receivedAt":"2025-02-28T09:26:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking of reftable iterators. As the low-level reftable\niterators already support seeking this change is straight-forward. Two\nnotes though:\n\n - We do not support seeking on reflog iterators. It is unclear what\n seeking would even look like in this context, as you typically would\n want to seek to a specific entry in the reflog for a specific ref.\n There is not currently a usecase for this, but if there ever is we\n can implement seeking in the future.\n\n - We start to check whether `reftable_stack_init_ref_iterator()` is\n successful.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 35 ++++++++++++++++++++++++++++++-----\n 1 file changed, 30 insertions(+), 5 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 06543f79c64..b0c09f34433 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -547,7 +547,7 @@ struct reftable_ref_iterator {\n \tstruct reftable_ref_record ref;\n \tstruct object_id oid;\n \n-\tconst char *prefix;\n+\tchar *prefix;\n \tsize_t prefix_len;\n \tchar **exclude_patterns;\n \tsize_t exclude_patterns_index;\n@@ -718,6 +718,20 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ITER_OK;\n }\n \n+static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct reftable_ref_iterator *iter =\n+\t\t(struct reftable_ref_iterator *)ref_iterator;\n+\n+\tfree(iter->prefix);\n+\titer->prefix = xstrdup_or_null(prefix);\n+\titer->prefix_len = prefix ? strlen(prefix) : 0;\n+\titer->err = reftable_iterator_seek_ref(&iter->iter, prefix);\n+\n+\treturn iter->err;\n+}\n+\n static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -744,10 +758,12 @@ static void reftable_ref_iterator_release(struct ref_iterator *ref_iterator)\n \t\t\tfree(iter->exclude_patterns[i]);\n \t\tfree(iter->exclude_patterns);\n \t}\n+\tfree(iter->prefix);\n }\n \n static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n \t.advance = reftable_ref_iterator_advance,\n+\t.seek = reftable_ref_iterator_seek,\n \t.peel = reftable_ref_iterator_peel,\n \t.release = reftable_ref_iterator_release,\n };\n@@ -806,8 +822,6 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_\n \n \titer = xcalloc(1, sizeof(*iter));\n \tbase_ref_iterator_init(&iter->base, &reftable_ref_iterator_vtable);\n-\titer->prefix = prefix;\n-\titer->prefix_len = prefix ? strlen(prefix) : 0;\n \titer->base.oid = &iter->oid;\n \titer->flags = flags;\n \titer->refs = refs;\n@@ -821,8 +835,11 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_\n \tif (ret)\n \t\tgoto done;\n \n-\treftable_stack_init_ref_iterator(stack, &iter->iter);\n-\tret = reftable_iterator_seek_ref(&iter->iter, prefix);\n+\tret = reftable_stack_init_ref_iterator(stack, &iter->iter);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_ref_iterator_seek(&iter->base, prefix);\n \tif (ret)\n \t\tgoto done;\n \n@@ -2015,6 +2032,13 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ITER_OK;\n }\n \n+static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n+\t\t\t\t\t const char *prefix UNUSED)\n+{\n+\tBUG(\"reftable reflog iterator cannot be seeked\");\n+\treturn -1;\n+}\n+\n static int reftable_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \t\t\t\t\t struct object_id *peeled UNUSED)\n {\n@@ -2033,6 +2057,7 @@ static void reftable_reflog_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable reftable_reflog_iterator_vtable = {\n \t.advance = reftable_reflog_iterator_advance,\n+\t.seek = reftable_reflog_iterator_seek,\n \t.peel = reftable_reflog_iterator_peel,\n \t.release = reftable_reflog_iterator_release,\n };\n\n-- \n2.49.0.rc0.375.gae4b89d849.dirty\n\n"},{"id":"513255","messageId":"20250228-pks-update-ref-optimization-v4-13-6425c04268b5@pks.im","threadId":"62965","inReplyTo":"20250228-pks-update-ref-optimization-v4-0-6425c04268b5@pks.im","subject":"[PATCH v4 13/16] refs/iterator: implement seeking for ref-cache iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T09:26:50Z","receivedAt":"2025-02-28T09:27:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking of ref-cache iterators. This is done by splitting most\nof the logic to seek iterators out of `cache_ref_iterator_begin()` and\nputting it into `cache_ref_iterator_seek()` so that we can reuse the\nlogic.\n\nNote that we cannot use the optimization anymore where we return an\nempty ref iterator when there aren't any references, as otherwise it\nwouldn't be possible to reseek the iterator to a different prefix that\nmay exist. This shouldn't be much of a performance concern though as we\nnow start to bail out early in case `advance()` sees that there are no\nmore directories to be searched.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/ref-cache.c | 79 ++++++++++++++++++++++++++++++++++++--------------------\n 1 file changed, 51 insertions(+), 28 deletions(-)\n\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 6457e02c1ea..c1f1bab1d50 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -362,9 +362,7 @@ struct cache_ref_iterator {\n \tstruct ref_iterator base;\n \n \t/*\n-\t * The number of levels currently on the stack. This is always\n-\t * at least 1, because when it becomes zero the iteration is\n-\t * ended and this struct is freed.\n+\t * The number of levels currently on the stack.\n \t */\n \tsize_t levels_nr;\n \n@@ -376,7 +374,7 @@ struct cache_ref_iterator {\n \t * The prefix is matched textually, without regard for path\n \t * component boundaries.\n \t */\n-\tconst char *prefix;\n+\tchar *prefix;\n \n \t/*\n \t * A stack of levels. levels[0] is the uppermost level that is\n@@ -389,6 +387,9 @@ struct cache_ref_iterator {\n \tstruct cache_ref_iterator_level *levels;\n \n \tstruct repository *repo;\n+\tstruct ref_cache *cache;\n+\n+\tint prime_dir;\n };\n \n static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n@@ -396,6 +397,9 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tstruct cache_ref_iterator *iter =\n \t\t(struct cache_ref_iterator *)ref_iterator;\n \n+\tif (!iter->levels_nr)\n+\t\treturn ITER_DONE;\n+\n \twhile (1) {\n \t\tstruct cache_ref_iterator_level *level =\n \t\t\t&iter->levels[iter->levels_nr - 1];\n@@ -444,6 +448,41 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t}\n }\n \n+static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct cache_ref_iterator *iter =\n+\t\t(struct cache_ref_iterator *)ref_iterator;\n+\tstruct cache_ref_iterator_level *level;\n+\tstruct ref_dir *dir;\n+\n+\tdir = get_ref_dir(iter->cache->root);\n+\tif (prefix && *prefix)\n+\t\tdir = find_containing_dir(dir, prefix);\n+\tif (!dir) {\n+\t\titer->levels_nr = 0;\n+\t\treturn 0;\n+\t}\n+\n+\tif (iter->prime_dir)\n+\t\tprime_ref_dir(dir, prefix);\n+\titer->levels_nr = 1;\n+\tlevel = &iter->levels[0];\n+\tlevel->index = -1;\n+\tlevel->dir = dir;\n+\n+\tif (prefix && *prefix) {\n+\t\tfree(iter->prefix);\n+\t\titer->prefix = xstrdup(prefix);\n+\t\tlevel->prefix_state = PREFIX_WITHIN_DIR;\n+\t} else {\n+\t\tFREE_AND_NULL(iter->prefix);\n+\t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -456,12 +495,13 @@ static void cache_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct cache_ref_iterator *iter =\n \t\t(struct cache_ref_iterator *)ref_iterator;\n-\tfree((char *)iter->prefix);\n+\tfree(iter->prefix);\n \tfree(iter->levels);\n }\n \n static struct ref_iterator_vtable cache_ref_iterator_vtable = {\n \t.advance = cache_ref_iterator_advance,\n+\t.seek = cache_ref_iterator_seek,\n \t.peel = cache_ref_iterator_peel,\n \t.release = cache_ref_iterator_release,\n };\n@@ -471,39 +511,22 @@ struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\n \t\t\t\t\t struct repository *repo,\n \t\t\t\t\t int prime_dir)\n {\n-\tstruct ref_dir *dir;\n \tstruct cache_ref_iterator *iter;\n \tstruct ref_iterator *ref_iterator;\n-\tstruct cache_ref_iterator_level *level;\n-\n-\tdir = get_ref_dir(cache->root);\n-\tif (prefix && *prefix)\n-\t\tdir = find_containing_dir(dir, prefix);\n-\tif (!dir)\n-\t\t/* There's nothing to iterate over. */\n-\t\treturn empty_ref_iterator_begin();\n-\n-\tif (prime_dir)\n-\t\tprime_ref_dir(dir, prefix);\n \n \tCALLOC_ARRAY(iter, 1);\n \tref_iterator = &iter->base;\n \tbase_ref_iterator_init(ref_iterator, &cache_ref_iterator_vtable);\n \tALLOC_GROW(iter->levels, 10, iter->levels_alloc);\n \n-\titer->levels_nr = 1;\n-\tlevel = &iter->levels[0];\n-\tlevel->index = -1;\n-\tlevel->dir = dir;\n+\titer->repo = repo;\n+\titer->cache = cache;\n+\titer->prime_dir = prime_dir;\n \n-\tif (prefix && *prefix) {\n-\t\titer->prefix = xstrdup(prefix);\n-\t\tlevel->prefix_state = PREFIX_WITHIN_DIR;\n-\t} else {\n-\t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n+\tif (cache_ref_iterator_seek(&iter->base, prefix) < 0) {\n+\t\tref_iterator_free(&iter->base);\n+\t\treturn NULL;\n \t}\n \n-\titer->repo = repo;\n-\n \treturn ref_iterator;\n }\n\n-- \n2.49.0.rc0.375.gae4b89d849.dirty\n\n"},{"id":"513256","messageId":"20250228-pks-update-ref-optimization-v4-14-6425c04268b5@pks.im","threadId":"62965","inReplyTo":"20250228-pks-update-ref-optimization-v4-0-6425c04268b5@pks.im","subject":"[PATCH v4 14/16] refs/iterator: implement seeking for packed-ref iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T09:26:51Z","receivedAt":"2025-02-28T09:27:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking of `packed-ref` iterators. The implementation is again\nstraight forward, except that we cannot continue to use the prefix\niterator as we would otherwise not be able to reseek the iterator\nanymore in case one first asks for an empty and then for a non-empty\nprefix. Instead, we open-code the logic to in `advance()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/packed-backend.c | 65 ++++++++++++++++++++++++++++++++++-----------------\n 1 file changed, 43 insertions(+), 22 deletions(-)\n\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 38a1956d1a8..f4c82ba2c7d 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -819,6 +819,8 @@ struct packed_ref_iterator {\n \n \tstruct snapshot *snapshot;\n \n+\tchar *prefix;\n+\n \t/* The current position in the snapshot's buffer: */\n \tconst char *pos;\n \n@@ -841,11 +843,9 @@ struct packed_ref_iterator {\n };\n \n /*\n- * Move the iterator to the next record in the snapshot, without\n- * respect for whether the record is actually required by the current\n- * iteration. Adjust the fields in `iter` and return `ITER_OK` or\n- * `ITER_DONE`. This function does not free the iterator in the case\n- * of `ITER_DONE`.\n+ * Move the iterator to the next record in the snapshot. Adjust the fields in\n+ * `iter` and return `ITER_OK` or `ITER_DONE`. This function does not free the\n+ * iterator in the case of `ITER_DONE`.\n */\n static int next_record(struct packed_ref_iterator *iter)\n {\n@@ -942,6 +942,9 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tint ok;\n \n \twhile ((ok = next_record(iter)) == ITER_OK) {\n+\t\tconst char *refname = iter->base.refname;\n+\t\tconst char *prefix = iter->prefix;\n+\n \t\tif (iter->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n \t\t !is_per_worktree_ref(iter->base.refname))\n \t\t\tcontinue;\n@@ -951,12 +954,41 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t\t\t\t &iter->oid, iter->flags))\n \t\t\tcontinue;\n \n+\t\twhile (prefix && *prefix) {\n+\t\t\tif (*refname < *prefix)\n+\t\t\t\tBUG(\"packed-refs backend yielded reference preceding its prefix\");\n+\t\t\telse if (*refname > *prefix)\n+\t\t\t\treturn ITER_DONE;\n+\t\t\tprefix++;\n+\t\t\trefname++;\n+\t\t}\n+\n \t\treturn ITER_OK;\n \t}\n \n \treturn ok;\n }\n \n+static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct packed_ref_iterator *iter =\n+\t\t(struct packed_ref_iterator *)ref_iterator;\n+\tconst char *start;\n+\n+\tif (prefix && *prefix)\n+\t\tstart = find_reference_location(iter->snapshot, prefix, 0);\n+\telse\n+\t\tstart = iter->snapshot->start;\n+\n+\tfree(iter->prefix);\n+\titer->prefix = xstrdup_or_null(prefix);\n+\titer->pos = start;\n+\titer->eof = iter->snapshot->eof;\n+\n+\treturn 0;\n+}\n+\n static int packed_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -979,11 +1011,13 @@ static void packed_ref_iterator_release(struct ref_iterator *ref_iterator)\n \t\t(struct packed_ref_iterator *)ref_iterator;\n \tstrbuf_release(&iter->refname_buf);\n \tfree(iter->jump);\n+\tfree(iter->prefix);\n \trelease_snapshot(iter->snapshot);\n }\n \n static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \t.advance = packed_ref_iterator_advance,\n+\t.seek = packed_ref_iterator_seek,\n \t.peel = packed_ref_iterator_peel,\n \t.release = packed_ref_iterator_release,\n };\n@@ -1097,7 +1131,6 @@ static struct ref_iterator *packed_ref_iterator_begin(\n {\n \tstruct packed_ref_store *refs;\n \tstruct snapshot *snapshot;\n-\tconst char *start;\n \tstruct packed_ref_iterator *iter;\n \tstruct ref_iterator *ref_iterator;\n \tunsigned int required_flags = REF_STORE_READ;\n@@ -1113,14 +1146,6 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \t */\n \tsnapshot = get_snapshot(refs);\n \n-\tif (prefix && *prefix)\n-\t\tstart = find_reference_location(snapshot, prefix, 0);\n-\telse\n-\t\tstart = snapshot->start;\n-\n-\tif (start == snapshot->eof)\n-\t\treturn empty_ref_iterator_begin();\n-\n \tCALLOC_ARRAY(iter, 1);\n \tref_iterator = &iter->base;\n \tbase_ref_iterator_init(ref_iterator, &packed_ref_iterator_vtable);\n@@ -1130,19 +1155,15 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \n \titer->snapshot = snapshot;\n \tacquire_snapshot(snapshot);\n-\n-\titer->pos = start;\n-\titer->eof = snapshot->eof;\n \tstrbuf_init(&iter->refname_buf, 0);\n-\n \titer->base.oid = &iter->oid;\n-\n \titer->repo = ref_store->repo;\n \titer->flags = flags;\n \n-\tif (prefix && *prefix)\n-\t\t/* Stop iteration after we've gone *past* prefix: */\n-\t\tref_iterator = prefix_ref_iterator_begin(ref_iterator, prefix, 0);\n+\tif (packed_ref_iterator_seek(&iter->base, prefix) < 0) {\n+\t\tref_iterator_free(&iter->base);\n+\t\treturn NULL;\n+\t}\n \n \treturn ref_iterator;\n }\n\n-- \n2.49.0.rc0.375.gae4b89d849.dirty\n\n"},{"id":"513257","messageId":"20250228-pks-update-ref-optimization-v4-15-6425c04268b5@pks.im","threadId":"62965","inReplyTo":"20250228-pks-update-ref-optimization-v4-0-6425c04268b5@pks.im","subject":"[PATCH v4 15/16] refs/iterator: implement seeking for files iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T09:26:52Z","receivedAt":"2025-02-28T09:27:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking for \"files\" iterators. As we simply use a ref-cache\niterator under the hood the implementation is straight-forward. Note\nthat we do not implement seeking on reflog iterators, same as with the\n\"reftable\" backend.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/files-backend.c | 16 ++++++++++++++++\n 1 file changed, 16 insertions(+)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 859f1c11941..4e1c50fead3 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -918,6 +918,14 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n+static int files_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct files_ref_iterator *iter =\n+\t\t(struct files_ref_iterator *)ref_iterator;\n+\treturn ref_iterator_seek(iter->iter0, prefix);\n+}\n+\n static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -936,6 +944,7 @@ static void files_ref_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable files_ref_iterator_vtable = {\n \t.advance = files_ref_iterator_advance,\n+\t.seek = files_ref_iterator_seek,\n \t.peel = files_ref_iterator_peel,\n \t.release = files_ref_iterator_release,\n };\n@@ -2294,6 +2303,12 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n+static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n+\t\t\t\t const char *prefix UNUSED)\n+{\n+\tBUG(\"ref_iterator_seek() called for reflog_iterator\");\n+}\n+\n static int files_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \t\t\t\t struct object_id *peeled UNUSED)\n {\n@@ -2309,6 +2324,7 @@ static void files_reflog_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable files_reflog_iterator_vtable = {\n \t.advance = files_reflog_iterator_advance,\n+\t.seek = files_reflog_iterator_seek,\n \t.peel = files_reflog_iterator_peel,\n \t.release = files_reflog_iterator_release,\n };\n\n-- \n2.49.0.rc0.375.gae4b89d849.dirty\n\n"},{"id":"513258","messageId":"20250228-pks-update-ref-optimization-v4-16-6425c04268b5@pks.im","threadId":"62965","inReplyTo":"20250228-pks-update-ref-optimization-v4-0-6425c04268b5@pks.im","subject":"[PATCH v4 16/16] refs: reuse iterators when determining refname availability","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-28T09:26:53Z","receivedAt":"2025-02-28T09:27:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When verifying whether refnames are available we have to verify whether\nany reference exists that is nested under the current reference. E.g.\ngiven a reference \"refs/heads/foo\", we must make sure that there is no\nother reference \"refs/heads/foo/*\".\n\nThis check is performed using a ref iterator with the prefix set to the\nnested reference namespace. Until now it used to not be possible to\nreseek iterators, so we always had to reallocate the iterator for every\nsingle reference we're about to check. This keeps us from reusing state\nthat the iterator may have and that may make it work more efficiently.\n\nRefactor the logic to reseek iterators. This leads to a sizeable speedup\nwith the \"reftable\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 39.8 ms ± 0.9 ms [User: 29.7 ms, System: 9.8 ms]\n Range (min … max): 38.4 ms … 42.0 ms 62 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 31.9 ms ± 1.1 ms [User: 27.0 ms, System: 4.5 ms]\n Range (min … max): 29.8 ms … 34.3 ms 74 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.25 ± 0.05 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n\nThe \"files\" backend doesn't really show a huge impact:\n\n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 392.3 ms ± 7.1 ms [User: 59.7 ms, System: 328.8 ms]\n Range (min … max): 384.6 ms … 404.5 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 387.7 ms ± 7.4 ms [User: 54.6 ms, System: 329.6 ms]\n Range (min … max): 377.0 ms … 397.7 ms 10 runs\n\n Summary\n update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.01 ± 0.03 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n\nThis is mostly because it is way slower to begin with because it has to\ncreate a separate file for each new reference, so the milliseconds we\nshave off by reseeking the iterator doesn't really translate into a\nsignificant relative improvement.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs.c | 12 +++++++-----\n 1 file changed, 7 insertions(+), 5 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 8eff60a2186..6cbb9decdb0 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2555,8 +2555,13 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \t\tif (!initial_transaction) {\n \t\t\tint ok;\n \n-\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n-\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n+\t\t\tif (!iter) {\n+\t\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n+\t\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n+\t\t\t} else if (ref_iterator_seek(iter, dirname.buf) < 0) {\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\n \t\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\t\t\tif (skip &&\n \t\t\t\t string_list_has_string(skip, iter->refname))\n@@ -2569,9 +2574,6 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \n \t\t\tif (ok != ITER_DONE)\n \t\t\t\tBUG(\"error while iterating over references\");\n-\n-\t\t\tref_iterator_free(iter);\n-\t\t\titer = NULL;\n \t\t}\n \n \t\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n\n-- \n2.49.0.rc0.375.gae4b89d849.dirty\n\n"},{"id":"513655","messageId":"CAOLa=ZR=+4rzudsXKQv0OT5NkY3KeM08PxUA=UFhERO0RpF1uA@mail.gmail.com","threadId":"62965","inReplyTo":"20250228-pks-update-ref-optimization-v4-2-6425c04268b5@pks.im","subject":"Re: [PATCH v4 02/16] object-name: allow skipping ambiguity checks in `get_oid()` family","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-03-06T13:21:47Z","receivedAt":"2025-03-06T13:21:50Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> When reading an object ID via `get_oid_basic()` or any of its related\n> functions we perform a check whether the object ID is ambiguous, which\n> can be the case when a reference with the same name exists. While the\n> check is generally helpful, there are cases where it only adds to the\n> runtime overhead without providing much of a benefit.\n>\n> Add a new flag that allows us to disable the check. The flag will be\n> used in a subsequent commit.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> hash.h | 1 +\n> object-name.c | 4 +++-\n> 2 files changed, 4 insertions(+), 1 deletion(-)\n>\n> diff --git a/hash.h b/hash.h\n> index 4367acfec50..79419016513 100644\n> --- a/hash.h\n> +++ b/hash.h\n> @@ -204,6 +204,7 @@ struct object_id {\n> #define GET_OID_ONLY_TO_DIE 04000\n> #define GET_OID_REQUIRE_PATH 010000\n> #define GET_OID_HASH_ANY 020000\n> +#define GET_OID_SKIP_AMBIGUITY_CHECK 040000\n>\n\nNit: not worth re-rolling for, but the other macros are aligned. Our\nstyling guide is to align them and this is only set in our\n'clang-format' [1].\n\n[1] : https://clang.llvm.org/docs/ClangFormatStyleOptions.html#alignconsecutiveassignments\n\n> #define GET_OID_DISAMBIGUATORS \\\n> \t(GET_OID_COMMIT | GET_OID_COMMITTISH | \\\n> diff --git a/object-name.c b/object-name.c\n> index 233f3f861e3..85444dbb15b 100644\n> --- a/object-name.c\n> +++ b/object-name.c\n> @@ -961,7 +961,9 @@ static int get_oid_basic(struct repository *r, const char *str, int len,\n> \tint fatal = !(flags & GET_OID_QUIETLY);\n>\n> \tif (len == r->hash_algo->hexsz && !get_oid_hex(str, oid)) {\n> -\t\tif (repo_settings_get_warn_ambiguous_refs(r) && warn_on_object_refname_ambiguity) {\n> +\t\tif (!(flags & GET_OID_SKIP_AMBIGUITY_CHECK) &&\n> +\t\t repo_settings_get_warn_ambiguous_refs(r) &&\n> +\t\t warn_on_object_refname_ambiguity) {\n> \t\t\trefs_found = repo_dwim_ref(r, str, len, &tmp_oid, &real_ref, 0);\n> \t\t\tif (refs_found > 0) {\n> \t\t\t\twarning(warn_msg, len, str);\n>\n> --\n> 2.49.0.rc0.375.gae4b89d849.dirty\n"},{"id":"513656","messageId":"CAOLa=ZTUb28f6Dr=cinGZN3XdwNr5bdeqZ_fruwj3YrY2tsO7A@mail.gmail.com","threadId":"62965","inReplyTo":"20250228-pks-update-ref-optimization-v4-4-6425c04268b5@pks.im","subject":"Re: [PATCH v4 04/16] refs: introduce function to batch refname availability checks","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-03-06T13:47:03Z","receivedAt":"2025-03-06T13:47:05Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n[snip]\n\n> diff --git a/refs.h b/refs.h\n> index a0cdd99250e..185aed5a461 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -124,6 +124,18 @@ int refs_verify_refname_available(struct ref_store *refs,\n> \t\t\t\t unsigned int initial_transaction,\n> \t\t\t\t struct strbuf *err);\n>\n> +/*\n> + * Same as `refs_verify_refname_available()`, but checking for a list of\n> + * refnames instead of only a single item. This is more efficient in the case\n> + * where one needs to check multiple refnames.\n> + */\n> +int refs_verify_refnames_available(struct ref_store *refs,\n> +\t\t\t\t const struct string_list *refnames,\n> +\t\t\t\t const struct string_list *extras,\n> +\t\t\t\t const struct string_list *skip,\n> +\t\t\t\t unsigned int initial_transaction,\n> +\t\t\t\t struct strbuf *err);\n> +\n> int refs_ref_exists(struct ref_store *refs, const char *refname);\n>\n> int should_autocreate_reflog(enum log_refs_config log_all_ref_updates,\n\nFYI: In my patch-series to add partial transaction support (based on top\nof this series), I move this function to 'refs-internal.h', because I\nalso pass in the transaction to it.\n\nThe patch looks good!\n"},{"id":"513657","messageId":"CAOLa=ZRUxYEOgsMOJCR1WGNFtkP2NU=PykRLCqYvQR=EnuUbUw@mail.gmail.com","threadId":"62965","inReplyTo":"20250228-pks-update-ref-optimization-v4-5-6425c04268b5@pks.im","subject":"Re: [PATCH v4 05/16] refs/reftable: batch refname availability checks","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-03-06T14:00:22Z","receivedAt":"2025-03-06T14:00:24Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> Refactor the \"reftable\" backend to batch the availability check for\n> refnames. This does not yet have an effect on performance as we\n> essentially still call `refs_verify_refname_available()` in a loop, but\n> this will change in subsequent commits.\n>\n\nI thought this patch removes it from the loop. Which loop are you\ntalking about?\n\n> Signed-off-by: Patrick Steinhardt \n> ---\n> refs/reftable-backend.c | 16 ++++++++++------\n> 1 file changed, 10 insertions(+), 6 deletions(-)\n>\n> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> index d39a14c5a46..2a90e7cb391 100644\n> --- a/refs/reftable-backend.c\n> +++ b/refs/reftable-backend.c\n> @@ -1069,6 +1069,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n> \t\treftable_be_downcast(ref_store, REF_STORE_WRITE|REF_STORE_MAIN, \"ref_transaction_prepare\");\n> \tstruct strbuf referent = STRBUF_INIT, head_referent = STRBUF_INIT;\n> \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n> +\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n> \tstruct reftable_transaction_data *tx_data = NULL;\n> \tstruct reftable_backend *be;\n> \tstruct object_id head_oid;\n> @@ -1224,12 +1225,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n> \t\t\t * can output a proper error message instead of failing\n> \t\t\t * at a later point.\n> \t\t\t */\n> -\t\t\tret = refs_verify_refname_available(ref_store, u->refname,\n> -\t\t\t\t\t\t\t &affected_refnames, NULL,\n> -\t\t\t\t\t\t\t transaction->flags & REF_TRANSACTION_FLAG_INITIAL,\n> -\t\t\t\t\t\t\t err);\n> -\t\t\tif (ret < 0)\n> -\t\t\t\tgoto done;\n> +\t\t\tstring_list_append(&refnames_to_check, u->refname);\n>\n> \t\t\t/*\n> \t\t\t * There is no need to write the reference deletion\n> @@ -1379,6 +1375,13 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n> \t\t}\n> \t}\n>\n> +\tstring_list_sort(&refnames_to_check);\n> +\tret = refs_verify_refnames_available(ref_store, &refnames_to_check, &affected_refnames, NULL,\n> +\t\t\t\t\t transaction->flags & REF_TRANSACTION_FLAG_INITIAL,\n> +\t\t\t\t\t err);\n> +\tif (ret < 0)\n> +\t\tgoto done;\n> +\n> \ttransaction->backend_data = tx_data;\n> \ttransaction->state = REF_TRANSACTION_PREPARED;\n>\n> @@ -1394,6 +1397,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n> \tstring_list_clear(&affected_refnames, 0);\n> \tstrbuf_release(&referent);\n> \tstrbuf_release(&head_referent);\n> +\tstring_list_clear(&refnames_to_check, 0);\n>\n> \treturn ret;\n> }\n>\n> --\n> 2.49.0.rc0.375.gae4b89d849.dirty\n"},{"id":"513658","messageId":"CAOLa=ZQAqVZeFhmQbDcT+j9vc0Zn=gSK3rzN-PigL_Bx0J6vPQ@mail.gmail.com","threadId":"62965","inReplyTo":"20250228-pks-update-ref-optimization-v4-7-6425c04268b5@pks.im","subject":"Re: [PATCH v4 07/16] refs/files: batch refname availability checks for initial transactions","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-03-06T14:10:13Z","receivedAt":"2025-03-06T14:10:15Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> The \"files\" backend explicitly carves out special logic for its initial\n> transaction so that it can avoid writing out every single reference as\n> a loose reference. While the assumption is that there shouldn't be any\n> preexisting references, we still have to verify that none of the newly\n> written references will conflict with any other new reference in the\n> same transaction.\n>\n> Refactor the initial transaction to use batched refname availability\n> checks. This does not yet have an effect on performance as we still call\n> `refs_verify_refname_available()` in a loop. But this will change in\n> subsequent commits and then impact performance when cloning a repository\n> with many references or when migrating references to the \"files\" format.\n>\n> This doesn't yet have an effect on performance as the underlying\n> logic simply calls This will improve performance when cloning a repository with\n\nSeems like this sentence needs to be re-written.\n\n> many references or when migrating references from any format to the\n> \"files\" format.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> refs/files-backend.c | 23 ++++++++++++++++-------\n> 1 file changed, 16 insertions(+), 7 deletions(-)\n>\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index 6ce79cf0791..11a620ea11a 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -3056,6 +3056,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n> \tsize_t i;\n> \tint ret = 0;\n> \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n> +\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n> \tstruct ref_transaction *packed_transaction = NULL;\n> \tstruct ref_transaction *loose_transaction = NULL;\n>\n> @@ -3105,11 +3106,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n> \t\t !is_null_oid(&update->old_oid))\n> \t\t\tBUG(\"initial ref transaction with old_sha1 set\");\n>\n> -\t\tif (refs_verify_refname_available(&refs->base, update->refname,\n> -\t\t\t\t\t\t &affected_refnames, NULL, 1, err)) {\n> -\t\t\tret = TRANSACTION_NAME_CONFLICT;\n> -\t\t\tgoto cleanup;\n> -\t\t}\n> +\t\tstring_list_append(&refnames_to_check, update->refname);\n>\n> \t\t/*\n> \t\t * packed-refs don't support symbolic refs, root refs and reflogs,\n> @@ -3145,8 +3142,19 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n> \t\t}\n> \t}\n>\n> -\tif (packed_refs_lock(refs->packed_ref_store, 0, err) ||\n> -\t ref_transaction_commit(packed_transaction, err)) {\n> +\tif (packed_refs_lock(refs->packed_ref_store, 0, err)) {\n> +\t\tret = TRANSACTION_GENERIC_ERROR;\n> +\t\tgoto cleanup;\n> +\t}\n> +\n> +\tif (refs_verify_refnames_available(&refs->base, &refnames_to_check,\n> +\t\t\t\t\t &affected_refnames, NULL, 1, err)) {\n> +\t\tpacked_refs_unlock(refs->packed_ref_store);\n> +\t\tret = TRANSACTION_NAME_CONFLICT;\n> +\t\tgoto cleanup;\n> +\t}\n> +\n> +\tif (ref_transaction_commit(packed_transaction, err)) {\n> \t\tret = TRANSACTION_GENERIC_ERROR;\n> \t\tgoto cleanup;\n> \t}\n> @@ -3167,6 +3175,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n> \t\tref_transaction_free(packed_transaction);\n> \ttransaction->state = REF_TRANSACTION_CLOSED;\n> \tstring_list_clear(&affected_refnames, 0);\n> +\tstring_list_clear(&refnames_to_check, 0);\n> \treturn ret;\n> }\n>\n>\n> --\n> 2.49.0.rc0.375.gae4b89d849.dirty\n"},{"id":"513659","messageId":"CAOLa=ZRg=qUV_9kZpUnx=pJ6nBhrPQjGaVHQRjwHSa2omx93fw@mail.gmail.com","threadId":"62965","inReplyTo":"CAOLa=ZRUxYEOgsMOJCR1WGNFtkP2NU=PykRLCqYvQR=EnuUbUw@mail.gmail.com","subject":"Re: [PATCH v4 05/16] refs/reftable: batch refname availability checks","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-03-06T14:12:41Z","receivedAt":"2025-03-06T14:12:45Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Karthik Nayak writes:\n\n> Patrick Steinhardt writes:\n>\n>> Refactor the \"reftable\" backend to batch the availability check for\n>> refnames. This does not yet have an effect on performance as we\n>> essentially still call `refs_verify_refname_available()` in a loop, but\n>> this will change in subsequent commits.\n>>\n>\n> I thought this patch removes it from the loop. Which loop are you\n> talking about?\n>\n\nLooking at future patches, maybe this 'loop' is a reference to how\n'refs_verify_refnames_available()' still loops over all references,\nwhich we start optimizing in patch 08 and onward?\n\n[snip]\n"},{"id":"513660","messageId":"CAOLa=ZT1ZVKGQ2KLiZY14hRH2qobaz6=V6qNXjcLkwhRvZPEnw@mail.gmail.com","threadId":"62965","inReplyTo":"20250228-pks-update-ref-optimization-v4-12-6425c04268b5@pks.im","subject":"Re: [PATCH v4 12/16] refs/iterator: implement seeking for reftable iterators","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-03-06T14:16:53Z","receivedAt":"2025-03-06T14:16:55Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> Implement seeking of reftable iterators. As the low-level reftable\n> iterators already support seeking this change is straight-forward. Two\n> notes though:\n>\n> - We do not support seeking on reflog iterators. It is unclear what\n> seeking would even look like in this context, as you typically would\n> want to seek to a specific entry in the reflog for a specific ref.\n> There is not currently a usecase for this, but if there ever is we\n> can implement seeking in the future.\n>\n\nNit: This last sentence reads a little weird, perhaps:\n\n There is currently no use case for this, but if one arises in the\n future, we can implement seeking.\n\n> - We start to check whether `reftable_stack_init_ref_iterator()` is\n> successful.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> refs/reftable-backend.c | 35 ++++++++++++++++++++++++++++++-----\n> 1 file changed, 30 insertions(+), 5 deletions(-)\n>\n\nThe patch looks good.\n\n[snip]\n"},{"id":"513661","messageId":"CAOLa=ZR-AEGuKGARzrhXSpeJFWQrDSoTzsq3EfMQzH5PDqCxkA@mail.gmail.com","threadId":"62965","inReplyTo":"20250228-pks-update-ref-optimization-v4-0-6425c04268b5@pks.im","subject":"Re: [PATCH v4 00/16] refs: batch refname availability checks","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-03-06T14:20:22Z","receivedAt":"2025-03-06T14:20:24Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> Hi,\n>\n> this patch series has been inspired by brian's report that the reftable\n> backend is significantly slower when writing many references compared to\n> the files backend. As explained in that thread, the underlying issue is\n> the design of tombstone references: when we first delete all references\n> in a repository and then recreate them, we still have all the tombstones\n> and thus we need to churn through all of them to figure out that they\n> have been deleted in the first place. The files backend does not have\n> this issue.\n>\n> I consider the benchmark itself to be kind of broken, as it stems from\n> us deleting all refs and then recreating them. And if you pack refs in\n> between then the \"reftable\" backend outperforms the \"files\" backend.\n>\n> But there are a couple of opportunities here anyway. While we cannot\n> make the underlying issue of tombstones being less efficient go away,\n> this has prompted me to have a deeper look at where we spend all the\n> time. There are three ideas in this series:\n>\n> - git-update-ref(1) performs ambiguity checks for any full-size object\n> ID, which triggers a lot of reads. This is somewhat pointless though\n> given that the manpage explicitly points out that the command is\n> about object IDs, even though it does know to parse refs. But being\n> part of plumbing, emitting the warning here does not make a ton of\n> sense, and favoring object IDs over references in these cases is the\n> obvious thing to do anyway.\n>\n> - For each ref \"refs/heads/bar\", we need to verify that neither\n> \"refs/heads\" nor \"refs\" exists. This was repeated for every refname,\n> but because most refnames use common prefixes this made us re-check\n> a lot of prefixes. This is addressed by using a `strset` of already\n> checked prefixes.\n>\n> - For each ref \"refs/heads/bar\", we need to verify that no ref\n> \"refs/heads/bar/*\" exists. We always created a new ref iterator for\n> this check, which requires us to discard all internal state and then\n> recreate it. The reftable library has already been refactored though\n> to have reseekable iterators, so we backfill this functionality to\n> all the other iterators and then reuse the iterator.\n>\n> With the (somewhat broken) benchmark we see a small speedup with the\n> \"files\" backend:\n>\n> Benchmark 1: update-ref (refformat = files, revision = master)\n> Time (mean ± σ): 234.4 ms ± 1.9 ms [User: 75.6 ms, System: 157.2 ms]\n> Range (min … max): 232.2 ms … 236.9 ms 10 runs\n>\n> Benchmark 2: update-ref (refformat = files, revision = HEAD)\n> Time (mean ± σ): 184.2 ms ± 2.0 ms [User: 62.8 ms, System: 119.9 ms]\n> Range (min … max): 181.1 ms … 187.0 ms 10 runs\n>\n> Summary\n> update-ref (refformat = files, revision = HEAD) ran\n> 1.27 ± 0.02 times faster than update-ref (refformat = files, revision = master)\n>\n> And a huge speedup with the \"reftable\" backend:\n>\n> Benchmark 1: update-ref (refformat = reftable, revision = master)\n> Time (mean ± σ): 16.852 s ± 0.061 s [User: 16.754 s, System: 0.059 s]\n> Range (min … max): 16.785 s … 16.982 s 10 runs\n>\n> Benchmark 2: update-ref (refformat = reftable, revision = HEAD)\n> Time (mean ± σ): 2.230 s ± 0.009 s [User: 2.192 s, System: 0.029 s]\n> Range (min … max): 2.215 s … 2.244 s 10 runs\n>\n> Summary\n> update-ref (refformat = reftable, revision = HEAD) ran\n> 7.56 ± 0.04 times faster than update-ref (refformat = reftable, revision = master)\n>\n> We're still not up to speed with the \"files\" backend, but considerably\n> better. Given that this is an extreme edge case and not reflective of\n> the general case I'm okay with this result for now.\n>\n> But more importantly, this refactoring also has a positive effect when\n> updating references in a repository with preexisting refs, which I\n> consider to be the more realistic scenario. The following benchmark\n> creates 10k refs with 100k preexisting refs.\n>\n> With the \"files\" backend we see a modest improvement:\n>\n> Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = master)\n> Time (mean ± σ): 478.4 ms ± 11.9 ms [User: 96.7 ms, System: 379.6 ms]\n> Range (min … max): 465.4 ms … 496.6 ms 10 runs\n>\n> Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n> Time (mean ± σ): 388.5 ms ± 10.3 ms [User: 52.0 ms, System: 333.8 ms]\n> Range (min … max): 376.5 ms … 403.1 ms 10 runs\n>\n> Summary\n> update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n> 1.23 ± 0.04 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = master)\n>\n> But with the \"reftable\" backend we see an almost 5x improvement, where\n> it's now ~15x faster than the \"files\" backend:\n>\n> Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = master)\n> Time (mean ± σ): 153.9 ms ± 2.0 ms [User: 96.5 ms, System: 56.6 ms]\n> Range (min … max): 150.5 ms … 158.4 ms 18 runs\n>\n> Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n> Time (mean ± σ): 32.2 ms ± 1.2 ms [User: 27.6 ms, System: 4.3 ms]\n> Range (min … max): 29.8 ms … 38.6 ms 71 runs\n>\n> Summary\n> update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n> 4.78 ± 0.19 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = master)\n>\n> The series is structured as follows:\n>\n> - Patches 1 to 4 implement the logic to skip ambiguity checks in\n> git-update-ref(1).\n>\n> - Patch 5 to 8 introduce batched checks.\n>\n> - Patch 9 deduplicates the ref prefix checks.\n>\n> - Patch 10 to 16 implement the infrastructure to reseek iterators.\n>\n> - Patch 17 starts to reuse iterators for nested ref checks.\n>\n> Changes in v2:\n> - Point out why we also have to touch up the `dir_iterator`.\n> - Fix up the comment explaining `ITER_DONE`.\n> - Fix up comments that show usage patterns of the ref and dir iterator\n> interfaces.\n> - Start batching availability checks in the \"files\" backend, as well.\n> - Improve the commit message that drops the ambiguity check so that we\n> also point to 25fba78d36b (cat-file: disable object/refname\n> ambiguity check for batch mode, 2013-07-12).\n> - Link to v1: https://lore.kernel.org/r/20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im\n>\n> Changes in v3:\n> - Fix one case where we didn't skip ambiguity checks in\n> git-update-ref(1).\n> - Document better that only the prefix can change on reseeking\n> iterators. Other internal state will remain the same.\n> - Fix a memory leak in the ref-cache iterator.\n> - Don't ignore errors returned by `packed_ref_iterator_seek()`.\n> - Link to v2: https://lore.kernel.org/r/20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im\n>\n> Changes in v4:\n> - A couple of clarifications in the commit message that disabled\n> ambiguity warnings.\n> - Link to v3: https://lore.kernel.org/r/20250225-pks-update-ref-optimization-v3-0-77c3687cda75@pks.im\n>\n> Thanks!\n>\n\nI did a review of the 4th version and it generally looks great. I've\nnoted some nits, but I don't see anything that warrants a re-roll.\nThanks\n\n[snip]\n"},{"id":"513675","messageId":"20250306-pks-update-ref-optimization-v5-0-dcb2ee037e97@pks.im","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im","subject":"[PATCH v5 00/16] refs: batch refname availability checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-06T15:08:31Z","receivedAt":"2025-03-06T15:08:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series has been inspired by brian's report that the reftable\nbackend is significantly slower when writing many references compared to\nthe files backend. As explained in that thread, the underlying issue is\nthe design of tombstone references: when we first delete all references\nin a repository and then recreate them, we still have all the tombstones\nand thus we need to churn through all of them to figure out that they\nhave been deleted in the first place. The files backend does not have\nthis issue.\n\nI consider the benchmark itself to be kind of broken, as it stems from\nus deleting all refs and then recreating them. And if you pack refs in\nbetween then the \"reftable\" backend outperforms the \"files\" backend.\n\nBut there are a couple of opportunities here anyway. While we cannot\nmake the underlying issue of tombstones being less efficient go away,\nthis has prompted me to have a deeper look at where we spend all the\ntime. There are three ideas in this series:\n\n - git-update-ref(1) performs ambiguity checks for any full-size object\n ID, which triggers a lot of reads. This is somewhat pointless though\n given that the manpage explicitly points out that the command is\n about object IDs, even though it does know to parse refs. But being\n part of plumbing, emitting the warning here does not make a ton of\n sense, and favoring object IDs over references in these cases is the\n obvious thing to do anyway.\n\n - For each ref \"refs/heads/bar\", we need to verify that neither\n \"refs/heads\" nor \"refs\" exists. This was repeated for every refname,\n but because most refnames use common prefixes this made us re-check\n a lot of prefixes. This is addressed by using a `strset` of already\n checked prefixes.\n\n - For each ref \"refs/heads/bar\", we need to verify that no ref\n \"refs/heads/bar/*\" exists. We always created a new ref iterator for\n this check, which requires us to discard all internal state and then\n recreate it. The reftable library has already been refactored though\n to have reseekable iterators, so we backfill this functionality to\n all the other iterators and then reuse the iterator.\n\nWith the (somewhat broken) benchmark we see a small speedup with the\n\"files\" backend:\n\n Benchmark 1: update-ref (refformat = files, revision = master)\n Time (mean ± σ): 234.4 ms ± 1.9 ms [User: 75.6 ms, System: 157.2 ms]\n Range (min … max): 232.2 ms … 236.9 ms 10 runs\n\n Benchmark 2: update-ref (refformat = files, revision = HEAD)\n Time (mean ± σ): 184.2 ms ± 2.0 ms [User: 62.8 ms, System: 119.9 ms]\n Range (min … max): 181.1 ms … 187.0 ms 10 runs\n\n Summary\n update-ref (refformat = files, revision = HEAD) ran\n 1.27 ± 0.02 times faster than update-ref (refformat = files, revision = master)\n\nAnd a huge speedup with the \"reftable\" backend:\n\n Benchmark 1: update-ref (refformat = reftable, revision = master)\n Time (mean ± σ): 16.852 s ± 0.061 s [User: 16.754 s, System: 0.059 s]\n Range (min … max): 16.785 s … 16.982 s 10 runs\n\n Benchmark 2: update-ref (refformat = reftable, revision = HEAD)\n Time (mean ± σ): 2.230 s ± 0.009 s [User: 2.192 s, System: 0.029 s]\n Range (min … max): 2.215 s … 2.244 s 10 runs\n\n Summary\n update-ref (refformat = reftable, revision = HEAD) ran\n 7.56 ± 0.04 times faster than update-ref (refformat = reftable, revision = master)\n\nWe're still not up to speed with the \"files\" backend, but considerably\nbetter. Given that this is an extreme edge case and not reflective of\nthe general case I'm okay with this result for now.\n\nBut more importantly, this refactoring also has a positive effect when\nupdating references in a repository with preexisting refs, which I\nconsider to be the more realistic scenario. The following benchmark\ncreates 10k refs with 100k preexisting refs.\n\nWith the \"files\" backend we see a modest improvement:\n\n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = master)\n Time (mean ± σ): 478.4 ms ± 11.9 ms [User: 96.7 ms, System: 379.6 ms]\n Range (min … max): 465.4 ms … 496.6 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 388.5 ms ± 10.3 ms [User: 52.0 ms, System: 333.8 ms]\n Range (min … max): 376.5 ms … 403.1 ms 10 runs\n\n Summary\n update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.23 ± 0.04 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = master)\n\nBut with the \"reftable\" backend we see an almost 5x improvement, where\nit's now ~15x faster than the \"files\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = master)\n Time (mean ± σ): 153.9 ms ± 2.0 ms [User: 96.5 ms, System: 56.6 ms]\n Range (min … max): 150.5 ms … 158.4 ms 18 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 32.2 ms ± 1.2 ms [User: 27.6 ms, System: 4.3 ms]\n Range (min … max): 29.8 ms … 38.6 ms 71 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 4.78 ± 0.19 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = master)\n\nThe series is structured as follows:\n\n - Patches 1 to 4 implement the logic to skip ambiguity checks in\n git-update-ref(1).\n\n - Patch 5 to 8 introduce batched checks.\n\n - Patch 9 deduplicates the ref prefix checks.\n\n - Patch 10 to 16 implement the infrastructure to reseek iterators.\n\n - Patch 17 starts to reuse iterators for nested ref checks.\n\nChanges in v2:\n - Point out why we also have to touch up the `dir_iterator`.\n - Fix up the comment explaining `ITER_DONE`.\n - Fix up comments that show usage patterns of the ref and dir iterator\n interfaces.\n - Start batching availability checks in the \"files\" backend, as well.\n - Improve the commit message that drops the ambiguity check so that we\n also point to 25fba78d36b (cat-file: disable object/refname\n ambiguity check for batch mode, 2013-07-12).\n - Link to v1: https://lore.kernel.org/r/20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im\n\nChanges in v3:\n - Fix one case where we didn't skip ambiguity checks in\n git-update-ref(1).\n - Document better that only the prefix can change on reseeking\n iterators. Other internal state will remain the same.\n - Fix a memory leak in the ref-cache iterator.\n - Don't ignore errors returned by `packed_ref_iterator_seek()`.\n - Link to v2: https://lore.kernel.org/r/20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im\n\nChanges in v4:\n - A couple of clarifications in the commit message that disabled\n ambiguity warnings.\n - Link to v3: https://lore.kernel.org/r/20250225-pks-update-ref-optimization-v3-0-77c3687cda75@pks.im\n\nChanges in v5:\n - Improve a couple of commit messages.\n - Align `GET_OID_*` flag values.\n - Link to v4: https://lore.kernel.org/r/20250228-pks-update-ref-optimization-v4-0-6425c04268b5@pks.im\n\nThanks!\n\nPatrick\n\n[1]: \n\n---\nPatrick Steinhardt (16):\n object-name: introduce `repo_get_oid_with_flags()`\n object-name: allow skipping ambiguity checks in `get_oid()` family\n builtin/update-ref: skip ambiguity checks when parsing object IDs\n refs: introduce function to batch refname availability checks\n refs/reftable: batch refname availability checks\n refs/files: batch refname availability checks for normal transactions\n refs/files: batch refname availability checks for initial transactions\n refs: stop re-verifying common prefixes for availability\n refs/iterator: separate lifecycle from iteration\n refs/iterator: provide infrastructure to re-seek iterators\n refs/iterator: implement seeking for merged iterators\n refs/iterator: implement seeking for reftable iterators\n refs/iterator: implement seeking for ref-cache iterators\n refs/iterator: implement seeking for packed-ref iterators\n refs/iterator: implement seeking for files iterators\n refs: reuse iterators when determining refname availability\n\n builtin/clone.c | 2 +\n builtin/update-ref.c | 15 ++--\n dir-iterator.c | 24 +++---\n dir-iterator.h | 11 +--\n hash.h | 23 +++---\n iterator.h | 2 +-\n object-name.c | 18 +++--\n object-name.h | 6 ++\n refs.c | 186 ++++++++++++++++++++++++++-----------------\n refs.h | 12 +++\n refs/debug.c | 20 +++--\n refs/files-backend.c | 117 +++++++++++++++++----------\n refs/iterator.c | 145 +++++++++++++++++----------------\n refs/packed-backend.c | 92 ++++++++++++---------\n refs/ref-cache.c | 88 ++++++++++++--------\n refs/refs-internal.h | 53 +++++++-----\n refs/reftable-backend.c | 85 +++++++++++---------\n t/helper/test-dir-iterator.c | 1 +\n 18 files changed, 539 insertions(+), 361 deletions(-)\n\nRange-diff versus v4:\n\n 1: 0d104cd63d5 = 1: 0fa8574b155 object-name: introduce `repo_get_oid_with_flags()`\n 2: 3fc310597db ! 2: 9bc87107e05 object-name: allow skipping ambiguity checks in `get_oid()` family\n @@ Commit message\n \n ## hash.h ##\n @@ hash.h: struct object_id {\n - #define GET_OID_ONLY_TO_DIE 04000\n - #define GET_OID_REQUIRE_PATH 010000\n - #define GET_OID_HASH_ANY 020000\n + \tint algo;\t/* XXX requires 4-byte alignment */\n + };\n + \n +-#define GET_OID_QUIETLY 01\n +-#define GET_OID_COMMIT 02\n +-#define GET_OID_COMMITTISH 04\n +-#define GET_OID_TREE 010\n +-#define GET_OID_TREEISH 020\n +-#define GET_OID_BLOB 040\n +-#define GET_OID_FOLLOW_SYMLINKS 0100\n +-#define GET_OID_RECORD_PATH 0200\n +-#define GET_OID_ONLY_TO_DIE 04000\n +-#define GET_OID_REQUIRE_PATH 010000\n +-#define GET_OID_HASH_ANY 020000\n ++#define GET_OID_QUIETLY 01\n ++#define GET_OID_COMMIT 02\n ++#define GET_OID_COMMITTISH 04\n ++#define GET_OID_TREE 010\n ++#define GET_OID_TREEISH 020\n ++#define GET_OID_BLOB 040\n ++#define GET_OID_FOLLOW_SYMLINKS 0100\n ++#define GET_OID_RECORD_PATH 0200\n ++#define GET_OID_ONLY_TO_DIE 04000\n ++#define GET_OID_REQUIRE_PATH 010000\n ++#define GET_OID_HASH_ANY 020000\n +#define GET_OID_SKIP_AMBIGUITY_CHECK 040000\n \n #define GET_OID_DISAMBIGUATORS \\\n 3: f702ed7ece6 = 3: 345af4b7012 builtin/update-ref: skip ambiguity checks when parsing object IDs\n 4: eead30e64d7 = 4: f1422bc1ef6 refs: introduce function to batch refname availability checks\n 5: 1bd5eb82167 ! 5: 1aa1b95b5d8 refs/reftable: batch refname availability checks\n @@ Commit message\n refs/reftable: batch refname availability checks\n \n Refactor the \"reftable\" backend to batch the availability check for\n - refnames. This does not yet have an effect on performance as we\n - essentially still call `refs_verify_refname_available()` in a loop, but\n - this will change in subsequent commits.\n + refnames. This does not yet have an effect on performance as\n + `refs_verify_refnames_available()` effectively still performs the\n + availability check for each refname individually. But this will be\n + optimized in subsequent commits, where we learn to optimize some parts\n + of the logic when checking multiple refnames for availability.\n \n Signed-off-by: Patrick Steinhardt \n \n 6: 02d02c9c219 = 6: 2cf0c6a3bbe refs/files: batch refname availability checks for normal transactions\n 7: a3c4a4f751d ! 7: 84d7830c5e5 refs/files: batch refname availability checks for initial transactions\n @@ Commit message\n subsequent commits and then impact performance when cloning a repository\n with many references or when migrating references to the \"files\" format.\n \n - This doesn't yet have an effect on performance as the underlying\n - logic simply calls This will improve performance when cloning a repository with\n - many references or when migrating references from any format to the\n - \"files\" format.\n + This will improve performance when cloning a repository with many\n + references or when migrating references from any format to the \"files\"\n + format once the availability checks have learned to optimize checks for\n + many references in a subsequent commit.\n \n Signed-off-by: Patrick Steinhardt \n \n 8: 595d98a6cd3 = 8: a0d99e8a3a0 refs: stop re-verifying common prefixes for availability\n 9: 7dd99377266 = 9: b73d1ffa761 refs/iterator: separate lifecycle from iteration\n10: 8935bed6fcb = 10: a1de5c7819c refs/iterator: provide infrastructure to re-seek iterators\n11: c9bfdde3c89 = 11: 90698a9237b refs/iterator: implement seeking for merged iterators\n12: 25fe519bff6 ! 12: 420d6e08fa9 refs/iterator: implement seeking for reftable iterators\n @@ Commit message\n - We do not support seeking on reflog iterators. It is unclear what\n seeking would even look like in this context, as you typically would\n want to seek to a specific entry in the reflog for a specific ref.\n - There is not currently a usecase for this, but if there ever is we\n - can implement seeking in the future.\n + There is currently no use case for this, but if one arises in the\n + future, we can still implement seeking at that later point.\n \n - We start to check whether `reftable_stack_init_ref_iterator()` is\n successful.\n13: 8e8d5b4b91e = 13: 0f16f1bc5f6 refs/iterator: implement seeking for ref-cache iterators\n14: 1670074af7c = 14: 01c6a5bf45d refs/iterator: implement seeking for packed-ref iterators\n15: 537ff4ae54c = 15: 4179771a9cb refs/iterator: implement seeking for files iterators\n16: 93084536803 = 16: e51b3985882 refs: reuse iterators when determining refname availability\n\n---\nbase-commit: e2067b49ecaef9b7f51a17ce251f9207f72ef52d\nchange-id: 20250217-pks-update-ref-optimization-15c795e66e2b\n\n"},{"id":"513676","messageId":"20250306-pks-update-ref-optimization-v5-1-dcb2ee037e97@pks.im","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-0-dcb2ee037e97@pks.im","subject":"[PATCH v5 01/16] object-name: introduce `repo_get_oid_with_flags()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-06T15:08:32Z","receivedAt":"2025-03-06T15:08:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new function `repo_get_oid_with_flags()`. This function\nbehaves the same as `repo_get_oid()`, except that it takes an extra\n`flags` parameter that it ends up passing to `get_oid_with_context()`.\n\nThis function will be used in a subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-name.c | 14 ++++++++------\n object-name.h | 6 ++++++\n 2 files changed, 14 insertions(+), 6 deletions(-)\n\ndiff --git a/object-name.c b/object-name.c\nindex 945d5bdef25..233f3f861e3 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -1794,18 +1794,20 @@ void object_context_release(struct object_context *ctx)\n \tstrbuf_release(&ctx->symlink_path);\n }\n \n-/*\n- * This is like \"get_oid_basic()\", except it allows \"object ID expressions\",\n- * notably \"xyz^\" for \"parent of xyz\"\n- */\n-int repo_get_oid(struct repository *r, const char *name, struct object_id *oid)\n+int repo_get_oid_with_flags(struct repository *r, const char *name,\n+\t\t\t struct object_id *oid, unsigned flags)\n {\n \tstruct object_context unused;\n-\tint ret = get_oid_with_context(r, name, 0, oid, &unused);\n+\tint ret = get_oid_with_context(r, name, flags, oid, &unused);\n \tobject_context_release(&unused);\n \treturn ret;\n }\n \n+int repo_get_oid(struct repository *r, const char *name, struct object_id *oid)\n+{\n+\treturn repo_get_oid_with_flags(r, name, oid, 0);\n+}\n+\n /*\n * This returns a non-zero value if the string (built using printf\n * format and the given arguments) is not a valid object.\ndiff --git a/object-name.h b/object-name.h\nindex 8dba4a47a47..cda4934cd5f 100644\n--- a/object-name.h\n+++ b/object-name.h\n@@ -51,6 +51,12 @@ void strbuf_repo_add_unique_abbrev(struct strbuf *sb, struct repository *repo,\n void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n \t\t\t int abbrev_len);\n \n+/*\n+ * This is like \"get_oid_basic()\", except it allows \"object ID expressions\",\n+ * notably \"xyz^\" for \"parent of xyz\". Accepts GET_OID_* flags.\n+ */\n+int repo_get_oid_with_flags(struct repository *r, const char *str,\n+\t\t\t struct object_id *oid, unsigned flags);\n int repo_get_oid(struct repository *r, const char *str, struct object_id *oid);\n __attribute__((format (printf, 2, 3)))\n int get_oidf(struct object_id *oid, const char *fmt, ...);\n\n-- \n2.49.0.rc0.416.g627208d89d.dirty\n\n"},{"id":"513677","messageId":"20250306-pks-update-ref-optimization-v5-2-dcb2ee037e97@pks.im","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-0-dcb2ee037e97@pks.im","subject":"[PATCH v5 02/16] object-name: allow skipping ambiguity checks in `get_oid()` family","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-06T15:08:33Z","receivedAt":"2025-03-06T15:08:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When reading an object ID via `get_oid_basic()` or any of its related\nfunctions we perform a check whether the object ID is ambiguous, which\ncan be the case when a reference with the same name exists. While the\ncheck is generally helpful, there are cases where it only adds to the\nruntime overhead without providing much of a benefit.\n\nAdd a new flag that allows us to disable the check. The flag will be\nused in a subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n hash.h | 23 ++++++++++++-----------\n object-name.c | 4 +++-\n 2 files changed, 15 insertions(+), 12 deletions(-)\n\ndiff --git a/hash.h b/hash.h\nindex 4367acfec50..5e3c462dc5e 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -193,17 +193,18 @@ struct object_id {\n \tint algo;\t/* XXX requires 4-byte alignment */\n };\n \n-#define GET_OID_QUIETLY 01\n-#define GET_OID_COMMIT 02\n-#define GET_OID_COMMITTISH 04\n-#define GET_OID_TREE 010\n-#define GET_OID_TREEISH 020\n-#define GET_OID_BLOB 040\n-#define GET_OID_FOLLOW_SYMLINKS 0100\n-#define GET_OID_RECORD_PATH 0200\n-#define GET_OID_ONLY_TO_DIE 04000\n-#define GET_OID_REQUIRE_PATH 010000\n-#define GET_OID_HASH_ANY 020000\n+#define GET_OID_QUIETLY 01\n+#define GET_OID_COMMIT 02\n+#define GET_OID_COMMITTISH 04\n+#define GET_OID_TREE 010\n+#define GET_OID_TREEISH 020\n+#define GET_OID_BLOB 040\n+#define GET_OID_FOLLOW_SYMLINKS 0100\n+#define GET_OID_RECORD_PATH 0200\n+#define GET_OID_ONLY_TO_DIE 04000\n+#define GET_OID_REQUIRE_PATH 010000\n+#define GET_OID_HASH_ANY 020000\n+#define GET_OID_SKIP_AMBIGUITY_CHECK 040000\n \n #define GET_OID_DISAMBIGUATORS \\\n \t(GET_OID_COMMIT | GET_OID_COMMITTISH | \\\ndiff --git a/object-name.c b/object-name.c\nindex 233f3f861e3..85444dbb15b 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -961,7 +961,9 @@ static int get_oid_basic(struct repository *r, const char *str, int len,\n \tint fatal = !(flags & GET_OID_QUIETLY);\n \n \tif (len == r->hash_algo->hexsz && !get_oid_hex(str, oid)) {\n-\t\tif (repo_settings_get_warn_ambiguous_refs(r) && warn_on_object_refname_ambiguity) {\n+\t\tif (!(flags & GET_OID_SKIP_AMBIGUITY_CHECK) &&\n+\t\t repo_settings_get_warn_ambiguous_refs(r) &&\n+\t\t warn_on_object_refname_ambiguity) {\n \t\t\trefs_found = repo_dwim_ref(r, str, len, &tmp_oid, &real_ref, 0);\n \t\t\tif (refs_found > 0) {\n \t\t\t\twarning(warn_msg, len, str);\n\n-- \n2.49.0.rc0.416.g627208d89d.dirty\n\n"},{"id":"513678","messageId":"20250306-pks-update-ref-optimization-v5-3-dcb2ee037e97@pks.im","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-0-dcb2ee037e97@pks.im","subject":"[PATCH v5 03/16] builtin/update-ref: skip ambiguity checks when parsing object IDs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-06T15:08:34Z","receivedAt":"2025-03-06T15:08:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Most of the commands in git-update-ref(1) accept an old and/or new\nobject ID to update a specific reference to. These object IDs get parsed\nvia `repo_get_oid()`, which not only handles plain object IDs, but also\nthose that have a suffix like \"~\" or \"^2\". More surprisingly though, it\neven knows to resolve arbitrary revisions, despite the fact that its\nmanpage does not mention this fact even once.\n\nOne consequence of this is that we also check for ambiguous references:\nwhen parsing a full object ID where the DWIM mechanism would also cause\nus to resolve it as a branch, we'd end up printing a warning. While this\ncheck makes sense to have in general, it is arguably less useful in the\ncontext of git-update-ref(1). This is due to multiple reasons:\n\n - The manpage is explicitly structured around object IDs. So if we see\n a fully blown object ID, the intent should be quite clear in\n general.\n\n - The command is part of our plumbing layer and not a tool that users\n would generally use in interactive workflows. As such, the warning\n will likely not be visible to anybody in the first place.\n\n - Users can and should use the fully-qualified refname in case there\n is any potential for ambiguity. And given that this command is part\n of our plumbing layer, one should always try to be as defensive as\n possible and use fully-qualified refnames.\n\nFurthermore, this check can be quite expensive when updating lots of\nreferences via `--stdin`, because we try to read multiple references per\nobject ID that we parse according to the DWIM rules. This effect can be\nseen both with the \"files\" and \"reftable\" backend.\n\nThe issue is not unique to git-update-ref(1), but was also an issue in\ngit-cat-file(1), where it was addressed by disabling the ambiguity check\nin 25fba78d36b (cat-file: disable object/refname ambiguity check for\nbatch mode, 2013-07-12).\n\nDisable the warning in git-update-ref(1), which provides a significant\nspeedup with both backends. The user-visible outcome is unchanged even\nwhen ambiguity exists, except that we don't show the warning anymore.\n\nThe following benchmark creates 10000 new references with a 100000\npreexisting refs with the \"files\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 467.3 ms ± 5.1 ms [User: 100.0 ms, System: 365.1 ms]\n Range (min … max): 461.9 ms … 479.3 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 394.1 ms ± 5.8 ms [User: 63.3 ms, System: 327.6 ms]\n Range (min … max): 384.9 ms … 405.7 ms 10 runs\n\n Summary\n update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.19 ± 0.02 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n\nAnd with the \"reftable\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 146.9 ms ± 2.2 ms [User: 90.4 ms, System: 56.0 ms]\n Range (min … max): 142.7 ms … 150.8 ms 19 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 63.2 ms ± 1.1 ms [User: 41.0 ms, System: 21.8 ms]\n Range (min … max): 61.1 ms … 66.6 ms 41 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 2.32 ± 0.05 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n\nNote that the absolute improvement with both backends is roughly in the\nsame ballpark, but the relative improvement for the \"reftable\" backend\nis more significant because writing the new table to disk is faster in\nthe first place.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/update-ref.c | 15 ++++++++++-----\n 1 file changed, 10 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/update-ref.c b/builtin/update-ref.c\nindex 4d35bdc4b4b..1d541e13ade 100644\n--- a/builtin/update-ref.c\n+++ b/builtin/update-ref.c\n@@ -179,7 +179,8 @@ static int parse_next_oid(const char **next, const char *end,\n \t\t(*next)++;\n \t\t*next = parse_arg(*next, &arg);\n \t\tif (arg.len) {\n-\t\t\tif (repo_get_oid(the_repository, arg.buf, oid))\n+\t\t\tif (repo_get_oid_with_flags(the_repository, arg.buf, oid,\n+\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\t\tgoto invalid;\n \t\t} else {\n \t\t\t/* Without -z, an empty value means all zeros: */\n@@ -197,7 +198,8 @@ static int parse_next_oid(const char **next, const char *end,\n \t\t*next += arg.len;\n \n \t\tif (arg.len) {\n-\t\t\tif (repo_get_oid(the_repository, arg.buf, oid))\n+\t\t\tif (repo_get_oid_with_flags(the_repository, arg.buf, oid,\n+\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\t\tgoto invalid;\n \t\t} else if (flags & PARSE_SHA1_ALLOW_EMPTY) {\n \t\t\t/* With -z, treat an empty value as all zeros: */\n@@ -299,7 +301,8 @@ static void parse_cmd_symref_update(struct ref_transaction *transaction,\n \t\t\tdie(\"symref-update %s: expected old value\", refname);\n \n \t\tif (!strcmp(old_arg, \"oid\")) {\n-\t\t\tif (repo_get_oid(the_repository, old_target, &old_oid))\n+\t\t\tif (repo_get_oid_with_flags(the_repository, old_target, &old_oid,\n+\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\t\tdie(\"symref-update %s: invalid oid: %s\", refname, old_target);\n \n \t\t\thave_old_oid = 1;\n@@ -772,7 +775,8 @@ int cmd_update_ref(int argc,\n \t\trefname = argv[0];\n \t\tvalue = argv[1];\n \t\toldval = argv[2];\n-\t\tif (repo_get_oid(the_repository, value, &oid))\n+\t\tif (repo_get_oid_with_flags(the_repository, value, &oid,\n+\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\tdie(\"%s: not a valid SHA1\", value);\n \t}\n \n@@ -783,7 +787,8 @@ int cmd_update_ref(int argc,\n \t\t\t * must not already exist:\n \t\t\t */\n \t\t\toidclr(&oldoid, the_repository->hash_algo);\n-\t\telse if (repo_get_oid(the_repository, oldval, &oldoid))\n+\t\telse if (repo_get_oid_with_flags(the_repository, oldval, &oldoid,\n+\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\tdie(\"%s: not a valid old SHA1\", oldval);\n \t}\n \n\n-- \n2.49.0.rc0.416.g627208d89d.dirty\n\n"},{"id":"513679","messageId":"20250306-pks-update-ref-optimization-v5-4-dcb2ee037e97@pks.im","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-0-dcb2ee037e97@pks.im","subject":"[PATCH v5 04/16] refs: introduce function to batch refname availability checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-06T15:08:35Z","receivedAt":"2025-03-06T15:08:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `refs_verify_refname_available()` functions checks whether a\nreference update can be committed or whether it would conflict with\neither a prefix or suffix thereof. This function needs to be called once\nper reference that one wants to check, which requires us to redo a\ncouple of checks every time the function is called.\n\nIntroduce a new function `refs_verify_refnames_available()` that does\nthe same, but for a list of references. For now, the new function uses\nthe exact same implementation, except that we loop through all refnames\nprovided by the caller. This will be tuned in subsequent commits.\n\nThe existing `refs_verify_refname_available()` function is reimplemented\non top of the new function. As such, the diff is best viewed with the\n`--ignore-space-change option`.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs.c | 169 +++++++++++++++++++++++++++++++++++++----------------------------\n refs.h | 12 +++++\n 2 files changed, 109 insertions(+), 72 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex f4094a326a9..5a9b0f2fa1e 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2467,19 +2467,15 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \treturn ret;\n }\n \n-int refs_verify_refname_available(struct ref_store *refs,\n-\t\t\t\t const char *refname,\n-\t\t\t\t const struct string_list *extras,\n-\t\t\t\t const struct string_list *skip,\n-\t\t\t\t unsigned int initial_transaction,\n-\t\t\t\t struct strbuf *err)\n+int refs_verify_refnames_available(struct ref_store *refs,\n+\t\t\t\t const struct string_list *refnames,\n+\t\t\t\t const struct string_list *extras,\n+\t\t\t\t const struct string_list *skip,\n+\t\t\t\t unsigned int initial_transaction,\n+\t\t\t\t struct strbuf *err)\n {\n-\tconst char *slash;\n-\tconst char *extra_refname;\n \tstruct strbuf dirname = STRBUF_INIT;\n \tstruct strbuf referent = STRBUF_INIT;\n-\tstruct object_id oid;\n-\tunsigned int type;\n \tint ret = -1;\n \n \t/*\n@@ -2489,79 +2485,91 @@ int refs_verify_refname_available(struct ref_store *refs,\n \n \tassert(err);\n \n-\tstrbuf_grow(&dirname, strlen(refname) + 1);\n-\tfor (slash = strchr(refname, '/'); slash; slash = strchr(slash + 1, '/')) {\n-\t\t/*\n-\t\t * Just saying \"Is a directory\" when we e.g. can't\n-\t\t * lock some multi-level ref isn't very informative,\n-\t\t * the user won't be told *what* is a directory, so\n-\t\t * let's not use strerror() below.\n-\t\t */\n-\t\tint ignore_errno;\n-\t\t/* Expand dirname to the new prefix, not including the trailing slash: */\n-\t\tstrbuf_add(&dirname, refname + dirname.len, slash - refname - dirname.len);\n+\tfor (size_t i = 0; i < refnames->nr; i++) {\n+\t\tconst char *refname = refnames->items[i].string;\n+\t\tconst char *extra_refname;\n+\t\tstruct object_id oid;\n+\t\tunsigned int type;\n+\t\tconst char *slash;\n+\n+\t\tstrbuf_reset(&dirname);\n+\n+\t\tfor (slash = strchr(refname, '/'); slash; slash = strchr(slash + 1, '/')) {\n+\t\t\t/*\n+\t\t\t * Just saying \"Is a directory\" when we e.g. can't\n+\t\t\t * lock some multi-level ref isn't very informative,\n+\t\t\t * the user won't be told *what* is a directory, so\n+\t\t\t * let's not use strerror() below.\n+\t\t\t */\n+\t\t\tint ignore_errno;\n+\n+\t\t\t/* Expand dirname to the new prefix, not including the trailing slash: */\n+\t\t\tstrbuf_add(&dirname, refname + dirname.len, slash - refname - dirname.len);\n+\n+\t\t\t/*\n+\t\t\t * We are still at a leading dir of the refname (e.g.,\n+\t\t\t * \"refs/foo\"; if there is a reference with that name,\n+\t\t\t * it is a conflict, *unless* it is in skip.\n+\t\t\t */\n+\t\t\tif (skip && string_list_has_string(skip, dirname.buf))\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (!initial_transaction &&\n+\t\t\t !refs_read_raw_ref(refs, dirname.buf, &oid, &referent,\n+\t\t\t\t\t &type, &ignore_errno)) {\n+\t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n+\t\t\t\t\t dirname.buf, refname);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\n+\t\t\tif (extras && string_list_has_string(extras, dirname.buf)) {\n+\t\t\t\tstrbuf_addf(err, _(\"cannot process '%s' and '%s' at the same time\"),\n+\t\t\t\t\t refname, dirname.buf);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\t\t}\n \n \t\t/*\n-\t\t * We are still at a leading dir of the refname (e.g.,\n-\t\t * \"refs/foo\"; if there is a reference with that name,\n-\t\t * it is a conflict, *unless* it is in skip.\n+\t\t * We are at the leaf of our refname (e.g., \"refs/foo/bar\").\n+\t\t * There is no point in searching for a reference with that\n+\t\t * name, because a refname isn't considered to conflict with\n+\t\t * itself. But we still need to check for references whose\n+\t\t * names are in the \"refs/foo/bar/\" namespace, because they\n+\t\t * *do* conflict.\n \t\t */\n-\t\tif (skip && string_list_has_string(skip, dirname.buf))\n-\t\t\tcontinue;\n+\t\tstrbuf_addstr(&dirname, refname + dirname.len);\n+\t\tstrbuf_addch(&dirname, '/');\n+\n+\t\tif (!initial_transaction) {\n+\t\t\tstruct ref_iterator *iter;\n+\t\t\tint ok;\n+\n+\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n+\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n+\t\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n+\t\t\t\tif (skip &&\n+\t\t\t\t string_list_has_string(skip, iter->refname))\n+\t\t\t\t\tcontinue;\n+\n+\t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n+\t\t\t\t\t iter->refname, refname);\n+\t\t\t\tref_iterator_abort(iter);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \n-\t\tif (!initial_transaction &&\n-\t\t !refs_read_raw_ref(refs, dirname.buf, &oid, &referent,\n-\t\t\t\t &type, &ignore_errno)) {\n-\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n-\t\t\t\t dirname.buf, refname);\n-\t\t\tgoto cleanup;\n+\t\t\tif (ok != ITER_DONE)\n+\t\t\t\tBUG(\"error while iterating over references\");\n \t\t}\n \n-\t\tif (extras && string_list_has_string(extras, dirname.buf)) {\n+\t\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n+\t\tif (extra_refname) {\n \t\t\tstrbuf_addf(err, _(\"cannot process '%s' and '%s' at the same time\"),\n-\t\t\t\t refname, dirname.buf);\n+\t\t\t\t refname, extra_refname);\n \t\t\tgoto cleanup;\n \t\t}\n \t}\n \n-\t/*\n-\t * We are at the leaf of our refname (e.g., \"refs/foo/bar\").\n-\t * There is no point in searching for a reference with that\n-\t * name, because a refname isn't considered to conflict with\n-\t * itself. But we still need to check for references whose\n-\t * names are in the \"refs/foo/bar/\" namespace, because they\n-\t * *do* conflict.\n-\t */\n-\tstrbuf_addstr(&dirname, refname + dirname.len);\n-\tstrbuf_addch(&dirname, '/');\n-\n-\tif (!initial_transaction) {\n-\t\tstruct ref_iterator *iter;\n-\t\tint ok;\n-\n-\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n-\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n-\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n-\t\t\tif (skip &&\n-\t\t\t string_list_has_string(skip, iter->refname))\n-\t\t\t\tcontinue;\n-\n-\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n-\t\t\t\t iter->refname, refname);\n-\t\t\tref_iterator_abort(iter);\n-\t\t\tgoto cleanup;\n-\t\t}\n-\n-\t\tif (ok != ITER_DONE)\n-\t\t\tBUG(\"error while iterating over references\");\n-\t}\n-\n-\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n-\tif (extra_refname)\n-\t\tstrbuf_addf(err, _(\"cannot process '%s' and '%s' at the same time\"),\n-\t\t\t refname, extra_refname);\n-\telse\n-\t\tret = 0;\n+\tret = 0;\n \n cleanup:\n \tstrbuf_release(&referent);\n@@ -2569,6 +2577,23 @@ int refs_verify_refname_available(struct ref_store *refs,\n \treturn ret;\n }\n \n+int refs_verify_refname_available(struct ref_store *refs,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const struct string_list *extras,\n+\t\t\t\t const struct string_list *skip,\n+\t\t\t\t unsigned int initial_transaction,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct string_list_item item = { .string = (char *) refname };\n+\tstruct string_list refnames = {\n+\t\t.items = &item,\n+\t\t.nr = 1,\n+\t};\n+\n+\treturn refs_verify_refnames_available(refs, &refnames, extras, skip,\n+\t\t\t\t\t initial_transaction, err);\n+}\n+\n struct do_for_each_reflog_help {\n \teach_reflog_fn *fn;\n \tvoid *cb_data;\ndiff --git a/refs.h b/refs.h\nindex a0cdd99250e..185aed5a461 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -124,6 +124,18 @@ int refs_verify_refname_available(struct ref_store *refs,\n \t\t\t\t unsigned int initial_transaction,\n \t\t\t\t struct strbuf *err);\n \n+/*\n+ * Same as `refs_verify_refname_available()`, but checking for a list of\n+ * refnames instead of only a single item. This is more efficient in the case\n+ * where one needs to check multiple refnames.\n+ */\n+int refs_verify_refnames_available(struct ref_store *refs,\n+\t\t\t\t const struct string_list *refnames,\n+\t\t\t\t const struct string_list *extras,\n+\t\t\t\t const struct string_list *skip,\n+\t\t\t\t unsigned int initial_transaction,\n+\t\t\t\t struct strbuf *err);\n+\n int refs_ref_exists(struct ref_store *refs, const char *refname);\n \n int should_autocreate_reflog(enum log_refs_config log_all_ref_updates,\n\n-- \n2.49.0.rc0.416.g627208d89d.dirty\n\n"},{"id":"513680","messageId":"20250306-pks-update-ref-optimization-v5-5-dcb2ee037e97@pks.im","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-0-dcb2ee037e97@pks.im","subject":"[PATCH v5 05/16] refs/reftable: batch refname availability checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-06T15:08:36Z","receivedAt":"2025-03-06T15:08:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor the \"reftable\" backend to batch the availability check for\nrefnames. This does not yet have an effect on performance as\n`refs_verify_refnames_available()` effectively still performs the\navailability check for each refname individually. But this will be\noptimized in subsequent commits, where we learn to optimize some parts\nof the logic when checking multiple refnames for availability.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 16 ++++++++++------\n 1 file changed, 10 insertions(+), 6 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex d39a14c5a46..2a90e7cb391 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1069,6 +1069,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE|REF_STORE_MAIN, \"ref_transaction_prepare\");\n \tstruct strbuf referent = STRBUF_INIT, head_referent = STRBUF_INIT;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n \tstruct reftable_transaction_data *tx_data = NULL;\n \tstruct reftable_backend *be;\n \tstruct object_id head_oid;\n@@ -1224,12 +1225,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t\t * can output a proper error message instead of failing\n \t\t\t * at a later point.\n \t\t\t */\n-\t\t\tret = refs_verify_refname_available(ref_store, u->refname,\n-\t\t\t\t\t\t\t &affected_refnames, NULL,\n-\t\t\t\t\t\t\t transaction->flags & REF_TRANSACTION_FLAG_INITIAL,\n-\t\t\t\t\t\t\t err);\n-\t\t\tif (ret < 0)\n-\t\t\t\tgoto done;\n+\t\t\tstring_list_append(&refnames_to_check, u->refname);\n \n \t\t\t/*\n \t\t\t * There is no need to write the reference deletion\n@@ -1379,6 +1375,13 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t}\n \t}\n \n+\tstring_list_sort(&refnames_to_check);\n+\tret = refs_verify_refnames_available(ref_store, &refnames_to_check, &affected_refnames, NULL,\n+\t\t\t\t\t transaction->flags & REF_TRANSACTION_FLAG_INITIAL,\n+\t\t\t\t\t err);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \ttransaction->backend_data = tx_data;\n \ttransaction->state = REF_TRANSACTION_PREPARED;\n \n@@ -1394,6 +1397,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \tstring_list_clear(&affected_refnames, 0);\n \tstrbuf_release(&referent);\n \tstrbuf_release(&head_referent);\n+\tstring_list_clear(&refnames_to_check, 0);\n \n \treturn ret;\n }\n\n-- \n2.49.0.rc0.416.g627208d89d.dirty\n\n"},{"id":"513681","messageId":"20250306-pks-update-ref-optimization-v5-6-dcb2ee037e97@pks.im","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-0-dcb2ee037e97@pks.im","subject":"[PATCH v5 06/16] refs/files: batch refname availability checks for normal transactions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-06T15:08:37Z","receivedAt":"2025-03-06T15:08:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Same as the \"reftable\" backend that we have adapted in the preceding\ncommit to use batched refname availability checks we can also do so for\nthe \"files\" backend. Things are a bit more intricate here though, as we\ncall `refs_verify_refname_available()` in a set of different contexts:\n\n 1. `lock_raw_ref()` when it hits either EEXISTS or EISDIR when creating\n a new reference, mostly to create a nice, user-readable error\n message. This is nothing we have to care about too much, as we only\n hit this code path at most once when we hit a conflict.\n\n 2. `lock_raw_ref()` when it _could_ create the lockfile to check\n whether it is conflicting with any packed refs. In the general case,\n this code path will be hit once for every (successful) reference\n update.\n\n 3. `lock_ref_oid_basic()`, but it is only executed when copying or\n renaming references or when expiring reflogs. It will thus not be\n called in contexts where we have many references queued up.\n\n 4. `refs_refname_ref_available()`, but again only when copying or\n renaming references. It is thus not interesting due to the same\n reason as the previous case.\n\n 5. `files_transaction_finish_initial()`, which is only executed when\n creating a new repository or migrating references.\n\nSo out of these, only (2) and (5) are viable candidates to use the\nbatched checks.\n\nAdapt `lock_raw_ref()` accordingly by queueing up reference names that\nneed to be checked for availability and then checking them after we have\nprocessed all updates. This check is done before we (optionally) lock\nthe `packed-refs` file, which is somewhat flawed because it means that\nthe `packed-refs` could still change after the availability check and\nthus create an undetected conflict. But unconditionally locking the file\nwould change semantics that users are likely to rely on, so we keep the\ncurrent locking sequence intact, even if it's suboptmial.\n\nThe refactoring of `files_transaction_finish_initial()` will be done in\nthe next commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/files-backend.c | 42 +++++++++++++++++++++++++++++++-----------\n 1 file changed, 31 insertions(+), 11 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 29f08dced40..6ce79cf0791 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -678,6 +678,7 @@ static void unlock_ref(struct ref_lock *lock)\n */\n static int lock_raw_ref(struct files_ref_store *refs,\n \t\t\tconst char *refname, int mustexist,\n+\t\t\tstruct string_list *refnames_to_check,\n \t\t\tconst struct string_list *extras,\n \t\t\tstruct ref_lock **lock_p,\n \t\t\tstruct strbuf *referent,\n@@ -855,16 +856,11 @@ static int lock_raw_ref(struct files_ref_store *refs,\n \t\t}\n \n \t\t/*\n-\t\t * If the ref did not exist and we are creating it,\n-\t\t * make sure there is no existing packed ref that\n-\t\t * conflicts with refname:\n+\t\t * If the ref did not exist and we are creating it, we have to\n+\t\t * make sure there is no existing packed ref that conflicts\n+\t\t * with refname. This check is deferred so that we can batch it.\n \t\t */\n-\t\tif (refs_verify_refname_available(\n-\t\t\t\t refs->packed_ref_store, refname,\n-\t\t\t\t extras, NULL, 0, err)) {\n-\t\t\tret = TRANSACTION_NAME_CONFLICT;\n-\t\t\tgoto error_return;\n-\t\t}\n+\t\tstring_list_insert(refnames_to_check, refname);\n \t}\n \n \tret = 0;\n@@ -2569,6 +2565,7 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n \t\t\t struct ref_update *update,\n \t\t\t struct ref_transaction *transaction,\n \t\t\t const char *head_ref,\n+\t\t\t struct string_list *refnames_to_check,\n \t\t\t struct string_list *affected_refnames,\n \t\t\t struct strbuf *err)\n {\n@@ -2597,7 +2594,7 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n \t\tlock->count++;\n \t} else {\n \t\tret = lock_raw_ref(refs, update->refname, mustexist,\n-\t\t\t\t affected_refnames,\n+\t\t\t\t refnames_to_check, affected_refnames,\n \t\t\t\t &lock, &referent,\n \t\t\t\t &update->type, err);\n \t\tif (ret) {\n@@ -2811,6 +2808,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \tsize_t i;\n \tint ret = 0;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n \tchar *head_ref = NULL;\n \tint head_type;\n \tstruct files_transaction_backend_data *backend_data;\n@@ -2898,7 +2896,8 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\tstruct ref_update *update = transaction->updates[i];\n \n \t\tret = lock_ref_for_update(refs, update, transaction,\n-\t\t\t\t\t head_ref, &affected_refnames, err);\n+\t\t\t\t\t head_ref, &refnames_to_check,\n+\t\t\t\t\t &affected_refnames, err);\n \t\tif (ret)\n \t\t\tgoto cleanup;\n \n@@ -2930,6 +2929,26 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\t}\n \t}\n \n+\t/*\n+\t * Verify that none of the loose reference that we're about to write\n+\t * conflict with any existing packed references. Ideally, we'd do this\n+\t * check after the packed-refs are locked so that the file cannot\n+\t * change underneath our feet. But introducing such a lock now would\n+\t * probably do more harm than good as users rely on there not being a\n+\t * global lock with the \"files\" backend.\n+\t *\n+\t * Another alternative would be to do the check after the (optional)\n+\t * lock, but that would extend the time we spend in the globally-locked\n+\t * state.\n+\t *\n+\t * So instead, we accept the race for now.\n+\t */\n+\tif (refs_verify_refnames_available(refs->packed_ref_store, &refnames_to_check,\n+\t\t\t\t\t &affected_refnames, NULL, 0, err)) {\n+\t\tret = TRANSACTION_NAME_CONFLICT;\n+\t\tgoto cleanup;\n+\t}\n+\n \tif (packed_transaction) {\n \t\tif (packed_refs_lock(refs->packed_ref_store, 0, err)) {\n \t\t\tret = TRANSACTION_GENERIC_ERROR;\n@@ -2972,6 +2991,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n cleanup:\n \tfree(head_ref);\n \tstring_list_clear(&affected_refnames, 0);\n+\tstring_list_clear(&refnames_to_check, 0);\n \n \tif (ret)\n \t\tfiles_transaction_cleanup(refs, transaction);\n\n-- \n2.49.0.rc0.416.g627208d89d.dirty\n\n"},{"id":"513682","messageId":"20250306-pks-update-ref-optimization-v5-7-dcb2ee037e97@pks.im","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-0-dcb2ee037e97@pks.im","subject":"[PATCH v5 07/16] refs/files: batch refname availability checks for initial transactions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-06T15:08:38Z","receivedAt":"2025-03-06T15:08:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"files\" backend explicitly carves out special logic for its initial\ntransaction so that it can avoid writing out every single reference as\na loose reference. While the assumption is that there shouldn't be any\npreexisting references, we still have to verify that none of the newly\nwritten references will conflict with any other new reference in the\nsame transaction.\n\nRefactor the initial transaction to use batched refname availability\nchecks. This does not yet have an effect on performance as we still call\n`refs_verify_refname_available()` in a loop. But this will change in\nsubsequent commits and then impact performance when cloning a repository\nwith many references or when migrating references to the \"files\" format.\n\nThis will improve performance when cloning a repository with many\nreferences or when migrating references from any format to the \"files\"\nformat once the availability checks have learned to optimize checks for\nmany references in a subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/files-backend.c | 23 ++++++++++++++++-------\n 1 file changed, 16 insertions(+), 7 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 6ce79cf0791..11a620ea11a 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3056,6 +3056,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \tsize_t i;\n \tint ret = 0;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n \tstruct ref_transaction *packed_transaction = NULL;\n \tstruct ref_transaction *loose_transaction = NULL;\n \n@@ -3105,11 +3106,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\t !is_null_oid(&update->old_oid))\n \t\t\tBUG(\"initial ref transaction with old_sha1 set\");\n \n-\t\tif (refs_verify_refname_available(&refs->base, update->refname,\n-\t\t\t\t\t\t &affected_refnames, NULL, 1, err)) {\n-\t\t\tret = TRANSACTION_NAME_CONFLICT;\n-\t\t\tgoto cleanup;\n-\t\t}\n+\t\tstring_list_append(&refnames_to_check, update->refname);\n \n \t\t/*\n \t\t * packed-refs don't support symbolic refs, root refs and reflogs,\n@@ -3145,8 +3142,19 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\t}\n \t}\n \n-\tif (packed_refs_lock(refs->packed_ref_store, 0, err) ||\n-\t ref_transaction_commit(packed_transaction, err)) {\n+\tif (packed_refs_lock(refs->packed_ref_store, 0, err)) {\n+\t\tret = TRANSACTION_GENERIC_ERROR;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif (refs_verify_refnames_available(&refs->base, &refnames_to_check,\n+\t\t\t\t\t &affected_refnames, NULL, 1, err)) {\n+\t\tpacked_refs_unlock(refs->packed_ref_store);\n+\t\tret = TRANSACTION_NAME_CONFLICT;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif (ref_transaction_commit(packed_transaction, err)) {\n \t\tret = TRANSACTION_GENERIC_ERROR;\n \t\tgoto cleanup;\n \t}\n@@ -3167,6 +3175,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\tref_transaction_free(packed_transaction);\n \ttransaction->state = REF_TRANSACTION_CLOSED;\n \tstring_list_clear(&affected_refnames, 0);\n+\tstring_list_clear(&refnames_to_check, 0);\n \treturn ret;\n }\n \n\n-- \n2.49.0.rc0.416.g627208d89d.dirty\n\n"},{"id":"513683","messageId":"20250306-pks-update-ref-optimization-v5-8-dcb2ee037e97@pks.im","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-0-dcb2ee037e97@pks.im","subject":"[PATCH v5 08/16] refs: stop re-verifying common prefixes for availability","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-06T15:08:39Z","receivedAt":"2025-03-06T15:08:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"One of the checks done by `refs_verify_refnames_available()` is whether\nany of the prefixes of a reference already exists. For example, given a\nreference \"refs/heads/main\", we'd check whether \"refs/heads\" or \"refs\"\nalready exist, and if so we'd abort the transaction.\n\nWhen updating multiple references at once, this check is performed for\neach of the references individually. Consequently, because references\ntend to have common prefixes like \"refs/heads/\" or refs/tags/\", we\nevaluate the availability of these prefixes repeatedly. Naturally this\nis a waste of compute, as the availability of those prefixes should in\ngeneral not change in the middle of a transaction. And if it would,\nbackends would notice at a later point in time.\n\nOptimize this pattern by storing prefixes in a `strset` so that we can\ntrivially track those prefixes that we have already checked. This leads\nto a significant speedup with the \"reftable\" backend when creating many\nreferences that all share a common prefix:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 63.1 ms ± 1.8 ms [User: 41.0 ms, System: 21.6 ms]\n Range (min … max): 60.6 ms … 69.5 ms 38 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 40.0 ms ± 1.3 ms [User: 29.3 ms, System: 10.3 ms]\n Range (min … max): 38.1 ms … 47.3 ms 61 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.58 ± 0.07 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n\nFor the \"files\" backend we see an improvement, but a much smaller one:\n\n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 395.8 ms ± 5.3 ms [User: 63.6 ms, System: 330.5 ms]\n Range (min … max): 387.0 ms … 404.6 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 386.0 ms ± 4.0 ms [User: 51.5 ms, System: 332.8 ms]\n Range (min … max): 380.8 ms … 392.6 ms 10 runs\n\n Summary\n update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.03 ± 0.02 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n\nThis change also leads to a modest improvement when writing references\nwith \"initial\" semantics, for example when migrating references. The\nfollowing benchmarks are migrating 1m references from the \"reftable\" to\nthe \"files\" backend:\n\n Benchmark 1: migrate reftable:files (refcount = 1000000, revision = HEAD~)\n Time (mean ± σ): 836.6 ms ± 5.6 ms [User: 645.2 ms, System: 185.2 ms]\n Range (min … max): 829.6 ms … 845.9 ms 10 runs\n\n Benchmark 2: migrate reftable:files (refcount = 1000000, revision = HEAD)\n Time (mean ± σ): 759.8 ms ± 5.1 ms [User: 574.9 ms, System: 178.9 ms]\n Range (min … max): 753.1 ms … 768.8 ms 10 runs\n\n Summary\n migrate reftable:files (refcount = 1000000, revision = HEAD) ran\n 1.10 ± 0.01 times faster than migrate reftable:files (refcount = 1000000, revision = HEAD~)\n\nAnd vice versa:\n\n Benchmark 1: migrate files:reftable (refcount = 1000000, revision = HEAD~)\n Time (mean ± σ): 870.7 ms ± 5.7 ms [User: 735.2 ms, System: 127.4 ms]\n Range (min … max): 861.6 ms … 883.2 ms 10 runs\n\n Benchmark 2: migrate files:reftable (refcount = 1000000, revision = HEAD)\n Time (mean ± σ): 799.1 ms ± 8.5 ms [User: 661.1 ms, System: 130.2 ms]\n Range (min … max): 787.5 ms … 812.6 ms 10 runs\n\n Summary\n migrate files:reftable (refcount = 1000000, revision = HEAD) ran\n 1.09 ± 0.01 times faster than migrate files:reftable (refcount = 1000000, revision = HEAD~)\n\nThe impact here is significantly smaller given that we don't perform any\nreference reads with \"initial\" semantics, so the speedup only comes from\nus doing less string list lookups.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs.c | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/refs.c b/refs.c\nindex 5a9b0f2fa1e..eaf41421f50 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2476,6 +2476,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n {\n \tstruct strbuf dirname = STRBUF_INIT;\n \tstruct strbuf referent = STRBUF_INIT;\n+\tstruct strset dirnames;\n \tint ret = -1;\n \n \t/*\n@@ -2485,6 +2486,8 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \n \tassert(err);\n \n+\tstrset_init(&dirnames);\n+\n \tfor (size_t i = 0; i < refnames->nr; i++) {\n \t\tconst char *refname = refnames->items[i].string;\n \t\tconst char *extra_refname;\n@@ -2514,6 +2517,14 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \t\t\tif (skip && string_list_has_string(skip, dirname.buf))\n \t\t\t\tcontinue;\n \n+\t\t\t/*\n+\t\t\t * If we've already seen the directory we don't need to\n+\t\t\t * process it again. Skip it to avoid checking checking\n+\t\t\t * common prefixes like \"refs/heads/\" repeatedly.\n+\t\t\t */\n+\t\t\tif (!strset_add(&dirnames, dirname.buf))\n+\t\t\t\tcontinue;\n+\n \t\t\tif (!initial_transaction &&\n \t\t\t !refs_read_raw_ref(refs, dirname.buf, &oid, &referent,\n \t\t\t\t\t &type, &ignore_errno)) {\n@@ -2574,6 +2585,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n cleanup:\n \tstrbuf_release(&referent);\n \tstrbuf_release(&dirname);\n+\tstrset_clear(&dirnames);\n \treturn ret;\n }\n \n\n-- \n2.49.0.rc0.416.g627208d89d.dirty\n\n"},{"id":"513684","messageId":"20250306-pks-update-ref-optimization-v5-11-dcb2ee037e97@pks.im","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-0-dcb2ee037e97@pks.im","subject":"[PATCH v5 11/16] refs/iterator: implement seeking for merged iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-06T15:08:42Z","receivedAt":"2025-03-06T15:08:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking on merged iterators. The implementation is rather\nstraight forward, with the only exception that we must not deallocate\nthe underlying iterators once they have been exhausted.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/iterator.c | 38 +++++++++++++++++++++++++++++---------\n 1 file changed, 29 insertions(+), 9 deletions(-)\n\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex 757b105261a..63608ef9907 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -96,7 +96,8 @@ int is_empty_ref_iterator(struct ref_iterator *ref_iterator)\n struct merge_ref_iterator {\n \tstruct ref_iterator base;\n \n-\tstruct ref_iterator *iter0, *iter1;\n+\tstruct ref_iterator *iter0, *iter0_owned;\n+\tstruct ref_iterator *iter1, *iter1_owned;\n \n \tref_iterator_select_fn *select;\n \tvoid *cb_data;\n@@ -160,13 +161,11 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tif (!iter->current) {\n \t\t/* Initialize: advance both iterators to their first entries */\n \t\tif ((ok = ref_iterator_advance(iter->iter0)) != ITER_OK) {\n-\t\t\tref_iterator_free(iter->iter0);\n \t\t\titer->iter0 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n \t\t}\n \t\tif ((ok = ref_iterator_advance(iter->iter1)) != ITER_OK) {\n-\t\t\tref_iterator_free(iter->iter1);\n \t\t\titer->iter1 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -177,7 +176,6 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t * entry:\n \t\t */\n \t\tif ((ok = ref_iterator_advance(*iter->current)) != ITER_OK) {\n-\t\t\tref_iterator_free(*iter->current);\n \t\t\t*iter->current = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -206,7 +204,6 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \n \t\tif (selection & ITER_SKIP_SECONDARY) {\n \t\t\tif ((ok = ref_iterator_advance(*secondary)) != ITER_OK) {\n-\t\t\t\tref_iterator_free(*secondary);\n \t\t\t\t*secondary = NULL;\n \t\t\t\tif (ok == ITER_ERROR)\n \t\t\t\t\tgoto error;\n@@ -226,6 +223,28 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ITER_ERROR;\n }\n \n+static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct merge_ref_iterator *iter =\n+\t\t(struct merge_ref_iterator *)ref_iterator;\n+\tint ret;\n+\n+\titer->current = NULL;\n+\titer->iter0 = iter->iter0_owned;\n+\titer->iter1 = iter->iter1_owned;\n+\n+\tret = ref_iterator_seek(iter->iter0, prefix);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n+\tret = ref_iterator_seek(iter->iter1, prefix);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n+\treturn 0;\n+}\n+\n static int merge_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -242,12 +261,13 @@ static void merge_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct merge_ref_iterator *iter =\n \t\t(struct merge_ref_iterator *)ref_iterator;\n-\tref_iterator_free(iter->iter0);\n-\tref_iterator_free(iter->iter1);\n+\tref_iterator_free(iter->iter0_owned);\n+\tref_iterator_free(iter->iter1_owned);\n }\n \n static struct ref_iterator_vtable merge_ref_iterator_vtable = {\n \t.advance = merge_ref_iterator_advance,\n+\t.seek = merge_ref_iterator_seek,\n \t.peel = merge_ref_iterator_peel,\n \t.release = merge_ref_iterator_release,\n };\n@@ -268,8 +288,8 @@ struct ref_iterator *merge_ref_iterator_begin(\n \t */\n \n \tbase_ref_iterator_init(ref_iterator, &merge_ref_iterator_vtable);\n-\titer->iter0 = iter0;\n-\titer->iter1 = iter1;\n+\titer->iter0 = iter->iter0_owned = iter0;\n+\titer->iter1 = iter->iter1_owned = iter1;\n \titer->select = select;\n \titer->cb_data = cb_data;\n \titer->current = NULL;\n\n-- \n2.49.0.rc0.416.g627208d89d.dirty\n\n"},{"id":"513685","messageId":"20250306-pks-update-ref-optimization-v5-9-dcb2ee037e97@pks.im","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-0-dcb2ee037e97@pks.im","subject":"[PATCH v5 09/16] refs/iterator: separate lifecycle from iteration","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-06T15:08:40Z","receivedAt":"2025-03-06T15:08:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The ref and reflog iterators have their lifecycle attached to iteration:\nonce the iterator reaches its end, it is automatically released and the\ncaller doesn't have to care about that anymore. When the iterator should\nbe released before it has been exhausted, callers must explicitly abort\nthe iterator via `ref_iterator_abort()`.\n\nThis lifecycle is somewhat unusual in the Git codebase and creates two\nproblems:\n\n - Callsites need to be very careful about when exactly they call\n `ref_iterator_abort()`, as calling the function is only valid when\n the iterator itself still is. This leads to somewhat awkward calling\n patterns in some situations.\n\n - It is impossible to reuse iterators and re-seek them to a different\n prefix. This feature isn't supported by any iterator implementation\n except for the reftable iterators anyway, but if it was implemented\n it would allow us to optimize cases where we need to search for\n specific references repeatedly by reusing internal state.\n\nDetangle the lifecycle from iteration so that we don't deallocate the\niterator anymore once it is exhausted. Instead, callers are now expected\nto always call a newly introduce `ref_iterator_free()` function that\ndeallocates the iterator and its internal state.\n\nNote that the `dir_iterator` is somewhat special because it does not\nimplement the `ref_iterator` interface, but is only used to implement\nother iterators. Consequently, we have to provide `dir_iterator_free()`\ninstead of `dir_iterator_release()` as the allocated structure itself is\nmanaged by the `dir_iterator` interfaces, as well, and not freed by\n`ref_iterator_free()` like in all the other cases.\n\nWhile at it, drop the return value of `ref_iterator_abort()`, which\nwasn't really required by any of the iterator implementations anyway.\nFurthermore, stop calling `base_ref_iterator_free()` in any of the\nbackends, but instead call it in `ref_iterator_free()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/clone.c | 2 +\n dir-iterator.c | 24 +++++------\n dir-iterator.h | 11 ++---\n iterator.h | 2 +-\n refs.c | 7 +++-\n refs/debug.c | 9 ++---\n refs/files-backend.c | 36 +++++------------\n refs/iterator.c | 95 ++++++++++++++------------------------------\n refs/packed-backend.c | 27 ++++++-------\n refs/ref-cache.c | 9 ++---\n refs/refs-internal.h | 29 +++++---------\n refs/reftable-backend.c | 34 ++++------------\n t/helper/test-dir-iterator.c | 1 +\n 13 files changed, 100 insertions(+), 186 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex fd001d800c6..ac3e84b2b18 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -426,6 +426,8 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\tstrbuf_setlen(src, src_len);\n \t\tdie(_(\"failed to iterate over '%s'\"), src->buf);\n \t}\n+\n+\tdir_iterator_free(iter);\n }\n \n static void clone_local(const char *src_repo, const char *dest_repo)\ndiff --git a/dir-iterator.c b/dir-iterator.c\nindex de619846f29..857e1d9bdaf 100644\n--- a/dir-iterator.c\n+++ b/dir-iterator.c\n@@ -193,9 +193,9 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \n \tif (S_ISDIR(iter->base.st.st_mode) && push_level(iter)) {\n \t\tif (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC)\n-\t\t\tgoto error_out;\n+\t\t\treturn ITER_ERROR;\n \t\tif (iter->levels_nr == 0)\n-\t\t\tgoto error_out;\n+\t\t\treturn ITER_ERROR;\n \t}\n \n \t/* Loop until we find an entry that we can give back to the caller. */\n@@ -211,11 +211,11 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\tint ret = next_directory_entry(level->dir, iter->base.path.buf, &de);\n \t\t\tif (ret < 0) {\n \t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n-\t\t\t\t\tgoto error_out;\n+\t\t\t\t\treturn ITER_ERROR;\n \t\t\t\tcontinue;\n \t\t\t} else if (ret > 0) {\n \t\t\t\tif (pop_level(iter) == 0)\n-\t\t\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\t\t\t\treturn ITER_DONE;\n \t\t\t\tcontinue;\n \t\t\t}\n \n@@ -223,7 +223,7 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t} else {\n \t\t\tif (level->entries_idx >= level->entries.nr) {\n \t\t\t\tif (pop_level(iter) == 0)\n-\t\t\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\t\t\t\treturn ITER_DONE;\n \t\t\t\tcontinue;\n \t\t\t}\n \n@@ -232,22 +232,21 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \n \t\tif (prepare_next_entry_data(iter, name)) {\n \t\t\tif (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC)\n-\t\t\t\tgoto error_out;\n+\t\t\t\treturn ITER_ERROR;\n \t\t\tcontinue;\n \t\t}\n \n \t\treturn ITER_OK;\n \t}\n-\n-error_out:\n-\tdir_iterator_abort(dir_iterator);\n-\treturn ITER_ERROR;\n }\n \n-int dir_iterator_abort(struct dir_iterator *dir_iterator)\n+void dir_iterator_free(struct dir_iterator *dir_iterator)\n {\n \tstruct dir_iterator_int *iter = (struct dir_iterator_int *)dir_iterator;\n \n+\tif (!iter)\n+\t\treturn;\n+\n \tfor (; iter->levels_nr; iter->levels_nr--) {\n \t\tstruct dir_iterator_level *level =\n \t\t\t&iter->levels[iter->levels_nr - 1];\n@@ -266,7 +265,6 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator)\n \tfree(iter->levels);\n \tstrbuf_release(&iter->base.path);\n \tfree(iter);\n-\treturn ITER_DONE;\n }\n \n struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags)\n@@ -301,7 +299,7 @@ struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags)\n \treturn dir_iterator;\n \n error_out:\n-\tdir_iterator_abort(dir_iterator);\n+\tdir_iterator_free(dir_iterator);\n \terrno = saved_errno;\n \treturn NULL;\n }\ndiff --git a/dir-iterator.h b/dir-iterator.h\nindex 6d438809b6e..ccd6a197343 100644\n--- a/dir-iterator.h\n+++ b/dir-iterator.h\n@@ -28,7 +28,7 @@\n *\n * while ((ok = dir_iterator_advance(iter)) == ITER_OK) {\n * if (want_to_stop_iteration()) {\n- * ok = dir_iterator_abort(iter);\n+ * ok = ITER_DONE;\n * break;\n * }\n *\n@@ -39,6 +39,7 @@\n *\n * if (ok != ITER_DONE)\n * handle_error();\n+ * dir_iterator_free(iter);\n *\n * Callers are allowed to modify iter->path while they are working,\n * but they must restore it to its original contents before calling\n@@ -107,11 +108,7 @@ struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags);\n */\n int dir_iterator_advance(struct dir_iterator *iterator);\n \n-/*\n- * End the iteration before it has been exhausted. Free the\n- * dir_iterator and any associated resources and return ITER_DONE. On\n- * error, free the dir_iterator and return ITER_ERROR.\n- */\n-int dir_iterator_abort(struct dir_iterator *iterator);\n+/* Free the dir_iterator and any associated resources. */\n+void dir_iterator_free(struct dir_iterator *iterator);\n \n #endif\ndiff --git a/iterator.h b/iterator.h\nindex 0f6900e43ad..6b77dcc2626 100644\n--- a/iterator.h\n+++ b/iterator.h\n@@ -12,7 +12,7 @@\n #define ITER_OK 0\n \n /*\n- * The iterator is exhausted and has been freed.\n+ * The iterator is exhausted.\n */\n #define ITER_DONE -1\n \ndiff --git a/refs.c b/refs.c\nindex eaf41421f50..8eff60a2186 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2476,6 +2476,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n {\n \tstruct strbuf dirname = STRBUF_INIT;\n \tstruct strbuf referent = STRBUF_INIT;\n+\tstruct ref_iterator *iter = NULL;\n \tstruct strset dirnames;\n \tint ret = -1;\n \n@@ -2552,7 +2553,6 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \t\tstrbuf_addch(&dirname, '/');\n \n \t\tif (!initial_transaction) {\n-\t\t\tstruct ref_iterator *iter;\n \t\t\tint ok;\n \n \t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n@@ -2564,12 +2564,14 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \n \t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n \t\t\t\t\t iter->refname, refname);\n-\t\t\t\tref_iterator_abort(iter);\n \t\t\t\tgoto cleanup;\n \t\t\t}\n \n \t\t\tif (ok != ITER_DONE)\n \t\t\t\tBUG(\"error while iterating over references\");\n+\n+\t\t\tref_iterator_free(iter);\n+\t\t\titer = NULL;\n \t\t}\n \n \t\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n@@ -2586,6 +2588,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \tstrbuf_release(&referent);\n \tstrbuf_release(&dirname);\n \tstrset_clear(&dirnames);\n+\tref_iterator_free(iter);\n \treturn ret;\n }\n \ndiff --git a/refs/debug.c b/refs/debug.c\nindex fbc4df08b43..a9786da4ba1 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -179,19 +179,18 @@ static int debug_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn res;\n }\n \n-static int debug_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void debug_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct debug_ref_iterator *diter =\n \t\t(struct debug_ref_iterator *)ref_iterator;\n-\tint res = diter->iter->vtable->abort(diter->iter);\n-\ttrace_printf_key(&trace_refs, \"iterator_abort: %d\\n\", res);\n-\treturn res;\n+\tditer->iter->vtable->release(diter->iter);\n+\ttrace_printf_key(&trace_refs, \"iterator_abort\\n\");\n }\n \n static struct ref_iterator_vtable debug_ref_iterator_vtable = {\n \t.advance = debug_ref_iterator_advance,\n \t.peel = debug_ref_iterator_peel,\n-\t.abort = debug_ref_iterator_abort,\n+\t.release = debug_ref_iterator_release,\n };\n \n static struct ref_iterator *\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 11a620ea11a..859f1c11941 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -915,10 +915,6 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\titer->iter0 = NULL;\n-\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\tok = ITER_ERROR;\n-\n \treturn ok;\n }\n \n@@ -931,23 +927,17 @@ static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator_peel(iter->iter0, peeled);\n }\n \n-static int files_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void files_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct files_ref_iterator *iter =\n \t\t(struct files_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->iter0)\n-\t\tok = ref_iterator_abort(iter->iter0);\n-\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n+\tref_iterator_free(iter->iter0);\n }\n \n static struct ref_iterator_vtable files_ref_iterator_vtable = {\n \t.advance = files_ref_iterator_advance,\n \t.peel = files_ref_iterator_peel,\n-\t.abort = files_ref_iterator_abort,\n+\t.release = files_ref_iterator_release,\n };\n \n static struct ref_iterator *files_ref_iterator_begin(\n@@ -1378,7 +1368,7 @@ static int should_pack_refs(struct files_ref_store *refs,\n \t\t\t\t iter->flags, opts))\n \t\t\trefcount++;\n \t\tif (refcount >= limit) {\n-\t\t\tref_iterator_abort(iter);\n+\t\t\tref_iterator_free(iter);\n \t\t\treturn 1;\n \t\t}\n \t}\n@@ -1386,6 +1376,7 @@ static int should_pack_refs(struct files_ref_store *refs,\n \tif (ret != ITER_DONE)\n \t\tdie(\"error while iterating over references\");\n \n+\tref_iterator_free(iter);\n \treturn 0;\n }\n \n@@ -1452,6 +1443,7 @@ static int files_pack_refs(struct ref_store *ref_store,\n \tpacked_refs_unlock(refs->packed_ref_store);\n \n \tprune_refs(refs, &refs_to_prune);\n+\tref_iterator_free(iter);\n \tstrbuf_release(&err);\n \treturn 0;\n }\n@@ -2299,9 +2291,6 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\titer->dir_iterator = NULL;\n-\tif (ref_iterator_abort(ref_iterator) == ITER_ERROR)\n-\t\tok = ITER_ERROR;\n \treturn ok;\n }\n \n@@ -2311,23 +2300,17 @@ static int files_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \tBUG(\"ref_iterator_peel() called for reflog_iterator\");\n }\n \n-static int files_reflog_iterator_abort(struct ref_iterator *ref_iterator)\n+static void files_reflog_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct files_reflog_iterator *iter =\n \t\t(struct files_reflog_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->dir_iterator)\n-\t\tok = dir_iterator_abort(iter->dir_iterator);\n-\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n+\tdir_iterator_free(iter->dir_iterator);\n }\n \n static struct ref_iterator_vtable files_reflog_iterator_vtable = {\n \t.advance = files_reflog_iterator_advance,\n \t.peel = files_reflog_iterator_peel,\n-\t.abort = files_reflog_iterator_abort,\n+\t.release = files_reflog_iterator_release,\n };\n \n static struct ref_iterator *reflog_iterator_begin(struct ref_store *ref_store,\n@@ -3837,6 +3820,7 @@ static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n \n out:\n+\tdir_iterator_free(iter);\n \tstrbuf_release(&sb);\n \tstrbuf_release(&refname);\n \treturn ret;\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex d25e568bf0b..aaeff270437 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -21,9 +21,14 @@ int ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator->vtable->peel(ref_iterator, peeled);\n }\n \n-int ref_iterator_abort(struct ref_iterator *ref_iterator)\n+void ref_iterator_free(struct ref_iterator *ref_iterator)\n {\n-\treturn ref_iterator->vtable->abort(ref_iterator);\n+\tif (ref_iterator) {\n+\t\tref_iterator->vtable->release(ref_iterator);\n+\t\t/* Help make use-after-free bugs fail quickly: */\n+\t\tref_iterator->vtable = NULL;\n+\t\tfree(ref_iterator);\n+\t}\n }\n \n void base_ref_iterator_init(struct ref_iterator *iter,\n@@ -36,20 +41,13 @@ void base_ref_iterator_init(struct ref_iterator *iter,\n \titer->flags = 0;\n }\n \n-void base_ref_iterator_free(struct ref_iterator *iter)\n-{\n-\t/* Help make use-after-free bugs fail quickly: */\n-\titer->vtable = NULL;\n-\tfree(iter);\n-}\n-\n struct empty_ref_iterator {\n \tstruct ref_iterator base;\n };\n \n-static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)\n {\n-\treturn ref_iterator_abort(ref_iterator);\n+\treturn ITER_DONE;\n }\n \n static int empty_ref_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n@@ -58,16 +56,14 @@ static int empty_ref_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \tBUG(\"peel called for empty iterator\");\n }\n \n-static int empty_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void empty_ref_iterator_release(struct ref_iterator *ref_iterator UNUSED)\n {\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable empty_ref_iterator_vtable = {\n \t.advance = empty_ref_iterator_advance,\n \t.peel = empty_ref_iterator_peel,\n-\t.abort = empty_ref_iterator_abort,\n+\t.release = empty_ref_iterator_release,\n };\n \n struct ref_iterator *empty_ref_iterator_begin(void)\n@@ -151,11 +147,13 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tif (!iter->current) {\n \t\t/* Initialize: advance both iterators to their first entries */\n \t\tif ((ok = ref_iterator_advance(iter->iter0)) != ITER_OK) {\n+\t\t\tref_iterator_free(iter->iter0);\n \t\t\titer->iter0 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n \t\t}\n \t\tif ((ok = ref_iterator_advance(iter->iter1)) != ITER_OK) {\n+\t\t\tref_iterator_free(iter->iter1);\n \t\t\titer->iter1 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -166,6 +164,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t * entry:\n \t\t */\n \t\tif ((ok = ref_iterator_advance(*iter->current)) != ITER_OK) {\n+\t\t\tref_iterator_free(*iter->current);\n \t\t\t*iter->current = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -179,9 +178,8 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t\titer->select(iter->iter0, iter->iter1, iter->cb_data);\n \n \t\tif (selection == ITER_SELECT_DONE) {\n-\t\t\treturn ref_iterator_abort(ref_iterator);\n+\t\t\treturn ITER_DONE;\n \t\t} else if (selection == ITER_SELECT_ERROR) {\n-\t\t\tref_iterator_abort(ref_iterator);\n \t\t\treturn ITER_ERROR;\n \t\t}\n \n@@ -195,6 +193,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \n \t\tif (selection & ITER_SKIP_SECONDARY) {\n \t\t\tif ((ok = ref_iterator_advance(*secondary)) != ITER_OK) {\n+\t\t\t\tref_iterator_free(*secondary);\n \t\t\t\t*secondary = NULL;\n \t\t\t\tif (ok == ITER_ERROR)\n \t\t\t\t\tgoto error;\n@@ -211,7 +210,6 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t}\n \n error:\n-\tref_iterator_abort(ref_iterator);\n \treturn ITER_ERROR;\n }\n \n@@ -227,28 +225,18 @@ static int merge_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator_peel(*iter->current, peeled);\n }\n \n-static int merge_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void merge_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct merge_ref_iterator *iter =\n \t\t(struct merge_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->iter0) {\n-\t\tif (ref_iterator_abort(iter->iter0) != ITER_DONE)\n-\t\t\tok = ITER_ERROR;\n-\t}\n-\tif (iter->iter1) {\n-\t\tif (ref_iterator_abort(iter->iter1) != ITER_DONE)\n-\t\t\tok = ITER_ERROR;\n-\t}\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n+\tref_iterator_free(iter->iter0);\n+\tref_iterator_free(iter->iter1);\n }\n \n static struct ref_iterator_vtable merge_ref_iterator_vtable = {\n \t.advance = merge_ref_iterator_advance,\n \t.peel = merge_ref_iterator_peel,\n-\t.abort = merge_ref_iterator_abort,\n+\t.release = merge_ref_iterator_release,\n };\n \n struct ref_iterator *merge_ref_iterator_begin(\n@@ -310,10 +298,10 @@ struct ref_iterator *overlay_ref_iterator_begin(\n \t * them.\n \t */\n \tif (is_empty_ref_iterator(front)) {\n-\t\tref_iterator_abort(front);\n+\t\tref_iterator_free(front);\n \t\treturn back;\n \t} else if (is_empty_ref_iterator(back)) {\n-\t\tref_iterator_abort(back);\n+\t\tref_iterator_free(back);\n \t\treturn front;\n \t}\n \n@@ -350,19 +338,10 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \n \twhile ((ok = ref_iterator_advance(iter->iter0)) == ITER_OK) {\n \t\tint cmp = compare_prefix(iter->iter0->refname, iter->prefix);\n-\n \t\tif (cmp < 0)\n \t\t\tcontinue;\n-\n-\t\tif (cmp > 0) {\n-\t\t\t/*\n-\t\t\t * As the source iterator is ordered, we\n-\t\t\t * can stop the iteration as soon as we see a\n-\t\t\t * refname that comes after the prefix:\n-\t\t\t */\n-\t\t\tok = ref_iterator_abort(iter->iter0);\n-\t\t\tbreak;\n-\t\t}\n+\t\tif (cmp > 0)\n+\t\t\treturn ITER_DONE;\n \n \t\tif (iter->trim) {\n \t\t\t/*\n@@ -386,9 +365,6 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\titer->iter0 = NULL;\n-\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\treturn ITER_ERROR;\n \treturn ok;\n }\n \n@@ -401,23 +377,18 @@ static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator_peel(iter->iter0, peeled);\n }\n \n-static int prefix_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void prefix_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct prefix_ref_iterator *iter =\n \t\t(struct prefix_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->iter0)\n-\t\tok = ref_iterator_abort(iter->iter0);\n+\tref_iterator_free(iter->iter0);\n \tfree(iter->prefix);\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n }\n \n static struct ref_iterator_vtable prefix_ref_iterator_vtable = {\n \t.advance = prefix_ref_iterator_advance,\n \t.peel = prefix_ref_iterator_peel,\n-\t.abort = prefix_ref_iterator_abort,\n+\t.release = prefix_ref_iterator_release,\n };\n \n struct ref_iterator *prefix_ref_iterator_begin(struct ref_iterator *iter0,\n@@ -453,20 +424,14 @@ int do_for_each_ref_iterator(struct ref_iterator *iter,\n \tcurrent_ref_iter = iter;\n \twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\tretval = fn(iter->refname, iter->referent, iter->oid, iter->flags, cb_data);\n-\t\tif (retval) {\n-\t\t\t/*\n-\t\t\t * If ref_iterator_abort() returns ITER_ERROR,\n-\t\t\t * we ignore that error in deference to the\n-\t\t\t * callback function's return value.\n-\t\t\t */\n-\t\t\tref_iterator_abort(iter);\n+\t\tif (retval)\n \t\t\tgoto out;\n-\t\t}\n \t}\n \n out:\n \tcurrent_ref_iter = old_ref_iter;\n \tif (ok == ITER_ERROR)\n-\t\treturn -1;\n+\t\tretval = -1;\n+\tref_iterator_free(iter);\n \treturn retval;\n }\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex a7b6f74b6e3..38a1956d1a8 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -954,9 +954,6 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\tok = ITER_ERROR;\n-\n \treturn ok;\n }\n \n@@ -976,23 +973,19 @@ static int packed_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t}\n }\n \n-static int packed_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void packed_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct packed_ref_iterator *iter =\n \t\t(struct packed_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n \tstrbuf_release(&iter->refname_buf);\n \tfree(iter->jump);\n \trelease_snapshot(iter->snapshot);\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n }\n \n static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \t.advance = packed_ref_iterator_advance,\n \t.peel = packed_ref_iterator_peel,\n-\t.abort = packed_ref_iterator_abort\n+\t.release = packed_ref_iterator_release,\n };\n \n static int jump_list_entry_cmp(const void *va, const void *vb)\n@@ -1362,8 +1355,10 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t */\n \titer = packed_ref_iterator_begin(&refs->base, \"\", NULL,\n \t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n-\tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n+\tif ((ok = ref_iterator_advance(iter)) != ITER_OK) {\n+\t\tref_iterator_free(iter);\n \t\titer = NULL;\n+\t}\n \n \ti = 0;\n \n@@ -1411,8 +1406,10 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t\t\t\t * the iterator over the unneeded\n \t\t\t\t * value.\n \t\t\t\t */\n-\t\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n+\t\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK) {\n+\t\t\t\t\tref_iterator_free(iter);\n \t\t\t\t\titer = NULL;\n+\t\t\t\t}\n \t\t\t\tcmp = +1;\n \t\t\t} else {\n \t\t\t\t/*\n@@ -1449,8 +1446,10 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t\t\t\t\t peel_error ? NULL : &peeled))\n \t\t\t\tgoto write_error;\n \n-\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n+\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK) {\n+\t\t\t\tref_iterator_free(iter);\n \t\t\t\titer = NULL;\n+\t\t\t}\n \t\t} else if (is_null_oid(&update->new_oid)) {\n \t\t\t/*\n \t\t\t * The update wants to delete the reference,\n@@ -1499,9 +1498,7 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t\t get_tempfile_path(refs->tempfile), strerror(errno));\n \n error:\n-\tif (iter)\n-\t\tref_iterator_abort(iter);\n-\n+\tref_iterator_free(iter);\n \tdelete_tempfile(&refs->tempfile);\n \treturn -1;\n }\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 02f09e4df88..6457e02c1ea 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -409,7 +409,7 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\tif (++level->index == level->dir->nr) {\n \t\t\t/* This level is exhausted; pop up a level */\n \t\t\tif (--iter->levels_nr == 0)\n-\t\t\t\treturn ref_iterator_abort(ref_iterator);\n+\t\t\t\treturn ITER_DONE;\n \n \t\t\tcontinue;\n \t\t}\n@@ -452,21 +452,18 @@ static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn peel_object(iter->repo, ref_iterator->oid, peeled) ? -1 : 0;\n }\n \n-static int cache_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void cache_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct cache_ref_iterator *iter =\n \t\t(struct cache_ref_iterator *)ref_iterator;\n-\n \tfree((char *)iter->prefix);\n \tfree(iter->levels);\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable cache_ref_iterator_vtable = {\n \t.advance = cache_ref_iterator_advance,\n \t.peel = cache_ref_iterator_peel,\n-\t.abort = cache_ref_iterator_abort\n+\t.release = cache_ref_iterator_release,\n };\n \n struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex aaab711bb96..74e2c03cef1 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -273,11 +273,11 @@ enum do_for_each_ref_flags {\n * the next reference and returns ITER_OK. The data pointed at by\n * refname and oid belong to the iterator; if you want to retain them\n * after calling ref_iterator_advance() again or calling\n- * ref_iterator_abort(), you must make a copy. When the iteration has\n+ * ref_iterator_free(), you must make a copy. When the iteration has\n * been exhausted, ref_iterator_advance() releases any resources\n * associated with the iteration, frees the ref_iterator object, and\n * returns ITER_DONE. If you want to abort the iteration early, call\n- * ref_iterator_abort(), which also frees the ref_iterator object and\n+ * ref_iterator_free(), which also frees the ref_iterator object and\n * any associated resources. If there was an internal error advancing\n * to the next entry, ref_iterator_advance() aborts the iteration,\n * frees the ref_iterator, and returns ITER_ERROR.\n@@ -293,7 +293,7 @@ enum do_for_each_ref_flags {\n *\n * while ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n * if (want_to_stop_iteration()) {\n- * ok = ref_iterator_abort(iter);\n+ * ok = ITER_DONE;\n * break;\n * }\n *\n@@ -307,6 +307,7 @@ enum do_for_each_ref_flags {\n *\n * if (ok != ITER_DONE)\n * handle_error();\n+ * ref_iterator_free(iter);\n */\n struct ref_iterator {\n \tstruct ref_iterator_vtable *vtable;\n@@ -333,12 +334,8 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator);\n int ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t struct object_id *peeled);\n \n-/*\n- * End the iteration before it has been exhausted, freeing the\n- * reference iterator and any associated resources and returning\n- * ITER_DONE. If the abort itself failed, return ITER_ERROR.\n- */\n-int ref_iterator_abort(struct ref_iterator *ref_iterator);\n+/* Free the reference iterator and any associated resources. */\n+void ref_iterator_free(struct ref_iterator *ref_iterator);\n \n /*\n * An iterator over nothing (its first ref_iterator_advance() call\n@@ -438,13 +435,6 @@ struct ref_iterator *prefix_ref_iterator_begin(struct ref_iterator *iter0,\n void base_ref_iterator_init(struct ref_iterator *iter,\n \t\t\t struct ref_iterator_vtable *vtable);\n \n-/*\n- * Base class destructor for ref_iterators. Destroy the ref_iterator\n- * part of iter and shallow-free the object. This is meant to be\n- * called only by the destructors of derived classes.\n- */\n-void base_ref_iterator_free(struct ref_iterator *iter);\n-\n /* Virtual function declarations for ref_iterators: */\n \n /*\n@@ -463,15 +453,14 @@ typedef int ref_iterator_peel_fn(struct ref_iterator *ref_iterator,\n \n /*\n * Implementations of this function should free any resources specific\n- * to the derived class, then call base_ref_iterator_free() to clean\n- * up and free the ref_iterator object.\n+ * to the derived class.\n */\n-typedef int ref_iterator_abort_fn(struct ref_iterator *ref_iterator);\n+typedef void ref_iterator_release_fn(struct ref_iterator *ref_iterator);\n \n struct ref_iterator_vtable {\n \tref_iterator_advance_fn *advance;\n \tref_iterator_peel_fn *peel;\n-\tref_iterator_abort_fn *abort;\n+\tref_iterator_release_fn *release;\n };\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 2a90e7cb391..06543f79c64 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -711,17 +711,10 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\tbreak;\n \t}\n \n-\tif (iter->err > 0) {\n-\t\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\t\treturn ITER_ERROR;\n+\tif (iter->err > 0)\n \t\treturn ITER_DONE;\n-\t}\n-\n-\tif (iter->err < 0) {\n-\t\tref_iterator_abort(ref_iterator);\n+\tif (iter->err < 0)\n \t\treturn ITER_ERROR;\n-\t}\n-\n \treturn ITER_OK;\n }\n \n@@ -740,7 +733,7 @@ static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn -1;\n }\n \n-static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void reftable_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct reftable_ref_iterator *iter =\n \t\t(struct reftable_ref_iterator *)ref_iterator;\n@@ -751,14 +744,12 @@ static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n \t\t\tfree(iter->exclude_patterns[i]);\n \t\tfree(iter->exclude_patterns);\n \t}\n-\tfree(iter);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n \t.advance = reftable_ref_iterator_advance,\n \t.peel = reftable_ref_iterator_peel,\n-\t.abort = reftable_ref_iterator_abort\n+\t.release = reftable_ref_iterator_release,\n };\n \n static int qsort_strcmp(const void *va, const void *vb)\n@@ -2017,17 +2008,10 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \t\tbreak;\n \t}\n \n-\tif (iter->err > 0) {\n-\t\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\t\treturn ITER_ERROR;\n+\tif (iter->err > 0)\n \t\treturn ITER_DONE;\n-\t}\n-\n-\tif (iter->err < 0) {\n-\t\tref_iterator_abort(ref_iterator);\n+\tif (iter->err < 0)\n \t\treturn ITER_ERROR;\n-\t}\n-\n \treturn ITER_OK;\n }\n \n@@ -2038,21 +2022,19 @@ static int reftable_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSE\n \treturn -1;\n }\n \n-static int reftable_reflog_iterator_abort(struct ref_iterator *ref_iterator)\n+static void reftable_reflog_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct reftable_reflog_iterator *iter =\n \t\t(struct reftable_reflog_iterator *)ref_iterator;\n \treftable_log_record_release(&iter->log);\n \treftable_iterator_destroy(&iter->iter);\n \tstrbuf_release(&iter->last_name);\n-\tfree(iter);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable reftable_reflog_iterator_vtable = {\n \t.advance = reftable_reflog_iterator_advance,\n \t.peel = reftable_reflog_iterator_peel,\n-\t.abort = reftable_reflog_iterator_abort\n+\t.release = reftable_reflog_iterator_release,\n };\n \n static struct reftable_reflog_iterator *reflog_iterator_for_stack(struct reftable_ref_store *refs,\ndiff --git a/t/helper/test-dir-iterator.c b/t/helper/test-dir-iterator.c\nindex 6b297bd7536..8d46e8ba409 100644\n--- a/t/helper/test-dir-iterator.c\n+++ b/t/helper/test-dir-iterator.c\n@@ -53,6 +53,7 @@ int cmd__dir_iterator(int argc, const char **argv)\n \t\tprintf(\"(%s) [%s] %s\\n\", diter->relative_path, diter->basename,\n \t\t diter->path.buf);\n \t}\n+\tdir_iterator_free(diter);\n \n \tif (iter_status != ITER_DONE) {\n \t\tprintf(\"dir_iterator_advance failure\\n\");\n\n-- \n2.49.0.rc0.416.g627208d89d.dirty\n\n"},{"id":"513686","messageId":"20250306-pks-update-ref-optimization-v5-10-dcb2ee037e97@pks.im","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-0-dcb2ee037e97@pks.im","subject":"[PATCH v5 10/16] refs/iterator: provide infrastructure to re-seek iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-06T15:08:41Z","receivedAt":"2025-03-06T15:08:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Reftable iterators need to be scrapped after they have either been\nexhausted or aren't useful to the caller anymore, and it is explicitly\nnot possible to reuse them for iterations. But enabling for reuse of\niterators may allow us to tune them by reusing internal state of an\niterator. The reftable iterators for example can already be reused\ninternally, but we're not able to expose this to any users outside of\nthe reftable backend.\n\nIntroduce a new `.seek` function in the ref iterator vtable that allows\ncallers to seek an iterator multiple times. It is expected to be\nfunctionally the same as calling `refs_ref_iterator_begin()` with a\ndifferent (or the same) prefix.\n\nNote that it is not possible to adjust parameters other than the seeked\nprefix for now, so exclude patterns, trimmed prefixes and flags will\nremain unchanged. We do not have a usecase for changing these parameters\nright now, but if we ever find one we can adapt accordingly.\n\nImplement the callback for trivial cases. The other iterators will be\nimplemented in subsequent commits.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/debug.c | 11 +++++++++++\n refs/iterator.c | 24 ++++++++++++++++++++++++\n refs/refs-internal.h | 24 ++++++++++++++++++++++++\n 3 files changed, 59 insertions(+)\n\ndiff --git a/refs/debug.c b/refs/debug.c\nindex a9786da4ba1..5390fa9c187 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -169,6 +169,16 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn res;\n }\n \n+static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct debug_ref_iterator *diter =\n+\t\t(struct debug_ref_iterator *)ref_iterator;\n+\tint res = diter->iter->vtable->seek(diter->iter, prefix);\n+\ttrace_printf_key(&trace_refs, \"iterator_seek: %s: %d\\n\", prefix ? prefix : \"\", res);\n+\treturn res;\n+}\n+\n static int debug_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -189,6 +199,7 @@ static void debug_ref_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable debug_ref_iterator_vtable = {\n \t.advance = debug_ref_iterator_advance,\n+\t.seek = debug_ref_iterator_seek,\n \t.peel = debug_ref_iterator_peel,\n \t.release = debug_ref_iterator_release,\n };\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex aaeff270437..757b105261a 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -15,6 +15,12 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ref_iterator->vtable->advance(ref_iterator);\n }\n \n+int ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t const char *prefix)\n+{\n+\treturn ref_iterator->vtable->seek(ref_iterator, prefix);\n+}\n+\n int ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t struct object_id *peeled)\n {\n@@ -50,6 +56,12 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)\n \treturn ITER_DONE;\n }\n \n+static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n+\t\t\t\t const char *prefix UNUSED)\n+{\n+\treturn 0;\n+}\n+\n static int empty_ref_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \t\t\t\t struct object_id *peeled UNUSED)\n {\n@@ -62,6 +74,7 @@ static void empty_ref_iterator_release(struct ref_iterator *ref_iterator UNUSED)\n \n static struct ref_iterator_vtable empty_ref_iterator_vtable = {\n \t.advance = empty_ref_iterator_advance,\n+\t.seek = empty_ref_iterator_seek,\n \t.peel = empty_ref_iterator_peel,\n \t.release = empty_ref_iterator_release,\n };\n@@ -368,6 +381,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n+static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct prefix_ref_iterator *iter =\n+\t\t(struct prefix_ref_iterator *)ref_iterator;\n+\tfree(iter->prefix);\n+\titer->prefix = xstrdup_or_null(prefix);\n+\treturn ref_iterator_seek(iter->iter0, prefix);\n+}\n+\n static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -387,6 +410,7 @@ static void prefix_ref_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable prefix_ref_iterator_vtable = {\n \t.advance = prefix_ref_iterator_advance,\n+\t.seek = prefix_ref_iterator_seek,\n \t.peel = prefix_ref_iterator_peel,\n \t.release = prefix_ref_iterator_release,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 74e2c03cef1..8f18274a165 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -327,6 +327,22 @@ struct ref_iterator {\n */\n int ref_iterator_advance(struct ref_iterator *ref_iterator);\n \n+/*\n+ * Seek the iterator to the first reference with the given prefix.\n+ * The prefix is matched as a literal string, without regard for path\n+ * separators. If prefix is NULL or the empty string, seek the iterator to the\n+ * first reference again.\n+ *\n+ * This function is expected to behave as if a new ref iterator with the same\n+ * prefix had been created, but allows reuse of iterators and thus may allow\n+ * the backend to optimize. Parameters other than the prefix that have been\n+ * passed when creating the iterator will remain unchanged.\n+ *\n+ * Returns 0 on success, a negative error code otherwise.\n+ */\n+int ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t const char *prefix);\n+\n /*\n * If possible, peel the reference currently being viewed by the\n * iterator. Return 0 on success.\n@@ -445,6 +461,13 @@ void base_ref_iterator_init(struct ref_iterator *iter,\n */\n typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);\n \n+/*\n+ * Seek the iterator to the first reference matching the given prefix. Should\n+ * behave the same as if a new iterator was created with the same prefix.\n+ */\n+typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix);\n+\n /*\n * Peels the current ref, returning 0 for success or -1 for failure.\n */\n@@ -459,6 +482,7 @@ typedef void ref_iterator_release_fn(struct ref_iterator *ref_iterator);\n \n struct ref_iterator_vtable {\n \tref_iterator_advance_fn *advance;\n+\tref_iterator_seek_fn *seek;\n \tref_iterator_peel_fn *peel;\n \tref_iterator_release_fn *release;\n };\n\n-- \n2.49.0.rc0.416.g627208d89d.dirty\n\n"},{"id":"513687","messageId":"20250306-pks-update-ref-optimization-v5-12-dcb2ee037e97@pks.im","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-0-dcb2ee037e97@pks.im","subject":"[PATCH v5 12/16] refs/iterator: implement seeking for reftable iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-06T15:08:43Z","receivedAt":"2025-03-06T15:08:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking of reftable iterators. As the low-level reftable\niterators already support seeking this change is straight-forward. Two\nnotes though:\n\n - We do not support seeking on reflog iterators. It is unclear what\n seeking would even look like in this context, as you typically would\n want to seek to a specific entry in the reflog for a specific ref.\n There is currently no use case for this, but if one arises in the\n future, we can still implement seeking at that later point.\n\n - We start to check whether `reftable_stack_init_ref_iterator()` is\n successful.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 35 ++++++++++++++++++++++++++++++-----\n 1 file changed, 30 insertions(+), 5 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 06543f79c64..b0c09f34433 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -547,7 +547,7 @@ struct reftable_ref_iterator {\n \tstruct reftable_ref_record ref;\n \tstruct object_id oid;\n \n-\tconst char *prefix;\n+\tchar *prefix;\n \tsize_t prefix_len;\n \tchar **exclude_patterns;\n \tsize_t exclude_patterns_index;\n@@ -718,6 +718,20 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ITER_OK;\n }\n \n+static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct reftable_ref_iterator *iter =\n+\t\t(struct reftable_ref_iterator *)ref_iterator;\n+\n+\tfree(iter->prefix);\n+\titer->prefix = xstrdup_or_null(prefix);\n+\titer->prefix_len = prefix ? strlen(prefix) : 0;\n+\titer->err = reftable_iterator_seek_ref(&iter->iter, prefix);\n+\n+\treturn iter->err;\n+}\n+\n static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -744,10 +758,12 @@ static void reftable_ref_iterator_release(struct ref_iterator *ref_iterator)\n \t\t\tfree(iter->exclude_patterns[i]);\n \t\tfree(iter->exclude_patterns);\n \t}\n+\tfree(iter->prefix);\n }\n \n static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n \t.advance = reftable_ref_iterator_advance,\n+\t.seek = reftable_ref_iterator_seek,\n \t.peel = reftable_ref_iterator_peel,\n \t.release = reftable_ref_iterator_release,\n };\n@@ -806,8 +822,6 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_\n \n \titer = xcalloc(1, sizeof(*iter));\n \tbase_ref_iterator_init(&iter->base, &reftable_ref_iterator_vtable);\n-\titer->prefix = prefix;\n-\titer->prefix_len = prefix ? strlen(prefix) : 0;\n \titer->base.oid = &iter->oid;\n \titer->flags = flags;\n \titer->refs = refs;\n@@ -821,8 +835,11 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_\n \tif (ret)\n \t\tgoto done;\n \n-\treftable_stack_init_ref_iterator(stack, &iter->iter);\n-\tret = reftable_iterator_seek_ref(&iter->iter, prefix);\n+\tret = reftable_stack_init_ref_iterator(stack, &iter->iter);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_ref_iterator_seek(&iter->base, prefix);\n \tif (ret)\n \t\tgoto done;\n \n@@ -2015,6 +2032,13 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ITER_OK;\n }\n \n+static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n+\t\t\t\t\t const char *prefix UNUSED)\n+{\n+\tBUG(\"reftable reflog iterator cannot be seeked\");\n+\treturn -1;\n+}\n+\n static int reftable_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \t\t\t\t\t struct object_id *peeled UNUSED)\n {\n@@ -2033,6 +2057,7 @@ static void reftable_reflog_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable reftable_reflog_iterator_vtable = {\n \t.advance = reftable_reflog_iterator_advance,\n+\t.seek = reftable_reflog_iterator_seek,\n \t.peel = reftable_reflog_iterator_peel,\n \t.release = reftable_reflog_iterator_release,\n };\n\n-- \n2.49.0.rc0.416.g627208d89d.dirty\n\n"},{"id":"513689","messageId":"20250306-pks-update-ref-optimization-v5-13-dcb2ee037e97@pks.im","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-0-dcb2ee037e97@pks.im","subject":"[PATCH v5 13/16] refs/iterator: implement seeking for ref-cache iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-06T15:08:44Z","receivedAt":"2025-03-06T15:08:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking of ref-cache iterators. This is done by splitting most\nof the logic to seek iterators out of `cache_ref_iterator_begin()` and\nputting it into `cache_ref_iterator_seek()` so that we can reuse the\nlogic.\n\nNote that we cannot use the optimization anymore where we return an\nempty ref iterator when there aren't any references, as otherwise it\nwouldn't be possible to reseek the iterator to a different prefix that\nmay exist. This shouldn't be much of a performance concern though as we\nnow start to bail out early in case `advance()` sees that there are no\nmore directories to be searched.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/ref-cache.c | 79 ++++++++++++++++++++++++++++++++++++--------------------\n 1 file changed, 51 insertions(+), 28 deletions(-)\n\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 6457e02c1ea..c1f1bab1d50 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -362,9 +362,7 @@ struct cache_ref_iterator {\n \tstruct ref_iterator base;\n \n \t/*\n-\t * The number of levels currently on the stack. This is always\n-\t * at least 1, because when it becomes zero the iteration is\n-\t * ended and this struct is freed.\n+\t * The number of levels currently on the stack.\n \t */\n \tsize_t levels_nr;\n \n@@ -376,7 +374,7 @@ struct cache_ref_iterator {\n \t * The prefix is matched textually, without regard for path\n \t * component boundaries.\n \t */\n-\tconst char *prefix;\n+\tchar *prefix;\n \n \t/*\n \t * A stack of levels. levels[0] is the uppermost level that is\n@@ -389,6 +387,9 @@ struct cache_ref_iterator {\n \tstruct cache_ref_iterator_level *levels;\n \n \tstruct repository *repo;\n+\tstruct ref_cache *cache;\n+\n+\tint prime_dir;\n };\n \n static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n@@ -396,6 +397,9 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tstruct cache_ref_iterator *iter =\n \t\t(struct cache_ref_iterator *)ref_iterator;\n \n+\tif (!iter->levels_nr)\n+\t\treturn ITER_DONE;\n+\n \twhile (1) {\n \t\tstruct cache_ref_iterator_level *level =\n \t\t\t&iter->levels[iter->levels_nr - 1];\n@@ -444,6 +448,41 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t}\n }\n \n+static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct cache_ref_iterator *iter =\n+\t\t(struct cache_ref_iterator *)ref_iterator;\n+\tstruct cache_ref_iterator_level *level;\n+\tstruct ref_dir *dir;\n+\n+\tdir = get_ref_dir(iter->cache->root);\n+\tif (prefix && *prefix)\n+\t\tdir = find_containing_dir(dir, prefix);\n+\tif (!dir) {\n+\t\titer->levels_nr = 0;\n+\t\treturn 0;\n+\t}\n+\n+\tif (iter->prime_dir)\n+\t\tprime_ref_dir(dir, prefix);\n+\titer->levels_nr = 1;\n+\tlevel = &iter->levels[0];\n+\tlevel->index = -1;\n+\tlevel->dir = dir;\n+\n+\tif (prefix && *prefix) {\n+\t\tfree(iter->prefix);\n+\t\titer->prefix = xstrdup(prefix);\n+\t\tlevel->prefix_state = PREFIX_WITHIN_DIR;\n+\t} else {\n+\t\tFREE_AND_NULL(iter->prefix);\n+\t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -456,12 +495,13 @@ static void cache_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct cache_ref_iterator *iter =\n \t\t(struct cache_ref_iterator *)ref_iterator;\n-\tfree((char *)iter->prefix);\n+\tfree(iter->prefix);\n \tfree(iter->levels);\n }\n \n static struct ref_iterator_vtable cache_ref_iterator_vtable = {\n \t.advance = cache_ref_iterator_advance,\n+\t.seek = cache_ref_iterator_seek,\n \t.peel = cache_ref_iterator_peel,\n \t.release = cache_ref_iterator_release,\n };\n@@ -471,39 +511,22 @@ struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\n \t\t\t\t\t struct repository *repo,\n \t\t\t\t\t int prime_dir)\n {\n-\tstruct ref_dir *dir;\n \tstruct cache_ref_iterator *iter;\n \tstruct ref_iterator *ref_iterator;\n-\tstruct cache_ref_iterator_level *level;\n-\n-\tdir = get_ref_dir(cache->root);\n-\tif (prefix && *prefix)\n-\t\tdir = find_containing_dir(dir, prefix);\n-\tif (!dir)\n-\t\t/* There's nothing to iterate over. */\n-\t\treturn empty_ref_iterator_begin();\n-\n-\tif (prime_dir)\n-\t\tprime_ref_dir(dir, prefix);\n \n \tCALLOC_ARRAY(iter, 1);\n \tref_iterator = &iter->base;\n \tbase_ref_iterator_init(ref_iterator, &cache_ref_iterator_vtable);\n \tALLOC_GROW(iter->levels, 10, iter->levels_alloc);\n \n-\titer->levels_nr = 1;\n-\tlevel = &iter->levels[0];\n-\tlevel->index = -1;\n-\tlevel->dir = dir;\n+\titer->repo = repo;\n+\titer->cache = cache;\n+\titer->prime_dir = prime_dir;\n \n-\tif (prefix && *prefix) {\n-\t\titer->prefix = xstrdup(prefix);\n-\t\tlevel->prefix_state = PREFIX_WITHIN_DIR;\n-\t} else {\n-\t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n+\tif (cache_ref_iterator_seek(&iter->base, prefix) < 0) {\n+\t\tref_iterator_free(&iter->base);\n+\t\treturn NULL;\n \t}\n \n-\titer->repo = repo;\n-\n \treturn ref_iterator;\n }\n\n-- \n2.49.0.rc0.416.g627208d89d.dirty\n\n"},{"id":"513688","messageId":"20250306-pks-update-ref-optimization-v5-14-dcb2ee037e97@pks.im","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-0-dcb2ee037e97@pks.im","subject":"[PATCH v5 14/16] refs/iterator: implement seeking for packed-ref iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-06T15:08:45Z","receivedAt":"2025-03-06T15:08:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking of `packed-ref` iterators. The implementation is again\nstraight forward, except that we cannot continue to use the prefix\niterator as we would otherwise not be able to reseek the iterator\nanymore in case one first asks for an empty and then for a non-empty\nprefix. Instead, we open-code the logic to in `advance()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/packed-backend.c | 65 ++++++++++++++++++++++++++++++++++-----------------\n 1 file changed, 43 insertions(+), 22 deletions(-)\n\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 38a1956d1a8..f4c82ba2c7d 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -819,6 +819,8 @@ struct packed_ref_iterator {\n \n \tstruct snapshot *snapshot;\n \n+\tchar *prefix;\n+\n \t/* The current position in the snapshot's buffer: */\n \tconst char *pos;\n \n@@ -841,11 +843,9 @@ struct packed_ref_iterator {\n };\n \n /*\n- * Move the iterator to the next record in the snapshot, without\n- * respect for whether the record is actually required by the current\n- * iteration. Adjust the fields in `iter` and return `ITER_OK` or\n- * `ITER_DONE`. This function does not free the iterator in the case\n- * of `ITER_DONE`.\n+ * Move the iterator to the next record in the snapshot. Adjust the fields in\n+ * `iter` and return `ITER_OK` or `ITER_DONE`. This function does not free the\n+ * iterator in the case of `ITER_DONE`.\n */\n static int next_record(struct packed_ref_iterator *iter)\n {\n@@ -942,6 +942,9 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tint ok;\n \n \twhile ((ok = next_record(iter)) == ITER_OK) {\n+\t\tconst char *refname = iter->base.refname;\n+\t\tconst char *prefix = iter->prefix;\n+\n \t\tif (iter->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n \t\t !is_per_worktree_ref(iter->base.refname))\n \t\t\tcontinue;\n@@ -951,12 +954,41 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t\t\t\t &iter->oid, iter->flags))\n \t\t\tcontinue;\n \n+\t\twhile (prefix && *prefix) {\n+\t\t\tif (*refname < *prefix)\n+\t\t\t\tBUG(\"packed-refs backend yielded reference preceding its prefix\");\n+\t\t\telse if (*refname > *prefix)\n+\t\t\t\treturn ITER_DONE;\n+\t\t\tprefix++;\n+\t\t\trefname++;\n+\t\t}\n+\n \t\treturn ITER_OK;\n \t}\n \n \treturn ok;\n }\n \n+static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct packed_ref_iterator *iter =\n+\t\t(struct packed_ref_iterator *)ref_iterator;\n+\tconst char *start;\n+\n+\tif (prefix && *prefix)\n+\t\tstart = find_reference_location(iter->snapshot, prefix, 0);\n+\telse\n+\t\tstart = iter->snapshot->start;\n+\n+\tfree(iter->prefix);\n+\titer->prefix = xstrdup_or_null(prefix);\n+\titer->pos = start;\n+\titer->eof = iter->snapshot->eof;\n+\n+\treturn 0;\n+}\n+\n static int packed_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -979,11 +1011,13 @@ static void packed_ref_iterator_release(struct ref_iterator *ref_iterator)\n \t\t(struct packed_ref_iterator *)ref_iterator;\n \tstrbuf_release(&iter->refname_buf);\n \tfree(iter->jump);\n+\tfree(iter->prefix);\n \trelease_snapshot(iter->snapshot);\n }\n \n static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \t.advance = packed_ref_iterator_advance,\n+\t.seek = packed_ref_iterator_seek,\n \t.peel = packed_ref_iterator_peel,\n \t.release = packed_ref_iterator_release,\n };\n@@ -1097,7 +1131,6 @@ static struct ref_iterator *packed_ref_iterator_begin(\n {\n \tstruct packed_ref_store *refs;\n \tstruct snapshot *snapshot;\n-\tconst char *start;\n \tstruct packed_ref_iterator *iter;\n \tstruct ref_iterator *ref_iterator;\n \tunsigned int required_flags = REF_STORE_READ;\n@@ -1113,14 +1146,6 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \t */\n \tsnapshot = get_snapshot(refs);\n \n-\tif (prefix && *prefix)\n-\t\tstart = find_reference_location(snapshot, prefix, 0);\n-\telse\n-\t\tstart = snapshot->start;\n-\n-\tif (start == snapshot->eof)\n-\t\treturn empty_ref_iterator_begin();\n-\n \tCALLOC_ARRAY(iter, 1);\n \tref_iterator = &iter->base;\n \tbase_ref_iterator_init(ref_iterator, &packed_ref_iterator_vtable);\n@@ -1130,19 +1155,15 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \n \titer->snapshot = snapshot;\n \tacquire_snapshot(snapshot);\n-\n-\titer->pos = start;\n-\titer->eof = snapshot->eof;\n \tstrbuf_init(&iter->refname_buf, 0);\n-\n \titer->base.oid = &iter->oid;\n-\n \titer->repo = ref_store->repo;\n \titer->flags = flags;\n \n-\tif (prefix && *prefix)\n-\t\t/* Stop iteration after we've gone *past* prefix: */\n-\t\tref_iterator = prefix_ref_iterator_begin(ref_iterator, prefix, 0);\n+\tif (packed_ref_iterator_seek(&iter->base, prefix) < 0) {\n+\t\tref_iterator_free(&iter->base);\n+\t\treturn NULL;\n+\t}\n \n \treturn ref_iterator;\n }\n\n-- \n2.49.0.rc0.416.g627208d89d.dirty\n\n"},{"id":"513690","messageId":"20250306-pks-update-ref-optimization-v5-15-dcb2ee037e97@pks.im","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-0-dcb2ee037e97@pks.im","subject":"[PATCH v5 15/16] refs/iterator: implement seeking for files iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-06T15:08:46Z","receivedAt":"2025-03-06T15:08:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking for \"files\" iterators. As we simply use a ref-cache\niterator under the hood the implementation is straight-forward. Note\nthat we do not implement seeking on reflog iterators, same as with the\n\"reftable\" backend.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/files-backend.c | 16 ++++++++++++++++\n 1 file changed, 16 insertions(+)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 859f1c11941..4e1c50fead3 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -918,6 +918,14 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n+static int files_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct files_ref_iterator *iter =\n+\t\t(struct files_ref_iterator *)ref_iterator;\n+\treturn ref_iterator_seek(iter->iter0, prefix);\n+}\n+\n static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -936,6 +944,7 @@ static void files_ref_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable files_ref_iterator_vtable = {\n \t.advance = files_ref_iterator_advance,\n+\t.seek = files_ref_iterator_seek,\n \t.peel = files_ref_iterator_peel,\n \t.release = files_ref_iterator_release,\n };\n@@ -2294,6 +2303,12 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n+static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n+\t\t\t\t const char *prefix UNUSED)\n+{\n+\tBUG(\"ref_iterator_seek() called for reflog_iterator\");\n+}\n+\n static int files_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \t\t\t\t struct object_id *peeled UNUSED)\n {\n@@ -2309,6 +2324,7 @@ static void files_reflog_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable files_reflog_iterator_vtable = {\n \t.advance = files_reflog_iterator_advance,\n+\t.seek = files_reflog_iterator_seek,\n \t.peel = files_reflog_iterator_peel,\n \t.release = files_reflog_iterator_release,\n };\n\n-- \n2.49.0.rc0.416.g627208d89d.dirty\n\n"},{"id":"513691","messageId":"20250306-pks-update-ref-optimization-v5-16-dcb2ee037e97@pks.im","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-0-dcb2ee037e97@pks.im","subject":"[PATCH v5 16/16] refs: reuse iterators when determining refname availability","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-06T15:08:47Z","receivedAt":"2025-03-06T15:08:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When verifying whether refnames are available we have to verify whether\nany reference exists that is nested under the current reference. E.g.\ngiven a reference \"refs/heads/foo\", we must make sure that there is no\nother reference \"refs/heads/foo/*\".\n\nThis check is performed using a ref iterator with the prefix set to the\nnested reference namespace. Until now it used to not be possible to\nreseek iterators, so we always had to reallocate the iterator for every\nsingle reference we're about to check. This keeps us from reusing state\nthat the iterator may have and that may make it work more efficiently.\n\nRefactor the logic to reseek iterators. This leads to a sizeable speedup\nwith the \"reftable\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 39.8 ms ± 0.9 ms [User: 29.7 ms, System: 9.8 ms]\n Range (min … max): 38.4 ms … 42.0 ms 62 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 31.9 ms ± 1.1 ms [User: 27.0 ms, System: 4.5 ms]\n Range (min … max): 29.8 ms … 34.3 ms 74 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.25 ± 0.05 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n\nThe \"files\" backend doesn't really show a huge impact:\n\n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 392.3 ms ± 7.1 ms [User: 59.7 ms, System: 328.8 ms]\n Range (min … max): 384.6 ms … 404.5 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 387.7 ms ± 7.4 ms [User: 54.6 ms, System: 329.6 ms]\n Range (min … max): 377.0 ms … 397.7 ms 10 runs\n\n Summary\n update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.01 ± 0.03 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n\nThis is mostly because it is way slower to begin with because it has to\ncreate a separate file for each new reference, so the milliseconds we\nshave off by reseeking the iterator doesn't really translate into a\nsignificant relative improvement.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs.c | 12 +++++++-----\n 1 file changed, 7 insertions(+), 5 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 8eff60a2186..6cbb9decdb0 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2555,8 +2555,13 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \t\tif (!initial_transaction) {\n \t\t\tint ok;\n \n-\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n-\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n+\t\t\tif (!iter) {\n+\t\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n+\t\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n+\t\t\t} else if (ref_iterator_seek(iter, dirname.buf) < 0) {\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\n \t\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\t\t\tif (skip &&\n \t\t\t\t string_list_has_string(skip, iter->refname))\n@@ -2569,9 +2574,6 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \n \t\t\tif (ok != ITER_DONE)\n \t\t\t\tBUG(\"error while iterating over references\");\n-\n-\t\t\tref_iterator_free(iter);\n-\t\t\titer = NULL;\n \t\t}\n \n \t\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n\n-- \n2.49.0.rc0.416.g627208d89d.dirty\n\n"},{"id":"513705","messageId":"Z8m7s_tyfro8VaJU@pks.im","threadId":"62965","inReplyTo":"CAOLa=ZRg=qUV_9kZpUnx=pJ6nBhrPQjGaVHQRjwHSa2omx93fw@mail.gmail.com","subject":"Re: [PATCH v4 05/16] refs/reftable: batch refname availability checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-06T15:13:55Z","receivedAt":"2025-03-06T15:14:01Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Mar 06, 2025 at 09:12:41AM -0500, Karthik Nayak wrote:\n> Karthik Nayak writes:\n> \n> > Patrick Steinhardt writes:\n> >\n> >> Refactor the \"reftable\" backend to batch the availability check for\n> >> refnames. This does not yet have an effect on performance as we\n> >> essentially still call `refs_verify_refname_available()` in a loop, but\n> >> this will change in subsequent commits.\n> >>\n> >\n> > I thought this patch removes it from the loop. Which loop are you\n> > talking about?\n> >\n> \n> Looking at future patches, maybe this 'loop' is a reference to how\n> 'refs_verify_refnames_available()' still loops over all references,\n> which we start optimizing in patch 08 and onward?\n\nYes, exactly. I'll clarify.\n\nPatrick\n"},{"id":"513711","messageId":"CAOLa=ZSUDq+uqBhbpf00k4i=mT9D8VFzjyR_a7K=3UzRiNj8eQ@mail.gmail.com","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-0-dcb2ee037e97@pks.im","subject":"Re: [PATCH v5 00/16] refs: batch refname availability checks","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-03-06T15:32:48Z","receivedAt":"2025-03-06T15:32:50Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n[snip]\n\n> Changes in v5:\n> - Improve a couple of commit messages.\n> - Align `GET_OID_*` flag values.\n> - Link to v4: https://lore.kernel.org/r/20250228-pks-update-ref-optimization-v4-0-6425c04268b5@pks.im\n>\n\nThe range-diff looks good (snipped to keep my email small and direct).\nThanks for addressing the comments!\n\nKarthik\n\n[snip]\n"},{"id":"514044","messageId":"Z9F6SFwh50s2MNkI@ArchLinux","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-2-dcb2ee037e97@pks.im","subject":"Re: [PATCH v5 02/16] object-name: allow skipping ambiguity checks in `get_oid()` family","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-03-12T12:12:56Z","receivedAt":"2025-03-12T12:12:47Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Thu, Mar 06, 2025 at 04:08:33PM +0100, Patrick Steinhardt wrote:\n> When reading an object ID via `get_oid_basic()` or any of its related\n> functions we perform a check whether the object ID is ambiguous, which\n> can be the case when a reference with the same name exists. While the\n> check is generally helpful, there are cases where it only adds to the\n> runtime overhead without providing much of a benefit.\n> \n\nWhen reading, I am wondering which cases. I somehow think that we may\ncombine this patch with the next patch. For this standalone commit, the\nreader cannot easily get the motivation until reading the next patch.\n\nHowever, I also understand why you use this standalone commit. So, both\nways are OK from my side.\n\n> Add a new flag that allows us to disable the check. The flag will be\n> used in a subsequent commit.\n> \n> Signed-off-by: Patrick Steinhardt \n"},{"id":"514045","messageId":"Z9F_3xu6RXeelXJK@ArchLinux","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-4-dcb2ee037e97@pks.im","subject":"Re: [PATCH v5 04/16] refs: introduce function to batch refname availability checks","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-03-12T12:36:47Z","receivedAt":"2025-03-12T12:36:38Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Thu, Mar 06, 2025 at 04:08:35PM +0100, Patrick Steinhardt wrote:\n> The `refs_verify_refname_available()` functions checks whether a\n> reference update can be committed or whether it would conflict with\n> either a prefix or suffix thereof. This function needs to be called once\n> per reference that one wants to check, which requires us to redo a\n> couple of checks every time the function is called.\n> \n> Introduce a new function `refs_verify_refnames_available()` that does\n> the same, but for a list of references. For now, the new function uses\n> the exact same implementation, except that we loop through all refnames\n> provided by the caller. This will be tuned in subsequent commits.\n> \n\nAfter reading this patch, I think we may could add more motivation here.\nWhat's the advantage of checking batch refname? From my understanding,\nwe want to check the group of refnames and if we find one which is not\ngood we will just return and we don't need to check further more thus\navoiding unnecessary checks which improves speed.\n\nWhen I read the commit message, I wonder if we will loop through all\nrefnames, we still need to handle the same number of refnames. So how do\nwe avoid this?\n\nI think we should make this clear.\n\n> The existing `refs_verify_refname_available()` function is reimplemented\n> on top of the new function. As such, the diff is best viewed with the\n> `--ignore-space-change option`.\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n> refs.c | 169 +++++++++++++++++++++++++++++++++++++----------------------------\n> refs.h | 12 +++++\n> 2 files changed, 109 insertions(+), 72 deletions(-)\n> \n> diff --git a/refs.c b/refs.c\n> index f4094a326a9..5a9b0f2fa1e 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -2467,19 +2467,15 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n> \treturn ret;\n> }\n> \n> -int refs_verify_refname_available(struct ref_store *refs,\n> -\t\t\t\t const char *refname,\n> -\t\t\t\t const struct string_list *extras,\n> -\t\t\t\t const struct string_list *skip,\n> -\t\t\t\t unsigned int initial_transaction,\n> -\t\t\t\t struct strbuf *err)\n> +int refs_verify_refnames_available(struct ref_store *refs,\n> +\t\t\t\t const struct string_list *refnames,\n> +\t\t\t\t const struct string_list *extras,\n> +\t\t\t\t const struct string_list *skip,\n> +\t\t\t\t unsigned int initial_transaction,\n> +\t\t\t\t struct strbuf *err)\n> {\n> -\tconst char *slash;\n> -\tconst char *extra_refname;\n> \tstruct strbuf dirname = STRBUF_INIT;\n> \tstruct strbuf referent = STRBUF_INIT;\n> -\tstruct object_id oid;\n> -\tunsigned int type;\n> \tint ret = -1;\n> \n> \t/*\n> @@ -2489,79 +2485,91 @@ int refs_verify_refname_available(struct ref_store *refs,\n> \n> \tassert(err);\n> \n> -\tstrbuf_grow(&dirname, strlen(refname) + 1);\n> -\tfor (slash = strchr(refname, '/'); slash; slash = strchr(slash + 1, '/')) {\n> -\t\t/*\n> -\t\t * Just saying \"Is a directory\" when we e.g. can't\n> -\t\t * lock some multi-level ref isn't very informative,\n> -\t\t * the user won't be told *what* is a directory, so\n> -\t\t * let's not use strerror() below.\n> -\t\t */\n> -\t\tint ignore_errno;\n> -\t\t/* Expand dirname to the new prefix, not including the trailing slash: */\n> -\t\tstrbuf_add(&dirname, refname + dirname.len, slash - refname - dirname.len);\n> +\tfor (size_t i = 0; i < refnames->nr; i++) {\n\nNit: we may just use `for_each_string_list_item` instead of use the raw\n\"for\" loop.\n\n> +\t\tconst char *refname = refnames->items[i].string;\n> +\t\tconst char *extra_refname;\n> +\t\tstruct object_id oid;\n> +\t\tunsigned int type;\n> +\t\tconst char *slash;\n> +\n> +\t\tstrbuf_reset(&dirname);\n> +\n> +\t\tfor (slash = strchr(refname, '/'); slash; slash = strchr(slash + 1, '/')) {\n> +\t\t\t/*\n> +\t\t\t * Just saying \"Is a directory\" when we e.g. can't\n> +\t\t\t * lock some multi-level ref isn't very informative,\n> +\t\t\t * the user won't be told *what* is a directory, so\n> +\t\t\t * let's not use strerror() below.\n> +\t\t\t */\n> +\t\t\tint ignore_errno;\n> +\n> +\t\t\t/* Expand dirname to the new prefix, not including the trailing slash: */\n> +\t\t\tstrbuf_add(&dirname, refname + dirname.len, slash - refname - dirname.len);\n> +\n> +\t\t\t/*\n> +\t\t\t * We are still at a leading dir of the refname (e.g.,\n> +\t\t\t * \"refs/foo\"; if there is a reference with that name,\n> +\t\t\t * it is a conflict, *unless* it is in skip.\n> +\t\t\t */\n> +\t\t\tif (skip && string_list_has_string(skip, dirname.buf))\n> +\t\t\t\tcontinue;\n> +\n> +\t\t\tif (!initial_transaction &&\n> +\t\t\t !refs_read_raw_ref(refs, dirname.buf, &oid, &referent,\n> +\t\t\t\t\t &type, &ignore_errno)) {\n> +\t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n> +\t\t\t\t\t dirname.buf, refname);\n> +\t\t\t\tgoto cleanup;\n> +\t\t\t}\n> +\n> +\t\t\tif (extras && string_list_has_string(extras, dirname.buf)) {\n> +\t\t\t\tstrbuf_addf(err, _(\"cannot process '%s' and '%s' at the same time\"),\n> +\t\t\t\t\t refname, dirname.buf);\n> +\t\t\t\tgoto cleanup;\n> +\t\t\t}\n> +\t\t}\n> \n> \t\t/*\n> -\t\t * We are still at a leading dir of the refname (e.g.,\n> -\t\t * \"refs/foo\"; if there is a reference with that name,\n> -\t\t * it is a conflict, *unless* it is in skip.\n> +\t\t * We are at the leaf of our refname (e.g., \"refs/foo/bar\").\n> +\t\t * There is no point in searching for a reference with that\n> +\t\t * name, because a refname isn't considered to conflict with\n> +\t\t * itself. But we still need to check for references whose\n> +\t\t * names are in the \"refs/foo/bar/\" namespace, because they\n> +\t\t * *do* conflict.\n> \t\t */\n> -\t\tif (skip && string_list_has_string(skip, dirname.buf))\n> -\t\t\tcontinue;\n> +\t\tstrbuf_addstr(&dirname, refname + dirname.len);\n> +\t\tstrbuf_addch(&dirname, '/');\n> +\n> +\t\tif (!initial_transaction) {\n> +\t\t\tstruct ref_iterator *iter;\n> +\t\t\tint ok;\n> +\n> +\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n> +\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n> +\t\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n> +\t\t\t\tif (skip &&\n> +\t\t\t\t string_list_has_string(skip, iter->refname))\n> +\t\t\t\t\tcontinue;\n> +\n> +\t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n> +\t\t\t\t\t iter->refname, refname);\n> +\t\t\t\tref_iterator_abort(iter);\n> +\t\t\t\tgoto cleanup;\n> +\t\t\t}\n> \n> -\t\tif (!initial_transaction &&\n> -\t\t !refs_read_raw_ref(refs, dirname.buf, &oid, &referent,\n> -\t\t\t\t &type, &ignore_errno)) {\n> -\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n> -\t\t\t\t dirname.buf, refname);\n> -\t\t\tgoto cleanup;\n> +\t\t\tif (ok != ITER_DONE)\n> +\t\t\t\tBUG(\"error while iterating over references\");\n> \t\t}\n> \n> -\t\tif (extras && string_list_has_string(extras, dirname.buf)) {\n> +\t\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n> +\t\tif (extra_refname) {\n> \t\t\tstrbuf_addf(err, _(\"cannot process '%s' and '%s' at the same time\"),\n> -\t\t\t\t refname, dirname.buf);\n> +\t\t\t\t refname, extra_refname);\n> \t\t\tgoto cleanup;\n> \t\t}\n> \t}\n> \n> -\t/*\n> -\t * We are at the leaf of our refname (e.g., \"refs/foo/bar\").\n> -\t * There is no point in searching for a reference with that\n> -\t * name, because a refname isn't considered to conflict with\n> -\t * itself. But we still need to check for references whose\n> -\t * names are in the \"refs/foo/bar/\" namespace, because they\n> -\t * *do* conflict.\n> -\t */\n> -\tstrbuf_addstr(&dirname, refname + dirname.len);\n> -\tstrbuf_addch(&dirname, '/');\n> -\n> -\tif (!initial_transaction) {\n> -\t\tstruct ref_iterator *iter;\n> -\t\tint ok;\n> -\n> -\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n> -\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n> -\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n> -\t\t\tif (skip &&\n> -\t\t\t string_list_has_string(skip, iter->refname))\n> -\t\t\t\tcontinue;\n> -\n> -\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n> -\t\t\t\t iter->refname, refname);\n> -\t\t\tref_iterator_abort(iter);\n> -\t\t\tgoto cleanup;\n> -\t\t}\n> -\n> -\t\tif (ok != ITER_DONE)\n> -\t\t\tBUG(\"error while iterating over references\");\n> -\t}\n> -\n> -\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n> -\tif (extra_refname)\n> -\t\tstrbuf_addf(err, _(\"cannot process '%s' and '%s' at the same time\"),\n> -\t\t\t refname, extra_refname);\n> -\telse\n> -\t\tret = 0;\n> +\tret = 0;\n> \n> cleanup:\n> \tstrbuf_release(&referent);\n> @@ -2569,6 +2577,23 @@ int refs_verify_refname_available(struct ref_store *refs,\n> \treturn ret;\n> }\n> \n> +int refs_verify_refname_available(struct ref_store *refs,\n> +\t\t\t\t const char *refname,\n> +\t\t\t\t const struct string_list *extras,\n> +\t\t\t\t const struct string_list *skip,\n> +\t\t\t\t unsigned int initial_transaction,\n> +\t\t\t\t struct strbuf *err)\n> +{\n> +\tstruct string_list_item item = { .string = (char *) refname };\n> +\tstruct string_list refnames = {\n> +\t\t.items = &item,\n> +\t\t.nr = 1,\n> +\t};\n> +\n> +\treturn refs_verify_refnames_available(refs, &refnames, extras, skip,\n> +\t\t\t\t\t initial_transaction, err);\n> +}\n> +\n> struct do_for_each_reflog_help {\n> \teach_reflog_fn *fn;\n> \tvoid *cb_data;\n> diff --git a/refs.h b/refs.h\n> index a0cdd99250e..185aed5a461 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -124,6 +124,18 @@ int refs_verify_refname_available(struct ref_store *refs,\n> \t\t\t\t unsigned int initial_transaction,\n> \t\t\t\t struct strbuf *err);\n> \n> +/*\n> + * Same as `refs_verify_refname_available()`, but checking for a list of\n> + * refnames instead of only a single item. This is more efficient in the case\n> + * where one needs to check multiple refnames.\n> + */\n\nShould we talk about more about why this is more efficient?\n\n> +int refs_verify_refnames_available(struct ref_store *refs,\n> +\t\t\t\t const struct string_list *refnames,\n> +\t\t\t\t const struct string_list *extras,\n> +\t\t\t\t const struct string_list *skip,\n> +\t\t\t\t unsigned int initial_transaction,\n> +\t\t\t\t struct strbuf *err);\n> +\n> int refs_ref_exists(struct ref_store *refs, const char *refname);\n> \n> int should_autocreate_reflog(enum log_refs_config log_all_ref_updates,\n> \n> -- \n> 2.49.0.rc0.416.g627208d89d.dirty\n"},{"id":"514046","messageId":"Z9GBut33q1puzliq@ArchLinux","threadId":"62965","inReplyTo":"Z9F_3xu6RXeelXJK@ArchLinux","subject":"Re: [PATCH v5 04/16] refs: introduce function to batch refname availability checks","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-03-12T12:44:42Z","receivedAt":"2025-03-12T12:44:33Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Wed, Mar 12, 2025 at 08:36:47PM +0800, shejialuo wrote:\n> On Thu, Mar 06, 2025 at 04:08:35PM +0100, Patrick Steinhardt wrote:\n> > The `refs_verify_refname_available()` functions checks whether a\n> > reference update can be committed or whether it would conflict with\n> > either a prefix or suffix thereof. This function needs to be called once\n> > per reference that one wants to check, which requires us to redo a\n> > couple of checks every time the function is called.\n> > \n> > Introduce a new function `refs_verify_refnames_available()` that does\n> > the same, but for a list of references. For now, the new function uses\n> > the exact same implementation, except that we loop through all refnames\n> > provided by the caller. This will be tuned in subsequent commits.\n> > \n> \n> After reading this patch, I think we may could add more motivation here.\n> What's the advantage of checking batch refname? From my understanding,\n> we want to check the group of refnames and if we find one which is not\n> good we will just return and we don't need to check further more thus\n> avoiding unnecessary checks which improves speed.\n> \n> When I read the commit message, I wonder if we will loop through all\n> refnames, we still need to handle the same number of refnames. So how do\n> we avoid this?\n> \n> I think we should make this clear.\n> \n\nI totally misunderstood. After reading the commit message of the next\npatch, I have realized that we just simply do a refactor. Ignore my\ncomment. Sorry.\n"},{"id":"514048","messageId":"Z9GD-mg7CrvUxgqH@ArchLinux","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-5-dcb2ee037e97@pks.im","subject":"Re: [PATCH v5 05/16] refs/reftable: batch refname availability checks","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-03-12T12:54:18Z","receivedAt":"2025-03-12T12:54:09Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Thu, Mar 06, 2025 at 04:08:36PM +0100, Patrick Steinhardt wrote:\n> Refactor the \"reftable\" backend to batch the availability check for\n> refnames. This does not yet have an effect on performance as\n> `refs_verify_refnames_available()` effectively still performs the\n> availability check for each refname individually. But this will be\n> optimized in subsequent commits, where we learn to optimize some parts\n> of the logic when checking multiple refnames for availability.\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n> refs/reftable-backend.c | 16 ++++++++++------\n> 1 file changed, 10 insertions(+), 6 deletions(-)\n> \n> diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> index d39a14c5a46..2a90e7cb391 100644\n> --- a/refs/reftable-backend.c\n> +++ b/refs/reftable-backend.c\n> @@ -1069,6 +1069,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n> \t\treftable_be_downcast(ref_store, REF_STORE_WRITE|REF_STORE_MAIN, \"ref_transaction_prepare\");\n> \tstruct strbuf referent = STRBUF_INIT, head_referent = STRBUF_INIT;\n> \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n> +\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n> \tstruct reftable_transaction_data *tx_data = NULL;\n> \tstruct reftable_backend *be;\n> \tstruct object_id head_oid;\n> @@ -1224,12 +1225,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n> \t\t\t * can output a proper error message instead of failing\n> \t\t\t * at a later point.\n> \t\t\t */\n> -\t\t\tret = refs_verify_refname_available(ref_store, u->refname,\n> -\t\t\t\t\t\t\t &affected_refnames, NULL,\n> -\t\t\t\t\t\t\t transaction->flags & REF_TRANSACTION_FLAG_INITIAL,\n> -\t\t\t\t\t\t\t err);\n> -\t\t\tif (ret < 0)\n> -\t\t\t\tgoto done;\n> +\t\t\tstring_list_append(&refnames_to_check, u->refname);\n> \n\nInstead of checking in `prepare`, we will just append the refname to the\nlist.\n\n> \t\t\t/*\n> \t\t\t * There is no need to write the reference deletion\n> @@ -1379,6 +1375,13 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n> \t\t}\n> \t}\n> \n> +\tstring_list_sort(&refnames_to_check);\n\nI am curious why we need to sort the refnames here. I think at current,\nwe don't optimize the `refs_verify_refnames_available` function. No\nmatter whether the `refnames_to_check` is sorted, it should not\nchange the result of `refs_verify_refnames_available` function, right? I\nguess this statement may be related to optimization part. If so, I think\nwe should delete this line and add in the later commit.\n\nHowever, I am not sure.\n\n> +\tret = refs_verify_refnames_available(ref_store, &refnames_to_check, &affected_refnames, NULL,\n> +\t\t\t\t\t transaction->flags & REF_TRANSACTION_FLAG_INITIAL,\n> +\t\t\t\t\t err);\n> +\tif (ret < 0)\n> +\t\tgoto done;\n> +\n> \ttransaction->backend_data = tx_data;\n> \ttransaction->state = REF_TRANSACTION_PREPARED;\n> \n> @@ -1394,6 +1397,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n> \tstring_list_clear(&affected_refnames, 0);\n> \tstrbuf_release(&referent);\n> \tstrbuf_release(&head_referent);\n> +\tstring_list_clear(&refnames_to_check, 0);\n> \n> \treturn ret;\n> }\n"},{"id":"514049","messageId":"Z9GFBl5AIQZGyI-9@ArchLinux","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-6-dcb2ee037e97@pks.im","subject":"Re: [PATCH v5 06/16] refs/files: batch refname availability checks for normal transactions","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-03-12T12:58:46Z","receivedAt":"2025-03-12T12:58:37Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Thu, Mar 06, 2025 at 04:08:37PM +0100, Patrick Steinhardt wrote:\n> @@ -2811,6 +2808,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n> \tsize_t i;\n> \tint ret = 0;\n> \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n> +\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n> \tchar *head_ref = NULL;\n> \tint head_type;\n> \tstruct files_transaction_backend_data *backend_data;\n> @@ -2898,7 +2896,8 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n> \t\tstruct ref_update *update = transaction->updates[i];\n> \n> \t\tret = lock_ref_for_update(refs, update, transaction,\n> -\t\t\t\t\t head_ref, &affected_refnames, err);\n> +\t\t\t\t\t head_ref, &refnames_to_check,\n> +\t\t\t\t\t &affected_refnames, err);\n> \t\tif (ret)\n> \t\t\tgoto cleanup;\n> \n> @@ -2930,6 +2929,26 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n> \t\t}\n> \t}\n> \n> +\t/*\n> +\t * Verify that none of the loose reference that we're about to write\n> +\t * conflict with any existing packed references. Ideally, we'd do this\n> +\t * check after the packed-refs are locked so that the file cannot\n> +\t * change underneath our feet. But introducing such a lock now would\n> +\t * probably do more harm than good as users rely on there not being a\n> +\t * global lock with the \"files\" backend.\n> +\t *\n> +\t * Another alternative would be to do the check after the (optional)\n> +\t * lock, but that would extend the time we spend in the globally-locked\n> +\t * state.\n> +\t *\n> +\t * So instead, we accept the race for now.\n> +\t */\n\nI am curious why we don't sort the `refnames_to_check` here. What is the\ndifference between the reftable backend and files backend?\n\n> +\tif (refs_verify_refnames_available(refs->packed_ref_store, &refnames_to_check,\n> +\t\t\t\t\t &affected_refnames, NULL, 0, err)) {\n> +\t\tret = TRANSACTION_NAME_CONFLICT;\n> +\t\tgoto cleanup;\n> +\t}\n> +\n> \tif (packed_transaction) {\n> \t\tif (packed_refs_lock(refs->packed_ref_store, 0, err)) {\n> \t\t\tret = TRANSACTION_GENERIC_ERROR;\n"},{"id":"514050","messageId":"Z9GGwNpfDly4_NIL@ArchLinux","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-7-dcb2ee037e97@pks.im","subject":"Re: [PATCH v5 07/16] refs/files: batch refname availability checks for initial transactions","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-03-12T13:06:08Z","receivedAt":"2025-03-12T13:05:59Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Thu, Mar 06, 2025 at 04:08:38PM +0100, Patrick Steinhardt wrote:\n> The \"files\" backend explicitly carves out special logic for its initial\n> transaction so that it can avoid writing out every single reference as\n> a loose reference. While the assumption is that there shouldn't be any\n> preexisting references, we still have to verify that none of the newly\n> written references will conflict with any other new reference in the\n> same transaction.\n> \n> Refactor the initial transaction to use batched refname availability\n> checks. This does not yet have an effect on performance as we still call\n> `refs_verify_refname_available()` in a loop. But this will change in\n> subsequent commits and then impact performance when cloning a repository\n> with many references or when migrating references to the \"files\" format.\n> \n> This will improve performance when cloning a repository with many\n> references or when migrating references from any format to the \"files\"\n> format once the availability checks have learned to optimize checks for\n> many references in a subsequent commit.\n\nI guess you forgot to delete some sentences for the commit message. This\nparagraph is a little redundant. In the second paragraph, I think we\nhave already talked about this.\n\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n> refs/files-backend.c | 23 ++++++++++++++++-------\n> 1 file changed, 16 insertions(+), 7 deletions(-)\n> \n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index 6ce79cf0791..11a620ea11a 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -3056,6 +3056,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n> \tsize_t i;\n> \tint ret = 0;\n> \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n> +\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n> \tstruct ref_transaction *packed_transaction = NULL;\n> \tstruct ref_transaction *loose_transaction = NULL;\n> \n> @@ -3105,11 +3106,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n> \t\t !is_null_oid(&update->old_oid))\n> \t\t\tBUG(\"initial ref transaction with old_sha1 set\");\n> \n> -\t\tif (refs_verify_refname_available(&refs->base, update->refname,\n> -\t\t\t\t\t\t &affected_refnames, NULL, 1, err)) {\n> -\t\t\tret = TRANSACTION_NAME_CONFLICT;\n> -\t\t\tgoto cleanup;\n> -\t\t}\n> +\t\tstring_list_append(&refnames_to_check, update->refname);\n> \n> \t\t/*\n> \t\t * packed-refs don't support symbolic refs, root refs and reflogs,\n> @@ -3145,8 +3142,19 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n> \t\t}\n> \t}\n> \n> -\tif (packed_refs_lock(refs->packed_ref_store, 0, err) ||\n> -\t ref_transaction_commit(packed_transaction, err)) {\n> +\tif (packed_refs_lock(refs->packed_ref_store, 0, err)) {\n> +\t\tret = TRANSACTION_GENERIC_ERROR;\n> +\t\tgoto cleanup;\n> +\t}\n> +\n\nStill the same question: why we don't sort the `refnames_to_check` here.\n\n> +\tif (refs_verify_refnames_available(&refs->base, &refnames_to_check,\n> +\t\t\t\t\t &affected_refnames, NULL, 1, err)) {\n> +\t\tpacked_refs_unlock(refs->packed_ref_store);\n> +\t\tret = TRANSACTION_NAME_CONFLICT;\n> +\t\tgoto cleanup;\n> +\t}\n> +\n> +\tif (ref_transaction_commit(packed_transaction, err)) {\n> \t\tret = TRANSACTION_GENERIC_ERROR;\n> \t\tgoto cleanup;\n> \t}\n> @@ -3167,6 +3175,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n> \t\tref_transaction_free(packed_transaction);\n> \ttransaction->state = REF_TRANSACTION_CLOSED;\n> \tstring_list_clear(&affected_refnames, 0);\n> +\tstring_list_clear(&refnames_to_check, 0);\n> \treturn ret;\n> }\n"},{"id":"514055","messageId":"Z9GKs9z4n4qgOP7n@ArchLinux","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-8-dcb2ee037e97@pks.im","subject":"Re: [PATCH v5 08/16] refs: stop re-verifying common prefixes for availability","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-03-12T13:22:59Z","receivedAt":"2025-03-12T13:22:50Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Thu, Mar 06, 2025 at 04:08:39PM +0100, Patrick Steinhardt wrote:\n> refs.c | 12 ++++++++++++\n> 1 file changed, 12 insertions(+)\n> \n> diff --git a/refs.c b/refs.c\n> index 5a9b0f2fa1e..eaf41421f50 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -2476,6 +2476,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n> {\n> \tstruct strbuf dirname = STRBUF_INIT;\n> \tstruct strbuf referent = STRBUF_INIT;\n> +\tstruct strset dirnames;\n> \tint ret = -1;\n> \n> \t/*\n> @@ -2485,6 +2486,8 @@ int refs_verify_refnames_available(struct ref_store *refs,\n> \n> \tassert(err);\n> \n> +\tstrset_init(&dirnames);\n> +\n> \tfor (size_t i = 0; i < refnames->nr; i++) {\n> \t\tconst char *refname = refnames->items[i].string;\n> \t\tconst char *extra_refname;\n> @@ -2514,6 +2517,14 @@ int refs_verify_refnames_available(struct ref_store *refs,\n> \t\t\tif (skip && string_list_has_string(skip, dirname.buf))\n> \t\t\t\tcontinue;\n> \n> +\t\t\t/*\n> +\t\t\t * If we've already seen the directory we don't need to\n> +\t\t\t * process it again. Skip it to avoid checking checking\n> +\t\t\t * common prefixes like \"refs/heads/\" repeatedly.\n> +\t\t\t */\n> +\t\t\tif (!strset_add(&dirnames, dirname.buf))\n> +\t\t\t\tcontinue;\n> +\n\nReading here, I think we should not sort the refnames for \"reftable\"\nbackend. Anyway, really a nice job for optimizing the speed.\n\n> \t\t\tif (!initial_transaction &&\n> \t\t\t !refs_read_raw_ref(refs, dirname.buf, &oid, &referent,\n> \t\t\t\t\t &type, &ignore_errno)) {\n> @@ -2574,6 +2585,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n> cleanup:\n> \tstrbuf_release(&referent);\n> \tstrbuf_release(&dirname);\n> +\tstrset_clear(&dirnames);\n> \treturn ret;\n> }\n> \n> \n> -- \n> 2.49.0.rc0.416.g627208d89d.dirty\n> \n"},{"id":"514057","messageId":"Z9GQDJxT1Gu9nzGp@ArchLinux","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-9-dcb2ee037e97@pks.im","subject":"Re: [PATCH v5 09/16] refs/iterator: separate lifecycle from iteration","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-03-12T13:45:48Z","receivedAt":"2025-03-12T13:45:40Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Thu, Mar 06, 2025 at 04:08:40PM +0100, Patrick Steinhardt wrote:\n\n> @@ -350,19 +338,10 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> \n> \twhile ((ok = ref_iterator_advance(iter->iter0)) == ITER_OK) {\n> \t\tint cmp = compare_prefix(iter->iter0->refname, iter->prefix);\n> -\n> \t\tif (cmp < 0)\n> \t\t\tcontinue;\n> -\n> -\t\tif (cmp > 0) {\n> -\t\t\t/*\n> -\t\t\t * As the source iterator is ordered, we\n> -\t\t\t * can stop the iteration as soon as we see a\n> -\t\t\t * refname that comes after the prefix:\n> -\t\t\t */\n> -\t\t\tok = ref_iterator_abort(iter->iter0);\n> -\t\t\tbreak;\n> -\t\t}\n> +\t\tif (cmp > 0)\n> +\t\t\treturn ITER_DONE;\n\nShould we maintain the above comment? Why do we delete the comment. I\nsomehow think the comment makes sense.\n\n> \n> \t\tif (iter->trim) {\n> \t\t\t/*\n"},{"id":"514058","messageId":"Z9GUJliUPpNPng7z@ArchLinux","threadId":"62965","inReplyTo":"20250306-pks-update-ref-optimization-v5-0-dcb2ee037e97@pks.im","subject":"Re: [PATCH v5 00/16] refs: batch refname availability checks","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-03-12T14:03:18Z","receivedAt":"2025-03-12T14:03:08Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Thu, Mar 06, 2025 at 04:08:31PM +0100, Patrick Steinhardt wrote:\n\n> Changes in v3:\n> - Fix one case where we didn't skip ambiguity checks in\n> git-update-ref(1).\n> - Document better that only the prefix can change on reseeking\n> iterators. Other internal state will remain the same.\n> - Fix a memory leak in the ref-cache iterator.\n> - Don't ignore errors returned by `packed_ref_iterator_seek()`.\n> - Link to v2: https://lore.kernel.org/r/20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im\n> \n> Changes in v4:\n> - A couple of clarifications in the commit message that disabled\n> ambiguity warnings.\n> - Link to v3: https://lore.kernel.org/r/20250225-pks-update-ref-optimization-v3-0-77c3687cda75@pks.im\n> \n> Changes in v5:\n> - Improve a couple of commit messages.\n> - Align `GET_OID_*` flag values.\n> - Link to v4: https://lore.kernel.org/r/20250228-pks-update-ref-optimization-v4-0-6425c04268b5@pks.im\n> \n> Thanks!\n> \n> Patrick\n\nI don't look at the range-diff, but review again in this version. I\nmainly carefully look at patch 1-8 which I haven't looked in the\nprevious versions and left some comments.\n\nThanks,\nJialuo\n"},{"id":"514063","messageId":"Z9GqAYy4E0JVU7DA@pks.im","threadId":"62965","inReplyTo":"Z9F_3xu6RXeelXJK@ArchLinux","subject":"Re: [PATCH v5 04/16] refs: introduce function to batch refname availability checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:36:33Z","receivedAt":"2025-03-12T15:36:40Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Mar 12, 2025 at 08:36:47PM +0800, shejialuo wrote:\n> On Thu, Mar 06, 2025 at 04:08:35PM +0100, Patrick Steinhardt wrote:\n> > diff --git a/refs.c b/refs.c\n> > index f4094a326a9..5a9b0f2fa1e 100644\n> > --- a/refs.c\n> > +++ b/refs.c\n> > @@ -2489,79 +2485,91 @@ int refs_verify_refname_available(struct ref_store *refs,\n> > \n> > \tassert(err);\n> > \n> > -\tstrbuf_grow(&dirname, strlen(refname) + 1);\n> > -\tfor (slash = strchr(refname, '/'); slash; slash = strchr(slash + 1, '/')) {\n> > -\t\t/*\n> > -\t\t * Just saying \"Is a directory\" when we e.g. can't\n> > -\t\t * lock some multi-level ref isn't very informative,\n> > -\t\t * the user won't be told *what* is a directory, so\n> > -\t\t * let's not use strerror() below.\n> > -\t\t */\n> > -\t\tint ignore_errno;\n> > -\t\t/* Expand dirname to the new prefix, not including the trailing slash: */\n> > -\t\tstrbuf_add(&dirname, refname + dirname.len, slash - refname - dirname.len);\n> > +\tfor (size_t i = 0; i < refnames->nr; i++) {\n> \n> Nit: we may just use `for_each_string_list_item` instead of use the raw\n> \"for\" loop.\n\nFair, can do.\n\n> > diff --git a/refs.h b/refs.h\n> > index a0cdd99250e..185aed5a461 100644\n> > --- a/refs.h\n> > +++ b/refs.h\n> > @@ -124,6 +124,18 @@ int refs_verify_refname_available(struct ref_store *refs,\n> > \t\t\t\t unsigned int initial_transaction,\n> > \t\t\t\t struct strbuf *err);\n> > \n> > +/*\n> > + * Same as `refs_verify_refname_available()`, but checking for a list of\n> > + * refnames instead of only a single item. This is more efficient in the case\n> > + * where one needs to check multiple refnames.\n> > + */\n> \n> Should we talk about more about why this is more efficient?\n\nI don't think the caller needs to be aware of why specifically it is\nfaster. All they should care for is that it does the same than the other\nfunction, but that it knows to optimize better.\n\nPatrick\n"},{"id":"514064","messageId":"Z9GqBsUdefVlNusR@pks.im","threadId":"62965","inReplyTo":"Z9GD-mg7CrvUxgqH@ArchLinux","subject":"Re: [PATCH v5 05/16] refs/reftable: batch refname availability checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:36:38Z","receivedAt":"2025-03-12T15:36:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Mar 12, 2025 at 08:54:18PM +0800, shejialuo wrote:\n> On Thu, Mar 06, 2025 at 04:08:36PM +0100, Patrick Steinhardt wrote:\n> > diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\n> > index d39a14c5a46..2a90e7cb391 100644\n> > --- a/refs/reftable-backend.c\n> > +++ b/refs/reftable-backend.c\n> > @@ -1379,6 +1375,13 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n> > \t\t}\n> > \t}\n> > \n> > +\tstring_list_sort(&refnames_to_check);\n> \n> I am curious why we need to sort the refnames here. I think at current,\n> we don't optimize the `refs_verify_refnames_available` function. No\n> matter whether the `refnames_to_check` is sorted, it should not\n> change the result of `refs_verify_refnames_available` function, right? I\n> guess this statement may be related to optimization part. If so, I think\n> we should delete this line and add in the later commit.\n> \n> However, I am not sure.\n\nYou're right, sorting shouldn't be necessary. It was in a previous\nversion of my patch series, but now that it's not we can drop this.\n\nWe may at one point introduce an optimization that does depend on refs\nbeing sorted. But until there is a need we should skip unnecessary work.\n\nPatrick\n"},{"id":"514065","messageId":"Z9GqCsobVxnFI-rc@pks.im","threadId":"62965","inReplyTo":"Z9GFBl5AIQZGyI-9@ArchLinux","subject":"Re: [PATCH v5 06/16] refs/files: batch refname availability checks for normal transactions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:36:42Z","receivedAt":"2025-03-12T15:36:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Mar 12, 2025 at 08:58:46PM +0800, shejialuo wrote:\n> On Thu, Mar 06, 2025 at 04:08:37PM +0100, Patrick Steinhardt wrote:\n> > @@ -2930,6 +2929,26 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n> > \t\t}\n> > \t}\n> > \n> > +\t/*\n> > +\t * Verify that none of the loose reference that we're about to write\n> > +\t * conflict with any existing packed references. Ideally, we'd do this\n> > +\t * check after the packed-refs are locked so that the file cannot\n> > +\t * change underneath our feet. But introducing such a lock now would\n> > +\t * probably do more harm than good as users rely on there not being a\n> > +\t * global lock with the \"files\" backend.\n> > +\t *\n> > +\t * Another alternative would be to do the check after the (optional)\n> > +\t * lock, but that would extend the time we spend in the globally-locked\n> > +\t * state.\n> > +\t *\n> > +\t * So instead, we accept the race for now.\n> > +\t */\n> \n> I am curious why we don't sort the `refnames_to_check` here. What is the\n> difference between the reftable backend and files backend?\n\nWe do sort because we use `string_list_insert()` here. But in fact, we\ndon't have to sort at all, so I'll stop sorting in the preceding commit\nand convert this callsite here to use `string_list_append()` instead.\n\nPatrick\n"},{"id":"514066","messageId":"Z9GqDjVO3VvFOT4x@pks.im","threadId":"62965","inReplyTo":"Z9GGwNpfDly4_NIL@ArchLinux","subject":"Re: [PATCH v5 07/16] refs/files: batch refname availability checks for initial transactions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:36:46Z","receivedAt":"2025-03-12T15:36:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Mar 12, 2025 at 09:06:08PM +0800, shejialuo wrote:\n> On Thu, Mar 06, 2025 at 04:08:38PM +0100, Patrick Steinhardt wrote:\n> > The \"files\" backend explicitly carves out special logic for its initial\n> > transaction so that it can avoid writing out every single reference as\n> > a loose reference. While the assumption is that there shouldn't be any\n> > preexisting references, we still have to verify that none of the newly\n> > written references will conflict with any other new reference in the\n> > same transaction.\n> > \n> > Refactor the initial transaction to use batched refname availability\n> > checks. This does not yet have an effect on performance as we still call\n> > `refs_verify_refname_available()` in a loop. But this will change in\n> > subsequent commits and then impact performance when cloning a repository\n> > with many references or when migrating references to the \"files\" format.\n> > \n> > This will improve performance when cloning a repository with many\n> > references or when migrating references from any format to the \"files\"\n> > format once the availability checks have learned to optimize checks for\n> > many references in a subsequent commit.\n> \n> I guess you forgot to delete some sentences for the commit message. This\n> paragraph is a little redundant. In the second paragraph, I think we\n> have already talked about this.\n\nNot quite: the second paragraph talks about how this does not yet have\nan effect, but will have an effect once we have optimized this. The\nthird paragraph on the other hand talks about which specific parts will\nbenefit from the optimizations.\n\nPatrick\n"},{"id":"514067","messageId":"Z9GqErzv39SCEb5v@pks.im","threadId":"62965","inReplyTo":"Z9GQDJxT1Gu9nzGp@ArchLinux","subject":"Re: [PATCH v5 09/16] refs/iterator: separate lifecycle from iteration","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:36:50Z","receivedAt":"2025-03-12T15:36:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Mar 12, 2025 at 09:45:48PM +0800, shejialuo wrote:\n> On Thu, Mar 06, 2025 at 04:08:40PM +0100, Patrick Steinhardt wrote:\n> \n> > @@ -350,19 +338,10 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> > \n> > \twhile ((ok = ref_iterator_advance(iter->iter0)) == ITER_OK) {\n> > \t\tint cmp = compare_prefix(iter->iter0->refname, iter->prefix);\n> > -\n> > \t\tif (cmp < 0)\n> > \t\t\tcontinue;\n> > -\n> > -\t\tif (cmp > 0) {\n> > -\t\t\t/*\n> > -\t\t\t * As the source iterator is ordered, we\n> > -\t\t\t * can stop the iteration as soon as we see a\n> > -\t\t\t * refname that comes after the prefix:\n> > -\t\t\t */\n> > -\t\t\tok = ref_iterator_abort(iter->iter0);\n> > -\t\t\tbreak;\n> > -\t\t}\n> > +\t\tif (cmp > 0)\n> > +\t\t\treturn ITER_DONE;\n> \n> Should we maintain the above comment? Why do we delete the comment. I\n> somehow think the comment makes sense.\n\nYeah, let's. I'll add it back in.\n\nPatrick\n"},{"id":"514068","messageId":"Z9GqFmLBX6SAkgZy@pks.im","threadId":"62965","inReplyTo":"Z9GKs9z4n4qgOP7n@ArchLinux","subject":"Re: [PATCH v5 08/16] refs: stop re-verifying common prefixes for availability","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:36:54Z","receivedAt":"2025-03-12T15:36:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Mar 12, 2025 at 09:22:59PM +0800, shejialuo wrote:\n> On Thu, Mar 06, 2025 at 04:08:39PM +0100, Patrick Steinhardt wrote:\n> > diff --git a/refs.c b/refs.c\n> > index 5a9b0f2fa1e..eaf41421f50 100644\n> > --- a/refs.c\n> > +++ b/refs.c\n> > @@ -2514,6 +2517,14 @@ int refs_verify_refnames_available(struct ref_store *refs,\n> > \t\t\tif (skip && string_list_has_string(skip, dirname.buf))\n> > \t\t\t\tcontinue;\n> > \n> > +\t\t\t/*\n> > +\t\t\t * If we've already seen the directory we don't need to\n> > +\t\t\t * process it again. Skip it to avoid checking checking\n> > +\t\t\t * common prefixes like \"refs/heads/\" repeatedly.\n> > +\t\t\t */\n> > +\t\t\tif (!strset_add(&dirnames, dirname.buf))\n> > +\t\t\t\tcontinue;\n> > +\n> \n> Reading here, I think we should not sort the refnames for \"reftable\"\n> backend. Anyway, really a nice job for optimizing the speed.\n\nAgreed, it's unnecessary for this optimization.\n\nPatrick\n"},{"id":"514071","messageId":"20250312-pks-update-ref-optimization-v6-0-f778e0414f55@pks.im","threadId":"62965","inReplyTo":"20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im","subject":"[PATCH v6 00/16] refs: batch refname availability checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:56:06Z","receivedAt":"2025-03-12T15:56:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis patch series has been inspired by brian's report that the reftable\nbackend is significantly slower when writing many references compared to\nthe files backend. As explained in that thread, the underlying issue is\nthe design of tombstone references: when we first delete all references\nin a repository and then recreate them, we still have all the tombstones\nand thus we need to churn through all of them to figure out that they\nhave been deleted in the first place. The files backend does not have\nthis issue.\n\nI consider the benchmark itself to be kind of broken, as it stems from\nus deleting all refs and then recreating them. And if you pack refs in\nbetween then the \"reftable\" backend outperforms the \"files\" backend.\n\nBut there are a couple of opportunities here anyway. While we cannot\nmake the underlying issue of tombstones being less efficient go away,\nthis has prompted me to have a deeper look at where we spend all the\ntime. There are three ideas in this series:\n\n - git-update-ref(1) performs ambiguity checks for any full-size object\n ID, which triggers a lot of reads. This is somewhat pointless though\n given that the manpage explicitly points out that the command is\n about object IDs, even though it does know to parse refs. But being\n part of plumbing, emitting the warning here does not make a ton of\n sense, and favoring object IDs over references in these cases is the\n obvious thing to do anyway.\n\n - For each ref \"refs/heads/bar\", we need to verify that neither\n \"refs/heads\" nor \"refs\" exists. This was repeated for every refname,\n but because most refnames use common prefixes this made us re-check\n a lot of prefixes. This is addressed by using a `strset` of already\n checked prefixes.\n\n - For each ref \"refs/heads/bar\", we need to verify that no ref\n \"refs/heads/bar/*\" exists. We always created a new ref iterator for\n this check, which requires us to discard all internal state and then\n recreate it. The reftable library has already been refactored though\n to have reseekable iterators, so we backfill this functionality to\n all the other iterators and then reuse the iterator.\n\nWith the (somewhat broken) benchmark we see a small speedup with the\n\"files\" backend:\n\n Benchmark 1: update-ref (refformat = files, revision = master)\n Time (mean ± σ): 234.4 ms ± 1.9 ms [User: 75.6 ms, System: 157.2 ms]\n Range (min … max): 232.2 ms … 236.9 ms 10 runs\n\n Benchmark 2: update-ref (refformat = files, revision = HEAD)\n Time (mean ± σ): 184.2 ms ± 2.0 ms [User: 62.8 ms, System: 119.9 ms]\n Range (min … max): 181.1 ms … 187.0 ms 10 runs\n\n Summary\n update-ref (refformat = files, revision = HEAD) ran\n 1.27 ± 0.02 times faster than update-ref (refformat = files, revision = master)\n\nAnd a huge speedup with the \"reftable\" backend:\n\n Benchmark 1: update-ref (refformat = reftable, revision = master)\n Time (mean ± σ): 16.852 s ± 0.061 s [User: 16.754 s, System: 0.059 s]\n Range (min … max): 16.785 s … 16.982 s 10 runs\n\n Benchmark 2: update-ref (refformat = reftable, revision = HEAD)\n Time (mean ± σ): 2.230 s ± 0.009 s [User: 2.192 s, System: 0.029 s]\n Range (min … max): 2.215 s … 2.244 s 10 runs\n\n Summary\n update-ref (refformat = reftable, revision = HEAD) ran\n 7.56 ± 0.04 times faster than update-ref (refformat = reftable, revision = master)\n\nWe're still not up to speed with the \"files\" backend, but considerably\nbetter. Given that this is an extreme edge case and not reflective of\nthe general case I'm okay with this result for now.\n\nBut more importantly, this refactoring also has a positive effect when\nupdating references in a repository with preexisting refs, which I\nconsider to be the more realistic scenario. The following benchmark\ncreates 10k refs with 100k preexisting refs.\n\nWith the \"files\" backend we see a modest improvement:\n\n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = master)\n Time (mean ± σ): 478.4 ms ± 11.9 ms [User: 96.7 ms, System: 379.6 ms]\n Range (min … max): 465.4 ms … 496.6 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 388.5 ms ± 10.3 ms [User: 52.0 ms, System: 333.8 ms]\n Range (min … max): 376.5 ms … 403.1 ms 10 runs\n\n Summary\n update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.23 ± 0.04 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = master)\n\nBut with the \"reftable\" backend we see an almost 5x improvement, where\nit's now ~15x faster than the \"files\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = master)\n Time (mean ± σ): 153.9 ms ± 2.0 ms [User: 96.5 ms, System: 56.6 ms]\n Range (min … max): 150.5 ms … 158.4 ms 18 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 32.2 ms ± 1.2 ms [User: 27.6 ms, System: 4.3 ms]\n Range (min … max): 29.8 ms … 38.6 ms 71 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 4.78 ± 0.19 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = master)\n\nThe series is structured as follows:\n\n - Patches 1 to 4 implement the logic to skip ambiguity checks in\n git-update-ref(1).\n\n - Patch 5 to 8 introduce batched checks.\n\n - Patch 9 deduplicates the ref prefix checks.\n\n - Patch 10 to 16 implement the infrastructure to reseek iterators.\n\n - Patch 17 starts to reuse iterators for nested ref checks.\n\nChanges in v2:\n - Point out why we also have to touch up the `dir_iterator`.\n - Fix up the comment explaining `ITER_DONE`.\n - Fix up comments that show usage patterns of the ref and dir iterator\n interfaces.\n - Start batching availability checks in the \"files\" backend, as well.\n - Improve the commit message that drops the ambiguity check so that we\n also point to 25fba78d36b (cat-file: disable object/refname\n ambiguity check for batch mode, 2013-07-12).\n - Link to v1: https://lore.kernel.org/r/20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im\n\nChanges in v3:\n - Fix one case where we didn't skip ambiguity checks in\n git-update-ref(1).\n - Document better that only the prefix can change on reseeking\n iterators. Other internal state will remain the same.\n - Fix a memory leak in the ref-cache iterator.\n - Don't ignore errors returned by `packed_ref_iterator_seek()`.\n - Link to v2: https://lore.kernel.org/r/20250219-pks-update-ref-optimization-v2-0-e696e7220b22@pks.im\n\nChanges in v4:\n - A couple of clarifications in the commit message that disabled\n ambiguity warnings.\n - Link to v3: https://lore.kernel.org/r/20250225-pks-update-ref-optimization-v3-0-77c3687cda75@pks.im\n\nChanges in v5:\n - Improve a couple of commit messages.\n - Align `GET_OID_*` flag values.\n - Link to v4: https://lore.kernel.org/r/20250228-pks-update-ref-optimization-v4-0-6425c04268b5@pks.im\n\nChanges in v6:\n - Use `for_each_string_list()` instead of manually iterating through\n the string list.\n - Stop sorting refs passed to `refs_verify_refnames_available()`.\n - Revive a comment that has been deleted during one of the\n refactorings.\n - Link to v5: https://lore.kernel.org/r/20250306-pks-update-ref-optimization-v5-0-dcb2ee037e97@pks.im\n\nThanks!\n\nPatrick\n\n[1]: \n\n---\nPatrick Steinhardt (16):\n object-name: introduce `repo_get_oid_with_flags()`\n object-name: allow skipping ambiguity checks in `get_oid()` family\n builtin/update-ref: skip ambiguity checks when parsing object IDs\n refs: introduce function to batch refname availability checks\n refs/reftable: batch refname availability checks\n refs/files: batch refname availability checks for normal transactions\n refs/files: batch refname availability checks for initial transactions\n refs: stop re-verifying common prefixes for availability\n refs/iterator: separate lifecycle from iteration\n refs/iterator: provide infrastructure to re-seek iterators\n refs/iterator: implement seeking for merged iterators\n refs/iterator: implement seeking for reftable iterators\n refs/iterator: implement seeking for ref-cache iterators\n refs/iterator: implement seeking for packed-ref iterators\n refs/iterator: implement seeking for files iterators\n refs: reuse iterators when determining refname availability\n\n builtin/clone.c | 2 +\n builtin/update-ref.c | 15 ++--\n dir-iterator.c | 24 +++---\n dir-iterator.h | 11 +--\n hash.h | 23 +++---\n iterator.h | 2 +-\n object-name.c | 18 +++--\n object-name.h | 6 ++\n refs.c | 187 ++++++++++++++++++++++++++-----------------\n refs.h | 12 +++\n refs/debug.c | 20 +++--\n refs/files-backend.c | 117 +++++++++++++++++----------\n refs/iterator.c | 150 ++++++++++++++++++----------------\n refs/packed-backend.c | 92 ++++++++++++---------\n refs/ref-cache.c | 88 ++++++++++++--------\n refs/refs-internal.h | 53 +++++++-----\n refs/reftable-backend.c | 84 ++++++++++---------\n t/helper/test-dir-iterator.c | 1 +\n 18 files changed, 544 insertions(+), 361 deletions(-)\n\nRange-diff versus v5:\n\n 1: d23737dfca2 = 1: 07ebc2f03d9 object-name: introduce `repo_get_oid_with_flags()`\n 2: 33f4548d2c0 = 2: b722efbe3dc object-name: allow skipping ambiguity checks in `get_oid()` family\n 3: ea0234047ab = 3: a0171a61808 builtin/update-ref: skip ambiguity checks when parsing object IDs\n 4: 3dfef6655f3 ! 4: 91f0c75ac7b refs: introduce function to batch refname availability checks\n @@ refs.c: int ref_transaction_commit(struct ref_transaction *transaction,\n \tstruct strbuf referent = STRBUF_INIT;\n -\tstruct object_id oid;\n -\tunsigned int type;\n ++\tstruct string_list_item *item;\n \tint ret = -1;\n \n \t/*\n @@ refs.c: int refs_verify_refname_available(struct ref_store *refs,\n -\t\tint ignore_errno;\n -\t\t/* Expand dirname to the new prefix, not including the trailing slash: */\n -\t\tstrbuf_add(&dirname, refname + dirname.len, slash - refname - dirname.len);\n -+\tfor (size_t i = 0; i < refnames->nr; i++) {\n -+\t\tconst char *refname = refnames->items[i].string;\n ++\tfor_each_string_list_item(item, refnames) {\n ++\t\tconst char *refname = item->string;\n +\t\tconst char *extra_refname;\n +\t\tstruct object_id oid;\n +\t\tunsigned int type;\n 5: 671ef35018c ! 5: 8caf4f99ecd refs/reftable: batch refname availability checks\n @@ refs/reftable-backend.c: static int reftable_be_transaction_prepare(struct ref_s\n \t\t}\n \t}\n \n -+\tstring_list_sort(&refnames_to_check);\n +\tret = refs_verify_refnames_available(ref_store, &refnames_to_check, &affected_refnames, NULL,\n +\t\t\t\t\t transaction->flags & REF_TRANSACTION_FLAG_INITIAL,\n +\t\t\t\t\t err);\n 6: 4ff7d2b821e ! 6: a10538db712 refs/files: batch refname availability checks for normal transactions\n @@ refs/files-backend.c: static int lock_raw_ref(struct files_ref_store *refs,\n -\t\t\tret = TRANSACTION_NAME_CONFLICT;\n -\t\t\tgoto error_return;\n -\t\t}\n -+\t\tstring_list_insert(refnames_to_check, refname);\n ++\t\tstring_list_append(refnames_to_check, refname);\n \t}\n \n \tret = 0;\n 7: 901664612bd = 7: dccee563e60 refs/files: batch refname availability checks for initial transactions\n 8: 1ddc9372a77 ! 8: c7b2ce15afe refs: stop re-verifying common prefixes for availability\n @@ Commit message\n \n ## refs.c ##\n @@ refs.c: int refs_verify_refnames_available(struct ref_store *refs,\n - {\n \tstruct strbuf dirname = STRBUF_INIT;\n \tstruct strbuf referent = STRBUF_INIT;\n + \tstruct string_list_item *item;\n +\tstruct strset dirnames;\n \tint ret = -1;\n \n @@ refs.c: int refs_verify_refnames_available(struct ref_store *refs,\n \n +\tstrset_init(&dirnames);\n +\n - \tfor (size_t i = 0; i < refnames->nr; i++) {\n - \t\tconst char *refname = refnames->items[i].string;\n + \tfor_each_string_list_item(item, refnames) {\n + \t\tconst char *refname = item->string;\n \t\tconst char *extra_refname;\n @@ refs.c: int refs_verify_refnames_available(struct ref_store *refs,\n \t\t\tif (skip && string_list_has_string(skip, dirname.buf))\n 9: b3cd0baed29 ! 9: 22730c09f01 refs/iterator: separate lifecycle from iteration\n @@ iterator.h\n \n ## refs.c ##\n @@ refs.c: int refs_verify_refnames_available(struct ref_store *refs,\n - {\n \tstruct strbuf dirname = STRBUF_INIT;\n \tstruct strbuf referent = STRBUF_INIT;\n + \tstruct string_list_item *item;\n +\tstruct ref_iterator *iter = NULL;\n \tstruct strset dirnames;\n \tint ret = -1;\n @@ refs/iterator.c: static int prefix_ref_iterator_advance(struct ref_iterator *ref\n -\t\t\tok = ref_iterator_abort(iter->iter0);\n -\t\t\tbreak;\n -\t\t}\n ++\t\t/*\n ++\t\t * As the source iterator is ordered, we\n ++\t\t * can stop the iteration as soon as we see a\n ++\t\t * refname that comes after the prefix:\n ++\t\t */\n +\t\tif (cmp > 0)\n +\t\t\treturn ITER_DONE;\n \n10: c1ceade2ba2 = 10: e65b53f5b52 refs/iterator: provide infrastructure to re-seek iterators\n11: 77642dfdade = 11: 66b8aa94cd7 refs/iterator: implement seeking for merged iterators\n12: fae0048ced3 = 12: 42192c4d4b6 refs/iterator: implement seeking for reftable iterators\n13: ff9384ee7b0 = 13: 76078359277 refs/iterator: implement seeking for ref-cache iterators\n14: 105a01b3a59 = 14: 83d6f90c90a refs/iterator: implement seeking for packed-ref iterators\n15: 1bf2e76b4c8 = 15: b66bf0a9966 refs/iterator: implement seeking for files iterators\n16: 24548c33e5c = 16: 67457a1c03f refs: reuse iterators when determining refname availability\n\n---\nbase-commit: e2067b49ecaef9b7f51a17ce251f9207f72ef52d\nchange-id: 20250217-pks-update-ref-optimization-15c795e66e2b\n\n"},{"id":"514072","messageId":"20250312-pks-update-ref-optimization-v6-1-f778e0414f55@pks.im","threadId":"62965","inReplyTo":"20250312-pks-update-ref-optimization-v6-0-f778e0414f55@pks.im","subject":"[PATCH v6 01/16] object-name: introduce `repo_get_oid_with_flags()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:56:07Z","receivedAt":"2025-03-12T15:56:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Introduce a new function `repo_get_oid_with_flags()`. This function\nbehaves the same as `repo_get_oid()`, except that it takes an extra\n`flags` parameter that it ends up passing to `get_oid_with_context()`.\n\nThis function will be used in a subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n object-name.c | 14 ++++++++------\n object-name.h | 6 ++++++\n 2 files changed, 14 insertions(+), 6 deletions(-)\n\ndiff --git a/object-name.c b/object-name.c\nindex 945d5bdef25..233f3f861e3 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -1794,18 +1794,20 @@ void object_context_release(struct object_context *ctx)\n \tstrbuf_release(&ctx->symlink_path);\n }\n \n-/*\n- * This is like \"get_oid_basic()\", except it allows \"object ID expressions\",\n- * notably \"xyz^\" for \"parent of xyz\"\n- */\n-int repo_get_oid(struct repository *r, const char *name, struct object_id *oid)\n+int repo_get_oid_with_flags(struct repository *r, const char *name,\n+\t\t\t struct object_id *oid, unsigned flags)\n {\n \tstruct object_context unused;\n-\tint ret = get_oid_with_context(r, name, 0, oid, &unused);\n+\tint ret = get_oid_with_context(r, name, flags, oid, &unused);\n \tobject_context_release(&unused);\n \treturn ret;\n }\n \n+int repo_get_oid(struct repository *r, const char *name, struct object_id *oid)\n+{\n+\treturn repo_get_oid_with_flags(r, name, oid, 0);\n+}\n+\n /*\n * This returns a non-zero value if the string (built using printf\n * format and the given arguments) is not a valid object.\ndiff --git a/object-name.h b/object-name.h\nindex 8dba4a47a47..cda4934cd5f 100644\n--- a/object-name.h\n+++ b/object-name.h\n@@ -51,6 +51,12 @@ void strbuf_repo_add_unique_abbrev(struct strbuf *sb, struct repository *repo,\n void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n \t\t\t int abbrev_len);\n \n+/*\n+ * This is like \"get_oid_basic()\", except it allows \"object ID expressions\",\n+ * notably \"xyz^\" for \"parent of xyz\". Accepts GET_OID_* flags.\n+ */\n+int repo_get_oid_with_flags(struct repository *r, const char *str,\n+\t\t\t struct object_id *oid, unsigned flags);\n int repo_get_oid(struct repository *r, const char *str, struct object_id *oid);\n __attribute__((format (printf, 2, 3)))\n int get_oidf(struct object_id *oid, const char *fmt, ...);\n\n-- \n2.49.0.rc2.394.gf6994c5077.dirty\n\n"},{"id":"514073","messageId":"20250312-pks-update-ref-optimization-v6-2-f778e0414f55@pks.im","threadId":"62965","inReplyTo":"20250312-pks-update-ref-optimization-v6-0-f778e0414f55@pks.im","subject":"[PATCH v6 02/16] object-name: allow skipping ambiguity checks in `get_oid()` family","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:56:08Z","receivedAt":"2025-03-12T15:56:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When reading an object ID via `get_oid_basic()` or any of its related\nfunctions we perform a check whether the object ID is ambiguous, which\ncan be the case when a reference with the same name exists. While the\ncheck is generally helpful, there are cases where it only adds to the\nruntime overhead without providing much of a benefit.\n\nAdd a new flag that allows us to disable the check. The flag will be\nused in a subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n hash.h | 23 ++++++++++++-----------\n object-name.c | 4 +++-\n 2 files changed, 15 insertions(+), 12 deletions(-)\n\ndiff --git a/hash.h b/hash.h\nindex 4367acfec50..5e3c462dc5e 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -193,17 +193,18 @@ struct object_id {\n \tint algo;\t/* XXX requires 4-byte alignment */\n };\n \n-#define GET_OID_QUIETLY 01\n-#define GET_OID_COMMIT 02\n-#define GET_OID_COMMITTISH 04\n-#define GET_OID_TREE 010\n-#define GET_OID_TREEISH 020\n-#define GET_OID_BLOB 040\n-#define GET_OID_FOLLOW_SYMLINKS 0100\n-#define GET_OID_RECORD_PATH 0200\n-#define GET_OID_ONLY_TO_DIE 04000\n-#define GET_OID_REQUIRE_PATH 010000\n-#define GET_OID_HASH_ANY 020000\n+#define GET_OID_QUIETLY 01\n+#define GET_OID_COMMIT 02\n+#define GET_OID_COMMITTISH 04\n+#define GET_OID_TREE 010\n+#define GET_OID_TREEISH 020\n+#define GET_OID_BLOB 040\n+#define GET_OID_FOLLOW_SYMLINKS 0100\n+#define GET_OID_RECORD_PATH 0200\n+#define GET_OID_ONLY_TO_DIE 04000\n+#define GET_OID_REQUIRE_PATH 010000\n+#define GET_OID_HASH_ANY 020000\n+#define GET_OID_SKIP_AMBIGUITY_CHECK 040000\n \n #define GET_OID_DISAMBIGUATORS \\\n \t(GET_OID_COMMIT | GET_OID_COMMITTISH | \\\ndiff --git a/object-name.c b/object-name.c\nindex 233f3f861e3..85444dbb15b 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -961,7 +961,9 @@ static int get_oid_basic(struct repository *r, const char *str, int len,\n \tint fatal = !(flags & GET_OID_QUIETLY);\n \n \tif (len == r->hash_algo->hexsz && !get_oid_hex(str, oid)) {\n-\t\tif (repo_settings_get_warn_ambiguous_refs(r) && warn_on_object_refname_ambiguity) {\n+\t\tif (!(flags & GET_OID_SKIP_AMBIGUITY_CHECK) &&\n+\t\t repo_settings_get_warn_ambiguous_refs(r) &&\n+\t\t warn_on_object_refname_ambiguity) {\n \t\t\trefs_found = repo_dwim_ref(r, str, len, &tmp_oid, &real_ref, 0);\n \t\t\tif (refs_found > 0) {\n \t\t\t\twarning(warn_msg, len, str);\n\n-- \n2.49.0.rc2.394.gf6994c5077.dirty\n\n"},{"id":"514074","messageId":"20250312-pks-update-ref-optimization-v6-3-f778e0414f55@pks.im","threadId":"62965","inReplyTo":"20250312-pks-update-ref-optimization-v6-0-f778e0414f55@pks.im","subject":"[PATCH v6 03/16] builtin/update-ref: skip ambiguity checks when parsing object IDs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:56:09Z","receivedAt":"2025-03-12T15:56:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Most of the commands in git-update-ref(1) accept an old and/or new\nobject ID to update a specific reference to. These object IDs get parsed\nvia `repo_get_oid()`, which not only handles plain object IDs, but also\nthose that have a suffix like \"~\" or \"^2\". More surprisingly though, it\neven knows to resolve arbitrary revisions, despite the fact that its\nmanpage does not mention this fact even once.\n\nOne consequence of this is that we also check for ambiguous references:\nwhen parsing a full object ID where the DWIM mechanism would also cause\nus to resolve it as a branch, we'd end up printing a warning. While this\ncheck makes sense to have in general, it is arguably less useful in the\ncontext of git-update-ref(1). This is due to multiple reasons:\n\n - The manpage is explicitly structured around object IDs. So if we see\n a fully blown object ID, the intent should be quite clear in\n general.\n\n - The command is part of our plumbing layer and not a tool that users\n would generally use in interactive workflows. As such, the warning\n will likely not be visible to anybody in the first place.\n\n - Users can and should use the fully-qualified refname in case there\n is any potential for ambiguity. And given that this command is part\n of our plumbing layer, one should always try to be as defensive as\n possible and use fully-qualified refnames.\n\nFurthermore, this check can be quite expensive when updating lots of\nreferences via `--stdin`, because we try to read multiple references per\nobject ID that we parse according to the DWIM rules. This effect can be\nseen both with the \"files\" and \"reftable\" backend.\n\nThe issue is not unique to git-update-ref(1), but was also an issue in\ngit-cat-file(1), where it was addressed by disabling the ambiguity check\nin 25fba78d36b (cat-file: disable object/refname ambiguity check for\nbatch mode, 2013-07-12).\n\nDisable the warning in git-update-ref(1), which provides a significant\nspeedup with both backends. The user-visible outcome is unchanged even\nwhen ambiguity exists, except that we don't show the warning anymore.\n\nThe following benchmark creates 10000 new references with a 100000\npreexisting refs with the \"files\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 467.3 ms ± 5.1 ms [User: 100.0 ms, System: 365.1 ms]\n Range (min … max): 461.9 ms … 479.3 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 394.1 ms ± 5.8 ms [User: 63.3 ms, System: 327.6 ms]\n Range (min … max): 384.9 ms … 405.7 ms 10 runs\n\n Summary\n update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.19 ± 0.02 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n\nAnd with the \"reftable\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 146.9 ms ± 2.2 ms [User: 90.4 ms, System: 56.0 ms]\n Range (min … max): 142.7 ms … 150.8 ms 19 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 63.2 ms ± 1.1 ms [User: 41.0 ms, System: 21.8 ms]\n Range (min … max): 61.1 ms … 66.6 ms 41 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 2.32 ± 0.05 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n\nNote that the absolute improvement with both backends is roughly in the\nsame ballpark, but the relative improvement for the \"reftable\" backend\nis more significant because writing the new table to disk is faster in\nthe first place.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/update-ref.c | 15 ++++++++++-----\n 1 file changed, 10 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/update-ref.c b/builtin/update-ref.c\nindex 4d35bdc4b4b..1d541e13ade 100644\n--- a/builtin/update-ref.c\n+++ b/builtin/update-ref.c\n@@ -179,7 +179,8 @@ static int parse_next_oid(const char **next, const char *end,\n \t\t(*next)++;\n \t\t*next = parse_arg(*next, &arg);\n \t\tif (arg.len) {\n-\t\t\tif (repo_get_oid(the_repository, arg.buf, oid))\n+\t\t\tif (repo_get_oid_with_flags(the_repository, arg.buf, oid,\n+\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\t\tgoto invalid;\n \t\t} else {\n \t\t\t/* Without -z, an empty value means all zeros: */\n@@ -197,7 +198,8 @@ static int parse_next_oid(const char **next, const char *end,\n \t\t*next += arg.len;\n \n \t\tif (arg.len) {\n-\t\t\tif (repo_get_oid(the_repository, arg.buf, oid))\n+\t\t\tif (repo_get_oid_with_flags(the_repository, arg.buf, oid,\n+\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\t\tgoto invalid;\n \t\t} else if (flags & PARSE_SHA1_ALLOW_EMPTY) {\n \t\t\t/* With -z, treat an empty value as all zeros: */\n@@ -299,7 +301,8 @@ static void parse_cmd_symref_update(struct ref_transaction *transaction,\n \t\t\tdie(\"symref-update %s: expected old value\", refname);\n \n \t\tif (!strcmp(old_arg, \"oid\")) {\n-\t\t\tif (repo_get_oid(the_repository, old_target, &old_oid))\n+\t\t\tif (repo_get_oid_with_flags(the_repository, old_target, &old_oid,\n+\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\t\tdie(\"symref-update %s: invalid oid: %s\", refname, old_target);\n \n \t\t\thave_old_oid = 1;\n@@ -772,7 +775,8 @@ int cmd_update_ref(int argc,\n \t\trefname = argv[0];\n \t\tvalue = argv[1];\n \t\toldval = argv[2];\n-\t\tif (repo_get_oid(the_repository, value, &oid))\n+\t\tif (repo_get_oid_with_flags(the_repository, value, &oid,\n+\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\tdie(\"%s: not a valid SHA1\", value);\n \t}\n \n@@ -783,7 +787,8 @@ int cmd_update_ref(int argc,\n \t\t\t * must not already exist:\n \t\t\t */\n \t\t\toidclr(&oldoid, the_repository->hash_algo);\n-\t\telse if (repo_get_oid(the_repository, oldval, &oldoid))\n+\t\telse if (repo_get_oid_with_flags(the_repository, oldval, &oldoid,\n+\t\t\t\t\t\t GET_OID_SKIP_AMBIGUITY_CHECK))\n \t\t\tdie(\"%s: not a valid old SHA1\", oldval);\n \t}\n \n\n-- \n2.49.0.rc2.394.gf6994c5077.dirty\n\n"},{"id":"514075","messageId":"20250312-pks-update-ref-optimization-v6-4-f778e0414f55@pks.im","threadId":"62965","inReplyTo":"20250312-pks-update-ref-optimization-v6-0-f778e0414f55@pks.im","subject":"[PATCH v6 04/16] refs: introduce function to batch refname availability checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:56:10Z","receivedAt":"2025-03-12T15:56:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `refs_verify_refname_available()` functions checks whether a\nreference update can be committed or whether it would conflict with\neither a prefix or suffix thereof. This function needs to be called once\nper reference that one wants to check, which requires us to redo a\ncouple of checks every time the function is called.\n\nIntroduce a new function `refs_verify_refnames_available()` that does\nthe same, but for a list of references. For now, the new function uses\nthe exact same implementation, except that we loop through all refnames\nprovided by the caller. This will be tuned in subsequent commits.\n\nThe existing `refs_verify_refname_available()` function is reimplemented\non top of the new function. As such, the diff is best viewed with the\n`--ignore-space-change option`.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs.c | 170 +++++++++++++++++++++++++++++++++++++----------------------------\n refs.h | 12 +++++\n 2 files changed, 110 insertions(+), 72 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex f4094a326a9..d91a2184e06 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2467,19 +2467,16 @@ int ref_transaction_commit(struct ref_transaction *transaction,\n \treturn ret;\n }\n \n-int refs_verify_refname_available(struct ref_store *refs,\n-\t\t\t\t const char *refname,\n-\t\t\t\t const struct string_list *extras,\n-\t\t\t\t const struct string_list *skip,\n-\t\t\t\t unsigned int initial_transaction,\n-\t\t\t\t struct strbuf *err)\n+int refs_verify_refnames_available(struct ref_store *refs,\n+\t\t\t\t const struct string_list *refnames,\n+\t\t\t\t const struct string_list *extras,\n+\t\t\t\t const struct string_list *skip,\n+\t\t\t\t unsigned int initial_transaction,\n+\t\t\t\t struct strbuf *err)\n {\n-\tconst char *slash;\n-\tconst char *extra_refname;\n \tstruct strbuf dirname = STRBUF_INIT;\n \tstruct strbuf referent = STRBUF_INIT;\n-\tstruct object_id oid;\n-\tunsigned int type;\n+\tstruct string_list_item *item;\n \tint ret = -1;\n \n \t/*\n@@ -2489,79 +2486,91 @@ int refs_verify_refname_available(struct ref_store *refs,\n \n \tassert(err);\n \n-\tstrbuf_grow(&dirname, strlen(refname) + 1);\n-\tfor (slash = strchr(refname, '/'); slash; slash = strchr(slash + 1, '/')) {\n-\t\t/*\n-\t\t * Just saying \"Is a directory\" when we e.g. can't\n-\t\t * lock some multi-level ref isn't very informative,\n-\t\t * the user won't be told *what* is a directory, so\n-\t\t * let's not use strerror() below.\n-\t\t */\n-\t\tint ignore_errno;\n-\t\t/* Expand dirname to the new prefix, not including the trailing slash: */\n-\t\tstrbuf_add(&dirname, refname + dirname.len, slash - refname - dirname.len);\n+\tfor_each_string_list_item(item, refnames) {\n+\t\tconst char *refname = item->string;\n+\t\tconst char *extra_refname;\n+\t\tstruct object_id oid;\n+\t\tunsigned int type;\n+\t\tconst char *slash;\n+\n+\t\tstrbuf_reset(&dirname);\n+\n+\t\tfor (slash = strchr(refname, '/'); slash; slash = strchr(slash + 1, '/')) {\n+\t\t\t/*\n+\t\t\t * Just saying \"Is a directory\" when we e.g. can't\n+\t\t\t * lock some multi-level ref isn't very informative,\n+\t\t\t * the user won't be told *what* is a directory, so\n+\t\t\t * let's not use strerror() below.\n+\t\t\t */\n+\t\t\tint ignore_errno;\n+\n+\t\t\t/* Expand dirname to the new prefix, not including the trailing slash: */\n+\t\t\tstrbuf_add(&dirname, refname + dirname.len, slash - refname - dirname.len);\n+\n+\t\t\t/*\n+\t\t\t * We are still at a leading dir of the refname (e.g.,\n+\t\t\t * \"refs/foo\"; if there is a reference with that name,\n+\t\t\t * it is a conflict, *unless* it is in skip.\n+\t\t\t */\n+\t\t\tif (skip && string_list_has_string(skip, dirname.buf))\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (!initial_transaction &&\n+\t\t\t !refs_read_raw_ref(refs, dirname.buf, &oid, &referent,\n+\t\t\t\t\t &type, &ignore_errno)) {\n+\t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n+\t\t\t\t\t dirname.buf, refname);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\n+\t\t\tif (extras && string_list_has_string(extras, dirname.buf)) {\n+\t\t\t\tstrbuf_addf(err, _(\"cannot process '%s' and '%s' at the same time\"),\n+\t\t\t\t\t refname, dirname.buf);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\t\t}\n \n \t\t/*\n-\t\t * We are still at a leading dir of the refname (e.g.,\n-\t\t * \"refs/foo\"; if there is a reference with that name,\n-\t\t * it is a conflict, *unless* it is in skip.\n+\t\t * We are at the leaf of our refname (e.g., \"refs/foo/bar\").\n+\t\t * There is no point in searching for a reference with that\n+\t\t * name, because a refname isn't considered to conflict with\n+\t\t * itself. But we still need to check for references whose\n+\t\t * names are in the \"refs/foo/bar/\" namespace, because they\n+\t\t * *do* conflict.\n \t\t */\n-\t\tif (skip && string_list_has_string(skip, dirname.buf))\n-\t\t\tcontinue;\n+\t\tstrbuf_addstr(&dirname, refname + dirname.len);\n+\t\tstrbuf_addch(&dirname, '/');\n+\n+\t\tif (!initial_transaction) {\n+\t\t\tstruct ref_iterator *iter;\n+\t\t\tint ok;\n+\n+\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n+\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n+\t\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n+\t\t\t\tif (skip &&\n+\t\t\t\t string_list_has_string(skip, iter->refname))\n+\t\t\t\t\tcontinue;\n+\n+\t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n+\t\t\t\t\t iter->refname, refname);\n+\t\t\t\tref_iterator_abort(iter);\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \n-\t\tif (!initial_transaction &&\n-\t\t !refs_read_raw_ref(refs, dirname.buf, &oid, &referent,\n-\t\t\t\t &type, &ignore_errno)) {\n-\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n-\t\t\t\t dirname.buf, refname);\n-\t\t\tgoto cleanup;\n+\t\t\tif (ok != ITER_DONE)\n+\t\t\t\tBUG(\"error while iterating over references\");\n \t\t}\n \n-\t\tif (extras && string_list_has_string(extras, dirname.buf)) {\n+\t\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n+\t\tif (extra_refname) {\n \t\t\tstrbuf_addf(err, _(\"cannot process '%s' and '%s' at the same time\"),\n-\t\t\t\t refname, dirname.buf);\n+\t\t\t\t refname, extra_refname);\n \t\t\tgoto cleanup;\n \t\t}\n \t}\n \n-\t/*\n-\t * We are at the leaf of our refname (e.g., \"refs/foo/bar\").\n-\t * There is no point in searching for a reference with that\n-\t * name, because a refname isn't considered to conflict with\n-\t * itself. But we still need to check for references whose\n-\t * names are in the \"refs/foo/bar/\" namespace, because they\n-\t * *do* conflict.\n-\t */\n-\tstrbuf_addstr(&dirname, refname + dirname.len);\n-\tstrbuf_addch(&dirname, '/');\n-\n-\tif (!initial_transaction) {\n-\t\tstruct ref_iterator *iter;\n-\t\tint ok;\n-\n-\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n-\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n-\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n-\t\t\tif (skip &&\n-\t\t\t string_list_has_string(skip, iter->refname))\n-\t\t\t\tcontinue;\n-\n-\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n-\t\t\t\t iter->refname, refname);\n-\t\t\tref_iterator_abort(iter);\n-\t\t\tgoto cleanup;\n-\t\t}\n-\n-\t\tif (ok != ITER_DONE)\n-\t\t\tBUG(\"error while iterating over references\");\n-\t}\n-\n-\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n-\tif (extra_refname)\n-\t\tstrbuf_addf(err, _(\"cannot process '%s' and '%s' at the same time\"),\n-\t\t\t refname, extra_refname);\n-\telse\n-\t\tret = 0;\n+\tret = 0;\n \n cleanup:\n \tstrbuf_release(&referent);\n@@ -2569,6 +2578,23 @@ int refs_verify_refname_available(struct ref_store *refs,\n \treturn ret;\n }\n \n+int refs_verify_refname_available(struct ref_store *refs,\n+\t\t\t\t const char *refname,\n+\t\t\t\t const struct string_list *extras,\n+\t\t\t\t const struct string_list *skip,\n+\t\t\t\t unsigned int initial_transaction,\n+\t\t\t\t struct strbuf *err)\n+{\n+\tstruct string_list_item item = { .string = (char *) refname };\n+\tstruct string_list refnames = {\n+\t\t.items = &item,\n+\t\t.nr = 1,\n+\t};\n+\n+\treturn refs_verify_refnames_available(refs, &refnames, extras, skip,\n+\t\t\t\t\t initial_transaction, err);\n+}\n+\n struct do_for_each_reflog_help {\n \teach_reflog_fn *fn;\n \tvoid *cb_data;\ndiff --git a/refs.h b/refs.h\nindex a0cdd99250e..185aed5a461 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -124,6 +124,18 @@ int refs_verify_refname_available(struct ref_store *refs,\n \t\t\t\t unsigned int initial_transaction,\n \t\t\t\t struct strbuf *err);\n \n+/*\n+ * Same as `refs_verify_refname_available()`, but checking for a list of\n+ * refnames instead of only a single item. This is more efficient in the case\n+ * where one needs to check multiple refnames.\n+ */\n+int refs_verify_refnames_available(struct ref_store *refs,\n+\t\t\t\t const struct string_list *refnames,\n+\t\t\t\t const struct string_list *extras,\n+\t\t\t\t const struct string_list *skip,\n+\t\t\t\t unsigned int initial_transaction,\n+\t\t\t\t struct strbuf *err);\n+\n int refs_ref_exists(struct ref_store *refs, const char *refname);\n \n int should_autocreate_reflog(enum log_refs_config log_all_ref_updates,\n\n-- \n2.49.0.rc2.394.gf6994c5077.dirty\n\n"},{"id":"514076","messageId":"20250312-pks-update-ref-optimization-v6-5-f778e0414f55@pks.im","threadId":"62965","inReplyTo":"20250312-pks-update-ref-optimization-v6-0-f778e0414f55@pks.im","subject":"[PATCH v6 05/16] refs/reftable: batch refname availability checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:56:11Z","receivedAt":"2025-03-12T15:56:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Refactor the \"reftable\" backend to batch the availability check for\nrefnames. This does not yet have an effect on performance as\n`refs_verify_refnames_available()` effectively still performs the\navailability check for each refname individually. But this will be\noptimized in subsequent commits, where we learn to optimize some parts\nof the logic when checking multiple refnames for availability.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 15 +++++++++------\n 1 file changed, 9 insertions(+), 6 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex d39a14c5a46..5c464b9d143 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -1069,6 +1069,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\treftable_be_downcast(ref_store, REF_STORE_WRITE|REF_STORE_MAIN, \"ref_transaction_prepare\");\n \tstruct strbuf referent = STRBUF_INIT, head_referent = STRBUF_INIT;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n \tstruct reftable_transaction_data *tx_data = NULL;\n \tstruct reftable_backend *be;\n \tstruct object_id head_oid;\n@@ -1224,12 +1225,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t\t * can output a proper error message instead of failing\n \t\t\t * at a later point.\n \t\t\t */\n-\t\t\tret = refs_verify_refname_available(ref_store, u->refname,\n-\t\t\t\t\t\t\t &affected_refnames, NULL,\n-\t\t\t\t\t\t\t transaction->flags & REF_TRANSACTION_FLAG_INITIAL,\n-\t\t\t\t\t\t\t err);\n-\t\t\tif (ret < 0)\n-\t\t\t\tgoto done;\n+\t\t\tstring_list_append(&refnames_to_check, u->refname);\n \n \t\t\t/*\n \t\t\t * There is no need to write the reference deletion\n@@ -1379,6 +1375,12 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \t\t}\n \t}\n \n+\tret = refs_verify_refnames_available(ref_store, &refnames_to_check, &affected_refnames, NULL,\n+\t\t\t\t\t transaction->flags & REF_TRANSACTION_FLAG_INITIAL,\n+\t\t\t\t\t err);\n+\tif (ret < 0)\n+\t\tgoto done;\n+\n \ttransaction->backend_data = tx_data;\n \ttransaction->state = REF_TRANSACTION_PREPARED;\n \n@@ -1394,6 +1396,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store,\n \tstring_list_clear(&affected_refnames, 0);\n \tstrbuf_release(&referent);\n \tstrbuf_release(&head_referent);\n+\tstring_list_clear(&refnames_to_check, 0);\n \n \treturn ret;\n }\n\n-- \n2.49.0.rc2.394.gf6994c5077.dirty\n\n"},{"id":"514077","messageId":"20250312-pks-update-ref-optimization-v6-6-f778e0414f55@pks.im","threadId":"62965","inReplyTo":"20250312-pks-update-ref-optimization-v6-0-f778e0414f55@pks.im","subject":"[PATCH v6 06/16] refs/files: batch refname availability checks for normal transactions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:56:12Z","receivedAt":"2025-03-12T15:56:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Same as the \"reftable\" backend that we have adapted in the preceding\ncommit to use batched refname availability checks we can also do so for\nthe \"files\" backend. Things are a bit more intricate here though, as we\ncall `refs_verify_refname_available()` in a set of different contexts:\n\n 1. `lock_raw_ref()` when it hits either EEXISTS or EISDIR when creating\n a new reference, mostly to create a nice, user-readable error\n message. This is nothing we have to care about too much, as we only\n hit this code path at most once when we hit a conflict.\n\n 2. `lock_raw_ref()` when it _could_ create the lockfile to check\n whether it is conflicting with any packed refs. In the general case,\n this code path will be hit once for every (successful) reference\n update.\n\n 3. `lock_ref_oid_basic()`, but it is only executed when copying or\n renaming references or when expiring reflogs. It will thus not be\n called in contexts where we have many references queued up.\n\n 4. `refs_refname_ref_available()`, but again only when copying or\n renaming references. It is thus not interesting due to the same\n reason as the previous case.\n\n 5. `files_transaction_finish_initial()`, which is only executed when\n creating a new repository or migrating references.\n\nSo out of these, only (2) and (5) are viable candidates to use the\nbatched checks.\n\nAdapt `lock_raw_ref()` accordingly by queueing up reference names that\nneed to be checked for availability and then checking them after we have\nprocessed all updates. This check is done before we (optionally) lock\nthe `packed-refs` file, which is somewhat flawed because it means that\nthe `packed-refs` could still change after the availability check and\nthus create an undetected conflict. But unconditionally locking the file\nwould change semantics that users are likely to rely on, so we keep the\ncurrent locking sequence intact, even if it's suboptmial.\n\nThe refactoring of `files_transaction_finish_initial()` will be done in\nthe next commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/files-backend.c | 42 +++++++++++++++++++++++++++++++-----------\n 1 file changed, 31 insertions(+), 11 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 29f08dced40..f798d8dae37 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -678,6 +678,7 @@ static void unlock_ref(struct ref_lock *lock)\n */\n static int lock_raw_ref(struct files_ref_store *refs,\n \t\t\tconst char *refname, int mustexist,\n+\t\t\tstruct string_list *refnames_to_check,\n \t\t\tconst struct string_list *extras,\n \t\t\tstruct ref_lock **lock_p,\n \t\t\tstruct strbuf *referent,\n@@ -855,16 +856,11 @@ static int lock_raw_ref(struct files_ref_store *refs,\n \t\t}\n \n \t\t/*\n-\t\t * If the ref did not exist and we are creating it,\n-\t\t * make sure there is no existing packed ref that\n-\t\t * conflicts with refname:\n+\t\t * If the ref did not exist and we are creating it, we have to\n+\t\t * make sure there is no existing packed ref that conflicts\n+\t\t * with refname. This check is deferred so that we can batch it.\n \t\t */\n-\t\tif (refs_verify_refname_available(\n-\t\t\t\t refs->packed_ref_store, refname,\n-\t\t\t\t extras, NULL, 0, err)) {\n-\t\t\tret = TRANSACTION_NAME_CONFLICT;\n-\t\t\tgoto error_return;\n-\t\t}\n+\t\tstring_list_append(refnames_to_check, refname);\n \t}\n \n \tret = 0;\n@@ -2569,6 +2565,7 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n \t\t\t struct ref_update *update,\n \t\t\t struct ref_transaction *transaction,\n \t\t\t const char *head_ref,\n+\t\t\t struct string_list *refnames_to_check,\n \t\t\t struct string_list *affected_refnames,\n \t\t\t struct strbuf *err)\n {\n@@ -2597,7 +2594,7 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n \t\tlock->count++;\n \t} else {\n \t\tret = lock_raw_ref(refs, update->refname, mustexist,\n-\t\t\t\t affected_refnames,\n+\t\t\t\t refnames_to_check, affected_refnames,\n \t\t\t\t &lock, &referent,\n \t\t\t\t &update->type, err);\n \t\tif (ret) {\n@@ -2811,6 +2808,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \tsize_t i;\n \tint ret = 0;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n \tchar *head_ref = NULL;\n \tint head_type;\n \tstruct files_transaction_backend_data *backend_data;\n@@ -2898,7 +2896,8 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\tstruct ref_update *update = transaction->updates[i];\n \n \t\tret = lock_ref_for_update(refs, update, transaction,\n-\t\t\t\t\t head_ref, &affected_refnames, err);\n+\t\t\t\t\t head_ref, &refnames_to_check,\n+\t\t\t\t\t &affected_refnames, err);\n \t\tif (ret)\n \t\t\tgoto cleanup;\n \n@@ -2930,6 +2929,26 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\t}\n \t}\n \n+\t/*\n+\t * Verify that none of the loose reference that we're about to write\n+\t * conflict with any existing packed references. Ideally, we'd do this\n+\t * check after the packed-refs are locked so that the file cannot\n+\t * change underneath our feet. But introducing such a lock now would\n+\t * probably do more harm than good as users rely on there not being a\n+\t * global lock with the \"files\" backend.\n+\t *\n+\t * Another alternative would be to do the check after the (optional)\n+\t * lock, but that would extend the time we spend in the globally-locked\n+\t * state.\n+\t *\n+\t * So instead, we accept the race for now.\n+\t */\n+\tif (refs_verify_refnames_available(refs->packed_ref_store, &refnames_to_check,\n+\t\t\t\t\t &affected_refnames, NULL, 0, err)) {\n+\t\tret = TRANSACTION_NAME_CONFLICT;\n+\t\tgoto cleanup;\n+\t}\n+\n \tif (packed_transaction) {\n \t\tif (packed_refs_lock(refs->packed_ref_store, 0, err)) {\n \t\t\tret = TRANSACTION_GENERIC_ERROR;\n@@ -2972,6 +2991,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n cleanup:\n \tfree(head_ref);\n \tstring_list_clear(&affected_refnames, 0);\n+\tstring_list_clear(&refnames_to_check, 0);\n \n \tif (ret)\n \t\tfiles_transaction_cleanup(refs, transaction);\n\n-- \n2.49.0.rc2.394.gf6994c5077.dirty\n\n"},{"id":"514078","messageId":"20250312-pks-update-ref-optimization-v6-7-f778e0414f55@pks.im","threadId":"62965","inReplyTo":"20250312-pks-update-ref-optimization-v6-0-f778e0414f55@pks.im","subject":"[PATCH v6 07/16] refs/files: batch refname availability checks for initial transactions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:56:13Z","receivedAt":"2025-03-12T15:56:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"files\" backend explicitly carves out special logic for its initial\ntransaction so that it can avoid writing out every single reference as\na loose reference. While the assumption is that there shouldn't be any\npreexisting references, we still have to verify that none of the newly\nwritten references will conflict with any other new reference in the\nsame transaction.\n\nRefactor the initial transaction to use batched refname availability\nchecks. This does not yet have an effect on performance as we still call\n`refs_verify_refname_available()` in a loop. But this will change in\nsubsequent commits and then impact performance when cloning a repository\nwith many references or when migrating references to the \"files\" format.\n\nThis will improve performance when cloning a repository with many\nreferences or when migrating references from any format to the \"files\"\nformat once the availability checks have learned to optimize checks for\nmany references in a subsequent commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/files-backend.c | 23 ++++++++++++++++-------\n 1 file changed, 16 insertions(+), 7 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex f798d8dae37..ab6f0af5502 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -3056,6 +3056,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \tsize_t i;\n \tint ret = 0;\n \tstruct string_list affected_refnames = STRING_LIST_INIT_NODUP;\n+\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n \tstruct ref_transaction *packed_transaction = NULL;\n \tstruct ref_transaction *loose_transaction = NULL;\n \n@@ -3105,11 +3106,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\t !is_null_oid(&update->old_oid))\n \t\t\tBUG(\"initial ref transaction with old_sha1 set\");\n \n-\t\tif (refs_verify_refname_available(&refs->base, update->refname,\n-\t\t\t\t\t\t &affected_refnames, NULL, 1, err)) {\n-\t\t\tret = TRANSACTION_NAME_CONFLICT;\n-\t\t\tgoto cleanup;\n-\t\t}\n+\t\tstring_list_append(&refnames_to_check, update->refname);\n \n \t\t/*\n \t\t * packed-refs don't support symbolic refs, root refs and reflogs,\n@@ -3145,8 +3142,19 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\t}\n \t}\n \n-\tif (packed_refs_lock(refs->packed_ref_store, 0, err) ||\n-\t ref_transaction_commit(packed_transaction, err)) {\n+\tif (packed_refs_lock(refs->packed_ref_store, 0, err)) {\n+\t\tret = TRANSACTION_GENERIC_ERROR;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif (refs_verify_refnames_available(&refs->base, &refnames_to_check,\n+\t\t\t\t\t &affected_refnames, NULL, 1, err)) {\n+\t\tpacked_refs_unlock(refs->packed_ref_store);\n+\t\tret = TRANSACTION_NAME_CONFLICT;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif (ref_transaction_commit(packed_transaction, err)) {\n \t\tret = TRANSACTION_GENERIC_ERROR;\n \t\tgoto cleanup;\n \t}\n@@ -3167,6 +3175,7 @@ static int files_transaction_finish_initial(struct files_ref_store *refs,\n \t\tref_transaction_free(packed_transaction);\n \ttransaction->state = REF_TRANSACTION_CLOSED;\n \tstring_list_clear(&affected_refnames, 0);\n+\tstring_list_clear(&refnames_to_check, 0);\n \treturn ret;\n }\n \n\n-- \n2.49.0.rc2.394.gf6994c5077.dirty\n\n"},{"id":"514079","messageId":"20250312-pks-update-ref-optimization-v6-8-f778e0414f55@pks.im","threadId":"62965","inReplyTo":"20250312-pks-update-ref-optimization-v6-0-f778e0414f55@pks.im","subject":"[PATCH v6 08/16] refs: stop re-verifying common prefixes for availability","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:56:14Z","receivedAt":"2025-03-12T15:56:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"One of the checks done by `refs_verify_refnames_available()` is whether\nany of the prefixes of a reference already exists. For example, given a\nreference \"refs/heads/main\", we'd check whether \"refs/heads\" or \"refs\"\nalready exist, and if so we'd abort the transaction.\n\nWhen updating multiple references at once, this check is performed for\neach of the references individually. Consequently, because references\ntend to have common prefixes like \"refs/heads/\" or refs/tags/\", we\nevaluate the availability of these prefixes repeatedly. Naturally this\nis a waste of compute, as the availability of those prefixes should in\ngeneral not change in the middle of a transaction. And if it would,\nbackends would notice at a later point in time.\n\nOptimize this pattern by storing prefixes in a `strset` so that we can\ntrivially track those prefixes that we have already checked. This leads\nto a significant speedup with the \"reftable\" backend when creating many\nreferences that all share a common prefix:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 63.1 ms ± 1.8 ms [User: 41.0 ms, System: 21.6 ms]\n Range (min … max): 60.6 ms … 69.5 ms 38 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 40.0 ms ± 1.3 ms [User: 29.3 ms, System: 10.3 ms]\n Range (min … max): 38.1 ms … 47.3 ms 61 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.58 ± 0.07 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n\nFor the \"files\" backend we see an improvement, but a much smaller one:\n\n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 395.8 ms ± 5.3 ms [User: 63.6 ms, System: 330.5 ms]\n Range (min … max): 387.0 ms … 404.6 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 386.0 ms ± 4.0 ms [User: 51.5 ms, System: 332.8 ms]\n Range (min … max): 380.8 ms … 392.6 ms 10 runs\n\n Summary\n update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.03 ± 0.02 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n\nThis change also leads to a modest improvement when writing references\nwith \"initial\" semantics, for example when migrating references. The\nfollowing benchmarks are migrating 1m references from the \"reftable\" to\nthe \"files\" backend:\n\n Benchmark 1: migrate reftable:files (refcount = 1000000, revision = HEAD~)\n Time (mean ± σ): 836.6 ms ± 5.6 ms [User: 645.2 ms, System: 185.2 ms]\n Range (min … max): 829.6 ms … 845.9 ms 10 runs\n\n Benchmark 2: migrate reftable:files (refcount = 1000000, revision = HEAD)\n Time (mean ± σ): 759.8 ms ± 5.1 ms [User: 574.9 ms, System: 178.9 ms]\n Range (min … max): 753.1 ms … 768.8 ms 10 runs\n\n Summary\n migrate reftable:files (refcount = 1000000, revision = HEAD) ran\n 1.10 ± 0.01 times faster than migrate reftable:files (refcount = 1000000, revision = HEAD~)\n\nAnd vice versa:\n\n Benchmark 1: migrate files:reftable (refcount = 1000000, revision = HEAD~)\n Time (mean ± σ): 870.7 ms ± 5.7 ms [User: 735.2 ms, System: 127.4 ms]\n Range (min … max): 861.6 ms … 883.2 ms 10 runs\n\n Benchmark 2: migrate files:reftable (refcount = 1000000, revision = HEAD)\n Time (mean ± σ): 799.1 ms ± 8.5 ms [User: 661.1 ms, System: 130.2 ms]\n Range (min … max): 787.5 ms … 812.6 ms 10 runs\n\n Summary\n migrate files:reftable (refcount = 1000000, revision = HEAD) ran\n 1.09 ± 0.01 times faster than migrate files:reftable (refcount = 1000000, revision = HEAD~)\n\nThe impact here is significantly smaller given that we don't perform any\nreference reads with \"initial\" semantics, so the speedup only comes from\nus doing less string list lookups.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs.c | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/refs.c b/refs.c\nindex d91a2184e06..111e3cf3aa9 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2477,6 +2477,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \tstruct strbuf dirname = STRBUF_INIT;\n \tstruct strbuf referent = STRBUF_INIT;\n \tstruct string_list_item *item;\n+\tstruct strset dirnames;\n \tint ret = -1;\n \n \t/*\n@@ -2486,6 +2487,8 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \n \tassert(err);\n \n+\tstrset_init(&dirnames);\n+\n \tfor_each_string_list_item(item, refnames) {\n \t\tconst char *refname = item->string;\n \t\tconst char *extra_refname;\n@@ -2515,6 +2518,14 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \t\t\tif (skip && string_list_has_string(skip, dirname.buf))\n \t\t\t\tcontinue;\n \n+\t\t\t/*\n+\t\t\t * If we've already seen the directory we don't need to\n+\t\t\t * process it again. Skip it to avoid checking checking\n+\t\t\t * common prefixes like \"refs/heads/\" repeatedly.\n+\t\t\t */\n+\t\t\tif (!strset_add(&dirnames, dirname.buf))\n+\t\t\t\tcontinue;\n+\n \t\t\tif (!initial_transaction &&\n \t\t\t !refs_read_raw_ref(refs, dirname.buf, &oid, &referent,\n \t\t\t\t\t &type, &ignore_errno)) {\n@@ -2575,6 +2586,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n cleanup:\n \tstrbuf_release(&referent);\n \tstrbuf_release(&dirname);\n+\tstrset_clear(&dirnames);\n \treturn ret;\n }\n \n\n-- \n2.49.0.rc2.394.gf6994c5077.dirty\n\n"},{"id":"514080","messageId":"20250312-pks-update-ref-optimization-v6-11-f778e0414f55@pks.im","threadId":"62965","inReplyTo":"20250312-pks-update-ref-optimization-v6-0-f778e0414f55@pks.im","subject":"[PATCH v6 11/16] refs/iterator: implement seeking for merged iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:56:17Z","receivedAt":"2025-03-12T15:56:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking on merged iterators. The implementation is rather\nstraight forward, with the only exception that we must not deallocate\nthe underlying iterators once they have been exhausted.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/iterator.c | 38 +++++++++++++++++++++++++++++---------\n 1 file changed, 29 insertions(+), 9 deletions(-)\n\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex ea4db59481d..766d96e795c 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -96,7 +96,8 @@ int is_empty_ref_iterator(struct ref_iterator *ref_iterator)\n struct merge_ref_iterator {\n \tstruct ref_iterator base;\n \n-\tstruct ref_iterator *iter0, *iter1;\n+\tstruct ref_iterator *iter0, *iter0_owned;\n+\tstruct ref_iterator *iter1, *iter1_owned;\n \n \tref_iterator_select_fn *select;\n \tvoid *cb_data;\n@@ -160,13 +161,11 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tif (!iter->current) {\n \t\t/* Initialize: advance both iterators to their first entries */\n \t\tif ((ok = ref_iterator_advance(iter->iter0)) != ITER_OK) {\n-\t\t\tref_iterator_free(iter->iter0);\n \t\t\titer->iter0 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n \t\t}\n \t\tif ((ok = ref_iterator_advance(iter->iter1)) != ITER_OK) {\n-\t\t\tref_iterator_free(iter->iter1);\n \t\t\titer->iter1 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -177,7 +176,6 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t * entry:\n \t\t */\n \t\tif ((ok = ref_iterator_advance(*iter->current)) != ITER_OK) {\n-\t\t\tref_iterator_free(*iter->current);\n \t\t\t*iter->current = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -206,7 +204,6 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \n \t\tif (selection & ITER_SKIP_SECONDARY) {\n \t\t\tif ((ok = ref_iterator_advance(*secondary)) != ITER_OK) {\n-\t\t\t\tref_iterator_free(*secondary);\n \t\t\t\t*secondary = NULL;\n \t\t\t\tif (ok == ITER_ERROR)\n \t\t\t\t\tgoto error;\n@@ -226,6 +223,28 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ITER_ERROR;\n }\n \n+static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct merge_ref_iterator *iter =\n+\t\t(struct merge_ref_iterator *)ref_iterator;\n+\tint ret;\n+\n+\titer->current = NULL;\n+\titer->iter0 = iter->iter0_owned;\n+\titer->iter1 = iter->iter1_owned;\n+\n+\tret = ref_iterator_seek(iter->iter0, prefix);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n+\tret = ref_iterator_seek(iter->iter1, prefix);\n+\tif (ret < 0)\n+\t\treturn ret;\n+\n+\treturn 0;\n+}\n+\n static int merge_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -242,12 +261,13 @@ static void merge_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct merge_ref_iterator *iter =\n \t\t(struct merge_ref_iterator *)ref_iterator;\n-\tref_iterator_free(iter->iter0);\n-\tref_iterator_free(iter->iter1);\n+\tref_iterator_free(iter->iter0_owned);\n+\tref_iterator_free(iter->iter1_owned);\n }\n \n static struct ref_iterator_vtable merge_ref_iterator_vtable = {\n \t.advance = merge_ref_iterator_advance,\n+\t.seek = merge_ref_iterator_seek,\n \t.peel = merge_ref_iterator_peel,\n \t.release = merge_ref_iterator_release,\n };\n@@ -268,8 +288,8 @@ struct ref_iterator *merge_ref_iterator_begin(\n \t */\n \n \tbase_ref_iterator_init(ref_iterator, &merge_ref_iterator_vtable);\n-\titer->iter0 = iter0;\n-\titer->iter1 = iter1;\n+\titer->iter0 = iter->iter0_owned = iter0;\n+\titer->iter1 = iter->iter1_owned = iter1;\n \titer->select = select;\n \titer->cb_data = cb_data;\n \titer->current = NULL;\n\n-- \n2.49.0.rc2.394.gf6994c5077.dirty\n\n"},{"id":"514081","messageId":"20250312-pks-update-ref-optimization-v6-10-f778e0414f55@pks.im","threadId":"62965","inReplyTo":"20250312-pks-update-ref-optimization-v6-0-f778e0414f55@pks.im","subject":"[PATCH v6 10/16] refs/iterator: provide infrastructure to re-seek iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:56:16Z","receivedAt":"2025-03-12T15:56:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Reftable iterators need to be scrapped after they have either been\nexhausted or aren't useful to the caller anymore, and it is explicitly\nnot possible to reuse them for iterations. But enabling for reuse of\niterators may allow us to tune them by reusing internal state of an\niterator. The reftable iterators for example can already be reused\ninternally, but we're not able to expose this to any users outside of\nthe reftable backend.\n\nIntroduce a new `.seek` function in the ref iterator vtable that allows\ncallers to seek an iterator multiple times. It is expected to be\nfunctionally the same as calling `refs_ref_iterator_begin()` with a\ndifferent (or the same) prefix.\n\nNote that it is not possible to adjust parameters other than the seeked\nprefix for now, so exclude patterns, trimmed prefixes and flags will\nremain unchanged. We do not have a usecase for changing these parameters\nright now, but if we ever find one we can adapt accordingly.\n\nImplement the callback for trivial cases. The other iterators will be\nimplemented in subsequent commits.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/debug.c | 11 +++++++++++\n refs/iterator.c | 24 ++++++++++++++++++++++++\n refs/refs-internal.h | 24 ++++++++++++++++++++++++\n 3 files changed, 59 insertions(+)\n\ndiff --git a/refs/debug.c b/refs/debug.c\nindex a9786da4ba1..5390fa9c187 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -169,6 +169,16 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn res;\n }\n \n+static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct debug_ref_iterator *diter =\n+\t\t(struct debug_ref_iterator *)ref_iterator;\n+\tint res = diter->iter->vtable->seek(diter->iter, prefix);\n+\ttrace_printf_key(&trace_refs, \"iterator_seek: %s: %d\\n\", prefix ? prefix : \"\", res);\n+\treturn res;\n+}\n+\n static int debug_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -189,6 +199,7 @@ static void debug_ref_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable debug_ref_iterator_vtable = {\n \t.advance = debug_ref_iterator_advance,\n+\t.seek = debug_ref_iterator_seek,\n \t.peel = debug_ref_iterator_peel,\n \t.release = debug_ref_iterator_release,\n };\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex d61474cba75..ea4db59481d 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -15,6 +15,12 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ref_iterator->vtable->advance(ref_iterator);\n }\n \n+int ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t const char *prefix)\n+{\n+\treturn ref_iterator->vtable->seek(ref_iterator, prefix);\n+}\n+\n int ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t struct object_id *peeled)\n {\n@@ -50,6 +56,12 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)\n \treturn ITER_DONE;\n }\n \n+static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n+\t\t\t\t const char *prefix UNUSED)\n+{\n+\treturn 0;\n+}\n+\n static int empty_ref_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \t\t\t\t struct object_id *peeled UNUSED)\n {\n@@ -62,6 +74,7 @@ static void empty_ref_iterator_release(struct ref_iterator *ref_iterator UNUSED)\n \n static struct ref_iterator_vtable empty_ref_iterator_vtable = {\n \t.advance = empty_ref_iterator_advance,\n+\t.seek = empty_ref_iterator_seek,\n \t.peel = empty_ref_iterator_peel,\n \t.release = empty_ref_iterator_release,\n };\n@@ -373,6 +386,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n+static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct prefix_ref_iterator *iter =\n+\t\t(struct prefix_ref_iterator *)ref_iterator;\n+\tfree(iter->prefix);\n+\titer->prefix = xstrdup_or_null(prefix);\n+\treturn ref_iterator_seek(iter->iter0, prefix);\n+}\n+\n static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -392,6 +415,7 @@ static void prefix_ref_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable prefix_ref_iterator_vtable = {\n \t.advance = prefix_ref_iterator_advance,\n+\t.seek = prefix_ref_iterator_seek,\n \t.peel = prefix_ref_iterator_peel,\n \t.release = prefix_ref_iterator_release,\n };\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 74e2c03cef1..8f18274a165 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -327,6 +327,22 @@ struct ref_iterator {\n */\n int ref_iterator_advance(struct ref_iterator *ref_iterator);\n \n+/*\n+ * Seek the iterator to the first reference with the given prefix.\n+ * The prefix is matched as a literal string, without regard for path\n+ * separators. If prefix is NULL or the empty string, seek the iterator to the\n+ * first reference again.\n+ *\n+ * This function is expected to behave as if a new ref iterator with the same\n+ * prefix had been created, but allows reuse of iterators and thus may allow\n+ * the backend to optimize. Parameters other than the prefix that have been\n+ * passed when creating the iterator will remain unchanged.\n+ *\n+ * Returns 0 on success, a negative error code otherwise.\n+ */\n+int ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t const char *prefix);\n+\n /*\n * If possible, peel the reference currently being viewed by the\n * iterator. Return 0 on success.\n@@ -445,6 +461,13 @@ void base_ref_iterator_init(struct ref_iterator *iter,\n */\n typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);\n \n+/*\n+ * Seek the iterator to the first reference matching the given prefix. Should\n+ * behave the same as if a new iterator was created with the same prefix.\n+ */\n+typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix);\n+\n /*\n * Peels the current ref, returning 0 for success or -1 for failure.\n */\n@@ -459,6 +482,7 @@ typedef void ref_iterator_release_fn(struct ref_iterator *ref_iterator);\n \n struct ref_iterator_vtable {\n \tref_iterator_advance_fn *advance;\n+\tref_iterator_seek_fn *seek;\n \tref_iterator_peel_fn *peel;\n \tref_iterator_release_fn *release;\n };\n\n-- \n2.49.0.rc2.394.gf6994c5077.dirty\n\n"},{"id":"514082","messageId":"20250312-pks-update-ref-optimization-v6-9-f778e0414f55@pks.im","threadId":"62965","inReplyTo":"20250312-pks-update-ref-optimization-v6-0-f778e0414f55@pks.im","subject":"[PATCH v6 09/16] refs/iterator: separate lifecycle from iteration","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:56:15Z","receivedAt":"2025-03-12T15:56:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The ref and reflog iterators have their lifecycle attached to iteration:\nonce the iterator reaches its end, it is automatically released and the\ncaller doesn't have to care about that anymore. When the iterator should\nbe released before it has been exhausted, callers must explicitly abort\nthe iterator via `ref_iterator_abort()`.\n\nThis lifecycle is somewhat unusual in the Git codebase and creates two\nproblems:\n\n - Callsites need to be very careful about when exactly they call\n `ref_iterator_abort()`, as calling the function is only valid when\n the iterator itself still is. This leads to somewhat awkward calling\n patterns in some situations.\n\n - It is impossible to reuse iterators and re-seek them to a different\n prefix. This feature isn't supported by any iterator implementation\n except for the reftable iterators anyway, but if it was implemented\n it would allow us to optimize cases where we need to search for\n specific references repeatedly by reusing internal state.\n\nDetangle the lifecycle from iteration so that we don't deallocate the\niterator anymore once it is exhausted. Instead, callers are now expected\nto always call a newly introduce `ref_iterator_free()` function that\ndeallocates the iterator and its internal state.\n\nNote that the `dir_iterator` is somewhat special because it does not\nimplement the `ref_iterator` interface, but is only used to implement\nother iterators. Consequently, we have to provide `dir_iterator_free()`\ninstead of `dir_iterator_release()` as the allocated structure itself is\nmanaged by the `dir_iterator` interfaces, as well, and not freed by\n`ref_iterator_free()` like in all the other cases.\n\nWhile at it, drop the return value of `ref_iterator_abort()`, which\nwasn't really required by any of the iterator implementations anyway.\nFurthermore, stop calling `base_ref_iterator_free()` in any of the\nbackends, but instead call it in `ref_iterator_free()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/clone.c | 2 +\n dir-iterator.c | 24 +++++------\n dir-iterator.h | 11 ++---\n iterator.h | 2 +-\n refs.c | 7 ++-\n refs/debug.c | 9 ++--\n refs/files-backend.c | 36 +++++-----------\n refs/iterator.c | 100 +++++++++++++++----------------------------\n refs/packed-backend.c | 27 ++++++------\n refs/ref-cache.c | 9 ++--\n refs/refs-internal.h | 29 ++++---------\n refs/reftable-backend.c | 34 ++++-----------\n t/helper/test-dir-iterator.c | 1 +\n 13 files changed, 105 insertions(+), 186 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex fd001d800c6..ac3e84b2b18 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -426,6 +426,8 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \t\tstrbuf_setlen(src, src_len);\n \t\tdie(_(\"failed to iterate over '%s'\"), src->buf);\n \t}\n+\n+\tdir_iterator_free(iter);\n }\n \n static void clone_local(const char *src_repo, const char *dest_repo)\ndiff --git a/dir-iterator.c b/dir-iterator.c\nindex de619846f29..857e1d9bdaf 100644\n--- a/dir-iterator.c\n+++ b/dir-iterator.c\n@@ -193,9 +193,9 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \n \tif (S_ISDIR(iter->base.st.st_mode) && push_level(iter)) {\n \t\tif (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC)\n-\t\t\tgoto error_out;\n+\t\t\treturn ITER_ERROR;\n \t\tif (iter->levels_nr == 0)\n-\t\t\tgoto error_out;\n+\t\t\treturn ITER_ERROR;\n \t}\n \n \t/* Loop until we find an entry that we can give back to the caller. */\n@@ -211,11 +211,11 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t\tint ret = next_directory_entry(level->dir, iter->base.path.buf, &de);\n \t\t\tif (ret < 0) {\n \t\t\t\tif (iter->flags & DIR_ITERATOR_PEDANTIC)\n-\t\t\t\t\tgoto error_out;\n+\t\t\t\t\treturn ITER_ERROR;\n \t\t\t\tcontinue;\n \t\t\t} else if (ret > 0) {\n \t\t\t\tif (pop_level(iter) == 0)\n-\t\t\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\t\t\t\treturn ITER_DONE;\n \t\t\t\tcontinue;\n \t\t\t}\n \n@@ -223,7 +223,7 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \t\t} else {\n \t\t\tif (level->entries_idx >= level->entries.nr) {\n \t\t\t\tif (pop_level(iter) == 0)\n-\t\t\t\t\treturn dir_iterator_abort(dir_iterator);\n+\t\t\t\t\treturn ITER_DONE;\n \t\t\t\tcontinue;\n \t\t\t}\n \n@@ -232,22 +232,21 @@ int dir_iterator_advance(struct dir_iterator *dir_iterator)\n \n \t\tif (prepare_next_entry_data(iter, name)) {\n \t\t\tif (errno != ENOENT && iter->flags & DIR_ITERATOR_PEDANTIC)\n-\t\t\t\tgoto error_out;\n+\t\t\t\treturn ITER_ERROR;\n \t\t\tcontinue;\n \t\t}\n \n \t\treturn ITER_OK;\n \t}\n-\n-error_out:\n-\tdir_iterator_abort(dir_iterator);\n-\treturn ITER_ERROR;\n }\n \n-int dir_iterator_abort(struct dir_iterator *dir_iterator)\n+void dir_iterator_free(struct dir_iterator *dir_iterator)\n {\n \tstruct dir_iterator_int *iter = (struct dir_iterator_int *)dir_iterator;\n \n+\tif (!iter)\n+\t\treturn;\n+\n \tfor (; iter->levels_nr; iter->levels_nr--) {\n \t\tstruct dir_iterator_level *level =\n \t\t\t&iter->levels[iter->levels_nr - 1];\n@@ -266,7 +265,6 @@ int dir_iterator_abort(struct dir_iterator *dir_iterator)\n \tfree(iter->levels);\n \tstrbuf_release(&iter->base.path);\n \tfree(iter);\n-\treturn ITER_DONE;\n }\n \n struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags)\n@@ -301,7 +299,7 @@ struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags)\n \treturn dir_iterator;\n \n error_out:\n-\tdir_iterator_abort(dir_iterator);\n+\tdir_iterator_free(dir_iterator);\n \terrno = saved_errno;\n \treturn NULL;\n }\ndiff --git a/dir-iterator.h b/dir-iterator.h\nindex 6d438809b6e..ccd6a197343 100644\n--- a/dir-iterator.h\n+++ b/dir-iterator.h\n@@ -28,7 +28,7 @@\n *\n * while ((ok = dir_iterator_advance(iter)) == ITER_OK) {\n * if (want_to_stop_iteration()) {\n- * ok = dir_iterator_abort(iter);\n+ * ok = ITER_DONE;\n * break;\n * }\n *\n@@ -39,6 +39,7 @@\n *\n * if (ok != ITER_DONE)\n * handle_error();\n+ * dir_iterator_free(iter);\n *\n * Callers are allowed to modify iter->path while they are working,\n * but they must restore it to its original contents before calling\n@@ -107,11 +108,7 @@ struct dir_iterator *dir_iterator_begin(const char *path, unsigned int flags);\n */\n int dir_iterator_advance(struct dir_iterator *iterator);\n \n-/*\n- * End the iteration before it has been exhausted. Free the\n- * dir_iterator and any associated resources and return ITER_DONE. On\n- * error, free the dir_iterator and return ITER_ERROR.\n- */\n-int dir_iterator_abort(struct dir_iterator *iterator);\n+/* Free the dir_iterator and any associated resources. */\n+void dir_iterator_free(struct dir_iterator *iterator);\n \n #endif\ndiff --git a/iterator.h b/iterator.h\nindex 0f6900e43ad..6b77dcc2626 100644\n--- a/iterator.h\n+++ b/iterator.h\n@@ -12,7 +12,7 @@\n #define ITER_OK 0\n \n /*\n- * The iterator is exhausted and has been freed.\n+ * The iterator is exhausted.\n */\n #define ITER_DONE -1\n \ndiff --git a/refs.c b/refs.c\nindex 111e3cf3aa9..3e65ccad7ac 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2477,6 +2477,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \tstruct strbuf dirname = STRBUF_INIT;\n \tstruct strbuf referent = STRBUF_INIT;\n \tstruct string_list_item *item;\n+\tstruct ref_iterator *iter = NULL;\n \tstruct strset dirnames;\n \tint ret = -1;\n \n@@ -2553,7 +2554,6 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \t\tstrbuf_addch(&dirname, '/');\n \n \t\tif (!initial_transaction) {\n-\t\t\tstruct ref_iterator *iter;\n \t\t\tint ok;\n \n \t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n@@ -2565,12 +2565,14 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \n \t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n \t\t\t\t\t iter->refname, refname);\n-\t\t\t\tref_iterator_abort(iter);\n \t\t\t\tgoto cleanup;\n \t\t\t}\n \n \t\t\tif (ok != ITER_DONE)\n \t\t\t\tBUG(\"error while iterating over references\");\n+\n+\t\t\tref_iterator_free(iter);\n+\t\t\titer = NULL;\n \t\t}\n \n \t\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n@@ -2587,6 +2589,7 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \tstrbuf_release(&referent);\n \tstrbuf_release(&dirname);\n \tstrset_clear(&dirnames);\n+\tref_iterator_free(iter);\n \treturn ret;\n }\n \ndiff --git a/refs/debug.c b/refs/debug.c\nindex fbc4df08b43..a9786da4ba1 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -179,19 +179,18 @@ static int debug_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn res;\n }\n \n-static int debug_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void debug_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct debug_ref_iterator *diter =\n \t\t(struct debug_ref_iterator *)ref_iterator;\n-\tint res = diter->iter->vtable->abort(diter->iter);\n-\ttrace_printf_key(&trace_refs, \"iterator_abort: %d\\n\", res);\n-\treturn res;\n+\tditer->iter->vtable->release(diter->iter);\n+\ttrace_printf_key(&trace_refs, \"iterator_abort\\n\");\n }\n \n static struct ref_iterator_vtable debug_ref_iterator_vtable = {\n \t.advance = debug_ref_iterator_advance,\n \t.peel = debug_ref_iterator_peel,\n-\t.abort = debug_ref_iterator_abort,\n+\t.release = debug_ref_iterator_release,\n };\n \n static struct ref_iterator *\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex ab6f0af5502..e97a267ad65 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -915,10 +915,6 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\titer->iter0 = NULL;\n-\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\tok = ITER_ERROR;\n-\n \treturn ok;\n }\n \n@@ -931,23 +927,17 @@ static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator_peel(iter->iter0, peeled);\n }\n \n-static int files_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void files_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct files_ref_iterator *iter =\n \t\t(struct files_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->iter0)\n-\t\tok = ref_iterator_abort(iter->iter0);\n-\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n+\tref_iterator_free(iter->iter0);\n }\n \n static struct ref_iterator_vtable files_ref_iterator_vtable = {\n \t.advance = files_ref_iterator_advance,\n \t.peel = files_ref_iterator_peel,\n-\t.abort = files_ref_iterator_abort,\n+\t.release = files_ref_iterator_release,\n };\n \n static struct ref_iterator *files_ref_iterator_begin(\n@@ -1378,7 +1368,7 @@ static int should_pack_refs(struct files_ref_store *refs,\n \t\t\t\t iter->flags, opts))\n \t\t\trefcount++;\n \t\tif (refcount >= limit) {\n-\t\t\tref_iterator_abort(iter);\n+\t\t\tref_iterator_free(iter);\n \t\t\treturn 1;\n \t\t}\n \t}\n@@ -1386,6 +1376,7 @@ static int should_pack_refs(struct files_ref_store *refs,\n \tif (ret != ITER_DONE)\n \t\tdie(\"error while iterating over references\");\n \n+\tref_iterator_free(iter);\n \treturn 0;\n }\n \n@@ -1452,6 +1443,7 @@ static int files_pack_refs(struct ref_store *ref_store,\n \tpacked_refs_unlock(refs->packed_ref_store);\n \n \tprune_refs(refs, &refs_to_prune);\n+\tref_iterator_free(iter);\n \tstrbuf_release(&err);\n \treturn 0;\n }\n@@ -2299,9 +2291,6 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\titer->dir_iterator = NULL;\n-\tif (ref_iterator_abort(ref_iterator) == ITER_ERROR)\n-\t\tok = ITER_ERROR;\n \treturn ok;\n }\n \n@@ -2311,23 +2300,17 @@ static int files_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \tBUG(\"ref_iterator_peel() called for reflog_iterator\");\n }\n \n-static int files_reflog_iterator_abort(struct ref_iterator *ref_iterator)\n+static void files_reflog_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct files_reflog_iterator *iter =\n \t\t(struct files_reflog_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->dir_iterator)\n-\t\tok = dir_iterator_abort(iter->dir_iterator);\n-\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n+\tdir_iterator_free(iter->dir_iterator);\n }\n \n static struct ref_iterator_vtable files_reflog_iterator_vtable = {\n \t.advance = files_reflog_iterator_advance,\n \t.peel = files_reflog_iterator_peel,\n-\t.abort = files_reflog_iterator_abort,\n+\t.release = files_reflog_iterator_release,\n };\n \n static struct ref_iterator *reflog_iterator_begin(struct ref_store *ref_store,\n@@ -3837,6 +3820,7 @@ static int files_fsck_refs_dir(struct ref_store *ref_store,\n \t\tret = error(_(\"failed to iterate over '%s'\"), sb.buf);\n \n out:\n+\tdir_iterator_free(iter);\n \tstrbuf_release(&sb);\n \tstrbuf_release(&refname);\n \treturn ret;\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex d25e568bf0b..d61474cba75 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -21,9 +21,14 @@ int ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator->vtable->peel(ref_iterator, peeled);\n }\n \n-int ref_iterator_abort(struct ref_iterator *ref_iterator)\n+void ref_iterator_free(struct ref_iterator *ref_iterator)\n {\n-\treturn ref_iterator->vtable->abort(ref_iterator);\n+\tif (ref_iterator) {\n+\t\tref_iterator->vtable->release(ref_iterator);\n+\t\t/* Help make use-after-free bugs fail quickly: */\n+\t\tref_iterator->vtable = NULL;\n+\t\tfree(ref_iterator);\n+\t}\n }\n \n void base_ref_iterator_init(struct ref_iterator *iter,\n@@ -36,20 +41,13 @@ void base_ref_iterator_init(struct ref_iterator *iter,\n \titer->flags = 0;\n }\n \n-void base_ref_iterator_free(struct ref_iterator *iter)\n-{\n-\t/* Help make use-after-free bugs fail quickly: */\n-\titer->vtable = NULL;\n-\tfree(iter);\n-}\n-\n struct empty_ref_iterator {\n \tstruct ref_iterator base;\n };\n \n-static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator)\n+static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)\n {\n-\treturn ref_iterator_abort(ref_iterator);\n+\treturn ITER_DONE;\n }\n \n static int empty_ref_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n@@ -58,16 +56,14 @@ static int empty_ref_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \tBUG(\"peel called for empty iterator\");\n }\n \n-static int empty_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void empty_ref_iterator_release(struct ref_iterator *ref_iterator UNUSED)\n {\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable empty_ref_iterator_vtable = {\n \t.advance = empty_ref_iterator_advance,\n \t.peel = empty_ref_iterator_peel,\n-\t.abort = empty_ref_iterator_abort,\n+\t.release = empty_ref_iterator_release,\n };\n \n struct ref_iterator *empty_ref_iterator_begin(void)\n@@ -151,11 +147,13 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tif (!iter->current) {\n \t\t/* Initialize: advance both iterators to their first entries */\n \t\tif ((ok = ref_iterator_advance(iter->iter0)) != ITER_OK) {\n+\t\t\tref_iterator_free(iter->iter0);\n \t\t\titer->iter0 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n \t\t}\n \t\tif ((ok = ref_iterator_advance(iter->iter1)) != ITER_OK) {\n+\t\t\tref_iterator_free(iter->iter1);\n \t\t\titer->iter1 = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -166,6 +164,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t * entry:\n \t\t */\n \t\tif ((ok = ref_iterator_advance(*iter->current)) != ITER_OK) {\n+\t\t\tref_iterator_free(*iter->current);\n \t\t\t*iter->current = NULL;\n \t\t\tif (ok == ITER_ERROR)\n \t\t\t\tgoto error;\n@@ -179,9 +178,8 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t\titer->select(iter->iter0, iter->iter1, iter->cb_data);\n \n \t\tif (selection == ITER_SELECT_DONE) {\n-\t\t\treturn ref_iterator_abort(ref_iterator);\n+\t\t\treturn ITER_DONE;\n \t\t} else if (selection == ITER_SELECT_ERROR) {\n-\t\t\tref_iterator_abort(ref_iterator);\n \t\t\treturn ITER_ERROR;\n \t\t}\n \n@@ -195,6 +193,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \n \t\tif (selection & ITER_SKIP_SECONDARY) {\n \t\t\tif ((ok = ref_iterator_advance(*secondary)) != ITER_OK) {\n+\t\t\t\tref_iterator_free(*secondary);\n \t\t\t\t*secondary = NULL;\n \t\t\t\tif (ok == ITER_ERROR)\n \t\t\t\t\tgoto error;\n@@ -211,7 +210,6 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t}\n \n error:\n-\tref_iterator_abort(ref_iterator);\n \treturn ITER_ERROR;\n }\n \n@@ -227,28 +225,18 @@ static int merge_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator_peel(*iter->current, peeled);\n }\n \n-static int merge_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void merge_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct merge_ref_iterator *iter =\n \t\t(struct merge_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->iter0) {\n-\t\tif (ref_iterator_abort(iter->iter0) != ITER_DONE)\n-\t\t\tok = ITER_ERROR;\n-\t}\n-\tif (iter->iter1) {\n-\t\tif (ref_iterator_abort(iter->iter1) != ITER_DONE)\n-\t\t\tok = ITER_ERROR;\n-\t}\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n+\tref_iterator_free(iter->iter0);\n+\tref_iterator_free(iter->iter1);\n }\n \n static struct ref_iterator_vtable merge_ref_iterator_vtable = {\n \t.advance = merge_ref_iterator_advance,\n \t.peel = merge_ref_iterator_peel,\n-\t.abort = merge_ref_iterator_abort,\n+\t.release = merge_ref_iterator_release,\n };\n \n struct ref_iterator *merge_ref_iterator_begin(\n@@ -310,10 +298,10 @@ struct ref_iterator *overlay_ref_iterator_begin(\n \t * them.\n \t */\n \tif (is_empty_ref_iterator(front)) {\n-\t\tref_iterator_abort(front);\n+\t\tref_iterator_free(front);\n \t\treturn back;\n \t} else if (is_empty_ref_iterator(back)) {\n-\t\tref_iterator_abort(back);\n+\t\tref_iterator_free(back);\n \t\treturn front;\n \t}\n \n@@ -350,19 +338,15 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \n \twhile ((ok = ref_iterator_advance(iter->iter0)) == ITER_OK) {\n \t\tint cmp = compare_prefix(iter->iter0->refname, iter->prefix);\n-\n \t\tif (cmp < 0)\n \t\t\tcontinue;\n-\n-\t\tif (cmp > 0) {\n-\t\t\t/*\n-\t\t\t * As the source iterator is ordered, we\n-\t\t\t * can stop the iteration as soon as we see a\n-\t\t\t * refname that comes after the prefix:\n-\t\t\t */\n-\t\t\tok = ref_iterator_abort(iter->iter0);\n-\t\t\tbreak;\n-\t\t}\n+\t\t/*\n+\t\t * As the source iterator is ordered, we\n+\t\t * can stop the iteration as soon as we see a\n+\t\t * refname that comes after the prefix:\n+\t\t */\n+\t\tif (cmp > 0)\n+\t\t\treturn ITER_DONE;\n \n \t\tif (iter->trim) {\n \t\t\t/*\n@@ -386,9 +370,6 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\titer->iter0 = NULL;\n-\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\treturn ITER_ERROR;\n \treturn ok;\n }\n \n@@ -401,23 +382,18 @@ static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn ref_iterator_peel(iter->iter0, peeled);\n }\n \n-static int prefix_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void prefix_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct prefix_ref_iterator *iter =\n \t\t(struct prefix_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n-\tif (iter->iter0)\n-\t\tok = ref_iterator_abort(iter->iter0);\n+\tref_iterator_free(iter->iter0);\n \tfree(iter->prefix);\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n }\n \n static struct ref_iterator_vtable prefix_ref_iterator_vtable = {\n \t.advance = prefix_ref_iterator_advance,\n \t.peel = prefix_ref_iterator_peel,\n-\t.abort = prefix_ref_iterator_abort,\n+\t.release = prefix_ref_iterator_release,\n };\n \n struct ref_iterator *prefix_ref_iterator_begin(struct ref_iterator *iter0,\n@@ -453,20 +429,14 @@ int do_for_each_ref_iterator(struct ref_iterator *iter,\n \tcurrent_ref_iter = iter;\n \twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\tretval = fn(iter->refname, iter->referent, iter->oid, iter->flags, cb_data);\n-\t\tif (retval) {\n-\t\t\t/*\n-\t\t\t * If ref_iterator_abort() returns ITER_ERROR,\n-\t\t\t * we ignore that error in deference to the\n-\t\t\t * callback function's return value.\n-\t\t\t */\n-\t\t\tref_iterator_abort(iter);\n+\t\tif (retval)\n \t\t\tgoto out;\n-\t\t}\n \t}\n \n out:\n \tcurrent_ref_iter = old_ref_iter;\n \tif (ok == ITER_ERROR)\n-\t\treturn -1;\n+\t\tretval = -1;\n+\tref_iterator_free(iter);\n \treturn retval;\n }\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex a7b6f74b6e3..38a1956d1a8 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -954,9 +954,6 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\treturn ITER_OK;\n \t}\n \n-\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\tok = ITER_ERROR;\n-\n \treturn ok;\n }\n \n@@ -976,23 +973,19 @@ static int packed_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t}\n }\n \n-static int packed_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void packed_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct packed_ref_iterator *iter =\n \t\t(struct packed_ref_iterator *)ref_iterator;\n-\tint ok = ITER_DONE;\n-\n \tstrbuf_release(&iter->refname_buf);\n \tfree(iter->jump);\n \trelease_snapshot(iter->snapshot);\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ok;\n }\n \n static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \t.advance = packed_ref_iterator_advance,\n \t.peel = packed_ref_iterator_peel,\n-\t.abort = packed_ref_iterator_abort\n+\t.release = packed_ref_iterator_release,\n };\n \n static int jump_list_entry_cmp(const void *va, const void *vb)\n@@ -1362,8 +1355,10 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t */\n \titer = packed_ref_iterator_begin(&refs->base, \"\", NULL,\n \t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n-\tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n+\tif ((ok = ref_iterator_advance(iter)) != ITER_OK) {\n+\t\tref_iterator_free(iter);\n \t\titer = NULL;\n+\t}\n \n \ti = 0;\n \n@@ -1411,8 +1406,10 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t\t\t\t * the iterator over the unneeded\n \t\t\t\t * value.\n \t\t\t\t */\n-\t\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n+\t\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK) {\n+\t\t\t\t\tref_iterator_free(iter);\n \t\t\t\t\titer = NULL;\n+\t\t\t\t}\n \t\t\t\tcmp = +1;\n \t\t\t} else {\n \t\t\t\t/*\n@@ -1449,8 +1446,10 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t\t\t\t\t peel_error ? NULL : &peeled))\n \t\t\t\tgoto write_error;\n \n-\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n+\t\t\tif ((ok = ref_iterator_advance(iter)) != ITER_OK) {\n+\t\t\t\tref_iterator_free(iter);\n \t\t\t\titer = NULL;\n+\t\t\t}\n \t\t} else if (is_null_oid(&update->new_oid)) {\n \t\t\t/*\n \t\t\t * The update wants to delete the reference,\n@@ -1499,9 +1498,7 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t\t get_tempfile_path(refs->tempfile), strerror(errno));\n \n error:\n-\tif (iter)\n-\t\tref_iterator_abort(iter);\n-\n+\tref_iterator_free(iter);\n \tdelete_tempfile(&refs->tempfile);\n \treturn -1;\n }\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 02f09e4df88..6457e02c1ea 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -409,7 +409,7 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\tif (++level->index == level->dir->nr) {\n \t\t\t/* This level is exhausted; pop up a level */\n \t\t\tif (--iter->levels_nr == 0)\n-\t\t\t\treturn ref_iterator_abort(ref_iterator);\n+\t\t\t\treturn ITER_DONE;\n \n \t\t\tcontinue;\n \t\t}\n@@ -452,21 +452,18 @@ static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn peel_object(iter->repo, ref_iterator->oid, peeled) ? -1 : 0;\n }\n \n-static int cache_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void cache_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct cache_ref_iterator *iter =\n \t\t(struct cache_ref_iterator *)ref_iterator;\n-\n \tfree((char *)iter->prefix);\n \tfree(iter->levels);\n-\tbase_ref_iterator_free(ref_iterator);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable cache_ref_iterator_vtable = {\n \t.advance = cache_ref_iterator_advance,\n \t.peel = cache_ref_iterator_peel,\n-\t.abort = cache_ref_iterator_abort\n+\t.release = cache_ref_iterator_release,\n };\n \n struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex aaab711bb96..74e2c03cef1 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -273,11 +273,11 @@ enum do_for_each_ref_flags {\n * the next reference and returns ITER_OK. The data pointed at by\n * refname and oid belong to the iterator; if you want to retain them\n * after calling ref_iterator_advance() again or calling\n- * ref_iterator_abort(), you must make a copy. When the iteration has\n+ * ref_iterator_free(), you must make a copy. When the iteration has\n * been exhausted, ref_iterator_advance() releases any resources\n * associated with the iteration, frees the ref_iterator object, and\n * returns ITER_DONE. If you want to abort the iteration early, call\n- * ref_iterator_abort(), which also frees the ref_iterator object and\n+ * ref_iterator_free(), which also frees the ref_iterator object and\n * any associated resources. If there was an internal error advancing\n * to the next entry, ref_iterator_advance() aborts the iteration,\n * frees the ref_iterator, and returns ITER_ERROR.\n@@ -293,7 +293,7 @@ enum do_for_each_ref_flags {\n *\n * while ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n * if (want_to_stop_iteration()) {\n- * ok = ref_iterator_abort(iter);\n+ * ok = ITER_DONE;\n * break;\n * }\n *\n@@ -307,6 +307,7 @@ enum do_for_each_ref_flags {\n *\n * if (ok != ITER_DONE)\n * handle_error();\n+ * ref_iterator_free(iter);\n */\n struct ref_iterator {\n \tstruct ref_iterator_vtable *vtable;\n@@ -333,12 +334,8 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator);\n int ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t struct object_id *peeled);\n \n-/*\n- * End the iteration before it has been exhausted, freeing the\n- * reference iterator and any associated resources and returning\n- * ITER_DONE. If the abort itself failed, return ITER_ERROR.\n- */\n-int ref_iterator_abort(struct ref_iterator *ref_iterator);\n+/* Free the reference iterator and any associated resources. */\n+void ref_iterator_free(struct ref_iterator *ref_iterator);\n \n /*\n * An iterator over nothing (its first ref_iterator_advance() call\n@@ -438,13 +435,6 @@ struct ref_iterator *prefix_ref_iterator_begin(struct ref_iterator *iter0,\n void base_ref_iterator_init(struct ref_iterator *iter,\n \t\t\t struct ref_iterator_vtable *vtable);\n \n-/*\n- * Base class destructor for ref_iterators. Destroy the ref_iterator\n- * part of iter and shallow-free the object. This is meant to be\n- * called only by the destructors of derived classes.\n- */\n-void base_ref_iterator_free(struct ref_iterator *iter);\n-\n /* Virtual function declarations for ref_iterators: */\n \n /*\n@@ -463,15 +453,14 @@ typedef int ref_iterator_peel_fn(struct ref_iterator *ref_iterator,\n \n /*\n * Implementations of this function should free any resources specific\n- * to the derived class, then call base_ref_iterator_free() to clean\n- * up and free the ref_iterator object.\n+ * to the derived class.\n */\n-typedef int ref_iterator_abort_fn(struct ref_iterator *ref_iterator);\n+typedef void ref_iterator_release_fn(struct ref_iterator *ref_iterator);\n \n struct ref_iterator_vtable {\n \tref_iterator_advance_fn *advance;\n \tref_iterator_peel_fn *peel;\n-\tref_iterator_abort_fn *abort;\n+\tref_iterator_release_fn *release;\n };\n \n /*\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 5c464b9d143..57d8512fe80 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -711,17 +711,10 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\tbreak;\n \t}\n \n-\tif (iter->err > 0) {\n-\t\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\t\treturn ITER_ERROR;\n+\tif (iter->err > 0)\n \t\treturn ITER_DONE;\n-\t}\n-\n-\tif (iter->err < 0) {\n-\t\tref_iterator_abort(ref_iterator);\n+\tif (iter->err < 0)\n \t\treturn ITER_ERROR;\n-\t}\n-\n \treturn ITER_OK;\n }\n \n@@ -740,7 +733,7 @@ static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \treturn -1;\n }\n \n-static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n+static void reftable_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct reftable_ref_iterator *iter =\n \t\t(struct reftable_ref_iterator *)ref_iterator;\n@@ -751,14 +744,12 @@ static int reftable_ref_iterator_abort(struct ref_iterator *ref_iterator)\n \t\t\tfree(iter->exclude_patterns[i]);\n \t\tfree(iter->exclude_patterns);\n \t}\n-\tfree(iter);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n \t.advance = reftable_ref_iterator_advance,\n \t.peel = reftable_ref_iterator_peel,\n-\t.abort = reftable_ref_iterator_abort\n+\t.release = reftable_ref_iterator_release,\n };\n \n static int qsort_strcmp(const void *va, const void *vb)\n@@ -2016,17 +2007,10 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \t\tbreak;\n \t}\n \n-\tif (iter->err > 0) {\n-\t\tif (ref_iterator_abort(ref_iterator) != ITER_DONE)\n-\t\t\treturn ITER_ERROR;\n+\tif (iter->err > 0)\n \t\treturn ITER_DONE;\n-\t}\n-\n-\tif (iter->err < 0) {\n-\t\tref_iterator_abort(ref_iterator);\n+\tif (iter->err < 0)\n \t\treturn ITER_ERROR;\n-\t}\n-\n \treturn ITER_OK;\n }\n \n@@ -2037,21 +2021,19 @@ static int reftable_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSE\n \treturn -1;\n }\n \n-static int reftable_reflog_iterator_abort(struct ref_iterator *ref_iterator)\n+static void reftable_reflog_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct reftable_reflog_iterator *iter =\n \t\t(struct reftable_reflog_iterator *)ref_iterator;\n \treftable_log_record_release(&iter->log);\n \treftable_iterator_destroy(&iter->iter);\n \tstrbuf_release(&iter->last_name);\n-\tfree(iter);\n-\treturn ITER_DONE;\n }\n \n static struct ref_iterator_vtable reftable_reflog_iterator_vtable = {\n \t.advance = reftable_reflog_iterator_advance,\n \t.peel = reftable_reflog_iterator_peel,\n-\t.abort = reftable_reflog_iterator_abort\n+\t.release = reftable_reflog_iterator_release,\n };\n \n static struct reftable_reflog_iterator *reflog_iterator_for_stack(struct reftable_ref_store *refs,\ndiff --git a/t/helper/test-dir-iterator.c b/t/helper/test-dir-iterator.c\nindex 6b297bd7536..8d46e8ba409 100644\n--- a/t/helper/test-dir-iterator.c\n+++ b/t/helper/test-dir-iterator.c\n@@ -53,6 +53,7 @@ int cmd__dir_iterator(int argc, const char **argv)\n \t\tprintf(\"(%s) [%s] %s\\n\", diter->relative_path, diter->basename,\n \t\t diter->path.buf);\n \t}\n+\tdir_iterator_free(diter);\n \n \tif (iter_status != ITER_DONE) {\n \t\tprintf(\"dir_iterator_advance failure\\n\");\n\n-- \n2.49.0.rc2.394.gf6994c5077.dirty\n\n"},{"id":"514083","messageId":"20250312-pks-update-ref-optimization-v6-12-f778e0414f55@pks.im","threadId":"62965","inReplyTo":"20250312-pks-update-ref-optimization-v6-0-f778e0414f55@pks.im","subject":"[PATCH v6 12/16] refs/iterator: implement seeking for reftable iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:56:18Z","receivedAt":"2025-03-12T15:56:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking of reftable iterators. As the low-level reftable\niterators already support seeking this change is straight-forward. Two\nnotes though:\n\n - We do not support seeking on reflog iterators. It is unclear what\n seeking would even look like in this context, as you typically would\n want to seek to a specific entry in the reflog for a specific ref.\n There is currently no use case for this, but if one arises in the\n future, we can still implement seeking at that later point.\n\n - We start to check whether `reftable_stack_init_ref_iterator()` is\n successful.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/reftable-backend.c | 35 ++++++++++++++++++++++++++++++-----\n 1 file changed, 30 insertions(+), 5 deletions(-)\n\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 57d8512fe80..6a60b26d1b9 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -547,7 +547,7 @@ struct reftable_ref_iterator {\n \tstruct reftable_ref_record ref;\n \tstruct object_id oid;\n \n-\tconst char *prefix;\n+\tchar *prefix;\n \tsize_t prefix_len;\n \tchar **exclude_patterns;\n \tsize_t exclude_patterns_index;\n@@ -718,6 +718,20 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ITER_OK;\n }\n \n+static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct reftable_ref_iterator *iter =\n+\t\t(struct reftable_ref_iterator *)ref_iterator;\n+\n+\tfree(iter->prefix);\n+\titer->prefix = xstrdup_or_null(prefix);\n+\titer->prefix_len = prefix ? strlen(prefix) : 0;\n+\titer->err = reftable_iterator_seek_ref(&iter->iter, prefix);\n+\n+\treturn iter->err;\n+}\n+\n static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -744,10 +758,12 @@ static void reftable_ref_iterator_release(struct ref_iterator *ref_iterator)\n \t\t\tfree(iter->exclude_patterns[i]);\n \t\tfree(iter->exclude_patterns);\n \t}\n+\tfree(iter->prefix);\n }\n \n static struct ref_iterator_vtable reftable_ref_iterator_vtable = {\n \t.advance = reftable_ref_iterator_advance,\n+\t.seek = reftable_ref_iterator_seek,\n \t.peel = reftable_ref_iterator_peel,\n \t.release = reftable_ref_iterator_release,\n };\n@@ -806,8 +822,6 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_\n \n \titer = xcalloc(1, sizeof(*iter));\n \tbase_ref_iterator_init(&iter->base, &reftable_ref_iterator_vtable);\n-\titer->prefix = prefix;\n-\titer->prefix_len = prefix ? strlen(prefix) : 0;\n \titer->base.oid = &iter->oid;\n \titer->flags = flags;\n \titer->refs = refs;\n@@ -821,8 +835,11 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_\n \tif (ret)\n \t\tgoto done;\n \n-\treftable_stack_init_ref_iterator(stack, &iter->iter);\n-\tret = reftable_iterator_seek_ref(&iter->iter, prefix);\n+\tret = reftable_stack_init_ref_iterator(stack, &iter->iter);\n+\tif (ret)\n+\t\tgoto done;\n+\n+\tret = reftable_ref_iterator_seek(&iter->base, prefix);\n \tif (ret)\n \t\tgoto done;\n \n@@ -2014,6 +2031,13 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ITER_OK;\n }\n \n+static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n+\t\t\t\t\t const char *prefix UNUSED)\n+{\n+\tBUG(\"reftable reflog iterator cannot be seeked\");\n+\treturn -1;\n+}\n+\n static int reftable_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \t\t\t\t\t struct object_id *peeled UNUSED)\n {\n@@ -2032,6 +2056,7 @@ static void reftable_reflog_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable reftable_reflog_iterator_vtable = {\n \t.advance = reftable_reflog_iterator_advance,\n+\t.seek = reftable_reflog_iterator_seek,\n \t.peel = reftable_reflog_iterator_peel,\n \t.release = reftable_reflog_iterator_release,\n };\n\n-- \n2.49.0.rc2.394.gf6994c5077.dirty\n\n"},{"id":"514084","messageId":"20250312-pks-update-ref-optimization-v6-14-f778e0414f55@pks.im","threadId":"62965","inReplyTo":"20250312-pks-update-ref-optimization-v6-0-f778e0414f55@pks.im","subject":"[PATCH v6 14/16] refs/iterator: implement seeking for packed-ref iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:56:20Z","receivedAt":"2025-03-12T15:56:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking of `packed-ref` iterators. The implementation is again\nstraight forward, except that we cannot continue to use the prefix\niterator as we would otherwise not be able to reseek the iterator\nanymore in case one first asks for an empty and then for a non-empty\nprefix. Instead, we open-code the logic to in `advance()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/packed-backend.c | 65 ++++++++++++++++++++++++++++++++++-----------------\n 1 file changed, 43 insertions(+), 22 deletions(-)\n\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 38a1956d1a8..f4c82ba2c7d 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -819,6 +819,8 @@ struct packed_ref_iterator {\n \n \tstruct snapshot *snapshot;\n \n+\tchar *prefix;\n+\n \t/* The current position in the snapshot's buffer: */\n \tconst char *pos;\n \n@@ -841,11 +843,9 @@ struct packed_ref_iterator {\n };\n \n /*\n- * Move the iterator to the next record in the snapshot, without\n- * respect for whether the record is actually required by the current\n- * iteration. Adjust the fields in `iter` and return `ITER_OK` or\n- * `ITER_DONE`. This function does not free the iterator in the case\n- * of `ITER_DONE`.\n+ * Move the iterator to the next record in the snapshot. Adjust the fields in\n+ * `iter` and return `ITER_OK` or `ITER_DONE`. This function does not free the\n+ * iterator in the case of `ITER_DONE`.\n */\n static int next_record(struct packed_ref_iterator *iter)\n {\n@@ -942,6 +942,9 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tint ok;\n \n \twhile ((ok = next_record(iter)) == ITER_OK) {\n+\t\tconst char *refname = iter->base.refname;\n+\t\tconst char *prefix = iter->prefix;\n+\n \t\tif (iter->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&\n \t\t !is_per_worktree_ref(iter->base.refname))\n \t\t\tcontinue;\n@@ -951,12 +954,41 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t\t\t\t &iter->oid, iter->flags))\n \t\t\tcontinue;\n \n+\t\twhile (prefix && *prefix) {\n+\t\t\tif (*refname < *prefix)\n+\t\t\t\tBUG(\"packed-refs backend yielded reference preceding its prefix\");\n+\t\t\telse if (*refname > *prefix)\n+\t\t\t\treturn ITER_DONE;\n+\t\t\tprefix++;\n+\t\t\trefname++;\n+\t\t}\n+\n \t\treturn ITER_OK;\n \t}\n \n \treturn ok;\n }\n \n+static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct packed_ref_iterator *iter =\n+\t\t(struct packed_ref_iterator *)ref_iterator;\n+\tconst char *start;\n+\n+\tif (prefix && *prefix)\n+\t\tstart = find_reference_location(iter->snapshot, prefix, 0);\n+\telse\n+\t\tstart = iter->snapshot->start;\n+\n+\tfree(iter->prefix);\n+\titer->prefix = xstrdup_or_null(prefix);\n+\titer->pos = start;\n+\titer->eof = iter->snapshot->eof;\n+\n+\treturn 0;\n+}\n+\n static int packed_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -979,11 +1011,13 @@ static void packed_ref_iterator_release(struct ref_iterator *ref_iterator)\n \t\t(struct packed_ref_iterator *)ref_iterator;\n \tstrbuf_release(&iter->refname_buf);\n \tfree(iter->jump);\n+\tfree(iter->prefix);\n \trelease_snapshot(iter->snapshot);\n }\n \n static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \t.advance = packed_ref_iterator_advance,\n+\t.seek = packed_ref_iterator_seek,\n \t.peel = packed_ref_iterator_peel,\n \t.release = packed_ref_iterator_release,\n };\n@@ -1097,7 +1131,6 @@ static struct ref_iterator *packed_ref_iterator_begin(\n {\n \tstruct packed_ref_store *refs;\n \tstruct snapshot *snapshot;\n-\tconst char *start;\n \tstruct packed_ref_iterator *iter;\n \tstruct ref_iterator *ref_iterator;\n \tunsigned int required_flags = REF_STORE_READ;\n@@ -1113,14 +1146,6 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \t */\n \tsnapshot = get_snapshot(refs);\n \n-\tif (prefix && *prefix)\n-\t\tstart = find_reference_location(snapshot, prefix, 0);\n-\telse\n-\t\tstart = snapshot->start;\n-\n-\tif (start == snapshot->eof)\n-\t\treturn empty_ref_iterator_begin();\n-\n \tCALLOC_ARRAY(iter, 1);\n \tref_iterator = &iter->base;\n \tbase_ref_iterator_init(ref_iterator, &packed_ref_iterator_vtable);\n@@ -1130,19 +1155,15 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \n \titer->snapshot = snapshot;\n \tacquire_snapshot(snapshot);\n-\n-\titer->pos = start;\n-\titer->eof = snapshot->eof;\n \tstrbuf_init(&iter->refname_buf, 0);\n-\n \titer->base.oid = &iter->oid;\n-\n \titer->repo = ref_store->repo;\n \titer->flags = flags;\n \n-\tif (prefix && *prefix)\n-\t\t/* Stop iteration after we've gone *past* prefix: */\n-\t\tref_iterator = prefix_ref_iterator_begin(ref_iterator, prefix, 0);\n+\tif (packed_ref_iterator_seek(&iter->base, prefix) < 0) {\n+\t\tref_iterator_free(&iter->base);\n+\t\treturn NULL;\n+\t}\n \n \treturn ref_iterator;\n }\n\n-- \n2.49.0.rc2.394.gf6994c5077.dirty\n\n"},{"id":"514085","messageId":"20250312-pks-update-ref-optimization-v6-13-f778e0414f55@pks.im","threadId":"62965","inReplyTo":"20250312-pks-update-ref-optimization-v6-0-f778e0414f55@pks.im","subject":"[PATCH v6 13/16] refs/iterator: implement seeking for ref-cache iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:56:19Z","receivedAt":"2025-03-12T15:56:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking of ref-cache iterators. This is done by splitting most\nof the logic to seek iterators out of `cache_ref_iterator_begin()` and\nputting it into `cache_ref_iterator_seek()` so that we can reuse the\nlogic.\n\nNote that we cannot use the optimization anymore where we return an\nempty ref iterator when there aren't any references, as otherwise it\nwouldn't be possible to reseek the iterator to a different prefix that\nmay exist. This shouldn't be much of a performance concern though as we\nnow start to bail out early in case `advance()` sees that there are no\nmore directories to be searched.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/ref-cache.c | 79 ++++++++++++++++++++++++++++++++++++--------------------\n 1 file changed, 51 insertions(+), 28 deletions(-)\n\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 6457e02c1ea..c1f1bab1d50 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -362,9 +362,7 @@ struct cache_ref_iterator {\n \tstruct ref_iterator base;\n \n \t/*\n-\t * The number of levels currently on the stack. This is always\n-\t * at least 1, because when it becomes zero the iteration is\n-\t * ended and this struct is freed.\n+\t * The number of levels currently on the stack.\n \t */\n \tsize_t levels_nr;\n \n@@ -376,7 +374,7 @@ struct cache_ref_iterator {\n \t * The prefix is matched textually, without regard for path\n \t * component boundaries.\n \t */\n-\tconst char *prefix;\n+\tchar *prefix;\n \n \t/*\n \t * A stack of levels. levels[0] is the uppermost level that is\n@@ -389,6 +387,9 @@ struct cache_ref_iterator {\n \tstruct cache_ref_iterator_level *levels;\n \n \tstruct repository *repo;\n+\tstruct ref_cache *cache;\n+\n+\tint prime_dir;\n };\n \n static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n@@ -396,6 +397,9 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \tstruct cache_ref_iterator *iter =\n \t\t(struct cache_ref_iterator *)ref_iterator;\n \n+\tif (!iter->levels_nr)\n+\t\treturn ITER_DONE;\n+\n \twhile (1) {\n \t\tstruct cache_ref_iterator_level *level =\n \t\t\t&iter->levels[iter->levels_nr - 1];\n@@ -444,6 +448,41 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t}\n }\n \n+static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct cache_ref_iterator *iter =\n+\t\t(struct cache_ref_iterator *)ref_iterator;\n+\tstruct cache_ref_iterator_level *level;\n+\tstruct ref_dir *dir;\n+\n+\tdir = get_ref_dir(iter->cache->root);\n+\tif (prefix && *prefix)\n+\t\tdir = find_containing_dir(dir, prefix);\n+\tif (!dir) {\n+\t\titer->levels_nr = 0;\n+\t\treturn 0;\n+\t}\n+\n+\tif (iter->prime_dir)\n+\t\tprime_ref_dir(dir, prefix);\n+\titer->levels_nr = 1;\n+\tlevel = &iter->levels[0];\n+\tlevel->index = -1;\n+\tlevel->dir = dir;\n+\n+\tif (prefix && *prefix) {\n+\t\tfree(iter->prefix);\n+\t\titer->prefix = xstrdup(prefix);\n+\t\tlevel->prefix_state = PREFIX_WITHIN_DIR;\n+\t} else {\n+\t\tFREE_AND_NULL(iter->prefix);\n+\t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -456,12 +495,13 @@ static void cache_ref_iterator_release(struct ref_iterator *ref_iterator)\n {\n \tstruct cache_ref_iterator *iter =\n \t\t(struct cache_ref_iterator *)ref_iterator;\n-\tfree((char *)iter->prefix);\n+\tfree(iter->prefix);\n \tfree(iter->levels);\n }\n \n static struct ref_iterator_vtable cache_ref_iterator_vtable = {\n \t.advance = cache_ref_iterator_advance,\n+\t.seek = cache_ref_iterator_seek,\n \t.peel = cache_ref_iterator_peel,\n \t.release = cache_ref_iterator_release,\n };\n@@ -471,39 +511,22 @@ struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\n \t\t\t\t\t struct repository *repo,\n \t\t\t\t\t int prime_dir)\n {\n-\tstruct ref_dir *dir;\n \tstruct cache_ref_iterator *iter;\n \tstruct ref_iterator *ref_iterator;\n-\tstruct cache_ref_iterator_level *level;\n-\n-\tdir = get_ref_dir(cache->root);\n-\tif (prefix && *prefix)\n-\t\tdir = find_containing_dir(dir, prefix);\n-\tif (!dir)\n-\t\t/* There's nothing to iterate over. */\n-\t\treturn empty_ref_iterator_begin();\n-\n-\tif (prime_dir)\n-\t\tprime_ref_dir(dir, prefix);\n \n \tCALLOC_ARRAY(iter, 1);\n \tref_iterator = &iter->base;\n \tbase_ref_iterator_init(ref_iterator, &cache_ref_iterator_vtable);\n \tALLOC_GROW(iter->levels, 10, iter->levels_alloc);\n \n-\titer->levels_nr = 1;\n-\tlevel = &iter->levels[0];\n-\tlevel->index = -1;\n-\tlevel->dir = dir;\n+\titer->repo = repo;\n+\titer->cache = cache;\n+\titer->prime_dir = prime_dir;\n \n-\tif (prefix && *prefix) {\n-\t\titer->prefix = xstrdup(prefix);\n-\t\tlevel->prefix_state = PREFIX_WITHIN_DIR;\n-\t} else {\n-\t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n+\tif (cache_ref_iterator_seek(&iter->base, prefix) < 0) {\n+\t\tref_iterator_free(&iter->base);\n+\t\treturn NULL;\n \t}\n \n-\titer->repo = repo;\n-\n \treturn ref_iterator;\n }\n\n-- \n2.49.0.rc2.394.gf6994c5077.dirty\n\n"},{"id":"514086","messageId":"20250312-pks-update-ref-optimization-v6-15-f778e0414f55@pks.im","threadId":"62965","inReplyTo":"20250312-pks-update-ref-optimization-v6-0-f778e0414f55@pks.im","subject":"[PATCH v6 15/16] refs/iterator: implement seeking for files iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:56:21Z","receivedAt":"2025-03-12T15:56:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Implement seeking for \"files\" iterators. As we simply use a ref-cache\niterator under the hood the implementation is straight-forward. Note\nthat we do not implement seeking on reflog iterators, same as with the\n\"reftable\" backend.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs/files-backend.c | 16 ++++++++++++++++\n 1 file changed, 16 insertions(+)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex e97a267ad65..5f921e85eb4 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -918,6 +918,14 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n+static int files_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *prefix)\n+{\n+\tstruct files_ref_iterator *iter =\n+\t\t(struct files_ref_iterator *)ref_iterator;\n+\treturn ref_iterator_seek(iter->iter0, prefix);\n+}\n+\n static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -936,6 +944,7 @@ static void files_ref_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable files_ref_iterator_vtable = {\n \t.advance = files_ref_iterator_advance,\n+\t.seek = files_ref_iterator_seek,\n \t.peel = files_ref_iterator_peel,\n \t.release = files_ref_iterator_release,\n };\n@@ -2294,6 +2303,12 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n+static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n+\t\t\t\t const char *prefix UNUSED)\n+{\n+\tBUG(\"ref_iterator_seek() called for reflog_iterator\");\n+}\n+\n static int files_reflog_iterator_peel(struct ref_iterator *ref_iterator UNUSED,\n \t\t\t\t struct object_id *peeled UNUSED)\n {\n@@ -2309,6 +2324,7 @@ static void files_reflog_iterator_release(struct ref_iterator *ref_iterator)\n \n static struct ref_iterator_vtable files_reflog_iterator_vtable = {\n \t.advance = files_reflog_iterator_advance,\n+\t.seek = files_reflog_iterator_seek,\n \t.peel = files_reflog_iterator_peel,\n \t.release = files_reflog_iterator_release,\n };\n\n-- \n2.49.0.rc2.394.gf6994c5077.dirty\n\n"},{"id":"514087","messageId":"20250312-pks-update-ref-optimization-v6-16-f778e0414f55@pks.im","threadId":"62965","inReplyTo":"20250312-pks-update-ref-optimization-v6-0-f778e0414f55@pks.im","subject":"[PATCH v6 16/16] refs: reuse iterators when determining refname availability","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-03-12T15:56:22Z","receivedAt":"2025-03-12T15:56:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When verifying whether refnames are available we have to verify whether\nany reference exists that is nested under the current reference. E.g.\ngiven a reference \"refs/heads/foo\", we must make sure that there is no\nother reference \"refs/heads/foo/*\".\n\nThis check is performed using a ref iterator with the prefix set to the\nnested reference namespace. Until now it used to not be possible to\nreseek iterators, so we always had to reallocate the iterator for every\nsingle reference we're about to check. This keeps us from reusing state\nthat the iterator may have and that may make it work more efficiently.\n\nRefactor the logic to reseek iterators. This leads to a sizeable speedup\nwith the \"reftable\" backend:\n\n Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 39.8 ms ± 0.9 ms [User: 29.7 ms, System: 9.8 ms]\n Range (min … max): 38.4 ms … 42.0 ms 62 runs\n\n Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 31.9 ms ± 1.1 ms [User: 27.0 ms, System: 4.5 ms]\n Range (min … max): 29.8 ms … 34.3 ms 74 runs\n\n Summary\n update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.25 ± 0.05 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)\n\nThe \"files\" backend doesn't really show a huge impact:\n\n Benchmark 1: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n Time (mean ± σ): 392.3 ms ± 7.1 ms [User: 59.7 ms, System: 328.8 ms]\n Range (min … max): 384.6 ms … 404.5 ms 10 runs\n\n Benchmark 2: update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD)\n Time (mean ± σ): 387.7 ms ± 7.4 ms [User: 54.6 ms, System: 329.6 ms]\n Range (min … max): 377.0 ms … 397.7 ms 10 runs\n\n Summary\n update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD) ran\n 1.01 ± 0.03 times faster than update-ref: create many refs (refformat = files, preexisting = 100000, new = 10000, revision = HEAD~)\n\nThis is mostly because it is way slower to begin with because it has to\ncreate a separate file for each new reference, so the milliseconds we\nshave off by reseeking the iterator doesn't really translate into a\nsignificant relative improvement.\n\nSigned-off-by: Patrick Steinhardt \n---\n refs.c | 12 +++++++-----\n 1 file changed, 7 insertions(+), 5 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 3e65ccad7ac..5f0dbee3aff 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2556,8 +2556,13 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \t\tif (!initial_transaction) {\n \t\t\tint ok;\n \n-\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n-\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n+\t\t\tif (!iter) {\n+\t\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n+\t\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n+\t\t\t} else if (ref_iterator_seek(iter, dirname.buf) < 0) {\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\n \t\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\t\t\tif (skip &&\n \t\t\t\t string_list_has_string(skip, iter->refname))\n@@ -2570,9 +2575,6 @@ int refs_verify_refnames_available(struct ref_store *refs,\n \n \t\t\tif (ok != ITER_DONE)\n \t\t\t\tBUG(\"error while iterating over references\");\n-\n-\t\t\tref_iterator_free(iter);\n-\t\t\titer = NULL;\n \t\t}\n \n \t\textra_refname = find_descendant_ref(dirname.buf, extras, skip);\n\n-- \n2.49.0.rc2.394.gf6994c5077.dirty\n\n"},{"id":"514152","messageId":"Z9JJpJkz66VVgDLy@ArchLinux","threadId":"62965","inReplyTo":"20250312-pks-update-ref-optimization-v6-0-f778e0414f55@pks.im","subject":"Re: [PATCH v6 00/16] refs: batch refname availability checks","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-03-13T02:57:40Z","receivedAt":"2025-03-13T02:57:30Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Wed, Mar 12, 2025 at 04:56:06PM +0100, Patrick Steinhardt wrote:\n> Changes in v6:\n> - Use `for_each_string_list()` instead of manually iterating through\n> the string list.\n> - Stop sorting refs passed to `refs_verify_refnames_available()`.\n> - Revive a comment that has been deleted during one of the\n> refactorings.\n> - Link to v5: https://lore.kernel.org/r/20250306-pks-update-ref-optimization-v5-0-dcb2ee037e97@pks.im\n> \n> Thanks!\n> \n> Patrick\n\nThe range-diff looks good to me. Thanks for your efforts.\n\nThanks,\nJialuo\n"},{"id":"515605","messageId":"CABPp-BFBqC_t5QSexRQpYsqXBa11WK+OqGt167E=K=xod=buQw@mail.gmail.com","threadId":"62965","inReplyTo":"20250312-pks-update-ref-optimization-v6-14-f778e0414f55@pks.im","subject":"Re: [PATCH v6 14/16] refs/iterator: implement seeking for packed-ref iterators","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2025-04-03T19:56:39Z","receivedAt":"2025-04-03T19:56:52Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Mar 12, 2025 at 11:42 PM Patrick Steinhardt wrote:\n>\n\n> @@ -951,12 +954,41 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> &iter->oid, iter->flags))\n> continue;\n>\n> + while (prefix && *prefix) {\n> + if (*refname < *prefix)\n> + BUG(\"packed-refs backend yielded reference preceding its prefix\");\n\nI just triggered this bug upon a \"git pull\" in an internal repository:\n\n$ git pull\nremote: Enumerating objects: 161255, done.\nremote: Counting objects: 100% (55884/55884), done.\nremote: Compressing objects: 100% (5518/5518), done.\nremote: Total 161255 (delta 54253), reused 50509 (delta 50364),\npack-reused 105371 (from 4)\nReceiving objects: 100% (161255/161255), 309.90 MiB | 16.87 MiB/s, done.\nResolving deltas: 100% (118048/118048), completed with 13416 local objects.\nFrom github.com:github/github\n 97ab7ae3f3745..8fb2f9fa180ed master\n -> origin/master\n[...snip many screenfuls of updates to origin remotes...]\nBUG: refs/packed-backend.c:984: packed-refs backend yielded reference\npreceding its prefix\nerror: fetch died of signal 6\n\n\n\nI made a backup of the repo with rsync.\n\nIt leaves around a lock file, which after I clear out and run `git\nfetch --update-head-ok` again, it'll hit the same error.\n\nIf I put a little debugging into the while loop:\n\n while (prefix && *prefix) {\n if (*refname < *prefix) {\n printf(\"Comparing %s to %s\\n\",\n refname, prefix);\n BUG(\"packed-refs backend yielded reference preceding\nits prefix\");\n\nthen my fetch produces:\n\nComparing 🍧 to zuora-params-orders-sorbet/\n(in case that doesn't come through well, that's a shaved-ice-emoji,\naccording to a google search, which someone decided to use within one\nof their branch names)\n\nIn this case, the utf-8 characters will have individual bytes whose\nvalues are greater than 127, which for a signed character will be\nrepresented by a negative number. I tried tweaking the loop to:\n\n while (prefix && *prefix) {\n if ((unsigned char)*refname < (unsigned char)*prefix) {\n printf(\"Comparing %d to %d\\n\",\n *refname, *prefix);\n printf(\"Comparing %s to %s\\n\",\n refname, prefix);\n BUG(\"packed-refs backend yielded reference preceding\nits prefix\");\n\nbut that just lets it proceed through the loop four times before dying:\n\nComparing 0 to 97\nComparing to a-params-orders-sorbet/\nBUG: refs/packed-backend.c:988: packed-refs backend yielded reference\npreceding its prefix\nAborted (core dumped)\n\nmeaning it eats up the four bytes from the shaved-ice-emoji and the\nfirst four bytes of prefix \"zuor\" and then hits the error, so I\nsuspect it needs to handle comparisons a little more broadly.\n"},{"id":"515614","messageId":"Z-8JN5ZXrgACgYvN@tapette.crustytoothpaste.net","threadId":"62965","inReplyTo":"CABPp-BFBqC_t5QSexRQpYsqXBa11WK+OqGt167E=K=xod=buQw@mail.gmail.com","subject":"Re: [PATCH v6 14/16] refs/iterator: implement seeking for packed-ref iterators","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2025-04-03T22:18:31Z","receivedAt":"2025-04-03T22:18:33Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2025-04-03 at 19:56:39, Elijah Newren wrote:\n> In this case, the utf-8 characters will have individual bytes whose\n> values are greater than 127, which for a signed character will be\n> represented by a negative number. I tried tweaking the loop to:\n> \n> while (prefix && *prefix) {\n> if ((unsigned char)*refname < (unsigned char)*prefix) {\n> printf(\"Comparing %d to %d\\n\",\n> *refname, *prefix);\n> printf(\"Comparing %s to %s\\n\",\n> refname, prefix);\n> BUG(\"packed-refs backend yielded reference preceding\n> its prefix\");\n\nI was mentioning this to Elijah earlier, but I think we might want to\nuse more emojis in testing things like refs and other text patterns in\nour testsuite. Because most of them are in astral planes, that will\nhelp us exercise our UTF-8 and UTF-16 surrogate handling, which is also\nrelevant to a variety of languages, like Chinese, which we obviously\nwant to support but which contributors may be less familiar with. Since\nemoji are more universal, it's easier for anyone to add a test for these\nkinds of cases and we might be able to catch some of these kinds of\nthings earlier.\n\nThat's in addition to the fact that many people use emojis in things\nlike commit messages because they're fun and we want to make those work.\n\nNext time I'm sending patches, I'll endeavour to cover a wider variety\nof codepoints in my testing.\n-- \nbrian m. carlson (they/them)\nToronto, Ontario, CA\n"},{"id":"515636","messageId":"Z--HtPsKissdMvuE@ArchLinux","threadId":"62965","inReplyTo":"Z-8JN5ZXrgACgYvN@tapette.crustytoothpaste.net","subject":"Re: [PATCH v6 14/16] refs/iterator: implement seeking for packed-ref iterators","fromName":"shejialuo","fromEmail":"shejialuo@gmail.com","sentAt":"2025-04-04T07:18:12Z","receivedAt":"2025-04-04T07:18:02Z","isPatch":true,"sender":{"key":"shejialuo@gmail.com","avatar":"https://avatars.githubusercontent.com/u/56911263?v=4"},"body":"On Thu, Apr 03, 2025 at 10:18:31PM +0000, brian m. carlson wrote:\n> On 2025-04-03 at 19:56:39, Elijah Newren wrote:\n> > In this case, the utf-8 characters will have individual bytes whose\n> > values are greater than 127, which for a signed character will be\n> > represented by a negative number. I tried tweaking the loop to:\n> > \n\nThat's good catch.\n\n> > while (prefix && *prefix) {\n> > if ((unsigned char)*refname < (unsigned char)*prefix) {\n> > printf(\"Comparing %d to %d\\n\",\n> > *refname, *prefix);\n> > printf(\"Comparing %s to %s\\n\",\n> > refname, prefix);\n> > BUG(\"packed-refs backend yielded reference preceding\n> > its prefix\");\n> \n> I was mentioning this to Elijah earlier, but I think we might want to\n> use more emojis in testing things like refs and other text patterns in\n> our testsuite. Because most of them are in astral planes, that will\n> help us exercise our UTF-8 and UTF-16 surrogate handling, which is also\n> relevant to a variety of languages, like Chinese, which we obviously\n> want to support but which contributors may be less familiar with. Since\n> emoji are more universal, it's easier for anyone to add a test for these\n> kinds of cases and we might be able to catch some of these kinds of\n> things earlier.\n> \n\nI somehow feel strange that people would like to use emoji(unicode) for\nref name, because if the user doesn't install the corresponding iconic\nfont, the user cannot see anything meaningful.\n\nAnyway, we may create unicode ref name or commit message to improve test\ncoverage.\n\n> That's in addition to the fact that many people use emojis in things\n> like commit messages because they're fun and we want to make those work.\n> \n> Next time I'm sending patches, I'll endeavour to cover a wider variety\n> of codepoints in my testing.\n> -- \n> brian m. carlson (they/them)\n> Toronto, Ontario, CA\n\nThanks,\nJialuo\n"},{"id":"515651","messageId":"Z--tomMthXftrdYA@pks.im","threadId":"62965","inReplyTo":"CABPp-BFBqC_t5QSexRQpYsqXBa11WK+OqGt167E=K=xod=buQw@mail.gmail.com","subject":"Re: [PATCH v6 14/16] refs/iterator: implement seeking for packed-ref iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-04T10:00:02Z","receivedAt":"2025-04-04T10:00:11Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Apr 03, 2025 at 12:56:39PM -0700, Elijah Newren wrote:\n> On Wed, Mar 12, 2025 at 11:42 PM Patrick Steinhardt wrote:\n> >\n> \n> > @@ -951,12 +954,41 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> > &iter->oid, iter->flags))\n> > continue;\n> >\n> > + while (prefix && *prefix) {\n> > + if (*refname < *prefix)\n> > + BUG(\"packed-refs backend yielded reference preceding its prefix\");\n> \n> I just triggered this bug upon a \"git pull\" in an internal repository:\n> \n> $ git pull\n> remote: Enumerating objects: 161255, done.\n> remote: Counting objects: 100% (55884/55884), done.\n> remote: Compressing objects: 100% (5518/5518), done.\n> remote: Total 161255 (delta 54253), reused 50509 (delta 50364),\n> pack-reused 105371 (from 4)\n> Receiving objects: 100% (161255/161255), 309.90 MiB | 16.87 MiB/s, done.\n> Resolving deltas: 100% (118048/118048), completed with 13416 local objects.\n> From github.com:github/github\n> 97ab7ae3f3745..8fb2f9fa180ed master\n> -> origin/master\n> [...snip many screenfuls of updates to origin remotes...]\n> BUG: refs/packed-backend.c:984: packed-refs backend yielded reference\n> preceding its prefix\n> error: fetch died of signal 6\n> \n> I made a backup of the repo with rsync.\n\nThanks, I can indeed reproduce the issue rather easily:\n\n\ttest_expect_success 'list refs with unicode characters' '\n\t\ttest_when_finished \"rm -rf repo\" &&\n\t\tgit init repo &&\n\t\t(\n\t\t\tcd repo &&\n\t\t\ttest_commit A &&\n\t\t\tgit update-ref refs/heads/ HEAD &&\n\t\t\tgit pack-refs --all &&\n\t\t\tgit for-each-ref refs/heads/z\n\t\t)\n\t'\n\nI'll investigate.\n\nPatrick\n"},{"id":"515652","messageId":"Z--u2DlaPij_j7zX@pks.im","threadId":"62965","inReplyTo":"Z--tomMthXftrdYA@pks.im","subject":"Re: [PATCH v6 14/16] refs/iterator: implement seeking for packed-ref iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-04T10:05:12Z","receivedAt":"2025-04-04T10:05:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Apr 04, 2025 at 12:00:07PM +0200, Patrick Steinhardt wrote:\n> On Thu, Apr 03, 2025 at 12:56:39PM -0700, Elijah Newren wrote:\n> > On Wed, Mar 12, 2025 at 11:42 PM Patrick Steinhardt wrote:\n> > >\n> > \n> > > @@ -951,12 +954,41 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> > > &iter->oid, iter->flags))\n> > > continue;\n> > >\n> > > + while (prefix && *prefix) {\n> > > + if (*refname < *prefix)\n> > > + BUG(\"packed-refs backend yielded reference preceding its prefix\");\n> > \n> > I just triggered this bug upon a \"git pull\" in an internal repository:\n> > \n> > $ git pull\n> > remote: Enumerating objects: 161255, done.\n> > remote: Counting objects: 100% (55884/55884), done.\n> > remote: Compressing objects: 100% (5518/5518), done.\n> > remote: Total 161255 (delta 54253), reused 50509 (delta 50364),\n> > pack-reused 105371 (from 4)\n> > Receiving objects: 100% (161255/161255), 309.90 MiB | 16.87 MiB/s, done.\n> > Resolving deltas: 100% (118048/118048), completed with 13416 local objects.\n> > From github.com:github/github\n> > 97ab7ae3f3745..8fb2f9fa180ed master\n> > -> origin/master\n> > [...snip many screenfuls of updates to origin remotes...]\n> > BUG: refs/packed-backend.c:984: packed-refs backend yielded reference\n> > preceding its prefix\n> > error: fetch died of signal 6\n> > \n> > I made a backup of the repo with rsync.\n> \n> Thanks, I can indeed reproduce the issue rather easily:\n> \n> \ttest_expect_success 'list refs with unicode characters' '\n> \t\ttest_when_finished \"rm -rf repo\" &&\n> \t\tgit init repo &&\n> \t\t(\n> \t\t\tcd repo &&\n> \t\t\ttest_commit A &&\n> \t\t\tgit update-ref refs/heads/ HEAD &&\n> \t\t\tgit pack-refs --all &&\n> \t\t\tgit for-each-ref refs/heads/z\n> \t\t)\n> \t'\n> \n> I'll investigate.\n> \n> Patrick\n\nOkay, below patch should fix the issue. The problem is that the sorting\nwe use for refnames is done via `cmp_packed_refname()`, which does the\nsame cast. And because the uppermost bit is set for the emoji character\nthis causes us to compare diferently in `packed_ref_iterator_advance()`\nand thus causes the bug.\n\nCould you please test whether this works for you? Once confirmed I'll\nsend a proper patch.\n\nThanks!\n\nPatrick\n\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex b4289a7d9ce..7e31904bd41 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -980,9 +980,9 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t\tcontinue;\n \n \t\twhile (prefix && *prefix) {\n-\t\t\tif (*refname < *prefix)\n+\t\t\tif ((unsigned char)*refname < (unsigned char)*prefix)\n \t\t\t\tBUG(\"packed-refs backend yielded reference preceding its prefix\");\n-\t\t\telse if (*refname > *prefix)\n+\t\t\telse if ((unsigned char)*refname > (unsigned char)*prefix)\n \t\t\t\treturn ITER_DONE;\n \t\t\tprefix++;\n \t\t\trefname++;\n"},{"id":"515655","messageId":"Z--7nfaKTpbBXbV9@pks.im","threadId":"62965","inReplyTo":"Z--u2DlaPij_j7zX@pks.im","subject":"Re: [PATCH v6 14/16] refs/iterator: implement seeking for packed-ref iterators","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-04-04T10:59:41Z","receivedAt":"2025-04-04T10:59:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Apr 04, 2025 at 12:05:12PM +0200, Patrick Steinhardt wrote:\n> On Fri, Apr 04, 2025 at 12:00:07PM +0200, Patrick Steinhardt wrote:\n> > On Thu, Apr 03, 2025 at 12:56:39PM -0700, Elijah Newren wrote:\n> > > On Wed, Mar 12, 2025 at 11:42 PM Patrick Steinhardt wrote:\n> > > >\n> > > \n> > > > @@ -951,12 +954,41 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> > > > &iter->oid, iter->flags))\n> > > > continue;\n> > > >\n> > > > + while (prefix && *prefix) {\n> > > > + if (*refname < *prefix)\n> > > > + BUG(\"packed-refs backend yielded reference preceding its prefix\");\n> > > \n> > > I just triggered this bug upon a \"git pull\" in an internal repository:\n> > > \n> > > $ git pull\n> > > remote: Enumerating objects: 161255, done.\n> > > remote: Counting objects: 100% (55884/55884), done.\n> > > remote: Compressing objects: 100% (5518/5518), done.\n> > > remote: Total 161255 (delta 54253), reused 50509 (delta 50364),\n> > > pack-reused 105371 (from 4)\n> > > Receiving objects: 100% (161255/161255), 309.90 MiB | 16.87 MiB/s, done.\n> > > Resolving deltas: 100% (118048/118048), completed with 13416 local objects.\n> > > From github.com:github/github\n> > > 97ab7ae3f3745..8fb2f9fa180ed master\n> > > -> origin/master\n> > > [...snip many screenfuls of updates to origin remotes...]\n> > > BUG: refs/packed-backend.c:984: packed-refs backend yielded reference\n> > > preceding its prefix\n> > > error: fetch died of signal 6\n> > > \n> > > I made a backup of the repo with rsync.\n> > \n> > Thanks, I can indeed reproduce the issue rather easily:\n> > \n> > \ttest_expect_success 'list refs with unicode characters' '\n> > \t\ttest_when_finished \"rm -rf repo\" &&\n> > \t\tgit init repo &&\n> > \t\t(\n> > \t\t\tcd repo &&\n> > \t\t\ttest_commit A &&\n> > \t\t\tgit update-ref refs/heads/ HEAD &&\n> > \t\t\tgit pack-refs --all &&\n> > \t\t\tgit for-each-ref refs/heads/z\n> > \t\t)\n> > \t'\n> > \n> > I'll investigate.\n> > \n> > Patrick\n> \n> Okay, below patch should fix the issue. The problem is that the sorting\n> we use for refnames is done via `cmp_packed_refname()`, which does the\n> same cast. And because the uppermost bit is set for the emoji character\n> this causes us to compare diferently in `packed_ref_iterator_advance()`\n> and thus causes the bug.\n> \n> Could you please test whether this works for you? Once confirmed I'll\n> send a proper patch.\n\nSent via [1]. Let's continue with the discussion over there.\n\nPatrick\n\n[1]: <20250404-b4-pks-packed-backend-seek-with-utf8-v1-1-6ceb694e3bd7@pks.im>\n"}]}