{"thread":{"id":"63722","subject":"[PATCH 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","startedAt":"2025-07-01T15:04:13Z","lastAt":"2025-07-24T22:30:28Z","messageCount":102,"participants":["Karthik Nayak","Junio C Hamano","Phillip Wood","Patrick Steinhardt","Andreas Schwab","Christian Couder","Kyle Lippincott","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"521052","messageId":"20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com","threadId":"63722","inReplyTo":null,"subject":"[PATCH 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-01T15:03:26Z","receivedAt":"2025-07-01T15:04:13Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `git-for-each-ref(1)` command is used to iterate over references\npresent in a repository. In large repositories with millions of\nreferences, it would be optimal to paginate this output such that we\ncan start iteration from a given reference. This would avoid having to\niterate over all references from the beginning each time when paginating\nthrough results.\n\nThis series adds a '--skip-until' option in 'git-for-each-ref(1)'. When\nused, the reference iteration seeks to the first matching reference and\niterates from there onward.\n\nThis enables efficient pagination workflows like:\n git for-each-ref --count=100\n git for-each-ref --count=100 --skip-until=refs/heads/branch-100\n git for-each-ref --count=100 --skip-until=refs/heads/branch-200\n\nTo add this functionality, we expose the `ref_iterator` outside the\n'refs/' namespace and modify the `ref_iterator_seek()` to actually seek\nto a given reference and only set the prefix when the `set_prefix` field\nis set.\n\nOn the reftable and packed backend, the changes are simple. But since\nthe files backend uses 'ref-cache' for reference handling, the changes\nthere are a little more involved, since we need to setup the right\nlevels and the indexing.\n\nInitally I was also planning to cleanup all the `refs_for_each...()`\nfunctions in 'refs.h' by simply using the iterator, but this bloated the\nseries. So I've left that for another day.\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/git-for-each-ref.adoc | 6 +-\n builtin/for-each-ref.c | 5 +\n ref-filter.c | 57 ++++++++----\n ref-filter.h | 1 +\n refs.c | 2 +-\n refs.h | 153 ++++++++++++++++++++++++++++++\n refs/debug.c | 7 +-\n refs/files-backend.c | 7 +-\n refs/iterator.c | 24 +++--\n refs/packed-backend.c | 15 +--\n refs/ref-cache.c | 95 +++++++++++++++----\n refs/ref-cache.h | 7 --\n refs/refs-internal.h | 152 ++----------------------------\n refs/reftable-backend.c | 17 ++--\n t/t6302-for-each-ref-filter.sh | 180 ++++++++++++++++++++++++++++++++++++\n 15 files changed, 507 insertions(+), 221 deletions(-)\n\nKarthik Nayak (4):\n refs: expose `ref_iterator` via 'refs.h'\n ref-cache: remove unused function 'find_ref_entry()'\n refs: selectively set prefix in the seek functions\n for-each-ref: introduce a '--skip-until' option\n\n\n\nbase-commit: cf6f63ea6bf35173e02e18bdc6a4ba41288acff9\nchange-id: 20250605-306-git-for-each-ref-pagination-0ba8a29ae646\n\nThanks\n- Karthik\n\n"},{"id":"521053","messageId":"20250701-306-git-for-each-ref-pagination-v1-2-4f0ae7c0688f@gmail.com","threadId":"63722","inReplyTo":"20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com","subject":"[PATCH 2/4] ref-cache: remove unused function 'find_ref_entry()'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-01T15:03:28Z","receivedAt":"2025-07-01T15:04:14Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The 'find_ref_entry' function is no longer used, so remove it.\n\nSigned-off-by: Karthik Nayak \n---\n refs/ref-cache.c | 14 --------------\n refs/ref-cache.h | 7 -------\n 2 files changed, 21 deletions(-)\n\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex c1f1bab1d5..8aaffa8c6b 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -194,20 +194,6 @@ static struct ref_dir *find_containing_dir(struct ref_dir *dir,\n \treturn dir;\n }\n \n-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname)\n-{\n-\tint entry_index;\n-\tstruct ref_entry *entry;\n-\tdir = find_containing_dir(dir, refname);\n-\tif (!dir)\n-\t\treturn NULL;\n-\tentry_index = search_ref_dir(dir, refname, strlen(refname));\n-\tif (entry_index == -1)\n-\t\treturn NULL;\n-\tentry = dir->entries[entry_index];\n-\treturn (entry->flag & REF_DIR) ? NULL : entry;\n-}\n-\n /*\n * Emit a warning and return true iff ref1 and ref2 have the same name\n * and the same oid. Die if they have the same name but different\ndiff --git a/refs/ref-cache.h b/refs/ref-cache.h\nindex 5f04e518c3..f635d2d824 100644\n--- a/refs/ref-cache.h\n+++ b/refs/ref-cache.h\n@@ -201,13 +201,6 @@ void free_ref_cache(struct ref_cache *cache);\n */\n void add_entry_to_dir(struct ref_dir *dir, struct ref_entry *entry);\n \n-/*\n- * Find the value entry with the given name in dir, sorting ref_dirs\n- * and recursing into subdirectories as necessary. If the name is not\n- * found or it corresponds to a directory entry, return NULL.\n- */\n-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname);\n-\n /*\n * Start iterating over references in `cache`. If `prefix` is\n * specified, only include references whose names start with that\n\n-- \n2.49.0\n\n"},{"id":"521054","messageId":"20250701-306-git-for-each-ref-pagination-v1-1-4f0ae7c0688f@gmail.com","threadId":"63722","inReplyTo":"20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com","subject":"[PATCH 1/4] refs: expose `ref_iterator` via 'refs.h'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-01T15:03:27Z","receivedAt":"2025-07-01T15:04:14Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `ref_iterator` is an internal structure to the 'refs/'\nsub-directory, which allows iteration over refs. All reference iteration\nis built on top of these iterators.\n\nExternal clients of the 'refs' subsystem use the various\n'refs_for_each...()' functions to iterate over refs. However since these\nare wrapper functions, each combination of functionality requires a new\nwrapper function. This is not feasible as the functions pile up with the\nincrease in requirements. Expose the internal reference iterator, so\nadvanced users can mix and match options as needed.\n\nSigned-off-by: Karthik Nayak \n---\n refs.h | 148 +++++++++++++++++++++++++++++++++++++++++++++++++++\n refs/refs-internal.h | 145 +------------------------------------------------\n 2 files changed, 150 insertions(+), 143 deletions(-)\n\ndiff --git a/refs.h b/refs.h\nindex 46a6008e07..c05be6d0ac 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -1190,4 +1190,152 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n \t\t\t\t unsigned int flags,\n \t\t\t\t struct strbuf *err);\n \n+/*\n+ * Reference iterators\n+ *\n+ * A reference iterator encapsulates the state of an in-progress\n+ * iteration over references. Create an instance of `struct\n+ * ref_iterator` via one of the functions in this module.\n+ *\n+ * A freshly-created ref_iterator doesn't yet point at a reference. To\n+ * advance the iterator, call ref_iterator_advance(). If successful,\n+ * this sets the iterator's refname, oid, and flags fields to describe\n+ * the next reference and returns ITER_OK. The data pointed at by\n+ * refname and oid belong to the iterator; if you want to retain them\n+ * after calling ref_iterator_advance() again or calling\n+ * ref_iterator_free(), you must make a copy. When the iteration has\n+ * been exhausted, ref_iterator_advance() releases any resources\n+ * associated with the iteration, frees the ref_iterator object, and\n+ * returns ITER_DONE. If you want to abort the iteration early, call\n+ * ref_iterator_free(), which also frees the ref_iterator object and\n+ * any associated resources. If there was an internal error advancing\n+ * to the next entry, ref_iterator_advance() aborts the iteration,\n+ * frees the ref_iterator, and returns ITER_ERROR.\n+ *\n+ * The reference currently being looked at can be peeled by calling\n+ * ref_iterator_peel(). This function is often faster than peel_ref(),\n+ * so it should be preferred when iterating over references.\n+ *\n+ * Putting it all together, a typical iteration looks like this:\n+ *\n+ * int ok;\n+ * struct ref_iterator *iter = ...;\n+ *\n+ * while ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n+ * if (want_to_stop_iteration()) {\n+ * ok = ITER_DONE;\n+ * break;\n+ * }\n+ *\n+ * // Access information about the current reference:\n+ * if (!(iter->flags & REF_ISSYMREF))\n+ * printf(\"%s is %s\\n\", iter->refname, oid_to_hex(iter->oid));\n+ *\n+ * // If you need to peel the reference:\n+ * ref_iterator_peel(iter, &oid);\n+ * }\n+ *\n+ * if (ok != ITER_DONE)\n+ * handle_error();\n+ * ref_iterator_free(iter);\n+ */\n+struct ref_iterator;\n+\n+/*\n+ * These flags are passed to refs_ref_iterator_begin() (and do_for_each_ref(),\n+ * which feeds it).\n+ */\n+enum do_for_each_ref_flags {\n+\t/*\n+\t * Include broken references in a do_for_each_ref*() iteration, which\n+\t * would normally be omitted. This includes both refs that point to\n+\t * missing objects (a true repository corruption), ones with illegal\n+\t * names (which we prefer not to expose to callers), as well as\n+\t * dangling symbolic refs (i.e., those that point to a non-existent\n+\t * ref; this is not a corruption, but as they have no valid oid, we\n+\t * omit them from normal iteration results).\n+\t */\n+\tDO_FOR_EACH_INCLUDE_BROKEN = (1 << 0),\n+\n+\t/*\n+\t * Only include per-worktree refs in a do_for_each_ref*() iteration.\n+\t * Normally this will be used with a files ref_store, since that's\n+\t * where all reference backends will presumably store their\n+\t * per-worktree refs.\n+\t */\n+\tDO_FOR_EACH_PER_WORKTREE_ONLY = (1 << 1),\n+\n+\t/*\n+\t * Omit dangling symrefs from output; this only has an effect with\n+\t * INCLUDE_BROKEN, since they are otherwise not included at all.\n+\t */\n+\tDO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 << 2),\n+\n+\t/*\n+\t * Include root refs i.e. HEAD and pseudorefs along with the regular\n+\t * refs.\n+\t */\n+\tDO_FOR_EACH_INCLUDE_ROOT_REFS = (1 << 3),\n+};\n+\n+/*\n+ * Return an iterator that goes over each reference in `refs` for\n+ * which the refname begins with prefix. If trim is non-zero, then\n+ * trim that many characters off the beginning of each refname.\n+ * The output is ordered by refname.\n+ */\n+struct ref_iterator *refs_ref_iterator_begin(\n+\t\tstruct ref_store *refs,\n+\t\tconst char *prefix, const char **exclude_patterns,\n+\t\tint trim, enum do_for_each_ref_flags flags);\n+\n+/*\n+ * Advance the iterator to the first or next item and return ITER_OK.\n+ * If the iteration is exhausted, free the resources associated with\n+ * the ref_iterator and return ITER_DONE. On errors, free the iterator\n+ * resources and return ITER_ERROR. It is a bug to use ref_iterator or\n+ * call this function again after it has returned ITER_DONE or\n+ * ITER_ERROR.\n+ */\n+int ref_iterator_advance(struct ref_iterator *ref_iterator);\n+\n+/*\n+ * Seek the iterator to the first reference with the given prefix.\n+ * The prefix is matched as a literal string, without regard for path\n+ * separators. If prefix is NULL or the empty string, seek the iterator to the\n+ * first reference again.\n+ *\n+ * This function is expected to behave as if a new ref iterator with the same\n+ * prefix had been created, but allows reuse of iterators and thus may allow\n+ * the backend to optimize. Parameters other than the prefix that have been\n+ * passed when creating the iterator will remain unchanged.\n+ *\n+ * Returns 0 on success, a negative error code otherwise.\n+ */\n+int ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t const char *prefix);\n+\n+/*\n+ * If possible, peel the reference currently being viewed by the\n+ * iterator. Return 0 on success.\n+ */\n+int ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t struct object_id *peeled);\n+\n+/* Free the reference iterator and any associated resources. */\n+void ref_iterator_free(struct ref_iterator *ref_iterator);\n+\n+/*\n+ * The common backend for the for_each_*ref* functions. Call fn for\n+ * each reference in iter. If the iterator itself ever returns\n+ * ITER_ERROR, return -1. If fn ever returns a non-zero value, stop\n+ * the iteration and return that value. Otherwise, return 0. In any\n+ * case, free the iterator when done. This function is basically an\n+ * adapter between the callback style of reference iteration and the\n+ * iterator style.\n+ */\n+int do_for_each_ref_iterator(struct ref_iterator *iter,\n+\t\t\t each_ref_fn fn, void *cb_data);\n+\n+\n #endif /* REFS_H */\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex f868870851..03f5df04d5 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -244,90 +244,8 @@ const char *find_descendant_ref(const char *dirname,\n #define SYMREF_MAXDEPTH 5\n \n /*\n- * These flags are passed to refs_ref_iterator_begin() (and do_for_each_ref(),\n- * which feeds it).\n- */\n-enum do_for_each_ref_flags {\n-\t/*\n-\t * Include broken references in a do_for_each_ref*() iteration, which\n-\t * would normally be omitted. This includes both refs that point to\n-\t * missing objects (a true repository corruption), ones with illegal\n-\t * names (which we prefer not to expose to callers), as well as\n-\t * dangling symbolic refs (i.e., those that point to a non-existent\n-\t * ref; this is not a corruption, but as they have no valid oid, we\n-\t * omit them from normal iteration results).\n-\t */\n-\tDO_FOR_EACH_INCLUDE_BROKEN = (1 << 0),\n-\n-\t/*\n-\t * Only include per-worktree refs in a do_for_each_ref*() iteration.\n-\t * Normally this will be used with a files ref_store, since that's\n-\t * where all reference backends will presumably store their\n-\t * per-worktree refs.\n-\t */\n-\tDO_FOR_EACH_PER_WORKTREE_ONLY = (1 << 1),\n-\n-\t/*\n-\t * Omit dangling symrefs from output; this only has an effect with\n-\t * INCLUDE_BROKEN, since they are otherwise not included at all.\n-\t */\n-\tDO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 << 2),\n-\n-\t/*\n-\t * Include root refs i.e. HEAD and pseudorefs along with the regular\n-\t * refs.\n-\t */\n-\tDO_FOR_EACH_INCLUDE_ROOT_REFS = (1 << 3),\n-};\n-\n-/*\n- * Reference iterators\n- *\n- * A reference iterator encapsulates the state of an in-progress\n- * iteration over references. Create an instance of `struct\n- * ref_iterator` via one of the functions in this module.\n- *\n- * A freshly-created ref_iterator doesn't yet point at a reference. To\n- * advance the iterator, call ref_iterator_advance(). If successful,\n- * this sets the iterator's refname, oid, and flags fields to describe\n- * the next reference and returns ITER_OK. The data pointed at by\n- * refname and oid belong to the iterator; if you want to retain them\n- * after calling ref_iterator_advance() again or calling\n- * ref_iterator_free(), you must make a copy. When the iteration has\n- * been exhausted, ref_iterator_advance() releases any resources\n- * associated with the iteration, frees the ref_iterator object, and\n- * returns ITER_DONE. If you want to abort the iteration early, call\n- * ref_iterator_free(), which also frees the ref_iterator object and\n- * any associated resources. If there was an internal error advancing\n- * to the next entry, ref_iterator_advance() aborts the iteration,\n- * frees the ref_iterator, and returns ITER_ERROR.\n- *\n- * The reference currently being looked at can be peeled by calling\n- * ref_iterator_peel(). This function is often faster than peel_ref(),\n- * so it should be preferred when iterating over references.\n- *\n- * Putting it all together, a typical iteration looks like this:\n- *\n- * int ok;\n- * struct ref_iterator *iter = ...;\n- *\n- * while ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n- * if (want_to_stop_iteration()) {\n- * ok = ITER_DONE;\n- * break;\n- * }\n- *\n- * // Access information about the current reference:\n- * if (!(iter->flags & REF_ISSYMREF))\n- * printf(\"%s is %s\\n\", iter->refname, oid_to_hex(iter->oid));\n- *\n- * // If you need to peel the reference:\n- * ref_iterator_peel(iter, &oid);\n- * }\n- *\n- * if (ok != ITER_DONE)\n- * handle_error();\n- * ref_iterator_free(iter);\n+ * Data structure for holding a reference iterator. See refs.h for\n+ * more details and usage instructions.\n */\n struct ref_iterator {\n \tstruct ref_iterator_vtable *vtable;\n@@ -337,42 +255,6 @@ struct ref_iterator {\n \tunsigned int flags;\n };\n \n-/*\n- * Advance the iterator to the first or next item and return ITER_OK.\n- * If the iteration is exhausted, free the resources associated with\n- * the ref_iterator and return ITER_DONE. On errors, free the iterator\n- * resources and return ITER_ERROR. It is a bug to use ref_iterator or\n- * call this function again after it has returned ITER_DONE or\n- * ITER_ERROR.\n- */\n-int ref_iterator_advance(struct ref_iterator *ref_iterator);\n-\n-/*\n- * Seek the iterator to the first reference with the given prefix.\n- * The prefix is matched as a literal string, without regard for path\n- * separators. If prefix is NULL or the empty string, seek the iterator to the\n- * first reference again.\n- *\n- * This function is expected to behave as if a new ref iterator with the same\n- * prefix had been created, but allows reuse of iterators and thus may allow\n- * the backend to optimize. Parameters other than the prefix that have been\n- * passed when creating the iterator will remain unchanged.\n- *\n- * Returns 0 on success, a negative error code otherwise.\n- */\n-int ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t const char *prefix);\n-\n-/*\n- * If possible, peel the reference currently being viewed by the\n- * iterator. Return 0 on success.\n- */\n-int ref_iterator_peel(struct ref_iterator *ref_iterator,\n-\t\t struct object_id *peeled);\n-\n-/* Free the reference iterator and any associated resources. */\n-void ref_iterator_free(struct ref_iterator *ref_iterator);\n-\n /*\n * An iterator over nothing (its first ref_iterator_advance() call\n * returns ITER_DONE).\n@@ -384,17 +266,6 @@ struct ref_iterator *empty_ref_iterator_begin(void);\n */\n int is_empty_ref_iterator(struct ref_iterator *ref_iterator);\n \n-/*\n- * Return an iterator that goes over each reference in `refs` for\n- * which the refname begins with prefix. If trim is non-zero, then\n- * trim that many characters off the beginning of each refname.\n- * The output is ordered by refname.\n- */\n-struct ref_iterator *refs_ref_iterator_begin(\n-\t\tstruct ref_store *refs,\n-\t\tconst char *prefix, const char **exclude_patterns,\n-\t\tint trim, enum do_for_each_ref_flags flags);\n-\n /*\n * A callback function used to instruct merge_ref_iterator how to\n * interleave the entries from iter0 and iter1. The function should\n@@ -520,18 +391,6 @@ struct ref_iterator_vtable {\n */\n extern struct ref_iterator *current_ref_iter;\n \n-/*\n- * The common backend for the for_each_*ref* functions. Call fn for\n- * each reference in iter. If the iterator itself ever returns\n- * ITER_ERROR, return -1. If fn ever returns a non-zero value, stop\n- * the iteration and return that value. Otherwise, return 0. In any\n- * case, free the iterator when done. This function is basically an\n- * adapter between the callback style of reference iteration and the\n- * iterator style.\n- */\n-int do_for_each_ref_iterator(struct ref_iterator *iter,\n-\t\t\t each_ref_fn fn, void *cb_data);\n-\n struct ref_store;\n \n /* refs backends */\n\n-- \n2.49.0\n\n"},{"id":"521055","messageId":"20250701-306-git-for-each-ref-pagination-v1-3-4f0ae7c0688f@gmail.com","threadId":"63722","inReplyTo":"20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com","subject":"[PATCH 3/4] refs: selectively set prefix in the seek functions","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-01T15:03:29Z","receivedAt":"2025-07-01T15:04:16Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The ref iterator exposes a `ref_iterator_seek()` function. The name\nsuggests that this would seek the iterator to a specific reference in\nsome ways similar to how `fseek()` works for the filesystem.\n\nHowever, the function actually sets the prefix for refs iteration. So\nfurther iteration would only yield references which match the particular\nprefix. This is a bit confusing.\n\nLet's add a 'set_prefix' field to the function, which when set, will set\nthe prefix for the iteration in-line with the existing behavior. But\nwhen the 'set_prefix' field is not set, the reference backends will\nsimply seek to the specified reference without setting prefix. This\nallows users to start iteration from a specific reference.\n\nIn the packed and reftable backend, since references are available in a\nsorted list, the changes are simply setting the prefix if needed. The\nchanges on the files-backend are a little more involved, since the files\nbackend uses the 'ref-cache' mechanism. We move out the existing logic\nwithin `cache_ref_iterator_seek()` to `cache_ref_iterator_set_prefix()`\nwhich is called when `set_prefix` is set. We then parse the provided\nseek string and set the required levels and their indexes to ensure that\nseeking is possible.\n\nSigned-off-by: Karthik Nayak \n---\n refs.c | 2 +-\n refs.h | 19 +++++++-----\n refs/debug.c | 7 +++--\n refs/files-backend.c | 7 +++--\n refs/iterator.c | 24 +++++++++------\n refs/packed-backend.c | 15 +++++----\n refs/ref-cache.c | 81 ++++++++++++++++++++++++++++++++++++++++++++++---\n refs/refs-internal.h | 7 +++--\n refs/reftable-backend.c | 17 ++++++-----\n 9 files changed, 134 insertions(+), 45 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex dce5c49ca2..a4220d3537 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2669,7 +2669,7 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs\n \t\t\tif (!iter) {\n \t\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n \t\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n-\t\t\t} else if (ref_iterator_seek(iter, dirname.buf) < 0) {\n+\t\t\t} else if (ref_iterator_seek(iter, dirname.buf, 1) < 0) {\n \t\t\t\tgoto cleanup;\n \t\t\t}\n \ndiff --git a/refs.h b/refs.h\nindex c05be6d0ac..c5e08db0ff 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -1300,20 +1300,25 @@ struct ref_iterator *refs_ref_iterator_begin(\n int ref_iterator_advance(struct ref_iterator *ref_iterator);\n \n /*\n- * Seek the iterator to the first reference with the given prefix.\n- * The prefix is matched as a literal string, without regard for path\n+ * Seek the iterator to the first reference matching the given seek string.\n+ * The seek string is matched as a literal string, without regard for path\n * separators. If prefix is NULL or the empty string, seek the iterator to the\n * first reference again.\n *\n- * This function is expected to behave as if a new ref iterator with the same\n- * prefix had been created, but allows reuse of iterators and thus may allow\n- * the backend to optimize. Parameters other than the prefix that have been\n- * passed when creating the iterator will remain unchanged.\n+ * When set_prefix is true, this function behaves as if a new ref iterator\n+ * with the same prefix had been created, setting the prefix for subsequent\n+ * iteration. When set_prefix is false, the iterator simply seeks to the\n+ * specified reference without changing the existing prefix, allowing\n+ * iteration to start from that specific reference.\n+ *\n+ * This function allows reuse of iterators and thus may allow the backend\n+ * to optimize. Parameters other than the prefix that have been passed when\n+ * creating the iterator will remain unchanged.\n *\n * Returns 0 on success, a negative error code otherwise.\n */\n int ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t const char *prefix);\n+\t\t const char *seek, int set_prefix);\n \n /*\n * If possible, peel the reference currently being viewed by the\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 485e3079d7..7c04bcba10 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -170,12 +170,13 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *seek, int set_prefix)\n {\n \tstruct debug_ref_iterator *diter =\n \t\t(struct debug_ref_iterator *)ref_iterator;\n-\tint res = diter->iter->vtable->seek(diter->iter, prefix);\n-\ttrace_printf_key(&trace_refs, \"iterator_seek: %s: %d\\n\", prefix ? prefix : \"\", res);\n+\tint res = diter->iter->vtable->seek(diter->iter, seek, set_prefix);\n+\ttrace_printf_key(&trace_refs, \"iterator_seek: %s set_prefix: %d: %d\\n\",\n+\t\t\t seek ? seek : \"\", set_prefix, res);\n \treturn res;\n }\n \ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex bf6f89b1d1..827b15981c 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -929,11 +929,11 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int files_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *seek, int set_prefix)\n {\n \tstruct files_ref_iterator *iter =\n \t\t(struct files_ref_iterator *)ref_iterator;\n-\treturn ref_iterator_seek(iter->iter0, prefix);\n+\treturn ref_iterator_seek(iter->iter0, seek, set_prefix);\n }\n \n static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n@@ -2316,7 +2316,8 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n-\t\t\t\t const char *prefix UNUSED)\n+\t\t\t\t const char *seek UNUSED,\n+\t\t\t\t int set_prefix UNUSED)\n {\n \tBUG(\"ref_iterator_seek() called for reflog_iterator\");\n }\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex 766d96e795..1f99045d40 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -16,9 +16,9 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n int ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t const char *prefix)\n+\t\t const char *seek, int set_prefix)\n {\n-\treturn ref_iterator->vtable->seek(ref_iterator, prefix);\n+\treturn ref_iterator->vtable->seek(ref_iterator, seek, set_prefix);\n }\n \n int ref_iterator_peel(struct ref_iterator *ref_iterator,\n@@ -57,7 +57,8 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)\n }\n \n static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n-\t\t\t\t const char *prefix UNUSED)\n+\t\t\t\t const char *seek UNUSED,\n+\t\t\t\t int set_prefix UNUSED)\n {\n \treturn 0;\n }\n@@ -224,7 +225,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *seek, int set_prefix)\n {\n \tstruct merge_ref_iterator *iter =\n \t\t(struct merge_ref_iterator *)ref_iterator;\n@@ -234,11 +235,11 @@ static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,\n \titer->iter0 = iter->iter0_owned;\n \titer->iter1 = iter->iter1_owned;\n \n-\tret = ref_iterator_seek(iter->iter0, prefix);\n+\tret = ref_iterator_seek(iter->iter0, seek, set_prefix);\n \tif (ret < 0)\n \t\treturn ret;\n \n-\tret = ref_iterator_seek(iter->iter1, prefix);\n+\tret = ref_iterator_seek(iter->iter1, seek, set_prefix);\n \tif (ret < 0)\n \t\treturn ret;\n \n@@ -407,13 +408,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *seek, int set_prefix)\n {\n \tstruct prefix_ref_iterator *iter =\n \t\t(struct prefix_ref_iterator *)ref_iterator;\n-\tfree(iter->prefix);\n-\titer->prefix = xstrdup_or_null(prefix);\n-\treturn ref_iterator_seek(iter->iter0, prefix);\n+\n+\tif (set_prefix) {\n+\t\tfree(iter->prefix);\n+\t\titer->prefix = xstrdup_or_null(seek);\n+\t}\n+\treturn ref_iterator_seek(iter->iter0, seek, set_prefix);\n }\n \n static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 7fd73a0e6d..dca886c5cc 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1004,19 +1004,22 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *seek, int set_prefix)\n {\n \tstruct packed_ref_iterator *iter =\n \t\t(struct packed_ref_iterator *)ref_iterator;\n \tconst char *start;\n \n-\tif (prefix && *prefix)\n-\t\tstart = find_reference_location(iter->snapshot, prefix, 0);\n+\tif (seek && *seek)\n+\t\tstart = find_reference_location(iter->snapshot, seek, 0);\n \telse\n \t\tstart = iter->snapshot->start;\n \n-\tfree(iter->prefix);\n-\titer->prefix = xstrdup_or_null(prefix);\n+\tif (set_prefix) {\n+\t\tfree(iter->prefix);\n+\t\titer->prefix = xstrdup_or_null(seek);\n+\t}\n+\n \titer->pos = start;\n \titer->eof = iter->snapshot->eof;\n \n@@ -1194,7 +1197,7 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \titer->repo = ref_store->repo;\n \titer->flags = flags;\n \n-\tif (packed_ref_iterator_seek(&iter->base, prefix) < 0) {\n+\tif (packed_ref_iterator_seek(&iter->base, prefix, 1) < 0) {\n \t\tref_iterator_free(&iter->base);\n \t\treturn NULL;\n \t}\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 8aaffa8c6b..656e6cd9ff 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -434,11 +434,9 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t}\n }\n \n-static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+static int cache_ref_iterator_set_prefix(struct cache_ref_iterator *iter,\n+\t\t\t\t\t const char *prefix)\n {\n-\tstruct cache_ref_iterator *iter =\n-\t\t(struct cache_ref_iterator *)ref_iterator;\n \tstruct cache_ref_iterator_level *level;\n \tstruct ref_dir *dir;\n \n@@ -469,6 +467,79 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n \treturn 0;\n }\n \n+static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *seek, int set_prefix)\n+{\n+\tstruct cache_ref_iterator *iter =\n+\t\t(struct cache_ref_iterator *)ref_iterator;\n+\n+\tif (set_prefix) {\n+\t\treturn cache_ref_iterator_set_prefix(iter, seek);\n+\t} else if (seek && *seek) {\n+\t\tstruct cache_ref_iterator_level *level;\n+\t\tconst char *slash = seek;\n+\t\tstruct ref_dir *dir;\n+\n+\t\tdir = get_ref_dir(iter->cache->root);\n+\n+\t\tif (iter->prime_dir)\n+\t\t\tprime_ref_dir(dir, seek);\n+\n+\t\titer->levels_nr = 1;\n+\t\tlevel = &iter->levels[0];\n+\t\tlevel->index = -1;\n+\t\tlevel->dir = dir;\n+\n+\t\t/*\n+\t\t * Breakdown the provided seek path and assign the correct\n+\t\t * indexing to each level as needed.\n+\t\t */\n+\t\tdo {\n+\t\t\tint len, idx;\n+\t\t\tint cmp = 0;\n+\n+\t\t\tsort_ref_dir(dir);\n+\n+\t\t\tslash = strchr(slash, '/');\n+\t\t\tlen = slash ? slash - seek : (int)strlen(seek);\n+\n+\t\t\tfor (idx = 0; idx < dir->nr; idx++) {\n+\t\t\t\tcmp = strncmp(seek, dir->entries[idx]->name, len);\n+\t\t\t\tif (cmp <= 0)\n+\t\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\t/* don't overflow the index */\n+\t\t\tidx = idx >= dir->nr ? dir->nr - 1 : idx;\n+\n+\t\t\tif (slash)\n+\t\t\t\tslash = slash + 1;\n+\n+\t\t\tlevel->index = idx;\n+\t\t\tif (dir->entries[idx]->flag & REF_DIR) {\n+\t\t\t\t/* push down a level */\n+\t\t\t\tdir = get_ref_dir(dir->entries[idx]);\n+\n+\t\t\t\tALLOC_GROW(iter->levels, iter->levels_nr + 1,\n+\t\t\t\t\t iter->levels_alloc);\n+\t\t\t\tlevel = &iter->levels[iter->levels_nr++];\n+\t\t\t\tlevel->dir = dir;\n+\t\t\t\tlevel->index = -1;\n+\t\t\t} else {\n+\t\t\t\t/* reduce the index so the leaf node is iterated over */\n+\t\t\t\tif (cmp <= 0 && !slash)\n+\t\t\t\t\tlevel->index = idx - 1;\n+\t\t\t\t/*\n+\t\t\t\t * while the seek path may not be exhausted, our\n+\t\t\t\t * match is exhausted at a leaf node.\n+\t\t\t\t */\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t} while (slash);\n+\t}\n+\n+\treturn 0;\n+}\n+\n static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -509,7 +580,7 @@ struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\n \titer->cache = cache;\n \titer->prime_dir = prime_dir;\n \n-\tif (cache_ref_iterator_seek(&iter->base, prefix) < 0) {\n+\tif (cache_ref_iterator_seek(&iter->base, prefix, 1) < 0) {\n \t\tref_iterator_free(&iter->base);\n \t\treturn NULL;\n \t}\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 03f5df04d5..cee377696c 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -353,11 +353,12 @@ void base_ref_iterator_init(struct ref_iterator *iter,\n typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);\n \n /*\n- * Seek the iterator to the first reference matching the given prefix. Should\n- * behave the same as if a new iterator was created with the same prefix.\n+ * Seek the iterator to the first matching reference. If set_prefix is set,\n+ * it would behave the same as if a new iterator was created with the same\n+ * prefix.\n */\n typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix);\n+\t\t\t\t const char *seek, int set_prefix);\n \n /*\n * Peels the current ref, returning 0 for success or -1 for failure.\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 4c3817f4ec..81fb6a9028 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -719,15 +719,17 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *seek, int set_prefix)\n {\n \tstruct reftable_ref_iterator *iter =\n \t\t(struct reftable_ref_iterator *)ref_iterator;\n \n-\tfree(iter->prefix);\n-\titer->prefix = xstrdup_or_null(prefix);\n-\titer->prefix_len = prefix ? strlen(prefix) : 0;\n-\titer->err = reftable_iterator_seek_ref(&iter->iter, prefix);\n+\tif (set_prefix) {\n+\t\tfree(iter->prefix);\n+\t\titer->prefix = xstrdup_or_null(seek);\n+\t\titer->prefix_len = seek ? strlen(seek) : 0;\n+\t}\n+\titer->err = reftable_iterator_seek_ref(&iter->iter, seek);\n \n \treturn iter->err;\n }\n@@ -839,7 +841,7 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_\n \tif (ret)\n \t\tgoto done;\n \n-\tret = reftable_ref_iterator_seek(&iter->base, prefix);\n+\tret = reftable_ref_iterator_seek(&iter->base, prefix, 1);\n \tif (ret)\n \t\tgoto done;\n \n@@ -2042,7 +2044,8 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n-\t\t\t\t\t const char *prefix UNUSED)\n+\t\t\t\t\t const char *seek UNUSED,\n+\t\t\t\t\t int set_prefix UNUSED)\n {\n \tBUG(\"reftable reflog iterator cannot be seeked\");\n \treturn -1;\n\n-- \n2.49.0\n\n"},{"id":"521056","messageId":"20250701-306-git-for-each-ref-pagination-v1-4-4f0ae7c0688f@gmail.com","threadId":"63722","inReplyTo":"20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com","subject":"[PATCH 4/4] for-each-ref: introduce a '--skip-until' option","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-01T15:03:30Z","receivedAt":"2025-07-01T15:04:16Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `git-for-each-ref(1)` command is used to iterate over references\npresent in a repository. In large repositories with millions of\nreferences, it would be optimal to paginate this output such that we\ncan start iteration from a given reference. This would avoid having to\niterate over all references from the beginning each time when paginating\nthrough results.\n\nThe previous commit added 'seek' functionality to the reference\nbackends. Utilize this and expose a '--skip-until' option in\n'git-for-each-ref(1)'. When used, the reference iteration seeks to the\nfirst matching reference and iterates from there onward.\n\nThis enables efficient pagination workflows like:\n git for-each-ref --count=100\n git for-each-ref --count=100 --skip-until=refs/heads/branch-100\n git for-each-ref --count=100 --skip-until=refs/heads/branch-200\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/git-for-each-ref.adoc | 6 +-\n builtin/for-each-ref.c | 5 +\n ref-filter.c | 57 ++++++++----\n ref-filter.h | 1 +\n t/t6302-for-each-ref-filter.sh | 180 ++++++++++++++++++++++++++++++++++++\n 5 files changed, 230 insertions(+), 19 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.adoc b/Documentation/git-for-each-ref.adoc\nindex 5ef89fc0fe..4bf7c66b8c 100644\n--- a/Documentation/git-for-each-ref.adoc\n+++ b/Documentation/git-for-each-ref.adoc\n@@ -14,7 +14,7 @@ SYNOPSIS\n \t\t [--points-at=]\n \t\t [--merged[=]] [--no-merged[=]]\n \t\t [--contains[=]] [--no-contains[=]]\n-\t\t [--exclude= ...]\n+\t\t [--exclude= ...] [--skip-until=]\n \n DESCRIPTION\n -----------\n@@ -108,6 +108,10 @@ TAB %(refname)`.\n --include-root-refs::\n \tList root refs (HEAD and pseudorefs) apart from regular refs.\n \n+--skip-until::\n+ Skip references up to the specified pattern. Cannot be used with\n+ general pattern matching.\n+\n FIELD NAMES\n -----------\n \ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 3d2207ec77..543013cd11 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -13,6 +13,7 @@ static char const * const for_each_ref_usage[] = {\n \tN_(\"git for-each-ref [--points-at ]\"),\n \tN_(\"git for-each-ref [--merged []] [--no-merged []]\"),\n \tN_(\"git for-each-ref [--contains []] [--no-contains []]\"),\n+\tN_(\"git for-each-ref [--skip-until ]\"),\n \tNULL\n };\n \n@@ -44,6 +45,7 @@ int cmd_for_each_ref(int argc,\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_INTEGER( 0 , \"count\", &format.array_opts.max_count, N_(\"show only matched refs\")),\n \t\tOPT_STRING( 0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n+\t\tOPT_STRING( 0 , \"skip-until\", &filter.seek, N_(\"skip-until\"), N_(\"skip references until\")),\n \t\tOPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n \t\tOPT_REF_FILTER_EXCLUDE(&filter),\n \t\tOPT_REF_SORT(&sorting_options),\n@@ -100,6 +102,9 @@ int cmd_for_each_ref(int argc,\n \t\tfilter.name_patterns = argv;\n \t}\n \n+\tif (filter.seek && filter.name_patterns && filter.name_patterns[0])\n+\t\tdie(_(\"cannot use --skip-until with patterns\"));\n+\n \tif (include_root_refs)\n \t\tflags |= FILTER_REFS_ROOT_REFS | FILTER_REFS_DETACHED_HEAD;\n \ndiff --git a/ref-filter.c b/ref-filter.c\nindex 7a274633cf..9d0255d5db 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2692,10 +2692,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t\t\t each_ref_fn cb,\n \t\t\t\t void *cb_data)\n {\n+\tstruct ref_iterator *iter;\n+\tint flags = 0, ret = 0;\n+\n \tif (filter->kind & FILTER_REFS_ROOT_REFS) {\n \t\t/* In this case, we want to print all refs including root refs. */\n-\t\treturn refs_for_each_include_root_refs(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t cb, cb_data);\n+\t\tflags |= DO_FOR_EACH_INCLUDE_ROOT_REFS;\n+\t\tgoto non_prefix_iter;\n \t}\n \n \tif (!filter->match_as_path) {\n@@ -2704,8 +2707,7 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t * prefixes like \"refs/heads/\" etc. are stripped off,\n \t\t * so we have to look at everything:\n \t\t */\n-\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t\"\", NULL, cb, cb_data);\n+\t\tgoto non_prefix_iter;\n \t}\n \n \tif (filter->ignore_case) {\n@@ -2714,20 +2716,28 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t * so just return everything and let the caller\n \t\t * sort it out.\n \t\t */\n-\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t\"\", NULL, cb, cb_data);\n+\t\tgoto non_prefix_iter;\n \t}\n \n \tif (!filter->name_patterns[0]) {\n \t\t/* no patterns; we have to look at everything */\n-\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"\", filter->exclude.v, cb, cb_data);\n+\t\tgoto non_prefix_iter;\n \t}\n \n \treturn refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n \t\t\t\t\t\t NULL, filter->name_patterns,\n \t\t\t\t\t\t filter->exclude.v,\n \t\t\t\t\t\t cb, cb_data);\n+\n+non_prefix_iter:\n+\titer = refs_ref_iterator_begin(get_main_ref_store(the_repository), \"\",\n+\t\t\t\t NULL, 0, flags);\n+\tif (filter->seek)\n+\t\tret = ref_iterator_seek(iter, filter->seek, 0);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\treturn do_for_each_ref_iterator(iter, cb, cb_data);\n }\n \n /*\n@@ -3200,6 +3210,8 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n \tif (!filter->kind)\n \t\tdie(\"filter_refs: invalid type\");\n \telse {\n+\t\tconst char *prefix = NULL;\n+\n \t\t/*\n \t\t * For common cases where we need only branches or remotes or tags,\n \t\t * we only iterate through those refs. If a mix of refs is needed,\n@@ -3207,19 +3219,28 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n \t\t * of filter_ref_kind().\n \t\t */\n \t\tif (filter->kind == FILTER_REFS_BRANCHES)\n-\t\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"refs/heads/\", NULL,\n-\t\t\t\t\t\t fn, cb_data);\n+\t\t\tprefix = \"refs/heads/\";\n \t\telse if (filter->kind == FILTER_REFS_REMOTES)\n-\t\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"refs/remotes/\", NULL,\n-\t\t\t\t\t\t fn, cb_data);\n+\t\t\tprefix = \"refs/remotes/\";\n \t\telse if (filter->kind == FILTER_REFS_TAGS)\n-\t\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"refs/tags/\", NULL, fn,\n-\t\t\t\t\t\t cb_data);\n-\t\telse if (filter->kind & FILTER_REFS_REGULAR)\n+\t\t\tprefix = \"refs/tags/\";\n+\n+\t\tif (prefix) {\n+\t\t\tstruct ref_iterator *iter;\n+\n+\t\t\titer = refs_ref_iterator_begin(get_main_ref_store(the_repository),\n+\t\t\t\t\t\t \"\", NULL, 0, 0);\n+\n+\t\t\tif (filter->seek)\n+\t\t\t\tret = ref_iterator_seek(iter, filter->seek, 0);\n+\t\t\telse if (prefix)\n+\t\t\t\tret = ref_iterator_seek(iter, prefix, 1);\n+\n+\t\t\tif (!ret)\n+\t\t\t\tret = do_for_each_ref_iterator(iter, fn, cb_data);\n+\t\t} else if (filter->kind & FILTER_REFS_REGULAR) {\n \t\t\tret = for_each_fullref_in_pattern(filter, fn, cb_data);\n+\t\t}\n \n \t\t/*\n \t\t * When printing all ref types, HEAD is already included,\ndiff --git a/ref-filter.h b/ref-filter.h\nindex c98c4fbd4c..9e97c65bc2 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -64,6 +64,7 @@ struct ref_array {\n \n struct ref_filter {\n \tconst char **name_patterns;\n+\tconst char *seek;\n \tstruct strvec exclude;\n \tstruct oid_array points_at;\n \tstruct commit_list *with_commit;\ndiff --git a/t/t6302-for-each-ref-filter.sh b/t/t6302-for-each-ref-filter.sh\nindex bb02b86c16..af2c60a2ce 100755\n--- a/t/t6302-for-each-ref-filter.sh\n+++ b/t/t6302-for-each-ref-filter.sh\n@@ -541,4 +541,184 @@ test_expect_success 'validate worktree atom' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'skip until with empty value' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/main\n+\trefs/heads/main_worktree\n+\trefs/heads/side\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --skip-until=\"\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'skip until to a specific reference' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/spot >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'skip until to a specific reference with partial match' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/sp >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'skip until just behind a specific reference' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/parrot >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'skip until to specific directory' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/odd >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'skip until to specific directory with trailing slash' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/lost >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'skip until just behind a specific directory' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'skip until overflow specific reference length' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/spotnew >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'skip until overflow specific reference path' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/spot/new >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'skip until used with a pattern' '\n+\tcat >expect <<-\\EOF &&\n+\tfatal: cannot use --skip-until with patterns\n+\tEOF\n+\ttest_must_fail git for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/spot refs/tags 2>actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n\n-- \n2.49.0\n\n"},{"id":"521089","messageId":"xmqqfrffltym.fsf@gitster.g","threadId":"63722","inReplyTo":"20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com","subject":"Re: [PATCH 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-01T17:08:33Z","receivedAt":"2025-07-01T17:08:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> The `git-for-each-ref(1)` command is used to iterate over references\n> present in a repository. In large repositories with millions of\n> references, it would be optimal to paginate this output such that we\n> can start iteration from a given reference.\n\nI haven't looked at the patches, but should the end-user's mental\nmodel of the process be like this?\n\n - We have a native order in which references are sorted and that is\n what \"git for-each-ref\" without \"--sort\" option gives them.\n\n - They can use the \"--skip-until\" option to seek in the above order\n and start iterating in the middle.\n\n - If they give \"--sort\", the set of refs to be shown would not\n change; skipping is done in the native order and then the\n remainder is given sorted.\n\nPlease make sure that the documentation is clear enough to avoid a\nmisunderstanding that this feature would kick in after we grab all\nrefs and sort them. If it worked that way, it would allow us to say\n\"going from newer to older, but skipping the most recent ones that\nwere touched within a week\", which would have been nice, but that is\nnot what we are doing with this feature---I think it is OK but we\nneed to be clear about it in the documentation.\n\n> This series adds a '--skip-until' option in 'git-for-each-ref(1)'. When\n> used, the reference iteration seeks to the first matching reference and\n> iterates from there onward.\n\nOK. Even the filesystem backed ones we internall sort after doing\nreaddir() loop, so this is feasible. Nice.\n\n> Initally I was also planning to cleanup all the `refs_for_each...()`\n> functions in 'refs.h' by simply using the iterator, but this bloated the\n> series. So I've left that for another day.\n\nOK.\n"},{"id":"521124","messageId":"xmqqms9niod7.fsf@gitster.g","threadId":"63722","inReplyTo":"20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com","subject":"Re: [PATCH 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-01T21:37:40Z","receivedAt":"2025-07-01T21:37:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Offtopic. After applying this topic, I asked clang-format if it\nwants to change anything.\n\n $ git clang-format --diff $(git merge-base HEAD master)\n\nThe result was disasterous. Can \"clang-format --diff\" mode be\ntaught a bit more focused to avoid touching existing entries in the\nsame array (in this case opts[] that has tons of options for the\n\"git for-each-ref\" command), when only one new entry was added, I\nwonder?\n\nAlso I am not impressed by the change it made to the code that is\ncommented out (in refs.h).\n\nLine wrapping it did to refs_ref_iterator_begin() is an improvement,\nbut those to ref_iterator_seek() and do_for_each_ref_iterator() are\nunnecessary (both of these were more readble in the original).\n\nEven though I found its output better for Toon's \"last-modified\"\nchanges, I am not impressed by what clang-format suggested for this\nseries.\n\n\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 543013cd11..39056557d4 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -8,7 +8,7 @@\n #include \"strbuf.h\"\n #include \"strvec.h\"\n \n-static char const * const for_each_ref_usage[] = {\n+static char const *const for_each_ref_usage[] = {\n \tN_(\"git for-each-ref [] []\"),\n \tN_(\"git for-each-ref [--points-at ]\"),\n \tN_(\"git for-each-ref [--merged []] [--no-merged []]\"),\n@@ -33,32 +33,41 @@ int cmd_for_each_ref(int argc,\n \tstruct option opts[] = {\n \t\tOPT_BIT('s', \"shell\", &format.quote_style,\n \t\t\tN_(\"quote placeholders suitably for shells\"), QUOTE_SHELL),\n-\t\tOPT_BIT('p', \"perl\", &format.quote_style,\n+\t\tOPT_BIT('p', \"perl\", &format.quote_style,\n \t\t\tN_(\"quote placeholders suitably for perl\"), QUOTE_PERL),\n-\t\tOPT_BIT(0 , \"python\", &format.quote_style,\n-\t\t\tN_(\"quote placeholders suitably for python\"), QUOTE_PYTHON),\n-\t\tOPT_BIT(0 , \"tcl\", &format.quote_style,\n+\t\tOPT_BIT(0, \"python\", &format.quote_style,\n+\t\t\tN_(\"quote placeholders suitably for python\"),\n+\t\t\tQUOTE_PYTHON),\n+\t\tOPT_BIT(0, \"tcl\", &format.quote_style,\n \t\t\tN_(\"quote placeholders suitably for Tcl\"), QUOTE_TCL),\n-\t\tOPT_BOOL(0, \"omit-empty\", &format.array_opts.omit_empty,\n-\t\t\tN_(\"do not output a newline after empty formatted refs\")),\n+\t\tOPT_BOOL(0, \"omit-empty\", &format.array_opts.omit_empty,\n+\t\t\t N_(\"do not output a newline after empty formatted refs\")),\n \n \t\tOPT_GROUP(\"\"),\n-\t\tOPT_INTEGER( 0 , \"count\", &format.array_opts.max_count, N_(\"show only matched refs\")),\n-\t\tOPT_STRING( 0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n-\t\tOPT_STRING( 0 , \"skip-until\", &filter.seek, N_(\"skip-until\"), N_(\"skip references until\")),\n+\t\tOPT_INTEGER(0, \"count\", &format.array_opts.max_count,\n+\t\t\t N_(\"show only matched refs\")),\n+\t\tOPT_STRING(0, \"format\", &format.format, N_(\"format\"),\n+\t\t\t N_(\"format to use for the output\")),\n+\t\tOPT_STRING(0, \"skip-until\", &filter.seek, N_(\"skip-until\"),\n+\t\t\t N_(\"skip references until\")),\n \t\tOPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n \t\tOPT_REF_FILTER_EXCLUDE(&filter),\n \t\tOPT_REF_SORT(&sorting_options),\n-\t\tOPT_CALLBACK(0, \"points-at\", &filter.points_at,\n-\t\t\t N_(\"object\"), N_(\"print only refs which points at the given object\"),\n+\t\tOPT_CALLBACK(0, \"points-at\", &filter.points_at, N_(\"object\"),\n+\t\t\t N_(\"print only refs which points at the given object\"),\n \t\t\t parse_opt_object_name),\n \t\tOPT_MERGED(&filter, N_(\"print only refs that are merged\")),\n \t\tOPT_NO_MERGED(&filter, N_(\"print only refs that are not merged\")),\n-\t\tOPT_CONTAINS(&filter.with_commit, N_(\"print only refs which contain the commit\")),\n-\t\tOPT_NO_CONTAINS(&filter.no_commit, N_(\"print only refs which don't contain the commit\")),\n-\t\tOPT_BOOL(0, \"ignore-case\", &icase, N_(\"sorting and filtering are case insensitive\")),\n-\t\tOPT_BOOL(0, \"stdin\", &from_stdin, N_(\"read reference patterns from stdin\")),\n-\t\tOPT_BOOL(0, \"include-root-refs\", &include_root_refs, N_(\"also include HEAD ref and pseudorefs\")),\n+\t\tOPT_CONTAINS(&filter.with_commit,\n+\t\t\t N_(\"print only refs which contain the commit\")),\n+\t\tOPT_NO_CONTAINS(&filter.no_commit,\n+\t\t\t\tN_(\"print only refs which don't contain the commit\")),\n+\t\tOPT_BOOL(0, \"ignore-case\", &icase,\n+\t\t\t N_(\"sorting and filtering are case insensitive\")),\n+\t\tOPT_BOOL(0, \"stdin\", &from_stdin,\n+\t\t\t N_(\"read reference patterns from stdin\")),\n+\t\tOPT_BOOL(0, \"include-root-refs\", &include_root_refs,\n+\t\t\t N_(\"also include HEAD ref and pseudorefs\")),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/refs.h b/refs.h\nindex c5e08db0ff..518b17c748 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -1229,7 +1229,8 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n *\n * // Access information about the current reference:\n * if (!(iter->flags & REF_ISSYMREF))\n- * printf(\"%s is %s\\n\", iter->refname, oid_to_hex(iter->oid));\n+ * printf(\"%s is %s\\n\", iter->refname,\n+ * oid_to_hex(iter->oid));\n *\n * // If you need to peel the reference:\n * ref_iterator_peel(iter, &oid);\n@@ -1284,10 +1285,11 @@ enum do_for_each_ref_flags {\n * trim that many characters off the beginning of each refname.\n * The output is ordered by refname.\n */\n-struct ref_iterator *refs_ref_iterator_begin(\n-\t\tstruct ref_store *refs,\n-\t\tconst char *prefix, const char **exclude_patterns,\n-\t\tint trim, enum do_for_each_ref_flags flags);\n+struct ref_iterator *refs_ref_iterator_begin(struct ref_store *refs,\n+\t\t\t\t\t const char *prefix,\n+\t\t\t\t\t const char **exclude_patterns,\n+\t\t\t\t\t int trim,\n+\t\t\t\t\t enum do_for_each_ref_flags flags);\n \n /*\n * Advance the iterator to the first or next item and return ITER_OK.\n@@ -1317,8 +1319,8 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator);\n *\n * Returns 0 on success, a negative error code otherwise.\n */\n-int ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t const char *seek, int set_prefix);\n+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,\n+\t\t int set_prefix);\n \n /*\n * If possible, peel the reference currently being viewed by the\n@@ -1339,8 +1341,7 @@ void ref_iterator_free(struct ref_iterator *ref_iterator);\n * adapter between the callback style of reference iteration and the\n * iterator style.\n */\n-int do_for_each_ref_iterator(struct ref_iterator *iter,\n-\t\t\t each_ref_fn fn, void *cb_data);\n-\n+int do_for_each_ref_iterator(struct ref_iterator *iter, each_ref_fn fn,\n+\t\t\t void *cb_data);\n \n #endif /* REFS_H */\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex 1f99045d40..2b7f019c3e 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -15,8 +15,8 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ref_iterator->vtable->advance(ref_iterator);\n }\n \n-int ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t const char *seek, int set_prefix)\n+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,\n+\t\t int set_prefix)\n {\n \treturn ref_iterator->vtable->seek(ref_iterator, seek, set_prefix);\n }\n@@ -57,8 +57,7 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)\n }\n \n static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n-\t\t\t\t const char *seek UNUSED,\n-\t\t\t\t int set_prefix UNUSED)\n+\t\t\t\t const char *seek UNUSED, int set_prefix UNUSED)\n {\n \treturn 0;\n }\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 656e6cd9ff..b812520dc7 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -525,7 +525,8 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n \t\t\t\tlevel->dir = dir;\n \t\t\t\tlevel->index = -1;\n \t\t\t} else {\n-\t\t\t\t/* reduce the index so the leaf node is iterated over */\n+\t\t\t\t/* reduce the index so the leaf node is iterated\n+\t\t\t\t * over */\n \t\t\t\tif (cmp <= 0 && !slash)\n \t\t\t\t\tlevel->index = idx - 1;\n \t\t\t\t/*\n"},{"id":"521171","messageId":"5e9b3ef1-931b-4b70-8275-5aed5da3d6f3@gmail.com","threadId":"63722","inReplyTo":"20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com","subject":"Re: [PATCH 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2025-07-02T14:14:07Z","receivedAt":"2025-07-02T14:14:10Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Karthik\n\nOn 01/07/2025 16:03, Karthik Nayak wrote:\n> \n> This enables efficient pagination workflows like:\n> git for-each-ref --count=100\n> git for-each-ref --count=100 --skip-until=refs/heads/branch-100\n> git for-each-ref --count=100 --skip-until=refs/heads/branch-200\n\nDoesn't that require you to know the name of the ref after the last one \nreturned by the previous batch? If the use case here is pagination then \nbeing able to provide a numeric offset might be a better fit. For example\n\n\tgit for-each-ref --count=100 --start=200\n\nwould show refs 200 to 300\n\nThanks\n\nPhillip\n\n"},{"id":"521184","messageId":"CAOLa=ZRs44wc7gtyKqn+nkOtm306FNCGFWsm5=sgDBSFoB52LA@mail.gmail.com","threadId":"63722","inReplyTo":"xmqqfrffltym.fsf@gitster.g","subject":"Re: [PATCH 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-02T16:45:51Z","receivedAt":"2025-07-02T16:45:55Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Karthik Nayak writes:\n>\n>> The `git-for-each-ref(1)` command is used to iterate over references\n>> present in a repository. In large repositories with millions of\n>> references, it would be optimal to paginate this output such that we\n>> can start iteration from a given reference.\n>\n> I haven't looked at the patches, but should the end-user's mental\n> model of the process be like this?\n>\n> - We have a native order in which references are sorted and that is\n> what \"git for-each-ref\" without \"--sort\" option gives them.\n>\n> - They can use the \"--skip-until\" option to seek in the above order\n> and start iterating in the middle.\n>\n> - If they give \"--sort\", the set of refs to be shown would not\n> change; skipping is done in the native order and then the\n> remainder is given sorted.\n>\n> Please make sure that the documentation is clear enough to avoid a\n> misunderstanding that this feature would kick in after we grab all\n> refs and sort them. If it worked that way, it would allow us to say\n> \"going from newer to older, but skipping the most recent ones that\n> were touched within a week\", which would have been nice, but that is\n> not what we are doing with this feature---I think it is OK but we\n> need to be clear about it in the documentation.\n>\n\nI totally didn't consider '--sort'. I do agree that we should document\nthat behavior if we tend to keep it as is. I wonder if it is worthwhile\nto even prevent the two from being used together. I find the whole \"we\nskip before sorting\" to be very confusing.\n\n>> This series adds a '--skip-until' option in 'git-for-each-ref(1)'. When\n>> used, the reference iteration seeks to the first matching reference and\n>> iterates from there onward.\n>\n> OK. Even the filesystem backed ones we internall sort after doing\n> readdir() loop, so this is feasible. Nice.\n>\n\nYup. We have 'sort_ref_dir()' to sort each directory parsed.\n\n>> Initally I was also planning to cleanup all the `refs_for_each...()`\n>> functions in 'refs.h' by simply using the iterator, but this bloated the\n>> series. So I've left that for another day.\n>\n> OK.\n"},{"id":"521192","messageId":"CAOLa=ZQZcX7Bh=63Bv5Sti89jsjq=FzVbkqqsP5Tgd7NzNr96g@mail.gmail.com","threadId":"63722","inReplyTo":"xmqqms9niod7.fsf@gitster.g","subject":"Re: [PATCH 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-02T18:19:10Z","receivedAt":"2025-07-02T18:19:13Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Offtopic. After applying this topic, I asked clang-format if it\n> wants to change anything.\n>\n> $ git clang-format --diff $(git merge-base HEAD master)\n>\n> The result was disasterous. Can \"clang-format --diff\" mode be\n> taught a bit more focused to avoid touching existing entries in the\n> same array (in this case opts[] that has tons of options for the\n> \"git for-each-ref\" command), when only one new entry was added, I\n> wonder?\n>\n\nI couldn't find any way to do something like this.\n\n> Also I am not impressed by the change it made to the code that is\n> commented out (in refs.h).\n>\n> Line wrapping it did to refs_ref_iterator_begin() is an improvement,\n> but those to ref_iterator_seek() and do_for_each_ref_iterator() are\n> unnecessary (both of these were more readble in the original).\n>\n> Even though I found its output better for Toon's \"last-modified\"\n> changes, I am not impressed by what clang-format suggested for this\n> series.\n>\n\nIt indeed looks really bad, I had a go with the new changes from\n'gitster/kn/clang-format-updates'. Which seems a lot better.\n\nHowever, this does show a problem with using 'RemoveBracesLLVM', where\nit formats the following:\n\n if (...) {\n ...\n ...\n } else {\n ...\n }\n\nto:\n\n if (...) {\n ...\n ...\n } else\n ...\n\nWhich isn't our style, I think we should completely drop this too, from\nmy patch series. Let me go ahead and do that. I really want to strip out\nas many rules as possible to make the number of false positives 0 so we\ncan actually start enforcing clang-format. Once we enforce it, we can\nslowly see what additional rules work well for us.\n\ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 543013cd11..2ec96eff74 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -8,7 +8,7 @@\n #include \"strbuf.h\"\n #include \"strvec.h\"\n\n-static char const * const for_each_ref_usage[] = {\n+static char const *const for_each_ref_usage[] = {\n \tN_(\"git for-each-ref [] []\"),\n \tN_(\"git for-each-ref [--points-at ]\"),\n \tN_(\"git for-each-ref [--merged []] [--no-merged []]\"),\n@@ -33,19 +33,19 @@ int cmd_for_each_ref(int argc,\n \tstruct option opts[] = {\n \t\tOPT_BIT('s', \"shell\", &format.quote_style,\n \t\t\tN_(\"quote placeholders suitably for shells\"), QUOTE_SHELL),\n-\t\tOPT_BIT('p', \"perl\", &format.quote_style,\n+\t\tOPT_BIT('p', \"perl\", &format.quote_style,\n \t\t\tN_(\"quote placeholders suitably for perl\"), QUOTE_PERL),\n-\t\tOPT_BIT(0 , \"python\", &format.quote_style,\n+\t\tOPT_BIT(0, \"python\", &format.quote_style,\n \t\t\tN_(\"quote placeholders suitably for python\"), QUOTE_PYTHON),\n-\t\tOPT_BIT(0 , \"tcl\", &format.quote_style,\n+\t\tOPT_BIT(0, \"tcl\", &format.quote_style,\n \t\t\tN_(\"quote placeholders suitably for Tcl\"), QUOTE_TCL),\n-\t\tOPT_BOOL(0, \"omit-empty\", &format.array_opts.omit_empty,\n-\t\t\tN_(\"do not output a newline after empty formatted refs\")),\n+\t\tOPT_BOOL(0, \"omit-empty\", &format.array_opts.omit_empty,\n+\t\t\t N_(\"do not output a newline after empty formatted refs\")),\n\n \t\tOPT_GROUP(\"\"),\n-\t\tOPT_INTEGER( 0 , \"count\", &format.array_opts.max_count, N_(\"show\nonly matched refs\")),\n-\t\tOPT_STRING( 0 , \"format\", &format.format, N_(\"format\"), N_(\"format\nto use for the output\")),\n-\t\tOPT_STRING( 0 , \"skip-until\", &filter.seek, N_(\"skip-until\"),\nN_(\"skip references until\")),\n+\t\tOPT_INTEGER(0, \"count\", &format.array_opts.max_count, N_(\"show only\n matched refs\")),\n+\t\tOPT_STRING(0, \"format\", &format.format, N_(\"format\"), N_(\"format to\nuse for the output\")),\n+\t\tOPT_STRING(0, \"skip-until\", &filter.seek, N_(\"skip-until\"),\nN_(\"skip references until\")),\n \t\tOPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n \t\tOPT_REF_FILTER_EXCLUDE(&filter),\n \t\tOPT_REF_SORT(&sorting_options),\ndiff --git a/refs.c b/refs.c\nindex a4220d3537..d492e1b423 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2669,23 +2669,21 @@ enum ref_transaction_error\nrefs_verify_refnames_available(struct ref_store *refs\n \t\t\tif (!iter) {\n \t\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n \t\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n-\t\t\t} else if (ref_iterator_seek(iter, dirname.buf, 1) < 0) {\n-\t\t\t\tgoto cleanup;\n-\t\t\t}\n+\t\t\t\telse if (ref_iterator_seek(iter, dirname.buf, 1) < 0) goto cleanup;\n\n-\t\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n-\t\t\t\tif (skip &&\n-\t\t\t\t string_list_has_string(skip, iter->refname))\n-\t\t\t\t\tcontinue;\n+\t\t\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n+\t\t\t\t\tif (skip &&\n+\t\t\t\t\t string_list_has_string(skip, iter->refname))\n+\t\t\t\t\t\tcontinue;\n\n-\t\t\t\tif (transaction && ref_transaction_maybe_set_rejected(\n-\t\t\t\t\t transaction, *update_idx,\n-\t\t\t\t\t REF_TRANSACTION_ERROR_NAME_CONFLICT))\n-\t\t\t\t\tcontinue;\n+\t\t\t\t\tif (transaction && ref_transaction_maybe_set_rejected(\n+\t\t\t\t\t\t\t\t transaction, *update_idx,\n+\t\t\t\t\t\t\t\t REF_TRANSACTION_ERROR_NAME_CONFLICT))\n+\t\t\t\t\t\tcontinue;\n\n-\t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n-\t\t\t\t\t iter->refname, refname);\n-\t\t\t\tgoto cleanup;\n+\t\t\t\t\tstrbuf_addf(err, _(\"'%s' exists; cannot create '%s'\"),\n+\t\t\t\t\t\t iter->refname, refname);\n+\t\t\t\t\tgoto cleanup;\n \t\t\t}\n\n \t\t\tif (ok != ITER_DONE)\ndiff --git a/refs.h b/refs.h\nindex c5e08db0ff..41fe96d688 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -1285,9 +1285,9 @@ enum do_for_each_ref_flags {\n * The output is ordered by refname.\n */\n struct ref_iterator *refs_ref_iterator_begin(\n-\t\tstruct ref_store *refs,\n-\t\tconst char *prefix, const char **exclude_patterns,\n-\t\tint trim, enum do_for_each_ref_flags flags);\n+\tstruct ref_store *refs,\n+\tconst char *prefix, const char **exclude_patterns,\n+\tint trim, enum do_for_each_ref_flags flags);\n\n /*\n * Advance the iterator to the first or next item and return ITER_OK.\n@@ -1342,5 +1342,4 @@ void ref_iterator_free(struct ref_iterator *ref_iterator);\n int do_for_each_ref_iterator(struct ref_iterator *iter,\n \t\t\t each_ref_fn fn, void *cb_data);\n\n-\n #endif /* REFS_H */\n"},{"id":"521204","messageId":"CAOLa=ZTwvOiCnYK18GTEUkcW0-YLHkJ=MBggdzOYsbTT+OHPwQ@mail.gmail.com","threadId":"63722","inReplyTo":"5e9b3ef1-931b-4b70-8275-5aed5da3d6f3@gmail.com","subject":"Re: [PATCH 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-02T20:33:47Z","receivedAt":"2025-07-02T20:33:49Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Phillip Wood writes:\n\nHello Phillip,\n\n> Hi Karthik\n>\n> On 01/07/2025 16:03, Karthik Nayak wrote:\n>>\n>> This enables efficient pagination workflows like:\n>> git for-each-ref --count=100\n>> git for-each-ref --count=100 --skip-until=refs/heads/branch-100\n>> git for-each-ref --count=100 --skip-until=refs/heads/branch-200\n>\n> Doesn't that require you to know the name of the ref after the last one\n> returned by the previous batch? If the use case here is pagination then\n> being able to provide a numeric offset might be a better fit. For example\n>\n\nIt does require that you know the last ref from the previous batch.\n\nThe reason for picking a reference offset is mostly for performance\noptimization. Our reference backends are built with prefix matching in\nmind, in short they do a binary search through the reference namespace\nto find the required prefix. By using a reference offset we can utilize\nthis binary search mechanism to arrive at offset.\n\nUsing a count offset would require iteration to reach the desired\noffset (basically a O(N) operation). This wouldn't really matter in\nrepositories with ~10^3 refs, but in larger repositories with around\n~10^6 refs this starts to make a large difference.\n\n> \tgit for-each-ref --count=100 --start=200\n>\n> would show refs 200 to 300\n>\n> Thanks\n>\n> Phillip\n"},{"id":"521227","messageId":"aGYSjf5H_ZBaVVJm@pks.im","threadId":"63722","inReplyTo":"CAOLa=ZTwvOiCnYK18GTEUkcW0-YLHkJ=MBggdzOYsbTT+OHPwQ@mail.gmail.com","subject":"Re: [PATCH 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-03T05:18:05Z","receivedAt":"2025-07-03T05:18:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Jul 02, 2025 at 03:33:47PM -0500, Karthik Nayak wrote:\n> Phillip Wood writes:\n> \n> Hello Phillip,\n> \n> > Hi Karthik\n> >\n> > On 01/07/2025 16:03, Karthik Nayak wrote:\n> >>\n> >> This enables efficient pagination workflows like:\n> >> git for-each-ref --count=100\n> >> git for-each-ref --count=100 --skip-until=refs/heads/branch-100\n> >> git for-each-ref --count=100 --skip-until=refs/heads/branch-200\n> >\n> > Doesn't that require you to know the name of the ref after the last one\n> > returned by the previous batch? If the use case here is pagination then\n> > being able to provide a numeric offset might be a better fit. For example\n> >\n> \n> It does require that you know the last ref from the previous batch.\n> \n> The reason for picking a reference offset is mostly for performance\n> optimization. Our reference backends are built with prefix matching in\n> mind, in short they do a binary search through the reference namespace\n> to find the required prefix. By using a reference offset we can utilize\n> this binary search mechanism to arrive at offset.\n> \n> Using a count offset would require iteration to reach the desired\n> offset (basically a O(N) operation). This wouldn't really matter in\n> repositories with ~10^3 refs, but in larger repositories with around\n> ~10^6 refs this starts to make a large difference.\n\nEven more importantly though, a numeric offset would be invalidated by a\nconcurrent write in case that write ends up inserting a ref in the range\nof commits you intend to skip now.\n\nPatrick\n"},{"id":"521228","messageId":"aGYbUpC9OEd90J4N@pks.im","threadId":"63722","inReplyTo":"20250701-306-git-for-each-ref-pagination-v1-3-4f0ae7c0688f@gmail.com","subject":"Re: [PATCH 3/4] refs: selectively set prefix in the seek functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-03T05:55:38Z","receivedAt":"2025-07-03T05:55:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Jul 01, 2025 at 05:03:29PM +0200, Karthik Nayak wrote:\n> The ref iterator exposes a `ref_iterator_seek()` function. The name\n> suggests that this would seek the iterator to a specific reference in\n> some ways similar to how `fseek()` works for the filesystem.\n> \n> However, the function actually sets the prefix for refs iteration. So\n> further iteration would only yield references which match the particular\n> prefix. This is a bit confusing.\n> \n> Let's add a 'set_prefix' field to the function, which when set, will set\n> the prefix for the iteration in-line with the existing behavior. But\n> when the 'set_prefix' field is not set, the reference backends will\n> simply seek to the specified reference without setting prefix. This\n> allows users to start iteration from a specific reference.\n> \n> In the packed and reftable backend, since references are available in a\n> sorted list, the changes are simply setting the prefix if needed. The\n> changes on the files-backend are a little more involved, since the files\n> backend uses the 'ref-cache' mechanism. We move out the existing logic\n> within `cache_ref_iterator_seek()` to `cache_ref_iterator_set_prefix()`\n> which is called when `set_prefix` is set. We then parse the provided\n> seek string and set the required levels and their indexes to ensure that\n> seeking is possible.\n\nThat solution makes sense.\n\n> diff --git a/refs.c b/refs.c\n> index dce5c49ca2..a4220d3537 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -2669,7 +2669,7 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs\n> \t\t\tif (!iter) {\n> \t\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n> \t\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n> -\t\t\t} else if (ref_iterator_seek(iter, dirname.buf) < 0) {\n> +\t\t\t} else if (ref_iterator_seek(iter, dirname.buf, 1) < 0) {\n> \t\t\t\tgoto cleanup;\n> \t\t\t}\n> \n\nThis is quite unreadable, as you have no idea what `1` could mean. Let's\nmake this a `unsigned flags` variable instead so that we can provide\nmeaningful names.\n\n> diff --git a/refs.h b/refs.h\n> index c05be6d0ac..c5e08db0ff 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -1300,20 +1300,25 @@ struct ref_iterator *refs_ref_iterator_begin(\n> int ref_iterator_advance(struct ref_iterator *ref_iterator);\n> \n> /*\n> - * Seek the iterator to the first reference with the given prefix.\n> - * The prefix is matched as a literal string, without regard for path\n> + * Seek the iterator to the first reference matching the given seek string.\n> + * The seek string is matched as a literal string, without regard for path\n> * separators. If prefix is NULL or the empty string, seek the iterator to the\n> * first reference again.\n> *\n> - * This function is expected to behave as if a new ref iterator with the same\n> - * prefix had been created, but allows reuse of iterators and thus may allow\n> - * the backend to optimize. Parameters other than the prefix that have been\n> - * passed when creating the iterator will remain unchanged.\n> + * When set_prefix is true, this function behaves as if a new ref iterator\n> + * with the same prefix had been created, setting the prefix for subsequent\n> + * iteration. When set_prefix is false, the iterator simply seeks to the\n> + * specified reference without changing the existing prefix, allowing\n> + * iteration to start from that specific reference.\n\nI think we should detangle this paragraph a bit.\n\n This function is expected to behave as if a new ref iterator has\n been created, but allows reuse of it\n"},{"id":"521229","messageId":"aGYbZcXP2voT2IYE@pks.im","threadId":"63722","inReplyTo":"20250701-306-git-for-each-ref-pagination-v1-4-4f0ae7c0688f@gmail.com","subject":"Re: [PATCH 4/4] for-each-ref: introduce a '--skip-until' option","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-03T05:55:49Z","receivedAt":"2025-07-03T05:55:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Jul 01, 2025 at 05:03:30PM +0200, Karthik Nayak wrote:\n> diff --git a/Documentation/git-for-each-ref.adoc b/Documentation/git-for-each-ref.adoc\n> index 5ef89fc0fe..4bf7c66b8c 100644\n> --- a/Documentation/git-for-each-ref.adoc\n> +++ b/Documentation/git-for-each-ref.adoc\n> @@ -14,7 +14,7 @@ SYNOPSIS\n> \t\t [--points-at=]\n> \t\t [--merged[=]] [--no-merged[=]]\n> \t\t [--contains[=]] [--no-contains[=]]\n> -\t\t [--exclude= ...]\n> +\t\t [--exclude= ...] [--skip-until=]\n> \n> DESCRIPTION\n> -----------\n> @@ -108,6 +108,10 @@ TAB %(refname)`.\n> --include-root-refs::\n> \tList root refs (HEAD and pseudorefs) apart from regular refs.\n> \n> +--skip-until::\n> + Skip references up to the specified pattern. Cannot be used with\n> + general pattern matching.\n> +\n> FIELD NAMES\n> -----------\n> \n\nIs it \"up to and including the specified pattern\" or \"up to but\nexcluding the specified pattern\"? It would help to make it very explicit\nwhether the pattern itself would be yielded or not.\n\n> diff --git a/ref-filter.c b/ref-filter.c\n> index 7a274633cf..9d0255d5db 100644\n> --- a/ref-filter.c\n> +++ b/ref-filter.c\n> @@ -2714,20 +2716,28 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n> \t\t * so just return everything and let the caller\n> \t\t * sort it out.\n> \t\t */\n> -\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n> -\t\t\t\t\t\t\"\", NULL, cb, cb_data);\n> +\t\tgoto non_prefix_iter;\n> \t}\n> \n> \tif (!filter->name_patterns[0]) {\n> \t\t/* no patterns; we have to look at everything */\n> -\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n> -\t\t\t\t\t\t \"\", filter->exclude.v, cb, cb_data);\n> +\t\tgoto non_prefix_iter;\n> \t}\n> \n> \treturn refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n> \t\t\t\t\t\t NULL, filter->name_patterns,\n> \t\t\t\t\t\t filter->exclude.v,\n> \t\t\t\t\t\t cb, cb_data);\n> +\n> +non_prefix_iter:\n> +\titer = refs_ref_iterator_begin(get_main_ref_store(the_repository), \"\",\n> +\t\t\t\t NULL, 0, flags);\n> +\tif (filter->seek)\n> +\t\tret = ref_iterator_seek(iter, filter->seek, 0);\n\nHm, this interface is somewhat weird now, as we have a split in what the\nprefix-string meeks when creating the iterator and seeking it. I think\nwe should align those two functions.\n\n> +\tif (ret)\n> +\t\treturn ret;\n> +\n> +\treturn do_for_each_ref_iterator(iter, cb, cb_data);\n> }\n> \n> /*\n> @@ -3200,6 +3210,8 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n> \tif (!filter->kind)\n> \t\tdie(\"filter_refs: invalid type\");\n> \telse {\n\nThe `if` branch now needs to be updated to have curly braces, as well.\n\nPatrick\n"},{"id":"521230","messageId":"xmqqsejdbywt.fsf@gitster.g","threadId":"63722","inReplyTo":"aGYSjf5H_ZBaVVJm@pks.im","subject":"Re: [PATCH 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-03T05:56:18Z","receivedAt":"2025-07-03T05:56:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> Even more importantly though, a numeric offset would be invalidated by a\n> concurrent write in case that write ends up inserting a ref in the range\n> of commits you intend to skip now.\n\nThat argument cuts both ways, no? You have shown up to some ref\nwhich you remember in the last cycle, and then while you are\nplanning to formulate another query with --skip-until naming that\nref, somebody removes that ref, then what happens? Or somebody\ninserts a new ref that sorts earlier than the ref you stopped at the\nlast time.\n\n"},{"id":"521238","messageId":"aGY9AyJ3c5wXpKaX@pks.im","threadId":"63722","inReplyTo":"xmqqsejdbywt.fsf@gitster.g","subject":"Re: [PATCH 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-03T08:19:15Z","receivedAt":"2025-07-03T08:19:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Jul 02, 2025 at 10:56:18PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > Even more importantly though, a numeric offset would be invalidated by a\n> > concurrent write in case that write ends up inserting a ref in the range\n> > of commits you intend to skip now.\n> \n> That argument cuts both ways, no? You have shown up to some ref\n> which you remember in the last cycle, and then while you are\n> planning to formulate another query with --skip-until naming that\n> ref, somebody removes that ref, then what happens?\n\nThis ref was already yielded, and it wouldn't and shouldn't be yielded\non the next page. This works as expected with the proposal, as\n`--skip-until` does not care whether the value itself actually exists.\n\n> Or somebody inserts a new ref that sorts earlier than the ref you\n> stopped at the last time.\n\nIt wouldn't and shouldn't be shown. When I have already yielded all refs\nup to refs/heads/something, I don't expect to see any ref that sorts\nbefore refs/heads/something on the next page.\n\nPatrick\n"},{"id":"521242","messageId":"CAOLa=ZRkBFdKH7ui+K=ocnYiTx0uJBppXHbFuhP8svCfWnN72w@mail.gmail.com","threadId":"63722","inReplyTo":"CAOLa=ZQZcX7Bh=63Bv5Sti89jsjq=FzVbkqqsP5Tgd7NzNr96g@mail.gmail.com","subject":"Re: [PATCH 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-03T08:41:50Z","receivedAt":"2025-07-03T08:41:52Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Karthik Nayak writes:\n\n> Junio C Hamano writes:\n>\n>> Offtopic. After applying this topic, I asked clang-format if it\n>> wants to change anything.\n>>\n>> $ git clang-format --diff $(git merge-base HEAD master)\n>>\n>> The result was disasterous. Can \"clang-format --diff\" mode be\n>> taught a bit more focused to avoid touching existing entries in the\n>> same array (in this case opts[] that has tons of options for the\n>> \"git for-each-ref\" command), when only one new entry was added, I\n>> wonder?\n>>\n>\n> I couldn't find any way to do something like this.\n>\n>> Also I am not impressed by the change it made to the code that is\n>> commented out (in refs.h).\n>>\n>> Line wrapping it did to refs_ref_iterator_begin() is an improvement,\n>> but those to ref_iterator_seek() and do_for_each_ref_iterator() are\n>> unnecessary (both of these were more readble in the original).\n>>\n>> Even though I found its output better for Toon's \"last-modified\"\n>> changes, I am not impressed by what clang-format suggested for this\n>> series.\n>>\n>\n> It indeed looks really bad, I had a go with the new changes from\n> 'gitster/kn/clang-format-updates'. Which seems a lot better.\n>\n> However, this does show a problem with using 'RemoveBracesLLVM', where\n> it formats the following:\n>\n> if (...) {\n> ...\n> ...\n> } else {\n> ...\n> }\n>\n> to:\n>\n> if (...) {\n> ...\n> ...\n> } else\n> ...\n>\n> Which isn't our style, I think we should completely drop this too, from\n> my patch series. Let me go ahead and do that. I really want to strip out\n> as many rules as possible to make the number of false positives 0 so we\n> can actually start enforcing clang-format. Once we enforce it, we can\n> slowly see what additional rules work well for us.\n>\n\nI did some more testing here, and it seems like this was because this\nparticular instance was more like\n\n if (...) {\n ...\n } else {\n ...\n }\n\nWhere both the clauses had single line statement, but we only modified\nthe 'else' part of the clause in this patch series, so clang-format,\nonly suggested removing the braces from the 'else' clause.\n\nSo all is good here, I think we can go ahead with the\n'gitster/kn/clang-format-updates' and merge it to 'next'. Sorry for\nbeing the false positive, I thought I missed testing a particular case\nand the series.\n"},{"id":"521243","messageId":"CAOLa=ZSN+Fvr0ixQWV0Becj-ELMRSkhm+POKF=BQ=F615sSj4A@mail.gmail.com","threadId":"63722","inReplyTo":"aGY9AyJ3c5wXpKaX@pks.im","subject":"Re: [PATCH 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-03T08:48:11Z","receivedAt":"2025-07-03T08:48:14Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Wed, Jul 02, 2025 at 10:56:18PM -0700, Junio C Hamano wrote:\n>> Patrick Steinhardt writes:\n>>\n>> > Even more importantly though, a numeric offset would be invalidated by a\n>> > concurrent write in case that write ends up inserting a ref in the range\n>> > of commits you intend to skip now.\n>>\n>> That argument cuts both ways, no? You have shown up to some ref\n>> which you remember in the last cycle, and then while you are\n>> planning to formulate another query with --skip-until naming that\n>> ref, somebody removes that ref, then what happens?\n>\n> This ref was already yielded, and it wouldn't and shouldn't be yielded\n> on the next page. This works as expected with the proposal, as\n> `--skip-until` does not care whether the value itself actually exists.\n\nThe current version of the series will include the reference provided to\n'--skip-until', if it exists. But your latter statement still holds, if\nthe reference doesn't exist, it will still work by finding the next\nreference in the default sort order.\n\n>> Or somebody inserts a new ref that sorts earlier than the ref you\n>> stopped at the last time.\n>\n> It wouldn't and shouldn't be shown. When I have already yielded all refs\n> up to refs/heads/something, I don't expect to see any ref that sorts\n> before refs/heads/something on the next page.\n>\n\nYeah, this was my thought too. Another way to think of this is that in a\ncursor based approach, a particular reference is guarateed never to\noccur again, even with modifications to the repository made between\nrequests. However in a count based approach this doesn't stand.\n\n> Patrick\n"},{"id":"521253","messageId":"CAOLa=ZQU7s2N+GqNKLQeAZVFGVXFKjVSeCLC35FGMqMZwt4zaw@mail.gmail.com","threadId":"63722","inReplyTo":"aGYbUpC9OEd90J4N@pks.im","subject":"Re: [PATCH 3/4] refs: selectively set prefix in the seek functions","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-03T09:40:50Z","receivedAt":"2025-07-03T09:40:52Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Tue, Jul 01, 2025 at 05:03:29PM +0200, Karthik Nayak wrote:\n>> The ref iterator exposes a `ref_iterator_seek()` function. The name\n>> suggests that this would seek the iterator to a specific reference in\n>> some ways similar to how `fseek()` works for the filesystem.\n>>\n>> However, the function actually sets the prefix for refs iteration. So\n>> further iteration would only yield references which match the particular\n>> prefix. This is a bit confusing.\n>>\n>> Let's add a 'set_prefix' field to the function, which when set, will set\n>> the prefix for the iteration in-line with the existing behavior. But\n>> when the 'set_prefix' field is not set, the reference backends will\n>> simply seek to the specified reference without setting prefix. This\n>> allows users to start iteration from a specific reference.\n>>\n>> In the packed and reftable backend, since references are available in a\n>> sorted list, the changes are simply setting the prefix if needed. The\n>> changes on the files-backend are a little more involved, since the files\n>> backend uses the 'ref-cache' mechanism. We move out the existing logic\n>> within `cache_ref_iterator_seek()` to `cache_ref_iterator_set_prefix()`\n>> which is called when `set_prefix` is set. We then parse the provided\n>> seek string and set the required levels and their indexes to ensure that\n>> seeking is possible.\n>\n> That solution makes sense.\n>\n>> diff --git a/refs.c b/refs.c\n>> index dce5c49ca2..a4220d3537 100644\n>> --- a/refs.c\n>> +++ b/refs.c\n>> @@ -2669,7 +2669,7 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs\n>> \t\t\tif (!iter) {\n>> \t\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n>> \t\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n>> -\t\t\t} else if (ref_iterator_seek(iter, dirname.buf) < 0) {\n>> +\t\t\t} else if (ref_iterator_seek(iter, dirname.buf, 1) < 0) {\n>> \t\t\t\tgoto cleanup;\n>> \t\t\t}\n>>\n>\n> This is quite unreadable, as you have no idea what `1` could mean. Let's\n> make this a `unsigned flags` variable instead so that we can provide\n> meaningful names.\n>\n\nYeah, that would make a lot more sense. Will amend.\n\n>> diff --git a/refs.h b/refs.h\n>> index c05be6d0ac..c5e08db0ff 100644\n>> --- a/refs.h\n>> +++ b/refs.h\n>> @@ -1300,20 +1300,25 @@ struct ref_iterator *refs_ref_iterator_begin(\n>> int ref_iterator_advance(struct ref_iterator *ref_iterator);\n>>\n>> /*\n>> - * Seek the iterator to the first reference with the given prefix.\n>> - * The prefix is matched as a literal string, without regard for path\n>> + * Seek the iterator to the first reference matching the given seek string.\n>> + * The seek string is matched as a literal string, without regard for path\n>> * separators. If prefix is NULL or the empty string, seek the iterator to the\n>> * first reference again.\n>> *\n>> - * This function is expected to behave as if a new ref iterator with the same\n>> - * prefix had been created, but allows reuse of iterators and thus may allow\n>> - * the backend to optimize. Parameters other than the prefix that have been\n>> - * passed when creating the iterator will remain unchanged.\n>> + * When set_prefix is true, this function behaves as if a new ref iterator\n>> + * with the same prefix had been created, setting the prefix for subsequent\n>> + * iteration. When set_prefix is false, the iterator simply seeks to the\n>> + * specified reference without changing the existing prefix, allowing\n>> + * iteration to start from that specific reference.\n>\n> I think we should detangle this paragraph a bit.\n>\n> This function is expected to behave as if a new ref iterator has\n> been created, but allows reuse of it\n\nSure, let me add this in. Thanks!\n"},{"id":"521254","messageId":"CAOLa=ZQvY6+fU-xDTH5jnrZxAsx2LxB2t-2QDu-8WpNrFpwjHQ@mail.gmail.com","threadId":"63722","inReplyTo":"aGYbZcXP2voT2IYE@pks.im","subject":"Re: [PATCH 4/4] for-each-ref: introduce a '--skip-until' option","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-03T10:02:03Z","receivedAt":"2025-07-03T10:02:05Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Tue, Jul 01, 2025 at 05:03:30PM +0200, Karthik Nayak wrote:\n>> diff --git a/Documentation/git-for-each-ref.adoc b/Documentation/git-for-each-ref.adoc\n>> index 5ef89fc0fe..4bf7c66b8c 100644\n>> --- a/Documentation/git-for-each-ref.adoc\n>> +++ b/Documentation/git-for-each-ref.adoc\n>> @@ -14,7 +14,7 @@ SYNOPSIS\n>> \t\t [--points-at=]\n>> \t\t [--merged[=]] [--no-merged[=]]\n>> \t\t [--contains[=]] [--no-contains[=]]\n>> -\t\t [--exclude= ...]\n>> +\t\t [--exclude= ...] [--skip-until=]\n>>\n>> DESCRIPTION\n>> -----------\n>> @@ -108,6 +108,10 @@ TAB %(refname)`.\n>> --include-root-refs::\n>> \tList root refs (HEAD and pseudorefs) apart from regular refs.\n>>\n>> +--skip-until::\n>> + Skip references up to the specified pattern. Cannot be used with\n>> + general pattern matching.\n>> +\n>> FIELD NAMES\n>> -----------\n>>\n>\n> Is it \"up to and including the specified pattern\" or \"up to but\n> excluding the specified pattern\"? It would help to make it very explicit\n> whether the pattern itself would be yielded or not.\n>\n\nIt is \"up to and including\", will modify to make this more clearer.\n\n>> diff --git a/ref-filter.c b/ref-filter.c\n>> index 7a274633cf..9d0255d5db 100644\n>> --- a/ref-filter.c\n>> +++ b/ref-filter.c\n>> @@ -2714,20 +2716,28 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n>> \t\t * so just return everything and let the caller\n>> \t\t * sort it out.\n>> \t\t */\n>> -\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n>> -\t\t\t\t\t\t\"\", NULL, cb, cb_data);\n>> +\t\tgoto non_prefix_iter;\n>> \t}\n>>\n>> \tif (!filter->name_patterns[0]) {\n>> \t\t/* no patterns; we have to look at everything */\n>> -\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n>> -\t\t\t\t\t\t \"\", filter->exclude.v, cb, cb_data);\n>> +\t\tgoto non_prefix_iter;\n>> \t}\n>>\n>> \treturn refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n>> \t\t\t\t\t\t NULL, filter->name_patterns,\n>> \t\t\t\t\t\t filter->exclude.v,\n>> \t\t\t\t\t\t cb, cb_data);\n>> +\n>> +non_prefix_iter:\n>> +\titer = refs_ref_iterator_begin(get_main_ref_store(the_repository), \"\",\n>> +\t\t\t\t NULL, 0, flags);\n>> +\tif (filter->seek)\n>> +\t\tret = ref_iterator_seek(iter, filter->seek, 0);\n>\n> Hm, this interface is somewhat weird now, as we have a split in what the\n> prefix-string meeks when creating the iterator and seeking it. I think\n> we should align those two functions.\n>\n\nThe `refs_ref_iterator_begin()` takes in a `prefix` string, which sets\nthe prefix.\n\nThe `ref_iterator_seek()` takes in a `seek` string, but a flag allows it\nalso set the prefix.\n\nI think this is okay since the naming matches what it does.\n\nThe alternate would be to `refs_ref_iterator_begin()` to also take in a\n`seek` string with a flag to also set the prefix. What do you think? I'm\nokay either ways.\n\n>> +\tif (ret)\n>> +\t\treturn ret;\n>> +\n>> +\treturn do_for_each_ref_iterator(iter, cb, cb_data);\n>> }\n>>\n>> /*\n>> @@ -3200,6 +3210,8 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n>> \tif (!filter->kind)\n>> \t\tdie(\"filter_refs: invalid type\");\n>> \telse {\n>\n> The `if` branch now needs to be updated to have curly braces, as well.\n>\n> Patrick\n\nYes, will add.\n\nThanks for the review!\n"},{"id":"521257","messageId":"aGZidwwlToWThkn8@pks.im","threadId":"63722","inReplyTo":"CAOLa=ZQvY6+fU-xDTH5jnrZxAsx2LxB2t-2QDu-8WpNrFpwjHQ@mail.gmail.com","subject":"Re: [PATCH 4/4] for-each-ref: introduce a '--skip-until' option","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-03T10:59:03Z","receivedAt":"2025-07-03T10:59:10Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Jul 03, 2025 at 03:02:03AM -0700, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> > On Tue, Jul 01, 2025 at 05:03:30PM +0200, Karthik Nayak wrote:\n> >> diff --git a/ref-filter.c b/ref-filter.c\n> >> index 7a274633cf..9d0255d5db 100644\n> >> --- a/ref-filter.c\n> >> +++ b/ref-filter.c\n> >> @@ -2714,20 +2716,28 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n> >> \t\t * so just return everything and let the caller\n> >> \t\t * sort it out.\n> >> \t\t */\n> >> -\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n> >> -\t\t\t\t\t\t\"\", NULL, cb, cb_data);\n> >> +\t\tgoto non_prefix_iter;\n> >> \t}\n> >>\n> >> \tif (!filter->name_patterns[0]) {\n> >> \t\t/* no patterns; we have to look at everything */\n> >> -\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n> >> -\t\t\t\t\t\t \"\", filter->exclude.v, cb, cb_data);\n> >> +\t\tgoto non_prefix_iter;\n> >> \t}\n> >>\n> >> \treturn refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n> >> \t\t\t\t\t\t NULL, filter->name_patterns,\n> >> \t\t\t\t\t\t filter->exclude.v,\n> >> \t\t\t\t\t\t cb, cb_data);\n> >> +\n> >> +non_prefix_iter:\n> >> +\titer = refs_ref_iterator_begin(get_main_ref_store(the_repository), \"\",\n> >> +\t\t\t\t NULL, 0, flags);\n> >> +\tif (filter->seek)\n> >> +\t\tret = ref_iterator_seek(iter, filter->seek, 0);\n> >\n> > Hm, this interface is somewhat weird now, as we have a split in what the\n> > prefix-string meeks when creating the iterator and seeking it. I think\n> > we should align those two functions.\n> >\n> \n> The `refs_ref_iterator_begin()` takes in a `prefix` string, which sets\n> the prefix.\n> \n> The `ref_iterator_seek()` takes in a `seek` string, but a flag allows it\n> also set the prefix.\n> \n> I think this is okay since the naming matches what it does.\n> \n> The alternate would be to `refs_ref_iterator_begin()` to also take in a\n> `seek` string with a flag to also set the prefix. What do you think? I'm\n> okay either ways.\n\nI just think that the interface is a bit confusing. It's weird that the\nneedle that we're seeking for may or may not be used to update internal\nstate, and that this is inconsistent with the similar fields that you\npass to the iterator when creating it. So after seeking it sometimes\nacts like you have created a new iterator with the needle, sometimes it\ndoes not becaus we retain internal state. This kind of inconsistency\ninvites mistakes.\n\nPatrick\n"},{"id":"521312","messageId":"20250704-306-git-for-each-ref-pagination-v2-0-bcde14acdd81@gmail.com","threadId":"63722","inReplyTo":"20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com","subject":"[PATCH v2 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-04T13:02:28Z","receivedAt":"2025-07-04T13:02:46Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `git-for-each-ref(1)` command is used to iterate over references\npresent in a repository. In large repositories with millions of\nreferences, it would be optimal to paginate this output such that we\ncan start iteration from a given reference. This would avoid having to\niterate over all references from the beginning each time when paginating\nthrough results.\n\nThis series adds a '--skip-until' option in 'git-for-each-ref(1)'. When\nused, the reference iteration seeks to the first matching reference and\niterates from there onward.\n\nThis enables efficient pagination workflows like:\n git for-each-ref --count=100\n git for-each-ref --count=100 --skip-until=refs/heads/branch-100\n git for-each-ref --count=100 --skip-until=refs/heads/branch-200\n\nTo add this functionality, we expose the `ref_iterator` outside the\n'refs/' namespace and modify the `ref_iterator_seek()` to actually seek\nto a given reference and only set the prefix when the `set_prefix` field\nis set.\n\nOn the reftable and packed backend, the changes are simple. But since\nthe files backend uses 'ref-cache' for reference handling, the changes\nthere are a little more involved, since we need to setup the right\nlevels and the indexing.\n\nInitially I was also planning to cleanup all the `refs_for_each...()`\nfunctions in 'refs.h' by simply using the iterator, but this bloated the\nseries. So I've left that for another day.\n\nChanges in v2:\n- Modify 'ref_iterator_seek()' to take in flags instead of a\n 'set_prefix' variable. This improves readability, where users would\n use the 'REF_ITERATOR_SEEK_SET_PREFIX' instead of simply passing '1'.\n- When the set prefix flag isn't usage, reset any previously set prefix.\n This ensures that the internal prefix state is always reset whenever\n we seek and unifies the behavior between 'ref_iterator_seek' and\n 'ref_iterator_begin'.\n- Don't allow '--skip-until' to be run with '--sort', since the seeking\n always takes place before any sorting and this can be confusing.\n- Some styling fixes:\n - Remove extra newline\n - Skip braces around single lined if...else clause\n - Add braces around 'if' clause\n - Fix indentation\n- Link to v1: https://lore.kernel.org/git/20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com/\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/git-for-each-ref.adoc | 6 +-\n builtin/for-each-ref.c | 8 ++\n ref-filter.c | 61 ++++++++----\n ref-filter.h | 1 +\n refs.c | 6 +-\n refs.h | 158 ++++++++++++++++++++++++++++++\n refs/debug.c | 7 +-\n refs/files-backend.c | 7 +-\n refs/iterator.c | 26 ++---\n refs/packed-backend.c | 17 ++--\n refs/ref-cache.c | 99 +++++++++++++++----\n refs/ref-cache.h | 7 --\n refs/refs-internal.h | 152 ++---------------------------\n refs/reftable-backend.c | 21 ++--\n t/t6302-for-each-ref-filter.sh | 188 ++++++++++++++++++++++++++++++++++++\n 15 files changed, 538 insertions(+), 226 deletions(-)\n\nKarthik Nayak (4):\n refs: expose `ref_iterator` via 'refs.h'\n ref-cache: remove unused function 'find_ref_entry()'\n refs: selectively set prefix in the seek functions\n for-each-ref: introduce a '--skip-until' option\n\nRange-diff versus v1:\n\n1: a20ef2c841 ! 1: d19553c365 refs: expose `ref_iterator` via 'refs.h'\n @@ refs.h: int repo_migrate_ref_storage_format(struct repository *repo,\n + * The output is ordered by refname.\n + */\n +struct ref_iterator *refs_ref_iterator_begin(\n -+\t\tstruct ref_store *refs,\n -+\t\tconst char *prefix, const char **exclude_patterns,\n -+\t\tint trim, enum do_for_each_ref_flags flags);\n ++\tstruct ref_store *refs,\n ++\tconst char *prefix, const char **exclude_patterns,\n ++\tint trim, enum do_for_each_ref_flags flags);\n +\n +/*\n + * Advance the iterator to the first or next item and return ITER_OK.\n @@ refs.h: int repo_migrate_ref_storage_format(struct repository *repo,\n + */\n +int do_for_each_ref_iterator(struct ref_iterator *iter,\n +\t\t\t each_ref_fn fn, void *cb_data);\n -+\n +\n #endif /* REFS_H */\n \n2: 96f3e6eb05 = 2: 1d3936132b ref-cache: remove unused function 'find_ref_entry()'\n3: fa19b53a37 ! 3: aab2011494 refs: selectively set prefix in the seek functions\n @@ Commit message\n further iteration would only yield references which match the particular\n prefix. This is a bit confusing.\n \n - Let's add a 'set_prefix' field to the function, which when set, will set\n - the prefix for the iteration in-line with the existing behavior. But\n - when the 'set_prefix' field is not set, the reference backends will\n - simply seek to the specified reference without setting prefix. This\n - allows users to start iteration from a specific reference.\n + Let's add a 'flags' field to the function, which when set with the\n + 'REF_ITERATOR_SEEK_SET_PREFIX' flag, will set the prefix for the\n + iteration in-line with the existing behavior. Otherwise, the reference\n + backends will simply seek to the specified reference and clears any\n + previously set prefix. This allows users to start iteration from a\n + specific reference.\n \n In the packed and reftable backend, since references are available in a\n sorted list, the changes are simply setting the prefix if needed. The\n changes on the files-backend are a little more involved, since the files\n backend uses the 'ref-cache' mechanism. We move out the existing logic\n within `cache_ref_iterator_seek()` to `cache_ref_iterator_set_prefix()`\n - which is called when `set_prefix` is set. We then parse the provided\n - seek string and set the required levels and their indexes to ensure that\n - seeking is possible.\n + which is called when the 'REF_ITERATOR_SEEK_SET_PREFIX' flag is set. We\n + then parse the provided seek string and set the required levels and\n + their indexes to ensure that seeking is possible.\n \n + Helped-by: Patrick Steinhardt \n Signed-off-by: Karthik Nayak \n \n ## refs.c ##\n @@ refs.c: enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs\n - \t\t\tif (!iter) {\n + \t\tif (!initial_transaction) {\n + \t\t\tint ok;\n + \n +-\t\t\tif (!iter) {\n ++\t\t\tif (!iter)\n \t\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n \t\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n -\t\t\t} else if (ref_iterator_seek(iter, dirname.buf) < 0) {\n -+\t\t\t} else if (ref_iterator_seek(iter, dirname.buf, 1) < 0) {\n ++\t\t\telse if (ref_iterator_seek(iter, dirname.buf,\n ++\t\t\t\t\t\t REF_ITERATOR_SEEK_SET_PREFIX) < 0)\n \t\t\t\tgoto cleanup;\n - \t\t\t}\n +-\t\t\t}\n \n + \t\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n + \t\t\t\tif (skip &&\n \n ## refs.h ##\n @@ refs.h: struct ref_iterator *refs_ref_iterator_begin(\n + */\n int ref_iterator_advance(struct ref_iterator *ref_iterator);\n \n ++enum ref_iterator_seek_flag {\n ++\t/*\n ++\t * Also set the seek pattern as a prefix for iteration. This ensures\n ++\t * that only references which match the prefix are yielded.\n ++\t */\n ++\tREF_ITERATOR_SEEK_SET_PREFIX = (1 << 0),\n ++};\n ++\n /*\n - * Seek the iterator to the first reference with the given prefix.\n - * The prefix is matched as a literal string, without regard for path\n +- * separators. If prefix is NULL or the empty string, seek the iterator to the\n + * Seek the iterator to the first reference matching the given seek string.\n + * The seek string is matched as a literal string, without regard for path\n - * separators. If prefix is NULL or the empty string, seek the iterator to the\n ++ * separators. If seek is NULL or the empty string, seek the iterator to the\n * first reference again.\n *\n - * This function is expected to behave as if a new ref iterator with the same\n - * prefix had been created, but allows reuse of iterators and thus may allow\n - * the backend to optimize. Parameters other than the prefix that have been\n - * passed when creating the iterator will remain unchanged.\n -+ * When set_prefix is true, this function behaves as if a new ref iterator\n -+ * with the same prefix had been created, setting the prefix for subsequent\n -+ * iteration. When set_prefix is false, the iterator simply seeks to the\n -+ * specified reference without changing the existing prefix, allowing\n -+ * iteration to start from that specific reference.\n ++ * This function is expected to behave as if a new ref iterator has been\n ++ * created, but allows reuse of existing iterators for optimization.\n + *\n -+ * This function allows reuse of iterators and thus may allow the backend\n -+ * to optimize. Parameters other than the prefix that have been passed when\n -+ * creating the iterator will remain unchanged.\n ++ * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is\n ++ * updated to match the seek string, affecting all subsequent iterations. If\n ++ * not, the iterator seeks to the specified reference and clears any previously\n ++ * set prefix.\n *\n * Returns 0 on success, a negative error code otherwise.\n */\n - int ref_iterator_seek(struct ref_iterator *ref_iterator,\n +-int ref_iterator_seek(struct ref_iterator *ref_iterator,\n -\t\t const char *prefix);\n -+\t\t const char *seek, int set_prefix);\n ++int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,\n ++\t\t unsigned int flags);\n \n /*\n * If possible, peel the reference currently being viewed by the\n @@ refs/debug.c: static int debug_ref_iterator_advance(struct ref_iterator *ref_ite\n \n static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator,\n -\t\t\t\t const char *prefix)\n -+\t\t\t\t const char *seek, int set_prefix)\n ++\t\t\t\t const char *seek, unsigned int flags)\n {\n \tstruct debug_ref_iterator *diter =\n \t\t(struct debug_ref_iterator *)ref_iterator;\n -\tint res = diter->iter->vtable->seek(diter->iter, prefix);\n -\ttrace_printf_key(&trace_refs, \"iterator_seek: %s: %d\\n\", prefix ? prefix : \"\", res);\n -+\tint res = diter->iter->vtable->seek(diter->iter, seek, set_prefix);\n -+\ttrace_printf_key(&trace_refs, \"iterator_seek: %s set_prefix: %d: %d\\n\",\n -+\t\t\t seek ? seek : \"\", set_prefix, res);\n ++\tint res = diter->iter->vtable->seek(diter->iter, seek, flags);\n ++\ttrace_printf_key(&trace_refs, \"iterator_seek: %s flags: %d: %d\\n\",\n ++\t\t\t seek ? seek : \"\", flags, res);\n \treturn res;\n }\n \n @@ refs/files-backend.c: static int files_ref_iterator_advance(struct ref_iterator\n \n static int files_ref_iterator_seek(struct ref_iterator *ref_iterator,\n -\t\t\t\t const char *prefix)\n -+\t\t\t\t const char *seek, int set_prefix)\n ++\t\t\t\t const char *seek, unsigned int flags)\n {\n \tstruct files_ref_iterator *iter =\n \t\t(struct files_ref_iterator *)ref_iterator;\n -\treturn ref_iterator_seek(iter->iter0, prefix);\n -+\treturn ref_iterator_seek(iter->iter0, seek, set_prefix);\n ++\treturn ref_iterator_seek(iter->iter0, seek, flags);\n }\n \n static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n @@ refs/files-backend.c: static int files_reflog_iterator_advance(struct ref_iterat\n static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n -\t\t\t\t const char *prefix UNUSED)\n +\t\t\t\t const char *seek UNUSED,\n -+\t\t\t\t int set_prefix UNUSED)\n ++\t\t\t\t unsigned int flags UNUSED)\n {\n \tBUG(\"ref_iterator_seek() called for reflog_iterator\");\n }\n \n ## refs/iterator.c ##\n @@ refs/iterator.c: int ref_iterator_advance(struct ref_iterator *ref_iterator)\n + \treturn ref_iterator->vtable->advance(ref_iterator);\n }\n \n - int ref_iterator_seek(struct ref_iterator *ref_iterator,\n +-int ref_iterator_seek(struct ref_iterator *ref_iterator,\n -\t\t const char *prefix)\n -+\t\t const char *seek, int set_prefix)\n ++int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,\n ++\t\t unsigned int flags)\n {\n -\treturn ref_iterator->vtable->seek(ref_iterator, prefix);\n -+\treturn ref_iterator->vtable->seek(ref_iterator, seek, set_prefix);\n ++\treturn ref_iterator->vtable->seek(ref_iterator, seek, flags);\n }\n \n int ref_iterator_peel(struct ref_iterator *ref_iterator,\n @@ refs/iterator.c: static int empty_ref_iterator_advance(struct ref_iterator *ref_\n static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n -\t\t\t\t const char *prefix UNUSED)\n +\t\t\t\t const char *seek UNUSED,\n -+\t\t\t\t int set_prefix UNUSED)\n ++\t\t\t\t unsigned int flags UNUSED)\n {\n \treturn 0;\n }\n @@ refs/iterator.c: static int merge_ref_iterator_advance(struct ref_iterator *ref_\n \n static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,\n -\t\t\t\t const char *prefix)\n -+\t\t\t\t const char *seek, int set_prefix)\n ++\t\t\t\t const char *seek, unsigned int flags)\n {\n \tstruct merge_ref_iterator *iter =\n \t\t(struct merge_ref_iterator *)ref_iterator;\n @@ refs/iterator.c: static int merge_ref_iterator_seek(struct ref_iterator *ref_ite\n \titer->iter1 = iter->iter1_owned;\n \n -\tret = ref_iterator_seek(iter->iter0, prefix);\n -+\tret = ref_iterator_seek(iter->iter0, seek, set_prefix);\n ++\tret = ref_iterator_seek(iter->iter0, seek, flags);\n \tif (ret < 0)\n \t\treturn ret;\n \n -\tret = ref_iterator_seek(iter->iter1, prefix);\n -+\tret = ref_iterator_seek(iter->iter1, seek, set_prefix);\n ++\tret = ref_iterator_seek(iter->iter1, seek, flags);\n \tif (ret < 0)\n \t\treturn ret;\n \n @@ refs/iterator.c: static int prefix_ref_iterator_advance(struct ref_iterator *ref\n \n static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,\n -\t\t\t\t const char *prefix)\n -+\t\t\t\t const char *seek, int set_prefix)\n ++\t\t\t\t const char *seek, unsigned int flags)\n {\n \tstruct prefix_ref_iterator *iter =\n \t\t(struct prefix_ref_iterator *)ref_iterator;\n @@ refs/iterator.c: static int prefix_ref_iterator_advance(struct ref_iterator *ref\n -\titer->prefix = xstrdup_or_null(prefix);\n -\treturn ref_iterator_seek(iter->iter0, prefix);\n +\n -+\tif (set_prefix) {\n ++\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX) {\n +\t\tfree(iter->prefix);\n +\t\titer->prefix = xstrdup_or_null(seek);\n +\t}\n -+\treturn ref_iterator_seek(iter->iter0, seek, set_prefix);\n ++\treturn ref_iterator_seek(iter->iter0, seek, flags);\n }\n \n static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,\n @@ refs/packed-backend.c: static int packed_ref_iterator_advance(struct ref_iterato\n \n static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,\n -\t\t\t\t const char *prefix)\n -+\t\t\t\t const char *seek, int set_prefix)\n ++\t\t\t\t const char *seek, unsigned int flags)\n {\n \tstruct packed_ref_iterator *iter =\n \t\t(struct packed_ref_iterator *)ref_iterator;\n @@ refs/packed-backend.c: static int packed_ref_iterator_advance(struct ref_iterato\n \n -\tfree(iter->prefix);\n -\titer->prefix = xstrdup_or_null(prefix);\n -+\tif (set_prefix) {\n -+\t\tfree(iter->prefix);\n ++\t/* Unset any previously set prefix */\n ++\tFREE_AND_NULL(iter->prefix);\n ++\n ++\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX)\n +\t\titer->prefix = xstrdup_or_null(seek);\n -+\t}\n +\n \titer->pos = start;\n \titer->eof = iter->snapshot->eof;\n @@ refs/packed-backend.c: static struct ref_iterator *packed_ref_iterator_begin(\n \titer->flags = flags;\n \n -\tif (packed_ref_iterator_seek(&iter->base, prefix) < 0) {\n -+\tif (packed_ref_iterator_seek(&iter->base, prefix, 1) < 0) {\n ++\tif (packed_ref_iterator_seek(&iter->base, prefix,\n ++\t\t\t\t REF_ITERATOR_SEEK_SET_PREFIX) < 0) {\n \t\tref_iterator_free(&iter->base);\n \t\treturn NULL;\n \t}\n @@ refs/ref-cache.c: static int cache_ref_iterator_seek(struct ref_iterator *ref_it\n }\n \n +static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n -+\t\t\t\t const char *seek, int set_prefix)\n ++\t\t\t\t const char *seek, unsigned int flags)\n +{\n +\tstruct cache_ref_iterator *iter =\n +\t\t(struct cache_ref_iterator *)ref_iterator;\n +\n -+\tif (set_prefix) {\n ++\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX) {\n +\t\treturn cache_ref_iterator_set_prefix(iter, seek);\n +\t} else if (seek && *seek) {\n +\t\tstruct cache_ref_iterator_level *level;\n @@ refs/ref-cache.c: static int cache_ref_iterator_seek(struct ref_iterator *ref_it\n +\t\tlevel->index = -1;\n +\t\tlevel->dir = dir;\n +\n ++\t\t/* Unset any previously set prefix */\n ++\t\tFREE_AND_NULL(iter->prefix);\n ++\n +\t\t/*\n +\t\t * Breakdown the provided seek path and assign the correct\n +\t\t * indexing to each level as needed.\n @@ refs/ref-cache.c: struct ref_iterator *cache_ref_iterator_begin(struct ref_cache\n \titer->prime_dir = prime_dir;\n \n -\tif (cache_ref_iterator_seek(&iter->base, prefix) < 0) {\n -+\tif (cache_ref_iterator_seek(&iter->base, prefix, 1) < 0) {\n ++\tif (cache_ref_iterator_seek(&iter->base, prefix,\n ++\t\t\t\t REF_ITERATOR_SEEK_SET_PREFIX) < 0) {\n \t\tref_iterator_free(&iter->base);\n \t\treturn NULL;\n \t}\n @@ refs/refs-internal.h: void base_ref_iterator_init(struct ref_iterator *iter,\n */\n typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,\n -\t\t\t\t const char *prefix);\n -+\t\t\t\t const char *seek, int set_prefix);\n ++\t\t\t\t const char *seek, unsigned int flags);\n \n /*\n * Peels the current ref, returning 0 for success or -1 for failure.\n @@ refs/reftable-backend.c: static int reftable_ref_iterator_advance(struct ref_ite\n \n static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,\n -\t\t\t\t const char *prefix)\n -+\t\t\t\t const char *seek, int set_prefix)\n ++\t\t\t\t const char *seek, unsigned int flags)\n {\n \tstruct reftable_ref_iterator *iter =\n \t\t(struct reftable_ref_iterator *)ref_iterator;\n @@ refs/reftable-backend.c: static int reftable_ref_iterator_advance(struct ref_ite\n -\titer->prefix = xstrdup_or_null(prefix);\n -\titer->prefix_len = prefix ? strlen(prefix) : 0;\n -\titer->err = reftable_iterator_seek_ref(&iter->iter, prefix);\n -+\tif (set_prefix) {\n -+\t\tfree(iter->prefix);\n ++\t/* Unset any previously set prefix */\n ++\tFREE_AND_NULL(iter->prefix);\n ++\titer->prefix_len = 0;\n ++\n ++\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX) {\n +\t\titer->prefix = xstrdup_or_null(seek);\n +\t\titer->prefix_len = seek ? strlen(seek) : 0;\n +\t}\n @@ refs/reftable-backend.c: static struct reftable_ref_iterator *ref_iterator_for_s\n \t\tgoto done;\n \n -\tret = reftable_ref_iterator_seek(&iter->base, prefix);\n -+\tret = reftable_ref_iterator_seek(&iter->base, prefix, 1);\n ++\tret = reftable_ref_iterator_seek(&iter->base, prefix,\n ++\t\t\t\t\t REF_ITERATOR_SEEK_SET_PREFIX);\n \tif (ret)\n \t\tgoto done;\n \n @@ refs/reftable-backend.c: static int reftable_reflog_iterator_advance(struct ref_\n static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n -\t\t\t\t\t const char *prefix UNUSED)\n +\t\t\t\t\t const char *seek UNUSED,\n -+\t\t\t\t\t int set_prefix UNUSED)\n ++\t\t\t\t\t unsigned int flags UNUSED)\n {\n \tBUG(\"reftable reflog iterator cannot be seeked\");\n \treturn -1;\n4: 0cfc879a93 ! 4: dc6ffa9002 for-each-ref: introduce a '--skip-until' option\n @@ Documentation/git-for-each-ref.adoc: TAB %(refname)`.\n \tList root refs (HEAD and pseudorefs) apart from regular refs.\n \n +--skip-until::\n -+ Skip references up to the specified pattern. Cannot be used with\n -+ general pattern matching.\n ++ Skip references up to but excluding the specified pattern. Cannot be used\n ++ with general pattern matching or custom sort options.\n +\n FIELD NAMES\n -----------\n @@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc,\n \t\tOPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n \t\tOPT_REF_FILTER_EXCLUDE(&filter),\n \t\tOPT_REF_SORT(&sorting_options),\n +@@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc,\n + \tif (verify_ref_format(&format))\n + \t\tusage_with_options(for_each_ref_usage, opts);\n + \n ++\tif (filter.seek && sorting_options.nr > 1)\n ++\t\tdie(_(\"cannot use --skip-until custom sort options\"));\n ++\n + \tsorting = ref_sorting_options(&sorting_options);\n + \tref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);\n + \tfilter.ignore_case = icase;\n @@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc,\n \t\tfilter.name_patterns = argv;\n \t}\n @@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \n /*\n @@ ref-filter.c: static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n - \tif (!filter->kind)\n + \tinit_contains_cache(&filter->internal.no_contains_cache);\n + \n + \t/* Simple per-ref filtering */\n +-\tif (!filter->kind)\n ++\tif (!filter->kind) {\n \t\tdie(\"filter_refs: invalid type\");\n - \telse {\n +-\telse {\n ++\t} else {\n +\t\tconst char *prefix = NULL;\n +\n \t\t/*\n @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '\n +\tcat >expect <<-\\EOF &&\n +\tfatal: cannot use --skip-until with patterns\n +\tEOF\n -+\ttest_must_fail git for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/spot refs/tags 2>actual &&\n ++\ttest_must_fail git for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/spot refs/tags 2>actual &&\n ++\ttest_cmp expect actual\n ++'\n ++\n ++test_expect_success 'skip until used with custom sort order' '\n ++\tcat >expect <<-\\EOF &&\n ++\tfatal: cannot use --skip-until custom sort options\n ++\tEOF\n ++\ttest_must_fail git for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/spot --sort=author 2>actual &&\n +\ttest_cmp expect actual\n +'\n +\n\n\nbase-commit: cf6f63ea6bf35173e02e18bdc6a4ba41288acff9\nchange-id: 20250605-306-git-for-each-ref-pagination-0ba8a29ae646\n\nThanks\n- Karthik\n\n"},{"id":"521313","messageId":"20250704-306-git-for-each-ref-pagination-v2-1-bcde14acdd81@gmail.com","threadId":"63722","inReplyTo":"20250704-306-git-for-each-ref-pagination-v2-0-bcde14acdd81@gmail.com","subject":"[PATCH v2 1/4] refs: expose `ref_iterator` via 'refs.h'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-04T13:02:29Z","receivedAt":"2025-07-04T13:02:47Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `ref_iterator` is an internal structure to the 'refs/'\nsub-directory, which allows iteration over refs. All reference iteration\nis built on top of these iterators.\n\nExternal clients of the 'refs' subsystem use the various\n'refs_for_each...()' functions to iterate over refs. However since these\nare wrapper functions, each combination of functionality requires a new\nwrapper function. This is not feasible as the functions pile up with the\nincrease in requirements. Expose the internal reference iterator, so\nadvanced users can mix and match options as needed.\n\nSigned-off-by: Karthik Nayak \n---\n refs.h | 147 +++++++++++++++++++++++++++++++++++++++++++++++++++\n refs/refs-internal.h | 145 +-------------------------------------------------\n 2 files changed, 149 insertions(+), 143 deletions(-)\n\ndiff --git a/refs.h b/refs.h\nindex 46a6008e07..7c21aaef3d 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -1190,4 +1190,151 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n \t\t\t\t unsigned int flags,\n \t\t\t\t struct strbuf *err);\n \n+/*\n+ * Reference iterators\n+ *\n+ * A reference iterator encapsulates the state of an in-progress\n+ * iteration over references. Create an instance of `struct\n+ * ref_iterator` via one of the functions in this module.\n+ *\n+ * A freshly-created ref_iterator doesn't yet point at a reference. To\n+ * advance the iterator, call ref_iterator_advance(). If successful,\n+ * this sets the iterator's refname, oid, and flags fields to describe\n+ * the next reference and returns ITER_OK. The data pointed at by\n+ * refname and oid belong to the iterator; if you want to retain them\n+ * after calling ref_iterator_advance() again or calling\n+ * ref_iterator_free(), you must make a copy. When the iteration has\n+ * been exhausted, ref_iterator_advance() releases any resources\n+ * associated with the iteration, frees the ref_iterator object, and\n+ * returns ITER_DONE. If you want to abort the iteration early, call\n+ * ref_iterator_free(), which also frees the ref_iterator object and\n+ * any associated resources. If there was an internal error advancing\n+ * to the next entry, ref_iterator_advance() aborts the iteration,\n+ * frees the ref_iterator, and returns ITER_ERROR.\n+ *\n+ * The reference currently being looked at can be peeled by calling\n+ * ref_iterator_peel(). This function is often faster than peel_ref(),\n+ * so it should be preferred when iterating over references.\n+ *\n+ * Putting it all together, a typical iteration looks like this:\n+ *\n+ * int ok;\n+ * struct ref_iterator *iter = ...;\n+ *\n+ * while ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n+ * if (want_to_stop_iteration()) {\n+ * ok = ITER_DONE;\n+ * break;\n+ * }\n+ *\n+ * // Access information about the current reference:\n+ * if (!(iter->flags & REF_ISSYMREF))\n+ * printf(\"%s is %s\\n\", iter->refname, oid_to_hex(iter->oid));\n+ *\n+ * // If you need to peel the reference:\n+ * ref_iterator_peel(iter, &oid);\n+ * }\n+ *\n+ * if (ok != ITER_DONE)\n+ * handle_error();\n+ * ref_iterator_free(iter);\n+ */\n+struct ref_iterator;\n+\n+/*\n+ * These flags are passed to refs_ref_iterator_begin() (and do_for_each_ref(),\n+ * which feeds it).\n+ */\n+enum do_for_each_ref_flags {\n+\t/*\n+\t * Include broken references in a do_for_each_ref*() iteration, which\n+\t * would normally be omitted. This includes both refs that point to\n+\t * missing objects (a true repository corruption), ones with illegal\n+\t * names (which we prefer not to expose to callers), as well as\n+\t * dangling symbolic refs (i.e., those that point to a non-existent\n+\t * ref; this is not a corruption, but as they have no valid oid, we\n+\t * omit them from normal iteration results).\n+\t */\n+\tDO_FOR_EACH_INCLUDE_BROKEN = (1 << 0),\n+\n+\t/*\n+\t * Only include per-worktree refs in a do_for_each_ref*() iteration.\n+\t * Normally this will be used with a files ref_store, since that's\n+\t * where all reference backends will presumably store their\n+\t * per-worktree refs.\n+\t */\n+\tDO_FOR_EACH_PER_WORKTREE_ONLY = (1 << 1),\n+\n+\t/*\n+\t * Omit dangling symrefs from output; this only has an effect with\n+\t * INCLUDE_BROKEN, since they are otherwise not included at all.\n+\t */\n+\tDO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 << 2),\n+\n+\t/*\n+\t * Include root refs i.e. HEAD and pseudorefs along with the regular\n+\t * refs.\n+\t */\n+\tDO_FOR_EACH_INCLUDE_ROOT_REFS = (1 << 3),\n+};\n+\n+/*\n+ * Return an iterator that goes over each reference in `refs` for\n+ * which the refname begins with prefix. If trim is non-zero, then\n+ * trim that many characters off the beginning of each refname.\n+ * The output is ordered by refname.\n+ */\n+struct ref_iterator *refs_ref_iterator_begin(\n+\tstruct ref_store *refs,\n+\tconst char *prefix, const char **exclude_patterns,\n+\tint trim, enum do_for_each_ref_flags flags);\n+\n+/*\n+ * Advance the iterator to the first or next item and return ITER_OK.\n+ * If the iteration is exhausted, free the resources associated with\n+ * the ref_iterator and return ITER_DONE. On errors, free the iterator\n+ * resources and return ITER_ERROR. It is a bug to use ref_iterator or\n+ * call this function again after it has returned ITER_DONE or\n+ * ITER_ERROR.\n+ */\n+int ref_iterator_advance(struct ref_iterator *ref_iterator);\n+\n+/*\n+ * Seek the iterator to the first reference with the given prefix.\n+ * The prefix is matched as a literal string, without regard for path\n+ * separators. If prefix is NULL or the empty string, seek the iterator to the\n+ * first reference again.\n+ *\n+ * This function is expected to behave as if a new ref iterator with the same\n+ * prefix had been created, but allows reuse of iterators and thus may allow\n+ * the backend to optimize. Parameters other than the prefix that have been\n+ * passed when creating the iterator will remain unchanged.\n+ *\n+ * Returns 0 on success, a negative error code otherwise.\n+ */\n+int ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t const char *prefix);\n+\n+/*\n+ * If possible, peel the reference currently being viewed by the\n+ * iterator. Return 0 on success.\n+ */\n+int ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t struct object_id *peeled);\n+\n+/* Free the reference iterator and any associated resources. */\n+void ref_iterator_free(struct ref_iterator *ref_iterator);\n+\n+/*\n+ * The common backend for the for_each_*ref* functions. Call fn for\n+ * each reference in iter. If the iterator itself ever returns\n+ * ITER_ERROR, return -1. If fn ever returns a non-zero value, stop\n+ * the iteration and return that value. Otherwise, return 0. In any\n+ * case, free the iterator when done. This function is basically an\n+ * adapter between the callback style of reference iteration and the\n+ * iterator style.\n+ */\n+int do_for_each_ref_iterator(struct ref_iterator *iter,\n+\t\t\t each_ref_fn fn, void *cb_data);\n+\n #endif /* REFS_H */\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex f868870851..03f5df04d5 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -244,90 +244,8 @@ const char *find_descendant_ref(const char *dirname,\n #define SYMREF_MAXDEPTH 5\n \n /*\n- * These flags are passed to refs_ref_iterator_begin() (and do_for_each_ref(),\n- * which feeds it).\n- */\n-enum do_for_each_ref_flags {\n-\t/*\n-\t * Include broken references in a do_for_each_ref*() iteration, which\n-\t * would normally be omitted. This includes both refs that point to\n-\t * missing objects (a true repository corruption), ones with illegal\n-\t * names (which we prefer not to expose to callers), as well as\n-\t * dangling symbolic refs (i.e., those that point to a non-existent\n-\t * ref; this is not a corruption, but as they have no valid oid, we\n-\t * omit them from normal iteration results).\n-\t */\n-\tDO_FOR_EACH_INCLUDE_BROKEN = (1 << 0),\n-\n-\t/*\n-\t * Only include per-worktree refs in a do_for_each_ref*() iteration.\n-\t * Normally this will be used with a files ref_store, since that's\n-\t * where all reference backends will presumably store their\n-\t * per-worktree refs.\n-\t */\n-\tDO_FOR_EACH_PER_WORKTREE_ONLY = (1 << 1),\n-\n-\t/*\n-\t * Omit dangling symrefs from output; this only has an effect with\n-\t * INCLUDE_BROKEN, since they are otherwise not included at all.\n-\t */\n-\tDO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 << 2),\n-\n-\t/*\n-\t * Include root refs i.e. HEAD and pseudorefs along with the regular\n-\t * refs.\n-\t */\n-\tDO_FOR_EACH_INCLUDE_ROOT_REFS = (1 << 3),\n-};\n-\n-/*\n- * Reference iterators\n- *\n- * A reference iterator encapsulates the state of an in-progress\n- * iteration over references. Create an instance of `struct\n- * ref_iterator` via one of the functions in this module.\n- *\n- * A freshly-created ref_iterator doesn't yet point at a reference. To\n- * advance the iterator, call ref_iterator_advance(). If successful,\n- * this sets the iterator's refname, oid, and flags fields to describe\n- * the next reference and returns ITER_OK. The data pointed at by\n- * refname and oid belong to the iterator; if you want to retain them\n- * after calling ref_iterator_advance() again or calling\n- * ref_iterator_free(), you must make a copy. When the iteration has\n- * been exhausted, ref_iterator_advance() releases any resources\n- * associated with the iteration, frees the ref_iterator object, and\n- * returns ITER_DONE. If you want to abort the iteration early, call\n- * ref_iterator_free(), which also frees the ref_iterator object and\n- * any associated resources. If there was an internal error advancing\n- * to the next entry, ref_iterator_advance() aborts the iteration,\n- * frees the ref_iterator, and returns ITER_ERROR.\n- *\n- * The reference currently being looked at can be peeled by calling\n- * ref_iterator_peel(). This function is often faster than peel_ref(),\n- * so it should be preferred when iterating over references.\n- *\n- * Putting it all together, a typical iteration looks like this:\n- *\n- * int ok;\n- * struct ref_iterator *iter = ...;\n- *\n- * while ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n- * if (want_to_stop_iteration()) {\n- * ok = ITER_DONE;\n- * break;\n- * }\n- *\n- * // Access information about the current reference:\n- * if (!(iter->flags & REF_ISSYMREF))\n- * printf(\"%s is %s\\n\", iter->refname, oid_to_hex(iter->oid));\n- *\n- * // If you need to peel the reference:\n- * ref_iterator_peel(iter, &oid);\n- * }\n- *\n- * if (ok != ITER_DONE)\n- * handle_error();\n- * ref_iterator_free(iter);\n+ * Data structure for holding a reference iterator. See refs.h for\n+ * more details and usage instructions.\n */\n struct ref_iterator {\n \tstruct ref_iterator_vtable *vtable;\n@@ -337,42 +255,6 @@ struct ref_iterator {\n \tunsigned int flags;\n };\n \n-/*\n- * Advance the iterator to the first or next item and return ITER_OK.\n- * If the iteration is exhausted, free the resources associated with\n- * the ref_iterator and return ITER_DONE. On errors, free the iterator\n- * resources and return ITER_ERROR. It is a bug to use ref_iterator or\n- * call this function again after it has returned ITER_DONE or\n- * ITER_ERROR.\n- */\n-int ref_iterator_advance(struct ref_iterator *ref_iterator);\n-\n-/*\n- * Seek the iterator to the first reference with the given prefix.\n- * The prefix is matched as a literal string, without regard for path\n- * separators. If prefix is NULL or the empty string, seek the iterator to the\n- * first reference again.\n- *\n- * This function is expected to behave as if a new ref iterator with the same\n- * prefix had been created, but allows reuse of iterators and thus may allow\n- * the backend to optimize. Parameters other than the prefix that have been\n- * passed when creating the iterator will remain unchanged.\n- *\n- * Returns 0 on success, a negative error code otherwise.\n- */\n-int ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t const char *prefix);\n-\n-/*\n- * If possible, peel the reference currently being viewed by the\n- * iterator. Return 0 on success.\n- */\n-int ref_iterator_peel(struct ref_iterator *ref_iterator,\n-\t\t struct object_id *peeled);\n-\n-/* Free the reference iterator and any associated resources. */\n-void ref_iterator_free(struct ref_iterator *ref_iterator);\n-\n /*\n * An iterator over nothing (its first ref_iterator_advance() call\n * returns ITER_DONE).\n@@ -384,17 +266,6 @@ struct ref_iterator *empty_ref_iterator_begin(void);\n */\n int is_empty_ref_iterator(struct ref_iterator *ref_iterator);\n \n-/*\n- * Return an iterator that goes over each reference in `refs` for\n- * which the refname begins with prefix. If trim is non-zero, then\n- * trim that many characters off the beginning of each refname.\n- * The output is ordered by refname.\n- */\n-struct ref_iterator *refs_ref_iterator_begin(\n-\t\tstruct ref_store *refs,\n-\t\tconst char *prefix, const char **exclude_patterns,\n-\t\tint trim, enum do_for_each_ref_flags flags);\n-\n /*\n * A callback function used to instruct merge_ref_iterator how to\n * interleave the entries from iter0 and iter1. The function should\n@@ -520,18 +391,6 @@ struct ref_iterator_vtable {\n */\n extern struct ref_iterator *current_ref_iter;\n \n-/*\n- * The common backend for the for_each_*ref* functions. Call fn for\n- * each reference in iter. If the iterator itself ever returns\n- * ITER_ERROR, return -1. If fn ever returns a non-zero value, stop\n- * the iteration and return that value. Otherwise, return 0. In any\n- * case, free the iterator when done. This function is basically an\n- * adapter between the callback style of reference iteration and the\n- * iterator style.\n- */\n-int do_for_each_ref_iterator(struct ref_iterator *iter,\n-\t\t\t each_ref_fn fn, void *cb_data);\n-\n struct ref_store;\n \n /* refs backends */\n\n-- \n2.49.0\n\n"},{"id":"521314","messageId":"20250704-306-git-for-each-ref-pagination-v2-2-bcde14acdd81@gmail.com","threadId":"63722","inReplyTo":"20250704-306-git-for-each-ref-pagination-v2-0-bcde14acdd81@gmail.com","subject":"[PATCH v2 2/4] ref-cache: remove unused function 'find_ref_entry()'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-04T13:02:30Z","receivedAt":"2025-07-04T13:02:48Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The 'find_ref_entry' function is no longer used, so remove it.\n\nSigned-off-by: Karthik Nayak \n---\n refs/ref-cache.c | 14 --------------\n refs/ref-cache.h | 7 -------\n 2 files changed, 21 deletions(-)\n\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex c1f1bab1d5..8aaffa8c6b 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -194,20 +194,6 @@ static struct ref_dir *find_containing_dir(struct ref_dir *dir,\n \treturn dir;\n }\n \n-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname)\n-{\n-\tint entry_index;\n-\tstruct ref_entry *entry;\n-\tdir = find_containing_dir(dir, refname);\n-\tif (!dir)\n-\t\treturn NULL;\n-\tentry_index = search_ref_dir(dir, refname, strlen(refname));\n-\tif (entry_index == -1)\n-\t\treturn NULL;\n-\tentry = dir->entries[entry_index];\n-\treturn (entry->flag & REF_DIR) ? NULL : entry;\n-}\n-\n /*\n * Emit a warning and return true iff ref1 and ref2 have the same name\n * and the same oid. Die if they have the same name but different\ndiff --git a/refs/ref-cache.h b/refs/ref-cache.h\nindex 5f04e518c3..f635d2d824 100644\n--- a/refs/ref-cache.h\n+++ b/refs/ref-cache.h\n@@ -201,13 +201,6 @@ void free_ref_cache(struct ref_cache *cache);\n */\n void add_entry_to_dir(struct ref_dir *dir, struct ref_entry *entry);\n \n-/*\n- * Find the value entry with the given name in dir, sorting ref_dirs\n- * and recursing into subdirectories as necessary. If the name is not\n- * found or it corresponds to a directory entry, return NULL.\n- */\n-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname);\n-\n /*\n * Start iterating over references in `cache`. If `prefix` is\n * specified, only include references whose names start with that\n\n-- \n2.49.0\n\n"},{"id":"521315","messageId":"20250704-306-git-for-each-ref-pagination-v2-3-bcde14acdd81@gmail.com","threadId":"63722","inReplyTo":"20250704-306-git-for-each-ref-pagination-v2-0-bcde14acdd81@gmail.com","subject":"[PATCH v2 3/4] refs: selectively set prefix in the seek functions","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-04T13:02:31Z","receivedAt":"2025-07-04T13:02:49Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The ref iterator exposes a `ref_iterator_seek()` function. The name\nsuggests that this would seek the iterator to a specific reference in\nsome ways similar to how `fseek()` works for the filesystem.\n\nHowever, the function actually sets the prefix for refs iteration. So\nfurther iteration would only yield references which match the particular\nprefix. This is a bit confusing.\n\nLet's add a 'flags' field to the function, which when set with the\n'REF_ITERATOR_SEEK_SET_PREFIX' flag, will set the prefix for the\niteration in-line with the existing behavior. Otherwise, the reference\nbackends will simply seek to the specified reference and clears any\npreviously set prefix. This allows users to start iteration from a\nspecific reference.\n\nIn the packed and reftable backend, since references are available in a\nsorted list, the changes are simply setting the prefix if needed. The\nchanges on the files-backend are a little more involved, since the files\nbackend uses the 'ref-cache' mechanism. We move out the existing logic\nwithin `cache_ref_iterator_seek()` to `cache_ref_iterator_set_prefix()`\nwhich is called when the 'REF_ITERATOR_SEEK_SET_PREFIX' flag is set. We\nthen parse the provided seek string and set the required levels and\ntheir indexes to ensure that seeking is possible.\n\nHelped-by: Patrick Steinhardt \nSigned-off-by: Karthik Nayak \n---\n refs.c | 6 ++--\n refs.h | 29 +++++++++++------\n refs/debug.c | 7 ++--\n refs/files-backend.c | 7 ++--\n refs/iterator.c | 26 ++++++++-------\n refs/packed-backend.c | 17 ++++++----\n refs/ref-cache.c | 85 ++++++++++++++++++++++++++++++++++++++++++++++---\n refs/refs-internal.h | 7 ++--\n refs/reftable-backend.c | 21 ++++++++----\n 9 files changed, 155 insertions(+), 50 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex dce5c49ca2..243e6898b8 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2666,12 +2666,12 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs\n \t\tif (!initial_transaction) {\n \t\t\tint ok;\n \n-\t\t\tif (!iter) {\n+\t\t\tif (!iter)\n \t\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n \t\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n-\t\t\t} else if (ref_iterator_seek(iter, dirname.buf) < 0) {\n+\t\t\telse if (ref_iterator_seek(iter, dirname.buf,\n+\t\t\t\t\t\t REF_ITERATOR_SEEK_SET_PREFIX) < 0)\n \t\t\t\tgoto cleanup;\n-\t\t\t}\n \n \t\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\t\t\tif (skip &&\ndiff --git a/refs.h b/refs.h\nindex 7c21aaef3d..7852ad36f3 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -1299,21 +1299,32 @@ struct ref_iterator *refs_ref_iterator_begin(\n */\n int ref_iterator_advance(struct ref_iterator *ref_iterator);\n \n+enum ref_iterator_seek_flag {\n+\t/*\n+\t * Also set the seek pattern as a prefix for iteration. This ensures\n+\t * that only references which match the prefix are yielded.\n+\t */\n+\tREF_ITERATOR_SEEK_SET_PREFIX = (1 << 0),\n+};\n+\n /*\n- * Seek the iterator to the first reference with the given prefix.\n- * The prefix is matched as a literal string, without regard for path\n- * separators. If prefix is NULL or the empty string, seek the iterator to the\n+ * Seek the iterator to the first reference matching the given seek string.\n+ * The seek string is matched as a literal string, without regard for path\n+ * separators. If seek is NULL or the empty string, seek the iterator to the\n * first reference again.\n *\n- * This function is expected to behave as if a new ref iterator with the same\n- * prefix had been created, but allows reuse of iterators and thus may allow\n- * the backend to optimize. Parameters other than the prefix that have been\n- * passed when creating the iterator will remain unchanged.\n+ * This function is expected to behave as if a new ref iterator has been\n+ * created, but allows reuse of existing iterators for optimization.\n+ *\n+ * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is\n+ * updated to match the seek string, affecting all subsequent iterations. If\n+ * not, the iterator seeks to the specified reference and clears any previously\n+ * set prefix.\n *\n * Returns 0 on success, a negative error code otherwise.\n */\n-int ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t const char *prefix);\n+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,\n+\t\t unsigned int flags);\n \n /*\n * If possible, peel the reference currently being viewed by the\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 485e3079d7..2ed8cff2aa 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -170,12 +170,13 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *seek, unsigned int flags)\n {\n \tstruct debug_ref_iterator *diter =\n \t\t(struct debug_ref_iterator *)ref_iterator;\n-\tint res = diter->iter->vtable->seek(diter->iter, prefix);\n-\ttrace_printf_key(&trace_refs, \"iterator_seek: %s: %d\\n\", prefix ? prefix : \"\", res);\n+\tint res = diter->iter->vtable->seek(diter->iter, seek, flags);\n+\ttrace_printf_key(&trace_refs, \"iterator_seek: %s flags: %d: %d\\n\",\n+\t\t\t seek ? seek : \"\", flags, res);\n \treturn res;\n }\n \ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex bf6f89b1d1..0e63013319 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -929,11 +929,11 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int files_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *seek, unsigned int flags)\n {\n \tstruct files_ref_iterator *iter =\n \t\t(struct files_ref_iterator *)ref_iterator;\n-\treturn ref_iterator_seek(iter->iter0, prefix);\n+\treturn ref_iterator_seek(iter->iter0, seek, flags);\n }\n \n static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n@@ -2316,7 +2316,8 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n-\t\t\t\t const char *prefix UNUSED)\n+\t\t\t\t const char *seek UNUSED,\n+\t\t\t\t unsigned int flags UNUSED)\n {\n \tBUG(\"ref_iterator_seek() called for reflog_iterator\");\n }\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex 766d96e795..f2364bd6e7 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -15,10 +15,10 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ref_iterator->vtable->advance(ref_iterator);\n }\n \n-int ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t const char *prefix)\n+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,\n+\t\t unsigned int flags)\n {\n-\treturn ref_iterator->vtable->seek(ref_iterator, prefix);\n+\treturn ref_iterator->vtable->seek(ref_iterator, seek, flags);\n }\n \n int ref_iterator_peel(struct ref_iterator *ref_iterator,\n@@ -57,7 +57,8 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)\n }\n \n static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n-\t\t\t\t const char *prefix UNUSED)\n+\t\t\t\t const char *seek UNUSED,\n+\t\t\t\t unsigned int flags UNUSED)\n {\n \treturn 0;\n }\n@@ -224,7 +225,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *seek, unsigned int flags)\n {\n \tstruct merge_ref_iterator *iter =\n \t\t(struct merge_ref_iterator *)ref_iterator;\n@@ -234,11 +235,11 @@ static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,\n \titer->iter0 = iter->iter0_owned;\n \titer->iter1 = iter->iter1_owned;\n \n-\tret = ref_iterator_seek(iter->iter0, prefix);\n+\tret = ref_iterator_seek(iter->iter0, seek, flags);\n \tif (ret < 0)\n \t\treturn ret;\n \n-\tret = ref_iterator_seek(iter->iter1, prefix);\n+\tret = ref_iterator_seek(iter->iter1, seek, flags);\n \tif (ret < 0)\n \t\treturn ret;\n \n@@ -407,13 +408,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *seek, unsigned int flags)\n {\n \tstruct prefix_ref_iterator *iter =\n \t\t(struct prefix_ref_iterator *)ref_iterator;\n-\tfree(iter->prefix);\n-\titer->prefix = xstrdup_or_null(prefix);\n-\treturn ref_iterator_seek(iter->iter0, prefix);\n+\n+\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX) {\n+\t\tfree(iter->prefix);\n+\t\titer->prefix = xstrdup_or_null(seek);\n+\t}\n+\treturn ref_iterator_seek(iter->iter0, seek, flags);\n }\n \n static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 7fd73a0e6d..11a363d246 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1004,19 +1004,23 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *seek, unsigned int flags)\n {\n \tstruct packed_ref_iterator *iter =\n \t\t(struct packed_ref_iterator *)ref_iterator;\n \tconst char *start;\n \n-\tif (prefix && *prefix)\n-\t\tstart = find_reference_location(iter->snapshot, prefix, 0);\n+\tif (seek && *seek)\n+\t\tstart = find_reference_location(iter->snapshot, seek, 0);\n \telse\n \t\tstart = iter->snapshot->start;\n \n-\tfree(iter->prefix);\n-\titer->prefix = xstrdup_or_null(prefix);\n+\t/* Unset any previously set prefix */\n+\tFREE_AND_NULL(iter->prefix);\n+\n+\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX)\n+\t\titer->prefix = xstrdup_or_null(seek);\n+\n \titer->pos = start;\n \titer->eof = iter->snapshot->eof;\n \n@@ -1194,7 +1198,8 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \titer->repo = ref_store->repo;\n \titer->flags = flags;\n \n-\tif (packed_ref_iterator_seek(&iter->base, prefix) < 0) {\n+\tif (packed_ref_iterator_seek(&iter->base, prefix,\n+\t\t\t\t REF_ITERATOR_SEEK_SET_PREFIX) < 0) {\n \t\tref_iterator_free(&iter->base);\n \t\treturn NULL;\n \t}\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 8aaffa8c6b..01dfbeb50c 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -434,11 +434,9 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t}\n }\n \n-static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+static int cache_ref_iterator_set_prefix(struct cache_ref_iterator *iter,\n+\t\t\t\t\t const char *prefix)\n {\n-\tstruct cache_ref_iterator *iter =\n-\t\t(struct cache_ref_iterator *)ref_iterator;\n \tstruct cache_ref_iterator_level *level;\n \tstruct ref_dir *dir;\n \n@@ -469,6 +467,82 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n \treturn 0;\n }\n \n+static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *seek, unsigned int flags)\n+{\n+\tstruct cache_ref_iterator *iter =\n+\t\t(struct cache_ref_iterator *)ref_iterator;\n+\n+\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX) {\n+\t\treturn cache_ref_iterator_set_prefix(iter, seek);\n+\t} else if (seek && *seek) {\n+\t\tstruct cache_ref_iterator_level *level;\n+\t\tconst char *slash = seek;\n+\t\tstruct ref_dir *dir;\n+\n+\t\tdir = get_ref_dir(iter->cache->root);\n+\n+\t\tif (iter->prime_dir)\n+\t\t\tprime_ref_dir(dir, seek);\n+\n+\t\titer->levels_nr = 1;\n+\t\tlevel = &iter->levels[0];\n+\t\tlevel->index = -1;\n+\t\tlevel->dir = dir;\n+\n+\t\t/* Unset any previously set prefix */\n+\t\tFREE_AND_NULL(iter->prefix);\n+\n+\t\t/*\n+\t\t * Breakdown the provided seek path and assign the correct\n+\t\t * indexing to each level as needed.\n+\t\t */\n+\t\tdo {\n+\t\t\tint len, idx;\n+\t\t\tint cmp = 0;\n+\n+\t\t\tsort_ref_dir(dir);\n+\n+\t\t\tslash = strchr(slash, '/');\n+\t\t\tlen = slash ? slash - seek : (int)strlen(seek);\n+\n+\t\t\tfor (idx = 0; idx < dir->nr; idx++) {\n+\t\t\t\tcmp = strncmp(seek, dir->entries[idx]->name, len);\n+\t\t\t\tif (cmp <= 0)\n+\t\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\t/* don't overflow the index */\n+\t\t\tidx = idx >= dir->nr ? dir->nr - 1 : idx;\n+\n+\t\t\tif (slash)\n+\t\t\t\tslash = slash + 1;\n+\n+\t\t\tlevel->index = idx;\n+\t\t\tif (dir->entries[idx]->flag & REF_DIR) {\n+\t\t\t\t/* push down a level */\n+\t\t\t\tdir = get_ref_dir(dir->entries[idx]);\n+\n+\t\t\t\tALLOC_GROW(iter->levels, iter->levels_nr + 1,\n+\t\t\t\t\t iter->levels_alloc);\n+\t\t\t\tlevel = &iter->levels[iter->levels_nr++];\n+\t\t\t\tlevel->dir = dir;\n+\t\t\t\tlevel->index = -1;\n+\t\t\t} else {\n+\t\t\t\t/* reduce the index so the leaf node is iterated over */\n+\t\t\t\tif (cmp <= 0 && !slash)\n+\t\t\t\t\tlevel->index = idx - 1;\n+\t\t\t\t/*\n+\t\t\t\t * while the seek path may not be exhausted, our\n+\t\t\t\t * match is exhausted at a leaf node.\n+\t\t\t\t */\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t} while (slash);\n+\t}\n+\n+\treturn 0;\n+}\n+\n static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -509,7 +583,8 @@ struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\n \titer->cache = cache;\n \titer->prime_dir = prime_dir;\n \n-\tif (cache_ref_iterator_seek(&iter->base, prefix) < 0) {\n+\tif (cache_ref_iterator_seek(&iter->base, prefix,\n+\t\t\t\t REF_ITERATOR_SEEK_SET_PREFIX) < 0) {\n \t\tref_iterator_free(&iter->base);\n \t\treturn NULL;\n \t}\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 03f5df04d5..6376a3b379 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -353,11 +353,12 @@ void base_ref_iterator_init(struct ref_iterator *iter,\n typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);\n \n /*\n- * Seek the iterator to the first reference matching the given prefix. Should\n- * behave the same as if a new iterator was created with the same prefix.\n+ * Seek the iterator to the first matching reference. If set_prefix is set,\n+ * it would behave the same as if a new iterator was created with the same\n+ * prefix.\n */\n typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix);\n+\t\t\t\t const char *seek, unsigned int flags);\n \n /*\n * Peels the current ref, returning 0 for success or -1 for failure.\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 4c3817f4ec..d627221b65 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -719,15 +719,20 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *seek, unsigned int flags)\n {\n \tstruct reftable_ref_iterator *iter =\n \t\t(struct reftable_ref_iterator *)ref_iterator;\n \n-\tfree(iter->prefix);\n-\titer->prefix = xstrdup_or_null(prefix);\n-\titer->prefix_len = prefix ? strlen(prefix) : 0;\n-\titer->err = reftable_iterator_seek_ref(&iter->iter, prefix);\n+\t/* Unset any previously set prefix */\n+\tFREE_AND_NULL(iter->prefix);\n+\titer->prefix_len = 0;\n+\n+\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX) {\n+\t\titer->prefix = xstrdup_or_null(seek);\n+\t\titer->prefix_len = seek ? strlen(seek) : 0;\n+\t}\n+\titer->err = reftable_iterator_seek_ref(&iter->iter, seek);\n \n \treturn iter->err;\n }\n@@ -839,7 +844,8 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_\n \tif (ret)\n \t\tgoto done;\n \n-\tret = reftable_ref_iterator_seek(&iter->base, prefix);\n+\tret = reftable_ref_iterator_seek(&iter->base, prefix,\n+\t\t\t\t\t REF_ITERATOR_SEEK_SET_PREFIX);\n \tif (ret)\n \t\tgoto done;\n \n@@ -2042,7 +2048,8 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n-\t\t\t\t\t const char *prefix UNUSED)\n+\t\t\t\t\t const char *seek UNUSED,\n+\t\t\t\t\t unsigned int flags UNUSED)\n {\n \tBUG(\"reftable reflog iterator cannot be seeked\");\n \treturn -1;\n\n-- \n2.49.0\n\n"},{"id":"521316","messageId":"20250704-306-git-for-each-ref-pagination-v2-4-bcde14acdd81@gmail.com","threadId":"63722","inReplyTo":"20250704-306-git-for-each-ref-pagination-v2-0-bcde14acdd81@gmail.com","subject":"[PATCH v2 4/4] for-each-ref: introduce a '--skip-until' option","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-04T13:02:32Z","receivedAt":"2025-07-04T13:02:50Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `git-for-each-ref(1)` command is used to iterate over references\npresent in a repository. In large repositories with millions of\nreferences, it would be optimal to paginate this output such that we\ncan start iteration from a given reference. This would avoid having to\niterate over all references from the beginning each time when paginating\nthrough results.\n\nThe previous commit added 'seek' functionality to the reference\nbackends. Utilize this and expose a '--skip-until' option in\n'git-for-each-ref(1)'. When used, the reference iteration seeks to the\nfirst matching reference and iterates from there onward.\n\nThis enables efficient pagination workflows like:\n git for-each-ref --count=100\n git for-each-ref --count=100 --skip-until=refs/heads/branch-100\n git for-each-ref --count=100 --skip-until=refs/heads/branch-200\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/git-for-each-ref.adoc | 6 +-\n builtin/for-each-ref.c | 8 ++\n ref-filter.c | 61 ++++++++----\n ref-filter.h | 1 +\n t/t6302-for-each-ref-filter.sh | 188 ++++++++++++++++++++++++++++++++++++\n 5 files changed, 243 insertions(+), 21 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.adoc b/Documentation/git-for-each-ref.adoc\nindex 5ef89fc0fe..e369fee9a1 100644\n--- a/Documentation/git-for-each-ref.adoc\n+++ b/Documentation/git-for-each-ref.adoc\n@@ -14,7 +14,7 @@ SYNOPSIS\n \t\t [--points-at=]\n \t\t [--merged[=]] [--no-merged[=]]\n \t\t [--contains[=]] [--no-contains[=]]\n-\t\t [--exclude= ...]\n+\t\t [--exclude= ...] [--skip-until=]\n \n DESCRIPTION\n -----------\n@@ -108,6 +108,10 @@ TAB %(refname)`.\n --include-root-refs::\n \tList root refs (HEAD and pseudorefs) apart from regular refs.\n \n+--skip-until::\n+ Skip references up to but excluding the specified pattern. Cannot be used\n+ with general pattern matching or custom sort options.\n+\n FIELD NAMES\n -----------\n \ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 3d2207ec77..aee2e7489a 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -13,6 +13,7 @@ static char const * const for_each_ref_usage[] = {\n \tN_(\"git for-each-ref [--points-at ]\"),\n \tN_(\"git for-each-ref [--merged []] [--no-merged []]\"),\n \tN_(\"git for-each-ref [--contains []] [--no-contains []]\"),\n+\tN_(\"git for-each-ref [--skip-until ]\"),\n \tNULL\n };\n \n@@ -44,6 +45,7 @@ int cmd_for_each_ref(int argc,\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_INTEGER( 0 , \"count\", &format.array_opts.max_count, N_(\"show only matched refs\")),\n \t\tOPT_STRING( 0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n+\t\tOPT_STRING( 0 , \"skip-until\", &filter.seek, N_(\"skip-until\"), N_(\"skip references until\")),\n \t\tOPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n \t\tOPT_REF_FILTER_EXCLUDE(&filter),\n \t\tOPT_REF_SORT(&sorting_options),\n@@ -79,6 +81,9 @@ int cmd_for_each_ref(int argc,\n \tif (verify_ref_format(&format))\n \t\tusage_with_options(for_each_ref_usage, opts);\n \n+\tif (filter.seek && sorting_options.nr > 1)\n+\t\tdie(_(\"cannot use --skip-until custom sort options\"));\n+\n \tsorting = ref_sorting_options(&sorting_options);\n \tref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);\n \tfilter.ignore_case = icase;\n@@ -100,6 +105,9 @@ int cmd_for_each_ref(int argc,\n \t\tfilter.name_patterns = argv;\n \t}\n \n+\tif (filter.seek && filter.name_patterns && filter.name_patterns[0])\n+\t\tdie(_(\"cannot use --skip-until with patterns\"));\n+\n \tif (include_root_refs)\n \t\tflags |= FILTER_REFS_ROOT_REFS | FILTER_REFS_DETACHED_HEAD;\n \ndiff --git a/ref-filter.c b/ref-filter.c\nindex 7a274633cf..56bb5312bd 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2692,10 +2692,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t\t\t each_ref_fn cb,\n \t\t\t\t void *cb_data)\n {\n+\tstruct ref_iterator *iter;\n+\tint flags = 0, ret = 0;\n+\n \tif (filter->kind & FILTER_REFS_ROOT_REFS) {\n \t\t/* In this case, we want to print all refs including root refs. */\n-\t\treturn refs_for_each_include_root_refs(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t cb, cb_data);\n+\t\tflags |= DO_FOR_EACH_INCLUDE_ROOT_REFS;\n+\t\tgoto non_prefix_iter;\n \t}\n \n \tif (!filter->match_as_path) {\n@@ -2704,8 +2707,7 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t * prefixes like \"refs/heads/\" etc. are stripped off,\n \t\t * so we have to look at everything:\n \t\t */\n-\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t\"\", NULL, cb, cb_data);\n+\t\tgoto non_prefix_iter;\n \t}\n \n \tif (filter->ignore_case) {\n@@ -2714,20 +2716,28 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t * so just return everything and let the caller\n \t\t * sort it out.\n \t\t */\n-\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t\"\", NULL, cb, cb_data);\n+\t\tgoto non_prefix_iter;\n \t}\n \n \tif (!filter->name_patterns[0]) {\n \t\t/* no patterns; we have to look at everything */\n-\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"\", filter->exclude.v, cb, cb_data);\n+\t\tgoto non_prefix_iter;\n \t}\n \n \treturn refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n \t\t\t\t\t\t NULL, filter->name_patterns,\n \t\t\t\t\t\t filter->exclude.v,\n \t\t\t\t\t\t cb, cb_data);\n+\n+non_prefix_iter:\n+\titer = refs_ref_iterator_begin(get_main_ref_store(the_repository), \"\",\n+\t\t\t\t NULL, 0, flags);\n+\tif (filter->seek)\n+\t\tret = ref_iterator_seek(iter, filter->seek, 0);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\treturn do_for_each_ref_iterator(iter, cb, cb_data);\n }\n \n /*\n@@ -3197,9 +3207,11 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n \tinit_contains_cache(&filter->internal.no_contains_cache);\n \n \t/* Simple per-ref filtering */\n-\tif (!filter->kind)\n+\tif (!filter->kind) {\n \t\tdie(\"filter_refs: invalid type\");\n-\telse {\n+\t} else {\n+\t\tconst char *prefix = NULL;\n+\n \t\t/*\n \t\t * For common cases where we need only branches or remotes or tags,\n \t\t * we only iterate through those refs. If a mix of refs is needed,\n@@ -3207,19 +3219,28 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n \t\t * of filter_ref_kind().\n \t\t */\n \t\tif (filter->kind == FILTER_REFS_BRANCHES)\n-\t\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"refs/heads/\", NULL,\n-\t\t\t\t\t\t fn, cb_data);\n+\t\t\tprefix = \"refs/heads/\";\n \t\telse if (filter->kind == FILTER_REFS_REMOTES)\n-\t\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"refs/remotes/\", NULL,\n-\t\t\t\t\t\t fn, cb_data);\n+\t\t\tprefix = \"refs/remotes/\";\n \t\telse if (filter->kind == FILTER_REFS_TAGS)\n-\t\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"refs/tags/\", NULL, fn,\n-\t\t\t\t\t\t cb_data);\n-\t\telse if (filter->kind & FILTER_REFS_REGULAR)\n+\t\t\tprefix = \"refs/tags/\";\n+\n+\t\tif (prefix) {\n+\t\t\tstruct ref_iterator *iter;\n+\n+\t\t\titer = refs_ref_iterator_begin(get_main_ref_store(the_repository),\n+\t\t\t\t\t\t \"\", NULL, 0, 0);\n+\n+\t\t\tif (filter->seek)\n+\t\t\t\tret = ref_iterator_seek(iter, filter->seek, 0);\n+\t\t\telse if (prefix)\n+\t\t\t\tret = ref_iterator_seek(iter, prefix, 1);\n+\n+\t\t\tif (!ret)\n+\t\t\t\tret = do_for_each_ref_iterator(iter, fn, cb_data);\n+\t\t} else if (filter->kind & FILTER_REFS_REGULAR) {\n \t\t\tret = for_each_fullref_in_pattern(filter, fn, cb_data);\n+\t\t}\n \n \t\t/*\n \t\t * When printing all ref types, HEAD is already included,\ndiff --git a/ref-filter.h b/ref-filter.h\nindex c98c4fbd4c..9e97c65bc2 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -64,6 +64,7 @@ struct ref_array {\n \n struct ref_filter {\n \tconst char **name_patterns;\n+\tconst char *seek;\n \tstruct strvec exclude;\n \tstruct oid_array points_at;\n \tstruct commit_list *with_commit;\ndiff --git a/t/t6302-for-each-ref-filter.sh b/t/t6302-for-each-ref-filter.sh\nindex bb02b86c16..3f1823e95b 100755\n--- a/t/t6302-for-each-ref-filter.sh\n+++ b/t/t6302-for-each-ref-filter.sh\n@@ -541,4 +541,192 @@ test_expect_success 'validate worktree atom' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'skip until with empty value' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/main\n+\trefs/heads/main_worktree\n+\trefs/heads/side\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --skip-until=\"\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'skip until to a specific reference' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/spot >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'skip until to a specific reference with partial match' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/sp >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'skip until just behind a specific reference' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/parrot >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'skip until to specific directory' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/odd >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'skip until to specific directory with trailing slash' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/lost >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'skip until just behind a specific directory' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'skip until overflow specific reference length' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/spotnew >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'skip until overflow specific reference path' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/spot/new >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'skip until used with a pattern' '\n+\tcat >expect <<-\\EOF &&\n+\tfatal: cannot use --skip-until with patterns\n+\tEOF\n+\ttest_must_fail git for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/spot refs/tags 2>actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'skip until used with custom sort order' '\n+\tcat >expect <<-\\EOF &&\n+\tfatal: cannot use --skip-until custom sort options\n+\tEOF\n+\ttest_must_fail git for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/spot --sort=author 2>actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n\n-- \n2.49.0\n\n"},{"id":"521319","messageId":"87ms9kcbtq.fsf@igel.home","threadId":"63722","inReplyTo":"20250704-306-git-for-each-ref-pagination-v2-0-bcde14acdd81@gmail.com","subject":"Re: [PATCH v2 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2025-07-04T13:41:53Z","receivedAt":"2025-07-04T13:51:44Z","isPatch":true,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"On Jul 04 2025, Karthik Nayak wrote:\n\n> This series adds a '--skip-until' option in 'git-for-each-ref(1)'. When\n> used, the reference iteration seeks to the first matching reference and\n> iterates from there onward.\n\nI would have named the option --start-with. It has the advantage that\nit is clear whether the matched ref is included.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1\n\"And now for something completely different.\"\n"},{"id":"521321","messageId":"CAOLa=ZS0uP+5xso_SEG2GJZHeac-0F2_wMJKtvbFj_wROKbBkw@mail.gmail.com","threadId":"63722","inReplyTo":"87ms9kcbtq.fsf@igel.home","subject":"Re: [PATCH v2 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-04T14:02:17Z","receivedAt":"2025-07-04T14:02:18Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Andreas Schwab writes:\n\n> On Jul 04 2025, Karthik Nayak wrote:\n>\n>> This series adds a '--skip-until' option in 'git-for-each-ref(1)'. When\n>> used, the reference iteration seeks to the first matching reference and\n>> iterates from there onward.\n>\n> I would have named the option --start-with. It has the advantage that\n> it is clear whether the matched ref is included.\n>\n\nWe did discuss this internally, some other names we thought of:\n--skip-to\n--start-after\n--start-from\n--seek\n--skip-before\n--start-at\n\nI think I was a bit against '--start-from' and '--start-at', because\nthey imply that the reference provided must exist.\n\nConsider the example\n\n $ git for-each-ref\n refs/heads/bar\n refs/heads/foo\n refs/heads/main\n\n $ git for-each-ref --seek=refs/heads/cat\n refs/heads/foo\n refs/heads/main\n\nYou can see that the reference doesn't have to exist. So implying that\nit should can be a bit confusing.\n\nBut I'm open to changing this, if we can conclude on any flag name...\n\n> --\n> Andreas Schwab, schwab@linux-m68k.org\n> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1\n> \"And now for something completely different.\"\n"},{"id":"521326","messageId":"87ikk8c8jr.fsf@igel.home","threadId":"63722","inReplyTo":"CAOLa=ZS0uP+5xso_SEG2GJZHeac-0F2_wMJKtvbFj_wROKbBkw@mail.gmail.com","subject":"Re: [PATCH v2 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2025-07-04T14:52:40Z","receivedAt":"2025-07-04T14:52:53Z","isPatch":true,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"On Jul 04 2025, Karthik Nayak wrote:\n\n> Consider the example\n>\n> $ git for-each-ref\n> refs/heads/bar\n> refs/heads/foo\n> refs/heads/main\n>\n> $ git for-each-ref --seek=refs/heads/cat\n> refs/heads/foo\n> refs/heads/main\n>\n> You can see that the reference doesn't have to exist.\n\nThat is even more confusing. What is the first matching ref if none of\nthem match? Doesn't that mean skipping _all_ refs?\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1\n\"And now for something completely different.\"\n"},{"id":"521328","messageId":"CAOLa=ZR2=5iZzUVTS1o81a5NhLLiyHLiJfznz2Us5q0VLU74og@mail.gmail.com","threadId":"63722","inReplyTo":"87ikk8c8jr.fsf@igel.home","subject":"Re: [PATCH v2 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-04T14:58:14Z","receivedAt":"2025-07-04T14:58:16Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Andreas Schwab writes:\n\n> On Jul 04 2025, Karthik Nayak wrote:\n>\n>> Consider the example\n>>\n>> $ git for-each-ref\n>> refs/heads/bar\n>> refs/heads/foo\n>> refs/heads/main\n>>\n>> $ git for-each-ref --seek=refs/heads/cat\n>> refs/heads/foo\n>> refs/heads/main\n>>\n>> You can see that the reference doesn't have to exist.\n>\n> That is even more confusing. What is the first matching ref if none of\n> them match? Doesn't that mean skipping _all_ refs?\n>\n\nWell the idea is it would seek to the offset where the reference would\nfit in.\n\nThis is to ensure that seeks to references which were deleted\nconcurrently doesn't leave the client hanging with no results while\npaginating over all references.\n\n> --\n> Andreas Schwab, schwab@linux-m68k.org\n> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1\n> \"And now for something completely different.\"\n"},{"id":"521330","messageId":"87ecuwc5n4.fsf@igel.home","threadId":"63722","inReplyTo":"CAOLa=ZR2=5iZzUVTS1o81a5NhLLiyHLiJfznz2Us5q0VLU74og@mail.gmail.com","subject":"Re: [PATCH v2 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2025-07-04T15:55:27Z","receivedAt":"2025-07-04T16:03:08Z","isPatch":true,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"On Jul 04 2025, Karthik Nayak wrote:\n\n> Andreas Schwab writes:\n>\n>> On Jul 04 2025, Karthik Nayak wrote:\n>>\n>>> Consider the example\n>>>\n>>> $ git for-each-ref\n>>> refs/heads/bar\n>>> refs/heads/foo\n>>> refs/heads/main\n>>>\n>>> $ git for-each-ref --seek=refs/heads/cat\n>>> refs/heads/foo\n>>> refs/heads/main\n>>>\n>>> You can see that the reference doesn't have to exist.\n>>\n>> That is even more confusing. What is the first matching ref if none of\n>> them match? Doesn't that mean skipping _all_ refs?\n>>\n>\n> Well the idea is it would seek to the offset where the reference would\n> fit in.\n>\n> This is to ensure that seeks to references which were deleted\n> concurrently doesn't leave the client hanging with no results while\n> paginating over all references.\n\nThen don't call it a pattern. Pattern matching is a set operation,\nindependent of sorting. What you really have is a marker that divides\nthe sorted list in two parts according to how the marker sorts. And\nthat makes --start-with more descriptive and less ambiguous.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1\n\"And now for something completely different.\"\n"},{"id":"521331","messageId":"xmqqa55jc3md.fsf@gitster.g","threadId":"63722","inReplyTo":"CAOLa=ZS0uP+5xso_SEG2GJZHeac-0F2_wMJKtvbFj_wROKbBkw@mail.gmail.com","subject":"Re: [PATCH v2 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-04T16:39:06Z","receivedAt":"2025-07-04T16:39:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> I think I was a bit against '--start-from' and '--start-at', because\n> they imply that the reference provided must exist.\n\nIt also implies that if the reference does exist, that would be the\nfirst one that is shown. But I do not think you want that, as ...\n\n> Consider the example\n>\n> $ git for-each-ref\n> refs/heads/bar\n> refs/heads/foo\n> refs/heads/main\n\n\n... after a paging application starts from the beginning and showed\na single page of some items, it knows the \"last\" one it showed.\nThat last entry may have been refs/heads/bar. The application may\nnot have seen the next entry (i.e. refs/heads/foo). So if it has to\nuse '--start-at=refs/heads/bar', the first entry it gets from such a\nrequest may be for refs/heads/bar again. The application needs to\nremember the \"last\" one it showed and skip that, which is a bit\nawkward, isn't it?\n\n> $ git for-each-ref --seek=refs/heads/cat\n> refs/heads/foo\n> refs/heads/main\n>\n> You can see that the reference doesn't have to exist. So implying that\n> it should can be a bit confusing.\n\nFor that reason, whatever verb you pick from seek or start or skip,\nit would be great if the option name also made it explicit that the\nnamed one, if exists, is not shown. Conceptually, it is \"skip\neverything that sorts before the named item, including the named\nitem itself\" that such a paging application would want, wouldn't it?\n\nThanks.\n\n"},{"id":"521417","messageId":"CAOLa=ZRSFH7F=cU2owPjhVrupSSubEArak6abmhRcP97wStVyA@mail.gmail.com","threadId":"63722","inReplyTo":"87ecuwc5n4.fsf@igel.home","subject":"Re: [PATCH v2 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-07T08:52:24Z","receivedAt":"2025-07-07T08:52:27Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Andreas Schwab writes:\n\n> On Jul 04 2025, Karthik Nayak wrote:\n>\n>> Andreas Schwab writes:\n>>\n>>> On Jul 04 2025, Karthik Nayak wrote:\n>>>\n>>>> Consider the example\n>>>>\n>>>> $ git for-each-ref\n>>>> refs/heads/bar\n>>>> refs/heads/foo\n>>>> refs/heads/main\n>>>>\n>>>> $ git for-each-ref --seek=refs/heads/cat\n>>>> refs/heads/foo\n>>>> refs/heads/main\n>>>>\n>>>> You can see that the reference doesn't have to exist.\n>>>\n>>> That is even more confusing. What is the first matching ref if none of\n>>> them match? Doesn't that mean skipping _all_ refs?\n>>>\n>>\n>> Well the idea is it would seek to the offset where the reference would\n>> fit in.\n>>\n>> This is to ensure that seeks to references which were deleted\n>> concurrently doesn't leave the client hanging with no results while\n>> paginating over all references.\n>\n> Then don't call it a pattern. Pattern matching is a set operation,\n> independent of sorting. What you really have is a marker that divides\n> the sorted list in two parts according to how the marker sorts. And\n> that makes --start-with more descriptive and less ambiguous.\n>\n\nFair enough. I'll change the documentation and description in the next\nversion.\n\n> --\n> Andreas Schwab, schwab@linux-m68k.org\n> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1\n> \"And now for something completely different.\"\n"},{"id":"521418","messageId":"CAOLa=ZTDcssjQcNcvDOA4-r-j2asp-XHCy2D_qoHEidz+KsKCQ@mail.gmail.com","threadId":"63722","inReplyTo":"xmqqa55jc3md.fsf@gitster.g","subject":"Re: [PATCH v2 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-07T08:59:06Z","receivedAt":"2025-07-07T08:59:08Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Karthik Nayak writes:\n>\n>> I think I was a bit against '--start-from' and '--start-at', because\n>> they imply that the reference provided must exist.\n>\n> It also implies that if the reference does exist, that would be the\n> first one that is shown. But I do not think you want that, as ...\n>\n>> Consider the example\n>>\n>> $ git for-each-ref\n>> refs/heads/bar\n>> refs/heads/foo\n>> refs/heads/main\n>\n>\n> ... after a paging application starts from the beginning and showed\n> a single page of some items, it knows the \"last\" one it showed.\n> That last entry may have been refs/heads/bar. The application may\n> not have seen the next entry (i.e. refs/heads/foo). So if it has to\n> use '--start-at=refs/heads/bar', the first entry it gets from such a\n> request may be for refs/heads/bar again. The application needs to\n> remember the \"last\" one it showed and skip that, which is a bit\n> awkward, isn't it?\n>\n\nI do agree, I was modelling this after what would be the best approach\nwithin the Git codebase. But I think it would be nicer for the clients\nif we skip the provided reference.\n\n>> $ git for-each-ref --seek=refs/heads/cat\n>> refs/heads/foo\n>> refs/heads/main\n>>\n>> You can see that the reference doesn't have to exist. So implying that\n>> it should can be a bit confusing.\n>\n> For that reason, whatever verb you pick from seek or start or skip,\n> it would be great if the option name also made it explicit that the\n> named one, if exists, is not shown. Conceptually, it is \"skip\n> everything that sorts before the named item, including the named\n> item itself\" that such a paging application would want, wouldn't it?\n>\n> Thanks.\n>\n\nWith that I think '--start-after' sounds like the best option. I'll\nmodify for the next version accordingly.\n\nThanks!\n"},{"id":"521423","messageId":"3d95d37c-7352-4e57-8d50-6fd4fcb77095@gmail.com","threadId":"63722","inReplyTo":"CAOLa=ZTDcssjQcNcvDOA4-r-j2asp-XHCy2D_qoHEidz+KsKCQ@mail.gmail.com","subject":"Re: [PATCH v2 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2025-07-07T09:45:45Z","receivedAt":"2025-07-07T09:45:47Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 07/07/2025 09:59, Karthik Nayak wrote:\n> Junio C Hamano writes:\n> \n>> Karthik Nayak writes:\n> I do agree, I was modelling this after what would be the best approach\n> within the Git codebase. \n\nThat was my fear when I asked about using a numeric offset. Patrick has \nmade a principled argument for using a ref name rather than a numeric \nofsset - I think you should build the motivation for this series around \nthat and the documentation should explain the implications of references \nbeing added and deleted while paging them.\n\n> With that I think '--start-after' sounds like the best option. I'll\n> modify for the next version accordingly.\nThat sounds like a good name\n\nThanks\n\nPhillip\n\n"},{"id":"521445","messageId":"xmqq5xg4588e.fsf@gitster.g","threadId":"63722","inReplyTo":"20250704-306-git-for-each-ref-pagination-v2-4-bcde14acdd81@gmail.com","subject":"Re: [PATCH v2 4/4] for-each-ref: introduce a '--skip-until' option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-07T15:30:25Z","receivedAt":"2025-07-07T15:30:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> +\tif (filter.seek && sorting_options.nr > 1)\n> +\t\tdie(_(\"cannot use --skip-until custom sort options\"));\n\nMissing \"with\" before \"custom sort\".\n\nWhen I commented on the previous iteration about sorting, I didn't\nmean to suggest making them incompatible---it may have some use case\nto grab a batch out of the underlying refstore, sort refs in that\nbatch, and then show them. But from usability's point of view, I\ntend to agree with this design decision. Such an unnatural batching\nand sorting operation is probably not worth supporting.\n"},{"id":"521459","messageId":"CAOLa=ZSSQJxm04x1RArP2VxbbxE5ySGiYZ+p-3PaUOu9hXUt0w@mail.gmail.com","threadId":"63722","inReplyTo":"xmqq5xg4588e.fsf@gitster.g","subject":"Re: [PATCH v2 4/4] for-each-ref: introduce a '--skip-until' option","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-07T18:31:06Z","receivedAt":"2025-07-07T18:31:08Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Karthik Nayak writes:\n>\n>> +\tif (filter.seek && sorting_options.nr > 1)\n>> +\t\tdie(_(\"cannot use --skip-until custom sort options\"));\n>\n> Missing \"with\" before \"custom sort\".\n>\n\nThanks, will fix in the next version.\n\n> When I commented on the previous iteration about sorting, I didn't\n> mean to suggest making them incompatible---it may have some use case\n> to grab a batch out of the underlying refstore, sort refs in that\n> batch, and then show them. But from usability's point of view, I\n> tend to agree with this design decision. Such an unnatural batching\n> and sorting operation is probably not worth supporting.\n\nI think so too, there might be some usecase, when that usecase is more\nconcrete we can probably re-visit this.\n\nFor now, it is confusing and it is just easier to not support them\ntogether.\n"},{"id":"521535","messageId":"CAOLa=ZRodyDB5_ZESeDe43Et29dXWNV2+RcNJ14D8BAp7WtJ_g@mail.gmail.com","threadId":"63722","inReplyTo":"3d95d37c-7352-4e57-8d50-6fd4fcb77095@gmail.com","subject":"Re: [PATCH v2 0/4] for-each-ref: introduce seeking functionality via '--skip-until'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-08T11:39:36Z","receivedAt":"2025-07-08T11:39:38Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Phillip Wood writes:\n\n> On 07/07/2025 09:59, Karthik Nayak wrote:\n>> Junio C Hamano writes:\n>>\n>>> Karthik Nayak writes:\n>> I do agree, I was modelling this after what would be the best approach\n>> within the Git codebase.\n>\n> That was my fear when I asked about using a numeric offset. Patrick has\n> made a principled argument for using a ref name rather than a numeric\n> ofsset - I think you should build the motivation for this series around\n> that and the documentation should explain the implications of references\n> being added and deleted while paging them.\n>\n\nYeah, I'll add something on those lines in the next version. I do\nappreciate these checks/questions.\n\n>> With that I think '--start-after' sounds like the best option. I'll\n>> modify for the next version accordingly.\n> That sounds like a good name\n>\n\nThanks!\n\n> Thanks\n>\n> Phillip\n"},{"id":"521540","messageId":"20250708-306-git-for-each-ref-pagination-v3-0-8cfba1080be4@gmail.com","threadId":"63722","inReplyTo":"20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com","subject":"[PATCH v3 0/4] for-each-ref: introduce seeking functionality via '--start-after'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-08T13:47:45Z","receivedAt":"2025-07-08T13:48:03Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `git-for-each-ref(1)` command is used to iterate over references\npresent in a repository. In large repositories with millions of\nreferences, it would be optimal to paginate this output such that we\ncan start iteration from a given reference. This would avoid having to\niterate over all references from the beginning each time when paginating\nthrough results.\n\nThis series adds a '--start-after' option in 'git-for-each-ref(1)'. When\nused, the reference iteration seeks to first reference following the\nmarker alphabetically. When paging, it should be noted that references\nmay be deleted, modified or added between invocations. Output will only\nyield those references which follow the marker lexicographically. If the\nmarker does not exist, output begins from the first reference that would\ncome after it alphabetically.\n\nThis enables efficient pagination workflows like:\n git for-each-ref --count=100\n git for-each-ref --count=100 --start-after=refs/heads/branch-100\n git for-each-ref --count=100 --start-after=refs/heads/branch-200\n\nTo add this functionality, we expose the `ref_iterator` outside the\n'refs/' namespace and modify the `ref_iterator_seek()` to actually seek\nto a given reference and only set the prefix when the `set_prefix` field\nis set.\n\nOn the reftable and packed backend, the changes are simple. But since\nthe files backend uses 'ref-cache' for reference handling, the changes\nthere are a little more involved, since we need to setup the right\nlevels and the indexing.\n\nInitially I was also planning to cleanup all the `refs_for_each...()`\nfunctions in 'refs.h' by simply using the iterator, but this bloated the\nseries. So I've left that for another day.\n\nChanges in v3:\n- Change the working of the command to exclude the marker provided. With\n this rename the flag to '--start-after'.\n- Extend the documentation to add a note about concurrent modifications\n to the reference database.\n- Link to v2: https://lore.kernel.org/r/20250704-306-git-for-each-ref-pagination-v2-0-bcde14acdd81@gmail.com\n\nChanges in v2:\n- Modify 'ref_iterator_seek()' to take in flags instead of a\n 'set_prefix' variable. This improves readability, where users would\n use the 'REF_ITERATOR_SEEK_SET_PREFIX' instead of simply passing '1'.\n- When the set prefix flag isn't usage, reset any previously set prefix.\n This ensures that the internal prefix state is always reset whenever\n we seek and unifies the behavior between 'ref_iterator_seek' and\n 'ref_iterator_begin'.\n- Don't allow '--skip-until' to be run with '--sort', since the seeking\n always takes place before any sorting and this can be confusing.\n- Some styling fixes:\n - Remove extra newline\n - Skip braces around single lined if...else clause\n - Add braces around 'if' clause\n - Fix indentation\n- Link to v1: https://lore.kernel.org/git/20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com/\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/git-for-each-ref.adoc | 11 +-\n builtin/for-each-ref.c | 8 ++\n ref-filter.c | 80 +++++++++++----\n ref-filter.h | 1 +\n refs.c | 6 +-\n refs.h | 158 +++++++++++++++++++++++++++++\n refs/debug.c | 7 +-\n refs/files-backend.c | 7 +-\n refs/iterator.c | 26 +++--\n refs/packed-backend.c | 17 ++--\n refs/ref-cache.c | 99 ++++++++++++++----\n refs/ref-cache.h | 7 --\n refs/refs-internal.h | 152 ++--------------------------\n refs/reftable-backend.c | 21 ++--\n t/t6302-for-each-ref-filter.sh | 194 ++++++++++++++++++++++++++++++++++++\n 15 files changed, 568 insertions(+), 226 deletions(-)\n\nKarthik Nayak (4):\n refs: expose `ref_iterator` via 'refs.h'\n ref-cache: remove unused function 'find_ref_entry()'\n refs: selectively set prefix in the seek functions\n for-each-ref: introduce a '--start-after' option\n\nRange-diff versus v2:\n\n1: c0ce873c35 = 1: dbb03c2aa9 refs: expose `ref_iterator` via 'refs.h'\n2: 2c50d1eba2 = 2: fa5a0cb722 ref-cache: remove unused function 'find_ref_entry()'\n3: fae849749f = 3: 9940d390cc refs: selectively set prefix in the seek functions\n4: a0725a6647 ! 4: ebe864095a for-each-ref: introduce a '--skip-until' option\n @@ Metadata\n Author: Karthik Nayak \n \n ## Commit message ##\n - for-each-ref: introduce a '--skip-until' option\n + for-each-ref: introduce a '--start-after' option\n \n The `git-for-each-ref(1)` command is used to iterate over references\n present in a repository. In large repositories with millions of\n @@ Commit message\n through results.\n \n The previous commit added 'seek' functionality to the reference\n - backends. Utilize this and expose a '--skip-until' option in\n + backends. Utilize this and expose a '--start-after' option in\n 'git-for-each-ref(1)'. When used, the reference iteration seeks to the\n - first matching reference and iterates from there onward.\n + lexicographically next reference and iterates from there onward.\n \n This enables efficient pagination workflows like:\n git for-each-ref --count=100\n - git for-each-ref --count=100 --skip-until=refs/heads/branch-100\n - git for-each-ref --count=100 --skip-until=refs/heads/branch-200\n + git for-each-ref --count=100 --start-after=refs/heads/branch-100\n + git for-each-ref --count=100 --start-after=refs/heads/branch-200\n +\n + Since the reference iterators only allow seeking to a specified marker\n + via the `ref_iterator_seek()`, we introduce a helper function\n + `start_ref_iterator_after()`, which seeks to next reference by simply\n + adding (char) 1 to the marker.\n +\n + We must note that pagination always continues from the provided marker,\n + as such any concurrent reference updates lexicographically behind the\n + marker will not be output. Document the same.\n \n Signed-off-by: Karthik Nayak \n \n @@ Documentation/git-for-each-ref.adoc: SYNOPSIS\n \t\t [--merged[=]] [--no-merged[=]]\n \t\t [--contains[=]] [--no-contains[=]]\n -\t\t [--exclude= ...]\n -+\t\t [--exclude= ...] [--skip-until=]\n ++\t\t [--exclude= ...] [--start-after=]\n \n DESCRIPTION\n -----------\n @@ Documentation/git-for-each-ref.adoc: TAB %(refname)`.\n --include-root-refs::\n \tList root refs (HEAD and pseudorefs) apart from regular refs.\n \n -+--skip-until::\n -+ Skip references up to but excluding the specified pattern. Cannot be used\n -+ with general pattern matching or custom sort options.\n ++--start-after::\n ++ Allows paginating the output by skipping references up to and including the\n ++ specified marker. When paging, it should be noted that references may be\n ++ deleted, modified or added between invocations. Output will only yield those\n ++ references which follow the marker lexicographically. If the marker does not\n ++ exist, output begins from the first reference that would come after it\n ++ alphabetically. Cannot be used with general pattern matching or custom\n ++ sort options.\n +\n FIELD NAMES\n -----------\n @@ builtin/for-each-ref.c: static char const * const for_each_ref_usage[] = {\n \tN_(\"git for-each-ref [--points-at ]\"),\n \tN_(\"git for-each-ref [--merged []] [--no-merged []]\"),\n \tN_(\"git for-each-ref [--contains []] [--no-contains []]\"),\n -+\tN_(\"git for-each-ref [--skip-until ]\"),\n ++\tN_(\"git for-each-ref [--start-after ]\"),\n \tNULL\n };\n \n @@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc,\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_INTEGER( 0 , \"count\", &format.array_opts.max_count, N_(\"show only matched refs\")),\n \t\tOPT_STRING( 0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n -+\t\tOPT_STRING( 0 , \"skip-until\", &filter.seek, N_(\"skip-until\"), N_(\"skip references until\")),\n ++\t\tOPT_STRING( 0 , \"start-after\", &filter.start_after, N_(\"start-start\"), N_(\"start iteration after the provided marker\")),\n \t\tOPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n \t\tOPT_REF_FILTER_EXCLUDE(&filter),\n \t\tOPT_REF_SORT(&sorting_options),\n @@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc,\n \tif (verify_ref_format(&format))\n \t\tusage_with_options(for_each_ref_usage, opts);\n \n -+\tif (filter.seek && sorting_options.nr > 1)\n -+\t\tdie(_(\"cannot use --skip-until custom sort options\"));\n ++\tif (filter.start_after && sorting_options.nr > 1)\n ++\t\tdie(_(\"cannot use --start-after with custom sort options\"));\n +\n \tsorting = ref_sorting_options(&sorting_options);\n \tref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);\n @@ builtin/for-each-ref.c: int cmd_for_each_ref(int argc,\n \t\tfilter.name_patterns = argv;\n \t}\n \n -+\tif (filter.seek && filter.name_patterns && filter.name_patterns[0])\n -+\t\tdie(_(\"cannot use --skip-until with patterns\"));\n ++\tif (filter.start_after && filter.name_patterns && filter.name_patterns[0])\n ++\t\tdie(_(\"cannot use --start-after with patterns\"));\n +\n \tif (include_root_refs)\n \t\tflags |= FILTER_REFS_ROOT_REFS | FILTER_REFS_DETACHED_HEAD;\n \n \n ## ref-filter.c ##\n +@@ ref-filter.c: static int filter_exclude_match(struct ref_filter *filter, const char *refname)\n + \treturn match_pattern(filter->exclude.v, refname, filter->ignore_case);\n + }\n + \n ++/*\n ++ * We need to seek to the reference right after a given marker but excluding any\n ++ * matching references. So we seek to the lexicographically next reference.\n ++ */\n ++static int start_ref_iterator_after(struct ref_iterator *iter, const char *marker)\n ++{\n ++\tstruct strbuf sb = STRBUF_INIT;\n ++\tint ret;\n ++\n ++\tstrbuf_addstr(&sb, marker);\n ++\tstrbuf_addch(&sb, 1);\n ++\n ++\tret = ref_iterator_seek(iter, sb.buf, 0);\n ++\n ++\tstrbuf_release(&sb);\n ++\treturn ret;\n ++}\n ++\n + /*\n + * This is the same as for_each_fullref_in(), but it tries to iterate\n + * only over the patterns we'll care about. Note that it _doesn't_ do a full\n @@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t\t\t each_ref_fn cb,\n \t\t\t\t void *cb_data)\n @@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,\n +non_prefix_iter:\n +\titer = refs_ref_iterator_begin(get_main_ref_store(the_repository), \"\",\n +\t\t\t\t NULL, 0, flags);\n -+\tif (filter->seek)\n -+\t\tret = ref_iterator_seek(iter, filter->seek, 0);\n ++\tif (filter->start_after)\n ++\t\tret = start_ref_iterator_after(iter, filter->start_after);\n ++\n +\tif (ret)\n +\t\treturn ret;\n +\n @@ ref-filter.c: static int do_filter_refs(struct ref_filter *filter, unsigned int\n +\t\t\titer = refs_ref_iterator_begin(get_main_ref_store(the_repository),\n +\t\t\t\t\t\t \"\", NULL, 0, 0);\n +\n -+\t\t\tif (filter->seek)\n -+\t\t\t\tret = ref_iterator_seek(iter, filter->seek, 0);\n ++\t\t\tif (filter->start_after)\n ++\t\t\t\tret = start_ref_iterator_after(iter, filter->start_after);\n +\t\t\telse if (prefix)\n +\t\t\t\tret = ref_iterator_seek(iter, prefix, 1);\n +\n @@ ref-filter.h: struct ref_array {\n \n struct ref_filter {\n \tconst char **name_patterns;\n -+\tconst char *seek;\n ++\tconst char *start_after;\n \tstruct strvec exclude;\n \tstruct oid_array points_at;\n \tstruct commit_list *with_commit;\n @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '\n \ttest_cmp expect actual\n '\n \n -+test_expect_success 'skip until with empty value' '\n ++test_expect_success 'start after with empty value' '\n +\tcat >expect <<-\\EOF &&\n +\trefs/heads/main\n +\trefs/heads/main_worktree\n @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '\n +\trefs/tags/three\n +\trefs/tags/two\n +\tEOF\n -+\tgit for-each-ref --format=\"%(refname)\" --skip-until=\"\" >actual &&\n ++\tgit for-each-ref --format=\"%(refname)\" --start-after=\"\" >actual &&\n +\ttest_cmp expect actual\n +'\n +\n -+test_expect_success 'skip until to a specific reference' '\n ++test_expect_success 'start after a specific reference' '\n +\tcat >expect <<-\\EOF &&\n -+\trefs/odd/spot\n +\trefs/tags/annotated-tag\n +\trefs/tags/doubly-annotated-tag\n +\trefs/tags/doubly-signed-tag\n @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '\n +\trefs/tags/three\n +\trefs/tags/two\n +\tEOF\n -+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/spot >actual &&\n ++\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/spot >actual &&\n +\ttest_cmp expect actual\n +'\n +\n -+test_expect_success 'skip until to a specific reference with partial match' '\n ++test_expect_success 'start after a specific reference with partial match' '\n +\tcat >expect <<-\\EOF &&\n +\trefs/odd/spot\n +\trefs/tags/annotated-tag\n @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '\n +\trefs/tags/three\n +\trefs/tags/two\n +\tEOF\n -+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/sp >actual &&\n ++\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/sp >actual &&\n +\ttest_cmp expect actual\n +'\n +\n -+test_expect_success 'skip until just behind a specific reference' '\n ++test_expect_success 'start after, just behind a specific reference' '\n +\tcat >expect <<-\\EOF &&\n +\trefs/odd/spot\n +\trefs/tags/annotated-tag\n @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '\n +\trefs/tags/three\n +\trefs/tags/two\n +\tEOF\n -+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/parrot >actual &&\n ++\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/parrot >actual &&\n +\ttest_cmp expect actual\n +'\n +\n -+test_expect_success 'skip until to specific directory' '\n ++test_expect_success 'start after with specific directory match' '\n +\tcat >expect <<-\\EOF &&\n +\trefs/odd/spot\n +\trefs/tags/annotated-tag\n @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '\n +\trefs/tags/three\n +\trefs/tags/two\n +\tEOF\n -+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/odd >actual &&\n ++\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd >actual &&\n +\ttest_cmp expect actual\n +'\n +\n -+test_expect_success 'skip until to specific directory with trailing slash' '\n ++test_expect_success 'start after with specific directory and trailing slash' '\n +\tcat >expect <<-\\EOF &&\n +\trefs/odd/spot\n +\trefs/tags/annotated-tag\n @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '\n +\trefs/tags/three\n +\trefs/tags/two\n +\tEOF\n -+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/lost >actual &&\n ++\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/lost >actual &&\n +\ttest_cmp expect actual\n +'\n +\n -+test_expect_success 'skip until just behind a specific directory' '\n ++test_expect_success 'start after, just behind a specific directory' '\n +\tcat >expect <<-\\EOF &&\n +\trefs/odd/spot\n +\trefs/tags/annotated-tag\n @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '\n +\trefs/tags/three\n +\trefs/tags/two\n +\tEOF\n -+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/ >actual &&\n ++\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/ >actual &&\n +\ttest_cmp expect actual\n +'\n +\n -+test_expect_success 'skip until overflow specific reference length' '\n ++test_expect_success 'start after, overflow specific reference length' '\n +\tcat >expect <<-\\EOF &&\n +\trefs/tags/annotated-tag\n +\trefs/tags/doubly-annotated-tag\n @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '\n +\trefs/tags/three\n +\trefs/tags/two\n +\tEOF\n -+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/spotnew >actual &&\n ++\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/spotnew >actual &&\n +\ttest_cmp expect actual\n +'\n +\n -+test_expect_success 'skip until overflow specific reference path' '\n ++test_expect_success 'start after, overflow specific reference path' '\n +\tcat >expect <<-\\EOF &&\n +\trefs/tags/annotated-tag\n +\trefs/tags/doubly-annotated-tag\n @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '\n +\trefs/tags/three\n +\trefs/tags/two\n +\tEOF\n -+\tgit for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/spot/new >actual &&\n ++\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/spot/new >actual &&\n ++\ttest_cmp expect actual\n ++'\n ++\n ++test_expect_success 'start after, last reference' '\n ++\tcat >expect <<-\\EOF &&\n ++\tEOF\n ++\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/tags/two >actual &&\n +\ttest_cmp expect actual\n +'\n +\n -+test_expect_success 'skip until used with a pattern' '\n ++test_expect_success 'start after used with a pattern' '\n +\tcat >expect <<-\\EOF &&\n -+\tfatal: cannot use --skip-until with patterns\n ++\tfatal: cannot use --start-after with patterns\n +\tEOF\n -+\ttest_must_fail git for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/spot refs/tags 2>actual &&\n ++\ttest_must_fail git for-each-ref --format=\"%(refname)\" --start-after=refs/odd/spot refs/tags 2>actual &&\n +\ttest_cmp expect actual\n +'\n +\n -+test_expect_success 'skip until used with custom sort order' '\n ++test_expect_success 'start after used with custom sort order' '\n +\tcat >expect <<-\\EOF &&\n -+\tfatal: cannot use --skip-until custom sort options\n ++\tfatal: cannot use --start-after with custom sort options\n +\tEOF\n -+\ttest_must_fail git for-each-ref --format=\"%(refname)\" --skip-until=refs/odd/spot --sort=author 2>actual &&\n ++\ttest_must_fail git for-each-ref --format=\"%(refname)\" --start-after=refs/odd/spot --sort=author 2>actual &&\n +\ttest_cmp expect actual\n +'\n +\n\n\nbase-commit: cf6f63ea6bf35173e02e18bdc6a4ba41288acff9\nchange-id: 20250605-306-git-for-each-ref-pagination-0ba8a29ae646\n\nThanks\n- Karthik\n\n"},{"id":"521541","messageId":"20250708-306-git-for-each-ref-pagination-v3-1-8cfba1080be4@gmail.com","threadId":"63722","inReplyTo":"20250708-306-git-for-each-ref-pagination-v3-0-8cfba1080be4@gmail.com","subject":"[PATCH v3 1/4] refs: expose `ref_iterator` via 'refs.h'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-08T13:47:46Z","receivedAt":"2025-07-08T13:48:04Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `ref_iterator` is an internal structure to the 'refs/'\nsub-directory, which allows iteration over refs. All reference iteration\nis built on top of these iterators.\n\nExternal clients of the 'refs' subsystem use the various\n'refs_for_each...()' functions to iterate over refs. However since these\nare wrapper functions, each combination of functionality requires a new\nwrapper function. This is not feasible as the functions pile up with the\nincrease in requirements. Expose the internal reference iterator, so\nadvanced users can mix and match options as needed.\n\nSigned-off-by: Karthik Nayak \n---\n refs.h | 147 +++++++++++++++++++++++++++++++++++++++++++++++++++\n refs/refs-internal.h | 145 +-------------------------------------------------\n 2 files changed, 149 insertions(+), 143 deletions(-)\n\ndiff --git a/refs.h b/refs.h\nindex 46a6008e07..7c21aaef3d 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -1190,4 +1190,151 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n \t\t\t\t unsigned int flags,\n \t\t\t\t struct strbuf *err);\n \n+/*\n+ * Reference iterators\n+ *\n+ * A reference iterator encapsulates the state of an in-progress\n+ * iteration over references. Create an instance of `struct\n+ * ref_iterator` via one of the functions in this module.\n+ *\n+ * A freshly-created ref_iterator doesn't yet point at a reference. To\n+ * advance the iterator, call ref_iterator_advance(). If successful,\n+ * this sets the iterator's refname, oid, and flags fields to describe\n+ * the next reference and returns ITER_OK. The data pointed at by\n+ * refname and oid belong to the iterator; if you want to retain them\n+ * after calling ref_iterator_advance() again or calling\n+ * ref_iterator_free(), you must make a copy. When the iteration has\n+ * been exhausted, ref_iterator_advance() releases any resources\n+ * associated with the iteration, frees the ref_iterator object, and\n+ * returns ITER_DONE. If you want to abort the iteration early, call\n+ * ref_iterator_free(), which also frees the ref_iterator object and\n+ * any associated resources. If there was an internal error advancing\n+ * to the next entry, ref_iterator_advance() aborts the iteration,\n+ * frees the ref_iterator, and returns ITER_ERROR.\n+ *\n+ * The reference currently being looked at can be peeled by calling\n+ * ref_iterator_peel(). This function is often faster than peel_ref(),\n+ * so it should be preferred when iterating over references.\n+ *\n+ * Putting it all together, a typical iteration looks like this:\n+ *\n+ * int ok;\n+ * struct ref_iterator *iter = ...;\n+ *\n+ * while ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n+ * if (want_to_stop_iteration()) {\n+ * ok = ITER_DONE;\n+ * break;\n+ * }\n+ *\n+ * // Access information about the current reference:\n+ * if (!(iter->flags & REF_ISSYMREF))\n+ * printf(\"%s is %s\\n\", iter->refname, oid_to_hex(iter->oid));\n+ *\n+ * // If you need to peel the reference:\n+ * ref_iterator_peel(iter, &oid);\n+ * }\n+ *\n+ * if (ok != ITER_DONE)\n+ * handle_error();\n+ * ref_iterator_free(iter);\n+ */\n+struct ref_iterator;\n+\n+/*\n+ * These flags are passed to refs_ref_iterator_begin() (and do_for_each_ref(),\n+ * which feeds it).\n+ */\n+enum do_for_each_ref_flags {\n+\t/*\n+\t * Include broken references in a do_for_each_ref*() iteration, which\n+\t * would normally be omitted. This includes both refs that point to\n+\t * missing objects (a true repository corruption), ones with illegal\n+\t * names (which we prefer not to expose to callers), as well as\n+\t * dangling symbolic refs (i.e., those that point to a non-existent\n+\t * ref; this is not a corruption, but as they have no valid oid, we\n+\t * omit them from normal iteration results).\n+\t */\n+\tDO_FOR_EACH_INCLUDE_BROKEN = (1 << 0),\n+\n+\t/*\n+\t * Only include per-worktree refs in a do_for_each_ref*() iteration.\n+\t * Normally this will be used with a files ref_store, since that's\n+\t * where all reference backends will presumably store their\n+\t * per-worktree refs.\n+\t */\n+\tDO_FOR_EACH_PER_WORKTREE_ONLY = (1 << 1),\n+\n+\t/*\n+\t * Omit dangling symrefs from output; this only has an effect with\n+\t * INCLUDE_BROKEN, since they are otherwise not included at all.\n+\t */\n+\tDO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 << 2),\n+\n+\t/*\n+\t * Include root refs i.e. HEAD and pseudorefs along with the regular\n+\t * refs.\n+\t */\n+\tDO_FOR_EACH_INCLUDE_ROOT_REFS = (1 << 3),\n+};\n+\n+/*\n+ * Return an iterator that goes over each reference in `refs` for\n+ * which the refname begins with prefix. If trim is non-zero, then\n+ * trim that many characters off the beginning of each refname.\n+ * The output is ordered by refname.\n+ */\n+struct ref_iterator *refs_ref_iterator_begin(\n+\tstruct ref_store *refs,\n+\tconst char *prefix, const char **exclude_patterns,\n+\tint trim, enum do_for_each_ref_flags flags);\n+\n+/*\n+ * Advance the iterator to the first or next item and return ITER_OK.\n+ * If the iteration is exhausted, free the resources associated with\n+ * the ref_iterator and return ITER_DONE. On errors, free the iterator\n+ * resources and return ITER_ERROR. It is a bug to use ref_iterator or\n+ * call this function again after it has returned ITER_DONE or\n+ * ITER_ERROR.\n+ */\n+int ref_iterator_advance(struct ref_iterator *ref_iterator);\n+\n+/*\n+ * Seek the iterator to the first reference with the given prefix.\n+ * The prefix is matched as a literal string, without regard for path\n+ * separators. If prefix is NULL or the empty string, seek the iterator to the\n+ * first reference again.\n+ *\n+ * This function is expected to behave as if a new ref iterator with the same\n+ * prefix had been created, but allows reuse of iterators and thus may allow\n+ * the backend to optimize. Parameters other than the prefix that have been\n+ * passed when creating the iterator will remain unchanged.\n+ *\n+ * Returns 0 on success, a negative error code otherwise.\n+ */\n+int ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t const char *prefix);\n+\n+/*\n+ * If possible, peel the reference currently being viewed by the\n+ * iterator. Return 0 on success.\n+ */\n+int ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t struct object_id *peeled);\n+\n+/* Free the reference iterator and any associated resources. */\n+void ref_iterator_free(struct ref_iterator *ref_iterator);\n+\n+/*\n+ * The common backend for the for_each_*ref* functions. Call fn for\n+ * each reference in iter. If the iterator itself ever returns\n+ * ITER_ERROR, return -1. If fn ever returns a non-zero value, stop\n+ * the iteration and return that value. Otherwise, return 0. In any\n+ * case, free the iterator when done. This function is basically an\n+ * adapter between the callback style of reference iteration and the\n+ * iterator style.\n+ */\n+int do_for_each_ref_iterator(struct ref_iterator *iter,\n+\t\t\t each_ref_fn fn, void *cb_data);\n+\n #endif /* REFS_H */\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex f868870851..03f5df04d5 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -244,90 +244,8 @@ const char *find_descendant_ref(const char *dirname,\n #define SYMREF_MAXDEPTH 5\n \n /*\n- * These flags are passed to refs_ref_iterator_begin() (and do_for_each_ref(),\n- * which feeds it).\n- */\n-enum do_for_each_ref_flags {\n-\t/*\n-\t * Include broken references in a do_for_each_ref*() iteration, which\n-\t * would normally be omitted. This includes both refs that point to\n-\t * missing objects (a true repository corruption), ones with illegal\n-\t * names (which we prefer not to expose to callers), as well as\n-\t * dangling symbolic refs (i.e., those that point to a non-existent\n-\t * ref; this is not a corruption, but as they have no valid oid, we\n-\t * omit them from normal iteration results).\n-\t */\n-\tDO_FOR_EACH_INCLUDE_BROKEN = (1 << 0),\n-\n-\t/*\n-\t * Only include per-worktree refs in a do_for_each_ref*() iteration.\n-\t * Normally this will be used with a files ref_store, since that's\n-\t * where all reference backends will presumably store their\n-\t * per-worktree refs.\n-\t */\n-\tDO_FOR_EACH_PER_WORKTREE_ONLY = (1 << 1),\n-\n-\t/*\n-\t * Omit dangling symrefs from output; this only has an effect with\n-\t * INCLUDE_BROKEN, since they are otherwise not included at all.\n-\t */\n-\tDO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 << 2),\n-\n-\t/*\n-\t * Include root refs i.e. HEAD and pseudorefs along with the regular\n-\t * refs.\n-\t */\n-\tDO_FOR_EACH_INCLUDE_ROOT_REFS = (1 << 3),\n-};\n-\n-/*\n- * Reference iterators\n- *\n- * A reference iterator encapsulates the state of an in-progress\n- * iteration over references. Create an instance of `struct\n- * ref_iterator` via one of the functions in this module.\n- *\n- * A freshly-created ref_iterator doesn't yet point at a reference. To\n- * advance the iterator, call ref_iterator_advance(). If successful,\n- * this sets the iterator's refname, oid, and flags fields to describe\n- * the next reference and returns ITER_OK. The data pointed at by\n- * refname and oid belong to the iterator; if you want to retain them\n- * after calling ref_iterator_advance() again or calling\n- * ref_iterator_free(), you must make a copy. When the iteration has\n- * been exhausted, ref_iterator_advance() releases any resources\n- * associated with the iteration, frees the ref_iterator object, and\n- * returns ITER_DONE. If you want to abort the iteration early, call\n- * ref_iterator_free(), which also frees the ref_iterator object and\n- * any associated resources. If there was an internal error advancing\n- * to the next entry, ref_iterator_advance() aborts the iteration,\n- * frees the ref_iterator, and returns ITER_ERROR.\n- *\n- * The reference currently being looked at can be peeled by calling\n- * ref_iterator_peel(). This function is often faster than peel_ref(),\n- * so it should be preferred when iterating over references.\n- *\n- * Putting it all together, a typical iteration looks like this:\n- *\n- * int ok;\n- * struct ref_iterator *iter = ...;\n- *\n- * while ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n- * if (want_to_stop_iteration()) {\n- * ok = ITER_DONE;\n- * break;\n- * }\n- *\n- * // Access information about the current reference:\n- * if (!(iter->flags & REF_ISSYMREF))\n- * printf(\"%s is %s\\n\", iter->refname, oid_to_hex(iter->oid));\n- *\n- * // If you need to peel the reference:\n- * ref_iterator_peel(iter, &oid);\n- * }\n- *\n- * if (ok != ITER_DONE)\n- * handle_error();\n- * ref_iterator_free(iter);\n+ * Data structure for holding a reference iterator. See refs.h for\n+ * more details and usage instructions.\n */\n struct ref_iterator {\n \tstruct ref_iterator_vtable *vtable;\n@@ -337,42 +255,6 @@ struct ref_iterator {\n \tunsigned int flags;\n };\n \n-/*\n- * Advance the iterator to the first or next item and return ITER_OK.\n- * If the iteration is exhausted, free the resources associated with\n- * the ref_iterator and return ITER_DONE. On errors, free the iterator\n- * resources and return ITER_ERROR. It is a bug to use ref_iterator or\n- * call this function again after it has returned ITER_DONE or\n- * ITER_ERROR.\n- */\n-int ref_iterator_advance(struct ref_iterator *ref_iterator);\n-\n-/*\n- * Seek the iterator to the first reference with the given prefix.\n- * The prefix is matched as a literal string, without regard for path\n- * separators. If prefix is NULL or the empty string, seek the iterator to the\n- * first reference again.\n- *\n- * This function is expected to behave as if a new ref iterator with the same\n- * prefix had been created, but allows reuse of iterators and thus may allow\n- * the backend to optimize. Parameters other than the prefix that have been\n- * passed when creating the iterator will remain unchanged.\n- *\n- * Returns 0 on success, a negative error code otherwise.\n- */\n-int ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t const char *prefix);\n-\n-/*\n- * If possible, peel the reference currently being viewed by the\n- * iterator. Return 0 on success.\n- */\n-int ref_iterator_peel(struct ref_iterator *ref_iterator,\n-\t\t struct object_id *peeled);\n-\n-/* Free the reference iterator and any associated resources. */\n-void ref_iterator_free(struct ref_iterator *ref_iterator);\n-\n /*\n * An iterator over nothing (its first ref_iterator_advance() call\n * returns ITER_DONE).\n@@ -384,17 +266,6 @@ struct ref_iterator *empty_ref_iterator_begin(void);\n */\n int is_empty_ref_iterator(struct ref_iterator *ref_iterator);\n \n-/*\n- * Return an iterator that goes over each reference in `refs` for\n- * which the refname begins with prefix. If trim is non-zero, then\n- * trim that many characters off the beginning of each refname.\n- * The output is ordered by refname.\n- */\n-struct ref_iterator *refs_ref_iterator_begin(\n-\t\tstruct ref_store *refs,\n-\t\tconst char *prefix, const char **exclude_patterns,\n-\t\tint trim, enum do_for_each_ref_flags flags);\n-\n /*\n * A callback function used to instruct merge_ref_iterator how to\n * interleave the entries from iter0 and iter1. The function should\n@@ -520,18 +391,6 @@ struct ref_iterator_vtable {\n */\n extern struct ref_iterator *current_ref_iter;\n \n-/*\n- * The common backend for the for_each_*ref* functions. Call fn for\n- * each reference in iter. If the iterator itself ever returns\n- * ITER_ERROR, return -1. If fn ever returns a non-zero value, stop\n- * the iteration and return that value. Otherwise, return 0. In any\n- * case, free the iterator when done. This function is basically an\n- * adapter between the callback style of reference iteration and the\n- * iterator style.\n- */\n-int do_for_each_ref_iterator(struct ref_iterator *iter,\n-\t\t\t each_ref_fn fn, void *cb_data);\n-\n struct ref_store;\n \n /* refs backends */\n\n-- \n2.49.0\n\n"},{"id":"521542","messageId":"20250708-306-git-for-each-ref-pagination-v3-2-8cfba1080be4@gmail.com","threadId":"63722","inReplyTo":"20250708-306-git-for-each-ref-pagination-v3-0-8cfba1080be4@gmail.com","subject":"[PATCH v3 2/4] ref-cache: remove unused function 'find_ref_entry()'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-08T13:47:47Z","receivedAt":"2025-07-08T13:48:05Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The 'find_ref_entry' function is no longer used, so remove it.\n\nSigned-off-by: Karthik Nayak \n---\n refs/ref-cache.c | 14 --------------\n refs/ref-cache.h | 7 -------\n 2 files changed, 21 deletions(-)\n\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex c1f1bab1d5..8aaffa8c6b 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -194,20 +194,6 @@ static struct ref_dir *find_containing_dir(struct ref_dir *dir,\n \treturn dir;\n }\n \n-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname)\n-{\n-\tint entry_index;\n-\tstruct ref_entry *entry;\n-\tdir = find_containing_dir(dir, refname);\n-\tif (!dir)\n-\t\treturn NULL;\n-\tentry_index = search_ref_dir(dir, refname, strlen(refname));\n-\tif (entry_index == -1)\n-\t\treturn NULL;\n-\tentry = dir->entries[entry_index];\n-\treturn (entry->flag & REF_DIR) ? NULL : entry;\n-}\n-\n /*\n * Emit a warning and return true iff ref1 and ref2 have the same name\n * and the same oid. Die if they have the same name but different\ndiff --git a/refs/ref-cache.h b/refs/ref-cache.h\nindex 5f04e518c3..f635d2d824 100644\n--- a/refs/ref-cache.h\n+++ b/refs/ref-cache.h\n@@ -201,13 +201,6 @@ void free_ref_cache(struct ref_cache *cache);\n */\n void add_entry_to_dir(struct ref_dir *dir, struct ref_entry *entry);\n \n-/*\n- * Find the value entry with the given name in dir, sorting ref_dirs\n- * and recursing into subdirectories as necessary. If the name is not\n- * found or it corresponds to a directory entry, return NULL.\n- */\n-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname);\n-\n /*\n * Start iterating over references in `cache`. If `prefix` is\n * specified, only include references whose names start with that\n\n-- \n2.49.0\n\n"},{"id":"521543","messageId":"20250708-306-git-for-each-ref-pagination-v3-3-8cfba1080be4@gmail.com","threadId":"63722","inReplyTo":"20250708-306-git-for-each-ref-pagination-v3-0-8cfba1080be4@gmail.com","subject":"[PATCH v3 3/4] refs: selectively set prefix in the seek functions","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-08T13:47:48Z","receivedAt":"2025-07-08T13:48:06Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The ref iterator exposes a `ref_iterator_seek()` function. The name\nsuggests that this would seek the iterator to a specific reference in\nsome ways similar to how `fseek()` works for the filesystem.\n\nHowever, the function actually sets the prefix for refs iteration. So\nfurther iteration would only yield references which match the particular\nprefix. This is a bit confusing.\n\nLet's add a 'flags' field to the function, which when set with the\n'REF_ITERATOR_SEEK_SET_PREFIX' flag, will set the prefix for the\niteration in-line with the existing behavior. Otherwise, the reference\nbackends will simply seek to the specified reference and clears any\npreviously set prefix. This allows users to start iteration from a\nspecific reference.\n\nIn the packed and reftable backend, since references are available in a\nsorted list, the changes are simply setting the prefix if needed. The\nchanges on the files-backend are a little more involved, since the files\nbackend uses the 'ref-cache' mechanism. We move out the existing logic\nwithin `cache_ref_iterator_seek()` to `cache_ref_iterator_set_prefix()`\nwhich is called when the 'REF_ITERATOR_SEEK_SET_PREFIX' flag is set. We\nthen parse the provided seek string and set the required levels and\ntheir indexes to ensure that seeking is possible.\n\nHelped-by: Patrick Steinhardt \nSigned-off-by: Karthik Nayak \n---\n refs.c | 6 ++--\n refs.h | 29 +++++++++++------\n refs/debug.c | 7 ++--\n refs/files-backend.c | 7 ++--\n refs/iterator.c | 26 ++++++++-------\n refs/packed-backend.c | 17 ++++++----\n refs/ref-cache.c | 85 ++++++++++++++++++++++++++++++++++++++++++++++---\n refs/refs-internal.h | 7 ++--\n refs/reftable-backend.c | 21 ++++++++----\n 9 files changed, 155 insertions(+), 50 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex dce5c49ca2..243e6898b8 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2666,12 +2666,12 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs\n \t\tif (!initial_transaction) {\n \t\t\tint ok;\n \n-\t\t\tif (!iter) {\n+\t\t\tif (!iter)\n \t\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n \t\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n-\t\t\t} else if (ref_iterator_seek(iter, dirname.buf) < 0) {\n+\t\t\telse if (ref_iterator_seek(iter, dirname.buf,\n+\t\t\t\t\t\t REF_ITERATOR_SEEK_SET_PREFIX) < 0)\n \t\t\t\tgoto cleanup;\n-\t\t\t}\n \n \t\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\t\t\tif (skip &&\ndiff --git a/refs.h b/refs.h\nindex 7c21aaef3d..7852ad36f3 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -1299,21 +1299,32 @@ struct ref_iterator *refs_ref_iterator_begin(\n */\n int ref_iterator_advance(struct ref_iterator *ref_iterator);\n \n+enum ref_iterator_seek_flag {\n+\t/*\n+\t * Also set the seek pattern as a prefix for iteration. This ensures\n+\t * that only references which match the prefix are yielded.\n+\t */\n+\tREF_ITERATOR_SEEK_SET_PREFIX = (1 << 0),\n+};\n+\n /*\n- * Seek the iterator to the first reference with the given prefix.\n- * The prefix is matched as a literal string, without regard for path\n- * separators. If prefix is NULL or the empty string, seek the iterator to the\n+ * Seek the iterator to the first reference matching the given seek string.\n+ * The seek string is matched as a literal string, without regard for path\n+ * separators. If seek is NULL or the empty string, seek the iterator to the\n * first reference again.\n *\n- * This function is expected to behave as if a new ref iterator with the same\n- * prefix had been created, but allows reuse of iterators and thus may allow\n- * the backend to optimize. Parameters other than the prefix that have been\n- * passed when creating the iterator will remain unchanged.\n+ * This function is expected to behave as if a new ref iterator has been\n+ * created, but allows reuse of existing iterators for optimization.\n+ *\n+ * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is\n+ * updated to match the seek string, affecting all subsequent iterations. If\n+ * not, the iterator seeks to the specified reference and clears any previously\n+ * set prefix.\n *\n * Returns 0 on success, a negative error code otherwise.\n */\n-int ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t const char *prefix);\n+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,\n+\t\t unsigned int flags);\n \n /*\n * If possible, peel the reference currently being viewed by the\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 485e3079d7..2ed8cff2aa 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -170,12 +170,13 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *seek, unsigned int flags)\n {\n \tstruct debug_ref_iterator *diter =\n \t\t(struct debug_ref_iterator *)ref_iterator;\n-\tint res = diter->iter->vtable->seek(diter->iter, prefix);\n-\ttrace_printf_key(&trace_refs, \"iterator_seek: %s: %d\\n\", prefix ? prefix : \"\", res);\n+\tint res = diter->iter->vtable->seek(diter->iter, seek, flags);\n+\ttrace_printf_key(&trace_refs, \"iterator_seek: %s flags: %d: %d\\n\",\n+\t\t\t seek ? seek : \"\", flags, res);\n \treturn res;\n }\n \ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex bf6f89b1d1..0e63013319 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -929,11 +929,11 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int files_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *seek, unsigned int flags)\n {\n \tstruct files_ref_iterator *iter =\n \t\t(struct files_ref_iterator *)ref_iterator;\n-\treturn ref_iterator_seek(iter->iter0, prefix);\n+\treturn ref_iterator_seek(iter->iter0, seek, flags);\n }\n \n static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n@@ -2316,7 +2316,8 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n-\t\t\t\t const char *prefix UNUSED)\n+\t\t\t\t const char *seek UNUSED,\n+\t\t\t\t unsigned int flags UNUSED)\n {\n \tBUG(\"ref_iterator_seek() called for reflog_iterator\");\n }\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex 766d96e795..f2364bd6e7 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -15,10 +15,10 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ref_iterator->vtable->advance(ref_iterator);\n }\n \n-int ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t const char *prefix)\n+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,\n+\t\t unsigned int flags)\n {\n-\treturn ref_iterator->vtable->seek(ref_iterator, prefix);\n+\treturn ref_iterator->vtable->seek(ref_iterator, seek, flags);\n }\n \n int ref_iterator_peel(struct ref_iterator *ref_iterator,\n@@ -57,7 +57,8 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)\n }\n \n static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n-\t\t\t\t const char *prefix UNUSED)\n+\t\t\t\t const char *seek UNUSED,\n+\t\t\t\t unsigned int flags UNUSED)\n {\n \treturn 0;\n }\n@@ -224,7 +225,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *seek, unsigned int flags)\n {\n \tstruct merge_ref_iterator *iter =\n \t\t(struct merge_ref_iterator *)ref_iterator;\n@@ -234,11 +235,11 @@ static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,\n \titer->iter0 = iter->iter0_owned;\n \titer->iter1 = iter->iter1_owned;\n \n-\tret = ref_iterator_seek(iter->iter0, prefix);\n+\tret = ref_iterator_seek(iter->iter0, seek, flags);\n \tif (ret < 0)\n \t\treturn ret;\n \n-\tret = ref_iterator_seek(iter->iter1, prefix);\n+\tret = ref_iterator_seek(iter->iter1, seek, flags);\n \tif (ret < 0)\n \t\treturn ret;\n \n@@ -407,13 +408,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *seek, unsigned int flags)\n {\n \tstruct prefix_ref_iterator *iter =\n \t\t(struct prefix_ref_iterator *)ref_iterator;\n-\tfree(iter->prefix);\n-\titer->prefix = xstrdup_or_null(prefix);\n-\treturn ref_iterator_seek(iter->iter0, prefix);\n+\n+\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX) {\n+\t\tfree(iter->prefix);\n+\t\titer->prefix = xstrdup_or_null(seek);\n+\t}\n+\treturn ref_iterator_seek(iter->iter0, seek, flags);\n }\n \n static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 7fd73a0e6d..11a363d246 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1004,19 +1004,23 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *seek, unsigned int flags)\n {\n \tstruct packed_ref_iterator *iter =\n \t\t(struct packed_ref_iterator *)ref_iterator;\n \tconst char *start;\n \n-\tif (prefix && *prefix)\n-\t\tstart = find_reference_location(iter->snapshot, prefix, 0);\n+\tif (seek && *seek)\n+\t\tstart = find_reference_location(iter->snapshot, seek, 0);\n \telse\n \t\tstart = iter->snapshot->start;\n \n-\tfree(iter->prefix);\n-\titer->prefix = xstrdup_or_null(prefix);\n+\t/* Unset any previously set prefix */\n+\tFREE_AND_NULL(iter->prefix);\n+\n+\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX)\n+\t\titer->prefix = xstrdup_or_null(seek);\n+\n \titer->pos = start;\n \titer->eof = iter->snapshot->eof;\n \n@@ -1194,7 +1198,8 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \titer->repo = ref_store->repo;\n \titer->flags = flags;\n \n-\tif (packed_ref_iterator_seek(&iter->base, prefix) < 0) {\n+\tif (packed_ref_iterator_seek(&iter->base, prefix,\n+\t\t\t\t REF_ITERATOR_SEEK_SET_PREFIX) < 0) {\n \t\tref_iterator_free(&iter->base);\n \t\treturn NULL;\n \t}\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 8aaffa8c6b..01dfbeb50c 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -434,11 +434,9 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t}\n }\n \n-static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+static int cache_ref_iterator_set_prefix(struct cache_ref_iterator *iter,\n+\t\t\t\t\t const char *prefix)\n {\n-\tstruct cache_ref_iterator *iter =\n-\t\t(struct cache_ref_iterator *)ref_iterator;\n \tstruct cache_ref_iterator_level *level;\n \tstruct ref_dir *dir;\n \n@@ -469,6 +467,82 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n \treturn 0;\n }\n \n+static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *seek, unsigned int flags)\n+{\n+\tstruct cache_ref_iterator *iter =\n+\t\t(struct cache_ref_iterator *)ref_iterator;\n+\n+\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX) {\n+\t\treturn cache_ref_iterator_set_prefix(iter, seek);\n+\t} else if (seek && *seek) {\n+\t\tstruct cache_ref_iterator_level *level;\n+\t\tconst char *slash = seek;\n+\t\tstruct ref_dir *dir;\n+\n+\t\tdir = get_ref_dir(iter->cache->root);\n+\n+\t\tif (iter->prime_dir)\n+\t\t\tprime_ref_dir(dir, seek);\n+\n+\t\titer->levels_nr = 1;\n+\t\tlevel = &iter->levels[0];\n+\t\tlevel->index = -1;\n+\t\tlevel->dir = dir;\n+\n+\t\t/* Unset any previously set prefix */\n+\t\tFREE_AND_NULL(iter->prefix);\n+\n+\t\t/*\n+\t\t * Breakdown the provided seek path and assign the correct\n+\t\t * indexing to each level as needed.\n+\t\t */\n+\t\tdo {\n+\t\t\tint len, idx;\n+\t\t\tint cmp = 0;\n+\n+\t\t\tsort_ref_dir(dir);\n+\n+\t\t\tslash = strchr(slash, '/');\n+\t\t\tlen = slash ? slash - seek : (int)strlen(seek);\n+\n+\t\t\tfor (idx = 0; idx < dir->nr; idx++) {\n+\t\t\t\tcmp = strncmp(seek, dir->entries[idx]->name, len);\n+\t\t\t\tif (cmp <= 0)\n+\t\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\t/* don't overflow the index */\n+\t\t\tidx = idx >= dir->nr ? dir->nr - 1 : idx;\n+\n+\t\t\tif (slash)\n+\t\t\t\tslash = slash + 1;\n+\n+\t\t\tlevel->index = idx;\n+\t\t\tif (dir->entries[idx]->flag & REF_DIR) {\n+\t\t\t\t/* push down a level */\n+\t\t\t\tdir = get_ref_dir(dir->entries[idx]);\n+\n+\t\t\t\tALLOC_GROW(iter->levels, iter->levels_nr + 1,\n+\t\t\t\t\t iter->levels_alloc);\n+\t\t\t\tlevel = &iter->levels[iter->levels_nr++];\n+\t\t\t\tlevel->dir = dir;\n+\t\t\t\tlevel->index = -1;\n+\t\t\t} else {\n+\t\t\t\t/* reduce the index so the leaf node is iterated over */\n+\t\t\t\tif (cmp <= 0 && !slash)\n+\t\t\t\t\tlevel->index = idx - 1;\n+\t\t\t\t/*\n+\t\t\t\t * while the seek path may not be exhausted, our\n+\t\t\t\t * match is exhausted at a leaf node.\n+\t\t\t\t */\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t} while (slash);\n+\t}\n+\n+\treturn 0;\n+}\n+\n static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -509,7 +583,8 @@ struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\n \titer->cache = cache;\n \titer->prime_dir = prime_dir;\n \n-\tif (cache_ref_iterator_seek(&iter->base, prefix) < 0) {\n+\tif (cache_ref_iterator_seek(&iter->base, prefix,\n+\t\t\t\t REF_ITERATOR_SEEK_SET_PREFIX) < 0) {\n \t\tref_iterator_free(&iter->base);\n \t\treturn NULL;\n \t}\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 03f5df04d5..6376a3b379 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -353,11 +353,12 @@ void base_ref_iterator_init(struct ref_iterator *iter,\n typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);\n \n /*\n- * Seek the iterator to the first reference matching the given prefix. Should\n- * behave the same as if a new iterator was created with the same prefix.\n+ * Seek the iterator to the first matching reference. If set_prefix is set,\n+ * it would behave the same as if a new iterator was created with the same\n+ * prefix.\n */\n typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix);\n+\t\t\t\t const char *seek, unsigned int flags);\n \n /*\n * Peels the current ref, returning 0 for success or -1 for failure.\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 4c3817f4ec..d627221b65 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -719,15 +719,20 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *seek, unsigned int flags)\n {\n \tstruct reftable_ref_iterator *iter =\n \t\t(struct reftable_ref_iterator *)ref_iterator;\n \n-\tfree(iter->prefix);\n-\titer->prefix = xstrdup_or_null(prefix);\n-\titer->prefix_len = prefix ? strlen(prefix) : 0;\n-\titer->err = reftable_iterator_seek_ref(&iter->iter, prefix);\n+\t/* Unset any previously set prefix */\n+\tFREE_AND_NULL(iter->prefix);\n+\titer->prefix_len = 0;\n+\n+\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX) {\n+\t\titer->prefix = xstrdup_or_null(seek);\n+\t\titer->prefix_len = seek ? strlen(seek) : 0;\n+\t}\n+\titer->err = reftable_iterator_seek_ref(&iter->iter, seek);\n \n \treturn iter->err;\n }\n@@ -839,7 +844,8 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_\n \tif (ret)\n \t\tgoto done;\n \n-\tret = reftable_ref_iterator_seek(&iter->base, prefix);\n+\tret = reftable_ref_iterator_seek(&iter->base, prefix,\n+\t\t\t\t\t REF_ITERATOR_SEEK_SET_PREFIX);\n \tif (ret)\n \t\tgoto done;\n \n@@ -2042,7 +2048,8 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n-\t\t\t\t\t const char *prefix UNUSED)\n+\t\t\t\t\t const char *seek UNUSED,\n+\t\t\t\t\t unsigned int flags UNUSED)\n {\n \tBUG(\"reftable reflog iterator cannot be seeked\");\n \treturn -1;\n\n-- \n2.49.0\n\n"},{"id":"521544","messageId":"20250708-306-git-for-each-ref-pagination-v3-4-8cfba1080be4@gmail.com","threadId":"63722","inReplyTo":"20250708-306-git-for-each-ref-pagination-v3-0-8cfba1080be4@gmail.com","subject":"[PATCH v3 4/4] for-each-ref: introduce a '--start-after' option","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-08T13:47:49Z","receivedAt":"2025-07-08T13:48:07Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `git-for-each-ref(1)` command is used to iterate over references\npresent in a repository. In large repositories with millions of\nreferences, it would be optimal to paginate this output such that we\ncan start iteration from a given reference. This would avoid having to\niterate over all references from the beginning each time when paginating\nthrough results.\n\nThe previous commit added 'seek' functionality to the reference\nbackends. Utilize this and expose a '--start-after' option in\n'git-for-each-ref(1)'. When used, the reference iteration seeks to the\nlexicographically next reference and iterates from there onward.\n\nThis enables efficient pagination workflows like:\n git for-each-ref --count=100\n git for-each-ref --count=100 --start-after=refs/heads/branch-100\n git for-each-ref --count=100 --start-after=refs/heads/branch-200\n\nSince the reference iterators only allow seeking to a specified marker\nvia the `ref_iterator_seek()`, we introduce a helper function\n`start_ref_iterator_after()`, which seeks to next reference by simply\nadding (char) 1 to the marker.\n\nWe must note that pagination always continues from the provided marker,\nas such any concurrent reference updates lexicographically behind the\nmarker will not be output. Document the same.\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/git-for-each-ref.adoc | 11 +-\n builtin/for-each-ref.c | 8 ++\n ref-filter.c | 80 +++++++++++----\n ref-filter.h | 1 +\n t/t6302-for-each-ref-filter.sh | 194 ++++++++++++++++++++++++++++++++++++\n 5 files changed, 273 insertions(+), 21 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.adoc b/Documentation/git-for-each-ref.adoc\nindex 5ef89fc0fe..e099d1ba7c 100644\n--- a/Documentation/git-for-each-ref.adoc\n+++ b/Documentation/git-for-each-ref.adoc\n@@ -14,7 +14,7 @@ SYNOPSIS\n \t\t [--points-at=]\n \t\t [--merged[=]] [--no-merged[=]]\n \t\t [--contains[=]] [--no-contains[=]]\n-\t\t [--exclude= ...]\n+\t\t [--exclude= ...] [--start-after=]\n \n DESCRIPTION\n -----------\n@@ -108,6 +108,15 @@ TAB %(refname)`.\n --include-root-refs::\n \tList root refs (HEAD and pseudorefs) apart from regular refs.\n \n+--start-after::\n+ Allows paginating the output by skipping references up to and including the\n+ specified marker. When paging, it should be noted that references may be\n+ deleted, modified or added between invocations. Output will only yield those\n+ references which follow the marker lexicographically. If the marker does not\n+ exist, output begins from the first reference that would come after it\n+ alphabetically. Cannot be used with general pattern matching or custom\n+ sort options.\n+\n FIELD NAMES\n -----------\n \ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 3d2207ec77..3f21598046 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -13,6 +13,7 @@ static char const * const for_each_ref_usage[] = {\n \tN_(\"git for-each-ref [--points-at ]\"),\n \tN_(\"git for-each-ref [--merged []] [--no-merged []]\"),\n \tN_(\"git for-each-ref [--contains []] [--no-contains []]\"),\n+\tN_(\"git for-each-ref [--start-after ]\"),\n \tNULL\n };\n \n@@ -44,6 +45,7 @@ int cmd_for_each_ref(int argc,\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_INTEGER( 0 , \"count\", &format.array_opts.max_count, N_(\"show only matched refs\")),\n \t\tOPT_STRING( 0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n+\t\tOPT_STRING( 0 , \"start-after\", &filter.start_after, N_(\"start-start\"), N_(\"start iteration after the provided marker\")),\n \t\tOPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n \t\tOPT_REF_FILTER_EXCLUDE(&filter),\n \t\tOPT_REF_SORT(&sorting_options),\n@@ -79,6 +81,9 @@ int cmd_for_each_ref(int argc,\n \tif (verify_ref_format(&format))\n \t\tusage_with_options(for_each_ref_usage, opts);\n \n+\tif (filter.start_after && sorting_options.nr > 1)\n+\t\tdie(_(\"cannot use --start-after with custom sort options\"));\n+\n \tsorting = ref_sorting_options(&sorting_options);\n \tref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);\n \tfilter.ignore_case = icase;\n@@ -100,6 +105,9 @@ int cmd_for_each_ref(int argc,\n \t\tfilter.name_patterns = argv;\n \t}\n \n+\tif (filter.start_after && filter.name_patterns && filter.name_patterns[0])\n+\t\tdie(_(\"cannot use --start-after with patterns\"));\n+\n \tif (include_root_refs)\n \t\tflags |= FILTER_REFS_ROOT_REFS | FILTER_REFS_DETACHED_HEAD;\n \ndiff --git a/ref-filter.c b/ref-filter.c\nindex 7a274633cf..2dfd385313 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2683,6 +2683,24 @@ static int filter_exclude_match(struct ref_filter *filter, const char *refname)\n \treturn match_pattern(filter->exclude.v, refname, filter->ignore_case);\n }\n \n+/*\n+ * We need to seek to the reference right after a given marker but excluding any\n+ * matching references. So we seek to the lexicographically next reference.\n+ */\n+static int start_ref_iterator_after(struct ref_iterator *iter, const char *marker)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret;\n+\n+\tstrbuf_addstr(&sb, marker);\n+\tstrbuf_addch(&sb, 1);\n+\n+\tret = ref_iterator_seek(iter, sb.buf, 0);\n+\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n /*\n * This is the same as for_each_fullref_in(), but it tries to iterate\n * only over the patterns we'll care about. Note that it _doesn't_ do a full\n@@ -2692,10 +2710,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t\t\t each_ref_fn cb,\n \t\t\t\t void *cb_data)\n {\n+\tstruct ref_iterator *iter;\n+\tint flags = 0, ret = 0;\n+\n \tif (filter->kind & FILTER_REFS_ROOT_REFS) {\n \t\t/* In this case, we want to print all refs including root refs. */\n-\t\treturn refs_for_each_include_root_refs(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t cb, cb_data);\n+\t\tflags |= DO_FOR_EACH_INCLUDE_ROOT_REFS;\n+\t\tgoto non_prefix_iter;\n \t}\n \n \tif (!filter->match_as_path) {\n@@ -2704,8 +2725,7 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t * prefixes like \"refs/heads/\" etc. are stripped off,\n \t\t * so we have to look at everything:\n \t\t */\n-\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t\"\", NULL, cb, cb_data);\n+\t\tgoto non_prefix_iter;\n \t}\n \n \tif (filter->ignore_case) {\n@@ -2714,20 +2734,29 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t * so just return everything and let the caller\n \t\t * sort it out.\n \t\t */\n-\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t\"\", NULL, cb, cb_data);\n+\t\tgoto non_prefix_iter;\n \t}\n \n \tif (!filter->name_patterns[0]) {\n \t\t/* no patterns; we have to look at everything */\n-\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"\", filter->exclude.v, cb, cb_data);\n+\t\tgoto non_prefix_iter;\n \t}\n \n \treturn refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n \t\t\t\t\t\t NULL, filter->name_patterns,\n \t\t\t\t\t\t filter->exclude.v,\n \t\t\t\t\t\t cb, cb_data);\n+\n+non_prefix_iter:\n+\titer = refs_ref_iterator_begin(get_main_ref_store(the_repository), \"\",\n+\t\t\t\t NULL, 0, flags);\n+\tif (filter->start_after)\n+\t\tret = start_ref_iterator_after(iter, filter->start_after);\n+\n+\tif (ret)\n+\t\treturn ret;\n+\n+\treturn do_for_each_ref_iterator(iter, cb, cb_data);\n }\n \n /*\n@@ -3197,9 +3226,11 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n \tinit_contains_cache(&filter->internal.no_contains_cache);\n \n \t/* Simple per-ref filtering */\n-\tif (!filter->kind)\n+\tif (!filter->kind) {\n \t\tdie(\"filter_refs: invalid type\");\n-\telse {\n+\t} else {\n+\t\tconst char *prefix = NULL;\n+\n \t\t/*\n \t\t * For common cases where we need only branches or remotes or tags,\n \t\t * we only iterate through those refs. If a mix of refs is needed,\n@@ -3207,19 +3238,28 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n \t\t * of filter_ref_kind().\n \t\t */\n \t\tif (filter->kind == FILTER_REFS_BRANCHES)\n-\t\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"refs/heads/\", NULL,\n-\t\t\t\t\t\t fn, cb_data);\n+\t\t\tprefix = \"refs/heads/\";\n \t\telse if (filter->kind == FILTER_REFS_REMOTES)\n-\t\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"refs/remotes/\", NULL,\n-\t\t\t\t\t\t fn, cb_data);\n+\t\t\tprefix = \"refs/remotes/\";\n \t\telse if (filter->kind == FILTER_REFS_TAGS)\n-\t\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"refs/tags/\", NULL, fn,\n-\t\t\t\t\t\t cb_data);\n-\t\telse if (filter->kind & FILTER_REFS_REGULAR)\n+\t\t\tprefix = \"refs/tags/\";\n+\n+\t\tif (prefix) {\n+\t\t\tstruct ref_iterator *iter;\n+\n+\t\t\titer = refs_ref_iterator_begin(get_main_ref_store(the_repository),\n+\t\t\t\t\t\t \"\", NULL, 0, 0);\n+\n+\t\t\tif (filter->start_after)\n+\t\t\t\tret = start_ref_iterator_after(iter, filter->start_after);\n+\t\t\telse if (prefix)\n+\t\t\t\tret = ref_iterator_seek(iter, prefix, 1);\n+\n+\t\t\tif (!ret)\n+\t\t\t\tret = do_for_each_ref_iterator(iter, fn, cb_data);\n+\t\t} else if (filter->kind & FILTER_REFS_REGULAR) {\n \t\t\tret = for_each_fullref_in_pattern(filter, fn, cb_data);\n+\t\t}\n \n \t\t/*\n \t\t * When printing all ref types, HEAD is already included,\ndiff --git a/ref-filter.h b/ref-filter.h\nindex c98c4fbd4c..f22ca94b49 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -64,6 +64,7 @@ struct ref_array {\n \n struct ref_filter {\n \tconst char **name_patterns;\n+\tconst char *start_after;\n \tstruct strvec exclude;\n \tstruct oid_array points_at;\n \tstruct commit_list *with_commit;\ndiff --git a/t/t6302-for-each-ref-filter.sh b/t/t6302-for-each-ref-filter.sh\nindex bb02b86c16..a43e099118 100755\n--- a/t/t6302-for-each-ref-filter.sh\n+++ b/t/t6302-for-each-ref-filter.sh\n@@ -541,4 +541,198 @@ test_expect_success 'validate worktree atom' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'start after with empty value' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/main\n+\trefs/heads/main_worktree\n+\trefs/heads/side\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=\"\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after a specific reference' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/spot >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after a specific reference with partial match' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/sp >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after, just behind a specific reference' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/parrot >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after with specific directory match' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after with specific directory and trailing slash' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/lost >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after, just behind a specific directory' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after, overflow specific reference length' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/spotnew >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after, overflow specific reference path' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/spot/new >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after, last reference' '\n+\tcat >expect <<-\\EOF &&\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/tags/two >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after used with a pattern' '\n+\tcat >expect <<-\\EOF &&\n+\tfatal: cannot use --start-after with patterns\n+\tEOF\n+\ttest_must_fail git for-each-ref --format=\"%(refname)\" --start-after=refs/odd/spot refs/tags 2>actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after used with custom sort order' '\n+\tcat >expect <<-\\EOF &&\n+\tfatal: cannot use --start-after with custom sort options\n+\tEOF\n+\ttest_must_fail git for-each-ref --format=\"%(refname)\" --start-after=refs/odd/spot --sort=author 2>actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n\n-- \n2.49.0\n\n"},{"id":"521565","messageId":"xmqq4ivmv3a0.fsf@gitster.g","threadId":"63722","inReplyTo":"20250708-306-git-for-each-ref-pagination-v3-4-8cfba1080be4@gmail.com","subject":"Re: [PATCH v3 4/4] for-each-ref: introduce a '--start-after' option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-08T20:25:11Z","receivedAt":"2025-07-08T20:25:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> The `git-for-each-ref(1)` command is used to iterate over references\n> present in a repository. In large repositories with millions of\n> references, it would be optimal to paginate this output such that we\n> can start iteration from a given reference. This would avoid having to\n> iterate over all references from the beginning each time when paginating\n> through results.\n>\n> The previous commit added 'seek' functionality to the reference\n> backends. Utilize this and expose a '--start-after' option in\n> 'git-for-each-ref(1)'. When used, the reference iteration seeks to the\n> lexicographically next reference and iterates from there onward.\n>\n> This enables efficient pagination workflows like:\n> git for-each-ref --count=100\n> git for-each-ref --count=100 --start-after=refs/heads/branch-100\n> git for-each-ref --count=100 --start-after=refs/heads/branch-200\n\nIt is a bit hard to understand how this leads to \"efficient\npagination\" unless the reader is told what the calling script does\nafter the first call before making the second call. It remembers\nthe last output from the Nth call and prepares the N+1th call by\nusing that last output entry.\n\nBut that probably belongs to the end-user facing documentation, not\nin the log message.\n\n> +--start-after::\n\n`--start-after=`::\n\n> + Allows paginating the output by skipping references up to and including the\n> + specified marker. When paging, it should be noted that references may be\n> + deleted, modified or added between invocations. Output will only yield those\n> + references which follow the marker lexicographically. If the marker does not\n> + exist, output begins from the first reference that would come after it\n> + alphabetically.\n\nIt is true that the first entry shown would be what would come\nimmediately _after_ the given , whether the marker does or\ndoes not exist. So \"If the marker does not exist, output begins...\"\n-> \"Output begins ...\"\n\nOther than that, looked pretty good to me.\n\nThanks, will queue.\n\n"},{"id":"521642","messageId":"CAOLa=ZRZaoqZfT_o1nxuMXc8gPdAiDwYF1sQD1zmnfm4CpvsXA@mail.gmail.com","threadId":"63722","inReplyTo":"xmqq4ivmv3a0.fsf@gitster.g","subject":"Re: [PATCH v3 4/4] for-each-ref: introduce a '--start-after' option","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-09T09:53:46Z","receivedAt":"2025-07-09T09:53:48Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Karthik Nayak writes:\n>\n>> The `git-for-each-ref(1)` command is used to iterate over references\n>> present in a repository. In large repositories with millions of\n>> references, it would be optimal to paginate this output such that we\n>> can start iteration from a given reference. This would avoid having to\n>> iterate over all references from the beginning each time when paginating\n>> through results.\n>>\n>> The previous commit added 'seek' functionality to the reference\n>> backends. Utilize this and expose a '--start-after' option in\n>> 'git-for-each-ref(1)'. When used, the reference iteration seeks to the\n>> lexicographically next reference and iterates from there onward.\n>>\n>> This enables efficient pagination workflows like:\n>> git for-each-ref --count=100\n>> git for-each-ref --count=100 --start-after=refs/heads/branch-100\n>> git for-each-ref --count=100 --start-after=refs/heads/branch-200\n>\n> It is a bit hard to understand how this leads to \"efficient\n> pagination\" unless the reader is told what the calling script does\n> after the first call before making the second call. It remembers\n> the last output from the Nth call and prepares the N+1th call by\n> using that last output entry.\n>\n> But that probably belongs to the end-user facing documentation, not\n> in the log message.\n>\n\nI added a small line in the commit message to clarify this. I'm not sure\nthis belongs in the user facing documentation. Mostly I see this in the\ncommit message to explain the intention behind adding the flag. The\ndocumentation already mentions how the flag can be used, so we should be\ngood there.\n\n>> +--start-after::\n>\n> `--start-after=`::\n>\n\nOops. Thanks\n\n>> + Allows paginating the output by skipping references up to and including the\n>> + specified marker. When paging, it should be noted that references may be\n>> + deleted, modified or added between invocations. Output will only yield those\n>> + references which follow the marker lexicographically. If the marker does not\n>> + exist, output begins from the first reference that would come after it\n>> + alphabetically.\n>\n> It is true that the first entry shown would be what would come\n> immediately _after_ the given , whether the marker does or\n> does not exist. So \"If the marker does not exist, output begins...\"\n> -> \"Output begins ...\"\n>\n> Other than that, looked pretty good to me.\n>\n\nGood point, will amend this.\n\n> Thanks, will queue.\n\nThank you for the review. I'll add the changes locally and push a new\nversion after a day or two.\n"},{"id":"521735","messageId":"aG9hMP9gEFLhVgJL@pks.im","threadId":"63722","inReplyTo":"20250708-306-git-for-each-ref-pagination-v3-3-8cfba1080be4@gmail.com","subject":"Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-10T06:44:00Z","receivedAt":"2025-07-10T06:44:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Jul 08, 2025 at 03:47:48PM +0200, Karthik Nayak wrote:\n> diff --git a/refs.h b/refs.h\n> index 7c21aaef3d..7852ad36f3 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -1299,21 +1299,32 @@ struct ref_iterator *refs_ref_iterator_begin(\n> */\n> int ref_iterator_advance(struct ref_iterator *ref_iterator);\n> \n> +enum ref_iterator_seek_flag {\n> +\t/*\n> +\t * Also set the seek pattern as a prefix for iteration. This ensures\n> +\t * that only references which match the prefix are yielded.\n> +\t */\n> +\tREF_ITERATOR_SEEK_SET_PREFIX = (1 << 0),\n> +};\n> +\n\nNit: I think it's a tiny bit confusing that the documentation of this\nenum is split up across here and the doc of `ref_iterator_seek()`. I\nthink it would be sensible to move the last paragraph of the function\nover here so that the whole behaviour of the enum is explained in a\nsingle place.\n\n> /*\n> - * Seek the iterator to the first reference with the given prefix.\n> - * The prefix is matched as a literal string, without regard for path\n> - * separators. If prefix is NULL or the empty string, seek the iterator to the\n> + * Seek the iterator to the first reference matching the given seek string.\n> + * The seek string is matched as a literal string, without regard for path\n> + * separators. If seek is NULL or the empty string, seek the iterator to the\n> * first reference again.\n> *\n> - * This function is expected to behave as if a new ref iterator with the same\n> - * prefix had been created, but allows reuse of iterators and thus may allow\n> - * the backend to optimize. Parameters other than the prefix that have been\n> - * passed when creating the iterator will remain unchanged.\n> + * This function is expected to behave as if a new ref iterator has been\n> + * created, but allows reuse of existing iterators for optimization.\n> + *\n> + * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is\n> + * updated to match the seek string, affecting all subsequent iterations. If\n> + * not, the iterator seeks to the specified reference and clears any previously\n> + * set prefix.\n> *\n> * Returns 0 on success, a negative error code otherwise.\n> */\n> -int ref_iterator_seek(struct ref_iterator *ref_iterator,\n> -\t\t const char *prefix);\n> +int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,\n> +\t\t unsigned int flags);\n\nAnother tiny nit: instead of calling the variable `seek` we can just\ncall it `refname`. That might give a bit more of a hint what you're\nactually seeking for.\n\nBut other than that I'm happy with the new behaviour, where we are now\nconsistently either setting or resetting the prefix depending on whether\nor not the caller set the flag.\n\n> diff --git a/refs/iterator.c b/refs/iterator.c\n> index 766d96e795..f2364bd6e7 100644\n> --- a/refs/iterator.c\n> +++ b/refs/iterator.c\n> @@ -407,13 +408,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> }\n> \n> static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,\n> -\t\t\t\t const char *prefix)\n> +\t\t\t\t const char *seek, unsigned int flags)\n> {\n> \tstruct prefix_ref_iterator *iter =\n> \t\t(struct prefix_ref_iterator *)ref_iterator;\n> -\tfree(iter->prefix);\n> -\titer->prefix = xstrdup_or_null(prefix);\n> -\treturn ref_iterator_seek(iter->iter0, prefix);\n> +\n> +\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX) {\n> +\t\tfree(iter->prefix);\n> +\t\titer->prefix = xstrdup_or_null(seek);\n> +\t}\n> +\treturn ref_iterator_se\n"},{"id":"521810","messageId":"CAOLa=ZSo==Ukdt546PHSPGUpD9zqLUn_=tkf1-5_pqnkTViDZA@mail.gmail.com","threadId":"63722","inReplyTo":"aG9hMP9gEFLhVgJL@pks.im","subject":"Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-11T09:44:56Z","receivedAt":"2025-07-11T09:44:59Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Tue, Jul 08, 2025 at 03:47:48PM +0200, Karthik Nayak wrote:\n>> diff --git a/refs.h b/refs.h\n>> index 7c21aaef3d..7852ad36f3 100644\n>> --- a/refs.h\n>> +++ b/refs.h\n>> @@ -1299,21 +1299,32 @@ struct ref_iterator *refs_ref_iterator_begin(\n>> */\n>> int ref_iterator_advance(struct ref_iterator *ref_iterator);\n>>\n>> +enum ref_iterator_seek_flag {\n>> +\t/*\n>> +\t * Also set the seek pattern as a prefix for iteration. This ensures\n>> +\t * that only references which match the prefix are yielded.\n>> +\t */\n>> +\tREF_ITERATOR_SEEK_SET_PREFIX = (1 << 0),\n>> +};\n>> +\n>\n> Nit: I think it's a tiny bit confusing that the documentation of this\n> enum is split up across here and the doc of `ref_iterator_seek()`. I\n> think it would be sensible to move the last paragraph of the function\n> over here so that the whole behaviour of the enum is explained in a\n> single place.\n>\n\nYeah I think that makes sense.\n\n>> /*\n>> - * Seek the iterator to the first reference with the given prefix.\n>> - * The prefix is matched as a literal string, without regard for path\n>> - * separators. If prefix is NULL or the empty string, seek the iterator to the\n>> + * Seek the iterator to the first reference matching the given seek string.\n>> + * The seek string is matched as a literal string, without regard for path\n>> + * separators. If seek is NULL or the empty string, seek the iterator to the\n>> * first reference again.\n>> *\n>> - * This function is expected to behave as if a new ref iterator with the same\n>> - * prefix had been created, but allows reuse of iterators and thus may allow\n>> - * the backend to optimize. Parameters other than the prefix that have been\n>> - * passed when creating the iterator will remain unchanged.\n>> + * This function is expected to behave as if a new ref iterator has been\n>> + * created, but allows reuse of existing iterators for optimization.\n>> + *\n>> + * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is\n>> + * updated to match the seek string, affecting all subsequent iterations. If\n>> + * not, the iterator seeks to the specified reference and clears any previously\n>> + * set prefix.\n>> *\n>> * Returns 0 on success, a negative error code otherwise.\n>> */\n>> -int ref_iterator_seek(struct ref_iterator *ref_iterator,\n>> -\t\t const char *prefix);\n>> +int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,\n>> +\t\t unsigned int flags);\n>\n> Another tiny nit: instead of calling the variable `seek` we can just\n> call it `refname`. That might give a bit more of a hint what you're\n> actually seeking for.\n>\n\nFair enough, let me change that.\n\n> But other than that I'm happy with the new behaviour, where we are now\n> consistently either setting or resetting the prefix depending on whether\n> or not the caller set the flag.\n>\n\nThanks for the review!\n\n>> diff --git a/refs/iterator.c b/refs/iterator.c\n>> index 766d96e795..f2364bd6e7 100644\n>> --- a/refs/iterator.c\n>> +++ b/refs/iterator.c\n>> @@ -407,13 +408,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n>> }\n>>\n>> static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,\n>> -\t\t\t\t const char *prefix)\n>> +\t\t\t\t const char *seek, unsigned int flags)\n>> {\n>> \tstruct prefix_ref_iterator *iter =\n>> \t\t(struct prefix_ref_iterator *)ref_iterator;\n>> -\tfree(iter->prefix);\n>> -\titer->prefix = xstrdup_or_null(prefix);\n>> -\treturn ref_iterator_seek(iter->iter0, prefix);\n>> +\n>> +\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX) {\n>> +\t\tfree(iter->prefix);\n>> +\t\titer->prefix = xstrdup_or_null(seek);\n>> +\t}\n>> +\treturn ref_iterator_se\n"},{"id":"521826","messageId":"20250711-306-git-for-each-ref-pagination-v4-0-ed3303ad5b89@gmail.com","threadId":"63722","inReplyTo":"20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com","subject":"[PATCH v4 0/4] for-each-ref: introduce seeking functionality via '--start-after'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-11T16:18:36Z","receivedAt":"2025-07-11T16:18:45Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `git-for-each-ref(1)` command is used to iterate over references\npresent in a repository. In large repositories with millions of\nreferences, it would be optimal to paginate this output such that we\ncan start iteration from a given reference. This would avoid having to\niterate over all references from the beginning each time when paginating\nthrough results.\n\nThis series adds a '--start-after' option in 'git-for-each-ref(1)'. When\nused, the reference iteration seeks to first reference following the\nmarker alphabetically. When paging, it should be noted that references\nmay be deleted, modified or added between invocations. Output will only\nyield those references which follow the marker lexicographically. If the\nmarker does not exist, output begins from the first reference that would\ncome after it alphabetically.\n\nThis enables efficient pagination workflows like:\n git for-each-ref --count=100\n git for-each-ref --count=100 --start-after=refs/heads/branch-100\n git for-each-ref --count=100 --start-after=refs/heads/branch-200\n\nTo add this functionality, we expose the `ref_iterator` outside the\n'refs/' namespace and modify the `ref_iterator_seek()` to actually seek\nto a given reference and only set the prefix when the `set_prefix` field\nis set.\n\nOn the reftable and packed backend, the changes are simple. But since\nthe files backend uses 'ref-cache' for reference handling, the changes\nthere are a little more involved, since we need to setup the right\nlevels and the indexing.\n\nInitially I was also planning to cleanup all the `refs_for_each...()`\nfunctions in 'refs.h' by simply using the iterator, but this bloated the\nseries. So I've left that for another day.\n\nChanges in v4:\n- Patch 3/4: Move around the documentation for the flag and rename the\n seek variable to refname.\n- Patch 4/4: Cleanup the commit message and also the documentation.\n- Link to v3: https://lore.kernel.org/r/20250708-306-git-for-each-ref-pagination-v3-0-8cfba1080be4@gmail.com\n\nChanges in v3:\n- Change the working of the command to exclude the marker provided. With\n this rename the flag to '--start-after'.\n- Extend the documentation to add a note about concurrent modifications\n to the reference database.\n- Link to v2: https://lore.kernel.org/r/20250704-306-git-for-each-ref-pagination-v2-0-bcde14acdd81@gmail.com\n\nChanges in v2:\n- Modify 'ref_iterator_seek()' to take in flags instead of a\n 'set_prefix' variable. This improves readability, where users would\n use the 'REF_ITERATOR_SEEK_SET_PREFIX' instead of simply passing '1'.\n- When the set prefix flag isn't usage, reset any previously set prefix.\n This ensures that the internal prefix state is always reset whenever\n we seek and unifies the behavior between 'ref_iterator_seek' and\n 'ref_iterator_begin'.\n- Don't allow '--skip-until' to be run with '--sort', since the seeking\n always takes place before any sorting and this can be confusing.\n- Some styling fixes:\n - Remove extra newline\n - Skip braces around single lined if...else clause\n - Add braces around 'if' clause\n - Fix indentation\n- Link to v1: https://lore.kernel.org/git/20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com/\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/git-for-each-ref.adoc | 10 +-\n builtin/for-each-ref.c | 8 ++\n ref-filter.c | 80 +++++++++++----\n ref-filter.h | 1 +\n refs.c | 6 +-\n refs.h | 155 ++++++++++++++++++++++++++++\n refs/debug.c | 7 +-\n refs/files-backend.c | 7 +-\n refs/iterator.c | 26 +++--\n refs/packed-backend.c | 17 ++--\n refs/ref-cache.c | 99 ++++++++++++++----\n refs/ref-cache.h | 7 --\n refs/refs-internal.h | 152 ++--------------------------\n refs/reftable-backend.c | 21 ++--\n t/t6302-for-each-ref-filter.sh | 194 ++++++++++++++++++++++++++++++++++++\n 15 files changed, 564 insertions(+), 226 deletions(-)\n\nKarthik Nayak (4):\n refs: expose `ref_iterator` via 'refs.h'\n ref-cache: remove unused function 'find_ref_entry()'\n refs: selectively set prefix in the seek functions\n for-each-ref: introduce a '--start-after' option\n\nRange-diff versus v3:\n\n1: eed39162f5 = 1: 9e6ecff291 refs: expose `ref_iterator` via 'refs.h'\n2: b9db49d31b = 2: 22f5222e4f ref-cache: remove unused function 'find_ref_entry()'\n3: 502e2696fd ! 3: 0e71d8ffd9 refs: selectively set prefix in the seek functions\n @@ refs.h: struct ref_iterator *refs_ref_iterator_begin(\n \n +enum ref_iterator_seek_flag {\n +\t/*\n -+\t * Also set the seek pattern as a prefix for iteration. This ensures\n -+\t * that only references which match the prefix are yielded.\n ++\t * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is\n ++\t * updated to match the provided string, affecting all subsequent iterations. If\n ++\t * not, the iterator seeks to the specified reference and clears any previously\n ++\t * set prefix.\n +\t */\n +\tREF_ITERATOR_SEEK_SET_PREFIX = (1 << 0),\n +};\n @@ refs.h: struct ref_iterator *refs_ref_iterator_begin(\n - * passed when creating the iterator will remain unchanged.\n + * This function is expected to behave as if a new ref iterator has been\n + * created, but allows reuse of existing iterators for optimization.\n -+ *\n -+ * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is\n -+ * updated to match the seek string, affecting all subsequent iterations. If\n -+ * not, the iterator seeks to the specified reference and clears any previously\n -+ * set prefix.\n *\n * Returns 0 on success, a negative error code otherwise.\n */\n -int ref_iterator_seek(struct ref_iterator *ref_iterator,\n -\t\t const char *prefix);\n -+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,\n ++int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *refname,\n +\t\t unsigned int flags);\n \n /*\n @@ refs/debug.c: static int debug_ref_iterator_advance(struct ref_iterator *ref_ite\n \n static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator,\n -\t\t\t\t const char *prefix)\n -+\t\t\t\t const char *seek, unsigned int flags)\n ++\t\t\t\t const char *refname, unsigned int flags)\n {\n \tstruct debug_ref_iterator *diter =\n \t\t(struct debug_ref_iterator *)ref_iterator;\n -\tint res = diter->iter->vtable->seek(diter->iter, prefix);\n -\ttrace_printf_key(&trace_refs, \"iterator_seek: %s: %d\\n\", prefix ? prefix : \"\", res);\n -+\tint res = diter->iter->vtable->seek(diter->iter, seek, flags);\n ++\tint res = diter->iter->vtable->seek(diter->iter, refname, flags);\n +\ttrace_printf_key(&trace_refs, \"iterator_seek: %s flags: %d: %d\\n\",\n -+\t\t\t seek ? seek : \"\", flags, res);\n ++\t\t\t refname ? refname : \"\", flags, res);\n \treturn res;\n }\n \n @@ refs/files-backend.c: static int files_ref_iterator_advance(struct ref_iterator\n \n static int files_ref_iterator_seek(struct ref_iterator *ref_iterator,\n -\t\t\t\t const char *prefix)\n -+\t\t\t\t const char *seek, unsigned int flags)\n ++\t\t\t\t const char *refname, unsigned int flags)\n {\n \tstruct files_ref_iterator *iter =\n \t\t(struct files_ref_iterator *)ref_iterator;\n -\treturn ref_iterator_seek(iter->iter0, prefix);\n -+\treturn ref_iterator_seek(iter->iter0, seek, flags);\n ++\treturn ref_iterator_seek(iter->iter0, refname, flags);\n }\n \n static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n @@ refs/files-backend.c: static int files_reflog_iterator_advance(struct ref_iterat\n \n static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n -\t\t\t\t const char *prefix UNUSED)\n -+\t\t\t\t const char *seek UNUSED,\n ++\t\t\t\t const char *refname UNUSED,\n +\t\t\t\t unsigned int flags UNUSED)\n {\n \tBUG(\"ref_iterator_seek() called for reflog_iterator\");\n @@ refs/iterator.c: int ref_iterator_advance(struct ref_iterator *ref_iterator)\n \n -int ref_iterator_seek(struct ref_iterator *ref_iterator,\n -\t\t const char *prefix)\n -+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,\n ++int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *refname,\n +\t\t unsigned int flags)\n {\n -\treturn ref_iterator->vtable->seek(ref_iterator, prefix);\n -+\treturn ref_iterator->vtable->seek(ref_iterator, seek, flags);\n ++\treturn ref_iterator->vtable->seek(ref_iterator, refname, flags);\n }\n \n int ref_iterator_peel(struct ref_iterator *ref_iterator,\n @@ refs/iterator.c: static int empty_ref_iterator_advance(struct ref_iterator *ref_\n \n static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n -\t\t\t\t const char *prefix UNUSED)\n -+\t\t\t\t const char *seek UNUSED,\n ++\t\t\t\t const char *refname UNUSED,\n +\t\t\t\t unsigned int flags UNUSED)\n {\n \treturn 0;\n @@ refs/iterator.c: static int merge_ref_iterator_advance(struct ref_iterator *ref_\n \n static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,\n -\t\t\t\t const char *prefix)\n -+\t\t\t\t const char *seek, unsigned int flags)\n ++\t\t\t\t const char *refname, unsigned int flags)\n {\n \tstruct merge_ref_iterator *iter =\n \t\t(struct merge_ref_iterator *)ref_iterator;\n @@ refs/iterator.c: static int merge_ref_iterator_seek(struct ref_iterator *ref_ite\n \titer->iter1 = iter->iter1_owned;\n \n -\tret = ref_iterator_seek(iter->iter0, prefix);\n -+\tret = ref_iterator_seek(iter->iter0, seek, flags);\n ++\tret = ref_iterator_seek(iter->iter0, refname, flags);\n \tif (ret < 0)\n \t\treturn ret;\n \n -\tret = ref_iterator_seek(iter->iter1, prefix);\n -+\tret = ref_iterator_seek(iter->iter1, seek, flags);\n ++\tret = ref_iterator_seek(iter->iter1, refname, flags);\n \tif (ret < 0)\n \t\treturn ret;\n \n @@ refs/iterator.c: static int prefix_ref_iterator_advance(struct ref_iterator *ref\n \n static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,\n -\t\t\t\t const char *prefix)\n -+\t\t\t\t const char *seek, unsigned int flags)\n ++\t\t\t\t const char *refname, unsigned int flags)\n {\n \tstruct prefix_ref_iterator *iter =\n \t\t(struct prefix_ref_iterator *)ref_iterator;\n @@ refs/iterator.c: static int prefix_ref_iterator_advance(struct ref_iterator *ref\n +\n +\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX) {\n +\t\tfree(iter->prefix);\n -+\t\titer->prefix = xstrdup_or_null(seek);\n ++\t\titer->prefix = xstrdup_or_null(refname);\n +\t}\n -+\treturn ref_iterator_seek(iter->iter0, seek, flags);\n ++\treturn ref_iterator_seek(iter->iter0, refname, flags);\n }\n \n static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,\n @@ refs/packed-backend.c: static int packed_ref_iterator_advance(struct ref_iterato\n \n static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,\n -\t\t\t\t const char *prefix)\n -+\t\t\t\t const char *seek, unsigned int flags)\n ++\t\t\t\t const char *refname, unsigned int flags)\n {\n \tstruct packed_ref_iterator *iter =\n \t\t(struct packed_ref_iterator *)ref_iterator;\n @@ refs/packed-backend.c: static int packed_ref_iterator_advance(struct ref_iterato\n \n -\tif (prefix && *prefix)\n -\t\tstart = find_reference_location(iter->snapshot, prefix, 0);\n -+\tif (seek && *seek)\n -+\t\tstart = find_reference_location(iter->snapshot, seek, 0);\n ++\tif (refname && *refname)\n ++\t\tstart = find_reference_location(iter->snapshot, refname, 0);\n \telse\n \t\tstart = iter->snapshot->start;\n \n @@ refs/packed-backend.c: static int packed_ref_iterator_advance(struct ref_iterato\n +\tFREE_AND_NULL(iter->prefix);\n +\n +\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX)\n -+\t\titer->prefix = xstrdup_or_null(seek);\n ++\t\titer->prefix = xstrdup_or_null(refname);\n +\n \titer->pos = start;\n \titer->eof = iter->snapshot->eof;\n @@ refs/ref-cache.c: static int cache_ref_iterator_seek(struct ref_iterator *ref_it\n }\n \n +static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n -+\t\t\t\t const char *seek, unsigned int flags)\n ++\t\t\t\t const char *refname, unsigned int flags)\n +{\n +\tstruct cache_ref_iterator *iter =\n +\t\t(struct cache_ref_iterator *)ref_iterator;\n +\n +\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX) {\n -+\t\treturn cache_ref_iterator_set_prefix(iter, seek);\n -+\t} else if (seek && *seek) {\n ++\t\treturn cache_ref_iterator_set_prefix(iter, refname);\n ++\t} else if (refname && *refname) {\n +\t\tstruct cache_ref_iterator_level *level;\n -+\t\tconst char *slash = seek;\n ++\t\tconst char *slash = refname;\n +\t\tstruct ref_dir *dir;\n +\n +\t\tdir = get_ref_dir(iter->cache->root);\n +\n +\t\tif (iter->prime_dir)\n -+\t\t\tprime_ref_dir(dir, seek);\n ++\t\t\tprime_ref_dir(dir, refname);\n +\n +\t\titer->levels_nr = 1;\n +\t\tlevel = &iter->levels[0];\n @@ refs/ref-cache.c: static int cache_ref_iterator_seek(struct ref_iterator *ref_it\n +\t\t\tsort_ref_dir(dir);\n +\n +\t\t\tslash = strchr(slash, '/');\n -+\t\t\tlen = slash ? slash - seek : (int)strlen(seek);\n ++\t\t\tlen = slash ? slash - refname : (int)strlen(refname);\n +\n +\t\t\tfor (idx = 0; idx < dir->nr; idx++) {\n -+\t\t\t\tcmp = strncmp(seek, dir->entries[idx]->name, len);\n ++\t\t\t\tcmp = strncmp(refname, dir->entries[idx]->name, len);\n +\t\t\t\tif (cmp <= 0)\n +\t\t\t\t\tbreak;\n +\t\t\t}\n @@ refs/refs-internal.h: void base_ref_iterator_init(struct ref_iterator *iter,\n */\n typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,\n -\t\t\t\t const char *prefix);\n -+\t\t\t\t const char *seek, unsigned int flags);\n ++\t\t\t\t const char *refname, unsigned int flags);\n \n /*\n * Peels the current ref, returning 0 for success or -1 for failure.\n @@ refs/reftable-backend.c: static int reftable_ref_iterator_advance(struct ref_ite\n \n static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,\n -\t\t\t\t const char *prefix)\n -+\t\t\t\t const char *seek, unsigned int flags)\n ++\t\t\t\t const char *refname, unsigned int flags)\n {\n \tstruct reftable_ref_iterator *iter =\n \t\t(struct reftable_ref_iterator *)ref_iterator;\n @@ refs/reftable-backend.c: static int reftable_ref_iterator_advance(struct ref_ite\n +\titer->prefix_len = 0;\n +\n +\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX) {\n -+\t\titer->prefix = xstrdup_or_null(seek);\n -+\t\titer->prefix_len = seek ? strlen(seek) : 0;\n ++\t\titer->prefix = xstrdup_or_null(refname);\n ++\t\titer->prefix_len = refname ? strlen(refname) : 0;\n +\t}\n -+\titer->err = reftable_iterator_seek_ref(&iter->iter, seek);\n ++\titer->err = reftable_iterator_seek_ref(&iter->iter, refname);\n \n \treturn iter->err;\n }\n @@ refs/reftable-backend.c: static int reftable_reflog_iterator_advance(struct ref_\n \n static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n -\t\t\t\t\t const char *prefix UNUSED)\n -+\t\t\t\t\t const char *seek UNUSED,\n ++\t\t\t\t\t const char *refname UNUSED,\n +\t\t\t\t\t unsigned int flags UNUSED)\n {\n \tBUG(\"reftable reflog iterator cannot be seeked\");\n4: a571579886 ! 4: e4e9dddd15 for-each-ref: introduce a '--start-after' option\n @@ Commit message\n 'git-for-each-ref(1)'. When used, the reference iteration seeks to the\n lexicographically next reference and iterates from there onward.\n \n - This enables efficient pagination workflows like:\n + This enables efficient pagination workflows, where the calling script\n + can remember the last provided reference and use that as the starting\n + point for the next set of references:\n git for-each-ref --count=100\n git for-each-ref --count=100 --start-after=refs/heads/branch-100\n git for-each-ref --count=100 --start-after=refs/heads/branch-200\n @@ Documentation/git-for-each-ref.adoc: TAB %(refname)`.\n --include-root-refs::\n \tList root refs (HEAD and pseudorefs) apart from regular refs.\n \n -+--start-after::\n ++--start-after=::\n + Allows paginating the output by skipping references up to and including the\n + specified marker. When paging, it should be noted that references may be\n + deleted, modified or added between invocations. Output will only yield those\n -+ references which follow the marker lexicographically. If the marker does not\n -+ exist, output begins from the first reference that would come after it\n -+ alphabetically. Cannot be used with general pattern matching or custom\n -+ sort options.\n ++ references which follow the marker lexicographically. Output begins from the\n ++ first reference that would come after the marker alphabetically. Cannot be\n ++ used with general pattern matching or custom sort options.\n +\n FIELD NAMES\n -----------\n\n\nbase-commit: cf6f63ea6bf35173e02e18bdc6a4ba41288acff9\nchange-id: 20250605-306-git-for-each-ref-pagination-0ba8a29ae646\n\nThanks\n- Karthik\n\n"},{"id":"521827","messageId":"20250711-306-git-for-each-ref-pagination-v4-2-ed3303ad5b89@gmail.com","threadId":"63722","inReplyTo":"20250711-306-git-for-each-ref-pagination-v4-0-ed3303ad5b89@gmail.com","subject":"[PATCH v4 2/4] ref-cache: remove unused function 'find_ref_entry()'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-11T16:18:38Z","receivedAt":"2025-07-11T16:18:46Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The 'find_ref_entry' function is no longer used, so remove it.\n\nSigned-off-by: Karthik Nayak \n---\n refs/ref-cache.c | 14 --------------\n refs/ref-cache.h | 7 -------\n 2 files changed, 21 deletions(-)\n\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex c1f1bab1d5..8aaffa8c6b 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -194,20 +194,6 @@ static struct ref_dir *find_containing_dir(struct ref_dir *dir,\n \treturn dir;\n }\n \n-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname)\n-{\n-\tint entry_index;\n-\tstruct ref_entry *entry;\n-\tdir = find_containing_dir(dir, refname);\n-\tif (!dir)\n-\t\treturn NULL;\n-\tentry_index = search_ref_dir(dir, refname, strlen(refname));\n-\tif (entry_index == -1)\n-\t\treturn NULL;\n-\tentry = dir->entries[entry_index];\n-\treturn (entry->flag & REF_DIR) ? NULL : entry;\n-}\n-\n /*\n * Emit a warning and return true iff ref1 and ref2 have the same name\n * and the same oid. Die if they have the same name but different\ndiff --git a/refs/ref-cache.h b/refs/ref-cache.h\nindex 5f04e518c3..f635d2d824 100644\n--- a/refs/ref-cache.h\n+++ b/refs/ref-cache.h\n@@ -201,13 +201,6 @@ void free_ref_cache(struct ref_cache *cache);\n */\n void add_entry_to_dir(struct ref_dir *dir, struct ref_entry *entry);\n \n-/*\n- * Find the value entry with the given name in dir, sorting ref_dirs\n- * and recursing into subdirectories as necessary. If the name is not\n- * found or it corresponds to a directory entry, return NULL.\n- */\n-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname);\n-\n /*\n * Start iterating over references in `cache`. If `prefix` is\n * specified, only include references whose names start with that\n\n-- \n2.49.0\n\n"},{"id":"521828","messageId":"20250711-306-git-for-each-ref-pagination-v4-1-ed3303ad5b89@gmail.com","threadId":"63722","inReplyTo":"20250711-306-git-for-each-ref-pagination-v4-0-ed3303ad5b89@gmail.com","subject":"[PATCH v4 1/4] refs: expose `ref_iterator` via 'refs.h'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-11T16:18:37Z","receivedAt":"2025-07-11T16:18:46Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `ref_iterator` is an internal structure to the 'refs/'\nsub-directory, which allows iteration over refs. All reference iteration\nis built on top of these iterators.\n\nExternal clients of the 'refs' subsystem use the various\n'refs_for_each...()' functions to iterate over refs. However since these\nare wrapper functions, each combination of functionality requires a new\nwrapper function. This is not feasible as the functions pile up with the\nincrease in requirements. Expose the internal reference iterator, so\nadvanced users can mix and match options as needed.\n\nSigned-off-by: Karthik Nayak \n---\n refs.h | 147 +++++++++++++++++++++++++++++++++++++++++++++++++++\n refs/refs-internal.h | 145 +-------------------------------------------------\n 2 files changed, 149 insertions(+), 143 deletions(-)\n\ndiff --git a/refs.h b/refs.h\nindex 46a6008e07..7c21aaef3d 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -1190,4 +1190,151 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n \t\t\t\t unsigned int flags,\n \t\t\t\t struct strbuf *err);\n \n+/*\n+ * Reference iterators\n+ *\n+ * A reference iterator encapsulates the state of an in-progress\n+ * iteration over references. Create an instance of `struct\n+ * ref_iterator` via one of the functions in this module.\n+ *\n+ * A freshly-created ref_iterator doesn't yet point at a reference. To\n+ * advance the iterator, call ref_iterator_advance(). If successful,\n+ * this sets the iterator's refname, oid, and flags fields to describe\n+ * the next reference and returns ITER_OK. The data pointed at by\n+ * refname and oid belong to the iterator; if you want to retain them\n+ * after calling ref_iterator_advance() again or calling\n+ * ref_iterator_free(), you must make a copy. When the iteration has\n+ * been exhausted, ref_iterator_advance() releases any resources\n+ * associated with the iteration, frees the ref_iterator object, and\n+ * returns ITER_DONE. If you want to abort the iteration early, call\n+ * ref_iterator_free(), which also frees the ref_iterator object and\n+ * any associated resources. If there was an internal error advancing\n+ * to the next entry, ref_iterator_advance() aborts the iteration,\n+ * frees the ref_iterator, and returns ITER_ERROR.\n+ *\n+ * The reference currently being looked at can be peeled by calling\n+ * ref_iterator_peel(). This function is often faster than peel_ref(),\n+ * so it should be preferred when iterating over references.\n+ *\n+ * Putting it all together, a typical iteration looks like this:\n+ *\n+ * int ok;\n+ * struct ref_iterator *iter = ...;\n+ *\n+ * while ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n+ * if (want_to_stop_iteration()) {\n+ * ok = ITER_DONE;\n+ * break;\n+ * }\n+ *\n+ * // Access information about the current reference:\n+ * if (!(iter->flags & REF_ISSYMREF))\n+ * printf(\"%s is %s\\n\", iter->refname, oid_to_hex(iter->oid));\n+ *\n+ * // If you need to peel the reference:\n+ * ref_iterator_peel(iter, &oid);\n+ * }\n+ *\n+ * if (ok != ITER_DONE)\n+ * handle_error();\n+ * ref_iterator_free(iter);\n+ */\n+struct ref_iterator;\n+\n+/*\n+ * These flags are passed to refs_ref_iterator_begin() (and do_for_each_ref(),\n+ * which feeds it).\n+ */\n+enum do_for_each_ref_flags {\n+\t/*\n+\t * Include broken references in a do_for_each_ref*() iteration, which\n+\t * would normally be omitted. This includes both refs that point to\n+\t * missing objects (a true repository corruption), ones with illegal\n+\t * names (which we prefer not to expose to callers), as well as\n+\t * dangling symbolic refs (i.e., those that point to a non-existent\n+\t * ref; this is not a corruption, but as they have no valid oid, we\n+\t * omit them from normal iteration results).\n+\t */\n+\tDO_FOR_EACH_INCLUDE_BROKEN = (1 << 0),\n+\n+\t/*\n+\t * Only include per-worktree refs in a do_for_each_ref*() iteration.\n+\t * Normally this will be used with a files ref_store, since that's\n+\t * where all reference backends will presumably store their\n+\t * per-worktree refs.\n+\t */\n+\tDO_FOR_EACH_PER_WORKTREE_ONLY = (1 << 1),\n+\n+\t/*\n+\t * Omit dangling symrefs from output; this only has an effect with\n+\t * INCLUDE_BROKEN, since they are otherwise not included at all.\n+\t */\n+\tDO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 << 2),\n+\n+\t/*\n+\t * Include root refs i.e. HEAD and pseudorefs along with the regular\n+\t * refs.\n+\t */\n+\tDO_FOR_EACH_INCLUDE_ROOT_REFS = (1 << 3),\n+};\n+\n+/*\n+ * Return an iterator that goes over each reference in `refs` for\n+ * which the refname begins with prefix. If trim is non-zero, then\n+ * trim that many characters off the beginning of each refname.\n+ * The output is ordered by refname.\n+ */\n+struct ref_iterator *refs_ref_iterator_begin(\n+\tstruct ref_store *refs,\n+\tconst char *prefix, const char **exclude_patterns,\n+\tint trim, enum do_for_each_ref_flags flags);\n+\n+/*\n+ * Advance the iterator to the first or next item and return ITER_OK.\n+ * If the iteration is exhausted, free the resources associated with\n+ * the ref_iterator and return ITER_DONE. On errors, free the iterator\n+ * resources and return ITER_ERROR. It is a bug to use ref_iterator or\n+ * call this function again after it has returned ITER_DONE or\n+ * ITER_ERROR.\n+ */\n+int ref_iterator_advance(struct ref_iterator *ref_iterator);\n+\n+/*\n+ * Seek the iterator to the first reference with the given prefix.\n+ * The prefix is matched as a literal string, without regard for path\n+ * separators. If prefix is NULL or the empty string, seek the iterator to the\n+ * first reference again.\n+ *\n+ * This function is expected to behave as if a new ref iterator with the same\n+ * prefix had been created, but allows reuse of iterators and thus may allow\n+ * the backend to optimize. Parameters other than the prefix that have been\n+ * passed when creating the iterator will remain unchanged.\n+ *\n+ * Returns 0 on success, a negative error code otherwise.\n+ */\n+int ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t const char *prefix);\n+\n+/*\n+ * If possible, peel the reference currently being viewed by the\n+ * iterator. Return 0 on success.\n+ */\n+int ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t struct object_id *peeled);\n+\n+/* Free the reference iterator and any associated resources. */\n+void ref_iterator_free(struct ref_iterator *ref_iterator);\n+\n+/*\n+ * The common backend for the for_each_*ref* functions. Call fn for\n+ * each reference in iter. If the iterator itself ever returns\n+ * ITER_ERROR, return -1. If fn ever returns a non-zero value, stop\n+ * the iteration and return that value. Otherwise, return 0. In any\n+ * case, free the iterator when done. This function is basically an\n+ * adapter between the callback style of reference iteration and the\n+ * iterator style.\n+ */\n+int do_for_each_ref_iterator(struct ref_iterator *iter,\n+\t\t\t each_ref_fn fn, void *cb_data);\n+\n #endif /* REFS_H */\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex f868870851..03f5df04d5 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -244,90 +244,8 @@ const char *find_descendant_ref(const char *dirname,\n #define SYMREF_MAXDEPTH 5\n \n /*\n- * These flags are passed to refs_ref_iterator_begin() (and do_for_each_ref(),\n- * which feeds it).\n- */\n-enum do_for_each_ref_flags {\n-\t/*\n-\t * Include broken references in a do_for_each_ref*() iteration, which\n-\t * would normally be omitted. This includes both refs that point to\n-\t * missing objects (a true repository corruption), ones with illegal\n-\t * names (which we prefer not to expose to callers), as well as\n-\t * dangling symbolic refs (i.e., those that point to a non-existent\n-\t * ref; this is not a corruption, but as they have no valid oid, we\n-\t * omit them from normal iteration results).\n-\t */\n-\tDO_FOR_EACH_INCLUDE_BROKEN = (1 << 0),\n-\n-\t/*\n-\t * Only include per-worktree refs in a do_for_each_ref*() iteration.\n-\t * Normally this will be used with a files ref_store, since that's\n-\t * where all reference backends will presumably store their\n-\t * per-worktree refs.\n-\t */\n-\tDO_FOR_EACH_PER_WORKTREE_ONLY = (1 << 1),\n-\n-\t/*\n-\t * Omit dangling symrefs from output; this only has an effect with\n-\t * INCLUDE_BROKEN, since they are otherwise not included at all.\n-\t */\n-\tDO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 << 2),\n-\n-\t/*\n-\t * Include root refs i.e. HEAD and pseudorefs along with the regular\n-\t * refs.\n-\t */\n-\tDO_FOR_EACH_INCLUDE_ROOT_REFS = (1 << 3),\n-};\n-\n-/*\n- * Reference iterators\n- *\n- * A reference iterator encapsulates the state of an in-progress\n- * iteration over references. Create an instance of `struct\n- * ref_iterator` via one of the functions in this module.\n- *\n- * A freshly-created ref_iterator doesn't yet point at a reference. To\n- * advance the iterator, call ref_iterator_advance(). If successful,\n- * this sets the iterator's refname, oid, and flags fields to describe\n- * the next reference and returns ITER_OK. The data pointed at by\n- * refname and oid belong to the iterator; if you want to retain them\n- * after calling ref_iterator_advance() again or calling\n- * ref_iterator_free(), you must make a copy. When the iteration has\n- * been exhausted, ref_iterator_advance() releases any resources\n- * associated with the iteration, frees the ref_iterator object, and\n- * returns ITER_DONE. If you want to abort the iteration early, call\n- * ref_iterator_free(), which also frees the ref_iterator object and\n- * any associated resources. If there was an internal error advancing\n- * to the next entry, ref_iterator_advance() aborts the iteration,\n- * frees the ref_iterator, and returns ITER_ERROR.\n- *\n- * The reference currently being looked at can be peeled by calling\n- * ref_iterator_peel(). This function is often faster than peel_ref(),\n- * so it should be preferred when iterating over references.\n- *\n- * Putting it all together, a typical iteration looks like this:\n- *\n- * int ok;\n- * struct ref_iterator *iter = ...;\n- *\n- * while ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n- * if (want_to_stop_iteration()) {\n- * ok = ITER_DONE;\n- * break;\n- * }\n- *\n- * // Access information about the current reference:\n- * if (!(iter->flags & REF_ISSYMREF))\n- * printf(\"%s is %s\\n\", iter->refname, oid_to_hex(iter->oid));\n- *\n- * // If you need to peel the reference:\n- * ref_iterator_peel(iter, &oid);\n- * }\n- *\n- * if (ok != ITER_DONE)\n- * handle_error();\n- * ref_iterator_free(iter);\n+ * Data structure for holding a reference iterator. See refs.h for\n+ * more details and usage instructions.\n */\n struct ref_iterator {\n \tstruct ref_iterator_vtable *vtable;\n@@ -337,42 +255,6 @@ struct ref_iterator {\n \tunsigned int flags;\n };\n \n-/*\n- * Advance the iterator to the first or next item and return ITER_OK.\n- * If the iteration is exhausted, free the resources associated with\n- * the ref_iterator and return ITER_DONE. On errors, free the iterator\n- * resources and return ITER_ERROR. It is a bug to use ref_iterator or\n- * call this function again after it has returned ITER_DONE or\n- * ITER_ERROR.\n- */\n-int ref_iterator_advance(struct ref_iterator *ref_iterator);\n-\n-/*\n- * Seek the iterator to the first reference with the given prefix.\n- * The prefix is matched as a literal string, without regard for path\n- * separators. If prefix is NULL or the empty string, seek the iterator to the\n- * first reference again.\n- *\n- * This function is expected to behave as if a new ref iterator with the same\n- * prefix had been created, but allows reuse of iterators and thus may allow\n- * the backend to optimize. Parameters other than the prefix that have been\n- * passed when creating the iterator will remain unchanged.\n- *\n- * Returns 0 on success, a negative error code otherwise.\n- */\n-int ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t const char *prefix);\n-\n-/*\n- * If possible, peel the reference currently being viewed by the\n- * iterator. Return 0 on success.\n- */\n-int ref_iterator_peel(struct ref_iterator *ref_iterator,\n-\t\t struct object_id *peeled);\n-\n-/* Free the reference iterator and any associated resources. */\n-void ref_iterator_free(struct ref_iterator *ref_iterator);\n-\n /*\n * An iterator over nothing (its first ref_iterator_advance() call\n * returns ITER_DONE).\n@@ -384,17 +266,6 @@ struct ref_iterator *empty_ref_iterator_begin(void);\n */\n int is_empty_ref_iterator(struct ref_iterator *ref_iterator);\n \n-/*\n- * Return an iterator that goes over each reference in `refs` for\n- * which the refname begins with prefix. If trim is non-zero, then\n- * trim that many characters off the beginning of each refname.\n- * The output is ordered by refname.\n- */\n-struct ref_iterator *refs_ref_iterator_begin(\n-\t\tstruct ref_store *refs,\n-\t\tconst char *prefix, const char **exclude_patterns,\n-\t\tint trim, enum do_for_each_ref_flags flags);\n-\n /*\n * A callback function used to instruct merge_ref_iterator how to\n * interleave the entries from iter0 and iter1. The function should\n@@ -520,18 +391,6 @@ struct ref_iterator_vtable {\n */\n extern struct ref_iterator *current_ref_iter;\n \n-/*\n- * The common backend for the for_each_*ref* functions. Call fn for\n- * each reference in iter. If the iterator itself ever returns\n- * ITER_ERROR, return -1. If fn ever returns a non-zero value, stop\n- * the iteration and return that value. Otherwise, return 0. In any\n- * case, free the iterator when done. This function is basically an\n- * adapter between the callback style of reference iteration and the\n- * iterator style.\n- */\n-int do_for_each_ref_iterator(struct ref_iterator *iter,\n-\t\t\t each_ref_fn fn, void *cb_data);\n-\n struct ref_store;\n \n /* refs backends */\n\n-- \n2.49.0\n\n"},{"id":"521829","messageId":"20250711-306-git-for-each-ref-pagination-v4-3-ed3303ad5b89@gmail.com","threadId":"63722","inReplyTo":"20250711-306-git-for-each-ref-pagination-v4-0-ed3303ad5b89@gmail.com","subject":"[PATCH v4 3/4] refs: selectively set prefix in the seek functions","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-11T16:18:39Z","receivedAt":"2025-07-11T16:18:47Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The ref iterator exposes a `ref_iterator_seek()` function. The name\nsuggests that this would seek the iterator to a specific reference in\nsome ways similar to how `fseek()` works for the filesystem.\n\nHowever, the function actually sets the prefix for refs iteration. So\nfurther iteration would only yield references which match the particular\nprefix. This is a bit confusing.\n\nLet's add a 'flags' field to the function, which when set with the\n'REF_ITERATOR_SEEK_SET_PREFIX' flag, will set the prefix for the\niteration in-line with the existing behavior. Otherwise, the reference\nbackends will simply seek to the specified reference and clears any\npreviously set prefix. This allows users to start iteration from a\nspecific reference.\n\nIn the packed and reftable backend, since references are available in a\nsorted list, the changes are simply setting the prefix if needed. The\nchanges on the files-backend are a little more involved, since the files\nbackend uses the 'ref-cache' mechanism. We move out the existing logic\nwithin `cache_ref_iterator_seek()` to `cache_ref_iterator_set_prefix()`\nwhich is called when the 'REF_ITERATOR_SEEK_SET_PREFIX' flag is set. We\nthen parse the provided seek string and set the required levels and\ntheir indexes to ensure that seeking is possible.\n\nHelped-by: Patrick Steinhardt \nSigned-off-by: Karthik Nayak \n---\n refs.c | 6 ++--\n refs.h | 26 +++++++++------\n refs/debug.c | 7 ++--\n refs/files-backend.c | 7 ++--\n refs/iterator.c | 26 ++++++++-------\n refs/packed-backend.c | 17 ++++++----\n refs/ref-cache.c | 85 ++++++++++++++++++++++++++++++++++++++++++++++---\n refs/refs-internal.h | 7 ++--\n refs/reftable-backend.c | 21 ++++++++----\n 9 files changed, 152 insertions(+), 50 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex dce5c49ca2..243e6898b8 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2666,12 +2666,12 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs\n \t\tif (!initial_transaction) {\n \t\t\tint ok;\n \n-\t\t\tif (!iter) {\n+\t\t\tif (!iter)\n \t\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n \t\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n-\t\t\t} else if (ref_iterator_seek(iter, dirname.buf) < 0) {\n+\t\t\telse if (ref_iterator_seek(iter, dirname.buf,\n+\t\t\t\t\t\t REF_ITERATOR_SEEK_SET_PREFIX) < 0)\n \t\t\t\tgoto cleanup;\n-\t\t\t}\n \n \t\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\t\t\tif (skip &&\ndiff --git a/refs.h b/refs.h\nindex 7c21aaef3d..e6780a8848 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -1299,21 +1299,29 @@ struct ref_iterator *refs_ref_iterator_begin(\n */\n int ref_iterator_advance(struct ref_iterator *ref_iterator);\n \n+enum ref_iterator_seek_flag {\n+\t/*\n+\t * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is\n+\t * updated to match the provided string, affecting all subsequent iterations. If\n+\t * not, the iterator seeks to the specified reference and clears any previously\n+\t * set prefix.\n+\t */\n+\tREF_ITERATOR_SEEK_SET_PREFIX = (1 << 0),\n+};\n+\n /*\n- * Seek the iterator to the first reference with the given prefix.\n- * The prefix is matched as a literal string, without regard for path\n- * separators. If prefix is NULL or the empty string, seek the iterator to the\n+ * Seek the iterator to the first reference matching the given seek string.\n+ * The seek string is matched as a literal string, without regard for path\n+ * separators. If seek is NULL or the empty string, seek the iterator to the\n * first reference again.\n *\n- * This function is expected to behave as if a new ref iterator with the same\n- * prefix had been created, but allows reuse of iterators and thus may allow\n- * the backend to optimize. Parameters other than the prefix that have been\n- * passed when creating the iterator will remain unchanged.\n+ * This function is expected to behave as if a new ref iterator has been\n+ * created, but allows reuse of existing iterators for optimization.\n *\n * Returns 0 on success, a negative error code otherwise.\n */\n-int ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t const char *prefix);\n+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *refname,\n+\t\t unsigned int flags);\n \n /*\n * If possible, peel the reference currently being viewed by the\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 485e3079d7..da300efaf3 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -170,12 +170,13 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *refname, unsigned int flags)\n {\n \tstruct debug_ref_iterator *diter =\n \t\t(struct debug_ref_iterator *)ref_iterator;\n-\tint res = diter->iter->vtable->seek(diter->iter, prefix);\n-\ttrace_printf_key(&trace_refs, \"iterator_seek: %s: %d\\n\", prefix ? prefix : \"\", res);\n+\tint res = diter->iter->vtable->seek(diter->iter, refname, flags);\n+\ttrace_printf_key(&trace_refs, \"iterator_seek: %s flags: %d: %d\\n\",\n+\t\t\t refname ? refname : \"\", flags, res);\n \treturn res;\n }\n \ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex bf6f89b1d1..8b282f2a60 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -929,11 +929,11 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int files_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *refname, unsigned int flags)\n {\n \tstruct files_ref_iterator *iter =\n \t\t(struct files_ref_iterator *)ref_iterator;\n-\treturn ref_iterator_seek(iter->iter0, prefix);\n+\treturn ref_iterator_seek(iter->iter0, refname, flags);\n }\n \n static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n@@ -2316,7 +2316,8 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n-\t\t\t\t const char *prefix UNUSED)\n+\t\t\t\t const char *refname UNUSED,\n+\t\t\t\t unsigned int flags UNUSED)\n {\n \tBUG(\"ref_iterator_seek() called for reflog_iterator\");\n }\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex 766d96e795..17ef841d8a 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -15,10 +15,10 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ref_iterator->vtable->advance(ref_iterator);\n }\n \n-int ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t const char *prefix)\n+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *refname,\n+\t\t unsigned int flags)\n {\n-\treturn ref_iterator->vtable->seek(ref_iterator, prefix);\n+\treturn ref_iterator->vtable->seek(ref_iterator, refname, flags);\n }\n \n int ref_iterator_peel(struct ref_iterator *ref_iterator,\n@@ -57,7 +57,8 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)\n }\n \n static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n-\t\t\t\t const char *prefix UNUSED)\n+\t\t\t\t const char *refname UNUSED,\n+\t\t\t\t unsigned int flags UNUSED)\n {\n \treturn 0;\n }\n@@ -224,7 +225,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *refname, unsigned int flags)\n {\n \tstruct merge_ref_iterator *iter =\n \t\t(struct merge_ref_iterator *)ref_iterator;\n@@ -234,11 +235,11 @@ static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,\n \titer->iter0 = iter->iter0_owned;\n \titer->iter1 = iter->iter1_owned;\n \n-\tret = ref_iterator_seek(iter->iter0, prefix);\n+\tret = ref_iterator_seek(iter->iter0, refname, flags);\n \tif (ret < 0)\n \t\treturn ret;\n \n-\tret = ref_iterator_seek(iter->iter1, prefix);\n+\tret = ref_iterator_seek(iter->iter1, refname, flags);\n \tif (ret < 0)\n \t\treturn ret;\n \n@@ -407,13 +408,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *refname, unsigned int flags)\n {\n \tstruct prefix_ref_iterator *iter =\n \t\t(struct prefix_ref_iterator *)ref_iterator;\n-\tfree(iter->prefix);\n-\titer->prefix = xstrdup_or_null(prefix);\n-\treturn ref_iterator_seek(iter->iter0, prefix);\n+\n+\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX) {\n+\t\tfree(iter->prefix);\n+\t\titer->prefix = xstrdup_or_null(refname);\n+\t}\n+\treturn ref_iterator_seek(iter->iter0, refname, flags);\n }\n \n static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 7fd73a0e6d..5fa4ae6655 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1004,19 +1004,23 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *refname, unsigned int flags)\n {\n \tstruct packed_ref_iterator *iter =\n \t\t(struct packed_ref_iterator *)ref_iterator;\n \tconst char *start;\n \n-\tif (prefix && *prefix)\n-\t\tstart = find_reference_location(iter->snapshot, prefix, 0);\n+\tif (refname && *refname)\n+\t\tstart = find_reference_location(iter->snapshot, refname, 0);\n \telse\n \t\tstart = iter->snapshot->start;\n \n-\tfree(iter->prefix);\n-\titer->prefix = xstrdup_or_null(prefix);\n+\t/* Unset any previously set prefix */\n+\tFREE_AND_NULL(iter->prefix);\n+\n+\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX)\n+\t\titer->prefix = xstrdup_or_null(refname);\n+\n \titer->pos = start;\n \titer->eof = iter->snapshot->eof;\n \n@@ -1194,7 +1198,8 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \titer->repo = ref_store->repo;\n \titer->flags = flags;\n \n-\tif (packed_ref_iterator_seek(&iter->base, prefix) < 0) {\n+\tif (packed_ref_iterator_seek(&iter->base, prefix,\n+\t\t\t\t REF_ITERATOR_SEEK_SET_PREFIX) < 0) {\n \t\tref_iterator_free(&iter->base);\n \t\treturn NULL;\n \t}\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 8aaffa8c6b..1d95b56d40 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -434,11 +434,9 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t}\n }\n \n-static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+static int cache_ref_iterator_set_prefix(struct cache_ref_iterator *iter,\n+\t\t\t\t\t const char *prefix)\n {\n-\tstruct cache_ref_iterator *iter =\n-\t\t(struct cache_ref_iterator *)ref_iterator;\n \tstruct cache_ref_iterator_level *level;\n \tstruct ref_dir *dir;\n \n@@ -469,6 +467,82 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n \treturn 0;\n }\n \n+static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *refname, unsigned int flags)\n+{\n+\tstruct cache_ref_iterator *iter =\n+\t\t(struct cache_ref_iterator *)ref_iterator;\n+\n+\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX) {\n+\t\treturn cache_ref_iterator_set_prefix(iter, refname);\n+\t} else if (refname && *refname) {\n+\t\tstruct cache_ref_iterator_level *level;\n+\t\tconst char *slash = refname;\n+\t\tstruct ref_dir *dir;\n+\n+\t\tdir = get_ref_dir(iter->cache->root);\n+\n+\t\tif (iter->prime_dir)\n+\t\t\tprime_ref_dir(dir, refname);\n+\n+\t\titer->levels_nr = 1;\n+\t\tlevel = &iter->levels[0];\n+\t\tlevel->index = -1;\n+\t\tlevel->dir = dir;\n+\n+\t\t/* Unset any previously set prefix */\n+\t\tFREE_AND_NULL(iter->prefix);\n+\n+\t\t/*\n+\t\t * Breakdown the provided seek path and assign the correct\n+\t\t * indexing to each level as needed.\n+\t\t */\n+\t\tdo {\n+\t\t\tint len, idx;\n+\t\t\tint cmp = 0;\n+\n+\t\t\tsort_ref_dir(dir);\n+\n+\t\t\tslash = strchr(slash, '/');\n+\t\t\tlen = slash ? slash - refname : (int)strlen(refname);\n+\n+\t\t\tfor (idx = 0; idx < dir->nr; idx++) {\n+\t\t\t\tcmp = strncmp(refname, dir->entries[idx]->name, len);\n+\t\t\t\tif (cmp <= 0)\n+\t\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\t/* don't overflow the index */\n+\t\t\tidx = idx >= dir->nr ? dir->nr - 1 : idx;\n+\n+\t\t\tif (slash)\n+\t\t\t\tslash = slash + 1;\n+\n+\t\t\tlevel->index = idx;\n+\t\t\tif (dir->entries[idx]->flag & REF_DIR) {\n+\t\t\t\t/* push down a level */\n+\t\t\t\tdir = get_ref_dir(dir->entries[idx]);\n+\n+\t\t\t\tALLOC_GROW(iter->levels, iter->levels_nr + 1,\n+\t\t\t\t\t iter->levels_alloc);\n+\t\t\t\tlevel = &iter->levels[iter->levels_nr++];\n+\t\t\t\tlevel->dir = dir;\n+\t\t\t\tlevel->index = -1;\n+\t\t\t} else {\n+\t\t\t\t/* reduce the index so the leaf node is iterated over */\n+\t\t\t\tif (cmp <= 0 && !slash)\n+\t\t\t\t\tlevel->index = idx - 1;\n+\t\t\t\t/*\n+\t\t\t\t * while the seek path may not be exhausted, our\n+\t\t\t\t * match is exhausted at a leaf node.\n+\t\t\t\t */\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t} while (slash);\n+\t}\n+\n+\treturn 0;\n+}\n+\n static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -509,7 +583,8 @@ struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\n \titer->cache = cache;\n \titer->prime_dir = prime_dir;\n \n-\tif (cache_ref_iterator_seek(&iter->base, prefix) < 0) {\n+\tif (cache_ref_iterator_seek(&iter->base, prefix,\n+\t\t\t\t REF_ITERATOR_SEEK_SET_PREFIX) < 0) {\n \t\tref_iterator_free(&iter->base);\n \t\treturn NULL;\n \t}\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 03f5df04d5..90de7837f8 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -353,11 +353,12 @@ void base_ref_iterator_init(struct ref_iterator *iter,\n typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);\n \n /*\n- * Seek the iterator to the first reference matching the given prefix. Should\n- * behave the same as if a new iterator was created with the same prefix.\n+ * Seek the iterator to the first matching reference. If set_prefix is set,\n+ * it would behave the same as if a new iterator was created with the same\n+ * prefix.\n */\n typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix);\n+\t\t\t\t const char *refname, unsigned int flags);\n \n /*\n * Peels the current ref, returning 0 for success or -1 for failure.\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 4c3817f4ec..c3d48cc412 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -719,15 +719,20 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *refname, unsigned int flags)\n {\n \tstruct reftable_ref_iterator *iter =\n \t\t(struct reftable_ref_iterator *)ref_iterator;\n \n-\tfree(iter->prefix);\n-\titer->prefix = xstrdup_or_null(prefix);\n-\titer->prefix_len = prefix ? strlen(prefix) : 0;\n-\titer->err = reftable_iterator_seek_ref(&iter->iter, prefix);\n+\t/* Unset any previously set prefix */\n+\tFREE_AND_NULL(iter->prefix);\n+\titer->prefix_len = 0;\n+\n+\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX) {\n+\t\titer->prefix = xstrdup_or_null(refname);\n+\t\titer->prefix_len = refname ? strlen(refname) : 0;\n+\t}\n+\titer->err = reftable_iterator_seek_ref(&iter->iter, refname);\n \n \treturn iter->err;\n }\n@@ -839,7 +844,8 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_\n \tif (ret)\n \t\tgoto done;\n \n-\tret = reftable_ref_iterator_seek(&iter->base, prefix);\n+\tret = reftable_ref_iterator_seek(&iter->base, prefix,\n+\t\t\t\t\t REF_ITERATOR_SEEK_SET_PREFIX);\n \tif (ret)\n \t\tgoto done;\n \n@@ -2042,7 +2048,8 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n-\t\t\t\t\t const char *prefix UNUSED)\n+\t\t\t\t\t const char *refname UNUSED,\n+\t\t\t\t\t unsigned int flags UNUSED)\n {\n \tBUG(\"reftable reflog iterator cannot be seeked\");\n \treturn -1;\n\n-- \n2.49.0\n\n"},{"id":"521830","messageId":"20250711-306-git-for-each-ref-pagination-v4-4-ed3303ad5b89@gmail.com","threadId":"63722","inReplyTo":"20250711-306-git-for-each-ref-pagination-v4-0-ed3303ad5b89@gmail.com","subject":"[PATCH v4 4/4] for-each-ref: introduce a '--start-after' option","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-11T16:18:40Z","receivedAt":"2025-07-11T16:18:48Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `git-for-each-ref(1)` command is used to iterate over references\npresent in a repository. In large repositories with millions of\nreferences, it would be optimal to paginate this output such that we\ncan start iteration from a given reference. This would avoid having to\niterate over all references from the beginning each time when paginating\nthrough results.\n\nThe previous commit added 'seek' functionality to the reference\nbackends. Utilize this and expose a '--start-after' option in\n'git-for-each-ref(1)'. When used, the reference iteration seeks to the\nlexicographically next reference and iterates from there onward.\n\nThis enables efficient pagination workflows, where the calling script\ncan remember the last provided reference and use that as the starting\npoint for the next set of references:\n git for-each-ref --count=100\n git for-each-ref --count=100 --start-after=refs/heads/branch-100\n git for-each-ref --count=100 --start-after=refs/heads/branch-200\n\nSince the reference iterators only allow seeking to a specified marker\nvia the `ref_iterator_seek()`, we introduce a helper function\n`start_ref_iterator_after()`, which seeks to next reference by simply\nadding (char) 1 to the marker.\n\nWe must note that pagination always continues from the provided marker,\nas such any concurrent reference updates lexicographically behind the\nmarker will not be output. Document the same.\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/git-for-each-ref.adoc | 10 +-\n builtin/for-each-ref.c | 8 ++\n ref-filter.c | 80 +++++++++++----\n ref-filter.h | 1 +\n t/t6302-for-each-ref-filter.sh | 194 ++++++++++++++++++++++++++++++++++++\n 5 files changed, 272 insertions(+), 21 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.adoc b/Documentation/git-for-each-ref.adoc\nindex 5ef89fc0fe..ae61ba642a 100644\n--- a/Documentation/git-for-each-ref.adoc\n+++ b/Documentation/git-for-each-ref.adoc\n@@ -14,7 +14,7 @@ SYNOPSIS\n \t\t [--points-at=]\n \t\t [--merged[=]] [--no-merged[=]]\n \t\t [--contains[=]] [--no-contains[=]]\n-\t\t [--exclude= ...]\n+\t\t [--exclude= ...] [--start-after=]\n \n DESCRIPTION\n -----------\n@@ -108,6 +108,14 @@ TAB %(refname)`.\n --include-root-refs::\n \tList root refs (HEAD and pseudorefs) apart from regular refs.\n \n+--start-after=::\n+ Allows paginating the output by skipping references up to and including the\n+ specified marker. When paging, it should be noted that references may be\n+ deleted, modified or added between invocations. Output will only yield those\n+ references which follow the marker lexicographically. Output begins from the\n+ first reference that would come after the marker alphabetically. Cannot be\n+ used with general pattern matching or custom sort options.\n+\n FIELD NAMES\n -----------\n \ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 3d2207ec77..3f21598046 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -13,6 +13,7 @@ static char const * const for_each_ref_usage[] = {\n \tN_(\"git for-each-ref [--points-at ]\"),\n \tN_(\"git for-each-ref [--merged []] [--no-merged []]\"),\n \tN_(\"git for-each-ref [--contains []] [--no-contains []]\"),\n+\tN_(\"git for-each-ref [--start-after ]\"),\n \tNULL\n };\n \n@@ -44,6 +45,7 @@ int cmd_for_each_ref(int argc,\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_INTEGER( 0 , \"count\", &format.array_opts.max_count, N_(\"show only matched refs\")),\n \t\tOPT_STRING( 0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n+\t\tOPT_STRING( 0 , \"start-after\", &filter.start_after, N_(\"start-start\"), N_(\"start iteration after the provided marker\")),\n \t\tOPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n \t\tOPT_REF_FILTER_EXCLUDE(&filter),\n \t\tOPT_REF_SORT(&sorting_options),\n@@ -79,6 +81,9 @@ int cmd_for_each_ref(int argc,\n \tif (verify_ref_format(&format))\n \t\tusage_with_options(for_each_ref_usage, opts);\n \n+\tif (filter.start_after && sorting_options.nr > 1)\n+\t\tdie(_(\"cannot use --start-after with custom sort options\"));\n+\n \tsorting = ref_sorting_options(&sorting_options);\n \tref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);\n \tfilter.ignore_case = icase;\n@@ -100,6 +105,9 @@ int cmd_for_each_ref(int argc,\n \t\tfilter.name_patterns = argv;\n \t}\n \n+\tif (filter.start_after && filter.name_patterns && filter.name_patterns[0])\n+\t\tdie(_(\"cannot use --start-after with patterns\"));\n+\n \tif (include_root_refs)\n \t\tflags |= FILTER_REFS_ROOT_REFS | FILTER_REFS_DETACHED_HEAD;\n \ndiff --git a/ref-filter.c b/ref-filter.c\nindex 7a274633cf..2dfd385313 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2683,6 +2683,24 @@ static int filter_exclude_match(struct ref_filter *filter, const char *refname)\n \treturn match_pattern(filter->exclude.v, refname, filter->ignore_case);\n }\n \n+/*\n+ * We need to seek to the reference right after a given marker but excluding any\n+ * matching references. So we seek to the lexicographically next reference.\n+ */\n+static int start_ref_iterator_after(struct ref_iterator *iter, const char *marker)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret;\n+\n+\tstrbuf_addstr(&sb, marker);\n+\tstrbuf_addch(&sb, 1);\n+\n+\tret = ref_iterator_seek(iter, sb.buf, 0);\n+\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n /*\n * This is the same as for_each_fullref_in(), but it tries to iterate\n * only over the patterns we'll care about. Note that it _doesn't_ do a full\n@@ -2692,10 +2710,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t\t\t each_ref_fn cb,\n \t\t\t\t void *cb_data)\n {\n+\tstruct ref_iterator *iter;\n+\tint flags = 0, ret = 0;\n+\n \tif (filter->kind & FILTER_REFS_ROOT_REFS) {\n \t\t/* In this case, we want to print all refs including root refs. */\n-\t\treturn refs_for_each_include_root_refs(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t cb, cb_data);\n+\t\tflags |= DO_FOR_EACH_INCLUDE_ROOT_REFS;\n+\t\tgoto non_prefix_iter;\n \t}\n \n \tif (!filter->match_as_path) {\n@@ -2704,8 +2725,7 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t * prefixes like \"refs/heads/\" etc. are stripped off,\n \t\t * so we have to look at everything:\n \t\t */\n-\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t\"\", NULL, cb, cb_data);\n+\t\tgoto non_prefix_iter;\n \t}\n \n \tif (filter->ignore_case) {\n@@ -2714,20 +2734,29 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t * so just return everything and let the caller\n \t\t * sort it out.\n \t\t */\n-\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t\"\", NULL, cb, cb_data);\n+\t\tgoto non_prefix_iter;\n \t}\n \n \tif (!filter->name_patterns[0]) {\n \t\t/* no patterns; we have to look at everything */\n-\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"\", filter->exclude.v, cb, cb_data);\n+\t\tgoto non_prefix_iter;\n \t}\n \n \treturn refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n \t\t\t\t\t\t NULL, filter->name_patterns,\n \t\t\t\t\t\t filter->exclude.v,\n \t\t\t\t\t\t cb, cb_data);\n+\n+non_prefix_iter:\n+\titer = refs_ref_iterator_begin(get_main_ref_store(the_repository), \"\",\n+\t\t\t\t NULL, 0, flags);\n+\tif (filter->start_after)\n+\t\tret = start_ref_iterator_after(iter, filter->start_after);\n+\n+\tif (ret)\n+\t\treturn ret;\n+\n+\treturn do_for_each_ref_iterator(iter, cb, cb_data);\n }\n \n /*\n@@ -3197,9 +3226,11 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n \tinit_contains_cache(&filter->internal.no_contains_cache);\n \n \t/* Simple per-ref filtering */\n-\tif (!filter->kind)\n+\tif (!filter->kind) {\n \t\tdie(\"filter_refs: invalid type\");\n-\telse {\n+\t} else {\n+\t\tconst char *prefix = NULL;\n+\n \t\t/*\n \t\t * For common cases where we need only branches or remotes or tags,\n \t\t * we only iterate through those refs. If a mix of refs is needed,\n@@ -3207,19 +3238,28 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n \t\t * of filter_ref_kind().\n \t\t */\n \t\tif (filter->kind == FILTER_REFS_BRANCHES)\n-\t\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"refs/heads/\", NULL,\n-\t\t\t\t\t\t fn, cb_data);\n+\t\t\tprefix = \"refs/heads/\";\n \t\telse if (filter->kind == FILTER_REFS_REMOTES)\n-\t\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"refs/remotes/\", NULL,\n-\t\t\t\t\t\t fn, cb_data);\n+\t\t\tprefix = \"refs/remotes/\";\n \t\telse if (filter->kind == FILTER_REFS_TAGS)\n-\t\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"refs/tags/\", NULL, fn,\n-\t\t\t\t\t\t cb_data);\n-\t\telse if (filter->kind & FILTER_REFS_REGULAR)\n+\t\t\tprefix = \"refs/tags/\";\n+\n+\t\tif (prefix) {\n+\t\t\tstruct ref_iterator *iter;\n+\n+\t\t\titer = refs_ref_iterator_begin(get_main_ref_store(the_repository),\n+\t\t\t\t\t\t \"\", NULL, 0, 0);\n+\n+\t\t\tif (filter->start_after)\n+\t\t\t\tret = start_ref_iterator_after(iter, filter->start_after);\n+\t\t\telse if (prefix)\n+\t\t\t\tret = ref_iterator_seek(iter, prefix, 1);\n+\n+\t\t\tif (!ret)\n+\t\t\t\tret = do_for_each_ref_iterator(iter, fn, cb_data);\n+\t\t} else if (filter->kind & FILTER_REFS_REGULAR) {\n \t\t\tret = for_each_fullref_in_pattern(filter, fn, cb_data);\n+\t\t}\n \n \t\t/*\n \t\t * When printing all ref types, HEAD is already included,\ndiff --git a/ref-filter.h b/ref-filter.h\nindex c98c4fbd4c..f22ca94b49 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -64,6 +64,7 @@ struct ref_array {\n \n struct ref_filter {\n \tconst char **name_patterns;\n+\tconst char *start_after;\n \tstruct strvec exclude;\n \tstruct oid_array points_at;\n \tstruct commit_list *with_commit;\ndiff --git a/t/t6302-for-each-ref-filter.sh b/t/t6302-for-each-ref-filter.sh\nindex bb02b86c16..a43e099118 100755\n--- a/t/t6302-for-each-ref-filter.sh\n+++ b/t/t6302-for-each-ref-filter.sh\n@@ -541,4 +541,198 @@ test_expect_success 'validate worktree atom' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'start after with empty value' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/main\n+\trefs/heads/main_worktree\n+\trefs/heads/side\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=\"\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after a specific reference' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/spot >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after a specific reference with partial match' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/sp >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after, just behind a specific reference' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/parrot >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after with specific directory match' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after with specific directory and trailing slash' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/lost >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after, just behind a specific directory' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after, overflow specific reference length' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/spotnew >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after, overflow specific reference path' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/spot/new >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after, last reference' '\n+\tcat >expect <<-\\EOF &&\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/tags/two >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after used with a pattern' '\n+\tcat >expect <<-\\EOF &&\n+\tfatal: cannot use --start-after with patterns\n+\tEOF\n+\ttest_must_fail git for-each-ref --format=\"%(refname)\" --start-after=refs/odd/spot refs/tags 2>actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after used with custom sort order' '\n+\tcat >expect <<-\\EOF &&\n+\tfatal: cannot use --start-after with custom sort options\n+\tEOF\n+\ttest_must_fail git for-each-ref --format=\"%(refname)\" --start-after=refs/odd/spot --sort=author 2>actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n\n-- \n2.49.0\n\n"},{"id":"521892","messageId":"CAP8UFD06KKF0A0WjBSw77A7fKp_pGJGVyw-OPd8-X92h=RSF+w@mail.gmail.com","threadId":"63722","inReplyTo":"20250711-306-git-for-each-ref-pagination-v4-3-ed3303ad5b89@gmail.com","subject":"Re: [PATCH v4 3/4] refs: selectively set prefix in the seek functions","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-07-14T10:34:17Z","receivedAt":"2025-07-14T10:34:32Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Jul 11, 2025 at 6:20 PM Karthik Nayak wrote:\n\n> diff --git a/refs/refs-internal.h b/refs/refs-internal.h\n> index 03f5df04d5..90de7837f8 100644\n> --- a/refs/refs-internal.h\n> +++ b/refs/refs-internal.h\n> @@ -353,11 +353,12 @@ void base_ref_iterator_init(struct ref_iterator *iter,\n> typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);\n>\n> /*\n> - * Seek the iterator to the first reference matching the given prefix. Should\n> - * behave the same as if a new iterator was created with the same prefix.\n> + * Seek the iterator to the first matching reference. If set_prefix is set,\n\ns/If set_prefix is set/If the REF_ITERATOR_SEEK_SET_PREFIX flag is set/\n\n> + * it would behave the same as if a new iterator was created with the same\n> + * prefix.\n\nMaybe: s/with the same prefix/at the same reference/\n\n> */\n> typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,\n> - const char *prefix);\n> + const char *refname, unsigned int flags);\n"},{"id":"521901","messageId":"xmqqecuivkqh.fsf@gitster.g","threadId":"63722","inReplyTo":"20250701-306-git-for-each-ref-pagination-v1-2-4f0ae7c0688f@gmail.com","subject":"Re: [PATCH 2/4] ref-cache: remove unused function 'find_ref_entry()'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-14T15:46:14Z","receivedAt":"2025-07-14T15:46:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> The 'find_ref_entry' function is no longer used, so remove it.\n\nIt seems that ba1c052f (ref_store: implement `refs_peel_ref()`\ngenerically, 2017-09-25) removed the last caller of it. This is\nlong overdue ;-)\n\n> Signed-off-by: Karthik Nayak \n> ---\n> refs/ref-cache.c | 14 --------------\n> refs/ref-cache.h | 7 -------\n> 2 files changed, 21 deletions(-)\n>\n> diff --git a/refs/ref-cache.c b/refs/ref-cache.c\n> index c1f1bab1d5..8aaffa8c6b 100644\n> --- a/refs/ref-cache.c\n> +++ b/refs/ref-cache.c\n> @@ -194,20 +194,6 @@ static struct ref_dir *find_containing_dir(struct ref_dir *dir,\n> \treturn dir;\n> }\n> \n> -struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname)\n> -{\n> -\tint entry_index;\n> -\tstruct ref_entry *entry;\n> -\tdir = find_containing_dir(dir, refname);\n> -\tif (!dir)\n> -\t\treturn NULL;\n> -\tentry_index = search_ref_dir(dir, refname, strlen(refname));\n> -\tif (entry_index == -1)\n> -\t\treturn NULL;\n> -\tentry = dir->entries[entry_index];\n> -\treturn (entry->flag & REF_DIR) ? NULL : entry;\n> -}\n> -\n> /*\n> * Emit a warning and return true iff ref1 and ref2 have the same name\n> * and the same oid. Die if they have the same name but different\n> diff --git a/refs/ref-cache.h b/refs/ref-cache.h\n> index 5f04e518c3..f635d2d824 100644\n> --- a/refs/ref-cache.h\n> +++ b/refs/ref-cache.h\n> @@ -201,13 +201,6 @@ void free_ref_cache(struct ref_cache *cache);\n> */\n> void add_entry_to_dir(struct ref_dir *dir, struct ref_entry *entry);\n> \n> -/*\n> - * Find the value entry with the given name in dir, sorting ref_dirs\n> - * and recursing into subdirectories as necessary. If the name is not\n> - * found or it corresponds to a directory entry, return NULL.\n> - */\n> -struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname);\n> -\n> /*\n> * Start iterating over references in `cache`. If `prefix` is\n> * specified, only include references whose names start with that\n"},{"id":"521902","messageId":"CAP8UFD1wRxZDCRC76VuuA8_rpNn__TQnL9RnNumCE33wAjSrMQ@mail.gmail.com","threadId":"63722","inReplyTo":"20250711-306-git-for-each-ref-pagination-v4-4-ed3303ad5b89@gmail.com","subject":"Re: [PATCH v4 4/4] for-each-ref: introduce a '--start-after' option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-07-14T16:04:21Z","receivedAt":"2025-07-14T16:04:36Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Jul 11, 2025 at 6:21 PM Karthik Nayak wrote:\n\n> /*\n> * This is the same as for_each_fullref_in(), but it tries to iterate\n> * only over the patterns we'll care about. Note that it _doesn't_ do a full\n> @@ -2692,10 +2710,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n> each_ref_fn cb,\n> void *cb_data)\n> {\n> + struct ref_iterator *iter;\n> + int flags = 0, ret = 0;\n> +\n> if (filter->kind & FILTER_REFS_ROOT_REFS) {\n> /* In this case, we want to print all refs including root refs. */\n> - return refs_for_each_include_root_refs(get_main_ref_store(the_repository),\n> - cb, cb_data);\n> + flags |= DO_FOR_EACH_INCLUDE_ROOT_REFS;\n> + goto non_prefix_iter;\n> }\n>\n> if (!filter->match_as_path) {\n> @@ -2704,8 +2725,7 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n> * prefixes like \"refs/heads/\" etc. are stripped off,\n> * so we have to look at everything:\n> */\n> - return refs_for_each_fullref_in(get_main_ref_store(the_repository),\n> - \"\", NULL, cb, cb_data);\n> + goto non_prefix_iter;\n> }\n>\n> if (filter->ignore_case) {\n> @@ -2714,20 +2734,29 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n> * so just return everything and let the caller\n> * sort it out.\n> */\n> - return refs_for_each_fullref_in(get_main_ref_store(the_repository),\n> - \"\", NULL, cb, cb_data);\n> + goto non_prefix_iter;\n> }\n>\n> if (!filter->name_patterns[0]) {\n> /* no patterns; we have to look at everything */\n> - return refs_for_each_fullref_in(get_main_ref_store(the_repository),\n> - \"\", filter->exclude.v, cb, cb_data);\n> + goto non_prefix_iter;\n> }\n>\n> return refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n> NULL, filter->name_patterns,\n> filter->exclude.v,\n> cb, cb_data);\n> +\n> +non_prefix_iter:\n> + iter = refs_ref_iterator_begin(get_main_ref_store(the_repository), \"\",\n> + NULL, 0, flags);\n> + if (filter->start_after)\n> + ret = start_ref_iterator_after(iter, filter->start_after);\n> +\n> + if (ret)\n> + return ret;\n> +\n> + return do_for_each_ref_iterator(iter, cb, cb_data);\n> }\n\nNit: I wonder if what is under the 'non_prefix_iter' label could be in\na new function and instead of `goto non_prefix_iter` we could return\nthe result of the new function.\n\n> /*\n> @@ -3197,9 +3226,11 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n> init_contains_cache(&filter->internal.no_contains_cache);\n>\n> /* Simple per-ref filtering */\n> - if (!filter->kind)\n> + if (!filter->kind) {\n> die(\"filter_refs: invalid type\");\n> - else {\n> + } else {\n\nNit: the `else` could be removed altogether here, but maybe that\nshould be done in a preparatory patch.\n\n> + const char *prefix = NULL;\n> +\n\n[...]\n\n> +test_expect_success 'start after with specific directory and trailing slash' '\n> + cat >expect <<-\\EOF &&\n> + refs/odd/spot\n> + refs/tags/annotated-tag\n> + refs/tags/doubly-annotated-tag\n> + refs/tags/doubly-signed-tag\n> + refs/tags/foo1.10\n> + refs/tags/foo1.3\n> + refs/tags/foo1.6\n> + refs/tags/four\n> + refs/tags/one\n> + refs/tags/signed-tag\n> + refs/tags/three\n> + refs/tags/two\n> + EOF\n> + git for-each-ref --format=\"%(refname)\" --start-after=refs/lost >actual &&\n\nI don't see a trailing slash.\n\n> + test_cmp expect actual\n> +'\n> +\n> +test_expect_success 'start after, just behind a specific directory' '\n> + cat >expect <<-\\EOF &&\n> + refs/odd/spot\n> + refs/tags/annotated-tag\n> + refs/tags/doubly-annotated-tag\n> + refs/tags/doubly-signed-tag\n> + refs/tags/foo1.10\n> + refs/tags/foo1.3\n> + refs/tags/foo1.6\n> + refs/tags/four\n> + refs/tags/one\n> + refs/tags/signed-tag\n> + refs/tags/three\n> + refs/tags/two\n> + EOF\n> + git for-each-ref --format=\"%(refname)\" --start-after=refs/odd/ >actual &&\n\nHere there is a trailing slash though.\n\n> + test_cmp expect actual\n> +'\n"},{"id":"521903","messageId":"xmqq8qkqvjnu.fsf@gitster.g","threadId":"63722","inReplyTo":"aG9hMP9gEFLhVgJL@pks.im","subject":"Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-14T16:09:25Z","receivedAt":"2025-07-14T16:09:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> + * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is\n>> + * updated to match the seek string, affecting all subsequent iterations. If\n>> + * not, the iterator seeks to the specified reference and clears any previously\n>> + * set prefix.\n>> *\n>> * Returns 0 on success, a negative error code otherwise.\n>> */\n>> -int ref_iterator_seek(struct ref_iterator *ref_iterator,\n>> -\t\t const char *prefix);\n>> +int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,\n>> +\t\t unsigned int flags);\n>\n> Another tiny nit: instead of calling the variable `seek` we can just\n> call it `refname`. That might give a bit more of a hint what you're\n> actually seeking for.\n>\n> But other than that I'm happy with the new behaviour, where we are now\n> consistently either setting or resetting the prefix depending on whether\n> or not the caller set the flag.\n\nI am not sure. The way the \"prefix\" is used, if I understand correctly, is\n\n - it is set by iterator-begin, typically to the area to iterate\n over (e.g. \"refs/heads/\" for iterating over branches) in the\n for_each_ref_*() family of helpers, and internally we seek to\n that area (skipping anything that come strictly before\n \"refs/heads/\" for example).\n\n - iterator-advance looks at it and decides we are done when the\n iterator points beyond that prefix\n\nSo if you are iterating inside \"refs/heads/\" hierarchy and seek to\n\"refs/heads/m\", don't you still want to stop when you step outside\n\"refs/heads/\" by keeping the original prefix, instead of unsetting\nthe prefix to empty? A postimage of this patch for packed backend\n(picked at random) reads like this:\n\n static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,\n const char *seek, unsigned int flags)\n {\n struct packed_ref_iterator *iter =\n (struct packed_ref_iterator *)ref_iterator;\n const char *start;\n\n if (seek && *seek)\n start = find_reference_location(iter->snapshot, seek, 0);\n else\n start = iter->snapshot->start;\n\n /* Unset any previously set prefix */\n FREE_AND_NULL(iter->prefix);\n\n if (flags & REF_ITERATOR_SEEK_SET_PREFIX)\n iter->prefix = xstrdup_or_null(seek);\n\nso after (true) seeking that does not have the SET_PREFIX flag on,\nwouldn't our iterator-advance run through the end since it no longer\nis aware of where to stop?\n\nThanks.\n"},{"id":"521905","messageId":"CAP8UFD0anB4dPDf99-w6dyZNNGuOKU-b30_k322D0EsWE0cKPQ@mail.gmail.com","threadId":"63722","inReplyTo":"20250711-306-git-for-each-ref-pagination-v4-0-ed3303ad5b89@gmail.com","subject":"Re: [PATCH v4 0/4] for-each-ref: introduce seeking functionality via '--start-after'","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-07-14T16:34:58Z","receivedAt":"2025-07-14T16:35:12Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, Jul 11, 2025 at 6:20 PM Karthik Nayak wrote:\n\n> Initially I was also planning to cleanup all the `refs_for_each...()`\n> functions in 'refs.h' by simply using the iterator, but this bloated the\n> series. So I've left that for another day.\n\nI wonder if there is a plan to add the '--start-after' option to `git\nbranch` and `git tag` too?\n\n> Karthik Nayak (4):\n> refs: expose `ref_iterator` via 'refs.h'\n> ref-cache: remove unused function 'find_ref_entry()'\n> refs: selectively set prefix in the seek functions\n> for-each-ref: introduce a '--start-after' option\n\nExcept for the few small comments I left on the two last patches and\none below, this looks good to me.\n\n[ ... ]\n\n> Range-diff versus v3:\n\n[ ... ]\n\n> + struct cache_ref_iterator *iter =\n> + (struct cache_ref_iterator *)ref_iterator;\n> +\n> + if (flags & REF_ITERATOR_SEEK_SET_PREFIX) {\n> -+ return cache_ref_iterator_set_prefix(iter, seek);\n> -+ } else if (seek && *seek) {\n> ++ return cache_ref_iterator_set_prefix(iter, refname);\n> ++ } else if (refname && *refname) {\n\nNit: the `else` here could be removed, but yeah it might be better to\ndo that in a preparatory patch.\n\n> + struct cache_ref_iterator_level *level;\n> -+ const char *slash = seek;\n> ++ const char *slash = refname;\n> + struct ref_dir *dir;\n"},{"id":"521907","messageId":"xmqqqzyiu3k6.fsf@gitster.g","threadId":"63722","inReplyTo":"CAP8UFD1wRxZDCRC76VuuA8_rpNn__TQnL9RnNumCE33wAjSrMQ@mail.gmail.com","subject":"Re: [PATCH v4 4/4] for-each-ref: introduce a '--start-after' option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-14T16:42:33Z","receivedAt":"2025-07-14T16:42:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n>> /* Simple per-ref filtering */\n>> - if (!filter->kind)\n>> + if (!filter->kind) {\n>> die(\"filter_refs: invalid type\");\n>> - else {\n>> + } else {\n>\n> Nit: the `else` could be removed altogether here, but maybe that\n> should be done in a preparatory patch.\n\nGood eyes. Thanks for carefully reading it over.\n"},{"id":"521910","messageId":"xmqqms96u38z.fsf@gitster.g","threadId":"63722","inReplyTo":"CAP8UFD0anB4dPDf99-w6dyZNNGuOKU-b30_k322D0EsWE0cKPQ@mail.gmail.com","subject":"Re: [PATCH v4 0/4] for-each-ref: introduce seeking functionality via '--start-after'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-14T16:49:16Z","receivedAt":"2025-07-14T16:49:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> On Fri, Jul 11, 2025 at 6:20 PM Karthik Nayak wrote:\n>\n>> Initially I was also planning to cleanup all the `refs_for_each...()`\n>> functions in 'refs.h' by simply using the iterator, but this bloated the\n>> series. So I've left that for another day.\n>\n> I wonder if there is a plan to add the '--start-after' option to `git\n> branch` and `git tag` too?\n\nGood question.\n\n\"git for-each-ref\" is for scripters, \"git branch/tag\" are for\nhumans. And humans do not page (outside \"more/less\").\n\nSo while it may be trivial to expose the feature to these Porcelain\ncommands, it is not obvious that it is a good idea worth cluttering\n\"git tag -h\" output.\n"},{"id":"521941","messageId":"CAOLa=ZRgrfs8vip20QuNNkx9CmpUK392EV0vBkqu3HCyn4fXxQ@mail.gmail.com","threadId":"63722","inReplyTo":"CAP8UFD06KKF0A0WjBSw77A7fKp_pGJGVyw-OPd8-X92h=RSF+w@mail.gmail.com","subject":"Re: [PATCH v4 3/4] refs: selectively set prefix in the seek functions","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-15T08:19:56Z","receivedAt":"2025-07-15T08:19:58Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Christian Couder writes:\n\n> On Fri, Jul 11, 2025 at 6:20 PM Karthik Nayak wrote:\n>\n>> diff --git a/refs/refs-internal.h b/refs/refs-internal.h\n>> index 03f5df04d5..90de7837f8 100644\n>> --- a/refs/refs-internal.h\n>> +++ b/refs/refs-internal.h\n>> @@ -353,11 +353,12 @@ void base_ref_iterator_init(struct ref_iterator *iter,\n>> typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);\n>>\n>> /*\n>> - * Seek the iterator to the first reference matching the given prefix. Should\n>> - * behave the same as if a new iterator was created with the same prefix.\n>> + * Seek the iterator to the first matching reference. If set_prefix is set,\n>\n> s/If set_prefix is set/If the REF_ITERATOR_SEEK_SET_PREFIX flag is set/\n>\n\nWill change, thanks.\n\n>> + * it would behave the same as if a new iterator was created with the same\n>> + * prefix.\n>\n> Maybe: s/with the same prefix/at the same reference/\n>\n\nChanged it to\n\n If the REF_ITERATOR_SEEK_SET_PREFIX flag is set, it would behave the\n same as if a new iterator was created with the provided refname as prefix.\n\n>> */\n>> typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,\n>> - const char *prefix);\n>> + const char *refname, unsigned int flags);\n"},{"id":"521950","messageId":"CAOLa=ZSyVh1cWZL=F8Gk-URh7p6GTnPpOpWjGwCbwYX=a+-TNQ@mail.gmail.com","threadId":"63722","inReplyTo":"CAP8UFD1wRxZDCRC76VuuA8_rpNn__TQnL9RnNumCE33wAjSrMQ@mail.gmail.com","subject":"Re: [PATCH v4 4/4] for-each-ref: introduce a '--start-after' option","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-15T08:42:31Z","receivedAt":"2025-07-15T08:42:33Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Christian Couder writes:\n\n> On Fri, Jul 11, 2025 at 6:21 PM Karthik Nayak wrote:\n>\n>> /*\n>> * This is the same as for_each_fullref_in(), but it tries to iterate\n>> * only over the patterns we'll care about. Note that it _doesn't_ do a full\n>> @@ -2692,10 +2710,13 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n>> each_ref_fn cb,\n>> void *cb_data)\n>> {\n>> + struct ref_iterator *iter;\n>> + int flags = 0, ret = 0;\n>> +\n>> if (filter->kind & FILTER_REFS_ROOT_REFS) {\n>> /* In this case, we want to print all refs including root refs. */\n>> - return refs_for_each_include_root_refs(get_main_ref_store(the_repository),\n>> - cb, cb_data);\n>> + flags |= DO_FOR_EACH_INCLUDE_ROOT_REFS;\n>> + goto non_prefix_iter;\n>> }\n>>\n>> if (!filter->match_as_path) {\n>> @@ -2704,8 +2725,7 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n>> * prefixes like \"refs/heads/\" etc. are stripped off,\n>> * so we have to look at everything:\n>> */\n>> - return refs_for_each_fullref_in(get_main_ref_store(the_repository),\n>> - \"\", NULL, cb, cb_data);\n>> + goto non_prefix_iter;\n>> }\n>>\n>> if (filter->ignore_case) {\n>> @@ -2714,20 +2734,29 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n>> * so just return everything and let the caller\n>> * sort it out.\n>> */\n>> - return refs_for_each_fullref_in(get_main_ref_store(the_repository),\n>> - \"\", NULL, cb, cb_data);\n>> + goto non_prefix_iter;\n>> }\n>>\n>> if (!filter->name_patterns[0]) {\n>> /* no patterns; we have to look at everything */\n>> - return refs_for_each_fullref_in(get_main_ref_store(the_repository),\n>> - \"\", filter->exclude.v, cb, cb_data);\n>> + goto non_prefix_iter;\n>> }\n>>\n>> return refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n>> NULL, filter->name_patterns,\n>> filter->exclude.v,\n>> cb, cb_data);\n>> +\n>> +non_prefix_iter:\n>> + iter = refs_ref_iterator_begin(get_main_ref_store(the_repository), \"\",\n>> + NULL, 0, flags);\n>> + if (filter->start_after)\n>> + ret = start_ref_iterator_after(iter, filter->start_after);\n>> +\n>> + if (ret)\n>> + return ret;\n>> +\n>> + return do_for_each_ref_iterator(iter, cb, cb_data);\n>> }\n>\n> Nit: I wonder if what is under the 'non_prefix_iter' label could be in\n> a new function and instead of `goto non_prefix_iter` we could return\n> the result of the new function.\n>\n\nYeah, that would work too. Let me do that and make it nicer! Thanks for\nthe suggestion.\n\n>> /*\n>> @@ -3197,9 +3226,11 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n>> init_contains_cache(&filter->internal.no_contains_cache);\n>>\n>> /* Simple per-ref filtering */\n>> - if (!filter->kind)\n>> + if (!filter->kind) {\n>> die(\"filter_refs: invalid type\");\n>> - else {\n>> + } else {\n>\n> Nit: the `else` could be removed altogether here, but maybe that\n> should be done in a preparatory patch.\n>\n\nIndeed, since I plan to re-roll with the changes you've suggested, I\nwill add this in too.\n\n>> + const char *prefix = NULL;\n>> +\n>\n> [...]\n>\n>> +test_expect_success 'start after with specific directory and trailing slash' '\n>> + cat >expect <<-\\EOF &&\n>> + refs/odd/spot\n>> + refs/tags/annotated-tag\n>> + refs/tags/doubly-annotated-tag\n>> + refs/tags/doubly-signed-tag\n>> + refs/tags/foo1.10\n>> + refs/tags/foo1.3\n>> + refs/tags/foo1.6\n>> + refs/tags/four\n>> + refs/tags/one\n>> + refs/tags/signed-tag\n>> + refs/tags/three\n>> + refs/tags/two\n>> + EOF\n>> + git for-each-ref --format=\"%(refname)\" --start-after=refs/lost >actual &&\n>\n> I don't see a trailing slash.\n>\n\n>> + test_cmp expect actual\n>> +'\n>> +\n>> +test_expect_success 'start after, just behind a specific directory' '\n>> + cat >expect <<-\\EOF &&\n>> + refs/odd/spot\n>> + refs/tags/annotated-tag\n>> + refs/tags/doubly-annotated-tag\n>> + refs/tags/doubly-signed-tag\n>> + refs/tags/foo1.10\n>> + refs/tags/foo1.3\n>> + refs/tags/foo1.6\n>> + refs/tags/four\n>> + refs/tags/one\n>> + refs/tags/signed-tag\n>> + refs/tags/three\n>> + refs/tags/two\n>> + EOF\n>> + git for-each-ref --format=\"%(refname)\" --start-after=refs/odd/ >actual &&\n>\n> Here there is a trailing slash though.\n\nI think these tests would make more sense in the newer versions with the\nvalues swapped. Let me do that.\n\nThanks Christian for the thorough review.\n"},{"id":"521957","messageId":"CAOLa=ZS6ASf1+nbUnCTeeH1Di=kgmhQUEQ5UXPZu051rfqx9Pw@mail.gmail.com","threadId":"63722","inReplyTo":"xmqq8qkqvjnu.fsf@gitster.g","subject":"Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-15T09:49:03Z","receivedAt":"2025-07-15T09:49:06Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Patrick Steinhardt writes:\n>\n>>> + * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is\n>>> + * updated to match the seek string, affecting all subsequent iterations. If\n>>> + * not, the iterator seeks to the specified reference and clears any previously\n>>> + * set prefix.\n>>> *\n>>> * Returns 0 on success, a negative error code otherwise.\n>>> */\n>>> -int ref_iterator_seek(struct ref_iterator *ref_iterator,\n>>> -\t\t const char *prefix);\n>>> +int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *seek,\n>>> +\t\t unsigned int flags);\n>>\n>> Another tiny nit: instead of calling the variable `seek` we can just\n>> call it `refname`. That might give a bit more of a hint what you're\n>> actually seeking for.\n>>\n>> But other than that I'm happy with the new behaviour, where we are now\n>> consistently either setting or resetting the prefix depending on whether\n>> or not the caller set the flag.\n>\n> I am not sure. The way the \"prefix\" is used, if I understand correctly, is\n>\n> - it is set by iterator-begin, typically to the area to iterate\n> over (e.g. \"refs/heads/\" for iterating over branches) in the\n> for_each_ref_*() family of helpers, and internally we seek to\n> that area (skipping anything that come strictly before\n> \"refs/heads/\" for example).\n>\n> - iterator-advance looks at it and decides we are done when the\n> iterator points beyond that prefix\n>\n\nThat's right.\n\n> So if you are iterating inside \"refs/heads/\" hierarchy and seek to\n> \"refs/heads/m\", don't you still want to stop when you step outside\n> \"refs/heads/\" by keeping the original prefix, instead of unsetting\n> the prefix to empty? A postimage of this patch for packed backend\n> (picked at random) reads like this:\n>\n> static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,\n> const char *seek, unsigned int flags)\n> {\n> struct packed_ref_iterator *iter =\n> (struct packed_ref_iterator *)ref_iterator;\n> const char *start;\n>\n> if (seek && *seek)\n> start = find_reference_location(iter->snapshot, seek, 0);\n> else\n> start = iter->snapshot->start;\n>\n> /* Unset any previously set prefix */\n> FREE_AND_NULL(iter->prefix);\n>\n> if (flags & REF_ITERATOR_SEEK_SET_PREFIX)\n> iter->prefix = xstrdup_or_null(seek);\n>\n> so after (true) seeking that does not have the SET_PREFIX flag on,\n> wouldn't our iterator-advance run through the end since it no longer\n> is aware of where to stop?\n>\n\nThat's also right and that is indeed the intention. We're trying to make\nthe actions more intentional.\n\nSo if a user sets a 'prefix' for the iterator, all previous state of the\niterator is reset. So, the same function for seeking an iterator should\nalso have the same side-effect of resetting the previous state.\n\nThere could be a usecase where we add support for keeping the prefix,\nwhile also seeking the iterator. That would be an explicit change\n(perhaps with a corresponding flag?) that we'd have to build, add tests\nfor and call out. Until then, we explicitly reset the state whenever a\nuser calls 'ref_iterator_seek()', they can be sure that any previous\nstate is reset.\n\n> Thanks.\n"},{"id":"521958","messageId":"CAOLa=ZRT70g90+e-Mhy_USaXqLpDVamRm5dXc-r_fF2PHPQyPw@mail.gmail.com","threadId":"63722","inReplyTo":"xmqqms96u38z.fsf@gitster.g","subject":"Re: [PATCH v4 0/4] for-each-ref: introduce seeking functionality via '--start-after'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-15T09:49:57Z","receivedAt":"2025-07-15T09:49:59Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Christian Couder writes:\n>\n>> On Fri, Jul 11, 2025 at 6:20 PM Karthik Nayak wrote:\n>>\n>>> Initially I was also planning to cleanup all the `refs_for_each...()`\n>>> functions in 'refs.h' by simply using the iterator, but this bloated the\n>>> series. So I've left that for another day.\n>>\n>> I wonder if there is a plan to add the '--start-after' option to `git\n>> branch` and `git tag` too?\n>\n> Good question.\n>\n> \"git for-each-ref\" is for scripters, \"git branch/tag\" are for\n> humans. And humans do not page (outside \"more/less\").\n>\n> So while it may be trivial to expose the feature to these Porcelain\n> commands, it is not obvious that it is a good idea worth cluttering\n> \"git tag -h\" output.\n\nAgreed, since the functionality is built into 'ref-filter', it should be\neasy to expand to these commands. But I don't see the need for it.\n\nKarthik\n"},{"id":"521966","messageId":"20250715-306-git-for-each-ref-pagination-v5-0-852d5a2f56e1@gmail.com","threadId":"63722","inReplyTo":"20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com","subject":"[PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-15T11:28:25Z","receivedAt":"2025-07-15T11:28:32Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `git-for-each-ref(1)` command is used to iterate over references\npresent in a repository. In large repositories with millions of\nreferences, it would be optimal to paginate this output such that we\ncan start iteration from a given reference. This would avoid having to\niterate over all references from the beginning each time when paginating\nthrough results.\n\nThis series adds a '--start-after' option in 'git-for-each-ref(1)'. When\nused, the reference iteration seeks to first reference following the\nmarker alphabetically. When paging, it should be noted that references\nmay be deleted, modified or added between invocations. Output will only\nyield those references which follow the marker lexicographically. If the\nmarker does not exist, output begins from the first reference that would\ncome after it alphabetically.\n\nThis enables efficient pagination workflows like:\n git for-each-ref --count=100\n git for-each-ref --count=100 --start-after=refs/heads/branch-100\n git for-each-ref --count=100 --start-after=refs/heads/branch-200\n\nTo add this functionality, we expose the `ref_iterator` outside the\n'refs/' namespace and modify the `ref_iterator_seek()` to actually seek\nto a given reference and only set the prefix when the `set_prefix` field\nis set.\n\nOn the reftable and packed backend, the changes are simple. But since\nthe files backend uses 'ref-cache' for reference handling, the changes\nthere are a little more involved, since we need to setup the right\nlevels and the indexing.\n\nInitially I was also planning to cleanup all the `refs_for_each...()`\nfunctions in 'refs.h' by simply using the iterator, but this bloated the\nseries. So I've left that for another day.\n\nChanges in v5:\n- Changes to the comments to refer to the flag\n 'REF_ITERATOR_SEEK_SET_PREFIX' instead of a variable used in older\n versions. Also other small grammar fixes.\n- Added a commit to remove an unnecessary else clause.\n- Move seeking functionality within `for_each_fullref_in_pattern` to its\n own function.\n- Fix incorrect naming in the tests.\n- Link to v4: https://lore.kernel.org/r/20250711-306-git-for-each-ref-pagination-v4-0-ed3303ad5b89@gmail.com\n\nChanges in v4:\n- Patch 3/4: Move around the documentation for the flag and rename the\n seek variable to refname.\n- Patch 4/4: Cleanup the commit message and also the documentation.\n- Link to v3: https://lore.kernel.org/r/20250708-306-git-for-each-ref-pagination-v3-0-8cfba1080be4@gmail.com\n\nChanges in v3:\n- Change the working of the command to exclude the marker provided. With\n this rename the flag to '--start-after'.\n- Extend the documentation to add a note about concurrent modifications\n to the reference database.\n- Link to v2: https://lore.kernel.org/r/20250704-306-git-for-each-ref-pagination-v2-0-bcde14acdd81@gmail.com\n\nChanges in v2:\n- Modify 'ref_iterator_seek()' to take in flags instead of a\n 'set_prefix' variable. This improves readability, where users would\n use the 'REF_ITERATOR_SEEK_SET_PREFIX' instead of simply passing '1'.\n- When the set prefix flag isn't usage, reset any previously set prefix.\n This ensures that the internal prefix state is always reset whenever\n we seek and unifies the behavior between 'ref_iterator_seek' and\n 'ref_iterator_begin'.\n- Don't allow '--skip-until' to be run with '--sort', since the seeking\n always takes place before any sorting and this can be confusing.\n- Some styling fixes:\n - Remove extra newline\n - Skip braces around single lined if...else clause\n - Add braces around 'if' clause\n - Fix indentation\n- Link to v1: https://lore.kernel.org/git/20250701-306-git-for-each-ref-pagination-v1-0-4f0ae7c0688f@gmail.com/\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/git-for-each-ref.adoc | 10 +-\n builtin/for-each-ref.c | 8 ++\n ref-filter.c | 116 ++++++++++++++-------\n ref-filter.h | 1 +\n refs.c | 6 +-\n refs.h | 155 ++++++++++++++++++++++++++++\n refs/debug.c | 7 +-\n refs/files-backend.c | 7 +-\n refs/iterator.c | 26 +++--\n refs/packed-backend.c | 17 ++--\n refs/ref-cache.c | 99 ++++++++++++++----\n refs/ref-cache.h | 7 --\n refs/refs-internal.h | 152 ++--------------------------\n refs/reftable-backend.c | 21 ++--\n t/t6302-for-each-ref-filter.sh | 194 ++++++++++++++++++++++++++++++++++++\n 15 files changed, 583 insertions(+), 243 deletions(-)\n\nKarthik Nayak (5):\n refs: expose `ref_iterator` via 'refs.h'\n ref-cache: remove unused function 'find_ref_entry()'\n refs: selectively set prefix in the seek functions\n ref-filter: remove unnecessary else clause\n for-each-ref: introduce a '--start-after' option\n\nRange-diff versus v4:\n\n1: dde167f421 = 1: f9c9a7fdd9 refs: expose `ref_iterator` via 'refs.h'\n2: e392e93520 = 2: 83bee35517 ref-cache: remove unused function 'find_ref_entry()'\n3: 711ffcac00 ! 3: 3b6019a1e7 refs: selectively set prefix in the seek functions\n @@ refs/refs-internal.h: void base_ref_iterator_init(struct ref_iterator *iter,\n /*\n - * Seek the iterator to the first reference matching the given prefix. Should\n - * behave the same as if a new iterator was created with the same prefix.\n -+ * Seek the iterator to the first matching reference. If set_prefix is set,\n -+ * it would behave the same as if a new iterator was created with the same\n -+ * prefix.\n ++ * Seek the iterator to the first matching reference. If the\n ++ * REF_ITERATOR_SEEK_SET_PREFIX flag is set, it would behave the same as if a\n ++ * new iterator was created with the provided refname as prefix.\n */\n typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,\n -\t\t\t\t const char *prefix);\n-: ---------- > 4: 3f89eeef26 ref-filter: remove unnecessary else clause\n4: 3a0c89acbe ! 5: 7ee7d83cf0 for-each-ref: introduce a '--start-after' option\n @@ ref-filter.c: static int filter_exclude_match(struct ref_filter *filter, const c\n +\tstrbuf_release(&sb);\n +\treturn ret;\n +}\n ++\n ++static int for_each_fullref_with_seek(struct ref_filter *filter, each_ref_fn cb,\n ++\t\t\t\t void *cb_data, unsigned int flags)\n ++{\n ++\tstruct ref_iterator *iter;\n ++\tint ret = 0;\n ++\n ++\titer = refs_ref_iterator_begin(get_main_ref_store(the_repository), \"\",\n ++\t\t\t\t NULL, 0, flags);\n ++\tif (filter->start_after)\n ++\t\tret = start_ref_iterator_after(iter, filter->start_after);\n ++\n ++\tif (ret)\n ++\t\treturn ret;\n ++\n ++\treturn do_for_each_ref_iterator(iter, cb, cb_data);\n ++}\n +\n /*\n * This is the same as for_each_fullref_in(), but it tries to iterate\n * only over the patterns we'll care about. Note that it _doesn't_ do a full\n @@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,\n - \t\t\t\t each_ref_fn cb,\n - \t\t\t\t void *cb_data)\n {\n -+\tstruct ref_iterator *iter;\n -+\tint flags = 0, ret = 0;\n -+\n \tif (filter->kind & FILTER_REFS_ROOT_REFS) {\n \t\t/* In this case, we want to print all refs including root refs. */\n -\t\treturn refs_for_each_include_root_refs(get_main_ref_store(the_repository),\n -\t\t\t\t\t\t cb, cb_data);\n -+\t\tflags |= DO_FOR_EACH_INCLUDE_ROOT_REFS;\n -+\t\tgoto non_prefix_iter;\n ++\t\treturn for_each_fullref_with_seek(filter, cb, cb_data,\n ++\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_ROOT_REFS);\n \t}\n \n \tif (!filter->match_as_path) {\n @@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t */\n -\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n -\t\t\t\t\t\t\"\", NULL, cb, cb_data);\n -+\t\tgoto non_prefix_iter;\n ++\t\treturn for_each_fullref_with_seek(filter, cb, cb_data, 0);\n \t}\n \n \tif (filter->ignore_case) {\n @@ ref-filter.c: static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t */\n -\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n -\t\t\t\t\t\t\"\", NULL, cb, cb_data);\n -+\t\tgoto non_prefix_iter;\n ++\t\treturn for_each_fullref_with_seek(filter, cb, cb_data, 0);\n \t}\n \n \tif (!filter->name_patterns[0]) {\n \t\t/* no patterns; we have to look at everything */\n -\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n -\t\t\t\t\t\t \"\", filter->exclude.v, cb, cb_data);\n -+\t\tgoto non_prefix_iter;\n ++\t\treturn for_each_fullref_with_seek(filter, cb, cb_data, 0);\n \t}\n \n \treturn refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n - \t\t\t\t\t\t NULL, filter->name_patterns,\n - \t\t\t\t\t\t filter->exclude.v,\n - \t\t\t\t\t\t cb, cb_data);\n -+\n -+non_prefix_iter:\n -+\titer = refs_ref_iterator_begin(get_main_ref_store(the_repository), \"\",\n -+\t\t\t\t NULL, 0, flags);\n -+\tif (filter->start_after)\n -+\t\tret = start_ref_iterator_after(iter, filter->start_after);\n -+\n -+\tif (ret)\n -+\t\treturn ret;\n -+\n -+\treturn do_for_each_ref_iterator(iter, cb, cb_data);\n - }\n +@@ ref-filter.c: void filter_is_base(struct repository *r,\n \n - /*\n -@@ ref-filter.c: static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n - \tinit_contains_cache(&filter->internal.no_contains_cache);\n + static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref_fn fn, void *cb_data)\n + {\n ++\tconst char *prefix = NULL;\n + \tint ret = 0;\n \n - \t/* Simple per-ref filtering */\n --\tif (!filter->kind)\n -+\tif (!filter->kind) {\n - \t\tdie(\"filter_refs: invalid type\");\n --\telse {\n -+\t} else {\n -+\t\tconst char *prefix = NULL;\n -+\n - \t\t/*\n - \t\t * For common cases where we need only branches or remotes or tags,\n - \t\t * we only iterate through those refs. If a mix of refs is needed,\n + \tfilter->kind = type & FILTER_REFS_KIND_MASK;\n @@ ref-filter.c: static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n - \t\t * of filter_ref_kind().\n - \t\t */\n - \t\tif (filter->kind == FILTER_REFS_BRANCHES)\n --\t\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n --\t\t\t\t\t\t \"refs/heads/\", NULL,\n --\t\t\t\t\t\t fn, cb_data);\n -+\t\t\tprefix = \"refs/heads/\";\n - \t\telse if (filter->kind == FILTER_REFS_REMOTES)\n --\t\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n --\t\t\t\t\t\t \"refs/remotes/\", NULL,\n --\t\t\t\t\t\t fn, cb_data);\n -+\t\t\tprefix = \"refs/remotes/\";\n - \t\telse if (filter->kind == FILTER_REFS_TAGS)\n --\t\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n --\t\t\t\t\t\t \"refs/tags/\", NULL, fn,\n --\t\t\t\t\t\t cb_data);\n --\t\telse if (filter->kind & FILTER_REFS_REGULAR)\n -+\t\t\tprefix = \"refs/tags/\";\n + \t * of filter_ref_kind().\n + \t */\n + \tif (filter->kind == FILTER_REFS_BRANCHES)\n +-\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n +-\t\t\t\t\t \"refs/heads/\", NULL,\n +-\t\t\t\t\t fn, cb_data);\n ++\t\tprefix = \"refs/heads/\";\n + \telse if (filter->kind == FILTER_REFS_REMOTES)\n +-\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n +-\t\t\t\t\t \"refs/remotes/\", NULL,\n +-\t\t\t\t\t fn, cb_data);\n ++\t\tprefix = \"refs/remotes/\";\n + \telse if (filter->kind == FILTER_REFS_TAGS)\n +-\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n +-\t\t\t\t\t \"refs/tags/\", NULL, fn,\n +-\t\t\t\t\t cb_data);\n +-\telse if (filter->kind & FILTER_REFS_REGULAR)\n ++\t\tprefix = \"refs/tags/\";\n +\n -+\t\tif (prefix) {\n -+\t\t\tstruct ref_iterator *iter;\n ++\tif (prefix) {\n ++\t\tstruct ref_iterator *iter;\n +\n -+\t\t\titer = refs_ref_iterator_begin(get_main_ref_store(the_repository),\n -+\t\t\t\t\t\t \"\", NULL, 0, 0);\n ++\t\titer = refs_ref_iterator_begin(get_main_ref_store(the_repository),\n ++\t\t\t\t\t \"\", NULL, 0, 0);\n +\n -+\t\t\tif (filter->start_after)\n -+\t\t\t\tret = start_ref_iterator_after(iter, filter->start_after);\n -+\t\t\telse if (prefix)\n -+\t\t\t\tret = ref_iterator_seek(iter, prefix, 1);\n ++\t\tif (filter->start_after)\n ++\t\t\tret = start_ref_iterator_after(iter, filter->start_after);\n ++\t\telse if (prefix)\n ++\t\t\tret = ref_iterator_seek(iter, prefix, 1);\n +\n -+\t\t\tif (!ret)\n -+\t\t\t\tret = do_for_each_ref_iterator(iter, fn, cb_data);\n -+\t\t} else if (filter->kind & FILTER_REFS_REGULAR) {\n - \t\t\tret = for_each_fullref_in_pattern(filter, fn, cb_data);\n -+\t\t}\n ++\t\tif (!ret)\n ++\t\t\tret = do_for_each_ref_iterator(iter, fn, cb_data);\n ++\t} else if (filter->kind & FILTER_REFS_REGULAR) {\n + \t\tret = for_each_fullref_in_pattern(filter, fn, cb_data);\n ++\t}\n \n - \t\t/*\n - \t\t * When printing all ref types, HEAD is already included,\n + \t/*\n + \t * When printing all ref types, HEAD is already included,\n \n ## ref-filter.h ##\n @@ ref-filter.h: struct ref_array {\n @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '\n +\trefs/tags/three\n +\trefs/tags/two\n +\tEOF\n -+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/lost >actual &&\n ++\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/ >actual &&\n +\ttest_cmp expect actual\n +'\n +\n @@ t/t6302-for-each-ref-filter.sh: test_expect_success 'validate worktree atom' '\n +\trefs/tags/three\n +\trefs/tags/two\n +\tEOF\n -+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/ >actual &&\n ++\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/lost >actual &&\n +\ttest_cmp expect actual\n +'\n +\n\n\nbase-commit: cf6f63ea6bf35173e02e18bdc6a4ba41288acff9\nchange-id: 20250605-306-git-for-each-ref-pagination-0ba8a29ae646\n\nThanks\n- Karthik\n\n"},{"id":"521967","messageId":"20250715-306-git-for-each-ref-pagination-v5-1-852d5a2f56e1@gmail.com","threadId":"63722","inReplyTo":"20250715-306-git-for-each-ref-pagination-v5-0-852d5a2f56e1@gmail.com","subject":"[PATCH v5 1/5] refs: expose `ref_iterator` via 'refs.h'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-15T11:28:26Z","receivedAt":"2025-07-15T11:28:33Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `ref_iterator` is an internal structure to the 'refs/'\nsub-directory, which allows iteration over refs. All reference iteration\nis built on top of these iterators.\n\nExternal clients of the 'refs' subsystem use the various\n'refs_for_each...()' functions to iterate over refs. However since these\nare wrapper functions, each combination of functionality requires a new\nwrapper function. This is not feasible as the functions pile up with the\nincrease in requirements. Expose the internal reference iterator, so\nadvanced users can mix and match options as needed.\n\nSigned-off-by: Karthik Nayak \n---\n refs.h | 147 +++++++++++++++++++++++++++++++++++++++++++++++++++\n refs/refs-internal.h | 145 +-------------------------------------------------\n 2 files changed, 149 insertions(+), 143 deletions(-)\n\ndiff --git a/refs.h b/refs.h\nindex 46a6008e07..7c21aaef3d 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -1190,4 +1190,151 @@ int repo_migrate_ref_storage_format(struct repository *repo,\n \t\t\t\t unsigned int flags,\n \t\t\t\t struct strbuf *err);\n \n+/*\n+ * Reference iterators\n+ *\n+ * A reference iterator encapsulates the state of an in-progress\n+ * iteration over references. Create an instance of `struct\n+ * ref_iterator` via one of the functions in this module.\n+ *\n+ * A freshly-created ref_iterator doesn't yet point at a reference. To\n+ * advance the iterator, call ref_iterator_advance(). If successful,\n+ * this sets the iterator's refname, oid, and flags fields to describe\n+ * the next reference and returns ITER_OK. The data pointed at by\n+ * refname and oid belong to the iterator; if you want to retain them\n+ * after calling ref_iterator_advance() again or calling\n+ * ref_iterator_free(), you must make a copy. When the iteration has\n+ * been exhausted, ref_iterator_advance() releases any resources\n+ * associated with the iteration, frees the ref_iterator object, and\n+ * returns ITER_DONE. If you want to abort the iteration early, call\n+ * ref_iterator_free(), which also frees the ref_iterator object and\n+ * any associated resources. If there was an internal error advancing\n+ * to the next entry, ref_iterator_advance() aborts the iteration,\n+ * frees the ref_iterator, and returns ITER_ERROR.\n+ *\n+ * The reference currently being looked at can be peeled by calling\n+ * ref_iterator_peel(). This function is often faster than peel_ref(),\n+ * so it should be preferred when iterating over references.\n+ *\n+ * Putting it all together, a typical iteration looks like this:\n+ *\n+ * int ok;\n+ * struct ref_iterator *iter = ...;\n+ *\n+ * while ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n+ * if (want_to_stop_iteration()) {\n+ * ok = ITER_DONE;\n+ * break;\n+ * }\n+ *\n+ * // Access information about the current reference:\n+ * if (!(iter->flags & REF_ISSYMREF))\n+ * printf(\"%s is %s\\n\", iter->refname, oid_to_hex(iter->oid));\n+ *\n+ * // If you need to peel the reference:\n+ * ref_iterator_peel(iter, &oid);\n+ * }\n+ *\n+ * if (ok != ITER_DONE)\n+ * handle_error();\n+ * ref_iterator_free(iter);\n+ */\n+struct ref_iterator;\n+\n+/*\n+ * These flags are passed to refs_ref_iterator_begin() (and do_for_each_ref(),\n+ * which feeds it).\n+ */\n+enum do_for_each_ref_flags {\n+\t/*\n+\t * Include broken references in a do_for_each_ref*() iteration, which\n+\t * would normally be omitted. This includes both refs that point to\n+\t * missing objects (a true repository corruption), ones with illegal\n+\t * names (which we prefer not to expose to callers), as well as\n+\t * dangling symbolic refs (i.e., those that point to a non-existent\n+\t * ref; this is not a corruption, but as they have no valid oid, we\n+\t * omit them from normal iteration results).\n+\t */\n+\tDO_FOR_EACH_INCLUDE_BROKEN = (1 << 0),\n+\n+\t/*\n+\t * Only include per-worktree refs in a do_for_each_ref*() iteration.\n+\t * Normally this will be used with a files ref_store, since that's\n+\t * where all reference backends will presumably store their\n+\t * per-worktree refs.\n+\t */\n+\tDO_FOR_EACH_PER_WORKTREE_ONLY = (1 << 1),\n+\n+\t/*\n+\t * Omit dangling symrefs from output; this only has an effect with\n+\t * INCLUDE_BROKEN, since they are otherwise not included at all.\n+\t */\n+\tDO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 << 2),\n+\n+\t/*\n+\t * Include root refs i.e. HEAD and pseudorefs along with the regular\n+\t * refs.\n+\t */\n+\tDO_FOR_EACH_INCLUDE_ROOT_REFS = (1 << 3),\n+};\n+\n+/*\n+ * Return an iterator that goes over each reference in `refs` for\n+ * which the refname begins with prefix. If trim is non-zero, then\n+ * trim that many characters off the beginning of each refname.\n+ * The output is ordered by refname.\n+ */\n+struct ref_iterator *refs_ref_iterator_begin(\n+\tstruct ref_store *refs,\n+\tconst char *prefix, const char **exclude_patterns,\n+\tint trim, enum do_for_each_ref_flags flags);\n+\n+/*\n+ * Advance the iterator to the first or next item and return ITER_OK.\n+ * If the iteration is exhausted, free the resources associated with\n+ * the ref_iterator and return ITER_DONE. On errors, free the iterator\n+ * resources and return ITER_ERROR. It is a bug to use ref_iterator or\n+ * call this function again after it has returned ITER_DONE or\n+ * ITER_ERROR.\n+ */\n+int ref_iterator_advance(struct ref_iterator *ref_iterator);\n+\n+/*\n+ * Seek the iterator to the first reference with the given prefix.\n+ * The prefix is matched as a literal string, without regard for path\n+ * separators. If prefix is NULL or the empty string, seek the iterator to the\n+ * first reference again.\n+ *\n+ * This function is expected to behave as if a new ref iterator with the same\n+ * prefix had been created, but allows reuse of iterators and thus may allow\n+ * the backend to optimize. Parameters other than the prefix that have been\n+ * passed when creating the iterator will remain unchanged.\n+ *\n+ * Returns 0 on success, a negative error code otherwise.\n+ */\n+int ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t const char *prefix);\n+\n+/*\n+ * If possible, peel the reference currently being viewed by the\n+ * iterator. Return 0 on success.\n+ */\n+int ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t struct object_id *peeled);\n+\n+/* Free the reference iterator and any associated resources. */\n+void ref_iterator_free(struct ref_iterator *ref_iterator);\n+\n+/*\n+ * The common backend for the for_each_*ref* functions. Call fn for\n+ * each reference in iter. If the iterator itself ever returns\n+ * ITER_ERROR, return -1. If fn ever returns a non-zero value, stop\n+ * the iteration and return that value. Otherwise, return 0. In any\n+ * case, free the iterator when done. This function is basically an\n+ * adapter between the callback style of reference iteration and the\n+ * iterator style.\n+ */\n+int do_for_each_ref_iterator(struct ref_iterator *iter,\n+\t\t\t each_ref_fn fn, void *cb_data);\n+\n #endif /* REFS_H */\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex f868870851..03f5df04d5 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -244,90 +244,8 @@ const char *find_descendant_ref(const char *dirname,\n #define SYMREF_MAXDEPTH 5\n \n /*\n- * These flags are passed to refs_ref_iterator_begin() (and do_for_each_ref(),\n- * which feeds it).\n- */\n-enum do_for_each_ref_flags {\n-\t/*\n-\t * Include broken references in a do_for_each_ref*() iteration, which\n-\t * would normally be omitted. This includes both refs that point to\n-\t * missing objects (a true repository corruption), ones with illegal\n-\t * names (which we prefer not to expose to callers), as well as\n-\t * dangling symbolic refs (i.e., those that point to a non-existent\n-\t * ref; this is not a corruption, but as they have no valid oid, we\n-\t * omit them from normal iteration results).\n-\t */\n-\tDO_FOR_EACH_INCLUDE_BROKEN = (1 << 0),\n-\n-\t/*\n-\t * Only include per-worktree refs in a do_for_each_ref*() iteration.\n-\t * Normally this will be used with a files ref_store, since that's\n-\t * where all reference backends will presumably store their\n-\t * per-worktree refs.\n-\t */\n-\tDO_FOR_EACH_PER_WORKTREE_ONLY = (1 << 1),\n-\n-\t/*\n-\t * Omit dangling symrefs from output; this only has an effect with\n-\t * INCLUDE_BROKEN, since they are otherwise not included at all.\n-\t */\n-\tDO_FOR_EACH_OMIT_DANGLING_SYMREFS = (1 << 2),\n-\n-\t/*\n-\t * Include root refs i.e. HEAD and pseudorefs along with the regular\n-\t * refs.\n-\t */\n-\tDO_FOR_EACH_INCLUDE_ROOT_REFS = (1 << 3),\n-};\n-\n-/*\n- * Reference iterators\n- *\n- * A reference iterator encapsulates the state of an in-progress\n- * iteration over references. Create an instance of `struct\n- * ref_iterator` via one of the functions in this module.\n- *\n- * A freshly-created ref_iterator doesn't yet point at a reference. To\n- * advance the iterator, call ref_iterator_advance(). If successful,\n- * this sets the iterator's refname, oid, and flags fields to describe\n- * the next reference and returns ITER_OK. The data pointed at by\n- * refname and oid belong to the iterator; if you want to retain them\n- * after calling ref_iterator_advance() again or calling\n- * ref_iterator_free(), you must make a copy. When the iteration has\n- * been exhausted, ref_iterator_advance() releases any resources\n- * associated with the iteration, frees the ref_iterator object, and\n- * returns ITER_DONE. If you want to abort the iteration early, call\n- * ref_iterator_free(), which also frees the ref_iterator object and\n- * any associated resources. If there was an internal error advancing\n- * to the next entry, ref_iterator_advance() aborts the iteration,\n- * frees the ref_iterator, and returns ITER_ERROR.\n- *\n- * The reference currently being looked at can be peeled by calling\n- * ref_iterator_peel(). This function is often faster than peel_ref(),\n- * so it should be preferred when iterating over references.\n- *\n- * Putting it all together, a typical iteration looks like this:\n- *\n- * int ok;\n- * struct ref_iterator *iter = ...;\n- *\n- * while ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n- * if (want_to_stop_iteration()) {\n- * ok = ITER_DONE;\n- * break;\n- * }\n- *\n- * // Access information about the current reference:\n- * if (!(iter->flags & REF_ISSYMREF))\n- * printf(\"%s is %s\\n\", iter->refname, oid_to_hex(iter->oid));\n- *\n- * // If you need to peel the reference:\n- * ref_iterator_peel(iter, &oid);\n- * }\n- *\n- * if (ok != ITER_DONE)\n- * handle_error();\n- * ref_iterator_free(iter);\n+ * Data structure for holding a reference iterator. See refs.h for\n+ * more details and usage instructions.\n */\n struct ref_iterator {\n \tstruct ref_iterator_vtable *vtable;\n@@ -337,42 +255,6 @@ struct ref_iterator {\n \tunsigned int flags;\n };\n \n-/*\n- * Advance the iterator to the first or next item and return ITER_OK.\n- * If the iteration is exhausted, free the resources associated with\n- * the ref_iterator and return ITER_DONE. On errors, free the iterator\n- * resources and return ITER_ERROR. It is a bug to use ref_iterator or\n- * call this function again after it has returned ITER_DONE or\n- * ITER_ERROR.\n- */\n-int ref_iterator_advance(struct ref_iterator *ref_iterator);\n-\n-/*\n- * Seek the iterator to the first reference with the given prefix.\n- * The prefix is matched as a literal string, without regard for path\n- * separators. If prefix is NULL or the empty string, seek the iterator to the\n- * first reference again.\n- *\n- * This function is expected to behave as if a new ref iterator with the same\n- * prefix had been created, but allows reuse of iterators and thus may allow\n- * the backend to optimize. Parameters other than the prefix that have been\n- * passed when creating the iterator will remain unchanged.\n- *\n- * Returns 0 on success, a negative error code otherwise.\n- */\n-int ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t const char *prefix);\n-\n-/*\n- * If possible, peel the reference currently being viewed by the\n- * iterator. Return 0 on success.\n- */\n-int ref_iterator_peel(struct ref_iterator *ref_iterator,\n-\t\t struct object_id *peeled);\n-\n-/* Free the reference iterator and any associated resources. */\n-void ref_iterator_free(struct ref_iterator *ref_iterator);\n-\n /*\n * An iterator over nothing (its first ref_iterator_advance() call\n * returns ITER_DONE).\n@@ -384,17 +266,6 @@ struct ref_iterator *empty_ref_iterator_begin(void);\n */\n int is_empty_ref_iterator(struct ref_iterator *ref_iterator);\n \n-/*\n- * Return an iterator that goes over each reference in `refs` for\n- * which the refname begins with prefix. If trim is non-zero, then\n- * trim that many characters off the beginning of each refname.\n- * The output is ordered by refname.\n- */\n-struct ref_iterator *refs_ref_iterator_begin(\n-\t\tstruct ref_store *refs,\n-\t\tconst char *prefix, const char **exclude_patterns,\n-\t\tint trim, enum do_for_each_ref_flags flags);\n-\n /*\n * A callback function used to instruct merge_ref_iterator how to\n * interleave the entries from iter0 and iter1. The function should\n@@ -520,18 +391,6 @@ struct ref_iterator_vtable {\n */\n extern struct ref_iterator *current_ref_iter;\n \n-/*\n- * The common backend for the for_each_*ref* functions. Call fn for\n- * each reference in iter. If the iterator itself ever returns\n- * ITER_ERROR, return -1. If fn ever returns a non-zero value, stop\n- * the iteration and return that value. Otherwise, return 0. In any\n- * case, free the iterator when done. This function is basically an\n- * adapter between the callback style of reference iteration and the\n- * iterator style.\n- */\n-int do_for_each_ref_iterator(struct ref_iterator *iter,\n-\t\t\t each_ref_fn fn, void *cb_data);\n-\n struct ref_store;\n \n /* refs backends */\n\n-- \n2.49.0\n\n"},{"id":"521968","messageId":"20250715-306-git-for-each-ref-pagination-v5-2-852d5a2f56e1@gmail.com","threadId":"63722","inReplyTo":"20250715-306-git-for-each-ref-pagination-v5-0-852d5a2f56e1@gmail.com","subject":"[PATCH v5 2/5] ref-cache: remove unused function 'find_ref_entry()'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-15T11:28:27Z","receivedAt":"2025-07-15T11:28:34Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The 'find_ref_entry' function is no longer used, so remove it.\n\nSigned-off-by: Karthik Nayak \n---\n refs/ref-cache.c | 14 --------------\n refs/ref-cache.h | 7 -------\n 2 files changed, 21 deletions(-)\n\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex c1f1bab1d5..8aaffa8c6b 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -194,20 +194,6 @@ static struct ref_dir *find_containing_dir(struct ref_dir *dir,\n \treturn dir;\n }\n \n-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname)\n-{\n-\tint entry_index;\n-\tstruct ref_entry *entry;\n-\tdir = find_containing_dir(dir, refname);\n-\tif (!dir)\n-\t\treturn NULL;\n-\tentry_index = search_ref_dir(dir, refname, strlen(refname));\n-\tif (entry_index == -1)\n-\t\treturn NULL;\n-\tentry = dir->entries[entry_index];\n-\treturn (entry->flag & REF_DIR) ? NULL : entry;\n-}\n-\n /*\n * Emit a warning and return true iff ref1 and ref2 have the same name\n * and the same oid. Die if they have the same name but different\ndiff --git a/refs/ref-cache.h b/refs/ref-cache.h\nindex 5f04e518c3..f635d2d824 100644\n--- a/refs/ref-cache.h\n+++ b/refs/ref-cache.h\n@@ -201,13 +201,6 @@ void free_ref_cache(struct ref_cache *cache);\n */\n void add_entry_to_dir(struct ref_dir *dir, struct ref_entry *entry);\n \n-/*\n- * Find the value entry with the given name in dir, sorting ref_dirs\n- * and recursing into subdirectories as necessary. If the name is not\n- * found or it corresponds to a directory entry, return NULL.\n- */\n-struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname);\n-\n /*\n * Start iterating over references in `cache`. If `prefix` is\n * specified, only include references whose names start with that\n\n-- \n2.49.0\n\n"},{"id":"521969","messageId":"20250715-306-git-for-each-ref-pagination-v5-3-852d5a2f56e1@gmail.com","threadId":"63722","inReplyTo":"20250715-306-git-for-each-ref-pagination-v5-0-852d5a2f56e1@gmail.com","subject":"[PATCH v5 3/5] refs: selectively set prefix in the seek functions","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-15T11:28:28Z","receivedAt":"2025-07-15T11:28:36Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The ref iterator exposes a `ref_iterator_seek()` function. The name\nsuggests that this would seek the iterator to a specific reference in\nsome ways similar to how `fseek()` works for the filesystem.\n\nHowever, the function actually sets the prefix for refs iteration. So\nfurther iteration would only yield references which match the particular\nprefix. This is a bit confusing.\n\nLet's add a 'flags' field to the function, which when set with the\n'REF_ITERATOR_SEEK_SET_PREFIX' flag, will set the prefix for the\niteration in-line with the existing behavior. Otherwise, the reference\nbackends will simply seek to the specified reference and clears any\npreviously set prefix. This allows users to start iteration from a\nspecific reference.\n\nIn the packed and reftable backend, since references are available in a\nsorted list, the changes are simply setting the prefix if needed. The\nchanges on the files-backend are a little more involved, since the files\nbackend uses the 'ref-cache' mechanism. We move out the existing logic\nwithin `cache_ref_iterator_seek()` to `cache_ref_iterator_set_prefix()`\nwhich is called when the 'REF_ITERATOR_SEEK_SET_PREFIX' flag is set. We\nthen parse the provided seek string and set the required levels and\ntheir indexes to ensure that seeking is possible.\n\nHelped-by: Patrick Steinhardt \nSigned-off-by: Karthik Nayak \n---\n refs.c | 6 ++--\n refs.h | 26 +++++++++------\n refs/debug.c | 7 ++--\n refs/files-backend.c | 7 ++--\n refs/iterator.c | 26 ++++++++-------\n refs/packed-backend.c | 17 ++++++----\n refs/ref-cache.c | 85 ++++++++++++++++++++++++++++++++++++++++++++++---\n refs/refs-internal.h | 7 ++--\n refs/reftable-backend.c | 21 ++++++++----\n 9 files changed, 152 insertions(+), 50 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex dce5c49ca2..243e6898b8 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2666,12 +2666,12 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs\n \t\tif (!initial_transaction) {\n \t\t\tint ok;\n \n-\t\t\tif (!iter) {\n+\t\t\tif (!iter)\n \t\t\t\titer = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,\n \t\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n-\t\t\t} else if (ref_iterator_seek(iter, dirname.buf) < 0) {\n+\t\t\telse if (ref_iterator_seek(iter, dirname.buf,\n+\t\t\t\t\t\t REF_ITERATOR_SEEK_SET_PREFIX) < 0)\n \t\t\t\tgoto cleanup;\n-\t\t\t}\n \n \t\t\twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\t\t\tif (skip &&\ndiff --git a/refs.h b/refs.h\nindex 7c21aaef3d..e6780a8848 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -1299,21 +1299,29 @@ struct ref_iterator *refs_ref_iterator_begin(\n */\n int ref_iterator_advance(struct ref_iterator *ref_iterator);\n \n+enum ref_iterator_seek_flag {\n+\t/*\n+\t * When the REF_ITERATOR_SEEK_SET_PREFIX flag is set, the iterator's prefix is\n+\t * updated to match the provided string, affecting all subsequent iterations. If\n+\t * not, the iterator seeks to the specified reference and clears any previously\n+\t * set prefix.\n+\t */\n+\tREF_ITERATOR_SEEK_SET_PREFIX = (1 << 0),\n+};\n+\n /*\n- * Seek the iterator to the first reference with the given prefix.\n- * The prefix is matched as a literal string, without regard for path\n- * separators. If prefix is NULL or the empty string, seek the iterator to the\n+ * Seek the iterator to the first reference matching the given seek string.\n+ * The seek string is matched as a literal string, without regard for path\n+ * separators. If seek is NULL or the empty string, seek the iterator to the\n * first reference again.\n *\n- * This function is expected to behave as if a new ref iterator with the same\n- * prefix had been created, but allows reuse of iterators and thus may allow\n- * the backend to optimize. Parameters other than the prefix that have been\n- * passed when creating the iterator will remain unchanged.\n+ * This function is expected to behave as if a new ref iterator has been\n+ * created, but allows reuse of existing iterators for optimization.\n *\n * Returns 0 on success, a negative error code otherwise.\n */\n-int ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t const char *prefix);\n+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *refname,\n+\t\t unsigned int flags);\n \n /*\n * If possible, peel the reference currently being viewed by the\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 485e3079d7..da300efaf3 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -170,12 +170,13 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int debug_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *refname, unsigned int flags)\n {\n \tstruct debug_ref_iterator *diter =\n \t\t(struct debug_ref_iterator *)ref_iterator;\n-\tint res = diter->iter->vtable->seek(diter->iter, prefix);\n-\ttrace_printf_key(&trace_refs, \"iterator_seek: %s: %d\\n\", prefix ? prefix : \"\", res);\n+\tint res = diter->iter->vtable->seek(diter->iter, refname, flags);\n+\ttrace_printf_key(&trace_refs, \"iterator_seek: %s flags: %d: %d\\n\",\n+\t\t\t refname ? refname : \"\", flags, res);\n \treturn res;\n }\n \ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex bf6f89b1d1..8b282f2a60 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -929,11 +929,11 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int files_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *refname, unsigned int flags)\n {\n \tstruct files_ref_iterator *iter =\n \t\t(struct files_ref_iterator *)ref_iterator;\n-\treturn ref_iterator_seek(iter->iter0, prefix);\n+\treturn ref_iterator_seek(iter->iter0, refname, flags);\n }\n \n static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n@@ -2316,7 +2316,8 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int files_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n-\t\t\t\t const char *prefix UNUSED)\n+\t\t\t\t const char *refname UNUSED,\n+\t\t\t\t unsigned int flags UNUSED)\n {\n \tBUG(\"ref_iterator_seek() called for reflog_iterator\");\n }\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex 766d96e795..17ef841d8a 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -15,10 +15,10 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ref_iterator->vtable->advance(ref_iterator);\n }\n \n-int ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t const char *prefix)\n+int ref_iterator_seek(struct ref_iterator *ref_iterator, const char *refname,\n+\t\t unsigned int flags)\n {\n-\treturn ref_iterator->vtable->seek(ref_iterator, prefix);\n+\treturn ref_iterator->vtable->seek(ref_iterator, refname, flags);\n }\n \n int ref_iterator_peel(struct ref_iterator *ref_iterator,\n@@ -57,7 +57,8 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator UNUSED)\n }\n \n static int empty_ref_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n-\t\t\t\t const char *prefix UNUSED)\n+\t\t\t\t const char *refname UNUSED,\n+\t\t\t\t unsigned int flags UNUSED)\n {\n \treturn 0;\n }\n@@ -224,7 +225,7 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *refname, unsigned int flags)\n {\n \tstruct merge_ref_iterator *iter =\n \t\t(struct merge_ref_iterator *)ref_iterator;\n@@ -234,11 +235,11 @@ static int merge_ref_iterator_seek(struct ref_iterator *ref_iterator,\n \titer->iter0 = iter->iter0_owned;\n \titer->iter1 = iter->iter1_owned;\n \n-\tret = ref_iterator_seek(iter->iter0, prefix);\n+\tret = ref_iterator_seek(iter->iter0, refname, flags);\n \tif (ret < 0)\n \t\treturn ret;\n \n-\tret = ref_iterator_seek(iter->iter1, prefix);\n+\tret = ref_iterator_seek(iter->iter1, refname, flags);\n \tif (ret < 0)\n \t\treturn ret;\n \n@@ -407,13 +408,16 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int prefix_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *refname, unsigned int flags)\n {\n \tstruct prefix_ref_iterator *iter =\n \t\t(struct prefix_ref_iterator *)ref_iterator;\n-\tfree(iter->prefix);\n-\titer->prefix = xstrdup_or_null(prefix);\n-\treturn ref_iterator_seek(iter->iter0, prefix);\n+\n+\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX) {\n+\t\tfree(iter->prefix);\n+\t\titer->prefix = xstrdup_or_null(refname);\n+\t}\n+\treturn ref_iterator_seek(iter->iter0, refname, flags);\n }\n \n static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 7fd73a0e6d..5fa4ae6655 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -1004,19 +1004,23 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int packed_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *refname, unsigned int flags)\n {\n \tstruct packed_ref_iterator *iter =\n \t\t(struct packed_ref_iterator *)ref_iterator;\n \tconst char *start;\n \n-\tif (prefix && *prefix)\n-\t\tstart = find_reference_location(iter->snapshot, prefix, 0);\n+\tif (refname && *refname)\n+\t\tstart = find_reference_location(iter->snapshot, refname, 0);\n \telse\n \t\tstart = iter->snapshot->start;\n \n-\tfree(iter->prefix);\n-\titer->prefix = xstrdup_or_null(prefix);\n+\t/* Unset any previously set prefix */\n+\tFREE_AND_NULL(iter->prefix);\n+\n+\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX)\n+\t\titer->prefix = xstrdup_or_null(refname);\n+\n \titer->pos = start;\n \titer->eof = iter->snapshot->eof;\n \n@@ -1194,7 +1198,8 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \titer->repo = ref_store->repo;\n \titer->flags = flags;\n \n-\tif (packed_ref_iterator_seek(&iter->base, prefix) < 0) {\n+\tif (packed_ref_iterator_seek(&iter->base, prefix,\n+\t\t\t\t REF_ITERATOR_SEEK_SET_PREFIX) < 0) {\n \t\tref_iterator_free(&iter->base);\n \t\treturn NULL;\n \t}\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 8aaffa8c6b..1d95b56d40 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -434,11 +434,9 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t}\n }\n \n-static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+static int cache_ref_iterator_set_prefix(struct cache_ref_iterator *iter,\n+\t\t\t\t\t const char *prefix)\n {\n-\tstruct cache_ref_iterator *iter =\n-\t\t(struct cache_ref_iterator *)ref_iterator;\n \tstruct cache_ref_iterator_level *level;\n \tstruct ref_dir *dir;\n \n@@ -469,6 +467,82 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n \treturn 0;\n }\n \n+static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n+\t\t\t\t const char *refname, unsigned int flags)\n+{\n+\tstruct cache_ref_iterator *iter =\n+\t\t(struct cache_ref_iterator *)ref_iterator;\n+\n+\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX) {\n+\t\treturn cache_ref_iterator_set_prefix(iter, refname);\n+\t} else if (refname && *refname) {\n+\t\tstruct cache_ref_iterator_level *level;\n+\t\tconst char *slash = refname;\n+\t\tstruct ref_dir *dir;\n+\n+\t\tdir = get_ref_dir(iter->cache->root);\n+\n+\t\tif (iter->prime_dir)\n+\t\t\tprime_ref_dir(dir, refname);\n+\n+\t\titer->levels_nr = 1;\n+\t\tlevel = &iter->levels[0];\n+\t\tlevel->index = -1;\n+\t\tlevel->dir = dir;\n+\n+\t\t/* Unset any previously set prefix */\n+\t\tFREE_AND_NULL(iter->prefix);\n+\n+\t\t/*\n+\t\t * Breakdown the provided seek path and assign the correct\n+\t\t * indexing to each level as needed.\n+\t\t */\n+\t\tdo {\n+\t\t\tint len, idx;\n+\t\t\tint cmp = 0;\n+\n+\t\t\tsort_ref_dir(dir);\n+\n+\t\t\tslash = strchr(slash, '/');\n+\t\t\tlen = slash ? slash - refname : (int)strlen(refname);\n+\n+\t\t\tfor (idx = 0; idx < dir->nr; idx++) {\n+\t\t\t\tcmp = strncmp(refname, dir->entries[idx]->name, len);\n+\t\t\t\tif (cmp <= 0)\n+\t\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\t/* don't overflow the index */\n+\t\t\tidx = idx >= dir->nr ? dir->nr - 1 : idx;\n+\n+\t\t\tif (slash)\n+\t\t\t\tslash = slash + 1;\n+\n+\t\t\tlevel->index = idx;\n+\t\t\tif (dir->entries[idx]->flag & REF_DIR) {\n+\t\t\t\t/* push down a level */\n+\t\t\t\tdir = get_ref_dir(dir->entries[idx]);\n+\n+\t\t\t\tALLOC_GROW(iter->levels, iter->levels_nr + 1,\n+\t\t\t\t\t iter->levels_alloc);\n+\t\t\t\tlevel = &iter->levels[iter->levels_nr++];\n+\t\t\t\tlevel->dir = dir;\n+\t\t\t\tlevel->index = -1;\n+\t\t\t} else {\n+\t\t\t\t/* reduce the index so the leaf node is iterated over */\n+\t\t\t\tif (cmp <= 0 && !slash)\n+\t\t\t\t\tlevel->index = idx - 1;\n+\t\t\t\t/*\n+\t\t\t\t * while the seek path may not be exhausted, our\n+\t\t\t\t * match is exhausted at a leaf node.\n+\t\t\t\t */\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t} while (slash);\n+\t}\n+\n+\treturn 0;\n+}\n+\n static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t struct object_id *peeled)\n {\n@@ -509,7 +583,8 @@ struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\n \titer->cache = cache;\n \titer->prime_dir = prime_dir;\n \n-\tif (cache_ref_iterator_seek(&iter->base, prefix) < 0) {\n+\tif (cache_ref_iterator_seek(&iter->base, prefix,\n+\t\t\t\t REF_ITERATOR_SEEK_SET_PREFIX) < 0) {\n \t\tref_iterator_free(&iter->base);\n \t\treturn NULL;\n \t}\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 03f5df04d5..40c1c0f93d 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -353,11 +353,12 @@ void base_ref_iterator_init(struct ref_iterator *iter,\n typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);\n \n /*\n- * Seek the iterator to the first reference matching the given prefix. Should\n- * behave the same as if a new iterator was created with the same prefix.\n+ * Seek the iterator to the first matching reference. If the\n+ * REF_ITERATOR_SEEK_SET_PREFIX flag is set, it would behave the same as if a\n+ * new iterator was created with the provided refname as prefix.\n */\n typedef int ref_iterator_seek_fn(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix);\n+\t\t\t\t const char *refname, unsigned int flags);\n \n /*\n * Peels the current ref, returning 0 for success or -1 for failure.\ndiff --git a/refs/reftable-backend.c b/refs/reftable-backend.c\nindex 4c3817f4ec..c3d48cc412 100644\n--- a/refs/reftable-backend.c\n+++ b/refs/reftable-backend.c\n@@ -719,15 +719,20 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int reftable_ref_iterator_seek(struct ref_iterator *ref_iterator,\n-\t\t\t\t const char *prefix)\n+\t\t\t\t const char *refname, unsigned int flags)\n {\n \tstruct reftable_ref_iterator *iter =\n \t\t(struct reftable_ref_iterator *)ref_iterator;\n \n-\tfree(iter->prefix);\n-\titer->prefix = xstrdup_or_null(prefix);\n-\titer->prefix_len = prefix ? strlen(prefix) : 0;\n-\titer->err = reftable_iterator_seek_ref(&iter->iter, prefix);\n+\t/* Unset any previously set prefix */\n+\tFREE_AND_NULL(iter->prefix);\n+\titer->prefix_len = 0;\n+\n+\tif (flags & REF_ITERATOR_SEEK_SET_PREFIX) {\n+\t\titer->prefix = xstrdup_or_null(refname);\n+\t\titer->prefix_len = refname ? strlen(refname) : 0;\n+\t}\n+\titer->err = reftable_iterator_seek_ref(&iter->iter, refname);\n \n \treturn iter->err;\n }\n@@ -839,7 +844,8 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_\n \tif (ret)\n \t\tgoto done;\n \n-\tret = reftable_ref_iterator_seek(&iter->base, prefix);\n+\tret = reftable_ref_iterator_seek(&iter->base, prefix,\n+\t\t\t\t\t REF_ITERATOR_SEEK_SET_PREFIX);\n \tif (ret)\n \t\tgoto done;\n \n@@ -2042,7 +2048,8 @@ static int reftable_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n }\n \n static int reftable_reflog_iterator_seek(struct ref_iterator *ref_iterator UNUSED,\n-\t\t\t\t\t const char *prefix UNUSED)\n+\t\t\t\t\t const char *refname UNUSED,\n+\t\t\t\t\t unsigned int flags UNUSED)\n {\n \tBUG(\"reftable reflog iterator cannot be seeked\");\n \treturn -1;\n\n-- \n2.49.0\n\n"},{"id":"521970","messageId":"20250715-306-git-for-each-ref-pagination-v5-4-852d5a2f56e1@gmail.com","threadId":"63722","inReplyTo":"20250715-306-git-for-each-ref-pagination-v5-0-852d5a2f56e1@gmail.com","subject":"[PATCH v5 4/5] ref-filter: remove unnecessary else clause","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-15T11:28:29Z","receivedAt":"2025-07-15T11:28:37Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"In 'ref-filter.c', there is an 'else' clause within `do_filter_refs()`.\nThis is unnecessary since the 'if' clause calls `die()`, which would\nexit the program. So let's remove the unnecessary 'else' clause. This\nimproves readability since the indentation is also reduced and flow is\nsimpler.\n\nSigned-off-by: Karthik Nayak \n---\n ref-filter.c | 60 ++++++++++++++++++++++++++++++------------------------------\n 1 file changed, 30 insertions(+), 30 deletions(-)\n\ndiff --git a/ref-filter.c b/ref-filter.c\nindex 7a274633cf..da663c7ac8 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -3199,37 +3199,37 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n \t/* Simple per-ref filtering */\n \tif (!filter->kind)\n \t\tdie(\"filter_refs: invalid type\");\n-\telse {\n-\t\t/*\n-\t\t * For common cases where we need only branches or remotes or tags,\n-\t\t * we only iterate through those refs. If a mix of refs is needed,\n-\t\t * we iterate over all refs and filter out required refs with the help\n-\t\t * of filter_ref_kind().\n-\t\t */\n-\t\tif (filter->kind == FILTER_REFS_BRANCHES)\n-\t\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"refs/heads/\", NULL,\n-\t\t\t\t\t\t fn, cb_data);\n-\t\telse if (filter->kind == FILTER_REFS_REMOTES)\n-\t\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"refs/remotes/\", NULL,\n-\t\t\t\t\t\t fn, cb_data);\n-\t\telse if (filter->kind == FILTER_REFS_TAGS)\n-\t\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"refs/tags/\", NULL, fn,\n-\t\t\t\t\t\t cb_data);\n-\t\telse if (filter->kind & FILTER_REFS_REGULAR)\n-\t\t\tret = for_each_fullref_in_pattern(filter, fn, cb_data);\n \n-\t\t/*\n-\t\t * When printing all ref types, HEAD is already included,\n-\t\t * so we don't want to print HEAD again.\n-\t\t */\n-\t\tif (!ret && !(filter->kind & FILTER_REFS_ROOT_REFS) &&\n-\t\t (filter->kind & FILTER_REFS_DETACHED_HEAD))\n-\t\t\trefs_head_ref(get_main_ref_store(the_repository), fn,\n-\t\t\t\t cb_data);\n-\t}\n+\t/*\n+\t * For common cases where we need only branches or remotes or tags,\n+\t * we only iterate through those refs. If a mix of refs is needed,\n+\t * we iterate over all refs and filter out required refs with the help\n+\t * of filter_ref_kind().\n+\t */\n+\tif (filter->kind == FILTER_REFS_BRANCHES)\n+\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n+\t\t\t\t\t \"refs/heads/\", NULL,\n+\t\t\t\t\t fn, cb_data);\n+\telse if (filter->kind == FILTER_REFS_REMOTES)\n+\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n+\t\t\t\t\t \"refs/remotes/\", NULL,\n+\t\t\t\t\t fn, cb_data);\n+\telse if (filter->kind == FILTER_REFS_TAGS)\n+\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n+\t\t\t\t\t \"refs/tags/\", NULL, fn,\n+\t\t\t\t\t cb_data);\n+\telse if (filter->kind & FILTER_REFS_REGULAR)\n+\t\tret = for_each_fullref_in_pattern(filter, fn, cb_data);\n+\n+\t/*\n+\t * When printing all ref types, HEAD is already included,\n+\t * so we don't want to print HEAD again.\n+\t */\n+\tif (!ret && !(filter->kind & FILTER_REFS_ROOT_REFS) &&\n+\t (filter->kind & FILTER_REFS_DETACHED_HEAD))\n+\t\trefs_head_ref(get_main_ref_store(the_repository), fn,\n+\t\t\t cb_data);\n+\n \n \tclear_contains_cache(&filter->internal.contains_cache);\n \tclear_contains_cache(&filter->internal.no_contains_cache);\n\n-- \n2.49.0\n\n"},{"id":"521971","messageId":"20250715-306-git-for-each-ref-pagination-v5-5-852d5a2f56e1@gmail.com","threadId":"63722","inReplyTo":"20250715-306-git-for-each-ref-pagination-v5-0-852d5a2f56e1@gmail.com","subject":"[PATCH v5 5/5] for-each-ref: introduce a '--start-after' option","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-15T11:28:30Z","receivedAt":"2025-07-15T11:28:39Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The `git-for-each-ref(1)` command is used to iterate over references\npresent in a repository. In large repositories with millions of\nreferences, it would be optimal to paginate this output such that we\ncan start iteration from a given reference. This would avoid having to\niterate over all references from the beginning each time when paginating\nthrough results.\n\nThe previous commit added 'seek' functionality to the reference\nbackends. Utilize this and expose a '--start-after' option in\n'git-for-each-ref(1)'. When used, the reference iteration seeks to the\nlexicographically next reference and iterates from there onward.\n\nThis enables efficient pagination workflows, where the calling script\ncan remember the last provided reference and use that as the starting\npoint for the next set of references:\n git for-each-ref --count=100\n git for-each-ref --count=100 --start-after=refs/heads/branch-100\n git for-each-ref --count=100 --start-after=refs/heads/branch-200\n\nSince the reference iterators only allow seeking to a specified marker\nvia the `ref_iterator_seek()`, we introduce a helper function\n`start_ref_iterator_after()`, which seeks to next reference by simply\nadding (char) 1 to the marker.\n\nWe must note that pagination always continues from the provided marker,\nas such any concurrent reference updates lexicographically behind the\nmarker will not be output. Document the same.\n\nSigned-off-by: Karthik Nayak \n---\n Documentation/git-for-each-ref.adoc | 10 +-\n builtin/for-each-ref.c | 8 ++\n ref-filter.c | 78 +++++++++++----\n ref-filter.h | 1 +\n t/t6302-for-each-ref-filter.sh | 194 ++++++++++++++++++++++++++++++++++++\n 5 files changed, 272 insertions(+), 19 deletions(-)\n\ndiff --git a/Documentation/git-for-each-ref.adoc b/Documentation/git-for-each-ref.adoc\nindex 5ef89fc0fe..ae61ba642a 100644\n--- a/Documentation/git-for-each-ref.adoc\n+++ b/Documentation/git-for-each-ref.adoc\n@@ -14,7 +14,7 @@ SYNOPSIS\n \t\t [--points-at=]\n \t\t [--merged[=]] [--no-merged[=]]\n \t\t [--contains[=]] [--no-contains[=]]\n-\t\t [--exclude= ...]\n+\t\t [--exclude= ...] [--start-after=]\n \n DESCRIPTION\n -----------\n@@ -108,6 +108,14 @@ TAB %(refname)`.\n --include-root-refs::\n \tList root refs (HEAD and pseudorefs) apart from regular refs.\n \n+--start-after=::\n+ Allows paginating the output by skipping references up to and including the\n+ specified marker. When paging, it should be noted that references may be\n+ deleted, modified or added between invocations. Output will only yield those\n+ references which follow the marker lexicographically. Output begins from the\n+ first reference that would come after the marker alphabetically. Cannot be\n+ used with general pattern matching or custom sort options.\n+\n FIELD NAMES\n -----------\n \ndiff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c\nindex 3d2207ec77..3f21598046 100644\n--- a/builtin/for-each-ref.c\n+++ b/builtin/for-each-ref.c\n@@ -13,6 +13,7 @@ static char const * const for_each_ref_usage[] = {\n \tN_(\"git for-each-ref [--points-at ]\"),\n \tN_(\"git for-each-ref [--merged []] [--no-merged []]\"),\n \tN_(\"git for-each-ref [--contains []] [--no-contains []]\"),\n+\tN_(\"git for-each-ref [--start-after ]\"),\n \tNULL\n };\n \n@@ -44,6 +45,7 @@ int cmd_for_each_ref(int argc,\n \t\tOPT_GROUP(\"\"),\n \t\tOPT_INTEGER( 0 , \"count\", &format.array_opts.max_count, N_(\"show only matched refs\")),\n \t\tOPT_STRING( 0 , \"format\", &format.format, N_(\"format\"), N_(\"format to use for the output\")),\n+\t\tOPT_STRING( 0 , \"start-after\", &filter.start_after, N_(\"start-start\"), N_(\"start iteration after the provided marker\")),\n \t\tOPT__COLOR(&format.use_color, N_(\"respect format colors\")),\n \t\tOPT_REF_FILTER_EXCLUDE(&filter),\n \t\tOPT_REF_SORT(&sorting_options),\n@@ -79,6 +81,9 @@ int cmd_for_each_ref(int argc,\n \tif (verify_ref_format(&format))\n \t\tusage_with_options(for_each_ref_usage, opts);\n \n+\tif (filter.start_after && sorting_options.nr > 1)\n+\t\tdie(_(\"cannot use --start-after with custom sort options\"));\n+\n \tsorting = ref_sorting_options(&sorting_options);\n \tref_sorting_set_sort_flags_all(sorting, REF_SORTING_ICASE, icase);\n \tfilter.ignore_case = icase;\n@@ -100,6 +105,9 @@ int cmd_for_each_ref(int argc,\n \t\tfilter.name_patterns = argv;\n \t}\n \n+\tif (filter.start_after && filter.name_patterns && filter.name_patterns[0])\n+\t\tdie(_(\"cannot use --start-after with patterns\"));\n+\n \tif (include_root_refs)\n \t\tflags |= FILTER_REFS_ROOT_REFS | FILTER_REFS_DETACHED_HEAD;\n \ndiff --git a/ref-filter.c b/ref-filter.c\nindex da663c7ac8..c8a6b7f1af 100644\n--- a/ref-filter.c\n+++ b/ref-filter.c\n@@ -2683,6 +2683,41 @@ static int filter_exclude_match(struct ref_filter *filter, const char *refname)\n \treturn match_pattern(filter->exclude.v, refname, filter->ignore_case);\n }\n \n+/*\n+ * We need to seek to the reference right after a given marker but excluding any\n+ * matching references. So we seek to the lexicographically next reference.\n+ */\n+static int start_ref_iterator_after(struct ref_iterator *iter, const char *marker)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint ret;\n+\n+\tstrbuf_addstr(&sb, marker);\n+\tstrbuf_addch(&sb, 1);\n+\n+\tret = ref_iterator_seek(iter, sb.buf, 0);\n+\n+\tstrbuf_release(&sb);\n+\treturn ret;\n+}\n+\n+static int for_each_fullref_with_seek(struct ref_filter *filter, each_ref_fn cb,\n+\t\t\t\t void *cb_data, unsigned int flags)\n+{\n+\tstruct ref_iterator *iter;\n+\tint ret = 0;\n+\n+\titer = refs_ref_iterator_begin(get_main_ref_store(the_repository), \"\",\n+\t\t\t\t NULL, 0, flags);\n+\tif (filter->start_after)\n+\t\tret = start_ref_iterator_after(iter, filter->start_after);\n+\n+\tif (ret)\n+\t\treturn ret;\n+\n+\treturn do_for_each_ref_iterator(iter, cb, cb_data);\n+}\n+\n /*\n * This is the same as for_each_fullref_in(), but it tries to iterate\n * only over the patterns we'll care about. Note that it _doesn't_ do a full\n@@ -2694,8 +2729,8 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n {\n \tif (filter->kind & FILTER_REFS_ROOT_REFS) {\n \t\t/* In this case, we want to print all refs including root refs. */\n-\t\treturn refs_for_each_include_root_refs(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t cb, cb_data);\n+\t\treturn for_each_fullref_with_seek(filter, cb, cb_data,\n+\t\t\t\t\t\t DO_FOR_EACH_INCLUDE_ROOT_REFS);\n \t}\n \n \tif (!filter->match_as_path) {\n@@ -2704,8 +2739,7 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t * prefixes like \"refs/heads/\" etc. are stripped off,\n \t\t * so we have to look at everything:\n \t\t */\n-\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t\"\", NULL, cb, cb_data);\n+\t\treturn for_each_fullref_with_seek(filter, cb, cb_data, 0);\n \t}\n \n \tif (filter->ignore_case) {\n@@ -2714,14 +2748,12 @@ static int for_each_fullref_in_pattern(struct ref_filter *filter,\n \t\t * so just return everything and let the caller\n \t\t * sort it out.\n \t\t */\n-\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t\"\", NULL, cb, cb_data);\n+\t\treturn for_each_fullref_with_seek(filter, cb, cb_data, 0);\n \t}\n \n \tif (!filter->name_patterns[0]) {\n \t\t/* no patterns; we have to look at everything */\n-\t\treturn refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t\t \"\", filter->exclude.v, cb, cb_data);\n+\t\treturn for_each_fullref_with_seek(filter, cb, cb_data, 0);\n \t}\n \n \treturn refs_for_each_fullref_in_prefixes(get_main_ref_store(the_repository),\n@@ -3189,6 +3221,7 @@ void filter_is_base(struct repository *r,\n \n static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref_fn fn, void *cb_data)\n {\n+\tconst char *prefix = NULL;\n \tint ret = 0;\n \n \tfilter->kind = type & FILTER_REFS_KIND_MASK;\n@@ -3207,19 +3240,28 @@ static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref\n \t * of filter_ref_kind().\n \t */\n \tif (filter->kind == FILTER_REFS_BRANCHES)\n-\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t \"refs/heads/\", NULL,\n-\t\t\t\t\t fn, cb_data);\n+\t\tprefix = \"refs/heads/\";\n \telse if (filter->kind == FILTER_REFS_REMOTES)\n-\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t \"refs/remotes/\", NULL,\n-\t\t\t\t\t fn, cb_data);\n+\t\tprefix = \"refs/remotes/\";\n \telse if (filter->kind == FILTER_REFS_TAGS)\n-\t\tret = refs_for_each_fullref_in(get_main_ref_store(the_repository),\n-\t\t\t\t\t \"refs/tags/\", NULL, fn,\n-\t\t\t\t\t cb_data);\n-\telse if (filter->kind & FILTER_REFS_REGULAR)\n+\t\tprefix = \"refs/tags/\";\n+\n+\tif (prefix) {\n+\t\tstruct ref_iterator *iter;\n+\n+\t\titer = refs_ref_iterator_begin(get_main_ref_store(the_repository),\n+\t\t\t\t\t \"\", NULL, 0, 0);\n+\n+\t\tif (filter->start_after)\n+\t\t\tret = start_ref_iterator_after(iter, filter->start_after);\n+\t\telse if (prefix)\n+\t\t\tret = ref_iterator_seek(iter, prefix, 1);\n+\n+\t\tif (!ret)\n+\t\t\tret = do_for_each_ref_iterator(iter, fn, cb_data);\n+\t} else if (filter->kind & FILTER_REFS_REGULAR) {\n \t\tret = for_each_fullref_in_pattern(filter, fn, cb_data);\n+\t}\n \n \t/*\n \t * When printing all ref types, HEAD is already included,\ndiff --git a/ref-filter.h b/ref-filter.h\nindex c98c4fbd4c..f22ca94b49 100644\n--- a/ref-filter.h\n+++ b/ref-filter.h\n@@ -64,6 +64,7 @@ struct ref_array {\n \n struct ref_filter {\n \tconst char **name_patterns;\n+\tconst char *start_after;\n \tstruct strvec exclude;\n \tstruct oid_array points_at;\n \tstruct commit_list *with_commit;\ndiff --git a/t/t6302-for-each-ref-filter.sh b/t/t6302-for-each-ref-filter.sh\nindex bb02b86c16..e097db6b02 100755\n--- a/t/t6302-for-each-ref-filter.sh\n+++ b/t/t6302-for-each-ref-filter.sh\n@@ -541,4 +541,198 @@ test_expect_success 'validate worktree atom' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'start after with empty value' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/heads/main\n+\trefs/heads/main_worktree\n+\trefs/heads/side\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=\"\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after a specific reference' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/spot >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after a specific reference with partial match' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/sp >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after, just behind a specific reference' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/parrot >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after with specific directory match' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after with specific directory and trailing slash' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after, just behind a specific directory' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/odd/spot\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/lost >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after, overflow specific reference length' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/spotnew >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after, overflow specific reference path' '\n+\tcat >expect <<-\\EOF &&\n+\trefs/tags/annotated-tag\n+\trefs/tags/doubly-annotated-tag\n+\trefs/tags/doubly-signed-tag\n+\trefs/tags/foo1.10\n+\trefs/tags/foo1.3\n+\trefs/tags/foo1.6\n+\trefs/tags/four\n+\trefs/tags/one\n+\trefs/tags/signed-tag\n+\trefs/tags/three\n+\trefs/tags/two\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/odd/spot/new >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after, last reference' '\n+\tcat >expect <<-\\EOF &&\n+\tEOF\n+\tgit for-each-ref --format=\"%(refname)\" --start-after=refs/tags/two >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after used with a pattern' '\n+\tcat >expect <<-\\EOF &&\n+\tfatal: cannot use --start-after with patterns\n+\tEOF\n+\ttest_must_fail git for-each-ref --format=\"%(refname)\" --start-after=refs/odd/spot refs/tags 2>actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'start after used with custom sort order' '\n+\tcat >expect <<-\\EOF &&\n+\tfatal: cannot use --start-after with custom sort options\n+\tEOF\n+\ttest_must_fail git for-each-ref --format=\"%(refname)\" --start-after=refs/odd/spot --sort=author 2>actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n\n-- \n2.49.0\n\n"},{"id":"522010","messageId":"xmqqms95if8e.fsf@gitster.g","threadId":"63722","inReplyTo":"CAOLa=ZS6ASf1+nbUnCTeeH1Di=kgmhQUEQ5UXPZu051rfqx9Pw@mail.gmail.com","subject":"Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-15T16:35:45Z","receivedAt":"2025-07-15T16:35:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n>> so after (true) seeking that does not have the SET_PREFIX flag on,\n>> wouldn't our iterator-advance run through the end since it no longer\n>> is aware of where to stop?\n>>\n>\n> That's also right and that is indeed the intention. We're trying to make\n> the actions more intentional.\n>\n> So if a user sets a 'prefix' for the iterator, all previous state of the\n> iterator is reset. So, the same function for seeking an iterator should\n> also have the same side-effect of resetting the previous state.\n\nPerhaps we have different definition of \"previous state\" in mind?\nSo let's imagine an iterator is walking over all branches (i.e. the\nprefix is set to refs/heads/, to allow it to stop once it steps\noutside refs/heads/ and moves over to refs/imerge). It starts\niterating and I see branches whose name sorts early in alphabetical\norder. I tell it to seek to refs/heads/master and keep iterating.\n\nWouldn't it be a lot more natural if it still stops iterating after\nit finishes showing the last branch, iow, a ref in refs/heads/\nhierarchy? In other words, I am not sure why ...\n\n> There could be a usecase where we add support for keeping the prefix,\n> while also seeking the iterator. That would be an explicit change\n\n... that is the optional and unimplemented feature, not the other\nway around. Is it just the ease of implementation?\n\n"},{"id":"522028","messageId":"xmqqple1gtyg.fsf@gitster.g","threadId":"63722","inReplyTo":"20250715-306-git-for-each-ref-pagination-v5-0-852d5a2f56e1@gmail.com","subject":"Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-15T19:00:39Z","receivedAt":"2025-07-15T19:00:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> Changes in v5:\n> - Changes to the comments to refer to the flag\n> 'REF_ITERATOR_SEEK_SET_PREFIX' instead of a variable used in older\n> versions. Also other small grammar fixes.\n> - Added a commit to remove an unnecessary else clause.\n> - Move seeking functionality within `for_each_fullref_in_pattern` to its\n> own function.\n> - Fix incorrect naming in the tests.\n> - Link to v4: https://lore.kernel.org/r/20250711-306-git-for-each-ref-pagination-v4-0-ed3303ad5b89@gmail.com\n\nThe two refactoring differences relative to the previous round do\nmake the result more plesant to eyes. Looking great.\n\nWill replace. Thanks.\n"},{"id":"522110","messageId":"CAOLa=ZTc2_g3+8MM8whW+eZAj-+36HNsnUFDoLgnN42ytFLwAw@mail.gmail.com","threadId":"63722","inReplyTo":"xmqqms95if8e.fsf@gitster.g","subject":"Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-16T14:40:30Z","receivedAt":"2025-07-16T14:40:32Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Karthik Nayak writes:\n>\n>>> so after (true) seeking that does not have the SET_PREFIX flag on,\n>>> wouldn't our iterator-advance run through the end since it no longer\n>>> is aware of where to stop?\n>>>\n>>\n>> That's also right and that is indeed the intention. We're trying to make\n>> the actions more intentional.\n>>\n>> So if a user sets a 'prefix' for the iterator, all previous state of the\n>> iterator is reset. So, the same function for seeking an iterator should\n>> also have the same side-effect of resetting the previous state.\n>\n> Perhaps we have different definition of \"previous state\" in mind?\n> So let's imagine an iterator is walking over all branches (i.e. the\n> prefix is set to refs/heads/, to allow it to stop once it steps\n> outside refs/heads/ and moves over to refs/imerge). It starts\n> iterating and I see branches whose name sorts early in alphabetical\n> order. I tell it to seek to refs/heads/master and keep iterating.\n>\n\nI get what you're saying and indeed that would be natural. Let me draw\nanother example to draw the contrast.\n\nLet's say a user is iterating with a prefix set to 'refs/heads/', this\nwould iterate over all the refs with that prefix. But mid-way the user\nrealizes that they only care about 'refs/heads/feature/' prefix and they\nask the iterator to set that as the prefix.\n\nIn such a situation, the iterator seeks to 'refs/heads/feature/' and\nwill only yield references with that prefix. In short, the previous\nprefix state was reset.\n\nSo to avoid the two scenarios:\n\n1. Only seek the iterator but maintain prefix\n2. Seek and set new prefix, loosing old prefix\n\nWhere one resets the prefix while the other doesn't. We make it explicit\nand say, whenever 'ref_iterator_seek' is called, any set prefix is\nreset. I do see the other way around too, where prefix isn't treated as\nprevious state.\n\nWhat I was trying to argue for, was that, there could be a situation\nlike what you mentioned, where a user might want to retain a prefix,\nthis should be an explicit requirement which not implemented in this\nseries. So as of this series, you cannot set a prefix and then seek and\nexpect to retain the prefix.\n\n> Wouldn't it be a lot more natural if it still stops iterating after\n> it finishes showing the last branch, iow, a ref in refs/heads/\n> hierarchy? In other words, I am not sure why ...\n>\n>> There could be a usecase where we add support for keeping the prefix,\n>> while also seeking the iterator. That would be an explicit change\n>\n> ... that is the optional and unimplemented feature, not the other\n> way around. Is it just the ease of implementation?\n\nThis series did start out that way around, so ease of implementation\nisn't it. It was more of a side-effect of not clearing state. But I\nwould be more comfortable if this wasn't a side-effect but rather a\nconscious choice with tests and adequate documentation.\n"},{"id":"522119","messageId":"xmqqjz48cfg7.fsf@gitster.g","threadId":"63722","inReplyTo":"CAOLa=ZTc2_g3+8MM8whW+eZAj-+36HNsnUFDoLgnN42ytFLwAw@mail.gmail.com","subject":"Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-16T15:39:52Z","receivedAt":"2025-07-16T15:39:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> Let's say a user is iterating with a prefix set to 'refs/heads/', this\n> would iterate over all the refs with that prefix. But mid-way the user\n> realizes that they only care about 'refs/heads/feature/' prefix and they\n> ask the iterator to set that as the prefix.\n\nBut is that user who changes their mind in the middle \"seeking\"?\n\nIt is more like \"I am abandoning the enumeration I started earlier\nover refs/heads/, and I want a different enumeration over\nrefs/heads/feature/, but because I know the implementation detail\nthat abandoning an iterator and creating another is more expensive,\nlet me reuse the one in use to repurpose it\".\n\nI wouldn't call it \"seeking\"; it sounds more like \"resetting\".\n"},{"id":"522125","messageId":"xmqqfrevc3b6.fsf@gitster.g","threadId":"63722","inReplyTo":"CAOLa=ZTc2_g3+8MM8whW+eZAj-+36HNsnUFDoLgnN42ytFLwAw@mail.gmail.com","subject":"Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-16T20:02:05Z","receivedAt":"2025-07-16T20:02:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> Let's say a user is iterating with a prefix set to 'refs/heads/', this\n> would iterate over all the refs with that prefix. But mid-way the user\n> realizes that they only care about 'refs/heads/feature/' prefix and they\n> ask the iterator to set that as the prefix.\n>\n> In such a situation, the iterator seeks to 'refs/heads/feature/' and\n> will only yield references with that prefix. In short, the previous\n> prefix state was reset.\n\nYes, even though I wouldn't call such an operation \"seek\", \"Ah, I do\nnot need the entire refs/heads/ walked, only refs/heads/feature/ is\nenough\" is an operation mode that makes sense.\n\nBut not for paging, though.\n\nIf your web application is showing all branches, one pageful at a\ntime, and the first page ended at refs/heads/feature/something and\nyou ended up \"seeking\" to refs/heads/feature/ to start the second\npage, you do not want your second page to end when the iteration\ngoes out of refs/heads/feature/ hierarchy, no?\n\nIt seems to me that the root cause of the confusion is because\nprefix, which is to let iteration finish way before the data runs\nout (instead finish when the iteration steps out of a given\nsubhierarchy denoted by the prefix), is somehow abused as the\ncurrent position of the cursor. Shouldn't they be two separate\nconcepts? The cursor needs to fall within the prefix while the\niterator is active, so they are not two totally independent things,\nbut prefix is pretty much static while the cursor position is very\ndynamic.\n\n> This series did start out that way around, so ease of implementation\n> isn't it. It was more of a side-effect of not clearing state.\n\nI am even more worried about usability and correctness aspect of\nwhat was described here now. After seeking to refs/heads/feature/,\ndo we continue to iterate and step out of refs/heads/feature/\nhierarchy or can we cut off a particular page that started with a\nref within refs/heads/feature/ subhierarchy when we exhaust refs in\nrefs/heads/feature/ and have to wait for getting asked for the next\npage before we show refs/heads/gsomething that is outside\nrefs/heads/feature/ and sorts after? The \"I reset to iterate over\nrefs/heads/feature/ because the entire refs/heads/ is not what I\ncare about\" example makes me worried about this.\n\nThanks.\n\n"},{"id":"522134","messageId":"CAO_smVg9TDakUnubepjPGmLyOzW6n8Z=MDbnZKvkwN2=kN2RRw@mail.gmail.com","threadId":"63722","inReplyTo":"xmqqple1gtyg.fsf@gitster.g","subject":"Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'","fromName":"Kyle Lippincott","fromEmail":"spectral@google.com","sentAt":"2025-07-17T01:19:32Z","receivedAt":"2025-07-17T01:19:45Z","isPatch":true,"sender":{"key":"spectral@google.com","avatar":"https://avatars.githubusercontent.com/u/6371650?v=4"},"body":"There's something in this series that's triggering an msan warning in\nt/t6302-for-each-ref-filter:\n\n==2147==WARNING: MemorySanitizer: use-of-uninitialized-value\n #0 0x562a64e923cc in cache_ref_iterator_advance refs/ref-cache.c:409:27\n #1 0x562a64e88dc0 in ref_iterator_advance refs/iterator.c:15:9\n #2 0x562a64e88dc0 in merge_ref_iterator_advance refs/iterator.c:164:13\n #3 0x562a64e8850b in ref_iterator_advance refs/iterator.c:15:9\n #4 0x562a64e85f5a in files_ref_iterator_advance refs/files-backend.c:902:15\n #5 0x562a64e88bb4 in ref_iterator_advance refs/iterator.c:15:9\n #6 0x562a64e88bb4 in do_for_each_ref_iterator refs/iterator.c:478:15\n #7 0x562a64e64593 in for_each_fullref_with_seek ref-filter.c:2718:9\n #8 0x562a64e5cfe8 in for_each_fullref_in_pattern ref-filter.c\n #9 0x562a64e5cfe8 in do_filter_refs ref-filter.c:3263:9\n #10 0x562a64e5d7fc in filter_and_format_refs ref-filter.c:3364:3\n #11 0x562a64af0235 in cmd_for_each_ref builtin/for-each-ref.c:115:2\n #12 0x562a64a3ebdc in run_builtin git.c:480:11\n #13 0x562a64a3d342 in handle_builtin git.c:746:9\n #14 0x562a64a3be33 in run_argv git.c:813:4\n #15 0x562a64a3be33 in cmd_main git.c:953:19\n #16 0x562a64c2f12f in main common-main.c:9:11\n\nSUMMARY: MemorySanitizer: use-of-uninitialized-value\nrefs/ref-cache.c:409:27 in cache_ref_iterator_advance\n\nUnfortunately I can't provide great instructions for reproducing this\nlocally, because it relies on our internal build stack (which uses\nblaze). Getting MemorySanitizer running can be quite annoying, though\nyou might not have any issues if this test doesn't invoke any third\nparty libraries (like zlib).\n\nI need to sign off for the night soon, but if this isn't sufficient\nenough information to identify what's happening here, I can try to dig\ndeeper tomorrow. This run was executed on an import of upstream commit\n4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc (Junio's merge of this\nseries)\n"},{"id":"522144","messageId":"20250717015402.GA2127425@coredump.intra.peff.net","threadId":"63722","inReplyTo":"CAO_smVg9TDakUnubepjPGmLyOzW6n8Z=MDbnZKvkwN2=kN2RRw@mail.gmail.com","subject":"Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-07-17T01:54:02Z","receivedAt":"2025-07-17T01:54:06Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 16, 2025 at 06:19:32PM -0700, Kyle Lippincott wrote:\n\n> Unfortunately I can't provide great instructions for reproducing this\n> locally, because it relies on our internal build stack (which uses\n> blaze). Getting MemorySanitizer running can be quite annoying, though\n> you might not have any issues if this test doesn't invoke any third\n> party libraries (like zlib).\n> \n> I need to sign off for the night soon, but if this isn't sufficient\n> enough information to identify what's happening here, I can try to dig\n> deeper tomorrow. This run was executed on an import of upstream commit\n> 4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc (Junio's merge of this\n> series)\n\nvalgrind can often find the same issues as MSan without as much headache\nto get it running (the downside is that it is _way_ slower). And indeed:\n\n git checkout 4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc &&\n make &&\n (cd t && ./t6302-for-each-ref-filter.sh --valgrind-only=48)\n\nyields:\n\n ==2177572== Conditional jump or move depends on uninitialised value(s)\n ==2177572== at 0x3BC380: cache_ref_iterator_advance (ref-cache.c:409)\n ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15)\n ==2177572== by 0x3B6CC3: merge_ref_iterator_advance (iterator.c:179)\n ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15)\n ==2177572== by 0x3A9770: files_ref_iterator_advance (files-backend.c:902)\n ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15)\n ==2177572== by 0x3B7457: do_for_each_ref_iterator (iterator.c:478)\n ==2177572== by 0x399B43: for_each_fullref_with_seek (ref-filter.c:2718)\n ==2177572== by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756)\n ==2177572== by 0x39B031: do_filter_refs (ref-filter.c:3263)\n ==2177572== by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364)\n ==2177572== by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115)\n ==2177572== Uninitialised value was created by a heap allocation\n ==2177572== at 0x484BDD0: realloc (vg_replace_malloc.c:1801)\n ==2177572== by 0x44E941: xrealloc (wrapper.c:140)\n ==2177572== by 0x3BCAD9: cache_ref_iterator_begin (ref-cache.c:580)\n ==2177572== by 0x3A988A: files_ref_iterator_begin (files-backend.c:995)\n ==2177572== by 0x3A295E: refs_ref_iterator_begin (refs.c:1776)\n ==2177572== by 0x399AF6: for_each_fullref_with_seek (ref-filter.c:2710)\n ==2177572== by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756)\n ==2177572== by 0x39B031: do_filter_refs (ref-filter.c:3263)\n ==2177572== by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364)\n ==2177572== by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115)\n ==2177572== by 0x128C90: run_builtin (git.c:480)\n ==2177572== by 0x1290EB: handle_builtin (git.c:746)\n\nBisecting doesn't tell us much, though (the first commit that introduces\nthe test shows the problem). I didn't dig further than that.\n\n-Peff\n"},{"id":"522145","messageId":"20250717020905.GA2193264@coredump.intra.peff.net","threadId":"63722","inReplyTo":"20250715-306-git-for-each-ref-pagination-v5-3-852d5a2f56e1@gmail.com","subject":"Re: [PATCH v5 3/5] refs: selectively set prefix in the seek functions","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-07-17T02:09:05Z","receivedAt":"2025-07-17T02:09:06Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 15, 2025 at 01:28:28PM +0200, Karthik Nayak wrote:\n\n> +static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n> +\t\t\t\t const char *refname, unsigned int flags)\n> [...]\n> +\t\tdo {\n> +\t\t\tint len, idx;\n> +\t\t\tint cmp = 0;\n> +\n> +\t\t\tsort_ref_dir(dir);\n> +\n> +\t\t\tslash = strchr(slash, '/');\n> +\t\t\tlen = slash ? slash - refname : (int)strlen(refname);\n\nI was looking at this code due to a nearby thread and noticed this funny\ncast to int. I guess you added it to silence -Wsign-compare, but Why are\nwe not using a size_t in the first place?\n\nThis kind of conversion can sometimes have security implications because\na very large \"refname\" would cause \"len\" to become negative (i.e., if\nit's between 2GB and 4GB).\n\nIn this particular case it ends up cast back to a size_t via strncmp:\n\n> +\t\t\tfor (idx = 0; idx < dir->nr; idx++) {\n> +\t\t\t\tcmp = strncmp(refname, dir->entries[idx]->name, len);\n> +\t\t\t\tif (cmp <= 0)\n> +\t\t\t\t\tbreak;\n> +\t\t\t}\n\nso we get the original value back. We'd still get truncation for a\nrefname value over 4GB, which would presumably give us a slightly wrong\nanswer. But I don't think we'd ever look outside the array.\n\nSuch sizes are probably unlikely if we are feeding filesystem paths. But\nwe probably should not set a bad example, and just do;\n\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 1d95b56d40..3949d145e8 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -498,13 +498,14 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n \t\t * indexing to each level as needed.\n \t\t */\n \t\tdo {\n-\t\t\tint len, idx;\n+\t\t\tsize_t len;\n+\t\t\tint idx;\n \t\t\tint cmp = 0;\n \n \t\t\tsort_ref_dir(dir);\n \n \t\t\tslash = strchr(slash, '/');\n-\t\t\tlen = slash ? slash - refname : (int)strlen(refname);\n+\t\t\tlen = slash ? slash - refname : strlen(refname);\n \n \t\t\tfor (idx = 0; idx < dir->nr; idx++) {\n \t\t\t\tcmp = strncmp(refname, dir->entries[idx]->name, len);\n\n-Peff\n"},{"id":"522179","messageId":"CAOLa=ZSKsdGDERNS_1hJrBMn0muSOJ46SZz2bn8OHZndJ2Hu3g@mail.gmail.com","threadId":"63722","inReplyTo":"xmqqfrevc3b6.fsf@gitster.g","subject":"Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-17T09:01:05Z","receivedAt":"2025-07-17T09:01:08Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Karthik Nayak writes:\n>\n>> Let's say a user is iterating with a prefix set to 'refs/heads/', this\n>> would iterate over all the refs with that prefix. But mid-way the user\n>> realizes that they only care about 'refs/heads/feature/' prefix and they\n>> ask the iterator to set that as the prefix.\n>>\n>> In such a situation, the iterator seeks to 'refs/heads/feature/' and\n>> will only yield references with that prefix. In short, the previous\n>> prefix state was reset.\n>\n> Yes, even though I wouldn't call such an operation \"seek\", \"Ah, I do\n> not need the entire refs/heads/ walked, only refs/heads/feature/ is\n> enough\" is an operation mode that makes sense.\n>\n> But not for paging, though.\n>\n> If your web application is showing all branches, one pageful at a\n> time, and the first page ended at refs/heads/feature/something and\n> you ended up \"seeking\" to refs/heads/feature/ to start the second\n> page, you do not want your second page to end when the iteration\n> goes out of refs/heads/feature/ hierarchy, no?\n>\n\nYup and this (we show all references beyond the seek) is the current\nimplementation. I was talking about the internal implementation of\n'refs_iteration_seek()' which is the function used for seek and setting\nthe prefix.\n\nTo clarify, this is the current implementation:\n\n$ git for-each-ref\n6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit\trefs/heads/bar\n6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit\trefs/heads/feature/x\n6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit\trefs/heads/feature/y\n6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit\trefs/heads/foo\n6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit\trefs/heads/goo/x\n6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit\trefs/heads/goo/y\n6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit\trefs/heads/master\n6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit\trefs/tags/tagged/2\n6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit\trefs/tags/tagged/3\n6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit\trefs/tags/v1\n6f4b58c4968eb82277cf5b1cf8775117e5b83de0 commit\trefs/tags/v2\n\n$ git for-each-ref --format=\"%(refname)\" --start-after=refs/heads/goo\nrefs/heads/goo/x\nrefs/heads/goo/y\nrefs/heads/master\nrefs/tags/tagged/2\nrefs/tags/tagged/3\nrefs/tags/v1\nrefs/tags/v2\n\n$ git for-each-ref --format=\"%(refname)\" --start-after=refs/heads/master\nrefs/tags/tagged/2\nrefs/tags/tagged/3\nrefs/tags/v1\nrefs/tags/v2\n\n$ git for-each-ref --format=\"%(refname)\" --start-after=refs/heads/goo/x\nrefs/heads/goo/y\nrefs/heads/master\nrefs/tags/tagged/2\nrefs/tags/tagged/3\nrefs/tags/v1\nrefs/tags/v2\n\n$ git for-each-ref --format=\"%(refname)\" refs/heads/feature\nrefs/heads/feature/x\nrefs/heads/feature/y\n\nYou can see we list all references beyond the seek.\n\n> It seems to me that the root cause of the confusion is because\n> prefix, which is to let iteration finish way before the data runs\n> out (instead finish when the iteration steps out of a given\n> subhierarchy denoted by the prefix), is somehow abused as the\n> current position of the cursor. Shouldn't they be two separate\n> concepts? The cursor needs to fall within the prefix while the\n> iterator is active, so they are not two totally independent things,\n> but prefix is pretty much static while the cursor position is very\n> dynamic.\n>\n\nThe prefix setup in 'ref_iteration_seek' does two things, let's consider\nprefix: 'refs/heads/feature'\n1. It sets the cursor to seek to 'refs/heads/feature'\n2. It also sets the internal prefix matching to 'refs/heads/feature'\n\nIn Contrast seeking via 'ref_iteration_seek' only sets the cursor to\n'refs/heads/feature'.\n\nTo make this simpler, we've changed 'ref_iteration_seek' to do:\n1. seek the cursor to the requested reference\n2. Set prefix if the REF_ITERATOR_SEEK_SET_PREFIX is set, and unset the\nprefix otherwise.\n\nThe state reset I was talking about in my previous emails refers to step\n#2 here, where when no 'REF_ITERATOR_SEEK_SET_PREFIX' is set, we remove\nany previous prefix set.\n\n>> This series did start out that way around, so ease of implementation\n>> isn't it. It was more of a side-effect of not clearing state.\n>\n> I am even more worried about usability and correctness aspect of\n> what was described here now. After seeking to refs/heads/feature/,\n> do we continue to iterate and step out of refs/heads/feature/\n> hierarchy or can we cut off a particular page that started with a\n> ref within refs/heads/feature/ subhierarchy when we exhaust refs in\n> refs/heads/feature/ and have to wait for getting asked for the next\n> page before we show refs/heads/gsomething that is outside\n> refs/heads/feature/ and sorts after? The \"I reset to iterate over\n> refs/heads/feature/ because the entire refs/heads/ is not what I\n> care about\" example makes me worried about this.\n>\n> Thanks.\n\nI think we're crossing paths and talking different things. I hope the\nexamples above clarify things. The current implementation doesn't\nsupport '--start-after' and prefix setting at the same time:\n\n$ git for-each-ref --format=\"%(refname)\"\n--start-after=refs/heads/master refs/heads\nfatal: cannot use --start-after with patterns\n\nHappy to clarify if this doesn't make sense.\n\nThanks\n"},{"id":"522206","messageId":"xmqqtt3a7u08.fsf@gitster.g","threadId":"63722","inReplyTo":"20250715-306-git-for-each-ref-pagination-v5-2-852d5a2f56e1@gmail.com","subject":"Re: [PATCH v5 2/5] ref-cache: remove unused function 'find_ref_entry()'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-17T14:48:55Z","receivedAt":"2025-07-17T14:48:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> The 'find_ref_entry' function is no longer used, so remove it.\n\nIf my spelunking is correct, ba1c052f (ref_store: implement\n`refs_peel_ref()` generically, 2017-09-25) is the commit that\nremoved the last caller of it. Which may be worth noting here.\n\n> Signed-off-by: Karthik Nayak \n> ---\n> refs/ref-cache.c | 14 --------------\n> refs/ref-cache.h | 7 -------\n> 2 files changed, 21 deletions(-)\n>\n> diff --git a/refs/ref-cache.c b/refs/ref-cache.c\n> index c1f1bab1d5..8aaffa8c6b 100644\n> --- a/refs/ref-cache.c\n> +++ b/refs/ref-cache.c\n> @@ -194,20 +194,6 @@ static struct ref_dir *find_containing_dir(struct ref_dir *dir,\n> \treturn dir;\n> }\n> \n> -struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname)\n> -{\n> -\tint entry_index;\n> -\tstruct ref_entry *entry;\n> -\tdir = find_containing_dir(dir, refname);\n> -\tif (!dir)\n> -\t\treturn NULL;\n> -\tentry_index = search_ref_dir(dir, refname, strlen(refname));\n> -\tif (entry_index == -1)\n> -\t\treturn NULL;\n> -\tentry = dir->entries[entry_index];\n> -\treturn (entry->flag & REF_DIR) ? NULL : entry;\n> -}\n> -\n> /*\n> * Emit a warning and return true iff ref1 and ref2 have the same name\n> * and the same oid. Die if they have the same name but different\n> diff --git a/refs/ref-cache.h b/refs/ref-cache.h\n> index 5f04e518c3..f635d2d824 100644\n> --- a/refs/ref-cache.h\n> +++ b/refs/ref-cache.h\n> @@ -201,13 +201,6 @@ void free_ref_cache(struct ref_cache *cache);\n> */\n> void add_entry_to_dir(struct ref_dir *dir, struct ref_entry *entry);\n> \n> -/*\n> - * Find the value entry with the given name in dir, sorting ref_dirs\n> - * and recursing into subdirectories as necessary. If the name is not\n> - * found or it corresponds to a directory entry, return NULL.\n> - */\n> -struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname);\n> -\n> /*\n> * Start iterating over references in `cache`. If `prefix` is\n> * specified, only include references whose names start with that\n"},{"id":"522209","messageId":"xmqqikjq7s16.fsf@gitster.g","threadId":"63722","inReplyTo":"20250715-306-git-for-each-ref-pagination-v5-5-852d5a2f56e1@gmail.com","subject":"Re: [PATCH v5 5/5] for-each-ref: introduce a '--start-after' option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-17T15:31:33Z","receivedAt":"2025-07-17T15:31:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> The `git-for-each-ref(1)` command is used to iterate over references\n> present in a repository. In large repositories with millions of\n> references, it would be optimal to paginate this output such that we\n> can start iteration from a given reference. This would avoid having to\n> iterate over all references from the beginning each time when paginating\n> through results.\n>\n> The previous commit added 'seek' functionality to the reference\n> backends. Utilize this and expose a '--start-after' option in\n> 'git-for-each-ref(1)'. When used, the reference iteration seeks to the\n> lexicographically next reference and iterates from there onward.\n>\n> This enables efficient pagination workflows, where the calling script\n> can remember the last provided reference and use that as the starting\n> point for the next set of references:\n> git for-each-ref --count=100\n> git for-each-ref --count=100 --start-after=refs/heads/branch-100\n> git for-each-ref --count=100 --start-after=refs/heads/branch-200\n>\n> Since the reference iterators only allow seeking to a specified marker\n> via the `ref_iterator_seek()`, we introduce a helper function\n> `start_ref_iterator_after()`, which seeks to next reference by simply\n> adding (char) 1 to the marker.\n>\n> We must note that pagination always continues from the provided marker,\n> as such any concurrent reference updates lexicographically behind the\n> marker will not be output. Document the same.\n>\n> Signed-off-by: Karthik Nayak \n> ---\n> Documentation/git-for-each-ref.adoc | 10 +-\n> builtin/for-each-ref.c | 8 ++\n> ref-filter.c | 78 +++++++++++----\n> ref-filter.h | 1 +\n> t/t6302-for-each-ref-filter.sh | 194 ++++++++++++++++++++++++++++++++++++\n> 5 files changed, 272 insertions(+), 19 deletions(-)\n>\n> diff --git a/Documentation/git-for-each-ref.adoc b/Documentation/git-for-each-ref.adoc\n> index 5ef89fc0fe..ae61ba642a 100644\n> --- a/Documentation/git-for-each-ref.adoc\n> +++ b/Documentation/git-for-each-ref.adoc\n> @@ -14,7 +14,7 @@ SYNOPSIS\n> \t\t [--points-at=]\n> \t\t [--merged[=]] [--no-merged[=]]\n> \t\t [--contains[=]] [--no-contains[=]]\n> -\t\t [--exclude= ...]\n> +\t\t [--exclude= ...] [--start-after=]\n\nNot a problem this patch introduces, but as I noticed it, let me\nleave a #leftoverbits comment here (it is OK to have a preliminary\nclean-up patch).\n\n * \"--exclude=\" should be enclosed inside a pair of\n (parentheses), just like the way how [(--sort=)...] is\n shown.\n\n * [--stdin | ...] should be moved to the end. There is no\n reason to require \"--stdin\" to be the end of dashed options, but\n the ... must be, as they are positional, not dashed.\n\n> +--start-after=::\n> + Allows paginating the output by skipping references up to and including the\n> + specified marker. When paging, it should be noted that references may be\n> + deleted, modified or added between invocations. Output will only yield those\n> + references which follow the marker lexicographically. Output begins from the\n> + first reference that would come after the marker alphabetically. Cannot be\n> + used with general pattern matching or custom sort options.\n\nIt is unclear what \"general\" in \"general pattern matching\" refers\nto.\n\n Cannot be used with `--sort=` or `--stdin` options, or\n the __ argument(s) to limit the refs.\n\nor something, perhaps? It is curious how `--exclude=`\ninteracts with the feature. Presumably the exclusion is done so\nlate in the output phase that it does not have any effect? It does\nnot have to be mentioned in this documentation if that is the case\nas it is a mere implementation detail. \n\n Side note. The limitation that sorting and name_patterns cannot\n be used with the feature also comes from implementation\n (i.e. the name_patterns optimization will compete with this\n feature to take advantage of the \"prefix\" thing in an\n incompatible way), so while the reason does not have to be\n stated in the end-user facing documentation, the effect needs\n documenting.\n\n> @@ -3189,6 +3221,7 @@ void filter_is_base(struct repository *r,\n> \n> static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref_fn fn, void *cb_data)\n> {\n> +\tconst char *prefix = NULL;\n> ...\n> +\n> +\tif (prefix) {\n> +\t\tstruct ref_iterator *iter;\n> +\n> +\t\titer = refs_ref_iterator_begin(get_main_ref_store(the_repository),\n> +\t\t\t\t\t \"\", NULL, 0, 0);\n> +\n> +\t\tif (filter->start_after)\n\nThe start_after of the filter comes from \"--start-after=\".\nCan it be true with non-NULL prefix at this point? Unless you add\nsupport for the option to \"git branch/tag\", it would not happen, I\nguess.\n\nMore importantly, when you do add support to \"git branch/tag\", the\ncode need to be updated to keep the original prefix while seeking\nthe cursor to the specified , instead of clearing it.\n\n> +\t\t\tret = start_ref_iterator_after(iter, filter->start_after);\n> +\t\telse if (prefix)\n> +\t\t\tret = ref_iterator_seek(iter, prefix, 1);\n\nWe have \"REF_ITERATOR_SEEK_SET_PREFIX\" for that \"1\"?\n"},{"id":"522218","messageId":"CAO_smVgdaOxiD_494qA+VxkmxNd6c=XqJDcCE2weCTknWfmkkA@mail.gmail.com","threadId":"63722","inReplyTo":"20250717015402.GA2127425@coredump.intra.peff.net","subject":"Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'","fromName":"Kyle Lippincott","fromEmail":"spectral@google.com","sentAt":"2025-07-17T17:08:18Z","receivedAt":"2025-07-17T17:08:32Z","isPatch":true,"sender":{"key":"spectral@google.com","avatar":"https://avatars.githubusercontent.com/u/6371650?v=4"},"body":"On Wed, Jul 16, 2025 at 6:54 PM Jeff King wrote:\n>\n> On Wed, Jul 16, 2025 at 06:19:32PM -0700, Kyle Lippincott wrote:\n>\n> > Unfortunately I can't provide great instructions for reproducing this\n> > locally, because it relies on our internal build stack (which uses\n> > blaze). Getting MemorySanitizer running can be quite annoying, though\n> > you might not have any issues if this test doesn't invoke any third\n> > party libraries (like zlib).\n> >\n> > I need to sign off for the night soon, but if this isn't sufficient\n> > enough information to identify what's happening here, I can try to dig\n> > deeper tomorrow. This run was executed on an import of upstream commit\n> > 4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc (Junio's merge of this\n> > series)\n>\n> valgrind can often find the same issues as MSan without as much headache\n> to get it running (the downside is that it is _way_ slower). And indeed:\n>\n> git checkout 4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc &&\n> make &&\n> (cd t && ./t6302-for-each-ref-filter.sh --valgrind-only=48)\n>\n> yields:\n>\n> ==2177572== Conditional jump or move depends on uninitialised value(s)\n> ==2177572== at 0x3BC380: cache_ref_iterator_advance (ref-cache.c:409)\n> ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15)\n> ==2177572== by 0x3B6CC3: merge_ref_iterator_advance (iterator.c:179)\n> ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15)\n> ==2177572== by 0x3A9770: files_ref_iterator_advance (files-backend.c:902)\n> ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15)\n> ==2177572== by 0x3B7457: do_for_each_ref_iterator (iterator.c:478)\n> ==2177572== by 0x399B43: for_each_fullref_with_seek (ref-filter.c:2718)\n> ==2177572== by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756)\n> ==2177572== by 0x39B031: do_filter_refs (ref-filter.c:3263)\n> ==2177572== by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364)\n> ==2177572== by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115)\n> ==2177572== Uninitialised value was created by a heap allocation\n> ==2177572== at 0x484BDD0: realloc (vg_replace_malloc.c:1801)\n> ==2177572== by 0x44E941: xrealloc (wrapper.c:140)\n> ==2177572== by 0x3BCAD9: cache_ref_iterator_begin (ref-cache.c:580)\n> ==2177572== by 0x3A988A: files_ref_iterator_begin (files-backend.c:995)\n> ==2177572== by 0x3A295E: refs_ref_iterator_begin (refs.c:1776)\n> ==2177572== by 0x399AF6: for_each_fullref_with_seek (ref-filter.c:2710)\n> ==2177572== by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756)\n> ==2177572== by 0x39B031: do_filter_refs (ref-filter.c:3263)\n> ==2177572== by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364)\n> ==2177572== by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115)\n> ==2177572== by 0x128C90: run_builtin (git.c:480)\n> ==2177572== by 0x1290EB: handle_builtin (git.c:746)\n>\n> Bisecting doesn't tell us much, though (the first commit that introduces\n> the test shows the problem). I didn't dig further than that.\n>\n> -Peff\n\nThanks for that, that helped me a bit too as it provides more\ninformation than I was getting out of MemorySanitizer (I suspect\nMemorySanitizer was producing the information it just wasn't going to\nstderr or something, or maybe I was missing a flag to get it to report\nmore). I'm not sure what the right fix would be; my guess is that the\nfix would be to modify the places where we set levels_nr and\ninitialize the other fields in level to also set it to prefix_state\n(around lines 488 and 527 in ref-cache.c); and indeed setting the\nprefix_state to PREFIX_CONTAINS_DIR (the 0 value of the enum) makes\nthe test pass even under valgrind. Unfortunately without a much more\nin-depth knowledge of the code and the enum values I can't\ndefinitively state that those are the correct values. I can say that\nsetting it to PREFIX_WITHIN_DIR causes both additional valgrind\nfailures and test failures even without valgrind, but setting it to\nPREFIX_EXCLUDES_DIR doesn't seem to be a problem. I also moved the\n`if` around like 409 into the following if, because that was the only\ntime entry_prefix_state was used, I'd been thinking that maybe it\nneeded the check for entry->flag & REF_DIR prior to referencing\nlevel->prefix_state, but that didn't resolve it on its own.\n\nI don't mind if anyone else picks up this fix and runs with it, but\nI'm not comfortable sending this patch myself because I don't have\nenough knowledge of this are of the code to know if it's right, just\nthat it fixes the issue we encountered, and I'm extremely overloaded\nright now and can't get that knowledge nor see the patch through to\nthe end.\n\n\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 1d95b56d40..24feb33fcb 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -391,7 +391,6 @@ static int cache_ref_iterator_advance(struct\nref_iterator *ref_iterator)\n &iter->levels[iter->levels_nr - 1];\n struct ref_dir *dir = level->dir;\n struct ref_entry *entry;\n- enum prefix_state entry_prefix_state;\n\n if (level->index == -1)\n sort_ref_dir(dir);\n@@ -406,16 +405,17 @@ static int cache_ref_iterator_advance(struct\nref_iterator *ref_iterator)\n\n entry = dir->entries[level->index];\n\n- if (level->prefix_state == PREFIX_WITHIN_DIR) {\n- entry_prefix_state =\noverlaps_prefix(entry->name, iter->prefix);\n- if (entry_prefix_state == PREFIX_EXCLUDES_DIR ||\n- (entry_prefix_state == PREFIX_WITHIN_DIR\n&& !(entry->flag & REF_DIR)))\n- continue;\n- } else {\n- entry_prefix_state = level->prefix_state;\n- }\n-\n if (entry->flag & REF_DIR) {\n+ enum prefix_state entry_prefix_state;\n+ if (level->prefix_state == PREFIX_WITHIN_DIR) {\n+ entry_prefix_state =\noverlaps_prefix(entry->name, iter->prefix);\n+ if (entry_prefix_state == PREFIX_EXCLUDES_DIR ||\n+ (entry_prefix_state ==\nPREFIX_WITHIN_DIR && !(entry->flag & REF_DIR)))\n+ continue;\n+ } else {\n+ entry_prefix_state = level->prefix_state;\n+ }\n+\n /* push down a level */\n ALLOC_GROW(iter->levels, iter->levels_nr + 1,\n iter->levels_alloc);\n@@ -489,6 +489,7 @@ static int cache_ref_iterator_seek(struct\nref_iterator *ref_iterator,\n level = &iter->levels[0];\n level->index = -1;\n level->dir = dir;\n+ level->prefix_state = PREFIX_EXCLUDES_DIR; //\nFIXME: PROBABLY NOT CORRECT\n\n /* Unset any previously set prefix */\n FREE_AND_NULL(iter->prefix);\n@@ -527,6 +528,7 @@ static int cache_ref_iterator_seek(struct\nref_iterator *ref_iterator,\n level = &iter->levels[iter->levels_nr++];\n level->dir = dir;\n level->index = -1;\n+ level->prefix_state =\nPREFIX_EXCLUDES_DIR; // FIXME: PROBABLY NOT CORRECT\n } else {\n /* reduce the index so the leaf node\nis iterated over */\n if (cmp <= 0 && !slash)\n"},{"id":"522224","messageId":"xmqqpldy67wf.fsf@gitster.g","threadId":"63722","inReplyTo":"CAOLa=ZSKsdGDERNS_1hJrBMn0muSOJ46SZz2bn8OHZndJ2Hu3g@mail.gmail.com","subject":"Re: [PATCH v3 3/4] refs: selectively set prefix in the seek functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-17T17:31:44Z","receivedAt":"2025-07-17T17:31:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n>> Yes, even though I wouldn't call such an operation \"seek\", \"Ah, I do\n>> not need the entire refs/heads/ walked, only refs/heads/feature/ is\n>> enough\" is an operation mode that makes sense.\n>>\n>> But not for paging, though.\n\nActually, you are not using the new value for prefix; you are\nunsetting prefix to nothing, so you do not stop when you go over the\nboundary of refs/heads/feature/ hierarchy but will run through to\nthe end, so my worry is unfounded.\n\nWhich is good.\n\nBut the application has to decide when to stop as \"We obtained an\niterator in order to list all branches in refs/heads/ and the last\nround we gave refs/heads/main out. Seek to it and continue the\noutput\" will no longer stop after making a callback with\nrefs/heads/zz but will continue yielding non branch refs after that.\n\n> The current implementation doesn't\n> support '--start-after' and prefix setting at the same time:\n>\n> $ git for-each-ref --format=\"%(refname)\"\n> --start-after=refs/heads/master refs/heads\n> fatal: cannot use --start-after with patterns\n\nGood---in that case, \"unsetting prefix to nothing\" does not make any\ndifference and cannot introduce any confusing behaviour, as it has to\nrun to the end with or without \"--start-after\" anyway.\n\nThanks.\n"},{"id":"522226","messageId":"CAOLa=ZQPOP0DkK9L5XLmA_uzH7jWwX-ti8AgaEcAF2Mon=w0BA@mail.gmail.com","threadId":"63722","inReplyTo":"CAO_smVgdaOxiD_494qA+VxkmxNd6c=XqJDcCE2weCTknWfmkkA@mail.gmail.com","subject":"Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-17T19:26:31Z","receivedAt":"2025-07-17T19:26:33Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Kyle Lippincott writes:\n\n> On Wed, Jul 16, 2025 at 6:54 PM Jeff King wrote:\n>>\n>> On Wed, Jul 16, 2025 at 06:19:32PM -0700, Kyle Lippincott wrote:\n>>\n>> > Unfortunately I can't provide great instructions for reproducing this\n>> > locally, because it relies on our internal build stack (which uses\n>> > blaze). Getting MemorySanitizer running can be quite annoying, though\n>> > you might not have any issues if this test doesn't invoke any third\n>> > party libraries (like zlib).\n>> >\n>> > I need to sign off for the night soon, but if this isn't sufficient\n>> > enough information to identify what's happening here, I can try to dig\n>> > deeper tomorrow. This run was executed on an import of upstream commit\n>> > 4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc (Junio's merge of this\n>> > series)\n>>\n>> valgrind can often find the same issues as MSan without as much headache\n>> to get it running (the downside is that it is _way_ slower). And indeed:\n>>\n>> git checkout 4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc &&\n>> make &&\n>> (cd t && ./t6302-for-each-ref-filter.sh --valgrind-only=48)\n>>\n>> yields:\n>>\n>> ==2177572== Conditional jump or move depends on uninitialised value(s)\n>> ==2177572== at 0x3BC380: cache_ref_iterator_advance (ref-cache.c:409)\n>> ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15)\n>> ==2177572== by 0x3B6CC3: merge_ref_iterator_advance (iterator.c:179)\n>> ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15)\n>> ==2177572== by 0x3A9770: files_ref_iterator_advance (files-backend.c:902)\n>> ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15)\n>> ==2177572== by 0x3B7457: do_for_each_ref_iterator (iterator.c:478)\n>> ==2177572== by 0x399B43: for_each_fullref_with_seek (ref-filter.c:2718)\n>> ==2177572== by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756)\n>> ==2177572== by 0x39B031: do_filter_refs (ref-filter.c:3263)\n>> ==2177572== by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364)\n>> ==2177572== by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115)\n>> ==2177572== Uninitialised value was created by a heap allocation\n>> ==2177572== at 0x484BDD0: realloc (vg_replace_malloc.c:1801)\n>> ==2177572== by 0x44E941: xrealloc (wrapper.c:140)\n>> ==2177572== by 0x3BCAD9: cache_ref_iterator_begin (ref-cache.c:580)\n>> ==2177572== by 0x3A988A: files_ref_iterator_begin (files-backend.c:995)\n>> ==2177572== by 0x3A295E: refs_ref_iterator_begin (refs.c:1776)\n>> ==2177572== by 0x399AF6: for_each_fullref_with_seek (ref-filter.c:2710)\n>> ==2177572== by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756)\n>> ==2177572== by 0x39B031: do_filter_refs (ref-filter.c:3263)\n>> ==2177572== by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364)\n>> ==2177572== by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115)\n>> ==2177572== by 0x128C90: run_builtin (git.c:480)\n>> ==2177572== by 0x1290EB: handle_builtin (git.c:746)\n>>\n>> Bisecting doesn't tell us much, though (the first commit that introduces\n>> the test shows the problem). I didn't dig further than that.\n>>\n>> -Peff\n>\n> Thanks for that, that helped me a bit too as it provides more\n> information than I was getting out of MemorySanitizer (I suspect\n> MemorySanitizer was producing the information it just wasn't going to\n> stderr or something, or maybe I was missing a flag to get it to report\n> more).\n>\n\nThanks both for raising the issue. Thanks Jeff for also the valgrind\ninstructions.\n\nOn a sidenote, was discussing this at work and Patrick also mentioned\nthat we could try clang's MemorySanitizer. This seems to also be raising\nissues on master, so it was hard to find the exact output that valgrind\nwas providing.\n\n$ git checkout master\n$ CC=clang meson setup --reconfigure memory_build . -Db_sanitize=memory\n$ cd memory_build\n$ meson test -i --test-args=\"-ix\" t6302-for-each-ref-filter\n...\n==3275333==WARNING: MemorySanitizer: use-of-uninitialized-value\n #0 0x557bd886f4bb in git_mkstemps_mode ../wrapper.c:487:27\n #1 0x557bd886fb55 in git_mkstemp_mode ../wrapper.c:509:9\n #2 0x557bd8100d1a in create_tmpfile ../object-file.c:736:7\n #3 0x557bd80f1630 in start_loose_object_common ../object-file.c:781:7\n #4 0x557bd80f5203 in write_loose_object ../object-file.c:881:7\n #5 0x557bd80f4875 in write_object_file_flags ../object-file.c:1086:6\n #6 0x557bd80f9f65 in write_object_file ../object-file.h:181:9\n #7 0x557bd8101eb8 in index_mem ../object-file.c:1177:9\n #8 0x557bd80f8bd5 in index_core ../object-file.c:1247:10\n #9 0x557bd80f731d in index_fd ../object-file.c:1274:9\n #10 0x557bd80f95e4 in index_path ../object-file.c:1295:7\n #11 0x557bd831132d in add_to_index ../read-cache.c:771:7\n #12 0x557bd8313cb1 in add_file_to_index ../read-cache.c:804:9\n #13 0x557bd73f892c in add_files ../builtin/add.c:355:7\n #14 0x557bd73f4752 in cmd_add ../builtin/add.c:578:18\n #15 0x557bd7a38b6f in run_builtin ../git.c:480:11\n #16 0x557bd7a31d54 in handle_builtin ../git.c:746:9\n #17 0x557bd7a36644 in run_argv ../git.c:813:4\n #18 0x557bd7a30e09 in cmd_main ../git.c:953:19\n #19 0x557bd7a3ca01 in main ../common-main.c:9:11\n #20 0x7f7e3f02a4d7 in __libc_start_call_main\n(/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a4d7)\n(BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a)\n #21 0x7f7e3f02a59a in __libc_start_main@GLIBC_2.2.5\n(/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a59a)\n(BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a)\n #22 0x557bd7352b34 in _start (git+0x5db34)\n\nPossibly something we need to look into cleaning up.\n\n> I'm not sure what the right fix would be; my guess is that the\n> fix would be to modify the places where we set levels_nr and\n> initialize the other fields in level to also set it to prefix_state\n> (around lines 488 and 527 in ref-cache.c); and indeed setting the\n> prefix_state to PREFIX_CONTAINS_DIR (the 0 value of the enum) makes\n> the test pass even under valgrind. Unfortunately without a much more\n> in-depth knowledge of the code and the enum values I can't\n> definitively state that those are the correct values. I can say that\n> setting it to PREFIX_WITHIN_DIR causes both additional valgrind\n> failures and test failures even without valgrind, but setting it to\n> PREFIX_EXCLUDES_DIR doesn't seem to be a problem. I also moved the\n> `if` around like 409 into the following if, because that was the only\n> time entry_prefix_state was used, I'd been thinking that maybe it\n> needed the check for entry->flag & REF_DIR prior to referencing\n> level->prefix_state, but that didn't resolve it on its own.\n>\n> I don't mind if anyone else picks up this fix and runs with it, but\n> I'm not comfortable sending this patch myself because I don't have\n> enough knowledge of this are of the code to know if it's right, just\n> that it fixes the issue we encountered, and I'm extremely overloaded\n> right now and can't get that knowledge nor see the patch through to\n> the end.\n>\n\nThanks for taking a stab at this, your inference is correct. Let me\nclairfy some parts of it.\n\nSo the 'ref-cache' iteration logic is used to provide iteration over\nloose refs (which consists of directories and entries). Anytime we come\nacross a directory, we add it to the level variable, which acts as\nstack, when all entries under the current level are yielded, we pop the\nstack to obtain the next level to iterate. This ensures we iterate over\nall directories recursively.\n\nBefore this series, the seek function was used to set the prefix for\niteration, which meant we need to find the directory for matching the\nprefix and only iterate over that level and its subdirs. If the prefix\nprovided was a directory like 'refs/heads/' then all refs under that\nwould be yielded (PREFIX_CONTAINS_DIR). If the prefix was\n'refs/heads/foo', then the level would be set to 'ref/heads/' with the\nPREFIX_WITHIN_DIR flag set since only some refs within the dir would\nmatch the prefix. Entries which didn't overlap the prefix are denoted by\nPREFIX_EXCLUDES_DIR.\n\nThis series allows the seek function to set the cursor without setting\nthe prefix, which is a requirement for pagination. So there is no need\nto set 'prefix_state' for this functionality. Which is why I didn't set\nit, since the default value of '0' (PREFIX_CONTAINS_DIR) would be the\ncorrect setting for all dirs. This causes the issue.\n\nSo the only fix required would be\n\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 1d95b56d40..ceef3a2008 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(struct\nref_iterator *ref_iterator,\n \t\t\t\tlevel = &iter->levels[iter->levels_nr++];\n \t\t\t\tlevel->dir = dir;\n \t\t\t\tlevel->index = -1;\n+\t\t\t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n \t\t\t} else {\n \t\t\t\t/* reduce the index so the leaf node is iterated over */\n \t\t\t\tif (cmp <= 0 && !slash)\n\n\nThe other location (Line 488), is not needed because that is the root\ndirectory and the 'prefix_state' for it is set in\n'cache_ref_iterator_set_prefix()' when the iterator begins.\n\n>\n> diff --git a/refs/ref-cache.c b/refs/ref-cache.c\n> index 1d95b56d40..24feb33fcb 100644\n> --- a/refs/ref-cache.c\n> +++ b/refs/ref-cache.c\n> @@ -391,7 +391,6 @@ static int cache_ref_iterator_advance(struct\n> ref_iterator *ref_iterator)\n> &iter->levels[iter->levels_nr - 1];\n> struct ref_dir *dir = level->dir;\n> struct ref_entry *entry;\n> - enum prefix_state entry_prefix_state;\n>\n> if (level->index == -1)\n> sort_ref_dir(dir);\n> @@ -406,16 +405,17 @@ static int cache_ref_iterator_advance(struct\n> ref_iterator *ref_iterator)\n>\n> entry = dir->entries[level->index];\n>\n> - if (level->prefix_state == PREFIX_WITHIN_DIR) {\n> - entry_prefix_state =\n> overlaps_prefix(entry->name, iter->prefix);\n> - if (entry_prefix_state == PREFIX_EXCLUDES_DIR ||\n> - (entry_prefix_state == PREFIX_WITHIN_DIR\n> && !(entry->flag & REF_DIR)))\n> - continue;\n> - } else {\n> - entry_prefix_state = level->prefix_state;\n> - }\n> -\n> if (entry->flag & REF_DIR) {\n> + enum prefix_state entry_prefix_state;\n> + if (level->prefix_state == PREFIX_WITHIN_DIR) {\n> + entry_prefix_state =\n> overlaps_prefix(entry->name, iter->prefix);\n> + if (entry_prefix_state == PREFIX_EXCLUDES_DIR ||\n> + (entry_prefix_state ==\n> PREFIX_WITHIN_DIR && !(entry->flag & REF_DIR)))\n> + continue;\n> + } else {\n> + entry_prefix_state = level->prefix_state;\n> + }\n> +\n> /* push down a level */\n> ALLOC_GROW(iter->levels, iter->levels_nr + 1,\n> iter->levels_alloc);\n> @@ -489,6 +489,7 @@ static int cache_ref_iterator_seek(struct\n> ref_iterator *ref_iterator,\n> level = &iter->levels[0];\n> level->index = -1;\n> level->dir = dir;\n> + level->prefix_state = PREFIX_EXCLUDES_DIR; //\n> FIXME: PROBABLY NOT CORRECT\n>\n> /* Unset any previously set prefix */\n> FREE_AND_NULL(iter->prefix);\n> @@ -527,6 +528,7 @@ static int cache_ref_iterator_seek(struct\n> ref_iterator *ref_iterator,\n> level = &iter->levels[iter->levels_nr++];\n> level->dir = dir;\n> level->index = -1;\n> + level->prefix_state =\n> PREFIX_EXCLUDES_DIR; // FIXME: PROBABLY NOT CORRECT\n> } else {\n> /* reduce the index so the leaf node\n> is iterated over */\n> if (cmp <= 0 && !slash)\n"},{"id":"522227","messageId":"CAOLa=ZRqLgdnUz4Eeof0qdD_RkBYOKOT4oOoXEb0P386KQ0g8w@mail.gmail.com","threadId":"63722","inReplyTo":"xmqqtt3a7u08.fsf@gitster.g","subject":"Re: [PATCH v5 2/5] ref-cache: remove unused function 'find_ref_entry()'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-17T19:31:07Z","receivedAt":"2025-07-17T19:31:09Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Karthik Nayak writes:\n>\n>> The 'find_ref_entry' function is no longer used, so remove it.\n>\n> If my spelunking is correct, ba1c052f (ref_store: implement\n> `refs_peel_ref()` generically, 2017-09-25) is the commit that\n> removed the last caller of it. Which may be worth noting here.\n>\n\nIndeed, I could also verify this by running\n\n$ git log -S find_ref_entry --oneline\n2c90b85801 ref-cache: remove unused function 'find_ref_entry()'\nba1c052fa6 ref_store: implement `refs_peel_ref()` generically\n9939b33d6a packed-backend: rip out some now-unused code\n....\n\nAnd looking at `ba1c052fa6`. I should've done this before. But thanks\nfor the digging!\n\nI plan to address a few comments on this version, but I also see that\nyou've merged it to master. Should I raise followups for noteworthy\nchanges or a new version?\n\nKarthik\n"},{"id":"522229","messageId":"CAO_smVj4e0XOQyQr5sDyMZ8WPvqcBe2Y33DFHrED7C=VJnm4eg@mail.gmail.com","threadId":"63722","inReplyTo":"CAOLa=ZQPOP0DkK9L5XLmA_uzH7jWwX-ti8AgaEcAF2Mon=w0BA@mail.gmail.com","subject":"Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'","fromName":"Kyle Lippincott","fromEmail":"spectral@google.com","sentAt":"2025-07-17T19:35:58Z","receivedAt":"2025-07-17T19:36:11Z","isPatch":true,"sender":{"key":"spectral@google.com","avatar":"https://avatars.githubusercontent.com/u/6371650?v=4"},"body":"On Thu, Jul 17, 2025 at 12:26 PM Karthik Nayak wrote:\n>\n> Kyle Lippincott writes:\n>\n> > On Wed, Jul 16, 2025 at 6:54 PM Jeff King wrote:\n> >>\n> >> On Wed, Jul 16, 2025 at 06:19:32PM -0700, Kyle Lippincott wrote:\n> >>\n> >> > Unfortunately I can't provide great instructions for reproducing this\n> >> > locally, because it relies on our internal build stack (which uses\n> >> > blaze). Getting MemorySanitizer running can be quite annoying, though\n> >> > you might not have any issues if this test doesn't invoke any third\n> >> > party libraries (like zlib).\n> >> >\n> >> > I need to sign off for the night soon, but if this isn't sufficient\n> >> > enough information to identify what's happening here, I can try to dig\n> >> > deeper tomorrow. This run was executed on an import of upstream commit\n> >> > 4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc (Junio's merge of this\n> >> > series)\n> >>\n> >> valgrind can often find the same issues as MSan without as much headache\n> >> to get it running (the downside is that it is _way_ slower). And indeed:\n> >>\n> >> git checkout 4ea3c74afd42a503b3e0d60e1fec33bc0431e7bc &&\n> >> make &&\n> >> (cd t && ./t6302-for-each-ref-filter.sh --valgrind-only=48)\n> >>\n> >> yields:\n> >>\n> >> ==2177572== Conditional jump or move depends on uninitialised value(s)\n> >> ==2177572== at 0x3BC380: cache_ref_iterator_advance (ref-cache.c:409)\n> >> ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15)\n> >> ==2177572== by 0x3B6CC3: merge_ref_iterator_advance (iterator.c:179)\n> >> ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15)\n> >> ==2177572== by 0x3A9770: files_ref_iterator_advance (files-backend.c:902)\n> >> ==2177572== by 0x3B69D7: ref_iterator_advance (iterator.c:15)\n> >> ==2177572== by 0x3B7457: do_for_each_ref_iterator (iterator.c:478)\n> >> ==2177572== by 0x399B43: for_each_fullref_with_seek (ref-filter.c:2718)\n> >> ==2177572== by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756)\n> >> ==2177572== by 0x39B031: do_filter_refs (ref-filter.c:3263)\n> >> ==2177572== by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364)\n> >> ==2177572== by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115)\n> >> ==2177572== Uninitialised value was created by a heap allocation\n> >> ==2177572== at 0x484BDD0: realloc (vg_replace_malloc.c:1801)\n> >> ==2177572== by 0x44E941: xrealloc (wrapper.c:140)\n> >> ==2177572== by 0x3BCAD9: cache_ref_iterator_begin (ref-cache.c:580)\n> >> ==2177572== by 0x3A988A: files_ref_iterator_begin (files-backend.c:995)\n> >> ==2177572== by 0x3A295E: refs_ref_iterator_begin (refs.c:1776)\n> >> ==2177572== by 0x399AF6: for_each_fullref_with_seek (ref-filter.c:2710)\n> >> ==2177572== by 0x399C09: for_each_fullref_in_pattern (ref-filter.c:2756)\n> >> ==2177572== by 0x39B031: do_filter_refs (ref-filter.c:3263)\n> >> ==2177572== by 0x39B2B7: filter_and_format_refs (ref-filter.c:3364)\n> >> ==2177572== by 0x18C1D2: cmd_for_each_ref (for-each-ref.c:115)\n> >> ==2177572== by 0x128C90: run_builtin (git.c:480)\n> >> ==2177572== by 0x1290EB: handle_builtin (git.c:746)\n> >>\n> >> Bisecting doesn't tell us much, though (the first commit that introduces\n> >> the test shows the problem). I didn't dig further than that.\n> >>\n> >> -Peff\n> >\n> > Thanks for that, that helped me a bit too as it provides more\n> > information than I was getting out of MemorySanitizer (I suspect\n> > MemorySanitizer was producing the information it just wasn't going to\n> > stderr or something, or maybe I was missing a flag to get it to report\n> > more).\n> >\n>\n> Thanks both for raising the issue. Thanks Jeff for also the valgrind\n> instructions.\n>\n> On a sidenote, was discussing this at work and Patrick also mentioned\n> that we could try clang's MemorySanitizer. This seems to also be raising\n> issues on master, so it was hard to find the exact output that valgrind\n> was providing.\n>\n> $ git checkout master\n> $ CC=clang meson setup --reconfigure memory_build . -Db_sanitize=memory\n> $ cd memory_build\n> $ meson test -i --test-args=\"-ix\" t6302-for-each-ref-filter\n> ...\n> ==3275333==WARNING: MemorySanitizer: use-of-uninitialized-value\n> #0 0x557bd886f4bb in git_mkstemps_mode ../wrapper.c:487:27\n> #1 0x557bd886fb55 in git_mkstemp_mode ../wrapper.c:509:9\n> #2 0x557bd8100d1a in create_tmpfile ../object-file.c:736:7\n> #3 0x557bd80f1630 in start_loose_object_common ../object-file.c:781:7\n> #4 0x557bd80f5203 in write_loose_object ../object-file.c:881:7\n> #5 0x557bd80f4875 in write_object_file_flags ../object-file.c:1086:6\n> #6 0x557bd80f9f65 in write_object_file ../object-file.h:181:9\n> #7 0x557bd8101eb8 in index_mem ../object-file.c:1177:9\n> #8 0x557bd80f8bd5 in index_core ../object-file.c:1247:10\n> #9 0x557bd80f731d in index_fd ../object-file.c:1274:9\n> #10 0x557bd80f95e4 in index_path ../object-file.c:1295:7\n> #11 0x557bd831132d in add_to_index ../read-cache.c:771:7\n> #12 0x557bd8313cb1 in add_file_to_index ../read-cache.c:804:9\n> #13 0x557bd73f892c in add_files ../builtin/add.c:355:7\n> #14 0x557bd73f4752 in cmd_add ../builtin/add.c:578:18\n> #15 0x557bd7a38b6f in run_builtin ../git.c:480:11\n> #16 0x557bd7a31d54 in handle_builtin ../git.c:746:9\n> #17 0x557bd7a36644 in run_argv ../git.c:813:4\n> #18 0x557bd7a30e09 in cmd_main ../git.c:953:19\n> #19 0x557bd7a3ca01 in main ../common-main.c:9:11\n> #20 0x7f7e3f02a4d7 in __libc_start_call_main\n> (/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a4d7)\n> (BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a)\n> #21 0x7f7e3f02a59a in __libc_start_main@GLIBC_2.2.5\n> (/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a59a)\n> (BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a)\n> #22 0x557bd7352b34 in _start (git+0x5db34)\n>\n> Possibly something we need to look into cleaning up.\n\nI also saw those msan issues when trying `make\nCFLAGS=-fsanitize=memory CC=clang`, but not with Google's internal\nmsan build. I don't know which variable in wrapper.c:487 it's\ncomplaining about - you'd think it'd be `letters`, but if it's `v`,\nthen that potentially comes from OpenSSL or some other library, and\nthat library would also need to be built with msan (which is why it's\nsuch a pain to get msan builds working - EVERY library needs to be\nbuilt with memory sanitizer).\n\n>\n> > I'm not sure what the right fix would be; my guess is that the\n> > fix would be to modify the places where we set levels_nr and\n> > initialize the other fields in level to also set it to prefix_state\n> > (around lines 488 and 527 in ref-cache.c); and indeed setting the\n> > prefix_state to PREFIX_CONTAINS_DIR (the 0 value of the enum) makes\n> > the test pass even under valgrind. Unfortunately without a much more\n> > in-depth knowledge of the code and the enum values I can't\n> > definitively state that those are the correct values. I can say that\n> > setting it to PREFIX_WITHIN_DIR causes both additional valgrind\n> > failures and test failures even without valgrind, but setting it to\n> > PREFIX_EXCLUDES_DIR doesn't seem to be a problem. I also moved the\n> > `if` around like 409 into the following if, because that was the only\n> > time entry_prefix_state was used, I'd been thinking that maybe it\n> > needed the check for entry->flag & REF_DIR prior to referencing\n> > level->prefix_state, but that didn't resolve it on its own.\n> >\n> > I don't mind if anyone else picks up this fix and runs with it, but\n> > I'm not comfortable sending this patch myself because I don't have\n> > enough knowledge of this are of the code to know if it's right, just\n> > that it fixes the issue we encountered, and I'm extremely overloaded\n> > right now and can't get that knowledge nor see the patch through to\n> > the end.\n> >\n>\n> Thanks for taking a stab at this, your inference is correct. Let me\n> clairfy some parts of it.\n>\n> So the 'ref-cache' iteration logic is used to provide iteration over\n> loose refs (which consists of directories and entries). Anytime we come\n> across a directory, we add it to the level variable, which acts as\n> stack, when all entries under the current level are yielded, we pop the\n> stack to obtain the next level to iterate. This ensures we iterate over\n> all directories recursively.\n>\n> Before this series, the seek function was used to set the prefix for\n> iteration, which meant we need to find the directory for matching the\n> prefix and only iterate over that level and its subdirs. If the prefix\n> provided was a directory like 'refs/heads/' then all refs under that\n> would be yielded (PREFIX_CONTAINS_DIR). If the prefix was\n> 'refs/heads/foo', then the level would be set to 'ref/heads/' with the\n> PREFIX_WITHIN_DIR flag set since only some refs within the dir would\n> match the prefix. Entries which didn't overlap the prefix are denoted by\n> PREFIX_EXCLUDES_DIR.\n>\n> This series allows the seek function to set the cursor without setting\n> the prefix, which is a requirement for pagination. So there is no need\n> to set 'prefix_state' for this functionality. Which is why I didn't set\n> it, since the default value of '0' (PREFIX_CONTAINS_DIR) would be the\n> correct setting for all dirs. This causes the issue.\n>\n> So the only fix required would be\n>\n> diff --git a/refs/ref-cache.c b/refs/ref-cache.c\n> index 1d95b56d40..ceef3a2008 100644\n> --- a/refs/ref-cache.c\n> +++ b/refs/ref-cache.c\n> @@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(struct\n> ref_iterator *ref_iterator,\n> level = &iter->levels[iter->levels_nr++];\n> level->dir = dir;\n> level->index = -1;\n> + level->prefix_state = PREFIX_CONTAINS_DIR;\n> } else {\n> /* reduce the index so the leaf node is iterated over */\n> if (cmp <= 0 && !slash)\n>\n>\n> The other location (Line 488), is not needed because that is the root\n> directory and the 'prefix_state' for it is set in\n> 'cache_ref_iterator_set_prefix()' when the iterator begins.\n>\n> >\n> > diff --git a/refs/ref-cache.c b/refs/ref-cache.c\n> > index 1d95b56d40..24feb33fcb 100644\n> > --- a/refs/ref-cache.c\n> > +++ b/refs/ref-cache.c\n> > @@ -391,7 +391,6 @@ static int cache_ref_iterator_advance(struct\n> > ref_iterator *ref_iterator)\n> > &iter->levels[iter->levels_nr - 1];\n> > struct ref_dir *dir = level->dir;\n> > struct ref_entry *entry;\n> > - enum prefix_state entry_prefix_state;\n> >\n> > if (level->index == -1)\n> > sort_ref_dir(dir);\n> > @@ -406,16 +405,17 @@ static int cache_ref_iterator_advance(struct\n> > ref_iterator *ref_iterator)\n> >\n> > entry = dir->entries[level->index];\n> >\n> > - if (level->prefix_state == PREFIX_WITHIN_DIR) {\n> > - entry_prefix_state =\n> > overlaps_prefix(entry->name, iter->prefix);\n> > - if (entry_prefix_state == PREFIX_EXCLUDES_DIR ||\n> > - (entry_prefix_state == PREFIX_WITHIN_DIR\n> > && !(entry->flag & REF_DIR)))\n> > - continue;\n> > - } else {\n> > - entry_prefix_state = level->prefix_state;\n> > - }\n> > -\n> > if (entry->flag & REF_DIR) {\n> > + enum prefix_state entry_prefix_state;\n> > + if (level->prefix_state == PREFIX_WITHIN_DIR) {\n> > + entry_prefix_state =\n> > overlaps_prefix(entry->name, iter->prefix);\n> > + if (entry_prefix_state == PREFIX_EXCLUDES_DIR ||\n> > + (entry_prefix_state ==\n> > PREFIX_WITHIN_DIR && !(entry->flag & REF_DIR)))\n> > + continue;\n> > + } else {\n> > + entry_prefix_state = level->prefix_state;\n> > + }\n> > +\n> > /* push down a level */\n> > ALLOC_GROW(iter->levels, iter->levels_nr + 1,\n> > iter->levels_alloc);\n> > @@ -489,6 +489,7 @@ static int cache_ref_iterator_seek(struct\n> > ref_iterator *ref_iterator,\n> > level = &iter->levels[0];\n> > level->index = -1;\n> > level->dir = dir;\n> > + level->prefix_state = PREFIX_EXCLUDES_DIR; //\n> > FIXME: PROBABLY NOT CORRECT\n> >\n> > /* Unset any previously set prefix */\n> > FREE_AND_NULL(iter->prefix);\n> > @@ -527,6 +528,7 @@ static int cache_ref_iterator_seek(struct\n> > ref_iterator *ref_iterator,\n> > level = &iter->levels[iter->levels_nr++];\n> > level->dir = dir;\n> > level->index = -1;\n> > + level->prefix_state =\n> > PREFIX_EXCLUDES_DIR; // FIXME: PROBABLY NOT CORRECT\n> > } else {\n> > /* reduce the index so the leaf node\n> > is iterated over */\n> > if (cmp <= 0 && !slash)\n"},{"id":"522230","messageId":"CAOLa=ZTdUsv-6jVhPSwEW5dYr3vEh2bLT5ECRZZ1ot3MGAYhBg@mail.gmail.com","threadId":"63722","inReplyTo":"20250717020905.GA2193264@coredump.intra.peff.net","subject":"Re: [PATCH v5 3/5] refs: selectively set prefix in the seek functions","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-17T19:49:33Z","receivedAt":"2025-07-17T19:49:35Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Jeff King writes:\n\n> On Tue, Jul 15, 2025 at 01:28:28PM +0200, Karthik Nayak wrote:\n>\n>> +static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n>> +\t\t\t\t const char *refname, unsigned int flags)\n>> [...]\n>> +\t\tdo {\n>> +\t\t\tint len, idx;\n>> +\t\t\tint cmp = 0;\n>> +\n>> +\t\t\tsort_ref_dir(dir);\n>> +\n>> +\t\t\tslash = strchr(slash, '/');\n>> +\t\t\tlen = slash ? slash - refname : (int)strlen(refname);\n>\n> I was looking at this code due to a nearby thread and noticed this funny\n> cast to int. I guess you added it to silence -Wsign-compare, but Why are\n> we not using a size_t in the first place?\n>\n\nThat's an oversight from my side.\n\n> This kind of conversion can sometimes have security implications because\n> a very large \"refname\" would cause \"len\" to become negative (i.e., if\n> it's between 2GB and 4GB).\n>\n\nIndeed, I didn't think of that.\n\n> In this particular case it ends up cast back to a size_t via strncmp:\n>\n>> +\t\t\tfor (idx = 0; idx < dir->nr; idx++) {\n>> +\t\t\t\tcmp = strncmp(refname, dir->entries[idx]->name, len);\n>> +\t\t\t\tif (cmp <= 0)\n>> +\t\t\t\t\tbreak;\n>> +\t\t\t}\n>\n> so we get the original value back. We'd still get truncation for a\n> refname value over 4GB, which would presumably give us a slightly wrong\n> answer. But I don't think we'd ever look outside the array.\n>\n> Such sizes are probably unlikely if we are feeding filesystem paths. But\n> we probably should not set a bad example, and just do;\n>\n\nAgreed.\n\n> diff --git a/refs/ref-cache.c b/refs/ref-cache.c\n> index 1d95b56d40..3949d145e8 100644\n> --- a/refs/ref-cache.c\n> +++ b/refs/ref-cache.c\n> @@ -498,13 +498,14 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n> \t\t * indexing to each level as needed.\n> \t\t */\n> \t\tdo {\n> -\t\t\tint len, idx;\n> +\t\t\tsize_t len;\n> +\t\t\tint idx;\n> \t\t\tint cmp = 0;\n>\n> \t\t\tsort_ref_dir(dir);\n>\n> \t\t\tslash = strchr(slash, '/');\n> -\t\t\tlen = slash ? slash - refname : (int)strlen(refname);\n> +\t\t\tlen = slash ? slash - refname : strlen(refname);\n>\n> \t\t\tfor (idx = 0; idx < dir->nr; idx++) {\n> \t\t\t\tcmp = strncmp(refname, dir->entries[idx]->name, len);\n>\n> -Peff\n\nThanks, I think we have to typecast `slash - refname` to size_t, but\nthis is the right way to do it. Thanks for the review!\n\nKarthik\n"},{"id":"522231","messageId":"xmqqldom5zjp.fsf@gitster.g","threadId":"63722","inReplyTo":"CAOLa=ZRqLgdnUz4Eeof0qdD_RkBYOKOT4oOoXEb0P386KQ0g8w@mail.gmail.com","subject":"Re: [PATCH v5 2/5] ref-cache: remove unused function 'find_ref_entry()'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-17T20:32:10Z","receivedAt":"2025-07-17T20:32:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> Junio C Hamano writes:\n>\n>> Karthik Nayak writes:\n>>\n>>> The 'find_ref_entry' function is no longer used, so remove it.\n>>\n>> If my spelunking is correct, ba1c052f (ref_store: implement\n>> `refs_peel_ref()` generically, 2017-09-25) is the commit that\n>> removed the last caller of it. Which may be worth noting here.\n>>\n>\n> Indeed, I could also verify this by running\n>\n> $ git log -S find_ref_entry --oneline\n> 2c90b85801 ref-cache: remove unused function 'find_ref_entry()'\n> ba1c052fa6 ref_store: implement `refs_peel_ref()` generically\n> 9939b33d6a packed-backend: rip out some now-unused code\n> ....\n>\n> And looking at `ba1c052fa6`. I should've done this before. But thanks\n> for the digging!\n>\n> I plan to address a few comments on this version, but I also see that\n> you've merged it to master. Should I raise followups for noteworthy\n> changes or a new version?\n\nYou mean it is now in 'next'? Yes, please give incremental patches.\nFinding some more gotchas even after v5 is a sign that they are\ntricky enough that deserve separate explanation from the main part\nof the series. And I think the msan one Kyle and Peff found is also\ntricky enough that evaded reviewers' eyes.\n\nThanks.\n\n"},{"id":"522245","messageId":"20250717215506.GA2244266@coredump.intra.peff.net","threadId":"63722","inReplyTo":"CAOLa=ZTdUsv-6jVhPSwEW5dYr3vEh2bLT5ECRZZ1ot3MGAYhBg@mail.gmail.com","subject":"Re: [PATCH v5 3/5] refs: selectively set prefix in the seek functions","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-07-17T21:55:06Z","receivedAt":"2025-07-17T21:55:13Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 17, 2025 at 12:49:33PM -0700, Karthik Nayak wrote:\n\n> > diff --git a/refs/ref-cache.c b/refs/ref-cache.c\n> > index 1d95b56d40..3949d145e8 100644\n> > --- a/refs/ref-cache.c\n> > +++ b/refs/ref-cache.c\n> > @@ -498,13 +498,14 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n> > \t\t * indexing to each level as needed.\n> > \t\t */\n> > \t\tdo {\n> > -\t\t\tint len, idx;\n> > +\t\t\tsize_t len;\n> > +\t\t\tint idx;\n> > \t\t\tint cmp = 0;\n> >\n> > \t\t\tsort_ref_dir(dir);\n> >\n> > \t\t\tslash = strchr(slash, '/');\n> > -\t\t\tlen = slash ? slash - refname : (int)strlen(refname);\n> > +\t\t\tlen = slash ? slash - refname : strlen(refname);\n> >\n> > \t\t\tfor (idx = 0; idx < dir->nr; idx++) {\n> > \t\t\t\tcmp = strncmp(refname, dir->entries[idx]->name, len);\n> >\n> > -Peff\n> \n> Thanks, I think we have to typecast `slash - refname` to size_t, but\n> this is the right way to do it. Thanks for the review!\n\nAh, yeah. I mistakenly test-compiled without DEVELOPER=1. ;)\n\nI do think that cast is a lesser evil, though. It is a ptrdiff_t, but we\nknow it is correctly unsigned because \"slash > refname\" via strchr. I\nwish there was a good way to use the type system to tell the compiler\nthat.\n\n-Peff\n"},{"id":"522246","messageId":"20250717220929.GB2244266@coredump.intra.peff.net","threadId":"63722","inReplyTo":"CAO_smVj4e0XOQyQr5sDyMZ8WPvqcBe2Y33DFHrED7C=VJnm4eg@mail.gmail.com","subject":"Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-07-17T22:09:29Z","receivedAt":"2025-07-17T22:09:32Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 17, 2025 at 12:35:58PM -0700, Kyle Lippincott wrote:\n\n> > ==3275333==WARNING: MemorySanitizer: use-of-uninitialized-value\n> > #0 0x557bd886f4bb in git_mkstemps_mode ../wrapper.c:487:27\n> > #1 0x557bd886fb55 in git_mkstemp_mode ../wrapper.c:509:9\n> > #2 0x557bd8100d1a in create_tmpfile ../object-file.c:736:7\n> > #3 0x557bd80f1630 in start_loose_object_common ../object-file.c:781:7\n> > #4 0x557bd80f5203 in write_loose_object ../object-file.c:881:7\n> > #5 0x557bd80f4875 in write_object_file_flags ../object-file.c:1086:6\n> > #6 0x557bd80f9f65 in write_object_file ../object-file.h:181:9\n> > #7 0x557bd8101eb8 in index_mem ../object-file.c:1177:9\n> > #8 0x557bd80f8bd5 in index_core ../object-file.c:1247:10\n> > #9 0x557bd80f731d in index_fd ../object-file.c:1274:9\n> > #10 0x557bd80f95e4 in index_path ../object-file.c:1295:7\n> > #11 0x557bd831132d in add_to_index ../read-cache.c:771:7\n> > #12 0x557bd8313cb1 in add_file_to_index ../read-cache.c:804:9\n> > #13 0x557bd73f892c in add_files ../builtin/add.c:355:7\n> > #14 0x557bd73f4752 in cmd_add ../builtin/add.c:578:18\n> > #15 0x557bd7a38b6f in run_builtin ../git.c:480:11\n> > #16 0x557bd7a31d54 in handle_builtin ../git.c:746:9\n> > #17 0x557bd7a36644 in run_argv ../git.c:813:4\n> > #18 0x557bd7a30e09 in cmd_main ../git.c:953:19\n> > #19 0x557bd7a3ca01 in main ../common-main.c:9:11\n> > #20 0x7f7e3f02a4d7 in __libc_start_call_main\n> > (/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a4d7)\n> > (BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a)\n> > #21 0x7f7e3f02a59a in __libc_start_main@GLIBC_2.2.5\n> > (/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a59a)\n> > (BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a)\n> > #22 0x557bd7352b34 in _start (git+0x5db34)\n> >\n> > Possibly something we need to look into cleaning up.\n> \n> I also saw those msan issues when trying `make\n> CFLAGS=-fsanitize=memory CC=clang`, but not with Google's internal\n> msan build. I don't know which variable in wrapper.c:487 it's\n> complaining about - you'd think it'd be `letters`, but if it's `v`,\n> then that potentially comes from OpenSSL or some other library, and\n> that library would also need to be built with msan (which is why it's\n> such a pain to get msan builds working - EVERY library needs to be\n> built with memory sanitizer).\n\nYeah, presumably it is \"v\" from csprng_bytes(). If there are only a few\nsuch spots, we can manually \"unpoison\" memory coming from libraries. On\nmy system, I didn't hit the case shown above but do have trouble with\nbytes coming back from zlib.\n\nApplying this ancient patch:\n\n https://lore.kernel.org/git/20171004101932.pai6wzcv2eohsicr@sigill.intra.peff.net/\n\nand building with \"make SANITIZE=memory CC=clang\" let me run t6302 to\ncompletion, modulo the bug that started this thread (and which I\nconfirmed goes away both with MSan and valgrind with the fix Karthik\nposted).\n\nProbably:\n\ndiff --git a/wrapper.c b/wrapper.c\nindex 2f00d2ac87..6a4c1c1c29 100644\n--- a/wrapper.c\n+++ b/wrapper.c\n@@ -482,6 +482,8 @@ int git_mkstemps_mode(char *pattern, int suffix_len, int mode)\n \t\tif (csprng_bytes(&v, sizeof(v), 0) < 0)\n \t\t\treturn error_errno(\"unable to get random bytes for temporary file\");\n \n+\t\tmsan_unpoison(&v, sizeof(v));\n+\n \t\t/* Fill in the random bits. */\n \t\tfor (i = 0; i < num_x; i++) {\n \t\t\tfilename_template[i] = letters[v % num_letters];\n\n\non top of that would fix the problem you guys are seeing. I don't know\nif that path leads to insanity, though. Using MSan-enabled libraries is\nprobably a better direction (should increase accuracy, and we don't have\nto carry these manual annotations around).\n\n-Peff\n"},{"id":"522247","messageId":"20250717221620.GA2782307@coredump.intra.peff.net","threadId":"63722","inReplyTo":"20250717220929.GB2244266@coredump.intra.peff.net","subject":"Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-07-17T22:16:20Z","receivedAt":"2025-07-17T22:16:22Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 17, 2025 at 06:09:29PM -0400, Jeff King wrote:\n\n> Probably:\n> \n> diff --git a/wrapper.c b/wrapper.c\n> index 2f00d2ac87..6a4c1c1c29 100644\n> --- a/wrapper.c\n> +++ b/wrapper.c\n> @@ -482,6 +482,8 @@ int git_mkstemps_mode(char *pattern, int suffix_len, int mode)\n> \t\tif (csprng_bytes(&v, sizeof(v), 0) < 0)\n> \t\t\treturn error_errno(\"unable to get random bytes for temporary file\");\n> \n> +\t\tmsan_unpoison(&v, sizeof(v));\n> +\n> \t\t/* Fill in the random bits. */\n> \t\tfor (i = 0; i < num_x; i++) {\n> \t\t\tfilename_template[i] = letters[v % num_letters];\n> \n> \n> on top of that would fix the problem you guys are seeing. I don't know\n> if that path leads to insanity, though. Using MSan-enabled libraries is\n> probably a better direction (should increase accuracy, and we don't have\n> to carry these manual annotations around).\n\nHmm, probably insanity. Just for fun I tried to run the whole suite, but\ngot this doozy:\n\n Uninitialized bytes in fopen64 at offset 0 inside [0x7020000109c0, 25)\n ==2568195==WARNING: MemorySanitizer: use-of-uninitialized-value\n #0 0x7f90fe14fa46 in BIO_new_file (/lib/x86_64-linux-gnu/libcrypto.so.3+0x14fa46) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4)\n #1 0x7f90fe1a659c (/lib/x86_64-linux-gnu/libcrypto.so.3+0x1a659c) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4)\n #2 0x7f90fe1a8453 in CONF_modules_load_file_ex (/lib/x86_64-linux-gnu/libcrypto.so.3+0x1a8453) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4)\n #3 0x7f90fe1a8807 (/lib/x86_64-linux-gnu/libcrypto.so.3+0x1a8807) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4)\n #4 0x7f90fe27274e (/lib/x86_64-linux-gnu/libcrypto.so.3+0x27274e) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4)\n #5 0x7f90fea01bc6 in __pthread_once_slow nptl/pthread_once.c:116:7\n #6 0x7f90fea01c38 in __pthread_once nptl/pthread_once.c:143:12\n #7 0x7f90fe287f3c in CRYPTO_THREAD_run_once (/lib/x86_64-linux-gnu/libcrypto.so.3+0x287f3c) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4)\n #8 0x7f90fe272fd9 in OPENSSL_init_crypto (/lib/x86_64-linux-gnu/libcrypto.so.3+0x272fd9) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4)\n #9 0x7f90fe78a6d7 in OPENSSL_init_ssl (/lib/x86_64-linux-gnu/libssl.so.3+0x396d7) (BuildId: a0d77cb273378dec1d74a115ac1c9e40306e675d)\n #10 0x7f90fee2a503 (/lib/x86_64-linux-gnu/libcurl.so.4+0x9e503) (BuildId: 61ee7a8d1799c0e6c38a99b4d739e0c90391a05f)\n #11 0x7f90fedc2e42 (/lib/x86_64-linux-gnu/libcurl.so.4+0x36e42) (BuildId: 61ee7a8d1799c0e6c38a99b4d739e0c90391a05f)\n #12 0x7f90fedc33c9 in curl_global_init (/lib/x86_64-linux-gnu/libcurl.so.4+0x373c9) (BuildId: 61ee7a8d1799c0e6c38a99b4d739e0c90391a05f)\n #13 0x55fbbe211623 in http_init http.c:1347:6\n #14 0x55fbbe1f5b98 in cmd_main remote-curl.c:1583:2\n #15 0x55fbbe244571 in main common-main.c:9:11\n #16 0x7f90fe993ca7 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16\n #17 0x7f90fe993d64 in __libc_start_main csu/../csu/libc-start.c:360:3\n #18 0x55fbbe15bb60 in _start (git-remote-http+0x43b60) (BuildId: dfc63b9261f6d575776d30b4e048b235389a7b20)\n \n SUMMARY: MemorySanitizer: use-of-uninitialized-value (/lib/x86_64-linux-gnu/libcrypto.so.3+0x14fa46) (BuildId: 07a8321bad67632b52b47ad026125c79b7ebaab4) in BIO_new_file\n\n\nSo MSan complaining about stuff deep within curl/openssl, and AFAICT not\nsomething we could influence or annotate as OK.\n\n-Peff\n"},{"id":"522248","messageId":"xmqqwm864fwk.fsf@gitster.g","threadId":"63722","inReplyTo":"CAOLa=ZQPOP0DkK9L5XLmA_uzH7jWwX-ti8AgaEcAF2Mon=w0BA@mail.gmail.com","subject":"Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-17T22:21:47Z","receivedAt":"2025-07-17T22:21:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> This series allows the seek function to set the cursor without setting\n> the prefix, which is a requirement for pagination. So there is no need\n> to set 'prefix_state' for this functionality. Which is why I didn't set\n> it, since the default value of '0' (PREFIX_CONTAINS_DIR) would be the\n> correct setting for all dirs. This causes the issue.\n>\n> So the only fix required would be\n>\n> diff --git a/refs/ref-cache.c b/refs/ref-cache.c\n> index 1d95b56d40..ceef3a2008 100644\n> --- a/refs/ref-cache.c\n> +++ b/refs/ref-cache.c\n> @@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(struct\n> ref_iterator *ref_iterator,\n> \t\t\t\tlevel = &iter->levels[iter->levels_nr++];\n> \t\t\t\tlevel->dir = dir;\n> \t\t\t\tlevel->index = -1;\n> +\t\t\t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n> \t\t\t} else {\n> \t\t\t\t/* reduce the index so the leaf node is iterated over */\n> \t\t\t\tif (cmp <= 0 && !slash)\n\nYup, that is inside the code added by this series. It does look\nlike it fixes the complaint from the checker.\n\nThanks.\n"},{"id":"522359","messageId":"CAOLa=ZR==xihNsZcNkFPtqb1JbC+EbMHtgF-RUzdwOQp55+MOw@mail.gmail.com","threadId":"63722","inReplyTo":"20250717220929.GB2244266@coredump.intra.peff.net","subject":"Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-21T14:27:45Z","receivedAt":"2025-07-21T14:27:47Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Jeff King writes:\n\n> On Thu, Jul 17, 2025 at 12:35:58PM -0700, Kyle Lippincott wrote:\n>\n>> > ==3275333==WARNING: MemorySanitizer: use-of-uninitialized-value\n>> > #0 0x557bd886f4bb in git_mkstemps_mode ../wrapper.c:487:27\n>> > #1 0x557bd886fb55 in git_mkstemp_mode ../wrapper.c:509:9\n>> > #2 0x557bd8100d1a in create_tmpfile ../object-file.c:736:7\n>> > #3 0x557bd80f1630 in start_loose_object_common ../object-file.c:781:7\n>> > #4 0x557bd80f5203 in write_loose_object ../object-file.c:881:7\n>> > #5 0x557bd80f4875 in write_object_file_flags ../object-file.c:1086:6\n>> > #6 0x557bd80f9f65 in write_object_file ../object-file.h:181:9\n>> > #7 0x557bd8101eb8 in index_mem ../object-file.c:1177:9\n>> > #8 0x557bd80f8bd5 in index_core ../object-file.c:1247:10\n>> > #9 0x557bd80f731d in index_fd ../object-file.c:1274:9\n>> > #10 0x557bd80f95e4 in index_path ../object-file.c:1295:7\n>> > #11 0x557bd831132d in add_to_index ../read-cache.c:771:7\n>> > #12 0x557bd8313cb1 in add_file_to_index ../read-cache.c:804:9\n>> > #13 0x557bd73f892c in add_files ../builtin/add.c:355:7\n>> > #14 0x557bd73f4752 in cmd_add ../builtin/add.c:578:18\n>> > #15 0x557bd7a38b6f in run_builtin ../git.c:480:11\n>> > #16 0x557bd7a31d54 in handle_builtin ../git.c:746:9\n>> > #17 0x557bd7a36644 in run_argv ../git.c:813:4\n>> > #18 0x557bd7a30e09 in cmd_main ../git.c:953:19\n>> > #19 0x557bd7a3ca01 in main ../common-main.c:9:11\n>> > #20 0x7f7e3f02a4d7 in __libc_start_call_main\n>> > (/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a4d7)\n>> > (BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a)\n>> > #21 0x7f7e3f02a59a in __libc_start_main@GLIBC_2.2.5\n>> > (/nix/store/g2jzxk3s7cnkhh8yq55l4fbvf639zy37-glibc-2.40-66/lib/libc.so.6+0x2a59a)\n>> > (BuildId: f117ee0f586dfa828cbdd08e37393c8f04f6480a)\n>> > #22 0x557bd7352b34 in _start (git+0x5db34)\n>> >\n>> > Possibly something we need to look into cleaning up.\n>>\n>> I also saw those msan issues when trying `make\n>> CFLAGS=-fsanitize=memory CC=clang`, but not with Google's internal\n>> msan build. I don't know which variable in wrapper.c:487 it's\n>> complaining about - you'd think it'd be `letters`, but if it's `v`,\n>> then that potentially comes from OpenSSL or some other library, and\n>> that library would also need to be built with msan (which is why it's\n>> such a pain to get msan builds working - EVERY library needs to be\n>> built with memory sanitizer).\n>\n> Yeah, presumably it is \"v\" from csprng_bytes(). If there are only a few\n> such spots, we can manually \"unpoison\" memory coming from libraries. On\n> my system, I didn't hit the case shown above but do have trouble with\n> bytes coming back from zlib.\n>\n> Applying this ancient patch:\n>\n> https://lore.kernel.org/git/20171004101932.pai6wzcv2eohsicr@sigill.intra.peff.net/\n>\n> and building with \"make SANITIZE=memory CC=clang\" let me run t6302 to\n> completion, modulo the bug that started this thread (and which I\n> confirmed goes away both with MSan and valgrind with the fix Karthik\n> posted).\n>\n> Probably:\n>\n> diff --git a/wrapper.c b/wrapper.c\n> index 2f00d2ac87..6a4c1c1c29 100644\n> --- a/wrapper.c\n> +++ b/wrapper.c\n> @@ -482,6 +482,8 @@ int git_mkstemps_mode(char *pattern, int suffix_len, int mode)\n> \t\tif (csprng_bytes(&v, sizeof(v), 0) < 0)\n> \t\t\treturn error_errno(\"unable to get random bytes for temporary file\");\n>\n> +\t\tmsan_unpoison(&v, sizeof(v));\n> +\n> \t\t/* Fill in the random bits. */\n> \t\tfor (i = 0; i < num_x; i++) {\n> \t\t\tfilename_template[i] = letters[v % num_letters];\n>\n>\n> on top of that would fix the problem you guys are seeing. I don't know\n> if that path leads to insanity, though. Using MSan-enabled libraries is\n> probably a better direction (should increase accuracy, and we don't have\n> to carry these manual annotations around).\n>\n\nI wonder if an alternate is to use '-fsanitize-ignorelist', since the\nMemorySanitizer is supposed to work with that too [1].\n\n[1]: https://clang.llvm.org/docs/MemorySanitizer.html#ignorelist\n\n> -Peff\n"},{"id":"522374","messageId":"20250721212221.GA818305@coredump.intra.peff.net","threadId":"63722","inReplyTo":"CAOLa=ZR==xihNsZcNkFPtqb1JbC+EbMHtgF-RUzdwOQp55+MOw@mail.gmail.com","subject":"Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-07-21T21:22:21Z","receivedAt":"2025-07-21T21:22:28Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jul 21, 2025 at 02:27:45PM +0000, Karthik Nayak wrote:\n\n> > Applying this ancient patch:\n> >\n> > https://lore.kernel.org/git/20171004101932.pai6wzcv2eohsicr@sigill.intra.peff.net/\n> >\n> > and building with \"make SANITIZE=memory CC=clang\" let me run t6302 to\n> > completion, modulo the bug that started this thread (and which I\n> > confirmed goes away both with MSan and valgrind with the fix Karthik\n> > posted).\n> [...]\n> \n> I wonder if an alternate is to use '-fsanitize-ignorelist', since the\n> MemorySanitizer is supposed to work with that too [1].\n\nI think you could do that, but it isn't quite what we want: it is\nannotating the access of those (false-positive) \"uninitialized\" bytes.\nSo you have to mark every spot that touches bytes that come from zlib,\nwhich in Git is a lot of places. And so the patch linked above was an\nattempt to silence all of those with a single line: marking the bytes\ncoming out of zlib as OK.\n\n-Peff\n"},{"id":"522400","messageId":"CAOLa=ZTPJCZSphi4fV9+NmJt60rMZBxx0YRdpeLE+ab114aF1Q@mail.gmail.com","threadId":"63722","inReplyTo":"xmqqikjq7s16.fsf@gitster.g","subject":"Re: [PATCH v5 5/5] for-each-ref: introduce a '--start-after' option","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-22T08:07:27Z","receivedAt":"2025-07-22T08:07:30Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Karthik Nayak writes:\n>\n>> The `git-for-each-ref(1)` command is used to iterate over references\n>> present in a repository. In large repositories with millions of\n>> references, it would be optimal to paginate this output such that we\n>> can start iteration from a given reference. This would avoid having to\n>> iterate over all references from the beginning each time when paginating\n>> through results.\n>>\n>> The previous commit added 'seek' functionality to the reference\n>> backends. Utilize this and expose a '--start-after' option in\n>> 'git-for-each-ref(1)'. When used, the reference iteration seeks to the\n>> lexicographically next reference and iterates from there onward.\n>>\n>> This enables efficient pagination workflows, where the calling script\n>> can remember the last provided reference and use that as the starting\n>> point for the next set of references:\n>> git for-each-ref --count=100\n>> git for-each-ref --count=100 --start-after=refs/heads/branch-100\n>> git for-each-ref --count=100 --start-after=refs/heads/branch-200\n>>\n>> Since the reference iterators only allow seeking to a specified marker\n>> via the `ref_iterator_seek()`, we introduce a helper function\n>> `start_ref_iterator_after()`, which seeks to next reference by simply\n>> adding (char) 1 to the marker.\n>>\n>> We must note that pagination always continues from the provided marker,\n>> as such any concurrent reference updates lexicographically behind the\n>> marker will not be output. Document the same.\n>>\n>> Signed-off-by: Karthik Nayak \n>> ---\n>> Documentation/git-for-each-ref.adoc | 10 +-\n>> builtin/for-each-ref.c | 8 ++\n>> ref-filter.c | 78 +++++++++++----\n>> ref-filter.h | 1 +\n>> t/t6302-for-each-ref-filter.sh | 194 ++++++++++++++++++++++++++++++++++++\n>> 5 files changed, 272 insertions(+), 19 deletions(-)\n>>\n>> diff --git a/Documentation/git-for-each-ref.adoc b/Documentation/git-for-each-ref.adoc\n>> index 5ef89fc0fe..ae61ba642a 100644\n>> --- a/Documentation/git-for-each-ref.adoc\n>> +++ b/Documentation/git-for-each-ref.adoc\n>> @@ -14,7 +14,7 @@ SYNOPSIS\n>> \t\t [--points-at=]\n>> \t\t [--merged[=]] [--no-merged[=]]\n>> \t\t [--contains[=]] [--no-contains[=]]\n>> -\t\t [--exclude= ...]\n>> +\t\t [--exclude= ...] [--start-after=]\n>\n> Not a problem this patch introduces, but as I noticed it, let me\n> leave a #leftoverbits comment here (it is OK to have a preliminary\n> clean-up patch).\n>\n> * \"--exclude=\" should be enclosed inside a pair of\n> (parentheses), just like the way how [(--sort=)...] is\n> shown.\n>\n> * [--stdin | ...] should be moved to the end. There is no\n> reason to require \"--stdin\" to be the end of dashed options, but\n> the ... must be, as they are positional, not dashed.\n>\n\nI'm sending in a series of these small fixes, so I'll add this in.\n\n>> +--start-after=::\n>> + Allows paginating the output by skipping references up to and including the\n>> + specified marker. When paging, it should be noted that references may be\n>> + deleted, modified or added between invocations. Output will only yield those\n>> + references which follow the marker lexicographically. Output begins from the\n>> + first reference that would come after the marker alphabetically. Cannot be\n>> + used with general pattern matching or custom sort options.\n>\n> It is unclear what \"general\" in \"general pattern matching\" refers\n> to.\n>\n> Cannot be used with `--sort=` or `--stdin` options, or\n> the __ argument(s) to limit the refs.\n>\n\nThis does read much better. I'll also add this.\n\n> or something, perhaps? It is curious how `--exclude=`\n> interacts with the feature. Presumably the exclusion is done so\n> late in the output phase that it does not have any effect? It does\n> not have to be mentioned in this documentation if that is the case\n> as it is a mere implementation detail.\n\nThat is correct indeed, while this doesn't have to be documented, I\nthink we can merit from a test. So I'll add that in.\n\n>\n> Side note. The limitation that sorting and name_patterns cannot\n> be used with the feature also comes from implementation\n> (i.e. the name_patterns optimization will compete with this\n> feature to take advantage of the \"prefix\" thing in an\n> incompatible way), so while the reason does not have to be\n> stated in the end-user facing documentation, the effect needs\n> documenting.\n>\n>> @@ -3189,6 +3221,7 @@ void filter_is_base(struct repository *r,\n>>\n>> static int do_filter_refs(struct ref_filter *filter, unsigned int type, each_ref_fn fn, void *cb_data)\n>> {\n>> +\tconst char *prefix = NULL;\n>> ...\n>> +\n>> +\tif (prefix) {\n>> +\t\tstruct ref_iterator *iter;\n>> +\n>> +\t\titer = refs_ref_iterator_begin(get_main_ref_store(the_repository),\n>> +\t\t\t\t\t \"\", NULL, 0, 0);\n>> +\n>> +\t\tif (filter->start_after)\n>\n> The start_after of the filter comes from \"--start-after=\".\n> Can it be true with non-NULL prefix at this point? Unless you add\n> support for the option to \"git branch/tag\", it would not happen, I\n> guess.\n>\n> More importantly, when you do add support to \"git branch/tag\", the\n> code need to be updated to keep the original prefix while seeking\n> the cursor to the specified , instead of clearing it.\n>\n\nExactly, if we do add '' and '--start-after' compatibility,\nwe'll have to make that change.\n\n>> +\t\t\tret = start_ref_iterator_after(iter, filter->start_after);\n>> +\t\telse if (prefix)\n>> +\t\t\tret = ref_iterator_seek(iter, prefix, 1);\n>\n> We have \"REF_ITERATOR_SEEK_SET_PREFIX\" for that \"1\"?\n\nYup, also the 'if (prefix)' can be dropped too.\n"},{"id":"522415","messageId":"CAOLa=ZTE_Hmnz5EMxAk94rSsvvT7+O9PrroLTbjPW_oe6AzCVw@mail.gmail.com","threadId":"63722","inReplyTo":"20250721212221.GA818305@coredump.intra.peff.net","subject":"Re: [PATCH v5 0/5] for-each-ref: introduce seeking functionality via '--start-after'","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-22T08:44:41Z","receivedAt":"2025-07-22T08:44:43Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Jeff King writes:\n\n> On Mon, Jul 21, 2025 at 02:27:45PM +0000, Karthik Nayak wrote:\n>\n>> > Applying this ancient patch:\n>> >\n>> > https://lore.kernel.org/git/20171004101932.pai6wzcv2eohsicr@sigill.intra.peff.net/\n>> >\n>> > and building with \"make SANITIZE=memory CC=clang\" let me run t6302 to\n>> > completion, modulo the bug that started this thread (and which I\n>> > confirmed goes away both with MSan and valgrind with the fix Karthik\n>> > posted).\n>> [...]\n>>\n>> I wonder if an alternate is to use '-fsanitize-ignorelist', since the\n>> MemorySanitizer is supposed to work with that too [1].\n>\n> I think you could do that, but it isn't quite what we want: it is\n> annotating the access of those (false-positive) \"uninitialized\" bytes.\n> So you have to mark every spot that touches bytes that come from zlib,\n> which in Git is a lot of places. And so the patch linked above was an\n> attempt to silence all of those with a single line: marking the bytes\n> coming out of zlib as OK.\n>\n\nThat makes sense, thanks for explaining.\n\n> -Peff\n"},{"id":"522608","messageId":"xmqqldoelgnd.fsf_-_@gitster.g","threadId":"63722","inReplyTo":"20250715-306-git-for-each-ref-pagination-v5-0-852d5a2f56e1@gmail.com","subject":"[PATCH] ref-iterator-seek: correctly initialize the prefix_state for a new level","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-23T21:51:50Z","receivedAt":"2025-07-23T21:51:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"When cache_ref_iterator_seek() \"jumps\" to a middle of the sorted ref\nlist, it forgets to set the .prefix_state member of the new\n(i.e. deeper) level it just initialized. This later causes\ncache_ref_iterator_advance() to look at this uninitialized member\nto base its decision on what to do next.\n\nKyle Lippincott [*] and Jeff King noticed this with MSAN and\nValgrind, and Karthik Nayak as the original author located exactly\nwhere the missing initialization is.\n\n[*] \n\nSigned-off-by: Junio C Hamano \n---\n refs/ref-cache.c | 1 +\n 1 file changed, 1 insertion(+)\n\n * I had this as \"fixup!\" on top of your topic for quite a while and\n forgot to ask you to send in an official fix. As Kyle's\n discovery was after the topic hit 'next' (understandable, as\n their internal edition of Git is based on 'next'), we need a\n separate fix on top.\n\n To prepare for merging down the whole thing to 'master', I wrote\n the proposed log message to help expedite the process. Comments?\n\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 1d95b56d40..ceef3a2008 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n \t\t\t\tlevel = &iter->levels[iter->levels_nr++];\n \t\t\t\tlevel->dir = dir;\n \t\t\t\tlevel->index = -1;\n+\t\t\t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n \t\t\t} else {\n \t\t\t\t/* reduce the index so the leaf node is iterated over */\n \t\t\t\tif (cmp <= 0 && !slash)\n-- \n2.50.1-521-gf11ee0bd80\n\n"},{"id":"522610","messageId":"CAO_smVjUyOit66LaseHd=0dt6KgnTrCHbrO+oWU1L5Wd1z8Y7A@mail.gmail.com","threadId":"63722","inReplyTo":"xmqqldoelgnd.fsf_-_@gitster.g","subject":"Re: [PATCH] ref-iterator-seek: correctly initialize the prefix_state for a new level","fromName":"Kyle Lippincott","fromEmail":"spectral@google.com","sentAt":"2025-07-23T21:57:29Z","receivedAt":"2025-07-23T21:57:43Z","isPatch":true,"sender":{"key":"spectral@google.com","avatar":"https://avatars.githubusercontent.com/u/6371650?v=4"},"body":"On Wed, Jul 23, 2025 at 2:51 PM Junio C Hamano wrote:\n>\n> When cache_ref_iterator_seek() \"jumps\" to a middle of the sorted ref\n> list, it forgets to set the .prefix_state member of the new\n> (i.e. deeper) level it just initialized. This later causes\n> cache_ref_iterator_advance() to look at this uninitialized member\n> to base its decision on what to do next.\n>\n> Kyle Lippincott [*] and Jeff King noticed this with MSAN and\n> Valgrind, and Karthik Nayak as the original author located exactly\n> where the missing initialization is.\n>\n> [*] \n>\n> Signed-off-by: Junio C Hamano \n\nLooks good to me, thanks!\n\n> ---\n> refs/ref-cache.c | 1 +\n> 1 file changed, 1 insertion(+)\n>\n> * I had this as \"fixup!\" on top of your topic for quite a while and\n> forgot to ask you to send in an official fix. As Kyle's\n> discovery was after the topic hit 'next' (understandable, as\n> their internal edition of Git is based on 'next'), we need a\n> separate fix on top.\n>\n> To prepare for merging down the whole thing to 'master', I wrote\n> the proposed log message to help expedite the process. Comments?\n>\n> diff --git a/refs/ref-cache.c b/refs/ref-cache.c\n> index 1d95b56d40..ceef3a2008 100644\n> --- a/refs/ref-cache.c\n> +++ b/refs/ref-cache.c\n> @@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n> level = &iter->levels[iter->levels_nr++];\n> level->dir = dir;\n> level->index = -1;\n> + level->prefix_state = PREFIX_CONTAINS_DIR;\n> } else {\n> /* reduce the index so the leaf node is iterated over */\n> if (cmp <= 0 && !slash)\n> --\n> 2.50.1-521-gf11ee0bd80\n>\n"},{"id":"522616","messageId":"20250723235234.GA592873@coredump.intra.peff.net","threadId":"63722","inReplyTo":"xmqqldoelgnd.fsf_-_@gitster.g","subject":"Re: [PATCH] ref-iterator-seek: correctly initialize the prefix_state for a new level","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-07-23T23:52:34Z","receivedAt":"2025-07-23T23:52:43Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 23, 2025 at 02:51:50PM -0700, Junio C Hamano wrote:\n\n> When cache_ref_iterator_seek() \"jumps\" to a middle of the sorted ref\n> list, it forgets to set the .prefix_state member of the new\n> (i.e. deeper) level it just initialized. This later causes\n> cache_ref_iterator_advance() to look at this uninitialized member\n> to base its decision on what to do next.\n> \n> Kyle Lippincott [*] and Jeff King noticed this with MSAN and\n> Valgrind, and Karthik Nayak as the original author located exactly\n> where the missing initialization is.\n\nThis explanation makes sense to me (from my admittedly rusty view of the\nref iteration code). And certainly the patch looks right.\n\n-Peff\n"},{"id":"522638","messageId":"CAOLa=ZRr4fpF0E8EM3nL37aSYzcHyCNnv5HzBzisM4C0UQUx8Q@mail.gmail.com","threadId":"63722","inReplyTo":"xmqqldoelgnd.fsf_-_@gitster.g","subject":"Re: [PATCH] ref-iterator-seek: correctly initialize the prefix_state for a new level","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-24T08:12:21Z","receivedAt":"2025-07-24T08:12:24Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> When cache_ref_iterator_seek() \"jumps\" to a middle of the sorted ref\n> list, it forgets to set the .prefix_state member of the new\n> (i.e. deeper) level it just initialized. This later causes\n> cache_ref_iterator_advance() to look at this uninitialized member\n> to base its decision on what to do next.\n>\n\nI think the explanation is correct. For reference I had some more\ndetails in my local patch, but this is totally okay.\n\n ref-cache: set prefix_state when seeking\n\n In 090eb5336c (refs: selectively set prefix in the seek functions,\n 2025-07-15) we separated the seeking functionality of reference\n iterators from the functionality to set prefix to an iterator. This\n allows users of ref iterators to seek to a particular reference to\n provide pagination support.\n\n The files-backend, uses the ref-cache iterator to iterate over loose\n refs. The iterator tracks directories and entries already processed via\n a stack of levels. Each level corresponds to a directory under the files\n backend. New levels are added to the stack, and when all entries from a\n level is yielded, the corresponding level is popped from the stack.\n\n To accommodate seeking, we need to populate and traverse the levels to\n stop the requested seek marker at the appropriate level and its entry\n index. Each level also contains a 'prefix_state' which is used for\n prefix matching, this allows the iterator to skip levels/entries which\n don't match a prefix. The default value of 'prefix_state' is\n PREFIX_CONTAINS_DIR, which yields all entries within a level. When\n purely seeking without prefix matching, we want to yield all entries.\n The commit however, skips setting the value explicitly. This causes the\n MemorySanitizer to issue a 'use-of-uninitialized-value' error when\n running 't/t6302-for-each-ref-filter'.\n\n Set the value explicitly to avoid to fix the issue.\n\n> Kyle Lippincott [*] and Jeff King noticed this with MSAN and\n> Valgrind, and Karthik Nayak as the original author located exactly\n> where the missing initialization is.\n>\n> [*] \n>\n> Signed-off-by: Junio C Hamano \n> ---\n> refs/ref-cache.c | 1 +\n> 1 file changed, 1 insertion(+)\n>\n> * I had this as \"fixup!\" on top of your topic for quite a while and\n> forgot to ask you to send in an official fix. As Kyle's\n> discovery was after the topic hit 'next' (understandable, as\n> their internal edition of Git is based on 'next'), we need a\n> separate fix on top.\n>\n> To prepare for merging down the whole thing to 'master', I wrote\n> the proposed log message to help expedite the process. Comments?\n>\n\nI had a set of patches locally, I just didn't get around to sending it.\nWill send the others, omitting this. Thanks for doing it!\n\n> diff --git a/refs/ref-cache.c b/refs/ref-cache.c\n> index 1d95b56d40..ceef3a2008 100644\n> --- a/refs/ref-cache.c\n> +++ b/refs/ref-cache.c\n> @@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n> \t\t\t\tlevel = &iter->levels[iter->levels_nr++];\n> \t\t\t\tlevel->dir = dir;\n> \t\t\t\tlevel->index = -1;\n> +\t\t\t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n> \t\t\t} else {\n> \t\t\t\t/* reduce the index so the leaf node is iterated over */\n> \t\t\t\tif (cmp <= 0 && !slash)\n> --\n> 2.50.1-521-gf11ee0bd80\n\nThe patch looks good.\n"},{"id":"522691","messageId":"xmqqbjp9ikvk.fsf@gitster.g","threadId":"63722","inReplyTo":"CAOLa=ZRr4fpF0E8EM3nL37aSYzcHyCNnv5HzBzisM4C0UQUx8Q@mail.gmail.com","subject":"Re: [PATCH] ref-iterator-seek: correctly initialize the prefix_state for a new level","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-24T17:01:03Z","receivedAt":"2025-07-24T17:01:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n>> * I had this as \"fixup!\" on top of your topic for quite a while and\n>> forgot to ask you to send in an official fix. As Kyle's\n>> discovery was after the topic hit 'next' (understandable, as\n>> their internal edition of Git is based on 'next'), we need a\n>> separate fix on top.\n>>\n>> To prepare for merging down the whole thing to 'master', I wrote\n>> the proposed log message to help expedite the process. Comments?\n>>\n>\n> I had a set of patches locally, I just didn't get around to sending it.\n> Will send the others, omitting this. Thanks for doing it!\n\nI do not mind discarding what I sent out at all. I actually prefer\nif it came from you.\n\nThanks.\n"},{"id":"522706","messageId":"20250724221136.693120-1-karthik.188@gmail.com","threadId":"63722","inReplyTo":"xmqqbjp9ikvk.fsf@gitster.g","subject":"[PATCH] ref-cache: set prefix_state when seeking","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-24T22:11:36Z","receivedAt":"2025-07-24T22:11:50Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"In 090eb5336c (refs: selectively set prefix in the seek functions,\n2025-07-15) we separated the seeking functionality of reference\niterators from the functionality to set prefix to an iterator. This\nallows users of ref iterators to seek to a particular reference to\nprovide pagination support.\n\nThe files-backend, uses the ref-cache iterator to iterate over loose\nrefs. The iterator tracks directories and entries already processed via\na stack of levels. Each level corresponds to a directory under the files\nbackend. New levels are added to the stack, and when all entries from a\nlevel is yielded, the corresponding level is popped from the stack.\n\nTo accommodate seeking, we need to populate and traverse the levels to\nstop the requested seek marker at the appropriate level and its entry\nindex. Each level also contains a 'prefix_state' which is used for\nprefix matching, this allows the iterator to skip levels/entries which\ndon't match a prefix. The default value of 'prefix_state' is\nPREFIX_CONTAINS_DIR, which yields all entries within a level. When\npurely seeking without prefix matching, we want to yield all entries.\nThe commit however, skips setting the value explicitly. This causes the\nMemorySanitizer to issue a 'use-of-uninitialized-value' error when\nrunning 't/t6302-for-each-ref-filter'.\n\nSet the value explicitly to avoid to fix the issue.\n\nReported-by: Kyle Lippincott \nHelped-by: Kyle Lippincott \nHelped-by: Jeff King \nSigned-off-by: Karthik Nayak \n---\n\nHere is my version of the same patch!\n\n refs/ref-cache.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 1d95b56d40..ceef3a2008 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n \t\t\t\tlevel = &iter->levels[iter->levels_nr++];\n \t\t\t\tlevel->dir = dir;\n \t\t\t\tlevel->index = -1;\n+\t\t\t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n \t\t\t} else {\n \t\t\t\t/* reduce the index so the leaf node is iterated over */\n \t\t\t\tif (cmp <= 0 && !slash)\n-- \n2.49.0\n\n"},{"id":"522708","messageId":"xmqqseilxlvi.fsf@gitster.g","threadId":"63722","inReplyTo":"20250724221136.693120-1-karthik.188@gmail.com","subject":"Re: [PATCH] ref-cache: set prefix_state when seeking","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-24T22:30:25Z","receivedAt":"2025-07-24T22:30:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> In 090eb5336c (refs: selectively set prefix in the seek functions,\n> 2025-07-15) we separated the seeking functionality of reference\n> iterators from the functionality to set prefix to an iterator. This\n> allows users of ref iterators to seek to a particular reference to\n> provide pagination support.\n>\n> The files-backend, uses the ref-cache iterator to iterate over loose\n> refs. The iterator tracks directories and entries already processed via\n> a stack of levels. Each level corresponds to a directory under the files\n> backend. New levels are added to the stack, and when all entries from a\n> level is yielded, the corresponding level is popped from the stack.\n>\n> To accommodate seeking, we need to populate and traverse the levels to\n> stop the requested seek marker at the appropriate level and its entry\n> index. Each level also contains a 'prefix_state' which is used for\n> prefix matching, this allows the iterator to skip levels/entries which\n> don't match a prefix. The default value of 'prefix_state' is\n> PREFIX_CONTAINS_DIR, which yields all entries within a level. When\n> purely seeking without prefix matching, we want to yield all entries.\n> The commit however, skips setting the value explicitly. This causes the\n> MemorySanitizer to issue a 'use-of-uninitialized-value' error when\n> running 't/t6302-for-each-ref-filter'.\n>\n> Set the value explicitly to avoid to fix the issue.\n>\n> Reported-by: Kyle Lippincott \n> Helped-by: Kyle Lippincott \n> Helped-by: Jeff King \n> Signed-off-by: Karthik Nayak \n> ---\n>\n> Here is my version of the same patch!\n\nThanks!\n\n>\n> refs/ref-cache.c | 1 +\n> 1 file changed, 1 insertion(+)\n>\n> diff --git a/refs/ref-cache.c b/refs/ref-cache.c\n> index 1d95b56d40..ceef3a2008 100644\n> --- a/refs/ref-cache.c\n> +++ b/refs/ref-cache.c\n> @@ -527,6 +527,7 @@ static int cache_ref_iterator_seek(struct ref_iterator *ref_iterator,\n> \t\t\t\tlevel = &iter->levels[iter->levels_nr++];\n> \t\t\t\tlevel->dir = dir;\n> \t\t\t\tlevel->index = -1;\n> +\t\t\t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n> \t\t\t} else {\n> \t\t\t\t/* reduce the index so the leaf node is iterated over */\n> \t\t\t\tif (cmp <= 0 && !slash)\n"}]}